The huge language model has a restricted range of application at the moment, but it can perform anything from reading the literature to producing and evaluating its own articles.
Is it possible for science to be completely mechanized? Researchers studying machine learning have now given it a go.
An “AI Scientist” is a person who conducts the whole research cycle, from reviewing the current literature on a subject and creating hypotheses for new advances to trying out solutions and producing a paper. The person was part of a team at the Tokyo startup Sakana AI as well as at academic labs in Canada and the United Kingdom. Even more, AI Scientist assesses its own output and performs some of the duties of peer reviewers.
The creation of AI beings that have at least partially automated the scientific method is being pursued by AI Scientists. Cong Lu, a machine-learning researcher at the University of British Columbia in Vancouver, Canada, and co-creator of AI Scientist, states, “To my knowledge, no one has yet done the entire scientific community, all in one system.” This month, the results1 were made available on the arXiv preprint service.
According to Jevin West, a computational social scientist at the University of Washington in Seattle, “it’s impressive that they’ve done this end-to-end.” Furthermore, I believe that these concepts should be explored further since they may advance science.
Thus far, the results are not particularly remarkable, and the system is limited to doing research within the domain of machine learning. Specifically, AI Scientist lacks the proficiency in laboratory work that most scientists would consider essential to doing science. Materials scientist Gerbrand Ceder of Lawrence Berkeley National Laboratory and the University of California, Berkeley, states that “there’s still a lot of work to go from AI that makes a hypothesis to implementing that in a robot scientist.” Nevertheless, Ceder continues, “I have no doubt in my mind that this is where much of science will go if you look into the future.”
Automated experiments
The foundation of AI Scientist is a large language model (LLM). Taking as a model a publication that outlines a machine learning algorithm, it first looks through the literature for related research. Inspired by Darwinian evolution’s natural selection and mutations, the team used a method known as evolutionary computation. It works in stages, making tiny, haphazard adjustments to an algorithm and choosing the ones that increase performance.
To achieve this, AI Scientist runs the algorithms and evaluates their performance, essentially doing its own “experiments.” Ultimately, it generates a document and assesses it using a process akin to automated peer review. Once the literature has been “augmented,” the algorithm can continue the cycle by expanding on its own findings.
The authors acknowledge that the research published in the articles by AI Scientists was only a step forward. On social media, a few other scholars left harsh comments. If I were a journal editor, I would probably desk-reject them. “If I were a reviewer, I would turn them down,” a user said on the Hacker News website.
West adds that the writers had a reductionist perspective on how scientists discover the status of their subject at the moment. Many of the things they know originate from other channels of communication, including attending conferences or striking up conversations with coworkers at the water cooler. According to West, “science is more than a pile of papers.” “A 5-hour study of the literature won’t be as beneficial as a 5-minute conversation.”
Shahan Memon, West’s colleague, concurs, yet both men commend the writers for making their code and findings completely transparent. They are now able to examine the AI Scientist’s findings thanks to this. For instance, they have discovered that it exhibits a “popularity bias” when it comes to the selection of past publications it includes as references, leaning more toward those with a large number of citations. Memon and West state that they are investigating whether or not the decisions made by AI scientists were the most important ones.
Repetitive tasks
Naturally, AI Scientist is not the first attempt to automate at least some aspects of a researcher’s work; according to Tom Hope, a computer scientist at the Allen Institute for AI in Jerusalem, the idea of automating scientific discovery dates back to the 1950s, when artificial intelligence first emerged. For instance, the Automatic Statistician2 was already capable of data analysis and paper writing ten years ago. Ceder and his associates have also automated certain bench tasks: the “robot chemist” they introduced the previous year is capable of creating novel materials and conducting experiments with them3.
As per Hope, the existing LLMs “cannot formulate new and practical scientific directions beyond basic superficial combinations of buzzwords.” However, Ceder asserts that while AI won’t be able to perform more creative tasks anytime soon, it will be able to automate many of the more tedious parts of research. “At a basic level, you’re attempting to analyze the nature of something and its behavior. Though it accounts for 90% of our work, that is hardly the creative side of science. Lu claims that many other researchers also gave him similar comments. “I have a hundred ideas, but I don’t have time for them,” people will remark. Assign such tasks to the AI scientist.
According to Lu, AI Scientist may require more methods in addition to language models in order to expand its capabilities, even into abstract domains outside of machine learning, such pure mathematics. Google Deep Mind’s recent accomplishments in arithmetic problem solving, for instance, demonstrate the effectiveness of combining LLMs with “symbolic” AI approaches, which include logical principles into a system instead of only depending on it to learn from statistical patterns in data. However, he notes that this version is just the beginning. He calls this the “GPT-1 of AI science,” alluding to an early big language model developed by San Francisco, California-based OpenAI.
According to West, the findings contribute to a discussion that is now consuming the minds of many scholars. “Everyone I work with in the sciences is attempting to determine how artificial intelligence fits into our work. It does compel us to consider what, if anything, science is and is not in the twenty-first century, he claims.
