VANCOUVER, CANADA—Artificial intelligence (AI) tools designed to execute end-to-end projects, from coming up with hypotheses to running and writing up experiments, are increasingly popular with researchers—and increasingly skilled. But a new study shows these tools can stealthily violate norms of research integrity.
Computer scientist Nihar Shah of Carnegie Mellon University and colleagues looked at two high-profile tools—Agent Laboratory and the AI Scientist v2—both developed recently to help computer scientists perform experiments within the field of machine learning. The AI Scientist made headlines earlier this year by being the first AI system to have an original research paper accepted by peer review.
But in a presentation at the World Conferences on Research Integrity here today, Shah reported that both systems engaged in acts that aren’t acceptable in research, including making up data and “p-hacking”: running an experiment multiple times but only reporting the best outcome. (The team’s results were previously posted as a preprint on arXiv.) The misbehaviors weren’t obvious and required a lot of sleuthing to track down, suggesting AI-assisted studies might fall victim to such problems without their authors’ knowledge.
“Their core findings are worth taking seriously,” says Samuel Schmidgall, a computer scientist who co-developed the Agent Laboratory while at Johns Hopkins University. Work like Shah’s is important, he says, to hammer home to researchers the ways in which AI can lead things astray. AI scientists already come with plenty of disclaimers stressing that human oversight is essential at every stage, he adds.
AI Scientist co-developer Jeff Clune, a computer scientist at the University of British Columbia, agrees. “We are not advocating that people simply use these systems to produce science and publish the outputs as is.” He says the AI Scientist adds a digital watermark to papers it produces to show their provenance.
To test the two agentic systems, Shah devised a simple experiment involving a hidden rule used to categorize data. First, Shah and colleagues came up with a batch of data sets, each consisting of thousands of strings of colored symbols such as red triangles or blue circles. Each data set was divided into two groups, depending on whether the strings conformed to a certain rule. Some rules were simple, such as “there are exactly three triangles”; others were more complicated. The rules were kept secret from the AIs. The team also introduced some “noise” by occasionally miscategorizing a string to make the task more challenging.
Shah then asked the AI agentic systems to develop algorithms to predict the categorizations. Each AI system completed the task, building their algorithms using a “training” portion of each data set and scoring themselves using the “test” data. They then wrote up a full research paper to describe their work and report how well their algorithms worked.
Worryingly, both systems sometimes invented data sets, for example if the data had somehow gotten lost as the different parts of the AI agent completed their tasks and passed them on to the next. “That was completely unexpected for us; we were extremely surprised,” Shah says.
Making matters worse, the AI was not up-front about it. The team only discovered this behavior when it devised a test to see whether the AIs, which sometimes reported impossibly good performances, were cheating by sneaking a look at the test data during training. Inspecting the trace code—the very long, full record of what the AIs did—revealed that, instead, the AIs were on occasion making up data. In the trace code, the AIs provided excuses such as saying they invented data to enable faster training. “This is really worrying,” Shah says.
Both systems were also guilty of using methods analogous to p-hacking. “You keep slicing and dicing data until you get a significant result,” Shah says. And both failed to choose representative data sets for their investigations. In one part of the experiment, Shah asked the AIs to analyze just four of 20 data sets, under the pretense that this would save on computing time. The AI Scientist v2 showed a preference for “easy” ones, with shorter data strings and simpler hidden rules; the Agent Laboratory tended to simply pick the four that were at the top of the list.
The good news was that neither system was guilty of cherry-picking the best metric to rate its own performance.
“It’s really interesting that these systems are inclined towards these practices,” says Renee Hoch, head of publication ethics for the nonprofit publisher Public Library of Science, who is attending the conference. Most publishers already require that the use of AI in research papers be disclosed, but the findings suggest that may not be sufficient. “It would be difficult to predict what guardrails we need, as there’s a bajillion ways it could go wrong,” she says.
Ironically, AI may be part of the solution for detecting hidden violations. The team could not tell by the final research papers alone what problems lay under the hood. But when a large language model read through both the final papers and the full trace code, which was too long for a human reviewer to reasonably digest, it detected research integrity issues with about 80% accuracy. Shah argues journals should ask for the full trace code when reviewing papers produced using AI agents. Schmidgall agrees. “The paper-only review paradigm is insufficient for AI-generated work,” he says.
The results also mean the output of AI agent systems should be handled with an abundance of caution, Shah says. “I’m really optimistic about the use of AI in science—I think it has a lot of potential.” But, he adds, “Right now, we need researchers who use this to be much more careful.”