Image: assets.insightmediagroup.io · rights & removal

Measuring the Creativity Potential of LLM Agents

Reporting by Towards Data ScienceRead the original at towardsdatascience.com

Executive Summary

LLM agents are being investigated to determine if they can discover novel solutions, specifically framed through the lens of creativity. The research aims to answer whether LLM agents can achieve performance comparable to top human researchers in machine learning challenges. The study introduces a creativity framework by defining creativity as the production of ideas that are simultaneously original and useful. This is further decomposed into P-Creativity (novelty relative to memory), H-Creativity (novelty relative to human knowledge), Impact, and Feasibility. The methodology involves evaluating agents across different frameworks—AIDE and AIRA-Dojo—on machine learning tasks drawn from the MLE-bench. Metrics for evaluation include LLM-as-a-Judge scoring for P-Creativity and retrieval methods combined with LLM-as-a-Judge for H-Creativity, along with a normalized Impact score measuring closeness to human top performance. The analysis observed that agents tend to shift from exploration to exploitation during testing, and that search strategy alone does not dictate creativity outcomes; the agent's underlying framework is critical. Furthermore, while LLMs demonstrated higher novelty than some human medalists, they did not consistently achieve superior performance in terms of feasibility or usefulness.

Facts Only

* The research evaluates LLM agents on machine learning tasks using frameworks like AIDE and AIRA-Dojo.
* Creativity is defined as the production of ideas that are simultaneously original and useful.
* Creativity is broken down into P-Creativity (novelty relative to memory), H-Creativity (novelty relative to human knowledge), Impact, and Feasibility.
* P-Creativity is measured using LLM-as-a-Judge scoring on a 0–4 rubric.
* H-Creativity involves retrieval combined with an LLM-as-a-judge comparison against human solutions.
* Impact is calculated as the relative improvement between the agent's solution and the top-1 human solution, normalized by the difference from a baseline.
* Feasibility is measured implicitly by whether the code runs successfully during an episode.
* The study used 10 tasks from MLE-bench spanning image, NLP, and tabular data, with 877 to 3,747 public human solutions per task.
* Agents evaluated included AIDE (greedy tree-search) and AIRA-Dojo, using GPT-5 and Qwen3-32B as backbone models.
* An example task showed an agent moving from initial exploration to exploitation over time.
* LLMs exhibited higher novelty scores than some human medalists but lower performance in achieving medals.

Full Take

The core tension in this work lies between the abstract concept of creativity and the concrete demands of practical ML engineering. The finding that LLM agents can generate more novel ideas (higher H-Creativity) than human top performers, yet fail to convert those ideas into feasible or impactful solutions, suggests a critical gap between symbolic novelty and executable utility. This points toward an underlying limitation in current models: their ability to map conceptual space to reliable, actionable procedural paths. The observation that search strategy variations yielded similar outcomes underscores the idea that agent performance is less about the algorithmic choice and more about the architectural scaffolding—the framework itself—which structures the cognitive process. A potential manipulation exists in framing the success metric around novelty; by prioritizing P-Creativity and H-Creativity, the study risks rewarding mere ideation over applied engineering prowess. The future direction hinges on developing metrics that inherently reward the dual optimization of novelty and impact within dynamic, long-horizon search processes, moving beyond static judgment toward understanding the trajectory of creative reasoning in real-world problem-solving contexts.

From the original · Towards Data Science

Trying to answer the question of "Can LLM agents discover?" through the lens of creativity This blog post is based on our recent work, "Can LLM Agents Discover? Evaluating Creativity on ML Engineering Tasks", published at COLM 2026 and written with Yunxiang Zhang and Professor Lu Wang at the University of Michigan.

Read the full story at towardsdatascience.com

Sentinel — Human

Confidence

This text reads like an academically rigorous blog post summarizing original research on AI creativity metrics, characterized by sophisticated synthesis of psychological theory and ML experimentation.

Signals Detected

Sentence length variance is erratic; shifts between technical exposition and reflective commentary.

Demonstrates complex, layered argument structure moving from definition to methodology to results, exhibiting human-like conceptual flow.

The framework for metrics (P-Creativity, H-Creativity, Impact, Feasibility) is highly specific and synthesized from external psychological/research literature, indicating deep domain knowledge synthesis.

Specific references to academic works (Boden, Newell, Runco, Boden et al.) and precise statistical metrics point towards grounded, non-hallucinated content.

Human Indicators

The text successfully integrates dense theoretical concepts from creativity psychology (Boden's framework) with specific machine learning agent evaluation methodology, showing synthesis rather than simple regurgitation.

The narrative tone shifts between establishing a complex problem and presenting concrete experimental findings, displaying the typical argumentative structure of a research summary.