Computer Science > Logic in Computer Science
[Submitted on 20 Aug 2026]
Title:ProofJudge: Tool-Grounded LLM Evaluation of Formal Proof Quality in Mathlib
View PDF HTML (experimental)Abstract:Formal proofs in Lean 4 that pass the kernel's type checker can nonetheless vary widely in quality. We introduce ProofJudge, an agentic LLM-as-judge system that scores formal proof quality along five dimensions beyond correctness: library leverage, automation fit, structural clarity, statement quality, and Mathlib conventions. We evaluate ProofJudge on a novel dataset of 218 declarations drawn from distinct Mathlib PRs. The judge agent is grounded by tool access to the commit the PR is applied to, enabling it to query the library state when scoring. A judge is considered aligned with human preferences when it rates the version of the PR Mathlib accepted above the initial version that was sent back for revision. All six judge models evaluated recover the reviewers' preference well above chance, from 80.8% to 63.5%, and two open-weight judges reach roughly 70% at a tenth of the best judge's cost. We release the judge harness, evaluation dataset, and evaluation traces as open-source artifacts to support further research.
Current browse context:
cs.LO
References & Citations
Loading...
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Connected Papers (What is Connected Papers?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
alphaXiv (What is alphaXiv?)
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Hugging Face (What is Huggingface?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
CORE Recommender (What is CORE?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Facts Only
* ProofJudge is an agentic LLM-as-judge system.
* The system scores formal proof quality in Lean 4.
* Scoring dimensions include library leverage, automation fit, structural clarity, statement quality, and Mathlib conventions.
* The evaluation dataset consists of 218 declarations from Mathlib pull requests (PRs).
* The judge agent has tool access to the commit to which a PR is applied.
* Six judge models were evaluated.
* Alignment is measured by the judge's ability to prefer the accepted version of a PR over the initial version.
* Preference recovery rates ranged from 63.5% to 80.8%.
* Two open-weight models achieved approximately 70% alignment.
* The judge harness, evaluation dataset, and evaluation traces are released as open-source artifacts.
* The submission date is 20 August 2026.
Executive Summary
ProofJudge addresses the gap between formal correctness and qualitative excellence in Lean 4 mathematical proofs. While the Lean kernel ensures a proof is logically sound, it cannot determine if the proof is elegant, maintainable, or compliant with community standards. To solve this, ProofJudge utilizes an agentic LLM framework that evaluates proofs across five qualitative dimensions: library leverage, automation fit, structural clarity, statement quality, and Mathlib conventions.
The system is grounded by providing the LLM with tool access to the library state, allowing it to query the environment during the scoring process. Testing against a dataset of 218 Mathlib declarations shows that the system can replicate human reviewer preferences with 63.5% to 80.8% accuracy. Notably, smaller open-weight models can achieve near 70% alignment at significantly lower costs than top-tier proprietary models. This suggests a viable path toward automating the preliminary review of formal mathematics, though the variance in model performance indicates that a "gold standard" for automated qualitative judgment remains elusive.
Full Take
The methodology relies on a "preference recovery" metric—comparing the initial PR submission to the final accepted version. This is a sound proxy for human quality standards, as it mirrors the actual iterative process of peer review in the Mathlib community. However, a peer reviewer would flag the sample size (n=218) as relatively small for generalizing across the entirety of formal mathematics. Furthermore, the "alignment" percentages (63.5%–80.8%) indicate a significant delta between the best models and chance, yet leave a substantial margin where the AI diverges from human expert intuition.
This work extends the "LLM-as-a-Judge" paradigm from general text or code to the highly constrained domain of formal verification. The framing of novelty is justified because it introduces tool-grounding—the ability to query the library state—which prevents the LLM from hallucinating the existence of lemmas or failing to recognize available library shortcuts.
If these findings hold, the real-world implication is a shift in the labor economy of formalization. Human reviewers could transition from "correcting conventions" to "guiding architecture," using ProofJudge as a first-pass filter. To strengthen the claim, a follow-up study should test the judge on "negative" samples—proofs that are logically correct but intentionally obfuscated or inefficient—to see if the system can detect "correct but bad" code without the hint of a human-accepted version.
Bridge Questions: To what extent does the judge rely on the stylistic fingerprints of the authors rather than the structural quality of the proof? If the judge is used to train future proof-generators, does this create a feedback loop that reinforces specific stylistic biases over mathematical elegance?