CausalForge Kills LLM Reviewers: Lean Proofs Fix AI Research
CausalForge uses Lean proof verification to automate causal inference research, eliminating the unreliability of LLM reviewers. This framework changes what is possible in AI-driven theoretical discovery.
The Bad Scientist paper (2025) showed that LLM reviewers accept fabricated papers and detect fakes at rates close to chance. Now CausalForge bypasses that problem entirely by grounding evaluation in Lean proof verification, creating a self-improving research loop that does not rely on the very models it is trying to improve.
- CausalForge replaces LLM reviewers with Lean proof verification, solving the hallucination problem in automated research evaluation.
- The framework generates candidate causal inference theorems, verifies them formally, and uses verified results to improve its own generation.
- This closes the research loop without relying on the same model type that produced the candidate results, breaking the circular validation trap.
Why Are LLM Reviewers Unreliable for Automated Research?
According to the Bad Scientist study published in 2025, LLM reviewers accept fabricated scientific papers at rates indistinguishable from chance, and detect real fake papers only slightly better than random guessing. This undermines the entire premise of closing the research loop with an LLM reviewer, because the reviewer cannot reliably distinguish valid results from plausible-sounding nonsense. The CausalForge authors explicitly cite this finding as the motivation for their alternative approach: formal proof verification in Lean.
How Does Lean Proof Verification Fix the Evaluation Problem?
CausalForge generates candidate theorems and proofs in causal inference, then submits them to the Lean proof assistant for formal verification. Lean checks each step of the proof against its foundational logic, accepting only fully verified results. This eliminates the need for a probabilistic LLM reviewer entirely. The authors report that this approach guarantees that every accepted result is mathematically sound, provided the Lean kernel is correct. This is a fundamentally different reliability guarantee than any LLM-based review can offer.
What Does the Self-Improving Loop Actually Look Like?
The framework operates in cycles: generate candidate results, verify them in Lean, and use the verified results as training data for the next generation of candidate generation. The authors describe this as a self-improving feedback loop where each cycle produces a growing body of formally verified causal inference knowledge. Unlike typical LLM fine-tuning on human-generated data, CausalForge's training data is guaranteed correct, which should prevent the model from learning incorrect reasoning patterns.
What Are the Limitations of This Approach?
The framework is limited to domains that can be formalized in Lean. Causal inference has a rich mathematical foundation that maps well to Lean, but many scientific domains do not. The authors acknowledge that the initial theorem library is small and that generating complex proofs remains computationally expensive. Additionally, Lean verification can only check correctness, not importance—a trivial but true theorem will pass verification just as easily as a groundbreaking one. The framework does not yet have a mechanism for ranking results by significance.
How Does CausalForge Compare to Existing Automated Research Systems?
| Feature | CausalForge | LLM-Reviewer Systems |
|---|---|---|
| Evaluation method | Lean proof verification | LLM judge |
| Hallucination risk in evaluation | Near zero (formal verification) | High (chance-level detection) |
| Domain generality | Formalizable domains only | Any text domain |
| Self-improvement | Verified results as training data | Unreliable feedback loop |
| Computational cost | High (proof checking) | Moderate (LLM inference) |
| Verdict | Winner for formal domains | Useful for exploration only |
My thesis is that CausalForge represents the first credible path to automated theoretical research in formal domains. The key insight is that by decoupling generation from evaluation—using a different, formally grounded system for verification—the framework avoids the circular validation trap that plagues LLM-only approaches. In the short term, this will accelerate causal inference research, particularly for theorems that can be expressed in existing Lean libraries. In the long term, if the framework scales, it could fundamentally change how mathematical and theoretical research is conducted, reducing the role of human intuition in generating new theorems. The losers here are any AI research automation systems that rely on LLM reviewers as their sole evaluation mechanism—they are now provably unreliable. The winners are researchers in causal inference who can use CausalForge to generate and verify theorems faster than any human team. I predict that within 18 months, at least one major causal inference journal will accept a paper whose core theorem was discovered by CausalForge.
- By Q4 2027, at least one top-tier causal inference conference (e.g., UAI or NeurIPS) will publish a paper whose main theoretical result was generated and verified by CausalForge.
- By mid-2028, at least two major AI labs (e.g., DeepMind or OpenAI) will adopt formal verification-based research agents for at least one subfield of their work, citing CausalForge as the proof of concept.
- By 2029, the Bad Scientist finding will be cited in every paper proposing an LLM-only automated research system as a limitation, making formal verification a de facto requirement for credible automated discovery claims.
- 2025Bad Scientist paper published
Demonstrated that LLM reviewers detect fabricated papers at rates close to chance.
- July 2026CausalForge preprint released
Authors propose Lean proof verification as alternative to LLM reviewers for automated research.
Estimated verification reliability comparison (2025-2026)
- CausalForge solves the verification problem that made LLM-only research agents unreliable, using Lean proof checking instead of another LLM as judge.
- The framework is limited to formalizable domains, but within those domains it offers a correctness guarantee that no LLM-based system can match.
- The self-improving loop uses only verified results as training data, preventing the model from learning incorrect reasoning patterns.
- Future work must address the ranking problem—how to identify which verified theorems are actually important, not just correct.
- The Bad Scientist (2025) paper is now a canonical citation for the failure of LLM-based review, and CausalForge directly responds to that failure.
Discussion
Add a comment