Biopharma Bench
AI agents evaluated on regulatory, CMC, quality, and clinical tasks built from real drug and device filings.
- expert tasks
- 81
- real drug & device products
- 24
- models evaluated
- 9
- tasks fully correct, best model
- 1/81
Leaderboard
| Rank | Model | Agent | Mean score | Full passes |
|---|---|---|---|---|
| 1 | GPT-6 Astra(high) | Codex | 59.5% | 1/81 |
| 2 | Claude Fable 5.1(high) | Claude Code | 58.9% | 1/81 |
| 3 | GPT-6.1 Sol(high) | Codex | 58.8% | 0/81 |
| 4 | Grok 4.7(high) | Cursor CLI | 54.0% | 0/81 |
| 5 | Grok 4.6(high) | Cursor CLI | 53.4% | 0/81 |
| 6 | DeepSeek V4.1 Flash(high) | Pi | 51.0% | 0/81 |
| 7 | Claude Opus 5.5(high) | Claude Code | 50.2% | 1/81 |
| 8 | GLM-5.3 Flash(high) | Pi | 42.6% | 0/81 |
| 9 | GPT-6.1 Terra(high) | Codex | 37.1% | 0/81 |
About the benchmark
Task coverage
81 tasks across regulatory filings, CMC, quality/MSAT, and clinical operations, spanning development through marketed products. Tasks are mapped in O*NET-style work-activity terms and model work that typically runs through CDMOs, CROs, and regulatory affairs teams on two-to-three-year program timelines.
Real environments
Built from 24 real drug and device programs. Each task desk holds hundreds of real documents and datasets across Veeva Vault, EDMS, QMS, LIMS, ERP, RIM, and workplace chat, dated to when the job started.
Rubrics from actual outcomes
Tasks and criteria are generated from what the person in that seat actually did next: the real next edit, filing, or agency response. Every item is checked by practitioners with 10+ years of filing experience in that jurisdiction (Lonza, Catalent, Pfizer, AstraZeneca, and other sponsors and CDMOs).
Public environments
Fictional sterile-injectables company · public
Agents take staff jobs: review lab results, assess suppliers, answer a regulator.
789 files · 120 colleagues on chat · 10 tasks · 40 scored runs
Real FDA partial clinical hold, rebuilt from the public record
Agents draft the hold response from the filed dossier, FDA's letter, and raw lab data.
294 files · Nothing synthetic · 4 tasks · 7 scored runs
Contribute as an expert
Review criteria and author rubrics through the AI Fellows program.
AI Fellows programBenchmark and data collaboration
Build a benchmark in your domain, or license tasks and environments from this one.