Biopharma Bench:
Can Frontier Agents Do Real Professional Work?

Imagine if an AI agent had to go back in time, sit in the regulatory and CMC seats of an innovative drug company, navigate years of conflicting internal records and agency requests, and bring a breakthrough therapy—like a first-in-class GLP-1 receptor agonist or an antibody-drug conjugate—through rounds of regulatory approval. Could it autonomously do the job without making a fatal mistake?

Most existing agent benchmarks evaluate models in simplified, synthetic conditions: a short prompt, a pre-extracted text snippet, and a narrow set of expected answers. While useful for measuring basic reasoning or coding syntax, these setups miss the core friction of professional knowledge work in regulated industry. What makes a benchmark valid? We anchor Biopharma Bench on three uncompromising pillars:

When a benchmark tests an agent on a curated prompt, it tests transcription and summarization. What makes an evaluation valid is whether an agent can uncover the facts, resolve contradictory data, and defend its conclusions against real statutory and operational standards.

The benchmark roster contains 71 assignments across 12 distinct company environments, built from authentic development milestones and private biopharma records. These environments span major health authorities (US FDA CDER, CBER, CDRH; the European Medicines Agency; Health Canada; Korea MFDS; and China NMPA / CDE), diverse modalities (mRNA vaccines, antibody-drug conjugates, targeted small molecules, peptide injectables, oral solid fixed-dose combinations, and IVD companion diagnostics), and essential company functions across regulatory affairs, quality compliance, MSAT, and clinical site operations.

Rather than answering an abstract question or parsing an isolated snippet, the agent sits down at an employee's desk at a specific historical moment in the company's journey. It receives the actual files, historical revisions, and background noise that a person in that seat would have had. Its job is to investigate the records and produce the required work product: an updated regulatory assessment, an out-of-specification investigation, or an audit sign-off package.

Alongside these private environments stand two open public showcases evaluated separately from the sealed 71-task benchmark: Havenor Therapeutics, a completely synthetic showcase company created so researchers can openly inspect full trajectories, coworker interactions, and deliverables in public without exposing confidential partner files, and the Broad Institute divalent siRNA program, rebuilt verbatim from Eric Minikel's public FDA IND submission. Havenor's records end in September 2026. Both are published in full and scored separately from the sealed 71-task benchmark.

You can explore curated run trajectories and actual deliverables from these showcases on the Biopharma Bench results page. The public Havenor desks and rubrics are on Hugging Face. To run the tasks with records, LIMS, and the verifier, use Harbor.

Figure 1 reports the headline leaderboard across all twelve frontier systems. Rank is determined by macro task mean—the primary metric, where every assignment carries equal weight. We also report Pass@1 (share of assignments where every single frozen substantive criterion passed on the first trial), absolute criteria passed, mean token cost, ATIF steps, tool calls, and execution time. You can sort the leaderboard by clicking any column header.

When an agent falls short on these tasks, it is almost never because the writing looks unpolished. The prose is usually fluent, confident, and neatly formatted. The failure almost always comes down to authority: which document actually governs?

You can see this failure mode clearly in our published showcase environment, Havenor Therapeutics (Task 03), where Claude Opus 5 (running in Claude Code) was assigned as a principal engineer to review sterile-filter validation studies and draft a technical brief ahead of a health authority audit.

Every benchmark attempt takes place inside an isolated container. The agent is assigned an employee seat with access to the local company files, relevant tools, and a standard set of instructions: explore the company records, make reasonable inferences when encountering ambiguity, and continue working unless genuinely blocked.

We evaluate complete model-and-harness pairings, not bare models in a vacuum. On the 71-task roster, Astra and GPT-6 Sol ran on Codex at high thinking, GPT-5.6 Sol on Codex at medium thinking, and Opus 5, Opus 5.5, and Fable 5.1 on Claude Code at high thinking. Grok 4.6, Grok 4.7, and Gemini ran on Cursor CLI at high thinking, DeepSeek and GLM on Pi at high thinking, and Kimi on Pi at max thinking. DeepSeek has no medium setting, and the Grok 4.6, Grok 4.7, and Gemini seats are the high model slugs. The harness governs how an agent inspects files, runs tools, and manages context, so results reflect the combined system.

Once an agent finishes, its deliverable is scored against frozen criteria that were quarantined away during the run. When a check is strictly quantitative—such as a specific calculated yield or a key spreadsheet value—we evaluate it with deterministic code.

Most professional work, however, cannot be captured by a regex. Two senior regulatory writers might structure a justification memo differently while both making sound, defensible arguments. Deliverables are scored using an automated rubric grader powered by GPT-5.6 Luna (medium thinking) executed within the Harbor evaluation harness. The grader evaluates each substantive criterion and aspect check against the submitted work alongside the underlying source documents and frozen truth specifications. A check passes only when the submitted work itself demonstrably supports it.

A brief word on what these scores mean: strong performance shows that an agent can navigate complex internal records, reconcile conflicting evidence, and draft defensible technical work. It does not certify regulatory compliance, and it does not mean an agent is ready to operate without accountable human oversight.