Polished work is not finished work.
Agents inside 12 real drug and device companies.

In drug and medical device development, evidence rarely comes in a neat, pre-packaged folder. It is scattered across regulatory submissions, quality deviation logs, clinical trial protocols, agency correspondence, and informal working drafts.

Those records frequently contradict one another, become superseded over time, or carry completely different legal and regulatory weight. An informal team memo cannot override an approved Common Technical Document (CTD) specification, and an exploratory Phase 1 protocol does not govern commercial release testing. Figuring out which document actually carries authority is the real job.

When a benchmark tests an agent on a clean prompt or a hand-picked evidence packet, it bypasses the hardest part of professional work. In the real world, typing the prose is easy. The challenge is investigating conflicting records and deciding what the company can defensibly do next.

The benchmark roster contains 71 assignments across 12 distinct company environments, built around authentic product histories and private biopharma records. The roster includes genuine development milestones—such as the complex dual-IND history of Allena Pharmaceuticals. Alongside these stands Havenor Therapeutics, a completely synthetic showcase company created so researchers can inspect full trajectories, coworker interactions, and deliverables in public without exposing confidential partner files.

These environments span major health authorities (US FDA CDER, CBER, CDRH; the European Medicines Agency; Health Canada; Korea MFDS; and China NMPA / CDE), diverse modalities (mRNA vaccines, antibody-drug conjugates, targeted small molecules, peptide injectables, oral solid fixed-dose combinations, and IVD companion diagnostics), and essential company disciplines across regulatory affairs, quality compliance, MSAT, and clinical site operations.

Rather than answering an abstract question, the agent sits down at an employee's desk at a specific moment in the company's history. It receives the actual files, tools, and background noise that a person in that seat would have had. Its job is to investigate the company state and produce the required work product: an updated regulatory assessment, an out-of-specification investigation, or a sign-off package. In Havenor Therapeutics, agents can also query simulated systems of record, pull lab data, and message coworkers while the work is underway.

You can explore curated run trajectories and actual deliverables from Havenor on the live benchmark page.

Figure 1 ranks all eight systems by macro task mean—the primary metric, where every assignment carries equal weight. We also show Pass@1 on every row: the percentage of assignments where an agent satisfied every single frozen criterion in its single scored trial. You can sort the leaderboard by macro mean, complete passes, or criteria passed.

When an agent falls short on these tasks, it is almost never because the writing looks unpolished. The prose is usually fluent, confident, and neatly formatted. The failure almost always comes down to authority: which document actually governs?

You can see this clearly in Havenor Therapeutics Task 03, where a principal engineer is asked to review sterile-filter validation studies and draft a technical brief ahead of a health authority audit.

Every benchmark attempt takes place inside an isolated container. The agent is assigned an employee seat with access to the local company files, relevant tools, and a standard set of instructions: explore the company records, make reasonable inferences when encountering ambiguity, and continue working unless genuinely blocked.

We evaluate complete model-and-harness pairings, not bare models in a vacuum. On the 71-task roster, Astra and Sol ran on Codex at medium thinking, Opus on Claude Code at medium thinking, Grok and Gemini on Cursor CLI at high thinking, DeepSeek and GLM on Pi at high thinking, and Kimi on a blend of Cursor CLI and Pi at the provider default. DeepSeek has no medium setting, and the Grok and Gemini seats are the high model slugs. The harness governs how an agent inspects files, runs tools, and manages context, so results reflect the combined system.

Once an agent finishes, its deliverable is scored against frozen criteria that were quarantined away during the run. When a check is strictly quantitative—such as a specific calculated yield or a key spreadsheet value—we evaluate it with deterministic code.

Most professional work, however, cannot be captured by a regex. Two senior regulatory writers might structure a justification memo differently while both making sound, defensible arguments. The scores on this page are a hand grade of each deliverable against those frozen criteria. A check passes only when the submitted work itself supports it.

A brief word on what these scores mean: strong performance shows that an agent can navigate complex internal records, reconcile conflicting evidence, and draft defensible technical work. It does not certify regulatory compliance, and it does not mean an agent is ready to operate without accountable human oversight.