Introducing
Biopharma Bench v0.2

After our first release, domain experts with an average of 10 years of experience and two approved drug or device products helped us review whether our grading requirements reflected sound professional judgment and identify additional tasks with practical economic value. Several pharmaceutical companies have also reached out about building an open benchmark together; we will share more on that soon.

Their feedback helped shape the expanded coverage in v0.2. Where v0.1 leaned on clinical site operations, v0.2 adds cell and gene therapy manufacturing, responses to GMP inspections, pharmacovigilance, market access and patent work, and medical-device registration.

Each environment is rebuilt from a real program’s records, such as FDA warning letters, approval packages and site files, and frozen at a date. The agent sees what the employee had then, and nothing that was written later.

To explore an environment yourself, our synthetic showcase company, Havenor Therapeutics, has an open desk, step-by-step agent runs and finished deliverables. It is scored separately from the private benchmark.

Reading v0.1 runs, we saw agents treat the desk as a puzzle to solve rather than a job to do, and some of the cues were ours. In one clinical data reconciliation workbook, every row holding a discrepancy was marked “Ready for lock”. No clinical data manager flags individual discrepant rows that way; it pointed straight at the error. Now only the submission package as a whole is marked ready for database lock, as a real handoff would be. An informed-consent draft carried the parenthetical “(use the study contact block)” right beside the outdated investigator phone number we wanted the agent to catch. We removed the parenthetical so the discrepancy sits in the document without a signpost.

The prompts had the same problem. Early versions listed the failure modes the grader would check and dictated file paths like a test specification. Prompts now read like an actual handoff from a Regulatory Affairs, CMC, or Clinical Operations lead: drafting the response to a health authority’s Information Request, reconciling primary efficacy populations ahead of a BLA freeze, or setting the sponsor’s position across clinical trial sites before an FDA BIMO inspection. The agent works directly from the controlled folders, regulatory correspondence, and study records on the employee’s desk and records any assumptions in the deliverable.

The files gave the setup away too. Office documents still carried the stamps of the Python libraries that generated them, and file timestamps showed when we built the environment. We stripped the stamps and reset every timestamp to the environment’s date.

We also observed evaluation awareness: agents recognizing the evaluation setup and searching for grading instructions or previous transcripts. In one run, a verifier-related installation file prompted the agent to identify the setup as an evaluation and start hunting for grading material.

To keep the assessment focused on the work, we removed exposed construction clues and separated the agent’s workspace from the verifier. The agent produces its deliverable in one sandbox; grading happens in another, keeping grading instructions and reward outputs outside the working environment.

We also restricted external access so agents would work from the supplied records rather than search the web. Built-in web search and fetch tools were disabled, and outbound connections were limited to an allowlist of required services, such as model API endpoints. These restrictions constrain external access without making the environment fully offline.

We shared these observations publicly on X, tagging the Harbor team. Harbor subsequently published a reward-hacking update describing further protections, including safer artifact transfers, removal of automatic verifier-log mounts, and neutral runtime names, alongside tools for checking tasks and reviewing completed runs.

In one environment, the agent drafts the root-cause section of a manufacturer’s response to an FDA warning letter about a potentially carcinogenic impurity found in a drug substance. The manufacturer’s own investigation blamed contamination in a process reagent. Our earlier requirement treated that explanation as the answer, so an agent that repeated it got credit.

The lot records on the same desk contradict it. A batch made with reagent whose suspected contaminant was below the detection limit still contained substantial impurity. The requirement now credits reporting the explanation as the manufacturer’s filed hypothesis, not as an established cause. A separate requirement asks the agent to conclude from the lot data that contamination in that reagent does not explain the impurity found in the drug substance.

We also revised how requirements divide up the work. Some separate items rewarded the same finding more than once; others gave credit for identifying a fact without drawing the conclusion it supported. We combined overlapping checks and kept dependent parts of a judgment together, so quoting the right number with the wrong conclusion does not earn credit. Independent findings are scored separately.

A benchmark’s name does not establish what its scores measure. Desai and colleagues found that shared scoring formats can influence model rankings alongside the capabilities a test claims to assess. In a review of 445 LLM benchmarks, Bean and colleagues likewise identified weaknesses in how tasks and metrics supported capability claims. For professional work, that means checking both whether an assignment reflects the job and whether its grading requirements capture the judgments that make the work correct.

We adapted selected construction checks from Terminal-Bench to document-based work: checking that required deliverables are identified, that the starting records support the expected work, and that failures across models are examined for task or grading defects before being interpreted as capability gaps. Later filings and regulatory reviews help establish the reference standard, while the employee’s starting records determine what could have been known at the assignment’s date.

The Agentic Benchmark Checklist distinguishes task validity (whether an assignment tests the intended capability) from outcome validity (whether a passing evaluation corresponds to successful completion). Our expert reviews examine both. Reading actual deliverables against the source records exposed requirements that rewarded unsupported claims and others that rejected sound answers by demanding evidence the employee never had.

One correction addressed a requirement that asked for evidence the employee did not have. In an injectable-drug filing with China’s drug regulator, the agent answers a reviewer’s question about fill volume. Our requirement asked for extractable-volume results compared against the reference product. No such result exists on the desk: the available reference-product testing addressed a different question. Agents that correctly said the study was still outstanding were failing. The requirement now credits a justified fill range relative to the labeled volume and a statement that the comparative study is still required, and it fails any claim that a comparison result exists.

Time causes the same problem. In a label assignment, the subgroup counts in a response table do not add up to the reported total. An explanation for the discrepancy appears only in a working outline written after the assignment’s date. Requiring it would have meant requiring the agent to know the future. The revised requirement credits flagging the discrepancy as unresolved and calling for the missing patient-level information. The later explanation is acceptable only as a hypothesis pending confirmation.

We are also expanding the documentation that links each requirement to its supporting source passages, so readers can check these judgments themselves.

Nine models have results on all 81 assignments. GPT-6 Astra leads at 59.5%, followed by Claude Fable 5.1 at 58.9% and GPT-6.1 Sol at 58.8%.

GPT-6 Astra averaged 70.4% on v0.1. v0.2 has different assignments and stricter requirements, so the two numbers are not directly comparable.

Mean scores show how much of the work agents complete; full passes show how often they finish an assignment against every requirement in its professional grading standard. For v0.2, expert review helped us revise those requirements to better distinguish a complete, supported deliverable from one that still needs substantive correction. Full passes are rare: Astra, Fable, and Opus 5.5 each passed 1 of 81 assignments, showing that completing the work consistently remains beyond the evaluated models. The lower full-pass rates reflect performance across a broader set of assignments assessed against a revised, expert-reviewed professional standard.

GPT-6 Luna judges each requirement twice. A requirement scores 1 when both judgments say it is met, 0.5 when they disagree, and 0 otherwise. An assignment’s score is the share of its requirements met, and the mean task score weights every assignment equally.

We score each model together with its agent software, or harness, because the harness shapes how it searches files, uses tools and manages its working context. The setting under each model name shows which harness and reasoning effort we used.

Browse the published benchmark and public showcases, or open the Havenor walkthrough to follow an agent from the records it read to the brief it produced. We will update the results as more runs finish.

If you are a domain expert interested in reviewing criteria or helping build the next open benchmark, apply to the AI Fellows program. For pharmaceutical and biotech teams interested in collaborating on open benchmarks or custom evaluations, reach out through our access page or at team@raycaster.ai.