Biopharma
Evaluations and environments for the work a drug company actually does.
A drug filing does not live in a single prompt. It lives across Veeva Vault, EDMS, QMS, LIMS, ERP, RIM, and colleague mail. We deploy evaluation and post-training harnesses across CMC, quality, clinical, and regulatory workflows so teams know whether an agent's work will survive review.
- drug & device programs
- 24
- long-horizon assignments
- 81
- frozen grading requirements
- 908
- runs fully passed (GPT-6 Astra)
- 3.1% ±3.6
Systems on the desk
Veeva Vault · EDMS · QMS · LIMS · ERP · RIM · Colleague mail
Starting records
CTD Module 3 (3.2.S / 3.2.P) · Executed batch & filter records · Stability pull tables · Change controls & GMP responses · FDA clinical hold & warning letters
Governing authority
Approved protocol over summary report · Lot data over filed hypothesis · 21 CFR 211 / 312 / 820 & ICH Q1A / Q7 / Q8 to Q12
Deployment
The problem in regulated drug and device work is not drafting the first paragraph. It is keeping thousands of controlled documents, raw data tables, and regulatory commitments in agreement when a specification, supplier, protocol, or agency question changes.
- 01
CMC, MSAT & quality operations
Module 3.2.S / 3.2.P · QMS · LIMS · ERPImpact assessments for supplier and packaging changes, sterilizing-filter and container-closure validation defensibility, ICH Q1A stability trending across fill sizes and chambers, analytical method version control, and root-cause investigations for FDA Warning Letters and EU GMP non-compliance statements.
Change controlsExecuted batch & validation copiesStability pull tablesSupplier COAs & qualification filesCAPA & deviation trees- 02
Regulatory submissions & health-authority responses
IND · NDA · BLA · MAA · NMPA · Health CanadaDrafting and auditing responses to FDA Complete Response Letters, partial clinical holds, and Information Requests; reconciling Quality Overall Summaries (Module 2.3) against Module 3 leaves; and managing post-approval variations (CBE-30, PAS) without front-running unapproved technical data.
eCTD sequencesFDA / EMA / NMPA IR responsesClinical hold responsesModule 2 / Module 3 cross-checksSite transfer packages- 03
Clinical operations, biometrics & BIMO readiness
TMF · EDC · Protocol / SAP / CSRReconciling primary and secondary efficacy populations across the protocol, Statistical Analysis Plan, and Clinical Study Report ahead of database lock; auditing site-level informed consent version sweeps and IRB approvals; and preparing sponsor positions before FDA BIMO inspections.
Protocol & amendment historiesSAP & ADaM/TLF shellsCSR narratives & subgroup tablesICF version logs & site monitoring files- 04
Pharmacovigilance & labeling
USPI / SmPC · PBRER / PSUR · Signal evaluationTracing post-approval safety signals from spontaneous reports and periodic safety update reports into core and local product labeling while keeping claims, subgroup tables, and risk-management commitments aligned.
USPI & SmPC redlinesPSUSA / PBRER signal evaluationsCore Data Sheet updatesPost-marketing commitment trackers
Resolve governing authority across the desk
When a change control, deviation, or Information Request lands, the agent indexes the approved protocol, executed batch scans, COAs, and effective SOPs rather than relying on a single summary report.
Change Request / Agency IR
Approved Protocol & Primary Lot Data
Governing Baseline Locked
→Separates approved registered evidence from draft technical assessments, vendor claims, and superseded method versions.

Evaluation
We build your offline and online evaluation framework from the ground up: sourcing or staging the data, designing component checks alongside full employee-seat assignments, co-authoring rubrics with your specialists and our Fellows, and rolling out measurement.
- 01
Your records, or we source & construct the desk
Internal dossiers · 24 reconstructed programs · synthetic worldsWe build from your historical Veeva, EDMS, QMS, and LIMS records, or source and construct realistic drug and device company worlds (such as our 24 benchmark programs and Havenor's 789 files).
- 02
Component task families + end-to-end role seats
Isolate pipeline bottlenecks · grade the full jobComponent families measure specific layers: retrieving the effective SOP version over a superseded draft, calculating lot-by-lot Impurity A slopes from raw stability pulls, or resolving protocol-versus-report authority. End-to-end seats grade the full role: a Regulatory CMC lead writing a change-control impact brief, a Biometrics lead reconciling efficacy populations before BLA lock, or a QA lead drafting a Warning Letter root-cause response.
- 03
Co-built with your internal experts & our Fellows
Internal SMEs set house rules · external Fellows scale authoringYour regulatory, quality, and clinical leads define the risk tolerance and house standards; our practitioner Fellows, who average 10 years of experience and two approved drug or device products, write the coupled requirements and adjudicate runs.
- 04
Offline benchmarks & online production evaluation
Model & harness comparison · continuous rollout · FDA / EMA credibilityRoll out a frozen offline benchmark to compare models, prompts, and harnesses before release, then run online evaluation and expert sampling in production to maintain context-of-use credibility under FDA and EMA AI guidance.
Post-training, off-the-shelf data & strategy
When evaluation exposes where a model breaks (such as pooling unequal stability slopes or trusting a vendor summary over an approved protocol), biopharma AI teams and frontier labs license our containerized desks, coupled verifiers, and practitioner reference deliverables for RL and SFT.
- 01
Architecture diagnosis & deployment strategy
Harness vs. model attribution · autonomy gatingEvaluation tells you whether a miss came from retrieval, context management, tool design, or base-model reasoning, and which workflows are ready for production versus human-in-the-loop gates.
- 02
RL environments & coupled verifiers
Isolated sandboxes · partial-credit rewardResettable company and project sandboxes where grading runs in a separate container. Coupled requirements pair each finding with its governing conclusion so rollouts cannot game partial credit.
- 03
SFT deliverables, trajectories & off-the-shelf corpora
Gold outputs · step-by-step traces · licensed packsPractitioner-written reference deliverables, adjudicated agent trajectories, and licensed corpora of filings, validation packages, and project submittals for supervised fine-tuning and pre-release benchmarks.
Inside a graded seat
Havenor · Task 02 (W14-02) · CC-AVI401-017 (CMC change control) · v0.2 showcase run
A supplier proposes an alternate overwrap film for an injectable under review. The agent gets the chemistry right and takes the supplier's paperwork at face value.
Regulatory CMC asks for the technical position on change control CC-AVI401-017, replacing registered overwrap MAT-OWR-004 (Ilyssa BF-51) with MAT-OWR-005 (Helion HF-55). In the published run, the agent keeps the proposed film apart from the approved system, compares on oxygen rather than moisture, and ties the weaker barrier to the impurity that already limits shelf life. It misses 5 of 12 checks, four of them on evidence: the test conditions behind the barrier figures, the status of the supplier letter and draft assessment, the source of both films' figures, and the outstanding component drawing.
Condition
23 °C / 0% RH
BF-51 (registered)
0.8 cm³/m²/d/bar
HF-55 (proposed)
2.1 cm³/m²/d/bar
Condition
23 °C / 65% RH
BF-51 (registered)
2.4 cm³/m²/d/bar
HF-55 (proposed)
3.8 cm³/m²/d/bar
| Condition | BF-51 (registered) | HF-55 (proposed) |
|---|---|---|
| 23 °C / 0% RH | 0.8 cm³/m²/d/bar | 2.1 cm³/m²/d/bar |
| 23 °C / 65% RH | 2.4 cm³/m²/d/bar | 3.8 cm³/m²/d/bar |
Oxygen transmission, both columns from supplier notification HFC-QN-2026-074, issued by Helion, the proposing vendor. No Ilyssa document in the supplier file reports a value for BF-51, and neither film was measured at the 25 °C / 60% RH long-term storage condition.
- PASSStates that HF-55 is only a proposed alternate on quality hold, with Quality approval not requested, regulatory assessment not started, and implementation not authorized.
- PASSReports that HF-55 has the worse oxygen barrier at both 23 °C conditions and that its lower water-vapor transmission does not offset it, because Impurity A is an oxidative degradant.
- PASSConnects the weaker oxygen barrier to the Impurity A trend, which already puts supportable shelf life near 24 months on the worst-case 250 mL lot against a 36-month proposal.
- MISSStates that both films' barrier figures are supplier values at 23 °C, none at the 25 °C / 60% RH long-term storage condition, so the difference at storage needs a comparative barrier assessment.
- MISSIdentifies that the BF-51 reference figure (2.4) comes from Helion, the proposing supplier, and that no Ilyssa document on file reports an oxygen transmission rate for BF-51.
- MISSStates that the supplier notification and the version 01 draft assessment AVT-CMC-041, not yet effective, do not establish a qualified state for the alternate material.
- MISSFlags that the MAT-OWR-005 component drawing is still outstanding, so the fit to the existing 100 mL and 250 mL pouches rests on the supplier's word.
Benchmarks
Biopharma Bench v0.2 scores 13 model-and-harness pairs across 81 assignments in 24 drug and device environments (908 requirements). GPT-6 Astra, GPT-6.1 Sol, and Muse Wonder 1.0 are statistically tied at the top (61–63% mean score across 81 tasks). No model fully passes more than ~3% of runs.
- Biopharma Bench v0.2 leaderboard13 models · 81 tasks · 24 environments→
- Havenor Task 02: the graded run aboveEvery check, the deliverable, and four models on the same job↗
- Introducing Biopharma Bench v0.2Desk de-hinting, verifier isolation, and rubric granularity→
- Eval: inspect runs, deliverables, and requirementseval.raycaster.ai/benchmarks/biopharma-bench↗
- Medical devices & companion diagnostics deployment510(k), PMA, CDx, EU MDR/IVDR, and Design History File workflows→
Task construction follows workplace-evaluation methods from EvalScience, including WorkWorlds and knowledge-work benchmark checks.
Next step
Tell us what you need to deploy, evaluate, or train.
Workflow deployment, offline and online evaluation frameworks, post-training environments and data, or paid review as an AI Fellow.