The Illusion of the Correct Answer.
Why High-Stakes Work Demands Document Engineering.

The most rigorous public evaluations of "agentic" professional work have evolved far beyond isolated prompts. They now manifest as structured worlds—labyrinths of files, applications, and tools governed by automated graders. It is less a trivia night and more a synthetic stint on the job.

The recent APEX-Agents benchmark (Vidgen et al.) is the most visible example in this class today. Consider the scale it demands of an agent:

• ~166 files per environment on average
• 9 apps and 63 tools to navigate and execute the work
• ~1.82 hours of estimated human labor required to complete a single task

This is hard in a way that mirrors real work: any one task usually needs a small subset of files, which means most of the environment is plausible-but-irrelevant context. The agent has to locate the right source, in the right version, then use it correctly.

The benchmark was also stress-tested during creation with frontier agents, so weak prompt-first strategies were exposed early.

Public scores are still low: top systems remain around 30% on Pass@1 and around 40% on Pass@8.

So the benchmark is already asking for navigation, retrieval, tool use, and an auditable final answer. That is a big upgrade from chat-only tests. It still is not the full burden of production: a trajectory can pass while producing a document no one should sign if review, traceability, and change control are treated as afterthoughts.

The real bottleneck is the system around the model. We are building the equivalent of Git, CI/CD, and code review for messy, real-world documents.

While model intelligence remains a prerequisite, once a competence floor is cleared, the chasm between a flashy demo and a deliverable that legal or compliance will actually sign off on is bridged by system design: how evidence is meticulously retrieved, how edits are bounded, and how seamlessly a human can verify or roll back changes.

To be precise: domains that are bound by raw reasoning or pure mathematical calculation will naturally benefit most from ongoing improvements at the model layer. But in document-heavy domains, outcomes are dominated by the messiness of the real world. Here, large swings come from system choices like:

• retrieval routing policy
• multimodal parsing quality
• deterministic workflow design
• version-controlled edit management
• reviewer-facing change control

Put plainly: changing architecture can move benchmark outcomes more than swapping models within the same class—and it is the same lever that decides whether outputs are fit for a real review queue.

That cuts two ways. Economically, runtime design that lets smaller models reach frontier-adjacent scores shifts the price-performance curve instead of defaulting to the largest checkpoint. For governance, raw capability is never sufficient: in regulated or high-stakes work, deliverables have to stay verifiable, reviewable, and reversible under explicit standards, or the work does not ship.

Software engineering pioneered this trajectory. As coding evaluation matured from toy completions to complex, issue-level workflows, three lessons became impossible to ignore:

1. benchmark construction determines signal quality
2. harness design can materially change outcomes for the same base model
3. verification loops (test/review/retry) are part of capability, not implementation detail

In coding, the stack evolved from raw generation to harnessed workflows: retrieval, test execution, code review, and merge gates. Treat those as inspiration, not metaphor only—document systems need the same progression: retrieval plus evidence checks, standards and checklists, reviewer-visible diffs, and approval gates before anything becomes the record.

Document agents inherit all three lessons and add harder surface area: scans, dense tables, diagrams, handwriting, and long revision histories across files that were never designed for code-style tracking.

Anthropic’s recent research on harness design for long-running applications drives this home: they found that even frontier models fail at long-horizon tasks, drifting out of scope or hallucinatively approving their own bad work, unless constrained by a strict architecture. Their solution was to separate the agent doing the work from an independent "evaluator" agent forced to grade against explicit criteria. The model's raw capability was unlocked only by the orchestration built around it.

A similar parallel exists for context and retrieval. Cursor's write-up on semantic search in the agent harness: on their offline Cursor Context Bench retrieval eval, the same models with semantic search in the toolset answer questions about codebases more accurately—about 12.5% on average, with a roughly 6.5%–23.5% spread by model. Online A/B tests on real agent traffic show the same idea in product metrics (e.g. code retention and fewer dissatisfied follow-ups when search is on; smaller effect sizes because not every task needs retrieval). They emphasize that grep plus semantic search together beats either alone—exactly the split we rely on in document workspaces (Improving agent with semantic search).

The transfer to document work is straightforward but harder: retrieval must handle prose, tables, scans, figures, and version drift across many file formats. That means benchmark uplifts are informative, but likely still a lower bound on the architecture effects you see in real enterprise work.

Against that backdrop—realistic worlds, but still grader-centric—we asked a narrower empirical question: if you keep the tasks and models fixed and swap in a document-native runtime (retrieval routing, workspace navigation, and change-aware tooling), how much moves?

Across consulting, investment banking, and law with four models, our runtime improved baseline on 20/24 model-domain-metric cells (Pass@1 + Mean).

For GPT 5.4 mini and GPT 5.4 nano, improvements were 12/12 cells, with average deltas of +4.60 Pass@1 and +7.48 Mean.

Where lift appears—and where it doesn't

The shape is instructive.

Consulting and law improve most, where retrieval quality, cross-document grounding, and disciplined edit workflows dominate. Investment banking is flatter. This is largely because the IB domain is overwhelmingly focused on financial modeling and analysis—tasks bounded by post-training capabilities, numerical precision, and strict calculation chains inside Excel.

That is entirely consistent with the story so far: better architecture lifts retrieval and grounding reliability where those dominate the task. However, for workflows that are purely analytical or mathematical, performance will scale naturally on the model layer as foundation models improve their reasoning capabilities. But for document-heavy work—the messy reality of synthesizing contracts, reconciling scattered PDF evidence, and auditing compliance rules—model improvements alone will never be enough.

There is a collateral benefit hidden in these numbers. We originally built our document engine for the grueling realities of biopharma and energy. We did not write custom optimizations for APEX's legal or management consulting tasks. Yet, the system performed materially better in those new domains simply because the fundamental physics of document work—finding the right page, anchoring on the exact clause, managing version drift—are universal. When you solve for the document, the domain follows.

The bar chart below summarizes lift across every model we ran in one glance. The image at the top of the article keeps the nano Pass@1 and list-pricing comparison against the APEX GPT-5.2 (High) reference (at approximately 11x lower cost) so we do not repeat that same figure in the body.

That nano result is the clearest proof of the claim. When a sub-tier model (GPT 5.4 nano) in a document-native harness matches the bare baseline of a frontier model (GPT 5.2 High), it breaks the assumption that complex knowledge work is just a function of scale.

Model capabilities will inevitably continue to rise, but raw intelligence alone cannot dexterously untangle the messy realities of document-heavy work: broken file references, scanned PDFs, handwritten notes in clinical trial batch records, or legal exhibits with missing pages. A massive model will hallucinate if it reads the wrong version of a contract; a tiny model will succeed if the architecture reliably hands it the exact governing clause. That is why trust is always built around the models, not just by them.

Yet, aggregate averages often obscure the underlying mechanics. Take one representative APEX-style task in management consulting: the prompt asks for a market cost-efficiency analysis using a defined methodology (Cost Structure Score) for a fictional brand (PureLife) across several international markets.

Completing it means threading a dense file tree of statements, market-sizing spreadsheets, and competitor benchmarks. The agent leans on semantic vector search and grep to locate the methodology and exact columns, then writes Python to load the master sheet, clean fields, and compute weighted scores that match the grader.

A successful trajectory like the one above is a tightly routed pipeline:

1. Semantic retrieval (Vector Search) to surface README files and presentations that explain the methodology and point to the correct master spreadsheet.
2. Exact search (Grep) to scan across dozens of Excel files and text documents to verify column names and variables like Import_Duty_% and Distribution_Cost_%.
3. Environment Navigation & Data Processing writing Python code to load the master P&L, clean the data, apply inverse min-max scaling, and calculate the weighted scores.
4. Deterministic computation recovering from missing files and syntax errors to arrive at the exact final values.
5. Reviewer-visible output displaying the exact formatted answer requested by the user.

Miss any step and the task fails, even if the prose sounds plausible. This is the hidden workload inside a single Pass@1 success bit: not one model call, but a routed pipeline. With the right architecture and context engineering, even small models can run it end to end.

Crucially, because the harness forces agents to show their work via explicit diffs and evidence links rather than just outputting a final string, it inherently doubles as a high-signal environment for evaluation and Reinforcement Learning (RL) on real-world tasks.

A grader's verdict is binary; the reality of production is anything but. A benchmark might mark a run as successful while inadvertently leaving you with silent file deletions, unreviewed redlines, or claims that feign authority but lack a tether to a versioned source.

This becomes exponentially more critical as we anticipate the immediate future: the rise of cloud agents. Soon, organizations will not just have one model answering a prompt sequentially; they will have dozens of autonomous agents working in parallel across the same enterprise knowledge base. When multiple agents and multiple humans, each with their own organizational permissions and context, are collaborating on a live filing, raw intelligence is not enough to resolve merge conflicts or share context safely. The HCI (Human-Computer Interaction) implications of this are staggering. If we do not have rigorous control surfaces—redlines, rollback, deterministic review—parallel agent work will devolve into chaos.

Our thesis is that reliable document agents require document-native infrastructure analogous to what Git enabled for software. Deployment quality is decided by these control surfaces, not by the score alone:

• Verifiable (The Audit Trail): a clear, step-by-step record showing exactly how the agent arrived at a change (which files it opened, what it searched for, and how it calculated the numbers). A legal claim or regulatory submission must cite the exact clause, table cell, and document version, not just a plausible summary.
• Reviewable (Tracked Changes): a proposed set of edits presented clearly, much like redlines or suggested edits in Word. Reviewers need to see exactly what was added or removed before approving.
• Reversible: if the system edits the wrong version or applies a bad change, teams need one-step rollback with full history preserved.
• Standards-driven (Automated Checkers): built-in rules that enforce domain standards (like a playbook, policy, or formatting guideline) consistently across every run.

This is not hypothetical. In their failure analysis, the APEX-Agents authors noted that "deletions are never requested in the task prompts"—yet "GPT-5.2 [deleted] the most files (n=21)" (APEX-Agents paper). This is exactly why review and rollback are first-class requirements, not UX polish. We must design for human-as-reviewer by default, treating it as the core operating model rather than a post-hoc safety layer.

That is the same handoff coding already institutionalized: tests and CI for correctness, pull requests for review, merges for consent. Document workflows need the analogue—explicit evidence, reviewer-facing diffs, and publish gates—whether or not the benchmark run happened to pass. Evaluation protocols must grade not only the final outcome, but the entire process: the proposed edits plus the audit trail of how they were made.

None of these requirements negate the need for headline task accuracy; rather, they constrain what that accuracy is allowed to mean in a professional context. Pass@1 (or Pass@k) should sit beside a small set of companion checks, much like unit tests sit beside integration tests in engineering. For document-heavy workflows, the minimum useful surface is:

1. Task Completion@1 (the benchmark bit you already report)
2. Consistency@k (rerun stability under fixed seeds and inputs)
3. Edit Integrity (unrelated sections and files unchanged)
4. Grounding Coverage (material claims traceable to sources and versions)
5. Approval@1 (first-pass human accept rate on real queues)
6. Severity-weighted Error Cost (not all mistakes weigh the same)
7. Cost and latency per successful run (economics of the full pipeline)

A single score optimizes for demos and leaderboard screenshots. The bundle optimizes for work you can defend in front of counsel or a quality unit—and it is the bundle coding orgs already expect before they call a release safe.

• APEX-Agents paper (Vidgen et al., 2026)
• APEX-Agents leaderboard
• OfficeQA Pro: An Enterprise Benchmark for End-to-End Grounded Reasoning
• Introducing OfficeQA (Databricks)
• Cursor: Improving agent with semantic search
• Anthropic: Harness design for long-running application development
• Git book: About Version Control