Status: v3.3.5 — hub doc for benchmark disclosure schema
Schema: shared/benchmark_report.schema.json
Validator: scripts/check_benchmark_report.py
Template: examples/benchmark_report_template.json
The Anthropic automated-w2s-researcher (2026) paper headlined a performance comparison: "2 researchers × 7 days (PGR=0.23) vs 9 agents × 5 days (PGR=0.97)."
Dramatic. Also: n=2, author-conducted, no independence, self-scored. Not a scientific comparison — a presentation artifact. The authors knew what the agent would produce before sitting down to do the human baseline; their "7 days" is implicitly benchmarked against their own foreknowledge. Unaware external researchers doing the same task cold would likely score higher. The gap may partly be a sandbagged human baseline.
ARS inherits this risk the moment a user publishes an "ARS beats manual" claim. This pattern defines a mandatory schema for anyone publishing an ARS benchmark comparison. Reports that don't satisfy the schema aren't "ARS benchmarks" — they're anecdotes.
Six required top-level fields in benchmark_report.schema.json:
ars_version — exact semver the benchmark ran against; ties results to a specific
skill version so readers can check whether that version is still current.task_definition — description, task_type (outcome-gradable or open-ended), and
outcome_gradable flag; distinguishes rubric-scored tasks from genuinely open-ended ones.human_baseline — the five provenance fields that determine whether the human side
of the comparison is credible (see field-by-field rationale below).ars_run — cost, time, skills used, and declared data access level; lets readers
check the "agents are cheap" or "agents are fast" claims against concrete numbers.metrics — primary metric name, numeric value, and scoring independence; forces
explicit disclosure of who scored the outputs.caveats — non-empty array of known limitations; at least one honest statement is
required by schema enforcement.human_baselineThis block exists because the human side of a comparison is where benchmark credibility lives or dies. A weak human baseline inflates an agent's apparent advantage.
sample_size (integer, minimum 1): Schema rejects zero, forces an explicit number.
n=1 or n=2 get a validator warning rather than a hard failure — some tasks are
expert-bounded — but the number must be stated. Silence is worse than admitting n=1.
author_independence (enum: author-conducted | author-blinded | third-party-conducted):
author-conducted is the "2 authors did it themselves" trap: allowed, but the schema
flags the downward-bias risk. Authors know what a good ARS output looks like; their
human effort is calibrated against that prior. author-blinded (authors worked without
seeing ARS outputs first) is better. third-party-conducted eliminates experimenter
bias entirely.
hours_spent (number, minimum 0): Lets third parties sanity-check the time claim.
"7 days" reads differently against 20 papers versus 400.
recruitment (non-empty string): One sentence forces self-description of selection
bias. "Authors," "hired on Upwork," "convenience sample" are all honest. Blank is not
permitted.
tools_allowed (array of strings): "No tools" vs "ChatGPT allowed" is not the same
comparison as ARS. If the human baseline used no AI assistance while ARS ran full
pipeline, the comparison is testing infrastructure access, not cognitive quality.
ars_runcost_usd (number, minimum 0): A run costing $500 in API tokens is not "cheap"
just because it took 3 hours. Hours and cost together are needed for a fair comparison.
data_access_level_declared (enum: raw | redacted | verified_only): If ARS ran
on raw data while the human baseline used redacted data, the benchmark is testing
data privilege, not skill. Cross-references ground-truth isolation from v3.3.2.
skills_used (array, minItems 1): A benchmark running academic-pipeline
full-mode is not comparable to one running only deep-research. Both are ARS;
they're different scope claims.
metrics.scoring_independenceFour enum values: authors-scored, third-party-scored, blind-scored, self-scored.
self-scored parses and exits 0, but the validator emits a warning to stderr. Self-scoring
is the worst-case alignment failure: the same agents that produced the output are being
asked whether the output is good. blind-scored (human raters without knowledge of which
output came from ARS versus human baseline) is the minimum credible standard for any
claim that will be shared publicly.
caveatsNon-empty array, items non-empty strings, minItems 1. The schema physically prevents an empty caveats field. A benchmark report with no caveats either has no known limitations (implausible for any real-world evaluation) or the author didn't think about limitations (which disqualifies the report more than any specific limitation would).
The caveats field is where sample size warnings, scorer bias, task-selection bias, and tool-access asymmetries should land when they don't rise to schema errors. It is the honest disclosure box.
examples/benchmark_report_template.json to your benchmark directory.FILL IN: field with real values.human_baseline.sample_size to the actual number (schema rejects 0).python scripts/check_benchmark_report.py your-report.jsonERROR: lines (schema violations, exit 1).WARNING: lines (stderr, exit 0). Either address them in your methodology or
document them explicitly in the caveats array.Not a quality gate for your methodology. A report can be schema-valid and still be methodologically weak. The schema forces disclosure; it does not evaluate task choice, rubric validity, or baseline skill. Compliance is necessary but not sufficient.
Not a way to launder self-scored or n=2 comparisons. self-scored and
sample_size: 2 produce a valid document. The warnings are loud. Schema compliance
means the claim is honest about its weaknesses — not that the claim is strong.
Not a substitute for peer review. The schema is the minimum bar for internal credibility. Peer review catches confounded variables, cherry-picked tasks, post-hoc metric selection — problems the schema cannot see.
Not frozen. A benchmark declaring ars_version: 3.3.5 is always checked against
the v3.3.5 schema. Future versions may add fields without invalidating existing reports.
These are behaviors the schema is specifically designed to make visible. They are not prohibited — the schema accepts all of them — but they will appear in validator output or be apparent to any reader of the JSON.
Omitting sample size. The field is required. There is no valid ARS benchmark report
without a stated human_baseline.sample_size. n=1 is worse than n=10, but it is
infinitely better than unstated.
Author-conducted with no caveat disclosure. Using author-conducted is allowed.
Not noting it in caveats as a limitation is an editorial failure, not a schema
failure — but reviewers will notice.
Self-scoring with no warning acknowledgment. self-scored produces a warning on
every validator run. If you publish a report with self-scored and don't address it
in caveats, the omission is visible in the JSON to anyone who checks.
Empty or single-word caveats. The schema requires minLength: 1 per item and
minItems: 1 for the array. A caveat of "none" is technically valid but signals
that the author did not engage seriously with limitations.
Claiming ARS advantage without disclosing cost_usd. The field is required. A claim that "ARS completed this in 4 hours vs 40 human hours" without a dollar figure omits the dimension where the trade-off may be reversed.
Per-skill benchmark templates (a deep-research-specific template, an
academic-paper-reviewer-specific template) are likely as v3.4 adds domain-specific
evaluation protocols. The cost_usd field may split into cost_usd_api and
cost_usd_compute once cloud execution costs become a factor. The
data_access_level_declared enum may expand to match the v3.4 ground-truth tier
vocabulary if that spec evolves. None of these changes will break existing v3.3.5
reports, which remain valid against the version they declared in ars_version.