Before the Benchmark: Make the Claimed Win Reviewable

The familiar failure

A good-looking chart is not a reviewable result.

Teams often begin with a real question, change a configuration, keep the most favorable output, and only later ask whether the comparison can be trusted. By then, the baseline may be unclear, the seed plan may be unrecoverable, failed trials may be absent, and the raw outputs needed to inspect a score may be gone.

The problem is not simply that the result may be wrong. It is that no one can determine what the result means without reconstructing the experiment from memory. A claimed win needs a declared comparison before it needs an audience.

A practical response

Preflight the experiment. Retain the evidence. Assess the claim.

1. Preflight the contract. State the question, baseline, candidate, datasets or splits, metrics, seed plan, stopping rule, and required evidence before results exist.
2. Retain each trial. Preserve the trial outputs, provenance, identifiers, failed attempts, and the inputs needed to check that the evidence belongs to the declared comparison.
3. Assess the supplied record. Check whether the retained evidence is complete and consistent enough to support the claimed scope. If it is not, say so without converting missing evidence into a performance conclusion.

The executable artifact

ACE turns that discipline into a checkable workflow.

ACE, the Assured Comparison Engine ↗, is an open-source Python package for preflighting experiment contracts and assessing retained JSON or CSV trial outputs. Its v0.1.0 release produces an ACCEPTED, REJECTED, or INCONCLUSIVE decision pack from the evidence supplied to it.

ACE checks declared identity, provenance, split, seed, baseline, metric, failed-trial, and required-statistical evidence. Essential gaps or mismatches fail closed. That is deliberately narrower than a benchmark runner and more useful than a post-hoc checklist that cannot see the underlying record.

First applied case

Token-Bleed is retaining evidence before making its comparison.

The Token-Bleed benchmark ↗ now retains multi-seed trial evidence and maps that record into ACE. Its local pilot is an implementation check, not a published comparative finding. The benchmark’s declared protocol requires retained runs across 20 distinct seeds per endpoint before a comparative claim is considered.

That constraint is the point. A smaller pilot can surface operational problems or missing evidence. It cannot be promoted into a conclusion because the toolchain happens to be available.

Related research

Trustworthy claims require the architecture around them.