Your benchmark is only as trustworthy as its verifier.
Tbench validates the task, environment, reference solution and verifier. One task or a complete set enters the same fourteen-stage workflow, with source-bound evidence and complete required coverage before a mark is issued.
- end-to-end workflow stages
- 14
- registered static rules
- 47
- explicit verdict states
- 5
- engine regression tests collected
- 704
Engine inventory snapshot: 2026-09-21. This records engine test discovery; each submission retains its own execution evidence.
Loading the published methodology catalogue…
Fourteen stages around one evidence boundary.
The workflow contains one source lock, twelve task validation stages and one submission evidence seal. Engine regression tests validate the implementation.
All fourteen stages apply to a single-task audit: one source lock, twelve task-level stages, and one evidence seal. There is no minimum batch size.
- 01 · Source
- Stage 01 · Git SHA & tree digest lock
- 02 · Contract
- Stages 02–04 · Static rules, traceability, corpus
- 03 · Execution
- Stages 05–06 · Oracle/NOP controls & Pass@2
- 04 · Adversarial
- Stages 07–12 · Mutation, witness & cheat-pass
- 05 · Attestation
- Stages 13–14 · Review & cryptographic seal
01Source lock and provenanceSource integrityRequireddeterministicOnce per audit
Binds the repository revision, task tree digest, tool version and audit policy.02Static checksStatic architectureRequireddeterministicOne task
Runs the registered rules against structure, metadata, instructions, launchers and verifier contracts.03Per-Check QC - TraceabilityRequirement mappingRequireddeterministicOne task
Maps graded requirements from the instruction through the solution to verifier assertions.04Duplicate checkCorpus integrityRequireddeterministicOne task
Compares the task with a pinned corpus rather than treating a single task as vacuously unique.05ValidationContainer calibrationRequiredcontainerOne task
Runs positive and negative controls in isolated images.06Pre-check (Pass@2)Difficulty calibrationRequiredmodelOne task
Runs the fixed two-attempt gate and separates solver failure from infrastructure failure.07AVA - Verifier AuditVerifier rigorRequiredcontainerOne task
Tests valid and invalid controls, and flags unclassified boundary mutations for semantic review.08Witness VerificationIndependent re-executionRequiredcontainerOne task
Re-executes the deliverable on fresh seeded inputs.09Per-Check QC - MutationMutation rigorRequiredcontainerOne task
Measures which reference mutations are rejected. Surviving mutations require review against the contract.10PROBEDifficulty evidenceRequiredmodelOne task
Preserves panel disagreement and empirical difficulty evidence.11Adversarial Cheat-PassAdversarial resistanceRequiredcontainerOne task
Attempts leakage, protected-path access and shortcuts that earn reward without doing the task.12Agent Trial Results (Pass@5)Empirical gauntletRequiredmodelOne task
Records each valid attempt, model, environment and observed outcome.13Automated ReviewAutomated reviewAdvisorymodelOne task
Adds an advisory second opinion without overriding deterministic evidence.14Evidence seal and submission decisionAttestation and registerRequiredaggregateOnce per audit
Binds stage records to exact digests and issues or withholds the submission result.
The checks inside the stages.
Forty-seven registered static rules are grouped by failure mode. Forty-five are active under the standard policy; two witness checks support fresh-input re-execution.
- Active policy
- v1.1 · 45 active rules
- Supporting
- 2 witness replay checks
- Enforcement
- Zero silent waivers on required gates
Reference solution correctness
Validates that the reference solution computes the graded artifact legitimately.
Contract coherence
Finds missing, ambiguous, or undisclosed graded requirements.
Verifier rigor and scoring
Tests graded coverage, tolerances, scoring semantics, and judge gating.
Determinism and fixtures
Checks fixtures, randomness, clocks, networks, and reproducible execution.
Isolation and anti-cheat
Detects answer leakage, writable trust boundaries, and unsafe file handling.
Scaffold and platform contract
Validates task layout, launch behavior, metadata, reports, and container contracts.
Witness verification
Checks independent re-execution on fresh witness inputs.
Five clear verdict states.
The most important distinction is between a task defect and an audit that could not complete. Both can withhold a result, but they demand different remedies and stay different in the evidence.
- 01Passed
The stage ran, produced evidence, and satisfied its published rule.
- 02Failed
The stage ran and established a task or verifier defect. A required failure withholds the mark.
- 03Needs review
Evidence is ready for a named reviewer to decide.
- 04Did not run
A prerequisite or tool was unavailable. A required skip blocks issuance until the stage is completed.
- 05Errored
Infrastructure failed before a verdict existed. The report names it separately from a defect in the task.
Coverage comes before the score.
Complete required coverage is the first issuance condition. Every stage result remains visible, and the public mark is issued after the required workflow is complete.
- 01
Tbench Certified
Every required stage ran and passed. The evidence seal binds the result to the submitted source and methodology revision.
- 02
Conditional
Every required stage ran, but evidence needs a named human decision. The limitations travel with the result.
- 03
Not issued
A required stage failed, skipped or errored. The record remains useful because it identifies the blocker and remediation path.
Audit one task. Scale when the evidence holds.
Start with a single PR or submit a full set. Every submission follows the same standard.