Terminal-Bench 5
The one profile that adds a requirement.
The platform's TB5 profile adds declared difficulty evidence to the shared task and verifier requirements. The reference inventory records TB5 intent from submission descriptions; this catalog publishes the contract rather than a downloadable TB5 task set.
The conditions a task is audited under.
Read from src/infinity/profile.py, which resolves the profile once at load time so every check reads it from the bundle rather than each deciding for itself.
- Agent egress
- Open, as TB3 and TB4 -- the newer generation does not close the agent's network.
- Verifier
- Separate container, no network, as under every benchmark profile here.
- Difficulty evidence
- Required. TB5 is the only profile that turns on S13, the TB5 difficulty-evidence check.
What a next-generation task has to be able to show.
The next-generation profile asks authors to declare four forms of difficulty evidence. S13 records these declarations for review. Verifying the underlying trials remains a separate evaluation step.
Independently parameterized policy
A second, independently parameterized policy passing the same workload provides evidence that acceptance limits describe the problem across reference solutions.
Held-out false-solution corpus
The failures a capable model actually produces are near-misses that satisfy the surface contract and violate one stated rule. The task declares where that corpus lives and how many candidates it holds.
Scored strong-model trials
Recorded, scored strong-model trials establish the basis for the declared difficulty.
Every trial a valid task failure
Each counted failure must be attributable to the task. Crashes, timeouts and infrastructure faults are classified separately from valid task failures.
S13 records all four declarations as REVIEW findings. Declaration review does not execute or verify the trials; difficulty qualification requires independently checked trial evidence.
Shared assurance requirements.
The verifier runs apart from the agent
Harbor makes a separate verifier sandbox opt-in. The audit treats it as required, because a verifier sharing a filesystem with the thing it is grading has no boundary left to test.
S11 in src/infinity/checks/separate.pyComplete required coverage before issuance
The report separates completed checks from skipped and errored stages. Every required check must complete before a mark can be issued.
grade() in platform/web/lib/certification.tsThe result names the revision it was graded under
Each result retains the methodology revision used for its audit, so later changes remain distinct from the original evidence.
METHODOLOGY_VERSION = 1.0 in platform/web/lib/brand.ts
Inspect the implementation sources.
Inspect the source definitions for the profile, checks and reference inventory.
src/infinity/checks/tb5.pyS13 and the four declarations it looks forsrc/infinity/profile.pyProfile.TB5, and the note that it adds a requirement rather than relaxing onecorpus/tb5/README.mdThe two TB5-aimed PRs, and why the corpus keys off prose rather than a version number
Submit the task. Get its recorded check evidence.
Tbench AI audits the task, environment, reference solution and verifier. Every completed audit retains its evidence report.