Skip to content
Crossing HurdlesCrossing HurdlesExpertsTerminal-Bench Research
ApplyExpert login
Experts · Local Authoring & Calibration

Build your task locally. Submit evidence through your project.

Prepare, calibrate, and rehearse your assigned Terminal-Bench 4.0 task locally with Linux Docker, Harbor 0.18.0, and pytest before opening or syncing your cohort pull request.

Open Experts workspaceQualification studioExpert playbook
Python 3.12+ · Harbor 0.18.0Linux Docker EngineUnix LF · Non-Root rwe
WORKSTATION CALIBRATION CHECKLIST
  1. Pinned local Harbor 0.18.0

    Run offline container rehearsals before pushing to your PR branch.

    0.18.0
  2. Positive & negative controls

    Verify Oracle = 1.0 and NOP = 0.0 inside clean Linux containers.

    1.0 / 0.0
  3. Automated PR stage tags

    Pushed commits sync CI: 14/14 Passed and reviewer status automatically.

    SHA LOCK
Local rehearsals never consume your project model-trial allowance

Four-step local workflow

Rehearse locally before opening your pull request.

Follow the four steps below so every pushed revision passes static hygiene and container calibration on the first CI run.

01 / Get the assigned workspace

Claim an available category opportunity and submit your proposal. After approval, accept the project’s GitHub invitation, fork the assigned repository where permitted, and use the task ID and submission branch shown on your active task page. The approved project invitation establishes your repository access.

Step 01 · Provisioned Branch Access

02 / Install and inspect the tools

Install Git, Python 3.12 and Docker with a Linux engine. Create a clean Python virtual environment and install the pinned local Harbor harness and pytest runner so you can rehearse both task containers offline on your workstation.

Step 02 · Pinned Local Harness

03 / Validate Oracle and Nop controls

Oracle must complete with reward 1 and Nop with reward 0. Use local rehearsals to validate your task before independent project review. Official certification requires the authorized audit process. Keep Tbench model-provider credentials and GitHub App keys off contributor devices.

Step 03 · Zero Credential Leakage

04 / Submit an immutable revision

Push to your assigned fork/submission branch and open the upstream PR. Independent static checks and Oracle/Nop calibration verify the submission. First-time private-fork workflows may need maintainer approval. Counted model trials remain commit-bound: pass@2 uses two of the ten allowed attempts per task cycle. Local rehearsals are separate from that allowance.

Step 04 · Commit-Bound CI & Trials

Workstation commands

Linux, macOS, and Windows PowerShell setup.

Keep Docker Desktop in Linux container mode and configure Git to commit pure Unix LF line endings across all shell scripts, Dockerfiles, and Python modules.

Linux & macOS (bash / zsh)

Create virtual environment & run controls

git config core.autocrlf input
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install "harbor==0.18.0" "pytest>=8.0"
docker info
harbor run --agent oracle tasks/<slug>
harbor run --agent nop tasks/<slug>

Windows PowerShell (Linux Docker Engine)

Activate environment & verify LF hygiene

git config core.autocrlf input
py -3.12 -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install "harbor==0.18.0" "pytest>=8.0"
docker info
harbor run --agent oracle tasks/<slug>
harbor run --agent nop tasks/<slug>

Ready to push your branch

Use your active task page for live PR status and stage evidence.

Use your active task page and project playbook for proposal status, repository links, automated PR stage tags, and remaining trial allowance.

About the Experts workflowRead the verifier methodology