Agent and harness benchmarks
Harnesses that run an agent, or a whole CLI coding agent, on tasks in a sandbox and grade the outcome, usually with tests the agent cannot see; use them to compare models, harnesses, and configurations on the same tasks.
- Harbor - Runs agents, including Claude Code, Codex CLI, Gemini CLI, OpenCode, and Cursor CLI, on containerized tasks and scores each trial with the task’s own test script, so harnesses and models can be compared on the same tasks, with Terminal-Bench and other datasets in its registry; Apache-2.0.
- τ²-bench - Runs a tool-using customer-service agent against an LLM-simulated user, who also has tools in some domains such as telecom, through tasks in airline, retail, telecom, and other domains, and by default scores each trial by whether the database ends in the gold state and whether required information was communicated, in text or voice; MIT.
- Inspect SWE - Wraps Claude Code, Codex CLI, Gemini CLI, OpenCode, and other CLI coding agents as Inspect agents that run in an Inspect sandbox with their model calls routed through Inspect, so any Inspect model, scorer, and log viewer applies to them; MIT.
- coder-eval - Runs Claude Code, Codex, OpenCode, and other CLI coding agents on YAML tasks in a sandbox and grades the workspace and trajectory with deterministic checks such as file contents, commands, and whether a given skill was loaded, or with LLM judges, including A/B runs across agents, models, and skills; Apache-2.0.
- SWE-bench - Harness that applies a candidate patch to a real GitHub repository inside a per-issue Docker image and runs the repository’s held-out FAIL_TO_PASS and PASS_TO_PASS tests to decide whether the issue is resolved; MIT.
- Inspect Evals - Ready-made Inspect tasks for published benchmarks, from GSM8K to agentic and cybersecurity suites, run against any Inspect-supported model with one
inspect evalcommand; MIT. - YYLO Benchmark - Runs coding prompts and tasks across models and harnesses on fresh checkouts, keeps every output unchanged, and lets you re-evaluate them later with new checks or judges.
- AgentDojo - ETH Zurich benchmark and framework for tool-using agents against prompt injection, with task suites and attack and defense pipelines that report utility and attack success.
- ClawBench - Live-web benchmark for evaluating browser and computer-use agents on about 280 everyday tasks, with request interception and recorded execution evidence.