Skip to content
Evaluation and testing
- Ragas - Metrics and test-set generation for RAG and agent pipelines.
- DeepEval - Pytest-style eval framework with G-Eval, hallucination, and relevancy metrics, a tool-correctness metric that compares an agent’s tool calls with the expected ones, and MCP metrics that judge whether an agent used an MCP server’s tools to complete a task.
- Promptfoo - Declarative CLI that runs test cases against prompts, models, and agent providers, including the Claude Agent SDK and OpenAI Codex SDK, and grades them with deterministic and model-graded assertions, plus LLM red-teaming and vulnerability scanning in CI; MIT, now part of OpenAI.
- TruLens - Instrumentation plus feedback functions that score app internals, not just final output.
- Evidently - Python framework for evals, drift detection, and monitoring across tabular, text, and GenAI systems.
- Inspect - UK AI Security Institute’s eval framework for agentic tasks, tool use, and sandboxed execution, with exact, pattern, and model-graded scorers; CLI coding agents run in it through Inspect SWE.
- Inspect Scout - Scans agent transcripts from Inspect eval logs and other sources with LLM, grep, or custom scanners and writes queryable results, for finding failure patterns across runs; MIT.
- Giskard - Checks and red-team scans for LLM agents: multi-turn scenarios, LLM-judge checks, and probes for prompt injection and harmful content (v3; v2 is unmaintained).
- Scenario - Simulates multi-turn users against an agent so conversations, not single calls, can be asserted on.
- Autoevals - Standalone library of model-graded and heuristic scorers usable outside Braintrust.
- OpenEvals - Readymade evaluators for LLM apps: LLM-as-judge prompts, exact and string-distance matchers, and structured-output checks, as functions that return scored results and plug into pytest or LangSmith, in Python and TypeScript.
- AgentEvals - Evaluators that score an agent’s tool-call trajectory against a reference with strict, unordered, subset, or superset matching, or with an LLM judge; MIT.
- BFCL - Evaluates a model’s function calling on single-turn, live, multi-turn, and agentic web-search and memory tasks, checking single-turn calls by AST match and running multi-turn and agentic calls against the benchmark’s API backends; Apache-2.0, part of the Gorilla repository.
- HermesGate - Runs a repository’s own checks and stores a receipt keyed to the content and tool versions checked; in
fast and review modes an identical receipt is reused as a PASS and any change re-runs the checks, giving an agent’s own PR a repeatable pre-review result.
- NovaFabric - Self-hosted CLI whose
nova capture wraps any command, such as an agent script, and records the run as a portable Run Capsule that you can replay and structurally diff against another run; Apache-2.0, pre-1.0.
- LEEVAR Battery - Grades agent conversation transcripts on 18 tests across six dimensions with a judge model you configure, reports tests the transcript cannot evidence as NOT TESTED and leaves them out of the score, and issues no letter grade when two thirds or fewer of the tests have evidence; MIT, and the same authors run an optional hosted scan.
- Agent-Causality - Records LangGraph and OpenCode agent runs as event graphs with declared causal parents, so a failing event can be sliced to the earlier events it structurally depends on and agent state rebuilt at any event with a SHA-256 state hash, and decisions recorded through its Python SDK can be replayed with changed inputs to test which ones flip the outcome; AGPL-3.0.