Tools, standards, and benchmarks for tracing and evaluating AI products (LLM apps, agents, coding-agent harnesses, Agent Skills, and MCP servers), with evaluations graded by LLM judges or deterministic checks.

For engineers who build or run AI agents and need to see what a run did, what it cost, and whether it was right. It covers OpenTelemetry conventions, tracing platforms, coding-agent telemetry, eval frameworks, agent and harness benchmarks, skill and MCP server testing, and guardrails. Every entry is held to the list’s requirements: alive, maintained, and at least 8 weeks old, and new entries are run hands-on before they are added; the next paragraph says which older entries could not be run.

Entries say what a project does, not what it markets, and carry its license where that is not plainly open source. Projects that were dormant, end-of-life, out of scope, or too young to judge were left out. Most entries were installed or built and run against a mock backend at the 2026-10-04 audit; a few could not be run (for example Kong, SigNoz, and anything needing Docker or a hosted account), so this list does not claim every entry was exercised. Entries added at the 2026-11-15 evals expansion, and in the follow-up later that day, were each run on a Linux box with local models or mocks only, and the commands and output are in evidence/. A weekly job re-checks that each repository still exists, is not archived, and has a recent commit.

Sections

Write-ups