RunLedger
CI harness for tool-using agents that records tool calls once, replays them in CI, and fails PRs on tool-order or JSON-schema contract violations, tool-call budget overruns, or baseline regressions, with JUnit and HTML reports. MIT licensed.
A CI harness that replays recorded agent tool calls and fails pull requests on tool-order, schema, or call-budget regressions.
What it is
RunLedger is an open-source MIT-licensed harness for testing tool-using AI agents in continuous integration. You define a suite and cases, then the runner launches your agent as a subprocess, in any language, that talks to the runner over a newline-delimited JSON protocol on stdin and stdout. In record mode, tool results are saved to a cassette file; in replay mode, those recorded results are returned to the agent, so runs are deterministic and do not need API keys for the recorded tools. The runner then checks assertions and budgets, compares results to a baseline, and writes JUnit and HTML reports. Replay only freezes the tool boundary: any model or network calls the agent makes on its own still run live unless you route them through the protocol or replace them with fixtures.
Key features
- Records tool calls once to a cassette and replays them in CI, matching repeated identical requests in order
- Asserts tool order with must_call, must_not_call, and call_order checks
- Validates final JSON output against a JSON Schema and required fields
- Enforces tool-call, tool-error, and wall-time budgets
- Compares success rate and latency against a stored baseline
- Writes summary.json, junit.xml, report.html, and run.jsonl artifacts
Use cases
- Failing a pull request when an agent skips a tool or calls tools in a new order
- Catching output changes that break a JSON contract before they merge
- Stopping prompt or code changes that push an agent past its tool-call budget
- Keeping CI runs repeatable by avoiding live tool calls
Who it's for
Developers maintaining tool-using AI agents who want pytest-style regression gates in CI. It suits teams that already commit test fixtures and want deterministic checks on agent behavior rather than quality scoring.
Getting started
The README describes a quickstart flow in which the runledger init command writes an example suite, cassette, schema, baseline, and GitHub workflow, and a replay run is then executed against that suite. The exact setup commands are not reproduced here; consult the repository README for the verbatim steps.
Alternatives
Related directory entries include no-mistakes, oai-smoke, e2e, and tokenwatt-verify. This listing does not indicate how those projects compare or whether the README endorses them.
Drafted with AI from the project's public documentation; verify against the source. Written 2 Nov 2026.