Record an agent run once. Replay it forever, offline, at $0.
AgentReplay works like a flight recorder for LangGraph agents. It captures every non-deterministic input during a real run, then feeds those inputs back so the agent behaves exactly the same way again. Any recorded run then works as a free regression test: change a prompt, a model or the code, and AgentReplay tells you exactly which step changed.
Agents are non-deterministic programs
A normal function returns the same output for the same input. An agent doesn't, because its output depends on four inputs that change between runs. That makes failures hard to reproduce and changes hard to test, and every “test run” against the live API costs money.
LLM responses
The same prompt can return a different completion. Even temperature=0 isn't guaranteed to be identical.
Tool & API results
A search or database call returns different data next week.
The clock
datetime.now() returns a new value on every call.
Randomness
random(), uuid4() and sampled IDs are different every run.
The core idea: if every impure input is recorded and replayed, the agent becomes deterministic, and deterministic programs can be tested.
Record → Replay → Diff → Regress
Record
Wrap your agent with one line. Every LLM call, tool call, clock read and random draw is saved as an ordered event in a trace file.
pays API cost once
Replay
Rebuild the same agent with a replay model. Each call is answered from the trace, so nothing reaches the network.
$0 · milliseconds
Diff
Compare what the agent did on replay with what it did when recorded: graph path, tool arguments and prompts.
3 tiers, all free
Regress
A pytest plugin turns every .trace.json file into a test. CI runs them on each pull request without an API key.
green / red in CI
How interception works
Every LLM call in LangGraph goes through LangChain's BaseChatModel interface. AgentReplay hands your graph a model that implements that interface. While recording, it calls the real model and logs the exchange. While replaying, it answers from the trace. Your agent code doesn't change, and no provider SDK is monkeypatched.
Clock & randomness, scoped
agentreplay.now(), random() and uuid4() are drop-in replacements that only record or replay inside a run's context manager. Outside a run they behave like the standard library, and nothing is patched globally in your process.
One JSON file per run
A trace is an ordered list of typed events. Each event has a global seq (its order, which matters for async graphs) and a hash of its input, which replay uses for matching. The schema is a set of Pydantic v2 models with a versioned schema_version. New versions only add optional fields (v2 adds latency, token usage, model id and graph node), so old traces keep validating. Large outputs are stored once in blobs and referenced by content hash.
{
"schema_version": 1,
"run_id": "a1b2c3d4-…",
"events": [
{ "event_type": "clock", "seq": 0, "recorded_time": "2026-07-09T14:22:03.120Z" },
{ "event_type": "llm_call", "seq": 1, "hash": "11c22d33…",
"model_name": "claude-sonnet-5",
"input_messages": [{ "type": "human", "content": "Plan research for the topic: quantum error correction" }],
"output": "I'll investigate the state of the art in quantum error correction…" },
{ "event_type": "tool_call", "seq": 2, "hash": "22d33e44…",
"tool_name": "web_search",
"input_args": { "query": "surface code error correction threshold 2026", "max_results": 5 },
"output": "1. Google Quantum AI demonstrates below-threshold surface code …" },
{ "event_type": "random", "seq": 3, "source": "uuid4", "value": "7f3c9a20-…" },
{ "event_type": "llm_call", "seq": 4, "hash": "44f55667…", "output": "…" }
],
"blobs": { }
}Which recorded answer goes with this call?
During replay, each call the agent makes has to be paired with the right recorded event. The two obvious approaches both fail on real agents, so AgentReplay combines them:
Hash lookup
Hash the call's input (messages or tool args) and look it up. Because the lookup uses content, not position, a matching call is found even if the code runs steps in a different order.
Ordered cursor
If the same prompt appears more than once, as in a retry loop, a per-hash cursor returns the recordings in their original order: 1st, then 2nd, then 3rd.
Miss = divergence
No recording matches, so the new code made a call the original run never made. AgentReplay reports it, and that report is the regression signal. It is not treated as a crash.
| Strategy | Reordered / parallel nodes | Identical retries |
|---|---|---|
| Order only | breaks | Yes |
| Hash only | Yes | can't tell them apart |
| Hash → cursor (AgentReplay) | Yes | Yes |
Three tiers of “what changed?”
Comparing final outputs as exact text is brittle, and it hides where things went wrong. AgentReplay compares the two runs step by step, starting with the highest-signal checks:
- T1
Structural
Did the agent take the same path through the graph? A skipped or extra step is the clearest sign of a regression.
- T2
Tool calls
Same tool, same arguments, same result? Changed arguments often show the root cause.
- T3
LLM calls
Same prompt, same response? Shown as a unified diff, so a one-word prompt edit is easy to spot.
All three tiers run locally and cost nothing. A semantic check (“did the meaning change?”) would need an LLM judge, which costs tokens, so it is left as an opt-in.
❌ 3 divergence(s) found.
Trace A: 5 events
Trace B: 5 events
── Tool Calls (1) ───────────────────────────
[web_search] Tool 'web_search' called with different arguments
- "query": "surface code error correction threshold 2026",
+ "query": "logical qubit overhead surface code 2026 fault tolerance",
── LLM Outputs (2) ──────────────────────────
[seq 1→1] output changed
-I'll investigate the state of the art in quantum error correction. …
+I'll investigate quantum error correction, focusing on logical qubit overhead …
[seq 4→4] output changed
-Quantum error correction has crossed a milestone: …
+Quantum error correction reached a turning point in 2025: …How do we know replay costs $0?
Saying “the bill looked small” doesn't prove anything. Instead, during replay the agent is built with a TripwireModel, a chat model that raises an exception the moment it is called.
If replay ever fell through to a live model, the run would crash. It never does: the test suite, the CI regression job and the demo script all replay to the same final state as the recording, with the tripwire left untouched.
- 148 unit + integration tests
- Golden-trace regression suite on every push
- Replay reproduces the recorded final state exactly
class TripwireModel(BaseChatModel):
def _generate(self, *args, **kwargs):
raise RuntimeError("TRIPWIRE: live call!")
engine = ReplayEngine(
recorded_trace=trace,
build_fn=build_agent,
model=TripwireModel(), # never fires
)Three snippets
import agentreplay
wrapped = agentreplay.wrap(
build_fn=build_my_agent,
model=ChatAnthropic(...),
)
wrapped.invoke({"topic": "..."})
# → traces/<run_id>.trace.json# pyproject.toml [tool.pytest.ini_options] agentreplay_traces = "traces/" agentreplay_agent = "app:build_agent" $ pytest traces/6eb6f580.trace.json PASSED
agentreplay show run.trace.json agentreplay diff old.json new.json agentreplay serve # this UI
Known limits (v1)
- · LangGraph only. CrewAI / raw OpenAI support is planned.
- · Replay assumes read-mostly tools. Writes aren't sandboxed.
- · Replays final messages, not token streams.
- · Renaming a graph node shows up as a structural diff.