Evaluate Agent Outputs With Full Visibility Into the Memory That Drove Them
Agent eval frameworks score outputs without knowing which memory the agent retrieved. A bad output might mean a bad model, a bad prompt, or bad memory — but eval can't tell. MemoryLake links every output to the memory used, so evaluation actually identifies root causes.
Evaluate Agent Outputs With Full Visibility Into the Memory That Drove Them
Get Started FreeFree forever · No credit card required
The problem: agent evaluation without memory context is blind
The eval framework flagged 12% of outputs as low quality. You don't know if the model failed, the prompt failed, or the retrieved memory failed. Without memory context per evaluation, fixing the right thing is guesswork.
How MemoryLake delivers memory-aware evaluation
Per-output memory provenance
Every evaluated output links to the memory it used.
Memory diff between good and bad outputs
See what memory access correlated with quality.
Evaluation against pinned memory snapshots
Test with controlled memory state.
Memory-grounded eval categories
Failures attributable to retrieval vs generation.
Free forever · No credit card required
How it works for memory-aware eval
- Connect — Wire MemoryLake into your eval pipeline.
- Structure — Each generated output records the memory used.
- Reuse — Eval analysis shows memory-retrieval failures separately from generation failures.
Before vs. after: agent eval with memory awareness
| DIY memory + eval | MemoryLake | |
|---|---|---|
| Identify retrieval vs generation failures | Hard | Built in |
| Memory diff between cohorts | Manual | Semantic |
| Eval against pinned memory | Custom | Snapshots |
| Root cause attribution | Guesswork | Direct evidence |
Who this is for
Engineering teams running agent evaluation pipelines who need to attribute failures correctly to fix the right thing — and where current eval treats memory as a black box.
Try this use case on your own account
A support assistant on MemoryRouter is graded on 8 questions to decide whether prompt v2 beats v1. Run the same call three times with nothing changed and the attached memory differs, so grades flip and the v1 → v2 verdict comes out differently almost every run. Pin one memory snapshot per question and the memory holds still: the prompt's real effect shows up every time, and every wrong answer gets a cause.
What you will see
- 60 support facts in one project, one boundary; every eval call is
router chat --boundary(gpt-4o-mini, temperature 0), graded against the facts a right answer must state - A/A through MemoryRouter: three identical calls got different amounts of memory (prompt tokens) in 71 of 88 question-runs and changed grade in 33; the reply never says which memories it was given. The "v2 broke / fixed" list came out 10 different ways in 11 runs
- Pinned snapshots:
search --types fact --top-k 8once per question, ids and text saved, then every prompt runs against the same snapshot (router chat --body, no boundary). Same memory 88 of 88, A/A flips 7 of 88 (the model's own noise), and v2's real regression (it drops the break-glass step) found in 11 of 11 runs - Every wrong answer against a snapshot is attributed: the grader's fact missing from the snapshot (retrieval, e.g. the SCIM fact never retrieved for "create and remove user accounts") or present but left out (generation). Plus
snapshot/replaysubcommands and a local web UI
You need: a MemoryLake API key · the memorylake CLI · Python 3.9+
Run it
git clone https://github.com/memorylake-ai/memorylake-usecases.git cd memorylake-usecases/agent-eval-memory export MEMORYLAKE_API_KEY=sk-… python3 demo.py # or: python3 web/server.py
Watch it run · real, unedited recordings
Related use cases
Frequently asked questions
Eval framework integrations?
Eval framework integrations?
RAGAS, OpenAI Evals, LangSmith, custom — all supported.
Memory-grounded eval categories?
Memory-grounded eval categories?
Retrieval recall, retrieval precision, conflict surfacing, provenance accuracy.
Self-host?
Self-host?
Yes — enterprise tier deploys in your VPC.