Engineering & DeveloperA/B testing agent memory strategies

Run Real A/B Tests on Agent Memory Strategies — Not Vibes Comparisons

You want to know if reflection memory is paying off, if longer retention helps, if a different retrieval strategy outperforms. Without controlled experiments, every decision is vibes. MemoryLake provides branched memory for A/B testing — same users, different memory strategies, measurable outcomes.

Get Started Free

Free forever · No credit card required

The problem: agent memory decisions usually have no evidence

Should you increase retention? Switch retrieval ranking? Add reflection memory? Most teams ship the change to all users and hope for the best. No control group means no real measurement.

How MemoryLake enables memory A/B tests

Branched memory per cohort

Cohort A uses strategy 1; Cohort B uses strategy 2; same users otherwise.

Per-cohort retrieval rules

Different memory types, retention, or ranking per cohort.

Outcome attribution via memory diff

Measure what changed between cohorts.

Promote winning branches to main

Roll out the winner with full audit.

Get Started Free

Free forever · No credit card required

How it works for memory A/B testing

Connect — Define cohorts in the workspace.
Structure — Each cohort uses a memory branch with different rules.
Reuse — Measure agent outcomes per cohort; merge winning branch.

Before vs. after: agent memory strategy decisions

	DIY memory	MemoryLake
Comparing memory strategies	Vibes	Real A/B test
Per-cohort memory rules	Hard	Native branches
Outcome attribution	Limited	Memory diff
Rollout of winning strategy	Manual migration	Merge branch

Who this is for

Product and engineering teams who want evidence-based memory strategy decisions instead of "we tried it and it felt better."

Runnable demoabout 5 minutes

Try this use case on your own account

An A/B test of four memory strategies for a release assistant, scored two ways over the same replies. Four platform standups make eight decisions with conditions; three of them (the staging rehearsal, the client-app migration, the cost review) are never mentioned again, so their outcome is not in memory in either direction. The assistant often answers "No, the staging rehearsal did not pass": the right verdict, with a reason nobody recorded. A verdict-only scorer counts that as a win; a grounded scorer does not, and the two pick different winners.

What you will see

Four GROUP conversations go in one after another; the extracted facts are copied into two branches (fact list → fact add), so the arms differ only by the strategy under test. Eight Yes / No / Unknown questions through router chat --boundary, 3 times each, in four arms
Over 6 runs, condition checks written by the app ("no record that it was met") got every verdict right (138 / 138), but only 93 / 138 were grounded: the model turned "no record" into "the rehearsal did not pass" in 45 of 54 answers on the three lapsed conditions
A careful system prompt: 118 / 130 by verdict, 94 / 130 grounded. Re-wording only the three lapsed checks to "not recorded either way; do not say it happened or that it did not": 138 / 138 by verdict and 138 / 138 grounded
The grounded scorer strips "no record that …" phrases and restated conditions, then looks for a claim about an outcome the arm's memory does not hold; it is checked against written cases before grading. Plus memory and ask subcommands and a local web UI

You need: a MemoryLake API key · the memorylake CLI · Python 3.9+

Run it

git clone https://github.com/memorylake-ai/memorylake-usecases.git
cd memorylake-usecases/no-record-memory
export MEMORYLAKE_API_KEY=sk-…
python3 demo.py          # or: python3 web/server.py

Open on GitHub Read the walkthrough