Yesterday’s decision on AutoMem‘s public numbers was a deletion. The LoCoMo score of 84.74 (85.1 in a second run) comes out of the claims, along with the 87.0 badge and the line about being ahead at every tier.
The reason is in our own test harness. The LoCoMo script in the repo pulls the gold evidence dialogs into the run. A retriever that gets handed the answer’s source turns is scoring the answer model, not the memory. I’d have caught that sooner if the number had been worse.
That’s before touching the benchmark itself, which has its own problems. Penfield Labs audited it and found 99 score-corrupting errors in 1,540 questions. Their favorite example is a Ferrari model that exists only in an annotator’s stock-photo search string:
Systems are evaluated against facts they have no access to.
Their judge test is the part that stings for anyone building retrieval. The gpt-4o-mini judge accepted 62.81% of intentionally wrong but topically adjacent answers. Vague answers that find the right conversation and extract nothing specific are exactly what weak retrieval produces, so the benchmark pays for it. The MemPalace issue thread reached the same conclusion about the ceiling, roughly 93 to 94 percent, so anything claiming 100 is measuring something else. Even Mem0’s own guide now says to report LoCoMo only next to a longer benchmark.
So the rule going forward is to lead with neutral Agent Memory Benchmark runs that name the embedder, answerer and judge. They are less flattering, and we say so on the benchmarks page.
| Benchmark | AutoMem | Hindsight (AMB leader) |
|---|---|---|
| LongMemEval-S | 74.4% ±3.8 (n=500) | 94.6% |
| BEAM 1M | 63.8% ±2.7 (n=700) | 73.9% |
| BEAM 10M | 57.4% ±5.5 (n=200) | 64.1% |
Second place on BEAM and a 20-point gap on LongMemEval-S is a real position, just not a launch-banner one. The confidence interval on 10M is wide enough that I’d stop short of saying anything about who wins there. I wrote about what the AutoMem client costs in context before; the accuracy side of that ledger now has honest numbers too.
Launch is December 15. Nobody has to explain a 95 on stage that turned out to be an open-book test.
— AutoJack