Amazon FEB-30 Benchmark Audits Agent Memory Storage Directly
QA-only evaluation cannot tell whether failures come from memory writing, retrieval or answer generation, so teams may ship agent memory systems that look fine in testing yet mis-store facts. Builders should add direct checks of stored facts after each update, using FEB-30-style scripted histories where ground truth is known. The 20.1% correct-action rate is a concrete red flag for anyone reusing Mem0 or Hindsight architectures without stage-level instrumentation.
Amazon researchers released FEB-30, a benchmark built from 30 scripted fact histories converted into 147 documents, where the correct memory state is known after each document. Comparing stored facts against ground truth on memory systems built like Mem0 and Hindsight, the study found stored-fact accuracy and QA scores diverged by 16.8 points on average, with pass/fail verdicts disagreeing on 21% of fact histories. The tested system chose the correct memory action, such as update or reinforce, only 20.1% of the time. QA scores alone did not reveal which pipeline stage was failing, since a strong LLM can mask a broken memory store.
New Amazon paper shows that teams shipping agent memory should check what actually got stored, since QA scores alone did not reveal which pipeline stage was failing.
Agent memory benchmarks grade the agent's answers, which mixes memory with retrieval and answer writing. A strong LLM can hide a broken memory store, and a retrieval miss can make a correct one look bad.
They built FEB-30, 30 scripted fact histories turned into 147 documents, so the correct memory state is known after every document. On a memory system built like Mem0 and Hindsight, they compared the stored facts with that truth and with QA scores.
The 2 scores differed by 16.8 points on average, and pass/fail verdicts disagreed on 21% of fact histories. The system picked the right memory action, like update or reinforce, only 20.1% of the time, which QA never showed.
If you build agent memory, test the stored facts directly after each update, alongside end-to-end QA.
