Mem0 releases DolphinBench for agent memory evaluation
Memory benchmarks have mostly measured recall on quiz-style questions, ignoring the cost and latency trade-offs that decide what teams actually ship. DolphinBench scores downstream actions after long histories, so results better reflect real agent behaviour. Builders choosing a memory layer for production agents can use it to compare systems on all three axes at once.
Mem0 released DolphinBench, a public benchmark for evaluating agent memory systems. It grades the action an agent takes after a long interaction history, rather than scoring memory through question-answering quizzes with precision and recall. The benchmark reports accuracy, cost and latency on a single leaderboard and maps the Pareto frontier across memory systems. Details are available at dolphinbench.ai.
Today we are releasing DolphinBench: Mapping the Pareto frontier of agent memory. Coding, tool use, and long-context retrieval already have strong public benchmarks. Memory still mostly gets graded as a quiz: ask a question, judge the answer, maybe report precision and recall. DolphinBench grades that action after long history, and puts accuracy, cost, and latency on one board. dolphinbench.ai Deshraj Yadav @deshrajdry Today we are releasing DolphinBench: Mapping the Pareto frontier of agent memory. Coding, tool use, and long-context retrieval already have strong public benchmarks. Memory still mostly gets graded as a quiz: ask a question, judge the answer, maybe report precision and recall. A lot of those boards are getting saturated. They also skip the two questions that matter when you pick a memory system to ship: At what cost? At what latency? Agents are not taking memory quizzes but they take actions. The fact that matters is usually missing from the request. Either it shows up in the tool call, or the agent does the wrong thing while sounding fluent. DolphinBench grades that action after long history, and puts accuracy, cost, and latency on one board. dolphinbench.ai 🔗 View Quoted Tweet 💬 3 🔄 4 ❤️ 29 👀 2117 📊 9 ⚡
