DolphinBench grades agent memory by action, cost and latency
Memory benchmarks that judge answers rather than actions reward fluency while hiding real shipping trade-offs; DolphinBench's three-axis scoring makes cost-versus-latency comparisons visible when choosing between memory backends. Builders picking a memory layer for production agents can use it to weigh accuracy against infrastructure spend and response time. It sits alongside existing public boards for coding and long-context retrieval rather than replacing them.
Deshraj Yadav and Taranjeet launched DolphinBench, a benchmark for agent memory systems, at dolphinbench.ai. Instead of grading memory as a question-answer quiz, it evaluates whether an agent takes the correct action after processing a long interaction history. The benchmark reports accuracy, cost and latency together on a single board to map the Pareto frontier of memory systems. It targets the gap left by existing benchmarks for coding, tool use and long-context retrieval, which the authors say are saturating and omit cost and latency figures.
Launching DolphinBench - agent memory benchmark that grades action after long history and measures accuracy, cost and latency on one board. Deshraj Yadav @deshrajdry Today we are releasing DolphinBench: Mapping the Pareto frontier of agent memory. Coding, tool use, and long-context retrieval already have strong public benchmarks. Memory still mostly gets graded as a quiz: ask a question, judge the answer, maybe report precision and recall. A lot of those boards are getting saturated. They also skip the two questions that matter when you pick a memory system to ship: At what cost? At what latency? Agents are not taking memory quizzes but they take actions. The fact that matters is usually missing from the request. Either it shows up in the tool call, or the agent does the wrong thing while sounding fluent. DolphinBench grades that action after long history, and puts accuracy, cost, and latency on one board. dolphinbench.ai 🔗 View Quoted Tweet 💬 3 🔄 4 ❤️ 23 👀 1174 📊 6 ⚡
