Graph Memory Evaluation for Long-term Agents
The paper provides concrete evidence on memory system performance for AI agents, useful for developers building long-term memory solutions.
Researchers tested graph memory against flat retrieval on LongMemEval benchmark. The graph approach achieved token F1 of 0.42 compared to 0.47 for flat vector baseline. A paired bootstrap over 500 questions showed a gap of -0.050 (95% CI -0.085 to -0.016). The forgetting module performed better, pruning 9.8% of nodes from a 27,021-node graph without affecting token F1.
Finally, a good paper testing if graph memory actually beats flat retrieval for long-term agents. (bookmark this one) Researchers extract each conversational turn into typed nodes and attributed edges, answer from a two-hop subgraph, and hold the candidate-generation budget fixed at five retrieval roots. On LongMemEval the graph gets token F1 0.42 against 0.47 for a flat vector baseline, and a paired bootstrap over 500 questions puts the gap at -0.050 (95% CI -0.085 to -0.016). The damage concentrates on questions that require recalling a specific prior assistant turn, where judged correctness falls from 0.911 to 0.607. Splitting a turn into entities discards the surface form those questions depend on. The forgetting module fares much better. One pruning pass over a persistent 27,021-node graph, scored on recency, access frequency, degree centrality and age, removes 9.8% of nodes and 9.5% of stored bytes with token F1 unchanged. Paper: arxiv.org/abs/2608.28978 Chat with Paper: academy.dair.ai/papers/selecti… 💬 0 🔄 0 ❤️ 2 👀 727 📊 1 ⚡
- arXiv cs.LG09-01 16:38原文
