Jev-Mem speeds long-conversation memory building 6.6x
Swapping LLM calls for a lightweight classifier in memory pipelines cuts latency by an order of magnitude without losing answer quality, a design pattern applicable to any RAG or agent system that overuses the LLM for trivial choices. Builders working on chat memory, retrieval routing or stopping criteria can replicate this by scoring candidates in batches and reserving the LLM for final generation. The numbers also set a concrete benchmark point against LoCoMo baselines.
A University of Texas paper introduces Jev-Mem, a memory system that hands small routing decisions to Jev, a fast scoring model that returns batched probabilities instead of generating text. The LLM is only called at the end to read selected memories and produce the answer. On the LoCoMo long-conversation benchmark, Jev-Mem scored 0.777 versus 0.700 for the best baseline. It built memory in 158 seconds versus 1,044 seconds for the fastest rival and answered in 0.93 seconds versus 1.47 seconds.
University of Texas paper finds that handing small memory decisions to a fast scoring model let an AI memory system build its memory of long conversations 6.6 times faster
Many memory systems ask an LLM to make small choices, like what to link or when to stop searching.
Generating text for each choice is slow, and those calls pile up during memory building and search.
Jev-Mem gives those choices to a fast scorer, Jev, that returns probabilities in batches instead of text. The LLM is called only at the end, to read the chosen memories and answer.
On the LoCoMo long-conversation benchmark, it got a 0.777 answer score versus 0.700 for the best baseline. It built memory in 158 seconds versus 1,044 for the fastest rival and answered in 0.93 seconds versus 1.47.
If your memory pipeline calls an LLM for routing, scoring, or stopping, try a fast classifier there and save the LLM for answers.
