研究人员用CLEVER框架测试了7种缓存淘汰策略，发现LFU在大多数情况下表现最好，而其他策略效果差距不大。他们还发现，实际可用的缓存命中率远低于理论值，因为很多命中答案不可替换。这个研究对优化LLM缓存很有参考价值。
该研究使用CLEVER框架，在三个查询语料库、三种缓存容量和两种编码器上，评估了FIFO、LRU、LFU、ARC、GDSF、SISO和语义冗余等七种缓存淘汰策略。在18种设置中，没有任何策略的命中率能超过LFU超过0.041个百分点。在容量紧张时，FIFO和流式SISO的命中率比LFU低最多8.67和8.55个百分点。研究指出，在精确查找和插入缺失项的协议下，新插入的条目无法在其命中半径内有驻留邻居，因此几何感知的淘汰规则收到的冗余信号很少。审计发现，在MiniLM的中位最近邻阈值下，只有2.1-3.9%的命中被认为是答案可替换的，将原始51-60%的命中率降至质量调整后的1.1-2.2%。跨编码器研究还表明，阈值在不同嵌入模型之间无法转移。LFU是该协议下最强大的简单默认选择。
Which Eviction Policy Should an LLM Cache Use? A Systematic Study Across Workloads, Capacities, and Encoders
Semantic caches reuse an LLM response when the incoming query embedding lies near a cached query, but proposed eviction policies have rarely been compared under one protocol. Using CLEVER, we evaluate FIFO, LRU, LFU, ARC, GDSF, a single-pass streaming adaptation of SISO, and a semantic-redundancy policy across three ordered, deduplicated query corpora, three cache capacities, and two encoders. No evaluated policy improves on LFU by more than 0.041 percentage points in any of the eighteen settings. Replacement is not irrelevant: FIFO and streaming SISO trail LFU by as much as 8.67 and 8.55 points, respectively, at tight capacity. We explain the missing upside with a conditional packing result. Under exact lookup and insert-on-miss, a newly inserted entry cannot have a resident neighbor within the hit radius, so a geometry-aware eviction rule receives little new redundancy signal. A separate audit exposes a larger problem with the evaluated operating point. At MiniLM's median nearest-neighbor threshold, only 2.1-3.9% of sampled LMSYS and QQP hits are judged answer-substitutable, reducing raw hit rates of 51-60% to quality-adjusted rates of 1.1-2.2%. The cross-encoder study further shows that thresholds do not transfer between embedding models. LFU is the strongest simple default in this protocol; deployment decisions should first establish answer validity and then test sub-point policy differences with exact search.