想降低LLM智能体长会话的推理成本？看看TokenPilot，它通过智能管理上下文缓存，在三个基准上省了61%-87%的费用，性能还不掉队。
TokenPilot提出了一种双粒度上下文管理框架，通过Ingestion-Aware Compaction稳定提示前缀并消除环境噪声，以及Lifecycle-Aware Eviction监控上下文段残余效用。在PinchBench和Claw-Eval基准测试中，TokenPilot在孤立模式下分别降低61%和56%的成本，连续模式下降低61%和87%，同时保持与先前系统相当的性能。该框架已集成到LightMem2中，可访问https://github.com/zjunlp/LightMem2。
TokenPilot: Cache-Efficient Context Management for LLM Agents
As LLM agents are deployed in long-horizon sessions, context accumulation drives up inference costs. Existing approaches utilize text pruning or dynamic memory eviction to minimize token footprints; however, their unconstrained sequence mutations alter layouts, introducing prefix mismatches and cache invalidation. This reveals a critical trade-off between text sparsity and prompt cache continuity. To address this, we present TokenPilot, a dual-granularity context management framework. Globally, Ingestion-Aware Compaction acts as a framework harness to stabilize prompt prefixes and eliminate open-world environmental noise at the ingestion gate. Locally, Lifecycle-Aware Eviction monitors the ongoing residual utility of context segments, enforcing a conservative batch-turn schedule to offload content segments only when task relevance expires. Experiments on PinchBench and Claw-Eval under both isolated and continuous modes demonstrate that TokenPilot reduces costs by 61% and 56% in isolated mode, and 61% and 87% in continuous mode, while maintaining competitive performance compared to prior systems. TokenPilot has been integrated into LightMem2 at https://github.com/zjunlp/LightMem2.