Cachelint Detects Silent Prompt-Caching Bugs in LLM Agent Systems
Silent and Costly: Characterizing and Detecting Prompt-Caching Mis-Integration in LLM Agents
The non-obvious takeaway is that caching bugs produce no errors at all — teams simply pay more per token while assuming caching works. Platform and backend engineers running agent workloads on OpenAI, Anthropic or other provider APIs can run CACHELINT on their own repos before costs accumulate. As an academic tool study with no pricing or regional availability angle, its closest comparison is other static-analysis approaches like CodeQL-based linting, which CACHELINT extends with provider-aware precondition checks.
A new empirical study analysed 120 prompt-caching-related pull requests from 10 popular open-source agent systems, finding that mis-integrations stem from request-side misconfiguration and read-side usage miscount. Most of these bugs fail silently without runtime errors, quietly inflating inference costs and latency. The researchers built CACHELINT, which uses CodeQL to locate caching invocation sites and checks preconditions against a dated provider capability oracle or an LLM guard. It achieved 0.71 precision and 0.78 recall on the 120 labelled cases, and found 32 mis-integrations across 571 agent projects at 0.86 precision in the wild. The work has drawn 21 developer confirmations and 18 upstream fixes, including one in Anthropic's official SDK.
Silent and Costly: Characterizing and Detecting Prompt-Caching Mis-Integration in LLM Agents
Large language model (LLM) services process massive volumes of user requests, resulting in substantial inference costs. To reduce both cost and latency, providers have introduced prompt caching. However, correctly integrating prompt caching is non-trivial due to heterogeneous provider-specific APIs and continuously evolving caching semantics. To understand prompt-caching mis-integration in downstream applications, we conduct the first empirical study of 120 prompt-caching-related pull requests from 10 popular open-source agent systems. Our findings show that prompt-caching mis-integrations arise in both request-side misconfiguration and read-side usage miscount. More importantly, most mis-integrations fail silently without causing observable runtime failures. Motivated by these findings, we design CACHELINT, which models a prompt-caching mis-integration as an invocation site together with the preconditions required for correct cache integration. CACHELINT locates caching invocation sites with CodeQL and discharges each precondition either against a dated provider capability oracle or with an LLM guard check. CACHELINT achieves 0.71 precision and 0.78 recall in identifying the 120 labeled mis-integrations across the 10 studied agent systems. Our evaluation further demonstrates the effectiveness of CACHELINT in real-world settings, identifying 32 mis-integrations across 571 agent projects with 0.86 precision. These findings have received 21 developer confirmations and led to 18 upstream fixes, including one in Anthropic's official SDK.
