arXiv 论文提出 isloss：检测 LLM API 转换中对话信息丢失的确定性测试方法
Testing Conversation Preservation Across LLM APIs: A Rule-Based Oracle for Detecting Silent Information Loss
做 LLM 网关、多模型切换或者翻译层的开发者该看看这篇：翻译后的请求能过 API 校验，但角色、工具调用信息可能已经悄悄丢了。论文给出的 ConvIR 中间表示加规则判定，不用额外调用模型就能测出问题，还附带真实故障率数据。
arXiv 论文 2611.15543v1 提出 isloss，一个用于测试 LLM API 对话信息保留的确定性 oracle。它将源请求与翻译后的请求、回复映射到共享中间表示 ConvIR，再用显式规则区分合法的表示变化与信息丢失，无需参考翻译，也无需额外模型调用。
评估采用基于规范的故障注入、人工审查以及对翻译器、网关和客户端库的实验：在留出测试集上检测出 125 个植入故障中的 95 个（76.0%），误报为 100 次中的 2 次（2.0%）。对五个翻译器代码库六种配置的合成请求套件测试发现，685 个完成翻译中有 127 个（18.5%）通过了目标 API 的静态校验，却在静默改变 prompt、角色或工具信息，而这些信息目标 API 本可保留。回复翻译中，558 个流里有 203 个（36.4%）在至少一个官方 OpenAI SDK 中引发异常或静默损坏。
针对六个模型的实验表明，当任务依赖受影响信息时，翻译引起的变化会改变模型行为。论文将对话信息保留确立为 LLM 互操作、智能体执行与模型评估之外的一项独立正确性要求。
Testing Conversation Preservation Across LLM APIs: A Rule-Based Oracle for Detecting Silent Information Loss
LLM applications increasingly rely on translators and compatibility layers to move conversations across APIs. Yet a translated request can be accepted by the target API while silently losing information about instruction roles, tool interactions, error states, or generation controls. We introduce isloss, a deterministic oracle for testing conversation-information preservation across LLM API translations. It maps source and translated requests and replies to a shared intermediate representation, ConvIR, and applies explicit rules to distinguish legitimate representation changes from information loss. Unlike schema validation or reference-based differential testing, isloss requires neither a reference translation nor additional model calls. It also distinguishes avoidable loss from degradation caused by target API limitations. We evaluate isloss using specification-based fault injection, human review, and experiments on translators, gateways, and client libraries. On held-out tests, it detects 95/125 planted faults (76.0%) with 2/100 false alarms (2.0%). Across a synthetic request suite spanning six configurations of five translator codebases, 127/685 completed translations (18.5%) pass static target checks while silently changing prompt, role, or tool information in ways the target API could have preserved. In translated replies, 203/558 streams (36.4%) cause exceptions or silent corruption in at least one tested official OpenAI SDK. Targeted experiments across six models show that translation-induced changes can alter model behaviour when tasks depend on the affected information. These results establish conversation preservation as a distinct correctness requirement for reliable LLM interoperability, agent execution, and model evaluation.
This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include LLM API, isloss, ConvIR, OpenAI SDK.
Primary source verification: published directly by arXiv: OpenAI with first-party authority. Cross-source corroboration: 11 independent sources have covered this development (including OpenRouter, TestingCatalog, IT之家), confirming industry-wide signal strength.
