同一段话，泰文要花英文 17 倍的 token，用 GPT 类 API 的人可以看看自己语言有没有被多收费。
Rest of World 报道称，主流 AI 模型处理非拉丁字母文本最多需要 17 倍于英文的 token 数。这意味着泰文、缅甸文等语言的用户在调用 API 或使用订阅服务时，成本与配额消耗远高于英语用户，形成所谓的 tokenizer tax。同一句话按 token 计费时，不同文字系统之间的价格差异可以非常大。
AI models can take up to 17 times more tokens to process non-Latin scripts than English, saddling users with a steep "tokenizer tax" https://t.co/eBEuMBLJX2 https://t.co/CZWqTv6eFf
Formal verification of the chat-template path is a new angle on a failure mode most teams debug only after RL runs diverge. Practitioners running RL fine-tuning pipelines can compare this Lean-plus-Rust approach against their Jinja-based message construction to see where mismatch creeps in. No benchmark numbers are reported yet, so it functions as a prototype reference rather than a drop-in replacement.
用泰文这类语言跑模型，token 消耗可能是英文的 17 倍，账单直接翻好几倍，用非英文语言的值得看看这篇。
一篇挺硬核的表示学习论文：作者把 tokenizer 和生成模型放一起算 flop 效率，结果比 DINO 和 LeJEPA 更抗塌缩，还给出了 scaling 规律。
This research reveals a significant economic disadvantage for French speakers in AI services, effectively creating a language tax that compounds with agentic use. The Baracoda FR prototype offers a potential solution, though its performance is currently limited to French and English. Developers working with multilingual applications should consider these tokenization inefficiencies when designing systems for diverse linguistic markets.
