论文实测：Odia 文本在 GPT-4o 上的 token 消耗是英文的 5 倍
The Indic Token Tax in 2026: Measuring Tokenizer. Disparity and Its Cost for the Languages of India
印度团队实测了 9 个主流 tokenizer 处理 25 种印度语言的开销，Odia 在 GPT-4o 上要多花 5 倍 token，Sarvam 的 tokenizer 差距最小。做多语言客服或按 token 预算的团队，选模型前值得查一下自己的语言在哪个 tokenizer 上最省。
一篇 arXiv 论文量化了主流 tokenizer 在印度语言上的"token 税"：同一内容在 GPT-4o 的 tokenizer 下，Odia 需要 5.01 倍于英文的 token，Hindi 为 1.57 倍，Santali（Ol Chiki 文字）高达 13.7 倍。研究覆盖 9 个公开 tokenizer（含 GPT-4、GPT-4o、Qwen2.5、DeepSeek-V3、Mistral-Nemo 和 4 个 Sarvam 模型），数据来自 FLORES-200 的 25 种印度语言（2,009 句对齐英文的专业译文）和 PHINC 的 13,732 对真人翻译 Hinglish 社交媒体文本。
Sarvam-30B/105B 是唯一在所有语言上都低于 2.6 倍（中位数 1.53）的第三方 tokenizer。按 2026 年 11 月 5 日的官方标价估算，一个每月处理 100,000 次对话的客服机器人，用 GPT-4o 跑英文约 INR 56,640，跑 Marathi 则约 INR 102,829。词内形态切分测试还显示，所有 tokenizer 在形态词边界上的 boundary F1 至多 0.48。
由于商业 API 按 token 计费，这直接意味着非英文文本更贵、推理更慢、固定上下文窗口里能装的文字更少。论文附带的仓库（HavonAI/indic-token-tax）公开了原始 token 计数和固定版本工件，配套论文提出了一个降低该差距的 tokenizer。
The Indic Token Tax in 2026: Measuring Tokenizer. Disparity and Its Cost for the Languages of India
Commercial language model APIs bill per token, so a tokenizer that needs more tokens for the same content in one language makes that language more expensive, slower, and able to fit less text in a fixed context window. We measure this disparity for nine public tokenizers, including those of GPT-4, GPT-4o, Qwen2.5, DeepSeek-V3, Mistral-Nemo and four Sarvam models, on all 25 languages of India available in FLORES-200 (2,009 professionally translated sentences aligned with English) and on 13,732 human-translated Hinglish-English social media pairs from PHINC. On GPT-4o's tokenizer the same content needs between 1.57 times (Hindi) and 5.01 times (Odia) the tokens of English, and 13.7 times for Santali in Ol Chiki script; GPT-4's tokenizer needs up to 12.8 times. Sarvam-30B/105B is the only third-party tokenizer that stays below 2.6 times on every language (median 1.53). In an illustrative scenario using official list prices accessed on 5 November 2026 and stated assumptions, a support bot handling 100,000 chats a month on GPT-4o would cost about INR 56,640 in English and INR 102,829 in Marathi; these are estimates, not measured production costs. Against an openly licensed Hindi and Marathi morphological segmentation set, no tokenizer places more than 35% of its word-internal boundaries on morpheme boundaries (boundary F1 at most 0.48). In a companion paper we present a tokenizer that reduces this tax. Code, pinned artifacts and raw token counts are available at https://github.com/HavonAI/indic-token-tax.
This report is categorized under AITOP's "Papers" desk, focusing on China's AI industry and domestic foundation model ecosystem. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include GPT-4o, tokenizer, Sarvam, 多模态.
Primary source verification: published directly by arXiv: DeepSeek with first-party authority.
