结构性沉默：AI基础设施如何忽视孟加拉语等低资源语言
Structural Silence: When AI Infrastructure Fails Speakers of Underrepresented Languages
这篇论文讲孟加拉语在AI基建里的吃亏，67:1的token差距和36.5%的农村联网率是硬数据。
论文以孟加拉语为例，分析AI教育工具在低连接环境中的结构性障碍。孟加拉语占全球人口近4%，但全球网页内容占比不足0.
5%。主要多语语料库中，英语与孟加拉语的训练token比例达67:1。
孟加拉语的元音附标文字还带来额外的token化惩罚，加剧数据短缺。农村地区个人互联网普及率为36.
5%，城市为71.4%，形成连接排斥。
Structural Silence: When AI Infrastructure Fails Speakers of Underrepresented Languages
Artificial intelligence tools for education and language support are increasingly framed as scalable responses to access gaps in under-resourced communities. Yet the infrastructure underlying these tools, including training corpora, tokenization schemes, evaluation benchmarks, and deployment architectures, can systematically disadvantage speakers of underrepresented languages before a model is trained. This paper examines these structural barriers through Bengali, one of the world's most widely spoken languages, focusing on AI-assisted education in low-connectivity environments. We identify four interlocking failures: a severe web presence gap, with Bengali accounting for less than 0.5% of global web content despite representing nearly 4% of the global population; a 67:1 training-token deficit between English and Bengali in major multilingual corpora; a tokenization penalty associated with Bengali's alphasyllabary script that compounds the data deficit through higher token fertility; and connectivity exclusion, with individual internet penetration at 36.5% in rural areas compared with 71.4% in urban areas. These failures reflect longstanding resource-allocation decisions, institutional priorities, and design defaults that did not center underrepresented languages in mainstream AI development. We argue that dataset scarcity should be understood as a structural barrier rather than an isolated technical limitation, and that offline-first design should be treated as an equity-oriented infrastructure strategy. We conclude with directions for linguistics and AI research aimed at reducing these structural inequalities.
This report is categorized under AITOP's "Papers" desk. Key entities and technical topics under continuous tracking include 孟加拉语, 低资源语言, tokenization, 多语语料库.
Source attribution: curated from arXiv cs.AI (@Avijit Roy, Proma Roy).