这篇论文讲孟加拉语在AI基建里的吃亏,67:1的token差距和36.5%的农村联网率是硬数据。
论文以孟加拉语为例,分析AI教育工具在低连接环境中的结构性障碍。孟加拉语占全球人口近4%,但全球网页内容占比不足0.5%。主要多语语料库中,英语与孟加拉语的训练token比例达67:1。孟加拉语的元音附标文字还带来额外的token化惩罚,加剧数据短缺。农村地区个人互联网普及率为36.5%,城市为71.4%,形成连接排斥。
Structural Silence: When AI Infrastructure Fails Speakers of Underrepresented Languages
Artificial intelligence tools for education and language support are increasingly framed as scalable responses to access gaps in under-resourced communities. Yet the infrastructure underlying these tools, including training corpora, tokenization schemes, evaluation benchmarks, and deployment architectures, can systematically disadvantage speakers of underrepresented languages before a model is trained. This paper examines these structural barriers through Bengali, one of the world's most widely spoken languages, focusing on AI-assisted education in low-connectivity environments. We identify four interlocking failures: a severe web presence gap, with Bengali accounting for less than 0.5% of global web content despite representing nearly 4% of the global population; a 67:1 training-token deficit between English and Bengali in major multilingual corpora; a tokenization penalty associated with Bengali's alphasyllabary script that compounds the data deficit through higher token fertility; and connectivity exclusion, with individual internet penetration at 36.5% in rural areas compared with 71.4% in urban areas. These failures reflect longstanding resource-allocation decisions, institutional priorities, and design defaults that did not center underrepresented languages in mainstream AI development. We argue that dataset scarcity should be understood as a structural barrier rather than an isolated technical limitation, and that offline-first design should be treated as an equity-oriented infrastructure strategy. We conclude with directions for linguistics and AI research aimed at reducing these structural inequalities.