马拉地语有8300万使用者但标注数据稀缺,这个新数据集和MahaBERT模型基准很实用,适合做低资源语言NLP的朋友参考。
L3Cube-MahaPOS是首个大型人工标注的马拉地语词性标注数据集,包含32,354句新闻文本,遵循16标签Universal Dependencies方案。研究在HMM、CRF、BiLSTM、BiLSTM+CharCNN、MuRIL和MahaBERT-v2六类模型上进行了基准测试。最佳模型MahaBERT-v2达到88.67%的token级准确率和81.67%的宏F1分数。该数据集和模型检查点已开源,可助力马拉地语NLP研究。
L3Cube-MahaPOS: A Marathi Part-of-Speech Tagging Dataset and BERT Models
Part-of-Speech (POS) tagging is a foundational NLP task underpinning machine translation, information extraction, and syntactic parsing. Despite Marathi being spoken by over 83 million people and ranking among the top twenty most spoken languages worldwide, it remains severely under-resourced in annotated corpora and standardised evaluation benchmarks. Marathi presents unique challenges for computational modelling owing to its rich morphology, relatively free word order, lack of capitalisation conventions, and pervasive code-mixing with Hindi and English. We introduce L3Cube-MahaPOS, a gold-standard POS tagging dataset for Marathi comprising 32,354 manually annotated sentences drawn from news text. Annotation was performed entirely manually by a team of Marathi-proficient annotators following a 16-tag Universal Dependencies-aligned scheme. A structured preprocessing pipeline covering Unicode normalisation, Devanagari-aware tokenisation, and noise filtering ensures label consistency across all splits. We benchmark the dataset across six model families spanning HMM, CRF, BiLSTM, BiLSTM+CharCNN, MuRIL, and the Marathi-specific transformer MahaBERT-v2. The best system achieves 88.67\% token-level accuracy and a macro-F1 of 81.67% over 15 evaluated tag classes. We release the dataset, annotation guidelines, and trained model checkpoints to foster further research in Marathi NLP.