这篇论文发现，把文档当图片直接预训练，比先转成纯文本再训练效果更好，感兴趣的可以看看实验细节。
这篇论文挑战语言模型必须依赖纯文本预训练的假设，系统性地比较了无监督视觉预训练与纯文本预训练在5个骨干网络和多项基准上的表现。实验表明，基于包含图形、公式和版式的视觉文档进行预训练，在相同语料下持续优于纯文本预训练，提升幅度在1.2%-4.7%之间。该方法不依赖文本提取，直接利用视觉特征学习语言智能，为大规模预训练提供了更高效的路径。
Scalable Visual Pretraining for Language Intelligence
The rapid progress of large foundation models has been driven predominantly by pretraining on large-scale text corpora. However, many forms of knowledge are conveyed through visual representations, where figures, typeset equations, and page layouts carry rich information that cannot be faithfully or completely captured by text alone. Yet current pretraining approaches discard these visual cues by converting visually rich sources, such as documents and web pages, into plain text for learning language intelligence. This paper challenges the default assumption that language models must be trained on text-only representations and shows that Visual Pretraining is a scalable learner for foundation model intelligence. To this end, we conduct a systematic study of unsupervised visual pretraining paradigms that directly leverage visual documents without text extraction. Across multiple backbones and benchmarks, visual pretraining on the same underlying corpora consistently outperforms text-only pretraining, offering an efficient pathway to scalable language intelligence.