CheMatE让模型同时懂SMILES和化学文献，两阶段训练后在分子性质预测和科学理解任务上都不输基线。
CheMatE是一个基于ModernBERT的化学嵌入模型，在同一个向量空间中联合表示SMILES分子结构和化学领域自然语言。训练分两步：先在大规模SMILES标注科学文档上继续掩码语言建模，语料来自FineWeb和ChemPile，分别含10.4B和11.5B token；再用Multiple Negative Ranking Loss进行Matryoshka对比学习。模型在分子性质预测和科学语言理解任务上测试，效果与专用化学模型和通用语言模型基线相当。
Bi-semantic Chemical Embedder for Joint Representation Learning of SMILES and Natural Language
Transformer models have revolutionized natural language processing (NLP), and text-based molecular representations like SMILES have successfully extended these architectures to chemistry. However, domain-adaptive pre-training often causes models to overfit to chemical syntax, catastrophically forgetting their foundational semantic capabilities. To address this challenge, we introduce CheMatE, a chemistry-oriented embedding model that jointly captures molecular structure and domain-specific natural language within the same representation space. Built on a ModernBERT backbone, CheMatE learns bi-semantic representations through a two-stage training procedure: continued masked language modeling (MLM) followed by a Matryoshka contrastive learning stage via Multiple Negative Ranking Loss (MNRL). First, we train the model using MLM on a novel, large-scale corpus of SMILES-annotated, long-context scientific documents that were constructed and curated from FineWeb and ChemPile (comprising 10.4B and 11.5B tokens, respectively). Subsequently, the model undergoes contrastive learning using a synthetic dataset of SMILES-text pairs algorithmically derived from our original training corpus. This design exposes the model to SMILES-enriched scientific literature, enabling bi-semantic understanding. We evaluate CheMatE across a range of downstream tasks covering molecular property prediction and scientific language understanding. Our results demonstrate that coupling our custom-curated datasets with this sequential training strategy yields robust, highly transferable representations. By effectively unifying structural and contextual signals within a single text-based framework, CheMatE achieves competitive performance across both specialized chemistry models and general-purpose language model baselines.