UltraBERT-380M 单次前向多选分类,测试准确率最高达 72.1%
UltraBERT-JEV: Encoder models for System One Applications
做分类或路由这类单选任务的开发者可以看看这份报告:同样数据下,UltraBERT-380M 编码器比 Qwen3-0.6B 解码器高近 5 个点,决策还快一倍多。文中 "yes" 减 "no" 打分这个细节最实用,直接能搬到自己的微调流程里。
arXiv 技术报告 UltraBERT-JEV 研究编码器模型在单次前向传播中从多个选项里做选择、不生成文本的用法,比较了微调 UltraBERT-380M 的四种打分与概率化方案,其中一种参考了 DeepSeek-V3 的专家门控。
用与 Qwen3-0.6B 相同的数据训练,Qwen3-0.
6B 的 LLM-as-Jev 配方测试准确率为 67.6%,编码器达到 69.
7% 至 72.1%。
把选项得分从单独的 "yes" 改为 "yes" 减 "no" 带来最大提升,测试集从 69.7% 升到 71.
3%,外部基准从 57.8% 升到 59.
6%,而解码器方案仅 56.1%;
Sigmoid 归一化再加 0.8 个点,square-root softplus 无增益。
NVIDIA A100 上单条请求每次决策 29 毫秒,对比解码器的 68 毫秒,但峰值内存更高(2.8 对 2.
2 GiB)。模型还用 SelfCite 改造的奖励训练的辅助头,在同一次前向中预测支撑答案的句子。
UltraBERT-JEV: Encoder models for System One Applications
We study encoder models that pick one of several options in a single forward pass, without generating text. This technical report compares recipes for fine-tuning UltraBERT-380M with one mask per option. The four recipes differ in how an option is scored and how the scores become probabilities; one follows the expert gate of DeepSeek-V3. Trained on the same data as Qwen3-0.6B with the LLM-as-Jev recipe (67.6% test accuracy), the encoder reaches 69.7% to 72.1%. Scoring an option by "yes" minus "no" instead of "yes" alone gives the larger gain, from 69.7% to 71.3% on the test set and from 57.8% to 59.6% on external benchmarks (56.1% for the decoder). Sigmoid normalisation adds 0.8 test points; a square-root softplus adds none. One request at a time on an NVIDIA A100, the encoder takes 29 ms per decision against 68 ms, but uses more peak memory (2.8 against 2.2 GiB). An auxiliary head, trained on human-marked evidence and on labels selected with a reward adapted from SelfCite, predicts in the same pass the sentences annotated as support for the answer.
This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include UltraBERT, DeepSeek-V3, Qwen3, BERT.
Primary source verification: published directly by arXiv: DeepSeek with first-party authority. Cross-source corroboration: 11 independent sources have covered this development (including AI21 Labs, NVIDIA AI, AWS Machine Learning Blog), confirming industry-wide signal strength.