AI模型精选

Diffusion Gemma 速度4倍提升但事实错误多6倍

Diffusion Gemma is 4x faster, but makes 6x more mi…

精选理由

想用更快的推理速度就得接受更多幻觉,Google官方也为此打预防针了。

AI 摘要

Diffusion Gemma 在单个H100(FP8)上速度达763 tok/s,比Gemma 4的218 tok/s快约4倍。但事实准确性测试中,Diffusion Gemma 33个事实正确、28个错误,而Gemma 4为45正确、5错误。话题越冷门错误越多:乔布斯传4错、俄罗斯方块12错、BeOS故事12错。Diffusion Gemma胡编了乔布斯的母亲名字和游戏同事名称,并将BeBox价格虚构为$9,999(实际$1,600)。

原文 · @atomic_chat_hq

Diffusion Gemma is 4x faster, but makes 6x more mi…

Diffusion Gemma is 4x faster, but makes 6x more mistakes!

We benchmarked the new diffusion LLM against its autoregressive twin on a single H100 (FP8). We gave each the same three tasks: write a Steve Jobs biography, the history of Tetris, and the story of BeOS - every next topic less popular than the previous one. Then we fact-checked every claim in every answer.

Gemma4 got 45 facts right, 5 wrong. DiffusionGemma got 33 right, 28 wrong. The less popular the topic, the worse it got: 4 mistakes on Jobs, 12 on Tetris, 12 on BeOS. It named Clara Clley as Steve Jobs' mother, invented a colleague for Pajitnov named Geri Gulovik and priced the BeBox at $9,999. The real one cost $1,600.

Outputs: Gemma4 26B A4B: 218 tok/s · 15.1s total · 45 facts · 5 mistakes DiffusionGemma 26B A4B: 763 tok/s · 3.7s total · 33 facts · 28 mistakes

The reason is simple. DiffusionGemma throws 256 tokens on the screen at once and polishes them pass after pass until the text sounds smooth. Smooth is all it cares about: a fake name, date or number sounds just as smooth as a real one, so it stays. Regular Gemma4 meanwhile writes one word at a time and checks every new word against everything before it. Google says it themselves in the launch post: quality is lower, use regular Gemma 4 when facts matter.