OpenAI没换硬件，靠软件就把推理成本砍了一半。这对模型成本和用户体验可能都有影响，值得了解。
OpenAI近期通过纯软件优化将推理成本降低了约50%，工程师利用更好的批处理和KV-cache管理提升了GPU利用率。该优化已部署到ChatGPT的登录用户流量中，GPU需求降至仅数百块。部分人猜测这是通过量化实现，并可能关联到近期ChatGPT质量下降的投诉。
OpenAI just halved inference costs using software …
OpenAI just halved inference costs using software alone. Engineers shipped new optimisations this month that cut compute spend by around 50% on certain models - no new hardware required.
Rolled out to logged-out ChatGPT traffic, GPU needs fell to just a few hundred. That is the highest-volume segment, now squeezed into a fraction of its previous compute footprint. The win is pure software - tighter GPU utilisation via batching, smarter KV-cache management, and similar tricks.
Reactions are split. Some see the profitability timeline pulling forward. Others connect it to the recent wave of ChatGPT quality complaints. Plenty are betting it is quantisation - and blaming it for why “5.5 became catastrophically stupid”.
The timing is the point: the software savings landed before OpenAI’s custom Jalapeño chip with Broadcom even ships. https://t.co/tosp5X2s3U