Meta 和 OpenAI 的推理成本计算方式不同，友利AI开源模型游戏成本降低5.5倍

Notes: - Stream 200 tokens to a user, then lose the GPU to maintenance, and you cannot just retry. ...

Notes: - Stream 200 tokens to a user, then lose the GPU to maintenance, and you cannot just retry. Meta measures inference by cost per successful task, not per token. - OpenAI's inference load balancer used to set its routing weights from a feedback loop. Shift traffic off a hot engine and it cools, the controller reads cool as spare capacity, and the traffic comes back. An optimizer with a view of every engine replaced it. - The same coding agent built the same tower defense game on a closed frontier model and on an open weight model. Both games worked. FriendliAI puts the open weight run at about five and a half times cheaper. - Superlinked's automated tuning loop produced a fine tune that cost 80 cents to train and lifted retrieval on German legal text by 18 percent, as a proof of concept. - In Red Hat's demo a first request took about three seconds. The next turn reused the same system prompt, hit the cache on the same machine, and took about one. A new system prompt on a different machine paid the full three again. Inference Engineering playlist: youtube.com/watch?v=7c9FSU… 💬 0 🔄 0 ❤️ 0 👀 303 ⚡