OpenAI推出Ultrafast实现每秒300tokens
精选理由
OpenAI用NVIDIA GPU实现了每秒300tokens的Ultrafast，和Cerebras的低延迟合作形成有趣对比。
OpenAI宣布推出Ultrafast，可实现每秒300tokens的处理速度。根据SemiAnalysis分析，该服务运行在NVIDIA GPU上。OpenAI采用低批量大小策略，优先处理单个请求的速度。这与OpenAI和Cerebras专门针对超低延迟推理的合作形成对比。
原文 · kimmonismus
OpenAI announced up to 300 tokens per second with Ultrafast. According to SemiAnalysis, it’s running on NVIDIA GPUs. That’s particularly interesting given its partnership with Cerebras specifically for ultra-low-latency inference.
According to SemiAnalysis, OpenAI is using NVIDIA GPUs at a low batch size, prioritizing speed for individual requests.
