Perplexity优化嵌入模型推理性能
Join us if you want to work on hard inference and infrastructure engineering problems!
Perplexity分享嵌入模型部署技巧，用LLM优化内核实现3倍性能提升，对AI工程师很有参考价值。
Perplexity分享了其pplx-embed模型的部署经验，通过复用LLM优化内核实现高效服务。他们在单H200上实现了比vLLM低3倍p50和4.8倍p99延迟，使用128 token的BGE-M3基准测试。系统采用批量索引计算绑定预填充，在线服务内存绑定解码，并优化了运行时性能。
Join us if you want to work on hard inference and infrastructure engineering problems!
Join us if you want to work on hard inference and infrastructure engineering problems! Denis Yarats @denisyarats check out our new blog post on how we serve embeddings and rerankers for our SOTA pplx-embed models over an exabyte-scale search index. embedding models have a similar serving profile to LLMs: batch indexing is compute-bound prefill, online serving is memory-bound decode. this lets us reuse the same optimized kernels we built for large LLMs and get great efficiency for free. on top of that, we optimized the runtime: whole-model CUDA graphs captured lazily as the engine serves, and a LazyTensor in Rust that overlaps CPU scheduling with GPU execution. result: up to 3x lower p50 and 4.8x lower p99 latency than vLLM on BGE-M3 at 128 tokens, single H200! if you are interested in working on problems like this, DM me or apply at perplexity.ai/hub/careers 🔗 View Quoted Tweet 💬 12 🔄 2 ❤️ 148 👀 16013 📊 18 ⚡
