Perplexity Optimizes Inference Infrastructure
Perplexity's optimization techniques for embedding models can benefit developers across ASEAN working on similar inference infrastructure challenges.
Perplexity optimized serving for their pplx-embed models over an exabyte-scale search index. They reused kernels from large LLMs for embedding models, achieving similar efficiency. The runtime optimization included lazy CUDA graphs and LazyTensor in Rust. Results showed up to 3x lower p50 and 4.8x lower p99 latency than vLLM on BGE-M3 at 128 tokens using a single H200 GPU.
Join us if you want to work on hard inference and infrastructure engineering problems! Denis Yarats @denisyarats check out our new blog post on how we serve embeddings and rerankers for our SOTA pplx-embed models over an exabyte-scale search index. embedding models have a similar serving profile to LLMs: batch indexing is compute-bound prefill, online serving is memory-bound decode. this lets us reuse the same optimized kernels we built for large LLMs and get great efficiency for free. on top of that, we optimized the runtime: whole-model CUDA graphs captured lazily as the engine serves, and a LazyTensor in Rust that overlaps CPU scheduling with GPU execution. result: up to 3x lower p50 and 4.8x lower p99 latency than vLLM on BGE-M3 at 128 tokens, single H200! if you are interested in working on problems like this, DM me or apply at perplexity.ai/hub/careers 🔗 View Quoted Tweet 💬 12 🔄 2 ❤️ 148 👀 16013 📊 18 ⚡
