LlamaIndex speeds up OCRia-1 doc parsing 10x on one B200
The practical lesson is that compounding many small, accuracy-checked optimizations can deliver a 10x throughput gain without new hardware, and the details are documented in a public blog post. Inference engineers and teams serving document pipelines can study the writeup to cut their own GPU costs. Since some improvements will land in open-source vLLM, the gains will eventually benefit anyone serving models, not just LlamaParse users.
LlamaIndex's infrastructure team raised OCRia-1's document parsing throughput from 285 tok/s to 2,800 tok/s on a single NVIDIA B200 GPU. The team applied quantization and speculative decoding with dflash2, plus 17 additional optimizations including replaySSM, kernel fusion and vLLM changes. Each change was tested for both speed and accuracy, with some contributing only 2-5% individually. The company plans to contribute some optimizations back to the open-source vLLM project and is applying the work to reduce latency in LlamaParse.
OCRia-1 is one of the fastest models ever for doc parsing. We sped up OCRia-1 by 10x on a single B200 (285 tok/s --> 2800 tok/s). there are a ton of useful learnings in this extremely detailed blog post for anyone looking to optimize models! * we made use of established practices, like quantization and spec decoding (dflash2), to simultaneously reduce memory usage and increase tokens per second * we made 17 other optimizations - replaySSM, kernel fusion, vLLM changes, and more. Huge shoutout to our research team. They've made it possible for us to serve billions of pages at scale. We've continuing to make massive improvements to bring down latency across the board in LlamaParse. Check it out: llamaindex.ai/blog/the-infer… Your browser does not support the video tag. 🔗 View on Twitter LlamaIndex 🦙 @llama_index We got OCRia-1 generating tokens nearly 10× faster on a single B200. Getting there meant making much better use of the same GPU. We tuned how OCRia-1 runs for the demands of document parsing, testing every change for both speed and accuracy. Some improvements were only 2–5%. Seventeen smaller optimizations later, “just a few percent” starts adding up. Our infrastructure team wrote up what worked, how they tested the changes, and what they learned about making document models run faster. We also plan to contribute some of these optimizations back to the open-source vLLM project. Read the full breakdown → llamaindex.ai/blog/the-infer… Watch the video below to see OCRia-1’s token generation before and after the optimizations. Your browser does not support the video tag. 🔗 View on Twitter 🔗 View Quoted Tweet 💬 6 🔄 1 ❤️ 9 👀 1075 📊 6 ⚡
- LlamaIndex16:49原文
