Marcel Rød 搞了个 Gigatoken，比 HuggingFace 快上千倍，比 tiktoken 快百倍，跑 tokenizer 的抓紧试试。
Marcel Rød 发布了 Gigatoken，自称是世界上最快的 tokenizer 实现。该实现比 HuggingFace 快 500-1000 倍，比 OpenAI 的 tiktoken 快约 100 倍，而这些基线已经是多线程 Rust 实现。Gigatoken 适用于大多数 tokenizer 定义和机器。Marcel 也是斯坦福 CS 336 课程（从头实现 LLM）的合著者。
@marcelroed casually just dropped the world’s fastest tokenizer implementation! He's also co-author ...
@marcelroed casually just dropped the world’s fastest tokenizer implementation! He's also co-author for Stanford’s CS 336 (LLMs from Scratch), where implementing a tokenizer from scratch is the first assignment. Fitting! 💪 Marcel Rød @marcelroed Introducing the world's fastest tokenizer implementation, Gigatoken! Gigatoken is ~500-1000x faster than HuggingFace, and ~100x faster than OpenAI's tiktoken for most tokenizer definitions on most machines. These baselines are already multithreaded Rust implementations! 🧵 🔗 View Quoted Tweet 💬 1 🔄 0 ❤️ 3 👀 734 📊 2 ⚡