精选理由
Gigatoken解决了tokenization的瓶颈，训练前不用再傻等，速度提升非常明显。
斯坦福AI实验室的Tatsunori Hashimoto介绍了Gigatoken，一个超快速的tokenizer。它通过贴近硬件和编写状态机，将tokenization速度提升了数量级。研究者表示再也不需要等待tokenization完成再开始训练。
原文 · Stanford AI Lab
Not having to wait for tokenization before training runs feels like paradigm shift
Not having to wait for tokenization before training runs feels like paradigm shift Tatsunori Hashimoto @tatsu_hashimoto Gigatoken is absurdly fast! Thanks to Marcel, I will (hopefully) never have to wait for tokenization again. It turns out that even mature components like tokenizers have order-of-magnitude improvements left if you get close to the hardware and write state machines. 🔗 View Quoted Tweet 💬 0 🔄 0 ❤️ 10 👀 2509 📊 2 ⚡