想给大模型训练或推理提速？这个HuggingFace分词器速度近千倍的开源Rust工具值得一试。
Gigatoken是一款MIT许可的Rust BPE分词器，在144核AMD EPYC 9565上实现GPT-2分词速度24.53 GB/s。相比HuggingFace分词器快989倍，比tiktoken快681倍，且这些基线本身已用多线程Rust实现。性能提升来自手写SWAR预分词器和预缓存技术，而非更快的BPE合并循环。该工具可大幅加速文本编码流程。
Meet Gigatoken: A Rust BPE Tokenizer that Encodes Text at 24.53 GB/s, up to 989x Faster than HuggingFace Tokenizers
Gigatoken is an MIT-licensed Rust BPE tokenizer that encodes text at gigabytes per second. On a 144-core AMD EPYC 9565, it runs GPT-2 tokenization at 24.53 GB/s. That is 989x HuggingFace tokenizers and 681x tiktoken, against baselines already written in multithreaded Rust. The gains come from a hand-written SWAR pretokenizer and pretoken caching, not a faster BPE merge loop. The post Meet Gigatoken: A Rust BPE Tokenizer that Encodes Text at 24.53 GB/s, up to 989x Faster than HuggingFace Tokenizers appeared first on MarkTechPost .
- Stanford AI Lab07-21 19:21原文