Event ·Official first-hand

Gigatoken: Rust BPE分词器，编码速度达24.53 GB/s，比HuggingFace快989倍

Gigatoken是一款MIT许可的Rust BPE分词器，在144核AMD EPYC 9565上实现GPT-2分词速度24.53 GB/s。相比HuggingFace分词器快989倍，比tiktoken快681倍，且这些基线本身已用多线程Rust实现。性能提升来自手写SWAR预分词器和预缓存技术，而非更快的BPE合并循环。该工具可大幅加速文本编码流程。

Current read

想给大模型训练或推理提速？这个HuggingFace分词器速度近千倍的开源Rust工具值得一试。

5 sources62° AI heatLast updated 2026/7/23 08:01:22

Evidence

5 sources

Primary sourcemarktechpost

2026/7/23 08:01:22View source

Related source 1Stanford AI Lab

2026/7/21 19:21:59View source

Related source 2IT之家

2026/7/24 06:02:51View source

Related source 3LlamaIndex

2026/7/24 17:30:07View source

Related source 4Geek

2026/7/25 03:45:00View source

Conflict check

Dedup data only shows that these sources discuss the same event, not that they agree. There is no structured supporting or refuting evidence yet, so no conflict is inferred automatically.

Open the story