huggingface/tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

[view on github]last commit: Oct 19, 2026

stars

11,187

7d

+19

30d

+137

90d

+215

## star history

## found in

awesome-machine-learning/Rust awesome-machine-learning/Rust Awesome Open Source AI/🧬 1. Core Frameworks & Libraries