huggingface/tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

[view on github]last commit: May 22, 2026

stars

10,761

7d

+9

30d

+90

90d

+215

## star history

## found in

awesome-machine-learning/Rust awesome-machine-learning/Rust Awesome Open Source AI/🧬 1. Core Frameworks & Libraries