huggingface/tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

[view on github]last commit: Apr 2, 2026

stars

10,589

7d

+17

30d

+38

90d

+92

## star history

## found in

awesome-machine-learning/Rust awesome-machine-learning/Rust Awesome Open Source AI/🧬 1. Core Frameworks & Libraries