huggingface/tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

[view on github]last commit: Jun 18, 2026

stars

10,832

7d

+20

30d

+76

90d

+264

## star history

## found in

awesome-machine-learning/Rust awesome-machine-learning/Rust Awesome Open Source AI/🧬 1. Core Frameworks & Libraries