huggingface/tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

[view on github]last commit: Apr 15, 2026

stars

10,633

7d

+27

30d

+71

90d

+112

## star history

## found in

awesome-machine-learning/Rust awesome-machine-learning/Rust Awesome Open Source AI/🧬 1. Core Frameworks & Libraries