tokenizers

Repository facts

Stars

11,183

Forks

1,226

Open issues and pull requests

198

Primary language

Rust

License

Apache License 2.0 (Apache-2.0)

Owner

huggingface · Organization

Created

Nov 1, 2019

Last push

Oct 18, 2026

Topics

• bert
• gpt
• language-model
• natural-language-processing
• natural-language-understanding
• nlp
• transformers

Rank #716 of 9,507 in AI on GitHub, by stars. Data from the GitHub API, fetched Oct 18, 2026; star counts change daily. Stars measure popularity, not quality or safety, and a listing is not an endorsement. How this list is made

Related projects

Projects that share this one’s GitHub topics.

• 4,789

  Rank 4,789: explosion/spacy-transformers

  🛸 Use pretrained transformers like BERT, XLNet and GPT-2 in spaCy

  1,413 starsPythonMITLast push Mar 27, 2026

• 4,772

  Rank 4,772: SKTBrain/KoBERT

  Korean BERT pre-trained cased (KoBERT)

  1,419 starsPythonApache-2.0Last push Jun 14, 2025

• 7,577

  Rank 7,577: ymcui/MacBERT

  Revisiting Pre-trained Models for Chinese Natural Language Processing (MacBERT)

  722 starsApache-2.0Last push Apr 19, 2026

• 3,890

  Rank 3,890: nlpodyssey/spago

  Self-contained Machine Learning and Natural Language Processing library in Go

  1,852 starsGoBSD-2-ClauseLast push Apr 1, 2025

• 7,463

  Rank 7,463: primeqa/primeqa

  The prime repository for state-of-the-art Multilingual Question Answering research and development.

  739 starsPythonApache-2.0Last push Sep 18, 2025

• 998

  Rank 998: jessevig/bertviz

  BertViz: Visualize Attention in Transformer Models

  8,200 starsPythonApache-2.0Last push Jan 8, 2026