gigatoken

Repository facts

Stars

4,129

Forks

236

Open issues and pull requests

20

Primary language

Rust

License

MIT License (MIT)

Owner

marcelroed · Personal account

Created

Nov 10, 2025

Last push

Sep 2, 2026

Topics

• llm
• nlp
• tokenization
• tokenizer

Rank #1,922 of 9,507 in AI on GitHub, by stars. Data from the GitHub API, fetched Oct 18, 2026; star counts change daily. Stars measure popularity, not quality or safety, and a listing is not an endorsement. How this list is made

Related projects

Projects that share this one’s GitHub topics.

• 7,928

  Rank 7,928: cbaziotis/ekphrasis

  Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation…

  673 starsPythonMITLast push Jun 2, 2025

• 216

  Rank 216: explosion/spaCy

  💫 Industrial-strength Natural Language Processing (NLP) in Python

  33,968 starsPythonMITLast push Oct 12, 2026

• 6,723

  Rank 6,723: explosion/spacy-streamlit

  👑 spaCy building blocks and visualizers for Streamlit apps

  863 starsPythonMITLast push Jul 29, 2024

• 7,103

  Rank 7,103: nlp-uoregon/trankit

  Trankit is a Light-Weight Transformer-based Python Toolkit for Multilingual Natural Language Processing

  798 starsPythonApache-2.0Last push Jul 22, 2025

• 9,177

  Rank 9,177: yooper/php-text-analysis

  PHP Text Analysis is a library for performing Information Retrieval (IR) and Natural Language Processing (NLP) tasks using the PHP language

  534 starsPHPMITLast push Dec 28, 2024

• 4,729

  Rank 4,729: roshan-research/hazm

  Persian NLP Toolkit

  1,433 starsPythonMITLast push Apr 1, 2026