text-dedup

Repository facts

Stars

764

Forks

81

Open issues and pull requests

1

Primary language

Python

License

Apache License 2.0 (Apache-2.0)

Owner

ChenghaoMou · Personal account

Created

Mar 13, 2021

Last push

Mar 9, 2026

Topics

• data-processing
• de-duplication
• nlp
• text-processing

Rank #7,290 of 9,507 in AI on GitHub, by stars. Data from the GitHub API, fetched Oct 18, 2026; star counts change daily. Stars measure popularity, not quality or safety, and a listing is not an endorsement. How this list is made

Related projects

Projects that share this one’s GitHub topics.

• 2,281

  Rank 2,281: lynote-ai/humanize-text

  Open-source pipeline for rewriting AI-generated text into natural human prose. Need it in your app? Try the Lynote Humanizer API → lynote.ai/ai-humanizer-api

  3,448 starsPythonMITLast push Oct 13, 2026

• 4,729

  Rank 4,729: roshan-research/hazm

  Persian NLP Toolkit

  1,433 starsPythonMITLast push Apr 1, 2026

• 7,928

  Rank 7,928: cbaziotis/ekphrasis

  Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation…

  673 starsPythonMITLast push Jun 2, 2025

• 4,472

  Rank 4,472: allenai/dolma

  Data and tools for generating and inspecting OLMo pre-training data.

  1,559 starsPythonApache-2.0Last push Aug 24, 2026

• 4,909

  Rank 4,909: pemistahl/lingua-go

  The most accurate natural language detection library for Go, suitable for short text and mixed-language text

  1,367 starsGoApache-2.0Last push Feb 6, 2025

• 7,768

  Rank 7,768: abadojack/whatlanggo

  Natural language detection library for Go

  695 starsGoMITLast push Mar 28, 2023