tiktoken
Project record
tiktoken is OpenAI's open-source byte pair encoding (BPE) tokenizer for use with OpenAI's models. It converts text to the token numbers a model sees and back, using named encodings such as o200k_base and cl100k_base, and can return the encoding that corresponds to a given OpenAI model name. It also includes an educational submodule for training and visualizing a simple BPE tokenizer, and a plugin mechanism for registering other encodings.14
- Repository: Repository (external site: github.com)
- License: LICENSE (MIT) (external site: github.com)
- Website: Package on PyPI (external site: pypi.org)
- Release notes: Changelog (external site: github.com)
Availability and license
Overall availability
Source is public on GitHub and prebuilt packages are on PyPI. The encoding files are not in the repository; the library downloads them, when first used, from the openaipublic.blob.core.windows.net addresses named in its code.174
Availability is separate from permission: read the license before using or redistributing.
Component reuse rights
- weights
- Unknown — no complete fact-level rights review
- code
- Reviewed qualifying license recorded — check scope and conditions
- data
- Unknown — no complete fact-level rights review
- documentation
- Unknown — no complete fact-level rights review
No complete system-rights review is recorded for this release.
Public materials checklist
| Item | Status | Notes and evidence |
|---|---|---|
| Source codeIs the source code publicly readable? | Public | Python package with a Rust core built through setuptools-rust; the encoding definitions are in tiktoken_ext/openai_public.py.34 |
| DocumentationIs user documentation published? | Public | The README covers basic use, BPE background, and extending tiktoken, and says the API is documented in tiktoken/core.py; it links an OpenAI Cookbook notebook for examples.1 |
| InstallationAre installation instructions or packages publicly available? | Public | Installable from PyPI with pip; requires Python 3.9 or later.17 |
| Supported platformsAre supported operating systems or hardware documented? | Public | PyPI provides wheels for macOS (x86_64 and arm64), Linux (manylinux and musllinux, x86_64 and aarch64), and Windows (amd64), and a source distribution.73 |
| Release statusAre versioned releases published? | Public | Versioned PyPI releases with a changelog; 0.14.0 was uploaded on 2026-08-17.76 |
What it is useful for
Counting or inspecting the tokens that a piece of text becomes for an OpenAI model, and learning how BPE tokenization works. The README links an OpenAI Cookbook notebook on counting tokens with tiktoken.1
Run and use notes
Organization context
U.S. eligibility
Eligible · basis: U.S. headquarters
The repository is in OpenAI's GitHub organization and its license names OpenAI as a copyright holder. OpenAI's operating company, OpenAI Group PBC, is described as a Delaware public benefit corporation at an address in San Francisco, California, in a February 2026 agreement filed with the SEC.128
Sources
This listing is not an endorsement, a safety assessment, or a federal approval.