petastorm
by uber (organization)
Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code. (description from the project’s GitHub page)
Repository facts
- Stars
- 1,895
- Forks
- 286
- Open issues and pull requests
- 184
- Primary language
- Python
- License
- Apache License 2.0 (Apache-2.0)
- Owner
- uber · Organization
- Created
- Jun 15, 2018
- Last push
- Jan 2, 2026
Topics
Related projects
Projects that share this one’s GitHub topics.
Rank 2,274: lakehq/sail
Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.
3,455 starsRustApache-2.0Last push Oct 18, 2026
Rank 1,911: JohnSnowLabs/spark-nlp
State of the Art Natural Language Processing
4,160 starsScalaApache-2.0Last push Oct 14, 2026
Rank 960: vaexio/vaex
Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per second 🚀
8,517 starsPythonMITLast push Apr 1, 2026
Rank 114: roboflow/supervision
We write your reusable computer vision tools. 💜
51,241 starsPythonMITLast push Oct 17, 2026
Rank 138: ray-project/ray
Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
44,061 starsPythonApache-2.0Last push Oct 18, 2026