Data Processing & Manipulation

The gold standard for data analysis and manipulation in Python.

Blazing-fast DataFrame library (Rust backend) - modern alternative to Pandas for large-scale workloads.

GPU DataFrame library from RAPIDS. Accelerates Pandas workflows on NVIDIA GPUs with zero code changes using cuDF.pandas accelerator mode.

Parallel computing for big data - scales Pandas/NumPy/scikit-learn to clusters.

LLM-ready data preparation system for turning raw PDFs, conversations, code, databases, and other sources into SFT, QA, and RAG-ready datasets.

Fundamental array computing library that powers almost every AI stack.

Scientific computing algorithms (optimization, linear algebra, statistics, signal processing).

NumPy and SciPy-compatible array library for GPU-accelerated computing in Python.

Creation, manipulation, and study of complex networks. The foundational graph analysis library for Python data science.

GPU graph analytics library with NetworkX-compatible API. 10-100x faster than CPU for large-scale graph algorithms. Apache 2.0 licensed.

High-performance large data visualization. Renders billions of points interactively without aggregation artifacts. BSD-3-Clause licensed.

Chunked, compressed, N-dimensional array storage. Scalable tensor data format optimized for cloud and parallel computing. MIT licensed.

GPU-accelerated data loading and augmentation library with highly optimized building blocks for deep learning applications. Apache 2.0 licensed.

Lightweight compatibility layer between DataFrame libraries. Write Polars-like code that works seamlessly across Pandas, Polars, cuDF, Modin, and more. MIT licensed.

Portable Python dataframe library with 20+ backends. Write Pandas-like code that runs locally with DuckDB or scales to production databases (BigQuery, Snowflake, PostgreSQL) by changing one line. Apache 2.0 licensed.

Machine learning with dataframes for dirty categorical data. Preprocessing and feature engineering for heterogeneous data with seamless Pandas/Polars integration. BSD-3-Clause licensed.

Lightning fast data version control for machine learning. Optimized for large datasets with efficient diffing, branching, and collaboration. Apache 2.0 licensed.

Statistical data testing and validation for dataframes. Pydantic-like API for Pandas, Polars, and other dataframe libraries with type hints and lazy validation. MIT licensed.

System for quickly generating training data with weak supervision. Programmatically label, build, and manage training data using labeling functions and probabilistic consensus models. Powers Snorkel Flow and used by Google, Apple, and Intel. Apache 2.0 licensed.

High-performance analytical in-process SQL database system. Fast, reliable, portable, and easy to use with rich SQL dialect support. Perfect for data processing and analytics workloads. MIT licensed.

Visual AI development toolkit for visualizing, labeling, and evaluating visual datasets and models. Supercharges computer vision workflows with dataset exploration and model analysis. Apache 2.0 licensed.

Multi-type data labeling and annotation tool with standardized output format. Configurable interface for images, text, audio, video, and time series with ML-assisted labeling. Apache 2.0 licensed.

Open-source storage framework enabling Lakehouse architecture with ACID transactions, scalable metadata handling, and unified batch/streaming processing. Apache 2.0 licensed.

High-performance open table format for huge analytic tables. Brings SQL table reliability to big data with time travel, hidden partitioning, and schema evolution. Works with Spark, Trino, Flink, Presto, Hive and Impala. Apache 2.0 licensed.