Luigi

Python module for building complex pipelines of batch jobs. Handles dependency resolution, workflow management, visualization, and Hadoop integration. Built at Spotify and battle-tested in production. Apache 2.0 licensed.

View on GitHub Unknown

Guide pending

We haven't written this guide yet.

Guides are drafted from the project's own README and metadata. Until one exists, check the product's repository and documentation for authoritative information.

Nearby on the map

Related projects

Pandas

pandas-dev/pandas

The gold standard for data analysis and manipulation in Python.

unknown unknownactivity unknown

Polars

pola-rs/polars

Blazing-fast DataFrame library (Rust backend) - modern alternative to Pandas for large-scale workloads.

unknown unknownactivity unknown

cuDF

rapidsai/cudf

GPU DataFrame library from RAPIDS. Accelerates Pandas workflows on NVIDIA GPUs with zero code changes using cuDF.pandas accelerator mode.

unknown unknownactivity unknown

Dask

dask/dask

Parallel computing for big data - scales Pandas/NumPy/scikit-learn to clusters.

unknown unknownactivity unknown

DataFlow

OpenDCAI/DataFlow

LLM-ready data preparation system for turning raw PDFs, conversations, code, databases, and other sources into SFT, QA, and RAG-ready datasets.

unknown unknownactivity unknown

NumPy

numpy/numpy

Fundamental array computing library that powers almost every AI stack.

unknown unknownactivity unknown