Apache Spark

Unified analytics engine for large-scale data processing. In-memory cluster computing with high-level APIs in Python, Scala, Java, and R. Powers MLlib for distributed machine learning and Structured Streaming for real-time data. Apache 2.0 licensed.

View on GitHub Unknown

Guide pending

We haven't written this guide yet.

Guides are drafted from the project's own README and metadata. Until one exists, check the product's repository and documentation for authoritative information.

Nearby on the map

Related projects

Pandas

pandas-dev/pandas

The gold standard for data analysis and manipulation in Python.

unknown unknownactivity unknown

Polars

pola-rs/polars

Blazing-fast DataFrame library (Rust backend) - modern alternative to Pandas for large-scale workloads.

unknown unknownactivity unknown

cuDF

rapidsai/cudf

GPU DataFrame library from RAPIDS. Accelerates Pandas workflows on NVIDIA GPUs with zero code changes using cuDF.pandas accelerator mode.

unknown unknownactivity unknown

Dask

dask/dask

Parallel computing for big data - scales Pandas/NumPy/scikit-learn to clusters.

unknown unknownactivity unknown

DataFlow

OpenDCAI/DataFlow

LLM-ready data preparation system for turning raw PDFs, conversations, code, databases, and other sources into SFT, QA, and RAG-ready datasets.

unknown unknownactivity unknown

NumPy

numpy/numpy

Fundamental array computing library that powers almost every AI stack.

unknown unknownactivity unknown