Software that processes large datasets often repeats expensive computations when any part of the input data or processing logic changes. Metaxy is about perfecting the art of doing nothing: only compute what changed, save time and money, and accelerate exploration. In machine learning pipelines that handle video, audio, and images, these computations run on Graphics Processing Units (GPUs) that cost 10 to 100 times more per hour than standard processors1.

A small change to one processing step can trigger unnecessary recomputation of unrelated steps, wasting both time and money. Metaxy is a Python library that tracks which specific data records need reprocessing after a change, rather than rerunning entire datasets. It builds a dependency graph that connects individual data fields across processing steps.

When a researcher modifies one step, Metaxy identifies exactly which records are affected and which can be skipped. This selective approach lets downstream systems avoid redundant GPU work when a change does not affect a record, while preserving complete lineage for reproducibility.

The library treats the metadata store (which persists version entries), the compute engine (which evaluates feature code over dataframes), and the orchestrator (which schedules execution) as pluggable abstractions. Concrete integrations for the widely used Python ecosystem ship with the package (Dagster Labs, 2025; Gorelli & others, 2025; Ibis Project, 2025; Moritz et al., 2018), so that any compatible orchestrator can consume Metaxy’s record-level diffs and schedule only the necessary GPU workloads.

D. Gafni, G. Heiler, Metaxy: Record-Level Feature Metadata Management for Multimodal ML Pipelines, The Journal of Open Source Software 11(122) (2026) 10449.