Topic

#data-pipelines

Loot, blog posts and adjacent themes connected to this topic. Follow the tag to keep it in your orbit.

#data-pipelines
Loot

More from this topic

Explore all loot

Run Rust-backed Python data pipelines with Rivers

0
Rivers is a new open-source orchestration platform for data and ML pipelines: Python definitions, Rust control plane, Kubernetes operator, OIDC, and forward-auth support. Rivers is worth a bookmark if you build data or ML pipelines and want a Python authoring layer without putting the whole control plane on Python runtime behavior. The project defines assets and repositories in Python, then uses a Rust core for graph resolution, execution planning, scheduling, partition handling, and the web UI. What to try Define simple assets as Python functions and materialize them locally with pip install rivers. Use rivers dev MODULEPATH for a single-process local setup with embedded storage, scheduler, and UI. Evaluate it against Dagster-style asset orchestration needs where startup speed, typed execution planning, and Kubernetes deployment matter. Check the Kubernetes path if you need CodeLocation CRDs, one-pod-per-step execution, Helm charts, and a separate operator. Review the authentication guide before exposing the UI: Rivers documents none, native OIDC with PKCE, and forward-auth mode behind a trusted proxy. Good fit Rivers looks most useful for teams experimenting with data asset orchestration, ML pipeline scheduling, or internal pipeline platforms that want Python ergonomics plus a compiled Rust scheduler/control plane. It also has practical production hooks: SurrealDB-backed state, Server-Sent Events for UI updates, Kubernetes-native execution, backfills, sensors, schedules, concurrency limits, and pluggable IO. Caveats This is still a young project. The GitHub repository was created in May 2026 and the PyPI package is at 0.4.0, so treat it as an evaluation candidate rather than a drop-in replacement for mature orchestrators. The repository license is AGPL-3.0, which matters for internal platform and SaaS use. LinkLoot has not performed a hands-on review; verify installation, security posture, and operational behavior in a disposable environment before using it with production workloads.
View
Free
Open
Blog

Related reads

Browse blog
No blog posts for #data-pipelines yet

There is no published article with this tag right now. Browse the blog for adjacent themes or follow the tag for future updates.