Argo

Argo in one line: Argo runs your data pipelines on Kubernetes – where each step in the workflow is a container.

What is Argo?

Argo is a workflow engine for orchestrating parallel jobs on Kubernetes. In some ways, it overlaps with Apache Airflow: both tools let you to define pipelines of tasks as DAGs (Directed Acyclic Graphs). If you have a complicated system of tasks that need to run in a specific order on a specific schedule, you can define this with Argo, and it will run these tasks for you using Kubernetes.

What problem does Argo solve?

Many machine learning pipelines (and other complicated systems) rely on data being in a specific state, or tasks running at specific times. While you can use tools like Cron to schedule tasks, the implicit dependencies between different tasks may quickly become unmanageable. By explicitly defining how your tasks depend on each other in the same place you define when they should run, it’s much easier to work out where something went wrong. Using orchestration tools, you’ll also spend less time setting up new tasks, fitting them into your existing workflows, and spinning up hardware to run them.