Millstream vs Airflow, Dagster and Prefect
Airflow, Dagster and Prefect orchestrate data work in Python: tasks in a DAG, software-defined assets, or decorated flows, usually over batches, tables and partitions. Millstream orchestrates single items through a declarative graph where some steps are code, some are LLM agents and some are a person's decision.
At a glance
| Millstream | Airflow, Dagster, Prefect | |
|---|---|---|
| Definition | pipeline.yaml, validated by millstream check before it runs |
Python code: Airflow DAGs, Dagster assets and ops, Prefect flows and tasks |
| Unit of work | One delivery of one item to one step | A task run, an asset materialisation or a partition |
| Languages for steps | Any program speaking a JSON-lines protocol over stdio; Go SDK | Python first |
| Agents | Agent actions with schemas, sandbox, transcripts, budgets and evals | Tasks that call a model, written by you |
| Human decisions | review builtin with an inbox, edits, reason codes, timeouts and sampled auto-approval |
Varies per tool; no shared review model |
| Lineage | Per item, across steps and pipelines | Per task run, or per asset in Dagster |
| Laptop to server | One binary: SQLite on a laptop, Postgres with a hot standby and remote nodes on a server | A scheduler or server, workers and a metadata database; hosted offerings exist |
| Licence | Apache-2.0 core, commercial edition | Apache-2.0 |
| Status | Pre-release: M1 to M5 built and tested | Released, widely used |
Where the models differ
One item at a time
A daily crawl that finds 400 listings is one task run in an orchestrator, and what happens to each listing inside it is your code's business. In Millstream each listing is an item with its own deliveries, retries, cost and history. One malformed listing becomes a dead letter on <step>.dlq while the other 399 carry on, and millstream replay --from dlq --run <id> puts it back into its original run once the action is fixed.
Thresholds are pipeline steps, visible in the file, so one scoring action serves every pipeline:
keep-good:
action: filter
in: listings.scored
out: listings.shortlist
config: { expr: "item.score >= vars.minScore" }
Steps in any language
An action is a program that reads hello and item messages on stdin and answers emit, done or error on stdout. Go, Python, a container image or an agent adapter all speak the same protocol, and millstream test action --conformance checks one against it. Schemas declared by producers and consumers are checked against each other by millstream check before anything runs.
Agents and people are step kinds
An agent step's output is validated against its schema, its transcript is stored, and its cost is recorded per delivery and capped by maxCostUsd, runUsd and dayUsd. A review step parks items for an analyst with a timeout, an onTimeout rule and the fields they may edit. Every decision records who, when, the reason code and the diff, and the intervention rate per step is a metric. The insights projector reports which agent spend a later filter or reviewer threw away, as a waste insight.
When to choose Airflow, Dagster or Prefect
- Your work is batch data engineering: loading, transforming and publishing tables, partitions and files on a schedule.
- You want asset-level lineage across a data platform, which is Dagster's model.
- Your team writes Python and wants the orchestration in the same language as the transformations.
- You depend on a large catalogue of ready-made integrations with warehouses, clouds and data tools.
- You need a released, widely used tool with hosted offerings. Millstream is self-hosted and pre-release.
Using them together
An orchestrator task can fire a Millstream run once its load lands, through a webhook trigger: POST /hooks/<pipeline>/<trigger> with a JSON body checked against the trigger's inputSchema, and an Idempotency-Key so a retried task does not open a second run. Millstream then handles the per-item agent and review work. The stock sink actions that come with the first release (millstream/sink-file, millstream/sink-webhook) write approved items out, for the orchestrator to pick up.