Quick start
This tour goes from nothing to a pipeline with a crawler, a deduplication, an LLM agent and a human review, on a laptop, with no model API key. Every command and output below comes from a real run of the full tutorial, which ships with the source. Outputs are trimmed, and ids and times will differ on yours.
Install
The first release ships binaries for Linux and macOS, a Homebrew formula and a container image. Until then, early-access teams build it from source with Go and make:
git clone git@github.com:monolithiclab/millstream.git
cd millstream
make ui # optional, needs Docker: builds the web UI the binary embeds
make build # writes build/millstream
export PATH="$PWD/build:$PATH"
Scaffold, check and run
init writes a package: the pipeline file, an action written with the Go SDK, a fixture, an eval case and a package
test. check validates it, graph draws it, and test pipeline runs it with stubbed actions.
millstream init pipeline listings
cd listings
millstream check
millstream graph
millstream test pipeline
warning steps.greet.out: stream greetings is written by step greet but no step reads it
listings: 0 errors, 1 warning
flowchart LR
n0["greet"]
n1[("greetings")]
n2(["trigger.adhoc"])
n0 -->|"greetings"| n1
n2 -->|"trigger.adhoc"| n0
PASS one name greeted
listings: 1 passed, 0 failed
run --once fires the manual triggers, waits for the runs to settle and prints what reached each stream:
millstream run --once 2>/dev/null
listings run_20261020T162508Z-f9927400_adhoc succeeded
greetings 1
trigger.adhoc 1
State lives in .millstream/millstream.db, a SQLite database beside the package. millstream run keeps the engine
running and serves the web UI on http://127.0.0.1:7070/: the live graph, runs, items with their lineage, and dead
letters.
Add steps, then operate them
The tutorial replaces the placeholder with a crawler, dedupe, a price filter and routing. On the second day,
dedupe holds back the listings it has already seen:
listings run_20261020T162558Z-7559f3e7_adhoc completedWithErrors
affordable.dlq 1
listings.raw 4
listings.ready 1
listings.renovate 1
listings.unique 4
trigger.adhoc 1
listings run_20261020T162559Z-902b5020_adhoc succeeded
listings.raw 4
listings.renovate 1
listings.unique 1
trigger.adhoc 1
One listing had no price and failed the filter's expression, so it waits as a dead letter. Fix the expression, apply the package as a new revision, and replay the dead letter into its original run:
millstream apply .
millstream replay --pipeline listings --from dlq --run run_20261020T162558Z-7559f3e7_adhoc --migrate
listings revision 3 83b5897356a1
1 dead letter replayed
Or see what a threshold change would do before you ship it, by replaying one step over the last hour with a variable changed for that run only:
millstream replay --pipeline listings --step affordable --since 1h --var maxPrice=200000
Add an agent step
An agent action declares its prompt, model, output schema and limits. The tutorial runs it against a small local stand-in for an OpenAI-compatible model, records the answers once, then replays them offline with no key, which is how Millstream's own CI runs agents.
kind: agent
runtime:
agent:
harness: api
provider: openai
baseUrl: http://127.0.0.1:8099/v1
model: fake-scorer
apiKeySecret: modelKey
network: [127.0.0.1]
maxCostUsd: 0.01
The step's lessons file adds rules to the prompt, and replay back-fills the new step over the day's items:
millstream replay --pipeline listings --step score --since 1d
millstream eval actions/score
CASE ASSERTIONS JUDGE COST SCORE
cheap-flat 2/2 - $0.0018 1.00
TOTAL 1.00
Lower maxCostUsd below the call's cost and the budget stops it, with the transcript kept:
cheap-flat: error: the action failed: budget: the attempt cost $0.0018, over the action's maxCostUsd of $0.001 (the attempt's transcript is its transcript.json artifact)
Add a human review
A review step parks the shortlisted listings in an inbox. The run waits on people, holding no slot:
millstream inbox
millstream inbox approve dl_10e2b8dac5c2cda48768323670c8dc18 --set /score=85 --text "Large flat, worth it"
millstream inbox reject dl_319ef7f15ebfe412643004daef54835f --reason bad-area
dl_10e2b8dac5c2cda48768323670c8dc18 approved (delivery done)
dl_319ef7f15ebfe412643004daef54835f rejected (delivery filtered)
The run report shows where the time went, the reviewer's wait included:
Critical path (19.184s):
STEP DELIVERY LATENCY WAIT WORK BACKOFF REVIEW SHARE
score dl_983e2427bcd5c45f31b107a7250ea642 395ms 362ms 33ms 0s 0s 2%
keep-good dl_ab3aed8f1c6bb0acd6631f1fefa0d20d 3ms 2ms 1ms 0s 0s 0%
analyst-review dl_1a3bf49e9f6088ea853eb8d150e1cb52 18.786s 1ms 0s 0s 18.785s 98%
What the platform already noticed
With no LLM, the insights projector has analysed the pipeline: the failed filter expression, the agent's cost per approved item, and the review wait on the critical path.
KIND SCOPE SEVERITY SUMMARY
failureCluster affordable error 1 failures (7% of attempts), 0% recovered on retry: expr: filter of step affordable …
cost score info $0.0216 for 12 outputs ($0.0018 each, $0.0072 per approved item), 1 turns per delivery, 0 schema failures
slowStep analyst-review info 75% of the critical path (queue 0%, work 0%, backoff 0%, review 100%); p95 18.786s, no baseline
From here, a development deployment turns insights like these into validated proposals.