The platform
Millstream is one binary that holds a pipeline engine, a CLI, a node that runs actions and the web UI. It turns a pipeline file into durable, observable work, and the same package runs on a laptop and on a server.
How the pieces fit
A pipeline is a graph of steps connected by persisted streams of items. A step runs an action:
a program in any language, a container, an LLM agent, or one of three builtins the engine runs itself (filter,
dedupe, review). A trigger (a schedule, a webhook, a manual firing, an insight from another pipeline) opens a
run, and every item belongs to exactly one run.
The engine schedules and dispatches deliveries; nodes pull them under fenced leases and run the actions. On a laptop one node is embedded in the engine. On a server, a standby engine waits to take over and nodes run on other hosts.
flowchart LR
trigger(["daily"]) --> crawl["crawl"] --> dedupe{{"dedupe"}} --> score["score: agent"]
score --> keep{{"filter"}} --> review[/"review: person"/] --> report["report"]
The parts
Pipelines
- A declarative YAML file, validated before it runs
- Streams instead of calls, so steps insert and move freely
- Routing, dead letters, replay and revisions
- Read more
Agents
- Prompt, model, schema and allowlists in a manifest
- A sandbox, budgets and transcripts for every call
- Evals and record-and-replay for deterministic tests
- Read more
Human review
- Inboxes with edits, reason codes and timeouts
- Rules that auto-approve, measured by samples
- The intervention rate, per step and per pipeline
- Read more
Self-improvement
- Insights computed from the event log, with no LLM
- Proposals drafted in development, validated on evidence
- Agents replaced by code where their answers are predictable
- Read more
Observability
- Lineage for every item, events for every change
- Metrics, run reports and Prometheus
- A live web UI in the open-source core
- Read more
Deployment
- SQLite on a laptop, Postgres on a server
- A hot standby and remote nodes
- Compose and systemd, retention and backups
- Read more
Security
- One deployment per customer, isolated pipelines
- Sandboxed agents and untrusted code
- Secrets redacted, personal data erasable
- Read more