Self-improvement

A Millstream pipeline learns from its production runs: what is slow, what fails and why, what reviewers reject or correct, and which agent steps have become predictable enough to replace with code. Learning happens on production data; changes are made in development, validated, and accepted by a person.

The loop

flowchart LR
  prod(["Production runs"]) --> insights["Insights"] --> evidence["Evidence"]
  evidence --> proposals["Proposals"] --> validation["Backtest and evals"]
  validation --> person{{"A person accepts"}} --> release(["Release"])

Nothing changes production behind its package. Every change reaches it as a new revision through your normal release, so revisions and replay stay deterministic, and every learning has a reviewer and a diff.

Insights, computed from the log

The insights projector reads events, deliveries, reviews and metric rollups at the end of every run and on a schedule. No LLM is involved, so insights run in every deployment, with no API key.

Each insight carries its figures and sample item ids, updates one row while the problem recurs, and can be acknowledged or muted. Every run also gets a report: critical path, slowest steps, failures by cluster, cost, and a comparison with the median of the last ten runs.

Evidence leaves production safely

Production is where the data is. millstream evidence export writes a bundle: the open insights, a stratified sample of items with their lineage, review decisions with reason codes and edit diffs, transcripts, and the production revision.

• Values at the JSON pointers each stream lists are replaced by a keyed hash, so they compare but cannot be read.
• Reviewer identities and free text never leave.
• Bundles expire, and development deletes them when they do.
• Erasure lists every bundle that held an erased key.

millstream evidence export --pipeline property-sourcing --since 30d
millstream evidence import evb_6c41e0.tar.gz      # in a development deployment

Proposals, validated before anyone reads them

In development, the improve pipeline turns insights and evidence into proposals: reviewable patches with a rationale, the insights they answer, the base revision and a validation report.

Validation replays the change over the evidence (a backtest, before and after, per changed step) and runs the evals. A person then accepts or rejects it in the UI or the CLI. A proposal accepted on a base revision makes overlapping ones stale, and a stale proposal is rebased and validated again.

improve is an ordinary pipeline package embedded in the binary. Its steps are actions that call the public API with a token limited to proposals, with budgets, transcripts and replay like any pipeline. You can read it, fork it with millstream init pipeline --from improve, or replace it.

Agents that become code

When a step has at least 500 deliveries, a bounded output schema, few turns, no external tools and, where reviewed, at least 95% agreement with reviewers, the distillable insight flags it, ranked by the monthly cost it would save.

The distill proposal splits the step's history by time and by entity into a training set and a holdout the drafting agent never sees. The agent writes a Go action with the SDK, with unit tests and evals, that may abstain on any item it is unsure of. Validation requires at least 0.98 agreement on the holdout items it covers, at least 60% coverage and 200 covered items. The patch makes the code the step's action and moves the agent under fallback:

score:
  action: ./actions/score-rules
  in: listings.unique
  out: listings.scored
  fallback:
    action: lab/score-listing@2
    when: [abstain, error]
    auditRate: 0.02

Audits keep checking a share of the code's answers against the agent, and a disagreement raises a drift insight.

Measured end to end

The platform's milestone acceptance test runs this loop through the binary, offline and deterministic: two production pipelines on synthetic listings export evidence, a development deployment imports it, drafts a lessons proposal and a distill proposal, accepts both, and runs the next day on the new revision.

• 0.58 → 1.00 agreement with reviewers after the lessons proposal
• 260 of 270 held-out items covered by the code
• 1.000 agreement on the covered items
• 3.7% of production's agent cost on the next day

The ten items the code abstained on, listings without a surface area, went to the agent.

Publishing safely

A proposal's patch was written by an agent that read untrusted data, so the engine enforces three rules:

• Paths per kind. A lessons proposal touches only its lessons file, a prompt proposal only a prompt.md, a distill only a new action directory and the pipeline file. Symbolic links, file modes and paths outside the package are refused when drafted and again when published.
• Separate environments. Proposed code builds and runs only in a sandbox with no network and no credentials. The publish action holds the forge credentials, never builds and never runs candidate code.
• The reviewed patch is the published patch. Publishing checks the patch's hash against the validated one. Pull requests carry the validation report and a Millstream-Proposal trailer.

Next

Agents Human review Why Millstream