Agents in production, a checklist

An agent that works in a notebook still has to survive production: inputs nobody anticipated, a page written to steer it, a bill at the end of the month, and a question about why it answered what it did. Four practices hold for any agent system. This page lists what each one asks of an agent step, and the Millstream key that provides it.

Before the first agent: pick the cheapest step

An agent step can start where the rules are not known. As deliveries and reviews accumulate, Millstream proposes replacing it with code, with the agent kept as the fallback. How pipelines learn

Treat the output as untrusted

• A tight output schema. Enums over free strings, bounds on numbers, every field a later step reads in required. Each answer is validated before anything is published, and a failed answer is retried.
• A way to abstain. The schema admits {"abstain": "<why>"}, the prompt says when to use it, and the step has a fallback: a stronger model, code or a person.
• Allowlists. tools, mcp and network in the manifest list only what the agent needs, and the agent runs in its sandbox container.
• A review before every external effect. Actions that send, write or pay declare effects: external, and millstream check warns when an agent's output can reach one without a review.
• Thresholds in filter steps, where they are visible in the pipeline file and changed without touching the prompt.

Keep planning apart from execution

• Limits per call. maxTurns and maxCostUsd in the manifest. The agent stops at either.
• Limits per pipeline. budget.runUsd and budget.dayUsd, reserved before each call starts. A pipeline that reaches one pauses its agent steps until someone resumes it.
• A fixed graph. The agent answers one item inside one step and never changes the pipeline. Changes arrive as proposals, are backtested on past items, and are accepted by a person.
• Runs pinned to a revision. A release never changes work in flight, and millstream rollback restores the previous revision.

Record everything

• A transcript per attempt, prompts, tool calls and answer, with the pipeline's secrets redacted, kept for retention.transcripts.
• Metrics per answer: tokens, cost, turns and duration, labelled with the model, served to Prometheus.
• Lineage per item, from the trigger to the report, through every step and decision.
• Recordings of every model call, so tests and evals replay offline with no API key.

Improve from what was recorded

• Evals in the action's evals/ directory: CEL assertions and an optional LLM judge, scored by millstream eval before a prompt or model change ships.
• Reviews that teach. reasons on every review step and editable on the fields people correct. Edits are attributed to the step that produced the value.
• Insights from the event log, with no model call: slow steps, failure clusters, cost, review patterns.
• Proposals drafted in development: lessons, a prompt, a threshold, a filter moved upstream, or code in place of the agent.
• Audits of distilled code: fallback.auditRate sends a sample to the agent, and disagreements raise an insight.

The list

• The output schema is tight, and every field a later step reads is required
• The schema admits abstain, and the step has a fallback
• tools, mcp and network list only what the agent needs
• A review sits before every effects: external action downstream of the agent
• Thresholds live in filter steps
• maxCostUsd is set, and the pipeline has budget.dayUsd
• Evals exist, with recordings committed, and pass offline
• Review steps have reasons, and editable lists the corrected fields
• retention.transcripts matches how long an answer must stay explainable

Next

How agents run How pipelines learn Read the quick start