AI agents: what has to be true before you run one on live operations
Evaluation, guardrails, observability and rollback. In that order, and before the agent.
We’re building agent capability and we aren’t selling it yet. This is the checklist we’re working through, and it’s a reasonable checklist to hold any vendor to.
An evaluation suite that exists before the agent does
A set of real cases with known-correct outcomes, run on every change. Without it you can’t tell an improvement from a regression, and you’re shipping on vibes.
Guardrails at the tool boundary
Constrain what the agent can do, not just what it’s told to do. Read-only by default. Irreversible actions behind explicit approval. Spend and rate limits enforced by the system rather than the prompt.
Observability over every action
Every tool call, input and output logged and reviewable. If you can’t reconstruct why the agent did something last Tuesday, you can’t operate it.
Rollback
A defined way to undo what it did. If undoing a bad run means a person manually repairing records for two days, the agent isn’t ready for that workflow.
Until those four are in place, the honest answer to "can we deploy an agent" isn’t yet. That’s the answer we’re giving.