Automation could ship the wrong order. Decide which outputs get reviewed, and who signs off.
By Chris McCarthy, Founder & Chief AI Officer, OB.1
Once AI starts acting on orders, part numbers or dispatch, a single mistake can repeat at machine speed. The safeguard is a review rule written before go-live that answers one question: what outputs get reviewed, and who signs off? Match the level of review to the cost of being wrong.
This is part six of our Pilot to Production series. This one is for operations and supply chain leads.
From suggestions to actions
The first wave of AI in most companies drafts things. A person reads the draft and decides. The next wave acts. An AI agent can enter a sales order, change a part number on a bill of materials, reroute a shipment or release a purchase order.
That shift changes the risk. A bad draft wastes a few minutes. A bad action ships the wrong part to your largest customer, triggers a contract penalty, or orders a month of material you do not need.
This shift is coming fast. Gartner predicts that by 2028, at least 15% of day-to-day work decisions will be made autonomously through agentic AI, up from none in 2024, and a third of enterprise software applications will include agentic AI. The bar for accuracy stays where it is: by WERC's 2026 benchmarks, top-performing distribution centers hold order-picking accuracy at 99.7% or better. An agent touching your orders has to meet the standard your people already meet.
Why review rules break down
Most companies say they keep "a human in the loop." In practice, that often means a person approves everything, so they stop reading. Or nobody defined what the human checks, so approvals become a click.
Review that adds no judgment is theater. It slows the work and catches little.
Security experts have mapped the risks. The OWASP Top 10 for Agentic Applications, published in December 2025, lists tool misuse and cascading failures among the most critical risks for AI agents. Both describe an agent doing the wrong thing with real permissions, over and over.
The question to ask: what outputs get reviewed, and who signs off?
Sort every action the AI can take by the cost of a mistake, then set the review to match.
| Risk level | Example actions | Review rule |
|---|---|---|
| Low | Draft an order-status reply | Spot checks on a sample each week |
| Medium | Enter a standard repeat order | Automatic within set limits, human review outside them |
| High | Change a part number, release a large PO | Named approver signs off before it happens |
| Critical | Change pricing, ship to a new address | Two people sign off, with a log |
Each row needs a named owner. Write the limits in numbers: order value, quantity change, new customer, new ship-to address.
What a good answer looks like
A review design that holds up in production has five parts:
- Every action the AI can take, sorted by risk
- Limits written as numbers that trigger human review
- Named approvers for each high-risk action, each with a backup
- A stop switch that pauses the agent, and a named person allowed to use it
- Logs of every action and approval, read by someone every week
How OB.1 fixes this
The Operational Blueprint traces how an order moves through your company today, from intake to shipment. We note every point where a person checks something now, and why. Those checks are the starting point for the AI's review rules.
You get a risk-sorted action list, numeric limits, named approvers and a stop procedure your team can sign. Build partners then put those controls into the system in the Architect phase. Our Govern service runs them after go-live: monitoring, escalation design and quarterly reports on what the agent did and what people caught.
Read the rest of the series
- Costs climb faster than your prices
- Your ERP data sits in silos nobody trusts
- The plant you just bought runs a different system
- Staff paste pricing into public AI tools
- Your best tech retires, and the fixes leave too
- Automation could ship the wrong order (this article)
- Teams lean on the system and lose the judgment to step in
Sources
- Gartner, press release on agentic AI forecasts, June 2025
- OWASP GenAI Security Project, Top 10 for Agentic Applications for 2026, December 2025
- WERC 2026 DC Measures, as summarized in Yale's DC metrics white paper
- ISO/IEC 42001:2023, Artificial intelligence management system, December 2023