Frameworks

Teams lean on the system and lose the judgment to step in. Decide who owns adopting new model capabilities.

Chris McCarthyNovember 16, 20265 min read
automation complacencyai model update governancecontinuous improvement aiai deskilling workforceevaluating new ai models

Teams lean on the system and lose the judgment to step in. Decide who owns adopting new model capabilities.

By Chris McCarthy, Founder & Chief AI Officer, OB.1

As AI takes on more routine work, fewer people practice the judgment needed when a sensor fails or an edge case appears. Every model upgrade moves that line again. The fix at scale is ownership: who owns the process for evaluating and adopting new model capabilities? Give that job a name, a test set and a schedule.

This is part seven, the last in our Pilot to Production series. This one is for continuous improvement leads and the operations leaders they report to.

The quiet risk of a system that works

The early risk with AI is that it gets things wrong. The later risk is that it gets things right so often that people stop checking. Operators trust the recommendation. Planners accept the schedule. Over months, the skill of catching a bad answer fades because nobody needs to use it.

Research is starting to measure this. A 2025 Microsoft Research study of 319 knowledge workers found that higher confidence in generative AI was associated with less critical thinking. In medicine, a 2025 study in The Lancet Gastroenterology & Hepatology found that experienced doctors' detection rate on procedures done without AI fell from 28.4% to 22.4% after AI assistance was introduced. Aviation learned the same lesson years ago, which is why pilots still train for manual flight.

Then the rare failure arrives: a sensor drifts, a supplier changes a material, a customer sends an order unlike any before. The system gives a confident answer, and fewer people can tell it is wrong.

Why model upgrades make it harder

AI vendors release new model versions often. Each one can be better on average and still behave differently on your edge cases. An upgrade that improves most answers can quietly change the few that matter most to your plant.

Most companies have no process for this. The vendor updates the model, or someone flips a setting, and behavior changes with no test and no owner.

The NIST AI Risk Management Framework treats this as continuing work. Its Measure and Manage functions call for testing and monitoring AI systems across their whole life, well past the first deployment.

The question to ask: who owns the process for evaluating and adopting new model capabilities?

The answer is a named owner with a defined process. In a mid-sized manufacturer, that owner often sits in continuous improvement, because the job looks like any other process change: test, compare, decide, train.

The process needs four parts:

PartWhat it means
A test setReal cases from your plant, including the hard edge cases, with known right answers
A comparisonRun the current and new model on the test set before any switch
A decision ruleWhat results justify the change, and who signs off
A skills checkRegular drills where people solve cases without the AI

What a good answer looks like

Treat AI capability like equipment. You would not swap a machine controller without a trial run and operator training. The same rule applies to a model.

Keep people's judgment sharp on purpose. Rotate operators through manual troubleshooting drills. Ask the AI to show its reasons so people can check them. Review the cases where a person overruled the system, and learn from them.

How OB.1 fixes this

The Operational Blueprint identifies where AI is making or shaping decisions today and which of those decisions would hurt most if they went wrong. Those become the edge cases in your test set.

You get a named owner, a test set built from your own operations, a decision rule for upgrades, and a skills plan for the people who need to step in. Our Govern service then runs the evaluation on a schedule, tracks changes across model versions and reports what moved. Our Rules Before Tools training keeps the people side current as the tools change.

Read the rest of the series

  1. Costs climb faster than your prices
  2. Your ERP data sits in silos nobody trusts
  3. The plant you just bought runs a different system
  4. Staff paste pricing into public AI tools
  5. Your best tech retires, and the fixes leave too
  6. Automation could ship the wrong order
  7. Teams lean on the system and lose the judgment to step in (this article)

Book a Blueprint Session

Sources

Frequently Asked Questions

What is automation complacency?

Automation complacency is the tendency to stop checking a system that is usually right. Over time, people get slower at spotting the rare case where the system is wrong.

How do you prevent deskilling when using AI?

Run regular drills where people solve real cases without the AI, ask the system to show its reasons, and review every case where a person overruled it.

Who should own AI model updates in a manufacturing company?

A named owner, often in continuous improvement or operations, who runs a test set before any change and signs off on the switch with the process owner.

How should we test a new AI model before adopting it?

Build a test set of real cases from your operation, including hard edge cases with known answers. Run the current and new model on it and compare before switching.

Get Started

Book a Blueprint Session.

That is the first conversation.

chris@ob1ai.co | (234) 602-0500

Talk to OB.1

Tell us where AI fits in your business. We start with the Operational Blueprint and Diagnostic AI Consulting, and we reply within one business day.