NEOEDGE AIOur AI pilot isn’t working. Fix it, stop it or scale it?
Decide on evidence from your own cases. Measure the pilot against the job it was meant to do, counting errors, rework, checking time and ownership. Fix it when one cause explains most failures. Stop it when it can’t reach your bar. Scale it only when it meets the bar and has a named owner.
What should we measure?
Measure the result the pilot was meant to change, such as time per file or errors caught. Then count how often people redo its work, how long checking takes, which cases it can’t handle, and whether people use it at all. Last, check that someone owns the pilot and can make the call. Without an owner, a pilot drifts. Some AI pilots stall between the demo and daily use, and a stalled pilot keeps costing licence fees, staff time spent checking its work and the team’s trust in the next tool.
When is a pilot worth fixing?
Fix it when most failures trace to one cause you can change, such as poor input data, a missing rule or the wrong source document. Sometimes the fix is a better data source. When I built a card-statement coding tool for a Canadian mortgage lender, we also moved to the bank’s own export, which is more reliable than reading scans. In one AI chatbot I worked on, staff asked questions the source documents didn’t cover. Adding the missing material made it useful, and more people kept using it.
When should we stop it?
The clearest sign is the team itself. If people don’t trust the results or don’t use the tool, another month won’t fix that on its own. Also stop it when it can’t reach the accuracy your team needs on your own test cases, when checking its work takes longer than doing the work, or when nobody will own it. Stop it too when the data it needs can’t be used safely. OSFI’s E-23, which takes effect May 1, 2027, treats retiring a model as a formal step. The NIST AI Risk Management Framework suggests that teams “decommission systems that exceed risk tolerances.”
Why is stopping a good result?
A stop answers the unknowns you started with, and a good decision needs those answers. A stopped pilot still leaves you a test set, a measured baseline and a clear reason. Together they make the next decision faster.
When is it ready to scale?
Scale it when it meets your bar on cases it hasn’t seen before, a named owner runs it, a person reviews the cases that need review, and someone tracks its accuracy after launch. People should be using it without being chased.
| Signal | Fix | Stop | Scale |
|---|---|---|---|
| Accuracy on your cases | Close, with one clear cause of errors | Well short, causes spread out | Meets your bar on new cases |
| Checking time | Falls once the cause is fixed | More than doing the work | Lower than the work it replaces |
| Owner | Named, with time to fix it | Nobody will own it | Named, and running it |
| Data | Fixable with a better source | Can’t be used safely | Approved and stable |
| Use | Low until the fix lands | Staff work around it | Used without chasing |
Where I can help
A Decision Sprint tests one AI decision on your real cases in about two weeks. You leave with a written call: build, buy, fix, continue, scale or stop, with the evidence and a named owner. Fixed fee from CAD 15,000.
More guides for lenders
- How to evaluate an AI vendor
- 10 AI tasks for mortgage operations
- Readiness audit or Decision Sprint?
Sources
- OSFI, Guideline E-23 Model Risk Management (2027), section D.2, model decommission
- NIST, AI RMF Playbook, MANAGE 2.3
This page is general guidance. It isn’t legal or regulatory advice.