NEOEDGE AIYou pay per resolution, and the only count you have is the vendor’s own. You want your own number before the renewal conversation starts.
← All AI decisions
Resolution Audit · For the AI agent renewal on your calendar
Your AI vendor counts the resolutions. Check the count.
In about three weeks, I re-check a sample of the outcomes you’re billed for against your own written definition of “resolved,” and hand you an evidence pack built for the renewal table.
Platforms such as Intercom Fin, Zendesk, Ada, Sierra and Decagon may be suitable, depending on the export available. I confirm we can link billed outcomes to conversation records before we agree the scope and fee.
Sam Talasila Former Head of LLMs & Data Products, Wealthsimple
What you leave with
An evidence pack your team owns.
Built to be read by your leadership — and shown to your vendor.
Audit readout · AI support agentSynthetic example
- Verified rate
- 71% of the sampled billed resolutions met your definition. Each sampled item was billed as a resolution; the check used your written definition.
- Tested
- 450 billed resolutions, drawn at random across three months: 320 confirmed, 89 did not meet your definition, 41 had no usable record.
- Next move
- Take the exhibits to renewal. Ask for the rate to be measured against your written definition, attached to the contract.
- Owner
- Support operations lead, with procurement at the renewal table.
Evidence pack · ContentsEvery audit
- The tie-out: every billed outcome matched to a conversation record. Anything billed without a record is flagged first.
- The adjudication ledger: every sampled outcome, the call, and the reason for it.
- Your verified resolution rate with its confidence interval, and the dollar figure extrapolated across your annual contract.
- Your written definition of “resolved” — drafted with you, usable as a contract appendix.
- Dispute-ready exhibits keyed to conversation IDs, so every claim points at a record.
- A negotiation readout: the asks the evidence supports, and the concessions that pay for the audit.
When it fits
Three moments where an audit earns its fee.
Each has an outcome-priced contract, a renewal date, and a number nobody independent has checked.
The dashboard and the floor disagree. Neither side has evidence the other will accept, and the argument repeats every quarter.
Before the contract grows, someone has to divide the invoice by resolutions that actually met your definition — not the vendor’s.
How the three weeks run
Nothing is judged until the definition is written.
- Before we start
Write the definition.
Most teams have no written definition of “resolved.” We write yours together in one working session. Your team arranges one export from the vendor platform — about 30 minutes of admin time.
- Week one
Tie the invoice to the work.
Every billed outcome is matched to a conversation record. Outcomes billed without a usable record are flagged before anything else is judged.
- Week two
Adjudicate the sample.
A stratified sample of 400–500 billed outcomes, re-checked against your definition. Every disagreement gets a reason and a record to point at.
- The last day
Read the readout.
We walk through the verified rate, the dollar gap, and the asks the evidence supports — with whoever will sit at the renewal table.
Why verify · Different definitions, different countsEvidence
The definition changes the number.
“What counts as resolved?”
Before comparing a dashboard with your own count, check that both use the same population, time window and success criteria. An outcome that falls short of your business definition is not automatically an incorrect charge under the contract.
A published study of 14 production support bots reported a median dashboard resolution rate of 64.5%. Using a separate measure — an answer followed by no same-issue human contact within seven days — the publisher measured median deflection of 41.2%. The comparison shows why measurement definitions matter; it does not establish an overbilling rate.
This was a small, selected consultancy sample, not a representative industry survey. The publisher built five of the systems and sells development services; client-level source records were not published. Read the July 2026 field study, methodology and limitations.
Billing rules also differ across platforms. See examples of published outcome-counting rules and published resolution-verification tiers. Check the rules that apply to your agreement before drawing a billing conclusion.
Who does the audit ↗︎Scope
Bounded on purpose.
A clear edge keeps the audit quick, affordable, and honest about what it can show.
What you bring
One export and a few hours.
- An owner for the renewal — operations, support, or procurement
- One export of billed outcomes with conversation records from your vendor platform
- 60–90 minutes across the three weeks: the definition session and the readout
- Your security reviewer, long enough to approve the export format and the data-handling agreement
Not part of the audit
Kept for later, or elsewhere.
- Real-time monitoring or live access to your systems
- Joining your ticketing or CRM system of record — that’s a deeper engagement, scoped separately
- Legal dispute representation or formal audit assurance
- Checking your human agents’ work
- Building or changing the AI system itself
After the audit
Three honest next moves.
Take it to the renewal table.
The exhibits are built to be shown to the vendor. Even a few points of overstatement on a six-figure contract pays for the audit several times over.
One number, both sides.
Your written definition becomes the appendix the next contract is measured against. The argument about what “resolved” means ends.
Check it quarterly.
If you want the rate re-verified before every renewal, that’s a smaller recurring engagement — only after a first audit stands on its own.
The offer
Everything to walk into renewal with your own number.
On our first call we size what’s at stake. If the contract isn’t big enough for the audit to pay for itself, I’ll tell you it isn’t worth it.
What’s included
- The adjudication ledgerEvery sampled outcome, the call, and the reason — keyed to conversation IDs.
- Verified resolution rateWith its confidence interval, and the dollar figure across your annual contract.
- Definition workshopYour written definition of “resolved,” drafted together in week one and yours to keep.
- Dispute-ready evidence packExhibits built to be shown to the vendor, not just read internally.
- Negotiation readoutThe asks the evidence supports, and the concessions that pay for the audit.
- 30-day follow-upOne review within 30 days after your renewal conversation to check the number held.
My guarantee
Decision-Grade Guarantee
You get a verified rate with its confidence interval and evidence keyed to records. We agree the written definition of “resolved,” required deliverables and acceptance criteria before we start. If the pack falls short of those criteria, I correct it at no extra cost within the agreed platform, export and definition. New data, a different definition or additional scope is agreed separately.
I guarantee the evidence is decision-grade, not that the vendor is wrong. Sometimes the audit confirms the invoice — renewing with proof instead of trust is worth knowing too.
Why now
I run two audits at a time. I do the adjudication myself, so places are limited.
We work back from your renewal date. Starting about eight weeks out leaves time to use the pack before the conversation, not after it.
From the builder’s sideI ran LLM and data products at Wealthsimple, where support automation had to be measured to be trusted. I know how these counts get made — and where they drift.
Questions
Before you book.
Which platforms does this work with?
Suitability depends on the export, not just the platform name. For Intercom Fin, Zendesk, Ada, Sierra, Decagon and other outcome-priced platforms, I check whether your export links billed outcomes to conversation records before we agree the scope and fee.
Does the vendor need to cooperate?
No direct vendor involvement is needed if your team can provide a usable export linking billed outcomes to conversation records. We check export availability before agreeing the scope and fee.
What if we don’t have a written definition of “resolved”?
Most teams don’t. Writing it is part of the audit — one working session with your leads in week one, and the definition is yours to keep either way.
How is our data handled?
One export, agreed fields, and a data-handling agreement signed before anything moves. Records are used for adjudication only and deleted on an agreed schedule.
What if the audit confirms the vendor’s count?
Then you renew with proof instead of trust, and your definition is on paper for the next contract. The cheapest negotiating position is both sides agreeing on the number.
Is there a smaller first step?
A 30-minute call. Bring your renewal date and your last invoice, and I’ll tell you on the call whether an audit pays for itself.
When is your renewal?
In 30 minutes, tell me what you’re billed per resolution, what the contract costs, and when it renews. We’ll see whether a Resolution Audit — or neither of my offers — is the right first step.