Guide · Buying AI

How to evaluate an AI vendor. Before you sign anything.

By Sam Talasila · Last checked on October 19, 2026

Why test on our own cases?

A demo shows the vendor’s best examples. Your files have your formats, your exceptions and the cases your team already handles by hand. At a Canadian mortgage lender, I compared more than 15 AI testing tools and recommended a short trial of two on the same tests, using made-up data so no client information left the company. Three things narrowed the field to two: the quality of each tool’s output on the same tests, a vendor willing to build the features the lender needed, and a free trial.

How do we run a fair trial?

Pick real cases, including the awkward ones, and write down the right answer for each before the trial starts. Run every shortlisted tool on the same cases. Count the errors that matter to you, such as a wrong figure, a missed condition or a made-up answer. Time how long a person spends checking each tool’s output, because that checking time is the real cost.

What should we ask the vendor?

1. What is your privacy policy? Where is our data stored and processed, and is it used to train any model?
2. How does your price grow with usage, and what would we pay at full volume?
3. Which subcontractors touch our data, including the AI model provider behind the tool?
4. How do you test for made-up answers, and can we see the results?
5. Will you tell us before the underlying model changes?
6. What happens to our data, and our configuration, when we leave?

What does OSFI expect when we buy an AI tool?

If OSFI regulates you, its third-party risk guideline, B-10, applies. It expects due diligence “prior to entering contracts,” a contract that gives you and OSFI “the right to evaluate the risk management practices” of the vendor, notice when a subcontractor is retained, and timely reporting of incidents. It also expects ongoing monitoring and an exit plan. Once E-23 takes effect on May 1, 2027, OSFI expects a vendor’s model to be rated on its own. OSFI’s July 2026 bulletin also suggests requiring vendors to tell you if and how they use AI.

What does privacy law expect?

Privacy law applies whether or not OSFI regulates you. Under PIPEDA, you stay “responsible for personal information in its possession or custody,” including information you send to a vendor for processing. You need a contract or other means “to provide a comparable level of protection” while the vendor has it. Quebec, Alberta and British Columbia have their own private-sector privacy laws. Check which tools your team is approved to use before anyone pastes borrower data into one.

What mistakes do lenders make when buying AI?

The two mistakes I see most are skipping a small pilot and not forecasting usage-based pricing. A tool priced per document or per question costs more as use grows, so model the bill at full volume before you sign. Some AI purchases never make it into daily use, and the cost goes beyond the fee: setup time, staff time spent checking the output and the team’s trust in the next tool.

When should we walk away?

Walk away when the vendor won’t test on your cases, can’t say where your data goes, or can’t meet your error bar in a fair trial. Also walk away when the checking time eats the time the tool saves. A clear no before signing costs less than a stalled rollout.

More guides for lenders

• Our AI pilot isn’t working: fix, stop or scale?
• 10 AI tasks for mortgage operations
• Readiness audit or Decision Sprint?

Sources

• OSFI, Guideline B-10 Third-Party Risk Management
• OSFI, Technology risk bulletin, “Generative and Agentic Artificial Intelligence,” July 2026
• PIPEDA, Schedule 1, clause 4.1.3
• OSFI, Guideline E-23 Model Risk Management (2027)

This page is general guidance. It isn’t legal or regulatory advice.