AI Vendor Evaluation Scorecard
Stop evaluating vendors by demo quality. Rate your two shortlisted vendors across 9 due-diligence dimensions and see which one wins, and where each carries hidden risk.
Listen to this briefing
Vetting AI Vendors Beyond the Demo
How the scorecard reads two vendors
Each vendor gets a 1 to 5 rating on nine dimensions. The ratings become a weighted score out of 100, using the weight you give each dimension. Any rating of 2 or below raises a risk flag whatever the total, because one weak dimension can outweigh a strong average.
A higher total can still carry a risk flag
A leads by 5 points. A gap of 10 or less means the dimension that matters most to you should break the tie, and A's flag needs a written mitigation before signing.
Illustrative ratings for two unnamed vendors. The score formula, the 2-or-below flag and the 10-point tie-break are the ones this scorecard applies, with every weight at Standard.
Next: name both vendors, then rate each dimension for each one. The comparison appears once each vendor has three ratings.
Rate at least three dimensions for each vendor to see the comparison and a recommendation.
Vendor A
–
out of 100
Vendor B
–
out of 100
This tool produces a self-assessment based on the ratings you enter. It does not constitute independent testing or verification of either vendor, and it is not procurement, contractual or legal advice. The recommended steps above are the starting point for your own due diligence, which remains your responsibility. Your ratings stay in your browser and are neither sent to Terence Kok nor reviewed by anyone.
The criteria that separate a good vendor choice from a costly mistake
Integration, TCO and speed
- Integration Fit is the hidden multiplier on every other cost. A technically capable product that requires 12 months of integration work to deploy is not a low-cost option.
- TCO is often understated in vendor proposals. As a directional pattern from practice, licensing commonly runs 40–60% of all-in cost. The rest is implementation labour, training, maintenance, and governance overhead.
- Time to Deploy matters because value not yet realised is value lost. A 6-month deployment difference has compounding ROI consequences.
Governance, portability and risk
- Data Governance scores below 3 are a legal and reputational risk. PDPA, GDPR, and sector-specific regulations make vendor data practices a board-level concern.
- Portability scores below 3 mean you are signing a long-term contract even if the paper says 12 months. Data lock-in, API dependencies, and proprietary formats all compound over time.
Support and proof of value
- Support Quality is where most vendor relationships fail post-signature. The account executive who closed the deal is rarely the person handling your production incidents at 2am.
- Proof of Value at score 1–2 means you are being asked to be a reference customer, not joining one. Ask for 3 references from organisations similar to yours before scoring above 3.
Tool-ecosystem security and pricing risk
- Tool-Ecosystem Security covers how the vendor secures the MCP servers, plugins, and third-party connectors its agents call. Unauthenticated MCP implementations produced real, documented vulnerabilities in 2026: ask specifically how tool connections are authenticated and scoped.
- Pricing Model Risk separates predictable pricing from token or credit-based models where the same usage pattern can produce wildly different invoices month to month. Ask for a worst-case cost scenario at your expected volume.
Vendor demos are designed to hide exactly the things this scorecard forces into the open, total cost of ownership, portability, what support looks like once the sales rep is gone. Rate two vendors side by side across nine dimensions and you will usually find the gap somewhere you did not expect. I added the risk flags after seeing too many contracts signed on a strong demo and a weak governance answer nobody pushed back on. Use it before the meeting where you're expected to have a recommendation.
Last updated 2 Oct 2026
More in Build, buy and run
- AI Build vs. Buy vs. PartnerShould we build this ourselves, buy a platform, or partner?
- Data Architecture Decision ToolLakehouse, data mesh or data fabric: which one fixes our data first?
- Fine-Tune vs. Retrieval-Only ClassifierIs this dataset safe to train into a model, or should it stay behind retrieval?
- AI Incident Response PlannerIf one of our AI systems fails tomorrow, who switches it off, and which regulator clock has started?
- Model Performance and Health DashboardIs our deployed model drifting, slowing down or failing?
- Human-AI Interaction and Decision QualityAre our people over-trusting or under-using the AI?
