Every vendor demo is a tour of the best case. Twenty questions move the conversation to the ordinary case and the bad one, which is where the difference between products actually lives — and a vendor who answers all twenty comfortably has told you more than any feature list.
This guide covers the twenty questions in five groups, what each answer reveals, which answers should stop the conversation, and how to score an evaluation.
Group 1 — Integration (5 questions)
Where most of the cost and nearly all of the disappointment lives.
- Which of my systems do you integrate with, by name and version? A generic list means a generic proposal.
- Is the connection real-time or batch, in seconds? A nightly sync cannot support a fast-response promise, whatever else is true.
- Which fields do you read, and which do you write? Read is cheap and safe. Write touches your system of record.
- Is integration priced per rooftop? The most common surprise, per what integration services contain.
- How do I know when the connection breaks? Integrations fail silently. If there is no alerting, you find out from a gap in lead flow.
Stop the conversation if: the sync is nightly and the pitch is about speed.
Group 2 — Behaviour and escalation (4 questions)
- What does it do with a price question? The right answer is that it escalates, and says so honestly.
- Show me your escalation trigger list. A product with a written list has thought about this. One without will improvise in front of your customers.
- What happens when it is not confident? "It makes its best guess" is a real answer and a bad one.
- How do I change the escalation rules without calling support? If the answer is a support ticket, every rule change takes a week.
Stop the conversation if: it quotes prices or trade values to customers.
Group 3 — Compliance (4 questions)
- Show me the audit log for a specific message, with the consent state at the time it was sent. Live, in the demo. Not a documentation claim.
- How are opt-outs handled across systems? An opt-out in service must stop sales messages too.
- Are contact-time windows enforced in the customer's time zone? The most common silent violation, covered in SMS compliance.
- Who are your subprocessors, and will you notify me when they change? Part of the standard service provider clauses.
Stop the conversation if: they cannot show a real log entry with consent attached.
Group 4 — Operation (4 questions)
- What does this require from my team weekly? A tool needing weekly configuration will stop working at a store with no admin.
- What happens if nobody picks up an escalation? The handoff is where front-of-funnel gains are lost.
- Does it work on a phone? Your people are on the lot.
- What does onboarding actually involve, in days? Compare against the implementation fee being quoted.
Stop the conversation if: it assumes a dedicated administrator you do not have.
Group 5 — Commercial and exit (3 questions)
- What is the single-rooftop price, and what is the billable unit? Definitions drive the bill — see pricing models.
- What has the renewal increase been for existing customers? The number nobody volunteers.
- If I leave, what do I take and what does it cost? Data export, notice period, and what breaks.
Stop the conversation if: there is no month-to-month option on an unproven tool.
How to score it
Mark each answer clear, vague or evasive. Then:
Pattern Reading Mostly clear, two vague Normal. Follow up on the two Clear on features, vague on 1–5 Integration will be the project, not the product Clear everywhere except 10–13 Compliance is your problem, not theirs Evasive on 18–20 The commercial terms are the product Clear on all twenty Rare, and worth a lot The pattern matters more than the count. Vagueness clustered in one group tells you exactly where this deployment will go wrong.
What the answers actually reveal
Clear answers on integration mean the vendor has done this with your systems before. Vague ones mean you will be the first, and you will pay for the learning.
A written escalation list is the single best proxy for product maturity in this category. It takes real deployments to discover the twelve things a system must never answer.
A live audit log separates products built for regulated retail from products adapted to it.
Honest exit terms correlate with confidence. A vendor who makes leaving easy expects you to stay.
Where evaluations go wrong
1. Evaluating on the demo. The demo is the best case, scripted.
2. Not asking for the escalation list. The most informative single request on the sheet.
3. Accepting documentation instead of a live log. "Our platform maintains comprehensive audit trails" is not a log entry.
4. Skipping the exit question. It feels adversarial and it is the one that tells you the most.
5. One person evaluating. The GM sees the dashboard, the BDC manager sees the daily reality. Both should be in the room.
6. Evaluating without measuring first. A checklist cannot tell you whether you need the product — only your own three numbers can.
What should you measure, afterwards?
| Metric | How to compute | What it checks |
|---|---|---|
| Answers verified in practice | Claims that held, after 60 days ÷ claims made | The honesty score |
| Escalation accuracy | Triggers fired correctly ÷ should have | Whether the list was real |
| Integration downtime | Hours the connection was down, unnoticed | Whether alerting exists |
| Time to change a rule | Request to live | Whether question 9 was answered honestly |
| Actual first-year cost ÷ quoted | Ratio | Whether group 5 was straight |
Run this at 60 and 180 days and keep the result. It is the most useful input to the next evaluation you run, regardless of vendor.
Frequently asked questions
What should I ask a dealership AI vendor before buying?
Twenty questions across five areas: integration by system name and sync latency, escalation behaviour including a written trigger list, compliance including a live audit log entry with consent state, day-to-day operation including what it needs weekly, and commercial terms including the billable unit, renewal history and exit cost.
What is the most revealing single question?
Ask to see the written escalation trigger list — the specific conditions on which the system stops and hands to a person. It takes real deployments to discover those, so a product that has one has been used in anger, and one that has not will improvise in front of your customers.
What answers should end the conversation?
A nightly sync paired with a speed pitch, a system that quotes prices or trade values to customers, an inability to show a real audit log entry with consent attached, an assumption that you have a dedicated administrator, and no month-to-month option on an unproven tool.
Why ask about subprocessors?
Because an AI vendor is a service provider handling your customer information, and the model provider, hosting and logging services behind it are handling it too. Knowing who they are, and being notified when the list changes, is a standard clause rather than an unusual request.
How do I check compliance claims rather than accept them?
Ask for a live demonstration: one real message, with the consent state that existed at the moment it was sent. Documentation describing comprehensive audit capabilities is not the same as a system that can produce the record when asked.
Who should be in the evaluation?
At minimum the GM and whoever runs the department the tool touches daily. The two see different things — the dashboard and the daily reality — and products that impress one frequently frustrate the other.
Does a checklist tell me whether I need the product?
No. It tells you whether a product is sound, not whether it addresses your largest leak. That requires measuring first-touch time by day-part, filtered missed calls and aged inventory, then buying against the biggest number rather than the best demo.
What should be checked after buying?
Which claims held after 60 days, escalation accuracy against the list that was shown, integration downtime that went unnoticed, how long a rule change actually takes, and actual first-year cost against the quote. The record is the most useful input to your next evaluation.
Conclusion
- Five groups, twenty questions. Integration, escalation, compliance, operation, exit.
- The written escalation list is the best maturity proxy in the category.
- Ask for a live audit log entry, not a documentation claim.
- Vagueness clustered in one group tells you where the deployment will go wrong.
- Score the answers again at 60 days. That record is worth more than the next demo.
Last updated: