Everyone with a slide deck is an AI consultancy now. The gap between firms that have shipped production AI systems and firms that have run workshops about them is large, invisible on a website, and expensive to discover after signature.
The questions to ask an AI consultant before you sign all do one job: they move the conversation from promises to evidence. Ask for a running system, a named failure, a written definition of quality, a precise data answer, a handoff plan, a view on your existing systems, and the conditions under which the firm would turn the work down. We are a vendor in this market, so weigh that. These are the questions we would ask before hiring anyone, including us, and each is on the list because we have watched a buyer skip it and pay for it.
What has the consultant actually shipped?
Proof of shipping is the first filter, because production AI leaves evidence that a demo never does.
1. "Walk me through an AI system you built that is running in production today." Not a pilot, not a demo, not "an engagement with a Fortune 500 company." Then follow up: how long has it run, what broke, and who fixed it? Real production systems produce stories about drift, cost surprises, and edge cases. A consultant with no such stories has no production systems, whatever the logo slide says. Case studies should name outcomes specific enough to check.
2. "What did you build that didn't work?" Anyone shipping real AI has failures: pilots killed at the gate, approaches abandoned mid-build, models that missed the accuracy bar. A firm that claims none has done too little work or lacks the candor to say so. Both disqualify. A firm that runs pilots with kill criteria, the approach in how to run an AI pilot that doesn't waste the budget, will have killed some and can tell you what it changed afterward.
How will they measure quality and protect your data?
Quality and data handling decide whether production AI survives review. Expect precise answers to both.
3. "How will we measure whether the output is good, and who defines it?" This question separates engineering from theater faster than any other. Evaluation is a written definition of acceptable output, measured continuously and owned by a named person. If the answer is a feeling ("our clients are very happy with the quality"), the production system will be graded on feelings too, and feelings do not survive your compliance team. A strong answer treats evaluation as a deliverable: built early, run automatically, and handed to you. Measure is one of the four core functions in the NIST AI Risk Management Framework, so it is fair to ask how the firm covers it.
4. "What happens to our data?" Ask which models see it, under what terms, in whose cloud, and whether it trains anything. OpenAI's enterprise privacy page, for example, states that by default it does not use business data to train its models. A consultant should know those terms for every model it proposes, and what can stay inside your network. In healthcare, finance, insurance, or government, a vague answer disqualifies the firm.
What happens after the consultant leaves?
A production AI system needs an owner after launch, and the answer shows whether you are buying a system or renting one.
5. "Who maintains this after you leave, and what do they need to know?" Prompts drift, costs move, and models retire on the vendor's schedule. Anthropic, for example, gives at least 60 days' notice before it retires a publicly released model. The right answer includes a handoff plan, documentation as a deliverable, and a frank account of what ownership costs. If the answer implies permanent dependence on the consultant, you are leasing a system on their terms. Ask for the exit plan before you sign.
6. "Why does our existing stack matter to you?" AI earns its keep by connecting to the systems your business runs on, and some of those systems are old. A consultant who waves at them ("we'll just call an API") has never pulled structured data out of a twenty-year-old application. Firms with modernization experience ask about your older systems early, because that is where implementation projects stall. It is why we treat ColdFusion and legacy modernization and AI implementation as one practice.
Would they tell you not to do the project?
A consultant worth hiring can name the conditions under which it would advise against its own engagement.
7. "What would make you tell us not to do this project?" This is the most revealing question on the list. A firm that cannot answer it is a sales organization. A credible answer names real disqualifiers: data nobody can access, no internal owner for the workflow, or a use case where errors are intolerable and evaluation is impossible. Hire the firm that has walked away from bad projects. The alternative is a firm that billed for them.
Firms that ship have this evidence on hand, and none of the seven questions will surprise them. To know your own weak points before the first call, an AI readiness assessment names the constraint that will bind first. Our answers are on the AI implementation framework page and in our case studies. Bring the list.