Most AI pilots do not fail. They conclude. A demo gets shown, everyone agrees it was interesting, and nothing changes. Six months later the budget line moves somewhere else and the organization files AI under "we tried it."
To run an AI pilot that does not waste the budget, scope it to produce a decision. Aim it at your binding constraint, measure a baseline first, write kill criteria in advance, and run it on real data with real users. A pilot that skips any of those can succeed and still tell you nothing.
Why AI pilots end without a decision
AI pilots end without a decision because they test what is easy to test instead of what will block production. In a July 29, 2024 press release, Gartner predicted that at least 30% of generative AI projects would be abandoned after proof of concept by the end of 2025. It named the causes: poor data quality, inadequate risk controls, escalating costs, and unclear business value. None of those is a model problem. Each is a question a pilot could have answered.
The version we see most often is a chatbot pilot run on public documentation, because access to the real knowledge base needed a security review. The pilot succeeds. The production version needs the security review anyway. The organization has learned that the vendor can build a chatbot, which it already knew, and pushed the hard question back by a quarter.
Find the binding constraint first
A binding constraint is the one condition that will stop AI in your organization before anything else does. For some organizations it is data access: the information the AI needs sits in systems nobody can get it out of cleanly. For others it is workflow ownership: nobody has the authority to change the process the AI should improve. For others it is evaluation: nobody can say what a good output looks like, so nobody can say whether the AI produced one.
A pilot that does not touch the binding constraint tells you nothing, however well it goes. That is why we run a readiness assessment before the pilot, not instead of it. The assessment names the constraint, and the pilot aims at it. What an AI readiness assessment actually tells you explains how to read the result.
What should an AI pilot measure?
An AI pilot should measure one decision against a baseline, with the stop conditions written down before work starts. Four scoping rules make that happen.
Name the decision. A good answer: "whether extraction accuracy on our real documents is high enough to remove the manual review step." A weak answer: "whether the team finds it useful." If no decision depends on the outcome, the pilot is theater.
Measure the baseline before the first prompt. Record time per task, error rate, or backlog size for the current process. A rough number is enough. Without one, nobody can claim improvement, and someone in the budget meeting will correctly say so.
Write kill criteria in advance. Kill criteria are the results, agreed before the pilot starts, that mean stop. Pilots without them do not end. They fade, and they use up attention and goodwill as they go. A clean kill is what makes the next pilot fundable.
Use real data and real users, or call it a rehearsal. Sanitized data and volunteer users produce sanitized results. Run the pilot inside the actual workflow, on the data production would see, security review included. If that costs too much for a pilot, that is the finding: your binding constraint sits upstream of the AI.
If your team has no evaluation method of its own, the Measure function of the NIST AI Risk Management Framework is a reasonable place to start.
The shape that works
The pilots that produce decisions share a shape: six to eight weeks, one workflow, a small group of real users, an explicit baseline, explicit kill criteria, and one accountable person on the client side who owns the workflow. Small enough to kill without embarrassment. Real enough that success means something.
Build it on the plumbing production would use. An orchestration layer like n8n is cheap to stand up for six weeks, and it keeps a pilot that works from being rebuilt from scratch on its way to production. Model debates, platform commitments, and licensing negotiations wait until the pilot shows that the workflow improves. Made earlier, they are expensive guesses.
Before the pilot starts, also agree on what a successful result commits the organization to do. If the pilot hits its numbers and the answer is still "we'll see," it was a way of appearing to act. AI that ships, not AI that demos, starts with a commitment someone makes in advance.
Our AI implementation framework treats the pilot as one stage of a production build, not a standalone science project. If you inherited a pilot that someone else built, productionizing someone else's AI proof of concept covers that handoff.