Some quarters I ship the pilot. In other quarters, I get hired to explain why last year’s pilot is still a demo.
IDC, in a study with Lenovo, found that 88% of AI proofs of concept never reach large-scale deployment. Their explanation never mentions model quality. The authors blame “the low level of organizational readiness in terms of data, processes and IT infrastructure.”
That definitely matches what I see. A pilot becomes a step toward production only if it was built like production from the start, with live data, named owners, a measurable definition of success, and a written answer for the day the AI model is wrong. Two engagements from my own docs show what happens when that groundwork is skipped. My own team missed some of this too, so it is not a story about clients.
The pilot I watched die was a safety-monitoring system for an industrial operator. Computer vision would monitor live camera feeds for unsafe behavior, and the company’s own reviewers would adjudicate any issues it flagged. We spent several months revising the proposal through several versions. A time-and-materials statement of work eventually went out covering a nine-role delivery team, and our pipeline tracker recorded the deal as won (someone had even picked the start date). It never became a delivery. A conventional software project for the same client, won in the same window, started on schedule and ran without issues.
POC built on curated data not reflecting production conditions
On paper, AI system looked ready, and the strongest model scored near-perfect during training and performed great during testing. The trouble started when an executive asked what its alerts were worth. When measured against realistic footage, roughly 19 of every 20 alerts pointed at innocent behavior, and a plain set of handwritten rules did better than every machine-learning model we tried.
The explanation was quite obvious once we saw it: violations are rare, so a system that almost never flags anyone is right almost all of the time. The score and the usefulness of its alerts had very little to do with each other.
The training footage was ready and staged, with people who knew they were being filmed. Production meant more than 25 categories of violations caught at bad angles in the cluttered corners of a working site, and nobody had measured the distance between the two until the project was already almost 6 months old.
No integration architecture, no workflow analysis
The second story is a voice AI receptionist we were asked to review, which was already live at a healthcare group and taking hundreds of calls a day across its clinics. The recording we analyzed made for uncomfortable listening. Less than half a minute in, a date-of-birth mismatch triggered a transfer to a receptionist, then the transfer failed, and the caller hit a dead end inside the product’s first workflow. Ten minutes in, the agent announced that a confirmation link had just been sent, though no send event existed anywhere in the system.
Neither failure had anything to do with the model, which was arguably the only component doing its job. Everything broke at the joins: a transfer with no fallback, and an action that claimed to execute without verification.
Vague success criteria
Partway through the safety project, our ML team sent the client 15 questions that needed answers before anyone could label a single training example, starting with where the line sat between ordinary movement and a violation.
Nobody objected to the questions. Eight days later, a file came back titled “Answers on ML requirements,” and it contained the same 15 questions with no answers attached. The client was not being difficult. They were discovering, question by question, that they did not know their own rules. Their prohibition list had been written for human reviewers to exercise judgment, listing conduct that was banned whenever a reviewer suspected or deemed it improper. No classifier can be trained on the word “suspected.”
One decision would have saved the project. Had we refused to write model code until the client labeled a few hundred real recorded sessions and signed off on every violation category. A stalled build would have shrunk into a modest data-readiness engagement, and the client would have gladly paid for it. Somehow, we ended up drafting several versions of the proposal instead of one uncomfortable email.
The gap
Strip away the technology and both stories fail in the same place. The causes people wrote down were technical, but each one traced back to an ownership question which nobody had answered. Nobody owns the system after the pilot. The vendor owns everything during a pilot, which feels efficient and helps explain why nothing survives the handover effort. Our safety client had a sponsor who could sign a purchase order, but no one was accountable for producing labeled data. Unfortunately, we let that stand because pointing it out felt like an obstacle to the sale.
The runbook for a wrong answer usually does not exist either. Given the false-alarm rate we measured, most of what the safety system flagged would have been an innocent worker, and no one had decided who reviews a flag, what the accused person is told, how a wrongful accusation gets reversed, or who carries the liability.
Degradation is the third gap. The voice agent ran hundreds of calls per day as ad hoc manual testing, with its prompts in a single Python file and no CI. If a prompt change broke identity verification, the first alarm would be a simple complaint.
Validate data readiness and governance gaps
To ensure success, we now run a facilitated workshop before we provide a quote for a build. Every candidate use case has to fit on one page: the problem in a single sentence, a definition of success someone can measure, the actual data sources by name, and an individual owner for the business, the technology, and the data. Most candidates cannot fill that page, and learning that early on is the point of the exercise.
Time-box the POC to a measurable business outcome
The proof of concept itself runs eight to ten weeks at a fixed, capped price, with success stated in the client’s own metrics, response times, and adoption rather than model scores. Nobody can be held to a goal like “demonstrate feasibility.” Scope gets signed only after the business and data understanding phases close, about two weeks in, so a bad data-readiness finding arrives while the plan is still cheap to change.
Treat POC-to-MVP as an engineering phase
Almost nobody budgets for this phase, and it is where most pilots typically and quietly end. The gap between a demo that works and a system a clinic can rely on is not closed by more modeling. For the voice agent, there were several hundred scripted call scenarios, from wrong-patient ambiguity to emergency symptoms, replayed with every change. Twelve gates evaluate each call trace deterministically instead of asking a model for its opinion, and a single failed gate blocks the build regardless of the overall pass rate. Test and production share one span schema, so any production call can be rerun through the harness. An incident stops being a meeting about what probably happened and becomes a recording you can walk through step by step.
Define human accountability upfront
The line between what the model decides on its own and where a human steps in has to be drawn before launch, in the architecture, where it can be enforced. The voice agent operates from a short, bounded list of permitted actions covering caller verification, booking, and escalation; anything absent from that list is forbidden by default. The same machinery enforces the human boundaries. Nothing is disclosed before identity is verified, destructive actions require explicit confirmation after readback, and emergency intent must reach a human within two turns, each and every time. Intending that a human reviews edge cases is not the same as proving that no booking happens before an identity is verified, and only the second kind of statement survives an audit.
At the end of the day …
The difference between the companies that ship and the ones that stall is mostly a matter of timing. Both groups eventually do the unglamorous work of data readiness, ownership, and hardening. One group does it before the pilot. The other proposes it afterward as a follow-on phase, which is the first thing cut during budget season.
The excuse is expiring anyway. When IDC and Lenovo went back to 800 decision-makers for the CIO Playbook 2026, they found that 46% of proofs of concept had already reached production. Two years ago, an 88% failure rate was simply the industry's condition. Now it is starting to look like a choice.
Here’s something you can do: put one question to the initiative on your desk. When the model is wrong at 3:00 AM on a Sunday, who gets paged, what does the customer see, how is the error reversed, and which document says so? If the answer takes a meeting rather than a sentence, what you are funding is a demo with an attached roadmap.
