Your AI Project Is a Data Cleanup Project Wearing a Costume

A client buys an AI agent the way some people buy a gym membership. The purchase feels like the accomplishment. The hard part, the part that actually changes anything, is the unglamorous work nobody photographs. With AI the unglamorous work is the data, and it has been hiding in plain sight under a costume that says "model problem" or "prompt problem" or "our people just won't adopt it." Peel the costume off most stalled AI initiatives and you find the same thing underneath: data that was never cleaned, never structured, and never governed for the job you just handed it.

The costume and what's under it

The numbers are blunt. Depending on whose survey you trust, somewhere between 70 and 85 percent of AI projects fail to reach their stated goals. MIT's NANDA initiative looked at enterprise generative AI specifically and found that 95 percent of pilots deliver no measurable return, with only 5 percent extracting real value. You can argue about methodology, but you cannot argue with the direction. Most of this does not work yet, and most of the reason is not the part everyone is looking at.

The Project Management Institute, not an organization given to drama, says "garbage in, garbage out" holds particularly true in AI. One analysis found that in over 70 percent of failed AI projects the data was to blame rather than the algorithm. This is the behavioral-economics part that I find quietly fascinating: we consistently misdiagnose the failure because the costume is more interesting than the body. A model is a shiny object you can demo. A deduplication pass on a customer table is not. So the budget, the attention, and the post-mortem all go to the shiny object, and the actual cause sits in a spreadsheet nobody opened.

Here is the thing the failure rate obscures. The pilots that work are usually not running better models. They are running on better data. Same model, different outcome, and the difference was upstream the whole time.

What data readiness actually means

"Data readiness" gets used like a mood. It is not a mood. It is a checklist, and the useful version is narrower than the enterprise version your data vendor wants to sell you. Readiness is not "do we have a lot of data." Plenty of companies are data-rich and readiness-poor. It is whether the specific data your agent will read is four things at once: accurate (it reflects reality), complete enough (the gaps are not in the fields that matter), structured for retrieval (a machine can find the right slice on demand), and governed (someone owns it and it stays clean after you leave).

The reason this matters more for agents than it did for dashboards is retrieval. When you point a retrieval-augmented system at your documents, it does not read them the way you do. It chops them into pieces, matches the pieces to a question, and hands the model whatever came back. If your source has three slightly different "official" versions of the same policy, the system cannot tell which one is real. It will confidently retrieve the wrong one, and the model will dutifully build an answer on top of it. Snowflake's Emily Spragg framed AI-ready data as needing "the right definitions, the right semantic, the right lineages, the right access controls" and then added the line every executive should tape to a wall: "there's no AI strategy without a data strategy." That is not a vendor platitude. It is the whole game.

A readiness audit you can run this week is short. Pick the one workflow you want the agent to own. List every data source it would need to touch. For each source ask: is there a single source of truth, or several pretending to be? Are duplicates and dead records common? Is it structured (tables, clear fields) or a pile of PDFs and free text? Who owns it, and would they know if it broke? If you cannot answer those for the sources behind your first use case, you are not ready, and no amount of prompt engineering will rescue you. I wrote more about the diagnostic side of this in your dashboard is lying to you, because the same rot that fools a chart fools an agent, only faster and with more authority.

The cleanup sequence that doesn't boil the ocean

The fastest way to kill a data cleanup effort is to make it an enterprise data quality program. Those have a way of becoming permanent, like a kitchen renovation that consumes the house. The trick is to scope cleanup to the agent, not the organization. Clean the slice you are about to use, ship something that works, and let the win fund the next slice.

The sequence I use, in order, because order matters: first, deduplicate and resolve entities, so "Acme Corp," "Acme Corporation," and "acme corp." stop being three customers. Second, fix accuracy at the source, not in a downstream copy, or you will be cleaning the same dirt forever. Third, standardize structure: consistent fields, consistent formats, consistent units, so retrieval has something to grab. Fourth, establish one source of truth per concept and retire the impostors, even if retiring them annoys the three people who liked their version. Only then do you connect the agent.

This is genuinely tedious, and the tedium is the reason it gets skipped and the reason it is worth paying for. I made the full case for the unglamorous economics of it in the database cleanup nobody wants to pay for, but the short version is this: the cleanup is not a cost you incur to do AI. It is most of the value, arriving early, in a form that helps your humans too. A deduplicated, governed customer table makes every team faster, with or without an agent on top.

Structuring and governing so it stays clean

Cleaning data once is like mowing a lawn once. It looks great for a week. Without governance, dirty data regenerates, because the same broken intake forms, the same copy-paste habits, and the same unowned tables that created the mess are still running. This is why 81 percent of companies still struggle with AI data quality according to Qlik, and why data quality has rocketed up the list of AI blockers, cited by 44 percent of teams in 2025, up from 19 percent the year before. The problem is not getting clean. It is staying clean.

Governance for normal companies, not Fortune 50 ones, can be light. Assign an owner to each data source that feeds the agent, so there is a name attached to its health. Validate data at the moment of creation, where a bad entry costs one correction instead of ten thousand. Keep a simple data dictionary so "active customer" means one thing across the company. And before you ever hand data to an outside model or vendor, run it through a basic hygiene and provenance check, the kind I outlined in three questions to ask before you feed data to an AI vendor. None of this requires a governance department. It requires deciding that data has owners, which is mostly a cultural choice wearing a technical hat.

Frequently Asked Questions

Why do AI projects fail because of data?

Because AI amplifies whatever you feed it. A model trained or prompted on inaccurate, duplicated, or unstructured data produces confident, well-formatted wrong answers, which are worse than no answer because people trust them. Surveys attribute the majority of AI project failures to data rather than the model itself, and 99 percent of projects hit data quality issues along the way. The model is rarely the bottleneck. The data behind it almost always is.

What does data readiness for AI mean?

Data readiness means the specific data your AI will use is accurate, complete enough for the task, structured so a machine can retrieve the right piece on demand, and governed so it stays that way. It is not about having a lot of data. It is about whether the data behind your actual use case can be trusted and found. A useful test: pick one workflow, and ask whether every source it depends on has a single source of truth and a clear owner.

How do you clean data before an AI implementation?

Scope it to the use case, then work in order: deduplicate and resolve duplicate records, fix accuracy at the source rather than in copies, standardize structure and formats, and establish one source of truth per concept. Clean only the slice your first agent touches, ship it, and let that win fund the next slice. Avoid turning it into a company-wide data lake project, which is how cleanup efforts quietly die.

How do you prepare data for AI retrieval (RAG)?

Retrieval systems chop your documents into pieces and match those pieces to a question, so messy sources produce messy answers. Prepare data by removing duplicate and conflicting versions, giving documents clear structure and metadata, and maintaining a single authoritative version of each fact. The cleaner and better-labeled the source, the more often the system retrieves the right slice instead of a plausible-looking wrong one.

References

• Dynatrace, Why AI projects fail (Gartner and Vanson Bourne figures)
• Tale of Data, Why AI projects fail: data quality
• Project Management Institute, Why Most AI Projects Fail
• MIT NANDA, The GenAI Divide: State of AI in Business 2025
• Qlik, Data Quality is Not Being Prioritized on AI Projects (2024)
• Bigeye, The Data Quality Crisis Killing AI Projects (2025)
• Gartner, How to Improve Your Data Quality ($12.9M/yr cost)