Your Data Isn't Ready for AI — and What to Do About It

There's a predictable moment in most AI projects, usually about a week in. Everyone is enthusiastic, the prototype looked great in the demo, and then someone connects it to the real data and discovers that the customer table has four spellings of the same company name, the "status" field contains eleven values that mean roughly three things, and nobody can explain why 8% of orders have a delivery date before their order date.

This isn't a sign the project is doomed. It's completely normal, and it's the actual work. The uncomfortable truth is that most AI and automation projects are data projects wearing a more exciting hat.

The good news is that you almost certainly don't need the enterprise data programme a consultancy will happily sell you. You need to fix a specific, small set of things.

What "ready" actually means

Ignore the maturity models. For a practical automation or AI project, data readiness comes down to five questions about the specific data that project touches.

Can you get to it? Is the data reachable by software — through an API, a database connection, an export that can be automated? Or does it live in a system where the only way out is someone clicking "download CSV" every morning? Accessibility is the first gate, and the one most likely to stop a project dead.

Is it consistent? Does the same real-world thing appear the same way each time? "ACME Pty Ltd", "Acme Pty. Ltd.", "ACME" and "acme pty ltd" are one customer to you and four to a computer.

Is it complete enough? Not perfect — enough. If the field you need to reason about is empty 60% of the time, no model will rescue that.

Is it current? Data that updates nightly is fine for reporting and useless for a system that answers live customer questions.

Can you tell what's authoritative? When the CRM and the accounting system disagree about an address, which one wins? If nobody can answer, every automation built on top will inherit the ambiguity.

Notice what isn't on that list: volume. You need far less data than people assume for most useful AI work. Modern models bring their own general capability — what they need from you is accurate, retrievable context, not a warehouse.

Scope the assessment to the project

The single biggest mistake we see is treating data readiness as an organisation-wide programme. Someone proposes a data strategy, a governance framework and a two-year roadmap, and the automation that would have saved twenty hours a week never gets built.

Do the opposite. Pick the one process you want to automate, and assess only the data it touches.

A realistic version of this takes a few days:

1. List the fields the automation actually needs. Usually fewer than fifteen. Be strict — "nice to have" fields cost real cleanup effort.
2. Pull a genuine sample. Several hundred real records, including recent ones and awkward ones. Not a curated extract.
3. Profile it. For each field: how often is it empty, how many distinct values, what are the outliers, what formats appear. This is an afternoon of scripting and it is remarkably revealing.
4. Check it against reality. Take twenty records and verify them against the source of truth — the actual invoice, the actual contract. This is where you find the problems profiling can't see.
5. Write down what's broken and what it would cost to fix.

That list is your project plan. Everything not on it can wait.

The problems worth fixing, and the ones that aren't

Not all data problems block automation. Knowing which are which saves months.

Fix these:

• Inconsistent identifiers. If you can't reliably match a record across systems, nothing downstream works. This is the highest-priority fix, every time.
• Fields that mean different things to different teams. When "active" means one thing to sales and another to finance, an automation will silently do the wrong thing for one of them.
• Free text holding structured data. Delivery instructions containing the actual delivery date. Notes fields carrying pricing. This is common and it's exactly where AI extraction can help — you can often turn the mess into structure rather than cleaning it by hand.
• Missing values in fields the logic depends on. Either backfill them or design the automation to handle absence explicitly.
• No timestamps. If you can't tell when a record changed, you can't sync incrementally and you can't debug anything.

Leave these alone for now:

• Historical data you'll never process. Clean what's live and what's coming.
• Cosmetic inconsistencies in fields nothing reads.
• Perfect deduplication. Get the matching good enough for the task, not flawless.
• Fields belonging to processes you aren't automating.

Clean the data the automation reads. Resist cleaning the data that merely offends you.

Fix it at the source, not in the pipeline

When you find a problem, there's a fork: fix it where it's created, or fix it every time it's read.

Fixing at the source is almost always better. If the reason you have four spellings of a customer name is that staff type it free-hand into three different systems, a cleanup script will produce clean data for exactly as long as it takes someone to create the next record. Add a dropdown, a validation rule, or an integration that populates the field automatically, and the problem stops recurring.

Pipeline-level cleaning is the right answer when the source is a system you don't control, when the data is historical and closed, or when the transformation is genuinely a matter of interpretation rather than an error.

A practical pattern: clean the history once, fix the input so it stops happening, and keep a light validation step in the pipeline to catch what slips through. That last part matters — data quality isn't a state you reach, it's a property you maintain.

Use AI on the data problem itself

There's a pleasing circularity here that's easy to miss. A lot of data cleanup work is exactly the kind of messy, judgement-based reading that modern models handle well:

• Entity matching. Deciding whether "ACME Pty Ltd, 12 Smith St" and "Acme Proprietary Limited, 12 Smith Street" are the same organisation.
• Extracting structure from free text. Pulling dates, amounts and references out of notes fields into proper columns.
• Classifying inconsistent categories into a clean taxonomy.
• Flagging implausible records for human review — delivery before order, prices orders of magnitude off, addresses that don't parse.

The pattern that works is the same as everywhere else: the model proposes, a person approves the ambiguous cases, and the confident matches go through automatically. For a one-off cleanup of tens of thousands of records this can turn a months-long manual exercise into a fortnight — and the same approach we describe for document processing applies directly.

Build in quality monitoring from the start

Once your automation is live, data problems become production incidents. An upstream system changes a field format, and your pipeline starts silently producing nonsense.

Worth having:

• Validation at the boundary. Check incoming records against expectations — required fields present, values in range, formats as expected — and route failures to a review queue rather than processing them.
• Volume alerts. If you normally process 200 records a day and today it's 4, something upstream broke. This one check catches an enormous proportion of real incidents.
• Distribution alerts. If a field is normally 5% blank and it's suddenly 40%, tell someone.
• A visible exception queue. Somewhere records go when they can't be handled, that a human actually looks at.

This is the same discipline described in our post on integrations that don't break, and it applies just as much to the data flowing through them as to the connections themselves.

What this costs

For a single automation, a focused data assessment and remediation typically runs $8k–$25k — a genuine line item, but small against the cost of building an automation on foundations that don't hold.

The wrong comparison is against doing nothing. The right one is against building the automation anyway and discovering the problems in production, where they cost more and damage trust in the system you're trying to get people to adopt.

Start here

If you're considering an AI or automation project and want to de-risk it cheaply: take 200 real records from the system it will touch, and have someone spend a day profiling them honestly. You'll know within 24 hours whether you have a data problem, how big it is, and whether it's worth solving before you commit to anything larger.

That day is the highest-return time you can spend at the start of one of these projects.

---

Wondering whether your data is in good enough shape to build on? Send us a sample — we'll profile it, tell you plainly what would need fixing, and whether the automation you have in mind is realistic on what you've got today.