Using AI Without Putting Your Customers' Data at Risk

The fastest way to get value from AI is to point it at your real data. That's also the fastest way to create a privacy problem, because a lot of that data belongs to your customers and staff rather than to you.

This isn't a reason to avoid AI. It's a reason to make a handful of deliberate decisions early, most of which are straightforward once someone actually asks the questions.

A note before we start: this is general information from a software consultancy, not legal advice. For anything material — particularly health, financial or children's data — get proper advice from a privacy lawyer.

What applies to you

In Australia, the Privacy Act 1988 (Cth) and the Australian Privacy Principles (APPs) set the baseline. Whether they bind you depends partly on size: organisations with an annual turnover of $3 million or less are generally exempt, though there are significant exceptions — health service providers, businesses that trade in personal information, credit reporting, and contractors to the Commonwealth are all covered regardless of turnover.

Two points worth internalising even if you're currently exempt. First, the exemption has been under sustained review, and reforms passed in 2024 have been tightening obligations — including new transparency requirements around automated decision-making that are directly relevant to anyone using AI on personal information. Second, your customers and enterprise clients increasingly ask these questions in procurement regardless of what the law requires of you.

The APPs that matter most for AI work:

• APP 1 — have an open, current privacy policy that actually describes what you do.
• APP 5 — tell people what you're collecting and why, at the time you collect it.
• APP 6 — only use personal information for the purpose you collected it for, or a purpose the person would reasonably expect.
• APP 8 — before disclosing personal information overseas, take reasonable steps to ensure the recipient handles it consistently with the APPs.
• APP 11 — take reasonable steps to protect information, and destroy or de-identify it when it's no longer needed.

APP 6 and APP 8 are the two that catch AI projects. Sending customer records to an overseas model provider is a cross-border disclosure, and using data collected for order fulfilment to train a model is very likely a secondary purpose.

Questions to ask any AI vendor

Get these answered in writing, in the contract or the data processing terms — not from a marketing page.

Is our data used for training? For enterprise and API tiers, major providers generally say no by default. Consumer tiers often differ. This distinction matters enormously and is frequently misunderstood inside businesses, where someone pastes a client contract into a free chat tool.

How long is data retained? Many providers hold inputs briefly for abuse monitoring. Find out how long, and whether zero-retention is available on your plan.

Where is it processed? Most major model providers process in the United States. Some offer regional processing. Either way, you need to know, because APP 8 requires you to.

Who can access it? Under what circumstances can vendor staff see your inputs?

What certifications exist? SOC 2 Type II and ISO 27001 aren't a guarantee of anything, but their absence is informative.

Can we get a data processing agreement? If a vendor won't sign one, that tells you how they think about their obligations.

Structure the system to reduce the risk

The most effective controls are architectural. If sensitive data never reaches the model, most of the questions above stop being difficult.

Send the minimum. This is the highest-leverage decision available. If you're classifying support emails by topic, the model doesn't need the customer's account number, address or payment history. Strip fields before they leave your systems, not after.

Mask identifiers before sending. Replace names, emails, phone numbers and account numbers with placeholders, run the model, then re-insert the real values in your own environment. For many use cases — summarising, classifying, drafting — the model genuinely doesn't need to know who the person is.

Keep the sensitive step local. In a retrieval system, the search over your documents happens in your infrastructure. Only the relevant excerpt goes to the model. That's a much smaller disclosure than handing over your whole corpus.

Use per-purpose pipelines. Resist a single general-purpose "ask anything about our data" tool with access to everything. Narrow tools with narrow data access are easier to reason about, easier to secure and easier to explain.

Log what you send. You cannot assess an incident if you don't know what left your systems. Log the fact and the shape of each request, with the payload itself retained only as long as you genuinely need it.

Set retention deliberately. Decide how long AI inputs and outputs live in your own systems, and enforce it automatically. Indefinite retention by default is how small problems become large ones.

Be honest with people

Transparency is both an obligation and, increasingly, a competitive position.

• Update your privacy policy to describe AI processing in plain language — what's processed, by whom, and where. Vague references to "third-party service providers" are no longer adequate.
• Disclose when people are interacting with AI. Customers dislike discovering it afterwards far more than they dislike being told.
• Explain automated decisions that affect people. If AI influences an outcome that matters — an application, a price, a priority — people should be able to understand that it happened and get a human review.
• Handle access and correction requests. If someone asks what you hold about them, AI-generated notes and classifications form part of the answer.

A useful test: if your customers read a plain description of exactly what you do with their data, would they be surprised? If yes, fix the practice, not the wording.

Where organisations actually get caught

Not usually in the carefully designed system. Usually in the gaps around it:

• Staff using consumer AI tools with client data, because nobody gave them an approved alternative. This is the most common real-world exposure by a wide margin.
• Prompts and logs retained indefinitely in a database nobody thought of as containing personal information.
• Training on production data without checking whether the original collection notice covers it.
• Third-party tools that quietly added AI features to a product you already use, changing where your data flows without you noticing.
• Test environments seeded with real customer data, with weaker access controls than production.

The fix for the first one is cultural as much as technical: publish a short, clear policy on which tools are approved for which data, and make sure there's a sanctioned option that's genuinely convenient. Prohibition without an alternative just moves the behaviour out of sight.

A short checklist before you ship

1. What personal information does this feature touch, and is all of it necessary?
2. Can we mask, truncate or aggregate before anything leaves our systems?
3. Where is it processed, and is that disclosed?
4. What's the retention period — at the vendor and with us — and is it enforced?
5. Does our privacy policy accurately describe this?
6. Are decisions that affect people reviewable by a human?
7. Do we log enough to investigate an incident?
8. Is there an approved tool for staff, so they don't improvise?

None of this is onerous once it's part of how you build. It's considerably more onerous retrofitted after an incident.

---

Building something that touches customer data and want a second opinion on the design? Get in touch. We work through these questions on every AI project we take on, and we'd rather help you get the structure right at the start than review it afterwards.