Article

How to Keep Sensitive Data Out of Your LLMs

Somewhere in your organization right now, someone is probably pasting a customer record, a patient note, or a spreadsheet of account numbers into an AI tool to get a task done faster. They're not trying to cause a breach — they're trying to hit a deadline. But the data doesn't know the difference.

Research from Salesforce found that 27% of enterprise employees have entered confidential company data into public AI tools, including customer records and financial information, and Cyberhaven's analysis of real workplace AI usage found that 11% of everything pasted into tools like ChatGPT contains sensitive or confidential information (Airia, "Shadow AI Statistics"). The volume is accelerating fast: the amount of data shared with AI tools grew 485% year-over-year, and Verizon's 2026 Data Breach Investigations Report found that regular AI use on corporate devices tripled in twelve months, from 15% of employees to 45% — two-thirds of it happening through personal accounts the organization can't see at all (LayerX, "2026 DBIR Shadow AI Findings"). If your data governance plan doesn't account for this, it isn't a plan — it's a hope.

The three paths sensitive data takes into an LLM

"Keeping data out of LLMs" isn't one problem, it's at least three, and each one needs a different fix:

  1. Prompts. An employee types or pastes sensitive information directly into a chat interface, whether that's a sanctioned enterprise AI tool or an unsanctioned public one.

  2. Retrieval and tool context. A RAG pipeline or AI agent pulls records from a database or document store into the model's context window to answer a question — often without anyone reviewing what got pulled in.

  3. Training and fine-tuning data. A team fine-tunes a model on real production data to improve accuracy for a specific use case, embedding whatever was in that dataset into the model's weights.

Each path has already been documented as a real leakage vector, not a theoretical one. Source code is the single most common type of sensitive data flowing into unauthorized AI tools, but structured data, images, and internal documentation are close behind (LayerX, "2026 DBIR Shadow AI Findings"). And on the training side, researchers using divergence-based extraction attacks found that a meaningful share of a fine-tuned model's outputs contained memorized PII that was traceable back to training data — meaning the risk doesn't disappear once the data has "gone into" the model (DataMasque, "Risks of Using Production Data in Test Environments").

Why policy alone doesn't solve this

Most organizations' first move is a policy: "don't paste sensitive data into AI tools." It's necessary, but it's not sufficient, for a simple reason — most of this activity happens somewhere security can't see it. Two-thirds of employee AI use happens through personal, non-corporate accounts, and only 37% of organizations have any policy in place to manage or detect shadow AI usage at all (LayerX, "2026 DBIR Shadow AI Findings"). A rule nobody can enforce isn't a control, it's a talking point.

There's also a vendor-side blind spot most teams underestimate. A 2026 industry analysis of 2,400 AI-powered software vendors found that 63.6% did not disclose their third-party AI subprocessors in legal documentation, and nearly a third of AI-enabled systems studied also engaged in at least one other high-risk activity, like processing sensitive personal data or supporting automated decisions (DataGrail, "Privacy and AI Trends Report 2026"). Even AI tools your organization officially sanctioned may be routing data somewhere you didn't approve.

A practical framework: redact at capture, not after the fact

The fix that actually holds up is architectural, not just administrative: make sure sensitive data never reaches an AI surface in its raw form in the first place.

  • Redact and mask before data leaves its source system. Field-level redaction and pattern-based detection should run at capture time — before a record enters a prompt, a tool call, or a fine-tuning dataset — rather than being cleaned up in logs after the fact. The Cloud Security Alliance recommends exactly this pattern for AI environments: deploy automated PII detection and redaction, using tools like Microsoft Presidio or Google's DLP API, before any human or model sees the data (Cloud Security Alliance, "Data Security within AI Environments").

  • Minimize what enters the context window. Favor retrieval that pulls only the fields a query actually needs over dumping full records into a prompt "just in case" (Amestris, "DLP for LLM Systems").

  • Discover shadow AI usage before you try to govern it. You can't write policy for tools you don't know are in use — start by identifying which AI tools and integrations are actually touching your data, sanctioned or not.

  • Use masked or synthetic data for fine-tuning by default. If a model needs realistic data to perform well, give it realistic masked data as the default option, not a manual exception someone has to request.

This is the same discovery-mask-deliver sequence that matters for securing production data before it reaches AI workflows generally — LLM prompts and RAG pipelines are just the newest and fastest-moving version of that problem.

Why this matters more in regulated industries

In healthcare and financial services, this isn't only a security question — it's a regulatory one. Entering PHI into an AI tool without a signed business associate agreement covering that use can itself be a HIPAA violation, independent of whether any breach ever occurs. The same logic applies under GLBA and PCI DSS for financial data: the obligation is about controlling access and use, not just preventing theft. Regulators have shown they're willing to treat "we didn't realize the AI vendor had access to that data" as a compliance failure, not an excuse. (Related reading: Data Masking for AI Training Data.)

A short checklist for your next AI rollout

  • Do you know which AI tools — sanctioned and unsanctioned — currently have access to sensitive data in your environment?

  • Is sensitive data redacted or masked before it reaches a prompt, tool call, or fine-tuning dataset, or does that happen manually and inconsistently?

  • Does your RAG or agent pipeline retrieve only what's needed for a given query, or does it pass full records into context?

  • Have you reviewed your AI vendors' subprocessor disclosures, or are you assuming their legal documentation covers what you think it does?

If you're not confident in the answers, that's the gap to close before your next model or agent goes into production — not after.

The bottom line

LLMs and AI agents aren't going away, and neither is the pressure to give them real, useful data. The organizations getting this right aren't the ones with the strictest policy documents — they're the ones who've made redaction and masking automatic, so sensitive data simply never reaches the model in a form it could leak.

C² Data Privacy Platform discovers sensitive and shadow data across your cloud, database, and SaaS environments and masks it before it ever reaches a test, analytics, or AI workflow — so your teams can build with real-feeling data without the exposure. Book a demo to see it run against your own schema.

Sources: Airia — "Shadow AI Statistics: Key Data Points Every CISO Needs in 2026", LayerX — "2026 DBIR Shadow AI Findings", DataGrail — "Privacy and AI Trends Report 2026", Cloud Security Alliance — "Data Security within AI Environments", Amestris — "DLP for LLM Systems", DataMasque — "Risks of Using Production Data in Test Environments", C² Data Technology.