Chapter 3 — Data Security and Privacy

Published on: 2026-05-12 Last updated on: 2026-07-13 Version: 2
Chapter 3 — Data Security and Privacy

Chapter 3 — Data Security and Privacy

Third post of the chapter-by-chapter walkthrough of LLM Primer VII: AI Security. The chapter that treats data as an asset with a lifecycle — from training corpora that the model has partially memorised, to user inputs that Samsung engineers pasted into ChatGPT before the incident had a name.

Why this chapter exists

The threat model of Chapter 2 named data as one of six asset categories. Data security in LLM systems has enough distinctive properties to deserve a chapter of its own. Training corpora contain copyrighted material, personal information, and licensed content whose terms drift over time. Trained models memorise fragments of that corpus in ways an attacker can extract. Production inputs are themselves sensitive data — the Samsung incident in April–May 2023 put that beyond dispute — and their handling is a security concern independent of the model's behaviour. The Italian Garante's action against ChatGPT in March 2023 established that data-protection law applies to these systems whether or not the developers designed for it. This chapter walks the data lifecycle from ingestion to deletion.

One line: A trained model is a lossy, distributed compression of its training data; a production system is a growing archive of user inputs; each is a data-security artefact whose confidentiality, integrity, and retention have to be managed as deliberately as any database.

3.1 Training corpora carry copyright, PII, and licence drift

Frontier models are trained on corpora too large for any human to read end to end — Common Crawl, Wikipedia, books, scraped web text, code, licensed feeds, synthetic data. The scale is the source of capability and the source of risk. The first risk is copyright: the legal status of training on copyrighted material has been contested since 2023, with major suits from news organisations, authors, image-rights holders, and code-licence enforcers, and positions from the US Copyright Office, the EU, and the UK converging on the recognition that training is at least sometimes a copyright-relevant activity. The second is personal information: web-scraped corpora inevitably contain names, contact details, professional histories, court records, leaked databases, and credentials. GDPR, CCPA/CPRA, PIPL, DPDPA, LGPD, and PIPEDA all apply to these regardless of the model provider's location. The Italian Garante's March 2023 action against ChatGPT was the first regulatory shot; more have followed. The third is licence drift — the gradual divergence between what an organisation believes it may train on and what the underlying contracts actually permit. A per-corpus licence register linked to the training-data manifest is the discipline that keeps that answer definite.

3.2 Memorisation is a property; extraction is an attack

A trained model is a lossy, distributed compression of its training data. Most training documents are not directly recoverable, but the compression is imperfect. Carlini and colleagues' 2021 USENIX paper "Extracting Training Data from Large Language Models" demonstrated that GPT-2 could be induced to emit verbatim training examples — names, phone numbers, email addresses, code snippets — by prompting with carefully chosen prefixes. The 2023 ICLR follow-up "Quantifying Memorization Across Neural Language Models" showed memorisation scales with model capacity, corpus size, and example duplication: bigger models memorise more, and de-duplication helps but does not close the gap. Nasr and colleagues' 2023 "Scalable Extraction of Training Data from (Production) Language Models" extended the attack to aligned production models including ChatGPT via a divergence attack that broke instruction-following and reverted the model to raw training-data output. It is worth distinguishing the two ideas: memorisation is a property of the model; extraction is what an adversary with API access does with that property. A model can memorise content that is never extracted, and an extraction attempt can fail against a model whose memorisation is high. The mitigation stack is de-duplication, alignment training that resists divergence, output filtering against verbatim matches to canary strings, and rate limits that raise the cost of high-query attacks.

3.3 User inputs are a data category that needs management

The Samsung incidents of April–May 2023 — three separate cases in which semiconductor engineers pasted confidential source into ChatGPT — made the point in the plainest possible terms. In an LLM-integrated system, user input is itself a data category. Every production deployment has to answer, in writing: which categories may users legitimately send; which categories must be detected and blocked or redacted before the prompt reaches the model; where are inputs stored, by whom, and for how long; are inputs used for any subsequent training and has the user consented; who can read them in stored form; what happens on deletion request. The absence of an answer is a policy gap. For systems serving regulated domains, a redaction pipeline between the user-facing layer and the model — Microsoft Presidio is one open-source toolkit — detects PII and masks, replaces, or refuses depending on policy. The operational disciplines that protect data elsewhere all apply: encryption at rest for the training, fine-tuning, prompt-template, retrieval-corpus, log, and cache stores; TLS or mTLS in transit; per-tenant isolation across storage, prompt-construction, and log paths; and a retention policy with a defined service level for GDPR Article 17 or CCPA deletion requests. Multi-tenant leakage is the failure mode that most reliably ends deployments; the discipline required to prevent it is comparable to multi-tenant database isolation, plus the wrinkle that the model itself is shared.

Worth holding onto: The training corpus, the fine-tuning data, and the production input stream all become part of the security boundary the moment they exist. Data lifecycle in LLM systems runs longer than in traditional systems because training data keeps influencing outputs long after the data itself has been "processed."

What Chapter 3 sets up

Chapters 1 through 3 complete Part I: foundations. Part II turns to the operational interior. Chapter 4 takes up prompt injection and jailbreaks — the OWASP LLM01 problem — drawing on Willison's original framing, Greshake's indirect-injection paper, the universal-suffix work of Zou et al., Wei et al.'s jailbreak taxonomy, and Wallace et al.'s instruction-hierarchy training. Chapter 5 develops the input-validation and output-filtering layers into operational discipline, with the guardrail tooling and adversarial-testing frameworks the field has converged on. Chapter 6 takes up retrieval-augmented generation specifically, where the data risks of this chapter and the injection risks of Chapter 4 meet. The connecting principle is that security in these systems is a property of the architecture, not a feature of the model.

Next — Chapter 4: Prompt Injection and Jailbreaks. Why the SQL-injection analogy only goes so far, the jailbreak families that have survived every model update, and the layered mitigation strategy the rest of Part II builds out.

Want the full picture? The book chapter includes the runnable Presidio redaction example, the full extraction-attack taxonomy from Carlini through Nasr, the regulatory mapping across GDPR, CCPA, PIPL, and DPDPA, and the In Plain English sidebars this article only summarises. View LLM Primer VII on Amazon →
LLM Primer VII data leakage memorization extraction attack GDPR PII redaction chapter 3
SHO
SHO
CTO of Receipt Roller Inc., he builds innovative AI solutions and writes to make large language models more understandable, sharing both practical uses and behind-the-scenes insights.