What Independent Security Researchers Say Actual AI Privacy Regulation Would Require

AI privacy regulation was written for a world in which the database was the most dangerous thing in the room. Independent researchers now argue that this framing misses where the actual harm happens: at inference, when a model reasons over data it already holds. This piece maps what those researchers say real regulation would require.

Why existing AI privacy law was built for a different threat model

AI privacy regulation currently operates on a two-layer architecture that predates the threats it now faces. The first layer, comprising GDPR, HIPAA, and CCPA, applies fully to AI agents that access personal data, imposing lawful-basis requirements, data minimization rules, and data subject rights. These instruments were written for databases, not generative models. The second layer, including the EU AI Act, U.S. state AI laws, and the NIST AI Risk Management Framework, adds risk assessments for high-risk systems, transparency requirements, human oversight, and training-data provenance documentation.

On paper, the two layers feel comprehensive. They cover collection, storage, processing, and deployment. An organization that follows both layers can point to a stack of documentation that demonstrates diligence across every stage of the data lifecycle. Yet both layers share the same design assumption: the privacy violation happens when data is collected or stored, so controlling collection and storage controls the harm. The gap between what these rules require and what organizations actually do is large, and it is not primarily because organizations are flouting the law. It is because the law was not written to reach the conduct that causes the harm in modern AI systems.

This mismatch matters because the two-layer architecture will continue to shape compliance behavior. Organizations will keep producing lawful-basis documentation and risk assessments, and those artifacts will satisfy auditors, while the mechanism that actually generates privacy violations operates untouched. Understanding why requires looking at where the violation physically occurs.

Where AI systems actually generate privacy violations: the inference stage

The privacy violation in modern AI systems typically occurs not when data enters the system but when the model reasons over it. This is the inference stage, and existing law does not reach it. A model can expose, deduce, or reconstruct personal information without that information ever being explicitly queried or intentionally disclosed. The harm is generated by computation, not by collection.

A systematic review published in Frontiers in Artificial Intelligence in February 2026 maps the primary privacy threat categories by domain, drawing on 94 research papers. For machine learning broadly, the threats are model inversion, membership inference, adversarial leakage, and repurposing of training data. For large language models specifically, the list extends to prompt-based data extraction, jailbreaks, model leaks of training data, and hallucinated sensitive personal information. For computer vision, researchers identify facial recognition re-identification, inference of sensitive attributes, surveillance, and tracking. The spread across domains is wide enough that no single data-handling rule addresses it.

The clearest example of why existing law misses this stage is attribute inference. A model can deduce health status, political views, or age from inputs that contain none of that information explicitly. No personal data was collected in the legally meaningful sense, yet a privacy harm occurred. NIST AI 100-2 E2025, published in March 2025, taxonomizes five privacy attack types against AI systems: model extraction, data reconstruction, membership inference, and property inference. That edition expanded coverage to generative AI, adding poisoning, direct prompting, and indirect prompt injection attacks, while the privacy attack taxonomy applies primarily to predictive AI systems. The implication researchers draw is direct: a regulatory framework that stops at the data-collection boundary will consistently miss the actual harm-generating mechanism.

The five specific attack surfaces that researchers say regulation must name

If regulation is to reach the inference stage, it must name the surfaces where harm occurs. The most operationally specific public document doing this work is the April 2025 EDPB report on AI Privacy Risks and Mitigations for Large Language Models, written by external expert Isabel Barberá under the EDPB's Support Pool of Experts program. The document goes beyond what prior opinions mentioned in passing and enumerates surfaces that each require distinct technical mitigations.

Training data leakage is the first. Personal data absorbed during training can be extracted through crafted queries. The mitigation requires differential privacy during training, red-teaming for extraction vulnerabilities, and access controls limiting who can probe the model. Prompt injection is the second surface: user inputs or retrieved content can override system prompts, causing data disclosure. Mitigation here requires input validation, output filtering, and architectural separation between trusted system instructions and untrusted user input. Attribute inference, the third, is the case where the model deduces sensitive characteristics from inputs that do not explicitly contain them. The required technical constraint is one that prevents inferred attributes from being stored or used downstream. Output leakage is the fourth: a model's response to one user can contain personal data from another user's context, and mitigation requires session isolation, context clearing between users, and output filtering for personal data patterns. Retrieval-augmented generation risks form the fifth surface, where the danger sits in what a system retrieves and feeds into a prompt.

What makes these five surfaces significant for regulation is that no single control addresses them. A general privacy policy requirement, the kind of statement that layer-one instruments already mandate, does not touch any of the five. Each surface requires a distinct technical mitigation, and those mitigations are infrastructure decisions, not documentation choices. The DSK RAG Guidelines offer a useful reference point here: they address retrieval-layer access control in the context of GDPR data protection obligations, though they are non-binding guidance rather than a settled legal standard.

What mandatory regulatory requirements would look like in practice

Translating these attack surfaces into regulation produces obligations that are technically verifiable and that differ substantially from current compliance practice. The through-line is that each requirement can be checked, not just asserted. Data minimization becomes a design-time obligation rather than a policy statement. Modern AI systems are frequently architected to collect and retain expansive datasets by default, even when narrower datasets would suffice. The Workplace Privacy Report from January 2026 identifies data minimization as one of the most operationally challenging issues for 2026 for exactly this reason. Effective regulation would require demonstrable architectural choices, documented as such, showing that data ingestion is scoped to what each specific inference task requires.

Mandatory data protection impact assessments follow. The EDPB's Support Pool of Experts document, whose views do not necessarily reflect the official EDPB position, notes that given the potentially systemic nature of processing through LLMs, a DPIA is in most cases mandatory under the WP29 DPIA guidelines and the national supervisory authority blacklists derived from them. Those blacklists identify as high-risk any processing involving innovative use of new technological solutions, large-scale processing, or automated decision-making with legal or similarly significant effects. This is not a recommended best practice but a legal requirement triggered by the architecture of LLM deployment itself. Prompt and output filtering then becomes a technical baseline: filters can operate at lexical, semantic, or syntactic levels, enhanced with NLP classification or named entity recognition, and they must be built into the system before deployment, not toggled on after a complaint. Red-teaming follows the same pattern: programs should assess whether an LLM might disclose information learned during training, and the testing must be documented, repeatable, and tied to ongoing risk management rather than a pre-launch checkbox.

Human oversight lands as a GDPR baseline, not an AI Act aspiration. California's AI and privacy laws for 2026 note that by January 2027, businesses using automated decision-making technology for significant decisions in employment, housing, credit, healthcare, and education must provide pre-use notices, opt-out rights, and access to decision logic. That trajectory runs parallel to the EDPB position. What distinguishes all of these requirements is that they are technically verifiable, which is what makes them actionable regulatory obligations rather than aspirational principles. Privacy must be built into AI architecture from the start, not layered on afterward. Regulation that requires verified technical controls rather than documented intentions is the operationalization of that principle.

Where the legal landscape is moving to enforce these requirements

A wave of binding instruments is beginning to operationalize inference-layer obligations even where the statutory text does not name them. Colorado offers a cautionary example: the Colorado Artificial Intelligence Act was repealed and replaced before it ever took effect. On May 14, 2026, Governor Polis signed SB 26-189, which superseded SB 24-205 and takes effect January 1, 2027 with a narrower, notice-based approach rather than the full risk management and reasonable care framework of its predecessor. That rollback shows the countervailing pressure, but the broader direction holds.

California's updated CCPA regulations, operative January 1, 2026, make annual cybersecurity audits mandatory through independent auditors, with certifications due to the California Privacy Protection Agency on a schedule staggered by revenue through April 2030. The EU AI Act's high-risk system obligations move toward full enforcement as of 2027, requiring pre-market testing, documentation, and human oversight. The EDPB has set the bar for LLM anonymity very high: for a model to be considered anonymous, both the likelihood of directly extracting personal data and the likelihood of obtaining personal data from queries must each be insignificant, taking into account all means reasonably likely to be used. In practice, LLMs will not generally be considered anonymous. The standard corporate defense that model weights are not personal data is unlikely to hold.

GDPR clarifications are already established. Scraping public data for model training does not automatically grant legitimate interest as a legal basis. The right to be forgotten may apply when personal data is integrated into model weights, depending on a case-by-case assessment of whether the model can be considered anonymous. Morrison Foerster predicted in December 2025 that regulators and litigants will increasingly test how consent, deletion, and opt-out rights apply to data used for model training and retraining. The answer will define not just compliance costs but the economics of generative AI itself. Across jurisdictions, the direction of travel treats privacy as a design and infrastructure problem, not a documentation exercise.

Why technical privacy-preserving techniques alone do not satisfy what regulation requires

The strongest objection to the researcher agenda is that technical techniques already solve the problem. Differential privacy and federated learning are the usual names, and both are real advances. But what regulation requires, as researchers define it, is not what either technique alone delivers.

On differential privacy, the privacy guarantee is governed by the epsilon value. As epsilon decreases and the guarantee strengthens, model accuracy declines and class prediction imbalances emerge. That tradeoff is not an engineering detail. It is a substantive policy question about what level of privacy degradation is legally acceptable, and regulation that mandates a specific technique without specifying the epsilon target is not operationally meaningful. On federated learning, experiments have shown that the distributed training architecture does not prevent attackers from extracting training data from LLMs. It does not close the membership inference or data extraction attack surfaces that the EDPB and NIST AI 100-2 E2025 both identify as requiring mitigation.

The Frontiers review frames AI as both a threat to privacy and a tool for enhancing it, but its own finding is that no single technique resolves the full attack surface. The complexity requires layered, interdisciplinary approaches. That has a specific implication for regulation: effective rules cannot simply mandate a category of technique. They must specify verifiable outcomes, including what level of re-identification risk is acceptable, what adversarial test methodology constitutes sufficient red-teaming, and what access control architecture satisfies Article 25. Technique mandates without outcome criteria are not enforceable. This is where architecture-level privacy decisions become regulatory. A system designed from the ground up to avoid retaining data it does not need is structurally more defensible than one that applies privacy techniques as a post-hoc layer, because the latter retains the attack surface even when the technique is applied correctly.

What a regulation-ready inference-layer privacy posture actually requires organizations to do

Organizations that wait for regulation to crystallize before addressing inference-layer risks are making a compliance bet that the trajectory of binding instruments does not support. The same two-layer framework that missed the inference stage is what most organizations are currently relying on, and the instruments documented here are already moving past it. The minimum viable posture converges on six requirements. Conduct a DPIA before deploying any LLM system, treating it as triggered by the architecture rather than by a case-by-case assessment of sensitivity. Implement prompt and output filtering as infrastructure, built in before deployment. Scope data ingestion and retrieval to the minimum necessary for each specific inference task, documented as an architectural decision, following the DSK RAG Guidelines model. Establish a repeatable, documented red-teaming program that tests for the five NIST AI 100-2 E2025 attack types, on a cadence tied to model updates rather than initial deployment alone. Build human review into any automated decision workflow that has material consequences for data subjects, treating it as a GDPR baseline requirement. And do not rely on the argument that model weights are not personal data, because the EDPB's anonymity standard makes that defense unlikely to hold for LLMs generally.

The deeper structural shift researchers argue for is an evaluation change: organizations should assess AI providers not just on capability and cost but on whether privacy is built into the system's architecture. A system that minimizes data retention and inference-stage exposure by design is structurally more defensible than one that applies privacy controls as a compliance layer. Morrison Foerster frames 2026 as the year AI governance moves from aspirational to enforceable, and documented processes with verified controls are what regulators and litigants will examine. Confidant's architecture, designed from the ground up to avoid collecting or retaining user data rather than applying privacy controls after the fact, reflects the design-first approach researchers identify as the structural answer to inference-layer risk. Users whose AI tools do not retain their data do not face the training-data leakage, output-leakage, or membership-inference exposure that the EDPB and NIST frameworks describe. That is the same diagnosis from two directions: the researchers name the problem, and the architecture removes it before the problem can occur.