Why GDPR Has Not Stopped AI Training Data Abuse in the EU

GDPR has not stopped AI training data abuse in the EU because its protections depend on assumptions that large-scale training breaks. The gap between the law and industry practice reflects three compounding failures, namely unsettled legal grounds for training, transparency duties that collection outpaces, and enforcement that moves more slowly than deployment. Understanding that mismatch points toward privacy built into AI systems, rather than compliance added after personal data has entered them.

Why GDPR's architecture was not built for AI training

GDPR’s core obligations were designed for personal data kept in databases that could be identified and moved through processing flows that were documented. In that setting, access controls protect discrete records, and a controller can locate the information to which a person’s rights apply. The regulation was designed in 2016 for a world of cookie banners, third-party trackers, and consumer profiling. It did not anticipate foundation models trained on the open web or autonomous agents whose prompts cross jurisdictions before returning a response. Its working model was data that could be identified, tracked, and acted upon.

Data subjects can remain unidentified when material is collected, purposes can emerge during processing, and the result of processing is a parameter space. The important distinction is not simply that AI processes more data. It changes the relationship between the original information and the system built from it. The legal framework still asks controllers to justify collection, explain use, and honour individual rights, but the technical anchors for those duties become harder to find.

The lawful basis problem is that training on personal data has no settled legal ground

For private-sector AI developers, the lawful basis debate has narrowed in practice to consent and legitimate interest. Consent gives people a direct say, but individually obtaining it for billions of web-scraped documents is not practicable. Article 7(3) withdrawal rights are harder to honour once the information has been used for training. Legitimate interest is another route, but it requires a three-part test that weighs genuine interest, necessity, and the balance against data subjects’ rights. It is not a blanket permission for collecting AI training data.

The EDPB’s Opinion adopted in late 2024 accepted that legitimate interest may support training in some cases, including fraud detection and threat prevention. But it left the balancing test open-ended and let authorities decide how to enforce it case by case. A comparative analysis of 19 regulatory guidelines issued between 2020 and 2024, published in International Data Privacy Law by Oxford University Press, found that all recognized legitimate interest as relevant. Their practical tests nevertheless diverged sharply, with only a few authorities, notably France’s CNIL and Germany’s LfDI, naming specific sufficient scenarios.

Courts have not supplied a uniform answer either. The Cologne Higher Regional Court rejected an application in May 2025 to prohibit Meta from training on publicly shared Facebook and Instagram data of adult users, holding that the processing was necessary to pursue a legitimate interest under Article 6(1)(f) GDPR. That permissive reading contrasts with the approach Italian and other authorities have taken. Legal uncertainty is normal around new technology, so disagreement alone would not establish a structural failure. The difference here is that processing occurs at massive scale while the question remains unresolved, allowing harm to accumulate before the legal boundary is settled.

Why transparency obligations cannot work at AI training scale

Transparency depends on a person learning that their data has been collected and understanding what happens to it. GDPR’s framework assumes controllers can identify and notify those people, but web scraping often breaks that assumption at collection. Article 14(5)(b) excuses individual notification when it would involve disproportionate effort. The EDPB’s ChatGPT Taskforce acknowledged that this exemption could apply to data scraped from publicly accessible sources, while treating it as a narrow exception that cannot be “routinely relied upon”. At pre-training scale, however, disproportionate effort is what it usually takes to notify people.

A duty to reach each person applies to data collected across whole populations, with no workable way to reach them all. The sensitivity of the material makes the gap more consequential. An academic audit of Common Crawl, the web archive forming the backbone of most frontier LLMs, found that a substantial share of audited URLs fell into sensitive-data categories. Article 9 categories, including health, religion, political opinion, and sexual orientation, are therefore not merely hypothetical concerns.

The AI Act introduces supplementary transparency through its Public Summary Template, published 24 July 2025 under Article 53(1)(d). It calls for disclosure of training sources, including domain names crawled, time periods, and general content descriptions, while limiting detail to protect trade secrets. Individuals can draw reasonable inferences from those sources but cannot verify that their specific data was used. The Model Documentation Form, introduced 10 July 2025 through the voluntary GPAI Code of Practice, provides detailed dataset provenance to regulators under confidentiality. These instruments improve visibility, but visibility into a corpus is not the same as notice to a person.

Why the right to erasure breaks down once data becomes model weights

During training, information from an example is distributed non-linearly across millions or billions of parameters, leaving no pointer to that example inside the model weights. For a model fine-tuned on personal data, surgical removal is not available in the way it is for a database record, while full retraining is economically impractical at frontier scale. Current regulatory guidance also leaves the operational response undefined.

The EDPB’s 2024 Opinion rejected the assumption that models trained with personal data can always be treated as anonymous. It requires case-by-case assessment of direct and probabilistic extraction risks, with a high burden of proof on the controller. Researchers including Carlini et al. have established that large language models memorize a share of training data verbatim. Research published in 2025 also found that data could sometimes be extracted after supposed exact unlearning, using differences between pre- and post-unlearning checkpoints to guide extraction. An attempted removal can therefore be less final than its name suggests.

Erasure, rectification, and access assume information can be located, changed, or deleted. Model weights do not straightforwardly support those operations. Better documentation may explain what happened, but it cannot turn the weights back into individually addressable records. Enforcement must thus address harm already embedded in the system.

How the one-stop-shop mechanism concentrates and delays enforcement

GDPR’s one-stop-shop mechanism was designed to simplify cross-border oversight by assigning a lead supervisory authority. Companies whose main EU establishment is in Ireland, including Meta, Google, Apple, Microsoft, TikTok, X, and OpenAI Ireland Limited since February 2024, answer to the Irish Data Protection Commission in that role. The DPC’s 2025 Annual Report recorded its highest-ever volume of new cases, a 45% increase from 2024. Concentrating complex AI cases in one authority creates a bottleneck regardless of that authority’s competence or intent. Other member states have limited scope to act unilaterally once the mechanism applies.

Meta’s training on EU users’ data illustrates the resulting cycle. Training paused for nearly a year while the DPC sought a formal EDPB Opinion and Meta substantially revised its approach. Those revisions included enhanced transparency notices, improved objection mechanisms, data de-identification, and output filtering. Meta resumed training in May 2025, but noyb challenged the revised approach through a cease-and-desist and indicated a potential class-action lawsuit.

Jurisdiction can also undo work already completed. Italy’s Garante fined OpenAI for training ChatGPT on personal data without a valid lawful basis, but the Court of Rome annulled that decision in early 2026 because OpenAI’s Irish establishment made the DPC the lead authority. The annulment did not vindicate the data practices or positively confirm DPC jurisdiction over conduct predating that establishment. Meanwhile, the DPC’s inquiry opened on 11 April 2025 into X’s use of EU/EEA public posts to train Grok remained pending as of October 2026. Deliberate administrative review has value, but models can be deployed, updated, and superseded before the disputed processing receives a final answer.

How fragmented enforcement lets violations persist even after fines are issued

A completed enforcement action is not necessarily a settled outcome. Appeals, jurisdictional disputes, and cross-border coordination failures can prevent a ruling in one place from stopping non-compliant processing elsewhere. Large fines matter, but their headline size does not show how broadly or promptly the law is enforced. The Irish DPC accounts for a disproportionate share of penalties, driven by a small number of very large fines against platform companies. That concentration makes the overall figure an imperfect measure of deterrence across sectors and company sizes.

Enforcement is nevertheless widening. In H1 2026, data-enrichment vendor Kaspr was fined €200,000 for scraping professional profiles without consent. Yet a mid-range fine against a smaller company leaves open the questions of lawful basis and how to honour rights in frontier AI training.

A share of enterprises have not verified whether their AI vendors use submitted data for model training. Sensitive information can therefore enter third-party systems because companies trust the contracts, without technical checks on how the data is used. A vendor may be liable on paper even when an audit is needed to find out how it uses the data. Enforcement can do little to deter the use of data for training when the organizations supplying it cannot tell whether that use is happening.

What the AI Act adds and where it still falls short

The AI Act is a meaningful legislative response to training-data opacity. Its transparency instruments, now in force, create public-facing disclosure and a more detailed route for regulatory oversight. Regulators can now draw on rules that were written with foundation-model training in mind. The framework gives regulators and individuals information they previously lacked.

Its limits become clearer when disclosure is separated from permission. A GPAI provider can publish a Public Summary Template in full compliance with Article 53 while the underlying processing still lacks a valid basis under Article 6 GDPR. Describing a dataset does not establish that collecting and using it was lawful. Confidential provenance shared with authorities also leaves people without a direct way to find their own information. The deliberate protection of trade secrets leaves access, erasure, and objection constrained even under the new framework.

Meta publicly announced that it would not sign the voluntary GPAI Code of Practice, arguing that its measures introduce legal uncertainties and go beyond the AI Act’s scope. The Act adds a useful disclosure layer, but it neither settles GDPR’s lawful basis question nor removes its enforcement bottleneck.

Why the structural gaps point toward architectural solutions rather than compliance layers

The common feature of these failures is their timing. Personal data enters training before legal uncertainty is resolved, before individuals can meaningfully inspect its use, and before enforcement can deliver a final outcome. Documentation, guidance, and fines all matter, but none alone can prevent a processing model from creating the problem first. Contracts and policies state limits on sovereignty, retention, and training. Enforcing those limits requires a system built to do so.

Privacy-by-design addresses that distinction before the data becomes difficult to recover or remove. Systems designed so that user data is not retained, used for training, or accessible to the provider reduce dependence on promises made after collection. If personal data never enters the training pipeline, the unsettled lawful basis question does not arise for that processing. This prevents one source of exposure while leaving other GDPR obligations and issues in a model’s earlier training unresolved.

For individuals and organizations choosing AI tools, the practical question is therefore whether a provider’s design makes privacy obligations easier or harder to satisfy. Data architecture should be a top concern for governance alongside security and availability, with more weight than the wording of a privacy policy. Architecture-first privacy remains the direction of a durable response, with its limited availability and the costs of switching providers making the choice difficult. Privacy protection is stronger when a system is built not to create the harm that the law must later try to repair.