09.06. 2026
Gernot Fritz, Tanja Pfleger
Artificial intelligence is no longer a topic for the future. In many companies, AI systems are already answering customer enquiries, summarising documents, searching internal knowledge bases, supporting HR processes or assisting with software development. This brings new efficiency gains – but also new legal and technical attack surfaces.
The key issue is not merely how AI systems are used, but how they are controlled: What information enters the system? Which sources may the AI access? Which permissions does an AI assistant inherit from the user? What data is stored in prompts, logs, chat histories or vector databases? And how can companies prevent manipulated inputs from disclosing internal information or triggering unwanted actions?
Prompt Injection: When External Instructions Hijack the System
One of the main attack vectors in the use of AI systems is prompt injection. This involves manipulating an AI system through an input so that it disregards its original instructions, follows external commands or reveals information that should remain confidential.
Indirect prompt injection is particularly dangerous. In such cases, the manipulative instruction does not come from the user directly. Instead, it is embedded in external content, such as an email, a document, a website, a support ticket, a calendar entry or even an image. When the AI system processes the content, it may mistakenly interpret the hidden instruction as a relevant command.
Security research and practical tests have shown that AI assistants can be manipulated through carefully crafted emails, websites or embedded content to disclose internal information or trigger unwanted actions. In multimodal systems, hidden instructions may even be embedded in seemingly harmless files or images. Once an AI system can read external content while also accessing internal data or tools, a new attack surface emerges.
Jailbreaking: When Safeguards Are Circumvented
Jailbreaking is closely related to prompt injection. It refers to inputs designed to bypass the safeguards of an AI system. As a result, a system may generate prohibited, inappropriate or confidential content even though its configuration is intended to prevent precisely that.
This becomes legally relevant where personal data is disclosed, security measures are bypassed or unlawful processing activities are triggered. Not every successful jailbreak automatically amounts to a data protection incident. However, where personal data is accessed, altered or disclosed without authorisation, this may constitute a personal data breach within the meaning of the GDPR.
Overprivileged AI Assistants: When the Bot Has Too Much Access
AI applications connected to email accounts, HR systems, CRM databases, SharePoint, document management systems or internal knowledge bases are particularly sensitive. The practical risk often lies not in the model itself, but in the permissions surrounding it.
Where an AI assistant has the same broad access rights as a user – or even more extensive rights – a manipulated input may be enough to expose documents that are not required for the specific task. This is especially relevant for retrieval-augmented generation (RAG) systems, which retrieve information from external or internal data sources and provide it to a language model as additional context.
RAG can reduce hallucinations and make answers more traceable. From a data protection perspective, however, it also creates an additional processing layer: documents are indexed, broken down into smaller units, converted into vectors, stored and retrieved in response to user queries. For companies, this means that the model itself is only part of the picture. Data ingestion, indexing, access controls, inherited permissions, tenant segregation, deletion processes and logging must also be assessed.
GDPR: Security Starts with the Architecture
AI systems do not operate in a data protection vacuum. The general principles and security requirements of the GDPR continue to apply.
Article 5(1)(f) GDPR requires personal data to be processed in a manner that ensures appropriate security, including protection against unauthorised or unlawful processing and against accidental loss, destruction or damage. Article 25 GDPR requires data protection by design and by default. Article 32 GDPR further specifies the requirement to ensure a level of security appropriate to the risk.
Applied to AI, this means that prompts, chat histories, tool calls, RAG repositories, API keys, logs and error messages are not merely technical by-products. They may contain personal data, trade secrets or other highly sensitive information and must be protected accordingly.
The CNIL emphasises this point in its recommendations on the development of AI systems. Its guidance addresses not only the model itself, but also risk assessments, access controls, datasets, model components, documentation and secure development environments. The CNIL also notes that AI models trained on personal data may themselves fall within the scope of the GDPR due to the risk of memorisation.
This holistic perspective is crucial. In practice, vulnerabilities often arise in the surrounding environment: insufficiently vetted libraries, poorly secured interfaces, copied test data, exposed logs, unsecured API keys or unclear responsibilities between providers, integrators and deployers.
A model-centred assessment alone will therefore rarely be sufficient. What matters is the entire technical and organisational environment.
AI Act: Not Every AI System Is High-Risk – but Security Matters
The AI Act follows a risk-based approach. Not every AI application qualifies as a high-risk AI system. The classification depends on the specific purpose, the role of the company and the context in which the system is used. Nevertheless, the Regulation already contains important points of reference for AI security.
Article 4 AI Act is particularly relevant. Providers and deployers of AI systems must, to their best extent, ensure that their staff and other persons dealing with the operation and use of AI systems on their behalf have a sufficient level of AI literacy. This includes not only a basic understanding of AI, but also an awareness of the specific context in which the system is used and the associated risks.
For high-risk AI systems, the AI Act is more specific. Article 15 AI Act requires an appropriate level of accuracy, robustness and cybersecurity throughout the system’s lifecycle. The provision expressly refers to AI-specific vulnerabilities and attacks, including data poisoning, model poisoning, adversarial examples, model evasion, confidentiality attacks and flaws in the model. High-risk AI systems must therefore be resilient against attempts by unauthorised third parties to alter their use, outputs or performance by exploiting system vulnerabilities.
Deployers of high-risk AI systems are subject to additional operational obligations. These include using the systems in accordance with the instructions of use, implementing appropriate technical and organisational measures, ensuring human oversight by competent persons and retaining automatically generated logs to the extent that these are under their control.
When Something Goes Wrong: AI Incident or Personal Data Breach?
Not every AI-related error constitutes a personal data breach. An incorrect answer, an inappropriate suggestion or a harmless jailbreak with no connection to personal data will not, by itself, trigger a notification obligation under the GDPR.
The position is different where personal data is disclosed, altered, lost or made accessible without authorisation. In that case, it must be assessed whether a personal data breach has occurred. Under Article 33 GDPR, such a breach must be notified to the competent supervisory authority unless it is unlikely to result in a risk to the rights and freedoms of natural persons. Processors must notify the controller without undue delay. Where the breach is likely to result in a high risk to affected individuals, an additional communication under Article 34 GDPR may be required.
Companies should therefore avoid treating AI incidents merely as “model errors”. A manipulated prompt, a compromised API key, a misconfigured RAG repository or an overprivileged connector can quickly escalate into a data protection incident.
What Companies Should Do Now
Effective protection against AI-related risks does not lie in a single filter. It requires a multi-layered approach.
The first step is to establish a complete overview: Which AI systems are being used? For which purposes? By which departments? With what data? Through which integrations? Provided by which vendors? And with which access rights?
Based on this overview, companies should assess, for each relevant use case, what data is processed in prompts, context windows, RAG systems, logs and outputs. Chat histories and telemetry data are often underestimated. They can be particularly revealing because they combine questions, context, internal documents and response patterns.
From a technical perspective, AI systems should be designed in accordance with the need-to-know principle. An assistant should only be permitted to retrieve the data required for the specific purpose. Write permissions, tool access and external actions should be restricted and, for sensitive operations, subject to human confirmation.
RAG systems require a dedicated access-control concept. A system should not have broader visibility merely because this is technically easier to implement. The decisive question is whether the specific user would also have access to the relevant content in the underlying source system.
Secure development and testing environments are equally important. Many risks arise not in production systems, but in pilot projects, sandboxes or demos where real-world data is combined with incomplete permission structures and deletion procedures.
Finally, incident-response processes should expressly cover AI-specific scenarios: prompt injection, jailbreaks, data leakage through outputs, manipulated knowledge sources, compromised API keys, erroneous tool calls and unlawful storage in logs. Only then can companies assess in time whether a notification to the data protection authority or a communication to affected individuals is required.
Conclusion
The key challenge when using AI is not merely whether the model provides the “right” answer. The broader questions are: What data can the system access? What conclusions does it draw from that data? What actions is it permitted to trigger? And who controls these processes?
The Omnibus proposals currently under discussion could give companies more time for certain high-risk AI systems, simplify reporting obligations and extend the GDPR notification deadline for personal data breaches. Whether and in what form these changes will ultimately be adopted remains to be seen. Regardless, the underlying principle remains unchanged: AI security is already an integral part of data protection compliance, IT security, contractual arrangements and governance.
Companies using AI in their operations should therefore not treat prompt injection, jailbreaking, RAG risks, logs, API keys and permissions as peripheral technical issues. They are at the heart of a legally sound AI architecture.
Companies looking to deploy AI in practice need clear responsibilities and robust processes. We are happy to assist you with the implementation.
