Peter A Clarke

The EDPB has released a report titled AI Privacy Risks & Mitigations Large Language Models (LLMs). A dry title on an important issue.

The AI Privacy Risks & Mitigations Large Language Models (LLMs) report sets out a comprehensive risk management methodology for LLM systems and, importantly, mitigation measures for common privacy risks in LLM systems.

LLMs is another important advance in artificial intelligence. They  process and generate human-like language trained on extensive datasets.

It is a long and very technical document but one that privacy practitioners should read.

Some of the interesting points:

• How LLMs work: LLMs are advanced deep learning models designed to process and generate human-like language. They rely on the transformer architecture, which uses attention mechanisms to understand context and relationships between words. Most state of the art LLMs rely on transformers due to their scalability and effectiveness there are alternatives that are based on RNN (Recurring Neural Networks) such as LSTM (Long-short Term Memory)
• Transformers dominate general-purpose language models.
• Training: The foundation of LLM training lies in the use of extensive datasets. Text is cleaned and normalized by removing inconsistencies. Text data is broken into smaller units called tokens, which can be words, subwords, or even individual characters. Tokenization algorithms transforms unstructured text into manageable sequences for computational processing. Tokens are converted into numerical IDs that represent their vocabulary position. These IDs are then transformed into word embeddings
• there are three types of transformer architecture: encoder-only, encoder- decoder, and decoder-only:
  • The encoder takes the input text and converts it into a contextualized representation by analyzing relationships between words
  • The decoder generates text by predicting one token at a time. It builds upon the encoder’s output (if used) and the sequence of tokens already generated
• The training phase of LLMs relies on a structured optimization loop to enhance the model’s ability to generate accurate outputs which involves:
  • Loss calculation: The model compares the output sequence to the target sequence. A loss function quantifies the error, providing a numerical measure of how far the predicted output deviates from the desired result (the loss).
  • Backward pass: The  loss value is used to compute gradients, which indicate how much each model parameter (e.g., weights and biases) contributed to the error.
  • Parameter update: Using an optimization algorithm the model’s parameters are adjusted to reduces the error for future predictions by refining the internal model weights.
• Post training involves:
  • Supervised Fine-Tuning (SFT): This involves training a pre-trained model on a labeled dataset tailored to a specific task, with adjustments made to some or all of its parameters to enhance performance for that task.
  • Instruction Tuning: This technique optimises the LLM for following user instructions and handling conversational tasks.
  • Reinforcement Learning with Human Feedback (RLHF):This method uses human feedback to train a reward model (RM), which helps guide the AI during its learning process.
  •  Group Relative Policy Optimization (GRPO) uses computer generated scores to guide the model’s learning process
    
  • Parameter-Efficient Fine-Tuning (PEFT): This technique adapts pre-trained models to new tasks by training only some of the model’s parameters, leaving the majority of the pre-trained model unchanged.
  • Retrieval-Augmented Generation (RAG):This method  integrates information retrieval capabilities, enabling the referencing of specific documents. It allows LLMs to incorporate domain-specific or updated information when responding to user queries
  • Transfer Learning: knowledge learned from a task is re-used in another model
  • Feedback loops: Real-world user feedback refines the model’s behavior, allowing it to adapt to new contexts or correct inaccuracies. It is collected through user behaviour. It can also be collected when users directly provide feedback on the model’s output, such as a thumbs-up/thumbs-down rating, qualitative comments, or error corrections. The LLM is then refined based on this feedback.
• Interference phase involves it generating outputs based on new inputs by:
  • Input: The user’s query is processed through tokenization and embedding, transforming it into a format the model can understand.
  • Processing: The input passes through the transformer architecture, where attention mechanisms and decoder layers predict the next tokens in the sequence. The decoder produces a vector of scores (called logits) for each word in the vocabulary. These scores are then passed through the Softmax function, which converts them into probabilities. The model selects the most probable token as the next word in the sequence, ensuring that the generated text is coherent and contextually relevant.
  • Output: The model produces probabilities for potential next words, selecting the most likely options based on the input and context. These predictions are combined to generate coherent and relevant responses
  • For an LLM to become part of an AI system, additional components such as a user interface, must be integrated to enable it to function as a complete system. 
  • The privacy risk occur during:
    • the collection of data: The training, testing and validation set could contain identifiable personal data, sensitive data or special category of data.
    • Inference: Generated outputs could inadvertently reveal private information or contain
    • RAG process: the knowledge bases containing sensitive data or identifiable personal data could be used without implementing proper safeguards.
• Regarding Agentic AI:

  • it is estimated that by 2027, 50% of companies leveraging generative AI are expected to have launched pilots or proofs of concept to implement agentic AI systems.

  • these systems are envisioned to function as intelligent assistants, capable of autonomously managing complex tasks with minimal human supervision.

  • AI Agents are autonomous systems that can be built on top of LLMs and can perform complex tasks by combining the capabilities of LLMs with reasoning, decision-making, and interaction capabilities.

  • AI agents are proactive, capable of goal-oriented behavior such as planning, executing tasks, and iterating based on feedback. They can:

    • operate independently and are designed to achieve specific objectives by orchestrating multiple actions in sequence.

    • also incorporate feedback to refine their actions or responses over time.

    • integrate capabilities from other AI systems, such as computer vision or audio processing, to handle diverse data inputs

  • its architecture:
    • focuses on critical components that work together to enable sophisticated behavior and adaptability in real-world scenarios.
    • is modular, involving distinct components for perception, reasoning, planning, memory management, and action. This modularity allows the system to handle complex tasks, interact dynamically with their environment, and refine performance iteratively.
  • the common modules are:
    • Perception module: it handles the agent’s ability to process inputs from the environment and format them into a structure that the LLM can understand. It converts raw inputs into embeddings or structured formats that can be processed by the reasoning module.
    • The reasoning module: it enables the agent to interpret input data, analyze its context, and decompose complex tasks into smaller, manageable subtasks. It leverages the LLM’s ability to understand and process natural language to make decisions. The reasoning mechanism enables the agent to analyze user inputs to determine the best course of action and leverage the appropriate tool or resource to achieve the desired outcome.
    • The planning module: it  determines how the agent will execute the subtasks identified by the reasoning module. It organizes and sequences actions to achieve a defined goal.
    • Memory and statement management: To maintain context and continuity, the agent keeps track of past interactions. Memory allows the AI agent to store and retrieve context, both within a single interaction and across multiple sessions.
      • Short-Term Memory: Maintains context within the current interaction to ensure coherence in responses.
      • Long-Term Memory: Stores user preferences, past interactions, and learned insights for personalization
    •  Action module: it is responsible for executing the plan and interacting with the external environment by carrying out the tasks identified and planned by earlier modules. The agent must have access to a defined set of tools, such as APIs, databases, or external systems, which it can use to accomplish the specific tasks.
  • • Feedback and iteration loop: it enables the agent to evaluate the success of its actions and adjust its behavior dynamically. It incorporates user corrections, system logs, and performance metrics to refine reasoning, planning, and execution over time.
• The growing adoption of AI agents powered by LLMs introduce significant privacy risks being:
  • AI agents often require access to a wide range of user data, such as
    • Internet activity: Browsing history, online searches, and frequently visited
    • Personal applications: Emails, calendars, and messaging apps for scheduling or communication tasks.
    • Third-party systems: Financial accounts, customer management platforms, or other organizational systems
  • AI agents are designed to make decisions autonomously, which can lead to errors or choices that users may disagree with
  • Like other AI systems, AI agents are susceptible to biases originating from their training data, algorithms and usage context
• LLMs have applications as:
  • Chatbots and AI Assistants: LLMs power virtual assistants like Siri, Alexa, and Google Assistant, understand and process natural language, interpret user intent, and generate responses.
  • Content generation: LLMs assist in creating articles, reports, and marketing materials by generating human-like text, thereby streamlining content creation processes.
  • Language translation: Advanced LLMs facilitate real-time translation
  • Sentiment analysis: Businesses use LLMs to analyze customer feedback and social media content, gaining insights into public sentiment and informing strategic
  • Code generation and debugging: Developers leverage LLMs to generate code snippets and identify errors, enhancing software development efficiency.
  • Educational support tools: LLMs play a key role in personalized learning by generating educational content, explanations, and answering student questions.
  • Customer support: Automating responses to customer inquiries and escalating complex cases to human agents.
• AI Lifecycle Phases and their Impact on Privacy are:
  • Inception and Design: In this phase, decisions are made regarding data requirements, collection methods, and processing The selection of data sources may introduce risks if sensitive or personal data is included without adequate safeguards.
  • Data Preparation and Preprocessing: Raw data is collected, cleaned, in some cases anonymized, and prepared for training or fine-tuning. Datasets are often sourced from diverse origins, including web-crawled data, public repositories, proprietary data, or datasets obtained through partnerships and collaborations.
    • Privacy risks:
      • Training data may inadvertently include personal details, confidential documents, or other sensitive information.
      • Inadequate anonymization or handling of identifiable data can lead to breaches or unintended inferences during later stages.
      • Biases present in the datasets can affect the model’s predictions, resulting in unfair or discriminatory outcomes.
      • Errors or gaps in training data can adversely impact the model’s performance, reducing its effectiveness and reliability.
      • The collection and use of training data may violate privacy rights, lack proper consent, or infringe on copyrights and other legal obligations.
  • Development, Model Training: Prepared datasets are used to train the model, which involves large-scale processing. The model may inadvertently memorize sensitive data, leading to potential privacy violations if such data is exposed in outputs.
  • Verification & Validation:¹³⁰ The model is evaluated using test datasets, often including real-world scenarios. Testing data may inadvertently expose sensitive user information, particularly if real- world datasets are used without anonymization.
  • Deployment: The model interacts with live data inputs from users, often in real-time applications that could integrate with other systems. Live data streams might include highly sensitive information, requiring strict controls on collection, transmission, and storage.
  • Operation and Monitoring: Continuous data flows into the system for monitoring, feedback, and performance Logs from monitoring systems may retain personal data such as user interactions, creating risks of data leaks or misuse.
  • Re-evaluation, Maintenance and Updates: Additional data may be collected for retraining or updating the model to improve accuracy or address new Using live user data for updates without proper consent or safeguards can violate privacy principles.
  • Retirement: Data associated with the model and its operations is archived or Failure to properly erase personal data during decommissioning can lead to long-term privacy vulnerabilities.
• Given the broad spectrum of risks associated with AI, methodologies like threat modeling play a pivotal role in systematically identifying privacy risks. These methodologies often leverage libraries of specific AI threats, hazards and vulnerabilities providing a structured evaluation of risks throughout the lifecycle of the AI system including those arising from both intended and unintended uses of the system
• LLMs can present a wide range of privacy and data protection risks. These risks arise from the specific use case, the context of application, and the risk factors and evidence identified during the assessment process.
• Once risks have been identified, the next crucial steps within the risk analysis phase are the estimation and evaluation of the risks. This involves the classification and prioritization of risks based on their probability and severity or potential impact. The actual risk level or classification will depend heavily on the specific use case, operational context, system monitoring, model evaluation results and the affected stakeholders

• Risk treatment involves developing strategies to mitigate identified risks and creating actionable implementation plans. The choice of an appropriate treatment option should be context-specific, guided by a feasibility analysis such as the following:

  • Evaluate the type of risk and the available mitigation measures that can be implemented.
  • Compare the potential benefits gained from implementing the mitigation against the costs and efforts involved and the potential impact.
  • Assess the impact on the intended purpose of the LLM system’s implementation.
  • Consider the reasonable expectations of individuals impacted by the
  • Perform a trade-off analysis to evaluate the impact of potential mitigations on aspects such as performance, transparency, and fairness, ensuring that processing remains ethical and compliant based on the specific use case.

• The most common risk treatment criteria are: Mitigate, Transfer, Avoid and Accept. For each identified risk one of the criteria options will be selected:

  • Mitigate – Implement measures to reduce the probability or the severity of the
  • Transfer – Shift responsibility for the risk to another party (e.g., through insurance or outsourcing).
  • Avoid – Eliminate the risk entirely by addressing its root
  • Accept – Decide to take no action, accepting the risk as is because it falls within acceptable limits as defined in the risk criteria.
• It is  important to maintain a dynamic risk register, containing risk records that are durable, easily accessible, clear, and that are consistently updated to ensure accuracy and relevance.

• Risks should also have clear ownership assigned, and regular reviews should be conducted to ensure that risk management practices remain proactive.

  • Findings from prior evaluations conducted before the deployment phase
  • The effectiveness of applied mitigation measures
  • Potential risks that may arise post-deployment obtained through monitoring
  • New risks identified during threat modeling sessions.
  • To analyze residual risk, the probability and severity of the remaining risks are reevaluated, providing a clear overview of the risks that remain after mitigation and taking into account:
• Once residual risks are identified, organizations must decide whether these risks fall within acceptable levels as defined by their risk tolerance and acceptance criteria. If residual risks are deemed acceptable, they can be formally acknowledged and documented in the risk register. However, if the risks exceed acceptable levels, further mitigation measures must be explored and implemented as well as documented. The process then returns to the risk treatment phase to identify the most appropriate treatment option for the risk.
• Reviewing the risk management process is essential to ensure that planned activities have been properly executed and that risk controls and mitigations are effective.
• Documenting risk assessments, mitigation measures, and residual risks throughout the lifecycle is essential for ensuring accountability, compliance, and continuous improvement.
• Once risk mitigation measures have been implemented, ongoing monitoring is essential to assess their effectiveness and identify any emerging risks.
• To ensure an effective risk management strategy, it is also important to implement incident response mechanisms that enable a timely and appropriate response to alerts and warnings generated through evaluations and monitoring, as these may indicate a potential privacy or data protection incident
• Effective risk management for LLMs must adopt an iterative approach that spans the entire lifecycle of the system—from design and development through deployment, monitoring, and eventual decommissioning