The UK Information Commissioner’s Office launches its updated Artificial Intelligence data protection risk toolkit.

May 6, 2022 |

Artificial Intelligence (“AI”) is becoming a significant issue for lawyers generally and regulators in particular.   Its impact on the law is apparent with the Full Bench, of 5 justices, ruling in Commissioner of Patents v Thaler [2022] FCAFC 62 last month that an inventor in terms of patent law must be a natural person, not AI.  This was an appeal from a decision of Justice Beach on 30 July 2021 in Thaler v Commissioner of Patents [2021] FCA 879 who relevantly ordered:

  • The determination of the Deputy Commissioner that s 15(1) of the Patents Act 1990 (Cth) is inconsistent with an artificial intelligence system or device being treated as an inventor be set aside.
  • The matter as to whether patent application no. 2019363177 satisfies the formalities under the Patents Regulations 1991 (Cth) and its examination be remitted to the Deputy Commissioner to be determined according to law in accordance with these reasons.

In its reasons the Full Court found  that identification of the “inventor” was central to the operation of the legislation. Under s 15, only the inventor or someone claiming through the inventor is entitled to a patent.

Thaler will probably make its way to the High Court. 

But the use of AI is more prosaic and ubiquitous than in inventing devices.  That is likely to be both a public good and a cause for concern.  At the moment the technology and its implementation is far outpacing the law and regulation.  That is a concern given the potential forseeable and unforseeable consequences.  In that regard I thoroughly recommend Machines Behaving Badly; the Morality of AI by Toby Walsh.   I attended a presentation by Professor Walsh organised by the Centre for Artificial Intelligence and Digital Ethics (CAIDE) last Wednesday

Regulators in the United Kingdom and Europe have been much more alive to the need for guidance and consideration of AI and its effect on privacy and data security than in Australia where the regulator takes a more languid approach and seems to be letting the ACCC to take the running on big tech issues.  In that vein the Information Commissioner’s Office (‘ICO’) announced, on 4 May 2022, that it had launched its updated AI and data protection risk toolkit. It is an important document for those interested in both privacy and data security. 

The announcement provides:

The innovation, opportunities and potential value to society of AI will not need emphasising to anyone reading this guidance.

Nor is there a need to underline the range of risks involved in the use of technologies that shift processing of personal data to complex computer systems with often opaque approaches and algorithms.

This guidance helps organisations mitigate the risks specifically arising from a data protection perspective, explaining how data protection principles apply to AI projects without losing sight of the benefits such projects can deliver.

What stands out in the following pages is that the underlying data protection questions for even the most complex AI project are much the same as with any new project. Is data being used fairly, lawfully and transparently? Do people understand how their data is being used? How is data being kept secure?

The legal principle of accountability, for instance, requires organisations to account for the risks arising from their processing of personal data –whether they are running a simple register of customers’ contact details or operating a sophisticated AI system to predict future consumer demand.

Aspects of the guidance should act as an aide memoire to those running AI projects. There should be no surprises in the requirements for data protection impact assessments or of documenting decisions. The guidance offers support and methodologies on how best to approach this work.

Other aspects of the law require greater thought. Data minimisation, for instance, may seem at odds with systems that allow machine learning to conclude what information is necessary from large data sets. As the guidance sets out though, there need not be a conflict here, and there are several techniques that can ensure organisations only process the personal data needed for their purpose.

Similarly, transparency of processing, mitigating discrimination, and ensuring individual rights around potential automated decision-making can pose difficult questions. Aspects of this are complemented by our existing guidance ‘Explaining decisions made with AI guidance’, published with the Alan Turing Institute in May 2020.

The common element to these challenging areas, and perhaps the headline takeaway, is the value of considering data protection at an early stage. Mitigation of risks must come at the design stage: retrofitting compliance as an end-of-project bolt-on rarely leads to comfortable compliance or practical products. This guidance should accompany that early engagement with compliance, in a way that ultimately benefits the people whose data AI approaches rely on.

The development and use of AI within our society is growing and evolving, and it feels as though we are at the early stages of a long journey. We will continue to focus on AI developments and their implications for privacy by building on this foundational guidance, and continuing to offer tools that promote privacy by design to those developing and using AI.

I must end with an acknowledgment of the excellent work of one of the document’s authors, Professor Reuben Binns. Prof Binns joined the ICO as part of a fellowship scheme designed to deepen my office’s understanding of this complex area, as part of our strategic priority of enabling good practice in AI. His time at the ICO, and this guidance in particular, is testament to the success of that fellowship, and we wish Prof Binns the best as he continues his career as Associate Professor of Computer Science at the University of Oxford.

The guide is useful in explaining that:

  • there are four options when scoring risks, i.e. ‘high’, ‘medium’, ‘low’, and ‘non-applicable’;
  • the assessment of risks will vary depending on the context, so companies should undertake their own assessments of the risks identified;

The toolkit deals with:

  • risk areas during different states of an artifical intelligence (‘AI’) lifecycle being:
    • business requirements and design,
    • data acquisition and preparation,
    • training and testing, and
    • deployment and monitoring,
  • provides a risk assessment summary,
  • steps an organisation can take to mitigate risks.
  • provides relevant guidance.

The ICO’s Executive Director for Regulatory Futures and Innovation, Stephen Bonner, was quoted as stating: 

“[o]rganisations can use our toolkit to assess the privacy risks associated with their use of AI. It provides practical controls to help reduce these risks, putting data protection principles into practice and helping to provide assurance to the organisation and the public about their compliance. The toolkit forms part of the ICO AI Auditing Framework, which is designed to give us a clearer method to assess the data protection compliance of AI systems”.

The toolkit is not intended to replace a Data Protection Impact Assessment (‘DPIA’) and  is voluntary.

The executive summary provides:

Applications of artificial intelligence (AI) increasingly permeate many aspects of our lives. We understand the distinct benefits that AI can bring, but also the risks it can pose to the rights and freedoms of individuals.

This is why we have developed a framework for auditing AI, focusing on best practices for data protection compliance – whether you design your own AI system, or implement one from a third party. It provides a clear methodology to audit AI applications and ensure they process personal data fairly. It comprises:

    • auditing tools and procedures that we will use in audits and investigations;
    • this detailed guidance on AI and data protection; and
    • a toolkit designed to provide further practical support to organisations auditing the compliance of their own AI systems (forthcoming).

This guidance is aimed at two audiences:

    • those with a compliance focus, such as data protection officers (DPOs), general counsel, risk managers, senior management, and the ICO’s own auditors; and
    • technology specialists, including machine learning experts, data scientists, software developers and engineers, and cybersecurity and IT risk managers.

The guidance clarifies how you can assess the risks to rights and freedoms that AI can pose from a data protection perspective; and the appropriate measures you can implement to mitigate them.

While data protection and ‘AI ethics’ overlap, this guidance does not provide generic ethical or design principles for your use of AI. It corresponds to data protection principles, and is structured as follows:

    • part one addresses accountability and governance in AI, including data protection impact assessments (DPIAs);
    • part two covers fair, lawful and transparent processing, including lawful bases, assessing and improving AI system performance, and mitigating potential discrimination;
    • part three addresses data minimisation and security; and
    • part four covers compliance with individual rights, including rights related to automated decision-making.

The accountability principle makes you responsible for complying with data protection and for demonstrating that compliance in any AI system that processes personal data. In an AI context, accountability requires you to:

    • be responsible for the compliance of your system;
    • assess and mitigate its risks; and
    • document and demonstrate how your system is compliant and justify the choices you have made.

You should consider these issues as part of your DPIA for any system you intend to use. You should note that, in the majority of cases, you are legally required to complete a DPIA if you use AI systems that process personal data. DPIAs offer you an opportunity to consider how and why you are using AI systems to process personal data and what the potential risks could be.

You also need to take care to identify and understand controller / processor relationships. This is due to the complexity and mutual dependency of the various kinds of processing typically involved in AI supply chains.

As part of striking the required balance between the right to data protection and other fundamental rights in the context of your AI systems, you will inevitably have to consider a range of competing considerations and interests. During the design stage, you need to identify and assess what these may be. You should then determine how you can manage them in the context of the purposes of your processing and the risks it poses to the rights and freedoms of individuals. You should however note that if your AI system processes personal data you always have to comply with the fundamental data protection principles, and cannot ‘trade’ this requirement away.

When you use AI to process personal data, you must ensure that it is lawful, fair and transparent. Compliance with these principles may be challenging in an AI context.

AI systems can exacerbate known security risks and make them more difficult to manage. They also present challenges for compliance with the data minimisation principle.

Two security risks that AI can increase are the potential for:

    • loss or misuse of the large amounts of personal data often required to train AI systems; and
    • software vulnerabilities to be introduced as a result of the introduction of new AI-related code and infrastructure.

By default, the standard practices for developing and deploying AI involve processing large amounts of data. There is a risk that this fails to comply with the data minimisation principle. A number of techniques exist which help both data minimisation and effective AI development and deployment.

The way AI systems are developed and deployed means that personal data is often managed and processed in unusual ways. This may make it harder to understand when and how individual rights apply to this data, and more challenging to implement effective mechanisms for individuals to exercise those rights.

In relation to how to approach AI from a governance and risk management perspective the Guide provides:

If used well, AI has the potential to make organisations more efficient, effective and innovative. However, AI also raises significant risks for the rights and freedoms of individuals, as well as compliance challenges for organisations.

Different technological approaches will either exacerbate or mitigate some of these issues, but many others are much broader than the specific technology. As the rest of this guidance suggests, the data protection implications of AI are heavily dependent on the specific use cases, the population they are deployed on, other overlapping regulatory requirements, as well as social, cultural and political considerations.

While AI increases the importance of embedding data protection by design and default into an organisation’s culture and processes, the technical complexities of AI systems can make this more difficult. Demonstrating how you have addressed these complexities is an important element of accountability.

You cannot delegate these issues to data scientists or engineering teams. Your senior management, including DPOs, are also accountable for understanding and addressing them appropriately and promptly (although overall accountability for data protection compliance lies with the controller, ie your organisation).

To do so, in addition to their own upskilling, your senior management will need diverse, well-resourced teams to support them in carrying out their responsibilities. You also need to align your internal structures, roles and responsibilities maps, training requirements, policies and incentives to your overall AI governance and risk management strategy.

It is important that you do not underestimate the initial and ongoing level of investment of resources and effort that is required. Your governance and risk management capabilities need to be proportionate to your use of AI. This is particularly true now while AI adoption is still in its initial stages, and the technology itself, as well as the associated laws, regulations, governance and risk management best practices are still developing quickly.

We have also developed a more general accountability framework. This is not specific to AI, but provides a baseline for demonstrating your accountability under the UK GDPR, on which you could build your approach to AI accountability.

How should we set a meaningful risk appetite?

The risk-based approach of data protection law requires you to comply with your obligations and implement appropriate measures in the context of your particular circumstances – the nature, scope, context and purposes of the processing you intend to do, and the risks this poses to individuals’ rights and freedoms.

Your compliance considerations therefore involve assessing the risks to the rights and freedoms of individuals and judging what is appropriate in those circumstances. In all cases, you need to ensure you comply with data protection requirements.

This applies to the use of AI just as to other technologies that process personal data. In the context of AI, the specific nature of the risks posed and the circumstances of your processing will require you to strike an appropriate balance between competing interests as you go about ensuring data protection compliance. This may in turn impact the outcome of your processing. It is unrealistic to adopt a ‘zero tolerance’ approach to risks to rights and freedoms, and indeed the law does not require you to do so. It is about ensuring that these risks are identified, managed and mitigated. We talk about trade-offs and how you should manage them below and provide examples of some trade-offs throughout the guidance.

To manage the risks to individuals that arise from processing personal data in your AI systems, it is important that you develop a mature understanding of fundamental rights, risks, and how to balance these and other interests. Ultimately, it is necessary for you to:

    • assess the risks to individual rights that your use of AI poses;
    • determine how you will address these; and
    • establish the impact this has on your use of AI.

You should ensure your approach fits both your organisation and the circumstances of your processing. Where appropriate, you should also use risk assessment frameworks.

This is a complex task, which can take time to get right. However, it will give you, as well as the ICO, a fuller and more meaningful view of your risk positions and the adequacy of your compliance and risk management approaches.

The following sections deal with the AI-specific implications of accountability including:

    • how you should undertake data protection impact assessments for AI systems;
    • how you can identify whether you are a controller or processor for specific processing operations involved in the development and deployment of AI systems and the resulting implications for your responsibilities;
    • how you should assess the risks to the rights and freedoms of individuals, and how you should address them when you design, or decide to use, an AI system; and
    • how you should justify, document and demonstrate the approach you take, including your decision to use AI for the processing in question.

What do we need to consider when undertaking data protection impact assessments for AI?

DPIAs are a key part of data protection law’s focus on accountability and data protection by design.

You should not see DPIAs as simply a box ticking compliance exercise. They can effectively act as roadmaps for you to identify and control the risks to rights and freedoms that using AI can pose. They are also an ideal opportunity for you to consider and demonstrate your accountability for the decisions you make in the design or procurement of AI systems.

Why are DPIAs required under the data protection law?

In the vast majority of cases, the use of AI will involve a type of processing likely to result in a high risk to individuals’ rights and freedoms, and will therefore trigger the legal requirement for you to undertake a DPIA. You will need to make this assessment on a case by case basis. In those cases where you assess that a particular use of AI does not involve high risk processing, you still need to document how you have made this assessment.

If the result of an assessment indicates residual high risk to individuals that you cannot sufficiently reduce, you must consult with the ICO prior to starting the processing.

In addition to conducting a DPIA, you may also be required to undertake other kinds of impact assessments or do so voluntarily. For example, public sector organisations are required to undertake equality impact assessments, while other organisations voluntarily undertake ‘algorithm impact assessments’. Similarly, the machine learning community has proposed ‘model cards’ and ‘datasheets’ which describe how ML models may perform under different conditions, and the context behind the datasets they are trained on, which may help inform an impact assessment. There is no reason why you cannot combine these exercises, so long as the assessment encompasses all the requirements of a DPIA.

The ICO has produced detailed guidance on DPIAs that explains when they are required and how to complete them. This section sets out some of the things you should think about when carrying out a DPIA for the processing of personal data in AI systems.

How do we decide whether to do a DPIA?

We acknowledge that not all uses of AI will involve types of processing that are likely to result in a high risk to rights and freedoms. However, you should note that Article 35(3)(a) of the UK GDPR requires you to undertake a DPIA if your use of AI involves:

    • systematic and extensive evaluation of personal aspects based on automated processing, including profiling, on which decisions are made that produce legal or similarly significant effects;
    • large-scale processing of special categories of personal data; or
    • systematic monitoring of publicly-accessible areas on a large scale.

Beyond this, AI can also involve several processing operations that are themselves likely to result in a high risk, such as use of new technologies or novel application of existing technologies, data matching, invisible processing, and tracking of location or behaviour. When these involve things like evaluation or scoring, systematic monitoring, and large-scale processing, the requirement to do a DPIA is triggered.

In any case, if you have a major project that involves the use of personal data it is also good practice to do a DPIA. Read our list of processing operations ‘likely to result in high risk’ for examples of operations that require a DPIA, and further detail on which criteria are high risk in combination with others.

What should we assess in our DPIA?

Your DPIA needs to describe the nature, scope, context and purposes of any processing of personal data. It needs to make clear how and why you are going to use AI to process the data. You need to detail:

    • how you will collect, store and use data;
    • the volume, variety and sensitivity of the data;
    • the nature of your relationship with individuals; and
    • the intended outcomes for individuals or wider society, as well as for you.

Whether a system using AI is generally more or less risky than a system not using AI depends on the specific circumstances. You therefore need to evaluate this based on your own context. Your DPIA should show evidence of your consideration of less risky alternatives, if any, that achieve the same purpose of the processing, and why you didn’t choose them. This consideration is particularly relevant where you are using public task or legitimate interests as a lawful basis. See ‘How do we identify our purposes and lawful basis’.

In the context of the AI lifecycle, a DPIA will best serve its purpose if you undertake it at the earliest stages of project development. It should feature, at a minimum, the following key components.

How do we describe the processing?

Your DPIA should include:

    • a systematic description of the processing activity, including data flows and the stages when AI processes and automated decisions may produce effects on individuals;
    • an explanation of any relevant variation or margins of error in the performance of the system which may affect the fairness of the personal data processing (see ‘What do we need to do about Statistical Accuracy’); and
    • a description of the scope and context of the processing, including:
      • what data you will process;
      • the number of data subjects involved;
      • the source of the data; and
      • how far individuals are likely to expect the processing.

Your DPIA should identify and record the degree of any human involvement in the decision-making process and at what stage this takes place. Where automated decisions are subject to human intervention or review, you should implement processes to ensure this is meaningful and also detail the fact that decisions can be overturned.

It can be difficult to describe the processing activity of AI systems, particularly when they involve complex models and data sources. However, such a description is necessary as part of a DPIA. In some cases, although it is not a legal requirement, it may be good practice for you to maintain two versions of an assessment, with:

  • the first presenting a thorough technical description for specialist audiences; and
  • the second containing a more high-level description of the processing and explaining the logic of how the personal data inputs relate to the outputs affecting individuals (this may also support you in fulfilling your obligation to explain AI decisions to individuals).

Your DPIA should set out your roles and obligations as a controller and include any processors involved. Where AI systems are partly or wholly outsourced to external providers, both you and any other organisations involved should also assess whether joint controllership exists under Article 26 of the UK GDPR; and if so, collaborate in the DPIA process as appropriate.

If you use a processor, you can illustrate some of the more technical elements of the processing activity in a DPIA by reproducing information from that processor. For example, a flow diagram from a processor’s manual. However, you should generally avoid copying large sections of a processor’s literature into your own assessment.

Do we need to consult anyone?

You must, where appropriate:

  • seek and document the views of individuals or their representatives, unless there is a good reason not to;
  • consult all relevant internal stakeholders;
  • consult with your processor, if you use one; and
  • consider seeking legal advice or other expertise.

Unless there is a good reason not to do so, you should seek and document the views of individuals, or their representatives, on the intended processing operation during a DPIA. It is therefore important that you can describe the processing in a way that those you consult can understand. However, if you can demonstrate that consultation would compromise commercial confidentiality, undermine security, or be disproportionate or impracticable, these can be reasons not to consult.

How do we assess necessity and proportionality?

The deployment of an AI system to process personal data needs to be driven by evidence that there is a problem, and a reasoned argument that AI is a sensible solution to that problem, not by the mere availability of the technology. By assessing necessity in a DPIA, you can evidence that you couldn’t accomplish these purposes in a less intrusive way.

A DPIA also allows you to demonstrate that your processing of personal data by an AI system is a proportionate activity. When assessing proportionality, you need to weigh up your interests in using AI against the risks it may pose to the rights and freedoms of individuals. For AI systems, you need to think about any detriment to individuals that could follow from bias or inaccuracy in the algorithms and data sets being used.

Within the proportionality element of a DPIA, you need to assess whether individuals would reasonably expect an AI system to conduct the processing. If AI systems complement or replace human decision-making, you should document in the DPIA how the project might compare human and algorithmic accuracy side-by-side to better justify its use.

You should also describe any trade-offs that are made, for example between statistical accuracy and data minimisation, and document the methodology and rationale for these.

How do we identify and assess risks to individuals?

The DPIA process will help you to objectively identify the relevant risks to individuals’ interests. You should assign a score or level to each risk, measured against the likelihood and the severity of the impact on individuals.

The use of personal data in the development and deployment of AI systems may not just pose risks to individuals’ information rights. When considering sources of risk, your DPIA should consider the potential impact of other material and non-material damage or harm on individuals.


How do we identify mitigating measures?

Against each identified risk to individuals’ interests, you should consider options to reduce the level of assessed risk further. Examples of this could be data minimisation techniques or providing opportunities for individuals to opt out of the processing.

You should ask your DPO (if you have one) for advice when considering ways to reduce or avoid these risks, and you should record in your DPIA whether your chosen measure reduces or eliminates the risk in question.

It is important that DPOs or other information governance professionals or both are involved in AI projects from the earliest stages. There must be clear and open channels of communication between them and the project teams. This will ensure that they can identify and address these risks early in the AI lifecycle.

Data protection should not be an afterthought, and a DPO’s professional opinion should not come as a surprise at the eleventh hour.

You can use a DPIA to document the safeguards you put in place to ensure the individuals responsible for the development, testing, validation, deployment, and monitoring of AI systems are adequately trained and have an understanding of the data protection implications of the processing.

Your DPIA can also evidence the organisational measures you have put in place, such as appropriate training, to mitigate risks associated with human error. You should also document any technical measures designed to reduce risks to the security and accuracy of personal data processed in your AI system.

Once you have introduced measures to mitigate the risks you have identified, the DPIA should document the residual levels of risk posed by the processing.

You are not required to eliminate every risk identified. However, if your assessment indicates a high risk to the data protection rights of individuals that you are unable to sufficiently reduce, you are required to consult the ICO before you can go ahead with the processing.

How do we conclude our DPIA?

You should record:

    • what additional measures you plan to take;
    • whether each risk has been eliminated, reduced or accepted;
    • the overall level of ‘residual risk’ after taking additional measures;
    • the opinion of your DPO, if you have one; and
    • whether you need to consult the ICO.

What happens next?

Although you must carry out your DPIA before the processing of personal data begins, you should also consider it to be a ‘live’ document. This means reviewing the DPIA regularly and undertaking a reassessment where appropriate (eg if the nature, scope, context or purpose of the processing, and the risks posed to individuals, alter for any reason).

For example, depending on the deployment, it could be that the demographics of the target population may shift, or that people adjust their behaviour over time in response to the processing itself. This is a phenomenon in AI known as ‘concept drift’ (for more, see ‘How should we define and prioritise different statistical accuracy measures?’).

    • You should take the time to assess, and document, the status of each organisation you work with in respect of all the personal data and processing activities you carry out.
    • If you exercise overall control of the purpose and means of the processing of personal data – you decide what data to process and why – you are a controller.
    • If you don’t have any purpose of your own for processing the data and you only act on a client’s instructions, you are likely to be a processor – even if you make some technical decisions about how you process the data.
    • Organisations that determine the purposes and means of processing will be controllers regardless of how they are described in any contract about processing services.

As AI usually involves processing personal data in several different phases or for several different purposes, it is possible that you may be a controller or joint controller for some phases or purposes, and a processor for others.

What type of decisions mean we are a controller?

Our guidance says that if you make any of the following overarching decisions, you will be a controller:

    • to collect personal data in the first place;
    • what types of personal data to collect;
    • the purpose or purposes the data are to be used for;
    • which individuals to collect the data about;
    • how long to retain the data; and
    • how to respond to requests made in line with individuals’ rights.

What type of decisions can we take as a processor?

Our guidance says that you are likely to be a processor if you don’t have any purpose of your own for processing the data and you only act on a client’s instructions. You may still be able to make some technical decisions as a processor. For example, where allowed in the contract, you may use your technical knowledge to decide:

    • the IT systems and methods you use to process personal data;
    • how you store the data;
    • the security measures that will protect it; and
    • how you retrieve, transfer, delete or dispose of that data.

Our plans to explore these issues in more detail

Our work has identified that, when AI systems involve a number of organisations in the processing of personal data, assigning the roles of controller and processor can become complex– for instance, when some of the processing happens in the cloud. This raises questions of policy, and we plan to consult with stakeholders including Government to explore these areas, with a view to addressing these issues in more detail when we revise our Cloud Computing Guidance in 2021. This Guidance will also be subject to external stakeholder consultation prior to its finalisation.

As we review our Cloud Computing Guidance, we will consult on the scenarios which could result in an organisation becoming a controller, which may include when organisations make decisions about:

    • the source and nature of the data used to train an AI model;
    • the target output of the model (what is being predicted or classified);
    • the broad kinds of ML algorithms that will be used to create models from the data (eg regression models, decision trees, random forests, neural networks);
    • feature selection – the features that may be used in each model;
    • key model parameters (eg how complex a decision tree can be, or how many models will be included in an ensemble);
    • key evaluation metrics and loss functions, such as the trade-off between false positives and false negatives; and
    • how any models will be continuously tested and updated: how often, using what kinds of data, and how ongoing performance will be assessed.

We will also consider questions regarding when an organisation is (depending on the terms of their contract) able to make decisions to support the provision of AI services, and still remain a processor, for instance in areas such as:

    • the specific implementation of generic ML algorithms, such as the programming language and code libraries they are written in;
    • how the data and models are stored, such as the formats they are serialised and stored in, and local caching;
    • measures to optimise learning algorithms and models to minimise their consumption of computing resources (eg by implementing them as parallel processes); and
    • architectural details of how models will be deployed, such as the choice of virtual machines, microservices, APIs.

As we develop our Cloud Computing Guidance, we will work with stakeholders to develop a range of scenarios when the organisation remains a data processor as it provides AI services. In our work to date we have developed some indicative example scenarios:

How should we manage competing interests when assessing AI-related risks?

Your use of AI must comply with the requirements of data protection law. However, there can be a number of different values and interests to consider, and these may at times pull in different directions. These are commonly referred to as ‘trade-offs’, and the risk-based approach of data protection law can help you navigate them. There are several significant examples relating to AI, which we discuss in detail elsewhere:

    • The interests in training a sufficiently accurate AI system and in reducing the quantity of personal data processed to train that system (see ‘How should we balance data minimisation and statistical accuracy’).
    • Producing an AI system which is sufficiently statistically accurate and which avoids discrimination (see ‘What are the technical approaches to mitigate discrimination risk in ML models?’).
    • Striking the appropriate balance between explainability and statistical accuracy, security, and commercial secrecy (see the explaining decisions made with AI guidance, and ‘What about AI security risks exacerbated by explainable AI?’).

If you are using AI to process personal data you therefore need to identify and assess these interests, as part of your broader consideration of the risks to the rights and freedoms of individuals and how you will meet your obligations under the law.

The right balance depends on the specific sectoral and social context you operate in, and the impact the processing may have on individuals. However, there are methods you can use to assess and mitigate trade-offs that are relevant to many use cases.

How can we manage these trade-offs?

In most cases, striking the right balance between these multiple trade-offs is a matter of judgement, specific to the use case and the context an AI system is meant to be deployed in.

Whatever choices you make, you need to be accountable for them. Your efforts should be proportionate to the risks the AI system you are considering to deploy poses to individuals. You should:

    • identify and assess any existing or potential trade-offs, when designing or procuring an AI system, and assess the impact it may have on individuals;
    • consider available technical approaches to minimise the need for any trade-offs;
    • consider any techniques which you can implement with a reasonable level of investment and effort;
    • have clear criteria and lines of accountability about the final trade-off decisions. This should include a robust, risk-based and independent approval process;
    • where appropriate, take steps to explain any trade-offs to individuals or any human tasked with reviewing AI outputs; and
    • review trade-offs on a regular basis, taking into account, among other things, the views of individuals (or their representatives) and any emerging techniques or best practices to reduce them.

You should document these processes and their outcomes to an auditable standard. This will help you to demonstrate that your processing is fair, necessary, proportionate, adequate, relevant and limited. This is part of your responsibility as a controller under Article 24 and your compliance with the accountability principle under Article 5(2). You must also capture them with an appropriate level of detail where required as part of a DPIA or a legitimate interests assessment (LIA).

You should also document:

    • how you have considered the risks to the individuals that are having their personal data processed;
    • the methodology for identifying and assessing the trade-offs in scope; the reasons for adopting or rejecting particular technical approaches (if relevant);
    • the prioritisation criteria and rationale for your final decision; and
    • how the final decision fits within your overall risk appetite.

You should also be ready to halt the deployment of any AI systems, if it is not possible to achieve a balance that ensures compliance with data protection requirements.

Outsourcing and third-party AI systems

When you either buy an AI solution from a third party, or outsource it altogether, you need to conduct an independent evaluation of any trade-offs as part of your due diligence process. You are also required to specify your requirements at the procurement stage, rather than addressing trade-offs afterwards.

Recital 78 of the UK GDPR says producers of AI solutions should be encouraged to:

    • take into account the right to data protection when developing and designing their systems; and
    • make sure that controllers and processors are able to fulfil their data protection obligations.

You should ensure that any system you procure aligns with what you consider to be the appropriate trade-offs. If you are unable to assess whether the use of a third party solution would be data protection compliant, then you should, as a matter of good practice, opt for a different solution. Since new risks and compliance considerations may arise during the course of the deployment, you should regularly review any outsourced services and be able to modify them or switch to another provider if their use is no longer compliant in your circumstances.

For example, a vendor may offer a CV screening tool which effectively scores promising job candidates but may ostensibly require a lot of information about each candidate to assist with the assessment. If you are procuring such a system, you need to consider whether you can justify collecting so much personal data from candidates, and if not, request the provider modify their system or seek another provider.

Culture, diversity and engagement with stakeholders

You need to make significant judgement calls when determining the appropriate trade-offs. While effective risk management processes are essential, the culture of your organisation also plays a fundamental role.

Undertaking this kind of exercise will require collaboration between different teams within the organisation. Diversity, incentives to work collaboratively, as well as an environment in which staff feel encouraged to voice concerns and propose alternative approaches are all important.

The social acceptability of AI in different contexts, and the best practices in relation to trade-offs, are the subject of ongoing societal debates. Consultation with stakeholders outside your organisation, including those affected by the trade-off, can help you understand the value you should place on different criteria.

What about mathematical approaches to minimise trade-offs?

In some cases, you can precisely quantify elements of the trade-offs. A number of mathematical and computer science techniques known as ‘constrained optimisation’ aim to find the optimal solutions for minimising trade-offs.

For example, the theory of differential privacy provides a framework for quantifying and minimising trade-offs between the knowledge that can be gained from a dataset or statistical model, and the privacy of the people in it. Similarly, various methods exist to create ML models which optimise statistical accuracy while also minimising mathematically defined measures of discrimination.

While these approaches provide theoretical guarantees, it can be hard to meaningfully put them into practice. In many cases, values like privacy and fairness are difficult to meaningfully quantify. For example, differential privacy may be able to measure the likelihood of an individual being uniquely identified from a particular dataset, but not the sensitivity of that identification. Therefore, they may not always be appropriate. If you do decide to use them, you should always supplement these methods with a more qualitative and holistic approach. But the inability to precisely quantify the values at stake does not mean you can avoid assessing and justifying the trade-off altogether; you still need to justify your choices.

In many cases trade-offs are not precisely quantifiable, but this should not lead to arbitrary decisions. You should perform contextual assessments, documenting and justifying your assumptions about the relative value of different requirements for specific AI use cases


Leave a Reply