Peter A Clarke

Artificial Intelligence (“AI”) is becoming a significant issue for lawyers generally and regulators in particular.   Its impact on the law is apparent with the Full Bench, of 5 justices, ruling in Commissioner of Patents v Thaler [2022] FCAFC 62 last month that an inventor in terms of patent law must be a natural person, not AI.  This was an appeal from a decision of Justice Beach on 30 July 2021 in Thaler v Commissioner of Patents [2021] FCA 879 who relevantly ordered:

• The determination of the Deputy Commissioner that s 15(1) of the Patents Act 1990 (Cth) is inconsistent with an artificial intelligence system or device being treated as an inventor be set aside.
• The matter as to whether patent application no. 2019363177 satisfies the formalities under the Patents Regulations 1991 (Cth) and its examination be remitted to the Deputy Commissioner to be determined according to law in accordance with these reasons.

In its reasons the Full Court found  that identification of the “inventor” was central to the operation of the legislation. Under s 15, only the inventor or someone claiming through the inventor is entitled to a patent.

Thaler will probably make its way to the High Court. 

But the use of AI is more prosaic and ubiquitous than in inventing devices.  That is likely to be both a public good and a cause for concern.  At the moment the technology and its implementation is far outpacing the law and regulation.  That is a concern given the potential forseeable and unforseeable consequences.  In that regard I thoroughly recommend Machines Behaving Badly; the Morality of AI by Toby Walsh.   I attended a presentation by Professor Walsh organised by the Centre for Artificial Intelligence and Digital Ethics (CAIDE) last Wednesday. 

Regulators in the United Kingdom and Europe have been much more alive to the need for guidance and consideration of AI and its effect on privacy and data security than in Australia where the regulator takes a more languid approach and seems to be letting the ACCC to take the running on big tech issues.  In that vein the Information Commissioner’s Office (‘ICO’) announced, on 4 May 2022, that it had launched its updated AI and data protection risk toolkit. It is an important document for those interested in both privacy and data security. 

The announcement provides:

The innovation, opportunities and potential value to society of AI will not need emphasising to anyone reading this guidance.

Nor is there a need to underline the range of risks involved in the use of technologies that shift processing of personal data to complex computer systems with often opaque approaches and algorithms.

This guidance helps organisations mitigate the risks specifically arising from a data protection perspective, explaining how data protection principles apply to AI projects without losing sight of the benefits such projects can deliver.

What stands out in the following pages is that the underlying data protection questions for even the most complex AI project are much the same as with any new project. Is data being used fairly, lawfully and transparently? Do people understand how their data is being used? How is data being kept secure?

The legal principle of accountability, for instance, requires organisations to account for the risks arising from their processing of personal data –whether they are running a simple register of customers’ contact details or operating a sophisticated AI system to predict future consumer demand.

Aspects of the guidance should act as an aide memoire to those running AI projects. There should be no surprises in the requirements for data protection impact assessments or of documenting decisions. The guidance offers support and methodologies on how best to approach this work.

Other aspects of the law require greater thought. Data minimisation, for instance, may seem at odds with systems that allow machine learning to conclude what information is necessary from large data sets. As the guidance sets out though, there need not be a conflict here, and there are several techniques that can ensure organisations only process the personal data needed for their purpose.

Similarly, transparency of processing, mitigating discrimination, and ensuring individual rights around potential automated decision-making can pose difficult questions. Aspects of this are complemented by our existing guidance ‘Explaining decisions made with AI guidance’, published with the Alan Turing Institute in May 2020.

The common element to these challenging areas, and perhaps the headline takeaway, is the value of considering data protection at an early stage. Mitigation of risks must come at the design stage: retrofitting compliance as an end-of-project bolt-on rarely leads to comfortable compliance or practical products. This guidance should accompany that early engagement with compliance, in a way that ultimately benefits the people whose data AI approaches rely on.

The development and use of AI within our society is growing and evolving, and it feels as though we are at the early stages of a long journey. We will continue to focus on AI developments and their implications for privacy by building on this foundational guidance, and continuing to offer tools that promote privacy by design to those developing and using AI.

I must end with an acknowledgment of the excellent work of one of the document’s authors, Professor Reuben Binns. Prof Binns joined the ICO as part of a fellowship scheme designed to deepen my office’s understanding of this complex area, as part of our strategic priority of enabling good practice in AI. His time at the ICO, and this guidance in particular, is testament to the success of that fellowship, and we wish Prof Binns the best as he continues his career as Associate Professor of Computer Science at the University of Oxford.

The guide is useful in explaining that:

• there are four options when scoring risks, i.e. ‘high’, ‘medium’, ‘low’, and ‘non-applicable’;
• the assessment of risks will vary depending on the context, so companies should undertake their own assessments of the risks identified;

The toolkit deals with:

• risk areas during different states of an artifical intelligence (‘AI’) lifecycle being:
  • business requirements and design,
  • data acquisition and preparation,
  • training and testing, and
  • deployment and monitoring,
• provides a risk assessment summary,
• steps an organisation can take to mitigate risks.
• provides relevant guidance.

The ICO’s Executive Director for Regulatory Futures and Innovation, Stephen Bonner, was quoted as stating: 

“[o]rganisations can use our toolkit to assess the privacy risks associated with their use of AI. It provides practical controls to help reduce these risks, putting data protection principles into practice and helping to provide assurance to the organisation and the public about their compliance. The toolkit forms part of the ICO AI Auditing Framework, which is designed to give us a clearer method to assess the data protection compliance of AI systems”.

The toolkit is not intended to replace a Data Protection Impact Assessment (‘DPIA’) and  is voluntary.

The executive summary provides:

In relation to how to approach AI from a governance and risk management perspective the Guide provides:

If used well, AI has the potential to make organisations more efficient, effective and innovative. However, AI also raises significant risks for the rights and freedoms of individuals, as well as compliance challenges for organisations.

Different technological approaches will either exacerbate or mitigate some of these issues, but many others are much broader than the specific technology. As the rest of this guidance suggests, the data protection implications of AI are heavily dependent on the specific use cases, the population they are deployed on, other overlapping regulatory requirements, as well as social, cultural and political considerations.

While AI increases the importance of embedding data protection by design and default into an organisation’s culture and processes, the technical complexities of AI systems can make this more difficult. Demonstrating how you have addressed these complexities is an important element of accountability.

You cannot delegate these issues to data scientists or engineering teams. Your senior management, including DPOs, are also accountable for understanding and addressing them appropriately and promptly (although overall accountability for data protection compliance lies with the controller, ie your organisation).

To do so, in addition to their own upskilling, your senior management will need diverse, well-resourced teams to support them in carrying out their responsibilities. You also need to align your internal structures, roles and responsibilities maps, training requirements, policies and incentives to your overall AI governance and risk management strategy.

It is important that you do not underestimate the initial and ongoing level of investment of resources and effort that is required. Your governance and risk management capabilities need to be proportionate to your use of AI. This is particularly true now while AI adoption is still in its initial stages, and the technology itself, as well as the associated laws, regulations, governance and risk management best practices are still developing quickly.

We have also developed a more general accountability framework. This is not specific to AI, but provides a baseline for demonstrating your accountability under the UK GDPR, on which you could build your approach to AI accountability.

How should we set a meaningful risk appetite?

The risk-based approach of data protection law requires you to comply with your obligations and implement appropriate measures in the context of your particular circumstances – the nature, scope, context and purposes of the processing you intend to do, and the risks this poses to individuals’ rights and freedoms.

Your compliance considerations therefore involve assessing the risks to the rights and freedoms of individuals and judging what is appropriate in those circumstances. In all cases, you need to ensure you comply with data protection requirements.

This applies to the use of AI just as to other technologies that process personal data. In the context of AI, the specific nature of the risks posed and the circumstances of your processing will require you to strike an appropriate balance between competing interests as you go about ensuring data protection compliance. This may in turn impact the outcome of your processing. It is unrealistic to adopt a ‘zero tolerance’ approach to risks to rights and freedoms, and indeed the law does not require you to do so. It is about ensuring that these risks are identified, managed and mitigated. We talk about trade-offs and how you should manage them below and provide examples of some trade-offs throughout the guidance.

To manage the risks to individuals that arise from processing personal data in your AI systems, it is important that you develop a mature understanding of fundamental rights, risks, and how to balance these and other interests. Ultimately, it is necessary for you to:

• • assess the risks to individual rights that your use of AI poses;
  • determine how you will address these; and
  • establish the impact this has on your use of AI.

You should ensure your approach fits both your organisation and the circumstances of your processing. Where appropriate, you should also use risk assessment frameworks.

This is a complex task, which can take time to get right. However, it will give you, as well as the ICO, a fuller and more meaningful view of your risk positions and the adequacy of your compliance and risk management approaches.

The following sections deal with the AI-specific implications of accountability including:

• • how you should undertake data protection impact assessments for AI systems;
  • how you can identify whether you are a controller or processor for specific processing operations involved in the development and deployment of AI systems and the resulting implications for your responsibilities;
  • how you should assess the risks to the rights and freedoms of individuals, and how you should address them when you design, or decide to use, an AI system; and
  • how you should justify, document and demonstrate the approach you take, including your decision to use AI for the processing in question.

What do we need to consider when undertaking data protection impact assessments for AI?

DPIAs are a key part of data protection law’s focus on accountability and data protection by design.

You should not see DPIAs as simply a box ticking compliance exercise. They can effectively act as roadmaps for you to identify and control the risks to rights and freedoms that using AI can pose. They are also an ideal opportunity for you to consider and demonstrate your accountability for the decisions you make in the design or procurement of AI systems.

Why are DPIAs required under the data protection law?

In the vast majority of cases, the use of AI will involve a type of processing likely to result in a high risk to individuals’ rights and freedoms, and will therefore trigger the legal requirement for you to undertake a DPIA. You will need to make this assessment on a case by case basis. In those cases where you assess that a particular use of AI does not involve high risk processing, you still need to document how you have made this assessment.

If the result of an assessment indicates residual high risk to individuals that you cannot sufficiently reduce, you must consult with the ICO prior to starting the processing.

In addition to conducting a DPIA, you may also be required to undertake other kinds of impact assessments or do so voluntarily. For example, public sector organisations are required to undertake equality impact assessments, while other organisations voluntarily undertake ‘algorithm impact assessments’. Similarly, the machine learning community has proposed ‘model cards’ and ‘datasheets’ which describe how ML models may perform under different conditions, and the context behind the datasets they are trained on, which may help inform an impact assessment. There is no reason why you cannot combine these exercises, so long as the assessment encompasses all the requirements of a DPIA.

The ICO has produced detailed guidance on DPIAs that explains when they are required and how to complete them. This section sets out some of the things you should think about when carrying out a DPIA for the processing of personal data in AI systems.

How do we decide whether to do a DPIA?

We acknowledge that not all uses of AI will involve types of processing that are likely to result in a high risk to rights and freedoms. However, you should note that Article 35(3)(a) of the UK GDPR requires you to undertake a DPIA if your use of AI involves:

• • systematic and extensive evaluation of personal aspects based on automated processing, including profiling, on which decisions are made that produce legal or similarly significant effects;
  • large-scale processing of special categories of personal data; or
  • systematic monitoring of publicly-accessible areas on a large scale.

Beyond this, AI can also involve several processing operations that are themselves likely to result in a high risk, such as use of new technologies or novel application of existing technologies, data matching, invisible processing, and tracking of location or behaviour. When these involve things like evaluation or scoring, systematic monitoring, and large-scale processing, the requirement to do a DPIA is triggered.

In any case, if you have a major project that involves the use of personal data it is also good practice to do a DPIA. Read our list of processing operations ‘likely to result in high risk’ for examples of operations that require a DPIA, and further detail on which criteria are high risk in combination with others.

What should we assess in our DPIA?

Your DPIA needs to describe the nature, scope, context and purposes of any processing of personal data. It needs to make clear how and why you are going to use AI to process the data. You need to detail:

• • how you will collect, store and use data;
  • the volume, variety and sensitivity of the data;
  • the nature of your relationship with individuals; and
  • the intended outcomes for individuals or wider society, as well as for you.

Whether a system using AI is generally more or less risky than a system not using AI depends on the specific circumstances. You therefore need to evaluate this based on your own context. Your DPIA should show evidence of your consideration of less risky alternatives, if any, that achieve the same purpose of the processing, and why you didn’t choose them. This consideration is particularly relevant where you are using public task or legitimate interests as a lawful basis. See ‘How do we identify our purposes and lawful basis’.

In the context of the AI lifecycle, a DPIA will best serve its purpose if you undertake it at the earliest stages of project development. It should feature, at a minimum, the following key components.

How do we describe the processing?

Your DPIA should include:

• • a systematic description of the processing activity, including data flows and the stages when AI processes and automated decisions may produce effects on individuals;
  • an explanation of any relevant variation or margins of error in the performance of the system which may affect the fairness of the personal data processing (see ‘What do we need to do about Statistical Accuracy’); and
  • a description of the scope and context of the processing, including:
    • what data you will process;
    • the number of data subjects involved;
    • the source of the data; and
    • how far individuals are likely to expect the processing.

Your DPIA should identify and record the degree of any human involvement in the decision-making process and at what stage this takes place. Where automated decisions are subject to human intervention or review, you should implement processes to ensure this is meaningful and also detail the fact that decisions can be overturned.

It can be difficult to describe the processing activity of AI systems, particularly when they involve complex models and data sources. However, such a description is necessary as part of a DPIA. In some cases, although it is not a legal requirement, it may be good practice for you to maintain two versions of an assessment, with:

Your DPIA should set out your roles and obligations as a controller and include any processors involved. Where AI systems are partly or wholly outsourced to external providers, both you and any other organisations involved should also assess whether joint controllership exists under Article 26 of the UK GDPR; and if so, collaborate in the DPIA process as appropriate.

If you use a processor, you can illustrate some of the more technical elements of the processing activity in a DPIA by reproducing information from that processor. For example, a flow diagram from a processor’s manual. However, you should generally avoid copying large sections of a processor’s literature into your own assessment.

Do we need to consult anyone?

You must, where appropriate:

Unless there is a good reason not to do so, you should seek and document the views of individuals, or their representatives, on the intended processing operation during a DPIA. It is therefore important that you can describe the processing in a way that those you consult can understand. However, if you can demonstrate that consultation would compromise commercial confidentiality, undermine security, or be disproportionate or impracticable, these can be reasons not to consult.

How do we assess necessity and proportionality?

The deployment of an AI system to process personal data needs to be driven by evidence that there is a problem, and a reasoned argument that AI is a sensible solution to that problem, not by the mere availability of the technology. By assessing necessity in a DPIA, you can evidence that you couldn’t accomplish these purposes in a less intrusive way.

A DPIA also allows you to demonstrate that your processing of personal data by an AI system is a proportionate activity. When assessing proportionality, you need to weigh up your interests in using AI against the risks it may pose to the rights and freedoms of individuals. For AI systems, you need to think about any detriment to individuals that could follow from bias or inaccuracy in the algorithms and data sets being used.

Within the proportionality element of a DPIA, you need to assess whether individuals would reasonably expect an AI system to conduct the processing. If AI systems complement or replace human decision-making, you should document in the DPIA how the project might compare human and algorithmic accuracy side-by-side to better justify its use.

You should also describe any trade-offs that are made, for example between statistical accuracy and data minimisation, and document the methodology and rationale for these.

How do we identify and assess risks to individuals?

The DPIA process will help you to objectively identify the relevant risks to individuals’ interests. You should assign a score or level to each risk, measured against the likelihood and the severity of the impact on individuals.

The use of personal data in the development and deployment of AI systems may not just pose risks to individuals’ information rights. When considering sources of risk, your DPIA should consider the potential impact of other material and non-material damage or harm on individuals.