AI & Technology

We Need Better Protection for Citizens in the Face of Automated Decision Making

The second Reading of my Private Members Bill tool place recently. It is designed to give greater rights to all of us who are subject to AI and Automated decision making in government which is becoming increasingly prevalent with the enthusiasm of the new Labour government to "digitally transform" our public services.

 I thank Big Brother Watch, the Public Law Project and the Ada Lovelace Institute, which, each in their own way, have provided the evidence and underpinned my resolve to ensure that we regulate the adoption of algorithmic and AI tools in the public sector, which are increasingly being used across it to make and support many of the highest-impact decisions affecting individuals, families and communities across healthcare, welfare, education, policing, immigration and many other sensitive areas of an individual’s life. I also thank the Public Bill Office, the Library and other members of staff for all their assistance in bringing this Bill forward and communicating its intent and contents, and I thank all noble Lords who have taken the trouble to come to take part in this debate this afternoon.

The speed and volume of decision-making that new technologies will deliver is unprecedented. They have the potential to offer significant benefits, including improved efficiency and cost effectiveness in government operations, enhanced service delivery and resource allocation, better prediction and support for vulnerable people and increased transparency in public engagement. However, the rapid adoption of AI in the public sector also presents significant risks and challenges, with the potential for unfairness, discrimination and misuse through algorithmic bias and the need for human oversight, a lack of transparency and accountability in automated decision-making processes and privacy and data protection concerns.

Incidents such as the 2020 A-level and GCSE grading fiasco, where an algorithmic approach saw students, particularly those from lower-income areas, unfairly miss out on university places when an algorithm was used to estimate grades from exams that were cancelled because of Covid-19, have starkly illustrated the dangers of unchecked algorithmic systems in public administration disproportionately affecting those from lower-income backgrounds. That led to widespread public outcry and a loss of trust in government use of technology.

Big Brother Watch’s investigations have revealed that councils across the UK are conducting mass profiling and citizen scoring of welfare and social care recipients. Its report, entitled Poverty Panopticon [The Hidden Algorithms Shaping Britain’s Welfare State], uncovered alarming statistics. Some 540,000 benefits applicants are secretly assigned fraud risk scores by councils’ algorithms before accessing housing benefit or council tax support. Personal data from 1.6 million people living in social housing is processed by commercial algorithms to predict rent non-payers. Over 250,000 people’s data is processed by secretive automated tools to predict the likelihood of abuse, homelessness or unemployment.

Big Brother Watch criticises the nature of these algorithms, stating that most are secretive, unevidenced, incredibly invasive and likely discriminatory. It argues that these tools are being used without residents’ knowledge, effectively creating tools of automated suspicion. The organisation rightly expressed deep concern that these risk-scoring algorithms could be disadvantaging and discriminating against Britain’s poor. It warns of potential violations of privacy and equality rights, drawing parallels to controversial systems like the Metropolitan Police’s gangs matrix database, which was found to be operating unlawfully. From a series of freedom of information requests last June, Big Brother Watch found that a flawed DWP algorithm wrongly flagged 200,000 housing benefit claimants for possible fraud and error, which meant that thousands of UK households every month had their housing benefit claims unnecessarily investigated.

In August 2020, the Home Office agreed to stop using an algorithm to help sort visa applications after it was discovered that the algorithm contained entrenched racism and bias, and following a challenge from the Joint Council for the Welfare of Immigrants and the digital rights group Foxglove. The algorithm essentially created a three-tier system for immigration, with a speedy boarding lane for white people from the countries most favoured by the system. Privacy International has raised concerns about the Home Office's use of a current tool called Identify and Prioritise Immigration Cases—IPIC—which uses personal data, including biometric and criminal records to prioritise deportation cases, arguing that it lacks transparency and may encourage officials to accept recommended decisions without proper scrutiny.

Automated decision-making has been proven to lead to harms in privacy and equality contexts, such as in the Harm Assessment Risk Tool, which was used by Durham Police until 2021, and which predicted reoffending risks partly based on an individual’s postcode in order to inform charging decisions. All these cases illustrate how ADM can perpetuate discrimination. The Horizon saga illustrates how difficult it is to secure proper redress once the computer says no.

There is no doubt that our new Government are enthusiastic about the adoption of AI in the public sector. Both the DSIT Secretary of State and Feryal Clark, the AI Minister, are on the record about the adoption of AI in public services. They have ambitious plans to use AI and other technologies to transform public service delivery. Peter Kyle has said:

“We’re putting AI at the heart of the government’s agenda to boost growth and improve our public services”,

and

“bringing together digital, data and technology experts from across Government under one roof, my Department will drive forward the transformation of the state”.—[Official Report, Commons, 2/9/24; col. 89.]

Feryal Clarke has emphasised the Administration’s desire to “completely transform digital Government” with DSIT. As the Government continue to adopt AI technologies, it is crucial to balance the potential benefits with the need for responsible and ethical implementation to ensure fairness, transparency and public trust.

The Ada Lovelace Institute warns of the unintended consequences of AI in the public sector, including the risk of entrenching existing practices, instead of fostering innovation and systemic solutions. As it says, the safeguards around automated decision-making, which exist only in data protection law, are therefore more critical than ever in ensuring people understand when a significant decision about them is being automated, why that decision is made, and have routes to challenge it, or ask for it to be decided by a human.

Our citizens need greater, not less, protection, but rather than accepting the need for these, we see the Government following in the footsteps of their predecessor by watering down such rights as there are under GDPR Article 22 not to be subject to automated decision-making. We will, of course, be discussing these aspects of the Data (Use and Access) Bill in Committee next week.

ADM safeguards are critical to public trust in AI, but progress has been glacial. Take the Algorithmic Transparency Recording Standard, which was created in 2022 and is intended to offer a consistent framework for public bodies to publish details of the algorithms used in making these decisions. Six records were published at launch, and only three more seem to have been published since then. The previous Government announced earlier this year that the implementation of the Algorithmic Transparency Recording Standard will be mandatory for departments. Minister Clark in the new Government has said,

“multiple records are expected to be published soon”,

but when will this be consistent across government departments? What teeth do the Central Digital and Data Office and the Responsible Technology Adoption Unit, now both within DSIT, have to ensure the adoption of the standard, especially in view of the planned watering down of the Article 22 GDPR safeguards? Where is the promised repository for ATRS records? What about the other public services in local government too?

The Public Law Project, which maintains a register called Tracking Automated Government, believes that in October last year there were more than 55 examples of public ADM systems use. Where is the transparency on those? The fact is that the Government’s Algorithmic Transparency Recording Standard, while a step in the right direction, remains voluntary and lacks comprehensive adoption or indeed a compliance mechanism or opportunity for redress. The current regulatory landscape is clearly inadequate to address these challenges. Despite the existing guidance and framework, there is no legally enforceable obligation on public authorities to be transparent about their use of ADM and algorithmic systems, or to rigorously assess their impact.

To address these challenges, several measures are needed. We need to see the creation of and adherence to ethical guidelines and accountability mechanisms for AI implementation; a clear regulatory framework and standards for use in the public sector; increased transparency and explainability of the adoption and use of AI systems; investment in AI education; and workforce development for public sector employees. We also need to see the right of redress, with a strengthened right for the individuals to challenge automated decisions.

My Bill aims to establish a clear mandatory framework for the responsible use of algorithmic and automated decision-making systems in the public sector. It will help to prevent the embedding of bias and discrimination in administrative decision-making, protect individual rights and foster public trust in government use of new technologies.

I will not adumbrate all the elements of the Bill. In an era when AI and algorithmic systems are becoming increasingly central to government ambitions for greater productivity and public service delivery, this Bill, I hope noble Lords agree, is crucial to ensuring that the benefits of these technologies are realised while safeguarding democratic values and individual rights. By ensuring that ADM systems are used responsibly and ethically, the Bill facilitates their role in improving public service delivery, making government operations more efficient and responsive.

The Bill is not merely a response to past failures but a proactive measure to guide the future use of technology within government and empower our citizens in the face of these powerful new technologies. I hope that the House and the Government will agree that this is the way forward.

---

by Tim Clement-Jones

Lord C-J Commentary on the new Government's Science and Technology Programme

Sadly we only had 5 minutes speaking time in the recent Kings Speech debate . Here is an an extended version of my speech which goes into greater depth as to what I believe the Government should be doing in this area if it is to fulfill its growth through innovation agenda and expresses some caveats about how they plan to do this.

When we debated the New government’s proposals in the Kings speech recently the House of Lords  gave  a particularly warm welcome to Lord Vallance of Balham-formerly Sir Patrick Vallance-  as the new Minister of State in the Department.  While the Government’s Chief Scientific Adviser we know from the book “the Long Shot” how he played an  critical role in the establishment of the UK Vaccine Taskforce, which was set up in April 2020 in response to the COVID-19 pandemic. He was pivotal in the recruitment of Dame Kate Bingham to chair the Vaccine Taskforce and in organizing the overall strategy for the UK development and distribution of COVID-19 vaccines. For that we should be eternally grateful. 

 I welcome the Government’s growth through innovation agenda and mission to enhance  public services through the deployment of new technology and also the  concentration of digital functions in DSIT  and that it will become  the centre for “digital expertise and delivery in government,improving how the government and public services interact with citizens.”  in the words of the new Secretary  of State, Peter Kyle. 

The Government is expanding the department’s scope and size by bringing in experts in data, digital, and AI from the Government Digital Service, the Incubator for AI (i.AI), and the Central Digital Data Office to unite efforts to implement digital transformation of public services under one roof.  There is great potential in justice, education, healthcare to name but three areas. 

This is crucial particularly in the adoption of  innovative technologies and tools in our healthcare for which Liberal Democrats believe there should be ring-fenced budgets. We need to be ensuring interoperability of IT systems too.

They government have committed too to modernising public sector procurement frameworks to enable start-ups and SMEs to drive public sector innovation and better public services. Will , however, clear, transparent framework of standards incorporating ethical principles be established? Public sector adoption is very desirable but requires trust on the part of the public/ and the citizen For instance we need to ensure that citizens can assert their rights when faced with automated decision making or live facial recognition

It has felt, under the previous regime, that universities have been under continual threat from government rather than valued as the engines of knowledge and growth and we need to be far more internationally outward looking, in particular fixing our relationship with the EU- using science and technology to address societal challenges for a more resilient and prosperous future in the words of the Royal Society.  

I welcome the new Industrial Strategy Council. Does this mean we can plan for 10 years of stability and opportunity creation in science and tech sector? Successive policy changes to the R&D tax regime over the past several years have created uncertainty and additional red-tape for SMEs, putting at risk the UK’s reputation as a location for innovative businesses.We need to give businesses certainty and incentivise them to invest in new technologies to grow the economy,  create good jobs and tackle the climate crisis. 

Opening up what can be a blocked  pipeline all the way from R & D to commercialisation, from university spinout through start up to scale up and IPO, and crowding in and derisking private investment through the National Wealth Fund, the British Business Bank  and post Mansion House pension reforms, are crucial with all the local, regional, national and UK wide aspects, recognizing the importance of innovation clusters and centres of excellence. We need to tackle regional disparities and develop the innovation clusters with greater devolution to combined authorities

Digital Skills and Digital literacy are also crucial but to deploy digital tools successfully we also need a  pipeline of creative collaborative and critical thinking skills. A massive skills and upskilling agenda is needed in the face of technology advances. The focus in training should be on lifelong skills grants, reforming the apprenticeship levy, and boosting vocational training and apprenticeships and many of the governments proposals in this respect are welcome. 

In this context, as the the chair of a university governing council I very much welcome the Government’s new tone on the value of universities, of long term settlements,  and of resetting relations with Europe and international research collaboration.

The role of university research and spinouts is crucial . The Research Excellence Framework has the perverse incentive of discouraging cooperation. We should be encouraging strategic partnerships in research especially internationally. We need to be full throated members of Horizon -the uncertainty has been extremely damaging to collaboration. I hope the government will now  commit to joining the European Innovation Council as well

Last year Labour set out its plan for the life sciences.It committed to the investment of £10bn into R&D. Further, the plan said that Labour would see the creation of 100,000 jobs in the life sciences sector by 2030. The document contains a range of further welcome pledges including strengthening the Office for Life Sciences and the Life Sciences Council, and  to bring laboratory clusters within the scope of the ‘Nationally significant infrastructure regime’ in England.

We need to ensure Government spending on R&D keeps pace with other nations, and establish a long-term strategy for science, research and innovation that commands cross party support.Research, development and innovation are crucial to driving productivity growth, yet our current levels of R&D investment and productivity lag the G7. I hope this means that we will soon see whether spending plans for government  R & D expenditure by 2030 and 2035 match their words. 

And disproportionately high overseas researcher visa costs  MUST be lowered as Lord Vallance recommended in his Digital Technology Review.  UK visa costs are up to 17 times higher than other leading science nations.The Royal Society have called this a  “punitive tax on talent”. 

But support for innovation should not be unconditional or at any cost. I hope this government will not fall into the trap of viewing regulation as necessarily the enemy of innovation. We need guardrails to ensure that, for example, AI adoption leads to public benefit.

I hope therefore that the reference to AI regulation in the King's Speech, but failure to announce a bill, is only a timing issue. What IS the Government’s intention especially given an AI  bill was heavily trailed in the media?  

With AI technologies continuing to develop at an exponential rate, clarity on regulation is needed by developers and adopters.There is the question too as to what extent the new government will depart from the current sectoral approach to regulating AI and adopt a cross-sectoral approach. What does the King's Speech reference to regulating "the most powerful artificial intelligence models" actually refer to? Will the government be launching yet another consultation on AI regulation?

There is no doubt we need to seize the opportunities of AI,  whilst making sure we mitigate the risks of AI, ensuring ethical standards for AI development and use are adopted.

 Liberal Democrats believe we need to create a clear, workable and well-resourced cross-sectoral regulatory framework for artificial intelligence that:

• Promotes innovation while creating certainty for AI users, developers and investors.
•  Establishes transparency and accountability for AI systems in the public sector.
•  Ensures the use of personal data and AI is unbiased, transparent and  accurate, and respects the privacy of innocent people

The government in particular should lead the way in ensuring that there is a high level of transparency and opportunity for redress when Algorithmic and automated systems are used in government. I commend my new private members bill (the Public Authority Algorithmic and Automated Decision-Making Systems Bill) to it! 

The government should also negotiate the UK’s participation in the Trade and Technology Council with the US  and the EU, so we can play a leading role in global AI regulation, and we should work with international partners in agreeing common global standards for AI risk and impact assessment, testing, training monitoring and audit. 

As regards AI regulation in  the Kings Speech itself we are promised  a Product Safety and  Metrology bill which could require alignment of AI driven products with the EU AI Act which seems to be putting the cart well in front of the AI regulatory horse. 

We do need however to ensure that high risk systems are mandated to adopt international ethical and safety standards.At the same time in In this age of IOT we should require all  suppliers to provide a short, clear version of their terms and conditions, setting  out the key facts as they relate to individuals’ data and privacy.

As regards the creative industries there are clearly great opportunities in relation to the use of AI but there are also challenges and big questions over authorship and intellectual property and many artists feel threatened-the root cause of the recent Hollywood writers and actors strike. What is the government’s approach?

We need to establish very clearly that Generative AI systems need a licence to ingest copyright material for training purposes-just as Mumsnet and the New York Times are asserting- and that there is an obligation of transparency in the use of data sets and original  content.

Lord Vallance is on record as wanting certainty in the relationship between IP rights and generative AI for innovator and investor confidence. And this should be the case for for creatives too. Copyright content needs to be properly remunerated by the tech platforms. The bill needs to make clear that platforms profit from content and need to pay properly and fairly, on benchmarked terms and with reference to value for end users when content is use for training Large Language Models.

And when will the government  set up the promised new Regulatory Innovation Office? This was promised as an organisation to help “regulators to update regulation, speed up approval timelines and co-ordinate issues that span existing boundaries”. and as a “pro-innovation body” designed to “set targets for tech regulators, end uncertainty for businesses, turbocharge output, and boost economic growth”. We need in particular to know whether it will replace the Digital Regulators Cooperation Forum.

We must also ensure we have the right climate for FDI. The Harrington Report called for a new Business investment Strategy for the Office for Investment. Despite the previous government’s Life  Sciences  Vision we have seen pharma company Eli Lilley pulling investment on laboratory space in London because the UK “does not invite inward investment at this time”.  Astra Zeneca decided to build its next plant in Ireland  because of the U.K.’s “discouraging” tax rate. 

We also need to modernise employment rights to make them fit for the age of the gig economy,including by establishing a new ‘dependent contractor’ employment status in between  employment and self-employment, with entitlements to basic rights such as  minimum earnings levels, sick pay and holiday entitlement.

There is a great need for need for greater  diversity and inclusion in the AI workforce and science and technology more broadly. Only one in four senior tech employees in the UK are women, and only 14% from ethnic minorities. 

I hope the Government too is fully committed despite its growth agenda to a full hearted support for the Competition and Markets Authority in the use of its powers under the new Digital Markets Act. I welcome the CMA’s market investigation into Cloud Services and its reassurance that it is looking broadly at the anti-competitive practices of the service providers such as vendor lock-in tactics and non-competitive procurement. 

Then again how will the government kickstart better progress on Project Gigabit? Given the competitive model for rollout of broadband services that has been chosen, investors in alternative providers to the incumbents need reassurance that their investment is going onto a level playing field and not one tilted in favour of the incumbents. 

Also in terms of vital cross departmental working, joining up government on Science and Technology policy we need to know what  the role will be of the National Science and Technology Council and what are its key priorities.

There no mention in Labour’s manifesto on the potential impact of AI on the  workplace.The TUC and Institute for the Future of Work are among those who have called for new legislation to create further legal protections for workers and employers in relation to the use of AI. The government should introduce safeguards against the invasion of privacy through surveillance technology and discriminatory algorithmic decision-making in the workplace along the lines of the TUC draft bill and algorithmic impact assessment along the lines of IFOW’s proposals. 

The Government’s will also need to decide how to follow up on the recommendations of recent key Reports such as

• Professor Dame Angela McLean’s Review of Life Sciences
• The Vallance Review of Pro-innovation Regulation of Digital Technologies
• The Independent Review of Research Bureaucracy by Professor Adam Tickell
• The Independent Review of the UKRI by Sir David Grant
• The Independent Review of the UK’s Research, Development and Innovation Landscape by Sir Paul Nurse
• The O’Shaughnessy Report on Clinical Trials
• The Independent Review of the Future of Compute by Professor Zoubin Ghahramani FRS and 
• The Independent Review of University Spin-out Companies by Professor Irene Tracey and Dr. Andrew Williamson

More broadly it will need to set out its  approach to the science and technology framework for DSIT set out by the previous government in 2023 with its 10 priority areas  Will this be revised? If so they need to set measurable targets and key outcomes in the priority areas. The  government will also  need to take a clear view on  the key technologies we should be assisting in developing and commercialising 

Then there are the pre existing financial commitments in the science and technology field. The Chancellor has said she will be checking all the previous government’s commitments for affordability. Which of  the previous Government’s financial commitments will she confirm? For instance 

The  £7.4 million upskilling fund pilot to help SMEs develop AI skills.

Investing up to £100 million in the Alan Turing Institute over the next five years (up from £50 million)

The £100 million investment by the British Business Bank into ICG,in respect of  the Long-term Investment for Technology and Science (LIFTS) initiative

The £1.1 billion funding for 65 Centres for Doctoral Training (CDTs) through the Engineering and Physical Sciences Research Council (EPSRC), covering key technologies like AI and engineering biology. 

As regards the bills in the Kings Speech I look forward to seeing the details but the Digital Information and Smart Data bill does seem to be heading in the right direction in the areas being reinstated. The retention and enhancement of public trust in data use and sharing is the overriding need so that  the potential of data can be unleashed through better trusted sharing of data.  It is really important that we do more to educate the public about how and where our data is used and what powers individuals have to find out this information

 I hope other than a few clarifications, especially in the research area, and in terms of the constitution of the ICO  we are not going exhume some of the worst areas  of the old DPDI bill and we have ditched the idea of a Brexit EU divergence Dividend by the watering down of so many data subject rights.

Will the Government give a firm commitment to safeguard our data adequacy with the EU? Will the bill  introduce the promised  ban on the creation of sexually explicit deepfakes?

I also hope that the Government will confirm that the intent of the reinstated Digital Verification provisions is not compulsory national Digital ID but the creation of a market in digital ID providers that give choice to the citizen.

Given that LinesearchbeforeUdig, or LSBUD is claimed to already achieve the aims of NAUR, to be more widely used than the National Underground Assert Register NUAR and be more cost-effective, I hope also that Ministers will meet LSBUD and provide us all with much greater clarity around these proposals. 

I hope that we can include other positive spects of the late unlamented DPDI Bill  in the bill: More action on online fraud, digital identity theft, deepfakes in elections Misinformation and disinformation, misogyny as a hate crime, there is quite a list of possibilities. Together with new models of personal data control which were advocated as long ago as 2017 with the Hall Pesenti review, especially through new data communities and institutions and an enhanced ability to exercise our right to data portability, especially in real-time and more regulatory oversight over use of biometrics and biometric technologies. 

 I of course welcome the pledge to give coroners more  powers to access information held by technology companies after a child’s death AND to banning the creation of sexually explicit deepfakes.

As regards the Cyber Security and Resilience Bill, events of recent days have made it clear we are not just talking about threats from bad actors. It reminds us how dependent we are on just a few overly dominant major tech companies. With Microsoft and AWS enjoying a combined UK market share of around 70-90%, according to the Competition and Markets Authority’s own research, the lack of competition presents a serious concerns for our nation's security and resilience. There needs to to be a rethink on critical national infrastructure such as cloud services and business software which are now essential public utilities and also how we are wholesale replacing reliable analogue communication with digital systems without backup. 

In the bill I hope will we see the long awaited amendment of the Computer Misuse Act to include a statutory public interest defence, as called for by Cyber Up, to allow white hat research into computer systems as the Vallance report recommended.  The rules for computer evidence must be changed too. We must have no more Horizon scandals!

 

---

by Tim Clement-Jones

Data Protection and Digital Information Bill lost in wash up-Hurray!

---

by Tim Clement-Jones

Lords Debate Report on AI in Weapon Systems

Recently the House of Lords Debated the Report of the AI in  Weapon Systems Committee Proceed with Caution.

This is an edited version of what I said

Autonomous weapon systems present some of the most emotive and high-risk challenges posed by AI. We have heard a very interesting rehearsal of some of the issues surrounding use and possible benefits, but particularly the risks. I believe that the increasing use of drones in particular, potentially autonomously, in conflicts such as Libya, Syria and Ukraine and now by Iran and Israel, together with AI targeting systems such as Lavender, highlights the urgency of addressing the governance of weapon systems.

The implications of autonomous weapons systems—AWS—are far-reaching. There are serious risks to consider, such as escalation and proliferation of conflict, accountability and lack of accountability for actions,

and cybersecurity vulnerabilities. There is the lack of empathy and kindness qualities that humans are capable of in making military decisions.  There is misinformation and disinformation, which is a new kind of warfare.

Professor Stuart Russell, in his Reith lecture on this subject in 2021, painted a stark picture of the risks posed by scalable autonomous weapons capable of destruction on a mass scale. This chilling scenario underlines the urgency with which we must approach the regulation of AWS. The UK military sees AI as a priority for the future, with plans to integrate “boots and bots” to quote a senior military officer.

The UK integrated review of 2021 made lofty commitments to ethical AI development. Despite this and the near global consensus on the need to regulate AWS, the UK has not yet endorsed limitations on their use. The UK’s defence AI strategy and its associated policy statement, Ambitious, Safe, Responsible, acknowledged the line that should not be crossed regarding machines making combat decisions but lacked detail on where this line is drawn, raising ethical, legal and indeed moral concerns.

As we explored this complex landscape as a committee—and it was quite a journey for many of us—we found that, while the term AWS is frequently used, its definition is elusive. The inconsistency in how we define and understand AWS has significant implications for the development and governance of these technologies. However, the committee demonstrated that a working definition is possible, distinguishing between fully and partially autonomous systems. This is clearly still resisted by the Government, as their response has shown.

The current lack of definition allows for the assertion that the UK neither possesses nor intends to develop fully autonomous systems, but the deployment of autonomous systems raises questions about accountability, especially in relation to international humanitarian law. The Government emphasise the sufficiency of existing international humanitarian law while a human element in weapon deployment is retained. The Government have consistently stated that UK forces do not use systems that deploy lethal force without human involvement, and I welcome that.

Despite the UK’s reluctance to limit AWS, the UN and other states advocate for specific regulation. The UN Secretary-General, António Guterres, has called autonomous weapons with life-and-death decision-making powers “politically unacceptable, morally repugnant” and deserving of prohibition, yet an international agreement on limitation remains elusive.

In our view, the rapid development and deployment of AWS necessitates regulatory frameworks that address the myriad of challenges posed by these technologies. The relationship between our own military and the  private sector makes it even more important that we address the challenges posed by these technologies and ensure compliance with international law to maintain ethical standards and human oversight. I share the optimism of the noble Lord, Lord Holmes, that this is both possible and necessary.

Human rights organisations have urged the UK to lead in establishing new international law on autonomous weapon systems to address the current deadlock in conventional weapons conventions, and we should do so. There is a clear need for the UK to play an active role in shaping the nature of future military engagement.

A historic moment arrived last November with the UN’s first resolution on autonomous weapons, affirming the application of international law to these systems and setting the stage for further discussion at the UN General Assembly. The UK showed support for the UN resolution that begins consultations on these systems, which I very much welcome. The Government have committed also to explicitly ensure human control at all stages of an AWS’s life cycle. It is essential to have human control over the deployment of the system, to ensure both human moral agency and compliance with international humanitarian law.

However, the Government still have a number of questions to answer. Will they respond positively to the call by the UN Secretary-General and the International Committee of the Red Cross that a legally binding instrument be negotiated by states by 2026? How do the Government intend to engage at the Austrian Government’s conference “Humanity at the Crossroads”, which is taking place in Vienna at the end of this month? What is the Government’s assessment of the implications of the use of AI targeting systems under international humanitarian law? Can the Government clarify how new international law on AWS would be a threat to our defence interests? What factors are preventing the Government adopting a definition of AWS, as the noble Lord, Lord Lisvane, asked? What steps are being taken to ensure meaningful human involvement throughout the life cycle of AI-enabled military systems? Finally, will the Government continue discussions at the Convention on Certain Conventional Weapons, and continue to build a common understanding of autonomous weapon systems and elements of the constraints that should be placed on them?

 The committee rightly warns that time is short for us to tackle the issues surrounding AWS. I hope the Government will pay close and urgent attention to its recommendations.

---

by Tim Clement-Jones

Lord Holmes Private Members bill a "stake in the ground" says Lord C-J

Lord Holmes of Richmond recently introduced his Private Members Bill -The Artificial Intelligence (Regulation) Bill.

This may not go as far in regulating AI as many want to see but it is a good start. This what Lord Holmes says about it on his own website 

https://lordchrisholmes.com/artificial-intelligence-regulation-bill/

and this is what I said at its second reading recently

My Lords, I congratulate the noble Lord, Lord Holmes, on his inspiring introduction and on stimulating such an extraordinarily good and interesting debate.

The excellent House of Lords Library guide to the Bill warns us early on:

“The bill would represent a departure from the UK government’s current approach to the regulation of AI”.

Given the timidity of the Government’s pro-innovation AI White Paper and their response, I would have thought that was very much a “#StepInTheRightDirection”, as the noble Lord, Lord Holmes, might say.

There is clearly a fair wind around the House for the Bill, and I very much hope it progresses and we see the Government adopt it, although I am somewhat pessimistic about that. As we have heard in the debate, there are so many areas where AI is and can potentially be hugely beneficial. However, as many noble Lords have emphasised, it also carries risks, not just of the existential kind, which the Bletchley Park summit seemed to address, but others mentioned by noble Lords today, such as misinformation, disinformation, child sexual abuse, and so on, as well as the whole area of competition—the issue of the power and the asymmetry of these big tech AI systems and the danger of regulatory capture.

It is disappointing that, after a long gestation of national AI policy-making, which started so well back in 2017 with the Hall-Pesenti review, contributed to by our own House of Lords Artificial Intelligence Committee, the Government have ended up by producing a minimalist approach to AI regulation. I liked the phrase used by the noble Lord, Lord Empey, “lost momentum”, because it certainly feels like that after this period of time.

The UK’s National AI Strategy, a 10-year plan for UK investment in and support of AI, was published in September 2021 and accepted that in the UK we needed to prepare for artificial general intelligence. We needed to establish public trust and trustworthy AI, so often mentioned by noble Lords today. The Government had to set an example in their use of AI and to adopt international standards for AI development and use. So far, so good. Then, in the subsequent AI policy paper, AI Action Plan, published in 2022, the Government set out their emerging proposals for regulating AI, in which they committed to develop

“a pro-innovation national position on governing and regulating AI”,

to be set out in a subsequent governance White Paper. The Government proposed several early cross-sectoral and overarching principles that built on the OECD principles on artificial intelligence: ensuring safety, security, transparency, fairness, accountability and the ability to obtain redress.

Again, that is all good, but the subsequent AI governance White Paper in 2023 opted for a “context-specific approach” that distributes responsibility for embedding ethical principles into the regulation of AI systems across several UK sector regulators without giving them any new regulatory powers. I thought the analysis of this by the noble Lord, Lord Young, was interesting. There seemed to be no appreciation that there were gaps between regulators. That approach was confirmed this February in the response to the White Paper consultation.

Although there is an intention to set up a central body of some kind, there is no stated lead regulator, and the various regulators are expected to interpret and apply the principles in their individual sectors in the expectation that they will somehow join the dots between them. There is no recognition that the different forms of AI are technologies that need a comprehensive cross-sectoral approach to ensure that they are transparent,

explainable, accurate and free of bias, whether they are in an existing regulated or unregulated sector. As noble Lords have mentioned, discussing existential risk is one thing, but going on not to regulate is quite another.

Under the current Data Protection and Digital Information Bill, data subject rights regarding automated decision-making—in practice, by AI systems—are being watered down, while our creatives and the creative industries are up in arms about the lack of support from government in asserting their intellectual property rights in the face of the ingestion of their material by generative AI developers. It was a pleasure to hear what the noble Lord, Lord Freyberg, had to say on that.

For me, the cardinal rules are that business needs clarity, certainty and consistency in the regulatory system if it is to develop and adopt AI systems, and we need regulation to mitigate risk to ensure that we have public trust in AI technology. Regulation is not necessarily the enemy of innovation; it can be a stimulus. That is something that we need to take away from this discussion.

This is where the Bill of the noble Lord, Lord Holmes, is an important stake in the ground, as he has described. It provides for a central AI authority that has a duty of looking for gaps in regulation; it sets out extremely well out the safety and ethical principles to be followed; it provides for regulatory sandboxes, which we should not forget are an innovation invented in the UK; and it provides for AI responsible officers and for public engagement. Importantly, it builds in a duty of transparency regarding data and IP-protected material where they are used for training purposes, and for labelling AI-generated material, as the noble Baroness, Lady Stowell, and her committee have advocated. By itself, that would be a major step forward, so, as the noble Lord knows, we on these Benches wish the Bill very well, as do all those with an interest in protecting intellectual property, as we heard the other day at the round table that he convened.

However, in my view what is needed at the end of the day is the approach that the interim report of the Science, Innovation and Technology Committee recommended towards the end of last year in its inquiry into AI governance: a combination of risk-based cross-sectoral regulation and specific regulation in sectors such as financial services, applying to both developers and adopters, underpinned by common trustworthy standards of risk assessment, audit and monitoring. That should also provide recourse and redress, as the Ada Lovelace Institute, which has done so much work in the area, asserts.

That should include the private sector, where there is no effective regulator for the workplace, mentioned, and the public sector, where there is no central or local government compliance mechanism; no transparency yet in the form of a public register of use of automated decision-making, despite the promised adoption of the algorithmic recording standard; and no recognition by the Government that explicit legislation and/or regulation for intrusive

AI technologies used in the public sector, such as live facial recognition and other biometric capture, is needed. Then, of course, we need to meet the IP challenge. We need to introduce personality rights to protect our artists, writers and performers. We need the labelling of AI-generated material alongside the kinds of transparency duties contained in the noble Lord’s Bill.

Then there is another challenge, which is more international. We have world-beating AI researchers and developers. How can we ensure that, despite differing regulatory regimes—for instance, between ourselves and the EU or the US—developers are able to commercialise their products on a global basis and adopters can have the necessary confidence that the AI product meets ethical standards?

The answer, in my view, lies in international agreement on common standards such as those of risk and impact assessment, testing, audit, ethical design for AI systems, and consumer assurance, which incorporate what have become common internationally accepted AI ethics. Having a harmonised approach to standards would help provide the certainty that business needs to develop and invest in the UK more readily, irrespective of the level of obligation to adopt them in different jurisdictions and the necessary public trust. In this respect, the UK has the opportunity to play a much more positive role with the Alan Turing Institute’s AI Standards Hub and the British Standards Institution. The OECD.AI group of experts is heavily involved in a project to find common ground between the various standards.

We need a combination of proportionate but effective regulation in the UK and the development of international standards, so, in the words of the noble Lord, Lord Holmes, why are we not legislating? His Bill is a really good start; let us build on it.

---

by Tim Clement-Jones

New Digital Markets Bill Must Not be Watered Down

The Digital Markets Competition and Consumer Bill had its Second Reading in the House of Lords on the 5th December 2023 and its 3rd Reading on the 26th March 2024  This is an edited version of what I said on each occasion

Second Reading

I thank the Minister for what I thought was a comprehensive introduction that really set the scene for the Bill. As my noble friend said, we very much welcome the Bill, broadly. It is an overdue offspring of the Furman review and, along with so many noble Lords around the House, he gave very cogent reasons, given the dominance that big tech has and the inadequate powers that our competition regulators have had to tackle them. It is absolutely clear around the House that there is great appetite for improving the Bill. I have knocked around this House for a few years, and I have never heard such a measure of agreement at Second Reading.

We seem to have repeated ourselves, but repetition is good. I am sure that in the Minister’s notebook he just has a list saying “agree, agree, agree” as we have gone through the Bill. I very much hope that he will follow the example that both he and the noble Lord, Lord Parkinson, demonstrated on the then Online Safety Bill and will engage across and around the Chamber with all those intervening today, so that we really can improve the Bill.

It is not just size that matters: we must consider behaviour, dominance, market failure and market power. We need to hold on to that. We need new, flexible pro-competition powers and the ability to act ex ante and on an interim basis—those are crucial powers for the CMA. As we have heard from all round the House, the digital landscape, whether it is app stores, cloud services or more, is dominated by the power of certain big tech companies, particularly in AI, with massive expenditure on compute power, advanced semiconductors, large datasets and the scarce technology skills forming a major barrier to entry where the development of generative AI is concerned. We can already see the future coming towards us.

In that context, I very much welcome Ofcom’s decision to refer the hyperscalers in cloud services for an investigation by the CMA. The CMA and the DMU have the capability to deliver the Bill’s aims.the It must have the ability to implement the new legislative powers. Unlike some other commentators, we believe, as my noble friend said, that the CMA played a positively useful role in the Activision Blizzard-Microsoft merger. It is crucial that the CMA is independent of government. All around the House, there was comment about the new powers of the Secretary of State in terms of guidance. The accountability to Parliament will also be crucial, and that was again a theme that came forward. We heard about the Joint Committee proposals made by both the committee of the noble Baroness, Lady Stowell, and the Joint Committee on the Online Safety Bill.

We need to ensure that that scrutiny is there and, as the Communications and Digital Committee also said, that the DMU is well resourced and communicates its priorities, work programmes and decisions regularly to external stakeholders and Parliament.

The common theme across this debate—to mention individual noble Lords, I would have to mention almost every speaker—has been that the Bill must not be watered down. In many ways, that means going back to the original form of the Bill before it hit Report in the Commons. We certainly very much support that approach, whether it is to do with the merits approach to penalties, the explicit introduction of proportionality or the question of deleting the indispensability test in the countervailing benefits provisions. We believe that, quite apart from coming back on the amendments from Report, the Bill could be further strengthened in a number of respects.

In the light of the recent Open Markets Institute report, we should be asking whether we are going far enough in limiting the power of big tech. In particular, as regards the countervailing benefits exemption, as my noble friend said, using the argument of countervailing benefits—even if we went back to the definition from Report—must not be used by big tech as a major loophole to avoid regulatory action. It is clear that many noble Lords believe, especially in the light of those amendments, that the current countervailing benefits exemption provides SMS firms with too much room to evade conduct requirements.

The key thing that unites us is the fact that, even though we must act in consumers’ interests, this is not about short-term consumer welfare but longer-term consumers’ interests; a number of noble Lords from across the House have made that really important distinction.

We believe that there should be pre-notification if a platform intends to rely on this exemption. The scope of the exemption should also be significantly curtailed to prevent its abuse, in particular by providing an exhaustive list of the types of countervailing benefits that SMS firms are able to claim. We would go further in limiting the way in which the exemption operates.

On strategic market status, one of the main strengths of the Bill is its flexible approach. However, the current five-year period does not account for dynamic digital markets that will not have evidence of the position in the market in five years’ time. We believe that the Bill should be amended so that substantial and entrenched market power is mainly based on past data rather than a forward-looking assessment, and that the latter is restricted to a two-year assessment period. The consultation aspect of this was also raised; there should be much greater rights on the consultation of businesses that are not of strategic market status under the Bill.

A number of noble Lords recognised the need for speed. It is not just a question of making sure that the CMA has the necessary powers; it must be able to move quickly. We believe that the CMA should be given the legal power to secure injunctions under the High Court timetable, enabling it to stop anti-competitive activities in days. This would be in addition to the CMA’s current powers.

We have heard from across the House about the final offer mechanism affecting the news media. We believe that a straightforward levy on big tech platforms, redistributed to smaller journalism enterprises, would be a far more equitable approach. However we need to consider in the context of the Bill the adoption by the CMA of the equivalent to Ofcom’s duty in the Communications Act 2003

“to further the interests of citizens”,

so that it must consider the importance of an informed democracy and a plural media when considering its remedies.

The Bill needs to make it clear that platforms need to pay properly and fairly for content, on benchmarked terms and with reference to value for end-users. Indeed, we believe that they must seek permission for the content that they use. As we heard from a number of noble Lords, that is becoming particularly important as regards the large language models currently being developed.

We also believe it is crucial that smaller publishers are not frozen out or left with small change while the highly profitable large publishers scoop the pool. I hope that we will deal with the Daily Telegraph ownership question and the mergers regime in the Enterprise Act as we go forward into Committee, to make sure that the accumulation of social media platforms is assessed beyond the purely economic perspective. The Enterprise Act powers should be updated to allow the Secretary of State to issue a public interest notice seeking Ofcom’s advice on digital media mergers, as well as newspapers, and at the lower thresholds proposed by this Bill.

There were a number of questions related to leveraging. We want to make sure that we have the right approach to that. The Bill does not seem to be drafted properly in allowing the CMA to prevent SMS firms using their dominance in designated activities to increase their power in non-designated activities. We want to kick the tyres on that.

Of course, there are a great many consumer protection issues here, which a number of noble Lords raised. They include fake reviews and the need for collective action. It is important that we allow collective action not just on competition rights but further, through consumer claims, data abuse claims and so on. We should cap the costs for claimants in the Competition Appeal Tribunal.These issues also include misleading packaging.

Nearly every speaker mentioned subscriptions. I do not think that I need to point out to the Minister the sheer unanimity on this issue. We need to get this right because there is clearly support across the House for making sure that we get the provisions right while protecting the income of charities.

There is a whole host of other issues that we will no doubt discuss in Committee: mid-contract price rises, drip pricing, ticket touting, online scams and reforming ADR. We want to see this Bill and the new competition and consumer powers make a real difference. However, we believe that we can do this only with some key changes being made to the Bill, which are clearly common ground between us all, as we have debated the Bill today. We look forward to the Committee proceedings next year—I can say that now—which will, I hope, be very productive, if both Ministers will it so.

Third Reading

I reiterate the welcome that we on these Benches gave to the Bill at Second Reading. We believe it is vital to tackle the dominance of big tech and to enhance the powers of our competition regulators to tackle it, in particular through the new flexible pro-competition powers and the ability to act ex ante and on an interim basis.

We were of the view, and still are, that the Bill needs strengthening in a number of respects. We have been particularly concerned about the countervailing benefits exemption under Clause 29. This must not be used by big tech as a major loophole to avoid regulatory action. A number of other aspects were inserted into the Bill on Report in the Commons about appeals standards and proportionality. During the passage of the Bill, we added a fourth amendment to ensure that the Secretary of State’s power to approve CMA guidance will not unduly delay the regime coming into effect.

As the noble Baroness, Lady Stowell, said, we are already seeing big tech take an aggressive approach to the EU Digital Markets Act. We therefore believe the Bill needs to be more robust in this respect. In this light, it is essential to retain the four key amendments passed on Report and that they are not reversed through ping-pong when the Bill returns to the Commons.

I thank both Ministers and the Bill team. They have shown great flexibility in a number of other areas, such as online trading standards powers, fake reviews, drip pricing, litigation, funding, cooling-off periods, subscriptions and, above all, press ownership, as we have seen today. They have been assiduous in their correspondence throughout the passage of the Bill, and I thank them very much for that, but in the crucial area of digital markets we have seen no signs of movement. This is regrettable and gives the impression that the Government are unwilling to move because of pressure from big tech. If the Government want to dispel that impression, they should agree with these amendments, which passed with such strong cross-party support on Report.

In closing, I thank a number of outside organisations that have been so helpful during the passage of the Bill—in particular, the Coalition for App Fairness, the Public Interest News Foundation, Which?, Preiskel & Co, Foxglove, the Open Markets Institute and the News Media Association. I also thank Sarah Pughe and Mohamed-Ali Souidi in our own Whips’ Office.

Given the coalition of interest that has been steadily building across the House during the debates on the Online Safety Bill and now this Bill, I thank all noble Lords on other Benches who have made common cause and, consequently, had such a positive impact on the passage of this Bill. As with the Online Safety Act, this has been a real collaborative effort in a very complex area.

---

by Tim Clement-Jones

Living with the Algorithm now published!

Living with the Algorithm

Servant or Master?

AI Governance and Policy for the Future

Tim Clement-Jones

Published March 2024

Paperback with flaps, £14.99 ISBN: 9781911397922

A comprehensive breakdown of the AI risks and how to address them.

The rapid proliferation of AI brings with it a potentially massive shift in how society interacts with the digital world. New opportunities and challenges are emerging in unprecedented fashion and speed. AI however, comes with its own risks, including the potential for bias and discrimination, reputational harm, and the potential for widescale redundancy of millions of jobs. Many prominent technologists have voiced their concern at the existential risks to humanity that AI pose. So how do we ensure that AI remains our servant and not our master?

The purpose in this book is to identify and address these key risks looking at current approaches to regulation and governance of AI internationally in both the public and private sector, how we meet and mitigate these challenges, avoid inadequate or ill considered regulatory approaches, and protect ourselves from the unforeseen consequences that could flow from unregulated AI development and adoption.

---

by Tim Clement-Jones

AI Regulation-It's All about Standards

I recently gave a talk to the Engineers' Association of my Alma Mater, Trinity College Cambridge. This is what I said

Video here: https://www.youtube.com/watch?v=2Wnf97_Zu5E

 

You may ask how and why I have been sucked into the world of AI. Well, 8 years ago I set up a cross-party group in the UK parliament because i thought parliamentarians didn’t know enough about it and then- based on, in the Kingdom of the Blind the fact that one eyed man is king- I was asked to chair the House of Lords Special Enquiry Select Committee on AI with the remit  “to consider the economic, ethical and social implications of advances in artificial intelligence. This produced its report “AI in the UK: Ready Willing and Able?”  in April 2018.  It took a close look at government policy towards AI and its ambitions in the very early days of its policy-making when the UK was definitely ahead of the pack.

Since then I have been lucky enough to have acted as an adviser to the Council of Europe’s working party on AI (CAHAI) the One AI Group of OECD AI Experts and helped establish the OECD Global Parliamentary Network on AI which helps in tracking developments in AI and the policy responses to it, which come thick and fast.

Artificial Intelligence presents opportunities in a whole variety of sectors. I am an enthusiast for the technology -the opportunities for AI are incredibly varied-and I recently wote an upbeat piece on the way that AI is already transforming healthcare.

Many people find it unhelpful to have such a variety of different types of machine learning, algorithms, neural networks,  or deep learning, labelled AI. But the expression has been been used since John McCarthy invented it in 1956 and I think we are stuck with it!

Nowadays barely a day goes by without some reference to AI in the news media-particularly some aspect of Large Language Models in the news. We saw  the excitement over ChatGPT from Open AI  and AI text to image applications such as DALL E and now we have GPT 4 from OpenAI, LlaMa from Meta, Claude from Anthropic, Gemini from Google, Stability Diffusion from Stability AI, Co-pilot from Microsoft, Cohere, Midjourney, -a whole eco system of LLM’s of various kinds.

Increasingly the benefits are not just seen around increasing efficiency, speed etc in terms of analysis, pattern detection and ability to predict but now, with generative AI  much  more about what creatively AI can add to human endeavour , how it can augment what we do.

But things can go wrong. This isn’t just any old technology.The degree of autonomy, its very versatility, its ability to create convincing fakes, lack of human intervention, the Black box nature of some systems makes it different from other tech. The challenge is to ensure that AI is our servant not our master especially before the advent of AGI.

Failure to tackle issues such as bias/discrimination, deepfakery and disinformation, and lack of transparency will lead to a lack of public/consumer trust, reputational damage and inability to deploy new technology. Public trust and trustworthy AI is fundamental to continued advances in technology.

It is clear that AI even in its narrow form will and should have a profound impact on and implications for corporate governance in terms of the need to ensure responsible or ethical AI adoption.The AI Safety Conference at Bletchley Park-where incidentally my parents met- ended with a voluntary corporate pledge.

This means a more value driven approach to the adoption of new technology needs to be taken. Engagement from boards through governance right through to policy implementation is crucial. This is not purely a matter that can be delegated to the CTO or CIO.

It means in particular  assessing the ethics of adoption of AI  and the ethical standards to be applied corporately : It may involve the establishment of an ethics advisory committee.It certainly involves clear Board accountability..

We have a pretty good common set of principles -OECD or G20- which are generally regarded as the gold standard which can be adopted which can help us ensure

• Quality of training data
• Freedom from Bias
• The impact on Individual civil and human rights
• Accuracy and robustness
• Transparency and Explainability which of course include the need for open communication where these technologies are deployed.

And now we have the G7 principles for Organizations Developing Advanced AI systems to back those up.

Generally in business and in the tech research and development world I think there is an appetite for adoption of common  standards  which incorporate  ethical principles such as for

• Risk management
• Impact assessment
• Testing
• AI audit
• Continuous Monitoring

And I am optimistic that common standards can be achieved internationally in all these areas. The OECD  Internationally is doing a great deal to scope the opportunity and enable convergence. Our own  AI Standards Hub run by the Alan Turing institute is heavily involved. As is NIST in the US and the EU’s CEN-CENELEC standards bodies too.

Agreement on the actual regulation of AI in terms of what elements of governance and application of standards should be mandatory or obligatory, however, is much more difficult.

In the UK there are already some elements of a legal framework in place. Even without specific legislation, AI deployment in the UK will interface with existing legislation and regulation in particular relating to

• Personal data under UK GDPR
• Discrimination and unfair treatment under the Human Rights Act and Equality Act
• Product safety and public safety legislation
• And various sector-specific regulatory regimes requiring oversight and control by persons undertaking regulated functions, the FCA for financial services, Ofcom in the future for social media for example.

But when it comes to legislation and regulation that is specific to AI such over transparency and explanation and liability that’s where some of the difficulties and disagreements start emerging especially given the UK’s approach in its recent White Paper and the government’s response to the consultation.

Rather than regulating in the face of clear current evidence of the risk of the many uses and forms of AI it says it’s all too early to think about tackling the clear risks in front of us.  More research is needed. We are expected to wait until we have complete understanding and experience of the risks involved. Effectively in my view we are being treated as guinea pigs to see what happens whilst the government talks about the existential risks of AGI instead.

And we shouldn’t just focus on existential long term risk or indeed risk from Frontier AI, predictive AI is important too in terms of automated decision making, risk of bias and lack of transparency.

The government says it wishes its regulation to be innovation friendly and context specific but sticking to their piecemeal context specific approach  the government are not suggesting immediate regulation nor any new powers for sector regulators

But regulation is not necessarily the enemy of innovation, it can in fact be the stimulus and be the key to gaining and retaining public trust around digital technology and its adoption so we can realise the benefits and minimise the risks.

The recent response to the AI White paper  has demonstrated the gulf between the government’s rhetoric  about being world leading in safe AI.

In my view we need a broad definition of AI  and early risk based overarching horizontal legislation across the sectors ensuring that there is  conformity with standards for a proper risk management framework and impact assessment when AI systems are developed and aded.

Depending on the extent of the risk and impact assessed, further regulatory requirements would arise. When the system is assessed as high risk there would be additional requirements to adopt standards of testing, transparency and independent audit.

What else is on my wish list? As regards its use of AI and automated decision making systems the government needs to firmly implant its the Algorithmic Transparency Recording Standard alongside risk assessment together with a public register of AI systems in use in government.

It also needs need to beef up the Data Protection Bill in terms of rights of data subjects relative to Automated Decision Making rather than water them down and retain and extend the Data Protection Impact Assessment and DPO for use in AI regulation.

I also hope the Gov will take strong note of the House of Lords report on the use of copyrighted works by LLM’s. The government has adopted its usual approach of relying on a voluntary approach. But it is clear that this is simply is not going to work.  It needs to  act decisively to make sure that these works are not ingested into training LLM’s without any return to rightsholders.

Luckily others such as the EU-and even the US- contrary to many forecasts  are grasping the nettle. The EU’s AI Act is an attempt to grapple with the here and now risks in a constructive way and even the US where the White House Executive Order and Congressional bi-partisan proposals show a much more proactive approach.

But the more we diverge when it comes to regulation from other jurisdictions the more difficult it gets for UK  developers and those who want to develop AI systems internationally.

International harmonization, interoperability or convergence, call it what you like, is in my view essential if we are to see developers able to commercialize their products on a global basis, assured that they are adhering to common ethical standards of regulation.This means working with the BSI ISO  OECD and others towards convergence of international standards.There are already several existing standards such as ISO 42001 and 42006 and NIST’s RMF  which can form the basis for this

What I have suggested I believe would help provide the certainty, convergence and consistency we need to develop and invest in responsible AI innovation in the UK more readily. That in my view is the way to get real traction!

 

 

 

 

 

 

---

by Tim Clement-Jones

Great prospects but ….. The potential and challenges for AI in healthcare.

I recently wrote a piece on AI in healthcare for the Journal of the Apothecaries Livery Company, which has among its membership a great many doctors and  health specialists. This is what I said.

In our House of Lords AI Select Committee report “AI in the UK: Ready Willing and Able?” back in 2018, reflecting the views of many of our witnesses about its likely profound impact, we devoted a full chapter to the potential for AI in its application to healthcare. Not long afterwards the Royal College of Physicians itself made several far-sighted recommendations relating to the incentives, scrutiny and regulation needed for AI development and adoption.2

At that time it was already clear that medical imaging and supporting administrative roles were key areas for adoption. Fast forward 5 years to the current enquiry- “Future Cancer-exploring innovations in cancer diagnosis and treatment”- by the House of Commons Health and Social Care Select Committee and the application of different forms of AI is very much already here and now in the NHS.

It is evident that this is a highly versatile technology. The Committee heard in particular from GRAIL Bio UK about its Galleri AI application which has the ability to detect a genetic signal that is shared by over 50 different types of incipient cancer, particularly more aggressive tumours.3 Over the past year, we have heard of other major breakthroughs —tripling stroke recovery rates with Brainomix4,  mental health support through the conversational AI application Wysa5  and  Eye2Gene a decision support system with genetic diagnosis of inherited retinal disease, and applications for remotely managing conditions at home.6 Mendelian has developed an AI tool, piloting in the NHS, to interrogate large volumes of electronic patient records to find people with symptoms that could be indicative of a rare disease.6A

We have seen the introduction of Frontier software designed to ease bed blocking by improving the patient discharge process.7 And just a few weeks ago we heard of how, using AI, Lausanne researchers have created a digital bridge from the brain and implanted spine electrodes which allow patients with spinal injuries to regain coordination and movement.8 It is also clear despite the recent fate of Babylon Health 9 that consumer AI-enabled health apps and devices can have a strong future too in terms of health monitoring and self-care. We now have large language models such as Med-PaLM developed by Google research which are designed to designed to provide high quality answers to medical questions. 9A

We are seeing the promise of the use of AI in training surgeons for more precise keyhole brain surgery.

Now it seems just around the corner could be foundation models for generalist medical artificial intelligence which are trained on massive, diverse datasets and will be able to perform a very wide range of tasks based on a broad range of data such as images, electronic health records, laboratory results, genomics, graphs or medical text and to provide communicate directly with patients. 10

We even have the promise of Smartphones being able to detect the onset of dementia 10A

Encouragingly—whatever one’s view of the current condition more broadly of the Health Service—successive Secretaries of State for Health have been aware of the potential and have responded by investing. Over the past few years, the Department of Health and the NHS have set up a number of mechanisms and structures designed to exploit and incentivize the development of AI technologies.  The pandemic, whilst diminishing treatment capacity in many areas, has also demonstrated that the NHS is capable of innovation and agile adoption of new technology.

Its performance has yet to be evaluated, but through the NHS AI Lab set up in 2019 11, designed to accelerate the safe, ethical and effective adoption of AI in health and social care, with its AI in Health and Care Awards over the past few years, some £123 million has been invested in 86 different AI technologies, including stroke diagnosis, cancer screening, cardiovascular monitoring, mental health, osteoporosis detection, early warning and clinician support tools for obstetrics applications for remotely managing conditions at home. 11

This June the Government announced a new £21 million AI Diagnostic Fund to accelerate deployment of the most promising AI decision support tools in all 5 stroke networks covering 37 hospitals by the end of 2023, given results showing more patients being treated, more efficient and faster pathways to treatment and better patient outcomes.12

In the wider life sciences research field—which our original Lords enquiry dwelt on less—there has been ground-breaking research work by DeepMind in its Alphafold discovery of protein structures13 and Insilico Medicine’s use of generative AI for drug discovery in the field of idiopathic pulmonary treatment which it claims saved 2 years in the route to market.14 GSK has developed a large language model Cerebras to analyse the data from genetic databases which it means can take a more predictive approach in drug discovery .15

Despite these developments as many clinicians and researchers have emphasized—not least recently to the Health and Social Care Committee—the rate of adoption of AI is still too slow.

There are a variety of systemic issues in the NHS which still need to be overcome. We lag far behind health systems such as Israel’s, pioneers of the virtual hospital16, and Estonia.17

The recent NHS Long Term Workforce Plan 18 rightly acknowledges that AI in augmenting human clinicians will be able to greatly relieve pressures on our Health Service, but one of the principal barriers is a lack of skills to exploit and procure the new technologies, especially in working alongside AI systems.  As recognized by the Plan, the introduction of AI into potentially so many healthcare settings has huge implications for healthcare professions especially in terms of the need to strike a balance between AI assistance/human augmentation and substitution with all its implications for future deskilling.

Addressing this in itself is a digital challenge that the NHS Digital Academy—a virtual academy set up in 2018 19- was designed to solve. These issues were tackled by the Topol Review “Preparing the Healthcare Workforce to Deliver the Digital Future”, instituted by Jeremy Hunt when Health Secretary, and reported in February 2019 20. Above all, it concluded that NHS organisations will need to develop an expansive learning environment and flexible ways of working that encourage a culture of innovation and learning.  Similarly the review by Sir Paul Nurse of the research, development and innovation organisational landscape 21 highlighted a skills and training gap across these different areas and siloed working.  The current reality on the ground however is that the adoption of AI and digital technology still does not feature in workforce planning and is not reflected in medical training which is very traditional in its approach.

More specific AI-related skills are being delivered through the AI Centre for Value-based Healthcare. This is led by King’s College London and Guy's and Thomas’ NHS Foundation Trust, alongside a number of  NHS Trusts, Universities and UK and multinational industry partners.22  Funded by grants from UK Research and Innovation (UKRI) and the Department of Health and Social Care (DHSC) and the Office of Life Sciences their Fellowship in Clinical AI is a year-long programme integrated part-time alongside the clinical training of doctors and dentists approaching consultancy. This was first piloted in London and the South East in 2022 and is now being rolled out more widely but although it has been a catalyst for collaboration it has yet to make an impact at any scale. The fact remains that outside the major health centres there is still insufficient financial or administrative support for innovation.

Set against these ambitions many NHS clinicians complain that the IT in the health service is simply not fit for purpose. This particularly applies in areas such as breast cancer screening.

One of the key areas where developers and adopters can also find frustrations is in the multiplicity of regulators and regulatory processes. Reports such as the review by Professor Dame Angela McLean the Government's Chief Scientific Adviser on the Pro Innovation Regulation of Technologies-Life Sciences 23 identified blockages in the regulatory process for the “innovation pathway”. At the same time however we need to be mindful of the patient safety findings— shocking it must be said—of the Cumberledge Review—the independent Medicines and Medical Device Safety Review.24

To streamline the AI regulatory process the NHS AI Lab has set up the AI and Digital Regulations Service (formerly the multi-agency advisory service) which is a collaboration between 4 regulators: The National Institute for Health and Care Excellence, The Medicines and Healthcare Products Regulatory Agency (the MHRA), The Health Research Authority and The Care Quality Commission.25

The MHRA itself through its Software and AI as a Medical Device Change Programme Roadmap is a good example of how an individual health regulator is gearing up for the AI regulatory future, the intention being to produce guidance in a variety of areas, including bringing clarity to distinctions such as software as a medical device versus wellbeing and lifestyle software products and versus medicines and companion diagnostics.26

Specific regulation for AI systems is another factor that healthcare AI developers and adopters will need to factor in going forward. There are no current specific proposals in the UK but the EU’s AI Act will set up a regulatory regime which will apply to high-risk AI systems and applications which are made available within the EU, used within the EU or whose output affects people in the EU.

Whatever regulatory regime applies the higher the impact on the patient an AI application has, the stronger the need for clear and continuing ethical governance to ensure trust in its use, including preventing potential bias, ensuring explainability, accuracy, privacy, cybersecurity, and reliability, and determining how much human oversight should be maintained. This becomes of even greater importance in the long term if AI systems in healthcare become more autonomous.27

In particular AI in healthcare will not be successfully deployed unless the public is confident that its health data will be used in an ethical manner, is of high quality, assigned its true value, and used for the greater benefit of UK healthcare. The Ada Lovelace Institute in conjunction with the NHS AI Lab, has developed an algorithmic impact assessment for data access in a healthcare environment which demonstrates the crucial risk and ethical factors that need to be considered in the context of  AI development and adoption .28

For consumer self-care products explainability statements, of the kind  developed by Best Practice AI  for Heathily’s AI smart symptom checker, which provides a non-technical explanation of the app to its customers, regulators and the wider public will need to become the norm.29 We have also just recently seen the Introduction of British Standard 30440 designed as a validation framework for the use of AI in healthcare 30, All these are steps in the right direction but there needs to be regulatory incentivisation of adoption and compliance with these standards if  the technology is to be trustworthy and patients to be safe.

The adoption of and alignment with global standards is a governance requirement of growing relevance too. The WHO in 2021 produced important guidance on the Governance of Artificial Intelligence for Health31

Issues relating to patient data access, which is so crucial for research and training of AI systems, in terms of public trust, procedures for sharing and access have long bedevilled progress on AI development. This was recognized in the Data Saves Lives strategy of 2022 32  and led to the Goldacre Review which reported earlier this year. 33

As a result, greater interoperability and a Federated Data Platform comprised of Secure Research Environments is now emerging with greater clarity on data governance requirements which will allow researchers to navigate access to data more easily whilst safeguarding patient confidentiality.

All this links to barriers to the ability of researchers and developers to conduct clinical trials which has been recognized as a major impediment to innovation. The UK has fallen behind in global research rankings as a result. The O’Shaughnessy review on clinical trials which reported earlier this year made a number of key recommendations 34

-A national participatory process on patient consent to examine how to achieve greater data usage for research in a way that commands public trust. Much greater public communication and engagement on this has long been called for by the National Data Guardian.

-Urgent publication of guidance for NHS bodies on engaging in research with industry. This is particularly welcome. At the time of our Lords report, the Royal Free was criticized by the Information Commissioner for its arrangement with DeepMind which developed its Streams app to diagnose acute kidney injury, as being in breach of data protection law,35 Given subsequent questionable commercial relationships which have been entered into by NHS bodies, a standard protocol for the use of NHS patient data for commercial purposes, ensuring benefit flows back into the health service has long been needed and is only just now emerging.

-Above all, it recommended a much-needed national directory of clinical trials to give much greater visibility to national trials activity for the benefit of patients, clinicians, researchers and potential trial sponsors.

Following the Health and Care Act of 2022, the impact of the recent reorganisation and merger of NHS X and NHS Digital into the Transformation Directorate of  NHS England, the former of which was specifically designed when set up in 2019  to speed up innovation and technology adoption in the NHS, is yet to be seen, but clearly, there is an ambition for the new structure to be more effective in driving innovation.

The pace of drug discovery through AI has undoubtedly quickened over recent years but there is a risky and rocky road in the AI healthcare investment environment.  Adopting AI techniques for drug discovery does not necessarily shortcut the uncertainties. The experience of drug discovery investor Benevolent AI is a case in point which recently announced that it would need to shed up to 180 staff out of 360.36

Pharma companies are adamant too that in the UK  the NHS branded drug pricing system is a disincentive to drug development although it remains to be seen what the newly negotiated voluntary and statutory agreements will deliver.

To conclude, expectations of AI in healthcare are high but where is the next real frontier and where should we be focusing our research and development efforts for maximum impact? Where are the gaps in adoption?

It is still the case that much unrealized AI potential lies in some of the non-clinical healthcare aspects such as workforce planning and mapping demand to future needs. This needs to be allied with day to day clinical AI predictive tools for patient care which can link data sets and combine analysis from imaging and patient records.

In my view too, of even greater significance than improvements in diagnosis and treatment, is a new emphasis on a preventative philosophy through the application of predictive AI systems to genetic data. As a result, long term risks can be identified and appropriate action taken to inform patients and clinicians about the likelihood of their getting a particular disease including common cancer types.

Our Future Health is a new such project working in association with the NHS. 37 The plan (probably overambitious and not without controversy in terms of its intention to link into government non health data) is to collect the vital health statistics of 5 million adult volunteers.

With positive results from this kind of genetic work, however, AI in primary care could come into its own, capturing the potential for illness and disease at a much earlier stage. This is where I believe that ultimately the greatest potential arises for impact on our national health and the opportunity for greater equity in life expectancy across our population lies. Alongside this, however, the greatest care needs to be taken in retaining public trust about personal data access and use and the ethics of AI systems used.

Footnotes

1.AI in the UK : Ready Willing and Able? House of Lords AI Select Committee 2018 Artificial Intelligence Committee - Summary

2. Royal College of Physicians, Artificial intelligence in healthcare Report of a working party, https://www.rcplondon.ac.uk/projects/outputs/artificial-intelligence-ai-health

3. House of Commons Health and Social Care Committee, Future cancer - Committees

4. Brainomix’s e-Stroke Software Triples Stroke Recovery Rates,https://www.brainomix.com/news/oahsn-interim-report/

5. Evidence-based Conversational AI for Mental Health Wysa

6.Eye2Gene wins Artificial Intelligence in Health and Care Award | UCL Institute of Ophthalmology - UCL – University College London

6A https://www.newstatesman.com/spotlight/healthcare/innovation/2023/10/ai-diagnosis-technology-artificial-intelligence-healthcare

7.AI cure for bed blocking can predict hospital stay

8.Clinical trial evaluates implanted tech that wirelessly stimulates spinal cord to restore movement after paralysis, Walking naturally after spinal cord injury using a brain–spine interface

9.Babylon the future of the NHS, goes into administration

9A https://sites.research.google/med-palm/

10. Foundation models for generalist medical artificial intelligence | Nature

10 A. https://www.thetimes.co.uk/article/ai-could-detect-dementia-long-before-doctors-claims-oxford-professor-slbdz70v3#:~:text=Michael%20Wooldridge%2C%20a%20professor%20of,possible%20sign%20of%20the%20condition.

11..The Artificial Intelligence in Health and Care Award - NHS AI Lab programmes

https://transform.england.nhs.uk/ai-lab/ai-lab-programmes/ai-health-and-care-award/ai-health-and-care-award-winners/

12.NHS invests £21 million to expand life-saving stroke care app, https://www.gov.uk/government/news/21-million-to-roll-out-artificial-intelligence-across-the-nhs

13.DeepMind, AlphaFold: a solution to a 50-year-old grand challenge in biology, 2020. AlphaFold: a solution to a 50-year-old grand challenge in biology

14.From Start to Phase 1 in 30 Months | Insilico Medicine

15.GlaxoSmithKline and Cerebras are Advancing the State of the Art in AI for Drug Discoveryl

16.Israeli virtual hospital is caring for Ukrainian refugees - ISRAEL21c

17. Estonia embraces new AI-based services in healthcare.

18. NHS Long Term Workforce Plan

19. NHS Digital Academy

20. The Topol Review: Preparing the Healthcare Workforce to Deliver the Digital Future

21.The Nurse Review:  Research, development and innovation (RDI) organisational landscape: an independent review - GOV.UK

22. The AI Centre for Value Based Healthcare

23 Pro-innovation Regulation of Technologies Review Life Sciences - GOV.UK

24 Department of Health and Social Care, First Do No Harm – The report of the Independent Medicines and Medical Devices Safety Review, 2020. The report of the IMMDSReview

25. AI and digital regulations service - AI Regulation - NHS Transformation Directorate

26. Software and AI as a Medical Device Change Programme - Roadmap - GOV.UK

27. AI Act: a step closer to the first rules on Artificial Intelligence | News | European Parliament

28. Algorithmic impact assessment in healthcare | Ada Lovelace Institute

29. Best Practice AI: Developing an explainability statement for an AI-enabled medical symptom checker - GOV.UK

30. Validation framework for the use of AI in healthcare: overview of the new British standard BS30440 | BMJ Health & Care Informatics

31. Ethics and governance of artificial intelligence for health

32. Data saves lives: reshaping health and social care with data - GOV.UK

33. Goldacre Review

34. Commercial clinical trials in the UK: the Lord O’Shaughnessy review - final report - GOV.UK

35 .Royal Free breached UK data law in 1.6m patient deal with Google's DeepMind.

36. BenevolentAI cuts half of its staff after drug trial flop

37.Our Future Health

 

 

 

 

 

 

 

 

11

 

 

 

 

---

by Tim Clement-Jones