Google DeepMind's first deal with the NHS was illegal, UK data regulator rules

The UK's data protection watchdog has ruled that a deal between DeepMind and an NHS trust "failed to comply with data protection law."

The Information Commissioner's Office (ICO) announced its verdict on the controversial data sharing-agreement between DeepMind and the Royal Free London NHS Trust on Monday after a year long investigation.

The agreement — quietly signed in September 2015 and revealed in full by New Scientist in April 2016 — gave the Google-owned artificial intelligence (AI) lab access to 1.6 million NHS patient records across three North London hospitals without patient's prior knowledge.

The deal (replaced with a new one last November) was signed to help DeepMind test and develop a kidney monitoring app called Streams, which sends an alert to a clinician's smartphone if a patient's condition deteriorates. It also allows clinicians to view a patient's medical records and see where patients are being looked after. It doesn't use any of the AI that DeepMind is known for. Through the agreement, DeepMind was able to see whether people are HIV-positive as well as details of drug overdoses and abortions. DeepMind insists that it has never shared patient data with parent company Google.

DeepMind and the Royal Free tried to justify the data-sharing deal by saying that "implied consent" was assumed because the Streams app was delivering "direct care" to patients.

The Streams app.
DeepMind
But the ICO, which launched its investigation last May after receiving at least one complaint from the public, said it found several shortcomings in how the data was handled, including that patients were not adequately informed that their data would be used.

While the ICO found the deal to be illegal, it has no plans to punish the Royal Free or DeepMind. The regulator has the power to give out fines of up to £500,000 and last October it fined UK telecoms firm TalkTalk £400,000 for security failings that allowed a cyber attacker to access customer data "with ease".

Elizabeth Denham, Information Commissioner, said in a statement: "There's no doubt the huge potential that creative use of data could have on patient care and clinical improvements, but the price of innovation does not need to be the erosion of fundamental privacy rights.

"Our investigation found a number of shortcomings in the way patient records were shared for this trial. Patients would not have reasonably expected their information to have been used in this way, and the Trust could and should have been far more transparent with patients as to what was happening.

"We've asked the Trust to commit to making changes that will address those shortcomings, and their co-operation is welcome. The Data Protection Act is not a barrier to innovation, but it does need to be considered wherever people's data is being used."

ICO says there are lessons to be learned

The ICO — who published a blog titled "Four lessons NHS Trusts can learn from the Royal Free case" to coincide with her ruling — has ordered the Royal Free to establish a proper legal basis for the deal and to set out how it will comply with its duty of confidence to patients in any future trial involving personal data. The trust must also commission an audit of the trial and share the results with the ICO.

Following the ruling, the Royal Free said in a statement on its website: "We accept the ICO's findings and have already made good progress to address the areas where they have concerns."

DeepMind said in a blog post on its website: "We welcome the ICO's thoughtful resolution of this case, which we hope will guarantee the ongoing safe and legal handling of patient data for Streams.

"Although today's findings are about the Royal Free, we need to reflect on our own actions too. In our determination to achieve quick impact when this work started in 2015, we underestimated the complexity of the NHS and of the rules around patient data."

We underestimated the complexity of the NHS and of the rules around patient data.

DeepMind also acknowledged that it underestimated the potential fears about a well-known tech company (Google) working in health.

"We were almost exclusively focused on building tools that nurses and doctors wanted, and thought of our work as technology for clinicians rather than something that needed to be accountable to and shaped by patients, the public and the NHS as a whole," the company added. "We got that wrong, and we need to do better."

The ICO was advised by the National Data Guardian (NDG) Dame Fiona Caldicott, the UK health data regulator, during its investigation.

"It is my view," Caldicott wrote in the letter dated 20 February, "that the purpose for the transfer of 1.6 million identifiable patient records to Google DeepMind was for the testing of the Streams application, and not for the provision of direct care to patients."

"Given that Streams was going through testing and therefore could not be relied up for patient care, any role the application might have played in supporting the provision of direct care would have been limited and secondary to the purpose of the data transfer."

Streams started out as an app that could monitor patients with acute kidney injury (AKI) but it can now be used for a range of medical tasks.
Google DeepMind

The deal has attracted criticism from academics, lawyers, and privacy campaigners.

Review panel set up to scrutinise DeepMind's work with NHS

DeepMind set up its own independent review panel last year to scrutinise the work it is doing with the NHS.

The panel is reviewing DeepMind's data sharing agreements, its privacy and security measures, and its product roadmaps. It is due to brief journalists about the findings of the report at the Science Media Centre in London on Tuesday, before releasing it on Wednesday.

"Working in healthcare requires regular and independent oversight," DeepMind writes on its website. "We have asked a number of respected public figures to act in the public interest as unpaid Independent Reviewers of DeepMind Health."

Mustafa Suleyman, DeepMind cofounder and head of DeepMind Health.
DeepMind
DeepMind cofounder Mustafa Suleyman defended the data-sharing agreement last year, saying: "As Googlers, we have the very best privacy and secure infrastructure for managing the most sensitive data in the world."

While these deals also give DeepMind access to patient records, the purposes for which DeepMind can use the data are far better constrained than in first deal. The amount of data being shared is also more proportional.