It’s the data drought that’s killing the NHS Academic development

There were many reasons I chose to jump the NHS ship and develop software in financial tech. One big motivator was that London is an international hub for financial tech as most of the financial transactions go through London and New York. I knew that I would be learning cutting edge tech and processes. Another big motivator was the lack of support in the NHS from senior clinicians and IT. In the fast-paced field of artificial intelligence and machine learning, I could not afford to hinder my development wasting time fighting for data, and convincing seniors who can’t even work out why you cannot add probabilities together, that a machine learning algorithm would give more insights, and that a multidimensional approach to data analysis is far better than a series of one-dimensional statistics. Doing a physics degree had created a void between me and the seniors who made decisions.

I’m not the only one. Other doctors and nurses are also educating themselves in advanced math, learning how to code, and reading around the subject of machine learning. I meet up with a number of them and I few I know of are now doing PhDs and have research positions. Should I have not made the jump? Was I wrong not to stick it out?

At first, when I heard these projects and positions I thought maybe. But after meeting with a number of them from universities like Cambridge, UCL, Oxford, and Imperial I realized that I had made the right call. Whilst these people were smart and motivated, the NHS environment had not fostered cutting-edge skills. They spent most of their time trying to convince clueless senior clinicians to even try a project. Obtaining data was a nightmare for them and took ages, even if it was anonymised, and they didn’t have the support of seasoned software engineers who did 2-week sprints in order to build a software platform. Collaborative coding with Git in terms of making branches from the main project, testing their development, and then merging it back into the main project was just not happening in their labs. And pre-packaged modules for data analysis were being used left right and center because they didn’t have the time to learn the math. Object orientated code structure was non-existent and their scripts were just long streams of unorganized code. Could they achieve this? Certainly, they could, however, the data cleaning component for their projects are such a big part, refining such skills was not a luxury they could afford.

On the flip side, I am able to obtain clean data within an hour easily. There is a team of software engineers ranging from front end to back end and even a guy who just does SQL and data handling in the office. Obtaining information and advice from them takes roughly about 5 minutes because all I have to do is lean over the desk and ask them. Decisions are quick. Whilst my friends are trying to clean god awful data, I am building part of the backend web platform, and developing an object orientated neural network from scratch in its own repository. Don’t get me wrong, I’m still keeping my hand in the NHS and I’m willing to share the neural network repository with academics who have the particular skills to contribute and use it. But there is a growing concern among the academics that I know. Will the NHS ever improve? Is it just a matter of time until professionals from more advanced industries come in? The sad irony is that the excessive regulation that keeps them safe from the more advanced industries, is the double-edged sword that stifles their growth.

if someone wants me to believe theres a data drought in the medical profession, theyre going to have to start with the admission that too much data can be a misleading distraction.

only then am i capable of taking the need for more data seriously. obviously i mean this in a certain context– *all medical practice comes from data.* but depending on what you do with big data, it either provides needles or more hay to bury them in. often its the latter. so the question is– what data? and if thats common sense, its not common enough practice.

I agree, medicine is in a strange position. There’s a lot of nuance. Someone who has no medical knowledge could be barking up the wrong tree and chasing pointless data. On the other hand, a lot of medics are outright terrible when it comes to basic math, let alone proper analysis, and have pushed ideals that seem like common sense even though the data doesn’t support it because they don’t understand basic statistics. On top of this, there’s a lot of ego at stake. The combination of this is a perfect recipe for stagnation when it comes to data analysis in medicine. It’s changing, but very slowly, it’s only a matter of time before the medical profession gives in, admits defeat, and asks for help from more mature fields. I’ve seen both sides of the coin, the medical departments who will do this, will out achieve the ones that don’t