Precision radiology, deep learning, artificial intelligence, oh my!

Last month I presented a talk on my PhD project at the 2016 Aus and NZ College of Radiology and Radiation Oncology annual meeting, which is the first time I have talked about it publicly. We have not published any major papers yet (our first big one is just about ready for submission), so there hasn’t been much information on what we are doing.

I will explain a bit about the project in this blog post, but feel free to watch the talk. It only goes for about 15 minutes (around 10 minutes of talk plus a few good questions), and it gives a pretty good overview of the topic for anyone not familiar with machine learning and image analysis.

On a side note, my talk actually got tweeted/meme’d by Dr Andrew Dixon, who is among other things the Academic Director for Radiopaedia. Every sensible medico knows Radiopaedia the greatest radiology resource in the world, and vastly outstrips most textbooks for quality and breadth.


7 retweets. So this is what it feels like to be famous.

Without further ado – my research.


Precision radiology is, probably obviously, the application of radiology to precision medicine. Kind of like putting -omic on the end of any discipline (i.e., genomics, proteomics, radiomics) makes it sexier, so does putting “precision” at the front. I also get to use the term “artificial intelligence” to talk about my research, so I’ve got the whole trifecta.

These terms seem to frustrate people and cause much controversy, but I personally think both -omic and “precision” labels are quite useful because they identify a unique way to do science.

Precision medicine is the latest label for the application of -omic techniques to medicine. –Omics (roughly meaning “all of”) are the use of huge volumes of data with objective biomarkers to identify subtle patterns that human doctors can’t really find because we don’t see enough patients, the relationships are too complex for our poor meaty brains, and we can’t explore the parts of biology we don’t understand. More concretely, if genetics investigates single genes and their mutations to identify cause, effect, outcomes and so on, genomics is investigating all the genetic material and its variation in one go. Considering there are 3.2 billion base-pairs in the human genome, exploring mutations one at a time would take a while.


Obligatory double helix picture, but made of lots of dots. Because dots are data, and we have so much of it. Huge data.

I’ll talk more about hypothesis-driven research and hypothesis-free research some other time, but what it boils down to is that wide scale, big data, high-throughput research of the -omic sort is very efficient at finding correlations, but can’t really get at causation. The two methods go hand in hand, it isn’t a case of -omic science replacing the traditional scientific method (people did get upset about this in the past). I like to think of these techniques as an efficient screening system to identify the needles in the haystack, the unexpected places where further research will be useful. You want to search through a million pieces of data, you want -omics. You want to prove a certain drug targets a certain protein, you turn off the computer and hit the lab bench.

So precision medicine is the application of these techniques to medical practice. We use known, robust correlations found by -omic techniques to identify patient subgroups that matter in medical practice. The classic example is BRCA positive patients, a genetic mutation strongly associated with breast cancer. These patients are now offered preventative mastectomy because the lifetime risk of fatal cancer is unacceptably high, even with aggressive surveillance. Since the discovery of the BRCA mutations, many thousands of lives have been saved by this targeted therapy option.

And that is the point – precisely targeting therapy. That isn’t to say that we haven’t been targeting therapy in clinical medicine, in fact every test we do is for this purpose. Diagnosis itself is an example of this concept – a diagnosis identifies a patient subgroup that will have a certain natural history and respond to certain treatments. Unfortunately, diagnosis is pretty blunt, and two people with the same diagnosis may have wildly different histories and treatment responses. The key difference in precision medicine is that we use an -omic method to gather as much information (data) as possible to reach some theoretical limit of how precisely we can define patient subgroups. Hence the value of the terms ‘-omics’ and ‘precision’.

The same concept has been called personalised medicine, individualised medicine and many other things. It is no exaggeration to say that this is the big medical idea of our age, and our best (and possibly only) shot at beating the major diseases that cause ill-health and death in Australia and worldwide. Which is why it is so hot right now – it isn’t just smoke without fire, this seriously going to be the future of medicine.

Radiology has so far been left behind as many other parts of medicine have jumped on the precision bandwagon. It hasn’t been entirely clear how we radiologists can be precise in an -omic sense, with the few forays in radiogenomics gaining high visibility but so far minimal clinical impact. It may just be too soon for these techniques to translate into clinical practice, but I will leave the discussion about the strengths and weaknesses of current methods in radiomics and radiogenomics to another day.


Precision medicine has a problem – most of the major successes have come from genomics. Don’t get me wrong, genomics is the greatest advance in medical science in recent history, but genomics has a major flaw when it comes to measuring human health. The genome isn’t actually responsible for most of our disease.

The majority of disease in the developed world (and increasingly in the developing world as well) is chronic disease, and the genome is only responsible for 20-30% of chronic disease risk. This is probably obvious to everyone – it is our lifestyles that lead to diabetes, emphysema, heart disease, and so on. Once we start looking at people over the age of about 60, chronic diseases start to account for something like 90% of morbidity and mortality, and a huge chunk of our healthcare costs. To throw in a few new words so I can sound very smart, the lifestyle component of health is called the exposome, which is the sum of lifetime environmental exposures. This includes things like diet, pollution, stress, and so on. The combination of the genome and exposome is called the phenotype, which describes the totality of variation in human health. Lifelong and lifestyle, nature and nurture.

So precision medicine has so far been unable to get at the bulk of disease that afflicts our populations because we don’t have a good way to assess the critical part of the phenotype. Many approaches have been attempted to quantify the changes of chronic disease – proteomics to measure circulating serum proteins, epigenomics to analyse changes to genomic expression and function post-conception, microbiomics to assess the commensal organisms that inhabit our bodies. And many, many, many more.


A small subset of -omic domains. I have no idea what orfeomics is.

But none of these techniques get at the underlying problem. Chronic disease is the build up of cellular and subcellular pathology or damage, such as protein aggregates in Alzheimer’s, cross links in arteriosclerosis, lipid-laden macrophages in heart disease. To measure chronic disease with any level of accuracy, we have to measure this damage.

Microscopy is the gold standard here, but we can’t go around subjecting everyone to dozens (hundreds?) of biopsies to get an accurate estimate of the burden of disease. Particularly because most organs (like the brain) don’t really like biopsies, and because a tiny sample of tissue via a biopsy needle isn’t actually very accurate for this sort of thing.

Cross-sectional imaging (CT, MRI) can visualise all of the tissue at once. It can identify and quantify the pre-clinical changes of chronic disease, like with coronary artery calcium scores. There is even evidence that sub-visual image features (things computers can see but humans cannot) correlate to cellular and molecular variations such as tumour subtype and genetic mutations. These were the hints that led me to my research, because these properties of images sound very much like the building blocks of a precision medicine test for chronic disease. If only we could find the important features in the images.


We need an -omic solution to images. Current radiomics methods go part of the way there, but so far haven’t really explored what we might call the “space of all possible image biomarkers” very thoroughly. If genomics is the science of measuring every change that can possibly occur in the genome, we need to work out what the imaging equivalent of a base-pair is. What are the fundamental building blocks of medical images, and how can we efficiently analyse the entirety of this mammoth domain? Again, a complex discussion for another day.

But this is what we are trying to do. We are using more traditional radiomics methods alongside “deep learning” to try and find out everything we can about human health variation from medical images. So far we have been able to identify image features that account for around 70% of a persons risk of death within 5 years, using CT images alone (with strong controls for highly predictive clinical confounders like age). This is preliminary work, but is very exciting for us because it hints that our approach is more than just theoretically attractive. We are about to scale up from our proof of concept experiments to a massive dataset, from tens of patients of tens of thousands. Because big data is better. Seriously, there are mathematical proofs of this sort of thing.

I will try to keep this blog moving along, with more explanation of the various concepts I have briefly touched on here. I might even weigh in on the question I get asked most – are radiologists going to be replaced by computers in the near future (spoiler: sort of).

And never fear, I will try to explain deep learning as well, because, well … it’s so hot right now.



Practice cases now available on education page

Hi all,

After several requests I have uploaded the practice cases for the various lectures to the education page (the same place you can get the lectures). Please remember to download the files to make full use of them, as the animations don’t work over slideshare.

There will be some errors in the presentations, I keep forgetting to note down what the mistakes I notice during the lectures, so please remind me if you spot anything. My email is lukeoakdenrayner at gmail dot com.

I just thought I would say something about this decision to publish these cases, because it was a good learning moment for me. Historically in radiology each consultant gathers a set of unique and interesting cases across their work lifetime, which they use for teaching. There is some strange sort of status and value that comes from having interesting cases, and certainly all of the ‘best’ radiologists have incredible personal film libraries.

But in the modern era with electronic cases and case libraries, this seems much less important now. I personally have no attachment to these sort of ideas anyway, being something of an idealist about the free and open availability of learning resources. Education should be for everyone, in my mind, with the learner in control.

My ‘decision’ to keep the cases locked away until the lectures is even more surprising because my only goal is to teach students. There is no reason I should limit that teaching to when they are in a room with me, but instead they should be able to learn when it is convenient and review the material at any time. It is in fact counter-productive for me to restrict access to the cases if my goal is for other to learn.

But for some reason, my default position was still the traditional one. It certainly wasn’t a conscious decision – I just didn’t think about it at all. It just goes to show that culture is such a strong force, biasing us to take decisions even when they are directly opposed to our conscious positions.

Anyway, thanks to the students who asked the question and gave my brain the kick it needed to realise that my position was silly. Hope the resources are useful, and feel free to pass the links around to whoever might want them.

Launch of the SA Radiology Research Network

Launch of the SA Radiology Research Network

So part of the reason blogging has been slow is that I have been working towards the launch of a new initiative – the South Australian Radiology Research Network (SARRN) is intended to support and promote research and evidence based practice amongst radiologists and associated health professionals in SA.

We are still in the early stages of this initiative. The goal is to make life easier for anyone engaged in radiology research, including providing guidance in research methods, formalising a network of junior researchers, supervisors, mentors and collaborators, and simplifying the logistics of research – ethics applications, grant proposals, and the other various hoops that we have to jump through.

More details can be found at our website, which includes mini-guides and resources for researchers, as well as a news section to keep up to date on all the various changes that will impact on research in radiology (with Transforming Health, Choosing Wisely and changes with SAMI, there is a lot going on!).

What motivates clinicians to do research?

One of the reasons I am writing this blog is to organise my thoughts as I prepare a set of talks for radiology trainees in South Australia, with the goal of demystifying and promoting research. One of the first talks I am planning is a question that doesn’t get considered much among doctors – why would you do research?

The reason this is rarely discussed is that most of the time the answer is simple; research is a requirement of training or practice. But what is it that changes someone from that to a self-motivated researcher?

This is an especially important question in radiology because there is no extrinsic motivation to do research once a fellowship is obtained. Post-fellowship, CME (continuing medical education) credits are generously offered for research activities, but this is a non-factor when the points are easily obtained by less onerous means.

And we aren’t just comparing research to other CME activities. We have to be up front about it, serious research is a big commitment. It sucks up time and energy that you may prefer to spend on other things. So to be fair to prospective researchers we must compare research with their non-medical hobbies and interests.

My experience is that post-fellowship radiologists split their time in several ways:

  • about 60% of radiologist time is spent on non-medical interests and hobbies. Travelling, food and wine appreciation and various sports and fitness activities seem popular.
  • 20% or so spend a significant portion of their time becoming an expert in a subfield of radiology. Maybe another 5% devote much of their life to this, and become truly impressive in their area of knowledge.
  • another 10% develop non-medical expertise. They take courses in management/business, history, cooking, art, and many other things.
  • the remaining few percent goes into research activities.

Maybe it is obvious to most readers why a radiologist who spends all day sitting in a dark room might prefer spending time outdoors when they get home instead of working on a research paper, but it is harder to explain why there is such an imbalance between research and developing expertise. These seem like similar tasks and probably have similar motivations.

Everyone is motivated by different things, and has different values. Some work for money or prestige, some for a fulfilling challenge, some for the human interaction. But when I think back to interviewing for medical school, most of us said it was helping other people that led us to medicine. Even if we didn’t know it at the time there are better ways to make money, other jobs are as challenging, and doctors and patients aren’t always the most social bunch. The unique aspect of medicine is that we affect people’s lives more directly than most other professions can. Or so it seems. More on that in a minute.

If helping people still drives doctors later in their careers, we have a clear reason to do research. Medical research is responsible for all of modern clinical medicine, all the lives saved, disabilities managed or averted, symptoms controlled. So why is there a disconnect, and it remains a minority of doctors who pursue significant research agendas?

The disconnect, I think, is that most people haven’t thought through what helping people actually looks like. I would argue that really improving other people’s lives is almost impossible for non-researchers. Since we have just noted that most doctors enter the field wanting to help people, that last statement might need some explaining.

We can frame this as a maths problem – how much good can you do? This question is surprisingly complex, and we have to turn to economic theory to understand the answer. The back-of-the-napkin calculation goes like this:

A clinical radiologist in a developed country might see a few hundred thousand films in a lifetime, maybe a million if they push. What percentage had important findings? What percentage of those would have been missed by any other radiologist? What percentage of those would have caused harm. Take this number and subtract how many harmful findings you will miss that someone else might have detected.

That is your marginal impact, the amount of human benefit that you personally add beyond the baseline radiologist who would otherwise be doing your job.

In a million studies, I might find a few thousand serious things that could have been missed. I will also miss a similar number of things that other radiologists might have detected. Even if we are generous, my marginal impact to patients over my entire career might be less than a thousand important findings, most of which would have been later detected by other tests when symptoms persisted.

A little bit depressing, right? In actual impact, doctors might not be any better at helping people than anyone else. If we truly care about helping people, there may be much more effective ways. Researchers have explored this topic in depth, if you want to read a good summary of the research “How much good do doctors do?” is a great place to start. There are definitely some questionable assumptions in this research (like any economic modelling), but taken at face value the point is pretty clear.

If you do important research, your marginal impact can be massive.

Let’s give a concrete example. My area of interest is medical informatics. I want to use computers to make radiologists more accurate and more productive. Lets imagine a diagnostic aid system that improves the sensitivity and specificity of mammography reading by a few percent, a small and achievable gain. Let’s do that same napkin maths:

Every year in my state, over a hundred thousand women are screened. The stats show that 15% of cancers are missed at the initial screening, and just under 10% of benign lesions are thought to be cancer. Over 90% of positive screening mammogram reports end up not being cancer, but many of these healthy women end up with harmful biopsies and even surgery.

We bump those numbers in the right direction by a few percent, and we might see thousands of less ultrasounds per year, several hundred less biopsies, a few dozen less surgeries. We might even see a small but noticeable reduction in cancer mortality, maybe one or two women per year surviving longer. And that is in one state, across the world we would multiply those numbers hundreds of times over. The cost saving and the reduction in harm would be more than a single radiologist could ever achieve.

Lets be conservative and say I was responsible for 10% of that research. Lets go even further and say that it was an idea whose time had come, that another research team will achieved better results one year later. Even then my marginal impact, 10% of 12 months of a slightly better system, would completely dominate the impact of my entire clinical career.

This is why research is so impactful. Research is generalisable. What affects one patient can affect all patients, and it is a big world. There are a lot of patients.

Research is also incremental – my above mammography system would have informed the research that improved upon it, another positive not accounted for on my scribble covered napkin. New problems can be solved because of past research, just like electricity lead us to the computer.

Which is all just a long-winded way of explaining my motivation to do research. I would say I want to make the world a better place, but Silicon Valley has forever ruined the phrase. But I do want to make a difference, so here I am.

Interested in radiology or research?

Just a quick note:

If you are in South Australia and you are interested in radiology or research, or even radiology research, feel free to contact me. I can answer any questions you have or maybe even connect you with researchers who need help.

And if anyone is willing to give me feedback on my teaching materials, feel free to contact me as well.

My email is my full name (no spaces or dashes) at gmail dot com, or you can leave a reply here.


edit (10/5/2016):

I have recently had several conversations with medical students about my own research agenda, and have been surprised with the amount of interest in data science, machine learning, artificial intelligence and automation in medicine. If anyone is interested in these topics and might want to engage in research along these lines down the track, I can help you on that path. If you have programming skills already, or if you are a beginner but are happy to put in the time to learn (it is easier than it sounds), contact me.

First steps as a clinician researcher

Welcome to the first post of my blog! It feels a bit self-indulgent, but documenting my path into academia might help others who may find themselves in the same situation I was a bit over two years ago: a medical doctor who has never considered a career in research.

To explain that, a bit about how I got here.

I am a medical doctor trained in Adelaide,and it is fair to say that research does not feature prominently in our undergraduate curriculum. This doesn’t appear to be the fault of the faculty, because the near universal response of any medical student to research methods education is disinterest or even frank annoyance. The few lectures we did get were grudgingly attended, with a special dislike of anything related to statistics.

It seems that for medical students, at least in my experience at the undergraduate level, research isn’t “medicine”.

I don’t blame them. I felt the same way.

Most of us sign up to medicine expecting certain things, usually informed by tropes about the caring doctor from TV shows and books. Many of these expectations turn out to be pretty reasonable, despite the fictional origins. We see patients, we diagnose patients, we treat patients. We feel like we are doing something worthwhile, which is reinforced by the feedback we get from patients and families.

But that isn’t all a doctor can do.

A bit over a year ago, I had cleared my final exams and I had one major project left to do as part of my training. The “Part Two Project”, the biggest research engagement for trainees, which must result in a talk at an international conference or a paper that is accepted for review in a journal. A fairly low bar, by the standards of some training programs, but considering the low appetite for research in radiology even this much is often seen as a burden.

For most, the part two project is an obstacle to overcome with the smallest effort possible. Find a consultant who wants something written up, spend a month or two doing the grunt work, wipe your hands.

For me, that didn’t seem very palatable. I’ve always hated the idea of busy work, so I decided to spend some time thinking about something I could do that was worthwhile. I had a long think about my interests, and my goals in life in the broadest sense, and tried to work out where a 6 to 12 month project might fit in. In the end, it took me over a year to finally strike upon my idea, but I am glad I did. The decision has changed my life for the better.

What was going to be a six month project has snowballed into a PhD, my idea has grown into an international collaboration, and my hobbies and interests have infected my work to the point that I find myself having fun and getting paid for it.

I could have made my decision to do research much sooner if I had just had someone to explain it to me. Research can be boring, if all you notice is dry stats and tedious rules without any obvious payoff. But when you are applying those rules to something you find important, to discover something worthwhile, it all changes.

Well, at least, it did for me.

I hope I can use this blog to expand on these ideas, and maybe even spark something in the minds of people like me. Who knows, a brief insight into the life of a clinician researcher might change your mind too.

About me

I am a medical doctor training in Radiology at the Royal Adelaide Hospital, and Ph.D in Medicine candidate with the School of Public Health at the University of Adelaide.

My research explores the intersection of medical imaging, computer science and future technologies, with specific interests in the application of machine learning to medical images and text, exploitation of our vast but unstructured stores of medical data, and technologies that potentiate education and research activities.

I am passionate about education and currently teach interpretation of medical images to University of Adelaide medical students, junior Radiology trainees and anyone else who has time to go through a few films.

I am interested in promoting research activity in Radiology in South Australia, and can provide assistance and resources to any budding researchers, or help you make contact with appropriate collaborators and supervisors.