Imagine you go to the doctor and get tested for a rare disease, one that only 1 in 1000 people have. You know that the test has a false positive rate of 5% and false negative rate of 0%. If you test positive, what’s the likelihood you actually have the disease? If you guessed 95% or 100% you would be very wrong– but you would be as wrong as the typical physician in America.
The real answer is actually a little less than 2%. Here’s why: If one in 1000 people has the disease and they all get tested, the person with the disease will test positive. But because the test has a 5% false positive rate, that means that 5 out of 100 people tested who do not have the disease will also test positive. So 50 other people out of 1000 will have a positive test. If you test positive, the likelihood of you being the 1 in 51 with the disease is a bit less than 2%. (See this video for a good visual depiction of the scenario.)
Probability is not always intuitive, and can be very tricky at times, but this field of knowledge is very important for medicine. Clinicians should know the rough probability of a patient having a certain disease based on their symptoms and test results. Yet health care practitioners are not as good as you might expect at predicting patients’ disease risk.
In a recent study, Dr. Daniel Morgan, professor of epidemiology & public health at the University of Maryland School of Medicine, and colleagues put primary care clinicians to the test. They surveyed more than 550 clinicians and asked them to estimate the probabilities of:
- pneumonia in a 35-year-old woman smoker presenting with fatigue, productive cough, worsening shortness of breath, fever, decreased breath sounds in the lower right field, and normal vital signs;
- breast cancer in a 45-year-old woman with no specific risk factors or symptoms for breast cancer;
- coronary artery disease in a 43-year-old woman with no risk factors, atypical chest pain, and normal ECG results; and
- urinary tract infection in a 65-year-old man with foul-smelling urine but no pain or difficulty with urination.
Clinicians were also asked to estimate the probability of these disease given that a diagnostic test (chest radiography for pneumonia, mammogram for breast cancer, stress test for cardiac ischemia, and urine culture for UTI) came out positive or negative.
In almost every case, most clinicians overestimated the likelihood of the condition before testing and after testing. For example, the typical estimate for likelihood that the 35-year old smoker had pneumonia after a positive chest x-ray was 95%–but evidence shows that at most the likelihood is just 65%. Similarly, a positive mammogram in a healthy 45-year-old with no risk factors means they are less than 10% likely to actually have breast cancer– clinicians estimated that probability at 50%. The figure below from the article shows that in almost every scenario, clinicians’ estimates of the likelihood of disease far higher than what the evidence actually shows.
Why do clinicians overestimate the likelihood of disease and the accuracy of diagnostic tests? Part of it has to do with training — medical schools and other continuing education don’t spend a lot of time teaching probabilities and how to interpret tests. Clinicians are also not immune to the persistent message from some medical societies that “screening saves lives,” nor from advertising from the makers of diagnostic tests that promise accuracy.
The study author Dr. Morgan offered some potential solutions to this problem on Twitter. Among them were:
- Include curricula on probability and diagnostic test interpretation in clinical training;
- Provide clinicians with tools like Testing Wisely that allow them to calculate pre-test probability and diagnostic test accuracy when they need it;
- Implement “diagnostic stewardship” programs to emphasize more careful use of diagnostic tests and avoid common cases of testing overuse, such as ordering of routine urine cultures for asymptomatic patients; and
- Help clinicians and patients understand and accept that tests cannot always give us the certainty of a diagnosis that we crave.
However, the first step toward improving is recognizing that we have a problem with diagnosis in medicine, Dr. Morgan asserts. Only then can we start tackling this important problem.