Is the new Covid-19 test a ‘game-changer’?
Many media reports have recently covered the new “rapid” tests for Covid-19. Using nose or throat swabs, these tests can detect the virus in only 15-30 minutes and many reports have described them as ‘game changers’.
Such a bold claim deserves a bit of questioning and that needs a bit of math(s). Because many readers won’t be jumping-for-joy at the thought of big numbers and sums I’ll start with the take-home message and follow on with a detailed explanation for the undaunted few.
The take home message:
Although the new rapid tests are highly accurate (97 to 99%), most of the people who test positive will not have Covid-19; there will be many false alarms.
The full story:
The World Health Organization recently approved two new rapid Covid-19 tests, saying they can be over 90% sensitive with a specificity between 97% and 99%. Sounds impressive, but what does it mean?
To answer, let’s imagine giving the test to 1000 randomly-selected people. We need to know two things for this calculation: the amount of infection within the population and how good the test is (ie, its specificity).
Let’s use 0.1% for the population rate of infection. This may seem low but remember that the test is looking for virus not antibodies so we’re only interested in those people infected at the moment and not those who’ve had the virus and recovered.
0.1% is roughly the rate of infection that was present in India during the first week of October and at the same time India was close to the world’s average for cases-per-million; so 0.1% seems a reasonable choice. (Details of the data and calculations can be found at the very end of this article.)
For the specificity of the test we’ll be generous to the manufacturers and take their best value of 99%.
So, what might we expect to find when giving a test with 99% specificity to 1000 people selected at random from a population where 0.1% of people currently have Covid-19?
Firstly, we’d expect that only 1 person in the group has Covid-19 (0.1% of 1000) and the other 999 people therefore won’t.
Now it gets a little interesting when we realise that a test with 99% specificity means that 99% of positive results will happen because the person truly has the virus and the other 1% of positive results will be errors (“false positives”) where the person tested does not have the virus.
This sets the cat among the pigeons because in the group of 1000 we won’t just detect the 1 case that has Covid-19; there will also be some false positives as well. How many will depend on the false-positive rate of the test (1%) and the number of non-infected people that we’re testing (999 out of 1000).
So, now we see that we should expect to get 10 false positives (1% of 999) as well as finding the 1 true positive in the group.
The false positives (‘false alarms’) will happen 10 times more often than the true positives, so that only about 9% of all positive tests will not be errors (1 out of 11). In other words, even though the test has an ‘accuracy’ of 99% we can’t have 99% faith in the result when the condition being tested for is not very common. (If the test were 98% specific then only about 5% of positives would not be errors.)
So it may be reasonable to ask: in what sense are these new tests ‘game-changers’?
Well, the speed of the test result will certainly make a big difference for those people who are not infected. They will get the ‘all-clear’ almost instantly whereas the standard test at the moment is not only much more expensive but may take a day or two to give a result. (Aside from the question of whether or not it is too sensitive at detecting pieces of the virus – a topic for another day.)
So with the new tests doctors will be able to reassure people that they are not infected or infectious almost immediately, and for this reason it will indeed be a game-changer.
The flip-side is that many people will test positive even though they don’t have Covid-19, so that they will have to quarantine for a day or two until cleared by a second, more specific test, ie the slow, expensive one currently in use.
The high ratio of false positives to true positives (10:1 in this example) is not a sign that the test is poor or that it’s failing in any way – although most of those who get a positive result will probably think so.
Surprisingly high rates of false positives are expected with any test whenever the condition being looked for is uncommon, even if the test itself is very accurate and performed correctly. The challenge for the authorities will be to maintain the public’s faith in the system once the media start reporting high rates of failure.
The idea that testing for rare conditions will generate many more false positives than true positives, even with highly specific tests, is an important concept not limited to Covid-19 testing. For instance, it’s one good reason why many of the cancer screening tests haven’t been rolled out across entire populations.
The sharp-eyed reader may be wondering about the role of other side of a test’s accuracy, namely its sensitivity. The WHO states ‘up to 90%’ sensitivity for these new tests, meaning that 10% of people who have the virus will be missed by the test. This sounds important, but is it really?
To look at this we need a bigger group, so let’s expand and apply the test to 10,000 people from the same population with the same 0.1% infection rate. In this new group we’d expect there to be 10 infected people and our test should pick out 90% of them, ie 9. So, it will give false reassurance to 1 person in 10,000 or 0.01%. This seems not so bad.
So, I’ll conclude this toe-dip into the murky waters of medical statistics with a second and final take-home message:
The rarer a condition is, the less you can trust a positive result, even with an accurate test.
Thank you for reading. This is a tough concept to take in so don’t worry if it’s been a bit challenging – I wish I had a certificate to offer you for getting this far! If you’re interested in finding out more (or if you want to find a better explanation!) then try putting “Bayes Theorem” into an internet search engine and you should find many helpful articles.
~ ~ ~
India, 1st week of October, 2020
Total diagnoses of Covid-19: 6,641,789
Recovered from Covid-19: 5,608,761
Deaths from Covid-19: 103,600
Currently infected: 929,428 (6,641,789 minus 5,608,761 minus 103,600)
Prevalence of active Covid-19 infection: 0.01% (actually 0.07%) (929,428 / 1,383,530,375)
Specificity of test (true positive rate): 99%
False positive rate: 1%
Size of the test group: 1000
Rate of active Covid-19 in the test population: 0.1%
True positives by testing: 1 (0.1% of 1000 people)
False positives by testing: 10 (1% of the 999 people in the group who don’t have Covid-19)
(Substitute different percentages to see the effects of more/less specific tests and more/less disease within the community being tested.)