Don’t try this at home: The mathematics of HIV testing in low-risk populations

This post considers potential pitfalls of home HIV testing. It provides an excuse to write slightly less nerdy column on the mathematics of screening tests.

My friend and co-author A. David Paltiel flew in from Yale to speak with my University of Chicago students. David is a national authority on medical cost-effectiveness, particularly in matters connected with HIV. For example, this beautiful New England Journal of Medicine piece showed that population HIV screening is surprisingly cost-effective, even in relatively low-risk populations. In significant measure due to this analysis, the Centers for Disease Control and Prevention modified national guidelines to promote much more aggressive HIV screening in a variety of settings. So if your primary care doctor or emergency department nurse asks you whether you’d like an HIV test–blame Paltiel.

David and I have published related work on issues surrounding home HIV tests, now under FDA review. (See a great earlier commentary by Walensky and Paltiel here.) To simplify things, the idea here is that you could go to your local Walgreen’s and buy a test kit for about $40. You swab the thing around your mouth. In about 20 minutes, with impressive “accuracy” (in a minute I’ll explain why the commonsense word “accuracy” is a slippery way to describe screening tests), the test will say whether you are HIV-infected.

One natural group of customers might be romantically-involved University of Chicago students: They go out on a date. It goes well. They buy a pair of test kits (maybe romantically sharing one) for a quick HIV test. If the tests come out well…. whatever one cares to happen can now proceed.

Is this test a good idea? In some ways, yes. This normalizes testing. Some people will get tested who would not otherwise obtain care. But there are a variety of reasons to worry. One issue concerns the ability of ordinary people not to mess the test up. A second issue concerns whether home HIV tests will lead people to avoid other medical and public health systems that could provide better counseling and (if needed) stronger post-test linkages to care.

Ignore all these genuine issues. Let’s just focus on the basic mathematics of the test. Of course I’m focusing on HIV tests here, but similar mathematics holds for many similar things: of course a variety of medical tests, but also the screening process used by child welfare authorities to identify parental abuse, the algorithms used by Visa to decide whether a given transaction is fraudulent, and more.

Compared with these other things, HIV tests are amazingly good. Suppose that under realistic conditions, a certain test is 99.3% sensitive. This means that if you are truly HIV infected, you have a 99.3% chance of seeing the unwelcome second red bar in the picture above.  Of course, there is a corresponding 0.7% probability of a false negative in this situation, as well.

Moreover, suppose the test is 99.8% specific. This means that if you are truly free of HIV infection, you have a 99.8% chance of not seeing that same unwelcome second red bar.  There is a corresponding 0.2% probability of a false positive in this situation, as well. Sensitivity and specificity are technical properties of the screening tests themselves: what kinds of HIV-related antibodies can it reliably detect? Are the same chemical products sometimes present when there is no HIV infection?

When we say that a test is “accurate” we are often referring to one of these two numbers. Yet that’s pretty sloppy, too, since a test with greater than 99% sensitivity and specificity can still yield some pretty screwy results when it is deployed within the wrong population.

Suppose we use the above test in a pretty high-risk population in which the prevalence of unknown HIV infection is about 3%. That’s really high. My colleagues and I published a study of six safety-net urban emergency departments and found rates of about 1%. But this high prevalence might be reasonable in (say) an STD clinic or within a population of high-risk injection drug users. For simplicity, suppose that we test 10,000 people.

Given these assumptions, it’s reasonably easy to draw out the following table. If you haven’t done this before, you start with the 300 in the bottom left box, note that 298 is 99.3% of that. So there are two undetected cases. In similar fashion, there are 9,700 truly uninfected people. 99.8 percent of those people will be correctly identified as HIV-.

Take a moment to soak in the profundity of this table. Once we have it, we can answer some pretty simple but important questions:

1. Suppose someone got a positive test. What is the probability that she is actually HIV-infected? It’s called positive predictive value (PPV), the proportion of all positive test results that are true positives. What is this probability? Right. 298/317, or about 94%. From a clinical perspective, this is incredibly important. That’s how worried patients should be if they get a positive test result.
2. Now suppose someone got a negative test. What is the probability that she is truly free from infection? From a clinical perspective, this is also incredibly important. It’s called negative bpredictive value (NPV). What is this probability?  Right. 9,681/9,683, or about 99.98%.*

In this population, the rapid test really seems to work well. It has high positive predictive value, and it has incredibly high NPV, too. Incidentally, you can now see why the commonsense term “accurate” is a pretty sloppy way to discuss screening tests. Depending on the context and underlying prevalence, that word could denote any one of these four things: sensitivity, specificity, PPV, or NPV.

In any event, what happens if we apply the same test within a lower-risk population, such as the population of young sexually active straight couples likely to be a key market for this test. Within this population, HIV prevalence is more like 0.2% rather than 3%.

How does this change things? Well the above table becomes more like the following:

Now what do you think of the test? Of course,9,980 people get the right positive or negative test results. But the test doesn’t seem so “accurate” anymore. The positive predictive value drops to 20/40, which Austin tells me equals 50%. That’s a serious problem. Half of the positive test results will be false positives.

Other consumer groups are even more problematic. Suppose you go out on a date, and you wake up the next morning with the sudden thought: “Who is this person I just slept with? Oy veh, I’d better run out to Walgreen’s for my HIV test!”

This is a problem for many reasons. Rapid HIV tests search for antibodies the human immune system produces—with some delay—after HIV infection. These tests do not search for the presence of HIV virus itself. They are unlikely to detect a very-recent infection. The morning-after-freakout pharmacy trip is thus likely to produce troubling false negative results. The positive predictive value gets worse, too, since we’ll miss some of the true positives while the number of false positives remains the same.

I look at these results, and I worry. Walensky and Paltiel conclude in an earlier commentary:

Home HIV testing will attract a predominantly affluent clientele, composed disproportionately of HIV-uninfected new couples and “worried well” persons, as well as very recently infected persons with undetectable disease. The authors illustrate how testing in these populations may have the perverse effect of increasing both false-positive and false-negative results. A poorly functioning home HIV test may thereby undermine confidence in the reliability of HIV testing more generally and weaken critical efforts to expand HIV detection and linkage to lifesaving care for the estimated 300 000 U.S. citizens with unidentified HIV infection.

That’s a bit harsh, but I have to agree. Broad-based HIV testing in medical settings: Great idea! Home testing in low-risk populations….not so much.

*Thanks Bill Gardner for noting bad typo there.

(HAP)