The Incidental Economist

There’s been some chatter about how the Oregon Medicaid study is or might be biased. That’s worth a post!

There’s a precise way in which the study is not biased. By design it estimated the effect of Medicaid on those who won the lottery and enrolled, relative to those who lost the lottery and did not. This estimate is unbiased for the contrast between precisely these two groups, but not necessarily for others. In econometric jargon, this is known as the “local average treatment effect” (LATE). The “treatment effect” part of “LATE” is clear, but what’s this “local average” business?

Sigh. I hate this terminology. It’s supposed to evoke the idea that the instrument (the lottery in this case) doesn’t have a “global” effect on study participants, causing all randomized to Medicaid (lottery winners) to be on and all those randomized to control (lottery losers) to not be. It has a more modest, “localized” effect. The other jargon used for this is that the LATE estimate is an estimate of the effect of treatment on “compliers.” That’s a more meaningful term to me. The compliers are those that do what randomization “tells” them to do, they enroll in Medicaid if randomized to do so and they don’t if not.

Of course, you can’t expect full compliance in this study (or many other RCTs) because some lottery winners turned out to be ineligible for Medicaid by the time they were permitted to enroll. Some had too high income. Some moved out of state. Some may have found other sources of coverage. (You had to have income below 100% FPL, live in state, and uninsured for 6 months to be permitted to enroll.) Also, enrollment wasn’t mandatory. So, if you just decided it wasn’t worth the trouble or didn’t receive or notice the letter inviting enrollment, you might have missed the window (45 days is all they gave you).

On the flip side, nobody was preventing lottery losers from enrolling on Medicaid if they became eligible in another way. The study pertained only to the expansion of Medicaid beyond the statutory requirements. If people ended up in one of the eligible categories (aged, blind, disabled, pregnant) they could get on Medicaid.

So, there was considerable “crossover” (lottery losers enrolling in Medicaid, lottery winners not) or “contamination” or “noncompliance,” all jargon for the same thing. This was not a perfect RCT. Few are.

What to do? The investigators did two things. First, they considered an “intent-to-treat” (ITT) approach, comparing lottery winners to losers no matter whether they enrolled in Medicaid or not. These results are in their first year paper. I’ve forgotten what they say specifically, though in general they’re much smaller effects than the LATE results. The concern with ITT is that all this crossover biases the results toward zero. There isn’t as much contrast between study arms due to noncompliance.

Next, the investigators provided LATE estimates, about which I wrote above. These are unbiased for contrast among compliers. In this study, they’re about four times the size of the ITT estimates by virtue of the mathematics (“instrumental variables“) of LATE. But they need not be the same as one would find in the absence of noncompliance. There may be bias in that sense. Why?

• Hypothesis 1: Those who took the trouble to enroll in Medicaid were sicker than those who didn’t. After all, why enroll if you don’t need it? Remember, even some lottery losers (18.5% of them) enrolled in Medicaid. The LATE estimate removes the effect of them since they are noncompliers. Also, some lottery winners didn’t enroll (most of them didn’t) and the LATE estimate removes their effect too. What’s left under this hypothesis is a comparison of relatively sicker people who did enroll in Medicaid with relatively healthier people who didn’t. The investigators actually found some evidence to suggest that Medicaid enrollees are sicker. Many other studies find that Medicaid enrollees are sicker to the point that some studies find an association of Medicaid with increased mortality. Under hypothesis 1, results are biased downward relative to what they would be under full compliance. Medicaid looks less effective than it might otherwise be. 
• Hypothesis 2: Those who are more organized, better planners, with higher cognitive function and literacy (including health) skills enroll. It takes some awareness and planning to enroll, so there is some face validity to this argument. I’m aware of no evidence to support it though. (Got any?) Under this hypothesis Medicaid enrollees would do a better job of getting and staying healthy even apart from whatever Medicaid does for them. This would bias results toward showing a larger Medicaid effect than would be true in general (under full compliance).

There may be other hypothetical sources of bias. The point I’d make about all of them is that we don’t know whether any of these biases actually exist and, if they do, how big an effect they have. It’s all speculation. Still, LATE is an unbiased (and causal) estimate of the effect of Medicaid on compliers. It does filter out some who want to be on Medicaid and can’t enroll (lost lottery, no other route) and filters out some who enroll but weren’t invited (lost lottery but became eligible another way). Some of these noncompliers could be unusually sick. Some noncompliers could be unusually organized and aware. LATE filters some of them out.

Some might wonder about another type of estimate one could do, the effect of “treatment on the treated.” Here one just compares Medicaid enrollees to non-enrollees, ignoring the lottery draw. Unfortunately, this just exacerbates whatever bias might exist. There is no random assignment at play here. There’s no filtering for selection at all. You get an association, not a causal estimate. This is the problem with many studies of Medicaid and insurance. Randomness is key. The lottery should be exploited in some fashion (either ITT or LATE).

Lastly, notice how complicated RCT interpretation is? Yes, it’s the gold standard, but it still has issues. Using an IV approach for a LATE estimate is, in my view, about the best you can do. But there may be bias when considering generalizing the findings outside the “local” effect of the instrument (lottery or random assignment). These concerns arise with any IV study. In this sense, IV and RCT are much closer cousins than one tends to think. Disparage one and you disparage the other.

Not all that’s gold glitters, but it is still valuable.

@afrakt

Last night I had a false start in updating power calculations for the Medicaid Oregon study. The final result with corrections and a new PDF is here. If you’re into this stuff, it’s worth a look. The bottom line is that the study was underpowered for the change in proportion with elevated glycated hemoglobin (GH) by a factor of 23. Yes, twenty-three. You can use what I posted to run the numbers for other outcomes yourself.

What happened last night is that I had failed to update the computation of R², which requires some algebra or a simulation, neither of which I was prepared to do at 10PM. I’ve done both this morning and the results are documented in the updated post and PDF linked therefrom.

Let me now state where we are. With respect to the statistically insignificant physical health measures in the study, we now know they were very underpowered. The sample was too small even for much larger effects. This renders them statistically uninformative in general and, in particular, uninformative about whether or how much Medicaid improves physical health. Uninformative means just that. No new information. No resetting of priors is justified on this question.

We also know, from the authors’ discussion and from Aaron’s posts, that the results include changes in blood sugar and blood pressure that are not unreasonable to have expected clinically. Thus, the results — or these two anyway, but I suspect it generalizes — are not clinically informative either. Again, no resetting of priors is warranted.

Given this, for the physical health measures only, I don’t understand the rush I’ve noticed in people updating what they expected Medicaid could do. These results really shouldn’t do that if they are, as I’ve said, uninformative both clinically and statistically. How did people make these judgements the day after the study was published? It’s taken me and Aaron almost two weeks to chase things down. I think it is time for people to take another look at what this study is saying, at their own priors, and, yes, at their own biases.

What I think we’re seeing is a re-expression of everyone’s priors. This study is an opportunity to do that, but it doesn’t and shouldn’t change what they are. The claims that people should be changing sides from pro- to/from anti-Medicaid expansion just make no sense based on the physical health measures in this study.

Meanwhile, yes, this study reconfirms some large financial and mental health benefits that we knew about from last year’s paper from the group. I’m not sure that’s a prior-updating event either.

This was (is) an excellent study done by a smart and capable team of investigators. The results, to the extent they are meaningful, should be viewed as among the most credible possible within the context of the study (in/around Portland, over 2 years, Medicaid circa 2009). And yet, much too much is being made of the set of results that just don’t tell us anything new.

I’m happy to be corrected on any of this if I’ve overlooked or misinterpreted anything. As always, I am merely trying to be scientific.

UPDATE: I mistakenly wrote “30″ instead of “23″. It was a very bad rounding from 22.9.

@afrakt

Sorry for the false start last night. More about that here. I still welcome comments if you find any errors.

This is a follow up to my several prior posts on how to adjust a power calculation to account for an instrumental variable (IV) design. The details are in a new PDF. (If you downloaded the one I posted last night, replace it with this one.) First, for the non-geeks:

• Skip the proof and jump to the example that begins on page two of the PDF. It runs through the numbers for the Medicaid study result for glycated hemoglobin (GH), which I had used to illustrate the power issues in my first post on this topic. (It’s a commentary on this blog’s readership that I can even consider this example suitable for non-geeks. I guess I mean geeks of a different order.)
• One thing you may notice is that the Medicaid and non-Medicaid groups are different sizes than you might have expected if you only read the paper and not the appendix. I refer you to appendix table S9 for the details. Suffice it to say, it is not true that 24.1% of the lottery winners took up Medicaid. There were a lot more Medicaid enrollees than that. (What is true is that 24.1% more lottery winners took up Medicaid than non-winners.)
• For that reason, and because I was targeting 95% power, my estimate in my first post was quite a bit off. I thought the study was underpowered by a factor of 5 for the GH measure. Actually, according to the methods in that post, and using the new numbers and targeting 80% power (which, I am told, is more standard), the study is only underpowered by a factor of 1.5.
• But, as I wrote in that post, I had not accounted for the IV design. The new calculation does so. And that, my friends, really wallops power and precision. The bottom line is, accounting for the design, the GH analysis was underpowered by about  a factor of 23 (yes, twenty-three!) meaning it’d have needed that multiple of sample to be able to detect a true Medicaid effect with 80% probability.
• You can run the numbers for other measures using this online tool. The underpowering will vary. Below is a screenshot for the inputs for the GH analysis. Follow the steps in the PDF for the rest. (Hint: multiply the sample sizes from the online tool by 14.8.)

Now, for the uber-geeks, the content of the PDF differs from my prior version of a few days ago in three ways:

1. It properly accounts for the fact that we were assuming all vectors were zero mean. That didn’t affect the result, but it does affect how you should simulate the first stage (which we’ve done for you for the Medicaid study in the document).
2. It references Wooldridge, who obtained the same result. (So, we’re right!)
3. It includes a complete example from the Medicaid study. However, don’t overlook the fact that this generalizes. Truth be told, I didn’t do all this to comment on the Medicaid study. I need this for my own work.

I should point out that the finding that the variance of the effect size in an RCT scales with the inverse of Np(1-p) is beautiful. It doesn’t just scale with 1/N because it is the mean of a difference. When p goes to zero, there are no treatments. When p goes to 1, there are no controls. Either way, the variance of the difference in effect size has to go to infinity. And, indeed it does. This is comforting intuition.

Finally, I’m grateful for the awesome feedback I’ve received from readers. Once again, the TIE community has hit this one out of the park. Thank you.

@afrakt

I spoke with Russ Roberts of EconTalk for an hour or so about the Oregon Medicaid study. The conversation took place last Thursday. As you can tell by reading more recent posts on this blog, we’ve learned more since then.

@afrakt

Superstar blogger Adrianna McIntyre of Project Millennial got me those Lurie et al papers. They are two reports of patients who were dropped from the Medi-Cal program in 1982. It’s a case-control study, one report 6 months following loss of coverage and the other at one year.

The first paper examined 186 patients who lost coverage and compared them to 109 patients who did not. Of these, 51 had hypertension in the group that lost coverage, and 38 had hypertension in the group that did not. They found that those patients in the “study” group had their diastolic (not systolic) pressures go up in the six months after losing coverage (+ 10.0 mm Hg) and those who maintained coverage saw them drop (- 5.0 mm Hg).

Where to start? This is a very small study, not an RCT. So it’s nowhere near as powerful as the RAND HIE or the Oregon Medicaid study. But, yes, losing Medi-Cal coverage was associated with an increase in systolic BP in those who had hypertension. But it’s more complicated than that. The analysis didn’t control for any covariates at all. A number of the people who had their coverage dropped got health insurance from other sources, or got their Medi-Cal reinstated. So… are we saying that Medi-Cal was better than those other insurance products? Moreover, this was a plugged in group of patients. Check this out:

More than 95 per cent of the patients in both groups had, and could identify, a regular source of medical care. Nearly 85 per cent of the patients in both groups said they thought they could get medical care whenever they needed it, and approximately 90 per cent were “extremely” or “very” satisfied with the care they received.

In fact, this study came about because the researchers cared for these patients and were helping them to regain coverage or get care elsewhere. I applaud the efforts. But this is not a random design conducted with a blinded and disinterested group. And there’s no way these patients are representative of those in the Oregon study.

It’s also a very small study. It was statistically significant because the difference seen was huge, and they didn’t control for any other factors. It was also after a 6 month period. So it’s worth looking at the follow-up study conducted at one year which managed to include between 81% and 86% of those in the first study. At this time point, the systolic BP had decreased 4 mm Hg from the 6-month point in the group that lost coverage. The systolic BP in the control group (which kept Medi-cal) had increased 3 mm Hg from the 6-month point. In other words, they saw a regression towards the mean, which is what you might see from a small study like this with surprisingly large initial results. So at one year, the difference between the two groups in systolic BP was 4 mm Hg (Table 2). Would it have continued to lessen by two years? We don’t know.

Nonetheless, this is a very small study of a select and non-random group of patients. They saw that losing coverage was associated with (not caused) a large difference in systolic BP at six months that was much reduced (to about 4 mm Hg diastolic) by one year in a simple, uncontrolled analysis. These were patients really plugged into the system, who had physicians who were actively involved in their care and coverage. I’d expect a bigger difference from a sudden disruption in phenomenal care than from the other end, where you take a patient with no insurance and potentially less access and give them some. In other words, this study could be more appropriately used to describe the dangers of dropping people from the Medicaid rolls, than to describe the benefits of giving them new coverage. And I still wouldn’t say it’s powerful enough to be slam-dunk representative.

I’m not the only one who thinks this way. These ideas were also included in a letter to the editor here.

Moreover, all my previous arguments hold. We’re talking about patients with hypertension here. Not a much larger group of people, some of whom had hypertension.

But still, this study is nothing like the RAND HIE or the Oregon Medicaid Study. I believe some of the authors were involved in both these papers and the RAND HIE. I bet they’d tell you that the RAND HIE is much, much more significant. And for my feelings on that, go back here.

This study does not convince me that we should see large results from giving people with relatively well controlled blood pressure Medicaid in Oregon. Tell me why I’m wrong.

@aaronecarroll

So some people have asked why I have focused on diabetes so much, and not blood pressure or Framingham composite results, when looking at the Oregon Medicaid study. There are a few reasons. First, I’m a pedatrician. I know much less about hypertension, as I don’t see much of it nor do I do any research on it. Second, I do research in diabetes, and know much more about that. Third, I think diabetes is a looming problem with more short term concerns. Finally, I can’t remember the last time I used Framingham stuff (other than to advise friends), and I try not to wax philosophic on things unless I feel like I know enough about them.

But it’s a fair point, and so I decided to look into the blood pressure results further. They don’t make me feel much better.

Let’s start with what the Oregon Medicaid study reported recently. It’s in their Table 2. They found that in the control group, the systolic BP was 119.3 mm Hg. Medicaid changed it by -0.52 mm Hg. That wasn’t statistically significant. But let’s acknowledge that this result is for all people in the study, with and without high blood pressure (a systolic BP of 140 mm Hg or higher). Obviously, most people didn’t have it. The average systolic BP in the study was under 120 mm Hg, which is normal. so I’m not sure how much you could expect the (normal) average BP of this group to go down.

There seem to be two main sources of “expectations”. The first are two non-RCT papers from Lurie et al, published in the mid-1980s following people who were kicked off Medi-Cal. I’m told they saw a big jump in blood pressure, but even my institutional access won’t grant me permission to see those papers online. I’ll keep trying. If someone wants to send them to me, I’d be thrilled.

But I did find the RAND HIE paper on BP. That study found that for people with hypertension, giving people free care resulted in a change of systolic BP of -1.9 mm Hg. It’s easy to find this result. It’s right there in their Table 2.  Then you can go to Table 3. Among low-income people with hypertension (which I’ll concede is arguably more comparable to Medicaid populations), the change was -3.5 mm Hg.

But those results were for people with hypertension. There were 856 of them in the study. The average systolic BP in the free care hypertensive group was 137.1 mm Hg. So there was lots of room forimprovement.

Table 2 in the Oregon study, on the other hand, is all participants. Not hypertensives. You can tell that from the average systolic BP (119.3 mm Hg). I don’t see how you can apply the results from the RAND HIE (-3.5 mm Hg) to that group, because that group is not all hypertensives.

I’ve been told to look at sub-analyses. So be it. I’ve scoured the Appendix. Table S14a is a subset of those in the Oregon study who were age 50-64. They were less healthy, in that their average control group systolic BP was 127.3 instead of 119.3. And their systolic BP changed -2.5 mm Hg. Still not significant, of course, but much closer to the RAND HIE effect. And again, their average systolic BP was 127.3, which is still much less hypertensive than those in the RAND HIE.

The most on-point subanalysis I can find is Table S14c. That’s the pre-randomization diagnosis grouping, and it includes 2225 people who had a diagnosis of hypertension before the study began. But there are a few things to consider here. First, these are people who may have been more likely to be plugged into the system already because they had diagnoses. So they might see less of an effect by getting Medicaid. Second, their control BP was 129.8 mm Hg, still much lower than the hypertensive subjects in the RAND HIE. In fact, only 38% of them had an elevated blood pressure. But there’s more. There were only 2225 people with a diagnosis of hypertension pre-randomization. That’s 2225 of the 74,922 who were randomized. That’s 3% of the sample. If the percentages hold (and they should if it’s random), then they accounted for only 3% of the 1903 OHP-enrolled survey responders in the intervention group.

How many is that? 56 or 57 people. Of whom, only 38% who had an elevated blood pressure. So how much effect could Medicaid have?

The more I look into this, the more baffled I get. Here’s how the RAND HIE defined people with hypertension:

Thus, participants were called “hypertensive” at enrollment if they (1) reported taking antihypertensive drugs, (2) had a repeated systolic  blood pressure greater than or equal to 160 mm Hg or diastolic blood pressure greater than or equal to 95 mm Hg at the examination, (3) had a repeated systolic blood pressure greater than or equal to 140 mm Hg or diastolic blood pressure greater than or equal to 90 mm Hg and reported that their physician had previously told them they had hypertension, or (4) reported that a physician had told them more than once they had hypertension and either were assigned to miss the examination or had systolic blood pressure greater than or equal to 130 mm Hg or diastolic blood pressure greater than or equal to 80 mm Hg. Others were called “hypertensive” at exit from the HIE if they met criteria 1, 2, or 3 at exit or if (5) they had both repeated enrollment and exit systolic blood pressure greater than or equal to 140 mm Hg or diastolic blood pressure greater than or equal to 90 mm Hg or (6) a physician had reported hypertension on an insurance claim form and the participants reported they had been diagnosed as hypertensive, or the physician had reported hypertension on two or more insurance claim forms.

Those people had hypertension. There were more than 850 of them in the study. They had average systolic BPs near 140 mm Hg. And they found that in the low income sub-group, giving free care resulted in a change of -3.5 mm Hg.

No matter how I slice it, I can’t generate the same enthusiasm for the Oregon study. The study had fewer people, and less power. The study doesn’t seem able to define or quantify the hypertensive group in a similar way. The study seems to feel that it should be able to see a drop in systolic blood pressure that is higher than what was seen in the RAND HIE. And the study seems to think that we should be able to see those drops in groups with starting systolic blood pressures that are much closer to normal than those seen in the RAND HIE.

It’s totally possible I’m wrong. If I am, I will post it right here. Tell me what I’m missing. Tell me why when we start with a population of people with much better control of their blood pressure, we should expect a larger absolute drop in systolic BP than the RAND HIE did. And tell me why this study should be powered to detect that difference with what seems like far fewer participants with hypertension.

I’m still going to look for the Lurie studies. But it seems that the RAND HIE, which is the only other real RCT here, should be most on point.

@aaronecarroll

I’ve spoken to a number of people in the media today about the Oregon Medicaid study. On the whole, they asked excellent questions, and made me think about my own interpretation of the results. They also forced me to think about how I would be speaking about the study if the results had been amazing. Would I still “quibble” about power?

Here’s the thing. If they had shows statistically significant improvements in health outcomes, I’d be impressed. If they had shown statistically significant harms, I’d be concerned. Non-significance doesn’t tell us that much, especially if the study is underpowered.

But am I being fair? Consistent? Some have accused me of changing course. So I looked back at what I’d written previously, when I said “we could even start talking causality”. Here’s what I said:

But I’d like to reiterate that this was a randomized controlled trial. An RCT is pretty much the best way to prove causality, especially if it’s well done. So if you wanted to prove that Medicaid causes bad outcomes (as many do), this would be the way to prove it.

Not too long ago, ACA opponents were claiming that Medicaid was bad for health. Some even claimed it killed people. So I was eager to see if an RCT would find that. The initial results were positive and statistically significant. So I concluded:

Randomized controlled trials on this scale happen rarely. We’re still talking about the RAND health insurance experiment, which occurred decades ago. Here’s one that shows that Medicaid is both good for health and provides a significant financial benefit for it’s recipients. Since it’s an RCT, we can even start talking causality.

There’s no such studies or evidence showing the opposite, that Medicaid is bad for health. We’ll see if that talking point goes away.

There is still no evidence that Medicaid is bad for health. But if anyone has moved the goalposts, it’s the people claiming that now Medicaid must prove it improves health, instead of just not harming people. I stand by everything I said before. I think I’ve been pretty consistent.

@aaronecarroll

This is a joint post by Austin Frakt and Aaron Carroll.

Ross Douthat has a measured and reasoned response to our support for a discussion about Medicaid reform simultaneous with encouragement of expansion. Stop now and go read it. Then come back.

Douthat makes some excellent points, with which we largely agree, and one point with which we do not.

America rarely just “tries out” major expansions of the welfare state: Rather, our history strongly suggests that programs in motion tend to stay in motion, and that the best time to change a potentially-dysfunctional system is before it gets entrenched — before interest groups organize themselves around perpetuating those dysfunctions, before voters become accustomed to the program’s guarantees, and before the political system learns to take its existence for granted and turns to other debates instead. Whereas once something becomes the Way We Redistribute, it’s both hard to pare back and harder to propose alternatives, no matter what the data ultimately show about the program’s actual effectiveness.

It’s true, as Frakt and Carroll note, that no alternative reform is likely to be implemented as quickly as Obamacare itself. But it’s also true that if you favor a substantially-different alternative, cheering on the law’s full implementation while participating in a “conversation about how to make [it] more efficient and effective” is likely to lead to that alternative being passed sometime around the Fourth of Never. And this reality means, in turn, that for all the dilemmas that the current state of the Republican Party creates for thoughtful opponents of the new health care law, they still have an obligation to oppose.

The “obligation to oppose” is an illogical leap from a premise with which we agree. It is true that it will be hard to change a redistributive program once it begins doing its redistributing. Path dependency is a real thing. The state of our health system and the difficulty in changing it demonstrates it.

But recognizing that does not by itself obligate anyone to oppose Medicaid expansion. It only suggests one should weigh its likely inertia and imperfections against the moral implications of doing nothing. We, ourselves, have confronted this dilemma. There are many things we’d like to change about Medicaid and other facets of our health system. We understand that it will be hard, and some things we dislike may never change. Yet we are not obligated to oppose any more than we’re required to support. One has to do the moral calculus for oneself. One can still come down on the side that doing nothing is preferable to expanding Medicaid in one of the forms currently permitted by law. But, by no means is one obligated to do so.

To suggest obligation is a direct appeal to political loyalty, is it not? If this is about politics then we will concede the point to Douthat and move on. But if we’re talking how to craft policy to help poor Americans, that’s a discussion in which we want to participate, but only if we shed this nonsensical idea that anyone is obligated to anything. Is this an open minded discussion or not?

We want to re-emphasize the context of this discussion. Medicaid expansion, and the ACA in general, isn’t arbitrary policy. It had the support of the a majority of the House of Representatives, a supermajority in the Senate, and the President of the United States, who ran with health reform as a major plank of his platform. And then he won re-election. The policy was even examined and modified by the Supreme Court, on which sit a majority of conservative-appointed Justices.

Moreover, it’s not like there isn’t room for movement within the law. Do you want more of a managed care option? That’s what Florida’s governor got. Do you want more private insurance options instead? That’s where Arkansas is headed. The Obama administration seems more than willing to negotiate. That some alternative to traditional Medicaid will be possible on the “Fourth of Never” is obviously not true. Beyond that, a future administration can use the same waiver process as the current one to further widen the scope of the possible. We doubt the next Republican president will take office on the “Fourth of Never.”

The ACA, and those who support it, want to make sure people who earn below 138% of the poverty line get assistance with the high price of health care that they could never pay out of pocket. Period. Opposing the Medicaid expansion – in any of its forms – would deny many poor people that help. Some even oppose Arkansas’s private option. They are denying people access to the type of private insurance that most of the rest of us enjoy because, yes, we can afford it. But we do so with a lot of government money. We give people at the higher end of the socioeconomic spectrum thousands of dollars a year in tax breaks for their employer-sponsored insurance. It seems reasonable that we should be able to do this for people far less fortunate than ourselves.

Just saying “no” is just that – saying “no”. We agree that Obamacare opponents need to do more than that. We don’t agree that they are obligated to oppose every policy option that isn’t exactly to their liking until they build a coalition in support of a preferred alternative. To conclude as much takes more reasoning than Douthat has offered.

@aaronecarroll and @afrakt

A few commenters have questioned my power calculation on the Oregon Medicaid study, claiming different results. Though I can’t be sure what they are doing wrong (if anything), I did take the time to do several more checks of my calculation. These are in the technical footnote to this post.* Even though it’s weedy, if you’ve followed this story this far, you might want to look. It shows how you can do power calculations at home, with no money down! Meanwhile, the offer stands: if you find an error in my work, please let me know, but read the footnote first.

The question has been raised about how the study’s blood pressure findings compare to that of the RAND Health Insurance Experiment. (Harold also discussed this.) First, let’s deal with power. The baseline rate of elevated blood pressure in the Oregon study was 16.3% and the point estimate of the effect of Medicaid was a reduction of 1.33 percentage points. These are both bigger than the blood sugar (glycated hemoglobin, GH, A1C) results, which was the focus of my power calculation. So, maybe the blood pressure analysis was sufficiently powered. We have a calculator. Let’s find out!* (Of course, the 95% confidence intervals give us an answer, but how underpowered is it?)

No, the blood pressure analysis was no more adequately powered than the blood sugar one. Even though the baseline rate is a lot higher, the hypothesized effect size isn’t. However, the study was powered at the 0.85% level to find a reduction in proportion of the population with high blood pressure of 3 percentage points (more than twice the point estimate effect size). See, power depends on what question you’re asking.

I’m told, but have not independently verified, that the RAND HIE did find statistically significant results on blood pressure. That study had a sample size of 7,700 across four levels of cost sharing and followed participants for 3-5 years. The design and analytic approach were different than the Oregon Medicaid study, which could explain a difference in statistical significance. Also, RAND’s effect size was larger.

About this, Kate Baicker, the lead author of the Oregon Medicaid paper, wrote me,

The confidence intervals of our estimates of the impact of Medicaid tell us what effect sizes we have the power to reject. This can be read off of our reported confidence intervals. Consider, for example, the case of blood pressure. Table 2 indicates that over 16 percent of our control group has elevated blood pressure. For diastolic blood pressure, we see in Table 2 that the lower end of our 95 percent confidence interval is -2.65 mm Hg. This means that we can reject a decline in diastolic blood pressure of more than 2.65 with 95 percent confidence.

For context, it is instructive to compare what we can reject to prior estimates of the impact of health insurance on blood pressure. In particular, the RAND Health Insurance Experiment – which varied only the generosity of insurance coverage among the insured and not whether enrollees had insurance at all, as in the Oregon Health Insurance Experiment – found a reduction of 3 mm Hg in diastolic blood pressure among low-income enrollees. Quasi-experimental studies (previously published in NEJM) of the one-year impact of the loss of Medicaid (Medi-Cal) coverage among low-income adults found changes in diastolic blood pressure of 6 – 9 mm Hg (Lurie et al. 1984, 1986). The estimates in Table 2 allow us to reject that Medicaid causes a decline in diastolic blood pressure of the magnitude of the effects found in these prior studies. (These RAND and Medi-Cal estimates are based on a sub-population in disproportionately poor health, so one might instead compare their estimates to our estimates in our Appendix Table S14c showing the impact of Medicaid on diastolic blood pressure among those diagnosed with hypertension prior to the lottery. For this group we can reject a decline in diastolic blood pressure of more than 3.2 mm Hg with 95% confidence).

I don’t know what else I can say about all this. If you want to know if the study could reject the possibility that Medicaid had no effect on the physical health measures examined at 2-years of follow-up with 95% confidence, the answer is “no.” At the same time, the sample size was too low to be able to do that for all but very large effects. That’s just a mathematical fact. For effect sizes one might reasonably consider appropriate (and that are certainly clinically meaningful), the study would have had to have been several multiples larger (a factor of five is what I get). Again, that’s just math.

Please stay for the technical footnote:

* TECHNICAL FOOTNOTE: In contrast to what most people may think, I largely post on TIE to further my own knowledge and understanding, not to convince anyone of anything. So, if anyone finds errors in what I’ve written, I’m happy for the correction. But, I also recognize that I’m posting for a wide audience, and so I worry about the validity of the content of my posts long after they’re public. I continued to worry about my sample size calculation yesterday and this morning.

To increase confidence I had not made a grave error, I did my sample size calculation two additional and independent ways. First, it turns out Stata’s sampsi can be used many ways to do the same thing. Some ways require less input than others, which is safer since it is always possible to misunderstand what the proper form of the input is. Nevertheless, no matter how I used sampsi, I got the same answer, which is comforting.

Second, I used an online sample size calculator for the difference in proportions. I used the one here, but if you Google around, you’ll find others. Again, I got the same result as with sampsi. I encourage you to try it yourself. Below is a screenshot of the inputs and outputs for the calculation in my post. The only thing I didn’t mention in my post is what alpha is. It’s the probability of rejecting the null hypothesis (that Medicaid had no effect) under the assumption that it is true, the “p-value” of an estimate. Typically one seeks a value of 0.05 or lower. (Super geeky aside, “power” is not the same thing as “p-value.” The former is the probability of rejecting the null when it is false, the latter of rejecting it when it is true.)

@afrakt

A follow-up to this post is here. It includes instructions on how to run your own power calculations. 

Kevin Drum:

Let’s do the math. In the Oregon study, 5.1 percent of the people in the control group had elevated GH [glycated hemoglobin, aka A1C, or colloquially, blood sugar] levels. Now let’s take a look at the treatment group. It started out with about 6,000 people who were offered Medicaid. Of that, 1,500 actually signed up. If you figure that 5.1 percent of them started out with elevated GH levels, that’s about 80 people. A 20 percent reduction would be 16 people.

So here’s the question: if the researchers ended up finding the result they hoped for (i.e., a reduction of 16 people with elevated GH levels), is there any chance that this result would be statistically significant? [...] The answer is almost certainly no. It’s just too small a number.

I plugged these numbers into Stata’s sample size calculation program (sampsi) to do a power calculation for the difference between two proportions. I found that the probability that we can reject the null hypothesis that Medicaid would have no effect on GH levels to be 0.35. Under ordinary interpretations of statistical significance, the null cannot be rejected. We knew this from the paper, and, hence, all the hubbub. (Never mind that we also cannot reject a much larger effect. The authors cover this in their discussion.)

The standard level of statistical significance is rejecting the null with 0.95 probability. Assuming the same baseline 5.1% elevated GH rate and a 20% reduction under Medicaid, what sample size would we need to achieve a 0.95 level of significance? Plugging and chugging, I get about 30,000 for the control group and a 7,500 treatment (Medicaid) group. (I’ve fixed the Medicaid take-up rate at 25%, as found in the study.) This is a factor of five bigger than the researchers had.

Now, caveats:

• I’m taking the baseline rate, 5.1% from the study itself. But we know it is estimated with some imprecision. Maybe a different, more reliable rate is available elsewhere, but I don’t know what it is. Suffice it to say, it would take a lot of error on this number to overcome a factor of five in sample size. Assuming, again, a 20% reduction due to the intervention, I calculated that if the baseline rate were about four times bigger (e.g., about 20% instead of 5.1%), then the sample in the paper would have been sufficient to reject the null at the 95% level.
• The analysis in the study is not as simple as a straight comparison of two proportions. There is some multivariate adjustment for observable factors. There are some tweaks to due to measurement of multiple outcomes and weighting for survey design. It’s also an IV analysis, which retains in the sample patients who were randomized to treatment (won the lottery) but didn’t enroll in Medicaid. (This actually decreases power.) For these reasons, my power calculation is not fully correct. But, still, it would take a lot to overcome a factor of five in sample size.
• It is always possible I’ve made an error. If so, I’m happy for someone to correct it. Below is my code and output, using sampsi. Anyone think I did something wrong?

Code

local r = 0.0093 /* absolute change in rate due to intervention, from the paper */

local p1 = 0.051 /* baseline rate, from the paper */

local p2 = `p1′ – `r’ /* treatment group rate */

local sd1 = sqrt(`p1′*(1-`p1′)) /* standard deviation of baseline rate */

local sd2 = sqrt(`p2′*(1-`p2′)) /* standard deviation of treatment group rate */

local n1 = 6000 /* control group size, approx from the paper */

local n2 = int(0.25*`n1′) /* treatment group size, approx from the paper */

sampsi `p1′ `p2′, sd1(`sd1′) sd2(`sd2′) n1(`n1′) n2(`n2′)

Output

Estimated power for two-sample comparison of means

Test Ho: m1 = m2, where m1 is the mean in population 1

and m2 is the mean in population 2

Assumptions:

alpha = 0.0500 (two-sided)

m1 = .051

m2 = .0417

sd1 = .219998

sd2 = .199903

sample size n1 = 6000

n2 = 1500

n2/n1 = 0.25

Estimated power:

power = 0.3517

UPDATE: Mentioned the power reducing effect of IV.

@afrakt