Last night I had a false start in updating power calculations for the Medicaid Oregon study. The final result with corrections and a new PDF is here. If you’re into this stuff, it’s worth a look. The bottom line is that the study was underpowered for the change in proportion with elevated glycated hemoglobin (GH) by a factor of 23. Yes, twenty-three. You can use what I posted to run the numbers for other outcomes yourself.
What happened last night is that I had failed to update the computation of R2, which requires some algebra or a simulation, neither of which I was prepared to do at 10PM. I’ve done both this morning and the results are documented in the updated post and PDF linked therefrom.
Let me now state where we are. With respect to the statistically insignificant physical health measures in the study, we now know they were very underpowered. The sample was too small even for much larger effects. This renders them statistically uninformative in general and, in particular, uninformative about whether or how much Medicaid improves physical health. Uninformative means just that. No new information. No resetting of priors is justified on this question.
We also know, from the authors’ discussion and from Aaron’s posts, that the results include changes in blood sugar and blood pressure that are not unreasonable to have expected clinically. Thus, the results — or these two anyway, but I suspect it generalizes — are not clinically informative either. Again, no resetting of priors is warranted.
Given this, for the physical health measures only, I don’t understand the rush I’ve noticed in people updating what they expected Medicaid could do. These results really shouldn’t do that if they are, as I’ve said, uninformative both clinically and statistically. How did people make these judgements the day after the study was published? It’s taken me and Aaron almost two weeks to chase things down. I think it is time for people to take another look at what this study is saying, at their own priors, and, yes, at their own biases.
What I think we’re seeing is a re-expression of everyone’s priors. This study is an opportunity to do that, but it doesn’t and shouldn’t change what they are. The claims that people should be changing sides from pro- to/from anti-Medicaid expansion just make no sense based on the physical health measures in this study.
Meanwhile, yes, this study reconfirms some large financial and mental health benefits that we knew about from last year’s paper from the group. I’m not sure that’s a prior-updating event either.
This was (is) an excellent study done by a smart and capable team of investigators. The results, to the extent they are meaningful, should be viewed as among the most credible possible within the context of the study (in/around Portland, over 2 years, Medicaid circa 2009). And yet, much too much is being made of the set of results that just don’t tell us anything new.
I’m happy to be corrected on any of this if I’ve overlooked or misinterpreted anything. As always, I am merely trying to be scientific.
UPDATE: I mistakenly wrote “30” instead of “23”. It was a very bad rounding from 22.9.
