• The Oregon Medicaid study and the Framingham risk score

    Yesterday I posted results of power calculations for the Oregon Health Insurance Experiment’s (OHIE’s) analysis of cholesterol. Previously I posted power calculations for glycated hemoglobin and blood pressure. The study was underpowered for all of them, with respect to effect sizes one should have expected based on the literature. I promised to come back to the Framingham risk score (FRS), provided I could find enough information and had time.

    Well, I had time, but the information just isn’t there, so far as I can tell. What I need is some base of literature that robustly suggests how big a decrease in FRS one should expect and for a population comparable to those analyzed on the OHIE. As the OHIE NEJM paper explains, FRS was examined for three groups.

    The Framingham risk score was used to predict the 10-year cardiovascular risk. Risk scores were calculated separately for men and women on the basis of the following variables: age, total cholesterol and HDL cholesterol levels, measured blood pressure and use or nonuse of medication for high blood pressure, current smoking status, and status with respect to a glycated hemoglobin level ≥6.5%. Framingham risk scores, which are calculated for persons 30 years of age or older, range from 0.99 to 30%. Samples sizes for risk scores were [1] 9525 participants overall, [2] 3099 participants with high-risk diagnoses, and [3] 3372 participants with an age of 50 to 64 years. A high-risk diagnosis was defined as a diagnosis of diabetes, hypertension, hypercholesterolemia, myocardial infarction, or congestive heart failure before the lottery (i.e., before March 2008). [Numbers in brackets added to identify the three groups.]

    Here’s a PDF of research on FRS that TIE assistant extraordinaire Jaskaran Bains dug up. As you can see for yourself, it’s pretty thin and all over the map as to what type of group is examined. If you’re aware of any other relevant studies, let me know in the comments. Unless we obtain more information, I can’t do a power calculation without making some weak assumptions that will drive the result. I don’t want to do that because it’s just not credible.

    It could very well be that we have no good prior on FRS. If that’s the case, then the OHIE at least tells us what FRS effect sizes we can rule out: basically anything outside its error bars. That’s meaningful, though the error bars on FRS straddle zero, so we can’t tell if Medicaid had no effect or if the study is underpowered. I’m happy to come back to this if and when I know more.


    • In other words, the Oregon study didn’t have the controls of a clinical trial so its results aren’t very useful for clinical purposes. Sure, it’s useful for social science purposes (such as determining motivations for signing up with Medicaid), but not for medicine. The social sciences have a place in medicine, but not for determining what makes people sick, better, or die.

    • Excellent analysis, the most rigorous I’ve seen in this debate. A commenter on a previous post of yours on this topic (May 13th on GH) had asked if it would be possible to calculate the effect size needed given 80% power and the actual sample size, and I’m curious if that’s something you’re still working on for these 4 measures. I think that would be a more intuitive way to illustrate the degree to which the study was underpowered, and it would certainly help my own understanding.

      If so, thanks, and looking forward to the results!