• Oregon and Medicaid and Evidence and CHILL, PEOPLE!

    This is a joint post by Aaron Carroll and Austin Frakt. Relevant to this post, recently we have published three papers arguing for expansion of Medicaid, not relative to all possible other reforms, but relative to the status quo.

    First of all, we’re somewhat annoyed that the NEJM sent out press releases and the study to journalists, but not people like us, because we now have to rebut the gazillion stories that have already been written on a study we just found out about an hour ago. Maybe they should let some knowledgeable people see it early, too. Or just wait until it goes live to tell everyone. But we digress. Let’s get into it.

    To recap: Oregon ran an RCT of Medicaid, because of a lack of funds to expand it fully. Early results showed some promising evidence that Medicaid improved process measures, self-reported health, and enhanced financial protection. This update, at 2 years, was intended to give us some harder outcomes. The results are “mixed”:

    We found no significant effect of Medicaid coverage on the prevalence or diagnosis of hypertension or high cholesterol levels or on the use of medication for these conditions. Medicaid coverage significantly increased the probability of a diagnosis of diabetes and the use of diabetes medication, but we observed no significant effect on average glycated hemoglobin levels or on the percentage of participants with levels of 6.5% or higher. Medicaid coverage decreased the probability of a positive screening for depression (−9.15 percentage points; 95% confidence interval, −16.70 to −1.60; P = 0.02), increased the use of many preventive services, and nearly eliminated catastrophic out-of-pocket medical expenditures.

    Let’s review. The good: Medicaid improved rates of diagnosis of depression, increased the use of preventive services, and improved the financial outlook for enrollees. The bad: It did not significantly affect the A1C levels of people with diabetes or levels of hypertension or cholesterol.

    This has led many to declare (and we’re not linking to them) that the ACA is now a failed promise, that Medicaid is bad, and that anyone who disagrees is a “Medicaid denier”. How many people saying that are ready to give up insurance for themselves or their family? If they are arguing that Medicaid needs to be reformed in some way, we’re open to that. If they’re arguing that insurance coverage shouldn’t be accessible to poor Americans in any form, we don’t agree. Medicaid may not be perfect, but we don’t think being uninsured is better. This new study supports this view, though certainly not as strongly as it might have.

    From our full reading of the paper, let us add the following to the conversation:

    1) Improvements in mental health are still improvements in health outcomes. The rate of positive screens for depression dropped from 30% to 21% in the Medicaid group. The rate of medication use for depression went from 16.8% to 22.3%. It wasn’t statistically significant (though it was close, p=0.07), but that doesn’t mean Medicaid failed. Which leads us to…

    2) Non-statistical significance does not mean failure. It means that either (a) there is no treatment effect or (b) the study is underpowered. Since there does not seem to be a power calculation, we can’t tell. How much of a difference would there need to be in order for statistical significance? We can’t tell. But just because this difference wasn’t significant in with the sample studied doesn’t mean it wouldn’t be significant with a larger sample. Indeed, the authors note this in the discussion:

    Hypertension, high cholesterol levels, diabetes, and depression are only a subgroup of the set of health outcomes potentially affected by Medicaid coverage. We chose these conditions because they are important contributors to morbidity and mortality, feasible to measure, prevalent in the low-income population in our study, and plausibly modifiable by effective treatment within a 2-year time frame. Nonetheless, our power to detect changes in health was limited by the relatively small numbers of patients with these conditions; indeed, the only condition in which we detected improvements was depression, which was by far the most prevalent of the four conditions examined. The 95% confidence intervals for many of the estimates of effects on individual physical health measures were wide enough to include changes that would be considered clinically significant — such as a 7.16-percentage-point reduction in the prevalence of hypertension. Moreover, although we did not find a significant change in glycated hemoglobin levels, the point estimate of the decrease we observed is consistent with that which would be expected on the basis of our estimated increase in the use of medication for diabetes.

    This is important, because the point estimates show that blood pressure did fall in Medicaid. Sure, it was a small amount. Medicaid lowered the percentage of people with elevated blood pressure from 16.3% to 15%. (p=0.65). It also increased the chance of being on meds from 13.9% to 14.6%. Remember that A1C “failure”? The percent of people with diabetes with a high A1C went from 5.1% off Medicaid to 4.2% (p=0.61). The percent of people with high total cholesterol went from 14.1% to 11.7% (p=0.45). In all of these, Medicaid improved the numbers, but not in a statistically significant manner. Was it powered to detect these differences? Moreover, what should we expect? Which brings us to…

    3) What is reasonable to expect? How much does private insurance affect these values? Do we know? No. There is no RCT of private insurance vs. no insurance. No one claims we have to have one. We just “know” private insurance works. The RAND HIE did not compare insurance to no insurance. It just looked at cost-sharing of insurance. That’s not the same.

    There has never been an RCT of Medicare vs. no insurance either, though we could point to some suggestive observational work (admitting that is not the same thing).

    So Medicaid, and Medicaid only, needs an RCT to prove it works. Never mind that it’s just intuitive that easier access to health care (by any insurance means) seems likely to improve your chance of getting it and getting it when you need to keep you healthy, if not alive.

    4) Financial hardship matters. Here Medicaid shined. It hugely reduced out of pocket spending, catastrophic expenditures, medical debt, and the need to borrow money or skip payments.

    5) Preventive care matters. We’ve been cautious about the ability of prevention to save money. But some preventive care improves outcomes. More people on Medicaid got colonoscopies, cholesterol screenings, and prostate cancer screens (whether or not you support them). The percent of women over 50 who got mammograms doubled from 28.9% to 58.6%. Results once again weren’t always “statistically significant”, so people can claim “Medicaid failed”. But colonscopies in people over 50 went from 10.4% to 14.6% (p=0.33). Failure?

    6) Health insurance is necessary, but not sufficient to improve health. It’s just the first step. We have never claimed that quality would go up just because of the ACA. Access will improve. We need to do a lot more work to improve quality. And, yes, maybe that will require a change to how Medicaid operates, but will quality improve if more poor people don’t have access to the means to afford care? We don’t see how.

    7) Most of these measures are still process measures. A1C is a marker. So is cholesterol. Did real outcomes change? Patient centered ones, like health related quality of life, did. Did mortality? Did morbidity? We still don’t know. That would take more time to see.

    So chill, people. This is another piece of evidence. It shows that some things improved for people who got Medicaid. For others, changes weren’t statistically significant, which isn’t the same thing as certainty of no effect. For still others, the jury is still out. But it didn’t show that Medicaid harms people, or that the ACA is a failure, or that anything supporters of Medicaid have said is a lie. Moreover, it certainly didn’t show that private insurance or Medicare succeeds in ways that Medicaid fails.

    People claiming otherwise need to go read the study and rebut these points.

    @aaronecarroll and @afrakt

    Share
    Comments closed
     
    • Brilliant. ‘Nuff said

    • Very good post.

      The continuing problem with much medical and social science research is the failure to distinguish between statistical insignificance and substantive insignificance.

      Now that the study is done, we don’t need a power analysis — we can simply look at the numbers and the standard errors, and look at the probabilities of different effects. However, because most of the effects you cite are proportions, it would be easily possible to simulate what a likely power analysis would have looked like prior to looking at the data.

      The relevant issue from a benefit cost analysis is: given the estimates, what is the expected ratio of benefits and costs, and what are the probabilities of different net benefits.

      Does any of the analysis in the paper look at joint significance of all these effects? Even if individual effects are insignificant, what is the statistical significance of all these effects given the various inter-correlations? Or alternatively, if we put a dollar weight on different outcomes, what is the statistical significance of expected net benefits?

    • Spin. ‘Nuff said.

      Health care is for sick people. Wellness means eat a balanced diet, exercise, don’t smoke and drink in moderation if at all.

      • I agree.

        Why are we paying doctors 150k per year to tell people stuff they already know? Eat smart, get exercise. We could hire high school people at 1/10th of that cost to do exactly the same thing.

    • Guys

      I posted the same at Don’s blog:

      I just read Annie Lowrey’s post
      http://economix.blogs.nytimes.com/2013/05/01/what-the-oregon-health-study-cant-tell/

      I read the study, but cant make heads or tails of her statement:

      “The study presents strong evidence that Medicaid recipients spend more on health care, and not just because of pent-up demand: they just seem to spend more, full stop.”

      Exposed group spent $1700 or a third more than controls. On what does she base “pent up demand” vs excess above pent up demand, ie, “just seem to spend more?”

      Did I miss something in the study discussion or tables?

      Brad

    • -Anyone know if the authors measured the magnitude of the mental health improvements over time? I believe that Robin Hanson claimed that something like 2/3 of the mental health benefit for the enrollees materialized before they’d had their first visit with a provider of any kind.

      That’s consistent with my interpretation, which is that you can’t use this study to disentangle the effects of Medicaid enrollment and the clinical efficacy of any health interventions delivered therein from giving people access to private health insurance with an equal actuarial value, or simply giving them health care vouchers or cash worth the same amount. Medicaid coverage might indeed lead to superior outcomes relative to the above alternatives in many or all dimensions, and be it might be possible to demonstrate this with other studies in the future,. All this study can tell us, thus far, is that some coverage is better than none.

      .
      -”If they’re arguing that insurance coverage shouldn’t be accessible to poor Americans in any form, we don’t agree.”
      Who, exactly, is making this argument?

      • “Who, exactly, is making this argument?”

        Anyone who is blocking Medicaid expansion in a state without championing an alternative that will actually pass. This is very common. You must have heard.

        “something like 2/3 of the mental health benefit for the enrollees materialized before they’d had their first visit with a provider of any kind.”

        That may be somewhat stylized (can’t recall precisely), but it is roughly the right thrust. The authors addressed this in one of their prior papers. I believe I wrote about it in some post. You could try searching TIE on Oregon and Baicker.

        • -Ah. Your conclusions about the practical implications of their policy stances rather than something that a real person has actually said. That’s fine – my thought was that I would certainly have heard of any legislator making such statements if they had actually done so.

          -I’m kind of surprised that the subject of Medicaid coverage and childbirth comes up so rarely in these discussions of whether there’s a net benefit from Medicaid coverage or not (I may have just missed them). Here in Washington state, roughly 1/2 of all births are covered by Medicaid, and unless you believe that modern obstetrics, pre and post natal care, etc has zero health benefit for either the mother or the child then you pretty much have to concede that a program that enables women who might not otherwise see a healthcare provider at all prior to delivery is saving lives, reducing acute complications and persistent disabilities in children, etc, etc, etc relative to a situation where they had no coverage at all.

          I think that catastrophic plans with zero deductible for high-value, low cost interventions coupled with HSAs that are either totally or partially financed by the government would be a superior alternative to Medicaid in many dimensions, but I’ve never been able to give any credence to the claims that Medicaid is worse than having no coverage at all. The Medicaid population is so much sicker and more dysfunctional than any other cohort that I can understand why it’s difficult to design studies that show an unequivocal benefit relative to any other group of people (who will invariably be healthier and higher functioning), but it’s puzzling that this argument is taking place on empirical/statistical turf when the clinical efficacy of modern medicine has been established beyond all dispute, which should have rendered claims that giving people access to it has no health benefits logically indefensible.

          Are there better ways to organize and deliver care than Medicaid? Most likely yes. Is the access and delivery of care via Medicaid better than no coverage of any kind – at least for the poor and dysfunctional – of course.

          • Wow. I never thought the logic “If you don’t give health insurance to people who can’t afford health insurance, then they won’t have health insurance” was so complicated and controversial that we simply cannot assume that it holds true and people who oppose Medicaid cannot be expected to have thought of this consequence.

            I’m all for giving people the benefit of the doubt, but at some point you have to just assume that people who run state governments just can’t be that stupid, collectively, while everyone is pointing out the truth. At some point you just have to say, no, they’re lying, they know what’s going to happen.

            Sure it’s a judgment call, but it keeps us from going down the rabbit hole of “Did that representative know that voting for tax cuts would decrease taxes?” or “Did that senator know that voting for the ‘Authorization of War Act’ would authorize a war?” While possible in theory, it just isn’t consistent with the real world I observe, where these people wear shoes with laces and can find their ways home without sewing their addresses to their coats.

    • I don’t understand your call for a “power calculation” — such a calculation only makes sense in the design phase of the study, and it’s pointless after the data has been collected. And where would you get the numbers for such a power calculation? You can’t use the observed point effects from the study itself; such a post-hoc power analysis is a well-known statistical fallacy. But if you really want an answer: assuming the point estimates that were observed are reasonably accurate, then a failure to achieve statistical significance means that you had an insufficient sample size, and hence were underpowered.

      I don’t think it’s plausible to argue that the weak findings of the study were due to small numbers. There were approximately 6,000 in each arm of the study, so even an outcome measure with a low prevalence such as A1C (5%) would result in 300 subjects in each group, and that’s plenty for detecting a reasonable effect size. No, the main reason why statistical significance was not achieved was because the effects were small.

      Forget about the issue of statistical significance. If we assume that the observed point estimates are really the true population values, what conclusions should we draw from this study? I agree that the reduction in depression is indeed a clinical measure, and that must be viewed as a success, but I’m afraid that the study is also providing strong evidence that Medicaid is at best leading to only incremental improvements of at best a few percentage point for many other outcomes.

      • … over 2 years.

      • No one, article or posts, seems willing to address the basic function of “statistical significance”- that is, why do we place so much value on the 95% confidence interval. The obvious answer is to eliminate random data fluctuations as the reason for difference in outcomes. All the postulations about small sample size accounting for lack of significance may be true, or it is also possible those differences do not exist and instead we are looking at artifacts based on random error.

        • By saying the study is underpowered, one is precisely saying that one cannot distinguish between those two hypotheses. This was raised in one of our posts explicitly.

    • I’m not sure if you understand the difference between statistical and clinical significance or how p values are calculated. You insinuate the effects were positive and clinically significant when in fact none were. That’s the point. They confirmed the null hypotheses.

      • Marcus C,

        The authors of this study include three of the top ten-or-so health economists in the country; if you think they’ve made an elementary mistake, it is vastly more likely that you haven’t followed what they’re doing.

        Read the footnotes to Table 2. This isn’t a simple difference in means test, but reports estimates based on two-stage least squares methods (which correct for the problem that not everyone who won the lottery actually enrolled in Medicaid). They also have sampling weights to adjust for the probability of being sampled. Both of these procedures increase standard errors.

        For details, see the supplementary appendix: http://www.nejm.org/doi/suppl/10.1056/NEJMsa1212321/suppl_file/nejmsa1212321_appendix.pdf .

        People interested in power should really be looking at the confidence intervals, which tells you what sorts of magnitudes might be plausible.

        George, you NEVER confirm the null hypothesis. You only fail to reject, and then you can think critically about whether you failed to reject because the effect is probably very small or because you simply have very little power. Confidence intervals tell you a lot about how precise the estimates are.

        • Right. And before anyone could actually say “Medicare does not work” they’d need to run tests with a null hypothesis that the impact of medicare was greater than 0, and find results that enabled them to reject the null.

    • Outstanding post. This post should settle the debate. Of course it won’t. Misuse of the concept of statistical significance will be half way around the world before you got your boots on. But you were fast, totally right and utterly wonderful.

      See also Kevin Drum’s further thoughts amplifying the main point of this post (he fair use copied a key table from the actual article which is behind a paywall)

    • I suspect that if these individuals had been given private insurance instead that the results would probably be similar. The question in my mind is: what does the increase in costs actually get you, and is that worth it? That question needs multiple answers, from multiple distinct perspectives: enrollee, payor, and society.

    • It would be nice if studies like these included a section on statistical power and cost-benefit analysis. In particular, if it is possible for the benefits to exceed the costs and still be statistically insignificant, then the results really don’t offer much policy advice at all.

    • Austin-

      The entire thrust of the President’s campaign for the ACA was about ‘paying for what works’ and not paying for what doesn’t. Evidence based medicine and all that…

      This study- peer reviewed- shows that, in the aggregate, Medicaid failed to meet the claims and expectations of those who suggest that opposing a $1 TRILLION spending program over 10 years (assuming estimates correct, which we both know will likely be off significantly) means opponents want people dying in the streets and are not compassionate.

      If supporters preach evidence based medicine– it should certainly apply to the health care policies as well–

      The results suggest to me:
      1. massive expansion should be put on hold until other ‘medicaid alternatives’ can be studied.
      2. agree that RCT for Medicare and private insurance also be tested BEFORE massive $1 TRILLION in exchange subsidies spent

      in other words, the entire premise of the ACA is brought into question– and the entire premise of the Democrats push– ‘Republicans want sick people to die quickly’ should be condemned by even the law’s supporters.

      Additionally, it brings into question criticisms of the Republicans’ ‘alternative’ ideas (of which there are many, some good, some not so good) as ‘inadequate’.

      Ultimately, your analysis is consistent with my conclusions– more data before more spending.

      • “…more data before more spending.”

        So we can assume you’re giving up health insurance for yourself and your family until it si proven effective by a RCT?

    • There was a pretty strong selection bias, wasn’t there? Many people refused Medicaid even when it was offered, so it’s likely that those who got onto Medicaid needed treatment the most.

      Interesting for another reason too, b/c the research I’ve seen on pent-up demand indicates that there’s not much impact on newly insured use of health care resources.

    • Your political inclinations are recoiling in horror at the implications of the empirical data.

      Every attempt to cut Medicaid results in extravagant rhetoric about how millions are being virtually sentenced to death. At the very least your study demonstrates that such claims are political in nature and not supported by the facts.

      Now, under the ACA, we are being forced to buy a huge new fleet of cars which may or may not run. Maybe we should find out before we sign the check.

      Is anyone studying what people are doing with their $ 2500 per family savings from ObamaCare?
      http://www.youtube.com/watch?v=_o65vMUk5so

      • Woudl you give up your health insurance? I thought so. Think about this: even in countries with national health care (and for people with health care here in America) people are getting fatter, lazier, and sicker. That will change when we change our habits. Insurance will not change habits. But it will do what it is supposed to do: make health care available to you; thus treating you with human dignity and respect. Of course, less financial worry is crucial; especially when the leading cause of bankruptcy in America is medical costs

        If health insurance made one healthy simply by its virtue then the 65-70 percent of Americans who have proper health care should be healthy as can be. They are clearly not. Duh.

        The problem is that the ACA was partly- if not mostly- pushed as a way to save lives. Maybe in the long run over decades it will; but its biggest benefit is helping people SUFFER LESS by being treated.

        What is wrong with that?

        • @elboku
          Maybe you should listen more and talk less. I don’t have health insurance, haven’t had it for ten years. You do not speak for me or for the uninsured, only yourself. If your pathway to “human dignity and respect” is to have the government force others to buy a product they don’t want to buy that is your personal problem. Glory in your own self righteous fascism.

          What’s wrong with that? Nobody made you the arbiter of everyones’ well being or spokesperson for the uninsured, so stop trying to pretend you hold the post.

          • Unless you have managed to save enough money to pay for your health expenses or you simply do not go to the doc or you will just allow yourself to die/suffer if you get sick, well the rest of us will be forced to pay for your medical care either through higher costs and/or taxes.

            There is no free ride. You are not an island. This is not “Lord of the Flies’ wherein only the strongest survive.

            Unless you are prepared to allow millions of Americans who are either uninsured or under-insured to be refused medical care and thus suffer and/or die, then we all need to pitch in. There is plenty of money in America to treat people’s medical conditions. We choose not to do so.

            • So you admit it has nothing to do with “human dignity and respect” as you originally claimed. Your concern is you might have to pay for someone else’s medical bills if they aren’t insured. Yet you have no problem coercing taxpayers to pay for other peoples’ medical care through Medicaid.

              The most likely explanation, once again, is that you simply enjoy forcing others to do what you want them to do. And are upset that the data doesn’t justify your authoritarian demands.

            • @DaMarv,

              Human dignity and respect comes in when you protect someone from having to choose between a house and medical care, or between lifesaving care and bankruptcy. You are apparently relatively healthy (I’m guessing on the cheap side of the 80/20 rule) and you definitely have a high risk tolerance at this stage of your life. Others are not healthy, whether through congenital conditions, chronic conditions or acute conditions. They need care that they can’t afford, or can’t without severe trade offs. And also, many just have a lower tolerance for risk and being unprotected causes them stress and anxiety.

              The improvement in mental health appears real, if tentative. That is one indication that it is sensible to treat the sense of security and financial protection provided by health insurance as a matter of human dignity and respect.

    • There is no way to spin these Oregon results as anything other than extremely disappointing. I urge you to stop straining to defend Medicaid expansion and instead devote your efforts to thinking about alternatives with more promise.

      • People on Medicaid will no more get healthier than people on private insurance- why would it be any different? If the goal of health insurance is to make people healthier, well, in America it has been a miserable, abject, total, absolute, horrific failure.

    • The proverbial “nothing to see here, move along” post.

      The study, two years in, is a catastrophic indictment of the “faith based” unscientific premises of the Obamacare expansion.

      • “The study, two years in, is a catastrophic indictment of the “faith based” unscientific premises of the Obamacare expansion.”

        Apparently I’m going to have to read the study for myself, because I have seen nothing ANYWHERE to warrant characterizing these results as a “catastrophic indictment”. I wonder if Apetra is the type of person who would describe luke-warm coffee as “a crime against humanity”.

    • The effects actually are statistically significant in many cases (e.g., HDL levels & overall cholesterol). The authors just don’t know how to test for the difference in proportions correctly.

    • For example, the authors show (Table 2) that the proportion of patients with high cholesterol is 14.1 in the control group and 11.67% in the medicaid group. The authors claim that this is a non-significant difference (p=.34) but in fact this difference in proportions is statistically significant (at p<.01) and would represent a 17% reduction in the rate of people with high cholesterol. Any doctor would be thrilled with such an improvement in the population.

      • I think you’re wrong in saying that this difference between 14.1% and 11.7% is statistically significant, and the authors are correct to say that the differences are not statistically significant. You may be assuming that the study is simply comparing a treatment group that gets Medicaid with a control group that does not. But in fact the study is comparing a treatment group that is OFFERED Medicaid with a control group that does not. Only about one-quarter of those offered Medicaid end up signing up. They correct for this issue, but this correction means the effective sample size is much less than the nominal sample size. In addition, the study corrects for the fact that not everyone responded to the survey, plus the fact that people in the control group began enrolling in Medicaid later in the study. All of this drives the effective sample size down further.

      • MarcusC, I checked that calculation too but it turns out to be more complicated since they are not just comparing means between the control and treatment groups. That is, since most of the people (58%) who were selected by the lottery don’t enroll and some (18%) who weren’t selected do enroll, the effective sample size becomes much smaller (and more difficult to report since the differences are tested through modeling with an instrumental variable). You might think that it gets small enough that it cannot detect meaningful differences for some outcomes, as you note.

        • Peter,
          Thanks for the clarification. If you are right (you appear to be) then their effective sample sizes for each comparison should have been reported more clearly in Table 2 (rather than indicating an N of over 12,000 in their notes section). The observed effects when measured as d statistics are quite large for many variables suggesting that their effect size must have been really small to get their reported p-values.

          Unfortunately the don’t report that and the take-home message will be “massive study based on over 12,000 individuals shows no effect” – rather than “positive effects were observed but our effective sample size was rather small”.

          • Marcus C,

            First, the article is still accessible to me.

            Part of the trouble here is that every quantitative field has its own reporting conventions, and this is an economics paper. It’s remarkable how different the language and practice of statistics can be across fields, for models based on the same underlying frequentist approach.

            Part of the trouble here is that NEJM articles are so incredibly short. The caption to Table 2 is being asked to do the work of 15 pages worth of a methods section. You really need to read the supplemental appendix. I expect the authors will publish a much more extensive analysis in an economics journal, and I rather wish that we had that more extensive analysis available now. In the mean time, you can get a much better understanding of the underlying framework and their methods if you read the authors’ Quarterly Journal of Economics piece on their first-year results.

            This study is based on randomization, but it doesn’t use the same methods as comparing a directly randomized control and treatment group. It’s a “randomized experiment” in the sense that the randomization element gives it the same ability to draw causal inference, but it takes different methods. People were not randomly assigned to Medicaid; they were randomly assigned to the right to apply for Medicaid. There was no pre-screening of lottery entrants to determine who was actually eligible; only people who won the lottery actually went through an application process to assess eligibility. The control group thus includes a lot of people who wouldn’t have been actually eligible for Medicaid (but who were probably quite poor. The authors thus go beyond simply comparing the treatment and control groups and instead model the selection process using instrumental variables, the workhorse method of econometrics. Table 2 reports estimated means from a two-stage least squares. The randomized lottery makes the instrument an extraordinarily good one, but it is still an instrument, and that means the formulas for difference-in-means tests go out the window.

            (I really do NOT understand the people calling for a power analysis. The precision of the tests is provided by the confidence intervals. You can read off of the CI’s exactly how large an effect size the study ended up having the power to detect. If the effect sizes found were not statistically significant, then it is ex post clear that the study was not powered to detect effects of this size.)

            • I agree with your assessment and thank you for taking the time to draft such a very detailed response. I still find it concerning that the reported estimated means in Table 2 often indicate fairly substantial effects that would be practically very meaningful (e.g. a near 20% reduction in the proportion of patients with high cholesterol) but that are interpreted as not significant. How much bigger would these effects have had to be in order to reach conventional standards of statistical significance? A 50% reduction? Is such a reduction even remotely possible in any 2-year medical intervention?

    • Weird.
      I wrote to the authors earlier today with my concerns and questions and now the article can no longer be accessed via the NEJM website. How odd.
      Probably just a glitch.

    • I just wanted to state that I think your assessment of the study is very impartial and objective. I like that you admitted the weaknesses it revealed in medicaid, but that you also explained that it is simply a single study with lots of limitations.

      I agree that we need to continue to do research on the effectiveness of these programs to help influence policy making.

    • A1c <6.5?

      Not that I am immersed in the literature but I have never seen that standard. In fact, my understanding is that for many populations there is concern that trying to push A1c below 7 can actually be harmful. In many cases <8 is considered adequate control.

      Did I miss something?

    • Too early but interesting.

    • This study has been around for a long while as an NBER working paper.
      Here is the link: http://www.nber.org/papers/w17190.

    • “4. Financial hardship matters.”

      Who can deny this?

      But surely the right takeaway is Peter Suderman’s at Reason.com:

      If the primary goal of a program like Medicaid is to protect individuals from financial shocks associated with medical expenses, then why not support a far, far cheaper subsidized catastrophic insurance program instead of low-deductible insurance through Medicaid? If what the poor really need is financial protection, rather than health services, then why not just give them cash?

      • Ian: Because another factor that matters is price negotiation. Being on Medicaid allows insured people to essentially negotiate as a bloc to bring down often-nonsensical base rates that exist even for basic primary care visits.

      • @Ian: people on Medicaid as a rule are close to the poverty line. They don’t have a lot of disposable income, and so a catastrophic policy with a $5,000 deductible (or similar) would not function the same way that a $5,000 deductible would function for a middle class person. Just to stick with the financial well-being angle, the poor generally don’t have $5,000 in the bank. They live paycheck to paycheck, if they have a paycheck, and so such a policy could bankrupt them, or force them to sell a home, etc. It would be disruptive and even when there is just the threat of disruption I would expect there to be a mental health “hit.”

        Maybe you meant that allowances would be made for certain types of preventive care, checkups and medications for chronic diseases, etc. That makes more sense, but that certainly shouldn’t be called a “catastrophic” policy.

        Also, like with health insurance generally, the bulk of the costs are for a small percentage of people with very serious conditions who require expensive treatment far above the deductible. Savings would be modest.

        Finally, it always should be mentioned in these discussions: other developed nations keep costs under control far better than the US has without going the route of catastrophic policies for the poor (or anyone, really). High deductibles are not how other nations control spending, and we already have more out-of-pocket cost-sharing than they do, so it is far from obvious that this is the path we should take.

        • Jonathan,

          Just to correct you on one point, most OECD countries have much higher out of pocket spending than the US. Also frequently employed are national drug formularies and other restrictions. They tend to ration healthcare much more than we do.

        • @ Jonathan

          The catastrophic health insurance I envisage would not ask for any deductibles for the poorest, and I endorsed cash grants. I hope those clarifications meet your objections.

          More seriously, I don’t think commentators have grasped how revolutionary the Oregon findings are. In response to the study, the fallback position of advocates of Medicaid expansion like Paul Krugman has been to argue that it misses the point that the primary goal of a program like Medicaid is to protect individuals from financial stress associated with medical expenses, But if those medical expenses have no benefits, then no one should pay for them (poor or rich), and so the raison d’etre of Medicaid disappears because their are no financial pressures to insure against. (The exception is catastrophic insurance, of course).

          For a long time, one of the standard criticisms of health care in the US has been that we spend more on it than other countries and get worse results. I think the Medicaid study hints at the reasons why. If the goals is to improve health, then we may have grossly misallocated our health care dollars. An expansion of Medicaid will only compound that mistake.

          If follow-up investigations corroborate the Oregon study, then I hope we will have the guts to perform radical surgery on our health care finance system, meaning replacing it with catastrophic coverage.

    • 2) Non-statistical significance does not mean failure. It means that either (a) there is no treatment effect or (b) the study is underpowered.

      Or (c) that the determination of “significance” is based on a completely arbitrary standard, selected without any real thought as to its implications.

    • “Financial hardship matters. Here Medicaid shined. It hugely reduced out of pocket spending, catastrophic expenditures, medical debt, and the need to borrow money or skip payments.”

      What matters most is freedom. Medicaid takes money from some (harming their financial situation) and spends it on behalf of others, restricting their freedom to control their own healthcare decisions and spending. This is unequivocally harmful to everyone involved.

      Lots of things in life matter (food, for example), but using government bureaucrats to “provide” such things by imposing their will on people is no solution – it destroys the very thing individuals require to live their lives: freedom.