• My reply to Jim Manzi

    In a follow-up to my EconTalk discussion with Russ Roberts about the Oregon Health Insurance Experiment (OHIE), he interviewed Jim Manzi about it this week. Russ invited me to submit a written response to the interview, which you’ll find below. It is linked to from the EconTalk page for the Manzi interview. I have no substantial disagreement with most of the content of the interview. The purpose of this reply is to add more information, not to debate any points.

    Like Jim, I did not have any numerically specific prior views about how much expansion of Medicaid would affect the physical health of non-elderly adults over two years. However, as Jim pointed out, the OHIE investigators suggested that we compare their diastolic blood pressure change results to findings from specific, prior work. Since my conversation with Russ, my colleague and physician Aaron Carroll examined that prior work and shared his thoughts in two posts. He concluded that, for a variety of reasons, we should not have expected the OHIE to reveal the size of change observed in those prior studies, the approximately 5 mm Hg in diastolic blood pressure change that Jim suggested as a rough average. You can read the details at the links for yourself.*

    A key point is that blood pressure reduction should only be expected in a population with initially elevated blood pressure, which was the focus of the prior literature referenced above. In contrast, the headline OHIE result is for all study subjects, only a small percentage of whom had elevated blood pressure at baseline. Unfortunately, there is no reported OHIE subanalysis focused exclusively on subjects with hypertension at time of randomization. Depending on which metrics from the published results you examine, between 3% and 16% of the sample had elevated blood pressure at baseline. Taking the high end, 16% x 5 mm Hg = 0.8 mm Hg is in the ballpark of a reasonable expectation of the reduction in diastolic blood pressure the OHIE could have found (it was also the study’s point estimate) were it adequately powered to do so. Was it?

    I worked with Aaron and fellow health economist Sam Richardson on this question. We found that the study had 80% power (the standard minimum for clinical studies) to detect a change in diastolic blood pressure of 2.82 mm Hg. Put another way, this means that the probability of failing to detect a true change of this size, the false negative rate, is 20%. For the more reasonable, expected 0.8 mm Hg change calculated above, the probability of a false negative is about 86%, or 14% power. This is underpowered by any reasonable criterion.

    But that’s just diastolic blood pressure. What about another measure? In the discussion of their paper, the investigators calculated the reduction in glycated hemoglobin level one might have expected from the clinical literature, 0.05 percentage points. That’s well within the 95% confidence interval of their estimate and corresponds to a false negative rate of 75%, or 25% power. So, the study was underpowered for this measure too.

    Aaron, Sam, and I have calculated power for other physical health measures reported from the OHIE and will share results soon. If you can’t wait, I have posted methods so you can do power calculations yourself. This is possible because power analysis methods are well-known and all necessary parameters are readily available in the published paper. The only difference between the methods I’ve posted and what Aaron, Sam, and I are doing is that we are incorporating a few higher-order nuances, like adjusting for the effect of the study’s survey weights.

    Now for the lightning round. Here are a few quick responses to other aspects of the interview:

    • Jim noted that 40% of lottery winners didn’t apply for Medicaid coverage. As discussed, this might be due to an expectation of low value from Medicaid or lack of follow-through skills (jointly, low prudence). However, half of the 60% who did apply were deemed ineligible. The investigators report that this was largely due to income above the 100% FPL threshold, but other potential reasons include moving out of state, securing other coverage within a six month look-back period, or aging out of eligibility. One might reasonably presume a similar proportion (half) of those who did not apply would have also been ineligible. Perhaps some knew that to be the case and spared themselves the fruitless exercise of completing the forms. It seems reasonable to me that those capable of weighing the value of Medicaid would also know whether their incomes were too high, they moved out of state, they secured other health insurance coverage, or were too old. Therefore, it is likely that substantially fewer than 40% of the non-applicants suffered a lack of prudence. Judging from the proportion of applicants deemed ineligible, perhaps the number of imprudent non-applicants is closer to 20%. This is speculation, but no less plausible than that Jim or Russ offered.
    • The RAND Health Insurance Experiment was not a study of health insurance coverage since it did not include any uninsured subjects. It was a study of cost sharing, capped at $1,000 (circa mid-1970s dollars) for all participants.
    • The OHIE depression reduction result was not observed largely or entirely in the first month after enrollment. The investigators didn’t conduct a depression screen in a one-month survey, but did in later surveys, as Adrianna McIntyre explains. However, self-reported health did improve substantially in the first month.
    • To what extent the findings are informative about Obamacare’s Medicaid expansion would be an excellent topic of discussion. Neither Jim, Russ, nor I got into this question very deeply. It’s properly a question of external validity, not bias, which is something else.

    In conclusion, I applaud Russ for devoting two episodes to the OHIE. It is an important study, both for its subject and methods, and it deserves at least that much attention. I also praise Jim for his addition of substantial value to the conversation. I hope I have helped clarify a few points.

    * Aaron’s posts largely focus on systolic blood pressure, though diastolic is mentioned and is also included in the cited studies. Suffice it to say, the same issues of expected effect size and insufficient power arise for systolic blood pressure as I discuss for diastolic. I focused on diastolic because it is what Jim mentioned and the lead investigator emailed me about.

    @afrakt

    Share
    Comments closed
     
    • One sentence in the above post says volumes about the limitations of means tested government programs…………I am referring to the reasons why many applicants for Medicaid were rejected: “moving out of state, exceeding 100% of FPL, securing other coverage in a 6 month lookback period…”

      Compare this to Medicare, only of course there is no comparison. No one loses Medicare because they move, or earn an extra $1000 (which they need to eat and pay rent), or have other coverage.

      I fear that some parts of the ACA will develoip the same way. Already we are seeing some workers being kept away from subsidies because their employers cover them and their children, yet not their spouses. We are seeing the manipulation of the 30 hours rule that the Senate put into the bill. And so on.

      Bob Hertz, The Health Care Crusade

      • Bob Hertz, you are right in your first paragraph and that is a problem. Dr. John Goodman had proposed some ideas to fix the problem with Medicaid that you are well aware of, but many consider that a right wing agenda instead of recognizing it as a humanitarian way of solving at least part of the mess we are in.

        As far the exchanges, maybe the ideas behind much of the ACA were faulty and the writers should have paid more attention to the details.

        Additionally as mentioned in an earlier thread [Finkelstein Study of 2005 "Our results suggest that, in its first 10 years, the establishment of universal health insurance for the elderly had no discernible impact on their mortality."] maybe we are placing too much emphasis on insurance and not enough on common sense.

        The ACA appears to be an extreme intervention that demands near first dollar coverage, demands fixing administrative costs, limits competition, etc. Such demanding type of legislation from our government means a considerable reduction in our freedoms. It has hurt many Americans including those that will now lose their insurance that they were happy with.

    • Thanks Emily.

      However, the only Americans I know who are happy with their insurance are those who get it for free (i.e. the employer or union pays the entire premium).

      Medicare recipiients are also happy with their insurance, if they have the funds or good luck to have a wraparound Medical Advantage plan.

      I am an agnostic on whether the ACA is correct to insist on first dollar coverage for some preventive care. The goal of these provisions is to avoid the grotesque incidence of people needing major surgery because they did not have the money to pay for preventive care. What is problematic is whether this is something that has to be addressed with health insurance.

      I say (and have said for a long time) that the reason for health insurance is financial protection. Whether insurance makes us any healthier is beside the point to me. Frankly, any health statistics in a broad population depends on how much testing is done. You can always find indicators of bad health if you do enough testing.

      • Bob, No one gets their insurance free from their employer. It is built into the compensation package. Unfortunately many do not know the true cost. However I think at the time the ACA was passed most were reasonably satisfied. I don’t think as many will be as satisfied after the law is in full swing especially if they are informed of the true costs. That is something we will have to wait for to prove one way or another.

        However, I think there would have been a lot more satisfaction if everyone could have purchased high deductible insurance and pocketed the difference.

        IMO First dollar care is almost an absolute waste except if it involves diseases that can be transmitted to others.

        “I say (and have said for a long time) that the reason for health insurance is financial protection.”

        If that is the case there is little reason to force people to buy insurance. Those that have assets will risk losing their assets and future earnings. Those that are poor can’t afford the premiums so they will need assistance one way or the other. The premium and costs for a large group will most likely be greater than the sum of the costs if that group were not insured.

    • I must confess that some of this feels like Charlie Brown must have felt when Lucy moved the football…

      I thought we were going to Obamacare to make us healthier – hence we cover a wide range of not really prove preventative procedures – just in case…

      We cove things that are really not going to put us at great financial risk – first dollar coverage in many cases.

      We take away medical savings accounts and high deductible catastrophic coverage options – that allow those with the resources to negotiate for lower prices [cash will often get you 50% or more off of list price].

      But now it seems we are simply concerned with “saving” folks from financial ruin.

      Would it not be better to focus on changing the bankruptcy laws to smooth the escape from medical expense driven bankruptcy than to insist that healthy young males subsidize the care of relatively better off older folks.

      I am confused.

      LL

      • I understand what you’re saying, but I don’t understand what it has to do with this post.

        • Sorry if I am off topic…

          I am honestly confused with some of what I have been reading hear and elsewhere. I spent the time listening to the Manzi podcast and find myself even more confused. I think we get lost in the details – “prudence” and underpowered dominate the discussion – when I think the real issues get shoved aside – such as…
          1. Why did the funds that were spent on the lottery winners in Oregon NOT have a significant and large effect – should we not expect more from the spending of “public” funds?
          2. What did the “losers” do to maintain their health – clearly they found the solutions they need to stay even with the “winners”. To me this was a major finding in Oregon that nobody much talks about. A large number of people were able to STAY as healthy as the “control” group with NO public funding. To me this makes the case for both Obamacare and Medic,aid expansion a bit shaky.
          3. It seems that many want to fall back on the “financial security” improvement as the justification both OC and ME – but as I said in my first post, if that is what we are trying to accomplish we might have found a better, easier way to do so that would be simpler and cheaper.

          Again, sorry for being so dense.

          • You can’t even begin to address 1-2 unless you appreciate whether the study was powered (had sufficient sample) to detect the expected effects on health. It didn’t.

            Falling back to 3 doesn’t make sense unless one has really learned something new. And, if that is the case, a different policy option is certainly worth considering. Though, even there, one should weigh the political obstacles and realities. The counterfactual is not some other ideal system. It very well could be no additional assistance for poor Americans. Whether that is OK or not is a personal, subjective judgement.

            • But power as I understand it is connected to effect size…

              Had the Oregon study found large reductions in the measures, the sample size might have been sufficient – or am I missing something?

            • You’re missing a notion of how big an effect size one should have reasonably expected. You can always say, “If only the effect size had been larger.” As I’ve now shown, for several measures at least, we already knew the effect size was likely to be too small to be detectable. Hence, this study is not informative on physical health. People are either attempting to objectively update ill-informed priors or merely repeating their subjective views on well-informed priors. Either way, there is, in fact, nothing new here.

          • Lonely Libertarian, you are not so alone on this issue:

            Weren’t there15 different criteria? Wouldn’t one think they would see a blip? As mentioned earlier this is not the only study to produce this type of result. We have Finkelstein and McKnight:

            [http://www.nber.org/papers/w11609.pdf?new_window=1] “Our results suggest that, in its first 10 years, the establishment of universal health insurance for the elderly had no discernible impact on their mortality.”

            Similar points were made by Rand and others.

            • Oof. It doesn’t matter how many criteria there were if the sample size is too small for all of them.

              Because of that, you’re doing the right thing by relying on prior work, which, as you know, I summarized recently.

            • “Because of that, you’re doing the right thing by relying on prior work, which, as you know, I summarized recently.”

              Of course you summarized that work and used Meltzer’s table just as I used Meltzer’s table to refresh my memory of the Finkelstein and McKnight study. So I was twice doing the right thing twice and did the same thing as you. I copied the abstract of Finkelstein and McKnight and posted it on the blog. We are both genius’s :-)

              Yes, your claim is that sample size was too small, but not everyone feels that way and I’ll make a bet some well regarded studies that you may or may not have agreed with might have been underpowered as well. But, maybe even in an underpowered study when we break things down to 15 units we might want to see a blip. A huge amount of money is involved and no blip. Other evidence reveals the same type of thing even if they were long enough and had the power, no blip. I would say the contention with regard to Medicaid is blip less :-)

            • Nope. Whether the study is underpowered for expected effect sizes (based on the prior literature the authors suggest) is not opinion. It is mathematical fact. Sorry.

              Also, I will not make any use of any study that is underpowered for effect sizes we might reasonably expect. I am totally consistent about this.

              You may attempt to disprove either of these, if you wish to waste your time. It will require math and evidence, not assertions.

            • Underpowered is one way of looking at things. Thus you will weight what you call a low powered study less than another study. But, add a few low powered studies together plus a few high one’s with the same conclusions and the conclusion becomes more likely, even likelier when the attempts to prove the opposite fail.

              I am happy to hear that you are consistent and I assume trying to evaluate studies fairly despite what you might believe at the onset of your search. It’s a tough job because so many well accepted high powered studies really should be weighted quite low because of problems other than power and perhaps more significant than power.

              Thus I don’t concern myself with proving the power calculation wrong. In this case I accepted it as correct from day one and counted it as a negative when drawing my own conclusions as to which side to place my bet on. I’m not fixed to one side or the other for though I am familiar with the subject I am neither an expert on it nor researching this particular topic. But, in the end based upon the evidence and lack of evidence I have heard about Medicaid’s performance I have to lean on the side believing that Medicaid insurance doesn’t offer the health benefits it was intended to provide.

            • I don’t have an issue with your assessment. I do have a goal of trying as best I can to help people understand research. I confess a sense of frustration when I observe a failure to communicate that understanding. This post is specifically about the Oregon Medicaid study (with one very brief mention of a fact about the RAND study). Most of the post is about how it was underpowered to detect the changes in physical health we should have expected (your “the health benefits it was intended to provide”). That specifically means that even if those health benefits exist, the study could not reliably distinguish them from no benefit due to small sample. If you understand that one point and use it to inform your thinking and commenting, I will be satisfied.

            • The whole reason behind the study, mechanisms of the study and everyone’s interest is that the federal government is in tremendous debt so we all want to make sure as much value can be obtained from each dollar possible while giving the maximum benefits to all our citizens. Along with listening to each and every point I am trying not to lose the big picture. If the big picture was not of interest this study would not be under such scrutiny and likely would have been of little interest.

              I am sure others that have discounted the power calculation did so with good reason. Health care studies, as you well recognize, are not black and white because we are dealing with human beings, long time frames, and many variables where each study is in danger of not recognizing the hidden variables. We have gone down the wrong path many times because of that lack of recognition.

              This study doesn’t stand alone either on your site or elsewhere. As stated earlier I found your power calculation to be compelling, but not compelling enough to overcome all the flaws that we see in all studies. I don’t think I posted when the subject was about power calculations because that didn’t involve other subjects.

              Please, don’t think I didn’t get your point. I did and it is exactly the type of point I am looking for. That is what makes one want to come back as long as there is a reasonable balance of ideas.

            • I agree that close examination of trees can overlook the forest. At the same time, far too many (I’m not saying you) observe only the forest and don’t recognize that the properties of the trees that comprise it sometimes suggests something different than a pure forest-level view does.

              Related, there is almost no other place anywhere for a discussion of the important property of the trees. We can engage on the forest, but that’s a conversation that’s well covered elsewhere. The value added of this blog is a closer examination of the details that few bother to appreciate (again, not saying you!).

              So, forgive me for sometimes trying to say very focused. It’s absolutely essential that I do that. Because almost nobody else will. I can’t take every post up to the 30,000 foot level. Sometimes, yes. Always, no.

            • “I agree that close examination of trees can overlook the forest.”

              It appears that the titles of the individual threads indicate that we were mostly examining both and thus I don’t think there was a significant amount of posts that missed the target. It was admirable for you to take the time and focus specifically on the trees which you did, but those threads where that was specifically done for the most part dealt with that problem alone.

              I for one was happy to hear all these discussions even though from the placement of the threads, their titles and content it might appear that some of that was written to defend a position. That might be why you feel that the target was missed, but I appreciated your argument because the best thing that can be accomplished is argument and counterargument. Who won, who lost? I don’t know and don’t care for needed information was provided on the blog.

              I wanted to see the sequence of the blogs so I went back to the archives figuring it would be an easy task. It wasn’t and proved my contention above. Here is part of the list. Pardon me for exceeding the space limit.

              *Bias and the Oregon Medicaid study
              *The Oregon Experiment, Irrationality, and Universal Coverage (not yours)
              * Updated power calculation (I didn’t comment)
              * For economist/biostats geeks (I didn’t comment)
              *A podcast on the Medicaid study
              *How much could we expect the Oregon Medicaid study to reduce blood pressure? – ctd. (not yours)
              * How much could we expect the Oregon Medicaid study to reduce blood pressure? (not yours)
              *For economist/biostats geeks – ctd. (I didn’t comment)
              * Oregon and Medicaid – how the debate has changed (not yours)
              *On the obligation to oppose
              *For economist/biostats geeks only (a bleg) I didn’t comment)
              *What about power for the blood pressure result? (And so much more) (I didn’t comment)
              *Health insurance is good for health (not yours)
              *Power calculation FAQ
              *Power calculations for the Oregon Medicaid study (I didn’t comment but Chris Conover made some important points with regards to Rand and power calculations)
              *Power calculations for the Oregon Medicaid study
              * Why not give cash instead of Medicaid?

              And finally I gave up trying to list them all with this one that says it all and explains why the forest and the trees have to be viewed together.

              *Oregon Medicaid experiment “is a Rorschach test of people’s views of the ACA”

              My power calculations register a very big blip on this subject. I never would have started to list them all if I knew from the beginning how many thread titles I would have to copy. Understand, no one was off topic because the essence of the debate was highlighted by this study where you posited your views and many including Chris Conover provided theirs and he expanded his in Forbes.

            • This is what the tags are for. Here’s the tag for the Oregon study posts: http://theincidentaleconomist.com/wordpress/tag/oregon-health-study/. Tags are at the bottom of every post.

            • Austin, I understand your use of tags, but I wanted to make it easy for you to see how the discussion was much greater than the trees because it sounded as if you had some disappointments. I tired out and couldn’t list them all.

              I think in the end the following title said it all: “*Oregon Medicaid experiment “is a Rorschach test of people’s views of the ACA”” I think you should be quite pleased that your discussion on power blossomed into a very vibrant and informative discussion where your point was made and heard.

    • I’ll admit I only got to listen to the first half of the Manzi interview today. (Give me a break…it’s a holiday and I have two young kids.) I’ll listen to the second half tomorrow, but what struck me in the first half is he seems to be thinking that selection into applying for Medicaid is biasing the results in a way that it isn’t.

      He kept saying that with the analysis of treatment-on-the-treated (TT), the authors were comparing the subset of lottery winners who actually signed up for Medicaid to all of the lottery losers. But this isn’t what the study’s authors were doing: they were comparing all lottery winners to all lottery losers, and then scaling up their estimates to account for non-compliance (ie not ending up on Medicaid, despite winning the lottery). The authors’ approach gives an unbiased estimate of the effect among compliers (those who would enroll if they won the lottery and would not end up on Medicaid if they lost the lottery).

      And in this case, the relevant policy is making people eligible for Medicaid. We should only care about effects on those who would actually sign up, because those who don’t sign up cost the state nothing. So the TT approach is the relevant one, though I will grant that other ways of increasing eligibility for Medicaid might make a different population actually sign up.

    • Austin
      The point you raise in bullet #1 merits attention. Manzi states in the podcast he considered the possibility, but the number of folks falling into the low prudence drop out category was low, or he suspected it (dont recall). Have you substantiated your suspicions via the investigators?
      Brad

      • I don’t recall this nuance to the interview. As far as I know we don’t know anything about the eligibility of lottery winners who did not apply. Is that the question? I could double check this with the investigators (as could anyone), but I haven’t to date.

        • Transcript at 8:23 mark:

          (Clarifying my above comment, Manzi states authors believe movement to another insurance product a small effect:)

          Guest: That’s right. And I think that some of the possible reasons beyond literally just saying, I’m not going to bother, are: they are moving frequently. So, it’s hard to contact them by mail. Another reason is they could have for one reason or another moved onto some eligibility for some other insurance. I think the authors think that second cause would be a relatively small number. But there are also, virtually certainly some of them who in fact the mail did arrive at the house where they lived and for some reason they didn’t submit it. I really went to great pains to say I wasn’t trying to make some kind of a judgment, a moral judgment. This isn’t about a moral judgment about these people who didn’t respond. It is purely about a marker for behavior which in my view is intuitively correlated with plausible failure to comply with chronic disease management regimen.

          • There are some results in the study about participants being on other insurance products. I’d have to look back at the details to comment further. But this is just one of many ways to fail to be eligible. So, I’m not sure it would settle anything if Manzi were correct or incorrect.

          • If many dropped out due to the ineligibility factors, either too much income or other insurance, doesn’t that demonstrate the mobility of what we consider the poor and tell us again that we might be spending a lot of money on a group that might not need the assistance?

    • It’s a bit embarrassing to listen to Russ Roberts fail to wrap his head around the concept of a p value.

    • I read recently that 140,000 persons in Tennessee made less than the FPL and had private insurance through work.

      Honestly, I thought the number would be closer to zero. Perhaps these persons worked part time in factories where everyone was covered.

      That is just of general interest.

      But I do have a further comment.

      The debate about health status of Medicaid enrollees seems to miss a point. People get onto regular Medicaid because of low incomes, not because of health problems. (I am excluding those who are disabled and get on Medicaid.)

      Medicaid is not a high risk pool. It is bound to include large numbers of healthy persons, because eligibiilty is based on income. It seems that a lot of debate has missed this point.

    • 1. The results are evidence just not much evidence.

      2. If lowering blood sugar and blood pressure is a goal why would you not subsidize medications for Diabetes and hypertension or make them available for the poor at the health department like vaccinations are?

      Because government is stupid.

      • Actually a large number of those with Hypertension can get there meds at Wal Mart for about $40/drug per year – if they are on a two drug regime that is less than what they might pay for Netflix – and FAR less than the annual cost of the Oregon Medicaid coverage.

        Throughout the evolution of Obamacare into the monstrosity it currently is I wrote – emailed – cajoled my rep and Senators to propose simple, manageable first steps – like the one you suggest and to consider easing the way for those with catastrophic costs to get these discharged without losing their homes/pensions

    • So let me try a different approach….

      Can we agree that all of the research to this point supports the conclusion that our healthcare system pays a lot for very little (some might say nothing, but I will settle for very little).

      • Is it not clear that I agree there is a lot of waste in our health system?

        • Perhaps that is why I keep coming back to this blog.

          I think we have a lot in common in how we view the current state – and I know you are way more engaged and knowledgeable than I am about all of this.

          Where we differ is I want to go to smaller, simpler, less progressive solutions. You seem to want to go elsewhere – or am I missing something?

          • I’d be comfortable with the assessment that I seem to be open to a wide range of options, place a bit more emphasis on equity at the expense of efficiency than some, and take political constraints and practicality more seriously than others.

            However, I’d also say that all of that is largely irrelevant, as my principal objective is to read and interpret the research literature as honestly, completely (strengths, limitations, and all), and objectively as possible.

            • “more emphasis on equity”

              Health care efficacy and benefits are hard enough, but now you are adding equity which is even more difficult to define. You are indeed a brave soul. The utilitarian view might not favor that poor unemployed person without the gumption to get a job or even the sickest one for the capital expended and removed from the capital markets might reduce productivity and thus taxes and funding so that the people as a whole have less funds to satisfy their needs.

            • We share a desire for a more “equitable” [fair?] system. In fact the things that scare me most about the ACA are some pretty big built in inequities.
              1. Forcing the young and healthy to pay for the old and wealthy.
              2. Forcing all to pay for the sexual practices of the relatively few – birth control and abortion coverage?
              3. Compelling compliance via a government agency that is already feared by many – for very good reasons.
              4. Giving big pharma a pass by forcing excessive drug coverage in the plans and continuing it in Medicare [and I assume, but am not sure Medicaid].
              5. Covering things that should be available – but discretionary.

            • There are many dimensions of equity. Like you, I can think of some approaches that would not include the inequities you mention. In some form, they would be fine with me!

              There are also things that states could do today to increase fairness under current (and foreseeable future) law, in my view. One of them is Medicaid expansion in some form as an alternative to no expansion whatsoever. Notice, I am not suggesting that is the best policy relative to all other possible policies. I am suggesting only that I prefer expansion to non-expansion given the reality that the limited options available today will not change for years.