• Survival, mortality, and the frustration of gated papers

    There are days when I think that journals that publish their manuscripts behind a paywall shouldn’t be allowed to send out press releases. In essence, what they are doing is telling the world, “trust us”. We won’t let you see the paper, we won’t let you examine the methods, we’ll just tell you the results and make you assume they are are valid.

    That’s crazy. Case in point: the paper I talked about yesterday comparing the survival from cancer in this country to Europe.

    I’ve been pretty upfront about this for years, but I don’t spend too much time in general responding to comments. I read them all (I swear), and approve many of them myself. In fact, I think it’s all Austin and me doing the approving these days. But I’m busy enough in my regular job, and – you know – blogging, that I don’t have too much time to also be a full time comment moderator.

    But when people start asking legitimate questions, I sometimes feel the need to defend my work. So I do. But now, in just 24 hours, I’ve likely spent more time defending my critique of this manuscript than the authors have spent defending the manuscript themselves. That’s just not ok.

    It’s especially frustrating because I inevitably get the same questions over and over. When you guys don’t like a paper, you try and pick it apart. Did they control for other variables? Did they randomize? Did they have a control group? Etc. In an ideal world, I’d tell you to go check. That’s what I do. But I recognize that I have access you don’t, and so I do my best to reassure you that I am really pretty good at this, and that I checked to make sure those things were true already. At some point, you have to trust me. I’ve got a public track record of thousands of posts backing me up. I really am rigorous about vetting the research. It’s what I do.

    Moreover, if a flaw in my argument is revealed to me, I post again, and invite discussion. I’m public. I’m approachable. I’m accountable.

    Gated papers, ironically, are often not.

    Do let me apologize in advance if I ever come off snarky in comments. I don’t mean to be. But sometimes you guys can be rude, and you assume I’ve missed something instead of asking me if I did. Or, even when you’re polite, you still imply that I’ve forgotten something basic, with no evidence, just because you disagree with me.

    So I’m going to answer some questions that have been emailed or posted in the last 24 hours about this paper.

    1) Are you saying that survival rates have no value at all?

    No. Please go read this post.

    2) Couldn’t survival rates be useful in [insert hypothetical study here that is NOT the study you are discussing]?

    Yes. But that’s not what I’m discussing. In this study, I think they used survival rates inappropriately.

    3) They said they controlled for lead-time bias.

    They did say that. But the actual work of doing so is not even in the original manuscript. It’s in an Appendix. There are some bizarre typos in it, and it focuses on breast and prostate cancer, and I’m not convinced by it at all.  Moreover, if they want to prove that all this money is curing disease (and many cancers can be cured), improvements would show up in adjusted mortality rates. They are getting data from sources that have mortality rates. Why not use the mortality rates?

    4) If these conflict of interest issues were important, wouldn’t they have been in the press releases or available to those who don’t have access to the manuscript?

    You would think.

    5) The abstract mentions prostate cancer and breast cancer as the biggest gains, but it doesn’t say anything about the other cancers. If those two were essentially the whole story, and no improvement was observed for the other cancers, then it would very tempting indeed to dismiss the findings as an artifact of lead-time bias. What were the findings for the other cancers?

    I’m not sure how Health Affairs would feel about me just reposting a gated chart. So I made my own as best I could from the paper. Here are the Additional US Survival gains compared to Europe. Bars to the right are where the US does better. Bars to the left are where Europe does better. When there is no bar there is no significant difference. Please note I’m leaving breast and prostate cancer out here , and I’m estimating AML and CML as best I can because they never reveal the actual numbers in the paper:

    Do you see a clear argument on how the US kicks butt? Me neither. Also, I found it interesting that this study ignored lung cancer, which kills off more people per year in the US than breast and prostate cancer combined. Now I’ll add in breast and prostate cancer. For the purists, YES THE AXIS IS CHANGING. I have to change the axis to let you see how breast and prstate cancers dominate.

    I’ll let you decide if these two cancers are the whole story.

    6) Why do you hate the US or the US health care system so much?

    I don’t. I love my country, and I have dedicated my working life to making the US health care system better. I’ll keep poking it with a stick. That’s how I show my love.

    @aaronecarroll

    Share
    Comments closed
     
    • Now you are sounding like many of my patients. If I don’t have answers immediately available, they get upset, even though they could look some of their questions up on the internet. But, I understand I have access to some information, or at least can look it up faster, than they do. I’ve seen thousand of patients over the years, and I think am pretty good at this, and you’d think people could trust me with that experience, but I’ll do the work as I can. And some patients are just plain rude. They just plain disagree with me.

      Sorry fot the satire, but I think everyone lives in the same boat here.

    • For wonk-nerds, posting a .txt West Wing transcript is like a shibboleth for believing in the power of positive antagonism. Adorable.

    • Someone has (cough) posted the paper here:

      http://pnhp.org/blog/2012/04/11/is-higher-u-s-health-care-spending-on-cancer-worth-it

      It’s a poor effort of course, not least that the data is old – there is much more recent data from the 2000s. The paper that did make a convincing case for US survival benefit in some cancers is:

      but it’s interesting to note that for the first time data from the US National Program fo Cancer Registries was added to the SEER data, and the US lead in some cancers then came down, a trend that would probably continue the more data is added from more and also poorer areas of the US as the dataset is still less than 50%.

      As the authors say:

      “Part of the international variation in survival is thus probably
      attributable to under-investment in health resources.
      The variation in survival might be considered intuitively
      obvious, given wide global variation in expenditure on
      health care, whether that is expressed in absolute terms or
      as a proportion of national resources. A parallel could be
      drawn with diff erences in survival between rich and poor
      patients with cancer in a given country, which have
      frequently been reported.”

    • Thanks for your work. I was glad to learn that my skeptical reaction to news reports of the study has some basis in reality.

    • Skepticism seems a common-sense stance from which to regard any findings that compare rate changes that span years and countries and measures and disease categories. The current report being a case in point: The bulk of the US survival gains stemmed from two cancers (prostate & breast) which had seen a sharp increase in North America (but not Europe) during the study years, which, even before the increase, were two of the most common cancers (especially among younger adults who have lots of survival years left if cured), for which screening is effective and which are relatively responsive to treatment. The apparent gains in the US could easily be due to a rate-denominator inflated by the increased incidence of a common, highly treatable cancer rather than a decrease in the rate-numerator representing cures. The underlying differences in trends and incidence rates of the different cancer rates across countries must be accounted for in any such comparison. Finally, aside from whether or not earlier screening is associated with a “lead-time bias,” earlier screening is certainly associated with cancer survival and thus, cannot be excluded from such an analysis.

    • As noted, lung cancer kills more Americans than eitiher breast cancer or prostate cancer.

      Why then do we hear so much more about breast and prostate cancer? Perhaps because, according to the CDC “black men
      are more likely to die of lung cancer than any other group.” http://www.cdc.gov/cancer/lung/statistics/race.htm

      Cigarette smoking is the leading cause of lung cancer, and low-income adults are far more likely to smoke than their more affluent peers. They may explain why, even though lung cancer kiills more people “lung cancer does not receive as much attention, both in funding and in public awareness, compared to other types of cancers.”

      We’ve “raised public awareness” of prostate cancer and breast cancer to a point where average- risk women have a greatly exaggerated sense of how likely it is that they will die of breast cancer, and a-symtomatic men over-react when told that they have “early stage prostate cancer.” When I say they over-react, I am referring to the fact that for many, “watchful waiting” would be the most prudent course. But instead, many men feel that Someone Must Do Something Now– so they submit to unnecessary treatments that can lead to life-changing side effects.

      Meanwhile, two huge and extremely lucrative industries have been built up around our fear of breast and prostate cancer.
      That the economist who wrote this Health Affairs paper was supported by a drug company that makes a cancer drug raises a red flag. The whole thrust of his paper is to suggest that we can never spend too much on these two cancers.

      Thanks to extensive screening, we’re diagnosing more prostate cancer and breast cancer, but it is not clear that the actual incidence of breast cancer and prostate cancer has increased.

      As Gilbert Welch and other have observed, over-diagnosis leads to diagnosis of “pseudo-disesase” in the form of breast cancers that would have disappeared as well as diagnosis of “early stage prostate cancer” in asymptomatic patients who never would have experienced symptoms. If they live long enough, most men die with prostate cancer, but not of prostate cancer.

      But over-diagnosis and over-treatment of these two cancers has become extremely profitable. Corporate interests have helped drive
      the “public awareness campaigns” that have made so many patients so fearful– and so vulnerable to over-diagnosis and unnecessarily invasive treatments. See Welch’s book: Are You Sure You Want To Be Tested for Cancer?