There are days when I think that journals that publish their manuscripts behind a paywall shouldn’t be allowed to send out press releases. In essence, what they are doing is telling the world, “trust us”. We won’t let you see the paper, we won’t let you examine the methods, we’ll just tell you the results and make you assume they are are valid.
That’s crazy. Case in point: the paper I talked about yesterday comparing the survival from cancer in this country to Europe.
I’ve been pretty upfront about this for years, but I don’t spend too much time in general responding to comments. I read them all (I swear), and approve many of them myself. In fact, I think it’s all Austin and me doing the approving these days. But I’m busy enough in my regular job, and – you know – blogging, that I don’t have too much time to also be a full time comment moderator.
But when people start asking legitimate questions, I sometimes feel the need to defend my work. So I do. But now, in just 24 hours, I’ve likely spent more time defending my critique of this manuscript than the authors have spent defending the manuscript themselves. That’s just not ok.
It’s especially frustrating because I inevitably get the same questions over and over. When you guys don’t like a paper, you try and pick it apart. Did they control for other variables? Did they randomize? Did they have a control group? Etc. In an ideal world, I’d tell you to go check. That’s what I do. But I recognize that I have access you don’t, and so I do my best to reassure you that I am really pretty good at this, and that I checked to make sure those things were true already. At some point, you have to trust me. I’ve got a public track record of thousands of posts backing me up. I really am rigorous about vetting the research. It’s what I do.
Moreover, if a flaw in my argument is revealed to me, I post again, and invite discussion. I’m public. I’m approachable. I’m accountable.
Gated papers, ironically, are often not.
Do let me apologize in advance if I ever come off snarky in comments. I don’t mean to be. But sometimes you guys can be rude, and you assume I’ve missed something instead of asking me if I did. Or, even when you’re polite, you still imply that I’ve forgotten something basic, with no evidence, just because you disagree with me.
So I’m going to answer some questions that have been emailed or posted in the last 24 hours about this paper.
1) Are you saying that survival rates have no value at all?
No. Please go read this post.
2) Couldn’t survival rates be useful in [insert hypothetical study here that is NOT the study you are discussing]?
Yes. But that’s not what I’m discussing. In this study, I think they used survival rates inappropriately.
3) They said they controlled for lead-time bias.
They did say that. But the actual work of doing so is not even in the original manuscript. It’s in an Appendix. There are some bizarre typos in it, and it focuses on breast and prostate cancer, and I’m not convinced by it at all. Moreover, if they want to prove that all this money is curing disease (and many cancers can be cured), improvements would show up in adjusted mortality rates. They are getting data from sources that have mortality rates. Why not use the mortality rates?
4) If these conflict of interest issues were important, wouldn’t they have been in the press releases or available to those who don’t have access to the manuscript?
You would think.
5) The abstract mentions prostate cancer and breast cancer as the biggest gains, but it doesn’t say anything about the other cancers. If those two were essentially the whole story, and no improvement was observed for the other cancers, then it would very tempting indeed to dismiss the findings as an artifact of lead-time bias. What were the findings for the other cancers?
I’m not sure how Health Affairs would feel about me just reposting a gated chart. So I made my own as best I could from the paper. Here are the Additional US Survival gains compared to Europe. Bars to the right are where the US does better. Bars to the left are where Europe does better. When there is no bar there is no significant difference. Please note I’m leaving breast and prostate cancer out here , and I’m estimating AML and CML as best I can because they never reveal the actual numbers in the paper:
Do you see a clear argument on how the US kicks butt? Me neither. Also, I found it interesting that this study ignored lung cancer, which kills off more people per year in the US than breast and prostate cancer combined. Now I’ll add in breast and prostate cancer. For the purists, YES THE AXIS IS CHANGING. I have to change the axis to let you see how breast and prstate cancers dominate.
I’ll let you decide if these two cancers are the whole story.
6) Why do you hate the US or the US health care system so much?
I don’t. I love my country, and I have dedicated my working life to making the US health care system better. I’ll keep poking it with a stick. That’s how I show my love.
