• Limitations of Randomized Trials

    A 1998 paper by Joe Newhouse and Mark McClellan in the Annual Review of Public Health is insightful on the limitations of randomized controlled trials (RCTs) of clinical interventions. They note that a trial can only occur in the

    window of time where there is enough belief that a treatment is efficacious so that it is considered ethical to randomize patients to the treatment group but not sufficient belief in the efficacy of the treatment that it would be considered unethical to withhold the treatment.

    Even when a trial can be conducted, its results may lack generality because it is conducted in a setting (major teaching hospital, say) that differs from the norm. That is,

    the trial may demonstrate that a procedure is efficacious (i.e. obtains desired results under optimal conditions), but it will not necessarily show that it is effective (obtains desired results under typical or standard conditions). Or, somewhat related to this point, in the time since the trial was conducted physicians may have become better at performing a procedure, such that the results of the trial are no longer relevant to current practice. (Bold mine.)

    As if that weren’t enough, sometimes the population included in a trial differs from that which will (or is) actually treated generally. Women, children, the elderly, and individuals with comorbidities are among the sub-populations historically excluded from certain types of trials. That is,

    the results of the trial may have internal validity (comparisons between the treatment and control groups are unbiased for the population being studied) but not external validity (results do not necessarily apply to other populations). (Bold mine.)

    For all these reasons, well-designed observational studies can enhance our understanding of treatment outcomes by examining results over a broader setting and population than might be available in a RCT. Moreover, observational studies are less expensive, can be conducted more quickly, and are applicable in cases for which an RCT would be unethical or impractical, due to problems of recruitment, for example. Thus, as Newhouse and McClellan put it, “[t]he results of a well-designed observational study are useful even if the results of a clinical trial are available.”

    Comments closed
    • In this weeks JAMA, a letter to the editor addresses a Feb 3 commentary piece written by Michael H. Alderman on salt consumption and risk of vascular morbidity and mortality (CVA and MI). He has published on several occasions against regulation due to lack of data from RCTs (which he continues to advocate for in his responses–articulately and professionally). Conversely, the authors of the letter discuss observational data, which is the preponderance of evidence on the table at this time.

      The salt debate is fascinating both from an evidence and policy perspective. I continue to watch carefully as this plays out.

      • @Brad F – I haven’t read any of this literature. I’m trying to think of a good salt consumption instrument (anything related to salt intake that is arguably not related to health outcomes). One idea is to use the efforts to reduce salt in restaurant food in certain cities. Still, that’s pretty weak since folks can always add salt and people don’t get all their food at restaurants. Maybe this is a case for which an RCT is the only way to go.

    • Quick op-ed Alderman wrote in NYT that gets to the heart of the matter.

      Counterpoint by Marion Nestle.

      Both short and sweet and will get anyone up to speed.