From “Prespecified Falsification End Points: Can They Validate True Observational Associations?” by Vinay Prasad and Anupam Jena:
[A]nalyses in large data sets are not necessarily correct simply because they are larger. Control groups might not eliminate potential confounders, or many varying definitions of exposure to the agent may be tested (alternative thresholds for dose or duration of a drug)—a form of multiple- hypothesis testing. Just as small, true signals can be identified by these analyses, so too can small, erroneous associations. For instance, several observational studies have found an association between use of PPIs and development of pneumonia, and it is biologically plausible that elevated gastric pH may engender bacterial colonization. […] In light of the increasing prevalence of such studies and their importance in shaping clinical decisions, it is important to know that the associations identified are true rather than spurious correlations. Prespecified falsification hypotheses may provide an intuitive and useful safeguard when observational data are used to find rare harms.
A falsification hypothesis is a claim, distinct from the one being tested, that researchers believe is highly unlikely to be causally related to the intervention in question. For instance, a falsification hypothesis may be that PPI use increases the rate of soft tissue infection or myocardial infarction. A confirmed falsification test—in this case, a positive association between PPI use and risks of these conditions—would suggest that an association between PPI use and pneumonia initially suspected to be causal is perhaps confounded by unobserved patient or physician characteristics. Ideally, several prespecified false hypotheses can be tested and, if found not to exist, can support the main study association of interest. In the case of PPIs, falsification analyses have shown that many improbable conditions—chest pain, urinary tract infections, osteoarthritis, rheumatoid arthritis flares, and deep venous thrombosis—are also linked to PPI use, making the claim of an increased risk of pneumonia related to use of the drug unlikely. […]
[F]alsification analysis is not a perfect tool for validating the associations in observational studies, nor is it intended to be. The absence of implausible falsification hypotheses does not imply that the primary association of interest is causal, nor does their presence guarantee that real relations do not exist. However, when many false relationships are present, caution is warranted in the interpretation of study findings.
More on falsification tests here.
