Discrepances Between Meta-Analyses and Subsequent Large Randomized, Controlled Trials

Reviewers: Leonard A. Farber, M.D., and John Chang M.D.
Source: The New EnglandJournal of Medicine, August 21, 1997 -- Volume 337, Number 8:536-542

From the research center, Hôtel-Dieu de MontréalHospital, and the Department of Medicine, Faculty of Medicine,University of Montreal.

In order for clinicians and patients to achieve an understandingof results of treatment studies presented in journal articles, theymust first understand the nature of the type of trial being conducted. This is applicable to those whose interestresides primarily in the field of cancer, and to the whole ofmedicine as well. At present, the gold standard for the evaluation ofefficacy of clinical intervention is generally considered to be alarge randomized, controlled trial. Clinicians may also turn tometa-analyses to provide evidence to support clinical strategies,especially if a controlled trial in not available. Attention has beendirected toward the intrinsic weakness of meta-analysis, especiallywith regards to the incorporation of biases of individual studies andtheir selection, as well as the heterogeneity among them.

There is, to date, a paucity of data systematically comparing theresults of meta-analyses of several small trials with those of largerandomized, controlled trials. The authors, LeLorier et al.,identified and analyzed the results of large randomized, controlledtrials that involved at least 1000 patients published from 1991through 1994, and in four specific journals: the New England Journalof Medicine, the Lancet, the Annals of Internal Medicine, and theJournal of the American Medical Association. These were compared withresults of meta-analyses published earlier on the same topics. Thefindings were then analyzed as to whether the findings of therandomized trials were in agreement with those of the correspondingmeta-analyses.

Twelve large randomized, controlled trials and 19 meta-analysesmet the criteria for the authors' review. Some trials andmeta-analyses reported on more than one outcome, and therefore atotal of 40 outcomes were studied. Overall, they found importantdiscrepancies between the two modalities. Agreement between themeta-analyses and the large clinical trials produced kappa values ina range considered to be "fair." In addition, in each case ofdisagreement, a statistically significant effect of treatment wasfound by one method, whereas no statistically effect was seen by theother.

The issues addressed in this paper are of importance forclinicians in all fields, as well as interested readers both withinand outside of the medical field. Furthermore, the results of thisstudy encourage readers at any level to look carefully at publishedreports and data, and to evaluate the consistency of their results.