Netting the Evidence

Systematic Reviews of Therapy

J Irlam
Primary Health Care Directorate
UCT Faculty of Health Sciences
jirlam@cormack.uct.ac.za

Clinical scenario - What can we do to control our son’s asthma?

John, a 3-year-old boy, recently developed moderate asthma. Skin prick testing confirmed sensitivity to house dust mite. His parents were advised to remove all rugs and feather bedding from their home, and to clean the house regularly with a high-suction vacuum cleaner and an anti-mite chemical cleaner. Despite these measures, John’s asthma remains poorly controlled. His parents seek your advice on what to do next.

1. Ask the right question:

2. Access the evidence:

MEDLINE (PubMed Clinical Queries: Systematic Reviews):house dust mite control asthma yields 19 results, of which the following are of particular interest:

1. Gotzsche PC, Hammarquist C, Burr M. House dust mite control measures in the management of asthma: meta-analysis. BMJ. 1998 Oct 24;317(7166):1105-10
2. Gotzsche P, Johansen H, Schmidt L, Burr M. House dust mite control measures for asthma. Cochrane Database Syst Rev. 2004 Oct 18;(4):CD001187.

What is a traditional review?

• Qualitative, narrative summaries of evidence on a given topic by an “expert” in the field.
• Typically use subjective and non-systematic methods to collect and interpret study data.

Problems with traditional reviews:

1. Bias in selection of studies
   • Personal (“file drawer”) bias.
   • Publication bias.
2. Indexing bias in MEDLINE.
3. May ignore variable quality of published studies.
4. May ignore effect of study design.
5. May ignore effect of study size.

What is a Systematic Review?

This is a review in which bias has been reduced by the:

• systematic identification,
• appraisal,
• synthesis,
• statistical aggregation/ pooling of results (if appropriate)

of all relevant studies on a specific topic according to a predetermined and explicit method.

Moher et al. Lancet; 354: 1896-900

What is a Cochrane review?

This is a systematic review produced by the Cochrane Collaboration (www.cochranelibrary.com). Its features are

• a standardised format;
• an extensive peer review;
• it is published electronically on the Cochrane Library (and indexed in Medline);
• it invites comments and criticism;
• it is kept up-to-date.

The quality and reporting is on the average better than other systematic reviews
...but they are not without flaws!

What is a Meta-analysis?

• A statistical procedure that pools the results of several independent studies considered sufficiently similar (homogeneous).
• Not appropriate if studies are too dissimilar (heterogeneous).
• Provides a quantitative summary of the overall treatment effect.

3. Appraise the evidence - 1

Are the results of the review valid?

1. Does the review address a focused clinical question?
2. Were the selection criteria for eligible studies explicit and appropriate?
3. Was a comprehensive and repeatable search for the evidence undertaken?
4. Was the methodological quality of all eligible studies critically and independently appraised for validity by the reviewers?
5. Were data (PICO) extracted from the studies independently by the reviewers?
6. Was a meta-analysis of the outcomes appropriate in this review?

3. Appraise the evidence - 2

What are the results?

1. How large are the effects of treatment (beneficial and harmful outcomes)?
   • Examine reductions in relative risk (RR) or absolute risk (AR)
2. How precise are the estimates of the effects?
   • Examine 95% confidence interval
3. Are the results clinically significant?
   • Calculate the Number Needed to Treat (NNT).

Is other evidence of causality discussed?

• Consider temporality, biological plausibility, consistency with other studies.

4. Apply the evidence:

How relevant are the results to my patient?

• Is the evidence appropriate to this patient?
• Is the treatment acceptable to this patient?
• Is the treatment affordable?
• Is the treatment available?
• Is the treatment ethical (benefits vs. harms)?