Evidence-based Practice (EBP)

Does the treatment work? Critically appraising health care interventions

J Irlam
Primary Health Care Directorate
UCT Faculty of Health Sciences
jirlam@cormack.uct.ac.za

Can longer needles reduce the incidence of adverse reactions to immunisation?

Mrs. Brown brings her child at 10 weeks of age for his second round of DPT and HiB injections, but is concerned about an adverse reaction since the child experienced local redness and swelling after his previous visit. A number of other parents have recently reported the same thing to you. A colleague advises that using a longer needle may help to prevent these reactions, so you wonder whether there is any evidence for this in the literature.

Figure 1

1. Ask the right question:

Figure 2

2. Access the evidence:

· MEDLINE (PubMed Clinical Queries: therapy): Search terms: needle length immunisation.

• 1 RCT:

  Diggle L, Deeks J.
  Effect of needle length on incidence of local reactions to routine immunisation in infants aged 4 months: randomised controlled trial. BMJ. 2000 Oct 14; 321(7266):931-3.

3. Appraise the evidence - 1:

Is the study design appropriate to the question?

• Each design has different strengths and weaknesses for evaluating interventions.
• The most appropriate study design will provide the most reliable evidence, provided it is of high quality.

3. Appraise the evidence - 2:

3. Appraise the evidence – 1

Is the study valid?

1. Were participants allocated to comparison groups in a truly random, unbiased manner (quality of randomisation)?
2. Was the random allocation sequence concealed from those administering treatment (allocation concealment)?
3. Were the groups similar at baseline wrt potential confounding variables (did randomisation produce equal groups)?
4. Were the groups dealt with equally during the trial (apart from the interventions)?

3. Appraise the evidence - 2:

Why are RCTs best?

Random allocation (randomization) and control group = best design for minimising:

· Bias in:

• Selection of participants.
• Performance of trial.
• Attrition / loss to follow-up of participants.
• Detection of outcomes.

· Confounding

BUT… may not be feasible (time, cost), ethical, able to detect harmful effects in the long-term, or produce readily generalisable results.

Potential sources of bias in an RCT

Figure 3

3. Appraise the evidence – 3

Is the study valid?

1. Were participants allocated to comparison groups in a truly random, unbiased manner (quality of randomisation)?
2. Was the random allocation sequence concealed from those administering treatment (allocation concealment)?
3. Were the groups similar at baseline wrt potential confounding variables (did randomisation produce equal groups)?

4. Were the groups dealt with equally during the trial (apart from the interventions)?

   • Were clinicians administering the treatment blinded?
   • Were trial procedures standardised?

   • Was there proper training and supervision of personnel?

5. Were sufficient (>80%) and equal numbers of participants followed up to completion of trial?
6. Were analyses based on intent to treat (ITT)? (regardless of whether all completed the intervention as originally intended)

7. Were outcome measurements accurate (unbiased and error-free)?

   • Were study participants blinded?
   • Were those assessing outcomes blinded?
   • Were measurement procedures standardised?
     

     What are the results?

8. How large are the effects of treatment (beneficial and harmful outcomes)?

   • Examine reductions in relative risk (RR) or absolute risk (AR)

9. How precise are the estimates of the effects?

   • Examine 95% confidence interval

10. Are the results clinically significant?

    • Calculate the Number Needed to Treat (NNT)

Is other evidence of causality discussed?

• Temporal association, biological plausibility, consistency with other studies

4. Apply the evidence:

How relvant are the results to my patients?

• Is the evidence appropriate to this patient?
• Is the treatment acceptable to this patient?
• Is the treatment affordable?
• Is the treatment available
• Is the treatment ethical?