Next Lesson - Patient Safety and Quality Control
Core
Evidence based practice involves the integration of individual clinical expertise (from the healthcare worker in question) with the best available external evidence from systematic research, and with the patient’s own values and preferences. Evidence based practice attempts to reduce the number of healthcare workers continuing to use ineffective practices or failing to take on practices that have been proved to be effective.
Evidence from many studies is usually brought together in two main ways:
- Systematic Review - an overview of primary studies (the original research studies) done using explicit and reproducible methods (the method is clearly described, so another team could repeat it)
- Meta-analysis - a quantitative synthesis of the results of two or more primary studies that addressed the same hypothesis in the same way.
This means that a meta-analysis combines the results of studies that are similar enough to be pooled. A systematic review can include a broader range of studies, even those that differ more in their patients, treatments or outcomes (this variation is called clinical heterogeneity).
Problems getting evidence into practice
- Evidence exists but healthcare professionals don’t know how to access it or choose not to use it.
- Organisational systems or resources are not available to implement the change in advice following new evidence.
- Patients might not want the recommended treatment.
- NICE guidelines offering one treatment as a first choice can lead to reluctance to fund other treatments.
Systematic reviews are a type of literature review (a summary of published research on a topic) that collects together and analyses multiple studies around the same topic. They attempt to provide a complete summary of the current research. Systematic reviews may use meta-analyses as part of their reasoning.
They exist to: avoid or reduce bias, highlight gaps in research, address clinical uncertainty (if studies on the same topic come to different conclusions), and to assess the quality of studies and their results.
They are useful to clinicians because they use methods designed to reduce bias, although they can still inherit the biases of the studies they include. They also reduce the delay between research and implementation (because all the research is collated), save clinicians from having to investigate the research themselves, and increase quality control and certainty.
However, it is important to note that not all systematic reviews are of equal quality. Therefore a poorly conducted systematic review can be just as harmful as a poorly conducted study. Furthermore, if a systematic review uses poor quality studies to create its conclusions then those conclusions will themselves be of poor quality. The phrase ‘rubbish in, rubbish out’ sums this issue up.
Criticisms of Systematic Reviews
- It can be difficult to keep a systematic review up to date as new studies are published.
- Doctors may feel they have less autonomy (freedom to use their own judgement) if systematic reviews are treated as rules they must follow. In practice, evidence is used alongside clinical judgement and the patient’s own preferences.
Meta-analyses are a collation of similar studies that have been collected to form one conclusion. They facilitate the collection of a large amount of study data and reduce problems of small sample size (too few participants in a single study). They also allow the researchers to quantify the size of an effect and the confidence interval around it (the range that shows how uncertain the result is) as a single pooled estimate. In practice, doing a meta-analysis means that the researchers look at all the data on one topic and work out a weighted average of the results, so that larger, more precise studies count for more. They then calculate the degree of uncertainty around this average (the confidence interval), and can release it to the public.
Meta-analyses also have formal protocols (written plans) that specify several things before the researchers start. Fixing these in advance reduces the risk of bias:
- The eligibility criteria for including studies (decided in advance, so studies are included or left out because of the question and not because of what they found)
- The outcome that all the studies have in common, so that their results can be combined
- Planned analyses looking for sources of variation between studies (for example, checking whether one study included only men)
Forest plots are the diagrammatic form of a meta-analysis.
Diagram of a Forest Plot. All values and studies are fictional.
SimpleMed Original by Dr. Maddie Swannack
- A study’s individual odds ratio (a measure that compares the odds of an outcome in one group with the odds in the other) is indicated by a square. The size of the square shows how much weight is given to that study. In general, a study’s weight depends on how precise its result is: a larger sample size and a narrower confidence interval give the study more weight, and so a bigger square.
- A study’s confidence interval is represented by the horizontal lines out the side of each box.
- The vertical no-effect reference line is at an odds ratio of 1, which means the odds are equal in the two groups.
- The x-axis shows the effect size (how big the difference between the groups is), here an odds ratio. For continuous outcomes (measurements on a scale, such as blood pressure), a mean difference may be used instead; the no-effect reference line is then at 0, meaning there is no difference between the group means.
- The diamond shows the pooled estimate of the odds ratio (the combined result of all the studies). The centre of the diamond, marked by the dotted line, is the pooled odds ratio, and the width of the diamond is the pooled confidence interval.
- Heterogeneity between studies (studies too different)
- The choice between a fixed effect and a random effects model (explained below) affects the pooled result
- Sub-group analysis (comparing results between subsets of studies or participants, explained below) can help to explore the variation, but it should be limited and interpreted cautiously
- Variable quality of studies - not all are carried out to the same standard, so some are at higher risk of bias
- Publication bias in selection of the studies - studies with non-significant or unfavourable results are less likely to be published, for example when funders or investigators have an interest in the result, which can bias the evidence available
Studies are pooled using one of two models, which treat variation between studies differently: the fixed effect model and the random effects model.
Fixed effect model - assumes that all the studies are estimating exactly the same true effect, so all the studies are aiming for the same ‘true effect’ value. Any differences between the study results are assumed to be due to chance (random error) alone. No 2 studies can be exactly the same or have the same populations so the exact same true effect is a sizeable assumption in this model.
Random effects model - assumes that the studies are estimating similar (but not the same) ‘true effect’ values. As a result the ‘true value’ is the mean of these values. This model is often considered when the studies differ in their patients, treatments or methods, but the choice of model depends on the review question and assumptions, and should not be made using a statistical test for heterogeneity.
Diagram showing the differences between the fixed effect (when all studies are aiming for one true effect) and random effects (when studies are aiming for their own effect value, and the true effect is a mean of these) models.
SimpleMed Original by Dr. Maddie Swannack
These different models can produce different results, so the choice of model can change the conclusions drawn:
- The final estimate for the odds ratio is similar with both methods when there is little heterogeneity, but it can differ when there is a lot, for example if smaller studies give systematically different results from larger ones.
- The confidence interval is often wider with the random effects model.
- The weighting of studies is more equal in the random effects model (more weight is given to smaller studies).
Random effects modelling accounts for variation but cannot explain it. Sub-group analysis (splitting the studies or participants into groups and comparing the results between them) is used to help explain the heterogeneity (the variation between the studies). This can be useful in understanding the effects of a certain intervention (a treatment or other action) or exposure (something people have come into contact with, such as a risk factor).
There are two main types of sub-group analysis:
- Study characteristics - year of publication (is a later study different), length of follow up (how long participants are observed: is the conclusion different if follow up is twice as long), % female participants (different conclusion if all female).
- Participant profile - all data is analysed separately by type of participant eg male vs female, adult vs child, one ethnic group vs another. Such analyses should be interpreted cautiously as they are prone to confounding (when the effects of two linked characteristics cannot be separated) and false positive findings.
Issues with Studies of Variable Study Quality
Variable quality can be due to:
- Poor study design (this can be judged from how the study was planned and reported)
- Poorly designed protocols
- Poor protocol implementation (this is hard to assess after the study is done).
Risk of bias (whether a study’s design or conduct could systematically distort its results) depends on the study design, how the study was conducted and how the result was measured; no design is always the least or most biased:
- Randomisation helps control confounding, but trials can still be affected by problems with allocation, missing data or outcome measurement.
- Non-randomised trials and cohort studies (which follow groups of people over time) require careful assessment of confounding, participant selection and measurement.
- Case-control studies (which compare people who have a disease with people who do not) using recalled exposure can be affected by recall bias: cases and controls may remember or report past exposure differently after their disease status is known. Control selection and confounding also need assessment.
Study selection and assessment of bias should be planned before the review begins:
- Define which study designs can answer the review question and specify the criteria for including studies. Cochrane reviews (systematic reviews produced by the international organisation Cochrane) may include non-randomised studies when these are appropriate for the question.
- Assess the risk of bias in each study, then use the assessment in two ways:
- Consider specific sources of bias when interpreting the pooled estimate. Cochrane does not recommend turning these judgements into a single numerical quality score to adjust each study's statistical weight.
- Use sensitivity analysis (repeating the analysis with some studies left out) to check whether the conclusions change when studies at high risk of bias are excluded.
Use a domain-based risk-of-bias tool appropriate to the study design: a checklist that looks at each separate way bias can arise (such as the randomisation process or missing data) instead of giving one overall quality score. Synthesise randomised and non-randomised intervention studies separately; combine other studies only when their questions and methods are sufficiently comparable. A study can remain in the review even when it is omitted from a particular synthesis or sensitivity analysis.
There are many things to look at when assessing the quality of randomised controlled trials:
- Allocation methods (eg randomisation)
- Blinding (whether participants and researchers knew which treatment was given) and outcome assessment
- Patient drop out (people lost before the end of the study: there is no safe cut-off, and the risk of bias depends on whether the loss is related to the treatment received and to the outcome)
- Intention to treat analysis (analysing people in the groups they were randomly assigned to, whether or not they completed the treatment)
At least two people should assess the quality of each study independently, which reduces errors and the influence of one person’s preconceptions. The assessors should also be blinded to the results of the study, but this can be difficult as they have often seen the study results before they begin.
Studies with statistically significant results (unlikely to be explained by chance alone) or ‘favourable’ results are more likely to be published than studies with non-significant or ‘unfavourable’ results, and this occurs especially with smaller studies. This means that any systematic review or meta-analysis can be flawed by this bias as some data is simply not available.
Reviewers should search for unpublished as well as published studies to reduce the risk of missing evidence. A funnel plot can help explore whether results differ between smaller and larger studies, but it does not remove publication bias. If studies are estimating the same underlying effect without important bias, smaller studies tend to show a wider spread of effect estimates because their results are less precise. Larger studies cluster more closely around the underlying effect, giving the plot its funnel shape.
Diagram showing two funnel plots, the left side being more symmetrical and the right showing asymmetry. If a funnel plot is broadly symmetrical, this suggests there is no clear evidence of publication bias or other small-study effects (a tendency for smaller studies to give different results from larger ones), but it does not exclude them. If a funnel plot shows gaps or asymmetry (as shown by the blue circle), this may indicate publication bias, particularly where smaller studies with unfavourable results are missing. Other explanations, such as heterogeneity or chance, should also be considered.
SimpleMed Original by Dr. Maddie Swannack
Edited by: Dr. Ben Appleby
Reviewed by: Dr. Marcus Judge
- 6729
