Research Methods in Pharmacology

1. Introduction to Evidence-Based Medicine
2. The Hierarchy of Evidence
3. Clinical Trials
   • Phases of a Clinical Trial
   • The Randomised Controlled Trial
   • Bias and Confounding
   • Randomisation
   • Blinding
   • Intention-to-Treat versus As-Treated
4. Outcomes and Effect Size
   • Types of Outcome
   • Measures of Effect
   • Number Needed to Treat
5. Statistical Significance
   • P-Values
   • Confidence Intervals
6. Systematic Reviews and Meta-Analyses
   • The Systematic Review
   • The Meta-Analysis
   • Forest Plots
   • Heterogeneity and Fixed-versus-Random-Effects Models
   • Publication Bias
7. Pragmatic and Explanatory Trials
8. Ethics of Clinical Trials
9. Summary
10. Quiz

• Evidence-based medicine (EBM) is the conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients, as defined by David Sackett and colleagues.
• The strongest single type of evidence that a drug works (its efficacy) comes from a randomised controlled trial (RCT). The strongest evidence overall comes from a systematic review (a structured search for, and summary of, all the studies on a question) and meta-analysis (combining the numbers from those studies) of several RCTs.
• Trials are valued for their ability to reduce bias (systematic error) and confounding (a third factor that distorts the apparent link between a treatment and an outcome). The main tools are randomisation (allocation by chance), allocation concealment (hiding the next allocation from the people enrolling patients), blinding and intention-to-treat analysis (analysing patients in the groups they were randomised to). Effect size is described by relative risk, absolute risk difference and the number needed to treat (NNT), all explained below.
• Statistical significance is summarised by the p-value (the probability of a result at least as extreme as the one seen if there were really no difference) and the 95% confidence interval (a range of values that fits the data reasonably well); the standard threshold for significance is p < 0.05.

Introduction to Evidence-Based Medicine

Almost every drug a doctor prescribes was licensed because of a clinical trial. Many recommendations in NICE guidance are based on a synthesis of several trials, alongside other evidence and the judgement of the committee. Understanding how the evidence is generated, summarised and weighed is therefore the foundation of safe, modern prescribing, and a substantial part of every UK medical school's pharmacology curriculum.

The classical definition, by David Sackett and colleagues in 1996, is worth quoting in full:

"Evidence-based medicine is the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients."

The three components are equally important. Conscientious means actively seeking the evidence rather than relying on memory; explicit means making the reasoning transparent; and judicious means tempering the evidence with clinical judgement and patient preference. EBM is not "treatment by checklist"; it is the disciplined integration of evidence, expertise and the individual patient.

The Hierarchy of Evidence

Different study designs answer different questions, and not all generate equally robust evidence. The hierarchy of evidence is conventionally drawn as a pyramid, with the most reliable evidence at the top:

Diagram: The hierarchy of evidence. Systematic reviews of RCTs sit at the top because they minimise bias; expert opinion and pre-clinical research sit at the bottom.

From top to bottom:

• Systematic reviews and meta-analyses of randomised controlled trials.
• Randomised controlled trials (RCTs).
• Cohort studies: observational (researchers watch what happens rather than assigning treatments). They follow groups exposed and not exposed to something, such as a drug, forwards in time.
• Case-control studies: observational, compare those with and without disease backwards in time.
• Cross-sectional studies: observational, snapshot in time.
• Case series and case reports: descriptions of a small group of patients, or of a single patient.
• Editorials and expert opinion.
• Animal and in-vitro research: laboratory studies, for example on cells in a dish.

The pyramid does not say RCTs are always better than observational studies. They are usually better at answering the question "does this treatment work?" Other questions (prognosis, prevalence, harm of a rare side effect) are sometimes best answered by other designs.

Clinical Trials

A clinical trial is, in the textbook definition, "any form of planned experiment which involves patients and is designed to elucidate the most appropriate method of treatment for future patients with a given medical condition." In plain terms, it is a planned experiment on patients, done to find the best treatment for future patients with the same condition.

Phases of a Clinical Trial

Drug development in the UK and internationally is usually described in four numbered phases, each with a different question. Some trials cover two phases at once (for example phase 1/2), and a small exploratory phase 0 is sometimes run first. The four numbered phases are:

• Phase I. A small number of people (usually 20 to 100), who are often healthy volunteers. For some drugs, such as cancer treatments, patients with the condition are used instead. The question is "is the drug safe in humans?" Researchers also study pharmacokinetics (what the body does to the drug), dose finding and tolerability (how well people cope with the side effects).
• Phase II. Several hundred patients with the condition of interest. The question is "does the drug appear to work, and at what dose?"
• Phase III. Hundreds to thousands of patients across multiple centres. The question is "is the drug better than the current standard treatment, or than a placebo?" Almost always a randomised controlled trial. This is the phase that supports licensing.
• Phase IV. Post-marketing surveillance (monitoring a drug once it is licensed and in use). The question is "what happens in the real world, and what rare adverse effects emerge?" The Yellow Card Scheme (the UK system for reporting suspected side effects) contributes here. Both are covered in Pharmacovigilance and Pharmacogenetics.

The Randomised Controlled Trial

The randomised controlled trial (RCT) is the design of choice for testing whether a drug works. The basic structure is:

1. Define the disease of interest and a clear research question (often framed using PICO: Population, Intervention, Comparator (what the new treatment is compared with), Outcome).
2. Identify eligible patients and define inclusion and exclusion criteria (the rules for who can and cannot take part).
3. Take baseline measurements of any variable that might confound the result (age, sex, severity of disease, and comorbidities, meaning other illnesses the patient has).
4. Randomly allocate patients to the new treatment or to the comparator (placebo or current standard).
5. Blind patients, clinicians and assessors where possible.
6. Pre-specify a single primary outcome (the main result the trial is designed to measure) and a small number of secondary outcomes (other results of interest).
7. Follow the patients for a defined period.
8. Analyse on an intention-to-treat basis.

Pre-specifying the primary and secondary outcomes is a critical safeguard against data-dredging. Data-dredging is the practice of running many statistical tests until one comes back "significant" by chance.

Bias and Confounding

Two distinct threats to the validity of a trial:

• Bias is a systematic error in the way the trial is conducted, analysed or reported. Examples include selection or allocation bias (the groups differ from the start, for example because the person enrolling patients influences who gets which treatment), recall bias (patients remember past events differently depending on their outcome), observer bias (assessors see what they expect to see) and reporting bias (some results are reported and others are not).
• Confounding is a third variable that is independently associated with both the exposure (the thing being studied, such as a drug or coffee) and the outcome, producing an apparent association that is not causal. The classic example: does coffee cause lung cancer? Coffee drinkers tend to smoke more, and smoking causes lung cancer. Smoking, not coffee, explains the apparent coffee-cancer link.

Four tools are used to limit these problems. Randomisation tends to balance confounders between the groups. Allocation concealment prevents selection bias when patients are enrolled. Blinding limits bias after allocation. Intention-to-treat analysis keeps the benefit of randomisation when the results are analysed.

Randomisation

Randomisation distributes confounders, both known and unknown, evenly between treatment and control groups, on average. Methods include simple coin tosses, random number tables, and (in modern trials) computer-generated allocation. The allocation should be concealed: the person enrolling the next patient must not be able to predict which group that patient will be allocated to, otherwise selection bias creeps in.

Blinding

Blinding hides the treatment allocation after randomisation, so that knowing it cannot change how participants behave or report symptoms, how clinicians care for them, or how outcomes are assessed. It limits distortion of the results by the placebo effect, observer bias and reporting bias:

• Single blind: the patient does not know which treatment they are receiving.
• Double blind: neither the patient nor the clinician/assessor knows.
• Triple blind: the data analyst is also blinded. The term is used inconsistently, so a good trial report states exactly who was blinded.

Where blinding is impossible (surgery, psychotherapy), researchers use sham procedures (fake versions of the treatment) and blinded outcome assessment to recover what they can.

Intention-to-Treat versus As-Treated

Once a trial is over, the question is how to analyse the results. There are two competing principles:

• Intention-to-treat (ITT) analysis: analyses every patient in the group they were originally randomised to, regardless of whether they completed treatment or switched groups. Patients who drop out are still included, which needs their outcomes to have been measured or a stated way of handling the missing data. ITT preserves the benefits of randomisation and reflects what would happen in the real world, where some patients do not complete treatment. ITT is the standard approach and is sometimes called a pragmatic analysis.
• As-treated and per-protocol analyses are two related alternatives. An as-treated analysis groups patients by the treatment they actually received, even if that differs from the group they were randomised to. A per-protocol analysis (sometimes described as an explanatory analysis) includes only the patients who followed the protocol as intended. It can answer "what is the maximum biological effect of the drug?". Both can be biased and lose the protection of randomisation, because patients who stick to the treatment plan can differ systematically from those who do not.

The two approaches answer different questions. An ITT analysis tells you how the drug will perform in clinical practice, where some patients do not follow the plan. A per-protocol analysis tells you how the drug performs in patients who do follow the plan, but the analysis is more open to bias. Both have their place. In superiority trials (trials that aim to show that one treatment is better than another), ITT is the usual primary analysis. In non-inferiority or equivalence trials (trials that aim to show that a new treatment is no worse than, or is similar to, the standard), ITT alone can make two treatments look more alike than they are. A per-protocol analysis therefore matters too, and results from both are usually reported.

Outcomes and Effect Size

Types of Outcome

Trial outcomes are conventionally divided into three types:

• Pathophysiological outcomes (measurements of body processes, often called surrogate outcomes): tumour size, blood pressure, HbA_1c (glycated haemoglobin, a measure of long-term blood sugar) and ejection fraction (the proportion of blood the left ventricle pumps out with each beat). Easy to measure, but only useful if they actually predict patient-relevant events.
• Clinical outcomes: death, myocardial infarction, stroke, hospital admission. These are the outcomes that most directly guide clinical decisions.
• Patient-focused outcomes: quality of life, symptom scores, functional ability, satisfaction. Increasingly recognised as an important measure of a treatment's value, because they capture what matters to patients.

An ideal outcome is appropriate (relevant to the question), valid (it measures what it claims to measure), sensitive (it picks up real change), specific (it is not changed by unrelated things), reliable (repeat measurements agree), simple, sustainable and timely. In practice, no single measure ticks every box, and most trials use a combination.

Measures of Effect

The size of a treatment effect can be expressed in two complementary ways:

Relative Risk (RR)  =  Risk in treatment group  ÷  Risk in control group

Absolute Risk Difference (ARD)  =  Risk in control group  −  Risk in treatment group

A relative risk of 1 means no difference between groups. For a bad outcome, such as stroke, a relative risk less than 1 means the treatment reduces the risk and greater than 1 means it increases the risk. If the outcome is a good one, such as recovery, the direction is reversed. Using the formula above, the absolute risk difference is positive when the treatment lowers the risk of a bad outcome.

Relative measures can mislead when the underlying risk is small. A drug that "halves your risk of stroke" sounds impressive. But suppose the baseline risk (the risk without treatment) was 2 in 1000. A 50% reduction then means only 1 fewer stroke per 1000 people treated. That is a useful effect, but not as large as the relative figure suggests. Always look at both the relative and the absolute numbers.

Number Needed to Treat

The number needed to treat (NNT) expresses additional benefit relative to a specified comparator: it is the average number of people who must be treated, rather than given the comparator, for one extra person to have the beneficial outcome over a defined follow-up period.

NNT  =  1  ÷  Absolute Risk Difference

An NNT of 10 means that treating 10 people with the intervention rather than the comparator is expected to produce one additional beneficial outcome over the specified period. It does not guarantee one benefit in every group of 10. The corresponding measure for harm is the number needed to treat for one additional harmful outcome, often called the number needed to harm (NNH). Like the NNT, it depends on the comparator, the outcome and the time period.

Three published examples follow. NNTs can range from tens to thousands, and each figure depends on the population, the baseline risk and the follow-up period, so treat these as illustrations:

• Aspirin in secondary prevention (preventing a further event in people who have already had one) after a heart attack (myocardial infarction): NNT ≈ 28 to prevent one serious vascular event (a non-fatal heart attack, a non-fatal stroke or a death from vascular disease) over two years. This comes from 36 fewer serious vascular events per 1000 people treated for two years after a previous heart attack, in a large meta-analysis of antiplatelet drugs (drugs that reduce blood clotting; aspirin was the most widely studied).
• Statins (cholesterol-lowering drugs) in secondary prevention: NNT ≈ 21 to prevent one major vascular event over about five years for each 1 mmol/L fall in LDL cholesterol, in people with existing coronary heart disease (heart disease caused by narrowed arteries). This comes from 48 fewer events per 1000 people in a meta-analysis of statin trials. The NNT is larger if the LDL cholesterol falls by less.
• Antibiotics for sore throat: the NNT to prevent one case of rheumatic fever (a serious complication of throat infection that can damage the heart valves) is very large in a high-income country such as the UK, because the illness is now rare. A Cochrane review found that antibiotics reduced rheumatic fever, but only about 1.7 in every 100 people given a placebo developed it even in trials reporting before 1961, and the rate has fallen since.

Statistical Significance

P-Values

The p-value is the probability of getting a result at least as extreme as the one observed, if the null hypothesis (no real difference) were true and the other assumptions of the analysis were correct. It is not the probability that the null hypothesis is true, and it is not the probability that the result was "due to chance". A p-value below the conventional threshold of 0.05 is described as "statistically significant".

Two important caveats:

• Statistical significance is not the same as clinical importance. A massive trial can find a tiny, clinically irrelevant difference to be highly statistically significant.
• The 0.05 threshold is a convention, not a fact. The probability that a "significant" finding is a false positive (a result that looks real but is not) depends on prior probability (how plausible the idea was beforehand), sample size and the number of tests performed. For example, if 20 independent tests are run on treatments that do nothing, about one is expected to reach p < 0.05 by chance.

Confidence Intervals

Strictly, a 95% confidence interval (CI) means this: if the study were repeated many times, 95% of the resulting CIs would contain the true effect. This is the frequentist view (it describes what happens over many repeats of the study). A common shortcut is to say "the true effect lies within this range with 95% probability". Strictly, that is a Bayesian statement (one that updates prior beliefs with the data), not what a confidence interval tells you, so it is worth avoiding in formal work. As a rule of thumb, though, the interval does show the range of effects that fit the data reasonably well. The CI is more informative than the p-value alone, because it shows both the magnitude and precision of the estimate.

Two rules to remember:

• For a relative measure (relative risk, or odds ratio, which compares the odds of an event in two groups): if the 95% CI includes 1, the result is not statistically significant.
• For an absolute measure (risk difference, or mean difference, which is the difference between the average values in two groups): if the 95% CI includes 0, the result is not statistically significant.

A wide confidence interval means the data are imprecise; a narrow interval means they are precise. A wide interval that only just excludes 1 (its nearer end is close to 1) is a far weaker finding than a narrow interval that excludes 1 by a large margin, even if both have p < 0.05.

Preview the Research Methods in Pharmacology quiz