Description: Comprehensive synthesis of all available evidence on a question.
Strengths: - Combines multiple studies for greater power - Reduces impact of single-study anomalies - Can identify patterns across studies - Quantifies overall effect size
Weaknesses: - Quality depends on included studies ("garbage in, garbage out") - Publication bias can distort findings - Heterogeneity may make pooling inappropriate - Can mask important differences between studies
Critical evaluation: - Was search comprehensive (multiple databases, grey literature)? - Were inclusion criteria appropriate and prespecified? - Was study quality assessed? - Was heterogeneity explored? - Was publication bias assessed (funnel plots, fail-safe N)? - Were appropriate statistical methods used?
Description: Experimental studies with random assignment to conditions.
Strengths: - Gold standard for establishing causation - Controls for known and unknown confounders - Minimizes selection bias - Enables causal inference
Weaknesses: - May not be ethical or feasible - Artificial settings may limit generalizability - Often short-term with selected populations - Expensive and time-consuming
Critical evaluation: - Was randomization adequate (sequence generation, allocation concealment)? - Was blinding implemented (participants, providers, assessors)? - Was sample size adequate (power analysis)? - Was intention-to-treat analysis used? - Was attrition rate acceptable and balanced? - Are results generalizable?
Description: Observational studies following groups over time.
Types: - Prospective: Follow forward from exposure to outcome - Retrospective: Look backward at existing data
Strengths: - Can study multiple outcomes - Establishes temporal sequence - Can calculate incidence and relative risk - More feasible than RCTs for many questions
Weaknesses: - Susceptible to confounding - Selection bias possible - Attrition can bias results - Cannot prove causation definitively
Critical evaluation: - Were cohorts comparable at baseline? - Was exposure measured reliably? - Was follow-up adequate and complete? - Were potential confounders measured and controlled? - Was outcome assessment blinded to exposure?
Description: Compare people with outcome (cases) to those without (controls), looking back at exposures.
Strengths: - Efficient for rare outcomes - Relatively quick and inexpensive - Can study multiple exposures - Useful for generating hypotheses
Weaknesses: - Cannot calculate incidence - Susceptible to recall bias - Selection of controls is challenging - Cannot prove causation
Critical evaluation: - Were cases and controls defined clearly? - Were controls appropriate (same source population)? - Was matching appropriate? - How was exposure ascertained (records vs. recall)? - Were potential confounders controlled? - Could recall bias explain findings?
Description: Snapshot observation at single point in time.
Strengths: - Quick and inexpensive - Can assess prevalence - Useful for hypothesis generation - Can study multiple outcomes and exposures
Weaknesses: - Cannot establish temporal sequence - Cannot determine causation - Prevalence-incidence bias - Survival bias
Critical evaluation: - Was sample representative? - Were measures validated? - Could reverse causation explain findings? - Are confounders acknowledged?
Description: Description of observations in clinical practice.
Strengths: - Can identify new diseases or effects - Hypothesis-generating - Details rare phenomena - Quick to report
Weaknesses: - No control group - No statistical inference possible - Highly susceptible to bias - Cannot establish causation or frequency
Use: Primarily for hypothesis generation and clinical description.
Description: Statements by recognized authorities.
Strengths: - Synthesizes experience - Useful when no research available - May integrate multiple sources
Weaknesses: - Subjective and potentially biased - May not reflect current evidence - Appeal to authority fallacy risk - Individual expertise varies
Use: Lowest level of evidence; should be supported by data when possible.
No plausible confounders
Multiple converging lines of evidence from different study types
Natural experiments approximating randomization
Conflicts of interest
Biased meta-analyses:
Poor search strategy
Not addressing the right question:
GRADE (Grading of Recommendations Assessment, Development and Evaluation) assesses evidence quality across four levels:
Definition: Very confident that true effect is close to estimated effect.
Characteristics: - Well-conducted RCTs - Overwhelming evidence from observational studies - Large, consistent effects - No serious limitations
Definition: Moderately confident; true effect likely close to estimated, but could be substantially different.
Downgrades from high: - Some risk of bias - Inconsistency across studies - Indirectness (different populations/interventions) - Imprecision (wide confidence intervals) - Publication bias suspected
Definition: Limited confidence; true effect may be substantially different.
Downgrades: - Serious limitations in above factors - Observational studies without special strengths
Definition: Very limited confidence; true effect likely substantially different.
Characteristics: - Very serious limitations - Expert opinion - Multiple serious flaws
Questions: - Was randomization adequate? - Was allocation concealed? - Were groups similar at baseline? - Was blinding implemented? - Was attrition minimal and balanced? - Was intention-to-treat used? - Were all outcomes reported?
Questions: - Is sample representative of target population? - Are inclusion/exclusion criteria too restrictive? - Is setting realistic? - Are results applicable to other populations? - Are effects consistent across subgroups?
Questions: - Was sample size adequate (power)? - Were statistical tests appropriate? - Were assumptions checked? - Were effect sizes and confidence intervals reported? - Were multiple comparisons addressed? - Was analysis prespecified?
Questions: - Were measures validated and reliable? - Was outcome defined clearly and appropriately? - Were assessors blinded? - Were exposures measured accurately? - Was timing of measurement appropriate?
RCTs: - Cochrane RoB 2 (Risk of Bias 2) — current standard for randomized trials - PEDro Scale (for trials in physical therapy) - Legacy: Cochrane RoB 1, Jadad Scale (historical; prefer RoB 2 for new reviews)
Observational Studies: - ROBINS-I (Risk of Bias in Non-randomized Studies — of Interventions) - Newcastle-Ottawa Scale
Diagnostic Studies: - QUADAS-2 (Quality Assessment of Diagnostic Accuracy Studies)
Systematic Reviews: - PRISMA 2020 checklist (reporting standard for systematic reviews) - AMSTAR-2 (A Measurement Tool to Assess Systematic Reviews)
All Study Types: - CASP Checklists (Critical Appraisal Skills Programme)
Hierarchy differs: 1. Multiple convergent lines of evidence 2. Mechanistic understanding 3. Reproducible experiments 4. Established theoretical framework
Key considerations: - Replication essential - Mechanistic plausibility - Consistency across model systems - Convergence of methods
Additional concerns: - Replication crisis - Publication bias particularly problematic - Small effect sizes often expected - Cultural context matters - Measures often indirect (self-report)
Strong evidence includes: - Preregistered studies - Large samples - Multiple measures - Behavioral (not just self-report) outcomes - Cross-cultural replication
Causal inference frameworks: - Bradford Hill criteria - Rothman's causal pies - Directed Acyclic Graphs (DAGs)
Strong observational evidence: - Dose-response relationships - Temporal consistency - Biological plausibility - Specificity - Consistency across populations - Large effects unlikely due to confounding
Challenges: - Complex interventions - Context-dependent effects - Measurement challenges - Ethical constraints on RCTs
Strengthening evidence: - Mixed methods - Natural experiments - Instrumental variables - Regression discontinuity designs - Multiple operationalizations
Strong evidence: - Multiple studies, different investigators - Different populations and settings - Different research designs converge - Different measurement methods
Weak evidence: - Single study - Only one research group - Conflicting results - Publication bias evident
Strengthens evidence: - Known mechanism - Consistent with other knowledge - Dose-response relationship - Coherent with animal/in vitro data
Weakens evidence: - No plausible mechanism - Contradicts established knowledge - Biological implausibility
Essential for causation: - Cause must precede effect - Cross-sectional studies cannot establish - Reverse causation must be ruled out
Moderate indicator: - Specific cause → specific effect strengthens causation - But lack of specificity doesn't rule out causation - Most causes have multiple effects
Strong evidence: - Large effects unlikely to be due to confounding - Dose-response relationships - All-or-none effects
Caution: - Small effects may still be real - Large effects can still be confounded
High-quality evidence: - Strong confidence in acting on findings - Reasonable to change practice/policy
Moderate-quality evidence: - Provisional conclusions - Consider in conjunction with other factors - May warrant action depending on stakes
Low-quality evidence: - Weak confidence - Hypothesis-generating - Insufficient for major decisions alone - Consider cost/benefit of waiting for better evidence
Very low-quality evidence: - Very uncertain - Should not drive decisions alone - Useful for identifying gaps and research needs
Strategies: 1. Weight by study quality 2. Look for systematic differences (population, methods) 3. Consider publication bias 4. Update with most recent, rigorous evidence 5. Conduct/await systematic review 6. Consider if question is well-formed
Avoid: - Absolute certainty ("proves") - False balance (equal weight to unequal evidence) - Ignoring uncertainty - Cherry-picking studies
Better: - Quantify uncertainty - Describe strength of evidence - Acknowledge limitations - Present range of evidence - Distinguish established from emerging findings - Be clear about what is/isn't known