Systematic framework for grading evidence quality, used by the source_verification_agent and bibliography_agent.
╱╲
╱ I ╲ Systematic Reviews / Meta-Analyses
╱──────╲
╱ II ╲ Randomized Controlled Trials
╱──────────╲
╱ III ╲ Controlled Studies (non-randomized)
╱──────────────╲
╱ IV ╲ Case-Control / Cohort Studies
╱──────────────────╲
╱ V ╲ Systematic Reviews of Descriptive Studies
╱──────────────────────╲
╱ VI ╲ Single Descriptive / Qualitative Studies
╱──────────────────────────╲
╱ VII ╲ Expert Opinion / Committee Reports
╱──────────────────────────────╲
Weight: Highest Description: Rigorous synthesis of all available evidence using predefined, systematic methods. Characteristics: - Pre-registered protocol (PROSPERO or similar) - Comprehensive search across multiple databases - Explicit inclusion/exclusion criteria - Quality assessment of included studies - Statistical pooling (meta-analysis) when appropriate - PRISMA reporting guidelines followed
Trusted Sources: Cochrane Library, Campbell Collaboration, JBI Evidence Synthesis
Caveats: Quality depends on included studies ("garbage in, garbage out"); may be outdated if field moves fast.
Weight: Very High Description: Experimental studies with random allocation to intervention/control groups. Characteristics: - Random assignment - Control/comparison group - Blinding (single, double, or triple) - Pre-registered protocol - Adequate sample size - Intention-to-treat analysis
Caveats: Not always feasible (especially in social science/education); ethical constraints; external validity concerns.
Weight: High Description: Quasi-experimental designs with comparison groups but no randomization. Characteristics: - Comparison group present - Pre-post measurements - Attempts to control confounds - Larger samples than case studies
Examples: Difference-in-differences, propensity score matching, regression discontinuity.
Caveats: Selection bias risk; confounding variables harder to control.
Weight: Moderate-High Description: Observational studies tracking groups over time or comparing cases to controls. Characteristics: - Longitudinal (cohort) or retrospective (case-control) - Natural variation, no researcher intervention - Large samples possible - Real-world context
Caveats: Cannot establish causation; confounders possible; recall bias (case-control).
Weight: Moderate Description: Rigorous synthesis of qualitative or descriptive research. Characteristics: - Systematic search and selection - Quality appraisal of included studies - Meta-synthesis or meta-ethnography techniques - Transparent methods
Caveats: Quality limited by included studies; interpretive layer adds subjectivity.
Weight: Low-Moderate Description: Individual case studies, ethnographies, surveys, descriptive analyses. Characteristics: - In-depth, context-rich - Exploratory or descriptive purpose - Small samples typical - Thick description
Caveats: Limited generalizability; researcher subjectivity; no causal claims warranted.
Weight: Lowest Description: Position papers, editorials, committee reports, guidelines based on expert consensus. Characteristics: - Based on expertise and experience - Often integrates multiple evidence types informally - May reflect institutional or ideological positions
Caveats: Not empirically tested; potential bias; "authority" ≠ "evidence."
Each source is assessed on two distinct axes that the columns below combine. Conflating them caps strong interpretive and qualitative work at a low overall grade no matter how excellent it is within its own tradition.
The Evidence Level criterion below grades fitness, not raw design level. A source earns Grade A on Evidence Level when its study-design level meets or exceeds its own field's gold standard (see the Field-Specific Adjustments table below), not only when it reaches Level I-II. So humanities work at Level VI, where Level VI is the field's gold standard, can score Grade A on this criterion. The other five criteria are field-neutral.
| Criterion | Grade A (Excellent) | Grade B (Good) | Grade C (Adequate) | Grade D (Weak) | Grade F (Unacceptable) |
|---|---|---|---|---|---|
| Evidence Level | Meets/exceeds the field's gold standard for the claim | One level below the field's gold standard | Two levels below | Well below field norm | Unclassifiable or fundamentally unfit for the claim |
| Peer Review | Rigorous peer review | Standard peer review | Editorial review | No formal review | Self-published |
| Methodology | Exemplary, replicable | Sound, described | Adequate | Questionable | Absent/flawed |
| Sample/Data | Large, representative | Adequate | Limited but justified | Small, convenience | Unspecified |
| Currency | < 3 years | 3-5 years | 5-10 years | > 10 years | Outdated for topic |
| Conflicts | None declared or detected | Minor, disclosed | Moderate, disclosed | Undisclosed potential | Clear undisclosed conflict |
"Currency" is claim-relative: a foundational or historical source is not penalized for age when the claim is about that source itself or its period. Apply the age bands only when recency bears on the claim (e.g., empirical state-of-the-field assertions).
The overall grade reflects fitness for the claim, not raw study-design level. A source whose design level is low (e.g., a Level VI primary source) can still earn an overall Grade A when it is excellent by its field's norms for the claim it supports.
Aggregation rule. Grade the six criteria above (each A-F), then combine:
So an interpretive study that is gold-standard within its discipline reaches overall Grade A, while an integrity-compromised (step 1) or methodologically broken (step 3 proviso) source does not, in any field.
Overall grade meaning:
Not all fields use the same evidence hierarchy. Adjust expectations:
| Field | Gold Standard | Common Level | Notes |
|---|---|---|---|
| Medicine/Health | Level I-II (RCTs, meta-analyses) | Level I-III | Evidence-based medicine tradition |
| Education | Level III-IV (quasi-experimental) | Level IV-VI | Randomization often impractical |
| Social Science | Level III-V | Level IV-VI | Mixed methods common |
| Policy | Level IV-V + VII (expert panels) | Level V-VII | Context-dependent; expert opinion valued |
| Humanities | Level VI (primary sources) | Level VI-VII | Different epistemology; "evidence" means different things |
| Technology | Level III + industry reports | Level V-VII | Fast-moving; peer review lags reality |