A cognitive framework for producing high-quality reviews. Teaches how to think about paper quality, not just what to score.
Every paper should be evaluated through three lenses simultaneously:
Ask in order: 1. What is the central claim? 2. What evidence is presented? 3. Is there a logical chain from evidence to claim? (Warrant) 4. Are there alternative explanations the authors didn't consider? 5. Would removing any single piece of evidence collapse the argument?
If #5 is yes: The argument depends on a single linchpin. Flag it — the paper's contribution is only as strong as that one piece of evidence.
Ask in order: 1. Who is the population of interest? 2. Does the sample represent that population? 3. Are the conditions replicable? 4. Would the findings hold in a different context/culture/time? 5. What are the boundary conditions the authors don't mention?
Judgment heuristic: Most authors overstate generalizability. If the sample is from one university in one country, findings cannot claim to apply "generally" without qualification.
Ask in order: 1. What did we know before this paper? 2. What do we know after? 3. Is the delta meaningful (not just statistically significant)? 4. Who benefits from knowing this? 5. What new questions does this open?
Judgment heuristic: If you can't articulate the delta in one sentence, either the contribution is weak or the paper hasn't communicated it clearly. Both are review-worthy observations.
| Trap | Description | How to Avoid |
|---|---|---|
| Methodological tunnel vision | Only critiquing methods, ignoring whether the question matters | Start with Lens 3 (contribution) before Lens 1 |
| Novelty bias | Penalizing replication or incremental work | Replication IS valuable; evaluate on execution quality |
| Expertise projection | Expecting the paper to use your preferred method | Evaluate the chosen method on its own terms |
| Positivity-severity oscillation | Being too nice in comments, too harsh in scores | Write the score first, then justify with comments |
| Missing forest for trees | Listing 20 minor issues, missing the one fatal flaw | Always state the single most important issue first |