← Về thư mục

Brainstorming and Elicitation Methods

Use methods as fit-for-purpose process choices, not creativity guarantees. The best-supported finding in classic laboratory work is narrow: interacting face-to-face groups often produce fewer nonredundant ideas than the pooled output of the same number of people working independently, with turn-taking (production blocking) an important mechanism. That does not establish that independent work is always better for learning, synthesis, commitment, selection, or every real-world task. See sources.md for studies and limitations.

Choose by purpose

Need	Suitable pattern	Main caution
Broad initial idea pool	Independent generation, then structured sharing	A larger pool is not automatically a better decision
Equal participation and same-day prioritization	Nominal group technique (NGT)	Votes show panel preference, not scientific truth
Iterative geographically distributed judgment	Delphi	Consensus can stabilize around shared bias
Quantitative uncertain values for a model	Structured expert elicitation	Expert judgment does not replace empirical evidence
Explore a combinatorial design space	Morphological analysis	Combinations may be infeasible or meaningless
Reframe an existing concept	SCAMPER or assumption reversal	Prompting heuristics have context-dependent evidence
Stress-test shortlisted ideas	Red team, premortem, alternative explanations	Critique needs an explicit response and owner

For consequential decisions, state why the chosen process fits the question, who is included, what information participants see, and how uncertainty and dissent will be retained.

Independent-then-interactive generation

This is the default for a live research session.

1. Give every participant the same neutral question, constraints, and time.
2. Ask them to write ideas privately and in parallel.
3. Capture each idea before anyone sees another participant's answer.
4. Pool ideas with stable IDs; optionally mask contributor identity.
5. Clarify in a round robin without advocacy or scoring.
6. Add a second private round after participants have seen the pool.
7. Cluster by a declared relation while preserving original text.
8. Move to a separately announced evaluation phase.

Why use it:

• Parallel work avoids waiting to speak.
• Human-first generation limits early leader, example, and AI anchors.
• The second private round permits stimulation from others' ideas without requiring immediate public performance.

Limits:

• Classic brainstorming studies often used short, artificial tasks and student samples.
• Pooled individual output may contain redundancy and miss benefits of dialogue, knowledge integration, or implementation commitment.
• Higher idea counts do not guarantee better final selections. One experiment found nominal groups produced more and more-original ideas, but selected ideas were not better than those of interactive groups.

Nominal group technique

NGT is a facilitated, usually synchronous process for eliciting and prioritizing contributions. A common four-stage form is:

1. Silent generation: participants independently answer one precise question.
2. Round-robin recording: each person contributes one item at a time until all items are recorded.
3. Clarification: discuss meaning, not merit; merge only with originators' agreement and preserve a merge log.
4. Independent rating or ranking: participants vote privately using predeclared rules.

Use NGT when equal airtime, traceability, and prompt prioritization matter. Report:

• recruitment and relevant perspectives;
• exact question and materials shown in advance;
• group size, facilitator, accessibility adaptations, and conflicts;
• how items were edited, merged, removed, or added;
• rating scale, consensus or retention rule, missing votes, ties, and abstentions;
• full distribution, not only top-ranked items.

Do not:

• call a ranked list “validated”;
• drop low-ranked minority concerns when they concern safety or ethics;
• infer population prevalence from a purposive panel;
• silently modify NGT and still imply a standardized procedure.

NGT is flexible, and reviews document substantial variation in implementation. Describe the procedure actually used.

Delphi

Delphi is an iterative, usually anonymous elicitation process with controlled feedback between rounds. It is useful when participants are dispersed, face-to-face status effects are a concern, or judgments need time for revision.

Minimal defensible design

1. Define why Delphi is appropriate and what decision it will inform.
2. Predefine “expertise” or stakeholder eligibility; sample multiple relevant perspectives rather than only prestigious titles.
3. Pilot unambiguous questions and scales.
4. Predefine the number or stopping logic for rounds, feedback statistics, consensus rule, missing-data handling, and treatment of new items.
5. Collect round 1 independently.
6. Return controlled feedback that includes the distribution and anonymized reasons, not only a mean.
7. Let participants retain or revise judgments and explain important changes.
8. Report attrition by round, disagreement, stability, and items without consensus.

Interpretation

• Consensus means convergence among this panel under this protocol.
• It does not establish correctness, causality, clinical effectiveness, or ethical acceptability.
• Anonymity can reduce interpersonal pressure but also removes conversational repair and may obscure conflicts of interest.
• Repeated feedback may manufacture agreement. Preserve rationales and minority estimates.
• “Modified Delphi” is not self-explanatory; list every modification.

Use the CREDES checklist or the 2023 RAND methodological guidance when a Delphi result will be published or relied upon. Those resources are indexed in sources.md.

Structured expert elicitation

Use structured expert elicitation when empirical evidence is incomplete and a decision model needs quantities or probability distributions—not simply a list of ideas. Follow domain guidance where available.

High-level sequence:

1. Define the target quantity, unit, conditioning information, time horizon, and resolution criterion.
2. Review available evidence systematically before asking for judgment.
3. Select experts for relevant and complementary expertise; disclose conflicts.
4. Train participants to express uncertainty and test the instrument.
5. Elicit individual judgments before group aggregation.
6. Ask for plausible bounds and reasons, then a central estimate; avoid presenting a preferred anchor.
7. Record assumptions, dependencies, and what evidence would change the estimate.
8. Apply a declared mathematical or behavioral aggregation method.
9. Test sensitivity to experts, aggregation rules, and assumptions.
10. Document the complete process and distinguish expert judgment from data.

EFSA's guidance emphasizes framing, expert selection, uncertainty elicitation, aggregation, and documentation because unaided judgment—especially about uncertainty—can be biased. Cooke's Classical Model and other protocols have additional requirements; do not imitate only their scoring labels.

Divergence and convergence

Treat divergence and convergence as facilitation modes, not cleanly separable mental faculties.

Divergence

Aim for a varied candidate set:

• defer comparative judgment for a fixed interval;
• vary scale, population, mechanism, measurement, time horizon, and level of intervention;
• request alternatives that predict different observations;
• include null mechanisms and “do nothing / measure first” options;
• capture assumptions and uncertainties while the idea is generated.

Transition

The facilitator explicitly closes generation, freezes the initial register, and introduces evaluation criteria. This process separation reduces premature evaluation, but evidence does not support claiming that it always improves selected idea quality.

Convergence

• clarify and cluster without erasing distinctions;
• define originality and usefulness for this decision;
• rate independently before discussion;
• show score distributions and qualitative reasons;
• run adversarial, evidence, feasibility, and ethics reviews;
• revisit ideas when criteria or evidence change.

Experiments on idea selection show that people can select poorly from their own pools. Explicit criteria can improve selection on the named dimension while producing trade-offs in satisfaction or perceived effectiveness. Therefore, keep criteria plural and make trade-offs visible.

Generative prompt families

These are scaffolds, not validated scientific methods.

Assumption inventory and reversal

1. List descriptive, causal, measurement, operational, and value assumptions.
2. Mark each as evidenced, conventional, required, or uncertain.
3. Reverse or remove one assumption.
4. Ask what observation would follow and whether the reversal is coherent.

Do not confuse a provocative reversal with a plausible hypothesis.

Scale and boundary shifts

Vary:

• spatial or organizational level;
• time scale and lag;
• population, environment, or boundary conditions;
• dose, intensity, or resolution;
• unit of analysis and measurement modality.

Ask which mechanisms remain invariant and which predictions change.

Cross-domain analogy

Write the mapping explicitly:

• source system and target system;
• relation being transferred;
• known mismatches;
• testable implication;
• evidence needed before transfer is credible.

An analogy generates a question; it is not evidence for the target mechanism.

Morphological analysis

1. Define independent dimensions of the problem.
2. List bounded options for each dimension.
3. Generate combinations systematically.
4. Remove combinations that violate stated constraints.
5. Sample remaining combinations transparently if the full product is too large.
6. Record why combinations were excluded.

Do not equate an unlisted combination with novelty. Check literature and feasibility.

SCAMPER

For an existing method or concept, ask whether to Substitute, Combine, Adapt, Modify, Put to another use, Eliminate, or Reverse/Rearrange. For each output, add a mechanism, expected observation, and failure mode. Avoid generic technology substitution without a scientific reason.

Constraint ladder

Run three rounds:

1. current real constraints;
2. one negotiable constraint removed;
3. a stricter safety, cost, time, or accessibility constraint added.

Compare which ideas survive and which assumptions drive the difference.

Premortem and alternatives

Assume the favored idea produced an uninterpretable or harmful result. List causes across theory, measurement, sampling, execution, analysis, governance, and misuse. Then ask for at least two mechanisms that predict the same apparent success. Convert each into a check or discriminating observation.

Method combinations

Useful combinations include:

• independent generation → NGT clarification/rating → adversarial review;
• morphological analysis → independent rating → feasibility gate;
• Delphi → structured uncertainty elicitation for unresolved quantitative items;
• human-first round → disclosed AI counterexamples → second human-only round;
• literature check → assumption reversal → updated decision matrix.

Never stack methods merely to look rigorous. Every stage should have a stated purpose, output, and stop rule.

Stop conditions

Pause or end the process when:

• the question cannot be scoped without confidential or controlled details;
• the group lacks a perspective essential to safety or interpretation;
• a clinical, ethics, biosafety, security, legal, or regulatory gate is triggered;
• participants cannot dissent safely;
• criteria or weights are being changed to favor a known option;
• a literature check is too incomplete to support a novelty claim;
• the accountable decision owner is absent.