Running a Delphi Study Where the Reasoning Survives the Rounds

The panel that converged on nothing anyone could explain

Round 3 of the Delphi study closes and the facilitator emails the panel: consensus reached. The median moved from 6.1 to 7.8 on the ten-year adoption question; standard deviation halved; the report writes itself. Then one panelist — a careful one — replies with the question that unravels it: "I revised my estimate because the Round-2 summary said the group had strong reasons for optimism. Can I see those reasons? I'd like to know what I agreed with."

There is no good answer. The rounds were surveys. Between them, the panel saw medians, quartiles and a facilitator's two-paragraph summary. The reasons — whatever specific arguments moved eleven experts to revise upward — were typed into free-text boxes, compressed into the summary, and lost. The panel converged; on what reasoning, nobody can say.

This is not a broken study; it is standard practice with standard tooling, and it wastes the method's best property. Delphi was designed around a precise insight: anonymity and independence remove the dominance and status effects that distort face-to-face panels, and controlled feedback lets genuine judgment shift without social pressure. Survey software delivers the independence and loses the substance of the feedback. This tutorial runs the same rounds with the reasoning kept.

What survey-based Delphi loses between rounds

A Delphi round produces two things: a distribution (where the panel stands) and a body of reasoning (why). Survey tools are built for the first. The second gets flattened into free-text fields and then into a facilitator's summary — which quietly inserts an unaccountable editor between the experts and each other. Panelists in Round 2 are not responding to their peers' arguments; they are responding to a paraphrase of them. For a method whose entire epistemology is structured collective judgment, that is a strange thing to lose — the history of collective methods is largely a history of fights over exactly this feedback step.

Setup: the question, the panel — and the anonymity decision

Before anything else, one decision that a methodologist will check first: what your panel actually requires — independence or anonymity — because they are not the same thing, and the platform gives you one of them by default.

1. 1Frame the proposition as the root claim — falsifiable, not open-ended: "X will be standard clinical practice by 2032", not "what is the future of X?". A vague root produces unrateable positions. Checkpoint: the root can be argued for and against.

Round 1: independent positions, then the numbers

1. 1Independent submission window. Each panelist adds supporting or opposing positions under the root — without seeing others' until submitted. This is the independence the method requires, enforced by the medium rather than by trust in the facilitator's process. Checkpoint: every panelist has ≥1 position; nobody has seen or edited anyone else's.
2. 2Optional: AI panelists, labelled. Seed additional positions from several models. Provenance is stamped on every argument, so machine positions are always distinguishable from expert ones. This widens the position space; it adds no expertise — the distinction matters and belongs in your report (see the hybrid-panel section below). Checkpoint: AI positions visibly labelled.
3. 3Round-1 ratings. Every panelist rates every position — one value plus a label, with the anchors you defined ("1 = implausible … 9 = near-certain"). This is the round's quantitative record. Checkpoint: complete rating matrix; anchors stated once, used consistently.

Controlled feedback that carries the reasoning

Now the step survey tools cannot do. The feedback artifact for Round 2 is not a summary the facilitator wrote — it is the tree itself: every position, in its author's (or model's) own words, with its rating distribution attached. Panelists see exactly which arguments the group found strong, which split the panel, and which sank — with no editorial paraphrase in between. The facilitator's job shrinks to logistics: announce the window, point at the tree.

Round 2: interrogate, challenge, reconcile

Round 2 is where classic Delphi says "revise in light of the feedback." With the reasoning visible, revision becomes active — three structured moves, each a four-turn dialogue with a position's author:

1. 1Interrogate — Q&A chain. Before revising toward a position, understand it: your question, the author's answer, follow-up, answer — complete. For AI positions, the model answers automatically, so the panel is never blocked waiting on one member. Checkpoint: positions that shifted opinion carry chains explaining why.
2. 2Challenge — Review chain. Evaluate a position as unsupported; the author responds; follow-up; response. N panelists challenging one position = N parallel chains — which is exactly the structure a Delphi wants: parallel independent critique, never one group thread for status to re-enter through. Checkpoint: contested positions carry completed challenges.
3. 3Reconcile near-duplicates — Compromise chain. Two experts holding almost the same position double-counts one view in the ratings. One proposes the merged formulation to the other, on the record. Checkpoint: no duplicate pair left unmerged or unexplained.
4. 4Round-2 ratings. Re-rate everything under the same anchors. Checkpoint: second complete matrix, ready for comparison.

Hybrid panels: adding models without adding expertise

Because AI-authored positions answer their own chains — routed back to the model that wrote them — a hybrid human/AI panel is genuinely practical: use models to broaden Round 1's position space, then let the human experts cross-examine the machine positions in Round 2 and get substantive replies without anyone staffing the machine's side. Be precise about what this buys: wider input, not added expertise. A model's position is a well-phrased pattern, not a career of judgment, and the labelling exists so your convergence numbers can be computed with and without the machine rows. When widening input helps and when it merely dilutes is its own literature — when crowds go wrong covers the failure modes; the flip side, why independent judgments aggregate so well when they are genuinely independent, is the Galton and Condorcet story.

Measuring convergence, reporting dissent

Convergence is the narrowing of rating distributions between rounds, and you measure it by comparing them — honestly, by hand or export: there is no built-in convergence statistic, no automatic Kendall's W, no stability test. Export the two matrices and compute what your field expects, or report the distributions qualitatively ("twelve of fourteen positions narrowed; two split further").

And report the dissent. A position where the panel stably disagrees across rounds — same spread, both poles defended in completed chains — is a finding, arguably the most valuable kind: it marks where expert judgment genuinely divides, with the reasoning for both poles attached. Averaging it away is the one unforgivable Delphi sin. The tree makes the dissent citable: the surviving objection, its author (or its anonymized public-tenant stand-in), and the challenges it survived.

Honest limitations

• ✗Independence is not anonymity — stated up front and worth restating: private-tenant arguments are attributed. If anonymity is methodologically essential, use the Argumentree.AI public tenant and say so in your write-up.
• ✗There is no round manager. No close-Round-1 button, no per-round visibility gating. Rounds are a facilitation convention you enforce with time-boxes and instructions — say so, and do not imply automation that is not there.
• ✗No built-in convergence statistic. Kendall's W, stability tests and the like are yours to compute from exported ratings.
• ✗Ratings are a single labelled value, not a Likert instrument with validated anchors. Define your anchors in the label and keep them constant across rounds.
• ✗A chain is four turns, then complete. An expert dispute that needs more gets a call — the chain records what was already asked and answered.

Practical lessons

• ✓Write the anchors before Round 1 and paste them into every window announcement. Half of all messy Delphi data is anchor drift.
• ✓Time-box windows generously across time zones — 72 hours per round works for international panels; the chains are async by construction.
• ✓Cap Round-1 positions per panelist (three is plenty). Delphi rewards considered positions, not volume.
• ✓Freeze the tree between rounds. Announce that submissions outside windows will be removed — the round convention only holds if the facilitator holds it.

The reply you can finally send

Back to the careful panelist's email: "Can I see the reasons? I'd like to know what I agreed with." In this version of the study, the answer is a link. Here are the fourteen positions; here is the one whose rating spread collapsed after two Review chains failed to dent it; here is the exchange that moved you and eleven others. Consensus with the reasoning attached — which was the method's promise all along.