How Not to Be Fooled (by Your Brain or Your Chatbot)
Install the course's decoding tool — the Ladder of Misinference — and discover, on yourself, that the shortcuts which mislead a judge mislead a language model too.
The hook
“A New York lawyer told a federal judge that ChatGPT “could not possibly be fabricating cases.” It had invented six. The shortcut that fooled him — a fluent, confident answer must be a true one — is the same shortcut your own mind runs a hundred times a day, and you are about to watch it run on you.”
What you'll be able to do
- Tell a statement from a fact, a fact from data, data from evidence, and evidence from proof — and apply that scrutiny to any confident claim, whether a person, a headnote, or a chatbot made it.
- Recognise System 1 and System 2 thinking, and feel anchoring move your own estimate before you watch it move a model's.
- Explain why identical fact patterns draw wildly different judgments — judicial sentencing variability as the live example — and why a single AI answer conceals the same variability.
- Leave with the first discipline of the course: never smuggle the answer you want into the question.
On the syllabus
- System 1 vs. System 2 thinking; anchoring and priming
- The Ladder of Misinference as a tool to interrogate any confident claim
- Why identical fact patterns can produce wildly different human judgments (judicial sentencing variability as a live example)
- How the same anchoring effects that sway juries and judges also sway an AI model's estimates
- The Indian stakes: cost orders, a recalled tribunal order, and a Supreme Court warning about fake AI-generated judgments
In short
The opening hour hands you the course's master decoding tool — the Ladder of Misinference: statement → fact → data → evidence → proof — and demonstrates, on the room itself, that the cognitive shortcuts which mislead human judgment are the shortcuts that mislead a large language model. Anchoring is run live on the class before it is run on a chatbot; the sentencing studies show how much noise sits inside expert judgment; and the Indian cost orders show what happens when a lawyer takes a fluent answer for a proven one. The hands-on half hour puts three confident legal claims — one human, one from a headnote, one you generate yourself — on the Ladder.
Why it matters for using AI well
Every later session climbs the Ladder on a real AI output. Once you can say which rung a claim actually sits on — and notice when an anchor has already moved you — a confident chatbot answer stops being persuasive on its own, and you start asking the only question that would have saved the lawyer in Mata: how do I actually know this is true?
What they leave with
The skill
Climb the Ladder on any confident claim, and write neutral prompts that do not smuggle in the answer you want.
The insight
You are not a neutral instrument, and neither is the machine — and both were moved by the same trick.
The moment they remember
The room is split without knowing it. Half see a low number, half a high one, and everyone then estimates the same figure. The two group averages go up on screen, often two or three times apart — and nobody in either half felt manipulated. Then the identical anchor is planted in a chatbot's prompt on the projector and its estimate moves the same way. The moment students remember is not a fact about AI; it is discovering that they were moved, silently, in the first fifteen minutes of the course.
In this session
- 01
System 1 and System 2: the fast, automatic mind that produces a confident answer before the slow, effortful mind has checked it — and why a chatbot's fluent first answer is precisely a System 1 product, delivered without the System 2 pass.
- 02
Anchoring and priming: a number or a framing you met seconds ago drags your estimate toward it. The sentencing-dice study described in Thinking, Fast and Slow — judges who rolled a higher number handed down longer sentences — is the legal version; the identical pull appears in juries, in damages awards, and in a model's estimate once a figure sits in the prompt.
- 03
The Ladder of Misinference (Edmans, May Contain Lies) as the tool for interrogating any confident claim: statement → fact → data → evidence → proof. Most claims present themselves several rungs higher than they actually sit, and a hallucinated citation is a statement costumed as binding proof.
- 04
Noise: why identical fact patterns produce wildly different human judgments. The sentencing-variability studies collected in Kahneman, Sibony & Sunstein's Noise — the same case file, sharply different sentences from different judges — are the live example, and the first warning that one AI answer is a sample rather than a verdict.
- 05
The cautionary cases that set the stakes: Mata v. Avianca (S.D.N.Y. 2023), where ChatGPT fabricated six cases; the Bombay High Court's ₹50,000 cost order for a non-existent judgment (2026); and the Supreme Court of India's observation, in a pending 2026 matter, that a decision built on fake AI-generated judgments would be misconduct.
- 06
Why AI inherits our shortcuts: a model trained on human-generated text learns the patterns of human reasoning, including its biases — which is why one set of thinking skills decodes both the brain and the machine, and why this is a thinking course before it is a technology course.
The four-step mirror
Run it on the class. Then on the machine.
An experiment on the room, the same effect explained in the model, a live demonstration on a real chatbot, and a named takeaway skill.
Anchoring on a legal estimate — the flagship experiment
On the class
The room is split in two. Half see a low number, half a high one, before everyone estimates the same figure — the likely damages in a fact pattern, or the share of bail applications that succeed. The two group averages go up on screen, and the gap between them is the anchor's work.
In the model
A language model is just as suggestible. A figure or a framing planted in the prompt drags its estimate toward it, exactly as the two halves of the room were dragged.
Live chatbot
The presenter gives a chatbot a high anchor, then in a fresh session a low one, before the same estimation question — and the answer moves with the anchor, live on the projector.
The skill
Write neutral prompts. Never smuggle the answer you want into the question — and notice when a figure in an opponent's submission is doing the same to you.
Climb the Ladder on a live answer
On the class
A real legal question is put to the room. Most people's instinct is to accept a confident, well-formatted answer — the over-trust the Mata lawyer showed.
In the model
A model delivers every statement in the register of settled authority; fluency and formatting signal a certainty the model has no basis for. The answer arrives pre-promoted to the top of the Ladder.
Live chatbot
The same question goes to a chatbot. Together, the class climbs the Ladder on its answer — demoting each claim to the rung the evidence actually supports.
The skill
Ask which rung you are actually on. A statement is not a fact; a fact is not proof — from a person or from a model.
Same facts, different sentences — noise in judgment
On the class
The whole room sentences the same short fact pattern — a first offence, a stated value, a plea — on a 0–10 scale of years. The spread goes up on screen. Identical facts, one room, a wide range of sentences: that is the noise the sentencing studies found in real judges.
In the model
A model's answer to a question of judgment is sampled, not fixed. Ask it to sentence the same fact pattern in fresh sessions and its figure moves too — delivered with the same confidence each time.
Live chatbot
The presenter gives a chatbot the room's fact pattern and asks for a sentence, then repeats the identical prompt in a fresh session. Two answers; neither flagged as uncertain.
The skill
Identical facts do not produce identical judgments — from a bench or from a model. Treat one answer as a sample: ask again, vary the framing, and look at the spread before you rely on it.
Hands-on · on your own laptop
Spot the Rung
Take three confident statements on a legal point — one made by a human, one drawn from a reported case headnote, and one you generate yourself with a chatbot on your own laptop — and climb the Ladder of Misinference on each, classifying where the claim actually sits rather than where it presents itself. Then name the one verification step that would promote each claim a rung.
Run of show · 30 minutes
- 0–5 min — On your laptop, prompt a chatbot for a confident assertion on a point in your own area of interest. Capture it verbatim, formatting and all.
- 5–15 min — Place all three statements on the Ladder: the advocate's assertion, the headnote, and the chatbot's answer. For each, mark the rung it claims and the rung the evidence actually supports.
- 15–25 min — For each statement, write the single verification step that would promote it one rung — and say honestly whether you could complete that step in the next ten minutes.
- 25–30 min — Three students read theirs out; the room votes on the rung before the author reveals their own classification.
Deliverable
A three-row Ladder worksheet: each statement placed on the rung it claims and the rung it earns, with a one-line justification and the specific verification step that would promote it.
Key sources & cases
Mata v. Avianca, Inc., 678 F. Supp. 3d 443 (S.D.N.Y. 2023)
Judge Castel; $5,000 sanction; six fabricated cases; the lawyer's fatal assumption that ChatGPT could not be fabricating cases. The flagship cautionary tale and the course's framing device.
Gummadi Usha Rani v. Sure Mallikarjuna Rao (SC of India, SLP (C) No. 7575/2026)
A pending Special Leave Petition in which the Supreme Court, on a trial court order built on fake AI-generated judgments, observed that such a decision “would be a misconduct and legal consequence shall follow” and issued notice. Not a final holding; the India-first anchor for why verification is a duty.
Deepak v. Heart & Soul Entertainment Ltd. (Bombay HC, 7 Jan 2026)
₹50,000 cost order for unverified AI-generated written submissions citing a non-existent judgment — a concrete Indian consequence for a skipped verification step.
Alex Edmans, May Contain Lies (2024)
Source of the Ladder of Misinference (statement → fact → data → evidence → proof) and of the warning that smart people are better at biased search.
Daniel Kahneman, Thinking, Fast and Slow (2011)
System 1 and System 2; anchoring and priming, including the sentencing-dice study on judges it describes (Englich, Mussweiler & Strack, 2006).
Kahneman, Sibony & Sunstein, Noise (2021)
Judicial sentencing variability — identical case files, widely different sentences — as the live example of noise in expert judgment.
Readings
- Alex Edmans, May Contain Lies (2024) — the Ladder of Misinference and biased search
- Daniel Kahneman, Thinking, Fast and Slow (2011) — System 1 / System 2; anchors
- Kahneman, Sibony & Sunstein, Noise (2021) — noise in judicial sentencing
- Mata v. Avianca, Inc., 678 F. Supp. 3d 443 (S.D.N.Y. 2023)
- Gummadi Usha Rani v. Sure Mallikarjuna Rao (SC of India, 2026, pending)
- Deepak v. Heart & Soul Entertainment Ltd. (Bombay HC, 2026)
Next session
Session 02 / 08
Inside the Machine — How It Learns, and Why It Must Get Things Wrong
Decode the machineSixteen hours, one professional discipline.
Using AI well is not a knack — it is competence, candour and verification, practised until they are habits you could defend in court.