The Lawyer's Blind Spots: Cognitive Bias, Statistics, and the Sycophantic Bot
The same biases that mislead judges and juries are baked into your prompts and your model's training — learn to counter both.
The hook
“A confident advocate plus a model trained to flatter you is biased search on steroids — and your client pays for it.”
What you'll be able to do
- Map human cognitive bias and statistical error onto both legal judgment (judges, juries, advocates) and LLM output, recognizing them as one body of material with multiple professional payoffs.
- Identify model sycophancy as a distinct professional trap: the danger of asking AI to confirm your client's position rather than test it.
- Spot base-rate neglect, the prosecutor's fallacy, and correlation/causation errors in evidence and expert testimony, and trace how the same framing effects enter AI prompts and training data.
- Adopt the counter-move — force the model to argue the other side rather than letting it confirm you.
In short
This module reframes the cognitive biases and statistical traps of human legal judgment as simultaneously flaws in lawyers, the failure modes of LLMs that learned from us, and subjects of substantive evidence law. Its centerpiece is sycophancy: because models tend to agree with and flatter the user's framing, a lawyer who asks AI to confirm a client's position gets confident agreement, not scrutiny. The skill is to make the model argue against you.
The AI bridge
The same framing effects that sway juries are baked into prompts and training data; a confident advocate plus a sycophantic model equals biased search on steroids. Knowing this, you prompt to disconfirm — making the model build the strongest case against your position so AI tests your reasoning instead of merely flattering it.
In this module
- 01
Anchoring distorts legal estimation — including sentencing and quantum anchors. A number smuggled into a prompt moves the model's output just as a number smuggled into argument moves a judge; never plant the answer you want.
- 02
Confirmation bias and motivated reasoning in human advocates have a direct machine parallel in LLM sycophancy: ask AI 'I'm right, aren't I?' and it tends to agree, so the danger is asking AI to confirm your client's position rather than stress-test it.
- 03
Base-rate neglect and the prosecutor's fallacy corrupt how statistics enter evidence and expert testimony; R v Sally Clark shows a wrongful conviction driven by the 'one in 73 million' statistical fallacy in expert evidence — the definitive law-meets-statistics cautionary tale. Verify any statistic an AI cites against base rates before relying on it.
- 04
Correlation is not causation, and the misuse of statistics in evidence and expert testimony is a substantive litigation risk; the same fragile inferences appear in confident AI output and must be interrogated, not accepted on fluency.
- 05
Noise — identical cases drawing different sentences — shows even expert legal judgment is inconsistent; an AI's single confident answer can mask the same variability, so re-run, vary the framing, and compare rather than trust one pass.
- 06
Edmans's point that smart people are better at biased search applies acutely to top law students paired with a sycophantic model: the more skilled you are at finding support, the more dangerous unverified AI agreement becomes.
The interactive demos
Every idea is a Mirror Move
Run it on the room, show it inside the machine, prove it live on a real AI, then name the skill.
The Sycophantic Bot
On us
Vote on a contested legal proposition using the live-polling app to surface where the room already leans.
In the machine
Explain that LLMs tend to agree with and flatter the user's framing — the machine mirror of human confirmation bias and motivated reasoning.
Live AI
Ask a chatbot 'my client's position is X — I'm right, aren't I?' then, in a fresh session, 'make the strongest case against X,' and compare the two answers side by side.
The skill
Make the model argue the other side; never let it just confirm you.
The lab
Steel-man your opponent
Students prompt AI to produce the strongest possible counter-argument to their own moot position, then critique the quality of what the model returns — practicing disconfirming prompting against sycophancy.
Deliverable
An AI-generated counter-argument to the student's own moot position plus a written critique assessing its strength, gaps, and any sycophantic weaknesses.
Key sources & cases
Kahneman, Thinking, Fast and Slow
Source for anchoring and the cognitive biases (incl. sentencing/quantum anchors) reframed as both human and machine failure modes.
Kahneman, Sibony & Sunstein, Noise
Used for noise — identical cases drawing different sentences — evidencing inconsistency in judges' legal judgment.
Edmans, May Contain Lies
The Ladder of Misinference and the point that smart people are better at biased search — the course's answer-decoder spine, acute for sharp law students.
Huff, How to Lie with Statistics
Source on the misuse of statistics; teach with awareness of the Huff credibility irony per the course's intellectual-honesty rules.
Spiegelhalter, The Art of Statistics
Used for base rates and the Harold Shipman detection example underpinning base-rate reasoning.
R v Sally Clark (UK)
Wrongful conviction driven by a statistical fallacy in expert evidence (the 'one in 73 million' error) — the definitive law-meets-statistics cautionary tale; the prosecutor's fallacy generally.
Readings
- Kahneman, Thinking, Fast and Slow (2011)
- Kahneman, Sibony & Sunstein, Noise (2021)
- Edmans, May Contain Lies (2024) — the Ladder of Misinference
- Huff, How to Lie with Statistics (1954)
- Spiegelhalter, The Art of Statistics (2019)
- R v Sally Clark (UK, 2003)
Next module
Module 04 / 08
Decoding the Answer: Verification & the Duty to Check
VerifyBring a credit-bearing AI course to your students
A 16-hour course that treats using AI well as a professional duty — one a council can approve, and a graduate can defend in court.