Inside the Machine — How It Learns, and Why It Must Get Things Wrong
Open the black box early: prediction, training data and sampling — and watch every failure mode you will meet for the rest of the course fall out of one mechanism.
The hook
“It is not a brain and it is not a database. It is a machine that predicts the next plausible word — and once you understand that single sentence, hallucination, sycophancy, noise and bias stop being surprises and become predictions you can make in advance.”
What you'll be able to do
- Explain, in plain terms a non-specialist could restate, what a language model does: training data, learned patterns, and prediction of the next plausible token — not retrieval of a stored fact.
- Derive the four predictable failure modes — hallucination, sycophancy, noise and inherited bias — from that mechanism, rather than memorising them as a list.
- Name automation bias as the lawyer's own occupational hazard: the human tendency to defer to a machine's output precisely when it is most confident.
- Distinguish a generative tool from a grounded one, and know what a knowledge cutoff tells you about where to verify hardest.
On the syllabus
- Training data, patterns and prediction — what a model actually does, and what it never does
- Why hallucination is structural: it knows the shape of a citation, not the contents of any database
- Why the same prompt gives different answers (sampling), and why it agrees with you (helpfulness training)
- “Garbage in, garbage out” and algorithmic bias — COMPAS, State v. Loomis, and the recruiting model that learned to prefer men
- Automation bias: the human half of the failure, and the one the profession keeps paying for
- Knowledge cutoffs, generative versus grounded tools, and what “reasoning” models do and do not change
In short
Session 2 opens the black box, and it opens it early — because everything the rest of the course teaches is derivable from one mechanism. A model is trained on an enormous body of human text, learns the statistical shape of it, and generates by predicting the next plausible token. It never looks anything up. From that single fact the session derives why hallucination is structural rather than accidental, why the same prompt gives different answers, why the model agrees with the way you framed your question, and why it inherits every prejudice in its training data. The hour ends on the distinction the whole back half of the course rests on — generative versus grounded — and on automation bias, the failure that lives in the lawyer rather than the machine. The hands-on half hour has students train a classifier, deliberately poison it, and then elicit and fail to verify a fabricated Indian citation.
Why it matters for using AI well
You stop being surprised. Understanding that the machine predicts rather than knows lets you say in advance where it will fail — citations, figures, recent events, anything niche, anything you framed for it — and that prediction is what tells you where the duty to verify actually bites.
What they leave with
The skill
Ask what a tool is actually doing — predicting or retrieving — and what it was trained on, before you ask what it thinks.
The insight
Hallucination is not a bug that will be patched; it is what a next-token predictor does when the training is thin, and it will always sound exactly as confident as the truth.
The moment they remember
Students train a working image classifier on the room in about two minutes, then deliberately feed it lopsided examples and watch it fail — confidently — on the very next image. Building and breaking a real machine-learning system inside a single hands-on segment turns COMPAS from a case in a reading list into something they have just done with their own hands. The line that lands is the one a student usually says out loud: “so it never knew anything at all.”
In this session
- 01
Training data → patterns → predictions: a model is not a database of facts but a system that has learned the statistical shape of an enormous body of human text. Shown with a classifier trained live on the room, with no mathematics required.
- 02
Overfitting and shortcut learning: a model that learns the wrong pattern — the detector that learned the weather rather than the tank, the image model that calls a goat in a tree a giraffe — is confidently wrong in ways nobody intended (Shane, You Look Like a Thing and I Love You).
- 03
Hallucination, derived: a model that predicts the next plausible token knows exactly what an Indian citation looks like — party names, reporter, year, page — but has no connection to any case database. On a niche point, the statistically likely output is a citation that is formally perfect and entirely fictional. This is the Mata mechanism, and it is a property of the design rather than a defect in it.
- 04
Noise and sycophancy, derived: output is sampled from a distribution, so the same prompt run twice can give two different answers; and a model trained to be helpful tends to agree with the framing it is given. Ask “I'm right, aren't I?” and you will usually be told that you are. Both are consequences of the mechanism, not moods.
- 05
Garbage in, garbage out: a model inherits the biases of its data. COMPAS, the recidivism risk tool ProPublica found produced racially skewed error rates (2016); State v. Loomis (Wis. 2016), where a defendant challenged sentencing on an opaque score; and the recruiting model Amazon scrapped after it learned to downgrade CVs containing the word “women's” (reported by Reuters, 2018). Where you cannot see how a score was reached, you cannot interrogate it.
- 06
Automation bias — the failure that is ours, not the machine's: people systematically over-defer to an automated output, and defer hardest when it is fluent, formatted and fast. For a profession whose product is judgment, this is the occupational hazard the rest of the course is built to counter.
- 07
Knowledge cutoffs and thin training: training stops at a date and is sparse in places, so a model will answer confidently about recent judgments, small jurisdictions and local practice it never saw. The cutoff is a map of where to verify hardest.
- 08
Generative versus grounded: a chatbot composes plausible text; a retrieval-grounded tool answers from a document it has actually fetched and can cite. Knowing which kind of tool you are holding tells you whether its citations can be trusted at all — the distinction Sessions 6 to 8 are built on. Newer “reasoning” models spend more computation before answering and do better on multi-step problems; they do not acquire a connection to a case database, and they hallucinate with the same confidence when the training is thin.
The four-step mirror
Run it on the class. Then on the machine.
An experiment on the room, the same effect explained in the model, a live demonstration on a real chatbot, and a named takeaway skill.
Train it, then break it
On the class
The room trains a small image classifier live in two minutes — then feeds it deliberately lopsided data and, after voting on whether it will, watches it fail on the next example.
In the model
Every model is a compression of its training data. Skew the data and you skew the predictions — the cleanest possible lesson in bias and garbage-in, garbage-out. COMPAS's skewed error rates and the recruiting model that learned to rank men higher are the same mechanism at scale.
Live chatbot
The presenter asks a chatbot to rank two identical CVs that differ only in the candidate's name, or to complete “The nurse said… / The engineer said…”, and shows the skew live — then names it: the toy classifier, the chatbot, COMPAS and the hiring model all inherited their data.
The skill
Biased data makes biased AI. Ask what the model was trained on before you ask what it thinks.
Confident confabulation — watch it invent the law
On the class
Before the reveal, the room predicts: asked for Indian case law on a deliberately niche point, will a generic chatbot refuse, give a real citation, or confidently produce something?
In the model
The model knows what an Indian citation looks like — party names, reporter, year, page — but has no connection to any case database. The statistically likely output is a fluent, formally perfect, fabricated citation, and the model has no internal representation of “I don't know” to fall back on.
Live chatbot
The presenter asks, and the chatbot obliges with plausible authorities. The same query goes to Indian Kanoon; the cases are not there.
The skill
A confident citation is a statement, not proof. Fluency is a property of the machine, not evidence about the world.
Same prompt, different answer — and the model that agrees with you
On the class
Vote: how confident are you that an AI gives the same answer to the same question every time? Most of the room is very confident. Second vote: will it change its answer if you push back?
In the model
Outputs are sampled rather than looked up, so they vary run to run; and helpfulness training makes the model tend toward the framing it was handed. Both fall out of the mechanism explained in the first half of the hour.
Live chatbot
The presenter pastes an identical legal question into three fresh sessions and reveals three different answers, then takes one answer and simply disagrees with it — and the model folds.
The skill
Run important prompts more than once, and treat agreement as worthless until you have made the model argue the other side.
Hands-on · on your own laptop
Break It and Catch It
Two exercises that together make the mechanism concrete. First, train a small classifier in the browser, deliberately skew its training data, and document the confident failure that follows. Second, elicit a fabricated Indian citation from a generic chatbot and attempt — and fail — to verify it on a grounded source, recording the tells.
Run of show · 30 minutes
- 0–10 min — In pairs, train a Teachable Machine classifier on two categories, then retrain it with deliberately lopsided examples and record what it now gets confidently wrong.
- 10–20 min — On your laptop, prompt a generic chatbot for authority on a deliberately niche point of Indian law, or on a judgment after its cutoff. Capture the output verbatim, including how certain it sounded.
- 20–27 min — Attempt to verify every citation on Indian Kanoon. Record the exact point at which verification failed.
- 27–30 min — Post your fabrication to the room's shared wall; the common tells are collected on screen.
Deliverable
A two-part record: the skew-and-fail note from your classifier, and one documented fabrication — the prompt, the fabricated citation verbatim, the verification attempt, the point of failure, and the tells that marked it.
Key sources & cases
Janelle Shane, You Look Like a Thing and I Love You (2019)
The accessible, funny on-ramp to how AI actually works and fails: giraffing, recipes calling for broken glass, and the tank/sunny-day shortcut.
Hannah Fry, Hello World (2018)
Plain-language framing of how algorithms and machine learning work and where they go wrong — in justice, medicine, and cars.
Cathy O'Neil, Weapons of Math Destruction (2016)
Opaque, biased, self-reinforcing algorithmic decision systems — in criminal justice, hiring, and credit.
ProPublica, “Machine Bias” (2016)
Angwin et al.'s investigation finding racial disparities in the COMPAS recidivism risk tool; the empirical backbone of the bias discussion (an investigative report, not a case).
State v. Loomis, 881 N.W.2d 749 (Wis. 2016)
A defendant challenged the use of an opaque COMPAS risk score in sentencing; the Wisconsin Supreme Court upheld its use, with limits. Comparative (US).
Reuters, “Amazon scraps secret AI recruiting tool that showed bias against women” (10 Oct 2018)
The well-known hiring case: a recruiting model trained on ten years of past CVs learned to penalise applications that mentioned women. A news report, not a judgment — verify to source before teaching.
Goddard, Roudsari & Wyatt, “Automation bias: a systematic review of frequency, effect mediators, and mitigators,” JAMIA 19(1) (2012) 121–127
The systematic review of automation bias — the tendency to over-rely on automated output — across clinical decision-support systems, covering its frequency, what makes it worse, and what mitigates it. Verified 2026-08-26 (JAMIA / PubMed 21685142; DOI 10.1136/amiajnl-2011-000089). Teach the mechanism, not a headline percentage: the review reports a range across studies, not one figure.
Google Teachable Machine
The browser tool used to train — and deliberately break — a classifier live with the room.
Readings
- Janelle Shane, You Look Like a Thing and I Love You (2019)
- Hannah Fry, Hello World (2018)
- Cathy O'Neil, Weapons of Math Destruction (2016)
- ProPublica, “Machine Bias” (2016)
- State v. Loomis, 881 N.W.2d 749 (Wis. 2016)
- Alex Edmans, May Contain Lies (2024) — why intelligence makes biased search more sophisticated
- Goddard, Roudsari & Wyatt, “Automation bias: a systematic review…,” JAMIA 19(1) (2012) 121–127
Next session
Session 03 / 08
Thinking Like a Computer I — Decomposition & Pattern Recognition
Direct the machineSixteen hours, one professional discipline.
Using AI well is not a knack — it is competence, candour and verification, practised until they are habits you could defend in court.