Inside the Machine — How It Works, Why It Fails, and How Not to Be Fooled
The mechanism (prediction, never retrieval) and its failure modes, so the Session 1 craft rests on understood limits.
- 90 min
- 60 taught + 30 lab
- Direct the machine, decode it
- Mirror · experiment on the class
- Your laptop · free tier
The lab · 30 min · Recall versus reading
Make It Invent, Then Make It Read. Authorities elicited and checked live on Indian Kanoon (one usually dissolves), then a supplied judgment answered reliably with pinpoints.
The full lab planThe hook
“On 2 July 2026 the Supreme Court of India set aside a tribunal's order because the 'authorities' it rested on did not exist. A machine had predicted them, and nobody checked. The machine was not broken; it was doing what it is built to do. By the end of this hour you will know why, and where it will do it next.”
What you'll be able to do
- Explain, in terms a judge would accept, what a language model does: it predicts the next plausible token from patterns in its training data and never looks anything up. Derive from that one fact why it fabricates, agrees with you, varies run to run, and moves toward a figure planted in your prompt.
- Predict, before prompting, which side of the line a task falls on: reshaping text you supply (its reliable ground) or recalling what it never stored (citations, figures, anything recent, anything niche).
- Place any confident output on the Ladder of Misinference (statement → fact → data → evidence → proof) and treat a fluent citation as a statement costumed as proof until it is traced.
- State the Indian legal stakes with precision: what the Supreme Court has now held, what it has only observed in a pending matter, and what the press attributed that the orders never said.
On the syllabus
- Prediction from patterns, and why it answers instead of admitting ignorance
- The failure modes derived: hallucination, sycophancy, noise, anchoring
- Strengths: reshaping supplied text versus recalling unsupplied facts
- The Ladder of Misinference, for any confident claim
- The Indian stakes: the 2026 holding, the pending matter, the cost orders
The session
How a language model works and why it makes things up. A model predicts likely next words from patterns in its training data; it does not look facts up, so it can invent citations, agree too readily, and give different answers to the same question. The session covers the main failure modes and the Indian case law on citing unverified AI output.
Why it matters
You stop being surprised, and you stop over-correcting. Knowing the mechanism tells you in advance where the model will fail (citations, figures, recency, anything you framed) and where it is strong, so you use it hard on the right tasks instead of avoiding it on all of them. That prediction, made before every prompt, separates a tool from a trap.
What you can do on Monday
Before you ask the model anything, say which side of the line the task is on (reshaping text you supply, or recalling what it never stored), and check its first citation on Indian Kanoon before you believe the second.
What you leave with
The skill
Predict the failure before you prompt: reshaping what you supply is safe ground; recall is not. Any authority the model names goes to Indian Kanoon before it goes anywhere near your notes.
The insight
Hallucination is what a next-token predictor trained to always answer does when the training is thin. No patch is coming, and it always sounds as confident as the truth.
The moment you'll remember
The anchoring split. Half the room sees a low number, half a high one, and everyone estimates the same figure; the two group averages go up on screen, often multiples apart, and nobody felt a thing. Then the same anchor is planted in a chatbot's prompt on the projector, and its estimate moves the same way. What students remember is not a fact about AI. It is that they were moved, without feeling it, in the session's first fifteen minutes, by the trick that moves the machine.
In this session
- 01
The mechanism in one sentence a non-specialist can carry: a model predicts the next plausible token from the statistical shape of everything it read. It does not consult a database, and it has no internal flag separating what it knows from what it is composing. (The apps can now bolt on web search and file reading; that is retrieval, Session 3's subject. The bare model still predicts.) Everything else this hour is a corollary.
- 02
Why it never says 'I don't know' unless invited: models are optimised like good test-takers, and under the scoring they are trained against, a confident best guess beats an honest blank (Kalai, Nachum, Vempala & Zhang, 'Why Language Models Hallucinate', 2025). Confidence is a property of the training and carries no evidence about the world. That is why permission to abstain, the line your Session 1 contract already carries, measurably changes behaviour.
- 03
Hallucination, derived: the model knows what an Indian citation looks like (party names, reporter, year, page) and has no connection to any case database. On a niche point the statistically likely output is a formally perfect, fictional authority. This is the mechanism behind Mata v. Avianca and behind the six 'authorities' in the order the Supreme Court set aside in July 2026.
- 04
Sycophancy and noise, derived: helpfulness training makes the model lean toward your framing, so it folds when you disagree, and 'I'm right, aren't I?' usually gets a yes. Output is sampled, so the identical prompt in three fresh chats returns three answers, each delivered with the same certainty. Run anything that matters more than once, and never treat agreement with you as evidence.
- 05
Anchoring, felt then shown: the room is split, given a low or a high number, and asked for the same estimate, and the gap between the halves goes on screen before anyone is told why. Then the same anchor is planted in a model's prompt, and its estimate moves the same way. Neither you nor the machine is a neutral instrument. Write prompts that do not smuggle in the answer you want.
- 06
'Reasoning' and 'thinking' modes: they spend more computation before answering and help on multi-step problems, and they still fabricate when the training is thin. OpenAI's own system card measured its o3 reasoning model hallucinating on 33% of person-fact questions against 16% for its predecessor. More thinking is not more knowing; Session 5 teaches when the extra computation is worth paying for.
- 07
The pattern of strengths, as a prediction you can make before prompting: strong wherever it reshapes text it has been given (summarise, restructure, translate register, compare); weak wherever it must supply what it was never given (citations, quotations, figures, arithmetic, anything after its training cutoff, anything niche). The room votes on five tasks before each is run, and the reveals sort themselves onto that line.
- 08
The Ladder of Misinference (Edmans): statement → fact → data → evidence → proof. Most claims present themselves rungs above where they sit, and a hallucinated citation is a statement costumed as binding proof. In its legal reading (statement → fact → authority → binding authority → settled law) the Ladder is the course's verification spine, drilled to a habit in Session 6.
- 09
The Indian stakes, stated with the precision this course practises. Pooja Ramesh Singh v. J&K Bank (2026 INSC 668, 2 July 2026): a reportable holding that citing unverified AI-generated authorities is misconduct for an advocate and a 'serious lapse' for a judge, and that a decision resting on 'an iota' of hallucinated material is 'no decision in the eyes of the law'. Gummadi Usha Rani (SLP (C) 7575/2026): the pending matter where the same bench first said so, as an observation rather than a holding. Deepak v. Heart & Soul (Bombay HC, 7 Jan 2026): ₹50,000 in costs on a litigant in person, with a warning extended to advocates. Buckeye Trust (ITAT Bengaluru): an order recalled within days for 'inadvertent errors'; the AI attribution comes from the press, and the Tribunal's order never makes it. Worldwide, a public database tracked roughly 1,700 such decisions by July 2026, up from about 700 in January. The discipline is to state each at its register: holding, observation, order, inference.
The mirror
Run on the class, then on the machine.
An experiment on the class, the same effect explained in the model, a live demonstration on a real tool, and a named takeaway skill.
The anchoring split — the flagship experiment
Experiment on the classOn the class
Split in two, half the room sees a low number and half a high one before everyone estimates the same figure (damages in a fact pattern, or the share of bail applications that succeed). Both averages go on screen; the gap is the anchor's work, and nobody felt it.
In the model
A model is just as suggestible, for the same statistical reason: a figure sitting in the prompt is context that drags the prediction toward it.
Live on the model
The presenter plants the high anchor, then the low, in two temporary chats before the same estimation question, and the model's answer moves with the anchor, live on the projector.
The skill
Write neutral prompts: never smuggle the answer you want into the question, and notice when an opponent's figure is doing the same to you.
Will it get this right? — the strengths map
Experiment on the classOn the class
Five tasks go up one at a time (summarise this paragraph, count its words, name the leading case on a point, rewrite this clause in plain English, quote the operative sentence of a judgment it has not seen) and the room predicts each before it runs.
In the model
The strengths are systematic: reshaping supplied text sits on one side of the line, supplying unsupplied facts on the other. The mechanism predicts every reveal.
Live on the model
The five tasks run in sequence; the room's wrong predictions, usually over-trusting recall and under-trusting reshaping, are the lesson.
The skill
Sort the task before you prompt: reshaping, or recall? Lean hard on the first; verify everything from the second.
Push back, and run it thrice
DemonstrationThe room predicts
The room votes twice: will the model give the same answer to the same legal question three times running, and will it hold its position when the presenter simply disagrees?
What is going on
Sampling makes outputs vary run to run; helpfulness training makes the model fold toward the user's framing. Both follow from the training.
Shown live
The identical question goes into three temporary chats and returns three answers. Then the presenter takes one answer and pushes back without any new argument, and the model concedes.
The skill
Run anything that matters more than once, and treat the model's agreement with you as worthless until you have made it argue the other side.
The legal thread
Since Pooja Ramesh Singh (2026 INSC 668), citing unverified AI-generated authority is professional misconduct in India, and a decision built on hallucinated material is no decision at all. The duty to verify follows from the mechanism taught this hour.
The lab · 30 minutes · demonstrated in class, repeated on your own
Make It Invent, Then Make It Read
Authorities elicited and checked live on Indian Kanoon (one usually dissolves), then a supplied judgment answered reliably with pinpoints. Afterwards, students run the same pair on their own point and submit the record.
1 · The demonstration
In class, on one realistic task.
On
ChatGPT (free tier, in a temporary chat so memory cannot contaminate the demo), with Indian Kanoon open in a second tab
The exact prompt
I am preparing a note for a senior. List three Indian High Court or Supreme Court decisions on whether a landlord under a leave-and-licence arrangement may forfeit the entire security deposit when the licensee leaves before the lock-in period ends. Give full citations and one line on what each held.
Point at
The citation format is immaculate (party names, reporter, year, page) and the register is that of settled authority. Then search each on Indian Kanoon in the second tab, in front of the room. Point at the moment the first search comes back empty, and at how differently the room now reads the remaining two.
What comes back
Two or three plausible authorities, typically at least one untraceable and one real but misdescribed. If all three check out, say so (that is a finding too) and teach from the pre-captured transcript instead.
If it misbehaves
The pre-captured fabrication transcript and its failed Indian Kanoon searches (fallback folder, S2); or run the four citations from page 28 of the Buckeye Trust order live, which are the real thing.
2 · Afterwards, on your own
The same technique on a task of your own; the deliverable goes to the platform.
Afterwards, on your own: verify the authority you brought from Session 1's homework, elicit two or three more on a point from your own work, log each verdict on Indian Kanoon (found / not found / says something else), then paste a judgment excerpt from the session set into a fresh chat and check one pinpoint. The record is the platform submission.
Step by step
- 01Start with the authority you brought from Session 1's homework, then, in a temporary chat (memory off), elicit two or three more on a niche point from your own work (your moot, your last internship task, your seminar topic) and capture the output verbatim, formatting and all.
- 02Verify each authority on Indian Kanoon and log the verdict: found / not found / found-but-says-something-else, with the query you used.
- 03Open one real judgment from the session's excerpt set, paste the relevant portion into a fresh temporary chat, and ask your question about that text, requiring paragraph numbers.
- 04Check one pinpoint against the actual text, and note the difference in reliability between the recalled answers and the read ones.
- 05Write one line naming the tell you will watch for next time: the thing that should have made you suspicious.
Tools · free tier
- ChatGPT or Gemini (free tier, temporary chat)
- Indian Kanoon (free; no account needed)
- The session's judgment-excerpt set (text-layer PDFs, sized for free context windows)
Free tier
Six to eight short, text-only prompts, inside every free tier and every free context window, because you paste an excerpt rather than the judgment. Indian Kanoon needs no account. If the free tier rate-limits, space the runs out; the checking, not the prompting, is the skill.
If the free tier locks you out
If the free tier rate-limits, space the runs out or draft offline and run them when the window resets; the checking, not the prompting, is the skill.
3 · The reveal
The room commits before each check: will this authority survive Indian Kanoon? The reveal is the tally of predictions against results, recall beside reading.
Deliverable
A one-page 'recall versus reading' record: each elicited authority verbatim with verdict and trail, the grounded re-run with its checked pinpoint, and one line naming the tell you will watch for next time.
Feeds the Lab deliverablesRun of show · 30 minutes
- 0–18 min — The demonstration: authorities elicited, the room predicting each one's survival, every citation checked live on Indian Kanoon.
- 18–26 min — The grounded re-run: a judgment excerpt supplied, pinpoints demanded, one opened against the PDF.
- 26–30 min — The assignment: the same recall-versus-reading record on your own point, due before Session 3; questions.
Verification checklist
- Every elicited authority was checked on a grounded database, and the verdict and query are logged.
- The grounded re-run used supplied text, demanded paragraph numbers, and at least one pinpoint was opened and read.
- The record distinguishes cleanly between what the model recalled and what it read.
- Nothing unverified is described in the record as true; unverified items are recorded as claims.
For the instructor · before the session
Why this shape
You cannot build-along a bias: the anchoring split and the sycophancy fold only teach if the room feels them first, so this is the course's purest mirror. It is also the one full treatment of the failure modes; every later session treats hallucination, sycophancy and noise as consequences to engineer around.
- Create fresh temporary-chat sessions and confirm memory/personalisation is off on the demo account; a contaminated context breaks both the anchoring and the fabrication demos.
- Pre-test the fabrication prompt that morning; models change weekly. Turn web search OFF for the fabrication demo (a search-grounded answer changes the failure mode; if the app insists on searching, show that, name it as retrieval, and use the pre-captured transcript). If the free tier declines to fabricate, switch to the transcript and say why.
- Open the Indian Kanoon searches for the demo citations in pinned tabs; keep direct document URLs for the excerpt set so sixty students are not searching at once.
- Load the fallback folder: fabrication transcript, Buckeye p.28 citation list, anchoring screenshots.
- Launch the Session 2 poll deck with the class size; run the warm-up question to assign A/B groups before the split.
- Post the judgment-excerpt set (five mid-length excerpts from different practice areas, text-layer PDFs) to the class channel before the session.
Key sources & cases
Pooja Ramesh Singh v. Jammu and Kashmir Bank Ltd., 2026 INSC 668 (SC, 2 July 2026)
Reportable; Narasimha and Aradhe JJ. Held: citing unverified AI-generated judgments is misconduct for an advocate and a serious lapse for a judge; a decision resting on even 'an iota' of fake or hallucinated material is 'no decision in the eyes of the law'. NCLT and NCLAT orders set aside; the Bar Council of India directed to frame guiding principles and disciplinary consequences. The course's India-first anchor. Verified 2026-08-27 (Indian Kanoon; signed PDF); re-check the api.sci.gov.in copy before filing.
Gummadi Usha Rani v. Sure Mallikarjuna Rao, SLP (C) No. 7575/2026 (SC, order of 27 Feb 2026; pending)
Origin story: a trial-court order rested on four non-existent SCC citations; the same bench observed that such a decision 'would be a misconduct and legal consequence shall follow', stayed reliance on the tainted report and directed a BCI expert committee. Interim order, pending matter; the holding now lives in Pooja Ramesh Singh. Verified 2026-08-27 (Record of Proceedings PDF).
Deepak v. Heart & Soul Entertainment Ltd. (Bombay HC, 7 Jan 2026, Sathaye J.; 2026:BHC-AS:828)
₹50,000 in costs for AI-generated written submissions citing an untraceable judgment. The detail matters: the sanctioned party was a company appearing through its director in person, not an advocate; the court's warning on verification extends to advocates as dictum. Caption and neutral citation from concordant secondary reports (Verdictum, LiveLaw); re-check the order before filing.
Buckeye Trust v. PCIT, ITA No. 1051/Bang/2024 (ITAT Bengaluru; order of 30 Dec 2024, recalled 7 Jan 2025)
The original order (read to source) rests on authorities that could not be traced; the Tribunal recalled it suo motu within days for 'inadvertent errors'. The recall order as reported names neither AI nor any tool; the ChatGPT attribution is the press's inference. A live classroom exercise: run its page-28 citations through the four-step check.
Mata v. Avianca, Inc., 678 F. Supp. 3d 443 (S.D.N.Y. 2023)
Six fabricated cases, a $5,000 sanction, and the fatal assumption that a fluent tool 'could not possibly be fabricating cases'. The comparative flagship, now one of roughly 1,700 such decisions worldwide (Charlotin, AI Hallucination Cases, July 2026 count; re-check each cohort).
Kalai, Nachum, Vempala & Zhang, 'Why Language Models Hallucinate' (2025)
The mechanism behind the confidence: models are optimised like test-takers, and under binary grading a confident guess strictly beats abstention, so they answer. Grounds the course's practical rule that explicit permission to say 'I am not certain' changes behaviour. Verified to the paper's abstract and text 2026-08-27.
OpenAI, o3 and o4-mini System Card (April 2025), §3.3
The vendor's own measurement that a newer reasoning model hallucinated more on person-fact benchmarks (PersonQA: o3 at 0.33 versus o1 at 0.16). The verified basis for teaching that reasoning modes do not confer knowledge. Read to the PDF 2026-08-27.
Alex Edmans, May Contain Lies (2024); Daniel Kahneman, Thinking, Fast and Slow (2011)
The Ladder of Misinference, and anchoring: the room experiment's pedigree. Quotes under 15 words and attributed; see the codebook's misattribution flags before quoting either.
Readings
- Pooja Ramesh Singh v. J&K Bank, 2026 INSC 668 (SC, 2 July 2026) — read paragraphs 1, 7 and 17 before the session
- Alex Edmans, May Contain Lies (2024) — the Ladder of Misinference
- Daniel Kahneman, Thinking, Fast and Slow (2011) — anchoring, System 1 and System 2
- Kalai et al., 'Why Language Models Hallucinate' (2025) — skim the abstract; the mechanism in four pages
- Mata v. Avianca, 678 F. Supp. 3d 443 (S.D.N.Y. 2023) — the comparative flagship
Sixteen hours, one professional discipline.
Craft, competence and verification, practised until they are habits you could defend in court.