Common LLM Failure Modes

A working reference, and how judging AI against the job helps you manage them

Large language models, the AI behind tools like ChatGPT and Claude, fail in recognizable ways. This page lists commonly reported failures, many of which I run into working with AI agents every day, and what you can do about each.

Compared to whoever would otherwise do the job, as they actually are, how does the AI do?

That question is the yardstick behind this list. Judge AI by the role it plays, and you manage it as you would a capable new hire: constructive, skeptical, and checking the work. The argument for it is in The wrong yardstick.

How to read it. The "how the yardstick helps" lines are my reasoning from practice, not findings. They are things to test. Each entry's Basis says what stands behind the failure itself: Measured means research has measured it in AI; Borrowed from human research means a documented human bias that models appear to reproduce; Observed in practice means a commonly reported pattern, not one this page can point to a measurement for.

Judgment failures · Working failures · Where it doesn't help · Sources

Judgment failures

The failures named in SPARRING, my published framework for AI decision-making.

Pleasing bias (sycophancy)

What it looks like: Agrees too easily, especially when you state a view first.
How the yardstick helps: You expect a junior to want your approval, so agreement alone doesn't count as confirmation.
Basis: Measured (Sharma et al. 2023; Perez et al. 2022)

Confirmation bias

What it looks like: Reads ambiguous evidence the way it expected to.
How the yardstick helps: You ask what evidence would cut the other way.
Basis: Borrowed from human research (Nickerson 1998)

Anchoring

What it looks like: Refines its first answer instead of rethinking it.
How the yardstick helps: You ask for alternatives, as you would with a junior's first draft.
Basis: Borrowed from human research (Tversky & Kahneman 1974)

Misreading the question

What it looks like: Answers a neighboring question well.
How the yardstick helps: You have it restate the assignment before relying on the work.
Basis: Observed in practice

Specialization blind spots

What it looks like: Misses risks outside its frame.
How the yardstick helps: You know what this "hire" doesn't cover, and you route around it.
Basis: Observed in practice

Hallucinated detail

What it looks like: Invents specifics that look real: a citation, a number, a name.
How the yardstick helps: You spot-check a junior's citations and numbers.
Basis: Observed in practice

Confidently wrong answers

What it looks like: Its tone doesn't track its accuracy.
How the yardstick helps: You judge by track record, not by how sure it sounds.
Basis: Observed in practice

Bandwagon contamination

What it looks like: Catches your enthusiasm or the mood of the conversation.
How the yardstick helps: You notice when it is only echoing you.
Basis: Observed in practice

Circular sourcing (self-citation)

What it looks like: The evidence for a claim traces back to the claim's own author.
How the yardstick helps: You ask where a claim really came from.
Basis: Observed in practice

Fake pushback (theatrical adversariality)

What it looks like: Told to push back, it produces critique-shaped text with nothing real behind it.
How the yardstick helps: You judge a critique by what it found, not by how tough it sounds.
Basis: Observed in practice

Working failures

Other problems that show up when AI does real work over time.

False claims of completion

What it looks like: Reports "done, tests pass" when they don't.
How the yardstick helps: You check the work, not the status report.
Basis: Observed in practice

Jagged ability

What it looks like: Strong on one task and weak on a similar one.
How the yardstick helps: You learn where its abilities end, as a good manager does with any new hire.
Basis: Measured in people using AI (Dell'Acqua et al. 2023)

Inconsistency between runs

What it looks like: Gives different answers to the same question.
How the yardstick helps: You compare it with people, who vary too, and you audit what matters.
Basis: Observed in practice

Scope creep

What it looks like: Does more than you asked, or changes things it shouldn't.
How the yardstick helps: You set limits and review its changes.
Basis: Observed in practice

Gaming the measure

What it looks like: Meets the metric but misses the point.
How the yardstick helps: You review the outcome, not just the score.
Basis: Observed in practice

Lost context

What it looks like: Forgets instructions or earlier decisions.
How the yardstick helps: You keep written notes and handoffs.
Basis: Observed in practice

Stale knowledge

What it looks like: States outdated facts as current.
How the yardstick helps: You have it check anything that may have changed.
Basis: Observed in practice

Where the yardstick doesn't help

A stance isn't a control. These need more than good management.

Prompt injection. The AI follows instructions hidden in a web page or document (Greshake et al. 2023). This is a security problem and needs technical controls. A new hire's judgment is a poor analogy here.
Errors compounding between agents. In an automated sequence, often nobody is watching each handoff, so one agent's mistake becomes the next one's premise. This needs structure and governance controls.
What reliance does to you. Relying on AI can change how you work without it. In one observational and contested study, the rate at which experienced endoscopists found precancerous growths on colonoscopies done without AI fell from 28.4% to 22.4% in the three months after AI arrived at their centers. Whether that is lost skill or lower effort isn't settled (Budzyń et al. 2025).
Scale. An intern's mistake lands on one desk. A model's mistake can land on every desk that uses it. Sometimes that sets a higher bar for a machine than for a person, and the yardstick should say so.

Sources

Budzyń, K., et al. (2025). Endoscopist deskilling risk after exposure to artificial intelligence in colonoscopy. The Lancet Gastroenterology & Hepatology. DOI
Dell'Acqua, F., et al. (2023). Navigating the jagged technological frontier: Field experimental evidence of the effects of artificial intelligence on knowledge worker productivity and quality. Harvard Business School Working Paper 24-013. DOI
Greshake, K., et al. (2023). Not what you've signed up for: Compromising real-world LLM-integrated applications with indirect prompt injection. AISec '23. DOI
Nickerson, R. S. (1998). Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology. DOI
Perez, E., et al. (2022). Discovering language model behaviors with model-written evaluations. arXiv
Sharma, M., et al. (2023). Towards understanding sycophancy in language models. arXiv
Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science. DOI

By Bart Niedner, with help from Claude, an AI made by Anthropic, which has a stake in how AI is judged. I build AI assistants for people, so I have a stake too. Last reviewed October 31, 2026. This is a working list, not a complete one. Corrections are welcome at Contact@ResourceForge.com.