AI failures: Slide deck draft

Aharon Logue · September 27, 2026

The work of making AI effective. A draft presentation for an insurance and banking conference.

Open or download the PDF

Slide 1: AI failures The work of making AI effective Aharon Logue Finance Agent Studio
Slide 2: Aharon Logue Five years researching finance technology at Gartner. Studied teams getting tangible results with AI. Now leading Finance Agent Studio.
Slide 3: Six types of AI failure
Slide 4: The (not so) long history of AI in spreadsheets Source table redrawn from Deng et al., ACL 2024, Table 17. Values preserved. The task: recreate this table with every value under the correct year.
Slide 5: The results were... not great Published GPT-4 output redrawn from Deng et al., ACL 2024, Table 18. Cell placement preserved. GPT-4 · 2024
Slide 6: Until they were GPT-6-family trial, September 2026 · Same source table; not a controlled rerun of the 2024 experiment. GPT-6 · 2026
Slide 7: Temporary failures
Slide 8: The infamous pelican benchmark GPT-4o · 2024 GPT-5.6 Sol · 2026 GPT-6 Astra · 2026 Simon Willison · Selected SVG outputs, 2024–2026
Slide 9: AI is always failing... until it isn’t Sources: OpenAI · SpreadsheetBench · FinSearchComp | Different tests and tools; scores are not directly comparable. Investment banking Mean score · internal evaluation SpreadsheetBench Task accuracy · 912 tasks GDPval Win or tie vs. professionals FinanceAgent v1.1 Financial research · score OfficeQA Document questions · score FinSearchComp Global financial search · average
Slide 10: The jagged frontier keeps expanding Tomas Pueyo, reproduced in Ethan Mollick’s One Useful Thing · Conceptual illustration
Slide 11: Taking advantage of a changing technology Revisit tasks that previously failed. Try more ambitious requests (but don’t bank on easy success). Learn from other people’s experiments.
Slide 12: Starting early helps you benefit from later improvements The initial benefits may be small. The setup work builds habits and infrastructure you can use again.
Slide 13:
Slide 14:
Slide 15: Context failures
Slide 16: FINANCIAL TIMES Amazon holds engineering meeting following AI-related outages Amazon says an engineer followed AI advice drawn from an outdated internal wiki. Financial Times · March 2026 | Context: Amazon’s subsequent clarification
Slide 17: Context at an organizational scale gets complicated, and risky AI has to find the right information for the specific question. Policies and regulations change; yesterday’s correct answer can become wrong. Ambiguity in shared context can spread the same mistake across many tasks.
Slide 18: Keeping context useful is ongoing work Pull in relevant context without burying it in unrelated information. Update shared sources when policies and regulations change. Test how AI interprets that context and catch misunderstandings early.
Slide 19: An AI-built economic model looked convincing Anson Ho · Epoch AI 6 February 2026
Slide 20: But its predictions were wrong! Original GATE interface · Epoch AI
Slide 21: Plausibility failures
Slide 22: Work can look finished before it is good enough to use Each step can look plausible. The problems emerge when it all has to work together. You may have much more work left than you thought.
Slide 23: Your expertise has to scale with your AI ambitions “Many employees report completing work previously outside their expertise—backend engineers building UIs; researchers creating visualizations.” Anthropic · How AI is transforming work at Anthropic Study of its own engineers and researchers · December 2025
Slide 24: AI doesn’t get you off the hook for good work Assign reviewers who understand the work, even when it crosses team boundaries. Test important assumptions early, before more work depends on them. Make review and editing part of producing AI work, before it is passed on.
Slide 25: ABC News · 10 August 2026
Slide 26: Ambition failures
Slide 27: AI’s capability to execute can exceed its judgment We want AI to take on more. It can pursue the goal through actions we never intended. Greater ambition requires better direction and supervision.
Slide 28: Experienced users changed how they supervised AI Anthropic · Measuring agent autonomy · February 2026
Slide 29: Decide how much freedom the agent should have Enforce boundaries through what it can access and do. Bring people in at consequential decisions. Avoid approval prompts so routine that people tune out.
Slide 30: OpenAI · Incident report · 26 August 2026
Slide 31: Security failures
Slide 32: AI creates new opportunities for attack... Instructions planted in emails or websites can redirect an agent (prompt injection). A hijacked agent can use its permissions to steal data or change files. Poisoned memory can carry malicious instructions into future tasks. OWASP Agentic Top 10 · NIST / CAISI, March 2026
Slide 33: ...and makes attackers more capable Attackers used AI agents to steal data from about 200 customers of one breached software provider. Another group used AI to rewrite malware when security tools detected it. Reported by Anthropic · Threat Intelligence Report, September 2026
Slide 34: Don’t be low-hanging fruit: security foundations still matter Limit access and protect credentials. Separate sensitive systems. Monitor activity and keep a way to stop the agent.
Slide 35: AI is part of cybersecurity now Test for prompt injection and poisoned memory. Investigate failures and monitor deployed systems. Use AI to help find and resolve vulnerabilities.
Slide 36: Conclusion
Slide 37: Five Takeaways About AI Failures Failures are inevitable, but changing fast. The more we try to do with AI, the more we have to change how we work. Directing, validating, and securing AI is work that you (or someone) has to do. Getting the most out of AI means pushing the limits of your AI use. That requires scaling expertise, and limiting AI’s autonomy. New technologies always mean new risks, and unfamiliar work. Done right, the benefits can outweigh them both.
Slide 38: Questions