Training module
AI Limitations & Failure Modes
Build practical AI failure-mode literacy across predictive, generative and workflow systems
Rely on AI only when the evidence holds up
Build practical judgement about AI uncertainty, output fallibility and common failure modes across predictive, generative and AI-enabled workflow systems.
Overview
What this module is about
AI systems do not produce verified facts by default. Their outputs are shaped by data, labels, prompts, retrieved sources, model or service behaviour, integration choices, workflow conditions and human use. When those limits are not understood, organisations either over-trust AI or block useful AI for the wrong reasons.
This live module builds practical AI uncertainty and failure-mode literacy. Using Northstar Integrated Services as a running case, participants learn how failures arise across predictive AI, generative AI, data pipelines and socio-technical workflows, and how to challenge evidence, vendor claims and AI-polished wording with realistic judgement.
Audience
Who it is for
Management system implementers and coordinators
Executives and department heads accountable for management system performance
Those responsible for processes, policies, assets, risks and controls
Auditors seeking insight into management-side practice, not audit technique
Consultants working on management system design, governance or improvement
Decision supportIs this module for you?
It is a good fit if you…
want to understand why AI outputs should be treated as fallible signals rather than verified facts.
need to recognise common AI failure modes in practical business, product, risk, compliance or audit conversations.
work with predictive AI, generative AI, vendor AI features or AI-enabled workflows and need better questions.
want to challenge vendor claims, pilot evidence and AI-polished wording without becoming an AI engineer.
are preparing for AI inventory, AI risk management, operational control, supplier assurance or audit work.
It may not be the best fit if you…
need a full AI risk assessment, harm assessment or control-design course.
expect coding, prompt engineering craft, MLOps configuration or model tuning.
want statistical model validation, red teaming or advanced AI safety methods.
need legal classification or formal compliance advice for specific AI Act obligations.
Agenda
What is taught
7 parts01AI outputs as fallible signals
Separate AI output from verified fact, source evidence and unsupported claim
Use Northstar examples to calibrate useful trust without blind reliance or blanket rejection
02Where uncertainty enters AI systems
Trace uncertainty through data, labels, prompts, retrieval, model or service behaviour and integration
Connect workflow conditions and human use to practical reliability limits
03Predictive AI failure modes
Recognise generalisation limits, spurious correlations and threshold trade-offs
Challenge pilot performance, false positives, false negatives and context-shift assumptions
04Generative AI and retrieval failure modes
Diagnose hallucination, unsupported claims, prompt sensitivity and overconfident wording
Check whether retrieved sources are current, relevant and sufficient for the answer
05Data and pipeline failure modes
Identify data leakage, training-serving skew, missingness and weak provenance
Read drift-like monitoring signals as questions rather than automatic conclusions
06System and socio-technical failure modes
Examine automation bias, misuse, gaming, feedback loops and brittle interfaces
Spot local configuration changes, fallback failure and agentic creep before they become invisible assumptions
07Evidence limits and downstream questions
Challenge vendor claims, pilot results and AI-polished assurance wording against the available evidence
Route failure-mode questions to inventory, risk, control, monitoring, supplier, audit or technical review
Outcomes
Learning outcomes
01
Explain why AI outputs are uncertain signals rather than verified facts
02
Trace where uncertainty enters across data, prompts, retrieval, model or service behaviour, integration, workflow and human use
03
Recognise common predictive, generative, data, pipeline and socio-technical AI failure modes in realistic artefacts
Challenge vendor claims, pilot results and AI-polished wording against the available evidence
Conduct a bounded failure-mode walkthrough without turning it into risk assessment or control design
Route failure-mode questions to AI inventory, risk management, operational control, monitoring, supplier assurance, audit or technical review
Materials
The content and the assessment
Written module
The full content in writing, complete in itself. Videos are recorded for parts of it as an alternative way through, and the written module always carries everything.
Exercises
Graded work on the case organisation's own registers: structured answers checked against the encoded case, written answers scored against a rubric traced to it, with a trainer holding the final word on every assessed item.
On completion
The assessed exercises must be passed before the certificate is issued.
Scheduling
No public run of AI Limitations & Failure Modes is scheduled at the moment. Tell us you are interested and we will let you know when the next one opens, or discuss running it in-house.
Module facts
- Module ID
- HAM-AI-DF-02
- Discipline
- Artificial Intelligence
- Part of tracks
- Management System Auditor · Executive · Management System Manager
Case organisation
You work inside a company that already has the problem
Exercises run on one case organisation, carried across modules rather than restarted, so what you build here is what the next module finds.
Northstar Integrated Services AG is a group headquartered in Zurich, providing digital operations platforms and managed services to regulated organisations across Europe. It has not always been that. It began in 2008 as a field-operations firm of about twenty-five people, and what first forced documented decisions and named accountability on it was not growth but a single regulated customer. It now runs an acquired business in Poland and the Czech Republic through a subsidiary that kept its own legal identity, which is where the interesting failures live: group instruments rolled out operationally and never put in force by the governing bodies of the subsidiary itself.
Linked registers
Organisation and people, risks, objectives, policies and documents, findings and reviews, third parties, processes
Not a case study
Nothing is summarised for you; the evidence is where it would really be
It has a history
The organisation has a timeline, and modules enter it at different points, so a structure can be studied before it broke as well as after
It crosses borders
A Swiss parent, an acquired operating business in Poland and the Czech Republic under its own legal entity, and customers in several jurisdictions
Why it matters
Judgement is not trained on tidy examples, and it is not trained on a fresh one each week. Northstar is deliberately untidy, and modules enter it at different points of its history, so you see a governance structure being built, outgrown and rebuilt rather than a finished one. What you decide in one module is what the next one finds.
Delivery & dates
How this module reaches you
Delivered live online, combining conceptual framing, discussion, case work and direct interaction with the trainer. In-house and contextualised delivery is available on request.
No public run is scheduled at the moment. Most modules run on request as well as on the public calendar, so tell us the timing you need.
Want this module scheduled?
We will tell you when the next run is scheduled, or run this module in-house with your own case material.
Not sure it is the right module?
Describe your context in a short message and we will tell you honestly.