AI can help us identify risks, challenge assumptions and assess controls faster than ever. But what happens when an analysis sounds convincing and is simply incomplete? In this blog, we explore why false assurance may be the greatest risk of using AI in risk management, and what it takes to keep human judgement firmly in the lead.
A story about a management team that learned to question its own confidence.
Imagine a mid-sized organisation, which we will call ProduLogic, in the middle of a routine review of its procurement process. The risk team had done thorough work. The main risks were mapped, the controls described, and the report was ready for the management team.
Then Sanne, the Head of Risk, did something slightly unusual. Before presenting the analysis, she gave it to an AI tool and asked three questions: which risks might we have overlooked, which assumptions are not properly substantiated, and under what circumstances would our controls fail?
The answers did not make her team look foolish, and they did not replace their expertise. But they raised points the team had never discussed: a quiet dependency on a single supplier, a weakness in the segregation of duties, and a set of authorisations nobody had looked at in years. The value was not that AI knew better than an experienced risk manager. It asked questions the team had stopped asking itself. For Sanne, AI became a sparring partner rather than a substitute.
Encouraged, ProduLogic went further. The team began using AI to review its controls. A periodic check of user access rights, which used to take days, now took hours. The AI compared granted rights with policy and flagged the deviations.
Then came a lesson in judgement. One flagged user turned out to hold an approved exception, and another was a temporary account that was perfectly explainable. A flag, it turned out, is not the same as a finding. The opposite proved just as true: when the AI reported no deviations at all, that did not prove the control worked. It only meant the AI had found nothing in the data it had been given.
A few months later, the lesson grew larger. An AI application analysed several hundred incident reports and concluded there were no structural shortcomings in a key process. The conclusion was well written and entirely convincing, and the management team relaxed.
It was Sanne who asked the awkward question: how many incidents never make it into the register at all? Nobody knew. The analysis was only as complete as the data behind it, and the data had gaps nobody had measured. The dashboard that showed no warnings looked like safety, but silence is not the same as the absence of risk.
This is what we regard as the greatest danger of AI in risk management: false assurance. A reliable AI application should therefore show not only what it has found, but also what is missing, which assumptions it made and where it is uncertain. A good tool tells us what we know, and above all what we don’t.
At first, ProduLogic’s use of AI was scattered. One employee used a chatbot to draft an analysis, then copied the results by hand into spreadsheets and reports. It saved some time, but the process itself barely changed.
The real shift came when management stopped treating AI as a loose tool and connected it to the process. Incidents, risks, controls and improvement actions were linked. When an incident was registered, AI proposed possible causes, pointed to related risks and suggested whether a control needed adjusting. A responsible employee reviewed those proposals and decided what to do.
That only worked because ProduLogic had first put its house in order: clear processes, defined responsibilities and data of reliable quality. A poorly designed process does not become a good one by adding AI to it. This is the conviction we hold at BPM Power: technology should support and improve business processes, not the other way around
As AI’s role grew, the board asked the question every organisation eventually faces. If an AI analysis turns out to be wrong, who is responsible? The employee who used it? The process owner? The risk department? The supplier?
The answer was simpler than expected: nothing about existing responsibilities changes. Under the Three Lines Model, the first line still owns the risks in its own processes, the second line still supports, advises and critically monitors, and the third line still provides independent assurance.
But ProduLogic also saw a subtler danger. If all three lines rely on the same AI model, the same assumptions and the same flawed data, they can appear to give three independent opinions while really repeating one shortcoming three times. Independence and segregation of duties must therefore be protected deliberately, with clear agreements on ownership, quality control, traceability and independent testing. Human responsibility cannot be outsourced to an algorithm.
Good analysis needs information, and that information is often sensitive: confidential business data, incident details, personal data, and sometimes details of system vulnerabilities. ProduLogic therefore had to decide carefully what could be shared with AI applications and on what terms. That meant secure AI environments, appropriate contractual arrangements, restricted access, data minimisation, anonymisation where possible and monitoring of use.
Regulation matters as well. The European AI Act is partially applicable and its obligations depend on the nature and classification of the application. Since 2 August 2026 an important part of the rules has applied, with later dates for certain high-risk systems. Organisations must also pay attention to AI literacy, and the GDPR continues to apply whenever personal data is processed.
Yet ProduLogic’s management understood that compliance does not guarantee control. An organisation can meet every legal obligation and still have too little grip on the reliability of its own AI analyses. Compliance is the starting point, not the end goal.
information, comparing documents, analysing trends and drafting first versions of reports. That frees time for the work that matters most, advising management strategically.
But it also changes what makes a good risk manager. Knowledge of risks, processes and controls remains essential, and alongside it come critical thinking, an understanding of AI’s limitations and the ability to assess information sources. A risk manager must be able to recognise when AI gives an answer that is plausible but wrong, and judge when human judgement should outweigh the output of a model.
ProduLogic did not wait until every question about AI had been answered. It started small and kept it controlled. First, AI critically reviewed an existing risk analysis. Then it proposed alternative controls and helped structure incident information. Each time, the results were compared with the judgement of experienced professionals.
The team measured more than time saved. It asked whether the analyses were actually better, whether new insights emerged and whether employees felt genuinely helped. And just as importantly, it investigated where AI fell short. That showed the organisation both the possibilities and the limits, and only then were the successful applications integrated step by step into existing processes.
Innovation and control did not get in each other’s way. Responsible experimentation turned out to be one of the best ways to understand the risks of AI.
ProduLogic’s experience reflects what we see in practice. AI does not make risk management better by itself. The greatest value does not come from leaving as much work as possible to AI, but from combining technology, well-designed processes and human expertise. AI can help identify risks, deepen analyses and assess controls critically. It can also draw wrong conclusions, miss important information and create false assurance.
Our advice to management is therefore simple: start small, experiment, learn, and then integrate AI in a targeted way into your risk management process. Insist on transparency, demonstrable reliability and clear ownership. And above all, keep thinking for yourself.
Because in our view, the future of risk management is not about artificial intelligence replacing people, but about human intelligence being put to better use thanks to AI.
Would you like to discuss this further, or do you recognise these challenges within your organisation? Get in touch and let’s find out together.
