Finally we have the information we need to challenge this, and find a better path that may actually reduce the frequency and impact of cyber incidents.

The failure of compulsory training is a problem I know from personal experience.

When I rolled out a cyber CBT to 40,000 employees with just 8 questions to answer, one employee took 73 times to pass and one 86 times, such was the desire to comply whilst avoiding reading the material. The message was clear: you can force people to tick boxes, but you can’t force people to learn.

At another company, I asked a room of 100 employees who had just done training how many of them knew where to report a cyber incident. Nearly everyone raised their hand. But when I asked who was sure, all hands went down apart from two. Allowing for development and delivery costs, and assuming each employee spent half an hour on the CBT, each of those hands cost over $7,000.

Phishing testing performance had similar results, with failure rates depending largely on how hard the phish was. There was no real improvement over time, and the same proportion of people continued to click on similar retests.

There is of course a role for these on occasion. Onboarding of employees might be an example, when people are highly receptive and not distracted by day-to-day pressures. However it has been clear for some time that most time and money spent on these goes down a bottomless hole.

On the other hand, when we visited other companies and learned from their ideas, we found our own creative and fun ways to engage people. At one point our awareness campaign had a 92% engagement rate, versus a 4% corporate average. That campaign cost less than $1 per person for the same result.

What I didn’t have, however, was clear empirical evidence. Now we do.

In a newly released peer-reviewed study, Ho et al. (UC San Diego, 2025) studied 19,500 employees and found this training to be largely futile. Their peer-reviewed research, Understanding the Efficacy of Phishing Training in Practice, showed that phishing click rates did not improve over time and varied only by phish difficulty. Many participants spent less than a minute on the training, and a third didn’t open it at all.

They are not the only researchers to reach this conclusion - Rozema & Davis (Purdue University, 2025) rapidly followed up, reaching very similar conclusions.

Some defenders of CBTs argue they serve a purpose — especially during onboarding, when staff are more receptive. Others point to regulatory obligations or legal defensibility: it’s easier to prove a policy was taught than a behaviour was learned. Even flawed training, they argue, is better than none.

But these arguments don’t hold up to scrutiny. If the goal is learning, not just liability shielding, the outcomes matter. And here the outcomes are dire: low retention, high cost, and little evidence of impact.

As organisations need to show compliance, they then improvise to counter employee tendencies to avoid training they know is largely futile. These attempts to drive compliance are generally designed for force, rather than facilitate, completion. Companies randomise quiz questions, move answer boxes around the screen, and lock the quiz until video content has been finished. They insert trick questions, monitor completion times, and punish people for finishing too quickly. Everyone knows it’s theatre, but no one wants to break the spell.

If everyone knows it doesn’t work, why does nobody say so?

Occasionally people do raise this. But quite simply, nobody has an incentive to address the problem and most have an incentive to let the status quo continue. For example:

• If you are an employee, telling your boss or HR that you are not doing the compulsory training properly will get you into trouble, and probably just result in disciplinary action or being reassigned the training again.

• If you are a line manager, complaining about the cost or time spent on corporate mandated processes or activity is usually ineffectual, so managers won’t waste limited political capital on a lost cause.

• If you are in Risk or Compliance, this can demonstrate everyone has had the information and the opportunity to learn what is expected, even if you know it doesn’t mean they’ve listened. You can use the fact that they should have understood to hold employees and managers accountable... which is one way of driving behaviour.

• If you are in Legal and you need to show the organisation has passed on its obligations to employees, this allows you to show that the company exercised your responsibilities in a way that reaches everyone, and therefore to defend against many claims. Showing the training didn’t work is much harder than showing it was done.

• If you are on a Board, or maybe a customer, CBTs and similar exercises provide simple and objective metrics to hold executive management to account and allow management to demonstrate to you that controls are in place. If it looks effective and management think it is the right thing to be doing, can you really invest precious time in unpicking this?

Why do we need to think differently?

There are two good reasons to care about this.

The first issue is cost. To work out how much your organisation is spending on CBTs and similar training, this is the basic calculation:

Total cost of CBT = (Average fully loaded cost per employee x time spent x number of employees) + (cost of technology + costs of licensing + cost of development + cost of deployment + cost of monitoring & reporting).

For example, if your CBT costs £10,000 to license and run, and you have 1,000 employees who cost £50 per hour and each spends an hour on the CBT, your cost is £70,000 - not £10,000. Yet often only the software cost is measured, because that’s the only bit that’s visible in a budget.

If you then do 20 hours of CBTs per employee per year across all disciplines (which is quite normal when you consider HR, health and safety, and similar topics), you’re actually spending £1 million of staff time on this stuff. Do you get £1m of value? I doubt it.

At one point a function I ran was spending more than £6m a year of company time requiring everyone to do a compulsory CBT. We cut it from 60 minutes to 12 minutes, because we couldn’t point to a single benefit (other than avoiding a conversation with a regulator). Yet cutting that cost didn’t change my departmental budget, and I don’t think the company really noticed because nobody else was measuring this. It’s important to make these costs visible to quantify the impact of a control.

The second issue is the negative impact.

CBTs irritate, annoy, and reduce employee morale. They take hours to build, maintain and deliver. Phishing tests require IT configuration to bypass the very controls you ought to be testing. CBTs require tech or management solutions so they can be accessed by everyone, even if they are on the shop floor or outdoors. The constant need for completion monitoring also teaches employees that what they will really be measured on is compliance with corporate diktats, rather than trying to make the best possible decisions they can. CBTs can actually reduce performance and increase risk: they disenfranchise, disincentivise, and disconnect.

It’s a big burden to overcome before the claimed risk reduction outweighs the costs, and yet as UC San Diego found, that claimed risk reduction is often not forthcoming at all.

Ineffective controls are not harmless. Bad controls that look like good ones can be dangerous: metrics plus mis-stated impact can result in false assurance. That false assurance is then what we pass to boards, markets and regulators. No wonder it causes trouble later when the behaviours we have tried to force out by compulsion still occur.

Is there a better way?

The good news is that it does not have to be this way. It is possible to reduce costs, improve efficiency, build positive staff engagement, and reduce risk from undesirable behaviours. That doesn’t have to come at the price of compliance, but it does mean understanding that some boxes are just not worth ticking. Instead, the focus should be on performance and risk.

Firstly, don’t fight employees. Systems and processes should be designed to work with employees, not against them. If you don’t want employees to click links in emails, don’t send them links in emails. Instead of telling employees off for clicking, use technology to have new links open in a secure sandbox. If you want to educate, do it with small prompts and nudges in line with work activity: did you really want to send this sensitive attachment to your personal email? Now you ask, maybe not.

Secondly, match incentives and outcomes. If your employees’ incentives don’t match the desired outcomes, you will always get the wrong result. Don’t tell employees off for not doing a CBT, as that just drives fear and a tick-box culture. Instead, highlight and reward positive behaviour. For example, instead of monitoring compliance with a phishing test, monitor the rate at which real phish are reported and run a leaderboard for the best departments.

Finally, help employees have fun. The most effective awareness campaign I ran was called Harry the Hacker. Harry was a cartoon character and a shameless stereotype of a hacker, who got into all sorts of trouble including involvement with organised crime groups. A team member came up with the concept and we hired an ex-Marvel cartoonist to bring him to life. The hero was never the CISO or the CEO: it was always an employee who did the right thing. It was low tech, with a simple monthly email. He was such a hit that one day a member of staff dressed up as Harry, and sent us a photo of himself dressed as Harry with his computer open on our fake malicious website. Alongside this we cut the cyber CBT from 1 hour to 12 minutes. The results were excellent.

Eventually this was shut down by Corporate Communications who first relegated Harry from an employee email to a dusty corner of the intranet, then wanted us to make everything purple. Our engagement rate plummeted: sometimes you can’t win everything. However, my experience was that trying to engage people with the message in creative ways was more than worth the time and money. It cost more to develop than a standard CBT, but saved employees time and stress, whilst saving the company a fortune and reducing risk.

It’s true there will be a few that don’t engage - but they are also the few who would do a CBT 86 times to avoid reading the content. Sometimes we have to accept that there will always be some human risk, and whilst technology controls can help, having a CBT completion stat that denies the residual risk would hardly be beneficial.

Most employees want to do the right thing. All we need to do is help them.

What then should you do with those CBT and phishing stats?

Instead of punishing the employee, use them to measure the performance of the functions that set them. Hold the commissioning team to account for the business impact. This is where cyber risk quantification techniques can be valuable, but the simple cost calculation above will usually suffice to make the point clearly.

Report the real costs of the delivery, and ask them to justify the investment with reference to real business impact. For example, by measuring voluntary engagement rates, or frequency of desired behaviours vs undesired. If the ROI on the activity is too low, treat it as you would any other under-performing activity. If your marketing didn’t work, how long would you keep doing the same thing? Not long.

The emperor’s new clothes?

In the cautionary folk tale The Emperor’s New Clothes, the emperor and his courtiers are told by charlatan weavers that the Emperor’s clothes are visible unless you are incompetent or stupid. As a result nobody says anything for fear of looking stupid, and so the emperor walks naked through the streets, to general public shock and humiliation. It is a warning not against following, but against following blindly.

We’ve all been told that these tools work, and of course relying on them does not indicate incompetence or stupidity. At one point, this was best practice. I also am not advocating dropping them without careful consideration and a clear plan.

But if we now know they don’t work well, yet we continue to do them?

One day we might find ourselves wearing that security incident a little too visibly for comfort.

There are few things more harmful to an organisation than having maintained that everything is fine, only for it to turn out after an incident that it wasn’t, and that management should have known this all along.

The knowledge is now there to show that whilst compulsory CBTs and phishing exercises are often effective at ticking boxes, they are rarely good at reducing risk - and it’s clear that they cost a fortune whilst making organisations worse places to work.

Yet this is all avoidable, if we try a better approach.

The risk isn’t just in what these tools fail to teach — it’s in what we’ve convinced ourselves they have already taught.

Let’s do better.

References
Ho, G., Mirian, A., Luo, E., Tong, K., Lee, E., Liu, L., Longhurst, C. A., Dameff, C., Savage, S. and Voelker, G. (2025). Understanding the Efficacy of Phishing Training in Practice. In Proceedings of IEEE Symposium on Security and Privacy. Available at: https://www.computer.org/csdl/proceedings-article/sp/2025/223600a076/21B7RjYyG9q

Rozema, A.T. and Davis, J.C. (2025) Anti-Phishing Training (Still) Does Not Work: A Large-Scale Reproduction of Phishing Training Inefficacy Grounded in the NIST Phish Scale. arXiv preprint. Available at: https://arxiv.org/abs/2506.19899

When the RMS Titanic hit an iceberg on 15 April 1912, she set off flares and her wireless operator sent out a distress call. The RMS Carpathia responded, but by the time she arrived, the Titanic had already sunk: only those who had already made it to the lifeboats could be saved. Some 1,500 people died.

Another ship was closer and could potentially have responded faster—perhaps even fast enough that more lives could have been saved. Yet despite seeing the flares, she did nothing.

The SS Californian was a cargo vessel captained by a 34-year-old Brit, Stanley Lord. As the Titanic sank, she was close by—between 5 and 20 miles from the Titanic’s position.

However, the captain was having a nap at the time.

While Lord’s ship was nearby, it’s not clear they knew an incident had occurred. Crew members saw white flares, but mistook them for celebratory—perhaps not an unreasonable assumption. No wireless signals were received, as the ship’s sole wireless operator worked the day shift.

You could argue there were some control limitations (should they have had 24x7 wireless cover? Maybe. Could they have justified that cost if not required to? I doubt it). But these aren’t things the captain could have changed. It was also perfectly reasonable for Lord to get some rest. When he went to bed, it was a quiet night. I’ve slept just fine while leading a 24/7 response to a critical cyber incident—after getting it under control, of course. For a leader not to rest under such circumstances would be dangerous.

These factors meant the SS Californian didn’t become aware of the incident until receiving a wireless message in the morning—by which time it was far too late to help.

After the sinking of the Titanic, Lord’s inaction caused a scandal. Two inquiries were undertaken: a United States Senate hearing and a British Wreck Commissioner’s inquiry. Nobody was impressed. The hearings were critical, and the media unforgiving.

Despite regular sleep being a normal part of the human condition, and imperfect organisational controls being routine (as all risks have to be managed with regard to cost), the debate was quite binary with many taking sides for or against Lord. This is not unusual. You may wonder how Heathrow CEO Thomas Woldbye felt after sleeping through a power outage earlier this year, despite having protocols in place allowing his COO to make critical decisions.

Rather than evaluating a young captain who followed sensible protocol — stopping after reaching an ice flow in the dark, then resting in the chart room before misinterpreting unclear signals — movies and media portrayed Captain Lord as an out of touch older man in his late 40s or 60s, sleeping peacefully in a comfy cabin while callously ignoring obvious cries for help.

No formal charges were brought against Captain Lord or his crew, but he spent the rest of his life fighting to clear his name.

So what can we learn from Captain Lord?

Not responding—or being seen not to respond—to an incident is hard to live down. Yet responding in uncertainty carries its own risks. At a minimum, the risk of overreaction. Had Lord responded decisively only to find a celebrating Titanic, would he have been the panicky captain who ruined his crew’s rest to chase after a firework? Worse, if he had attempted to sail through an ice field in the dark, he could have put his ship and crew at risk with the potential loss of even more lives. With a duty of care to consider, responding to uncertain signals is itself a risk. Indeed if reacting to every signal had been Lord’s usual behaviour, he would likely have lost the confidence of both his company and his crew.

His actions then appear reasonable, if perhaps not optimal—and that was the eventual judgement. Years later, a further analysis suggested that even if Lord had responded and got underway as promptly as could reasonably be expected, the SS Californian would simply have arrived alongside the RMS Carpathia, collecting survivors rather than saving more lives.

That may not be the whole story.

Lord did have some options. He knew the signals were present and relied on his own interpretation. Suppose instead he had woken his wireless operator and asked him to listen out? The Californian might then have heard the Titanic’s distress call. That may or may not have changed the number of lives lost—but what did he have to lose by seeking better information?

During the 2013 Target data breach, the company’s security team received alerts from their outsourced security operations centre (SOC) about suspicious activity. These were forwarded from the security team in Bangalore to the team in Minneapolis but were deemed not to require immediate action. That decision allowed the breach—which compromised the personal and credit card information of millions—to continue for an extended period. Inaction has a price.

In triaging signals that indicate a potential incident, then, there are three options—not two.

First, you can treat the signals as noise and take no action. Often (usually, even), that’s right—but not always. This was Lord’s choice, and he lived to regret it.

Second, you can take action immediately. Lord could have taken a heading from the flares, woken his crew, fired up the engines, and sailed full speed for the Titanic. This would have been a risk.

Third, you can seek more data. This option is often forgotten amid the pressure and urgency of an incident. But it’s usually possible to seek further information, and the cost of doing so is typically lower than the cost of immediate action and more useful than inaction. As long as more information helps you make a better decision, it’s a good course of action. At worst, you understand the signals better and mount a more informed response. Lord could have woken his wireless operator. Target could have checked for further signs of compromise. And Mr Woldbye at Heathrow? His COO was on duty and made the necessary decision in his absence. Whether it was right or wrong, time may tell. But in acting quickly on the basis of the best information they could obtain, they at least maintained control of the incident.

If in doubt, then?

Do something.

And if you’re in bed, on a ship surrounded by ice in the middle of the night, when you see an unexpected light in the distance?

You have three options — make your decision.

Subscribe now

Incidentally. The word itself sounds inadvertent; secondary. And in theory incidents are exactly that - a distraction from purpose, a painful moment in an otherwise rewarding journey.

But that’s not really true. We take risks not for fun, but to make progress. Incidents are simply a risk materialising; their impact is not inevitable, nor their probability -but their existence is.

Risk is good. Without risk, there could have been be no tea clippers to India, and there could be no shuttles to space. With new developments come the inevitable risk of disasters, and so we - the organisations and individuals taking these risks - need to minimise the chance of this happening, but also to control the costs when they do.

Because we can do that we survive to tell the story, and so progress is made.

Progress therefore requires that risk is taken, and that incidents must always be possible, just as any progress in technology or information also depends on the possibility of serious cyber security incidents, the risk of which must be acknowledged and managed.

This issue is not so much about cyber security as about the human condition. Our fears, and our desires. That which drives us forward. Our striving for more, or better, or different.

And in so doing we take a risk, and survive the consequences, and learn.

Incidents are about survival, and about safety and security. They are about the care we take for others and our organisations, about our need for achievement, about purpose and meaning, and about seeing our world differently. In Maslow’s hierarchy of needs, incidents illustrate all.

So not really incidental at all.

Given how important it is to manage the impact of incidents, there should be more written and shared about it. But perhaps we prefer to move on to happier thoughts; after all who wants to dwell on the things we didn’t get right? However we learn little or nothing from success. We learn from when we fail, because the truth of our decisions is finally made visible to us in the cost to ourselves and others.

“Truth is incontrovertible. Panic may resent it. Ignorance may deride it. Malice may distort it. But there it is.” - Winston Churchill

With misinformation and ‘fake news’ we are increasingly lied to, the malice of those both at home and abroad distorting our understanding. The modern pace of change makes us inevitably ignorant, and as a result sober and balanced warnings are often met with derision. When all else fails us and there is no choice but to accept our situation, panic too often grips us and distorts our outlook: we freeze or under-respond, further worsening our situation to the benefit of those who would cause us harm.

But the truth remains incontrovertible, if only we can find it. Now is the time to look.

A resource for responders, for leaders, for learners and teachers

This exercise is not about me giving you the answers. I do not have a magical filter through which to see the signals in the noise. You may know as much as I do, or more, or less: it does not matter. You do not need or want to be lectured. My goal instead is that others find this interesting, or thought provoking, or just plain wrong. If you do, I trust you will join the conversation, share your perspective, and help all us learn together.

This occasional newsletter will be of value to those who do respond to cyber incidents, to those who may do so, and to those who hope they never will.

It will be no help at all to those who are certain it will not happen to them. If this is you, you are in the wrong place: try this strategy instead.

There is a high probability of periodic insight in this newsletter, but there are also risks, including non-delivery and poor performance. These risks may be controlled through expertise and prioritisation, but they are not nil. Please tell me if I do not meet your expectations.

I’ve not imported subscribers from my previous newsletter, but I will email them and give them the choice.

Who am I?

Formerly a fortune 500 global CISO, I have served as a technology and cyber security leader for some of the world’s leading financial institutions, and had the privilege to advise both countries and corporations.

At Jersey Cyber Security Centre, my role is to lead Jersey’s cyber defence, overseeing the direction of JCSC to promote and improve cyber resilience across critical national infrastructure, businesses, communities, and citizens.

In all these roles, I have dealt with many incidents. Some successfully, and others less so. I no longer relish the next one, but I know it is coming. Perhaps it is already here.

Part of my role is to improve the way we handle cyber incidents and communicate good practice.

My goal is to do that with this newsletter too. If you read it and share it, I will be encouraged. So my time is in your hands.

Read me, share me!

Just as it took you time to read this, it took me time to write it. In exchange for my giving this to you for free, I ask only that if you find it useful, you forward it to a friend and suggest that they subscribe.

Regards,

Matt