NIST and No-notice: Finding the Goldilocks zone for phishing simulation difficulty

Earlier this year, the National Institute for Standards and Technology (NIST) published updated recommendations for phishing simulations in security awareness training programs. We discussed it on our Community page soon after the updated standards were released, but the substance of the change bears repeating.

“Practical exercises include no-notice social engineering attempts to collect information, gain unauthorized access, or simulate the adverse impact of opening malicious email attachments or invoking, via spear-phishing attacks, malicious web links.” – NIST SP 800-53, Rev. 5, Section 5.3 (pg. 60)

This update includes a recommendation for “no-notice” phishing simulations to be delivered at the beginning of security awareness training programs to more accurately gauge the readiness of a set of users to recognize a phishing attempt.

The thinking obviously being that letting users in on the phishing simulation game will heighten suspicion of their inbox and skew baseline results. This concern can be thought as a spin-off of the well-studied “Observer Effect” known in many scientific fields; observing the behavior of something necessarily changes that behavior.

While it might be tempting for a Chief Information Security Officer (CISO) or other IT professional to take high grades on a phishing simulation a sign of a job well done, that can be a dangerous conclusion to draw. Phishing tests that are too easy do little to address a problem that’s become one of the most common methods of entry for ransomware attacks.¹ If IT professionals grade on a curve here, they’re doing very little to improve their organization’s overall cyber resilience.

Combatting this false sense of confidence about users’ ability to spot phishing attacks requires making sure simulations aren’t too easy to spot.

What makes a phishing simulation too easy?

After putting some thought into that question, NIST researchers published a paper last year in the Journal of Cybersecurity citing three key criteria for determining if a phishing simulation makes for good training.

According to the authors, “low click rates do not necessarily indicate training effectiveness and may instead mean the phishing emails” were:

1. Too obvious – Either errors were too overt or these templates were running something akin to the Nigerian Prince scam. Either way, they won’t help an employee overcome today’s more sophisticated phishing attempts
2. Not relevant to staff – We’re all busy at work. So deleting an email offering 25% off at Ed’s Golf Cart Repair Shop doesn’t mean a user is an expert at spotting scams. It just means there was nothing in the simulation that enticed anyone to click.
3. The phish was repeated or similar to one that was – Phish me once, shame on me…but seriously, this drives home the importance of having a wide range of phishing templates. These programs work best when they’re ongoing, so it’s important to switch it up.

On the other hand, a phishing simulation is convincing if it does the following to some degree:

• Mimics a workplace process or practice
• Has workplace relevance
• Aligns with other situations or events, including those external to the workplace
• Presents consequences for NOT clicking (e.g., buy gift cards or we lose the client)
• References targeted training, specific warnings or other exposure

Tip: NIST has devised a weighted version of this scale, “the phish scale,” you can use to determine the difficulty of your simulations. A phishing simulation that has all of the above characteristics would be considered extremely difficult. That’s good, right?

Too much difficulty can be dangerous, too

Any security awareness training program that’s too difficult is liable to leave learners feeling put off, resigned to failure, or worse, coming away without any practical security learnings. This is especially true if users are punished too harshly for failing to spot a difficult phishing simulation.

Any program that’s both difficult and relying on a stick rather than a carrot for motivation runs the risk of:

• Reinforcing negative stereotypes of security training programs
• Encouraging employees to “game” the system by sharing information about tests
• Fostering animosity towards the organization’s overall security posture
• Inviting legal trouble from dissatisfied employees

For security awareness training to be successful, it has to be collaborative. Learners should feel like they’re part of something constructive, rather than just subjected to another type of performance review.

Hitting the sweet spot

Finding the appropriate difficulty level for phishing simulations is one of the reasons the initial, no-notice NIST recommendation is so important. It helps administrators establish baseline results that most accurately reflect users’ real understanding of phishing attacks. But we don’t recommend a training program be hidden from employees forever.

Instead, after initial results have been established, it’s better to announce the program publicly along with its goals, evaluation criteria and a point of contact for those interested in learning more. Once users are in the know, subsequent phishing simulations can focus on incremental improvements over the baseline results. As scores rise across the board, the difficulty can be gradually increased over time.

One essential recommendation: Always report publicly on positive results. Let users know they’re managing to catch more and more difficult simulations. Be as specific as possible, as in, “click-through rates dropped from A to B in this exercise.” This will help establish a sense of shared responsibility for organizational security and “gamify” the experience.

Calibrating your security awareness training is an ongoing experience. Don’t be afraid to adjust your simulations based on results. Happy learning.

Ready to establish your own successful security awareness training? Try us out free for 30 days. 

1. ^{Hiscox. “Cyber Readiness Report 2021.” (April 2021)}