Sanne

Ash Hunt, Group Head of Information Security, Sanne

Having provided sagacious advice for generations, few would challenge the carpenter’s old adage.

The role of diligence before execution-measuring repeatedly before acting-remains sound judgement for most tasks in life, from cutting wood to deploying intricate designs into production –– accurately measuring cyber loss is no exception.

Unfortunately, vast coteries of sceptics across the profession peddle the superficial view that this is a Sisyphean challenge –– no tool, including pervasive, well-trodden management templates, can measure the myriad complexities of cyber risk. Of course, this is nonsense; the reverse is true. Cyber risk is measurable, no less than complex scenarios in pharmacology or financial risk. This misconception is rooted in conflating problematic methods with a worthy (and achievable) objective –– how much is an organisation’s cyber loss exposure and where will it yield the greatest ROI on allocating resources to mitigate cyber risk?

A gamut of research has exposed problematic methods of measurement to be cyber risk’s equivalent of Iron Sulphide. Colour charts; RAG matrices; ordinal scores — qualitative labels that mask reality, lulling unwitting audiences into ascribing certainty to something ill-defined, and thus delivering trivial value. The risk matrix–the principal siren–continues to lure practitioners with colourful formality and structure whilst inevitably increasing the perception of confidence in estimates. Ironically, this leads to diminishing returns — practitioners feel more secure in their estimates but simultaneously drift further from reflecting reality. This ‘analysis placebo’ emboldens harmful heuristics and cognitive biases, as well as a litany of interfering components, including partition dependence and lie factors. It’s a worrying reality these approaches routinely prop up thousands and millions of pounds of investment expenditure –– hardly ingratiating cyber teams to the CFO et. Al.

Frustratingly, the alternatives are numerous, trialled successfully over decades. Quantitative techniques–those delivering meaningful measurement­–serve to validate a hypothesis using a model whose efficacy can be continuously tested and measured.

Start with the individual. Cognitively, practitioners are inherently poor at projecting the future benefits of current choices, which exacerbates over time. Calibration addresses this by providing a measurable cognitive structure for deriving accurate estimates. Using confidence intervals to model an estimated range of plausible values (e.g. a 90 percent Confidence Interval estimating a data breach cost between £7500 and £250,000), Calibration checks against biases, overconfidence and inconsistency, enabling practitioners to demonstrate increasing improvement in successive estimates. A critical panacea against Feynman’s shrewd observation that “you’re the easiest person to fool.”

Effective measurement techniques are also mutually supportive –– Calibration facilitates feedback. The feedback loop for cyber risk is conventionally slow and inconsistent; practitioners don’t receive large samples of rare events or the effectiveness of mitigations against them. With the only validation being clinical trials of techniques and models, a robust feedback model promises accumulative value by absorbing feedback to improve future forecasting, enabling practitioners to stress test the accuracy of estimates and efficacy of investment decisions over time.

Beyond the individual, quantitative techniques structure an analytical model that underpins the measurement of complex factors in cyber risk. Scenario analysis offers a useful example.

Save the vague, audit-esque terminology of cyber risk, practitioners need to frame a specific loss scenario –– this should scope the threat profiled, threat event/attack technique(s) leveraged, asset(s) targeted, vulnerabilities exploited and type of loss caused.

With a defined scenario, data needs to be collected for given parameters. This often denotes the stopping point for sceptics, who brandish the critique of not having enough data. No such arbitrary threshold exists, mathematically or otherwise. Hoards of data samples exist across industry and within organisations. Practitioners need to leverage opinion and judgement to establish relationships, and condition those with empirical data.

"Beyond the individual, quantitative techniques structure an analytical model that underpins the measurement of complex factors in cyber risk."

Measuring factors within a cyber risk scenario is as much an art as it is a science. Irrespective of what factors are selected, practitioners need to determine how probable a loss event is to occur (i.e. frequency) and the extent of loss inflicted (i.e. productivity, response and replacement cost, reputational damage, legal & regulatory fines, and competitive advantage loss). Once measured, leveraging Monte Carlo analysis to simulate the scenario enables practitioners to ‘experience’ the risk thousands or millions of times, providing a distributed range of probable outcomes –– something that could not be afforded in reality. And forget the archaic notion of ‘inherent’ and ‘residual’ risk, particularly on the suggestion of omitting controls (this would be fictitious for almost every organisation). Instead, focus on modelling the organisation’s loss exposure as it stands and once again with hypothetical control improvements (this could be reflected in the reduced frequency of the loss event or a decrease in the loss inflicted). These two world views empower practitioners to measure the delta in loss and, with known control enhancement costs, the ROI on investment initiatives. Repeating this process over time across multiple scenarios yields an increasingly accurate forecast of where lies the greatest bang for buck in mitigating cyber risk.

Of course, no model is perfect. But limitations of scientific models are no argument for random intuition. Even naive statistical models routinely outperform intuition –– the objective is selecting a model that is measurably better at forecasting outcomes than others. A carpenter wouldn’t guess before cutting, nor should we.