Cyber risk is still treated as a sledgehammer, called in only after a breach
Less than half of organizations account for cloud-connected and mobile devices in their risk assessments, the speaker notes, even as ransomware attacks like Colonial Pipeline and JBS reach physical infrastructure. For security and risk teams, that gap means risk gets attention only after a breach. Priya Mouli, a technology risk director at KPMG, connects risk appetite, key risk indicators and reporting, then asks what makes a risk report leaders can actually act on.
Learn to build KRIs with thresholds and trigger actions, and turn them into reports boards can use
Priya Mouli, Director of Technology Risk Advisory at KPMG, explains how to build cyber risk metrics on a foundation of risk appetite and turn them into reporting that drives decisions. She covers ground rules for designing key risk indicators (KRIs), the data quality problems that undermine them, and the principles and automation that make risk reports useful from operations teams up to the board.
“Now, it is important to understand that it is these trigger actions when certain thresholds are reached that can be automated and not the metrics themselves.”
“KRIs need to be viewed together to give you the visibility into your true risk exposure.”
What you will learn
Anchor key risk indicator thresholds to your organization's risk appetite so metrics reflect what the business is willing to accept.
Apply ground rules for building KRIs: trusted data, a shared risk and controls taxonomy, alignment to a key risk and an up-to-date data dictionary.
Spot the data quality failures that most often break risk metrics, such as stale data, missing owners, no system of record and disconnected tools.
Design risk reports tailored to each audience that mix quantitative metrics with owner commentary and flag emerging risks.
Plan end-to-end automation of risk reporting, from source data to dashboards, connected to a golden source such as a GRC tool.
Key takeaways from the speaker
1) The What, Why and How of Risk Metrics
2) Understanding the Essence of Risk Reporting
3) What Makes Risk Reporting Efficient and Effective
Details
Level
Intermediate
Format
Talk
Technical depth
Practitioner
For
Mid-career, Senior individual contributor, People manager
Length
17:19
Published
June 2021
Questions this talk answers
What is technology risk management?
It deals with any risk that touches data or technology assets in an organization. It works as an umbrella framework across disciplines such as identity and access management, application security, third-party security and enterprise resilience.
Why does risk appetite matter when building risk metrics?
Risk appetite is the amount and type of risk an organization is willing to pursue or retain, and risk metric thresholds should be set in line with it. Thresholds with trigger actions then give visibility into risks before they occur.
What needs to be in place to build good key risk indicators?
Data that stakeholders trust, a risk and controls taxonomy that standardizes language, metrics aligned to a key risk, and an up-to-date data dictionary of formulae, thresholds, sources, owners and reports. KRIs also need a governance process because the build process is a cycle.
Why do risk metrics so often fail?
KRIs are often read in isolation instead of together, and poor data quality is what most companies struggle with. Problems arise when data is not refreshed, not updated by the right owners, has no system of record or sits in tools that do not talk to each other.
What makes risk reporting effective?
Reporting should be timely at every level, tailored to the audience, forward-looking, and mix qualitative commentary with quantitative metrics. Automating it end to end improves data confidence and saves many hours each reporting cycle.
About this talk
The presentation will cover the below areas :
1) Risk Appetite as a Foundation for Risk Metrics
2) The What, Why and How of Risk Metrics
3) Communicating Risk through Risk Reporting and Automation
Why KRIs need thresholds and trigger actions (34 s)
The data quality problem behind failed risk metrics (45 s)
What makes a risk report useful to the board (49 s)
Speaker: Okay. Hey, everyone. First of all, I thank you all for taking the time and joining us for this session today. And a special thanks to the Women Tech Conference team for having me. First of all, I hope you and your families are all staying safe and healthy and are all vaccinated. Like many of you, I too cannot wait to just get out and start traveling.
Speaker: Okay. Some housekeeping items before we get started. This session is around 20 minutes. I will cover the presentation material in about 15 minutes. So there's at least five minutes for questions and answers. So as questions come along, do feel free to type them into the chat window here. By way of introduction, I am Priya Mouli. I'm based out of New York and I'm a director in KPMG's technology risk consulting practice. I cover the areas of cybersecurity, privacy, data protection, and regulatory compliance. Most of my experience has been with supporting financial services and technology companies with their cybersecurity strategy. So I've worked with the three lines of defense on building and really operationalizing their tech risk programs.
My recent focus has been leading engagements around continuous controls monitoring, streamlining processes between risk and compliance management, and really working with senior management and the board to really think through that middle ground between security risk and agility in the environment. So I think that's one of the things that we're doing. So I think that's one of the things that I do for fun. I love watching movies in theaters. So with COVID around, I binge-watch Netflix and particularly tune into Grace Anatomy. And I spend my free time writing poems, learning new languages, learning new languages. And learning about wildlife, particularly predators. Okay. So let's move on to our discussion for today. Let me move into the agenda slide. So before we jump into the world of risk
metrics, I wanted to start off with an introduction of what is technology risk management. And after that, we will move on to risk appetite to set the foundations and then move on to the what, why, and the building of risk metrics. And ultimately we will move into risk reporting. But what is technology risk management? I know the slide has a lot, but then simply put, it deals with any risk that touches data or technology assets in an organization. Technology risk management is meant to be an umbrella framework to help with managing risk across multiple disciplines, such as identity and access management, application security, third party security, enterprise resilience, and so on. So why are we talking about tech risk and why now? We are we are revolution today. And based on a recent survey, we're still seeing that for the most part, technology
risk is still seen as a sledgehammer, so to speak, or as an internal firm regulator, as a result of which they are engaged only after an issue surfaces or a breach occurs. Also, less than 50% of organizations account for cloud connected devices and mobile device and application management in their risk assessments. Now, this is a shockingly low number, given the huge uptick in digital transformation, especially in a post COVID world. But it is important, especially now to not let cyber risk take the backseat. Also, hackers are just waiting for the right moment to strike. And we've already seen ransomware attacks to this effect. The most recent ones being colonial pipeline and JBS where hackers are really looking beyond the data into targeting the physical assets. And here we see cases in point around
targeting the critical infrastructure and supply chain. So it only makes sense to start thinking of proactive risk management and risk measures or metrics as a way to achieve just that. With that said, let's enter the world of risk metrics. So as I mentioned earlier, let's set the foundations right with looking at risk appetite. So risk appetite is the amount and type of risk that an organization is willing to pursue or can retain or can take. And what I've seen across my clients is that risk appetite is generally a set of overarching qualitative statements that depend on the company's business context. For example, keeping the customer transaction availability is important for a retail bank with an omnichannel presence.
The OCC heightened standards has some guidance around how to set risk appetite within the organization and who within the organization is responsible for setting this and ensuring that it stays current. OCC, by the way, stands for Office of the Comptroller of Currency. Essentially, what this is, is that it's an independent bureau within the United States Department of Treasury that serves to regulate and supervise all national banks and licensed agencies of foreign banks to ensure the safety and soundness of the US banking system. So you might be wondering, why are we talking about risk appetite as it concerns metrics? Well, we generally want risk metric thresholds to be in line with their risk appetite. And this brings me to why we even need metrics. So risk metrics, as you see on the slide, are measures, numbers, percentages, or trends
that serve as a measure of the risk that an organization faces. And this can be tailored at varying levels of granularity. And risk metrics with thresholds and corresponding trigger actions to find can enable companies to stay ahead of the curve by allowing them to get visibility into risks before they occur and hence enables proactive risk management. Now, it is important to understand that it is these trigger actions when certain thresholds are reached that can be automated and not the metrics themselves.
Now, let's move on to the KRI build process, the how of risk metrics. Now, this slide lays out four basic steps. But note that this is a cycle, right, as KRIs need to be refreshed periodically. Let's talk some ground rules for developing them. First off, data should be available to support the metric, and key stakeholders, such as your risk control and process owners should have high confidence in this data. It would also be helpful to have a risk and controls taxonomy, namely, a domain hierarchy or structuring or framework. And what this means is that the intent is to really standardize the risk and controls language to enable a common understanding of risk across business functions. Another thing to keep in mind is that metrics work best when they are aligned to a key risk and serve as an effective measure of
that risk. For example, absolute numbers in the case of audit or regulatory findings work better, and relative trends such as percentage increase or percentage over a period work better when you're measuring, say, the percentage of emergency changes or the change failures. So, KRIs could also serve as, you know, strategic or aspirational measures, you know, strategic or aspirational measures, meaning what an organization may want to focus on in the near future. For example, metrics on insider threat.
You can also have metrics around, you know, that could be special purpose or trigger based in cases of a merger or acquisition or divestiture, right? The reason I bring this up is that with COVID, we see a lot of consolidation across industries. Another point I want to mention is that metrics could also serve as maturity indicators of your program. For example, if your organization has a higher number of automated versus manual controls, you know that you're at the higher end of the maturity curve.
And last but not least, right? But the most important, in my view, is to have an up-to-date data dictionary, capturing the entire data, such as the KRI formulae, thresholds, the data elements that constitute the formulae, where they are sourced, their update frequency, owners, which reports they land on, et cetera. Now let's go through some examples of KRIs to make it real. One metric that comes to mind is, you know, the percentage of incidents impacting systems housing, meaning personally identifiable information.
Another metric could be the number of privileged systems you might want to monitor. And you can see if this is trending up or down. You might also want to use composite or complex metrics. For example, in the same privileged accounts case, you might want to break it down by employees, meaning full-time and part-time versus third parties, including your vendors, contractors, business partners, and also slice it down by their concentration by geography and have the split by the critical application systems and platforms they have access to. Another timely KRI that comes to mind is the percentage of unplanned or emergency changes and the percentage increase in the procurement of collaboration tools, such as Zoom or Microsoft Teams with certain contractual classes relaxed. Now, I brought these
specific examples, recognizing that some firms may be relaxing their thresholds given the current situation. What I'd like to highlight is that to resume to original thresholds once things get better, we should be okay. Like I said, the KRI development process laid out is a cycle and there needs to be an associated governance process for any changes to the metrics, say proposing new ones, changing formulae and related data, including retiral of these metrics. Now that we've covered the how, let's discuss some of the common challenges in using metrics and how to make them more effective. This slide is pretty self-explanatory, but three things I'd like to emphasize are this one here. KRIs need to be viewed together to give you the visibility into your true risk exposure. For example, you might have a metric around the percentage of assets
that are end of life or coming to end of life. Another metric you might have might be around the percentage of assets that you really have insight into. For example, if you have a number of gaps in your asset inventory reconciliation process, you know you have a problem. The second most important point is poor data quality. Now, there is a reason as to why I've blown this up on the slide. I can't emphasize enough on this, as this is what most companies struggle with. Not necessarily because they have bad data to start with, but issues start arising when this data is not refreshed regularly, when it's not updated by the right owners, when there is no system of record or, you know, having a disconnected ecosystem, meaning tools
not talking to each other. So let's land into the world of risk reporting, right? Because now we've spoken about risk metrics. Let's talk about how the next step, the next steps of how KRIs, their trends, and related data will be used to really drive risk decisioning. Now, risk reporting is generally considered to, yeah, it's generally considered to be an upper management item, but it is important that reporting happens timely across all organizational levels, right from the day-to-day operational teams to middle management, to leadership, and ultimately to the board. Now, there are quite some guiding principles on the slide, but four points I'd like to call out are relevant. Understanding audience is very important in terms of fine-tuning the content and the level of detail and determining the appropriate frequency of metrics,
especially for, you know, executive leadership and the board, the content should be made available succinctly across devices and platforms, with drill down abilities into the detail as and when they need it. The second point, which is point number four here, is advanced and aspirational. Now, risk reports should have a forward-looking agenda in terms of bringing attention to emerging risks based on industry trends and your internal environment. The third point is around numbers and words. So, well, what I mean to say here is that risk reports should have the right mix of qualitative data, meaning commentary from risk and control owners with respect to certain metric values, your trends, any action items planned, and quantitative data, meaning the actual metrics and their values themselves. Now, the critical element on the slide is
around automation of the risk reporting process, soup to nuts, right from pulling source data to translating them to risk metrics, gathering commentary from stakeholders, and pushing them into visualization tools or dashboards using technologies such as, you know, a Tableau, a ClickView, or a Power BI. And this works best for two reasons. A, it improves data confidence by reducing manual intervention in the data. And B, the number of man hours saved for every reporting cycle could be huge. And this is based on what I've seen at almost every organization I've worked with. And I'm certain you might have too. Now, the other aspect I would touch on is that some of the large banks I work with are seeing more regulatory scrutiny in involvement of their boards in providing credible challenge and weighing in on cyber risk related decisions. And what
this means is that the reporting going to the boards, right, should be timely, clear, concise, absorbable, and needs to be more transparent from companies management, even if it means delivering bad news things without any fear of repercussions. Okay, so now, we've covered the guidance. Let's talk about the inputs and outputs of risk reports. This slide has a lot to say, but a couple of things I'd like to highlight or call your attention to are, make sure that when you have risk reports, you cover the entire tech risk universe, meaning across domains, right, such as I mentioned, application security, third-party security, enterprise resilience, outside of focusing just on compliance issues.
The second point I'll touch on is that perform. It is really critical to perform a period of analysis by taking your risks, issues, control risk, failures, problems, however your organization calls them, to understand themes and trends, to really pinpoint those areas And this can potentially show you where you can tune down your existing controls, resulting in cost savings.
With that said, we will move on to the next slide. This basically has some CEO survey results on stats with respect to what companies are seeing with respect to some of the common challenges they are facing in seeing the true risk reporting. Now, let's talk about some of the ways to overcome this. A is around having a uniform or standardized risk and controls language and taxonomy, as I mentioned earlier, and B is having a golden source, meaning having a system of record like GRC, meaning a governance risk and compliance tool, to store all risk and control related data.
And it might not be your single repository, but it's also important to make sure that this system of record that you're using for your risk reports is connected to your other tools, like say your Jira issue tracker and your service now technology tickets that you're housing. Okay, so with that said, I want to part off with some concluding thoughts before we move on to questions now we may sometimes over engineer and sometimes reinvent the wheel, but on risk management, but if you take a step back, it might actually not be that complex to start with.
It's important to start with the basics no doubt and once you have the foundations right, like having a uniform taxonomy, enabling a common understanding of risk across business functions in an organization, we can potentially embrace and unleash new technologies at scale and we've already seen a huge uptick in digital transformation in a COVID world. All right, well, thank you so much for your time.
That's all I had.
Transcribed automatically; names checked against the speaker record. Select a time to jump there.
Article based on this talk
How to build cyber risk metrics and reports that leaders can act on
KPMG's Priya Mouli shows why risk appetite, trusted data and automated reporting decide whether key risk indicators help or just add noise.
In short
Cyber risk metrics only work when they rest on a clear risk appetite, trusted data and a shared risk language, argues Priya Mouli, Director of Technology Risk Advisory at KPMG. She walks through how to build key risk indicators (KRIs) with thresholds and trigger actions, why most organizations stumble on data quality, and what makes risk reporting useful from operational teams all the way to the board.
Key takeaways
The what, why and how of risk metrics
Understanding the essence of risk reporting
What makes risk reporting efficient and effective
Why technology risk cannot wait
Mouli starts with a plain definition: technology risk management deals with any risk that touches data or technology assets in an organization. It acts as an umbrella framework across disciplines such as identity and access management, application security, third-party security and enterprise resilience.
The problem, she says, is how organizations still treat it. Citing a recent survey, she notes that technology risk is often seen as a sledgehammer or an internal regulator, engaged only after an issue surfaces or a breach occurs. She adds that less than 50% of organizations account for cloud-connected devices and mobile device and application management in their risk assessments, which she calls shockingly low given the surge in digital transformation. Ransomware attacks such as Colonial Pipeline and JBS show attackers moving beyond data to physical assets, critical infrastructure and the supply chain.
"Also, hackers are just waiting for the right moment to strike." (3:22)
Her answer is proactive risk management, with metrics as the tool to achieve it.
Risk appetite comes first
Before any metric, Mouli sets the foundation. "So risk appetite is the amount and type of risk that an organization is willing to pursue or can retain or can take." (4:14) Across her clients, risk appetite usually takes the form of overarching qualitative statements tied to business context, such as keeping customer transactions available for a retail bank with an omnichannel presence. She points to the OCC heightened standards as guidance on how to set risk appetite and who is responsible for keeping it current.
The link to metrics is direct: KRI thresholds should be in line with risk appetite. Risk metrics are measures, numbers, percentages or trends that can be tailored to different levels of granularity, and when paired with thresholds and trigger actions they give visibility into risks before they occur. She draws a precise line on automation:
"Now, it is important to understand that it is these trigger actions when certain thresholds are reached that can be automated and not the metrics themselves." (6:04)
Ground rules for building KRIs
Mouli presents the KRI build process as a cycle, because indicators need regular refreshing. Her ground rules:
Data must be available to support the metric, and risk, control and process owners must trust it.
A risk and controls taxonomy standardizes language so business functions share an understanding of risk.
Metrics should align to a key risk. Absolute numbers suit audit or regulatory findings; relative trends suit measures like emergency changes or change failures.
KRIs can be strategic or aspirational (insider threat), special purpose (mergers, acquisitions or divestitures) or maturity indicators (automated versus manual controls).
Most important in her view: an up-to-date data dictionary capturing formulae, thresholds, data elements, sources, update frequency, owners and the reports each metric lands on.
Her examples make this concrete: the percentage of incidents impacting systems that house personally identifiable information, privileged accounts broken down by employees and third parties, geography and critical systems, and the procurement of collaboration tools such as Zoom or Microsoft Teams. Any change to metrics, from new proposals to retirement, needs a governance process.
Two challenges stand out. First, "KRIs need to be viewed together to give you the visibility into your true risk exposure." (10:01) End-of-life assets mean little if you also lack visibility into parts of your asset inventory. Second, data quality: "I can't emphasize enough on this, as this is what most companies struggle with." (10:55) The trouble usually comes not from bad data at the start but from data that is not refreshed, not updated by the right owners, lacks a system of record or lives in disconnected tools.
Reporting that drives decisions
Risk reporting is often seen as an upper management item, but Mouli argues it must happen in a timely way at every level, from day-to-day operations to middle management, leadership and the board. Her guiding principles include knowing the audience, so content, detail and frequency fit, with succinct, drill-down views for executives and the board. Reports should also look ahead:
"Now, risk reports should have a forward-looking agenda in terms of bringing attention to emerging risks based on industry trends and your internal environment." (12:34)
She calls for the right mix of numbers and words: metrics alongside commentary from risk and control owners on trends and planned actions. The critical element is automation from source data through to dashboards in tools such as Tableau, QlikView or Power BI, which improves data confidence by reducing manual intervention. "And B, the number of man hours saved for every reporting cycle could be huge." (13:23)
She also notes that large banks she works with face more regulatory scrutiny of board involvement in cyber risk decisions, so board reporting must be timely, clear, concise and transparent, even when the news is bad. Reports should cover the whole technology risk universe, not just compliance issues, and analysis of issues and control failures can reveal themes and even where controls can be tuned down to save cost.
Golden sources and getting the basics right
To overcome common reporting challenges, Mouli recommends a standardized risk and controls taxonomy and a golden source, such as a governance, risk and compliance (GRC) tool, connected to systems like Jira and ServiceNow. Her closing point is that risk management is often over-engineered:
"It's important to start with the basics no doubt and once you have the foundations right, like having a uniform taxonomy, enabling a common understanding of risk across business functions in an organization, we can potentially embrace and unleash new technologies at scale and we've already seen a huge uptick in digital transformation in a COVID world." (16:48)
Watch the talk
"Demystifying Cyber Risk Metrics and Reporting" by Priya Mouli (17 minutes): https://www.womentech.net/video/priya-mouli-demystifying-cyber-risk-metrics-and-reportingapply-speak
About the speaker
Priya Mouli is Director, Technology Risk Advisory at KPMG in New York, with 14 years of experience supporting global organizations with IT risk strategy across cybersecurity, regulatory compliance, privacy and data protection. She holds an MBA from IIM-B and a CISSP, and has helped build and operationalize technology risk programs from the ground up, bringing process improvement and automation to them.
Questions this talk answers
What is technology risk management?
Why does risk appetite matter when building risk metrics?
What needs to be in place to build good key risk indicators?
Why do risk metrics so often fail?
What makes risk reporting effective?
Drafted with AI from the talk transcript. Editor review pending.