Tools & Techniques - Key Performance Indicators

Introduction

To date, I've facilitated senior level reporting on the performance of security driven activity in almost every position I've held. For the best part, this has been a green-field requirement which has meant that I've been able to set out and make a case for reporting that drives improvement in the real-world security posture (RWSP) of my charge from the get-go. I believe, no matter what else is going on, that if you can inform the changes and behaviours that lead to your organisation protecting itself, you've had a good day. Secondly to driving improvement, KPIs and metrics provide your leadership team with the insight and visibility into the efforts and valuable work of your security team which might otherwise be hidden from them. 

There is no shortage of materials and musings about the importance of security KPIs however, despite this, many organisations struggle in a number of ways to define and/or employ them. I thought then I'd share some thoughts on the subject and set out some simple parameters for defining KPIs.

What is a Key Performance Indicator (KPI)?

Lets stars with a definition: 
  • A Key Performance Indicator (KPI) is a measurable value that demonstrates how effectively your organisation is achieving key objectives. To this end then, organisations can use KPIs to evaluate their success at reaching targets.
  • A KPI is not the turn-key reporting from every product or service managed by the security team or that the security team has an interest in...
  • A KPI is not a dashboard representing the collective outputs from every client, server and device across your estate (aka dashboard fever)...
Understanding what a KPI should be and why/how it should be used compared with the raw data that might inform it is vital. KPIs should be specific to your organisation and some time and effort should be needed to design them. For a lot of organisations though, the need for KPIs has become confused with the need for teams, products and services to demonstrate their own intrinsic value. Vendors too recognise the need to demonstrate their value and typically include 'Executive' or 'Management' reporting or dashboards which typically become presented as KPIs. 

Getting Started - Defining your Objectives

To develop effective KPIs you first need to identify your objectives. The amount of steer and support you receive from your leadership team is fundamentally important. A lack of management commitment will not only effect your ability to define KPIs but also your ability to make the necessary changes in their pursuit. KPIs should reflect specific business objectives. If you don't have the support of the business then KPIs, and the work that informs them just becomes another *thing* security tries to get people to do. 

KPI Parameters 

KPIs should comply with these parameters:
  • KPIs should be aligned with the strategic goals of the business. They should have a clear, documented objective (i.e. "Lowering non-compliant hosts below 1% total hosts will reduce our vulnerable surface area in line with the Executive's risk appetite").
  • KPIs should be attainable and their pursuit achievable. KPIs should not be aspirational or beyond the reach of your current capabilities.
  • KPIs should be acute and should help keep everyone on the same page and moving in the same direction. They should be approved by stakeholders before being tracked and reported on.
  • KPIs should be accurate, based on trustworthy and reliable data sources. 
  • KPIs should be actionable and should provide insight and information into the business it's processes.
  • KPIs should be alive and reviewed, updated or amended periodically in line with the business objectives or emerging situations.

Developing KPIs

In addition to being customised, KPIs need to be developed over time as your organisation's objectives change and as the pursuit of these effects how security considerations and requirements adapt. Pursuing poorly conceived KPIs, or KPIs that are impossible, unrealistic or outdated can result in significant cost and disruption. Pursuing the wrong KPIs is detrimental and can even instigate 'bad' behaviours. If you don't receive feedback the relevance of KPI sets, ask...

Thoughts on Presentation 

Executive level updates are likely to form a small part in a larger set of cross-business updates. To this end, brevity and impact are essential when considering how to effectively communicate with business leaders. Visual aids can be used to convey large amounts of information quickly. Traffic lights, scales and binary measures have always served me well. Know your audience...

Popular posts from this blog

Tools & Techniques - Cloud Firewalls (DigitalOcean)

Image
My home lab is (probably) typical of most security professionals: a beefy workstation running VMWare workstation, a beefy-ish workstation running ESXi and a bunch of laptops, switches and other devices.  I utilise a couple of VPS providers for hosting and exposing VMs to the cloud. My VPS provider of choice has been DigitalOcean for the last couple of years (going by my billing history) and to date, they've been excellent. They recently introduced and advertised a new service feature called 'Cloud Firewalls' and I had chance to have a play with them today. Essentially, they've incorporated a network level firewall service to their VPS offering which can be used as an alternative or in addition to host-based firewalls like iptables, firewalld, etc.  Initial Impressions - Pros, Cons and Limitations Pros No Cost (free!!) - Cloud Firewalls are available at no additional cost. Availability - Cloud Firewalls are available in ever region DigitalOcean operate.
Read more

PCI-DSS: Your CDE may be getting bigger

Image
In December, the PCI Security Standards Council released guidance around scoping and network segmentation for card data environment (CDE) assets. In their own words: “Many organizations struggle to understand where PCI DSS controls are required and which systems need to be protected. This document provides guidance to help organizations identify the systems that, at a minimum, need to be included in scope for PCI DSS. Additionally, the document provides guidance on how segmentation can be used to help reduce the number of systems that require PCI DSS controls.” The guidance goes on to visualise a scoping process, considering each category of system in turn: I believe it’s worth highlighting the ‘broadness’ of the “Connected to OR Security Impacting Systems” . These are cases where the: System component is on a different network (or subnet or VLAN), but can connect to or access the CDE (e.g., via internal network connectivity). System component can connect
Read more

WAR GAMES - Simulating Security Incidents

Image
Why it's a good idea There are a myriad of reasons to test in peace time, the controls and processes which collectively represent your incident readiness. These include: Validating your Incident Readiness - testing can confirm that you are as ready as you think you are, and that nothing has changed resulting in an end state which prevents you from initiating IR.  Assess Controls Coverage and Identify Gaps - testing can confirm your controls coverage is adequate as well as highlighting those gaps which you'd rather not have in the event of a real issue. Demonstrate value of investment - you've probably spent a lot of time and other people's money acquiring controls, attracting talent and preparing for the eventually of an incident. Internal stakeholders will likely already be looking for assurance that their investment has been worth while.  Demonstrate investment and commitment to interested parties - internal stakeholders considered, you'll lik
Read more