Vulnerability Management and Developer Toil

There have been several catch phrases in the cybersecurity industry that have become frequently cited in the last several years. Among them are things such as:

• Changing security from the office of “no”

• Minimizing friction on Developers

• Being a business enabler

• Guardrails over gates

You get the picture.

A common theme is the need to shift security from being a cost center that imposes unnecessary friction on the business, to an entity that functions as a business enabler, revenue protector and a group that works symbiotically, rather than contentiously with Developers (e.g. DevSecOps).

While there are many aspects to cybersecurity, one key activity is that of Vulnerability Management. That is, trying to help the organization manage vulnerabilities associated with their systems, technologies and environments. To begin with, what is a vulnerability? To have a shared language, let’s refer to NIST’s definition:

While vulnerabilities manifest in many forms, the most notable in the industry is in what are called Common Vulnerabilities and Enumerations (CVE)’s, that are available in a vulnerability database known as the NIST National Vulnerability Database (NVD).

We’ve previously written extensively about the Vulnerability Database ecosystem, including NVD and CVE’s, which can be found here, so I will not be diving into that in this article.

That said, I did want to take a look and highlight some findings from one of the largest Vulnerability Management/Security vendors, Qualys, from their 2023 TRURISK Research Report, which can be found here.

Given companies such as Qualys’ size, they are able to provide insights and metrics that would be difficult to find from other sources. To given an idea of the scale/scope they operate, they have conducted over 6 billion scans, managed 90 million agents and deployed 45 million patches, in just 2022.

They utilized insight from these activities to create the report while anonymizing the findings and metrics to protect the privacy of their customers while still providing the insights to the industry. The report overall utilizes over 13 trillion data points to determine the vulnerabilities that pose the most risk to organizations.

One major problem in our industry is that most organizations, including the U.S. Federal Government and Department of Defense (DoD) (for a detailed look at Federal Vulnerability Management, use my friend Walter Haydock’s article “Revealing the government’s approach to vulnerability management”) utilize blunt instruments such as Common Vulnerability Scoring System (CVSS) severity scores to drive vulnerability priorization.

It often manifests in the form of deployment gates in the CI/CD context (e.g. no Critical vulnerabilities can go to production), or drives vulnerability remediation timeline requirements for runtime/production environments (e.g. all Critical vulnerabilities must be resolved in 7 days, High’s in 30 days and so on).

The reason this is problematic is that most CVE’s are never actually exploited, there are significant issues with the CVSS scoring system itself and this approach also lacks context about the systems use within the business context, compensating/mitigating controls and more.

To recap some of the key activities and findings of their report, they detected more than 2.3 billion vulnerabilities in 2022. They conducted over 6 billion IP Scans/Audits, using 50,000+ scanner appliances. They also used more than 84 million cloud agencies that aggregated over 2 trillion security events.

While the report covers some great insights related to Ransomware, Initial Access and other key findings, I wanted to focus on the insights it provides regarding the explosive growth of vulnerabilities and the reality that a very small subset of all known vulnerabilities ever go on to be exploited.

The report shows how the number of known vulnerabilities (e.g. CVE’s) grew from less than 3,000 in the 1990’s to over 190,000 in 2022. Years between those periods saw growth in vulnerabilities in ranges such as 796% to 5,116% (yes, really).

(Source: Qualys 2023 TRURISK Threat Report)

On one hand, this shouldn’t be surprising. The ubiquity of technology and software also saw explosive growth during that time, now touching nearly every aspect of our life from mundane entertainment to critical infrastructure and national security systems. However, on the other hand these numbers should be concerning.

Even though as the image shows, the percentage of vulnerabilities that actually get exploited is small, but as the total number of vulnerabilities grows, as do the total number of vulnerabilities that get exploited.

The challenge is, helping organizations prioritize the vulnerabilities that pose real risk, and not waste incredible amounts of time focusing on non-exploitable/exploited vulnerabilities using narrowly focused approaches like CVSS scores alone, which unfortunately is the industry norm.

Keep this in mind as we continue the discussion and think about the burden the legacy approach to vulnerability management, including by our largest enterprise environments and Federal agencies and department and others such as PCI-DSS poses to developers, engineers and system owners.

The numbers are even more bleak when you look at 2022 alone, which saw over 25,000 known vulnerabilities but .36% or 93 exploited by Malware and .09%, or 23, exploited by threat actors and only .28% or 73 making their way to the CISA Known Exploited Vulnerabilities (KEV).

(An interesting aside is that one of the most frequently cited vendors on the CISA KEV also proclaim themselves to be one of the biggest security vendors in the world by revenue, fireman, meet arsonist - but I digress).

That said, it is useful to remember that vulnerabilities don’t get weaponized and exploited in only a calendar year (CY). As the Qualys report points out, plenty of vulnerabilities published prior to 2022 ended up being weaponized in 2022.

This demonstrates that malicious actors aren’t just focused on new vulnerabilities but also old vulnerabilities yet to be weaponized that they can leverage against organizations. Given the industries track record of patch management and the pervasiveness of vulnerabilities with patches available but unpatched, it isn’t surprising malicious actors are pulling old vulnerabilities out and utilizing them.

For example see below:

Qualys points out that 539 vulnerabilities were newly weaponized in 2022 and of those 118 were older than three years. This demonstrates that malicious actors see value in weaponizing older vulnerabilities because they are well aware that many organizations haven’t addressed these vulnerabilities, despite being more than 36 months old.

Qualys’ report goes on to describe some of the most exploited CVE’s during CY 2022. I won’t be diving into those here but I strongly recommend reading the full report for that context and insight.

However, Qualys’ report highlights some other aspects I wanted to highlight as well. One of which is the asymmetry that exists between organizations ability to patch weaponized vulnerabilities

While all of these insights are interesting, the part I particularly wanted to highlight goes back to the total number of vulnerabilities, using 2022 as an example, compared to actually exploited vulnerabilities, whether via Malware, present on CISA’s Known Exploited Vulnerability list (KEV), by Threat Actors or via Ransomware. All of these share one thing in common. They represent less than 1% of all known vulnerabilities in 2022.

Now keep in mind, as previously mentioned, organizations generally take blanket CVSS approaches to vulnerability prioritization and remediation requirements. Examples such as all Critical findings must be resolved in 7 days, High’s in 30 days etc.

These requirements exist and are pushed onto engineerings and developers regardless of whether the vulnerability is known to be exploited, or has a high probability of being exploited via sources such as the Exploit Prediction Scoring System (EPSS) (which I covered here).

Couple that with the reality of the complex security tooling landscape we have in the DevSecOps and CI/CD paradigm where we have SAST, DAST, IAST/RASP, Container Vulnerability Scanning, Secrets Scanning, SCA/SBOM and the list goes on. Granted, all of these tools and their associated findings don’t utilize CVE’s and CVSS for every finding, but many of them do.

You can begin to get a feel for the mountain of vulnerability data that we (Security) dump onto our Development/Engineering peers with no context related to the systems use within the business, the vulnerabilities exploitability or how to prioritize other than CVSS scores and it is easy to see why security is generally referred to as:

• A blocker

• The office of “no”

• Referred to by Developers as “a soul withering chore”

We largely put Developers in a “guilty until proven innocent” scenario, drowning them with vulnerability data that creates tremendous toil and at the end of the day, poses no real risk to the business, because we’ve provided no context for the findings aside from CVSS severity scores. All the while imposing cognitive load and detracting from other business beneficial activities such as feature development and delivery or worse, addressing vulnerabilities that actually do pose risk to the business/mission.

This isn’t to say that CVE’s and Vulnerabilities are irrelevant and should be neglected, but it is a call for us an industry to mature our approach to Vulnerability Management and provide actionable data to our peers that represents real risk to the business, not just vanity metrics of driving down large swaths of non-exploited and unlikely to ever be exploited CVE’s.

Security, particularly compliance often get a bad reputation for performing what has been dubbed “Security Theater” and there are few better examples than making Developers spend significant portions of time chasing down non-exploited/exploitable vulnerabilities with arbitrary remediation timelines when organizations are already enduring a war for technical talent, including the industry itself experiencing a massive shortfall of cybersecurity talent of 3.4 million people worldwide, per sources such as ISC2.

The uncomfortable reality is that our inefficient and illogical approaches to vulnerability management actually cause more risk than they mitigate, due to imposing cognitive overload on Security and Development teams, distracting from real activities that mitigate organizational risk and are irrational.

If we (Security) want to be seen as a partner to our peers, its time to start acting like it.