If there was one topic that dominated the cybersecurity industry in 2023 it was definitely AI. This included both the fears and concerns of how AI will be leveraged by malicious actors, as well as how AI can be leveraged by defenders to mitigate risk, protect organizations and innovate and accelerate traditional cybersecurity activities. 

As cyber practitioners scramble to up-skill themselves on the topic of AI security, as their organizations quickly adopt AI tools, platforms, applications and services, luckily there are various resources they can use to learn.

One we are going to discuss today is the OWASP AI Exchange. OWASP has positioned themselves as a go to resource AI security knowledge, including publishing the OWASP LLM Top 10 list in 2023, which documents the top 10 risks for LLM systems and recommendations on how to mitigate those risks. 

The OWASP AI Exchange is an incredible resource, serving as an open source collaborative effort to progress the development and sharing of global AI security standards, regulations and knowledge. It covers AI threats, vulnerabilities and controls. 

So let’s dive in and take a look at some of the core aspects of the OWASP AI Exchange, threats, vulnerabilities and controls it covers.

I also had a chance to interview Rob van der Veer, one of the leaders of the OWASP AI Exchange, and thought leaders more broadly in the AI Security space, contributing to key efforts such as the AI Exchange, OWASP LLM Top 10 and more. You can find that interview here.

AI Exchange Navigator and General Recommendations

Given the complexity of securing the AI landscape, it is great to have a resource to quickly overview the various threats, and controls and how they relate, luckily OWASP AI Exchange provides that with their “Navigator” which covers general controls, controls against runtime threats, controls against development time threats and runtime application security threats. 

OWASP AI Exchange also starts off by making some general recommendations. These include activities such as implementing AI governance, extending security and development practices into data science, and adding oversight of AI based on its specific use cases and nuances. There are many ways AI can and is already being leveraged, and various threats, such as data leaks, poisoning and even attacking the AI supply chain. They are captured in the below image from the AI Exchange.

The OWASP AI Exchange also provides a comprehensive way for organizations to identify relevant threats and accompanying controls. This includes identifying the threats, through activities such as threat modeling, determining responsibility within the organizations to address the threats, as well as external responsibilities such as when leveraging service providers, software and suppliers.

From there, organizations should select controls that mitigate the identified threats and cross-reference those controls to existing and emerging standards. Finally, the organization will be in a position to make an informed decision around risk acceptance and ongoing monitoring and maintenance of the identified and accepted risks.

Much like any other system, AI system development follows a lifecycle, which is why the AI Exchange recommends activities throughout the phases of the AI SDLC. These phases include Secure Design, Development, Deployment and Operations and Maintenance.

These phases are captured well in other frameworks as well, such as NIST’s Secure Software Development Framework (SSDF), which has phases with accompanying tasks and activities to secure system and software development throughout the SDLC. For deep dive into SSDF, you can read my previous article on the topic. I also have an article that dives into leading AppSec maturity models, such as OWASP SAMM and BSIMM, which can be found here.

Given that, let's take a look at some of the general threats as well as those associated with specific phases of the SDLC, such as development and runtime.

General Controls

General controls include activities that are overarching, such as governance. This includes assigning responsibilities for accountability around models and data, as well as risk governance. These efforts are aimed at ensuring that AI initiatives and use aren’t overlooked as part of broader information security management.

Specific general controls include items such as minimizing the data and fields that are unnecessary for the application to avoid potential leaks. It also includes ensuring only the permitted and authorized data is used as part of model training activities or in AI systems and platforms. Additionally, data should have a defined lifecycle and not be retained or accessible longer than necessary to minimize risk.

Methods such as tokenization, masking and encryption can and should be used to protect sensitive information in training datasets and to avoid inadvertent disclosure of sensitive data.

It’s also critical for organizations to implement controls to limit the effects of unwanted behavior, such as impacts on training data, manipulation of AI systems and detecting unwanted behavior and either correcting it or halting the unintended or malicious activities before they can have an impact and notifying respective system maintainers.

Threats Through Use

Threats through use are defined as taking place through normal interaction with AI models and systems such as providing input or receiving output. AI Exchange recommends monitoring use of models and capturing metrics such as input, date, time and user in logs for incident response. These may be improper model functioning, suspicious behavior patterns or malicious inputs. Attackers may also make attempts to abuse inputs through frequency, making controls such as rate limiting API’s. 

Attackers may also look to impact the integrity of model behavior leading to unwanted model outputs, such as failing fraud detection or making decisions that can have safety and security implications. Recommended controls here include items such as detecting odd input, adversarial input and choosing an evasion-robust model design. 

Development-time Threats

In the context of AI systems, OWASP’s AI Exchange discusses development-time threats as the development environment used for data and model engineering outside of the regular applications development scope. This includes activities such as collecting, storing and preparing data and models and protecting against attacks such as data leaks, poisoning and supply chain attacks. 

Specific controls cited include development data protection and using methods such as encrypting data-at-rest, implementing access control to data, including least privileged access and implementing operational controls to protect the security and integrity of stored data. 

Additional controls include development security for the systems involved, this includes the people, processes and technologies involved. Implementing controls such as personnel security for developers and protecting source code and configurations of development environments, as well as their endpoints through mechanisms such as virus scanning and vulnerability management, as in traditional application security practices. Compromises of development endpoints could lead to impacts to development environments and associated training data.

It’s easy to see why other industry trends such as zero trust, DevSecOps and secure software development and supply chain security have significant value to securing AI development and use as well, given the above.

The AI Exchange also makes mention of AI and ML Bills of Materials (BOM)’s, to assist with mitigating supply chain threats. It recommends utilizing MITRE ATLAS’s ML Supply Chain Compromise as a resource to mitigate against provenance and pedigree concerns, and also conducting activities such as verifying signatures and utilizing dependency verification tools. 

Runtime AppSec Threats

The AI Exchange points out that AI systems are ultimately IT systems and can have similar weaknesses and vulnerabilities that aren’t AI specific but impact the IT systems AI is part of. These controls of course are addressed by long standing application security standards and best practices, such as OWASP’s Application Security Verification Standard (ASVS).

That said, AI systems have some unique attack vectors which are addressed as well, such as runtime model poisoning and theft, as well as insecure output handling and direct prompt injection, the later of which was also cited in the OWASP LLM Top 10, claiming the top spot among the threats/risks listed. This is due to the popularity of GenAI and LLM platforms in the last 12-24 months. 

To address some of these AI specific runtime AppSec threats, the AI Exchange recommends controls such as runtime model and input/output integrity to address model poisoning. For runtime model theft, controls such as runtime model confidentiality (e.g. access control, encryption) and model obfuscation, making it difficult for attackers to understand the model in a deployed environment and extract insights to fuel their attacks. 

To address insecure output handling, recommended controls include encoding model output to avoid traditional injection attacks. 

Prompt injection attacks can be particularly nefarious for LLM systems, aiming to craft inputs to cause the LLM to unknowingly execute the attackers objectives either via direct or indirect prompt injections. These methods can be used to get the LLM to disclose sensitive data (e.g. personal data, intellectual property etc). 

To deal with direct prompt injection, again the OWASP LLM Top 10 is cited, and key recommendations to prevent its occurrence include enforcing privileged control for LLM access to backend systems, segregating external content from user prompts and establishing trust boundaries between the LLM and external sources.

Lastly, the AI Exchange discusses the risk of leaking sensitive input data at runtime. Think GenAI prompts being disclosed to a party they shouldn’t be, such as through an attack of attacker-in-the-middle scenario. The GenAI prompts may contain sensitive data, such as company secrets or personal information that attackers may want to capture. Controls here include protecting the transport and storage of model parameters through techniques such as access control, encryption and minimizing the retention of ingested prompts. 

Conclusion

As the industry continues the journey towards the adoption and exploration of AI capabilities, it is critical that the security community continue to learn how to secure AI systems and their use. This includes internally developed applications and systems with AI capabilities as well as organizational interaction with external AI platforms and vendors as well. 

The OWASP AI Exchange is an excellent freely open resource for practitioners to dig into to better understand both the risks and potential attack vectors as well as recommended controls and mitigations to address AI specific risks. As OWASP AI Exchange pioneer and AI security leader Rob van der Veer stated recently, a big part of AI security is the work of data scientists and AI security standards and guidelines such as the AI Exchange can help. 

Security professionals should primarily focus on the blue and green controls listed in the OWASP AI Exchange Navigator, which includes often incorporating longstanding AppSec and cybersecurity controls and techniques into systems utilizing AI.