As artificial intelligence (AI) becomes more integral in businesses, a new breed of vulnerabilities has quickly emerged, lurking in the shadows of open-source libraries and AI models––some of the most popular and widely used today.
At Oligo, we have named these Shadow Vulnerabilities, a topic we have written and spoken about extensively. Shadow Vulnerabilities are not your typical software vulnerability – they more often than not lack a Common Vulnerabilities and Exposures (CVE) identifier and are not picked up by traditional vulnerability scanning tools. In fact, these flaws are often not known to the open-source software’s maintainers, and are sometimes even considered functionality that makes open-source components "vulnerable by design.”
Here, we are going to dive into how Shadow Vulnerabilities impact open-source AI libraries and models, including what these vulnerabilities look like in practice, their exploit & traversal paths, their impact on AI security, and the limitations of CVEs in addressing these threats.
Why Shadow Vulnerabilities Thrive in AI: The CVE Blind Spot
The rapid adoption of AI technology, coupled with a "velocity over security" culture among practitioners, has created a fertile ground for these vulnerabilities. AI models, essentially complex code, are a goldmine for attackers, and the breakneck pace of development often leaves security as an afterthought. By definition, if you’re prioritizing velocity, you are not prioritizing security. As a result, the allure of rapid innovation often overshadows the need for robust security practices, leaving AI systems exposed to hidden dangers.
CVEs provide the backbone for the way modern security tooling works, including scanners that validate software versus known lists of vulnerabilities (i.e. CVEs) to verify whether your software is secure. However, these CVE-less vulnerabilities are frequently known to the maintainers of the software but are left unpatched, leaving the software intentionally "vulnerable-by-design." This issue is compounded by silent misconfigurations that, although documented, are frequently overlooked by developers who may not read the full documentation.
CVEs, while valuable for tracking known vulnerabilities, are ineffective against shadow vulnerabilities, as these vulnerabilities often lack a CVE or are disputed, making them invisible to traditional security tools. This blind spot creates a false sense of security, as organizations may believe their systems are protected when, in fact, they are exposed to these threats that lurk in the shadows.
The Multifaceted Nature of Shadow Vulnerabilities
Shadow vulnerabilities are particularly nefarious, as they can manifest in various ways, each posing unique challenges to AI security. Let’s take a look at some of the most common ways shadow vulnerabilities manifest:
- Silent Misconfigurations: These vulnerabilities often hide in plain sight, lurking within the documentation that few users bother to read. A seemingly innocuous configuration setting can inadvertently open the door to exploitation, allowing attackers to compromise AI systems.
- Intentional "Features": Some vulnerabilities are intentionally incorporated into the design of libraries, with the responsibility for safe usage shifted onto the user. This approach, while potentially offering flexibility, can be a double-edged sword, as it assumes a high level of security awareness and expertise among users.
- Exploitation of Legitimate Functionality: Attackers can cleverly exploit legitimate features of AI libraries and frameworks to achieve malicious ends. For instance, the SnakeYAML default constructor, designed for convenience, can be manipulated to execute arbitrary code, leading to remote code execution (RCE) attacks.
Real-World Examples: A Glimpse into the Shadows
We recently did a talk at CloudNativeSecurityCon that sheds light on just one of these popular Shadow Vulnerabilities - ShadowRay - that focuses on a shadow vulnerability in a popular open source framework called Ray. This, alongside several real-world examples that we have uncovered through extensive research, underscore the severity and prevalence of shadow vulnerabilities.
Below, we’ll dive into some other real-world examples of Shadow Vulnerabilities in AI and ML libraries so that you can get an idea of how they manifest, take the necessary measures to get protected, and learn to identify others before they’re exploited.
PyTorch
ShellTorch
The Oligo Security research team discovered critical vulnerabilities in TorchServe, and did the responsible disclosure work to ensure this vulnerability receives the necessary CVE identifier so it is no longer lurking in the shadows. This resulted in high-severity CVE-2023-43654, which is able to lead to RCE exposing thousands of instances, including those belonging to major global organizations. These vulnerabilities allow unauthorized access and the insertion of malicious AI models, potentially leading to a full server takeover. Despite the severity, some issues remained unresolved, leaving systems at risk. Mitigation steps and tools to minimize exposure are provided, with updates from Google, Amazon, and Meta indicating actions taken to address the risks.
This chain of vulnerabilities in TorchServe, a tool for serving PyTorch models, exposed a critical weakness in AI infrastructure. Attackers could exploit these vulnerabilities to achieve remote code execution, steal valuable models, and even poison them with malicious data, compromising the integrity and reliability of AI systems.
Read about the details of this shadow vulnerability here: Shelltorch Explained: Multiple Vulnerabilities in Pytorch Model Server (Torchserve) (CVSS 9.9, CVSS 9.8) Walkthrough.
Keras
Deserialization & Downgrade Attacks
TensorFlow and its built-in Keras framework, widely used in AI, contain a critical security flaw involving the unsafe serialization of models through Lambda layers. Despite a recent fix that introduced a "safe_mode" to prevent code execution, the vulnerability persists in older H5 file formats, leaving systems exposed to remote code execution (RCE). This flaw, unaddressed in older formats, makes TensorFlow and related tools like TFLite Converter and Apache Beam vulnerable by design. Traditional scanning tools miss these "shadow vulnerabilities," highlighting the need for runtime detection and proactive measures to protect AI systems.
The insecure deserialization mechanism in Keras, a popular deep learning library, allowed attackers to execute arbitrary code by crafting malicious model files. This vulnerability highlighted the risks associated with loading untrusted models into AI systems.
The downgrade attack exploits a vulnerability in the TensorFlow Lite (TFLite) converter, enabling attackers to downgrade Keras models to older, potentially vulnerable versions. This attack demonstrates the importance of version control and patching in AI security.
Learn more about this detailed shadow vulnerability in this post TensorFlow Keras Downgrade Attack: CVE-2024-3660 Bypass.
Ray
ShadowRay
The Oligo research team uncovered an ongoing attack campaign exploiting a critical vulnerability in Ray, a popular open-source AI framework. This vulnerability, which remains unpatched due to a dispute, has left thousands of organizations' AI infrastructure exposed, enabling attackers to hijack computing resources and steal sensitive data. Prior to coming to light, the flaw had been actively exploited for seven months, impacting industries such as education, cryptocurrency, and biopharma. Organizations using Ray should urgently assess their environments for exposure and investigate any unusual activity.
By exploiting misconfigurations in Ray clusters, attackers could gain unauthorized access to sensitive data, including API tokens, cloud credentials, and even private SSH keys. This incident served as a stark reminder of the potential consequences of shadow vulnerabilities in real-world AI deployments.
Read about ShadowRay: First Known Attack Campaign Targeting AI Workloads Actively Exploited In The Wild to discover the full attack scenario and mitigation recommendations.
Jinja
Jinja2 Template Injection
Found in Jinja2, a popular templating engine, this shadow vuln allowed attackers to inject malicious code into AI applications. Jinja2 is often used in web applications and AI frameworks to dynamically generate content based on user input. However, if not properly secured, it can become a vector for attackers to inject and execute arbitrary code on the server.
The issue stemmed from the way Jinja2 processes template expressions. When user input is incorporated into templates without adequate validation or sanitization, it can be manipulated to execute malicious code within the context of the application. This type of attack is known as template injection, and it can lead to severe consequences, including RCE, unauthorized access, data breaches, and more. This attack highlighted the need for robust input validation and sanitization in AI systems.
Protecting Against the Unknown Unknowns
In the context of AI applications, the impact of these vulnerabilities is particularly concerning. AI systems often handle sensitive data and perform critical operations, making them high-value targets for attackers. The ability to inject malicious code or perform remote execution into an AI application could compromise the integrity of the AI models, alter their behavior, steal intellectual property, or expose confidential data.
To effectively mitigate the risks posed by shadow vulnerabilities, a multi-layered approach is essential. Runtime solutions that continuously monitor AI systems for anomalous behavior can detect and respond to attacks, even without a CVE. Sandboxing open-source libraries can create an isolated environment, limiting the potential damage from exploits. Additionally, fostering a security-conscious culture among AI practitioners and prioritizing security throughout the development lifecycle can significantly reduce the risk of introducing shadow vulnerabilities.
Shadow vulnerabilities represent a significant and evolving challenge in AI security. By understanding their nature, recognizing their potential impact, and adopting proactive defense strategies, we can better protect AI systems from these hidden threats. The key lies in acknowledging the limitations of CVEs and embracing a comprehensive approach that combines runtime monitoring, sandboxing, and a security-first mindset. As AI continues to advance, so too must our efforts to secure it from the shadows.
