Cybersecurity researchers have discovered a serious security flaw in the Vanna.AI library that could be exploited to achieve a remote code execution vulnerability using brute force injection techniques.
The vulnerability, tracked as CVE-2024-5565 (CVSS score: 8.1), relates to an exploit in the “ask” function that can be used to force a library to execute arbitrary commands, according to supply chain security firm JFrog said.
Vanna is based on Python machine learning library which allows users to communicate with their SQL database to obtain information by “just asking questions” (aka prompts) that are translated into an equivalent SQL query using a large language model (LLM).
The rapid proliferation of generative artificial intelligence (AI) models in recent years has brought to the fore the risk of exploitation by malicious actors who can weaponize the tools by providing adversarial inputs to bypass their built-in security mechanisms.
One such known class of attack is quick injectionwhich refers to the type Jailbreak AI which may be used to bypass fences set up by LLM providers to prevent the production of offensive, harmful or illegal content, or to execute instructions that defeat the intended purpose of the application.
Such attacks can be indirect, where the system processes data controlled by a third party (such as incoming emails or editable documents) to launch a malicious payload that jailbreaks the AI.
They can also take the form of a so-called a multiple escapes from prison or the multi-turn jailbreak (aka Crescendo), in which the operator “starts an innocuous dialogue and gradually steers the conversation toward an intended, forbidden goal.”
This approach can be extended further to implement another new jailbreak attack known as Skeleton Key.
“This AI jailbreak method works by using a multi-turn (or multi-step) strategy to force the model to ignore its fences,” Mark Russinovich, CTO of Microsoft Azure, said. “By ignoring the fence, the model will not be able to detect malicious or unauthorized requests from others.”
Skeleton Key also differs from Crescendo in that, after a successful jailbreak and system rule change, the model can generate answers to questions that would otherwise be forbidden, regardless of ethical and security risks.
“If the Skeleton Key jailbreak is successful, the model recognizes that it has updated its guidelines and will subsequently follow the instructions to create any content, regardless of how much it violates the original responsible AI guidelines,” Rusinovich said.
“Unlike other jailbreaks like Crescendo, where models have to request tasks indirectly or via coding, Skeleton Key puts models in a mode where the user can request tasks directly. Additionally, the model’s output appears completely unfiltered and shows the model’s degree of knowledge or ability to produce the requested content.”
JFrog’s latest findings – too independently disclosed Tong Liu – show how immediate injections can have serious consequences, especially when they are linked to command execution.
CVE-2024-5565 exploits the fact that Vanna facilitates the generation of text in SQL to create SQL queries that are then executed and graphically presented to the user using the Plotly graphics library.
This is achieved by using an ask function. – for example, vn.ask(“What are the 10 customers with the most sales?”) is one of the main API endpoints that allows you to generate SQL queries against the database.
The above behavior, combined with Plotly’s dynamic code generation, creates a security hole that allows a threat actor to send a specially crafted prompt with a command embedded to be executed on the host system.
“The Vanna library uses a hint function to present rendered results to the user, it is possible to change the hint with a quick injection and run arbitrary Python code instead of the intended rendering code,” JFrog said.
“Specifically, allowing external input to the library’s “ask” method with “render” to True (the default behavior) results in remote code execution.”
After responsible disclosure, Vanna released a tempering guide which warns users that the Plotly integration can be used to generate arbitrary Python code and that users who expose this feature should do so in a sandboxed environment.
“This discovery demonstrates that the risks of widespread use of GenAI/LLM without proper governance and security can have dramatic consequences for organizations,” Shachar Menashe, senior director of security research at JFrog, said in a statement.
“The dangers of rapid injection are not yet well known, but they are easy to perform. Companies should not rely on advance hints as a foolproof protection mechanism and should use more robust mechanisms when LLM interacts with mission-critical resources such as databases or dynamic code generation.”
