Anthropic Researchers Challenge AI Ethics with Persistent Questioning

How to Manipulate an AI with Unintentional Queries: The Emergence of Many-Shot Jailbreaking

Wondering how to prompt an AI to provide answers it typically wouldn't share? Researchers from Anthropic have uncovered a new method known as “many-shot jailbreaking.” This technique reveals how a large language model (LLM) can be nudged into providing sensitive information, such as instructions for building a bomb, by initially asking a series of less harmful questions.

The researchers documented their findings in a detailed paper and communicated these vulnerabilities within the AI community to encourage preventive measures.

This newfound vulnerability stems from the expanded “context window” in the latest generation of LLMs. Context windows refer to the volume of information these models can manage in what resembles short-term memory—previously just a few sentences, these now encompass thousands of words or even entire books.

What Anthropic discovered is that LLMs with larger context windows perform more effectively on various tasks when provided ample examples within the prompt. For instance, when given numerous trivia questions, the model’s responses improve over time. A fact it may initially struggle with could be answered correctly if it appears later in the sequence of prompts.

However, an unexpected side effect of this “in-context learning” is that models also become more adept at responding to inappropriate queries. If a user directly asks the model to create a bomb, it will reject the request. Yet, if the prompt includes it answering 99 benign queries beforehand, the likelihood of compliance increases significantly.

(Update: Initially, I misunderstood the nature of the research, thinking the model engaged with the priming prompts directly. Instead, the questions and answers are integrated into the prompt, clarifying the process and prompting this revision.)

Why Does This Happen?

The underlying processes within an LLM are complex and not fully understood, but a discernible mechanism appears to guide it toward meeting user expectations based on the prompt's content. When asked trivia, the model seems to progressively activate its latent capabilities in that domain as more inquiries are made. Curiously, the same enhancement effect occurs with inappropriate questions, requiring users to provide both prompts and corresponding answers to create this influence.

Anthropic has proactively alerted both peers and competitors about this potential exploit, with hopes of encouraging a collaborative culture where vulnerabilities are shared openly among LLM developers and researchers.

To counteract this vulnerability, the team has discovered that while limiting the context window can mitigate the issue, it also negatively affects the overall performance of the model. Thus, they are exploring techniques to classify and contextualize queries prior to submitting them to the model. However, this raises the challenge of developing new frameworks to secure against evolving attack methods—a common expectation in AI security progress.

Stay informed on the latest in AI developments and implications.