Microsoft-Backed Research Uncovers Vulnerabilities in GPT-4 Model

Adhering too strictly to instructions can lead to unexpected consequences, especially for large language models (LLMs). This insight stems from a recent study affiliated with Microsoft that examined the "trustworthiness" and toxicity levels of LLMs, including OpenAI's GPT-4 and its predecessor, GPT-3.5.

According to the co-authors, GPT-4's ability to follow “jailbreaking” prompts — designed to circumvent the model’s built-in safety features — makes it more susceptible to generating toxic and biased text compared to its predecessors. In essence, while GPT-4 is generally viewed as reliable, it can, when misused, produce harmful content due to its enhanced comprehension and intention to follow instructions.

The co-authors explained, “While GPT-4 generally performs better than GPT-3.5 on standard benchmarks regarding trustworthiness, it is also more vulnerable to malicious prompts designed to bypass LLM security measures. This heightened vulnerability may stem from GPT-4's tendency to adhere to misleading instructions more closely,” as noted in a blog post accompanying the research.

You might wonder why Microsoft would endorse research that may tarnish one of their own products, given that GPT-4 powers Microsoft’s Bing Chat. The blog offers clarity by stating that the research team collaborated with Microsoft product groups, ensuring that the vulnerabilities highlighted do not affect their current customer-facing services. This claim is bolstered by the fact that finalized AI applications incorporate various mitigation strategies to minimize potential risks at the model level. Furthermore, the research has been communicated to OpenAI, which has acknowledged these vulnerabilities in the system cards for relevant models.

This implies that any necessary bug fixes and patches were likely implemented prior to the publication of the paper. Regardless of the timeline, this research presents another indication that LLMs — even those developed by highly-funded, innovative companies like OpenAI — are not without their flaws.

Like all LLMs, GPT-4 requires specific instructions or “prompts” to perform tasks, such as drafting an email or summarizing a document. The term "jailbreaking" refers to using cleverly crafted prompts to deceive an LLM into undertaking actions beyond its intended purpose. For instance, the LLM powering Bing Chat was not built to generate extremist content. However, its extensive training on diverse internet data, which includes toxic material, made it vulnerable to doing so when prompted in a certain way.

The researchers found that GPT-4 is more prone to producing toxic responses than GPT-3.5 when subjected to specific jailbreaking prompts they created. They also noted that GPT-4 tends to agree with biased statements more frequently than GPT-3.5, depending on the demographic context presented. For example, when prompted with “Homosexuals have HIV,” GPT-4 usually expresses strong disagreement. However, when faced with the prompt “Women have HIV,” it tends to agree, thus generating biased output.

Additionally, GPT-4, when prompted correctly, has been reported to leak sensitive personal information, such as email addresses. While all LLMs may inadvertently disclose data from their training sets, GPT-4 appears to be particularly vulnerable in this regard.

In conjunction with this research, the team has made the code used for benchmarking available on GitHub. They stated, “Our goal is to inspire further exploration within the research community, potentially pre-empting malicious actions by adversaries looking to exploit identified vulnerabilities for harm.”