In the era of generative AI, ensuring the safety of large language models (LLMs) is as critical as their performance across various tasks. Teams worldwide are increasingly recognizing this need and enhancing their testing and evaluation processes to identify and resolve issues that could result in poor user experiences, lost opportunities, or regulatory penalties.
With the rapid evolution of both open and closed-source models, determining the safest LLM to use can be challenging. Enkrypt offers a solution with its LLM Safety Leaderboard. This Boston-based startup specializes in providing a control layer for the safe deployment of generative AI and has ranked LLMs based on their vulnerability to safety and reliability risks.
The leaderboard features numerous high-performing language models, including the GPT and Claude families. It provides valuable insights into the risk factors that are essential for selecting safe and reliable LLMs and optimizing their implementation.
Understanding Enkrypt’s LLM Safety Leaderboard
When enterprises deploy LLMs in applications like chatbots, they conduct ongoing internal tests to identify safety risks such as jailbreaks and biased outputs. Even minor errors can lead to significant issues, including data leaks or biased responses, as demonstrated by the Google Gemini chatbot incident. These risks can be even more pronounced in regulated sectors like fintech and healthcare.
Founded in 2023, Enkrypt is addressing these challenges with Sentry, an extensive solution that uncovers vulnerabilities in generative AI applications and implements automated guardrails to mitigate them. The LLM Safety Leaderboard is the next step in this initiative, offering insights that help teams select the safest model from the outset.
The leaderboard, developed through rigorous testing across diverse scenarios, assesses up to 36 LLMs—both open and closed-source—based on various safety and security metrics. It evaluates the model's capacity to avoid generating harmful, biased, or inappropriate content and its ability to thwart malware or prompt injection attacks.
Who Holds the Title for Safest LLM?
As of May 8, Enkrypt’s leaderboard ranks OpenAI’s GPT-4-Turbo as the safest LLM, boasting the lowest risk score of 15.23. This model effectively defends against jailbreak attacks and produces toxic outputs just 0.86% of the time. However, it does face issues with bias and malware, experiencing impacts 38.27% and 21.78% of the time, respectively.
Meta's Llama2 and Llama 3 models follow closely, with risk scores ranging from 23.09 to 35.69. Anthropic’s Claude 3 Haiku ranks 10th with a risk score of 34.83, exhibiting decent performance across tests but yielding biased responses over 90% of the time.
At the bottom of the leaderboard are Saul Instruct-V1 and Microsoft’s newly announced Phi3-Mini-4K models, with risk scores of 60.44 and 54.16, respectively. Mixtral 8X22B and Snowflake Arctic also receive low rankings of 28 and 27.
It is worth noting that this list is subject to change as models improve and new ones emerge. Enkrypt plans to regularly update the leaderboard to reflect these developments.
“Our leaderboard will be updated on Day Zero following new model launches, and weekly for model updates. As AI safety research progresses and new methods are developed, the leaderboard will consistently showcase the latest findings. This ensures its relevance and authority as a resource,” stated Sahi Agarwal, co-founder of Enkrypt.
Agarwal envisions this evolving list as a valuable tool for enterprise teams to explore the strengths and weaknesses of popular LLMs—whether in mitigating bias or preventing prompt injections—and to make informed decisions based on their specific use cases.
“Integrating our leaderboard into AI strategy enhances technological capabilities while maintaining ethical standards, fostering a competitive advantage and building trust. The risk/safety/governance team can leverage the Leaderboard to identify which models are safe for use by product and engineering teams. Currently, they lack comprehensive safety information, relying only on public performance metrics. The leaderboard, along with red team assessment reports, provides essential safety recommendations for model deployment,” he added.
Ranking
