It doesn't take much for Generative AI (GenAI) to spread misinformation and inaccuracies. Just last week, Microsoft and Google’s chatbots prematurely declared a Super Bowl winner before the event even took place. The real concern arises when GenAI's fabrications become harmful— endorsing torture, perpetuating ethnic and racial stereotypes, and promoting conspiracy theories.
A growing number of companies, from established players like Nvidia and Salesforce to emerging startups such as CalypsoAI, offer solutions that they claim can eliminate harmful content generated by GenAI. However, these solutions often operate as black boxes; without independent testing, it’s challenging to gauge how they stack up against each other—and whether they genuinely fulfill their promises.
Shreya Rajpal recognized this pressing issue and founded Guardrails AI to address it. “Most organizations face similar challenges regarding the responsible deployment of AI applications and struggle to identify the most effective solutions,” Rajpal shared in an email interview. “They frequently find themselves reinventing the wheel when it comes to managing the essential risks that concern them.”
Surveys support Rajpal's assertion, indicating that complexity—and subsequently, risk—hinders many organizations from fully integrating GenAI. A recent poll from Intel’s Cnvrg.io revealed that around 25% of companies implementing GenAI applications shared concerns about compliance, privacy, reliability, high implementation costs, and a lack of technical expertise. In a separate survey by Riskonnect, over half of executives expressed worries about employees making decisions based on inaccurate information from GenAI tools.
Rajpal, who previously worked at Drive.ai before its acquisition by Apple, co-founded Guardrails along with Diego Oppenheimer, Safeer Mohiuddin, and Zayd Simjee. Oppenheimer led Algorithmia, a platform for machine learning operations, while Mohiuddin and Simjee held technology and engineering leadership roles at AWS.
In many ways, Guardrails offers solutions similar to existing market options. The startup's platform serves as a wrapper around GenAI models, including both open-source and proprietary text generation models like OpenAI's GPT-4, aiming to enhance their reliability and security.
Where Guardrails stands out is its open-source business model—its codebase is available on GitHub, making it free for users—and its crowdsourced approach. Through a marketplace known as the Guardrails Hub, developers can contribute modular components called “validators” that evaluate GenAI models against specific behavioral, compliance, and performance metrics. These validators can be deployed, repurposed, and reused by other developers and customers, forming the building blocks for customized GenAI moderation solutions.
“With the Hub, we aim to foster an open forum for sharing knowledge and discovering effective strategies to promote AI adoption while constructing a set of reusable guardrails any organization can implement,” Rajpal explained.
The validators in the Guardrails Hub range from straightforward rule-based checks to advanced algorithms designed to detect and mitigate issues within models. Currently, about 50 validators span numerous functions, including detecting hallucinations and policy violations, along with filtering proprietary information and insecure code.
“Most companies typically perform broad, one-size-fits-all checks for profanity and personally identifiable information,” Rajpal noted. “Yet, there’s no universal standard for what constitutes acceptable use for specific organizations. Unique organizational risks must be monitored—different organizations have varying communications policies. With the Hub, we enable users to implement our solutions out of the box or as a starting point to customize to meet their specific needs.”
While the concept of a hub for model guardrails is intriguing, I wonder whether developers will actively contribute to a relatively new platform without any compensation. Rajpal remains optimistic, believing that developers will be driven by recognition and a desire to collaboratively make GenAI safer.
“The Hub enables developers to understand the risks other enterprises face and the guardrails they implement to mitigate those risks,” she added. “The validators provide an open-source solution that organizations can tailor to their specific use cases.”
Guardrails AI, which currently offers its services at no cost, recently secured $7.5 million in seed funding led by Zetta Venture Partners, with participation from Factory, Pear VC, Bloomberg Beta, GitHub Fund, and noted AI expert Ian Goodfellow. Rajpal stated that the funding will help expand Guardrails’ six-person team and support additional open-source projects.
“We engage with various stakeholders—large enterprises, small startups, and individual developers—who are hampered in launching GenAI applications due to a lack of assurance and necessary risk mitigation,” she emphasized. “This is a novel challenge that has emerged at this scale due to the rise of ChatGPT and foundational models. We aspire to be at the forefront of resolving this issue.”
Ranking
