OpenAI and Anthropic Collaborate to Share AI Models with the U.S. AI Safety Institute for Enhanced Safety and Oversight

OpenAI and Anthropic have reached a significant agreement to share their AI models—both prior to and after their public release—with the newly established US AI Safety Institute. Formed by an executive order from President Biden in 2023, this agency will provide crucial safety feedback to improve the companies' models. OpenAI CEO Sam Altman hinted at this collaboration earlier this month.

While the US AI Safety Institute has not mentioned other companies involved with AI, a Google spokesperson indicated that the company is in discussions with the agency and will provide further details in due time. Notably, Google has begun rolling out updates to its chatbot and image generator models under the Gemini program.

"Safety is essential to advancing breakthrough technological innovation. With these agreements, we look forward to technical collaborations with Anthropic and OpenAI to enhance AI safety," stated Elizabeth Kelly, director of the US AI Safety Institute. "These agreements represent just the beginning of our efforts to responsibly guide the future of AI."

The US AI Safety Institute operates within the National Institute of Standards and Technology (NIST) and is responsible for creating guidelines, benchmark tests, and best practices for evaluating potentially hazardous AI systems. Vice President Kamala Harris emphasized the dual nature of AI's impact in late 2023, highlighting its potential for both significant advancements and serious risks, including AI-driven cyber-attacks and bioweapons.

This groundbreaking agreement is formalized through a non-binding Memorandum of Understanding, allowing the agency to access each company's major new models both before and after public release. The US AI Safety Institute characterizes these agreements as collaborative research efforts focused on assessing capabilities and ensuring safety. Furthermore, it will collaborate with the UK AI Safety Institute.

As both federal and state regulators aim to establish AI safety measures, the California state assembly recently passed an AI safety bill (SB 10147). This legislation mandates safety testing for AI models costing over $100 million to develop or requiring substantial computing power. It also requires AI developers to implement kill switches to shut down models that become “unwieldy or uncontrollable.”

Unlike the voluntary agreement with the federal government, the California bill includes enforceable provisions, allowing the state’s attorney general to take legal action against non-compliant AI developers, particularly during critical threat events. The bill awaits one final processing vote and the signature of Governor Gavin Newsom, who has until September 30 to approve it.