Arthur Launches Open Source Tool to Assist Companies in Selecting the Ideal LLM for Their Needs

Arthur, a startup specializing in machine learning monitoring, has capitalized on the growing interest in generative AI this year. The company is introducing Arthur Bench, an open-source tool designed to help organizations identify the most suitable large language models (LLMs) for their specific datasets.

Adam Wenchel, CEO and co-founder of Arthur, notes that the surge in generative AI and LLMs has prompted the company to focus intensively on product development.

“Even with the rapid rise of platforms like ChatGPT, many companies still lack a systematic approach to evaluate the effectiveness of various models. This gap is precisely what led to the creation of Arthur Bench,” Wenchel explained.

Arthur Bench addresses a crucial challenge faced by numerous clients: with so many model options available, how does one determine which is best suited for their unique application? “This tool enables users to rigorously assess performance across multiple models and helps you understand which prompts work most effectively with specific LLMs,” Wenchel stated.

The platform provides a comprehensive suite of tools to methodically test model performance, allowing users to explore the effectiveness of various prompts tailored to their applications. “You can evaluate up to 100 different prompts, comparing how models like Anthropic and OpenAI respond to the queries your users will most likely issue,” Wenchel added. This capability allows businesses to scale their testing efforts, ultimately facilitating more informed decisions about which model aligns best with their needs.

Arthur Bench is now available as an open-source tool, with a forthcoming SaaS version for customers seeking a hassle-free experience in managing their testing requirements or those with larger datasets who prefer a paid option. For the time being, Wenchel indicated that the company's primary focus will be on enhancing the open-source project.

In addition to this launch, Arthur recently unveiled Arthur Shield in May, an LLM firewall designed to detect model hallucinations while safeguarding against harmful content and the unauthorized sharing of private data.