Yahoo Separates Vespa, Its Search Technology, as a Standalone Company

Yahoo has announced the spin-off of Vespa, its renowned big data serving engine, into an independent venture.

Jon Bratseth, previously a VP architect in Yahoo’s big data and AI division and a key contributor to Vespa, has been appointed as the new CEO.

Yahoo plans to continue investing in Vespa, maintaining its position as the largest customer post-spin-off. Additionally, Yahoo will retain an ownership stake in Vespa and hold a seat on its board of directors.

Vespa was initially developed by Yahoo in 2005 following the acquisition of Overture, a paid search service that included the Norwegian search engine AlltheWeb.com. In collaboration with its e-commerce division, the team behind AlltheWeb, comprising around 30 members, transformed their search technology into a versatile tool for Yahoo developers to analyze large-scale datasets in real time.

Over the following decade, Yahoo enhanced Vespa to manage a wider variety of inputs, personalize content based on user engagement, and integrate machine learning capabilities. In 2017, Yahoo took a significant step by open sourcing Vespa, aiming to build developer support for the software and cultivate an ecosystem both internally and externally.

This was one of Yahoo's most significant open-source software initiatives since the release of Hadoop in 2006. The team in Norway undertook an extensive rewrite of the Vespa codebase, which proved beneficial. Today, Vespa powers search functionalities and recommendation systems on Yahoo’s platforms, including Yahoo Sports, Yahoo Finance, and Yahoo News, while also facilitating ad targeting across Yahoo's advertising network.

Yahoo reports that approximately 150 applications utilize Vespa, collectively serving a user base of 1 billion people and processing an impressive 800,000 queries per second.

Outside of Yahoo, thousands of brands—including Spotify, OkCupid, and Wix—leverage either the open-source version of Vespa or the fully managed service called Vespa Cloud, resulting in over 10 million downloads of the open-source package.

While alternatives like Solr and Elasticsearch exist, Yahoo asserts that Vespa delivers "several steps further" in capabilities. “The key differentiator stems from our vision to meet the comprehensive needs of applications that integrate data and AI at scale online,” Bratseth explained via email. “Our experienced team is dedicated to advancing Vespa towards this vision, fostering a culture of continuous improvement, and we are still outpacing our competitors.”

So, why is Vespa becoming its own company? Bratseth points to growing demand. “An increasing number of companies are interested in deploying applications through our cloud service, and becoming a standalone entity will enable us to enhance this functionality,” he said.

I can't help but wonder if Yahoo's ambitions to enter the public markets played a role in this decision. However, neither Bratseth nor Lara Davis, Yahoo’s chief strategy officer, confirmed or denied this speculation. (I attempted to reach Yahoo CEO Jim Lanzone for his insights, but a spokesperson informed me he was currently traveling.)

“Since 2017, Vespa has empowered clients to execute various AI-driven functions, ranging from searching vast document databases within global organizations to enhancing data-driven online advertising and supporting scalable AI-based language applications,” Davis noted. “We recognize Vespa as the leading technology for large-scale deployment, making it advantageous for Yahoo to remain both an investor and customer, actively supporting their growth. As part of this transition, we are migrating critical proprietary technology to Vespa, continuing our support during their journey to independence.”

Lanzone echoed Davis's sentiments in a prepared statement: “For many years, Vespa has been essential to Yahoo's AI and machine learning capabilities across all our properties. By remaining Vespa’s largest customer and key investor, we will leverage its offerings while creating new business opportunities for other companies to utilize this technology as a standalone entity.”