It's a well-acknowledged principle in software development that the individuals who write the code should not also be responsible for testing it. For many developers, this responsibility is often unwelcome, and as is the case with effective auditing practices, those who create the work should not be the same people who validate it.
As a result, testing code—be it through usability assessments, language or task-specific evaluations, or comprehensive end-to-end testing—has become a key area of focus for a wave of generative AI startups.
Among these innovative startups is Nova AI, a graduate of the Unusual Academy accelerator that recently completed a $1 million pre-seed funding round. The company aims to surpass its competitors in end-to-end testing tools by challenging conventional Silicon Valley startup norms, according to founder and CEO Zach Smith.
Unlike the typical Y Combinator strategy of starting small, Nova AI targets mid-size to large enterprises that often grapple with complex codebases and immediate testing needs. Smith refrained from naming specific customers but described them as primarily late-stage (Series C or later) venture-backed enterprises in sectors such as e-commerce, fintech, and consumer products, where user experience is critical and downtime can be costly.
Nova AI’s technology automatically generates tests by analyzing customers’ code using generative AI. Its focus is particularly on continuous integration and continuous delivery/deployment (CI/CD) environments, where engineers frequently integrate changes into their production code.
The inspiration for Nova AI stems from Smith's and co-founder Jeffrey Shih's experiences as engineers at major tech companies. Smith, a former Google employee, was involved in cloud-related initiatives that helped clients leverage automation technologies. Shih, with a background at Meta, Unity, and Microsoft, specializes in AI applications involving synthetic data. They have since brought in a third co-founder, AI data scientist Henry Li.
Another guiding principle that Nova AI is sidestepping is the heavy reliance on OpenAI’s leading models. Instead, the company minimizes its use of OpenAI’s Chat GPT-4, ensuring no customer data is provided to OpenAI.
Despite OpenAI's assurance that enterprise customers' data is not used for model training, many companies remain skeptical, Smith notes. "When we're communicating with large enterprises, they express concerns about not wanting their data accessible to OpenAI," he explains.
This sense of distrust isn't limited to engineering teams; OpenAI is also facing multiple lawsuits from entities that object to their work being utilized in AI model training without consent or compensation.
In response, Nova AI predominantly utilizes open-source models such as Llama from Meta and StarCoder, developed by the BigCoder community in collaboration with ServiceNow and Hugging Face. While they haven’t yet implemented Google’s Gemma with customers, testing has shown promising outcomes, according to Smith.
For instance, he points out that OpenAI provides models for vector embeddings, which convert text into numerical values for various language model operations. Nova AI opts for open-source alternatives for embedding, focusing on customer-specific code without feeding data into OpenAI. The company employs OpenAI’s tools solely for certain coding and labeling tasks, ensuring customer data remains protected.
"Instead of relying on OpenAI's embedding models, we deploy our own open-source models. This way, when we analyze files, we're not sending them to OpenAI," Smith clarifies.
By not transmitting customer data to OpenAI, Nova AI alleviates concern among enterprises, while the use of open-source AI models proves to be not only cost-effective but also adequate for specific tasks such as writing tests.
"The open LLM space is demonstrating its potential to outperform GPT-4 and other large providers in specialized applications," he asserts. "We don’t need to create vast models to cater to every possible query; our goal is to write efficient tests, and our models are finely tuned for that purpose."
Moreover, open-source models continue to evolve rapidly, with Meta recently unveiling an upgraded version of Llama that has garnered praise in tech communities, fostering increased interest in alternatives to OpenAI among AI startups.
Ranking
