Runware Delivers Rapid AI Inference with Custom Hardware and Advanced Orchestration Techniques

Sometimes, experiencing a demo is the best way to grasp a product's potential. This is certainly true for Runware. By visiting Runware’s website, entering a prompt, and hitting enter, you can witness image generation in action—remarkably, it takes less than a second to produce results.

Runware is an emerging player in the generative AI and AI inference market. The company is designing its own servers and refining the software layer to eliminate bottlenecks and enhance inference speeds for image generation models. They have successfully secured $3 million in funding from notable investors, including Andreessen Horowitz’s Speedrun, LakeStar’s Halo II, and Lunar Ventures.

Rather than reinvent the wheel, Runware aims to optimize its performance. The startup manufactures its own servers equipped with numerous GPUs on a single motherboard. They have developed a custom cooling system and manage their data centers to ensure efficiency.

For running AI models on its servers, Runware has significantly optimized the orchestration layer, utilizing BIOS and operating system tweaks to improve cold start times. The team has crafted specialized algorithms to effectively allocate inference workloads.

The demonstration itself is compelling. Now, the company is eager to leverage its research and development efforts to establish a thriving business. Unlike many GPU hosting services that charge based on GPU usage time, Runware believes in encouraging clients to accelerate their workloads. To this end, Runware presents an image generation API with a cost-per-API-call pricing model, drawing from popular AI frameworks such as Flux and Stable Diffusion.

“If you consider platforms like Together AI, Replicate, and Hugging Face—they sell compute resources based on GPU time,” noted co-founder and CEO Flaviu Radulescu. “When you compare the time we take to generate images against theirs, along with our pricing, you’ll clearly see we are both faster and more cost-effective.”

“It will be nearly impossible for them to replicate this performance,” he continued. “Especially as cloud providers operate in virtualized environments, which introduce additional delays.”

By examining the complete inference pipeline—optimizing both hardware and software—Runware aspires to integrate GPUs from various manufacturers in the near future. This is crucial, as Nvidia currently dominates the GPU market, making its products relatively pricey for startups.

“We currently rely solely on Nvidia GPUs. However, this should be an abstraction at the software level,” Radulescu explained. “Our technology allows for rapid model switching in GPU memory, enabling us to serve multiple customers on the same GPUs efficiently.”

“In contrast to our competitors who load a model into the GPU for a specific task, our software solution facilitates the swift toggling of models within the GPU memory during inference.”

If AMD and other GPU manufacturers develop compatibility layers for standard AI workloads, Runware will be well positioned to create a hybrid cloud solution that incorporates GPUs from multiple vendors. This strategy will undoubtedly help them maintain a competitive edge in AI inference pricing.