After a demanding week of coding, one might expect San Francisco’s tech enthusiasts to escape to the Bay Area's mountains, beaches, or vibrant nightlife. However, the weekend signals the start of AI hackathons.
In recent years, San Francisco has become a hub for AI hackathons. Each Saturday and Sunday, technology professionals gather to discuss the latest advancements in artificial intelligence, network, and—most importantly—transform innovative ideas into functional prototypes. While some hackathons offer cash prizes or cloud credits, the true champions leave with the foundational seeds of a startup.
“There’s no better city in the world to embark on your most ambitious project than San Francisco,” says Alex Reibman, co-founder of Agency AI. “You’ll often find numerous competitions—like hackathons—but they’re less about competing against each other. It’s a unique blend of collaboration and competition.”
Last summer at a San Francisco hackathon, Reibman experimented with creating AI agents capable of web scraping. With the AI boom in full swing in Silicon Valley, agents have become a hot topic. Though the term lacks a precise definition, it broadly refers to AI-driven bots performing tasks autonomously within environments that were not originally intended for automation—essentially offloading repetitive tasks that once required human effort.
However, Reibman and his team quickly faced challenges. “They struggled,” Reibman remarked during an interview. “The agents failed around 30 to 40% of the time, often in surprising ways.”
To address these issues, Reibman’s team developed internal debugging instruments to identify the agents’ shortcomings. Ultimately, while they improved the agent’s performance, the debugging tools themselves garnered attention and won the hackathon.
“I began showcasing the tools at various hackathons and events in San Francisco, and soon, people were eager to access them,” said Reibman. “That was the affirmation I needed: Instead of developing an agent ourselves, we should focus on creating tools that simplify agent construction.”
Thus, Reibman launched Agency with co-founders Adam Silverman and Shawn Qiu, offering tools designed to analyze AI agents' behavior and pinpoint errors. A year later, these tools became Agency’s flagship product, the AgentOps platform, now utilized by thousands of teams monthly, Reibman shared.
Chief Operating Officer Adam Silverman describes AgentOps as “multi-device management for agents,” meticulously monitoring every action of the agent to prevent erratic behavior.
“You need to understand whether your agent will operate as intended and identify the limitations to enforce,” noted Silverman during an interview. “The key is to visually represent the guardrails in place and confirm the agent adheres to them before deploying into production.”
Agency collaborates with AI model developers like Cohere and Mistral to enhance its offerings. This partnership enables users to leverage the AgentOps dashboard for insights into how agents interact with their environment and the associated costs. Agency is model-agnostic, compatible with various AI agent frameworks, and integrates seamlessly with leading tools like Microsoft’s AutoGen, crewAI, and AutoGPT.
In addition to the AgentOps dashboard, Agency provides consulting services (Reibman previously worked at EY) to aid businesses in developing their AI agents. While they chose not to disclose specific clients, they highlighted that hedge funds, consultants, and marketing firms are among the users of their tools.
For instance, Reibman mentioned that Agency developed an AI agent that generates blog posts about client companies. This client now tracks the agent’s performance and costs through the AgentOps dashboard.
As industry giants like OpenAI and Google are poised to enhance their agent offerings soon, AI startups like Agency must navigate these developments and find ways to coexist with them.
“With so many layers in the technology stack, it’s unlikely that any single LLM provider will dominate every aspect,” Reibman explained. “While OpenAI and Anthropic are focused on building agent creators, various supporting layers are essential to ensure a robust production-ready codebase.”
Ranking
