The Media AI Impact Tour recently visited San Francisco, focusing on the real-world applications of generative AI and the intricacies of deploying such initiatives. Matt Marshall, CEO of a media, led the discussion with industry leaders including Ed Anuff, Chief Product Officer at DataStax; Nicole Kaufman, Chief Transformation Officer at Genesis Health; and Tisson Matthew, CEO of Skypoint.
This conversation is pivotal as enterprises transition from exploratory phases of generative AI towards mature implementations. Companies are moving beyond simple experiments with tools like ChatGPT and are now tackling the critical question: how do we harness this technology and integrate it with our essential business data for production use?
“We're witnessing the emergence of an AI maturity model,” Anuff noted. “Organizations are shifting from one-off projects aimed at quick wins to critical AI initiatives led by business champions focused on high-impact, high-visibility deployments. These efforts might take longer to develop, but their transformative potential is significant.”
Generative AI can be applied across various use cases, from back-office operations to public-facing websites and mobile applications. While companies may still reference terms like "chatbots" or "conversational interfaces," they ultimately aim to create knowledge applications that allow for interactive data retrieval in appropriate contexts. The primary decision is whether to develop these solutions in-house or to leverage off-the-shelf products.
Pre-Production Considerations
For customer support or financial analysis, many organizations seek to utilize generative AI to create applications that derive insights from internal data. Anuff explained, “Depending on your data volume and the specific requirements of your customized interface, off-the-shelf solutions can be quite effective. Companies like Amazon offer platforms that enable you to upload documents for immediate chatbot responses, allowing for a quick and straightforward implementation.”
However, as enterprises expand their focus from smaller, back-office applications to critical use cases tied to core business functions—especially those that are externally facing—basic solutions may not suffice. Anuff cited healthcare applications that require real-time connections to data sources for accurate responses based on updated patient information. He also highlighted AI agents in Asia-Pac financial institutions that provide direct access to chat-based financial planning informed by live data from financial statements.
“That’s a custom-built AI RAG (retrieval augmented generation) application linked to your core data,” he asserted. “High-profile retailers like Home Depot or Best Buy invest in dedicated web engineering teams to develop tailored experiences that align with their brand and business operations.”
Calculating Readiness and Cost
As organizations progress past the ideation stage, they encounter two main challenges: relevancy and cost.
“Relevancy is a new key metric for many in the data space,” Anuff explained. “Organizations need to assess how suitable their AI-generated responses are. Relevancy issues often require a reevaluation of your entire data architecture.”
This, in turn, influences the second challenge—cost. Finding a way to deliver relevant, clean results is expensive, and organizations must further evaluate the costs associated with scaling into production.
“Discussions around these issues provide a realistic gauge of how prepared teams are for production,” he noted. “If relevancy continues to be a hurdle, it indicates that teams have moved past initial architecture challenges but face new complexities with production costs, which often go hand in hand.”
Hallucinations, Data, and the Importance of RAG
The term “hallucinations” is commonly used when AI responses appear incorrect. While it serves as a useful colloquial term, not every misguided response is a hallucination; some result from errors in the training data. Hallucinations occur when a large language model (LLM) extrapolates beyond its training and generates vague or misleading content. Anuff highlighted that there are effective solutions to these issues, particularly through RAG.
RAG combines knowledge retrieval with generative AI, allowing a system to process and consolidate data from an internal knowledge base, providing context-aware responses in natural language instead of merely summarizing information.
“A large language model excels in two areas,” Anuff stated. “First, it understands the nuances of language. Second, it serves as a knowledge repository. A programmer can control how much of its intrinsic knowledge is utilized by imposing limits on responses, known as grounding. This significantly reduces the risk of hallucinations by keeping the model focused on relevant data.”
Additionally, Anuff emphasized that RAG is vital for securely and accurately integrating real-time company data into the model during inference.
“While other methods of data integration exist, they often lack safety, real-time capability, and security,” he asserted. “For this reason, using a model-database coupling—regardless of whether we label it RAG—will remain prevalent in the industry moving forward.”
Ranking
