Over 40% of marketing, sales, and customer service organizations have embraced generative AI, making it the second most adopted technology after IT and cybersecurity. Among generative AI technologies, conversational AI is poised for rapid expansion within these sectors, as it effectively addresses communication gaps between businesses and customers.
Despite its potential, many marketing leaders are unsure how to initiate the implementation of this technology. They grapple with choosing the right large language model (LLM), deciding between open source and closed source options, and managing concerns about costs associated with this emerging technology.
While companies can purchase off-the-shelf conversational AI tools, those looking to integrate it as a core part of their business have the option to develop custom solutions in-house.
To demystify this process, I’d like to share insights from our internal research on selecting the ideal LLM for building conversational AI. We evaluated various LLM providers and estimated the costs associated with different models based on usage expectations.
For our comparison, we focused on two leading LLMs: GPT-4o (OpenAI) and Llama 3 (Meta). These models represent a closed source and an open source option, respectively, and they stand out as some of the highest quality LLMs available.
Calculating LLM Costs for Conversational AI
When choosing an LLM, two primary financial considerations emerge: setup costs and processing costs.
- Setup Costs: This includes all expenses necessary to launch the LLM, covering development and operational costs.
- Processing Costs: These are incurred per conversation once your AI is live.
The cost-to-value ratio for setup heavily depends on the LLM's intended use and expected frequency. If speed is a priority, a model like GPT-4o, which requires minimal setup, may be preferred. While Llama 3 demands more time for setup, it offers greater control and scalability for businesses managing a high volume of client interactions.
Foundational Costs of Each LLM
Before diving into operational costs, it’s essential to assess the foundational costs involved.
GPT-4o is a closed-source model operated via a straightforward API call, necessitating minimal setup time and cost.
Llama 3, as an open-source model, must be hosted on your own infrastructure or cloud provider, which introduces hosting costs. Most providers charge based on compute hours, while AWS's Bedrock charges per token processed, potentially making it a cost-effective option for lower usage volumes.
Operating Llama 3 requires more investment in time and resources for server setup, maintenance, and monitoring. Key factors influencing your cost-to-value ratio include the time to deploy, usage levels, and the necessity for control over data.
Costs per Conversation for Major LLMs
Now, let’s examine the costs associated with each conversation unit.
Using a heuristic of 1,000 words equating to 7,515 characters and approximately 1,870 tokens, we estimated an average consumer conversation comprises 16 messages, totaling around 30,390 tokens.
- For GPT-4o:
- Input Tokens: 29,920 at $0.005 per 1,000 tokens = $0.1496
- Output Tokens: 470 at $0.015 per 1,000 tokens = $0.00705
- Total Cost per Conversation: approximately $0.15665
- For Llama 3 on AWS Bedrock:
- Input Tokens: 29,920 at $0.00265 per 1,000 tokens = $0.07929
- Output Tokens: 470 at $0.00350 per 1,000 tokens = $0.00165
- Total Cost per Conversation: approximately $0.08093
In summary, once fully set up, Llama 3 conversations cost nearly 50% less than those generated by GPT-4o, not accounting for any additional server costs involved with Llama 3.
This analysis captures only a fraction of the overall costs tied to each LLM, as many variables will influence your specific implementation, including the complexity of prompts used.
For businesses considering conversational AI as a key offering, the investment in developing an in-house solution may not justify the effort compared to the quality available from off-the-shelf products.
Ultimately, whichever solution you choose, implementing conversational AI can greatly enhance operations. Always stay aligned with your company’s goals and your customers' needs.
Sam Oliver is a Scottish tech entrepreneur and serial startup founder.
Ranking
