Reducing inferencing costs is becoming a critical objective for businesses leveraging large language models. Nvidia has introduced a hardware-based solution with its new H200 chips, which promise to cut costs by half. In parallel, the team behind the Vicuna AI model has developed an innovative software-based method designed to further decrease inferencing costs and reduce latency. This technique, known as lookahead decoding, focuses on minimizing the number of decoding steps required to generate responses, resulting in both lower operational costs and faster processing times.
Developed by LMSYS Org—an open research group founded by academics—lookahead decoding aims to optimize GPU performance by addressing the inefficiencies seen in models like GPT-4 and Llama 2, which utilize autoregressive decoding. By implementing this new approach, users can streamline the decoding process, allowing the model to predict multiple tokens simultaneously instead of generating them one at a time. This innovation significantly reduces generation time while enhancing model efficiency.
According to a blog post from LMSYS Org, “Lookahead decoding provides a substantial reduction in latency, ranging from 1.5x to 2.3x, with negligible computational overhead. More importantly, it allows for a trade-off between computational expenses and reduced latency, though this advantage does come with diminishing returns.”
The lookahead decoding process resembles preparing layers of a cake. Traditionally, the Jacobi iteration method has been a preferred approach for non-linear systems, enabling parallel token generation without a draft model. However, LMSYS Org argues that using Jacobi iterations often leads to limitations, as achieving successful simultaneous decoding and accurate positioning of multiple tokens proves challenging.
In contrast, lookahead decoding creates new tokens informed by historical values from previous iterations. This allows the model to achieve parallel decoding while verifying promising n-grams from the cache. By accepting an n-gram, the model can advance multiple tokens in a single step, significantly expediting the decoding process.
To validate this new approach, Vicuna developers conducted tests using two Llama models—LLaMA-2-Chat and CodeLLaMA. They evaluated versions of each model with parameter sets of 7B, 13B, and 33B on a single Nvidia A100 GPU, alongside a 70 billion version tested on two A100 GPUs. The results were promising; employing lookahead decoding notably improved inferencing speeds across various benchmarks including MT-Bench, HumanEval, and GSM8K. For instance, LLaMA-Chat recorded a 1.5x speedup on MT-Bench, while CodeLLaMA achieved a 2x latency reduction on HumanEval. Moreover, CodeLLama-Instruct solved math problems from GSM8K with a significant 1.8x latency reduction.
Developers interested in implementing lookahead decoding can access this innovative method through the LMSYS Org's GitHub page. The organization has confirmed that lookahead decoding is available under the Apache 2.0 license, enabling developers to integrate it into their commercial models and systems effectively.
By adopting lookahead decoding, businesses and researchers can harness enhanced performance capabilities while maintaining cost-effectiveness in their AI models, paving the way for more efficient and scalable applications of large language models.
Ranking
