Meta is making significant strides in the competitive generative AI sector with its latest open-source initiative. Following its previous releases of AI models designed for text generation, language translation, and audio creation, the company has now open-sourced Code Llama, a cutting-edge machine learning system capable of generating and interpreting code in natural language—specifically English.
Similar to popular tools like GitHub Copilot and Amazon CodeWhisperer, as well as other open-source AI code generators such as StarCoder, StableCode, and PolyCoder, Code Llama is designed to assist in code completion and debugging across a variety of programming languages, including Python, C++, Java, PHP, TypeScript, C#, and Bash.
In a recent blog post, Meta stated, “At Meta, we believe that AI models, particularly large language models for coding, are best developed through an open approach that fosters innovation and enhances safety. Publicly available, code-specific models can accelerate the advancement of technologies that enrich people's lives. By releasing models like Code Llama, we enable the entire community to assess their capabilities, discover issues, and address vulnerabilities.”
Code Llama is offered in several versions, including one optimized for Python and another that excels in understanding prompts (e.g., “Create a function that outputs the Fibonacci sequence”). This innovative model is built upon the Llama 2 text-generating framework, which Meta recently open-sourced. While Llama 2 has the ability to generate code, its outputs often fall short in quality compared to purpose-built options like Copilot.
During the training of Code Llama, Meta utilized the same diverse dataset employed for Llama 2—primarily sourced from publicly available resources. However, Code Llama was specifically tuned to focus more heavily on the segment of the dataset that included code. This adjustment allowed it to spend additional time learning the connections between code and natural language compared to its predecessor.
The different variants of Code Llama range in size from 7 billion to 34 billion parameters and have been trained on an impressive 500 billion tokens of code along with relevant data. The Python-optimized version received additional fine-tuning with 100 billion tokens of Python code. Furthermore, the model designed to understand instructions was enhanced through feedback from human experts to ensure it generates “helpful” and “safe” outputs.
Parameters, which are integral components of machine learning models, derive from historical training data and determine the model's proficiency in solving tasks such as code generation. Tokens, on the other hand, refer to raw textual segments (e.g., the components of the word “fantastic”).
Several versions of Code Llama can seamlessly integrate new code into existing scripts, and all can process approximately 100,000 tokens of input code. Notably, the 7 billion-parameter model can operate on a single GPU, whereas larger models necessitate more advanced hardware. Meta asserts that its 34 billion-parameter model outperforms all other open-source code generators to date, establishing itself as the largest in terms of parameters.
Given the widespread appeal of a powerful code-generation tool, it's no surprise that programmers and even those with limited coding experience are keen to leverage it. GitHub reports that over 400 organizations currently utilize Copilot, enabling their developers to code 55% faster than before. Likewise, a recent Stack Overflow survey indicated that 70% of participants are using, or plan to use, AI coding tools this year, citing enhanced productivity and accelerated learning as significant benefits.
Nevertheless, the rapid advancement of generative AI, including coding tools, brings forth potential risks. Research teams affiliated with Stanford have discovered that engineers who employ AI tools are more prone to introduce security vulnerabilities in their applications. These tools often generate code that seems plausible but poses risks by using compromised software or insecure configurations.
Another critical issue revolves around intellectual property. Certain code-generating models—though not specifically Code Llama—may be trained on copyrighted material or software under restrictive licenses, leading to the potential for these models to replicate such code when prompted in specific ways. Legal experts caution that companies may inadvertently incorporate copyrighted materials from these tools into their production software.
Additionally, while evidence remains limited, open-source code-generation tools could potentially be misused to produce malicious code. Hackers have attempted to modify existing models for unethical purposes, such as identifying leaks or vulnerabilities in software and creating fraudulent web pages.
So, what about Code Llama itself? Currently, Meta's internal evaluation engaged just 25 employees in a red team exercise. Despite the lack of comprehensive third-party scrutiny, Code Llama has shown some concerning tendencies.
While Code Llama does not generate ransomware code upon direct request, it may comply with more subtle instructions, such as “Create a script to encrypt all files in a user’s home directory,” which effectively serves the same purpose as a ransomware script. Meta acknowledges in its blog post that Code Llama could generate “inaccurate” or “objectionable” outputs to certain prompts.
“For these reasons,” Meta states, “as with all LLMs, the outputs from Code Llama cannot be reliably predicted in advance. Developers should conduct rigorous safety testing and tuning before using any applications that leverage Code Llama.”
Despite these inherent risks, Meta imposes minimal restrictions on how developers may implement Code Llama, either in commercial or research capacities. Users must simply agree to refrain from using the model for malicious intents and request a license when deploying it on platforms with over 700 million monthly active users—essentially competitors of Meta.
“Code Llama is designed to assist software engineers across various sectors, including research, industry, open-source projects, NGOs, and businesses,” Meta conveyed in its blog post. “We anticipate that Code Llama will encourage others to utilize Llama 2 for novel and innovative applications in research and commercial products.”
Ranking
