Introducing LLaMA: A Powerful Language Model by Meta AI
LLaMA (Large Language Model Meta AI) is a cutting-edge large language model (LLM) released by Meta AI in February 2023. With model sizes ranging from 7 billion to 65 billion parameters, LLaMA has demonstrated impressive performance on various NLP benchmarks. The 13 billion parameter model even outperformed the much larger GPT-3 (with 175 billion parameters), while the largest model proved competitive with state-of-the-art models like PaLM and Chinchilla.
Unlike other powerful LLMs, which are often accessible only through limited APIs, Meta released LLaMA’s model weights to the research community under a noncommercial license. However, within a week of its release, LLaMA’s weights were leaked to the public on 4chan via BitTorrent.
LLaMA’s Diverse Training Data
LLaMA was trained on a massive dataset of 1.4 trillion tokens, sourced from various publicly available data sources, including:
- Webpages scraped by CommonCrawl
- Open source repositories of source code from GitHub
- Wikipedia in 20 different languages
- Public domain books from Project Gutenberg
- The LaTeX source code for scientific papers uploaded to ArXiv
- Questions and answers from Stack Exchange websites
Release, Leak, and Reactions
LLaMA was announced on February 23, 2023, through a blog post and a paper detailing the model’s training, architecture, and performance. The code used to train the model was publicly released under the open-source GPL 3 license. Access to the model’s weights was managed by an application process, granting access on a case-by-case basis to academic researchers, government, civil society, and academia-affiliated organizations, and industry research laboratories worldwide.
Reactions to the leak were mixed. Some speculated that the model could be used for malicious purposes, such as sophisticated spam, while others celebrated its accessibility and potential to promote further research developments. LLaMA has been compared to Stable Diffusion, a text-to-image model that was openly distributed, leading to a rapid proliferation of associated tools, techniques, and software.
LLaMA’s Impact on Conversational AI Models
Since its release, LLaMA has become the foundation for various conversational AI models, such as Stanford’s Alpaca and Databricks’ Dolly. The latest addition to this list is Vicuna, a collaboration between researchers from UC Berkeley, CMU, Stanford, and UC San Diego.
Vicuna-13B is a new open-source chatbot developed to address the lack of training and architecture details in existing LLMs like OpenAI’s ChatGPT. Vicuna-13B is fine-tuned using a LLaMA base model and approximately 70,000 user-shared conversations from ShareGPT.com, resulting in an enhanced dataset. Preliminary evaluations show that Vicuna-13B achieves over 90% quality of OpenAI ChatGPT and Google Bard while outperforming other models like LLaMA and Stanford Alpaca in more than 90% of cases.
Vicuna is an open-source chatbot fine-tuned from a LLaMA base model using approximately 70,000 user-shared conversations collected from ShareGPT.com with public APIs. The research team ensured data quality by converting HTML back to markdown and filtering out inappropriate or low-quality samples. They also divided lengthy conversations into smaller segments to fit the model’s maximum context length. After fine-tuning, Vicuna generated more detailed and well-structured answers compared to Alpaca, with quality on par with ChatGPT.
You can browse various models on: https://huggingface.co/models?other=llama&p=1&sort=downloads
And see a comparison and benchamark of AI models here: https://lmsys.org/blog/2023-05-03-arena/
Leave a Reply