Meta Llama LLM Models Context Window Details

Llama 1 (February 2023)

Parameter Size	Context Window	Key Features
7B, 13B, 33B, 65B	2048 tokens	Initial release of Llama models Strong performance on various NLP tasks Open-source, allowing for community innovations Trained on publicly available data Efficient inference, suitable for research and applications

Notable Achievements:

• Comparable performance to GPT-3 with significantly fewer parameters
• Sparked a wave of open-source AI development and fine-tuning projects

Llama 2 (July 2023)

Parameter Size	Context Window	Key Features
7B, 13B, 70B	4096 tokens	Significant improvements over Llama 1 Fine-tuned versions for chat (Llama 2 Chat) Enhanced safety measures and reduced toxicity Commercial usage allowed for companies with <700M MAU

Improvements:

• 40% more training data than Llama 1
• 2x longer context window (4096 tokens vs 2048)
• Constitutional AI techniques applied for better alignment
• Llama 2 Chat versions fine-tuned with over 1 million human annotations

Code Llama (August 2023 - January 2024)

Parameter Size	Context Window	Key Features
7B, 13B, 34B, 70B	4096 tokens	Specialized for code-related tasks Available in base, Python-specialized, and instruction-following variants Supports popular programming languages

Capabilities:

• Code completion
• Code generation from natural language descriptions
• Code explanation and debugging
• Support for 20+ programming languages including Python, C++, Java, PHP, Typescript, C#, Bash

Llama 3 (April 2024)

Parameter Size	Context Window	Key Features
8B, 70B	8192 tokens	Latest generation with significant improvements Enhanced reasoning and instruction-following capabilities Improved coding abilities Better factual accuracy and reduced hallucinations

Planned Future Developments:

• Multimodal capabilities (text, image, audio)
• Expanded multilingual support
• Extended context window
• Overall performance enhancements across various tasks