Llama 3.2: Open-Weight Vision & Lightweight Models

What Is Llama 3.2?

Llama 3.2 is a groundbreaking language model family featuring enhanced capabilities, broader applicability, and multimodal image support. This release includes small and medium-sized vision LLMs (11B and 90B), and lightweight, text-only models (1B and 3B) that fit onto edge and mobile devices, in both pre-trained and instruction-tuned versions.

The 11B and 90B are the first Llama models to support vision tasks, with a new model architecture that integrates image encoder representations into the language model. The 1B and 3B models are enabled on day one for Qualcomm and MediaTek hardware and optimized for Arm processors.

Available Sizes and Variants

The Llama 3.2 collection offers models in various sizes, from lightweight text-only 1B and 3B parameter models suitable for edge devices to small and medium-sized 11B and 90B parameter models capable of sophisticated reasoning tasks including multimodal support for high resolution images.

Text-Only Models (Pre-trained and Instruction-tuned):

• Llama 3.2 1B comprises 1.23B parameters; 3B comprises 3.21B parameters
• Both available in BF16 and quantized variants (INT4/INT8)
• Quantized instruct versions have a reduced context length of 8k

Vision-Language Models (Multimodal):

• 11B and 90B vision variants for image understanding and text generation
• The 11B base model supports tile sizes of 448, while the instruct version and the 90B models use tile sizes of 560

Hardware Requirements

For Inference (text-only 1B/3B models):

• The 3B model needs about 7 GB of VRAM for FP16 inference
• Quantized to INT4, the 3B model needs about 1.8 GB of VRAM, so it fits on a 24 GB RTX 4090 with room for a small batch
• Recommended: GPU with CUDA support (16GB VRAM or higher); RAM: at least 32GB (64GB for larger models)
• Minimum 50GB of free disk space for the model and dependencies

For Vision Models (11B/90B):

• Vision models require significantly more resources; quantization (INT4/QLoRA) is recommended for local deployment

CPU/System:

• Operating System: Linux (preferred), macOS, or Windows; Python: Version 3.8 or higher; CUDA Toolkit: Required for GPU acceleration (11.6 or newer)

Benchmark Highlights

In the MMLU test with chain-of-thought prompting, Llama 3.2 90B scores 86.0, slightly outperforming Claude 3 Haiku and close to GPT-4o-mini.

On MMLU (5-shot), Llama 3.2 3B scores 63.4, outperforming Gemma 2B IT at 57.8. On GPQA (0-shot, CoT), Llama 3.2 90B scores 46.7, better than Claude 3 Haiku but behind GPT-4o-mini.

In the IFEval test, Llama 3.2 3B scores an impressive 77.4, beating Gemma 2B IT (61.9) and Phi-3.5-mini IT (59.2). On the MGSM multilingual benchmark, Llama 3.2 90B scores 86.9, a solid result comparable to GPT-4o-mini at 87.0.

How to Run It Locally

Using Ollama (Quickest Method):

For the most popular 3B model:

ollama run llama3.2:3b

For the 1B model (even lighter):

ollama run llama3.2:1b

Alternative Local Runners:

• LM Studio - provides a GUI interface with built-in model search and management
• Jan - streamlined interface optimized for conversational use
• GPT4All - desktop application with one-click model downloads
• Msty - full-featured client with prompt templates and history management

All these applications include Llama 3.2 in their default model catalogs and handle quantization variants automatically.

Manual Setup (Advanced):

Models were trained with up to 9 trillion tokens and support a long context length of 128k tokens. Download BF16 or quantized weights from Hugging Face (meta-llama/Llama-3.2-1B, meta-llama/Llama-3.2-3B, etc.) and use transformers + torch, or specialized frameworks like vLLM for optimized serving.

Best Use Cases

Text-Only Models (1B/3B):

• On-device use cases like summarization, instruction following, and rewriting tasks running locally at the edge
• Prompt rewriting, multilingual knowledge retrieval, summarization tasks, tool usage, and locally running assistants
• Mobile AI-powered writing assistants
• Personal knowledge management systems
• Real-time processing on phones, tablets, and smart devices

Vision Models (11B/90B):

• Visual reasoning and image understanding tasks, exceeding closed models such as Claude 3 Haiku
• Document analysis (charts, graphs, financial reports)
• Image captioning and visual grounding
• Multimodal document understanding and search

FAQ

Can I use Llama 3.2 commercially?

Llama 3.2 is granted under a non-exclusive, worldwide, non-transferable and royalty-free limited license allowing use, reproduction, distribution, copy, creation of derivative works, and modifications. If monthly active users exceed 700 million, a separate commercial agreement with Meta is required, which may involve fees or usage-based royalties.

What is the context length supported?

All Llama 3.2 models support a 128K context length, maintaining the expanded token capacity introduced in Llama 3.1. Note that quantized instruct versions have a reduced context length of 8k.

Which model size should I choose for my device?

For inference on consumer GPUs like an RTX 4090, the 3B model needs 7GB in FP16 but only 1.8GB when quantized to INT4. Choose 1B for maximum portability on phones/edge; choose 3B for better quality when you have 2GB+ available. For desktop with 8GB+ VRAM, use the full-precision 3B or quantized 11B vision model.

Are multimodal models restricted?

With respect to multimodal models, the rights granted under the Llama 3.2 Community License Agreement are not being granted if you are an individual domiciled in, or a company with principal place of business in, the European Union. This restriction does not apply to end users of a product or service that incorporates these multimodal models.