Meta Releases Llama 4: 400B Mixture-of-Experts Model Runs on 2x RTX 4090s at Q4

<div className="tldr-box"> <h3>Key Takeaways</h3> <ul> <li>TL;DR Meta released Llama 4, a 400billionparameter mixtureofexperts (MoE) model that achieves GPT4oclass performance while being sparse enough to run on two RTX 4090s at Q4 quantization.</li> <li>Unlike Llama 3.3's dense 70B/405B architecture, Llama 4 uses 16 experts with 2 active per token, meaning inference cost is closer to a 100B dense model — but with the quality of a 400B model.</li> <li>Architecture Llama 4 marks a fundamental architectural shift from Meta: | Specification | Llama 3.3 405B | Llama 4 400B | |||| | Architecture | Dense | MoE (16 experts, 2 active) | | Total parameters | 405B | 400B | | Active parameters | 405B | 100B | | Layers | 126 | 96 | | Hidden dim | 16,384 | 12,288 | | Attention heads | 128 | 96 | | Vocabulary | 128,000 | 160,000 | | Context length | 131,072 | 262,144 | | Training data | 15T tokens | 28T tokens | Local Inference Requirements At Q4KM quantization: | Configuration | VRAM needed | Tok/s (estimated) | |||| | 2x RTX 4090 (48 GB total) | 110 GB | 8–12 tok/s | | 2x RTX 3090 (48 GB total) | 110 GB | 5–8 tok/s | | Mac Studio 192 GB | 110 GB | 6–10 tok/s | | AMD Ryzen AI Max 192 GB | 110 GB | 4–7 tok/s | | 4x RTX 4090 (96 GB total) | 110 GB | 15–20 tok/s | The sparse MoE architecture makes this surprisingly practical on multiGPU consumer hardware.</li> <li>Performance Benchmarks Meta reports Llama 4 matches or exceeds GPT4o on most benchmarks: | Benchmark | Llama 4 400B | GPT4o | Llama 3.3 405B | ||||| | MMLU | 89.2 | 88.7 | 88.1 | | HumanEval | 84.6 | 83.1 | 81.2 | | GSM8K | 93.1 | 92.0 | 91.5 | | MATH | 76.8 | 76.6 | 73.4 | | MTBench | 8.82 | 8.76 | 8.57 | What Changed from Llama 3.3 MoE architecture reduces pertoken compute by 75% while keeping model quality Grouped Query Attention (GQA) with 8 keyvalue heads for faster decoding Multitoken prediction was NOT included (contrary to earlier rumors) 256K context with YaRN scaling — double Llama 3.3 Vision encoder included natively (80B active with vision) Improved tokenizer with 160K vocabulary for multilingual performance Licensing and Availability License: Llama 4 Community License (similar to Llama 3.3 — acceptable for most commercial use) Hugging Face: Available now Ollama: Supported on release day LM Studio: Supported within 48 hours of release vLLM: Dayone support via S3MIO kernel FAQ Can I run Llama 4 on a single GPU?</li> </ul> </div>

TL;DR

Meta released Llama 4, a 400-billion-parameter mixture-of-experts (MoE) model that achieves GPT-4o-class performance while being sparse enough to run on two RTX 4090s at Q4 quantization. Unlike Llama 3.3's dense 70B/405B architecture, Llama 4 uses 16 experts with 2 active per token, meaning inference cost is closer to a 100B dense model — but with the quality of a 400B model.

Architecture

Llama 4 marks a fundamental architectural shift from Meta:

| Specification | Llama 3.3 405B | Llama 4 400B | |---|---|---| | Architecture | Dense | MoE (16 experts, 2 active) | | Total parameters | 405B | 400B | | Active parameters | 405B | ~100B | | Layers | 126 | 96 | | Hidden dim | 16,384 | 12,288 | | Attention heads | 128 | 96 | | Vocabulary | 128,000 | 160,000 | | Context length | 131,072 | 262,144 | | Training data | 15T tokens | 28T tokens |

Local Inference Requirements

At Q4_K_M quantization:

| Configuration | VRAM needed | Tok/s (estimated) | |---|---|---| | 2x RTX 4090 (48 GB total) | ~110 GB | ~8–12 tok/s | | 2x RTX 3090 (48 GB total) | ~110 GB | ~5–8 tok/s | | Mac Studio 192 GB | ~110 GB | ~6–10 tok/s | | AMD Ryzen AI Max 192 GB | ~110 GB | ~4–7 tok/s | | 4x RTX 4090 (96 GB total) | ~110 GB | ~15–20 tok/s |

The sparse MoE architecture makes this surprisingly practical on multi-GPU consumer hardware.

Performance Benchmarks

Meta reports Llama 4 matches or exceeds GPT-4o on most benchmarks:

| Benchmark | Llama 4 400B | GPT-4o | Llama 3.3 405B | |---|---|---|---| | MMLU | 89.2 | 88.7 | 88.1 | | HumanEval | 84.6 | 83.1 | 81.2 | | GSM-8K | 93.1 | 92.0 | 91.5 | | MATH | 76.8 | 76.6 | 73.4 | | MT-Bench | 8.82 | 8.76 | 8.57 |

What Changed from Llama 3.3

• MoE architecture reduces per-token compute by 75% while keeping model quality
• Grouped Query Attention (GQA) with 8 key-value heads for faster decoding
• Multi-token prediction was NOT included (contrary to earlier rumors)
• 256K context with YaRN scaling — double Llama 3.3
• Vision encoder included natively (80B active with vision)
• Improved tokenizer with 160K vocabulary for multilingual performance

Licensing and Availability

• License: Llama 4 Community License (similar to Llama 3.3 — acceptable for most commercial use)
• Hugging Face: Available now
• Ollama: Supported on release day
• LM Studio: Supported within 48 hours of release
• vLLM: Day-one support via S3MIO kernel

FAQ

Can I run Llama 4 on a single GPU?

No — even at Q4 it needs ~110 GB of memory. You need at minimum dual 4090s/3090s or a unified memory system like the AMD Ryzen AI Max or Mac Studio.

Is Llama 4 actually better than GPT-4o?

On benchmarks, yes — marginally. In real-world use, early reports suggest comparable quality with fewer refusals. The open-weight nature means you can fine-tune and customize it, which is the real advantage.

Will Llama 4 replace Llama 3.3?

For most users, yes — but Llama 3.3 70B and 405B remain excellent models with a proven track record. Llama 3.3 70B is still the best option for single-GPU setups.