Best Local LLMs for 8GB VRAM: Real Hardware Benchmarks (2026)

TL;DR: Best Local LLMs for 8GB VRAM in 2026

After benchmarking five models on an RTX 3070 with llama.cpp, the verdict is clear: Qwen3.5-9B (Q4_K_M) is the best local LLM for 8GB VRAM by a significant margin. It is faster, smarter, and the only model that runs entirely in GPU memory at all four tested context sizes including 32K. As per our empirical testing, it is even capable of operating at a massive 200K+ context window with minimal performance penalty on 8GB cards. GLM-4.6V-Flash matches it at shorter contexts but hits a wall at 32K. Everything else involves a serious performance compromise.

• Winner: Qwen3.5-9B Q4_K_M (Vision-capable) - Full GPU offload at all contexts (4K–32K), 200K+ context capable with minimal penalty, 54–58 t/s decode, 6.96 GB peak at 32K, top intelligence index in its weight class, and the best frontend code in the practical test.
• Runner-up: GLM-4.6V-Flash Q4_K_M (Vision-capable) - Slightly faster prefill than Qwen at short contexts (2,376 vs 1,932 t/s at 4K), full GPU up to 16K, but decode speed collapses 70% to 17.4 t/s at 32K as it spills 4 layers to CPU.
• Nemotron Nano 12B v2 Q4_K_M - Technically fits (7.75 GB at 32K) but always partially CPU-offloaded, delivering 6.6–10.9 t/s. Choose if you need the higher intelligence ceiling and can tolerate slower output.
• Gemma 3 12B Q4_K_M (Vision-capable) - Heavy partial offload at all contexts (10.02 GB total at 32K), 4.3–8.6 t/s decode. A functional fallback if you specifically need Google's architecture.
• Phi-4 14B Reasoning Plus Q4_K_M - Included purely as a hardware scaling reference point. Released roughly a year ago, this older 14B parameter architecture is deeply CPU-bound on an 8GB card, bottlenecking at a near-unusable 1.8 t/s at 32K context.

Choosing your GPU? See our complete GPU comparison for LLM inference. Running 16GB VRAM? Check best local LLMs for 16GB VRAM where larger MoE models open up.

Intelligence Benchmarks for 8GB VRAM Models

Before testing hardware performance, it is worth establishing which models are actually intelligent enough to be worth running. The chart below compares our full 15-model shortlist (curated from the Artificial Analysis leaderboard) across three composite indices: overall intelligence, coding, and math. We ultimately benchmarked the five models that achieved the best coverage of the 7B–14B VRAM-safe range.

Key Benchmark Insights

Qwen3.5 Dominates Across the Board

Qwen3.5-9B scores 32.4 on the AA Intelligence Index, a commanding 38% lead over the runner-up GLM-4.6V (23.4). While it currently lacks a published AA Math Index score, its exceptionally high coding and general reasoning metrics suggest it performs on par with, or better than, the larger 12B models in mathematical logic. Alibaba's architecture improvements and training data quality have largely invalidated the "bigger is always smarter" rule in the sub-10B tier.

Larger Models Provide Diminishing Returns

Raw parameter count does not guarantee better reasoning. Despite taking up drastically more VRAM, Gemma 3 12B scores a dismal 8.8 on the intelligence index. Similarly, the older generation Phi-4 14B (included here purely to demonstrate the hardware penalty of running 14B architectures on 8GB VRAM) scores just 10.4. While Microsoft's new Phi-4-reasoning-vision-15B family promises a massive leap in capability, we omitted it here due to a lack of formalized benchmarks. Ultimately, pushing past 9B parameters on an 8GB GPU offers little reward for the massive throughput penalty incurred.

How We Tested: Inference Benchmark Methodology

We built llama.cpp from source to ensure correct CUDA 13.1 compatibility with the RTX 3070 (Compute Capability 8.6).

Hardware Configuration

Benchmark Settings

Every model was benchmarked with identical core flags to ensure a fair, apples-to-apples comparison. The inference engine was started via the command line like so:

llama-server \
  -m /path/to/model.Q4_K_M.gguf \  # The quantized model file (Q4_K_M for all variants)
  --fit on \                      # Automatically calculates and loads max layers into VRAM
  --flash-attn on \               # Enables Flash Attention to reduce KV cache memory usage
  -c 4096 \                       # Context window size (tested at 4096, 8192, 16384, and 32768)

The 8GB VRAM Reality: What Actually Fits (Text-Only)

The RTX 3070 has 7,842 MiB of usable VRAM. When llama.cpp loads a text-only model, it allocates three pools: model weights (fixed), KV cache (grows linearly with context), and a compute buffer (fixed overhead for attention calculations). If the sum exceeds your available VRAM, llama.cpp automatically reduces the number of GPU layers, spilling the remainder to system RAM over PCIe, and decode speed tanks accordingly (note that this auto-offload only happens if you use --fit on with llama.cpp, but it is the default behavior in both Ollama and LM Studio).

The table below shows the exact VRAM breakdown llama.cpp reported for every text-only model at every tested context size. Total Mem (VRAM + RAM) is the sum of GPU model weights, KV cache, and roughly 500-1500 MiB of variable compute buffer necessary for calculations. Physical VRAM Used includes the CUDA runtime baseline (~200 MiB). Models shown in bold achieved full GPU offload.

The table reveals a critical feature in action: Whenever the model and its KV cache require more than 8GB of VRAM, the engine packs the GPU as full as it safely can to circumvent a crash, and dumps the overflow into standard system CPU RAM. This automatic layer splitting is triggered entirely because of the --fit on flag we used in our benchmark settings (which is also the default fail-safe behavior in both Ollama and LM Studio). Without this flag, using -ngl 99 would have resulted in a simple out-of-memory crash on load. "--fit on" rescues usability, but the price is paid in speed: every decode step requires fetching the layers across the PCIe bus, which is what causes that dramatic throughput collapse.

Decode Speed: The Real-World Performance Gap

Decode speed, the rate at which a model generates output tokens, is the metric users feel most directly. A model at 10 t/s produces roughly one word every 0.7 seconds; at 55 t/s, responses stream in faster than most people read. The difference between 55 t/s and 5 t/s is the difference between a snappy coding assistant and an experience that makes you reach for the phone.

Decode speed (tokens/sec) across all 5 benchmarked models at four context sizes. RTX 3070 · llama.cpp · Q4_K_M · Flash Attention ON

The GLM-4.6V-Flash result at 32K deserves special attention. At 16K it matched Qwen3.5-9B almost exactly (54.9 t/s), but at 32K it spilled just 4 out of 41 layers to CPU. This seemingly minor change cut decode speed from 55 t/s to 17.4 t/s. This is the PCIe bottleneck in action: even one or two CPU layers force the decode path to synchronise across the bus on every single token, creating a serialisation penalty that dwarfs the compute cost.

Phi-4 14B at 32K context (1.8 t/s) generated 719 tokens in 422 seconds, just under 7 minutes for a single response. Because it is a reasoning model that typically generates extended chain-of-thought traces, real usage at 32K context on 8GB VRAM is effectively impractical.

The CPU Spill Penalty — and Why Full GPU Offload Is Non-Negotiable

Why does spilling even a few layers to CPU hurt so badly on what is otherwise a fast system? The i7-8700K has 64 GB of DDR4 RAM with roughly 40 GB/s bandwidth. The RTX 3070 has 385.7 GB/s memory bandwidth, nearly 10× faster. During the decode phase, each new token requires reading every model weight that participates in that token's forward pass. If some weights live in GPU memory and others in system RAM, the decode loop must repeatedly transfer data across the PCIe 3.0 x8 interface (theoretical ~8 GB/s effective, far less in practice with latency). The GPU sits idle waiting for RAM. This creates a bottleneck that is architectural. No tuning flag can fix it.

Total VRAM vs context window. The red dashed line marks the RTX 3070's 8 GB limit. Models above the line require CPU spill, and the performance penalty is severe.

GPU model weights (dark) plus KV cache (bright top segment) per model at each context size. The KV cache doubles with every context doubling, a predictable linear growth.

Prefill Speed: Prompt Processing Rate

Prefill speed (tokens processed per second during prompt ingestion) determines how long you wait before the model begins streaming output. For short prompts it is imperceptible, but for long RAG contexts or document analysis tasks with thousands of input tokens it becomes significant. Unlike decode speed, prefill is parallelisable across GPU and CPU simultaneously, which is why even partially-offloaded models show respectable prefill rates.

Prefill speed (prompt tokens/sec) at 4K, 16K, and 32K context. Higher means shorter wait before the first generated token appears.

GLM-4.6V-Flash and Qwen3.5-9B both exceed 2,000 t/s at 4K context, meaning a 2,000-token prompt is processed in under one second. The 12B and 14B models land in the 170–570 t/s range at 32K, translating to 20–70 seconds of prefill delay before the model begins replying. For a 12,000-token code review, that is a meaningful UX difference.

Model Deep Dives

Qwen3.5-9B Q4_K_M - The Clear Winner

File size: 5.68 GB  |  AA Intelligence Index: 32.4  |  Peak VRAM (32K): 6.96 GB

Qwen3.5-9B is the only model in this test that fits entirely in GPU memory at every context size tested, including 32K. Its 33 transformer layers load completely onto the RTX 3070, leaving the PCIe bus uninvolved in the decode loop. The result is a consistent 55–58 t/s across all context sizes, barely degrading as context grows because KV cache reads are served from GDDR6 (385.7 GB/s) rather than system DDR4 RAM.

The secret to this "punching above its weight" hardware footprint lies in its efficient hybrid architecture. Qwen3.5 deploys Gated Delta Networks combined with sparse Mixture-of-Experts, allowing it to deliver high-throughput inference with minimal latency and drastically reduced VRAM overhead for the KV cache compared to traditional architectures. As per our empirical testing, Qwen3.5-9B is uniquely capable of operating at a full 200K+ context window with minimal performance penalty even on 8GB cards without KV cache quantization!

The secret to expanding beyond the 32K limits lies in the architecture itself, which reduces the memory cost of increasing context length. To further reduce memory requirements or to improve inference performance with longer contexts, the Qwen engineering team officially states: The Qwen3.5 series maintains near-lossless accuracy under 4-bit weight and KV cache quantization. Because of this exceptionally resilient architecture, you can aggressively quantize the KV cache (starting with 8-bit, and dropping to 4-bit depending on practical performance) to drastically reduce memory overhead without the severe degradation seen in other models.

The intelligence story is equally strong. Qwen3.5-9B tops the AA Intelligence Index for the sub-10B category with a score of 32.4, decisively outperforming 12B models from NVIDIA and Google. Its reasoning mode supports natively generating chain-of-thought inference via the /think tag. A 32K context window in practice requires 6.96 GB VRAM, leaving just 850 MiB headroom. This is tight but stable on a headless GPU instance.

For 8GB VRAM setups, this is unequivocally the default choice.

GLM-4.6V-Flash Q4_K_M - Fast to 16K, Cliff at 32K

File size: 6.17 GB  |  AA Intelligence Index: 23.4  |  Math Index: 85.3  |  Peak VRAM (32K): 7.16 GB total

GLM-4.6V-Flash is the fastest prefill model in this test, reaching 2,376 t/s at 4K context — 23% faster than Qwen3.5-9B at the same context. It also leads on the math index (85.3), making it the best choice for quantitative reasoning tasks among the models tested. At 4K, 8K, and 16K context it matches Qwen3.5-9B closely on decode speed (54.9–58.2 t/s). However, at 32K the KV cache growth pushes total required memory just past what the RTX 3070 can hold with all 41 layers in GPU. llama.cpp spills just 4 layers to CPU, and decode speed collapses 70% to 17.4 t/s. For workflows under 16K context, GLM-4.6V-Flash is an excellent Qwen alternative. For 32K, stick with Qwen3.5-9B.

Nemotron Nano 12B v2 Q4_K_M - High Ceiling, Slow Output

File size: 7.49 GB  |  AA Intelligence Index: 14.9  |  Math Index: 75.0  |  Peak VRAM (32K): 7.75 GB total system

Nemotron Nano 12B v2 has an unusually deep architecture at 63 transformer layers, and llama.cpp consistently offloads 15–19 layers to CPU regardless of context size. Physical VRAM Used stays at approximately 6.2 GB flat, but CPU RAM absorbs 2–2.3 GB of weights. Decode ranges from 6.6–10.9 t/s. Its math index of 75.0 makes it worth considering if quantitative reasoning is your priority over generation speed. At 10.9 t/s it is conversational at short contexts; at 32K it drops to 6.6 t/s, roughly one word every 0.7 seconds.

Gemma 3 12B Q4_K_M - Functional but Outclassed

File size: 7.30 GB  |  AA Intelligence Index: 8.8  |  Peak VRAM (32K): 10.02 GB total system

Gemma 3 12B is the most CPU-spilled model in this test. At 32K context, only 26 of its 49 layers remain on GPU (the majority of model weight sits in system RAM), and total system memory reaches 10.02 GB. Decode degrades from 8.6 t/s at 4K to 4.3 t/s at 32K, approximately 2 minutes per 500-token response. Combined with an intelligence index of just 8.8, there is no compelling reason to choose Gemma 3 12B for an 8GB VRAM system. It is a good model on 16GB+ VRAM where all 49 layers fit comfortably in GPU memory.

AA Intelligence Index for the 5 benchmarked models. Qwen3.5-9B scores 32.4, nearly 4× Gemma 3 12B (8.8), despite being the physically smallest model tested.

Practical Test: Smart Home Dashboard Challenge

Benchmarks measure what a model knows. Practical tests measure what it can do. We gave each model a single identical prompt: write a complete, functional smart-home dashboard in a single self-contained HTML file with dark mode, sidebar navigation, interactive light buttons, a temperature slider, and a lock/unlock toggle with all CSS and JavaScript inline, and no external libraries. The output had to open directly in a browser with no build step.

We recorded a full video walkthrough of each model's output. Watch to see the quality gap between first and last place:

Qwen3.5-9B is the clear winner for frontend coding. Despite being the smallest model tested (9B parameters), it produced the most complete, polished, and bug-free output. It utilizes inline SVGs for icons, a proper CSS toggle switch with animation, a colour-coded temperature display that shifts between warning states, card hover animations, a user profile header, and a full CSS variable design system. Phi-4 14B earns second place with the most informative status display, but is noticeably simpler in visual design. GLM-4.6V-Flash and Gemma 3 12B are roughly tied. Both are functional with minor issues. Nemotron is the weakest entry, shipping with a broken FontAwesome dependency, just one light button, and no status text on the lock toggle. The ranking confirms that for single-file frontend coding tasks, Qwen3.5-9B punches well above its weight class, outperforming models with 1.3–1.6× more parameters.

Quick Start: Run Qwen3.5-9B on Google Colab

If you want to test the official multimodal Qwen3.5-9B (which includes vision encoders) but don't have enough local VRAM, you can easily run it on a free Google Colab T4 GPU instance. Just drop these commands into sequential notebook cells:

Cell 1: Install Dependencies

!sudo apt-get update
!sudo apt-get install zstd

Cell 2: Install Ollama

!curl -fsSL https://ollama.com/install.sh | sh

Cell 3: Start the Backend Server

# Run in the background so Colab doesn't block
!nohup ollama serve &

Cell 4: Pull the Multimodal Model

# This downloads the official tag, which includes the vision projectors
!ollama pull qwen3.5:9b

Cell 5: Run Inference

# The --verbose flag prints throughput tokens/sec after generating
!ollama run qwen3.5:9b --verbose "How are you"

For precise VRAM estimates for your specific model and context window, use our interactive VRAM calculator. For a full explanation of how context length affects memory, see our llama.cpp VRAM requirements guide.

Special Test: The Hidden Memory Cost of Vision Encoders

A common point of confusion is the VRAM difference between a standard text-only model and its multimodal (vision) equivalent. To measure this exactly, we ran a secondary benchmark using Ollama on a 16GB Tesla T4 GPU. We compared a strictly text-only unsloth compile of Qwen3.5-9B-Q4_K_M against the official multimodal qwen3.5:9b tag from the Ollama registry.

The results show that the vision encoder and its projector layers consume an enormous amount of VRAM simply by existing in memory, even when only processing text prompts.

The vision encoder adds a massive 1.38 GB (1,380 MiB) of fixed VRAM overhead across all context sizes. Furthermore, passing an actual image through the multimodal model at a 4K context dropped the prefill speed from 738 t/s to 569 t/s, and nearly doubled the Time to First Token (from 3.8s to 6.7s) as the vision transformer processed the image patches. If your workflow does not strictly require image analysis, downloading text-only GGUFs saves a massive amount of VRAM, especially on an 8GB card.

Conclusion

The 8GB VRAM tier is genuinely capable for local LLM inference in 2026, but it demands model selection discipline. The RTX 3070 delivers 55+ tokens per second on the right model. On the wrong model pushed past its memory limit, it degrades to 1.8 tokens per second: a 30× performance collapse caused entirely by PCIe bandwidth constraints between GPU and system RAM.

• Always check total VRAM requirement, not just file size. A 7.5 GB GGUF file does not mean 7.5 GB of VRAM. The FP16 KV cache at 32K context adds another 1–2 GB depending on architecture.
• Full GPU offload is non-negotiable on 8GB. Even 4 layers spilling to CPU (GLM-4.6V-Flash at 32K) cuts decode speed by 70%. Either the model fits entirely in VRAM, or you accept a severe performance penalty.
• Stick to Q4_K_M for 8GB cards. It provides the perfect balance of model intelligence and memory footprint, allowing Qwen3.5-9B to comfortably run a 32K context inside 6.96 GB VRAM.
• Qwen3.5-9B is the benchmark to beat until a sub-10B model surpasses it on the AA Intelligence Index while maintaining full GPU offload at 32K on 8GB VRAM.

One area where this class of models genuinely shines, and which is easy to overlook, is long-document workflows. Tasks like RAG (Retrieval-Augmented Generation), chatting with PDFs, summarising research papers, and reviewing lengthy codebases are all excellent fits for Qwen3.5-9B on an 8GB card. Its ability to operate at 200K+ context fully in GPU memory means it can hold an entire document in its attention window and reason over it at 55+ tokens per second. For knowledge-base Q&A, local document assistants, and private research tools, this model-hardware combination is arguably the best price-to-capability option available in 2026.

On the coding front, 8GB models are excellent for generating code snippets, short utility functions, boilerplate scaffolding, and explaining existing code. However, if your goal is true agentic coding - multi-file edits, autonomous debugging loops, architectural refactoring, or long-horizon tool-use, this weight class will hit its intelligence ceiling. Tasks of that complexity benefit significantly from 20B+ parameter models that can maintain coherent reasoning across very large contexts and multi-step plans. We have covered the best options for that tier in our Best Local LLMs for 24GB VRAM guide and the llama.cpp VRAM requirements guide, which maps model sizes to hardware configurations across the full range.

Frequently Asked Questions

What is the best local LLM for 8GB VRAM in 2026?

Qwen3.5-9B (Q4_K_M) is the clear winner. In our RTX 3070 benchmarks it delivered 54–58 tokens per second across all context sizes, fits entirely in GPU memory at 32K context (6.96 GB), and topped the Artificial Analysis Intelligence Index for sub-10B models. GLM-4.6V-Flash is a strong alternative at contexts up to 16K, but its decode speed collapses from 55 to 17.4 t/s at 32K when it spills to system RAM.

How much VRAM does a 9B model actually use with llama.cpp?

With Q4_K_M quantization, Qwen3.5-9B uses approximately 6.09 GB at 4K context, 6.21 GB at 8K, 6.46 GB at 16K, and 6.96 GB at 32K on the RTX 3070. Model weights (5.06 GB on GPU) stay fixed; only the KV cache grows linearly, doubling every time the context size doubles.

Can I run a 12B model on 8GB VRAM?

Yes, but performance suffers significantly. Nemotron 12B and Gemma 3 12B both ran on our RTX 3070, but llama.cpp spilled 15–23 layers to system RAM. Decode speeds dropped to 4–11 t/s versus 55–58 t/s for fully-offloaded 9B models. A 9B model at full GPU speed produces substantially more useful output per hour than a 12B model throttled by PCIe bandwidth.

What context length can I use with 8GB VRAM?

For a standard 9B model running Q4_K_M (with default FP16 KV Cache), 32K context is comfortably achievable, as our benchmarks confirmed 6.96 GB at 32K for Qwen3.5-9B. For larger 12B models, practical context is limited to 4K–16K on 8GB VRAM before forcing a massive CPU-spill penalty.

However, if you are running Qwen3.5 specifically, their engineering team has officially confirmed the model maintains "near-lossless accuracy" under aggressive KV cache quantization. By dropping your KV cache to 8-bit or 4-bit precision, you can safely scale Qwen3.5 up to 200K+ context windows natively inside 8GB VRAM. Use our interactive VRAM calculator to estimate your specific configuration and quantization levels.

What is the best vllm, llama.cpp, LM Studio and Ollama model for 8GB VRAM chatbot or assistant?

Qwen3.5-9B (Q4_K_M) is the best model for an 8GB VRAM chatbot or local assistant, and it works perfectly with vllm, llama.cpp, LM Studio, and Ollama. It delivers 55+ tokens per second fully in GPU memory, which translates to a smooth, near-instant conversational experience on a single RTX 3070. Its 32K+ context window means it can hold a long multi-turn conversation without losing earlier context, and it supports a /think reasoning mode for harder queries. GLM-4.6V-Flash is a strong second choice for shorter conversations. Both models run identically in Ollama, LM Studio, or directly via llama.cpp since all three use the same underlying inference engine.

What quantization should I use for 8GB VRAM?

Q4_K_M is the recommended baseline. It compresses a 9B model to approximately 5 GB while retaining around 95% of capability. This enables full GPU offload for deep long-context reasoning workflows. For a deeper explanation of the math, see our quantization explained guide.