How to Calculate GPU VRAM for Quantized Local LLMs (GGUF & vLLM)

Running local Large Language Models (LLMs) with Ollama, llama.cpp, LM Studio, or vLLM requires understanding that GPU memory is divided into three distinct allocations: 1. Model Weights Memory: The static byte footprint required to hold the model parameters in VRAM. This is determined by parameter count and quantization bit-depth. 2. KV Cache (Context Memory): Dynamic memory consumed by storing attention key-value vectors for every token in the conversation history. KV cache grows linearly with context length and batch size. 3. CUDA Runtime & Activation Overhead: Memory reserved by PyTorch or CUDA for computation scratchpads, kernel caches, and activation buffers (typically 1.0GB to 2.5GB).

Quantization compresses floating-point model weights into smaller integer representations, dramatically cutting memory requirements: - FP16 (Half Precision - 16 bits / ~2.0 bytes per param): Full precision baseline. A 70B model requires ~140GB VRAM just for weights. - Q8_0 (8-bit Quantization - ~1.05 bytes per param): Near-zero perplexity loss. A 70B model requires ~75GB VRAM. - Q4_K_M (Medium 4-bit Quantization - ~0.56 bytes per param): The sweet spot for local inference. Retains over 99% of original benchmark performance while reducing a 70B model to ~39GB VRAM.

def estimate_vram(params_billions, bits_per_weight=4.5, context_len=8192):
    # Weight memory in GB (with 15% CUDA runtime overhead)
    weight_gb = (params_billions * (bits_per_weight / 8.0)) * 1.15
    # Approximate KV cache memory for standard 32-layer architecture in FP16
    kv_cache_gb = (context_len / 1024) * 0.25
    total_vram_needed = weight_gb + kv_cache_gb
    return round(total_vram_needed, 2)

print(f"Llama 3 70B (Q4_K_M, 8k context): {estimate_vram(70, 4.5, 8192)} GB")