Understanding GPU VRAM

Imagine attempting to run a marathon in shoes two sizes too small — uncomfortable and inefficient. Similarly, deploying Large Language Models without adequate VRAMUnderstanding VRAM requirements helps you choose the right GPU, optimize your models, and avoid out-of-memory errors (Training typically requires 3-4X more VRAM than inference!). can lead to frustrating bottlenecks.

As the popularity of the LLMs are increasing, running large language models locally has become increasingly popular. So it's important to understand the VRAM requirements for training and inference. As it is the basic requirement to run any model.

What is VRAM and Why Does It Matter?

In machine learningA field of AI where models learn patterns from data to make predictions or generate new content., VRAM is critical because:

• Model Storage: Your model's parametersLearnable numbers (weights & biases) inside a neural network that define how it behaves. must fit inside VRAM
• Data Loading: Input data batches are copied to VRAM during training and inference
• Intermediate Results: ActivationsTemporary outputs produced at each layer of the model while processing data., gradientsThe derivatives of the loss function used to adjust weights during training., and optimizer states all take space
• Speed: VRAM is far faster than CPU memory or disk access

Training vs. Inference: The Memory Difference

One of the biggest distinctions in ML is that training requires significantly more VRAM than inference.

Inference (Making Predictions)

When running a trained model, you only store:

• Model parametersThe weights and biases that define your model's behavior.
• Input data
• Intermediate activationsOutputs from each layer as data flows through the network.

Training (Learning from Data)

Training adds much more memory pressure because it also stores:

• Gradients: DerivativesMathematical slopes that tell us how to adjust each parameter to reduce error. for every parameter
• Optimizer States: Adam optimizerA popular training algorithm that stores momentum and variance for each parameter. keeps 2 additional copies of parameters
• Larger Batches: Training often uses bigger batch sizesThe number of training samples processed together before updating model weights. for better stability

Factors Involved in VRAM (GPU Memory) Calculation

GPU VRAMVideo Random Access Memory — dedicated memory on your GPU for handling data during computation. usage in machine learning depends on several components. These can be grouped into five major categories that together determine how much memory your model consumes.

1. Model Parameters (Weights + Biases)

Depends on:

Factor	Description
Number of Parameters	Total learnable weights and biases (e.g., 110M for BERT Base, 7B for LLaMA 7B)
Precision / Data Type	Determines bytes per parameter. FP32 → 4 bytes, FP16/BF16 → 2 bytes, INT8 → 1 byte

2. Activations (Intermediate Outputs)

Depends on:

Factor	Description
Batch Size	Number of samples processed simultaneously. Larger = more activations.
Sequence Length / Resolution	Input size (tokens per sequence or image pixels). Larger = higher memory use.
Hidden Size	Width of each layer (e.g., 768 for BERT, 4096 for GPT).
Number of Layers	Deeper networks require more activation storage.
Precision	FP32 uses more memory than FP16 or BF16.

3. Gradients

Each parameter has a corresponding gradient (same size, same precision). So, if model weights occupy 4 GB, gradients typically consume another ~4 GB.

4. Optimizer States

Optimizer	Memory Overhead
SGD	~1× model size
Adam / AdamW	~2× model size (momentum + variance)
Adagrad / RMSProp	~2× model size
8-bit Optimizers (bitsandbytes)	~0.25× model size

5. Framework & GPU Overheads

• CUDA kernels and buffers
• cuDNN workspace
• Framework runtime cache (PyTorch / TensorFlow)
• Temporary tensors (reshaping, attention cache, etc.)

Putting It All Together

Inference VRAM

Typically about 1.2× Model Memory.

Example: LLaMA 7B Inference

Training VRAM

Or approximately:

Example: GPT-2 Medium Training

Quick Reference Formula

Symbol	Meaning	Typical Value
M	Multiplier (optimizer overhead)	3–4
A	Activation Memory	0.5–2× model memory
O	Framework Overhead	1–2 GB

Memory Optimization Strategies

When you're running out of VRAM, here are proven techniques to reduce memory usage:

Factor	Description	Effect
Reduce Precision (FP16/BF16)	Uses smaller data type	Use FP1616-bit floating point — cuts memory in half compared to FP32. or INT88-bit integer quantization — reduces memory by 75% compared to FP32. instead of FP32 to cut memory by 50-75%.
Gradient Checkpointing	Recomputes activations during backprop	Trade computation for memory by recomputing activationsIntermediate layer outputs that are normally stored for backpropagation. during the backward pass instead of storing them.
Model Quantization (INT8/INT4)	Compresses model weights	Convert trained models to INT8 for inference — achieves 4× memory reduction with minimal accuracy loss.
Batch Accumulation	Processes small batches iteratively	Reduce batch sizeNumber of samples processed together — smaller batches use less memory but may affect training stability. to lower activation memory (may affect training stability).
Model Parallelism	Splits model across GPUs	Split your model across multiple GPUs when it doesn't fit on a single device.
Offloading (CPU/NVMe)	Moves inactive tensors off GPU	Frees GPU VRAM dynamically
Gradient Accumulation	Simulates larger batches	Simulate larger batches by accumulating gradientsUpdates to model parameters — accumulating them over multiple small batches mimics a large batch. over multiple small batches.

Refrences

• Introduction to vrams
• calculate vram requirements
• how much vram need fine-tuning