Choosing quantization formats: GGUF, AWQ, or EXL2 compared

When running large language models locally or on your own servers, the original 16-bit floating-point format (FP16 or BF16) is rarely practically feasible. A modern open-weights model of 70 billion parameters quickly requires more than 140 GB of video memory uncompressed, purely to load the static weights. Anyone seeking efficiency turns to weight quantization: reducing numerical precision to 8, 4, or even 2 bits per parameter. Anyone who wants to know how the underlying compression mathematics and rounding techniques work can read quantization explained in the practical guide on hardware efficiency.

Yet choosing the right file and runtime format is not a universal decision. A format that excels on a laptop with Apple Silicon often performs poorly in a Linux server cluster with multiple Nvidia A100 or H100 GPUs. The three dominant formats in the current ecosystem — GGUF, AWQ, and EXL2 — represent three fundamentally different philosophies around hardware support, precision distribution, memory bandwidth, and runtime engines. To determine how much video memory is needed for a specific configuration, the article on converting parameters to VRAM helps provide direct insight into the absolute minimum requirements.

The technical core of GGUF, AWQ, and EXL2

The landscape of model quantization has evolved from simple post-training rounding to advanced algorithms that account for activation outliers and per-layer error optimization. Each of the three leading formats approaches this problem from a different perspective.

GGUF (GPT-Generated Unified Format) is designed as the binary successor to GGML by the llama.cpp project. It is a binary container format that not only stores the quantized weights but also unifies all necessary metadata, tokenizer configurations, and hyperparameters into a single file. GGUF uses k-quants (such as Q4_K_M or Q5_K_S), where different layers and components within a transformer block (such as attention mechanisms versus feed-forward networks) are kept at different bit precisions to minimize quality loss.

AWQ (Activation-aware Weight Quantization) opts for a hardware-optimized matrix approach. Instead of treating all weights equally, AWQ observes during a calibration step which weights correspond to the 1% most significant activations in the model. These 'salient weights' are protected against aggressive rounding errors by applying scaling instead of leaving them in FP16. The result is a homogeneous 4-bit format that can be natively decoded on GPU tensor cores via specialized CUDA or Triton kernels, without complex mixed-precision logic during inference.

EXL2 (ExLlamaV2) was developed with one specific goal in mind: maximum throughput and flexible bit depths on Nvidia consumer and datacenter GPUs. EXL2 builds on the GPTQ algorithm (Generalized Post-Training Quantization) but introduces the ability for variable sub-bit quantization (for example 3.2, 4.25, or 6.0 bits per weight). This allows a model to be tuned exactly to the hard physical limits of a specific GPU memory capacity, such as 24 GB on an RTX 3090 or RTX 4090.

Hardware compatibility and offloading strategies

The hardware architecture on which the model runs is the most decisive factor in the format choice. When a model does not fit entirely into the available video memory, major operational differences arise between the runtimes.

GGUF excels in environments with heterogeneous memory. Thanks to the architecture of llama.cpp, a model can be arbitrarily split across the system RAM and the VRAM of one or more graphics cards. If a 70B model in Q4_K_M requires around 43 GB and only a 16 GB GPU is available, llama.cpp can, for example, send 18 transformer layers to the GPU and run the remaining 62 layers on the system CPU. On platforms with unified memory, such as Apple Silicon Macs, GGUF reads directly from shared memory via Metal kernels, resulting in excellent performance without CPU-GPU transfer overhead. For those setting up small-scale systems, the overview on small models on the device practical rules of thumb about memory constraints.

AWQ and EXL2, on the other hand, are strictly designed for GPU inference. AWQ requires that the entire model fit into the VRAM of the connected GPUs (possibly split via Tensor Parallelism). When memory fills up, the runtime crashes with an Out-Of-Memory (OOM) error; there is no built-in fallback to system RAM. EXL2 does have experimental CPU offloading, but in practice the processing speed then collapses so severely that it is unusable for production purposes. EXL2 requires specific Nvidia GPU architectures (Pascal or newer) to execute its custom-built CUDA kernels with maximum efficiency.

Memory bandwidth and quantization precision (PPL vs. VRAM)

The performance of a language model during the generation phase (token-by-token generation) is almost exclusively bounded by memory bandwidth, not by the compute power of the compute cores. Lowering the precision directly reduces the number of bytes that must be transported over the memory bus per generated token.

Reducing the bit depth, however, introduces quantization noise, measurable via perplexity (PPL). A lower perplexity indicates a model that stays closer to the original FP16 baseline output. For larger models (from 30B and 70B parameters), the quality loss at 4-bit and 5-bit quantization is negligibly small. For smaller models (such as 3B to 8B), aggressive quantization below the 4-bit threshold leads to noticeable degradation in reasoning ability, coding syntax, and multilingual consistency.

AWQ generally maintains a very low perplexity increase at 4 bits because it accounts for activation patterns. GGUF compensates for this via k-quants: in a Q4_K_M model, critical components such as the v_proj and output tensors are kept at higher precision (6-bit), while less sensitive feed-forward matrices are reduced to 4-bit. EXL2 offers the highest granularity here: instead of being locked into steps of 1 bit (such as the jump from 4 to 5 bit), a user can target exactly 4.65 bits per weight. This makes it possible to maximally utilize the remaining 2 GB of free space on a graphics card for a larger context window or slightly higher precision.

Inference speed: single-stream versus high concurrency

The choice of format depends heavily on the type of workload: does the server serve a single user who expects low latency per token (single-stream), or must the system process dozens of simultaneous requests via continuous batching?

In single-stream scenarios on Nvidia GPUs, EXL2 is generally the fastest runtime. Because ExLlamaV2 uses specially written kernels optimized for batch sizes 1 to 4, a 70B model on a dual-GPU setup achieves significantly higher tokens per second than an equivalent GGUF configuration via llama.cpp. The overhead of the kernel calls is minimal and the decode step is extremely streamlined.

In production environments with hundreds of simultaneous API calls, the advantage shifts entirely to AWQ in combination with engines such as vLLM or SGLang. These engines make use of PagedAttention and advanced continuous batching. AWQ kernels integrate seamlessly with Tensor Parallelism across 2, 4, or 8 GPUs and maintain high computational density as batch size increases. GGUF and EXL2 are usable for small-scale concurrency, but scale less efficiently under high concurrent load due to limitations in dynamic memory allocation for the KV cache.

# Starten van een AWQ model in vLLM met geoptimaliseerde tensor parallelism
python3 -m vllm.entrypoints.openai.api_server \
  --model casperhansen/llama-3.3-70b-instruct-awq \
  --quantization awq \
  --tensor-parallel-size 2 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.95 \
  --port 8000

Impact on the KV cache and context length

A common misconception is that model quantization automatically solves the entire memory problem. Quantization of weights compresses only the static model parameters. As the context length increases to 32k, 64k, or 128k tokens, the Key-Value cache (KV cache) begins to become a dominant factor in the total VRAM footprint. How a context window builds up and why this directly impacts dynamic memory usage is explained in more detail in the article on the context window and its importance for modern LLM architectures.

For example, when an 8B parameter model occupies 5.5 GB VRAM at 4-bit precision, an uncompressed FP16 KV cache at a context of 64k tokens can easily require an additional 8 to 12 GB of dynamic memory on top of that. Without sufficient free VRAM space, this inevitably leads to out-of-memory errors.

The formats show different solutions here:

• GGUF (llama.cpp): Offers native support for quantized KV caches at Q8_0 and Q4_0 precision (via the -ctk and -ctv flags). This halves the memory footprint of long contexts with minimal quality loss.
• AWQ (within vLLM): Makes use of PagedAttention and supports FP8- or INT8-quantized KV caches on supported GPU architectures (Ada Lovelace, Hopper).
• EXL2 (ExLlamaV2): Supports native 4-bit, 6-bit, and 8-bit KV cache allocation directly from the engine configuration, keeping long contexts manageable on consumer cards.

Production, serving, and ecosystem integration

In addition to pure computational performance, tooling, deployment ease, and ecosystem support play a decisive role in IT architecture choices.

GGUF is the undisputed standard for desktop and edge applications. The single-file format makes distribution simple: one file contains the architecture, layers, metadata, and chat template. Runtimes such as Ollama, LM Studio, Jan, and LocalAI rely almost exclusively on GGUF. For developers building local assistants, internal office chatbots, or embedded systems, GGUF minimizes configuration complexity to a minimum.

AWQ is the standard choice for cloud-native and enterprise environments. Because AWQ models are stored in the standard Hugging Face Safetensors structure with explicit configuration files, they integrate directly with professional serving frameworks such as vLLM, TensorRT-LLM, and Text Generation Inference (TGI). Monitoring via Prometheus, dynamic batching, and distributed serving across Kubernetes clusters work flawlessly with AWQ.

EXL2 occupies a specialized niche. It is immensely popular among self-hosters, enthusiasts, and SME developers who want to squeeze maximum performance out of a fixed set of Nvidia RTX 3090 or 4090 GPUs. Integration with server tooling such as TabbyAPI makes it possible to serve OpenAI-compatible API endpoints with extremely low latency, although support in large enterprise orchestrators lags behind AWQ.

# Voorbeeld van llama.cpp server met GGUF en 8-bit KV-cache compressie
./llama-server \
  -m models/Meta-Llama-3.1-8B-Instruct-Q5_K_M.gguf \
  -c 32768 \
  -ngl 99 \
  -ctk q8_0 \
  -ctv q8_0 \
  --host 0.0.0.0 \
  --port 8080

Decision tree: which format do you choose when?

To make an informed choice between GGUF, AWQ, and EXL2, the following operational criteria can be applied:

Choose GGUF if:

• The target hardware consists of Apple Silicon (M1/M2/M3/M4 Macs) or systems without a powerful dedicated Nvidia GPU.
• The model is larger than the available VRAM, making partial offloading to system RAM (CPU memory) necessary.
• Deployment takes place via desktop applications such as Ollama or LM Studio.
• Simple distribution via a single file is required without complex Python dependencies.

Choose AWQ if:

• The model runs on dedicated Nvidia datacenter GPUs (such as A10, A100, L40S, H100).
• There is a multi-user production environment with high concurrency where frameworks such as vLLM or TGI are deployed.
• The model must be split across multiple GPUs via Tensor Parallelism.
• Full integration with the standard Hugging Face ecosystem and automated CI/CD pipelines is desired.

Choose EXL2 if:

• The workload runs on a fixed setup of consumer GPUs (such as 1x, 2x, or 4x RTX 3090/4090).
• Single-user latency and maximum token speed per second take priority over heavy concurrent batching.
• A model just barely does not fit into VRAM with standard 4-bit or 5-bit quantization, making an intermediate precision (such as 3.5 or 4.25 bits) the only way to prevent OOM errors.

Quantization in production: operational pitfalls

When deploying quantized models in production environments, teams must be alert to a number of structural pitfalls. First, 'small language models' (up to 8 billion parameters) respond much more sensitively to precision loss than models of 70 billion parameters. Quantizing an 8B model to below 4 bits regularly leads to syntax errors in structured JSON output or logical breaks in reasoning steps.

Second, the calibration dataset used during the quantization process must be taken into account. Both AWQ and EXL2 rely on calibration sets to determine the scaling factors of weights. When a model is calibrated exclusively on English-language texts, the perplexity on specialized Dutch legal documents or complex source code can increase disproportionately.

Finally, the combination of quantization and long contexts requires constant monitoring. A system that runs stably with short prompts can fail abruptly as soon as users submit documents of tens of thousands of tokens and the dynamic KV cache fully claims the remaining video memory. A robust architecture should therefore always reserve at least 20% to 30% of the total VRAM as a buffer for context and batch expansion.