Post-Training Quantization for LLMs: Guide to 8-Bit and 4-Bit Methods

Running a 70-billion parameter Large Language Model (LLM) on a single GPU used to be a pipe dream. But Post-Training Quantization is a technique that reduces the numerical precision of pre-trained model parameters after training, allowing models to run with significantly less memory and higher speed without retraining. It’s the secret sauce behind why you can now chat with massive AI models on consumer hardware or cut your cloud inference bills in half. If you’re dealing with high latency or memory bottlenecks, this is likely the first lever you should pull.

The core problem is simple: modern LLMs are huge. A 7B parameter model in standard FP16 format needs about 14GB of VRAM. A 70B model? That’s over 140GB. Most GPUs choke there. Quantization is the process of mapping high-precision floating-point numbers to lower-precision integers like INT8 or INT4. Post-Training Quantization (PTQ) specifically lets you do this *after* the model is trained. You don’t need the original training data. You don’t need weeks of compute. You just need a small calibration set and some clever math.

Why Precision Matters Less Than You Think

Neural networks are surprisingly robust to noise. When we train them in FP16 (16-bit floating point), we assume every bit matters. In reality, many weights and activations have distributions that allow us to throw away precision with minimal impact on output quality. The challenge isn’t just reducing bits; it’s doing so without breaking the model’s ability to understand context.

This is where the two main approaches come in: 8-bit and 4-bit quantization. They serve different purposes and trade off accuracy differently. Understanding when to use which is critical for your deployment strategy.

8-Bit Quantization: The Safe Bet

8-bit quantization is a method that maps weights and/or activations to 8-bit integers (INT8), typically resulting in a 2x memory reduction compared to FP16. This is the "safe" option. For most general-purpose tasks, the accuracy drop is negligible-often less than 0.5% on standard benchmarks like MMLU.

The biggest hurdle for 8-bit quantization has historically been activation outliers. In models like OPT-13B, some activation values spike way higher than others (sometimes exceeding magnitude 70). These spikes make it hard to scale the entire tensor to fit into 8 bits without losing information in the smaller values. Enter SmoothQuant is a mathematical technique developed by MIT-IBM Watson AI Lab that migrates quantization difficulty from activations to weights by applying channel-wise scaling. By smoothing out those activation spikes, SmoothQuant makes W8A8 (8-bit weights, 8-bit activations) feasible. It’s widely supported in frameworks like NVIDIA TensorRT and Hugging Face Optimum.

4-Bit Quantization: Pushing the Limits

If 8-bit saves you 50% of memory, 4-bit quantization is a more aggressive compression technique that maps parameters to 4-bit integers, reducing memory usage by up to 4x compared to FP16. This is where things get interesting. You can fit a 70B model into roughly 35-40GB of VRAM, making it viable for high-end consumer cards or single-node enterprise servers.

However, 4-bit is riskier. Naive quantization at this level often causes significant accuracy drops (5-10%). To fix this, methods like AWQ (Activation-aware Weight Quantization) is a method that identifies and preserves 'salient weights' aligned with high-magnitude activations while aggressively quantizing less important ones. AWQ doesn’t treat all weights equally. It looks at which weights actually matter during inference and keeps those at higher precision (or applies better scaling) while crushing the rest. Another popular approach is GPTQ is an efficient second-order quantization algorithm that uses Hessian information to minimize error during weight rounding. GPTQ is generally considered superior for 4-bit weights but requires more careful calibration.

Comparing the Methods: Which One Should You Pick?

Choosing between these methods depends on your hardware constraints and tolerance for accuracy loss. Here’s a quick breakdown:

Note that hybrid approaches are becoming the new standard. By combining SmoothQuant’s handling of activations with AWQ or GPTQ’s weight preservation, you can achieve 4-bit weight efficiency with near-FP16 accuracy. This is particularly powerful if you’re using hardware that supports mixed-precision inference well.

Implementation: What You Actually Need to Do

You don’t need to write C++ kernels to get started. Most major libraries handle the heavy lifting. Here’s the typical workflow:

1. Select Your Library: For PyTorch users, Hugging Face Optimum is a library providing wrappers for various quantization techniques including SmoothQuant, AWQ, and GPTQ. For production inference, vLLM is a high-throughput serving engine optimized for quantized LLMs. NVIDIA users might look at TensorRT Model Optimizer.
2. Prepare Calibration Data: You need a small dataset that represents your actual use case. Research suggests 128-256 samples are sufficient for 7B-13B models. For larger 70B+ models, aim for 512+ samples. Using fewer than 64 samples risks a 3-5% accuracy drop.
3. Run the Quantization Script: Most tools provide a simple command-line interface or Python API. You specify the base model, the target bit width, and the calibration dataset path.
4. Benchmark: Don’t trust the paper alone. Run your specific test cases. Measure both latency (tokens per second) and accuracy (perplexity or task-specific metrics).

A common pitfall is forgetting to check long-context performance. As Dr. Tim Dettmers noted, 4-bit quantization can introduce error accumulation in long-context generation, particularly beyond 4K tokens. If your app relies on reading whole documents, test with longer inputs.

Hardware Considerations and Real-World Performance

The benefits of PTQ aren’t just theoretical. On a consumer-grade NVIDIA RTX 4090 (24GB VRAM), a 4-bit quantized LLaMA-70B model can achieve 18-22 tokens per second. The same model in FP16 would either not fit or run at under 5 tokens per second. That’s a massive difference for user experience.

In enterprise settings, the savings are financial. Reducing memory footprint by 4x means you can serve four times as many concurrent requests on the same GPU cluster. According to recent industry surveys, organizations adopting PTQ report 40-60% reductions in inference costs. With the LLM quantization market projected to grow significantly through 2028, this cost efficiency is driving rapid adoption among enterprises.

Frequently Asked Questions