Image: Lin, Tang, Tang, Yang, and colleagues, AWQ paper, 2024. Official photo of a 4-bit LLaMA-7B chat model running on a laptop.
Open-weight models are no longer stuck in the cloud. In October 2026, this site reported that Black Forest Labs released FLUX 3 Image Klein with smaller FP8, INT8, and NVFP4 builds made with NVIDIA. The company said the INT8 version can run on GeForce RTX cards with as little as 8GB of video memory, if those compressed weights and extra tricks are used.
That is the problem quantization is meant to solve. A model is a huge list of numbers, called weights. Memory use is roughly the number of parameters times the bits used to store each one. Hugging Face's Transformers documentation says models are usually loaded in bfloat16, which takes 2 bytes per parameter, so an 8 billion parameter model needs about 16GB for its weights, about 8GB at 8 bits, and about 4 to 5GB at 4 bits. The SmoothQuant paper notes that GPT-3, with 175 billion parameters, consumes at least 350GB if stored in 16-bit floating point.
Quantization shrinks that bill by rounding. Imagine a shop's price list kept to the nearest paisa, then rewritten to whole rupees. Most items would still be in the right order, but a few odd prices would move. Neural network weights are those prices. Training usually produces 16-bit floating-point values, often called FP16 or BF16. An 8-bit integer format, often called INT8, keeps 256 possible values. A 4-bit integer keeps only 16. Methods typically set a scale, and sometimes a zero point, for a layer or a small block of weights, then snap each number to the nearest allowed step. Most downloads use post-training quantization, which compresses an already trained model without rerunning the original training job.
The hard part is the rare, very large weight. In 2022, Tim Dettmers and colleagues described systematic outlier features in a paper titled LLM.int8. Those outliers, they found, appear across transformer layers once models reach about 6.7 billion parameters, and they wreck naive 8-bit rounding. Their method keeps more than 99.9 percent of the values in 8-bit multiplication and isolates the outlier dimensions in 16-bit mixed precision. With that split, they reported that a 175 billion parameter 16 or 32-bit checkpoint could be converted to Int8 and used at once without performance degradation, cutting inference memory by half and making models such as OPT-175B and BLOOM runnable on a single server with consumer GPUs.
Eight-bit weights were soon not the only target. In GPTQ, Elias Frantar and colleagues introduced a one-shot, weight-only method that uses approximate second-order information to decide how to round. They reported that GPTQ could quantize 175 billion parameter models in about four GPU hours down to 3 or 4 bits per weight, with negligible accuracy degradation relative to the uncompressed baseline, and that a 175 billion parameter model could then run inside a single GPU for generative inference for the first time. They also reported speedups over FP16 of around 3.25 times on an NVIDIA A100 and 4.5 times on an NVIDIA A6000. For 2-bit or even ternary weights, they claimed only reasonable accuracy, a weaker result than at 3 and 4 bits.
SmoothQuant, from Guangxuan Xiao, Ji Lin, and colleagues, focused on activations. Weights are relatively easy to quantize, they argued, but activations are not, because of the same large-magnitude outliers that appear beyond about 6.7 billion parameters. The method migrates some of that difficulty from activations to weights with a mathematically equivalent transformation, so both can be stored as 8-bit integers, a scheme known as W8A8. The authors reported up to 1.56 times speedup and 2 times memory reduction with negligible loss in accuracy, and said the approach can serve a 530 billion parameter model within a single node.
Four-bit is now where many users stop, because the memory saving is large and the quality loss is often still small. AWQ, or Activation-aware Weight Quantization, from Ji Lin, Jiaming Tang, and colleagues, is a weight-only method built around that tradeoff. The authors found that protecting only 1 percent of salient weights can greatly reduce quantization error. Those important channels are identified from activation statistics, then scaled so rounding damages them less. They also reported more than a 3 times speedup over Hugging Face's FP16 implementation on desktop and mobile GPUs, 30 tokens per second for a 13 billion parameter model on a laptop RTX 4070 with 8GB of memory, and deployment of Llama-2 70B on a mobile GPU.
A 2024 empirical study of LLaMA3 quantization by Huang, Zheng, Ma, Qin, and colleagues found that 4-bit post-training methods had a slight performance decrease of about 2 percent compared with the original 16-bit model. At 3 bits, simple round-to-nearest quantization faced substantial losses, more than 10 percent lower than 4 bits, while GPTQ, AWQ, and some later methods stayed within 5 percent of their 4-bit results. At 2 bits, many of the same methods collapsed. On LLaMA3-8B, WikiText2 perplexity was 6.1 for the 16-bit model, 6.5 for GPTQ at 4 bits, 8.2 at 3 bits, and 210 at 2 bits. AWQ was similar at 4 and 3 bits and blew up at 2 bits. When both weights and activations were quantized with SmoothQuant, 8-bit stayed close to the original, and 4-bit collapsed on the 8 billion parameter model.
QLoRA is a different use of 4-bit weights. Tim Dettmers and colleagues showed that a frozen 4-bit model can still be fine-tuned by sending gradients into a small set of Low Rank Adapters, a method known as LoRA. That cut the memory needed to finetune a 65 billion parameter model from more than 780GB to less than 48GB, enough for a single 48GB GPU, without degrading runtime or predictive performance compared with a 16-bit fully finetuned baseline, they reported. The work introduced 4-bit NormalFloat, or NF4, a data type they describe as information-theoretically optimal for normally distributed weights, plus Double Quantization, which compresses the quantization constants themselves. Their Guanaco models reached 99.3 percent of ChatGPT's level on the Vicuna benchmark after 24 hours of finetuning on a single GPU, according to the paper, which also warns that current chatbot benchmarks are not trustworthy.
Going below 4 bits without a collapse usually means changing how the model is trained, not only how a finished file is stored. In a 2024 paper, Shuming Ma, Hongyu Wang, and colleagues introduced BitNet b1.58, in which every weight is ternary, taking only the values -1, 0, or 1, or about 1.58 bits. Unlike GPTQ or AWQ, it is trained from scratch, with 1.58-bit weights and 8-bit activations. The authors reported that it matches a full-precision FP16 or BF16 Transformer of the same size and training token count. At 3 billion parameters, their BitNet was 2.71 times faster and used 3.55 times less GPU memory than their LLaMA-style FP16 baseline, with slightly better perplexity, 9.91 against 10.04, after training on the RedPajama dataset for 100 billion tokens. It is a recipe for new models, not a switch for an existing 16-bit download.
What gets lost depends on the bit width, the method, and the model. The LLaMA3 study tests a newer open model that Meta had pretrained on more than 15 trillion tokens. The authors write that LLaMA3 still suffers from non-negligible degradation, particularly under ultra-low bit widths, and that a 2-bit LLaMA3-based multimodal model could fail to produce usable answers, with repetitive character strings. They say the degradation can lead to larger declines, and they tie that to LLaMA3's heavy pre-training, compared with earlier LLaMA and LLaMA2 models. They also found that low-rank fine-tuning on the small Alpaca dataset could not make up for quantization loss on LLaMA3 the way 4-bit LoRA fine-tunes sometimes could for those earlier models. The 4-bit QLoRA version of LLaMA3-8B scored 57.0 on MMLU, against 64.8 for the 16-bit model. The paper does not prove that newer models trained on more data always lose more when quantized. Some of its 4-bit results stay close to the 16-bit baseline, but 2-bit and some 3-bit settings can fail badly.
For people who download weights, the names on the files are now part of the problem. A common package is GGUF, a single-file format designed to store a model for inference with GGML and tools built on it, including llama.cpp. The GGUF specification describes it as a successor to earlier GGML file types. Hugging Face's Transformers documentation treats GGUF as a single file that holds the weights, config, and tokenizer. llama.cpp, at the time of this writing, lists integer quantization at 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit, aimed at running large models locally on consumer GPUs, Apple silicon, and CPUs.
The labels inside those files follow a compact code. In the project's quantize documentation, Q4_K_M is listed at 4.58G with a perplexity increase of 0.1754 on Llama-3-8B, and Q4_K is an alias for Q4_K_M. The same listing puts Q8_0 at 7.96G with a 0.0026 perplexity increase, Q5_K_M at 5.33G and 0.0569, Q3_K_M at 3.74G and 0.6569, and Q2_K at 2.96G and 3.5199. A table in the same documentation, measured on Llama-3.1-8B, gives Q4_K_M as 4.8944 bits per weight and 4.58 GiB, against 16.0005 bits per weight and 14.96 GiB for 16-bit floating point. For larger models, the tool lists a 70B Q4_K_M file at 43.1 GB, against an original size of 280.9 GB.
The letters in a name like Q4_K_M come from the k-quants design merged into llama.cpp, in a change still cited by the official quantize documentation. Q4 is the nominal bit width. K marks the k-quant family, which groups weights into super-blocks and stores scales, and sometimes minimums, in compact form. S, M, and L are mixes. Q4_K_S uses 4-bit k-quant tensors throughout. Q4_K_M, the medium mix, keeps some more sensitive tensors at higher precision. In the original design, that meant using 6-bit k-quant for half of the attention value and feed-forward down-projection tensors, and 4-bit k-quant for the rest. Q4_K_M is therefore a 4-bit GGUF file with a medium mix, often the default for a balance of size and quality. The documentation warns that requantizing an already quantized file can severely reduce quality, and it offers an importance-matrix option, called imatrix, to reduce error.
If a model barely fits, a 3-bit or 2-bit file may be the only option, but the official perplexity jumps and the LLaMA3 study's 2-bit collapses are a reminder of the cost. If memory allows, Q5_K_M or Q6_K stays closer to the 16-bit original, and Q8_0 is very close. Q4_K_M sits in the middle, which is why it is widely used. AWQ and GPTQ files are also usually 4-bit, but they are not the same format as a GGUF k-quant, and a program that loads one will not necessarily load the other.
Labs often report scores for a 16-bit, or occasionally 8-bit, checkpoint running in a controlled setup. A laptop download is frequently a 4-bit GGUF or AWQ file produced later, sometimes requantized or run with a different setup. Hugging Face notes that fewer bits cost some accuracy, and that each method makes a different tradeoff among memory, speed, and accuracy. NVIDIA's NVFP4 build of MiniMax's M3, covered here in June 2026, scored 91.92 on GPQA Diamond against 92.53 for the FP8 baseline, a small official gap on one test, while cutting disk and GPU memory use by about half. Liquid AI later said its 1.7 billion parameter LFM3 text model, also covered here, fits in about 1GB at 4-bit precision and ships GGUF files for tools such as llama.cpp.
A quantized model can therefore be the same family and still not be the model in the benchmark table. Eight-bit and carefully made 4-bit files often stay close. Two-bit and some 3-bit files can lose a lot, especially on already strong, heavily trained models and on harder multimodal work. The useful habit, when a file is named Q4_K_M, 4-bit, GGUF, or AWQ, is to read those labels as a memory choice, then test the actual file on the work you care about, rather than assuming it will match the cloud version.
Comments