inferwire
/
AI·4 min read

GSQ: Simple Scalar Quantization Nears the 2-Bit Accuracy Frontier

Researchers from ISTA, ETH Zürich and Red Hat AI use Gumbel-Softmax sampling to learn how each weight is rounded. At 2 bits, Llama-3.1-70B keeps most of its accuracy in a standard format.

TL;DR

  • GSQ learns how to round each LLM weight and beats other simple scalar quantizers at 2 and 3 bits per parameter [^1].
  • Its output runs on existing scalar inference kernels, and it compressed the trillion-parameter Kimi-K2.5 to 2 bits, where vector methods are hard to apply [^1].

Background

Quantization stores a model's weights with fewer bits; local setups now routinely run models at 2 to 3 bits per parameter [^1]. Simple scalar methods such as GPTQ, which brought 175-billion-parameter models to 3 or 4 bits in about four GPU hours, round each weight to a small grid [^2]. The GSQ authors say these hit an error wall around 3 to 4 bits, while vector and trellis methods like QTIP do better below that but are hard to scale [^1].

What happened

The new method, from ISTA, ETH Zürich and Red Hat AI, keeps the simple format: symmetric weights from a small uniform grid, one shared scale per group of 128 weights and no zero-points [^1]. What changes is how each weight picks its grid point. GSQ gives every weight a few trainable scores, one per grid level, and uses Gumbel-Softmax sampling, which turns a discrete choice into a smooth one that gradients can optimize [^1]. Scores and scales are trained together so that each quantized block reproduces the original block's output, and as a temperature setting is lowered, the soft choices settle on hard grid points [^1]. At 2 bits each weight has only four levels, which keeps the problem manageable; at higher bit-widths GSQ considers only a few nearby points [^1].

On Llama-3.1-70B-Instruct at 2 bits, GSQ averaged 75.57 across five zero-shot benchmarks, against 78.99 for the 16-bit model, 71.43 for EfficientQAT, the best scalar baseline, and 57.38 for GPTQ [^1]. That puts it 4.14 points above the best scalar method and 1.68 and 0.70 points behind the vector methods QTIP and PV-Tuning [^1]. On the 8B model, a ternary GSQ version at 1.58 bits beat 2-bit GPTQ and QuIP by more than 20 points and nearly matched 2-bit EfficientQAT [^1]. Served with vLLM and Humming kernels on NVIDIA L40s GPUs, the 2-bit 70B model ran up to 6.2 times faster per GPU than BF16 [^1].

The team also applied GSQ to Kimi-K2.5, a mixture-of-experts model with 1T parameters that ships in 4-bit, and brought its expert weights down to 2 bits [^1]. LiveCodeBench v6 rose from 61.37 to 69.37 and MATH500 from 96.68 to 97.32, while AIME 25 slipped from 95.33 to 93.00 and GPQA Diamond fell from 89.29 to 76.57 [^1]. The authors attribute that drop to their calibration data, OpenThoughts, which is heavily skewed toward math and code [^1]. On long-context retrieval, the 2-bit model matched or beat the original up to 32k tokens but slipped slightly beyond, for example from 46.04 to 44.91 in the 128k to 256k range [^1].

Why it matters

The authors' central claim is that the accuracy gap between scalar and vector quantization is mostly an optimization gap, not a limit of the format [^1]. That matters because scalar formats are what people actually deploy: they have by far the broadest adoption, while vector and trellis formats gave only very small decoding speedups over BF16 because of their complexity [^1]. A method that raises accuracy inside the scalar format works as a drop-in replacement and needs no new kernels [^1].

The paper also shows the limits. GSQ is calibration-based and expensive to run: its extra trainable scores take 2 to 5 times the memory of the weights being quantized, and the experiments ran mainly on nodes with eight H200 or B300 GPUs [^1]. Each configuration was run only once because of that cost, and the Kimi-K2.5 results compare GSQ only with the original model, not with other quantizers [^1]. The GPQA drop is a reminder the authors draw themselves: what goes into calibration shapes what the small model keeps [^1].

Practical example

Say Lena runs a small analytics firm that hosts Llama-3.1-70B-Instruct on its own GPUs and wants to cut serving costs by dropping to 2 bits per parameter. With plain GPTQ at that setting, the paper measured an average of 57.38 against 78.99 for the full model, a loss the firm's users would notice [^1].

Lena's team quantizes the model with GSQ instead. The weights end up in the same simple scalar format, so the firm can keep serving them with vLLM and a scalar kernel; in the paper's L40s setup the 2-bit model ran up to 6.2 times faster per GPU than BF16, at an average of 75.57 [^1]. The quantization run itself is the heavy part, since the paper's experiments used nodes with eight H200 or B300 GPUs [^1].

The catch is the calibration data. Lena's firm mostly answers chemistry and biology questions for clients. If the team calibrates on a math-and-code set, as the authors did for Kimi-K2.5, the science scores can sag the way GPQA Diamond did, from 89.29 to 76.57 [^1]. So the team calibrates on text like the firm's real workload and checks the quantized model on the firm's own questions before switching over.

Related gear

We recommend this book because it covers the gradient-based optimization and probability basics that GSQ uses to learn how each weight is rounded.

AdvertisementAmazon

Deep Learning (Adaptive Computation and Machine Learning series)

Sources

  1. [1]arXiv — GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling
  2. [2]arXiv — GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers