inferwire
/
AI·4 min read

Why Your Local LLM Feels Dumber: Precision Tests on Qwen3.6-27B

Tests on Qwen3.6-27B show that the attention backend, KV-cache quantization and 4-bit weights all shift a local model's output, and some setups broke tool calls on a 100k-token agent task.

TL;DR

  • In a Level1Techs forum post, user thr3e found that changing only the attention backend made Qwen3.6-27B pick different next tokens late in long prompts [^1].
  • Quantizing the KV cache to int4, or the weights to 4 bits, broke tool calls on a 100,000-token agent task; 8-bit setups got them right [^1].

Background

Open-weight models often run at home quantized, with weights or caches stored at lower precision to save memory. When such a setup disappoints, the model may not be the problem. Every combination of hardware and software computes the next token a little differently, even with the same weights; the nightly vLLM container for these tests held 734 packages [^1]. One way to measure the difference is comparing logits, the scores a model gives each possible next token, against a baseline.

What happened

The tests ran the official BF16 checkpoint of Qwen3.6-27B on an RTX PRO 6000 Blackwell GPU with a pinned nightly build of vLLM [^1]. The workload, "Prompt 2", was a context of roughly 100,000 tokens captured from a real lab workstream with multiple tool calls, which doesn't appear in any benchmark or training set. Logits were recorded every 32 prompt tokens [^1]. The post uses two measures: KL divergence, how far the probability distribution has moved from the baseline, and top-1 agreement, whether the highest-scoring token is the same. Lower divergence means closer to the baseline, not automatically smarter [^1].

In the first test, only the attention backend changed: FlashAttention 2, FlashInfer or Triton [^1]. For the first several thousand tokens all three agreed on every next token; later in the prompt they began to disagree. Repeated runs on the same backend produced bit-for-bit identical logits, so the differences came from each backend's arithmetic, not from random noise [^1]. The disagreements came in clusters that varied with the content of the prompt rather than rising smoothly with its length [^1].

The second test kept the weights at BF16 and quantized only the KV cache. That produced a reproducible tool-calling error: with a BF16 cache the model was fine, with int8 it eventually recovered, with int4 it did not [^1]. The third test compared four quantized versions of the weights against the BF16 reference. A community INT8 W8A16 version tracked the reference most closely, ahead of Qwen's official FP8 release. NVIDIA's NVFP4 version came last, with about 50 percent of next tokens flipped by 88,000 tokens of context, although vLLM ran it with a weight-only fallback rather than native FP4 arithmetic on this setup [^1]. The NVFP4 and AWQ 4-bit versions failed to close their tool calls properly and ran the wrong Cisco command, "show run" instead of "show arp"; the FP8 and INT8 versions made the correct calls [^1].

Why it matters

A model that seems worse at home than in its maker's benchmarks may be running a different computation rather than being a worse model [^1]. In these tests the effects showed up late in long contexts and during tool calls, which is where local agents spend their time, and quick checks miss them: the post warns against setting the temperature to zero, pasting in three test prompts and calling the model good or bad [^1]. It also cautions against impossibly low divergence figures on quantized model cards, which mean little unless the author discloses the reference checkpoint, the evaluation text, the context lengths and how the measurements were aggregated [^1].

The INT8 result has a partial explanation: that version keeps activations at 16 bits and leaves the model's linear-attention projections unquantized [^1]. The post is part one of a series, and more experiments are planned [^1].

Practical example

Say you run Qwen3.6-27B locally as a coding agent and it starts mangling tool calls deep into long sessions. Before blaming the model, you check the sampler settings and chat template on the model card, since a temperature set too low can leave Qwen looping in its thinking output [^1]. Then you check whether your KV cache is quantized to 4 bits and which quantized weights you downloaded. In these tests, a BF16 or int8 cache and 8-bit weights were the setups that got the tool calls right [^1].

Related gear

We recommend this book because it explains the exact inner mechanics of tokenizers, context windows, and sampling parameters that dictate local model performance.

AdvertisementAmazon

Build a Large Language Model (From Scratch)

Sources

  1. [1]Level1Techs Forums — Why your local LLM feels dumber than it is
  2. [2]Hacker News — Why your local LLM feels dumber than it is