Paper Argues Transformers Build Prompt-Specific Transformations
Two researchers argue against the "stochastic parrot" view: Transformers, they say, generate transformations from the prompt during inference, and this share grows with prompt length.
TL;DR
- Two researchers argue that Transformers build prompt-specific transformations during inference, rejecting the view that LLMs only reproduce training statistics [^1].
- They say this dynamic share grows with prompt length and can match fixed weights, which bears on interpretability, control and smaller systems [^1].
Background
The "stochastic parrot" view holds that large language models merely reproduce statistical regularities learned in training [^1]. These models are Transformers, an architecture built on attention [^2]. An attention function computes each output as a weighted sum of values, and the weight given to each value comes from comparing a query with a key [^2]. In self-attention, the queries, keys and values all come from the output of the previous layer, so these weights depend on the input, not only on training [^2].
What happened
Marco Giunti and Fabrizia Giulia Garavaglia posted "The Transformer Revolution, Part 1" on arXiv on August 4 [^1]. The paper offers a new interpretation of what a Transformer does during inference. Against the stochastic parrot view, the authors argue that Transformers construct and apply prompt-dependent transformations whose parameters are generated during inference. They call this form of computation SIDPP, short for Sequence-level Interactive Dynamic Parallel Processing [^1]. In their picture, the Transformer "transforms concepts by means of concepts": token vectors are the concepts being transformed, and the matrices and vectors that define transformations are the transforming concepts. These are static when training fixes them and dynamic when they are generated from the input sequence. Mechanically, they correspond to groups of simple neural networks [^1].
The authors locate the Transformer's novelty in what they call output-weight interconnections. Alongside ordinary connections, where one network's output becomes another's input, the outputs of some networks determine the weights of others [^1]. Through these links, the system builds transformations from the prompt and uses them to change token representations. According to the paper, the contribution of this dynamic processing grows with prompt length and may equal or exceed that of static processing, which the authors call strong prompt sensitivity [^1]. They go further and conjecture that human language processing may itself be a form of SIDPP, since the human nervous system has the mechanisms needed to implement it [^1].
Why it matters
The paper is an interpretation rather than a benchmark study, and its abstract reports no measured results [^1]. Its value lies in the framing. If, as the authors argue, a long prompt can contribute as much to the computation as the trained weights, then studying the weights alone explains only part of what a model does with a given input. The authors themselves say the account bears on interpretability, predictability and control, and on designing smaller, more sustainable systems [^1].
The claims differ in strength. The description of output-weight interconnections is a reading of the architecture, while the link to the cerebral cortex is explicitly a conjecture, which the authors say may hold "in principle" [^1]. The title marks it as the first part of a larger work.
Practical example
Say you maintain the prompt for a support chatbot and paste three pages of new policy text into it. On the authors' account, a longer prompt shifts more of the model's processing to transformations built from that prompt, and their share can equal or exceed that of the fixed weights [^1]. In practice, you'd treat a big prompt edit like a code change: you rerun your usual test questions after each edit instead of assuming the model will answer the way it did before.
Related gear
We recommend this book because it provides a balanced, expert perspective on whether AI models truly understand the world or are simply sophisticated pattern matchers.