A Conditional No-Hair Theorem for Transformers: How Language Models Forget Their Prompt
Abstract
Autoregressive transformers retain prompt influence via a power-law decay driven by attention dilution and relay mechanisms, with trained models forgetting faster than untrained ones.
We ask how much an autoregressive transformer retains of its prompt after generating t tokens. Consider two different prompts followed by the same continuation text, so that any difference in their hidden states comes from the prompt alone, and we find that the distance between their hidden states decays approximately as a power law t^{-α}, where α=1 is the rate at which a fixed prompt's share of uniform attention shrinks and smaller values mean longer memory. Trained models forget fast (αapprox0.55--0.7), untrained ones slowly (αapprox0.2--0.5). We prove that this offset is Obig(P(log t)^{L-1}/tbig) for a prompt of length P and depth L. The bound holds for finite-depth pre-LN softmax transformers with bounded attention logits, under one additional condition: the pre-norm activations stay away from zero. The mechanism is geometric: the prompt's direct influence is diluted as the context grows, and relay --- continuation tokens passing on what earlier tokens read from the prompt --- can only add logarithmic factors. This is what we call the conditional no-hair theorem: given the text generated so far, the prompt's influence on the next token vanishes as t grows, so any property of the prompt that survives, such as its language, must be carried by the text itself. An exactly solvable linear relay model shows that the bound cannot be improved and reproduces the rise of the local exponent towards 1 seen in untrained networks. Cancelling the dilution with a logit bias stops the decay early in training and reverses it after, so dilution is necessary for the forgetting we measure; untrained networks and the solvable model show it is also sufficient.
Community
We ask how much an autoregressive transformer retains of its prompt after generating t tokens. Consider two different prompts followed by the same continuation text, so that any difference in their hidden states comes from the prompt alone, and we find that the distance between their hidden states decays approximately as a power law t−α, where α=1 is the rate at which a fixed prompt's share of uniform attention shrinks and smaller values mean longer memory. Trained models forget fast (α≈0.55--0.7), untrained ones slowly (α≈0.2--0.5). We prove that this offset is O(P(logt)L−1/t) for a prompt of length P and depth L. The bound holds for finite-depth pre-LN softmax transformers with bounded attention logits, under one additional condition: the pre-norm activations stay away from zero. The mechanism is geometric: the prompt's direct influence is diluted as the context grows, and \emph{relay} --- continuation tokens passing on what earlier tokens read from the prompt --- can only add logarithmic factors. This is what we call the \emph{conditional no-hair theorem}: given the text generated so far, the prompt's influence on the next token vanishes as t grows, so any property of the prompt that survives, such as its language, must be carried by the text itself. An exactly solvable linear relay model shows that the bound cannot be improved and reproduces the rise of the local exponent towards 1 seen in untrained networks. Cancelling the dilution with a logit bias stops the decay early in training and reverses it after, so dilution is necessary for the forgetting we measure; untrained networks and the solvable model show it is also sufficient.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Hybrid Latent Attention for Looped Language Models (2026)
- Emergent Structure in the Marginal Attention Space of Language Models (2026)
- Language Models Share a Readout Bottleneck in Working Memory (2026)
- Where Prefill Expert Reduction Hurts MoE Models: A Few Prompt Tokens Carry Most of the Short-Answer F1 Loss (2026)
- Representation Drift in Continual Learning: LoRA versus N-gram Memory (2026)
- The Residual Stream's Effective Depth (2026)
- Underscoring the Problem: Why Softpick Fails at Initialization (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Very interesting work! We have performed a closely related study, however, using autograd to to measure the sensitivity of output logits to earlier tokens.
We also observe a power-law-type decay of token influence, with exponents typically around 0.6 to 0.9, across different language models (Pythia, Qwen2.5, GPT-Neo, Llama 3.1, Gemma 3, SmolLM2 and Mistral families, for 14M to 32B).
Your approach and ours differ:
We measure output-logit sensitivities the embeddings of individual token by taking the norm of the gradient (infinitesimal changes). On the other hand, you investigate differences in hidden states induced by finite changes in the prompt. These quantities are mathematically related but not the same.
One interesting difference concerns random initialization. For this case we did not observe power-law profiles, whereas your results do suggest power-law-like decay even in untrained models. This might be related to the different observables and aggregation methods.
It would be interesting to compare both approaches on the same models and inputs. We would be very happy to exchange ideas!
Here is the link to our paper:
Brändel, Köhler, and Rheinbach, "How Token Influence Decays with Causal Distance: Evidence from Autoregressive Language Models from 14M to 32B Parameters", Machine Learning: Science and Technology (Oct 2026), DOI: 10.1088/2632-2153/aebb8c (preprint: arXiv:2606.29139, June 2026).
Here is the diagram from our paper which shows the exponents in the power-law-type decay for different LLMs (Fig. 3 in Brändel, Köhler, and Rheinbach, MLST 2026). DOI: 10.1088/2632-2153/aebb8c, CC BY 4.0
Get this paper in your agent:
hf papers read 2610.24133 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper