Language Models are Injective and Hence Invertible

TL;DR

Everyone “knows” Transformers throw information away: softmax, LayerNorm, and many-to-one attention all look lossy, so people assume you can’t recover the original prompt from a model’s internal activations. This paper proves the opposite. For standard decoder-only LMs, the map from a prompt to its last-token hidden vector is injective almost surely — distinct prompts give distinct vectors — and this holds at random initialization and stays true through any finite amount of gradient-descent training. They back the theorem with ~5 billion collision tests on six real models (GPT-2, Gemma-3, Llama-3.1, Mistral, Phi-4, TinyStories) and find zero collisions. Then they ship SipIt, an algorithm that exactly reconstructs the input text from hidden states in provably linear time — recovering 20-token GPT-2 prompts with 100% accuracy in ~28 seconds versus a baseline that takes over an hour and fails. The practical punchline: hidden states are recoverable user data, with real privacy, deletion, and compliance consequences.

Problem & Motivation

Here’s the concrete pain. You build an agentic product. To make it fast and observable you cache hidden states (KV-cache), you log intermediate activations for debugging, you ship embeddings to a vector store, maybe you fine-tune a probe on layer-12 representations. The prevailing intuition — and the basis of at least one regulator’s position — is that these internal vectors are abstractions: lossy, scrambled, not “the text.” LayerNorm divides away the mean and scale. Softmax attention is many-to-one. Residual streams can cancel. Stack 12–80 of these and surely the original tokens are gone.

If that intuition were true, two things follow: (1) you could treat stored activations as safe-ish, de-identified artifacts, and (2) interpretability would be fundamentally limited — if a probe can’t find some information, maybe the information was destroyed by the architecture.

The paper’s claim is that both conclusions are wrong, and not by a little. The discrete-to-continuous map (token sequence → hidden vector) is injective for essentially every model you’d ever train. The “lossiness” intuition comes from analyzing components in isolation on continuous inputs in R^d. But the real object of interest is the map from the discrete prompt space (finite vocabulary, finite context) into R^d, and over that domain the whole composition behaves very differently.

What’s New (Core Contribution)

Three contributions, each closing a gap left by prior work.

• A theorem, not a belief. Before: injectivity of LM representations was an informal hope, and the one prior proof (Sutter et al., 2025) only covered random initialization and the full hidden matrix. Now: a proof that injectivity holds with respect to the parameters, at the task-relevant last-token state, and — critically — that it survives training. The guarantee is “almost sure” (it fails only on a measure-zero set of pathological weights) and finite (holds at real width, depth, and after any finite number of steps).

• Massive empirical confirmation. Before: no large-scale test of whether collisions actually happen. Now: ~5 billion pairwise comparisons across six SOTA models plus an exhaustive 343-billion-pair stress test on the hardest near-collision prompts. Zero collisions, with margins orders of magnitude above the numerical threshold.

• SipIt: injectivity made operational. Before: existing inversion methods are approximate, need a trained auxiliary inverter, and work from logits/logprobs in black-box settings (Morris et al.), or do gradient-based approximate prompt search (HardPrompts). Now: a training-free algorithm that exactly recovers the prompt from raw hidden states with a provable O(T·|V|) worst-case bound and ~linear wall-clock time in practice.

The genuinely new bit is the bridge: a clean theorem about analytic functions turned into a practical decode loop, plus a sharp legal/privacy reframing (“hidden states are the prompt in disguise”).

How It Works (Technically)

There are two engines here: (A) the proof that the map is injective, and (B) the SipIt algorithm that inverts it. Let me demystify both.

Part A — Why distinct prompts give distinct vectors

The whole proof rests on one property: every building block of a Transformer is a real-analytic function of its parameters. “Real-analytic” just means: locally equal to its Taylor series — infinitely smooth, no kinks, no jumps. Why does each block qualify?

• Embeddings / projections / residual adds → polynomials and affine maps (analytic).
• Attention → exp and softmax (analytic).
• LayerNorm with ε > 0 → involves 1/sqrt(variance + ε); because ε > 0 you never divide by zero, so the reciprocal-square-root is analytic.
• MLP with GELU or tanh activation → analytic. (ReLU is not analytic — it has a kink — which is why the assumption names smooth activations.)

Real-analytic functions are closed under addition, multiplication, division, and composition. A Transformer is a finite composition of these blocks, so the entire map (s, θ) → r(s; θ) is real-analytic in the parameters. That’s Theorem 2.1.

Now the magic trick. Fix two different prompts s ≠ s' and define the squared distance between their last-token vectors as a function of the weights:

h(θ) = ‖ r(s; θ) − r(s'; θ) ‖²

Translate that to English: “as I vary the weights, how far apart do these two prompts land?” By Theorem 2.1, h is itself real-analytic. There’s a deep fact about analytic functions (the identity theorem / measure-zero dichotomy): an analytic function is either identically zero everywhere, or its zero set is infinitesimally thin (Lebesgue measure zero — like a curve inside a plane; the chance of randomly landing on it is exactly 0).

A collision between s and s' means h(θ) = 0. So collisions live on the zero set of h. To prove that set is thin, you only have to rule out the “identically zero” case — i.e., exhibit one single weight setting where the two prompts don’t collide. The authors construct one by hand: if the prompts differ at the last position, freeze the net so the output is just “embedding + position,” pick distinct rows, done; if they differ earlier, tune one attention head to point the last position at the first mismatch so its token leaks into the output. Either way h is not identically zero → its zero set is measure zero → a randomly initialized model (Gaussian/Xavier/uniform all have densities) lands there with probability 0. That’s Theorem 2.2: injective at initialization, almost surely.

The last piece (Theorem 2.3) is the one prior work missed: training doesn’t break it. A gradient step is the map ϕ(θ) = θ − η ∇L(θ). Because the loss is analytic, ϕ is analytic, and its Jacobian determinant is analytic and not-identically-zero. Where the determinant is nonzero, the Inverse Function Theorem says ϕ is a smooth, locally invertible change of coordinates — it can stretch and bend parameter space but cannot crush a region of positive volume down onto a thin zero-measure sheet. So a “spread-out” (absolutely continuous) distribution of weights stays spread-out after each step. Since you start spread-out (random init) and each step keeps you spread-out, after any finite number of steps the probability of sitting on the collision set is still exactly 0. Corollaries extend this to SGD, mini-batch, even adversarial batch ordering, and to whole finite sets of prompts being mutually distinct.

The honest fine print: this is almost sure, not always. You can hand-build collisions: tie two tokens to the identical embedding row, or use non-analytic operations like aggressive quantization or weight tying. Those are deliberate, measure-zero choices — they don’t happen by accident under normal training.

Architecture & data flow

flowchart LR
  subgraph FWD["Forward map (proven injective)"]
    S["Prompt s\n(discrete tokens\nfrom vocab V)"] --> E[Embed + Positional]
    E --> B1[Transformer blocks\nattn + LayerNorm + MLP\nall real-analytic]
    B1 --> R["Last-token hidden\nstate r(s) in R^d"]
  end
  R -. "almost surely\nno two s collide" .-> R
  subgraph INV["SipIt (inverts it)"]
    H["Observed hidden\nstates H-hat"] --> L["For t = 1..T:\ntry candidate tokens,\nkeep prefix that\nmatches h_t"]
    L --> OUT["Recovered prompt\ns-hat == s"]
  end
  R --> H

Schematic of the measure-zero argument: each gray curve is the "collision set" where two specific prompts would map to the same vector. As you sample more random weight settings (dots), they essentially never land *on* a curve — that's why injectivity holds with probability 1. Click to drop more samples.

Part B — SipIt, the inversion algorithm

SipIt = Sequential Inverse Prompt via Iterative updates. It exploits the causal structure of decoders: the hidden state at position t depends only on tokens 1..t. So if you already know the prefix, the observed hidden state at position t uniquely pins down token t (by the local injectivity argument — with a fixed prefix, any two candidate next-tokens give different states).

So you decode left to right. At each position you loop over candidate tokens, run the model on prefix + candidate, and check whether the resulting hidden state matches the observed one (within a tiny ε-ball to absorb float error). The unique match is the true token. Append it, move on. Worst case you try the whole vocabulary at each of T positions → O(T·|V|) model calls, linear in sequence length.

The naive version (BRUTEFORCE) just enumerates the vocab in random order. The fast version uses a gradient-guided policy: it keeps a continuous relaxation of the next token (a soft distribution over the vocab), uses the gradient of “distance to the observed state” to rank which tokens to try first, and periodically projects back to the nearest real token. This usually finds the match in a handful of trials instead of |V|/2. The gradient guidance is just a search heuristic for ordering — correctness is guaranteed by the verifier regardless of policy, so the linear-time worst-case bound always holds.

Step through SipIt decoding. The prefix is fixed; the algorithm tests candidate tokens (highlighted) against the observed hidden state until one matches the target (green), commits it, and advances. Watch how it reconstructs the sequence one token at a time.

The algorithm, simplified

# SipIt: recover the exact prompt from observed hidden states at layer L.
# model(tokens) -> H, where H[t] is the hidden vector at position t (layer L fixed).
# Correctness comes from causal structure + almost-sure local injectivity.

def sipit(observed_H, model, vocab, eps=1e-6):
    prefix = []                                  # recovered tokens so far
    for t in range(len(observed_H)):             # decode left to right
        target = observed_H[t]                   # the vector we must reproduce
        for tok in policy(vocab, prefix, target, model):   # ordering only — see below
            h_t = model(prefix + [tok])[t]       # forward pass on prefix+candidate
            if l2(h_t, target) <= eps:           # unique match by injectivity
                prefix.append(tok)               # commit this token...
                break                            # ...and move to position t+1
    return prefix                                # == the original prompt, a.s.

def policy(vocab, prefix, target, model):
    # BRUTEFORCE: just `return shuffled(vocab)` — correct but slow (~|V|/2 tries).
    # GRADIENT-GUIDED: relax the next token to a soft vector p over the vocab,
    # take grad of l2(model(prefix+soft(p))[t], target) wrt p, rank tokens by it,
    # project p back to nearest real token each round. Same guarantee, far fewer tries.
    return rank_by_gradient(vocab, prefix, target, model)

Built on Prior Work

Results & Evidence

Collision search. 100k prompts sampled from Wikipedia + C4 + The Pile + GitHub code; last-token states extracted across all layers of six models; ~5 billion pairwise distance checks. Minimum distances sit orders of magnitude above the 10⁻⁶ collision threshold (e.g., Llama-3.1-8B min distance 0.001 at layer 1 rising to 0.620 at the last layer; TinyStories 0.029 → 2.793). Distances tend to grow with depth and stabilize after a moderate sequence length. No collisions, anywhere.

Exhaustive stress test. They took the 10 closest near-collision prompts, appended every vocabulary token, and compared all resulting states — 343 billion pairs per model. Still no collisions; the boxplots are “boringly flat,” which is exactly the point.

Inversion (the money table). On 100 GPT-2 Small prompts (20 tokens, 90% natural / 10% random tokens), recovering from a fixed layer’s hidden states:

SipIt is ~140× faster than its own brute-force ablation and infinitely better than the approximate baseline (which gets exact recovery 0% of the time). Inversion time rises only mildly with layer depth and scales gracefully to 200-token prompts.

What the evidence does NOT establish. (1) The inversion experiments are GPT-2 Small only — the collision tests span six models, but exact-recovery timing is one small model. (2) Everything assumes clean hidden states; the paper itself flags noise/quantization robustness as open. (3) The theorem needs analytic activations and ε > 0 LayerNorm — true for mainstream models, but heavily quantized or ReLU-only or weight-tied setups can violate it. (4) “Almost surely” is a probability-1 statement, not a literally-always one; adversarial constructions exist. None of this undermines the core claim, but it bounds how far you should generalize the wall-clock numbers.

How You’d Use It

For an AI services company, this paper is less a feature and more a risk-and-capability map. Two directions:

As a compliance/security lens (immediately relevant). If you store, cache, log, or transmit hidden states or embeddings derived from user prompts, you are — provably — holding recoverable user text. That changes the answer to “is this PII?” from “probably fine, it’s just vectors” to “yes, treat it like the raw prompt.” Concretely:

• Vector DBs of prompt embeddings, KV-cache dumps, activation logs in your observability stack → in scope for deletion requests, retention policies, and data-residency rules.
• “We deleted the prompt” is not enough if you kept the embeddings. SipIt is the existence proof an auditor (or attacker) could point to.
• You can productize this: an embedding-leakage audit offering — run SipIt-style recovery against a client’s stored representations and show them, viscerally, that the text is still in there.

As an interpretability foundation (medium-term). Injectivity means “the information is definitely in the last-token state.” So if a probe or a feature-extraction method fails to find something, the failure is in your method, not in the model destroying the info. That gives you a clean baseline for interpretability/auditing work you sell to clients building regulated agentic systems. It also makes hidden states a sound substrate for caching/dedup: identical states ⟺ identical prompts, so state-keyed caches are exact.

Where it slots into a multi-agent system: agents that pass around or persist intermediate representations (shared memory, context handoff) are passing around recoverable text. Design accordingly — encrypt at rest, scope retention, don’t treat embeddings as anonymization.

Build Your Own (Minimal Recipe)

You can build a working SipIt for a small open model in an afternoon. The 80% version:

1. Pick a model with hidden-state access. Any HuggingFace transformers decoder (GPT-2, Pythia, a small Llama) with output_hidden_states=True. Fix one target layer L.
2. Generate ground truth. Run a known prompt through the model, grab hidden_states[L] → that’s your observed_H (shape [T, d]).
3. Write the verifier. A function matches(prefix, tok, t, target) that runs model(prefix + [tok]), reads position t at layer L, and checks L2 distance ≤ ε. Start with eps = 1e-4 for fp32; you may need to tune for fp16/bf16.
4. Decode left to right with BruteForce first. Loop positions; loop the vocab; commit on match. This is correct and proves your pipeline before you optimize. It’ll be slow.
5. Add the gradient policy (the one hard part). Replace vocab enumeration with: a soft one-hot p over the vocab, embed as p @ embedding_matrix, forward to position t, loss = ||h_t − target||², backprop to p, rank tokens by ascending predicted distance, try them in that order, re-project to nearest real token each round. This is where ~all the speedup lives.

The two genuinely hard parts: (a) numerical tolerance — float nondeterminism across batch sizes/devices means ε is finicky, especially in low precision; verify on fp32 first. (b) Making the gradient policy actually rank well — getting the relaxation/projection loop stable is the difference between 28s and 65 minutes. Everything else is plumbing.

Libraries to reach for: transformers for the model + hidden states, torch autograd for the policy, that’s it. No training, no extra data.

How to Improve It

Limitations are leverage. Five concrete, testable directions:

• Noisy / quantized inversion (the authors’ own open problem). Real production states are quantized (int8/fp8) and may be perturbed. Test how recovery degrades vs. perturbation magnitude relative to the per-step margin the paper computes; build a robust verifier that does approximate nearest-token matching instead of an ε-ball, and report the recovery/SNR curve.
• Beam-search policy. SipIt commits the unique match greedily. Under noise the “unique” match may be wrong; carry a beam of top-k prefixes and backtrack when the per-step margin is small. Testable: exact-match rate under added Gaussian noise, greedy vs. beam.
• Multimodal extension. The proof needs analytic, discrete-to-continuous structure. Vision/audio Transformers take continuous inputs, so injectivity over the input space is a different question — but token-quantized modalities (VQ-VAE codes, audio codecs) may inherit the result. Concrete test: run the collision search on a music/vision-token Transformer.
• Throughput. The gradient policy does a forward+backward per ranking round. A learned token-ranker (distilled once per model) could replace per-step backprop, trading the “training-free” purity for speed — measure tokens/sec vs. accuracy.
• Defensive direction (sellable). If hidden states are recoverable text, can you break injectivity on purpose for privacy without hurting task performance? Test deliberate weight-tying or controlled quantization that collapses only sensitive distinctions. This flips the paper’s threat into a feature: provably non-invertible deployment.

Glossary

• Injective (one-to-one) — different inputs always produce different outputs; nothing collapses together, so the input is recoverable from the output.
• Last-token representation r(s) — the hidden vector at the final position of the sequence; it’s what drives next-token prediction, so it’s the operationally important state.
• Real-analytic — a function that locally equals its own Taylor series: perfectly smooth, no kinks (ReLU fails this; GELU/tanh pass).
• Measure zero — an infinitesimally thin set (like a line inside a plane); randomly sampling a continuous distribution lands there with probability exactly 0.
• Almost surely (a.s.) — with probability 1; true except on a measure-zero set of exceptions.
• Absolutely continuous distribution — a distribution with a density (Gaussian, uniform, Xavier); it puts zero probability on any measure-zero set.
• Inverse Function Theorem — if a smooth map’s Jacobian determinant is nonzero, the map is locally invertible and can’t crush volume to a lower dimension.
• LayerNorm ε — the small constant added inside the normalization’s square root to avoid divide-by-zero; with ε > 0 the op stays analytic.
• Causal / decoder-only — each position attends only to earlier positions, so hidden state t depends only on tokens 1..t; this is what lets SipIt decode left to right.
• SipIt — Sequential Inverse Prompt via Iterative updates: the paper’s left-to-right exact prompt-recovery algorithm.
• Policy (in SipIt) — the rule for ordering which candidate tokens to test; random or gradient-guided. Affects speed, never correctness.
• Collision — two distinct prompts mapping to the identical hidden state; the event the paper proves is almost impossible.