Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models

TL;DR

Steering a language model away from toxic output usually means training extra classifier models, fine-tuning the LM’s weights, or writing long example-based prompts that eat up the context window. This paper shows you can skip the extra models: just place a block of “polite” example text and a block of “toxic” example text in front of the prompt, run the same frozen LM three times per token, and contrast the three resulting probability distributions with Bayes’ rule to steer generation — no training at all. The catch is that a convincing toxic example text runs ~900 tokens, which is expensive and crowds out the context window. Their fix, “prompt compression,” trains a soft prompt (as few as a single learned vector) to mimic what that 900-token block does to the model’s predictions. Surprisingly, the compressed version usually matches or beats the original text at reducing toxicity, occasionally hitting toxicity scores competitive with PPLM (the decode-time state of the art at the time) using 900x less context — while retaining only the “gist” of the original prompt, not its details.

Problem & Motivation

Reducing toxicity in LM output takes one of two well-worn paths, and both are expensive:

• Retrain the weights. Fine-tune or RL-tune the model on curated/labeled “clean” data (CTRL, Quark, PPO-style RLHF). This works well but needs a reward model or labeled data, GPU time for fine-tuning, and you now maintain a second set of weights.
• Steer at decode time with an auxiliary model. GEDI trains a discriminator LM; DEXPERTs trains an “expert” and “anti-expert” LM and combines their logits; PPLM backprops through the base model’s hidden states toward a desired attribute. All three need you to train, store, and run at least one extra neural network.

There’s a cheaper decode-time trick available: LMs are good few-shot pattern-matchers. Show GPT-2 a paragraph of polite sentences and a paragraph of vulgar sentences, and its own next-token probabilities already “know” which paragraph a candidate word sounds more like it belongs to — no extra model needed, just prompt engineering. The problem: to reliably cover the many ways text can be toxic (racist, sexist, profane, vulgar), the example paragraph has to be long — the authors’ toxic example block is about 900 tokens. On a GPT-2-era 1024-token context window, that block alone eats most of your budget, and every generated token now costs three forward passes over a much longer sequence (attention cost scales O(n²) with context length). If you can’t shrink the example text without breaking its persuasive power, this “free” approach isn’t actually cheap.

What’s New (Core Contribution)

• Prompt compression, formalized. Before: soft prompts (Lester et al., 2021) were trained from labeled task data to make a frozen LM perform a downstream task well. Now: a soft prompt is trained to imitate a specific fixed piece of text — minimize the KL divergence between what the real text predicts and what the learned vectors predict, across a broad sample of continuations. This repurposes prompt tuning as a distillation target for a context, not a model.
• Contrastive contexts as a training-free attribute classifier. Before: GEDI/DEXPERTs-style decode-time steering needs an auxiliary discriminative LM, trained separately. Now: build the “classifier” purely by prompting — run the same frozen LM once with a positive exemplar block prepended, once with a negative exemplar block prepended, and take the ratio of the two resulting token likelihoods as the attribute score. Zero backprop, zero fine-tuning, zero extra model.
• The two ideas combined, empirically validated. Compressing the positive/negative contrastive contexts doesn’t just preserve their steering power — it usually improves it. A single learned vector (900x compression of the ~900-token toxic context) frequently gives the lowest expected-max-toxicity of any prompt length tested, matching contemporaneous SOTA (PPLM).
• A systematic scaling and fluency study. The method is tested across four GPT-2 sizes (117M–1.5B), including using small models to steer large ones, and reports the toxicity/fluency (perplexity) trade-off explicitly rather than reporting toxicity numbers alone.

How It Works (Technically)

There are two mechanisms here, trained/used at different times: prompt compression (an offline training step, done once per prompt you want to reuse) and contrastive conditioning (an online decode-time step, done for every token you generate). Toxicity reduction is what happens when you feed compressed contrastive contexts into the decode-time steering loop.

1. Prompt compression: distilling a text block into a few vectors

A language model defines a distribution over next tokens conditioned on everything before them: p(x_t | x_1…x_{t-1}). Call the fixed conditioning text you want to compress the hard prompt, x_h. A soft prompt (Lester et al., 2021) is a block of n trainable embedding vectors, θ_n — not tied to any real vocabulary token — that gets prepended directly to the embedding sequence and fed through the transformer exactly like real token embeddings would be. It induces its own distribution over continuations, q(x_t:k | θ_n).

Training objective (Eq. 1):

min_θ E[ KL( p(x_t:k | x_h) || q(x_t:k | θ_n) ) ]

Plain English: sample a large, diverse set of continuations x_t:k (the paper draws them from The Pile). For each one, compare two things: (a) how likely that continuation is when the model is conditioned on the real hard-prompt text, versus (b) how likely it is when the model is conditioned on the n learned vectors instead. Adjust the vectors (by gradient descent, Adam, 75,000 steps, LR schedule from 0.1, ~1–4 GPU-hours per prompt) so that (b) matches (a) as closely as possible, for any plausible continuation — not just one. This is knowledge distillation, but the “knowledge” being distilled is a single context, not a whole model, and the “student” is a handful of free parameters rather than a network.

The LM’s own weights are frozen throughout; only θ_n is trained. Once trained, θ_n is a fixed, reusable artifact — swap it in wherever you’d have used the original hard prompt, at a fraction of the sequence length (and therefore a fraction of the O(n²) attention cost).

What survives compression, and what doesn’t. The paper runs two diagnostics. First, a reading-comprehension test: ask questions about a paragraph, answered via the compressed prompt instead of the real one. General/thematic questions (“what do Frank and Cindy love to do?”) degrade gracefully as n shrinks; specific-detail questions (“what continent have they not visited?”) collapse fast. Second, a reconstruction test: ask the model to literally repeat the paragraph, and measure per-token likelihood normalized between a “no context” floor and a “full hard context” ceiling. At n=64, most salient nouns/phrases survive; at n=1, almost everything is lost except a couple of high-salience tokens (names, the country). The authors describe what remains as a “semantic eigenvector” — the compressed prompt keeps the single dominant signal a piece of text carries, and sheds the details around it. This turns out to be exactly the right property for a toxicity classifier, where “is this polite or vulgar” is a single dominant signal buried inside 900 tokens of specific examples.

They also show compressed prompts are partially composable: separately compress a “talk about cats” context and a “be negative” context, then concatenate the two compressed prompts. Generations shift toward exhibiting both attributes at once, roughly interpolating between the individual effects — evidence the compressed vectors aren’t a single indivisible blob.

2. Contrastive contexts: a classifier made of prompts, not weights

Steering at decode time (following GEDI/PPLM) uses Bayes’ rule to reweight the LM’s normal next-token distribution by how well each candidate token expresses some desired attribute a:

p(x_t | a, x_h) ∝ p(a | x_h, x_t)^ω · p(x_t | x_h) (Eq. 2)

Plain English: take the model’s normal (“prior”) probability for the next token, and multiply it by an attribute-classifier score raised to a temperature ω. Larger ω = stronger steering toward the attribute; ω=0 recovers plain, unsteered generation. GEDI and DEXPERTs build p(a|x_h,x_t) with a separately trained model. This paper builds it with prompting:

p(a | x_h, x_t) ≡ p(x_t | x+ x_h) / [ p(x_t | x+ x_h) + p(x_t | x- x_h) ] (Eq. 3)

Plain English: prepend a block of positive exemplar text (x+ — polite, kind sentences) in front of the running history and ask the LM how likely the candidate token is; separately prepend a block of negative exemplar text (x- — racist/sexist/profane sentences) and ask the same question. The attribute score is just the normalized contrast between the two: does this token look more at-home after the polite examples, or the vulgar ones? Three forward passes of the same frozen model per generated token — prior, positive, negative — combined by Eq. 2/3, then sampled normally (nucleus sampling, beam search, whatever you’d normally use).

For toxicity reduction specifically: x+ is a hand-written block of kind/polite sentences, x- is a hand-written block (~900 tokens) of racist, sexist, profane, and vulgar snippets, deliberately varied in spelling/capitalization/grammar to avoid the classifier keying on superficial formatting instead of content.

Where compression plugs in

x- is long precisely because it has to cover many kinds of toxicity. That’s expensive per Sec. 1 above (context budget, 3x forward-pass cost at length ~900). So: compress x+ and x- with the Sec. 1 procedure, and use the resulting soft prompts in place of the hard text inside Eq. 3. The measurement in Sec. 6 (below) is whether steering quality survives that swap — and the surprising finding is that it doesn’t just survive, it often improves.

Architecture & data flow

flowchart LR
  subgraph "Offline: prompt compression (Sec. 3, once per prompt)"
    XH["Hard prompt x_h<br/>(e.g. ~900-token toxic exemplars)"] --> Teacher["LM pass (frozen weights)"]
    Teacher --> PD["p(continuation | x_h)"]
    Theta["Soft prompt θ_n<br/>n learned vectors"] --> Student["LM pass (frozen weights)"]
    Student --> QD["q(continuation | θ_n)"]
    PD --> KL["KL divergence loss"]
    QD --> KL
    KL -.->|backprop into θ_n only, 75k steps| Theta
  end
  Theta -->|swap in for x_h| Steering
  subgraph "Online: contrastive steering (Sec. 5, every generated token)"
    Steering["3 LM passes: prior / positive / negative"] --> Final["Combine via Eq. 2 & 3"]
    Final --> Sample["Sample next token"]
  end

The contrastive-context mechanism (Eq. 2 & 3) on the paper's own toy example: "The party was ___." Three token distributions (prior, primed by a positive exemplar block, primed by a negative exemplar block) combine into one steered distribution. Drag the ω slider to see how steering strength trades off against staying close to the prior.

Schematic of the paper's central surprise (Sec. 6.3, Fig. 7): as the compressed toxic/positive contexts shrink from 64 learned vectors down to a single one, expected-max-toxicity does not get worse — it usually stays flat or improves, matching or beating the full 900-token hard prompt. Illustrative curve built from the paper's reported pattern, not digitized figure data.

The algorithm, simplified

# Decode-time contrastive steering (Sec. 5). One call per generated token.
# prior / pos / neg all use the SAME frozen LM — only the conditioning text differs.
# pos_ctx / neg_ctx can be real text (hard prompt) or a trained soft-prompt's
# embedding vectors (compressed prompt) — the steering math doesn't care which.
def steer_step(lm, history, pos_ctx, neg_ctx, omega=10.0):
    prior_p = lm.next_token_probs(history)                  # p(x_t | x_h)          -- Eq. 2 prior
    pos_p   = lm.next_token_probs(concat(pos_ctx, history))  # p(x_t | x+, x_h)      -- Eq. 3
    neg_p   = lm.next_token_probs(concat(neg_ctx, history))  # p(x_t | x-, x_h)      -- Eq. 3

    attribute_p = pos_p / (pos_p + neg_p + 1e-9)             # Eq. 3: contrast, not raw magnitude
    steered = prior_p * (attribute_p ** omega)                # Eq. 2: Bayes reweighting by ω
    return steered / steered.sum()                            # renormalize, then sample as usual


def generate(lm, prompt, pos_ctx, neg_ctx, n_tokens=20, omega=10.0):
    history = prompt
    for _ in range(n_tokens):
        probs = steer_step(lm, history, pos_ctx, neg_ctx, omega)
        next_token = sample_nucleus(probs)        # any standard decoding strategy
        history = concat(history, next_token)
    return history

# Prompt compression (Sec. 3). Run once, offline, per hard prompt you want to reuse.
# theta: n trainable embedding vectors, prepended in place of real token embeddings.
def compress_prompt(lm, hard_prompt, n_vectors, corpus, steps=75_000, lr=0.1):
    theta = init_embeddings(n_vectors)                       # the only thing being trained
    optimizer = Adam([theta], lr=lr, schedule="linear")
    for step in range(steps):
        continuation = sample_continuation(corpus)            # e.g. a Pile snippet
        target_logp  = lm.logprob(continuation, prefix=hard_prompt)   # teacher: real text
        student_logp = lm.logprob(continuation, prefix_embeds=theta)  # student: learned vectors
        loss = kl_divergence(target_logp, student_logp)        # Eq. 1
        loss.backward()                                         # LM weights stay frozen
        optimizer.step()
    return theta   # a reusable, ~n-token substitute for hard_prompt

Built on Prior Work

Results & Evidence

Evaluated on RealToxicityPrompts (RTP): a fixed, balanced subset of 2000 prompts, 25 generated continuations each (20 tokens), scored with Perspective API for expected-max-toxicity and average toxicity, across all four GPT-2 sizes.

What held up:

• Hard-prompt contrastive steering reduces toxicity as ω increases, competitive with PPLM at the strongest settings, with a bigger effect on smaller models than larger ones.
• Compressed-prompt steering matches or beats hard-prompt steering at essentially every setting tested — and the smallest compression (a single learned vector, ~900x smaller than the hard toxic context) is frequently the best performer, not just an acceptable approximation.
• There’s a real toxicity/fluency trade-off (measured via GPT-J-6B perplexity as ω increases), but soft prompts sit on an equal-or-better point of that trade-off curve than hard prompts at a matched perplexity.
• Small models make better “steerers” of large models than same-size or larger steerers, for both hard and compressed prompts — consistent with prior DEXPERTs/GEDI observations, now shown to hold for this training-free method too.

What the evidence does NOT establish (the paper’s own caveats):

• Only tested on GPT-2 (117M–1.5B parameters) — nothing here is verified on instruction-tuned, RLHF’d, or much larger modern models, where the base “few-shot pattern completion” behavior this method leans on may work differently.
• Toxicity is reduced, never eliminated — the authors explicitly say this should not be deployed where zero-tolerance for toxic output is required.
• The toxic/positive contexts were built by hand and only lightly tuned (three variants tried); alignment between “what the authors think reads as toxic” and what Perspective API scores as toxic wasn’t rigorously validated.
• The method targets blatant profanity/racism/sexism; the authors flag that subtler bias (microaggressions, coded language) is unlikely to be caught by this approach without more careful prompt engineering.
• Weight-modifying alternatives (Quark) achieve a lower absolute toxicity floor — this method’s advantage is being training-and-weight-free, composable, and cheap to swap, not being the best possible detoxifier.
• Why a single compressed token sometimes beats the full hard prompt is explicitly called “not well understood” by the authors — it’s a reported empirical pattern with a plausible hypothesis (compression forces distillation down to the single dominant signal), not a proven mechanism.

How You’d Use It

• A cheap steering/safety layer on top of a model you fully control. If you’re serving an open-weights model (GPT-2-class or larger, anything you can run a custom per-token decode loop against), this gives you toxicity/tone steering with no fine-tuning pipeline and no extra model to host — just three forward passes per token and a couple of prompt blocks. That’s a realistic pattern for a self-hosted, brand-safe generation feature in your own product.
• Composable “prompt libraries” as a lightweight tone-control layer. Once trained, a compressed prompt is a tiny, reusable artifact (as small as one vector). You could maintain a library — “no profanity,” “always polite,” “match brand voice X” — and mix-and-match per surface or per channel using the compositionality result (Sec. 4.3), without re-training the base model for each variant.
• Where it doesn’t fit: this needs per-token logit access and control over the sampling loop. It’s not usable behind a plain hosted chat-completion API (OpenAI/Anthropic-style) that only returns finished text — you’d need self-hosted or logit-exposing infrastructure.
• The training-free half is useful on its own, even without compression. “Score a candidate token by contrasting its likelihood under a positive vs. negative exemplar block” is a generally useful few-shot classifier pattern — brand-tone matching, mild PII-avoidance nudging, style steering — worth keeping as a pattern independent of whether you ever bother compressing the prompts.

Build Your Own (Minimal Recipe)

Components:

1. An open-weights causal LM you can run locally with access to raw per-token probabilities (any HF transformers GPT-2/GPT-J/Llama-class model).
2. A decode loop you control token-by-token — not a vendor chat-completion endpoint.
3. Hand-written positive and negative exemplar text blocks for whatever attribute you want to steer.
4. (Optional — only build this once the above works and context budget/latency actually hurts) a soft-prompt trainer.

Build order:

1. Implement Eq. 2/3 contrastive steering with plain text (hard) contexts first. This alone gets you training-free decode-time steering — validate it on your real use case (a profanity filter, a tone nudge) before touching compression at all.
2. Only add prompt compression once you’ve confirmed the hard-prompt version works, and only if it’s actually the bottleneck — long x+/x- eating your context budget, or the 3x-forward-pass cost being too slow at the context lengths you need.
3. To compress: for a batch of generic continuations (any large diverse corpus works as a stand-in for The Pile — C4, a Wikipedia dump, your own domain corpus), run the hard prompt through the frozen LM as a “teacher,” run n learnable embedding vectors through the same frozen LM as a “student,” and backprop only into the vectors to minimize per-token KL divergence between the two.

The genuinely hard parts:

• Getting embedding-level access to prepend raw vectors (not real tokens) before the transformer layers — in HuggingFace transformers, this is the inputs_embeds argument instead of input_ids; most causal LM classes support it but plumbing it through generation utilities takes care.
• Picking a good, diverse continuation-sampling distribution for the KL objective. If your training corpus doesn’t resemble what you’ll actually condition on at inference time, the compressed prompt won’t generalize — this is the one place where “just use whatever text you have lying around” will quietly hurt you.

Libraries: HuggingFace transformers (inputs_embeds), PyTorch + Adam for the compression training loop, Perspective API or an open toxicity classifier (e.g. Detoxify) for evaluation, any standard nucleus-sampling implementation for generation.

How to Improve It

1. Re-test on modern instruction-tuned / RLHF’d models. GPT-2 is small and has no alignment training; it’s unknown whether “smaller compression steers better” survives on a Llama-3/Mistral-class model whose base behavior is already shaped by RLHF — that alignment training might suppress or interact badly with this decode-time trick.
2. Learn an amortized compressor. The paper flags this itself: instead of 1–4 GPU-hours of optimization per prompt, train a small network that maps any hard prompt directly to its soft-prompt vectors in one forward pass — directly analogous to fast neural style transfer replacing per-image optimization.
3. Directly test the “semantic eigenvector” hypothesis. Train a soft prompt to match only the topic/sentiment marginal of the hard prompt’s completions (rather than the full next-token distribution) and see whether that alone reproduces the single-vector detox effect — this would turn an unexplained empirical result into a testable mechanism.
4. Systematically study compositionality. The cats+negativity result (Sec. 4.3) is one anecdote. Build a proper benchmark: compress N independent attributes, test all pairwise (and triple) compositions, and measure how additively the effects combine versus how much they interfere.
5. Chain it with a weight-modifying method. Use cheap compressed contrastive steering as a first-pass filter, and reserve a slower RL/fine-tuning pass (Quark-style) only for cases that slip through — potentially getting closer to a zero-toxicity floor without paying full RLHF cost on every request.

Glossary

• Soft prompt — a block of trainable embedding vectors prepended to a model’s input, not tied to any real vocabulary token; learned via backprop while the model’s own weights stay frozen.
• Hard prompt — an ordinary text prompt made of real tokens, as opposed to a soft prompt’s learned vectors.
• Prompt tuning — the original technique (Lester et al., 2021) of training a soft prompt on labeled task data to make a frozen LM perform a downstream task.
• Prompt compression — this paper’s technique: training a soft prompt to imitate a fixed hard prompt’s effect on the output distribution, rather than to perform a task.
• KL divergence (Kullback-Leibler divergence) — a measure of how different two probability distributions are; here, how differently the soft prompt and hard prompt predict the same continuations. Zero means identical predictions.
• Bayesian attribute classifier framework — a decode-time steering method that reweights the LM’s normal next-token probabilities by an attribute score, via Bayes’ rule (Eq. 2).
• Contrastive contexts — this paper’s method of building that attribute score purely from prompting: contrasting token likelihood under a positive exemplar block versus a negative one, with no trained classifier.
• ω (omega) — the temperature/strength parameter controlling how aggressively the attribute score reshapes the prior distribution; ω=0 is unsteered generation.
• GEDI — prior decode-time steering method using Bayes’ rule with a trained discriminative LM as the attribute classifier; this paper’s closest technical relative.
• PPLM (Plug and Play Language Models) — prior decode-time steering method that backprops through the LM’s hidden states toward a desired attribute; the fluency/toxicity SOTA baseline here.
• DEXPERTs — prior method combining a trained “expert” and “anti-expert” LM via product-of-experts to steer generation.
• Quark — a reinforcement-learning-based detoxification method that fine-tunes the LM’s weights; achieves lower absolute toxicity at the cost of requiring training.
• RealToxicityPrompts (RTP) — a benchmark dataset/protocol (Gehman et al., 2020) of 100,000 prompts used to evaluate how toxic a model’s continuations are.
• Expected max toxicity — an RTP metric: the highest toxicity score among 25 sampled continuations for a prompt, averaged over prompts.
• Perspective API — Google/Jigsaw’s toxicity-scoring API, used as the (imperfect but standard) toxicity measurement tool throughout.
• Perplexity — a standard measure of how well a model predicts text (lower = more fluent/predictable); used here as a fluency proxy to show the toxicity/fluency trade-off.
• Nucleus sampling — a text-generation decoding strategy that samples from the smallest set of tokens whose cumulative probability exceeds a threshold, avoiding both greedy repetition and low-probability nonsense.
• The Pile — an 800GB diverse text corpus (Gao et al., 2021) used here as the source of generic continuations for training compressed prompts.
• Knowledge distillation — training a smaller “student” to reproduce a larger “teacher’s” output distribution; the general category of technique prompt compression borrows from.
• inputs_embeds — the HuggingFace transformers argument that lets you feed raw embedding vectors directly into a model, bypassing the normal tokenizer/embedding lookup — the practical hook needed to use soft prompts.