build: who the model is and what the job is, what it knows, how it thinks, and how you make the answer stable. Every known prompt trick slots into one of those four layers.
TL;DR
This is a survey, not a new method. The authors sort the zoo of prompt techniques (role prompts, few-shot, RAG, chain-of-thought, tool use, self-refine, self-consistency, and so on) into four layers borrowed from how agent systems are usually split up: profile and instruction, knowledge, reasoning and planning, and reliability. The order matters: you start with a clean base prompt, add knowledge and reasoning only when the task needs them, and finish with a reliability step that makes repeated runs agree. They then map real LLM applications (chatbots, search, legal and medical Q&A, code, robotics, creative tools) onto those layers. There are no new experiments; the value is a checklist and a vocabulary you can design and debug prompts with.
Problem & Motivation
Prompt engineering grew as a pile of named tricks. Each paper reports a gain on some benchmark, and each earlier survey sorted the tricks by scenario (“prompts for QA”, “prompts for code”). That sorting breaks the moment you move to a new task: you cannot tell which tricks transfer, which overlap, and which order to apply them in.
The practical pain for a builder is familiar:
- A prompt that works in a demo gives different answers on the next run (content bias).
- A prompt that is vague gets long, generic answers; one that is specific gets hallucinated facts.
- You bolt on RAG, then CoT, then a critic, and you no longer know which piece is doing the work.
- There is no shared language on a team for “what kind of fix is this?”
The paper’s claim is that a taxonomy built on function (what the technique does in the pipeline) rather than scenario (where it was first used) gives you a design process that transfers.
What’s New (Core Contribution)
- A function-based taxonomy. Before: surveys sorted techniques by task (Sahoo et al. used 12 scenario buckets). Now: four functional layers mapped to the agent split of profile, memory, planning, and action. This is the one genuinely useful idea. It is a re-framing, not a discovery.
- An ordering, not just a list. Profile and instruction comes first (the base prompt), reliability comes last (stabilize the output), and knowledge plus reasoning sit in the middle as optional enhancers chosen per task. The paper calls this a “reasonable process flow for designing prompts”.
- An application map. Each surveyed system (Sparrow, Copilot, HyDE, ChatLaw, ChemCrow, PaLM-E, and others) is tagged with the layers it uses. Useful as a lookup table; shallow as analysis.
- A scope decision. They only cover hard (discrete text) prefix prompts for decoder-only models. Soft prompts, cloze prompts, and prompt tuning are out. That keeps the survey about things you can type into a prompt, which is what most builders actually control.
What is not new: every technique in the paper predates it, and there are no new benchmarks or ablations. Treat it as a well-organized map, not as evidence.
How It Works (Technically)

For a survey, “how it works” is the taxonomy itself plus the mechanism behind the few techniques that carry the most weight. Here is the structure, then three mini deep-dives.
The four layers
flowchart TD
T[Task] --> P
subgraph P[1. Profile and instruction - base prompt]
P1[Personality: role, tone, expertise]
P2[Task info: intent, domain, demand, environment]
P3[Demonstrations: zero-shot or few-shot]
end
P --> K{Needs outside facts?}
K -->|yes| KN[2. Knowledge: basic RAG, pre-retrieval rewrite, post-retrieval filter, memory]
K -->|no| R
KN --> R{Needs multi-step thinking?}
R -->|yes| RP[3. Reasoning and planning: decomposition, CoT, tools, feedback]
R -->|no| REL
RP --> REL[4. Reliability: prompt ensembles, verifiers, detox chains]
REL --> O[Stable answer]
The full taxonomy as a 3D tree you can orbit. The centre is "prompt", the four coloured branches are the layers, and the leaves are the named techniques the survey files under each. Drag to rotate, scroll to zoom. Structure is from the paper's Fig. 1; positions are schematic.
Layer 1: Profile and instruction. This is the base prompt, built from three parts.
- Personality information: the role (“you are a contracts lawyer”). The paper’s explanation of why this works is simple and correct enough: text near the word “lawyer” in training data is legal text, so naming the role shifts which part of the model’s knowledge gets attention.
- Task information: split into intent (the goal), domain (where the information comes from), and demand (the detailed requirements and output format). Add task environment for agents: what objects exist, what actions cost, and what each action does (LLM+P writes this as PDDL, a formal planning language; DEPS spells out action preconditions and outcomes).
- Demonstration information: zero-shot (no examples) or few-shot (a few input-output pairs). The important practical findings: choose examples that are similar to the current query (KNN retrieval over an example bank), keep them diverse (Vote-K), and watch for label bias, where the mix and order of example labels pushes the model toward one answer. Calibration methods fix this by measuring what the model outputs on a content-free input and subtracting that lean.
Layer 2: Knowledge. RAG, viewed only through what it does to the prompt.
- Basic RAG: chunk documents, embed them, retrieve the top-k by similarity, and paste them into a template like “Answer based on: Segment 1 … Segment 2 …”.
- Pre-retrieval: fix the query before you search. Query2doc and HyDE ask the model to write a fake answer document first, then search with that, because a fake answer looks more like the real documents than the short question does. Query2doc reports +3% relative on MS MARCO and +15% on TREC DL with plain BM25.
- Post-retrieval: fix the documents before you prompt. Rewrite them toward the query (Chain of Note), reorder them by relevance tier, or have the model strip irrelevant text first (System 2 Attention, which lifted factuality on TriviaQA from 62.8% to 80.3% in the cited work).
- Memory: keep past retrievals and conversation turns as a store the next retrieval can use (external memory, as in Selfmem), or let the model monitor and critique its own retrieval (internal memory, as in MetaRAG).
Layer 3: Reasoning and planning. Three families.
- Target decomposition: plan-then-execute (Least-to-Most: break the question into sub-questions, solve them in order, feed earlier answers forward) or iterative (Chain-of-Thought, plus its automated and structured descendants: Auto-CoT, Faithful CoT, Tree and Graph of Thought, XoT).
- External support: hand the parts LLMs are bad at to something exact. Code interpreters (PAL, Program of Thoughts: the model writes Python, Python does the math), physics simulators (Mind’s Eye), and general tool frameworks (Toolformer, MRKL routers, ART, HuggingGPT, Gorilla).
- Feedback: internal (Self-Refine: draft, critique, revise in a loop) or external (CRITIC: check the draft with a search engine or interpreter, then revise). The paper flags a real catch: self-feedback helps some tasks and hurts others, and models show self-bias, rating their own outputs too kindly. External checks fix more than internal ones do.
Layer 4: Reliability. Two kinds of bias.
- Content bias: the answer drifts from what the task asked for, or changes between runs. The fix is prompt ensembling, borrowed from classic ensemble learning. Bagging runs many prompts or samples and combines them (Self-Consistency majority vote; DiVeRSe adds a step-level verifier; AMA reformats the question several ways and aggregates with weak supervision). Boosting builds the prompt set iteratively, focusing each new prompt on the cases the last ones got wrong (PromptBoosting, PREFER).
- Value bias: harmful, toxic, or stereotyped output. The prompt-side tools here are thin (Detox-Chain), and the paper admits most real fixes still happen at training time.
Mini deep-dive: why majority voting works, in one equation
Self-Consistency is the workhorse of the reliability layer. If one sampled answer is right with
probability p, and you take a majority vote over n independent samples, the vote is right with
probability
P(majority correct) = sum over k > n/2 of C(n, k) * p^k * (1 - p)^(n - k)
Plain English: count every way that more than half the samples could be right, and add up how likely
each of those is. What it does operationally: when p is above 0.5, more samples push the vote
toward certainty fast. When p is below 0.5, more samples push it toward certain failure. Voting
amplifies whatever the model already leans toward. Two real-world catches the formula hides: samples
from one model are not independent (they share the same blind spots), and free-form answers must be
normalized before they can be counted as “the same answer”. That is exactly the gap DiVeRSe and AMA
try to close with verifiers and smarter aggregation.
Interactive: drag the slider to set the single-sample accuracy p. The curve shows majority-vote accuracy as you add samples (n = 1 to 25, odd only). Above p = 0.5 the curve climbs; below it, the curve falls. This is computed from the binomial formula above, not from the paper's data.
Mini deep-dive: the query rewrite trick
HyDE and Query2doc fix a mismatch most RAG builders hit: questions are short and phrased like questions, documents are long and phrased like answers. Embedding the question puts it in the wrong neighbourhood. So you ask the model to write a plausible answer first (it may contain wrong facts; that is fine), embed that, and search with it. The fake answer has the right vocabulary and shape, so it lands near the real answer. The cost is one extra LLM call per query.
Mini deep-dive: the ordering as a debugging method
The most usable idea in the paper is the order of layers. When a prompt fails, ask which layer failed, in order:
- Is the role, task, and output format clear? (Profile and instruction)
- Does the model lack a fact it needed? (Knowledge)
- Did it have the facts but botch the steps? (Reasoning and planning)
- Is it right sometimes but not reliably? (Reliability)
Fixing layer 4 when layer 1 is broken just gives you a confident vote for a misunderstood task.
Data flow: one request through all four layers
sequenceDiagram
participant U as User
participant B as Prompt builder
participant R as Retriever
participant M as LLM
participant T as Tool or checker
U->>B: question
B->>B: add role, intent, domain, demand, k similar examples
B->>M: write a hypothetical answer
M-->>B: pseudo-document
B->>R: search with question + pseudo-document
R-->>B: top-k chunks, filtered and reordered
loop n samples
B->>M: full prompt with "think step by step"
M->>T: run code or look up fact
T-->>M: exact result
M-->>B: candidate answer
end
B->>B: normalize and majority vote
B-->>U: final answer
The algorithm, simplified
The survey has no single algorithm, so this is the four-layer process written as one function. Each block is one layer, and each is optional except the first.
def layered_prompt(question, cfg):
# contracts: llm(prompt, temperature) -> str ; embed(text) -> vector ; search(vec, k) -> list[str]
# LAYER 1: profile + instruction (always on)
examples = nearest(cfg.example_bank, embed(question), k=3) # similar demos beat random ones
base = (f"You are {cfg.role}.\n" # personality
f"Goal: {cfg.intent}\nSource: {cfg.domain}\n" # task: intent + domain
f"Requirements: {cfg.demand}\n" # task: demand / output format
+ "".join(f"Q: {q}\nA: {a}\n" for q, a in examples))
# LAYER 2: knowledge (only if the task needs outside facts)
if cfg.needs_facts:
fake_doc = llm(f"Write a short passage answering: {question}", 0.0) # HyDE-style rewrite
chunks = search(embed(question + "\n" + fake_doc), k=8)
chunks = [c for c in chunks if llm(f"Relevant to '{question}'? yes/no\n{c}", 0) == "yes"]
base += "Answer using only:\n" + "\n".join(chunks) + "\n"
# LAYER 3: reasoning (only if the task is multi-step)
if cfg.multi_step:
base += "Think step by step. Write any arithmetic as Python in ```py``` blocks.\n"
# LAYER 4: reliability (sample n times, vote on normalized answers)
answers = []
for _ in range(cfg.n_samples): # n odd, e.g. 5
draft = llm(base + f"Q: {question}\nA:", 0.7) # temperature > 0 so samples differ
draft = run_code_blocks(draft) if cfg.multi_step else draft # tool does exact math
answers.append(normalize(extract_final(draft))) # "42.0", "42" -> "42"
best, count = Counter(answers).most_common(1)[0]
confidence = count / len(answers) # agreement rate is your free confidence score
return best, confidence
Built on Prior Work
This is a map of the field, so lineage here means “where each layer comes from”.
| Prior idea | What it gave | What this paper changes |
|---|---|---|
| Liu et al. 2021, prompt-based learning survey | Hard vs soft prompts, cloze vs prefix | Narrows to hard prefix prompts for decoder-only LLMs |
| Agent function split (profile, memory, planning, action) | A complete split of what an agent does | Re-uses it as the skeleton of the prompt taxonomy |
| Sahoo et al. 2024, scenario survey | 12 scenario buckets | Replaces scenarios with four functional layers |
| Wei et al. 2022, Chain-of-Thought | Step-by-step reasoning in the prompt | Files CoT and descendants under “iterative decomposition” |
| Lewis et al. 2020 and Gao et al. 2023, RAG | Retrieve, then generate | Keeps only the prompt-facing parts: rewrite, filter, memory |
| Wang et al. 2022, Self-Consistency; classic bagging and boosting | Majority vote over samples | Groups all multi-prompt methods as “prompt ensembling” for reliability |
| Madaan et al. 2023, Self-Refine; Gou et al. 2023, CRITIC | Draft, critique, revise | Splits feedback into internal (self) vs external (tool) |
Results & Evidence
There are no experiments. All numbers are quoted from the cited papers. The useful ones:
- Query2doc: +3% relative on MS MARCO, +15% relative on TREC DL with BM25 and no fine-tuning.
- System 2 Attention: TriviaQA factuality 62.8% to 80.3%; GSM-IC accuracy 51.7% to 61.3%.
- MetaRAG: +34.6% accuracy on 2WikiMultihopQA, +26% on HotpotQA.
- Self-Refine: 5% to 40% absolute gains across tasks; up to 13% on code with Codex.
Caveats you should keep in mind:
- No head-to-head comparisons. You cannot use this paper to decide between, say, Self-Refine and CRITIC. The numbers come from different models, tasks, and years.
- Some claims are garbled or wrong. BloombergGPT is described as 5 billion parameters; it is about 50 billion. The “irrelevant documents improve accuracy by 30%” line is a real but narrow finding from one study and should not be read as advice to pad your context with noise. “Self-Refine outperforms GPT-3.5 and GPT-4” means it improves outputs of those models.
- The taxonomy is not tested. The claim that the four layers are “comprehensive and orthogonal” is asserted, not shown. Tool use sits awkwardly under “reasoning”, and memory is split between “knowledge” and agent design.
- It is already dated in one way the authors note themselves. Reasoning models (o1 and its successors) internalize CoT and self-checking. For those models, much of layer 3 and part of layer 4 moves from the prompt into the model. Layers 1 and 2 still matter fully.
How You’d Use It
Three lenses fit this paper well.
Your harness. Turn the four layers into four explicit slots in your prompt builder instead of one
big string. Concretely: a profile block (role, intent, domain, demand), a knowledge block filled
by your retriever, a reasoning block (CoT or tool instructions), and a reliability policy (number
of samples, voting rule, verifier). It improves traceability: when a run fails, your logs show which
slot was empty or wrong. Cost: a day of refactoring if your prompts are already templated. It pays
off when you have more than a handful of agents or prompts and people other than the author edit them.
Your automations and business processes. For any unattended extraction or classification step (invoice fields, ticket routing, document tagging), add the cheapest reliability layer first: sample 3 to 5 times at a moderate temperature and majority-vote. Use the agreement rate as a confidence score and route low-agreement items to a human. This costs 3x to 5x tokens on that step and needs a deterministic way to normalize answers. It pays off when a wrong answer is expensive and your single-sample accuracy is already well above 50%; per the voting curve, it hurts when it is below.
Your workflows and methodologies. Use the layer order as a prompt-debugging checklist in code review or eval triage: role and format first, missing facts second, reasoning steps third, stability last. Label each failing eval case with the layer that failed. After a week you will know whether to invest in better retrieval, better instructions, or more samples, instead of guessing.
Your applications. If you ship a RAG feature, the two cheapest wins in the knowledge layer are a HyDE-style query rewrite (one extra call) and a relevance filter on retrieved chunks before they hit the prompt. Measure both against your current retrieval with a small labelled query set before you keep them.
Build Your Own (Minimal Recipe)
A “layered prompt builder” you can stand up in an afternoon:
- Prompt spec object. A small dataclass with
role,intent,domain,demand,output_schema,example_bank,needs_facts,multi_step,n_samples. Render it to text in a fixed order. Version it in git like code. - Example selector. Embed your example bank once; at request time pick the top 3 nearest examples. Shuffle their order across samples so order bias averages out.
- Retriever hook. Optional query rewrite (HyDE), top-k search, an LLM or cross-encoder relevance filter, then paste into the knowledge slot.
- Reasoning hook. For numeric or logic tasks, ask for code and execute it in a sandbox (PAL / Program of Thoughts style). Do not trust the model’s arithmetic.
- Reliability wrapper. Sample
n, normalize, vote, return answer plus agreement rate. Add a verifier call (CRITIC style: check against a tool) only for high-stakes outputs. - Eval harness. 30 to 100 labelled cases, and a run that toggles each layer on and off. This is the part the paper does not give you and the part that tells you what each layer is worth.
The hard parts are (a) answer normalization for voting on free text (use a structured output
schema so you vote on fields, not prose), and (b) knowing when a layer hurts: RAG on a question
the model already knows can add noise, and CoT can raise harmful output rates on sensitive topics.
Only the ablation run catches that. Libraries: any LLM SDK, an embedding model, a vector store or
even an in-memory numpy index, and collections.Counter for the vote. DSPy is worth a look if you
want the layers as composable modules with automatic example selection.
How to Improve It
- Test the taxonomy instead of asserting it. Take 10 tasks, build each prompt layer by layer, and measure the gain each layer adds. Publish which layers matter for which task types. That turns a map into a decision rule.
- Add a “budget” axis. Every layer costs tokens and latency (few-shot examples, retrieved chunks, n samples). A version of the taxonomy that tags each technique with its cost multiplier would let you pick the cheapest fix that works.
- Split tool use out. Tools act on the world and on context, not just on reasoning. Treating “action” as its own layer, matching the agent split the authors started from, would fit modern agent harnesses better.
- Re-cut it for reasoning models. Mark which techniques are redundant with o1-style models (plain CoT prompting, simple self-refine) and which still help (role, task spec, retrieval, external verification, voting on structured outputs).
- Make the reliability layer cheaper. Adaptive sampling: stop early when the first 3 samples agree, and only spend more samples when they disagree. This keeps most of the voting gain at a fraction of the cost.
Glossary
- Hard (discrete) prompt — a prompt made of real text tokens you can read and type.
- Soft (continuous) prompt — learned vectors fed into the model instead of text; out of scope here.
- Prefix prompt — the input comes first and the model continues it (as opposed to fill-in-the-blank cloze prompts).
- In-context learning (ICL) — the model picks up a task from examples in the prompt without any weight updates.
- Zero-shot / few-shot — no examples in the prompt / a few input-output examples in the prompt.
- Label bias — when the mix or order of example labels pushes the model toward one answer regardless of the input.
- Calibration — correcting the model’s output probabilities by subtracting its lean on a content-free input.
- RAG (Retrieval-Augmented Generation) — retrieve relevant documents and paste them into the prompt before generating.
- HyDE / Query2doc — write a fake answer document first, then search with it to find real documents.
- BM25 — a classic keyword-based search scoring method; no neural network.
- Chain-of-Thought (CoT) — prompting the model to write out intermediate reasoning steps before the answer.
- Least-to-Most — break the question into easier sub-questions, solve them in order, feed answers forward.
- PAL / Program of Thoughts — the model writes code for the computation and an interpreter runs it.
- Self-Refine — draft, have the model critique the draft, revise, repeat.
- CRITIC — like Self-Refine, but the critique comes from an external tool such as search or a code run.
- Self-bias — a model’s tendency to rate its own outputs as better than they are.
- Self-Consistency — sample several reasoning paths and take a majority vote on the final answer.
- Bagging / boosting — ensemble methods: combine many independent tries / build tries one by one, each fixing the last one’s mistakes.
- Content bias / value bias — the paper’s terms for answers that drift from the task / answers that are harmful or prejudiced.
- PDDL — Planning Domain Definition Language, a formal way to describe states, actions, and goals for a planner.
- Reasoning model (o1-style) — a model trained to think at length before answering, which builds CoT into the model itself.