Reliability and Effectiveness of Autonomous AI Agents in Supply Chain Management

TL;DR

The MIT Beer Game is the 60-year-old classroom simulation where four players (retailer, wholesaler, distributor, factory) each just decide “how much do I order this week,” and small demand blips turn into huge upstream order swings — the textbook “bullwhip effect.” This paper swaps in LLM agents for the humans and asks two questions: can GenAI agents actually run this well, and can you trust them to run it the same way twice? The answer to the first is yes — a strong reasoning model beats human student teams out of the box, and with the right prompt, guardrails, and shared information it cuts costs by up to 67%. The answer to the second is the paper’s real contribution: even holding everything fixed — same demand, same prompt, same model — repeated runs produce meaningfully different order decisions, and this “agent bullwhip effect” gets worse the further upstream you go and the longer the simulation runs. Averaging over more samples (the standard test-time fix) does not help. What does help is retraining the model itself: a Group Relative Policy Optimization (GRPO) run on a small open-weight model, using total supply-chain cost as the reward, turns a volatile agent into a cheap, stable one that beats much larger frontier models on both cost and consistency.

Problem & Motivation

Two separate, compounding problems motivate this paper:

1. Can LLM agents run a multi-echelon supply chain at all, without hand-built architecture around them? Most prior LLM-for-supply-chain work bolts a single model onto one narrow task (forecasting, one replenishment decision) inside a larger hand-engineered system. Nobody had cleanly measured “if you just hand each tier of a real coordination problem to an off-the-shelf LLM and let it decide,” how good is that, and against a real human baseline?

2. Even if it’s good on average, can you trust it? This is the sharper problem. LLMs are stochastic — sample the same prompt twice and you may get a different order quantity. In a single-turn benchmark that’s a nuisance. In a supply chain it’s dangerous: player k’s order becomes player k+1’s input, delayed by lead time, filtered through inventory and backlog math, for weeks. A decision that differs slightly across two “identical” runs doesn’t just create two different weekly numbers — it perturbs the whole trajectory of inventory positions and backlogs, and those perturbations don’t cancel out, they compound. A system that looks great in expectation but occasionally produces a wildly wrong order is, in the authors’ words, “practically unviable,” because a real firm can’t tell in advance which run it’s about to get.

Existing bullwhip literature (Lee et al. 1997, Chen et al. 2000) explains amplification of demand noise as it moves upstream. It has nothing to say about a supply chain where the decision-makers themselves are a new source of noise, independent of demand. That’s the gap this paper fills, and it’s a gap that generalizes past supply chains to almost any multi-agent LLM pipeline with a feedback loop.

What’s New (Core Contribution)

• A controlled, human-benchmarked GenAI Beer Game. Before: LLM-supply-chain papers mostly report task performance in isolation, with no apples-to-apples human comparison under identical rules. Now: the same 4-role, 20-week, 2-2-2-lead-time game is run by both 100+ Georgia Tech students (12 cohorts, 3 years) and by various LLM configurations, giving a real cost benchmark ($3,207 average human cost) to beat.
• Four named, ranked inference-time levers. Before: “prompt engineering” and “multi-agent orchestration” are treated as a grab-bag of tricks. Now: the paper isolates and ranks four specific levers — model selection, guardrails (a hard budget cap), a curating orchestrator, and prompt framing — and shows their effect sizes vary hugely by model tier (a weak model needs all four; a strong reasoning model barely needs any).
• Agent bullwhip and its decomposition into demand vs. decision bullwhip. Before: “bullwhip” meant order-variance amplification driven by real demand signals. Now: the paper defines a second, orthogonal amplification: run-to-run variance in the same agent’s decision under the same state, and proves (via a law-of-total-variance split) that this decision-driven noise amplifies upstream by the same feedback mechanism as demand noise — even when demand is literally held constant. This is the paper’s most exportable idea: any deployed multi-agent LLM pipeline with feedback has a bullwhip-shaped reliability metric waiting to be measured.
• Evidence that repeated sampling — the standard fix — doesn’t work here. Before: self-consistency / majority-voting / best-of-N is the default cheap remedy for LLM output noise. Now: the paper runs majority vote over 10 and then 100 samples per decision and shows the run-to-run order dispersion barely changes. That’s a useful negative result: it tells you the noise isn’t decoding-level jitter, it’s a property of the policy itself.
• A working GRPO recipe that fixes it. Before: GRPO is established for single-agent reasoning tasks (math, code) with one reward per rollout. Now: the paper trains one shared LLM backbone across all four roles simultaneously, using system-level (whole-supply-chain) cost as reward with per-agent, per-week (“rollout”/return-to-go) credit assignment, then deploys four independent copies of that trained backbone — a genuine centralized-training/decentralized-execution (CTDE) recipe built entirely out of prompting + RL, no custom value network.

How It Works (Technically)

The environment. Four tiers in a line: Retailer ← Wholesaler ← Distributor ← Factory ← (unlimited outside supplier). Each week, every tier sees only its own local state (on-hand inventory, backlog, last order placed, last delivery received) and must emit one number: how much to order upstream. Orders take a week to arrive at the upstream tier; shipments take two weeks to arrive back. Holding excess inventory costs money every week; having a backlog (unmet demand) costs more. Nobody can see anyone else’s state unless a “central orchestrator” agent chooses to share it. This is literally the same game Forrester and Sterman used in the 1960s–80s to demonstrate that rational local decisions produce irrational system-level behavior — the authors just replace the humans with LLM calls (each tier is one LLM prompt per week, parsed for a JSON order_quantity).

The four levers, in the order they matter:

1. Model selection — dominant. A reasoning model (GPT-5 mini) beats the $3,207 human average by 33% completely out of the box. A non-reasoning model (GPT-4o mini) is over 2x worse than humans out of the box, and some weaker models (Phi-4, DeepSeek-R1-0528 at the time tested) simply broke the output-format contract in >25% of turns.
2. Guardrails — a hard per-week budget cap that makes it physically impossible to place a panic order beyond what you can afford. This is the single cheapest, highest-leverage fix: it cut costs 25–41% across models and, importantly, cut reliability problems too (Llama 4 Maverick’s run-to-run coefficient of variation dropped from 46% to 37%).
3. Orchestration — a non-deciding agent with full visibility that hands each tier a curated slice of information. Sharing raw real-time customer demand helps every model (18–38% cost reduction). Sharing more — a 5-week demand history plus volatility stats — helps weak models a lot (69% for GPT-4o mini) but actively hurts strong models, who get distracted by it. More context is not free.
4. Prompt framing — reframing the objective from vague (“minimize total cost”) to specific (“minimize the weighted average of backlog and holding cost”) helps weak models (33–44% reduction) and does almost nothing for strong ones.

The headline: stack all four levers optimally and GPT-5 mini / Llama 4 Maverick 17B beat the human teams by 50–67% on cost.

But averages hide the real story — agent bullwhip. Run the exact same configuration 30 times. If the model had a stable internal policy, you’d expect near-identical trajectories. Instead, order quantities disperse — and they disperse in a structured way along two axes:

• Across echelons, at a fixed week: the retailer’s orders barely move run to run; by the time you get to the factory, the spread is huge. Formally, letting σ²_k,t be the run-to-run (not run-to-time) variance of tier k’s order in week t, they define Ψ_k(t) = σ²_k,t / σ²_{k-1,t}. Ψ_k(t) > 1 means tier k is more unreliable than the tier just downstream of it — the noise amplifies moving upstream, exactly like the classical bullwhip amplifies demand, except this is decision noise.
• Over time, within one echelon: even the same tier gets less reliable the longer the game runs. Φ_k(t) = σ²_{k,t+1} / σ²_{k,t} > 1 means tier k’s instability is still growing in week t+1 relative to week t. Interacting with the environment repeatedly does not make the agent more consistent — a genuinely counterintuitive result if you assumed “feedback” would help an LLM self-correct.

Why doesn’t averaging fix it? The demand/decision split. The key mechanistic insight is a variance decomposition — plain old law of total variance, applied to a new place:

Var(order) = Var-across-demand-scenarios( E[order | demand] )   +   E-over-demand[ Var-across-runs(order | demand) ]
             \_________________ demand bullwhip _________________/   \_________ decision bullwhip _________/

In words: total order variance splits into (a) how much the order would vary if you only changed the demand path and averaged out the model’s internal randomness — this is the classical bullwhip — plus (b) how much the order varies even after fixing the demand path, purely from the agent’s own run-to-run noise. Repeated sampling / majority voting can only ever attack term (b), and only if it drives that term toward zero. The paper shows empirically (constant-demand runs) that decision noise is still large and still amplifies upstream even under literally identical demand — so term (b) is not decoding jitter that averages away, it’s baked into the policy.

The theoretical model tells you why term (b) doesn’t decay. In an appendix, the authors build a stylized linear order-up-to policy (q_k,t = θ_k·forecast_k,t + shock_k,t − inventory_position_k,t, forecast built by exponential smoothing) and derive its transfer function H_k(L) in the lag operator L (think of L as “shift this series back one week”; H_k(L) is a small filter, roughly 1 + gain·(1−L)/(1−(1−λ)L), that turns tier k−1’s order series into tier k’s). Every unit of noise injected anywhere — demand or decision shock — gets passed through one or more copies of this filter as it moves upstream, and the filter has gain Γ = 1 + 2θλ + 2θ²λ²/(2−λ) > 1. That single number Γ > 1 is the whole mechanism: any noise source, injected at any tier, gets multiplied by roughly Γ every tier it crosses — that’s Theorem 1 for demand noise (already known) and Theorem 2 for decision noise (the paper’s extension: decision noise amplifies at the same rate). A companion result (Proposition 3) shows that within one fixed tier, variance under a fixed demand path is non-decreasing over time — it never voluntarily shrinks. So there is no free lunch: the same feedback loop that makes the classical bullwhip happen also amplifies whatever randomness an LLM agent injects, and nothing about “more time” or “more samples” changes that without changing the policy itself.

The fix: GRPO on a shared backbone, system-level reward. If the problem is the policy, fix the policy. GRPO (Group Relative Policy Optimization) is attractive here specifically because a standard actor-critic value function would need to predict expected future cost from a partially-observed, four-way-coupled, lead-time-delayed state — a genuinely hard prediction problem. GRPO sidesteps this: instead of learning “how good is this state,” it just compares several sampled trajectories from the same starting point against each other.

Architecture & data flow

flowchart LR
  C[Customer demand] --> R["Retailer agent (LLM)"]
  R -->|order| W["Wholesaler agent (LLM)"]
  W -->|order| D["Distributor agent (LLM)"]
  D -->|order| F["Factory agent (LLM)"]
  F -->|order| Sup[("Unlimited outside supplier")]
  subgraph Levers["Inference-time levers"]
    O["Orchestrator: shares curated demand info"]
    G["Guardrail: hard budget cap"]
    P["Prompt: specific cost objective"]
  end
  O -.curated data.-> R
  O -.-> W
  O -.-> D
  O -.-> F
  G -.caps order size.-> R
  G -.-> W
  G -.-> D
  G -.-> F

The GRPO training loop

flowchart TD
  Pi["Shared LLM policy πθ<br/>(one backbone, 4 roles)"] --> Roll["Roll out G episodes<br/>4 agents x T weeks each"]
  Roll --> Cost["Record realized cost<br/>per agent, per week"]
  Cost --> Adv["Normalize within the group:<br/>Adv = (cost − group mean) / group std"]
  Adv --> Grad["Weight each decision's log-prob<br/>by its own Adv"]
  Grad --> Update["Gradient step, KL-penalized<br/>toward a frozen reference model"]
  Update --> Pi

Schematic of agent bullwhip: each column is a week, each row an echelon. Box width shows how much a tier's order for that week disperses across repeated, otherwise-identical runs. Toggle between "off-the-shelf" and "after GRPO post-training" to see dispersion collapse — illustrative, built to match the paper's reported pattern (spread grows upstream and over time; post-training compresses it), not the paper's raw numbers.

Reading the two key equations.

Group-relative advantage:

Adv_k,t^(i) = ( r_k,t^(i) − mean_j(r_k,t^(j)) ) / ( std_j(r_k,t^(j)) + ε )

Plain English: take agent k’s reward at week t in episode i (reward = negative cost, so lower cost = higher reward), subtract the average reward that same agent, same week got across the other G sampled episodes in this training step, and divide by the spread. This is z-scoring a decision against its own peer group instead of against a learned value estimate. A positive Adv means “this decision did better than its peers, given everything downstream” — reinforce it. A negative one — suppress it. No critic network required.

The objective:

J(θ) = E[ (1/G) Σ_i (1/(T·|A|)) Σ_t Σ_k  Adv_k,t^(i) · log πθ(y_k,t^(i) | x_k,t^(i))  −  β·KL(πθ ‖ π_ref) ]

This is just “policy gradient, weighted by the advantage above, averaged over every agent-week-episode combination, with a penalty (β·KL) that stops the model from drifting so far from its starting point that it forgets how to speak English or format JSON.” The crucial detail: the sum runs over k (all four roles) and they all update the same parameters θ. So a good decision by the retailer and a good decision by the factory, in the same training step, both push the one shared model in a coordinating direction — that’s how a “centrally trained” model ends up implicitly encoding cross-echelon coordination despite being deployed as four independent, locally-blind copies later.

Credit assignment matters. The paper tests reward at two granularities: episode-level (every decision in a 20-week run gets the same single final-cost signal) vs. rollout / return-to-go (each week’s decision is scored by the cumulative cost from that week to the end of the game only) — and at two scopes: system-level (shared total cost) vs. agent-level (each tier judged on its own cost). The best-performing combination is agent-level, return-to-go: each tier’s week-t decision is judged by what happens to that tier’s own costs from week t onward, which is the most temporally and causally precise signal available.

The algorithm, simplified

# GRPO for a shared 4-role Beer Game policy. Reward = -cost, so lower cost = higher reward.
def grpo_step(policy, ref_policy, roles, T, G, beta, lr):
    episodes = [rollout(policy, roles, T) for _ in range(G)]   # G full 20-week games
    # episodes[i][role][t] = {"logprob": ..., "cost": ...}      # per agent, per week

    grad = zero_like(policy.params)
    for role in roles:
        for t in range(T):
            # return-to-go: sum this role's own cost from week t to the end
            returns = [-sum(ep[role][tau]["cost"] for tau in range(t, T)) for ep in episodes]
            mean_r, std_r = mean(returns), std(returns) + 1e-6
            for i, ep in enumerate(episodes):
                adv = (returns[i] - mean_r) / std_r            # group-relative advantage
                logprob = ep[role][t]["logprob"]
                grad += adv * grad_of(logprob, wrt=policy.params) / (G * T * len(roles))

    kl = kl_divergence(policy, ref_policy)                      # stay close to the starting model
    grad -= beta * grad_of(kl, wrt=policy.params)
    policy.params += lr * grad                                  # ascent: reinforce high-advantage decisions
    return policy

That inner loop is the paper: no value network, just “compare this trajectory’s role-specific future cost to its peers’, and nudge the shared model toward whichever decisions came out ahead.”

Built on Prior Work

Results & Evidence

What was actually measured. Human baseline: 11 recorded runs across two Georgia Tech cohorts (100+ students), average total cost $3,207. AI configurations: GPT-5 mini, Llama 4 Maverick 17B, GPT-4o mini, GPT-4.1 mini, Llama 3.3 70B, Qwen-3 4B, plus assorted weaker models (Phi-4, DeepSeek-R1-0528) evaluated mostly for the “breaks the format” finding. Everything run for 30 identical repeated trials per configuration to measure both mean cost and run-to-run variability (coefficient of variation).

Headline numbers:

• Best AI configuration (Llama 4 Maverick 17B + curated demand + budget guardrail): 67% cheaper than the human average.
• GPT-5 mini completely out of the box (no levers): already 33% cheaper than humans.
• Weak models out of the box (GPT-4o mini): 2.2x more expensive than humans — until orchestration/guardrails/prompting bring them roughly back to human parity.
• Reliability, out of the box: GPT-5 mini and Llama 4 Maverick 17B post 37–46% coefficient of variation across 30 runs — i.e., a materially bad run is a real possibility, not a tail curiosity.
• GRPO post-training (Qwen-3 4B, a small open model): mean cost falls from 1,585 → 952 (a 40% reduction), coefficient of variation falls from 26% → 13%, and the worst observed run across 30 trials falls from 2,847 → 1,353. The trained 4B model then beats GPT-5 mini (3,927 avg, 45% CoV) and Llama 4 Maverick 17B (4,026 avg, 52% CoV) on both cost and reliability — with no budget guardrail or orchestrator needed.

A caveat worth flagging yourself: coefficient of variation alone can mislead. GPT-4o mini out of the box has one of the lowest CoVs in the whole paper (12.6%) — not because it’s reliable in any useful sense, but because it’s reliably bad (costs >2x human, every single run). Low variance and low quality are not the same thing; always read CoV next to the mean, which the paper does but a skimming reader might not.

What the evidence does not establish:

• Topology. Every result is on the classic four-tier serial chain. Real supply/ops networks branch and merge; nothing here tests whether decision bullwhip behaves the same way in a non-serial graph.
• No classical-heuristic baseline for the guardrail. The budget cap is compared against “no guardrail,” not against a well-tuned classical inventory policy (e.g., a properly parameterized order-up-to rule). It’s plausible a good non-LLM heuristic would already close much of this gap on its own — the paper doesn’t test it.
• Training-cost transparency. Hyperparameters for the GRPO run (learning rate, KL coefficient β, number of training steps, total rollout count) aren’t reported in the excerpted main text, and neither is wall-clock/compute cost — you can’t yet do an ROI calculation of “guardrails” vs. “train it” from what’s published.
• Generalization of the trained policy. The model trains on synthetic Poisson/truncated-normal demand curricula and is evaluated on the single classic step-demand pattern (4,4,4,4,8,…,8). Whether the reliability gain survives genuinely out-of-curriculum shocks (a demand spike, a supply disruption) is untested.
• Ecological validity of “beats humans.” The human benchmark is undergraduate students playing a classroom game, not supply-chain professionals making real financial decisions — a reasonable proxy, but not the same as beating experienced practitioners.
• Sample size and significance. 30 runs per configuration is enough to see gross dispersion patterns in a boxplot, but the paper doesn’t report confidence intervals or significance tests on the headline percentage changes.

How You’d Use It

This maps cleanly onto building and running any multi-agent LLM system, not just supply chains:

• Run the reliability audit before you ship, not just the demo. Any multi-agent pipeline with a feedback loop (an agent’s output becomes another agent’s input, repeated over time) has a bullwhip-shaped risk hiding under a good average-case demo. Run N identical repeated trials and compute the demand/decision variance decomposition at every hop. This is a concrete, buildable diagnostic — “here’s your system’s decision bullwhip curve” — most teams shipping on a single good-looking demo never run.
• Hard numeric guardrails are still the cheapest lever, in your harness. Any agent that emits a magnitude with financial or operational consequences (spend, order quantity, resource allocation, position size) should get a hard, non-LLM-enforced cap. This paper’s single highest-leverage, lowest-effort intervention generalizes directly to any multi-agent system with an “actuator” agent.
• Curate, don’t dump, context between agents. The orchestrator finding — real-time demand alone beats real-time demand plus a volatility analysis, for strong models — validates a “less is more” context design for coordinator agents, worth testing explicitly in your own harness rather than assuming more shared state helps.
• A small, RL-post-trained open model can beat a big frontier model on your narrow task. If you can build (or already have) a simulator of your own operational loop, GRPO-style post-training of a cheap open-weight model against system-level reward gets you lower per-call cost and better reliability than calling a frontier model zero-shot forever. The catch (see caveats) is that you need a decent simulator/reward signal, which is the real infrastructure investment.
• Centralized-training, decentralized-execution is a reusable pattern for any multi-role agent team whose roles interact through shared state: train one backbone on pooled, cross-role reward; deploy separate cheap instances per role at runtime.

Build Your Own (Minimal Recipe)

You can build the reliability-measurement half in a day, and the training half in about a week if you already have simulation infrastructure.

Components, in build order:

1. The simulator. Beer Game logic is short: inventory position, backlog, shipment lead time, order lead time, holding/backorder cost. Port the paper’s Appendix B recursions directly (IP_{k,t+1} = IP_{k,t} + q_{k,t} − q_{k-1,t}, shipments capped by on-hand + inbound).
2. The agent harness. One prompt template per role (mirror Appendix A.2 — strict “output one JSON object, nothing else” contract), each agent seeing only its own local state each week.
3. The reliability metric — build this before you optimize anything. Run R (say, 30) identical trials under a fixed demand path. For each tier/week, compute the run-to-run variance; compute Ψ_k(t) (across-tier ratio) and Φ_k(t) (within-tier, across-time ratio). This alone tells you where your system’s bullwhip lives, and it’s a useful diagnostic on its own.
4. The cheap levers. Add a hard budget/order cap; add a thin orchestrator that shares one or two curated signals (not everything); tighten the prompt’s stated objective. Re-measure. Often this is enough.
5. GRPO, only if you still need it. Reach for TRL, verl, or OpenRLHF for the GRPO loop; vLLM for fast rollout sampling; a small open-weight model (Qwen3-4B-class) as the trainable backbone. Reward = negative cumulative cost, agent-level + return-to-go attribution (the paper’s best-performing combination). Train across a demand curriculum broader than your evaluation distribution.

The 1–2 genuinely hard parts:

• Building a reward signal you actually trust. GRPO itself is a fairly mechanical loop; the real engineering risk is having a simulator/cost function faithful enough that “reward = -cost” actually reflects what you care about in production. Get this wrong and you’ll train a confidently wrong policy.
• Credit assignment across roles sharing one backbone. The paper’s ablation shows agent-level, return-to-go rewards beat episode-level, system-level rewards by a wide margin — getting this granularity right (rather than defaulting to “one number per game”) is the difference between a useful training signal and mush.

How to Improve It

1. Test non-serial topologies. Both the empirical Beer Game and the theoretical linear-benchmark model are single serial chains. Extend the simulator to a branching/merging network and check whether the same Γ > 1 amplification logic (and the GRPO fix) still holds.
2. Add a classical-heuristic baseline. Compare the LLM-plus-guardrail setup against a properly tuned, non-LLM order-up-to policy. If a good classical heuristic gets most of the way there, the paper’s “LLM agents beat humans” framing needs a stronger comparison point than “LLM agents beat undergrads.”
3. Publish the training compute budget. Report GRPO hyperparameters, rollout counts, and wall-clock/compute cost so a reader can actually compute “train it” vs. “guardrail it” ROI — right now the recipe is directionally clear but not reproducible from the paper alone.
4. Stress-test out-of-curriculum. Evaluate the trained policy against demand shocks, supply disruptions, or adversarial inputs it never saw during Poisson/truncated-normal training, to see whether the reliability gain is genuine generalization or curriculum overfitting.
5. Ablate shared-backbone vs. per-role specialization. The paper never tests whether four separately fine-tuned per-role models would beat one shared backbone trained with pooled reward — a natural next experiment given how much the paper leans on the CTDE framing.

Glossary

• Beer Game — a four-role (retailer, wholesaler, distributor, factory) supply-chain simulation from the 1960s where each player just decides a weekly order quantity under lead-time delays and holding/backlog costs.
• Echelon / tier — one stage of the supply chain (e.g., “the wholesaler”); tiers are numbered upstream from the customer.
• Bullwhip effect — the classical finding that small demand fluctuations produce progressively larger order swings as you move upstream through a supply chain.
• Agent bullwhip — this paper’s extension: run-to-run decision instability (not demand-driven) that also amplifies upstream and over time when LLM agents make the decisions.
• Decision bullwhip vs. demand bullwhip — splitting total order variance (via the law of total variance) into the part caused by external demand changes (demand bullwhip) vs. the part caused purely by the agent’s own run-to-run randomness under fixed demand (decision bullwhip).
• Coefficient of variation (CoV) — standard deviation divided by the mean; used here to measure how unreliable a model’s cost is across repeated identical runs, independent of scale.
• Order-up-to policy — a classic inventory rule: order enough this period to bring your inventory position up to a target level based on your demand forecast.
• Inventory position — on-hand inventory, plus what you’ve already ordered but not received, minus your backlog.
• Lead time — the delay between placing an order/shipment and it arriving; the Beer Game has separate order and shipment lead times.
• Lag operator (L) — notation where L·x_t = x_{t-1}; used to write “how today’s decision depends on yesterday’s” compactly as a small algebraic filter.
• Transfer function / gain (Γ) — the multiplier a noise source picks up each time it passes through one tier’s ordering rule; Γ > 1 means noise grows as it moves upstream.
• Group Relative Policy Optimization (GRPO) — an RL post-training method that skips a learned value function and instead scores each sampled trajectory relative to a group of peer trajectories sampled at the same training step.
• Advantage — how much better (or worse) a specific decision’s outcome was than the average of its peer group; positive advantages get reinforced, negative ones suppressed.
• KL penalty / reference policy — a term in the training objective that keeps the updated model from drifting too far from its original (pre-training) behavior, preventing it from “forgetting” basic competence while chasing reward.
• Rollout / return-to-go attribution — scoring a decision made at week t using the cumulative cost from week t through the end of the episode, rather than one flat reward for the whole game.
• Centralized-training, decentralized-execution (CTDE) — training one policy with full, pooled information across all roles, then deploying separate copies that each act on local information only.
• Orchestrator — in this paper, a non-deciding agent with full visibility that selectively shares curated information with the deciding agents.