InvAgent: A Large Language Model based Multi-Agent System for Inventory Management in Supply Chains

TL;DR

Multi-echelon inventory management — the classic “beer distribution game” problem of deciding how much to order at each stage of a supply chain without knowing exact future demand — has historically been solved with either hand-tuned heuristics (simple but rigid) or reinforcement learning (adaptive but expensive to train and opaque). This paper asks a simpler question: what if you just describe the problem to GPT-4 in natural language, once per stage per round, and let it decide? InvAgent wires up one LLM “agent” per supply-chain stage (retailer, wholesaler, distributor, manufacturer) plus an orchestrating “user proxy” that runs the simulation loop, feeds each agent its current inventory/backlog/demand state as a formatted prompt, and reads back both a chain-of-thought justification and an order quantity. Across five demand scenarios (constant, variable, high-variance, seasonal, and normally distributed), InvAgent lands in the middle of the pack — beating simple heuristics, competitive with but usually slightly behind a fully-trained multi-agent RL policy (MAPPO) — while requiring no training data, no reward engineering, and producing a human-readable reason for every order it places.

Problem & Motivation

Inventory management in a multi-stage supply chain is a sequential decision problem under uncertainty: each stage (say, manufacturer → distributor → wholesaler → retailer) has to decide how much to order from its upstream supplier every period, without knowing exactly what the next period’s downstream demand will be, while carrying costs for unsold inventory, penalties for backlogged (unfulfilled) orders, and lead times that mean today’s order doesn’t arrive for several periods. Order too much and you eat holding costs; order too little and you stock out and backlog builds. Coordinate badly across stages and small demand fluctuations amplify into wild oscillations upstream — the well-known bullwhip effect.

Two families of solutions existed before this paper, and both have real costs:

• Heuristic policies (base-stock, tracking-demand) are simple, fast, and require no training — but they’re static formulas. They don’t reason about context, can’t explain why a particular order makes sense, and degrade when demand patterns get complex (seasonal jumps, high variance).
• Reinforcement learning (IPPO, MAPPO) can learn genuinely adaptive, high-performing policies — but only after extensive training with a simulator, careful reward and hyperparameter tuning, and the result is a black-box policy network. You can’t ask it why it ordered 6 units instead of 4.

The gap the authors target: nobody had systematically tested whether an LLM, using nothing but its pretrained knowledge and a well-structured prompt (zero-shot, no fine-tuning, no simulator-based training), could act as the decision-maker at each stage and hold its own against both families — while adding the one thing neither heuristics nor RL give you for free: a written explanation of the reasoning behind each order.

What’s New (Core Contribution)

• LLMs as zero-shot multi-agent inventory controllers. Before: every prior SCM automation approach that adapts to context needed either a hand-built heuristic formula or a trained policy (hundreds to thousands of simulated episodes). Now: each stage is an LLM prompted once per round with its current state; no training loop, no reward function, no simulator access during “training” — the model reasons from the prompt alone, using knowledge already baked in from pretraining.

• Explainability via chain-of-thought as a first-class output. Before: heuristic policies are a formula (no “why”); RL policies are a black-box network (no “why” beyond a Q-value). Now: every agent is asked to state its reasoning in 1–2 sentences before committing to an action — turning a numeric decision into an auditable sentence a supply-chain manager could actually read and sanity-check.

• A structured, reusable prompt template for stage-level inventory decisions. Before: ad hoc single-LLM SCM demos existed (see Prior Work), but none formalized a repeatable four-part prompt (state, demand, downstream order, strategy) tested across a multi-agent simulation loop. Now: the paper defines and ablates a specific prompt anatomy — state description, demand description, downstream-order signal, and an optional hand-written “golden rule” strategy tip — and measures which parts actually help.

• A rigorous like-for-like comparison against heuristic and MARL baselines across five demand regimes, using the same OR-Gym-style multi-echelon simulator for every method. This is the part that makes the paper’s honesty about not beating MAPPO valuable — most LLM-agent papers skip a fair RL baseline entirely.

Be clear-eyed about what’s not new: the multi-echelon inventory formalism (Equations 1–5) is lifted essentially unchanged from OR-Gym (Hubbs et al. 2020) and the classic beer-game literature; the RL baselines are off-the-shelf PPO variants; and “put an LLM in the loop of a simulation” is by now a well-worn pattern (economic and trading multi-agent LLM systems predate this paper by a year). The genuine contribution is applying that pattern carefully and specifically to inventory management, with a real ablation, not just a demo.

How It Works (Technically)

The environment (the part that’s borrowed, not invented). The supply chain is modeled as M stages numbered 0 (retailer, closest to the customer) through M-1 (the stage with unlimited raw-material supply). Each stage has an inventory holding area and a production area; one unit of inventory makes one unit of product; there’s a lead time L_m between stages for shipments to arrive. Every period, four things happen in order: (1) incoming shipments that cleared their lead time arrive, (2) each stage places a replenishment order upstream and the retailer receives customer demand, (3) orders/demand get fulfilled as far as inventory and upstream production capacity allow (shortfalls become backlog), (4) profit is computed. That’s it — that’s the whole simulator, and it’s identical in spirit to the beer distribution game every supply-chain course teaches.

The agents (the actual contribution). InvAgent puts one LLM “stage agent” at every stage, plus a user proxy that plays orchestrator/referee. The user proxy doesn’t make any decisions itself — it resets the environment, pulls the current state for each stage, hands that state to the corresponding stage agent as a prompt, collects the returned order quantities, steps the environment forward with all of them at once, and repeats until the episode ends.

Architecture & data flow

flowchart LR
  ENV[("Environment<br/>multi-echelon<br/>inventory simulator")]
  UP["User Proxy<br/>(orchestrator)"]
  R["Retailer Agent<br/>(LLM, stage 1)"]
  W["Wholesaler Agent<br/>(LLM, stage 2)"]
  D["Distributor Agent<br/>(LLM, stage 3)"]
  M["Manufacturer Agent<br/>(LLM, stage 4)"]
  ENV <-->|state, reward| UP
  UP <-->|prompt / action| R
  UP <-->|prompt / action| W
  UP <-->|prompt / action| D
  UP <-->|prompt / action| M
  R -.->|downstream order signal| W
  W -.->|downstream order signal| D
  D -.->|downstream order signal| M

The solid arrows are the mechanical loop (state out, action in); the dotted arrows are the one piece of inter-agent context each stage gets about its neighbor — “here’s what your downstream stage just ordered” — which is not the raw chat history, just one number, injected by the user proxy into the next prompt.

The per-period environment mechanics (Equations 1–5, demystified). Skip past the subscripts: this is four bookkeeping rules applied every period, per stage m:

• Inventory update — I[m,t] = I[m,t-1] + R[m, t-L_m] - S[m,t]. Plain English: today’s inventory is yesterday’s inventory, plus whatever shipment finally arrived after sitting in transit for the lead time L_m, minus whatever you sold/shipped out this period.
• Fulfilled order (R) — how much of what you asked your supplier for actually shows up, capped by three things at once: your supplier’s leftover backlog obligation to you, your supplier’s production capacity, and your supplier’s total available inventory (their stock plus their own incoming shipments). Fulfilled order is the minimum of those three — you never get more than the tightest bottleneck allows. At the very top of the chain, orders are always fully filled (infinite raw material assumed).
• Sales (S) — for every stage except the retailer, sales just equal whatever the downstream stage successfully drew from you (R one stage down). At the retailer, sales are capped by the same three-way minimum logic but against customer demand instead of a downstream order.
• Backlog (B) — running total of unmet demand: previous backlog plus new orders/demand, minus what actually got fulfilled this period. It only grows if you can’t keep up.
• Profit (P) — sales revenue − procurement cost − backlog penalty − holding cost, computed independently at every stage every period. This is the number a heuristic or an RL policy is explicitly optimizing; the LLM agent never sees it as a reward signal at all — it only sees state and is asked to reason about cost and stockouts in words.

The state each agent actually receives is the vector s = [c_m, p_m, r_m, k_m, h_m, L_m, I(t-1), B(t-1), B_upstream(t-1), recent sales history (zero-padded to L_max), incoming deliveries] — but the agent never sees this as a vector. It’s rendered into English: lead time, inventory level, current backlog, upstream backlog, previous sales from oldest to newest, and arriving deliveries from nearest to farthest. The zero-padding on sales history exists purely so every stage’s prompt has a fixed-length “recent sales” list even on round 1, when there’s no history yet — a bookkeeping detail, not a modeling choice the LLM needs to reason about.

The prompt anatomy (Figure 4 in the paper) has four swappable parts:

1. State description — the rendered state vector above.
2. Demand description — plain-English description of the demand distribution the retailer faces this scenario (e.g., “a discrete uniform distribution U{0,4} for all 12 rounds”). Every stage gets this, not just the retailer, so upstream stages can reason about the ultimate source of variability.
3. Downstream order description — “your downstream order from stage X for this round is Y” — a fast, low-latency signal that lets an upstream agent react to what just happened one hop away without waiting for it to propagate through inventory changes.
4. Strategy description — an optional, hand-written paragraph stating the “golden rule” (open orders should equal expected downstream demand plus backlog), warnings about lead time and the bullwhip effect, and a nudge to spread orders over multiple rounds rather than dumping a big order at once.

The response format is deliberately constrained: state your reason in 1–2 sentences, then give the order quantity as a non-negative integer in brackets, e.g. [4]. That bracket format is what makes the response machine-parseable while still forcing chain-of-thought before the number — one sentence enforces post-hoc justification for the model, but ordering “reason first, then answer” is what actually improves the answer (the CoT ablation confirms this matters).

One round, traced end to end

sequenceDiagram
  participant Env as Environment
  participant UP as User Proxy
  participant Ret as Retailer Agent (LLM)
  Env-->>UP: state(round=1, stage=1): inv=12, backlog=0, lead_time=2, sales=[0,0]
  UP->>Ret: prompt = system_msg + state + demand + downstream_order + strategy
  Note over Ret: "Inventory covers ~3 rounds of max demand,<br/>lead time is 2 rounds -> don't over-order."
  Ret-->>UP: "Reason: ... Action: [0]"
  UP->>UP: parse bracketed integer -> action = 0
  UP->>Env: submit actions for all 4 stages
  Env-->>UP: next_state, reward (Eq. 1-5 applied), done?
  Note over UP,Env: loop repeats for 12 rounds, or until episode ends

This is the paper’s own worked example (Figure 9, GPT-4, constant-demand scenario): with 12 units of inventory, zero backlog, and a 2-round lead time against a demand that never exceeds 4/round, the retailer agent reasons its way to ordering nothing this round — exactly the kind of context-sensitive restraint a fixed base-stock formula can’t express (base-stock would order up to capacity, overshooting).

The algorithm, simplified

# The InvAgent decision loop for one round, one stage. The environment step
# (Eq. 1-5) is standard OR-Gym-style bookkeeping; THIS loop is the contribution.

def build_state_description(state):
    # state: dict with inventory, backlog, upstream_backlog, lead_time,
    # recent_sales (zero-padded), incoming_deliveries
    return (
        f"- Lead Time: {state['lead_time']} round(s)\n"
        f"- Inventory Level: {state['inventory']} unit(s)\n"
        f"- Current Backlog (you owing downstream): {state['backlog']} unit(s)\n"
        f"- Upstream Backlog (your upstream owing you): {state['upstream_backlog']}\n"
        f"- Previous Sales (old to new): {state['recent_sales']}\n"
        f"- Arriving Deliveries (near to far): {state['incoming_deliveries']}"
    )

def build_prompt(round_i, stage, n_stages, state, demand_desc,
                  downstream_order, use_strategy=True):
    prompt = (
        f"Now this is round {round_i}, and you are at stage {stage} of "
        f"{n_stages} in the supply chain. Given your current state:\n"
        f"{build_state_description(state)}\n\n"
        f"{demand_desc}\n{downstream_order}\n"
        "What is your action (order quantity) for this round?\n"
    )
    if use_strategy:
        prompt += GOLDEN_RULE_STRATEGY_TEXT           # human-crafted, optional
    prompt += (
        "\nPlease state your reason in 1-2 sentences first, then give your "
        "action as a non-negative integer in brackets (e.g. [0])."
    )
    return prompt

def parse_action(llm_response: str) -> int:
    # pull the integer out of the last [..] in the response; forced non-negative
    import re
    match = re.findall(r"\[(\d+)\]", llm_response)
    return int(match[-1]) if match else 0

def run_episode(agents, env, n_rounds, use_strategy=True):
    state = env.reset()
    history = {a: [] for a in agents}                  # kept across the whole episode
    for t in range(1, n_rounds + 1):
        actions = {}
        for stage, agent in agents.items():
            prompt = build_prompt(t, stage, len(agents), state[stage],
                                   env.demand_description(),
                                   env.downstream_order_description(stage),
                                   use_strategy)
            response = llm(system=agent.system_message,
                            history=history[stage], prompt=prompt)   # chat, not stateless
            history[stage].append((prompt, response))
            actions[stage] = parse_action(response)
        state, reward, done = env.step(actions)         # applies Eq. 1-5 for every stage
        if done:
            break
    return reward

The whole trick is in build_prompt and the fact that llm(...) is called with the running chat history for that stage, not a fresh context each time — the ablation shows dropping history costs a meaningful chunk of performance, because the agent loses track of its own recent reasoning and commitments.

A schematic 4-stage chain running the paper's own bookkeeping equations (1)–(5) under a simple reactive ordering rule. Press play and watch how a demand spike at the retailer amplifies as it propagates upstream — the bullwhip effect the "golden rule" strategy text is explicitly trying to talk the LLM out of causing. Illustrative simulation built from the paper's equations, not the paper's own logged run.

How the fixed-length state vector shown to each agent is assembled from a variable amount of history via left-zero-padding. Slide the round number to see the sales/delivery windows fill in as the episode progresses — a bookkeeping detail worth seeing once so the prompt template stops looking arbitrary.

Built on Prior Work

Results & Evidence

Across five demand scenarios (constant, variable U{0,4}, larger U{0,8}, seasonal jump, and normal N(4,2²)), InvAgent’s mean episode reward (higher/less negative is better; rewards are negative because they’re net costs) lands roughly in the middle of the six methods tested:

• MAPPO wins outright in four of five scenarios (constant, larger, seasonal, normal); its centralized value function clearly pays off when it’s had the training budget to use it.
• InvAgent (without the hand-crafted strategy) wins the variable-demand scenario outright — the one case where reacting flexibly to noisy, low-magnitude demand mattered more than optimizing to a learned formula.
• InvAgent consistently beats both heuristic baselines (base-stock, tracking-demand) in most scenarios, which is the more relevant comparison: it’s a fairer fight (neither method trains) and InvAgent wins it.
• The ablation (Table 4, variable-demand scenario) is the most informative result in the paper. Removing the hand-crafted strategy text improved the reward by ~12%; removing the demand description hurt by ~4%; removing the downstream-order signal hurt by ~24%; dropping chain-of-thought hurt by ~13–15%; dropping chat history hurt by ~20%; and swapping GPT-4 for GPT-4-Turbo cratered performance by ~89%. In other words: the downstream-order signal, CoT, and conversation history are doing the real work — the “golden rule” strategy paragraph the authors hand-wrote can actually get in the way when demand is just noisy rather than seasonal, and model choice matters enormously.

What this does and doesn’t establish. It establishes that a genuinely zero-shot, untrained LLM system is competitive with heuristics and in the ballpark of trained MARL on this specific, small (4-stage, 12-period, single-product) benchmark — a real and useful data point. It does not establish that this scales to realistic SKU counts, longer horizons, multiple products competing for the same production capacity, or partial/asymmetric information between stages. Results are averaged over only 5 episodes per LLM configuration (100 for the trained baselines), so the reported standard deviations are wide and some scenario “wins” (e.g., variable demand by 41.6 points) should be read as suggestive, not conclusive. There’s also no cost analysis — InvAgent needs a live LLM call per stage per round (48 calls for a 12-round, 4-stage episode), which is nontrivial to run at real supply-chain scale and frequency compared to a trained policy’s near-zero inference cost.

How You’d Use It

This is a clean template for a class of problem broader than inventory: any recurring, structured, multi-party operational decision in your own operation that currently runs on a fixed formula or spreadsheet, where the decision would benefit from contextual judgment and an audit trail.

• A “reasoning layer” over your existing planning system, not a replacement for it — a concrete automation to add to your ops pipeline. If you already run a base-stock or reorder-point system in your ERP, don’t rip it out — put an LLM agent next to it that sees the same state, produces a recommended deviation plus a written reason, and a human (or a rules gate) approves or overrides it. That’s a low-risk way to pilot this in your own operation: “explainable second opinion on every reorder decision,” not “replace your MRP system” — and it fails safe, since the existing system keeps running underneath it.
• The four-part prompt anatomy is directly reusable in your harness for other multi-echelon or multi-party coordination problems inside your own automations: staffing/shift planning across locations, cash allocation across business units, capacity allocation across production lines. State description + peer-signal description (“downstream order”) + optional strategy nudge + forced reason-then-answer is a pattern, not an inventory-specific trick — it slots in wherever you already have per-stage or per-unit state to describe.
• The ablation is a workflow worth stealing. Knowing exactly which part of a prompt is carrying the performance — versus just shipping an agent because “it seems to work” — is the difference between an agent you can trust with real decisions and one you’re guessing about. Bake this kind of ablation into any agentic pilot before you let it touch production, and expect a day or two of extra harness work to wire up the comparisons.
• Where it loses to RL, plan around it. If your own operation already has enough historical data and stable-enough demand patterns to train MAPPO/IPPO, that will likely still outperform an LLM agent on raw cost — the honest case for InvAgent-style systems in your stack is speed-to-deploy, zero training data requirement, and explainability, not “beats the trained model.” That only pays off if explainability and fast setup actually matter more to you than squeezing out the last few points of cost.

Build Your Own (Minimal Recipe)

This is one of the cheaper “real” multi-agent systems to prototype — a working toy is a day’s work if you already have LLM API access.

Components, in build order:

1. The environment. Implement Equations 1–5 directly as a small Python class with a .reset() and .step(actions_dict) -> (state, reward, done) — OR-Gym’s InvManagement environments are literally this, or write it yourself in ~80 lines; it’s just the four-step bookkeeping described above per stage per period.
2. The prompt builder. Port the four-part template (state, demand, downstream order, strategy) verbatim from Figure 4 — it’s given in full in the paper and reproduced above. Start with the strategy text included; you’ll want to ablate it later.
3. One LLM agent per stage, each with its own system message (a one-paragraph role description: “you are stage N of M, minimize total cost”) and its own running chat history — this is exactly AutoGen’s ConversableAgent pattern, or a plain dict of message lists if you don’t want the dependency.
4. The orchestrator (user proxy). A simple loop: for each round, for each stage, build the prompt, call the LLM, parse the bracketed integer with a regex, collect all actions, step the environment, repeat. No framework required — the “AutoGen” framing in the paper is really just this loop wearing a library’s clothing.
5. Baselines, so you know if it’s working. Implement base-stock (Eq. 8) in 3 lines; it’s your sanity floor. If InvAgent can’t beat base-stock, something’s wrong with the prompt or the parsing.

The 1–2 genuinely hard parts:

• Action parsing robustness. The paper’s [N] bracket format is simple but brittle in practice — models occasionally hedge, give a range, or bury the number mid-paragraph. Budget time for a retry-with-clarification loop or a switch to structured output (JSON mode / function calling) rather than regex-on-free-text; this is the single most common silent failure mode in these systems.
• Deciding what goes in the “strategy” text, and testing whether it helps. The paper’s own ablation shows their hand-written strategy paragraph hurts in the variable-demand case and helps in the seasonal case. Don’t assume more guidance is better — treat every strategy sentence as a hypothesis to A/B test against a no-strategy baseline, exactly as this paper did.

Reach for: any tool-calling-capable model (the paper shows GPT-4 clearly outperforming GPT-4-Turbo here, so don’t assume “newer/cheaper model” is a safe swap without testing), AutoGen or a plain message-history dict for the multi-agent bookkeeping, and Gymnasium’s environment interface if you want the eval harness to be swappable with RL baselines later.

How to Improve It

1. Replace the free-text bracket parse with structured output. Force the model to return {"reason": str, "action": int} via tool calling / JSON schema. Removes an entire failure class for near-zero cost and would likely improve the reported numbers on its own — a very testable, very cheap first experiment.
2. Make the strategy text conditional instead of static. The ablation shows the golden-rule paragraph helps on seasonal demand and hurts on variable demand. Add a lightweight classifier or a meta-prompt that detects “how patterned is recent demand” and decides whether to include the strategy text per-episode, per-stage. This directly attacks the paper’s own most interesting negative result.
3. Compress the chat history instead of keeping it raw. The ablation shows dropping history costs ~20%, but raw full-episode history doesn’t scale past 12 rounds or a handful of products. Swap it for a rolling structured summary (last N rounds’ key stats plus a running one-paragraph memory) and test whether you recover the performance at a fraction of the context cost — a direct, measurable trade you can benchmark against the paper’s own ablation numbers.
4. Close the gap to MAPPO with cheap offline learning, as the authors suggest. Log the (state, CoT reasoning, action, resulting reward) tuples across many InvAgent episodes, then either fine-tune on the highest-reward trajectories (rejection sampling / DPO-style) or use them as few-shot exemplars in the prompt. This targets the paper’s own stated future work and is directly measurable against Table 3.
5. Test on real demand data and larger networks. Every scenario here is synthetic (uniform/normal distributions) and 4 stages deep. Swap in real SKU-level demand history and a wider/deeper network topology (branching, not just a chain) to see whether the prompt template and the “downstream order” signal still carry the same weight — the paper explicitly flags this as untested.

Glossary

• Multi-echelon inventory system — a supply chain with several sequential stages (e.g., manufacturer → distributor → wholesaler → retailer), each holding its own inventory and ordering from the stage above it.
• Lead time — the number of periods between placing an order and it arriving as usable inventory.
• Backlog — demand or orders you couldn’t fulfill this period, carried forward and owed next period.
• Bullwhip effect — the tendency for small demand fluctuations at the retail end of a supply chain to amplify into large order swings further upstream.
• Base-stock policy — a heuristic that always orders up to a fixed target inventory level equal to production capacity.
• Tracking-demand policy — a heuristic that sets the target inventory based on a moving average of recent sales plus current backlog, rather than a fixed level.
• Zero-shot learning (in this context) — the LLM makes decisions using only its pretrained knowledge and the current prompt, with no task-specific training or examples.
• Chain-of-thought (CoT) — prompting the model to write out its reasoning before giving a final answer, used here to both improve decision quality and produce an explanation.
• User proxy — the non-decision-making orchestrator agent that manages the simulation loop: pulling state, distributing prompts, collecting actions, stepping the environment.
• IPPO (Independent Proximal Policy Optimization) — a multi-agent RL setup where each agent trains its own PPO policy independently (with shared parameters here), without explicit coordination.
• MAPPO (Multi-Agent PPO) — an extension of PPO for multi-agent settings that uses a centralized value function seeing all agents’ information, improving training stability and typically outperforming IPPO.
• AutoGen — Microsoft’s open-source framework for building multi-agent LLM conversations (system messages, agent roles, message-passing orchestration); used here to implement the agent topology.
• OR-Gym — an open-source library providing Gymnasium-style RL environments for operations-research problems, including the multi-echelon inventory formulation this paper reuses.
• Episode reward — the cumulative profit (or, since it’s usually negative here, net cost) summed across all stages and all periods of one full simulation run.