Large Language Models for Supply Chain Decisions

TL;DR

Big companies run expensive optimization solvers to make supply-chain decisions (where to ship hardware, how to route around a factory outage, what a tariff change costs). The solvers work, but a planner who is not a data scientist can’t understand why the solver decided something, can’t ask “what if demand jumps 15%?”, and can’t tweak the model when reality changes — every such request becomes a multi-day ticket to the engineering team. This paper’s idea is to wrap the existing solver in three LLM-powered capabilities: explain (translate plan + data questions into SQL/queries), what-if (translate a question into a small change to the optimization model, re-solve, and explain the delta), and interactive planning (let the planner mutate the model directly). The headline production result: deployed at Microsoft Azure’s cloud supply chain on GPT-4, ~90% answer accuracy, ~23% reduction in fulfillment-investigation time, week-long analyses collapsed to minutes. The key architectural trick: the LLM emits code, the code runs against your data and solver locally, and your data is never sent to the LLM.

Problem & Motivation

A modern supply chain is a graph: suppliers → factories → warehouses → retailers, with costs, capacities, lead times, and contractual constraints on every edge. Companies already solve this well with mathematical optimization — you write the business problem as a mathematical program (minimize total cost subject to constraints) and a solver finds the optimal plan. This is mature technology; Microsoft, Amazon, and Google all run it at scale.

The pain is not the math. The pain is the human interface to the math. Three concrete failure modes:

1. “Why did it decide this?” A planner gets a fulfillment plan with thousands of decisions and tens of interdependent constraints. They can’t see the reasoning behind any single assignment, so they file a ticket and wait for an engineer.
2. “What if X changes?” Demand shifts, a factory goes down in a storm, a new tariff lands. Answering “what’s the cost impact?” means an engineer hand-edits the model and re-runs it — days of back-and-forth.
3. “The model is stale.” The real business changed; the model hasn’t. Updating it requires the data-science team.

Every one of these routes a business question through a scarce technical resource. The decision-makers (planners, executives) have the business context but not the optimization or SQL skills, so they’re permanently dependent on a bottleneck. The result: decisions that could be made in minutes take days or weeks, and the expensive optimization tooling is underused because nobody can drive it.

What’s New (Core Contribution)

This is an applied/systems chapter, not an algorithms paper. Its contributions are an architecture and a production validation, not a new model. Be honest about that — the novelty is in the integration pattern, which is genuinely useful.

• LLM-as-translator, solver-as-truth (the central idea). Before: you either trust the solver (opaque) or you ask an LLM to reason about numbers directly (unreliable, and you’d have to hand it your data). Now: the LLM never computes the answer — it translates human language into a small code change to the existing math model, the trusted solver computes, and the LLM translates the numeric result back to English. The solver remains the source of truth; the LLM is just bidirectional glue.
• Data stays home. Before: RAG-style approaches stuff your proprietary data into the prompt and ship it to a third-party model. Now: the LLM emits code that runs locally against your SQL/solver; only the question and a few generic examples ever touch the LLM. This is the single most commercially important design choice in the paper.
• In-context learning over a question→code library. Before: fine-tune a model on your domain (costly, brittle). Now: maintain a repository of (question, code) pairs; at query time, retrieve a few relevant pairs, append them to the prompt as examples, and let GPT-4 pattern-match. No training run required.
• A production deployment with real numbers. Before: most “LLM for supply chain” claims are demos. Now: shipped inside Azure’s cloud supply chain — GPT-4, ~90% accuracy, ~23% time savings, plus procurement case studies (an OEM found millions in unclaimed contract volume discounts). This is the evidence that the pattern survives contact with production.

How It Works (Technically)

The system is a loop with four stations. Walk one real question through it.

Question: “What would be the additional cost if retailer R can only use products from factory F?”

Station 1 — Question handler. The raw question comes in. The handler preprocesses it and, critically, retrieves a few relevant (question, code) example pairs from a curated repository and appends them to the prompt. This is in-context learning: instead of teaching the model your domain via fine-tuning, you show it 3–5 worked examples right before the real task. The examples teach it your schema names, your solver’s API, and the shape of the code you expect back.

Station 2 — The LLM. GPT-4 receives [examples] + [user question] and outputs application code — not an answer. For our question, it recognizes this as a constraint addition: “force retailer R to source only from F” becomes, in optimization terms, “prohibit every other factory from shipping to R.” Mechanically that’s adding constraints like flow[f', R] == 0 for all f' != F to the existing model. The crucial design point repeated throughout the paper: the LLM produces a small diff to a known-good model, not a model from scratch. A diff is verifiable and hard to get catastrophically wrong; a from-scratch model is neither (the authors explicitly say they cannot yet validate full models the LLM writes).

Station 3 — The application. The emitted code executes locally. It may (a) hit the data repository (a SQL query for “explain” questions), or (b) invoke the optimization solver with the modified model (for “what-if” questions). For our example, the solver re-runs with the extra constraint and returns a new plan and a new total cost. This is where the actual computation happens — on trusted infrastructure, with real data, by deterministic tools.

Station 4 — Interpreter. The numeric output (old cost vs. new cost, demand lost, service-level impact) is handed back to the LLM with a “explain this to a planner” instruction. The LLM emits a plain-English sentence: “Restricting R to factory F raises total cost by $42,000 (+3.1%) and leaves 4% of R’s demand unfilled.”

Notice the LLM appears twice — once on the way in (English → code) and once on the way out (numbers → English) — and the data flows through neither. The arrow that doesn’t exist (data → LLM) is the whole privacy story.

There is essentially no new mathematics in this paper. The “math” is standard mathematical programming: an objective function (minimize cost) and constraints (capacity, demand satisfaction, compatibility). What-if questions map cleanly onto small constraint/coefficient edits:

• “demand +15%” → multiply the demand parameter vector by 1.15 and re-solve.
• “shut down factory F” → add production[F, *] == 0.
• “raw material T is $1 cheaper” → decrement that cost coefficient and re-solve.
• “R can only use F” → zero out competing flow variables.

The insight worth internalizing: the universe of common what-if questions is a small, enumerable set of model edits. That’s exactly why in-context learning works here — you only need a few dozen example transformations to cover most real queries.

Architecture & data flow

flowchart LR
  U[Planner asks in English] --> QH[Question Handler]
  QH -->|append k example<br/>question-code pairs| LLM1[LLM: English to code]
  LLM1 -->|application code / model diff| APP[Application Runtime]
  APP -->|SQL query| DB[(Proprietary Data)]
  APP -->|modified model| SOLVER[Optimization Solver]
  DB --> APP
  SOLVER --> APP
  APP -->|numeric result| LLM2[LLM: numbers to English]
  LLM2 --> ANS[Plain-English answer to planner]
  DB -. never sent .-x LLM1

Step through the request loop. Click "Next" to advance the token of work through the four stations — watch where the proprietary data lives (it never crosses into the LLM boxes). Schematic, not the paper's actual traffic.

The algorithm, simplified

The contribution is the loop, so here it is as runnable-looking Python. The novel parts (example retrieval, model-diff, dual LLM passes, data isolation) are spelled out; the boring parts (the actual solver, the DB) are stubbed.

# Stubs: llm(prompt)->str runs the model. solve(model)->Result runs the trusted optimizer.
# query_db(sql)->rows hits proprietary data. NONE of these send data to the llm.

EXAMPLE_LIBRARY = load_examples()   # list of (question_text, code_string) pairs you curate

def answer_supply_chain_question(question, base_model):
    # 1. In-context learning: pull the few most similar worked examples.
    #    This is what replaces fine-tuning — you SHOW the model your schema + solver API.
    shots = retrieve_similar(question, EXAMPLE_LIBRARY, k=5)
    prompt = format_examples(shots) + "\n\nQ: " + question + "\nCode:"

    # 2. LLM pass #1: English -> code. The LLM emits a DIFF, not a fresh model.
    code = llm(prompt)                       # e.g. "model.add(flow[f,R]==0 for f!=F); resolve()"

    # 3. Execute locally against trusted infra. Data stays home.
    if is_data_question(code):
        result = query_db(code)              # "explain" path: SQL against the repository
    else:
        modified = apply_diff(base_model, code)   # "what-if" path: small change to the model
        result = solve(modified)             # the TRUSTED solver computes the real answer
        result = compare(result, solve(base_model))  # delta vs. the existing plan

    # 4. LLM pass #2: numbers -> English. Now the LLM only sees aggregate results,
    #    framed as "explain this to a non-expert planner."
    return llm(f"Explain to a planner in one paragraph:\n{result}")

Two things make this work in production and are easy to miss: retrieve_similar (your example library is your domain adaptation — curate it well) and the apply_diff discipline (constrain the LLM to edit a known model, never author one). Get those right and a mid-tier model often suffices.

Built on Prior Work

The chapter is a popular-audience expansion of the authors’ own technical work; the real machinery lives in the cited papers.

Results & Evidence

What they tested. A general evaluation harness: for each scenario, build a set of test questions — including deliberately hard ones (bad grammar, atypical phrasings). They compared in-context learning on large models (GPT-4) against fine-tuning smaller open models (Llama-2, Phi-2), across the Azure cloud supply chain plus synthetic manufacturing scenarios (suppliers/factories/retailers).

Headline numbers.

• Production deployment runs GPT-4 at ~90% accuracy.
• ~23% reduction in fulfillment-investigation time.
• Demand-drift analysis: ~1 week → minutes.
• Procurement case (an automotive OEM): mining thousands of contracts surfaced unclaimed volume discounts worth millions of dollars — pure data-discovery value, no solver involved.

What the evidence does establish. The integration pattern works at real enterprise scale and delivers concrete time/money savings; planners genuinely adopt it (the quoted testimonials read like real users). The privacy architecture is viable in a large regulated org.

What it does NOT establish — read these as a practitioner.

• No rigorous baseline table. “~90% accuracy” has no published breakdown by question type, no confidence interval, and no head-to-head numbers for the Llama-2/Phi-2 alternatives. We’re told fine-tuning small models was tried; we’re not told how well.
• 10% is a lot. One in ten answers is wrong, on a system feeding business decisions. The paper is candid that error detection and recovery is an open problem — but in production that 10% needs a human gate.
• Scope is deliberately narrow. Deployment “supports the most common what-if questions” via gradual rollout. The hard case (LLM authors a full model from scratch) is explicitly not solved — they lack tools to validate generated models and say it only handles “simple” structures today.
• Single-vendor, self-reported. It’s Microsoft reporting on Microsoft, in a thought-leadership chapter. The 23% figure is “estimated.” Treat the magnitudes as directional, not benchmarked.

How You’d Use It

This pattern is almost perfectly shaped into a productizable AI-services offering, and the moat is real because of the data-isolation design.

• The offering: “talk to your optimizer.” Any client running an optimization/planning tool (logistics, manufacturing scheduling, workforce planning, inventory, pricing) is a candidate. You wrap their existing solver with the four-station loop. You are not replacing their expensive, trusted model — an easy sell to the engineering team who built it.
• Privacy as the differentiator. Most “chat with your data” pitches choke in security review because data flows to a third-party model. Here you lead with: the LLM only ever sees questions and generic code examples; your data and your model run on your infrastructure. That clears procurement and InfoSec, which is where most enterprise AI deals die.
• In your multi-agent terms. Map the stations to roles you already know: a Translator agent (NL→code), an Executor/tool agent (runs SQL or the solver), and an Explainer agent (numbers→NL), with the example library as shared memory. The “model diff, not model from scratch” rule is exactly the kind of guardrail you’d give a tool-using agent to keep it inside a verifiable action space.
• Start where they did. Sell the explain / data-discovery tier first (Section 3) — it’s low-risk, needs no solver integration, just SQL generation over a clean schema, and the procurement contract-mining example shows it can pay for itself immediately. Upsell the what-if tier once the optimizer has a clean API.

Build Your Own (Minimal Recipe)

The 80%-value version is a weekend-to-two-weeks build if the client’s data and solver are already clean.

1. Pick the explain tier first. Stand up NL→SQL over one well-structured table or view. Components: an LLM API, a prompt with the schema + 5–10 example (question, SQL) pairs, a read-only DB connection, and a result→English pass. This is the whole loop minus the solver.
2. Curate the example library. This is the product’s intelligence. Write 20–50 (question, code) pairs that cover the real questions planners ask. Store them with embeddings so you can retrieve the k most similar at query time. Budget most of your effort here.
3. Add the what-if tier. Requires the optimizer to expose a programmatic interface (e.g., a Python modeling layer like Pyomo, PuLP, OR-Tools, or Gurobi’s API). Constrain the LLM to emit diffs against a fixed base model — add/remove constraints, scale coefficients — never a fresh model. Always solve base + modified and report the delta.
4. Wrap with a verification gate. Re-solve and sanity-check (feasibility, cost monotonicity where expected); for anything risky, show the planner the generated code/diff before executing.

Libraries/models to reach for: GPT-4-class model (or a strong open model + your example library) for translation; an optimization layer (OR-Tools/Pyomo/Gurobi); a vector store (FAISS/Chroma) for example retrieval; an agent framework you already use (e.g., LangGraph) to wire the stations.

The two genuinely hard parts: (1) the example library — coverage and quality determine accuracy far more than model choice; (2) verification and graceful failure on the ~10% the model gets wrong — feasibility checks, code review surfacing, and “I’m not sure, here’s what I’d change” fallbacks.

How to Improve It

The paper hands you its open problems; each is a concrete extension you could attack and demo.

1. Clarify-then-act loop. Ambiguous questions (“can we use factory F better?”) have multiple meanings. Add a step where the Translator agent, when confidence is low, generates a clarifying question and waits — turning a 90%-accuracy one-shot into a higher-accuracy dialog. Testable: measure accuracy with vs. without the clarify step on the hard-question set.
2. Self-verifying what-if. Before returning, have the system re-derive the same answer two ways (e.g., LP duality / shadow prices for marginal-cost questions vs. a full re-solve) and flag disagreement. This directly chips at the 10% error rate with a deterministic check.
3. Swap GPT-4 for a tuned SLM on the constrained task. The action space (model diffs over a known schema) is small and structured — ideal for a fine-tuned small model. The authors flag this themselves (ref [5]). Win: order-of-magnitude cheaper per query, possibly higher accuracy on the narrow task.
4. Constrained decoding / grammar-guided generation. Since valid outputs are code against a known API, force the LLM to emit only syntactically valid model edits (via a grammar or function-calling schema). Eliminates a whole class of the 10% errors — malformed code that won’t run.
5. Validated model-from-scratch. The unsolved frontier. Pair the generator with an automatic validator: generate a model, then auto-check it against held-out historical decisions (does it reproduce known-good plans?) before trusting it. This is the path from “answers what-if on a fixed model” to “builds the model,” and it’s wide open.

Glossary

• Optimization solver / mathematical program — software that, given an objective (minimize cost) and constraints (capacities, demand), computes the provably best plan. The trusted “source of truth” here.
• Constraint — a rule the plan must obey (e.g., “factory F can’t ship more than 100 units”). What-if questions usually map to adding/removing/editing constraints.
• In-context learning (ICL) — adapting an LLM by putting worked examples in the prompt rather than retraining its weights. No training run, instant, but uses up context.
• Fine-tuning — actually updating a model’s weights on domain data. More permanent than ICL, but costs a training run and produces a model you must host.
• RAG (Retrieval-Augmented Generation) — fetch relevant text/examples at query time and append to the prompt. Used here to pull the right (question, code) examples; downside is longer, pricier prompts.
• SLM (Small Language Model) — a compact model that can match big-model accuracy on a narrow task at a fraction of the cost. The proposed path to cheaper deployment.
• S&OP (Sales & Operations Planning) — the recurring process of aligning demand forecasts with supply/production plans.
• Demand drift — month-over-month change in the demand plan; analyzing why it changed was the week-long manual task LLMs collapsed to minutes.
• Fulfillment plan — the solver’s output assigning/shipping actual hardware (clusters of servers) to data centers, minimizing shipping + delay cost under constraints.
• Model diff — a small edit to an existing optimization model (add a constraint, change a coefficient) rather than a whole new model. Verifiable; central to keeping the LLM safe.