TL;DR
Most supply-chain monitoring tools only watch a company’s direct (Tier-1) suppliers, but over a third of real disruptions start deeper in the network — a fire at a Tier-4 chemical plant, a sanctioned Tier-2 mining company — and stay invisible until the damage reaches Tier-1. This paper builds a pipeline of seven LLM agents that reads a news article, classifies the disruption, walks a multi-tier supplier knowledge graph to find which of the company’s Tier-1 suppliers are secretly exposed, computes a deterministic risk score for each, and drafts a mitigation plan (replace the supplier, increase monitoring, or do nothing) that a human then approves. Tested on 30 hand-built scenarios across three car makers (Tesla, Mercedes-Benz, BMW), the four core agents hit F1 scores of 0.96–0.99, the whole pipeline runs in a mean of 3.83 minutes for $0.08, versus an industry benchmark of five days for a manual analyst review. The catch: there’s no real-world disruption dataset (the scenarios were hand-written by two experts), no live news feed (articles are still pasted in manually), and the knowledge graph the entire system depends on has to already exist — which for most companies, it doesn’t.
Problem & Motivation
The pain is concrete: a company can usually see and control its Tier-1 suppliers, but disruptions don’t originate there. A semiconductor fab fire, a raw-material export ban, a strike at a logistics hub feeding five different upstream manufacturers — these events happen two, three, four tiers removed from the company that eventually feels the impact. The paper cites research showing over a third, and in some cases up to half, of supply chain disruptions emerge beyond Tier-1. By the time the effect surfaces at Tier-1, where a company can actually act, the disruption has already been propagating for days or weeks.
Existing tools all stop short of this problem in the same way:
- Enterprise visibility platforms (inventory/shipment dashboards) only track direct suppliers.
- Control towers trigger alerts on structured, internal data against predefined rules — again Tier-1 only, and they can’t read a news article.
- Digital supply chain twins simulate Tier-1 and internal operations; deep tiers are outside the simulated boundary.
- Graph-based risk propagation models (the academic gold standard for multi-tier analysis) are the closest fit structurally — they can model Tier-4 → Tier-1 cascades — but they require someone to hand them a pre-built adjacency matrix and tell them which node is already disrupted. They have no way to read “Ukraine halts neon gas exports” and figure out which node that is.
- Traditional multi-agent systems (MAS) coordinate decentralized software agents, but every agent’s behavior is hard-coded if/then logic against a static ontology. A disruption type the designer didn’t anticipate simply isn’t handled, and none of it can parse free-text news.
So the actual capability gap is an unbroken chain: (1) read unstructured signals (news, filings, advisories), (2) map the entities mentioned onto the company’s real, multi-tier supplier network, (3) trace how the disruption propagates toward Tier-1, (4) quantify exposure for the suppliers the company can actually act on, (5) turn that into a decision. No existing system does all five end to end — network models need step 2 done for them, MAS can’t do step 1 or 2 at all, and prior LLM-based supply-chain systems (inventory optimization, procurement negotiation) start from the assumption that “we already know a disruption happened” and never touch steps 1–3.
Today, this is done by supply chain analysts manually reading news feeds and maintaining spreadsheets — only 40% of companies even have dedicated tooling for it, per the Deloitte survey cited. For a firm with hundreds of Tier-1 suppliers, thousands of Tier-2, and tens of thousands beyond that, this doesn’t scale.
What’s New (Core Contribution)
This is a systems paper, so the novelty is in the integration, not any single algorithm. Three explicit contributions, plus a fourth architectural pattern that’s arguably the most important design decision in the paper:
-
First end-to-end agentic pipeline spanning detection → mapping → risk → decision. Before: network models started from “a node is disrupted” (structured input required); LLM-powered SC systems (SHIELD, InvAgent, procurement agents) started from “we know a disruption happened” and optimized downstream (inventory, sourcing, negotiation). Now: one pipeline goes from raw news text all the way to a reviewed action plan, closing the gap between where disruptions are reported (unstructured text) and where impact must be assessed (a structured multi-tier network). Table 1 in the paper is blunt about this: network models, MAS, and LLM systems each satisfy 2–4 of seven required capabilities; this framework claims all seven.
-
Seven modular, chain-of-thought prompts as the actual “agents.” Each of the seven agents is really a role-specific system prompt (persona + task boundary + strict JSON output schema) plus whatever tools it’s handed — not a separately-trained model. The prompts use few-shot examples and explicit chain-of-thought scaffolding (“generate 3 expert reasoning statements, then 3 action-planning thoughts”) to keep GPT-4o’s output both interpretable and machine-parseable.
-
First synthesized evaluation dataset for this task: 30 scenarios (10 per company) across Tesla, Mercedes-Benz, and BMW, covering 5 disruption types and all four supplier tiers, with expert-generated ground truth for four of the seven agents. There was no existing benchmark for “detect + map + score + decide” as one task, so they built one — useful for anyone else building in this space, though see the caveats below.
-
The real architectural insight (not listed as a headline contribution, but it’s what makes the accuracy numbers believable): split reasoning from computation. Everything that must be reproducible — graph traversal, risk scoring, the risk→action mapping — is a deterministic function the agent calls as a tool, not something the LLM computes by “thinking.” The LLM’s job is narrower than it sounds: extract entities, choose which tool to call, interpret results, write the narrative. This is the standard “LLM orchestrates, code computes” pattern, but it’s applied with unusual discipline here — even the supplier replacement decision is a hard threshold (
risk ≥ 0.6→ replace), not an LLM judgment call.
How It Works (Technically)
The four-stage mental model, and the seven agents that actually implement it. The paper frames the problem as four sequential stages — Event Detection, Relevance Filtering, Risk Assessment, Action Planning — each stage consuming the previous stage’s output. In the actual implementation these four stages are realized by seven specialized agents built on the CrewAI framework, all running GPT-4o, communicating exclusively through strict JSON payloads (no free text between agents — this is what makes deterministic evaluation possible at all).
Every agent shares the same internal shape: tools (functions it can call), memory (short-term for the current run, long-term across runs), planning (chain-of-thought, self-critique, sub-goal decomposition), and actions (the structured output it hands to the next agent). This is standard agent architecture — if you’ve built a ReAct-style agent before, nothing here is new at the single-agent level. What’s interesting is how the seven are wired together.
Architecture & data flow
flowchart LR
NEWS[News article] --> A1["Agent 1<br/>Disruption Monitoring<br/>(classify + extract entities)"]
A1 -->|entities + diagnostic Qs| A2
KG[("Neo4j KG<br/>Tier-1..4 suppliers")] <-->|Cypher BFS| A2["Agent 2<br/>KG Query<br/>(trace disrupted paths)"]
A2 -->|tier-annotated paths| A3["Agent 3<br/>Product Search<br/>(annotate materials)"]
A3 --> A4["Agent 4<br/>Network Visualizer"]
A3 -->|paths + products| A5["Agent 5<br/>Risk Manager<br/>(deterministic scoring)"]
A5 -->|top-10 Tier-1 risk scores| A6["Agent 6<br/>CSCO<br/>(risk→action + narrative)"]
A6 --> H1{"Human review"}
H1 -->|approved| A7["Agent 7<br/>Alternative Sourcing<br/>(find + verify replacement)"]
A7 <-->|re-verify clean path| A2
A7 --> H2{"Human final check"}
Walking the diagram left to right: Agent 1 (Disruption Monitoring) is the only agent that touches raw unstructured text. It reads a news article as a “senior supply chain risk analyst,” classifies the disruption type (geopolitical, economic crisis, natural disaster, cybersecurity), extracts countries/industries/companies mentioned, and — critically — writes a diagnostic question structured so a graph database can answer it (“Which of {company}‘s Tier-1 suppliers are based in Russia or Ukraine?”). This is the handoff point from language to structure.
Agent 2 (Knowledge Graph Query) never touches the LLM for the actual traversal. It resolves the company name to a graph node, then runs a breadth-first search over a Neo4j graph (nodes = companies/countries/industries, edges = suppliesTo and locatedIn) out to Tier-4, filtered by the countries/industries Agent 1 flagged. The LLM’s role here is translating “which Tier-1 suppliers are exposed” into the right sequence of Cypher queries and interpreting results — the traversal itself is deterministic graph code.
Agent 3 (Product Search) walks every supplier edge the KG agent found and web-searches (via SerperDevTool) what material or component actually flows across it — “Norilsk Nickel supplies Nickel, Palladium, Platinum to Johnson Matthey.” This turns a generic “supplier X is at risk” into “the palladium in your catalytic converters is at risk,” which is the difference between an alert nobody acts on and one procurement can act on immediately.
Agent 4 (Network Visualizer) renders the annotated subgraph as an interactive HTML diagram — node size/color by risk, edges labeled by product. Purely a communication aid; no analytical role.
Agent 5 (Risk Manager) is where the paper’s most defensible engineering choice lives — see the math below. It aggregates everything discovered so far into a single risk number per Tier-1 supplier, using a fixed weighted formula, not an LLM guess.
Agent 6 (CSCO — Chief Supply Chain Officer) takes the risk scores and maps them to an action using fixed thresholds: risk ≥ 0.6 → replace the supplier, 0.45–0.59 → increase monitoring, < 0.45 → do nothing. The LLM’s only job is writing the executive narrative (disruption summary, network impact, justification) around a decision that was already made by arithmetic. The plan then goes to a human for approval, revision, or override — the first of two human-in-the-loop gates.
Agent 7 (Alternative Sourcing) only runs for suppliers flagged for replacement. It web-searches for candidate replacements, then calls back into Agent 2 to re-run the Tier-1→Tier-3 BFS on each candidate — confirming the replacement isn’t secretly exposed to the same disruption before recommending it. Second human-in-the-loop gate happens here.
The actual Mercedes-Benz subgraph from the paper's Russia–Ukraine case study, in 3D. Drag to orbit. Mercedes-Benz (Tier-0, center) has three direct Tier-1 suppliers; the disruption is invisible at that layer. It only becomes visible two and four hops out, at Norilsk Nickel (Russia) and Novatek (Russia) — nodes colored red. This is the concrete shape of "over a third of disruptions originate beyond Tier-1."
Demystifying the risk-scoring math
This is the one piece of real math in the paper, and it’s deliberately simple — the authors chose interpretability over sophistication, which is the right call for a system whose output triggers “replace this supplier” decisions.
For every Tier-1 supplier touched by a disruption, the Risk Manager Agent computes:
$$\text{risk} = 0.35 \cdot \text{breadth} + 0.25 \cdot \text{dependency} + 0.20 \cdot \text{criticality} + 0.10 \cdot \text{centrality} + 0.10 \cdot \text{depth}$$
Translated, each term is:
- Exposure breadth (35%, the biggest lever) — how many disrupted downstream (Tier-2/3/4) suppliers feed into this Tier-1 supplier, weighted so closer tiers count more. Operationally: how much of what’s broken actually flows through this supplier.
- Dependency ratio (25%) — how reliant the Tier-1 supplier is on the specific disrupted downstream suppliers versus its total supplier base. A supplier with ten backup sources for a disrupted material scores lower than one that’s single-sourced.
- Downstream criticality (20%) — the max of the disrupted downstream suppliers’ own centrality or PageRank in the network. A disruption at a structurally important hub (many other companies also depend on it) scores this component higher regardless of the Tier-1 supplier’s own position.
- Tier-1 supplier centrality (10%) — the Tier-1 supplier’s own degree centrality in the whole network. More connected suppliers are structurally more important to protect.
- Exposure depth (10%) — how deep the furthest disrupted node is (Tier-4 vs Tier-2), normalized by 4. Deeper disruptions get a small extra weight, reflecting that they’re harder to see coming and often harder to unwind.
The output isn’t just a number — it’s threshold-mapped into HIGH (≥0.6), MEDIUM (0.45–0.59), or LOW (<0.45), and that categorical label, not the raw score, is what Agent 6 uses to decide an action. Because every one of these five inputs is a graph metric (centrality, path counts, PageRank) computed by deterministic code, not the LLM, the same disruption always produces the same score — which is exactly what you want when the output triggers a “replace this supplier” recommendation.
Interactive: drag "exposure breadth" to see how it moves Johnson Matthey PLC's risk score across the HIGH/MEDIUM/LOW thresholds. Component weights are fixed at the paper's 35/25/20/10/10 split; the other four components are held at illustrative values chosen to reproduce the paper's reported score of ~0.52 for this supplier.
The algorithm, simplified
The genuinely reusable idea here isn’t any one agent — it’s the discipline of routing every step through either “LLM reasons over unstructured input” or “deterministic function computes a reproducible number,” and never letting the LLM do both at once.
# The core idea: LLM agents handle unstructured reasoning and tool selection;
# a deterministic function (never the LLM) computes anything that must be
# reproducible — graph traversal, risk scores, and the risk-to-action mapping.
def disruption_pipeline(article_text, focal_company, kg):
# Agent 1: LLM reads the article, returns structured entities plus a
# diagnostic question phrased so a graph query can answer it.
event = llm_extract_disruption(article_text) # -> {type, countries, industries, question}
# Agent 2: deterministic BFS over the knowledge graph, filtered by the
# entities Agent 1 found. No LLM computation here, only retrieval.
paths = kg.bfs_supply_chains(
start=focal_company, max_tier=4,
country_filter=event["countries"], industry_filter=event["industries"],
) # -> tier-annotated paths, e.g. [Mercedes, JohnsonMatthey(T1), Norilsk(T2)]
# Agent 3: LLM web search enriches each edge with the actual material
# flowing through it (palladium, catalysts, microchips, ...).
paths = llm_annotate_products(paths)
# Agent 5: pure math, no LLM. This is what makes the output reproducible.
tier1_risk = {}
for supplier in tier1_suppliers(paths):
breadth = downstream_disrupted_breadth(supplier, paths) # 0..1, tier-weighted
dependency = dependency_ratio(supplier, paths) # 0..1
criticality = max(downstream_centrality(supplier), pagerank(supplier))
centrality = degree_centrality(supplier, kg)
depth = max_disrupted_tier(supplier, paths) / 4.0
tier1_risk[supplier] = (0.35 * breadth + 0.25 * dependency +
0.20 * criticality + 0.10 * centrality +
0.10 * depth)
# Agent 6: the decision is a fixed threshold, not an LLM judgment call.
# The LLM only writes the executive narrative around it.
plan = []
for supplier, score in top_n(tier1_risk, n=10):
action = "replace" if score >= 0.6 else "monitor" if score >= 0.45 else "hold"
plan.append(llm_write_action(supplier, score, action))
approved = human_review(plan) # human-in-the-loop gate #1
if not approved:
return plan
# Agent 7: search for replacements, then re-run Agent 2's BFS on each
# candidate to prove it isn't secretly exposed to the same disruption.
alternatives = [llm_find_alternative(s) for s in replace_list(plan)]
alternatives = [a for a in alternatives
if kg.bfs_supply_chains(a, max_tier=3, country_filter=event["countries"]) == []]
return human_review(alternatives) # human-in-the-loop gate #2
That if not approved: return plan and the two human_review calls are the paper’s safety story: retrieval-grounding (every fact traces back to the KG, not the LLM’s training data), deterministic computation (the risk formula and the threshold mapping), and human sign-off before anything executes.
Built on Prior Work
| Prior idea | What it gave | What this paper changes |
|---|---|---|
| Graph-based risk propagation (Craighead 2007; Kim et al. 2015; Brintrup et al. 2018; Tabachová et al. 2024; Sun & Liao 2025) | Rigorous, quantitative multi-tier cascade math — centrality, PageRank, epidemic-style propagation | Keeps the same style of graph metrics for the risk score, but removes the requirement that a human first tell the model which node is disrupted — that’s now Agent 1 + 2’s job |
| Traditional MAS for supply chain coordination (Ferber & Weiss 1999; Wooldridge 2009; Giannakis & Louis 2011; Bi et al. 2022, 2024) | Decentralized agent coordination patterns, message-passing protocols | Replaces static ontologies and hard-coded if/then rules with LLM reasoning agents that can interpret situations never explicitly encoded |
| LLM-powered point solutions: SHIELD (Cheng et al. 2024), InvAgent (Quan & Liu 2024), procurement agents (Jannelli et al. 2024) | Proof that LLM agents can reason over supply-chain sub-problems (schema induction, inventory, negotiation) | None of these detect disruptions from news or map them to a multi-tier network — they assume the disruption context is already known. This paper chains detection → mapping → risk → decision into one pipeline |
| AlMahri et al. 2024 (same authors’ prior work) | The multi-tier Neo4j knowledge graph schema and construction methodology (companies/countries/industries, suppliesTo/locatedIn) that this paper’s Agent 2 queries | Reuses the graph as required infrastructure; this paper is the agentic reasoning layer built on top of it |
| Chain-of-thought and structured-output prompting (Wei et al. 2022; GPT-4o Structured Outputs) | Reliable, schema-conforming LLM outputs and elicited step-by-step reasoning | Applied as the shared backbone technique across all seven agent prompts — this is how the pipeline stays machine-parseable end to end |
Results & Evidence
30 scenarios (10 each for Tesla, Mercedes-Benz, BMW) over a real 6,596-node / 23,888-edge knowledge graph, 23 true-positive scenarios (a disrupted path genuinely exists) and 7 true-negative (it doesn’t), weighted toward Tier-4 (15 of 30 scenarios) specifically to stress-test deep-tier detection. Ground truth was hand-built by two domain experts independently, disagreements resolved by re-evaluation — a reasonable process, but it means the “ground truth” and the “test scenarios” were both authored by the same two people, which is worth keeping in mind when reading F1 = 0.99.
The headline numbers (Table 5, macro-averaged over 30 scenarios):
| Agent | Precision | Recall | F1 |
|---|---|---|---|
| Disruption Monitoring (Agent 1) | 0.983 ± 0.051 | 1.000 ± 0.000 | 0.991 ± 0.028 |
| KG Query (Agent 2) | 1.000 ± 0.000 | 0.975 ± 0.137 | 0.980 ± 0.110 |
| Risk Manager (Agent 5) | 1.000 ± 0.000 | 0.962 ± 0.196 | 0.962 ± 0.196 |
| CSCO (Agent 6) | 0.950 ± 0.201 | 0.893 ± 0.288 | 0.899 ± 0.277 |
Runtime and cost (Table 8, GPT-4o at $5/$15 per million input/output tokens): mean 3.83 minutes, mean $0.0836 per full end-to-end scenario (min 1.67 min / $0.035, max 6.78 min / $0.125). Against the cited industry benchmark of ~5 days for a human-led response (Kinaxis 2024 survey of 1,800 supply chain leaders), that’s roughly a 3-order-of-magnitude speedup. For 30 scenarios total: 115 minutes and $2.51.
The error-propagation finding is the most useful result in the paper, and it’s honestly reported: performance cascades. When Agent 1’s entity extraction fails, Agent 2 has nothing to search for and returns zero paths, so Agent 5 has nothing to score, so Agent 6 has nothing to decide on. But when Agent 1 succeeds, everything downstream is near-perfect — Agent 2 hits perfect precision (1.000), because Cypher queries are deterministic given correct entities. The system’s reliability is therefore entirely bottlenecked on Agent 1’s text-extraction accuracy, which is the one place the LLM does unconstrained reasoning over messy input. Every other agent’s high score is really a statement about deterministic graph code being correct, not about the LLM being reliable.
The qualitative evaluation of the CSCO agent’s prose is the one place the paper’s own numbers undercut its narrative. Rubric-scored by human evaluators across three sections: disruption summary (0.811), replacement recommendations (0.830, with actionability at 0.910), but network impact analysis scored only 0.486, driven by an accuracy sub-score of just 0.317 — meaning the narrative describing what the knowledge graph actually found was frequently inaccurate, even though the underlying graph query was itself correct. Translating structured multi-tier path data into prose is apparently harder for GPT-4o than either summarizing the news or writing an action recommendation.
A discrepancy worth flagging directly, because it matters if you’re evaluating whether to trust this system’s outputs: in the worked case study (Section 5), the CSCO agent’s own action plan calls Johnson Matthey PLC (risk score 0.52) and Siemens AG (risk score 0.50) “high-risk suppliers” requiring dual-sourcing action. But by the paper’s own stated thresholds (HIGH ≥ 0.6, MEDIUM 0.45–0.59), both suppliers fall squarely in the MEDIUM band, which should trigger “increased monitoring,” not the more aggressive replace-oriented actions described. Either the thresholds shifted between the evaluation section and the case study, or the LLM’s narrative in Agent 6 drifted from the deterministic label it was supposed to be reporting on. Given that the entire safety argument of the paper rests on “the LLM narrates, the threshold decides,” this is exactly the kind of inconsistency a production deployment needs a hard check for — and the paper doesn’t flag it.
What the evidence does not establish: there is no comparison against a simpler baseline (e.g., plain few-shot GPT-4o with no tool grounding, or a rules-based NER pipeline) — so you can’t tell from this paper how much of the 0.99 F1 comes from “agentic orchestration” versus “GPT-4o is good at structured extraction when you force strict JSON schemas.” There’s no scalability test under concurrent load. There’s no evaluation on non-automotive industries. And critically, the scenarios are synthesized, not drawn from real disruption events with real news articles at scale — the one real example (Section 5, Russia-Ukraine) is a single manually-selected walkthrough, not a statistical test.
How You’d Use It
The honest read: the agents are the cheap, easy part; the knowledge graph is the actual product.
- The hybrid “LLM reasons, deterministic code computes” pattern is the reusable architecture, not the supply-chain domain specifics. Anywhere in your own stack you need an LLM-orchestrated decision that has to be auditable and reproducible — insurance claims triage, credit exposure monitoring, compliance flagging, vendor risk scoring — this exact shape (LLM extracts → deterministic function scores → LLM narrates around a fixed threshold → human approves) is the piece worth lifting straight into your harness, independent of the supply-chain content.
- The real build effort is a data-engineering project wearing an agentic-AI hat. The entire system is inert without a pre-built, multi-tier (Tier-1 through Tier-4) supplier knowledge graph — and the paper is explicit that most companies’ ERP systems only capture Tier-1. Before you stand up the seven-agent pipeline for your own supply chain or vendor network, you’re building (or buying access to) Bloomberg/FactSet/Panjiva data, or running a supplier-disclosure and graph-construction project yourself. That’s the actual bottleneck — budget for it before you get excited about the agent demo.
- The two human-in-the-loop gates (CSCO plan approval, alternative-supplier final check) are a template for any automation that touches money or vendor relationships. If you’re automating a decision your own leadership is nervous about handing to AI, this is the shape to copy: the system can’t execute a replacement without a person clicking approve, twice, and every claim traces back to a graph query that’s inspectable, not a black-box LLM judgment call.
- The Agent-1-is-the-bottleneck finding tells you exactly where to spend your own engineering effort if you build something like this. Don’t over-invest in the graph-query or risk-scoring layers (they’re already near-deterministic); invest in making entity extraction from messy news text more robust — ensemble extraction, confidence scoring, or a cheaper first-pass filter before the expensive multi-agent pipeline runs at all.
Build Your Own (Minimal Recipe)
You can get to roughly 80% of the value with far less than seven agents, if you accept a smaller graph and fewer disruption types to start.
Components, in build order:
- A toy multi-tier supply graph. Even a hand-entered
networkxgraph with 50–200 nodes (companies, tiered bysuppliesToedges) is enough to prove the concept before you go anywhere near Neo4j or a paid data provider. - The entity-extraction agent. One LLM call with a strict JSON schema (use OpenAI’s or Anthropic’s structured-output / tool-use features) that reads a news snippet and returns
{disruption_type, countries, industries, companies}. This is the one component the paper shows is your reliability bottleneck — spend your testing budget here. - The deterministic risk function. Literally the
tier1_risk[...]block from the pseudocode above. You don’t need the paper’s exact weights — pick something interpretable, document it, and treat the weights as a configuration you’ll tune to whatever risk appetite your own deployment needs (the paper itself notes the thresholds are meant to be adjustable). - The threshold-mapped decision + narrative agent. One more LLM call, given the score and the action already decided by your threshold, to write the paragraph. Resist the temptation to let the LLM also decide the action — that’s the discipline this paper gets right.
- The human approval step. Even a Slack message with an approve/reject button covers this for a v1.
The 1–2 genuinely hard parts:
- Acquiring or building the multi-tier graph is harder than everything else combined. The paper treats this as a “prerequisite,” which is honest — it is not part of the framework’s contribution, and it’s the part that will actually consume your project budget.
- Entity resolution at scale is nastier than it looks in the paper’s toy example. “Johnson Matthey PLC” vs “Johnson Matthey” vs a ticker symbol vs a subsidiary name are all the same company in reality but different strings in news text and in your graph. The paper’s
resolve_entity_structtool is a black box in the write-up; budget real time for fuzzy matching / an entity-linking step here.
Reach for: LangGraph or CrewAI for orchestration, Neo4j (or networkx for a prototype) for the graph, an LLM with reliable structured outputs (GPT-4o, Claude with tool use), and Serper/Tavily for the web-search enrichment step.
How to Improve It
The paper is unusually honest about its own limitations (Section 4.4 and Section 6 both read like a punch list), which makes the improvement path clear:
- Fix the Agent-1 single point of failure. Since the paper’s own analysis shows every downstream failure traces back to entity extraction, add ensemble extraction (run Agent 1 twice with different prompts, reconcile disagreements) or a cheap rules-based NER pre-filter to catch the obvious misses before the expensive pipeline runs. Directly testable: re-run the 30 scenarios with and without ensembling and see if F1 moves.
- Add the confidence scores the paper recommends but never implements. They explicitly cite prior work on LLM confidence calibration and say “we recommend them for practical deployments” without building them. This is a two-line addition (ask the LLM for a 0–100 confidence alongside its extraction) that would let a human reviewer triage low-confidence outputs instead of reviewing everything uniformly.
- Run the ablation the paper skips. Test the same 30 scenarios with a non-agentic baseline — one big prompt, or a classic NER + graph-query script with no LLM reasoning at all — to find out how much of the 0.96–0.99 F1 is actually attributable to “agentic” orchestration versus “GPT-4o is good at JSON extraction.” This is the single most valuable missing experiment and directly answerable with the existing dataset.
- Add a consistency checker between the deterministic risk label and the LLM’s narrative, closing the exact gap this breakdown flagged in Results (Johnson Matthey scored MEDIUM but was narrated as “high-risk”). A simple regex/keyword check — does the generated text’s risk language match the threshold label it was given? — would catch this class of error before it reaches a human reviewer.
- Move from a static KG snapshot to a versioned/temporal one, which the paper lists as future work. Concretely: timestamp every
suppliesToedge, and re-run risk scoring against the graph state at the time of the news article, not today’s graph — otherwise a supplier relationship that ended last year can still trigger a false alarm.
Glossary
- Agentic AI — LLM-driven agents that plan, call tools, and coordinate, as opposed to hard-coded rule-based software agents.
- Multi-agent system (MAS) — multiple autonomous agents coordinating on a task; the “traditional” (rule-based) version is what this paper contrasts itself against.
- Knowledge graph (KG) — a database of entities (nodes) and relationships (edges); here, companies/countries/industries connected by
suppliesToandlocatedIn. - Neo4j / Cypher — a graph database and its query language, used to store and traverse the multi-tier supplier network.
- Tier-1/2/3/4 supplier — Tier-1 supplies the focal company directly; Tier-2 supplies Tier-1; and so on, deeper into the supply network.
- BFS (breadth-first search) — a graph traversal that explores all neighbors at the current depth before going deeper; used here to walk out from a company to Tier-4.
- Centrality / PageRank — graph metrics measuring how structurally important a node is (how connected, or how much “importance” flows to it); used as inputs to the risk score.
- Retrieval-augmented grounding — verifying LLM-stated facts against an external source of truth (the KG) rather than trusting the model’s training data.
- Deterministic tool orchestration — having the LLM call fixed, reproducible functions for critical computations instead of computing them itself.
- Human-in-the-loop — a required human approval checkpoint before a system’s recommendation is acted on.
- CrewAI — the multi-agent orchestration framework used to build and run the seven agents.
- Precision / Recall / F1 — precision = correct-positives ÷ all-predicted-positives; recall = correct-positives ÷ all-actual-positives; F1 = their harmonic mean. Standard way to score an extraction or classification task against ground truth.
- Jaccard similarity — a set-overlap measure (intersection ÷ union); used here to match predicted supply paths to ground-truth paths even with minor variations.
- CSCO — Chief Supply Chain Officer; the persona/role given to Agent 6, which synthesizes an executive action plan.
- Structured outputs — an LLM API feature (used here via GPT-4o) that guarantees a response matches a specified JSON schema.
- Chain-of-thought (CoT) prompting — instructing a model to reason step by step before producing a final answer, used throughout the seven agent prompts.
- Few-shot prompting — including example input/output pairs in a prompt to demonstrate the desired format and reasoning style.
- TTS / TTR — time-to-survive / time-to-recover; standard supply-chain resilience metrics this framework aims to improve by acting earlier.