A Practical Guide for Designing, Developing, and Deploying Production-Grade Agentic AI Workflows

TL;DR

Agentic prototypes are trivial to build in a notebook and miserable to run in production: the same prompt flickers between success and failure, agents pick the wrong tool, and outputs drift as models update. This paper isn’t a new algorithm — it’s a hard-won operations manual. The authors built a multi-agent system that scrapes live news, drafts podcast scripts with three different LLMs, reconciles them with a “reasoning” agent, renders audio/video, and opens a GitHub PR — then extracted nine best practices that made it stable. The headline lessons: push non-reasoning work out of the LLM and into plain functions, give each agent exactly one job and one tool, run multiple models in parallel and have a referee model consolidate them for Responsible-AI, and keep the whole thing aggressively simple. The “evidence” is a single qualitative case study, not a benchmark — but for someone who sells agentic systems, the patterns are immediately usable and mostly correct.

Problem & Motivation

Here’s the pain in one sentence: a multi-agent workflow that works 9 times out of 10 in a demo is worthless in production, and the 10% failure is non-deterministic, so you can’t even reproduce it to debug.

The authors are precise about where the brittleness comes from. Every time you hand a decision to an LLM — which tool to call, how to format the parameters, whether to call a tool at all — you inject variance. A demo hides this because you run it twice and it happens to work. Production exposes it because you run it ten thousand times under load, with model versions silently changing underneath you. Their concrete failures (Section 3.1–3.4):

• An agent talking to the GitHub MCP server made “ambiguous tool-selection decisions,” guessed invocation parameters inconsistently, and hit “non-deterministic MCP responses.” They tuned the prompt repeatedly; the flickering persisted.
• A single agent given two tools (scrape_markdown + publish_markdown) would call only one, call them out of order, or skip both entirely — especially as input size grew.
• A single agent told to both build a Veo-3 JSON spec and generate the video would emit malformed JSON, mix prose with JSON, and hallucinate file paths for videos it never produced.

The meta-problem: people import enterprise-software instincts (layered abstractions, microservices, MCP-everywhere) into agentic systems, and those instincts actively make things worse because every abstraction is another place for an LLM to get confused. There was no consolidated engineering discipline for this. This paper is that discipline.

What’s New (Core Contribution)

Be honest about what this is: it’s a curated best-practices paper with a worked example, not a novel method. The genuine contributions:

1. Nine concrete, opinionated production patterns — Before: scattered blog-post folklore and vendor docs. Now: a single ranked checklist with a stated failure → fix for each, grounded in one real system. The opinionated parts (prefer pure functions over tool calls; one-agent-one-tool) are sharper than the usual “it depends.”
2. A “demote the LLM” philosophy, made explicit — Before: “agentic” implied more LLM autonomy is better. Now: an argued case that the LLM should be used only where language reasoning is genuinely required, and everything else (timestamps, API posts, git commits) belongs in deterministic code. This inverts the hype.
3. The model-consortium-plus-reasoning-agent pattern as a Responsible-AI mechanism — Before: ensembling is an accuracy trick. Now: framed as bias mitigation, hallucination reduction, and drift robustness — a governance argument, not just a quality one.
4. A full open-source reference implementation — workflow + MCP-server repos, Dockerfiles, K8s manifests, built on the OpenAI Agents SDK. This is the part with the most reuse value: a runnable blueprint, not a diagram.

What’s repackaged: “single responsibility,” “KISS,” “separation of concerns,” and “containerize it” are classic software engineering. The novelty is applying them specifically to where LLM nondeterminism bites and showing the failure modes.

How It Works (Technically)

There’s no equation to demystify here — the “mechanism” is an architecture and a set of decision rules. The deep insight worth internalizing is the determinism gradient, so let me make that the spine.

The determinism gradient (the real idea under all nine rules)

Think of every step in your workflow as sitting somewhere on a spectrum of how much you let the LLM decide:

Six of the nine practices are really one instruction: slide every step as far right (toward “pure function”) as it will go without losing the language reasoning you actually need. That’s the whole game.

The nine practices, decoded

1. Tool calls over MCP. MCP adds an abstraction layer: the agent must read tool metadata from the protocol and reason about it, which raises “cognitive load” (read: token budget spent on plumbing, not the task) and adds nondeterminism. Inside a fixed pipeline where you know exactly what you’re calling, that flexibility is pure cost. Their fix: replace the GitHub MCP integration with a plain PR-creation function.

2. Direct function calls over tool calls. Even a clean tool call forces the LLM to map natural language → function arguments, burning tokens and risking misformatting. For “operations that do not require language reasoning” — posting to an API, a DB write, a timestamp — skip the LLM entirely. They deleted the “PR Agent” and called create_github_pr() straight from the controller. A pure function here means: deterministic, no hidden side effects, cheap, fast, unit-testable. This is the single highest-leverage rule.

3. One agent, one tool. Multiple tools on one agent forces a tool-selection decision before the parameter decision — two coupled sources of error. Their two-tool agent skipped, reordered, or dropped calls. Split into two single-tool agents → deterministic. Each agent now only has to do parameter inference, not tool routing.

4. Single-responsibility agents. The conceptual cousin of #3. An agent told to “build the Veo JSON and generate the video” blurs planning (design the spec) and execution (call the API, save the file). The LLM is great at the first and terrible at the second (it hallucinated success). Fix: a VeoJSONBuilder agent (output contract: valid Veo-3 JSON, nothing else) + a non-agent script_to_video() function that handles the API, retries, and file I/O deterministically.

5. Externalize prompts, load at runtime. Prompts live in a separate GitHub repo, pulled in at runtime — not baked into source. This decouples prompt iteration from code deploys, lets non-engineers (policy, domain experts) edit agent behavior, and unlocks version pinning, rollback, A/B tests, and prompt red-teaming as ordinary git operations.

6. Responsible-AI via model consortium + reasoning agent. This is the most interesting pattern. Run N different LLMs (Claude, GPT-5, Gemini, Llama…) on the same task in parallel — each has different biases and training distributions, so you get diverse drafts. Then a dedicated reasoning agent (they use GPT-oss) acts as a referee/auditor: it does not write new content. It compares drafts, resolves conflicts, removes anything not consistently supported across drafts, deduplicates, and strips speculation. The governance claim: grounding the final output in cross-model agreement mitigates single-model hallucination and bias, and survives model drift better than betting on one model.

7. Separate the workflow from the MCP server. Three clean layers: (a) the workflow backend (the agent pipeline) served as a REST API, (b) a thin MCP server that only forwards MCP tool calls to that API, (c) MCP clients (Claude Desktop, VS Code, LM Studio). The MCP server holds zero business logic, so it stays stable while the backend iterates and each layer scales independently.

8. Containerized deployment. Docker + Kubernetes for the workflow and the MCP server, so you get portability, autoscaling, self-healing, RBAC/secrets, observability (Prometheus/Grafana/OpenTelemetry), and CI/CD with blue-green and canary deploys. Standard cloud-native hygiene, applied per-component.

9. KISS. Counter-intuitively, agentic workflows want flat, function-driven code, not deep enterprise abstraction. Why: the cognitive work lives in the LLMs, so internal complexity adds bug surface without value — and simpler code is easier for AI coding tools (Claude Code, Copilot) to patch and refactor, and easier to migrate to new models.

Architecture & data flow

flowchart TB
  U([User: topic + source URLs]) --> WS[Web Search Agent<br/>RSS + MCP search]
  WS --> TF[Topic Filtering Agent<br/>keep relevant URLs]
  TF --> SC[Web Scrape Agent<br/>HTML to clean Markdown]
  SC --> CON[/Model Consortium<br/>parallel drafts/]
  CON --> A1[Claude script]
  CON --> A2[GPT-5 script]
  CON --> A3[Gemini script]
  A1 --> RA[Reasoning Agent / GPT-oss<br/>compare - reconcile - dedupe - ground]
  A2 --> RA
  A3 --> RA
  RA --> SCR[(Consolidated script)]
  SCR --> AV[Audio/Video Script Agent]
  AV --> VEO[Veo JSON Builder Agent<br/>strict Veo-3 JSON]
  AV --> TTS[[TTS function -> MP3]]
  VEO --> V2V[[script_to_video function -> MP4]]
  SCR --> PRF[[create_github_pr function<br/>pure: branch + commit + PR]]
  TTS --> PRF
  V2V --> PRF
  PRF --> GH([GitHub Pull Request])

  classDef fn fill:#e8f0ff,stroke:#2d6cdf;
  class TTS,V2V,PRF fn;

Note the deliberate split: rounded/double-bracket nodes in blue are pure functions (no LLM); the rest are agents. That visual separation is the paper’s thesis.

Interactive: drag each workflow step along the determinism gradient (MCP → tool call → single-tool agent → pure function) and watch the predicted reliability/cost/token-use bars respond. Schematic — illustrates the paper's argument, not measured numbers.

Interactive: the model-consortium + reasoning-agent pattern. Toggle individual model drafts on/off and see how the referee keeps only claims with cross-model support. Schematic.

The algorithm, simplified

The core idea — practices #6 (consortium + referee) and #2/#4 (pure functions for side effects) — as a controller you’d actually write:

# One workflow step: generate a grounded script, then render + publish deterministically.
# llm(model, prompt) -> str is the ONLY nondeterministic call. Everything else is pure code.

MODELS = ["claude-sonnet", "gpt-5", "gemini-pro"]          # the consortium

def generate_grounded_script(topic, markdown_context):
    # Practice #6: N diverse models draft IN PARALLEL -> diverse, biased-differently drafts
    draft_prompt = load_prompt("podcast_draft")            # #5: prompt fetched at runtime, not baked in
    drafts = [llm(m, draft_prompt.format(topic=topic, ctx=markdown_context))
              for m in MODELS]

    # The referee. It does NOT write new content — it consolidates.
    referee_prompt = load_prompt("reasoning_consolidate")
    final_script = llm("gpt-oss", referee_prompt.format(drafts=drafts))
    # keeps only claims supported across drafts; strips speculation; grounds in markdown_context
    return final_script

def run_workflow(topic, urls):
    urls   = topic_filter_agent(urls, topic)              # single-responsibility agent
    md     = scrape_agent(urls)                            # one agent, one tool (#3)
    script = generate_grounded_script(topic, md)

    # Practice #4: planning (LLM) vs execution (pure code) are SEPARATE
    veo_json = veo_json_builder_agent(script)              # agent: output contract = valid JSON only
    mp4      = script_to_video(veo_json)                   # PURE fn: API call, retries, file I/O
    mp3      = text_to_speech(script)                      # PURE fn

    # Practice #2: no "PR Agent" — the LLM has no business here
    create_github_pr(branch=f"podcast/{slug(topic)}",      # PURE fn: deterministic, testable
                     files=[script, mp3, mp4, veo_json])

Read it once and the philosophy is obvious: llm(...) appears only where language reasoning genuinely happens; every infrastructure action is a boring, testable function call.

Built on Prior Work

The team has a clear lineage of their own: many references are their own prior “fine-tuned LLM consortium + reasoning LLM” medical/diagnostic systems [9,15,23]. The consortium pattern is their house style, now generalized into engineering guidance.

Results & Evidence

Be clear-eyed: this is a qualitative case study, not a benchmark. There is no table of accuracy numbers, no A/B comparison, no statistical test, no baseline system run head-to-head. The “evaluation” (Section 5) is essentially: we built it, here are screenshots of the prompts and outputs, and here’s our narrative of what improved.

What the evidence does establish:

• The pipeline runs end to end and produces coherent multimodal output (scripts, MP3, MP4, valid Veo-3 JSON, a GitHub PR).
• The three consortium models do produce qualitatively different drafts (Llama concise/structured, OpenAI detailed/narrative, Gemini stylistic) — a believable, observed diversity.
• The Veo-3 JSON builder “consistently produced well-formed JSON… without requiring manual correction” across “multiple test runs” — but “multiple” and “consistently” are not quantified.
• MCP integration with LM Studio works (end-to-end interop screenshots).

What it does not establish:

• How much each practice helps. The failure→fix stories (flickering MCP, dropped tool calls, hallucinated file paths) are plausible and ring true to anyone who’s shipped agents — but they’re anecdotes, not measured failure-rate deltas.
• Whether the consortium genuinely reduces hallucination vs. a single strong model + good prompting. No comparison is run. The Responsible-AI claims are architectural arguments, not demonstrated outcomes.
• Cost. Running 3+ models per task plus a referee is expensive; there’s no cost/latency accounting against the determinism it buys.
• Generality. One domain (news→podcast), one team’s stack.

The honest read: the patterns are credible because they match practitioner experience and basic determinism logic, not because the paper proves them. Treat it as a well-organized expert opinion with a reference implementation — which, for a practical guide, is a reasonable bar.

How You’d Use It

For an AI services company, this is a delivery playbook and a sales differentiator, not a research finding. Concretely:

• As a design checklist on every engagement. Before writing code, classify each workflow step on the determinism gradient. Anything that doesn’t need language reasoning → pure function. This alone kills most of the “it worked in the demo” failures clients complain about.
• As a productized “Responsible-AI” tier. The consortium + referee pattern is a clean, explainable story to sell to regulated clients (finance, healthcare, gov): “we don’t trust a single model; we run several and reconcile them, grounded in your source documents, with an audit trail.” That’s a real, defensible upsell — and the paper’s own authors come from exactly these domains.
• As a deployment template. The Docker + K8s + thin-MCP-adapter + externalized-prompts architecture is a reusable skeleton you can clone per client. Externalized prompts in git is genuinely valuable: it lets client domain experts edit agent behavior under review without touching your code.
• As a debugging doctrine for existing brittle systems. When you inherit a flaky agent, the nine rules are a triage list: too many tools per agent? mixed planning/execution? MCP where a function would do? prompts hardcoded? Each one is a refactor with a predictable reliability win.

Where it slots in: this is the engineering discipline layer on top of whatever orchestration framework (OpenAI Agents SDK, LangGraph, your own) you already use. It doesn’t replace tools; it tells you how to wield them.

Build Your Own (Minimal Recipe)

The 80%-value version is small. You can stand up a credible “production-grade agentic workflow” demo for a client in a few days.

Components (build in this order):

1. A flat orchestration controller — plain Python (or your stack), no framework heroics. This is the spine; it calls agents and functions in sequence.
2. Pure functions for every side effect — create_github_pr, write_db, post_api, save_file. Unit-test these. They are the determinism backbone.
3. Single-tool, single-responsibility agents — one per genuine reasoning step. Give each a strict output contract (e.g., “return only valid JSON matching this schema”).
4. Externalized prompts — a /prompts git repo (or even a folder), loaded at runtime via a tiny load_prompt(name) helper.
5. The consortium + referee — call 2–3 models in parallel, then one reasoning model to consolidate. Start with 2 models to control cost.
6. A thin REST API + optional MCP adapter — only if a client needs IDE/desktop access. Skip until needed (that’s KISS).
7. Dockerfile + a basic K8s manifest — last, for deploy.

The 1–2 genuinely hard parts:

• Enforcing output contracts. Getting an agent to reliably return only valid JSON is the real work — use structured-output / JSON-mode features, schema validation, and a retry-on-invalid loop. This is where most of your engineering time goes.
• Writing the referee prompt. The consolidation prompt is the crown jewel and the paper hides it in a figure. It must instruct the model to only keep cross-supported claims, drop speculation, and ground in source — and not invent. Expect to iterate hard here; this prompt is your Responsible-AI moat.

Reach for: OpenAI Agents SDK (what they used) or LangGraph for orchestration; Pydantic for schema-validated output contracts; Firecrawl/Trafilatura for HTML→Markdown scraping; provider SDKs for Claude/GPT/Gemini; Docker + a managed K8s (or even just Docker Compose for v1).

How to Improve It

The paper’s weaknesses are your roadmap:

1. Quantify the patterns (the obvious missing science). Run the with vs. without ablation the paper skips: measure failure rate, token cost, and latency for one-tool vs. multi-tool agents, MCP vs. direct calls, consortium vs. single-model. This turns folklore into evidence — and into a credible client report. Highest-value next step.
2. Make the consortium adaptive, not fixed. Right now it always runs all N models. Add a router: easy/low-stakes tasks → one model; high-stakes or low-agreement tasks → escalate to the full consortium + referee. Massive cost savings; the referee’s disagreement signal is a natural trigger.
3. Turn cross-model agreement into a confidence score / guardrail. When drafts diverge sharply, that’s a hallucination/uncertainty signal. Surface it: low agreement → flag for human review, refuse, or re-retrieve. The paper grounds in agreement but never measures or acts on the degree of agreement.
4. Add self-monitoring / eval-in-the-loop (the paper names this as future work). Pipe traces (OpenTelemetry) into an automated LLM-judge that scores each run for grounding/contract compliance and alerts on drift after a model update — closing the “models change underneath you” loop they identified but didn’t solve.
5. Attack the referee single point of failure. The whole Responsible-AI story rests on one reasoning model you trust to be the auditor. Rotate referees, or use a small panel of referees with majority vote on contested claims, to avoid replacing single-model bias with single-referee bias.

Glossary

• Agentic workflow — A pipeline where multiple LLM-powered agents (each with a role) plus tools/functions cooperate to autonomously complete a multi-step task.
• MCP (Model Context Protocol) — A standardized protocol for exposing tools/services to LLM clients; here argued to add nondeterminism inside fixed workflows and best used only as a thin external adapter.
• Tool call — An LLM deciding to invoke a named function and formatting its arguments from natural language; structured but still nondeterministic.
• Pure function — Code run directly by the workflow (no LLM): deterministic, side-effect-controlled, cheap, fast, unit-testable. The paper’s preferred home for any non-reasoning step.
• Single-responsibility agent — An agent that does exactly one conceptual task, with a single clear output contract (vs. one agent juggling generation + validation + side effects).
• Model consortium — Several different LLMs run in parallel on the same task to get diverse, differently-biased drafts.
• Reasoning / referee agent — A dedicated model that consolidates the consortium’s drafts (resolves conflicts, drops unsupported claims, dedupes) rather than generating new content; the system’s auditor.
• Responsible AI — Here, operational properties: bias mitigation, hallucination reduction, drift robustness, accountability, and verifiability — pursued architecturally via consortium + grounding.
• Output contract — The strict, validated shape an agent must return (e.g., “valid Veo-3 JSON only”); central to making agents deterministic enough to chain.
• LLM drift — Behavior changes over time as the underlying model is updated by the provider, silently breaking workflows tuned to the old version.
• Veo-3 — Google’s text-to-video model; here driven by agent-generated JSON specifications.
• KISS — “Keep It Simple, Stupid”: for agentic systems, prefer flat, function-driven code over enterprise abstraction, since the cognitive load lives in the LLMs.
• Blue-green / canary deployment — Release strategies (run new version alongside old / roll out to a small slice first) enabled by containerization for safe iteration.