Four Multi-Agent Orchestration Patterns in Python

Multi-agent orchestration in Python, measured: sequential, supervisor, fan-out and debate over one job, with token spend, wall-clock and accuracy for each.

Cover Image for Four Multi-Agent Orchestration Patterns in Python

An incident has been resolved, and you want an agent to turn the notes into a briefing. The timeline, root cause and mitigation each need different evidence, so dividing the work among specialist agents sounds useful. You might let a supervisor assign the sections, write them in parallel, or ask several agents to compare drafts. Each arrangement changes how much context gets read and how long the briefing takes to finish.

Dividing the briefing improves coverage here, but each call adds context cost. The scripted single agent gets one of six sections right for $0.004. All four decomposing patterns recover four sections. Fan-out costs about $0.012, against debate's $0.033. We build their call graphs and count context and dependencies to explain the difference.

Anthropic's December 2024 "Building Effective Agents" recommends choosing the simplest composition that works and using several agents when a task decomposes cleanly. AutoGen, MetaGPT and the multiagent-debate papers describe different compositions. We instrument the calls to compare their context costs, scheduling dependencies and resulting answers on one task.

The measurement is cheap at small scale. We compare four call patterns: a sequential chain, a supervisor delegating to specialists, a parallel fan-out with a reduce step, and a debate panel with a judge. This post builds all four over one job against one scripted model, counting every token and every dependency between calls. You will see why the fan-out's speed-up is 1.3 times rather than six, watch a rate limit turn it into a slowdown, and find the whole table beaten on accuracy and cost at once by one agent with a better retriever. Everything runs in about a second with no API key.

Let's Build It

Open this in Google Colab and run it top to bottom.

Open In Colab

Here are the five schedules against one clock. Each bar is a model call, drawn where the scheduler runs it. Watch the green synthesis bars, and watch how long the debate lane spends filling before it can start one.

Animation of five orchestrations of the same six-section briefing running against a shared simulated clock. The one-agent lane is a single blue bar finishing at 2.83 seconds. The sequential lane is six blue bars end to end, finishing at 4.85 seconds. The supervisor lane runs six short grey router bars together, then six blue worker bars together, then one long green synthesis bar to 4.16 seconds. The fan-out lane runs six blue worker bars together, then one long green reduce bar to 3.75 seconds. The debate lane runs eighteen amber drafter bars together, then six teal judge bars, then a green synthesis bar to 4.68 seconds. A scoreboard on the right shows each lane's finish time and call count.

The job is one incident briefing with six sections, written from a corpus of eighteen documents. Six of those documents carry the facts a section needs. The other twelve are the ordinary contents of a company wiki, including two earlier incidents whose postmortems use the same vocabulary.

SECTIONS = ["timeline", "root_cause", "mitigation",
            "blast_radius", "comms", "regression_risk"]

Each pattern returns the same object: a list of Call records forming a dependency graph. That is the whole instrument.

@dataclass
class Call:
    call_id: str
    role: str                    # single, worker, router, drafter, judge, reduce
    deps: tuple[str, ...]        # the calls that must finish first
    prompt_tokens: int
    output_tokens: int

Cost is a sum over the graph. Wall-clock is the graph's critical path, which is a scheduler question and not a token question.

for pattern in ("single", "sequential", "supervisor", "parallel", "debate"):
    run = PATTERNS[pattern]("keyword-2")
    print(f"{pattern:12s} {len(run.calls):2d} calls  {run.tokens:5d} tokens  "
          f"${run.cost:.5f}  {run.wall_clock():.2f}s  {run.correct}/6 right")
single        1 calls    555 tokens  $0.00411  2.83s  1/6 right
sequential    6 calls   4190 tokens  $0.01501  4.85s  4/6 right
supervisor   13 calls   3159 tokens  $0.01464  4.16s  4/6 right
parallel      7 calls   2277 tokens  $0.01170  3.75s  4/6 right
debate       25 calls   7061 tokens  $0.03336  4.68s  4/6 right

Four of the five decompose the job, and all four get the same four sections right. Between the cheapest and the dearest of those four sits a factor of three in tokens.

Five call graphs side by side. One agent is a single node. Sequential is a vertical chain of six worker nodes. Supervisor is a row of six router nodes, each feeding a worker node below it, with all six workers converging on one synthesis node. Fan-out is a row of six worker nodes converging on one reduce node. Debate is six groups of three drafter nodes, each group feeding a judge node, with the six judges converging on one synthesis node. Each panel is captioned with its call count, token spend and wall-clock time.

What Just Happened?

The job and the corpus

Every section names the domain it belongs to and the question it answers. The briefing is structured input, which matters later.

Section(
    name="regression_risk",
    domain="risk",
    brief="risk of the same failure recurring and what still has to change",
    source="risk/change-register-0142",
)

source is the ground truth: the one document that carries the facts for that section. The corpus holds it alongside seventeen others, three of which were written to beat it on keyword overlap.

The scripted model and its limits

The stand-in model follows one rule.

def answer_section(section, context):
    """Right when the source document is in the context, wrong otherwise."""
    if section.source in context:
        return section.answer, True
    return PLAUSIBLE_ERROR[section.name], False

That rule buys reproducibility: no API key, no sampling, byte-identical figures on every machine, and a token count that does not drift between runs. It costs realism in exactly one column. A model that can only fail by missing its source makes the accuracy figures a measurement of what each topology does to the context, and not a measurement of what a real model does with the context once it has it.

Read the accuracy column that way throughout. The cost and latency columns carry no such caveat, because call counts, token counts and the dependency graph are properties of the shape, and a live model changes none of them.

Every wrong answer is fluent and on topic, drawn from a decoy the retriever did put in the context. None is flagged as low confidence, which is the behaviour that makes this failure mode expensive in production.

The instrumentation

Token counting is a self-contained approximation of a byte-pair count, and latency comes from three published serving figures.

TTFT = 0.35                        # time to first token
SECONDS_PER_OUTPUT_TOKEN = 0.012   # about 83 tokens per second
SECONDS_PER_PROMPT_TOKEN = 0.00008 # about 12,500 tokens per second
PRICE_IN, PRICE_OUT = 3.00, 15.00  # dollars per million tokens

Prices and speeds are for a mid-tier hosted model. Change them and every absolute number moves together, while the ratios between patterns stay where they are. The LLM cost calculator does the same arithmetic against current published rates, and the KV cache calculator covers the memory side of the same serving picture.

The scheduler is list scheduling over the dependency graph, with a concurrency cap.

def schedule(calls, concurrency=None):
    """Return the timeline and the makespan.

    With no cap the makespan is the graph's critical path. With a cap it is
    what an account with a rate limit gets.
    """

Going Deeper

The critical path, and what sits on it

The fan-out runs six workers at once and finishes in 3.75 seconds against the chain's 4.85. That is a speed-up of 1.30 times, from six-way parallelism. The missing factor is visible the moment the schedule is drawn.

Three Gantt charts of the same job. The sequential chain runs six worker bars end to end and finishes at 4.85 seconds. The fan-out with unlimited concurrency runs six short worker bars together in under one second, then a single long green reduce bar occupying the rest of the time to 3.75 seconds. The fan-out capped at two calls at a time staggers the six workers across two and a half seconds before the reduce bar, finishing at 5.22 seconds.

The six workers write one or two sentences each and take between 0.68 and 0.94 seconds. The synthesis call reads all six sections and writes the whole briefing, 203 output tokens, and takes 2.81 seconds on its own. Parallelising the workers compresses the part of the job that was already short and leaves the long pole untouched.

This is Amdahl's law wearing agent clothing. The serial fraction of a fan-out is the reduce step, and the reduce step is where the writing happens. Any orchestration that ends by having one model write the deliverable inherits that floor, no matter how wide the fan-out above it.

The same arithmetic explains the supervisor's numbers. Its thirteen calls take 9.93 seconds of model time and 4.16 seconds of wall-clock, because the routers run together, the workers run together, and only the synthesis is alone. The debate panel compresses hardest of all, 21.41 seconds of model time into 4.68 seconds, which is the strongest argument in its favour.

The rate limit

Unlimited concurrency is an assumption, and it is the assumption that carries the fan-out's entire case. Cap the same graphs at two calls in flight and the ordering changes.

Three panel figure. The first panel shows stacked prompt and output tokens per briefing for the five patterns, with cost labels: one agent 555 tokens at $0.004, sequential 4190 at $0.015, supervisor 3159 at $0.015, fan-out 2277 at $0.012, debate 7061 at $0.033. The second panel shows wall-clock seconds under unlimited concurrency against two calls at a time, where debate rises from 4.7 to 12.3 seconds and fan-out rises from 3.8 to 5.2 seconds while sequential and one agent are unchanged. The third panel shows sections correct of six: one agent 1, and sequential, supervisor, fan-out and debate all 4.

At two concurrent calls the fan-out takes 5.22 seconds and the chain still takes 4.85, so the pattern chosen for speed is now the slower of the two. The debate panel goes from 4.68 seconds to 12.27. The chain and the single agent do not move at all, because neither ever asked for concurrency.

Before adopting a fan-out for latency, check the rate limit it will run under and count the calls it will issue per job. A fan-out wider than the limit is a chain with extra bookkeeping.

The chain's context bill

The sequential pattern spends 4,190 tokens against the fan-out's 2,277 for identical output. The difference is not the call count, which is lower. Each call in a chain carries everything the chain has already retrieved and everything it has already written.

Two line charts. The left chart plots total tokens against the number of sections in the briefing for four patterns; sequential curves upward to 4190 tokens at six sections while fan-out rises roughly linearly to 2277 and one agent stays flat near 555. The right chart plots tokens per section, where sequential rises from 288 to 698, annotated as a factor of 2.42, supervisor falls slightly from 587 to 527, fan-out falls from 435 to 380 annotated as a factor of 0.87, and one agent falls steeply from 297 to 93.

Cut the job into more sections and the chain's cost per section climbs by a factor of 2.42 between one section and six. The fan-out's falls to 0.87, because each worker sees its own section and nothing else, and the fixed system prompt is amortised over more work. The two lines cross between two sections and three. Past that the chain is paying for memory it mostly does not use.

Two things soften this in production. Prompt caching makes the chain's repeated prefix much cheaper to re-read, which is the single most effective fix and the reason the pattern survives at all. The mechanics are in KV cache memory and LLM serving. A chain whose later steps depend on earlier ones is also buying something with those tokens. The chain here does not, because the six sections are independent, which is the case where a fan-out belongs.

The router's two costs

The supervisor adds a routing call before every worker. Each one is cheap, 148 prompt tokens and a four-token answer, and together they are under a tenth of the bill. The real costs are elsewhere.

The first is structural: a router is a serial hop. No worker can start until its router finishes, which adds 0.41 seconds to the critical path and gets worse the moment routers and workers contend for the same rate limit.

The second is that the router is wrong sometimes, and a specialist cannot recover from it. The specialist cards here are the most frequent terms in each domain's own documents, which is how such descriptions get written in practice.

route(comms)  # -> 'infra'   (the section's documents live in 'customer')

Five of six sections route correctly. The comms section asks what customers were told about this incident, and the word "incident" belongs to the infrastructure specialist's vocabulary because infrastructure documents talk about incidents. The routed specialist owns only infrastructure documents, so the status note it needed was never reachable. No retrieval budget inside the wrong corpus recovers it.

That caps the supervisor at five of six sections on every retriever tried, including the best one, while every other decomposing pattern reaches six. A router does not add a failure mode so much as move the whole job's accuracy ceiling to whatever the router achieves.

The price of a panel

Debate is the most expensive pattern by a distance: 25 calls, 7,061 tokens, three times the fan-out's spend. On this workload it returns the same four sections.

The reason is mechanical. Three drafters given the same documents and the same deterministic model produce the same draft, so the majority is that draft, and the vote is a formality. Running one configuration three times and voting cannot beat running it once.

That much is a tautology in a deterministic testbed, so it does not settle the general question. The general question concerns drafters that differ by sampling instead, and it turns on how correlated their errors are. The chart below models it rather than measures it: each drafter's latent score is a shared component plus a private one, mixed by a correlation ρ\rho, with the marginal accuracy held fixed so only the correlation changes.

Line chart of majority-vote accuracy against the correlation between two drafters errors, for single-drafter accuracies of 60, 70 and 80 per cent, each with panels of three and five. Every curve starts well above the single-drafter line at zero correlation and descends to meet it at correlation one. The 70 per cent panel of three starts at 0.783 and ends at 0.697. A shaded band from 0.8 to 1.0 is labelled same model, same prompt, sampled again.

A drafter that is right 70% of the time, voted three ways, reaches 78.3% when the errors are independent. That is a real gain, and Condorcet's jury theorem promises it. At a correlation of 0.9 the same panel reaches 70.3%, a gain of three tenths of a point, for four times the calls. At a correlation of 1.0 it is worth nothing.

The shaded band is where three samples from one model at one prompt sit. Repeated samples share the training data, the tokeniser, the system prompt and the retrieved context, and their mistakes share all of those causes. A panel earns its cost through disagreement that comes from somewhere, so give the seats different retrievers, different tools or different models. Three temperature draws are three votes from one voter.

Retrieval against topology

All four decomposing patterns get four of six sections right on the naive retriever, and both failures are retrieval failures. The comms and regression_risk sections lose their source documents to decoys.

Run the whole grid, five retrievers against six orchestrations, and the picture separates cleanly.

Scatter plot of sections correct out of six against dollars per briefing for thirty configurations, with marker shape for the orchestration and colour for the retriever. Points span roughly $0.004 to $0.058 horizontally and one to six sections vertically. A green annotation marks one agent with a section index at six of six for $0.006, and a red annotation marks debate on the widest retriever at the same six of six for $0.058.

Across the four topologies that decompose the job, switching topology moves mean accuracy by 0.4 sections. Switching retriever moves it by 1.75, more than four times as much. The vertical position of a point is set almost entirely by its colour.

Seven configurations reach six of six. The cheapest is one agent making one call against a hand-built section index, at $0.00597. The next cheapest is the fan-out on the same index at $0.01140, 1.9 times the price for the same result. The dearest is debate on the widest retriever at $0.05757, 9.6 times the price.

The section index is not an oracle. It names two candidate documents per section, and for four of the six one of the two is the wrong document. It took an afternoon with the wiki. Retrieving eight documents by keyword instead of two also reaches six of six, and costs nothing but prompt tokens. Either fix beats every orchestration in the table, and BM25 against embeddings is where the retrieval side of that gets its own treatment.

The uncomfortable reading is the one worth taking away. The supervisor pattern looks like specialisation, and its accuracy advantage on a naive retriever comes from routing giving each worker a domain-scoped index. That is a retrieval improvement wearing an orchestration costume, and the brief already declared the domain, so the router was recomputing a field its input contained.

Failure exposure

Cost and accuracy are the two columns people ask for. The third asks what a topology does with a wrong answer once it has one.

Two panel bar chart. The left panel shows later calls that read a wrong section, per section: one agent 0.0, sequential 2.5, supervisor 1.0, fan-out 1.0, debate 2.0, with sequential and debate in red. The right panel shows independent answers held by the first call that reads it: one agent, sequential, supervisor and fan-out all show one, labelled nothing to compare it against, while debate shows three, labelled three rival drafts of the same section.

In a chain, a wrong section sits in the prompt of every call after it: 2.5 later calls on average, and five for the first section. In a fan-out or under a supervisor, one call sees it, the reduce step, and that call has no independent source to weigh it against.

The right panel is where debate earns something the cost table does not show. Its wrong draft reaches a judge that is holding two rivals for the same section, which is the only place in any of these five shapes where a wrong answer meets a competing one. Whether the judge picks correctly is the correlation question again. The structure at least makes the comparison possible.

Six rules, in order

The measurements collapse into an order of operations.

  1. Fix retrieval before choosing a topology. On this workload the retriever moved accuracy more than four times as far as the orchestration did, and the cheapest fix was one call with a better index. Orchestration that compensates for bad retrieval is expensive and it does not converge.
  2. Find the reduce step and time it. It is the serial floor under every fan-out. Six-way parallelism bought 1.3 times here because one synthesis call held 2.81 of the 3.75 seconds. Shorten the deliverable or split the write, or accept the floor.
  3. Count the calls against the rate limit. At two concurrent calls the fan-out became slower than the chain it replaced. A fan-out wider than the limit is a chain with extra bookkeeping.
  4. Use a chain only where later steps read earlier ones. Independent subtasks in a chain pay a context bill that rises with every step added, 2.42 times per section from one section to six. Prompt caching reduces the bill without changing the shape of the curve.
  5. Do not add a router for a field you already have. The router costs a serial hop before every worker and caps the job's accuracy at its own. Where the input already names the specialist, read it.
  6. Buy a panel only where the seats differ for a reason. Three samples of one model at one prompt sit near a correlation of one, where the majority is worth four tenths of a point at four times the cost. Different retrievers, tools or models make the vote mean something.

Where This Comes From

Anthropic's "Building Effective Agents" (December 2024) is the practitioner document this post is arguing with and mostly agreeing with. It separates workflows, where the steps are fixed in code, from agents, where a model chooses its own path, and its recommendation is to start with the simplest composition that works:

"When building applications with LLMs, we recommend finding the simplest solution possible, and only increasing complexity when needed."

Its named patterns map onto the four measured here: prompt chaining is the sequential chain, routing is the supervisor, parallelisation covers both the fan-out and the voting panel, and orchestrator-workers is the supervisor with a dynamic set of subtasks. The measurement above adds the arithmetic that the essay asserts.

Wu et al.'s "AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation" (2023, arXiv:2308.08155) is the framework paper that made conversable agents a default. Its contribution is a programming model, agents that send each other messages with programmable reply behaviour, and the paper says plainly that the design space it opens is larger than the evidence for any point in it. Every topology in this post can be written in it, which is the point and also the problem, since the framework does not tell you which one to pick.

Du et al.'s "Improving Factuality and Reasoning in Language Models through Multiagent Debate" (2023, arXiv:2305.14325) is the strongest published case for the fourth pattern. Multiple instances propose and critique answers over several rounds, and the paper reports gains on arithmetic and factuality benchmarks. Read it next to the correlation chart above. The gains are real and they arrive through disagreement, so the operative question for a system is how much independent disagreement its panel can generate, and repeated samples of one model at one prompt generate little.

Hong et al.'s "MetaGPT: Meta Programming for a Multi-Agent Collaborative Framework" (2023, arXiv:2308.00352) takes the specialisation idea furthest, encoding standard operating procedures and role definitions so that agents pass structured artefacts instead of free conversation. Structured handoffs are the part worth borrowing, and the section index in this post rests on the same idea.

The oldest result here belongs to none of them. Amdahl's law (1967) sets the speed-up available from parallelising a fraction of a job, and the synthesis call is the fraction that will not parallelise. Condorcet's jury theorem (1785) sets the accuracy available from a majority vote, and its independence assumption is the one that repeated sampling breaks.

Further reading

  • The practitioner baseline: Anthropic, "Building Effective Agents", for the pattern catalogue and the argument for restraint
  • The framework: Wu et al. (2023), sections 2 and 3 for the conversable-agent programming model
  • The case for panels: Du et al. (2023), for what debate buys and on which benchmarks
  • Structured handoffs: Hong et al. (2023), for encoding roles and artefacts rather than conversation
  • Next topic to learn: evaluation for agent systems, since every number above is only as good as the ground truth behind it

Interactive Tools

  • LLM Cost Calculator: Price your own orchestration against current published rates, since the factor of three between the cheapest and dearest pattern here is a token count you can reproduce
  • KV Cache Calculator: Size the memory behind the context a sequential chain keeps re-reading, which is the other half of the chain's bill
  • AI Readiness Assessment: Covers the governance questions a multi-agent design raises, including who approves what an agent does and what it is allowed to reach

Stay in the loop

Get notified when new tutorials and deep-dives are published. No spam, unsubscribe anytime.

Related Posts

Frequently Asked Questions