Prompt chains: deterministic pipelines that beat premature agents

Most shipped LLM features do not need an agent. They need a fixed sequence of model calls with typed handoffs, a trace that explains what happened, and a budget that stops the thing when it misbehaves. That pattern is the prompt chain, and it sits between a single mega-prompt doing everything at once and an autonomous agent choosing its own next move each turn.

That middle ground is where a lot of production work lands, because it is where latency, cost, and behavior can be reasoned about before the code runs. A chain is not a weaker agent; it is a different commitment — adaptivity traded away on purpose, predictability handed back.

The original note

What “prompt chaining” is

“A prompt chain is a deterministic sequence of LLM calls (and optional tools) where each step’s output becomes the next step’s input.”

It is simpler than a full agent system, and it gives a stable workflow like clarify → rewrite → retrieve → synthesize → verify → format. Reach for it when predictable latency and cost matter, when the task fits a fixed decomposition such as Q&A with citations, and when guardrails are needed without agent complexity.

Common chaining patterns:

• Sequential (A → B → C)
• Branching/Router (route by intent/domain)
• Map-Reduce (parallel sub-prompts then merge)
• Self-Ask/Decompose (create sub-questions, synthesize)
• Reflect/Verify (second pass critiques or citation checks)

Chain vs. agent: who decides the next step

There is exactly one structural difference between a chain and an agent: a chain’s control flow is fixed at author time; an agent’s control flow is decided at run time by the model. In a chain the sequence of steps is a property of the source code, readable in a diff. In an agent the sequence is an output of the model — which tool, how many times, when to stop. Every practical difference follows from that.

Cost is bounded by reading the code. Six steps at two attempts each cannot exceed twelve model calls. An agent’s call count is bounded only by the harness loop cap, and the distribution inside that cap shifts whenever the prompt, the tool descriptions, or the model version change. Budgeting a chain is arithmetic; budgeting an agent is forecasting.

Traces are comparable across runs. A chain emits the same step names in the same order every time, so two runs line up row by row and a regression is legible: one step’s output changed, everything downstream inherited it. Agent traces vary in length and order, so comparison means aligning the sequences first.

Steps are testable in isolation. Each step is a function from state to a value. Substitute a stub completion function returning canned strings and the whole control flow runs with no network and no model, asserting that wiring, validators, retries, and budget behave as written. Testing an agent’s control flow means simulating a policy — larger, and far less stable.

Failure localizes and blast radius stays small. When a chain answers badly the trace names the step, and that step’s validator had a chance to reject the value and retry. When an agent answers badly the cause is usually a decision several turns back, and that decision was a plan: wrong tool, extra loop, budget spent before the real work started.

The trade is real: a chain cannot handle an input whose decomposition was never anticipated, and if task shape varies beyond a small set of enumerated branches, fixed control flow degrades into something generic.

One clarification, since this is where the line gets drawn wrong: model-driven branching does not make a chain an agent. If the model picks a label from a menu written in advance and that branch leads into another fixed sequence, the model is choosing from the flow, not composing it.

A runner small enough to keep

Chains do not need a framework. The useful core is a step record, a retry policy, and a trace. State is a plain dictionary and each step declares the one key it writes, so data dependencies live in the assembly rather than in prompts.

from __future__ import annotations

import time
from dataclasses import dataclass
from typing import Any, Callable, Optional

State = dict[str, Any]
Complete = Callable[[str], str]
Validator = Callable[[Any, State], None]


@dataclass
class StepResult:
    name: str
    inputs: State
    output: Any
    seconds: float
    attempts: int
    error: Optional[str] = None


@dataclass
class Step:
    name: str
    run: Callable[[State], Any]
    writes: str
    validate: Optional[Validator] = None
    max_attempts: int = 2
    backoff: float = 0.5


class ChainError(RuntimeError):
    """Raised when a step runs out of attempts or the budget ends."""

    def __init__(self, message: str, trace: list[StepResult]) -> None:
        super().__init__(message)
        self.trace = trace


@dataclass
class Chain:
    steps: list[Step]
    budget_seconds: float = 30.0

    def run(self, state: State) -> tuple[State, list[StepResult]]:
        state = dict(state)
        trace: list[StepResult] = []
        start = time.monotonic()
        for step in self.steps:
            value, record = self._attempt(step, dict(state))
            trace.append(record)
            if record.error is not None:
                raise ChainError(
                    f"{step.name} failed: {record.error}", trace
                )
            state[step.writes] = value
            if time.monotonic() - start > self.budget_seconds:
                raise ChainError(
                    f"budget spent after {step.name}", trace
                )
        return state, trace

    def _attempt(
        self, step: Step, inputs: State
    ) -> tuple[Any, StepResult]:
        start = time.monotonic()
        for attempt in range(1, step.max_attempts + 1):
            try:
                value = step.run(inputs)
                if step.validate is not None:
                    step.validate(value, inputs)
            except Exception as exc:
                if attempt < step.max_attempts:
                    time.sleep(step.backoff * attempt)
                    continue
                return None, StepResult(
                    name=step.name,
                    inputs=inputs,
                    output=None,
                    seconds=time.monotonic() - start,
                    attempts=attempt,
                    error=f"{type(exc).__name__}: {exc}",
                )
            return value, StepResult(
                name=step.name,
                inputs=inputs,
                output=value,
                seconds=time.monotonic() - start,
                attempts=attempt,
            )
        raise AssertionError("max_attempts must be at least 1")

Three details carry the weight. The validator sees the candidate value and the state that produced it, so checks can be relative to the inputs. Retries are per step, so a flaky call costs one call rather than a rerun. And ChainError carries the trace — a chain that blows up without one has discarded the only artifact that explains why.

The model client never appears in the runner; it arrives as a complete(prompt: str) -> str callable. A test can inject a function returning fixed strings and assert that a validator rejection triggers a second attempt, or that the budget fires after the right step — no model required, nothing flaky.

The five patterns

Sequential (A → B → C)

The base case: each step’s output becomes the next step’s input. It applies when the decomposition is known in advance and does not depend on the data — question answering with citations is canonical, since clarifying, retrieving, drafting, and checking never change.

import re

MAX_DOCS = 6
Search = Callable[[str], list[str]]


def non_empty(value: Any, _: State) -> None:
    if not isinstance(value, str) or not value.strip():
        raise ValueError("expected a non-empty string")


def non_empty_list(value: Any, _: State) -> None:
    if not isinstance(value, list) or not value:
        raise ValueError("retrieval returned no documents")


def citations_in_range(value: Any, inputs: State) -> None:
    non_empty(value, inputs)
    found = {int(n) for n in re.findall(r"\[(\d+)\]", value)}
    limit = len(inputs["docs"])
    bad = sorted(n for n in found if n < 1 or n > limit)
    if not found or bad:
        raise ValueError(f"bad citations: {bad or 'none found'}")


def build_qa_chain(complete: Complete, search: Search) -> Chain:
    def sources(docs: list[str]) -> str:
        return "\n".join(
            f"[{i}] {d}" for i, d in enumerate(docs, start=1)
        )

    def clarify(s: State) -> str:
        return complete(
            "Restate the question in one sentence. List each "
            "ambiguity on its own AMBIGUITY: line.\n\n"
            f"Question: {s['question']}"
        )

    def rewrite(s: State) -> str:
        return complete(
            "Turn this into one standalone search query. Reply "
            "with the query and nothing else.\n\n"
            f"Clarified: {s['clarified']}"
        )

    def retrieve(s: State) -> list[str]:
        return search(s["query"].strip())[:MAX_DOCS]

    def synthesize(s: State) -> str:
        return complete(
            "Answer using only the sources. Cite every claim "
            "as [n].\n\n"
            f"Question: {s['question']}\n\n"
            f"Sources:\n{sources(s['docs'])}"
        )

    def verify(s: State) -> str:
        return complete(
            "Delete or rewrite any sentence the sources do not "
            "support. Keep the [n] citations. Reply with the "
            "corrected answer only.\n\n"
            f"Draft: {s['draft']}\n\n"
            f"Sources:\n{sources(s['docs'])}"
        )

    def render(s: State) -> str:
        cited = [
            f"[{i}] {d[:70]}"
            for i, d in enumerate(s["docs"], start=1)
        ]
        return "\n".join([s["verified"].strip(), "", *cited])

    return Chain(
        steps=[
            Step("clarify", clarify, "clarified", non_empty),
            Step("rewrite", rewrite, "query", non_empty),
            Step(
                "retrieve", retrieve, "docs", non_empty_list,
                max_attempts=3,
            ),
            Step(
                "synthesize", synthesize, "draft",
                citations_in_range,
            ),
            Step(
                "verify", verify, "verified", citations_in_range,
                max_attempts=3,
            ),
            Step("format", render, "formatted", non_empty),
        ],
        budget_seconds=20.0,
    )


# complete and search are injected by the caller.
final, trace = build_qa_chain(complete, search).run(
    {"question": "Which sources describe the rollout window?"}
)
for record in trace:
    print(f"{record.name:<11}{record.attempts} try "
          f"{record.seconds:6.2f}s")
print(final["formatted"])

The format step is deterministic Python with no model call, which is the point: a step is a unit of work, not a unit of inference. The cost of the pattern is latency — the sum of every step plus every retry. Smaller, easier-to-write steps buy clarity with wall-clock time.

Branching / router

A router puts one classification call in front of several fixed chains and dispatches by intent or domain. It applies when requests fall into a handful of known kinds that want different treatment.

Runner = Callable[[State], tuple[State, list[StepResult]]]


def build_router(
    complete: Complete, routes: dict[str, Chain], fallback: str
) -> Runner:
    labels = ", ".join(sorted(routes))

    def dispatch(state: State) -> tuple[State, list[StepResult]]:
        raw = complete(
            "Classify the request into exactly one label. Reply "
            f"with the label only.\nLabels: {labels}\n\n"
            f"Request: {state['question']}"
        ).strip()
        chosen = raw.lower() if raw.lower() in routes else fallback
        return routes[chosen].run(
            {**state, "route": chosen, "route_raw": raw}
        )

    return dispatch

The costs are a classification call on the critical path and misroutes that fail quietly: the wrong-but-plausible chain answers a slightly different question, confidently. Keeping the raw label in state makes the fallback rate measurable. Label sets also rot, since each new destination makes the existing ones less mutually exclusive — a router with a dozen labels is usually two routers.

Map-reduce

Fan out sub-prompts over independent pieces of input, then merge. It applies when the input is wider than one prompt should carry and the pieces can be processed without reference to each other.

from concurrent.futures import ThreadPoolExecutor


def map_reduce(
    complete: Complete,
    question: str,
    chunks: list[str],
    workers: int = 4,
) -> str:
    def extract(chunk: str) -> str:
        return complete(
            "Extract only facts that help answer the question. "
            "Reply NONE if there are none.\n\n"
            f"Question: {question}\n\nText: {chunk}"
        )

    with ThreadPoolExecutor(max_workers=workers) as pool:
        partials = list(pool.map(extract, chunks))

    notes = [
        p.strip() for p in partials
        if p.strip().upper() != "NONE"
    ]
    if not notes:
        raise ValueError("no chunk produced usable notes")
    joined = "\n\n".join(f"- {n}" for n in notes)
    return complete(
        "Merge these notes into one answer. Drop duplicates "
        "and flag contradictions.\n\n"
        f"Question: {question}\n\nNotes:\n{joined}"
    )


merge_step = Step(
    name="map_reduce",
    run=lambda s: map_reduce(complete, s["question"], s["chunks"]),
    writes="merged",
    validate=non_empty,
    max_attempts=2,
)

pool.map returns results in input order, which keeps the reduce prompt byte-identical for identical inputs; collecting futures as they complete would make the merge order-dependent. The costs: spend scales with fan-out, the merge sees only summaries so cross-chunk reasoning is lost before it happens, latency floors at the slowest chunk, and one exception inside pool.map fails the whole fan-out — which the enclosing retry repeats in full.

Self-ask / decompose

The model generates its own sub-questions, each is answered, and the answers are synthesized. It applies to multi-hop questions where the intermediate facts cannot be named in advance.

This is the pattern that leaks agent properties into a chain: the number of sub-questions is a model decision, so cost stops being knowable from the source. The fix is a hard cap in code — take the first k sub-questions — which truncates genuinely deep questions but restores the budget. The subtler cost is that a wrong decomposition is invisible: every sub-answer looks correct and the synthesis is confidently wrong, because the question was cut along the wrong seam.

Reflect / verify

A second pass critiques, repairs, or checks the first. It applies where a wrong answer is expensive and a specific failure can be named — unsupported claims, missing citations, a format contract.

The chain above uses it twice over: a model repair pass at verify, and a deterministic citations_in_range validator rejecting any answer that cites a source which does not exist. Whatever code can check should be checked in code; model critique belongs to judgments code cannot make. The cost is roughly doubling the guarded step, plus the risk that a stochastic critic “corrects” an answer that was already right — so repair loops need the cap max_attempts gives them.

A playbook for building one

1. Define the step contract before the prompts. For each step: its name, the state keys it reads, the single key it writes, the shape of that value. If that table cannot be filled in, the task has no fixed decomposition yet.
2. Make every output typed and validated. Parse each result into a structure and reject it at the boundary. Free text flowing into the middle of the next prompt is where drift hides, since prompts tolerate garbage instead of failing on it.
3. Retry per step, never per chain. A retried step costs one call; a retried chain costs all of them, including the steps that already succeeded. Only retry what is safe to repeat — a step with a side effect needs an idempotency key, not a retry count.
4. Trace every step. Name, attempts, latency, inputs, output, kept even when the chain raises. The trace is the only artifact that explains a bad answer afterward.
5. Set a hard budget and enforce it between steps. Wall clock, call count, and a size cap on the state carried forward. Exceeding it should fail loudly with the trace attached, not degrade into a costlier version of the same answer.
6. Version the prompts with the code, and record the version in the trace. An untracked prompt edit is a control-flow change that leaves no diff.

Failure modes

The arithmetic

Per-step reliability multiplies. A six-step chain whose steps are each right 95% of the time is not 95% reliable end to end — it is 0.95^6: 0.95 × 0.95 = 0.9025, × 0.95 = 0.8574, × 0.95 = 0.8145, × 0.95 = 0.7738, × 0.95 = 0.7351. Roughly 74%, so about one run in four is wrong somewhere. Twelve steps gives 0.7351^2 ≈ 0.54, a coin flip. Backwards is worse: for six steps to reach 95% end to end, each must be right about 99.15% of the time, since 0.95^(1/6) ≈ 0.9915.

That is the argument for per-step validation and retry. If failures are independent, two attempts move a 95% step to 1 − 0.05^2 = 0.9975, and 0.9975^6 ≈ 0.985 across the chain — paid for with one extra call on only the steps that failed. Retrying the whole chain instead reaches 1 − 0.265^2 ≈ 0.93 while paying for every step twice. Independence is an assumption, though: a step failing because its prompt is wrong fails identically every attempt.

Context bloat. Each output lands in state, and the lazy default is to paste all of state into the next prompt. Cost then grows with the square of the step count while relevance falls. The discipline: steps read named keys, and long intermediates are dropped once consumed.

Silent schema drift. A prompt gets edited, the output format shifts, and the downstream step keeps working — models are forgiving readers — until it does not. Nothing fails; quality decays. Validators are the only defense, and a parse failure has to be an error, not a logged warning.

Reaching for an agent when a router would do. The stated reason is almost always “the task varies too much for a fixed flow.” Written out, the variation is usually three to five known kinds of request — a router, not a planner. Enumerating the branches keeps the cost bound and the tests.

When to graduate to an agent

The signal is not difficulty; it is enumerability. When branches can no longer be listed in advance, when the step count depends on something discovered mid-run at unknown depth, or when tool selection cannot be predicted from the input, fixed control flow stops paying for itself.

Even then the chain does not disappear — it moves down a level. Let the agent decide which unit of work runs next, and let each unit be a chain: fixed steps, typed outputs, per-step retries, its own trace and budget. Control flow at the top is dynamic because the problem demands it; everything underneath stays testable, bounded, and legible in a diff. That layering keeps flexibility where it earns its cost.