Graph-RAG needs eight evaluation gates, not one

Retrieval-augmented generation — RAG, the pattern where a model answers from documents fetched at query time rather than from its weights — is usually evaluated in one place: the end, where a judge scores the answer. For a plain vector pipeline that is defensible, because the machinery in between is thin — embed, search, paste the top-k chunks into a prompt — so when an answer is wrong the suspect list is short.

Graph-RAG has no such luxury. Between the raw documents and the answer sit four heavy transformations, each able to destroy information silently while everything downstream keeps producing confident output. An end-to-end score can say the system got worse this week; it cannot say that entity resolution stopped merging two spellings of the same company and split the community containing them. The fix is a gate at every stage boundary, measuring the artefact that stage produces.

The original note

Source Data → Data QA: doc lineage, versioning

Ingestion → Chunk QA: length distribution, overlap histograms

Chunking → IE QA: precision/recall vs gold triples, type consistency

Entity & Relationship Extraction → Graph QA: degree distribution, orphan rate, duplicate entity rate

Build Graph → Community QA: modularity Q, number of communities, stability across seeds

Community Detection → Hierarchy QA: depth balance, cross-edge leakage

Community Hierarchy → Summary Eval: coverage, redundancy, hallucination

Community-Level Summaries → Gen Eval: factuality, faithfulness, BLEU/ROUGE, latency/cost, citation accuracy

At query time: Query → Retrieve → Generate → Answer

What Graph-RAG adds, and what it costs

Vector RAG retrieves passages: the unit is a chunk of text and relevance is proximity in an embedding space, which suits questions whose answer sits in one place. Graph-RAG changes the unit of retrieval to entities and the relationships between them — documents are parsed into triples, (subject, predicate, object) statements such as (payments service, depends on, ledger API), merged into a graph that retrieval walks by neighbours and paths.

Two capabilities follow. Multi-hop questions become tractable, because the link between two entities exists as an edge even when no passage mentions both. And partitioning the graph into communities of densely connected entities, then summarising each, produces global answers — “what are the themes here” — that top-k chunk retrieval structurally cannot, because no chunk contains a corpus-level theme.

That is bought with four failure surfaces a vector index does not have:

Extraction. Triples are model output: wrong, inconsistently typed, or missing entirely.
Graph construction. Merging triples needs entity resolution; over-merging collapses distinct entities, under-merging splits one into several.
Community detection. A stochastic algorithmic choice that decides what the global summaries summarise.
Summarisation. Summaries are generated text that later answers are grounded in — a hallucination here is a bad retrieval corpus, not just a bad answer.

None of these are visible at the answer layer until they are already expensive.

Walking the gates

Stage	Gate	Primary signal
Source data	Data QA	Lineage, version
Ingestion	Chunk QA	Length, overlap
Extraction	IE QA	Precision, recall
Graph build	Graph QA	Orphans, duplicates
Detection	Community QA	Modularity, stability
Hierarchy	Hierarchy QA	Depth, leakage
Summaries	Summary Eval	Coverage, grounding
Answers	Gen Eval	Faithfulness, cost

Data QA and Chunk QA

The first two gates are cheap bookkeeping. Data QA means every document carries a stable identifier, a content hash, and a version, so any triple traces back to the revision of the text it came from; without lineage a wrong edge is just a wrong edge. Chunk QA watches the length distribution and overlap histogram of the chunks fed to the extractor: both drift when an ingestion parser changes, and one that loses paragraph breaks turns a tidy distribution into a long tail. Track percentiles; the mean stays flat while the tail rots.

IE QA

Information extraction is scored against a gold set: a few hundred triples hand-labelled from documents held out of any prompt tuning. Precision is the share of extracted triples that are correct, recall the share of gold triples found, F1 their harmonic mean. Comparison must happen after normalisation, or trivial surface differences (“Depends On” vs depends_on) register as errors.

Type consistency is the other half: if a depends on edge must run service to service, a triple whose object is a person is a violation — countable without gold labels, and often a faster warning than F1.

"""IE QA gate: score extracted triples against a gold set."""

import re
import unicodedata

_SPACE = re.compile(r"\s+")


def canon(text: str) -> str:
    """Canonical surface form used for set comparison."""
    text = unicodedata.normalize("NFKC", text).casefold()
    text = text.replace("_", " ").replace("-", " ")
    return _SPACE.sub(" ", text).strip()


def canon_triple(triple) -> tuple:
    subject, predicate, obj = triple
    return (canon(subject), canon(predicate), canon(obj))


def ie_quality(predicted, gold, sample: int = 20) -> dict:
    """Micro precision, recall and F1 over (s, p, o) triples."""
    pred_set = {canon_triple(t) for t in predicted}
    gold_set = {canon_triple(t) for t in gold}
    hits = pred_set & gold_set

    precision = len(hits) / len(pred_set) if pred_set else 0.0
    recall = len(hits) / len(gold_set) if gold_set else 0.0
    denom = precision + recall
    f1 = (2 * precision * recall / denom) if denom else 0.0

    return {
        "precision": precision,
        "recall": recall,
        "f1": f1,
        "spurious": sorted(pred_set - gold_set)[:sample],
        "missed": sorted(gold_set - pred_set)[:sample],
    }


def type_violations(triples, allowed_types, entity_type) -> list:
    """Triples whose endpoints break the predicate's schema."""
    bad = []
    for subject, predicate, obj in triples:
        rule = allowed_types.get(canon(predicate))
        if rule is None:
            bad.append((subject, predicate, obj, "unknown predicate"))
            continue
        src_ok = entity_type.get(subject) in rule["subject"]
        dst_ok = entity_type.get(obj) in rule["object"]
        if not (src_ok and dst_ok):
            bad.append((subject, predicate, obj, "type mismatch"))
    return bad

Graph QA

Three numbers describe the health of a freshly built graph, each mapping to an upstream defect.

Orphan rate is the fraction of entities with no edges. An entity extracted but never connected is dead weight: it joins no traversal and belongs to no community. A rising orphan rate usually means the extractor is emitting entity mentions without relations, or that resolution failed to link a new spelling to the node holding the edges.

Duplicate entity rate is the mirror image: one real thing present as several nodes — “AWS”, “Amazon Web Services”, “Amazon Web Services, Inc.” — each holding a share of the edges. Every duplicate splits a neighbourhood, and split neighbourhoods degrade traversal and detection alike. A cheap proxy is blocking: group nodes by a normalised name key and count the excess per block.

Degree distribution catches two structural pathologies. A hub swallowing everything — one node adjacent to a large share of all edges — usually means a generic term became an entity, so “system” links to everything and the communities collapse into one. The opposite is a shattered graph: uniformly low degree, many disconnected pieces, no structure to detect. The median, the 95th percentile, and the largest node’s degree share distinguish them.

"""Graph QA gate: structural health of a freshly built graph."""

import re
import statistics
import unicodedata
from collections import Counter

import networkx as nx

_PUNCT = re.compile(r"[^a-z0-9 ]+")
_SPACE = re.compile(r"\s+")


def normalize_name(name: str) -> str:
    """Fold case, drop accents and punctuation, collapse spaces."""
    folded = unicodedata.normalize("NFKD", name)
    folded = folded.encode("ascii", "ignore").decode("ascii")
    folded = _PUNCT.sub(" ", folded.lower())
    return _SPACE.sub(" ", folded).strip()


def graph_qa(graph: nx.Graph) -> dict:
    """Orphan rate, duplicate rate and degree-distribution stats."""
    n = graph.number_of_nodes()
    if n == 0:
        raise ValueError("graph QA needs at least one node")

    degrees = sorted(d for _, d in graph.degree())
    orphans = sum(1 for d in degrees if d == 0)

    # Blocking proxy for entity resolution failure: nodes sharing a
    # normalised name are counted as one entity, the rest as excess.
    blocks = Counter()
    for node, data in graph.nodes(data=True):
        blocks[normalize_name(data.get("name", str(node)))] += 1
    excess = sum(count - 1 for count in blocks.values() if count > 1)

    total_degree = sum(degrees)
    top_degree = degrees[-1]
    p95_index = max(0, int(0.95 * (n - 1)))

    return {
        "nodes": n,
        "edges": graph.number_of_edges(),
        "orphan_rate": orphans / n,
        "duplicate_entity_rate": excess / n,
        "degree_median": statistics.median(degrees),
        "degree_p95": degrees[p95_index],
        "degree_max": top_degree,
        "hub_share": top_degree / total_degree if total_degree else 0.0,
        "components": nx.number_connected_components(graph),
    }

Community QA

Modularity Q measures how cleanly a partition separates the graph: it compares the edge weight falling inside communities against the weight expected if the same degrees were wired at random. A value near zero means the partition is no better than chance, and useful community structure sits well above that. There is no universal pass mark, so the honest gate is relative: compute Q for the real graph and for a degree-preserving shuffle of it, and require clear separation.

Stability across seeds is more revealing and more commonly skipped. Louvain and Leiden, the standard modularity-optimising detection algorithms, are stochastic and order-dependent: two runs with different seeds can score almost the same Q while assigning entities to different communities. Since summaries are built per community, an unstable partition means the answer to “what are the main themes” changes when the data did not.

Comparing two partitions is not label matching: community indices are arbitrary. The right tools are the adjusted Rand index (ARI), which counts pairs of nodes grouped the same way in both partitions and corrects for chance agreement, and normalised mutual information (NMI), which measures shared information between the two labellings. Both run from about zero (unrelated) to one (identical).

"""Community QA gate: modularity plus stability under reseeding."""

import itertools
import statistics

import networkx as nx
from networkx.algorithms.community import louvain_communities, modularity

# The chance correction in ARI (and the entropy normalisation in NMI)
# is easy to get subtly wrong by hand, so use the reference versions.
from sklearn.metrics import adjusted_rand_score
from sklearn.metrics import normalized_mutual_info_score


def partition_labels(communities, nodes) -> list:
    """Label each node with the index of its community."""
    lookup = {}
    for index, members in enumerate(communities):
        for node in members:
            lookup[node] = index
    return [lookup[node] for node in nodes]


def community_stability(graph, seeds=(0, 1, 2, 3, 4), resolution=1.0):
    """Detect communities under several seeds and compare pairs."""
    if len(seeds) < 2:
        raise ValueError("stability needs at least two seeds")

    nodes = sorted(graph.nodes())
    runs = []
    for seed in seeds:
        parts = louvain_communities(
            graph, resolution=resolution, seed=seed
        )
        runs.append(
            {
                "labels": partition_labels(parts, nodes),
                "count": len(parts),
                "q": modularity(graph, parts),
            }
        )

    pairs = list(itertools.combinations(runs, 2))
    aris = [adjusted_rand_score(a["labels"], b["labels"]) for a, b in pairs]
    nmis = [
        normalized_mutual_info_score(a["labels"], b["labels"])
        for a, b in pairs
    ]

    return {
        "mean_ari": statistics.fmean(aris),
        "min_ari": min(aris),
        "mean_nmi": statistics.fmean(nmis),
        "mean_modularity": statistics.fmean(r["q"] for r in runs),
        "community_counts": [r["count"] for r in runs],
    }


def modularity_baseline(graph, communities, seed=0) -> dict:
    """Compare real Q against a degree-preserving null model."""
    real_q = modularity(graph, communities)
    null = nx.configuration_model(
        [d for _, d in graph.degree()], seed=seed
    )
    null = nx.Graph(null)
    null.remove_edges_from(nx.selfloop_edges(null))
    null_parts = louvain_communities(null, seed=seed)
    return {
        "q": real_q,
        "null_q": modularity(null, null_parts),
    }

Hierarchy QA

Detection is usually run at several resolutions: broad themes at the top, specific clusters at the leaves. Two things go wrong.

Depth balance describes whether the tree is even. When one branch runs many levels deep and its sibling is a single flat layer, summaries at a given level are not comparable, and a retriever selecting “level two communities” mixes coarse themes with fine detail.

Cross-edge leakage is the share of edge weight crossing community boundaries. Some is inherent — real graphs are not cleanly separable — but a hierarchy hiding heavy traffic between two communities produces two summaries each missing the other’s context. The heaviest crossing pair names the communities that should have been merged.

"""Hierarchy QA gate: boundary leakage and depth balance."""


def cross_edge_leakage(graph, communities) -> dict:
    """Fraction of edge weight crossing community boundaries."""
    lookup = {}
    for index, members in enumerate(communities):
        for node in members:
            lookup[node] = index

    inside = 0.0
    crossing = 0.0
    per_pair = {}
    for u, v, data in graph.edges(data=True):
        weight = float(data.get("weight", 1.0))
        cu, cv = lookup[u], lookup[v]
        if cu == cv:
            inside += weight
            continue
        crossing += weight
        key = (min(cu, cv), max(cu, cv))
        per_pair[key] = per_pair.get(key, 0.0) + weight

    total = inside + crossing
    worst = max(per_pair.items(), key=lambda kv: kv[1], default=None)
    return {
        "leakage": crossing / total if total else 0.0,
        "worst_pair": worst[0] if worst else None,
        "worst_pair_weight": worst[1] if worst else 0.0,
    }


def depth_balance(leaf_depths) -> dict:
    """Spread between the shallowest and deepest hierarchy leaf."""
    if not leaf_depths:
        return {"min": 0, "max": 0, "spread": 0}
    low, high = min(leaf_depths), max(leaf_depths)
    return {"min": low, "max": high, "spread": high - low}

Summary Eval and Gen Eval

Summary Eval scores each community summary against its own member subgraph. Coverage is the share of member entities and relations the summary mentions; low coverage means it cannot support questions about its own entities. Redundancy is overlap between sibling summaries — when siblings say the same things, the hierarchy adds no resolution. Hallucination is the count of claims with no supporting node or edge, the most damaging of the three because the summary becomes evidence for later answers.

Gen Eval sits at the answer. Factuality asks whether a claim is true against a source of truth; faithfulness asks whether it is supported by the context actually retrieved. A system can be faithful to bad retrieval, and separating the two says whether to fix the retriever or the corpus. BLEU and ROUGE measure n-gram overlap with reference answers — weak proxies, best used as regression tripwires rather than targets. Latency and cost are first-class because traversal plus multi-summary synthesis is expensive. Citation accuracy closes the loop: when an answer cites a community, that node must actually contain the support.

Instrumenting this on an existing pipeline

Make every artefact addressable. Give each build a run identifier and persist the chunks, triples, graph snapshot, partition, and summaries under it. Gates that cannot compare two runs are dashboards, not gates.
Build a small gold set first. A few hundred hand-labelled triples from held-out documents unblocks IE QA, which explains most graph-level anomalies.
Run the cheap structural gates on every build. Orphan rate, duplicate rate, degree summary, and component count cost seconds and need no labels.
Reseed detection and record the agreement. Three to five seeds, pairwise ARI or NMI, stored alongside modularity. Report the minimum pairwise score — the mean hides one divergent run.
Score summaries against member subgraphs, not prose. Extract claims and check each for a supporting node or edge — the only check that catches a hallucination before it becomes retrievable context.
Turn history into thresholds. Baseline each metric on the current healthy build, warn on a meaningful relative move, and block only on metrics with a clear failure semantic — a jump in orphan rate, a collapse in stability.

Failure modes in practice

The characteristic failures are quiet, and each has a gate that catches it early:

A prompt change halves recall. The graph still builds, communities still form, answers still read well — a class of relation is simply missing. IE QA catches it; end-to-end scoring often does not, because the questions that would expose it are the ones nobody has asked yet.
Entity resolution loosens. Duplicate rate drops, which looks like a win, while a hub appears in the degree distribution and communities collapse. Reading duplicate rate with hub share separates the two readings.
A resolution parameter shifts. Modularity barely moves; ARI between old and new partitions falls sharply, and every global summary is now about something slightly different.
A summary invents a relationship. It is retrieved for every thematic question in that community from then on, and the answer layer still looks faithful — the model faithfully reported what the corpus said.

What the gates buy you

Not correctness — no set of metrics delivers that. They buy localisation: when answer quality drops, the gate that moved names the stage that broke, and the question narrows from “why is the system worse” to “why did orphan rate jump in Tuesday’s build”. They also make it safe to change one stage at a time, judging a new extraction prompt on the artefact it produces rather than on answer quality several transformations downstream.

The argument is structural. Every arrow between two stages is a place where information can vanish without anything raising an error, and every such place deserves a number attached to it. One end-to-end score says something is wrong. Eight gates say where.