Query Planning for Multi-Source Enterprise RAG

A team ships its first enterprise RAG assistant and it looks great in demos. Ask a policy question and it retrieves a few chunks from a vector index. Ask a product support question and it cites the help center. Ask for an employee handbook detail and it quotes the right section. Then it meets real users.

A finance manager asks: “Which of our EMEA vendors had contract renewals delayed due to security review in Q2, and what were the approved exceptions?” Now the answer requires vendor records from a structured system, security-review workflow data from another system, contracts in a document repository, probably regional metadata from a master-data table, and policy exceptions buried in PDF approvals or ticket comments. A single retrieval call against a vector index won’t get you there. Even worse, a broad “search everything” strategy often returns a noisy pile of half-relevant evidence, drives latency through the roof, and still misses the key join condition.

This is where enterprise RAG stops being “retrieve top-k chunks and synthesize” and becomes a query-planning problem.

If your assistant must answer questions that span document corpora, search indexes, warehouses, ticketing systems, knowledge graphs, and APIs, then retrieval is no longer one operation. It is a plan: possibly decomposed, often branched, sometimes sequential, and almost always constrained by latency, access control, and provenance requirements.

The engineering challenge is not just “how do I connect more tools?” It is “how do I decide which retrieval steps to execute, in what order, under what budget, and how do I combine the results into something answerable and auditable?”

In production, good query planning is the difference between a system that feels smart and one that feels erratic. This article covers the practical patterns that emerge when you build enterprise RAG over multiple backends: when to fan out, when to sequence, how to join evidence, where naive planner designs fail, and how to evaluate and observe multi-source plans before they become an expensive mystery.

The pattern: not all questions are one retrieval problem

The first mistake teams make is treating every user query as if it has the same retrieval shape. In practice, enterprise questions tend to fall into a few buckets.

Single-source semantic lookup. These are classic RAG questions. “What is our travel reimbursement policy for meals?” One corpus, maybe one search index, maybe metadata filters.

Parallel multi-source evidence gathering. The user needs corroboration or coverage across systems. “What changed in product pricing and where was it announced?” You may need the pricing database, release notes, and customer communications. These can often be fetched independently and merged later.

Sequential dependency retrieval. One step determines the next. “For the customers affected by last week’s billing incident, what credits were issued?” You first identify affected customers from incident records, then use those IDs to query billing/credits. Doing both at once either fails or explodes search space.

Entity-centric joins across structured and unstructured evidence. “Which vendors in Germany have open risk waivers and contracts expiring this quarter?” This requires joining canonical entities from structured systems with supporting evidence from documents.

Constraint-heavy or policy-gated retrieval. Questions that imply region, time window, department, confidentiality tier, or document status. Planning matters because the wrong initial retrieval can swamp the system with irrelevant or unauthorized candidates.

Analytical questions disguised as search. “How many enterprise customers escalated SLA breaches after the September release, and what were the common root causes?” This is not a chunk-retrieval problem. It may require SQL aggregation, issue clustering, and only then supporting text retrieval.

Once you see these patterns, the architecture changes. Your system should not always route to a single “retrieve()” function. It should select a retrieval strategy.

Why the naive approach fails

There are a few common naive strategies, and they all break in predictable ways.

Naive strategy 1: Search everything at once

The simplest pattern is to send the user query to every backend in parallel: vector search, BM25, SQL tool, graph lookup, ticket search, document search, maybe a web connector. Then dump the top results into the model.

This fails for three reasons.

First, recall is often fake. Searching every system does not mean you found the answer. If the answer depends on an intermediate entity resolution step, broad search may never surface the relevant identifiers.

Second, ranking is incoherent across systems. A cosine score from a vector DB, a BM25 score from OpenSearch, and a confidence score from a graph expansion are not directly comparable. Teams often normalize them poorly or just concatenate results. Then a highly relevant SQL row may get drowned by semantically similar but useless document chunks.

Third, latency and cost balloon. Querying six systems on every request is fine in a proof of concept. In production, it blows your p95 and your tool spend. Worse, most user questions do not need six systems.

Naive strategy 2: Let the LLM plan everything dynamically

After “search everything” disappoints, teams swing to the other extreme: give the LLM a tool catalog and let it reason its way through each request.

This can work surprisingly well in low-volume environments, but it fails under production pressure.

Plans become nondeterministic. Similar queries may take different paths, making debugging miserable.

The model overuses expensive tools. If the planner sees an option, it tends to try it unless heavily constrained.

Tool schemas become the bottleneck. If system capabilities, filters, or join keys are poorly described, the model makes subtly bad choices.

It hides policy and budget logic inside prompts. That becomes unmaintainable. Latency ceilings, source preferences, and ACL requirements should not exist only as natural-language hints.

Naive strategy 3: Force a fixed linear pipeline for every question

Another reaction is to define a rigid flow: classify query, run semantic search, run SQL if entities found, rerank, answer. This is better operationally, but it underperforms because it assumes the same dependency structure for all requests.

Some questions need fan-out. Some need sequential narrowing. Some need no documents at all after a SQL answer. Fixed pipelines waste work on easy queries and still miss hard ones.

The deeper issue is this: enterprise RAG is not just retrieval. It is constrained information access and evidence assembly. You need query planning the way a database optimizer needs query planning, simpler than a relational optimizer, but guided by the same instincts: push down filters, avoid unnecessary scans, sequence dependent operations, and join on stable keys.

A better approach: policy-bounded query planning

The design that holds up in practice is not “purely dynamic agent” and not “one static pipeline.” It is a policy-bounded planner.

That means:

You define a small set of retrieval plan shapes that cover most enterprise questions.
A planner chooses among those shapes using query features, source metadata, and budget constraints.
Each shape has explicit execution rules for fan-out, sequencing, joins, provenance, and fallback.
The planner can fill parameters, not invent arbitrary workflows.

Think of it as constrained orchestration rather than open-ended agency.

A useful top-level architecture looks like this:

Intent and constraint extraction
- Detect question type: lookup, comparison, aggregation, explanation, compliance, troubleshooting.
- Extract entities, time windows, region, department, product, confidentiality hints.
- Detect whether the question implies counting, joining, or sequence dependencies.
Source capability registry
- For each backend, track supported entity types, filter dimensions, freshness, cost, latency, access rules, and output schema.
- Example: “Contract DB supports vendor_id, region, expiry_date filters; median 200 ms; high precision for contract metadata; no semantic text.”
Plan selector
- Choose from a finite menu such as:
  - Single-source retrieve
  - Parallel fan-out retrieve + merge
  - Structured-first sequence
  - Unstructured-first sequence
  - Entity-resolution then fan-out
  - Aggregate-then-evidence
Execution engine
- Runs steps with budgets, parallelization rules, retries, and cancellation.
- Enforces ACL checks and provenance capture at each step.
Evidence joiner/reranker
- Joins records and chunks using canonical IDs or inferred linkage.
- Produces a ranked evidence set with source attributions and confidence metadata.
Answer synthesis layer
- Generates an answer only from admissible evidence.
- Can abstain if joins are weak or evidence conflicts.

This architecture matters because it separates three concerns that many teams mix together:

deciding what plan shape to use,
deciding how to execute that plan under constraints,
deciding how to synthesize a response from joined evidence.

When to fan out across systems

Fan-out is appropriate when sub-retrieval tasks are mostly independent and the final answer benefits from breadth or corroboration more than dependency.

Typical cases:

“What changed?” questions spanning release notes, roadmap docs, and support notices.
“What is the latest status?” questions across incident tracker, status page, and internal runbooks.
Questions where the same entity exists in multiple systems and you want multiple evidence types in parallel.

A good fan-out plan has these properties:

Shared normalized query frame. Before issuing queries, normalize the user request into a structured frame:

intent = change_summary
entity = pricing
audience = enterprise customers
time_window = last 30 days
region = global

Each backend adapter then maps that frame into its own query language.

Filter pushdown. Do not retrieve broadly and filter later if the backend can filter. Push time, region, product line, document type, tenant, and ACL filters down into the source query.

Per-source budgets. Set top-k, timeout, and max-bytes per backend. The help center might return top 5 chunks in 300 ms; the ticket search might get 2 seconds but only for severe incidents.

Source-specific ranking before global merge. Rank within each source first. Then combine via a learned or heuristic fusion layer that incorporates source trust, freshness, and task relevance.

A practical global merge score might combine:

source prior weight
normalized source-specific relevance
metadata match strength
freshness bonus
evidence diversity bonus
provenance quality bonus

Do not treat all chunks as interchangeable. A canonical SQL record and a speculative ticket comment should not compete on the same score alone.

When fan-out goes wrong

Fan-out hurts when you have hidden dependencies. For example:

“Which open customer escalations are tied to accounts whose renewal is at risk due to unresolved security findings?”

If you fan out to escalations, renewals, and security findings independently using only the raw text query, you will retrieve lots of noisy candidates and miss exact account relationships. This should be sequenced around account IDs or account entities.

The rule of thumb: fan out when systems can answer the same query frame independently; sequence when one system is needed to determine the candidate set for another.

When to sequence retrieval steps instead of doing everything at once

Sequencing is the most underused pattern in enterprise RAG. It solves the common problem where one retrieval step must narrow or parameterize the next.

There are four sequence patterns that show up repeatedly.

1. Structured-first sequence

Use a structured system first when it can cheaply identify the exact entities, time windows, or states relevant to the query.

Example: “Which APAC customers downgraded after the August outage, and what reasons did CSMs document?”

Plan:

Query account events/CRM for APAC customers with downgrade events after outage date.
Get list of affected account IDs.
Query CSM notes and tickets only for those account IDs.
Summarize common reasons with citations.

Why it works:

Structured data sharply reduces the search space.
Unstructured retrieval becomes precise because you search with account IDs plus semantic terms.
Latency is often lower than global fan-out because the expensive search happens over a small candidate set.

2. Unstructured-first sequence

Sometimes the structured system cannot be queried until you know the right entities or codes, which are easiest to discover from text.

Example: “Do we have any approved exceptions to the standard encryption requirement for legacy payment connectors?”

Plan:

Search policy corpus and exception approval docs for “legacy payment connectors” and encryption exceptions.
Extract exception IDs, connector names, system names, or policy references.
Query compliance database for current approval status and scope.
Return joined answer with authoritative status plus documentary evidence.

Why it works:

The user’s language may map poorly to database fields.
Text retrieval helps discover the right canonical handles.

3. Entity-resolution then fan-out

If the user refers to ambiguous names, aliases, or natural-language descriptions, resolve entities first.

Example: “What issues are blocking the Northstar launch customer from going live?”

“Northstar” might be a codename, internal initiative, or customer nickname.

Plan:

Resolve “Northstar launch customer” to account_id or project_id via CRM/project metadata.
Fan out to blockers across tickets, onboarding checklist, risk tracker, and email summaries.
Merge by account/project ID.

4. Aggregate-then-evidence

Questions involving counts, cohorts, trends, or outliers often need structured aggregation before document retrieval.

Example: “How many SOC 2 questionnaire requests increased after the new security whitepaper launch, and what questions changed most?”

Plan:

Query request volume by week from ticket system or warehouse.
Detect increase window and identify common changed questionnaire items.
Retrieve representative supporting conversations/documents for those items.
Answer with metrics plus evidence.

This pattern prevents the common failure where the LLM tries to estimate counts from retrieved text.

Joining results without losing provenance

Joining is where many RAG systems quietly become untrustworthy. Teams retrieve from multiple systems, summarize across them, and only later realize they can no longer say which sentence came from which source or whether two pieces of evidence actually refer to the same entity.

You need a deliberate evidence model.

Use a canonical evidence envelope

Every retrieved item, SQL row, graph node, API response, document chunk, ticket comment, should be converted into a standard envelope such as:

evidence_id
source_system
source_type
source_locator (URL, record key, doc ID, query reference)
retrieval_query
retrieved_at
entity_refs [canonical IDs]
temporal_scope
permissions_context
content_payload
extraction_payload (structured facts extracted from content)
provenance_confidence
relevance_score

This sounds bureaucratic until you need to debug a bad answer six weeks later. Then it becomes essential.

Prefer joins on canonical IDs, not text similarity

The best joins are exact joins using stable keys:

customer_id
vendor_id
contract_id
incident_id
policy_id
employee_id

If a document chunk mentions a vendor name, run entity extraction and resolution to attach vendor_id before joining.

Text-similarity joins are last resort. If you must use them, label them as inferred joins with lower confidence and different synthesis rules.

Preserve one-to-many relationships

A common anti-pattern is flattening evidence too early.

Suppose one vendor has three contracts, one security exception, and seven ticket comments. If you flatten all of that into a blob per vendor, you lose the ability to answer precisely. Instead, keep relational structure long enough to support:

per-entity summaries,
exception scoping,
conflict detection,
targeted citation.

Separate authoritative facts from supporting narrative

Not all evidence serves the same purpose.

Structured systems often provide authoritative state: status, date, owner, amount.
Documents and comments often provide context and explanation: why, risk, rationale, discussion.

Your join layer should mark these roles explicitly. During synthesis, instruct the answer model to prefer structured records for factual fields and use unstructured text for explanation and nuance.

Track provenance at the claim level if possible

The gold standard is claim-level attribution: each generated claim is linked to one or more evidence envelopes. Not every team can build this on day one, but even lightweight claim-to-evidence mapping dramatically improves reviewability.

Planner vs fixed-policy tradeoffs

There is no universally correct amount of planner intelligence.

Fixed-policy routing

In fixed-policy systems, engineers define deterministic routing logic:

if query contains count/trend words -> structured aggregate path
if entity type = contract/vendor and requires status + rationale -> structured-first sequence
if informational policy query -> document search path

Advantages:

predictable,
easy to test,
easy to bound cost and latency,
simpler observability.

Disadvantages:

brittle coverage,
hard to handle novel query forms,
grows into a rule jungle over time.

LLM planner

An LLM planner decides:

which sources to query,
whether to decompose the question,
what sequence to use,
what join keys matter.

Advantages:

handles linguistic variation,
adapts to nuanced requests,
often better at decomposition than rules.

Disadvantages:

nondeterministic,
harder to evaluate,
can over-call tools,
difficult to guarantee policy adherence.

The middle ground that usually works

Use an LLM for interpretation and parameter filling, not unconstrained control.

Concretely:

The model extracts intent, entities, constraints, and possible subquestions.
The model may score candidate plan shapes.
A deterministic policy layer approves one of the known plan templates based on source capabilities, cost ceilings, ACLs, and latency budget.
The execution engine fills in tool calls from validated parameters.

This gives you the language robustness of an LLM without letting it invent unsafe or pathological plans.

A simple plan schema might look like:

json
{
  "plan_type": "structured_first_sequence",
  "goal": "identify vendors with delayed renewals due to security review and approved exceptions",
  "constraints": {
    "region": "EMEA",
    "time_window": "2024-Q2",
    "latency_budget_ms": 5000,
    "max_sources": 4
  },
  "steps": [
    {"kind": "query_sql", "source": "contracts_dw", "outputs": ["vendor_id", "contract_id", "renewal_status"]},
    {"kind": "query_api", "source": "security_reviews", "join_on": "vendor_id"},
    {"kind": "retrieve_docs", "source": "exceptions_index", "join_on": ["vendor_id", "contract_id"]}
  ]
}

The point is not the exact schema. The point is that planning outputs should be inspectable, constrainable, and executable without more free-form reasoning.

Implementation details that matter more than teams expect

Build a source capability registry

Do not rely on prompt descriptions of tools alone. Maintain machine-readable metadata for each backend:

supported query modes: keyword, semantic, exact lookup, aggregation
filterable fields
joinable identifiers
freshness SLA
p50/p95 latency
cost per call or tokenized estimate
failure modes
ACL model
evidence authority level

This registry powers better planning and better evals. It also helps avoid absurd plans, like sending aggregation questions to a vector index.

Introduce retrieval IR, not just natural-language tool calls

Use an intermediate representation between the user query and backend calls. Example:

json
{
  "entities": [{"type": "vendor_region", "value": "EMEA"}],
  "time_range": {"start": "2024-04-01", "end": "2024-06-30"},
  "intent": "find_delayed_contract_renewals_with_security_exception_context",
  "required_fields": ["vendor_id", "contract_id", "delay_reason", "exception_status"],
  "evidence_needs": ["authoritative_status", "approval_documentation"]
}

Adapters map this IR to SQL, search DSL, graph traversals, or API requests.

This reduces prompt fragility and makes plan replay possible.

Budget by phase, not just by request

A request-level timeout is not enough. Allocate budgets across planning and execution phases.

Example 5-second budget:

200 ms for intent extraction and routing
800 ms for structured candidate narrowing
2500 ms for parallel evidence retrieval
700 ms for join/rerank
800 ms for answer synthesis

If a sequence step returns too many candidates, prune early. If a slow source misses its budget, proceed with a degraded answer or abstain.

Support partial answers explicitly

In enterprise settings, waiting for all sources can be worse than returning a qualified partial answer.

Good systems can say:

“I found 8 EMEA vendors with delayed renewals tied to security review from the contracts and security systems. Exception approval documents are still loading; current answer excludes manual approvals after June 28.”

This requires the answer layer to know what evidence classes are complete versus missing.

Use source-specific retrievers

A single dense retriever over all enterprise content is tempting but rarely enough.

Use the right retrieval mode per source:

BM25/hybrid for exact policy names, SKUs, error codes
dense retrieval for semantically phrased FAQs or design docs
SQL for state/aggregation/filter-heavy requests
graph traversal for dependency or relationship queries
API lookup for live operational status

The planner should choose retrieval mode as well as source.

Rerank after join, not only before

Teams often rerank raw chunks before joining, but in multi-source retrieval the more important rerank is after assembling entity-centered evidence bundles.

For example, rank vendor bundles by:

confidence that delay reason is security review,
existence of approved exception evidence,
freshness,
completeness of fields,
contradiction risk.

This bundle-level ranking often improves answer quality more than chunk-level reranking.

Failure handling in production

Multi-source plans fail in messier ways than single-index RAG.

Common failures

Source timeout or outage. One backend misses SLA.

Entity resolution failure. The planner cannot map “Northstar launch customer” confidently.

Join explosion. An early step returns thousands of candidate IDs, making downstream retrieval too expensive.

Conflicting evidence. Structured status says approved; latest document says pending review.

Planner misclassification. The request needed sequencing but got fan-out.

Permission mismatch. The user may access summary records but not supporting documents.

Practical mitigations

Timeout-aware degradation paths. Each plan shape should define which steps are optional, which are required, and what fallback answer is allowed.

Cardinality guards. If candidate set > threshold, trigger refinement:

ask a clarifying question,
add stronger filters,
sample or aggregate instead of full retrieval.

Confidence-gated joins. If entity resolution confidence is below threshold, do not silently proceed. Ask for clarification or present top interpretations.

Conflict surfacing. If sources disagree, expose that disagreement rather than averaging it away. Enterprise users often prefer “systems disagree” over fabricated certainty.

Plan retry with alternate shape. If a fan-out plan returns low-joinability evidence, retry a structured-first or entity-resolution-first plan under a smaller residual budget.

Per-source ACL enforcement. Do not retrieve first and filter later for sensitive systems. Enforce access before retrieval when possible; otherwise, mark evidence non-synthesizable.

Evaluation design for multi-source retrieval plans

Evaluating enterprise query planning is harder than evaluating a single retriever because you must assess not just retrieved relevance, but plan correctness.

You need evals at four levels.

1. Routing and plan-shape accuracy

Given a query, did the system choose the right plan family?

Label a benchmark set with expected plan shape:

single-source,
fan-out,
structured-first,
unstructured-first,
entity-resolution-first,
aggregate-then-evidence.

Metrics:

top-1 plan accuracy,
cost-weighted accuracy,
latency-adjusted utility.

A wrong but cheap plan may be acceptable for low-risk use cases; a wrong and expensive plan is not.

2. Step-level retrieval quality

For each plan step, measure whether the right intermediate artifacts were produced:

entities resolved correctly,
filters extracted correctly,
candidate IDs recall,
per-source evidence recall/precision,
join-key extraction quality.

This is where many teams discover that answer failures are really intermediate-state failures.

3. Join quality

Measure whether evidence from multiple systems was correctly associated.

Metrics:

exact join accuracy on canonical IDs,
inferred join precision/recall,
contradiction detection rate,
provenance completeness.

If the system finds the right evidence but joins it to the wrong entity, your answer quality metric may hide the problem until users complain.

4. End-to-end answer utility

Finally, measure:

answer correctness,
citation correctness,
completeness,
abstention quality,
latency,
cost.

For enterprise systems, I strongly recommend a utility score that combines quality with operational metrics. An answer that is 2% more accurate but 4x slower and 6x more expensive may be a regression.

Build scenario-based eval sets

Do not rely only on synthetic single-hop questions. Curate scenarios that stress planning:

hidden dependency questions,
alias-heavy entity resolution,
conflicting source cases,
incomplete source availability,
ACL-restricted evidence,
analytical questions requiring aggregate-then-evidence,
questions with distracting but semantically similar corpora.

Ground truth should include:

correct plan shape,
key intermediate artifacts,
acceptable fallback behaviors,
expected evidence sources,
final answer.

Compare planner policies, not just models

A useful experiment matrix:

fixed policy vs policy-bounded LLM planner,
all-at-once fan-out vs structured-first sequence,
source subsets,
rerank strategies,
timeout policies.

This usually teaches more than comparing one foundation model to another.

Observability patterns for debugging bad plans

Without observability, multi-source RAG becomes folklore. People say things like “the planner got confused” or “retrieval was weak” without evidence.

Instrument the system so every request leaves a trace you can inspect.

What to log per request

raw user query
normalized query IR
chosen plan type
candidate plan scores if applicable
source list considered and rejected
per-step start/end times
per-step queries issued
per-step result counts
join keys extracted and confidence
join graph statistics
evidence envelopes selected for synthesis
missing/failed steps
final answer and claim-to-evidence links if available
token usage and source call costs

This is the minimum for production debugging.

Useful dashboards

Plan distribution dashboard

percentage of requests by plan type
drift over time
failure rate by plan type

Latency waterfall dashboard

p50/p95 by planning phase and source
timeout hotspots

Join health dashboard

average candidate cardinality
inferred-join rate
provenance completeness
contradiction frequency

Source utility dashboard

how often each source contributes evidence to accepted answers
marginal quality lift vs marginal latency/cost

These dashboards often reveal dead weight sources that add cost without helping.

Replay and diffing

Store replayable plan artifacts so you can rerun old requests against:

a new planner policy,
updated source metadata,
a new reranker,
a different model.

Being able to diff “old plan vs new plan” for the same request is enormously valuable. It turns debugging from vibes into engineering.

Model and tool choices: practical tradeoffs

The planner model does not need to be your biggest model. In many systems, a smaller fast model is enough for intent extraction, entity hints, and plan-template scoring. Save larger models for final synthesis or hard decomposition cases.

A practical stack might look like:

Small/fast model for query understanding, parameter extraction, plan-template proposal.
Deterministic policy layer for source eligibility, budgets, and ACL constraints.
Source-native retrievers rather than one universal tool.
Optional stronger reranker or extraction model for join-key extraction on ambiguous evidence.
Higher-quality answer model only after evidence is assembled.

Why this split works:

planning is latency-sensitive and usually structured,
synthesis benefits most from richer reasoning,
expensive models should not be wasted on blind retrieval.

For retrieval tools, hybrid search is usually safer than dense-only for enterprise corpora full of product names, IDs, policy references, and abbreviations. Structured sources should stay structured. Resist the urge to dump everything into embeddings and hope retrieval learns your schema.

Cost and latency tradeoffs you should make explicitly

Multi-source planning is an optimization problem, not just a quality problem.

Three tradeoffs matter most.

Breadth vs precision

More sources increase possible recall but also cost, merge complexity, and noise. Favor narrower plans when there is a strong candidate-narrowing step available.

Parallelism vs waste

Parallel fan-out reduces latency when sources are independently useful. But it wastes money when half the calls are unnecessary. Sequence when upstream results significantly constrain downstream work.

Completeness vs responsiveness

Some use cases need every relevant source before answering. Others can tolerate partial answers with caveats. Define this by workflow:

compliance review assistant: favor completeness and abstention
internal support copilot: favor responsiveness with clear caveats
executive dashboard Q&A: favor authoritative structured answers plus optional evidence links

If you do not make these tradeoffs explicit, the planner will make them accidentally.

Takeaways

Enterprise RAG query planning is really about respecting dependency structure.

When answers span multiple systems, do not ask “which retriever should I use?” Ask:

Is this question independent enough for fan-out?
Does one source need to narrow the candidate set for another?
What are the join keys?
Which source is authoritative for which claim?
What plan gets me the best evidence under this latency and cost budget?

The systems that work in production are usually not fully autonomous agents and not rigid one-path pipelines. They are policy-bounded planners with a small set of well-understood plan shapes, source capability metadata, explicit budgets, and strong provenance handling.

If you build those pieces, you get more than better answers. You get a system you can debug, evaluate, and trust.

And that is the real dividing line in enterprise RAG: not whether the model can speak fluently about many systems, but whether your architecture can retrieve across them deliberately.