October 15, 2026-12 min read

Listen to this briefing

Speech is not supported in this browser. Try Chrome or Edge.

Infographic of Cache-Augmented Generation: documents prefilled into cache, then the LLM generates answers — CAG prefills knowledge into context or KV cache, then generates. RAG retrieves fresh chunks per question.

CAG Explained: Cache-Augmented Generation vs RAG

A newspaper-style briefing on Cache-Augmented Generation (CAG): how prefilled context and KV cache differ from classic RAG, when to use each, hybrid patterns, tools, and how Compile Vision can build it for you.

After RAG became the default way to ground LLMs in company data, another pattern entered the search charts: CAG, or Cache-Augmented Generation. The idea is simple. Instead of retrieving a few chunks on every question, you load a curated knowledge set into the model context (or reuse a prompt/KV cache) and answer from that warm memory.

Long-context models made CAG practical. If your handbook, policy pack, or product FAQ fits comfortably in the window, repeated retrieval can be slower and noisier than answering from a prefilled cache. Teams still reach for RAG when the corpus is huge, multi-tenant, or changes every hour.

This briefing explains how CAG works, how it differs from vector RAG and Agentic RAG, which variants matter, and how Compile Vision helps you choose or combine both.

What Cache-Augmented Generation actually does

In classic RAG, each user question triggers search over an index, then the model sees only the top passages. In CAG, you select the documents that matter for a product or workspace, place them in the prompt (or a cached prefix), and let the model attend over that fixed working set for many questions.

Vendor prompt caching and KV-cache reuse make this cheaper on the second and third call: the expensive prefills of the shared docs are paid once, then subsequent questions append only the new user turn. That is why CAG often feels faster for stable FAQs and internal copilots.

CAG vs RAG: pick with your eyes open

RAG wins when knowledge is large, fragmented, or frequently updated, and when tenants must never see each other's docs. CAG wins when the working set is small-to-medium, relatively stable, and asked about all day by the same team.

Many production systems are hybrids: cache the core policy pack with CAG, and fall back to RAG (or Agentic RAG) when the question needs the long tail of tickets, wiki pages, or fresh releases. Compile Vision designs that decision as a product rule, not a slogan.

How you can build CAG

Curate a knowledge pack with owners and a refresh schedule. Measure token size against your model's context limit and leave headroom for the conversation. Enable prompt caching where the vendor supports it. Add eval questions that must be answered only from the pack, and a path to retrieve or refuse when the pack is silent.

Operationally, treat cache invalidation like a deploy: when legal updates a policy PDF, rebuild the prefill and bump a cache key. Stale CAG is still hallucination with better confidence.

You can prototype this in a week. For production tenancy, refresh jobs, and hybrid RAG fallback, contact Compile Vision and we will ship the pipeline with monitoring.

Key takeaways

CAG prefills or caches knowledge; RAG retrieves per question.

Use CAG for stable, mid-sized packs; use RAG for huge or fast-changing corpora.

Prompt/KV caching makes repeated CAG calls cheaper and faster.

Hybrids are common: cache the core, retrieve the long tail.

Compile Vision can design and ship CAG, RAG, or both with evals.

Types of CAG (and how they differ)

CAG is a family of patterns around prefilled or reused context. These are the shapes teams actually discuss next to RAG.

Full-doc CAG (prefill pack)

How it works: Concatenate the curated docs into the system or first-turn context; answer many questions from that pack.
Best for: Handbooks, SOPs, product FAQs that fit the context window.
How it differs: Simplest mental model. Limited by context size and refresh discipline.

Prompt-cache / KV-cache CAG

How it works: Same shared prefix (docs + instructions) is cached by the provider; later turns reuse the cache.
Best for: High-traffic copilots with repeated system+doc prefixes.
How it differs: Cuts cost and latency after the first prefill. Depends on vendor cache rules and TTLs.

Selective / layered CAG

How it works: Cache only hot modules (billing policy, returns, security); leave cold topics to RAG.
Best for: Support desks with a small set of must-be-perfect policies.
How it differs: Balances freshness and speed. Needs clear ownership of what is hot.

Hybrid CAG + RAG

How it works: Try CAG first; if confidence or coverage is low, retrieve from a vector/hybrid index.
Best for: Most real enterprises with a core pack plus a long tail.
How it differs: Best of both. More moving parts: routing rules and dual evals.

Hybrid CAG + Agentic RAG

How it works: Agent decides whether to use the cache, rewrite a retrieval query, or call tools.
Best for: Multi-step questions across docs, tickets, and CRM.
How it differs: Most flexible; highest need for guardrails and tracing.

Session CAG

How it works: Cache conversation-specific materials uploaded for one meeting or project space.
Best for: Deal rooms, incident war rooms, short-lived project knowledge.
How it differs: Cache lifetime equals the session. Must isolate tenants strictly.

Compressed / summary CAG

How it works: Store distilled summaries or outlines in cache when full docs exceed the window.
Best for: Large manuals that must still feel instant.
How it differs: Fits more topics; risk of losing rare details that RAG would fetch.

Tools & stack options

Practical pieces teams use to prototype and ship.

Long-context LLMs (Claude / Gemini / GPT)
Hold the curated knowledge pack in one window
Prompt caching APIs
Reuse KV/prefill for shared system+doc prefixes
Doc pack builder
Curate, version, and token-count the CAG corpus
pgvector / hybrid RAG (fallback)
Retrieve the long tail when cache coverage fails
LangGraph / routers
Decide CAG vs RAG vs tools per question
Eval harness
Groundedness, cache-hit rate, and stale-doc tests

What you can build

Internal policy copilot with a cached handbook and RAG for tickets
Customer support bot that caches returns/shipping rules for speed
Sales enablement pack cached per product line
Incident war-room assistant with session CAG over runbooks
Cost-optimized FAQ where prompt cache hits dominate traffic
Hybrid CAG + Agentic RAG for complex multi-hop questions

Architecture & diagrams

CAG vs classic RAG

Side-by-side mental model: prefill once versus retrieve every time.

Hybrid CAG + RAG router

A practical production pattern Compile Vision often ships.

Cache refresh lifecycle

Treat knowledge-pack updates like a deploy so answers do not go stale.

References & further reading

Compile Vision: RAG explained for builders
Companion piece on vector, hybrid, GraphRAG, and Agentic RAG.
Compile Vision: production RAG pipelines
How to harden retrieval when CAG alone is not enough.
Anthropic prompt caching
Vendor pattern for reusing long prefixes (related to CAG economics).
OpenAI prompt caching
How repeated prompts can reuse cached tokens on hosted APIs.

Want CAG, RAG, or a hybrid built for your docs?

Contact Compile Vision. We curate the knowledge pack, wire prompt caching, add RAG fallback and evals, and ship a copilot your team can trust. Start at /contact or email contact@compilevision.com.

Contact Compile Vision