Prompt Caching vs Semantic Caching: Two Different Mechanisms Confusingly Named

Two completely different mechanisms, confusingly named

Both have "cache" in the name. Both reduce cost. They share almost nothing else. Teams that confuse them either miss huge savings or break correctness.

This is Part 7 of the AI Engineer Series. Prompt caching cuts the cost of repeated context. Semantic caching cuts the cost of repeated queries. You usually want both, applied at different layers. Let us look at what each actually does, when to use which, and the gotchas that cost real money.

Prompt caching: provider-side, prefix-based

Prompt caching is something the provider does for you. Anthropic, OpenAI, and Google all offer it. The mechanism is the same in spirit: when you send a prompt, the model's internal state (KV cache, which we cover in Part 9) for the prefix gets stored. The next request that shares the prefix skips reprocessing it.

The savings are substantial. On Anthropic, cache reads are 10% of base input cost. On OpenAI, cached input is 50% off. On Google, cached input is 25-90% off depending on the model. TTFT drops dramatically too, often from several seconds to a few hundred milliseconds for prompts with long prefixes.

The rules that decide whether your cache works:

• Caches are prefix-based. The cache match must be byte-identical from the start of the prompt. Change a single character at position one and the cache misses entirely.
• Order matters. Put stable content first (system prompt, fixed instructions, retrieved documents), variable content last (the user's current question).
• Caches have a TTL. Anthropic offers 5-minute and 1-hour caches, OpenAI roughly 5-10 minutes. Idle traffic means cold caches.
• Cache writes cost more than reads. Anthropic charges 1.25x base for a 5-minute write, 2x for a 1-hour write, then 0.1x for reads. The math only works if hit rate is high enough.
• You may need explicit breakpoints. Anthropic uses cache_control markers; OpenAI and Google cache automatically when prefixes are long enough.

from anthropic import Anthropic

client = Anthropic()

# System prompt + long retrieved context = the prefix to cache
SYSTEM_PROMPT = "You are a helpful assistant that answers based on provided context."
context = retrieve_relevant_docs(user_query)  # ~10K tokens, stable across requests in a session

response = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": SYSTEM_PROMPT,
        },
        {
            "type": "text",
            "text": f"\n{context}\n",
            "cache_control": {"type": "ephemeral"},  # mark cache breakpoint
        },
    ],
    messages=[{"role": "user", "content": user_query}],
)

# Inspect cache usage
print(response.usage.cache_creation_input_tokens)  # cache write
print(response.usage.cache_read_input_tokens)      # cache hit

The audit that pays for itself

Open your prompt template right now. Scan for any value that changes per request: timestamps, request IDs, rotating session IDs, current date strings, dynamic user metadata. If any of those appear before stable content like the system prompt or retrieved context, they are silently destroying your cache hit rate.

The fix is order. Stable content first, dynamic content last. This single audit recovers 30-90% cost on long-prompt workloads. It is the single highest-ROI change in this entire post.

Semantic caching: application-side, query-based

Semantic caching is something you implement. The mechanism: when a query arrives, embed it, check if a semantically similar query has been answered before, return the cached answer if similarity exceeds a threshold.

The savings are total. The entire LLM call is skipped. But the correctness risk is also total, because you are answering a new question with a previously-answered one. The threshold and use case selection are everything.

import numpy as np
from typing import Optional

class SemanticCache:
    def __init__(self, embed_fn, threshold: float = 0.92):
        self.embed_fn = embed_fn
        self.threshold = threshold
        self.entries = []  # list of (embedding, query, answer)
    
    def lookup(self, query: str) -> Optional[str]:
        if not self.entries:
            return None
        query_emb = self.embed_fn(query)
        embeddings = np.array([e[0] for e in self.entries])
        similarities = embeddings @ query_emb
        best_idx = int(np.argmax(similarities))
        if similarities[best_idx] >= self.threshold:
            return self.entries[best_idx][2]
        return None
    
    def store(self, query: str, answer: str):
        emb = self.embed_fn(query)
        self.entries.append((emb, query, answer))


cache = SemanticCache(embed_fn=embed_text, threshold=0.92)

def cached_answer(query: str) -> str:
    hit = cache.lookup(query)
    if hit:
        return hit
    answer = expensive_llm_call(query)
    cache.store(query, answer)
    return answer

The threshold of 0.92 is conservative. In practice, you tune it by running both cached and uncached responses on a labeled set and measuring false-hit rate (cache returned a wrong answer) versus hit rate. The target is zero false hits, which usually means a threshold above 0.93 for OpenAI text-embedding-3 and above 0.88 for newer rotation-invariant embeddings.

When semantic caching is safe

Only when all three conditions hold:

• The answer does not depend on time. "What is our refund policy?" qualifies. "What is the current weather in Boston?" does not.
• The answer does not depend on the user. Public FAQ, general knowledge. Anything personalized is unsafe at the global cache layer (you can do per-user caches, but they have low hit rates).
• The answer does not depend on hidden context. If the answer is constructed from retrieved documents that change frequently, the cached answer goes stale.

Violations are how teams ship subtle bugs. The classic case: a chatbot caches "What is my account balance?" because the embedding looks like a generic question. Now every user gets the first user's balance. Always check the request domain before reaching for semantic caching.

When to use each

The composition pattern

Layer semantic cache on top of prompt cache. Try semantic cache first. On miss, call the LLM with prompt caching enabled.

def answer(query: str) -> str:
    # Layer 1: semantic cache (skip the LLM entirely)
    hit = semantic_cache.lookup(query)
    if hit:
        return hit
    
    # Layer 2: LLM call with prompt caching (skip the prefill)
    answer = llm_call_with_prompt_caching(query)
    
    semantic_cache.store(query, answer)
    return answer

The two layers attack different parts of the cost curve. Semantic cache eliminates the call entirely for recurring questions. Prompt cache reduces the cost of the calls that actually happen. Together, on the right workload, they can cut LLM costs by 5-10x.

Observability for caches

You cannot manage what you cannot see. Log per request:

• Semantic cache: hit/miss, similarity score, latency saved
• Prompt cache: input tokens (cached and uncached separately), cache write tokens, cache hit rate per prompt template

The metrics that matter: semantic cache hit rate (target 20%+ for FAQ workloads), false-hit rate (target zero, measured by sampling), and prompt cache hit rate (target 80%+ for prompts with stable prefixes).

What to build this week

Two concrete experiments:

• Audit your top three prompt templates for cache-defeating dynamic content. Reorder to put stable content first. Add cache_control markers if using Anthropic. Measure the cost drop over a week.
• Identify one user-facing endpoint where queries repeat (an FAQ bot, a search interface, a customer support assistant). Add semantic caching with a conservative 0.94 threshold. Log every hit. Spot-check 30 hits manually. If any answers wrong, raise the threshold.

Most teams discover their prompt cache hit rate is much lower than they expected, usually because of timestamp or session ID contamination. The fix is one PR and saves real money.

What is next

Part 8: cost attribution per feature. Caching reduces the cost. Attribution tells you which features actually drive the cost in the first place. We will look at tagging discipline, unit economics, and the hidden cost drivers that explain why your bill spiked last month.

Previous in the series

• Part 1: Harness Engineering
• Part 2: Context Engineering & Retrieval Quality
• Part 3: Structured Outputs and Fallback Chains
• Part 4: Evals
• Part 5: LLM Observability
• Part 6: Latency and Throughput Engineering

References

1. Anthropic Prompt Caching documentation. The reference for cache_control, TTL options, and pricing multipliers.
2. OpenAI Prompt Caching guide. Automatic prefix caching, no markers required.
3. Gemini Context Caching documentation. Google's implicit and explicit cache options.
4. GPTCache. The reference open-source semantic caching library.
5. Anthropic prompt caching launch post. The original announcement with cost and latency benchmarks.
6. Eugene Yan on caching. A broader treatment of caching in LLM systems.