Prompt Caching: The Trick That Cuts Your LLM Bill by 10x

Is your LLM bill spiraling out of control? Are you sending the same system instructions, the same few-shot examples, the same RAG context with every request? There's a solution that can cut your costs by 90% with Anthropic and 50% with OpenAI -- and most developers aren't using it yet.

Welcome to the world of prompt caching.

TL;DR

• Prompt caching = the LLM provider caches the prefix of your prompt so it doesn't reprocess it every request
• Anthropic: -90% on cached tokens (most aggressive), minimum 1,024 tokens, 5-minute TTL
• OpenAI: -50% on cached tokens, fully automatic, minimum 1,024 tokens
• Google: -75% on cached tokens, explicit context caching, minimum 32,768 tokens
• For RAG: cache the system prompt + few-shot examples + static context, only vary the query
• Typical savings: 40-80% reduction on monthly LLM bill

How Prompt Caching Works

The principle: prefix matching

On each LLM call, the provider compares the beginning of your prompt with recent prompts. If an identical prefix is found in cache, only the new portion is processed at full price.

Request 1:
┌──────────────────────────────────────────────────────────┐
│ System prompt (2000 tokens) │ Few-shots (1000) │ Query 1 │
│ ████████████████████████████ │ ████████████████ │ ████    │
│        Processed normally        │                │         │
└──────────────────────────────────────────────────────────┘
Cost: 3000 tokens × normal price = $0.045

Request 2 (same prefix):
┌──────────────────────────────────────────────────────────┐
│ System prompt (2000 tokens) │ Few-shots (1000) │ Query 2 │
│ ░░░░░░░░░░░░░░░░░░░░░░░░░░ │ ░░░░░░░░░░░░░░░ │ ████    │
│        CACHED (reduced rate)     │                │ Normal  │
└──────────────────────────────────────────────────────────┘
Cost: 3000 tokens × cached price + 50 tokens × normal price = much less

Conditions for cache hits

Anthropic: The Caching Champion (-90%)

How it works

Anthropic offers the most aggressive reduction: 90% off cached tokens, with a slight 25% surcharge on the initial cache write.

Detailed pricing: Claude 3.5 Sonnet

Anthropic implementation

DEVELOPERpython
import anthropic

client = anthropic.Anthropic()

# The system prompt will be cached after the first call
SYSTEM_PROMPT = """You are an expert assistant for {company_name}.
You answer only from the provided documents.
You always cite your sources with [Source: document_name].
You never hallucinate. If you don't know, say so.

## Formatting rules
- Concise responses (3-5 sentences max)
- Use bullet points for lists
- Cite the source in brackets

## Expected response examples

Q: What is the return policy?
A: Our return policy allows returns within 30 days
   for any unused item [Source: return-policy.pdf].
   Return shipping costs are the customer's responsibility
   except for defective products [Source: terms-2026.pdf].

Q: How do I configure SSO?
A: To configure SSO, go to Settings > Security > SSO.
   Select your provider (Okta, Azure AD, Google)
   and paste the metadata URL [Source: admin-guide.pdf].
""" + context_documents  # Add your RAG context here

def query_with_caching(user_query: str, rag_context: str):
    """Query with Anthropic prompt caching."""
    response = client.messages.create(
        model="claude-sonnet-4-20250514",
        max_tokens=1024,
        system=[
            {
                "type": "text",
                "text": SYSTEM_PROMPT,
                "cache_control": {"type": "ephemeral"}  # Enable caching
            }
        ],
        messages=[
            {
                "role": "user",
                "content": f"RAG Context:\n{rag_context}\n\nQuestion: {user_query}"
            }
        ],
    )

    # Check cache hit
    usage = response.usage
    print(f"Cache read: {usage.cache_read_input_tokens} tokens")
    print(f"Cache write: {usage.cache_creation_input_tokens} tokens")
    print(f"Uncached input: {usage.input_tokens} tokens")

    return response.content[0].text

Multi-breakpoint strategy with Anthropic

DEVELOPERpython
def query_with_multi_cache(user_query: str, rag_context: str):
    """Multiple cache levels to maximize savings."""
    response = client.messages.create(
        model="claude-sonnet-4-20250514",
        max_tokens=1024,
        system=[
            {
                "type": "text",
                "text": STATIC_SYSTEM_PROMPT,  # Rarely changes
                "cache_control": {"type": "ephemeral"}
            },
            {
                "type": "text",
                "text": FEW_SHOT_EXAMPLES,  # Changes sometimes
                "cache_control": {"type": "ephemeral"}
            }
        ],
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": rag_context,  # Changes often but shared
                        "cache_control": {"type": "ephemeral"}
                    },
                    {
                        "type": "text",
                        "text": user_query,  # Always unique
                    }
                ]
            }
        ],
    )
    return response

OpenAI: Automatic Caching (-50%)

How it works

OpenAI launched automatic caching: no configuration needed, the system automatically detects and caches identical prefixes.

Detailed pricing: GPT-4o

OpenAI implementation

DEVELOPERpython
from openai import OpenAI

client = OpenAI()

# Caching is AUTOMATIC with OpenAI
# Just structure the prompt with a stable prefix

SYSTEM_PROMPT = """You are an expert assistant for {company_name}.
... (same long system prompt) ...
"""

def query_with_openai_caching(user_query: str, rag_context: str):
    """Caching is automatic - just keep the prefix identical."""
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            # Tip: put stable RAG context as first user message
            {"role": "user", "content": f"Context:\n{rag_context}"},
            {"role": "assistant", "content": "Context noted."},
            {"role": "user", "content": user_query},
        ],
        max_tokens=1024,
    )

    # Check cached tokens in usage
    usage = response.usage
    cached = getattr(usage, 'prompt_tokens_details', {})
    if cached:
        print(f"Cached tokens: {cached.get('cached_tokens', 0)}")

    return response.choices[0].message.content

Maximizing cache hit rate with OpenAI

DEVELOPERpython
# BAD: message order changes → no cache hit
messages_v1 = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": f"Context: {context_A}\nQuery: {query_1}"},
]
messages_v2 = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": f"Context: {context_B}\nQuery: {query_2}"},
]
# context_A ≠ context_B → cache miss from 2nd message

# GOOD: separate stable and variable context
messages_v1 = [
    {"role": "system", "content": SYSTEM},  # ← Always identical
    {"role": "user", "content": "Context: " + STATIC_CONTEXT},  # ← Identical
    {"role": "assistant", "content": "OK."},
    {"role": "user", "content": query_1},  # ← Only variable element
]
messages_v2 = [
    {"role": "system", "content": SYSTEM},  # ← Cache hit!
    {"role": "user", "content": "Context: " + STATIC_CONTEXT},  # ← Cache hit!
    {"role": "assistant", "content": "OK."},
    {"role": "user", "content": query_2},  # ← Only variable element
]

Google: Explicit Context Caching (-75%)

How it works

Google offers explicit context caching with configurable TTLs and a 75% discount.

Detailed pricing: Gemini 1.5 Pro

Google implementation

DEVELOPERpython
import google.generativeai as genai
from google.generativeai import caching
import datetime

genai.configure(api_key="GOOGLE_API_KEY")

# Create an explicit cache
cache = caching.CachedContent.create(
    model="models/gemini-1.5-pro-002",
    display_name="rag-system-prompt",
    system_instruction=SYSTEM_PROMPT,
    contents=[
        # Pre-load static RAG context (must be > 32K tokens)
        {"role": "user", "parts": [{"text": LARGE_RAG_CONTEXT}]},
        {"role": "model", "parts": [{"text": "Context loaded."}]},
    ],
    ttl=datetime.timedelta(hours=1),  # Cache for 1 hour
)

# Use the cache for queries
model = genai.GenerativeModel.from_cached_content(cached_content=cache)

def query_with_google_caching(user_query: str):
    """Query using Google context cache."""
    response = model.generate_content(user_query)

    # Check cache usage
    print(f"Cached tokens: {response.usage_metadata.cached_content_token_count}")
    print(f"Total tokens: {response.usage_metadata.total_token_count}")

    return response.text

# Delete the cache when no longer needed
cache.delete()

Provider Comparison

Summary table

Cost simulator: 10,000 requests / day

Assumption: 3,000-token system prompt + 200-token query, GPT-4o / Claude 3.5 Sonnet / Gemini 1.5 Pro.

Note: Actual savings depend on cache hit rate. With a stable system prompt and steady traffic, hit rate exceeds 90%.

Which provider for caching?

RAG-Specific Caching Strategies

Optimal architecture for RAG

┌─────────────────────────────────────────────────────┐
│               RAG PROMPT STRUCTURE                    │
├─────────────────────────────────────────────────────┤
│                                                       │
│  ┌─────────────────────────────────┐ ← CACHED        │
│  │   System prompt (instructions)  │   (identical     │
│  │   + formatting rules            │    for all       │
│  │   + tone of voice               │    requests)     │
│  └─────────────────────────────────┘                  │
│                                                       │
│  ┌─────────────────────────────────┐ ← CACHED        │
│  │   Few-shot examples (3-5)       │   (rarely        │
│  │   + expected format             │    changes)      │
│  └─────────────────────────────────┘                  │
│                                                       │
│  ┌─────────────────────────────────┐ ← PARTIALLY     │
│  │   Dynamic RAG context           │   CACHED         │
│  │   (retrieved documents)         │   (if same docs) │
│  └─────────────────────────────────┘                  │
│                                                       │
│  ┌─────────────────────────────────┐ ← NOT CACHED    │
│  │   User query                    │   (always        │
│  │                                 │    unique)       │
│  └─────────────────────────────────┘                  │
└─────────────────────────────────────────────────────┘

Strategy 1: Cache system prompt + few-shots

The simplest and most effective strategy:

DEVELOPERpython
# System prompt and few-shots never change
# → Cache hit rate close to 100%

CACHED_PREFIX = f"""{SYSTEM_PROMPT}

## Examples
{FEW_SHOT_EXAMPLES}
"""

# Only the RAG context and query change
def build_prompt(rag_docs: list, query: str) -> str:
    return f"""{CACHED_PREFIX}

## Reference documents
{format_documents(rag_docs)}

## Question
{query}
"""

Strategy 2: Cache frequently accessed RAG context

For recurring questions that return the same documents:

DEVELOPERpython
from functools import lru_cache

@lru_cache(maxsize=100)
def get_top_documents(topic: str) -> str:
    """Local cache of most requested documents."""
    docs = search_qdrant(topic, limit=5)
    return format_documents(docs)

# If 2 users ask questions on the same topic
# → same RAG context → cache hit on the prefix

Strategy 3: Grouping by tenant / category

DEVELOPERpython
# Multi-tenant: each tenant has its own cached system prompt
TENANT_PROMPTS = {
    "tenant_A": "You are TechCorp's assistant...",
    "tenant_B": "You are RetailCo's assistant...",
}

# As long as requests from the same tenant arrive within the TTL window
# → cache works perfectly

Savings Calculator

Calculation formula

Monthly savings =
  (num_requests × cached_tokens × normal_price × (1 - cache_discount))
  - (num_requests × cached_tokens × cache_price)
  - cache_write_cost

Concrete examples

Pitfalls and Best Practices

Common mistakes

Optimization checklist

1. Structure the prompt: static at top, dynamic at bottom
2. Minimize changes in the prefix
3. Monitor cache hit rate
4. Group requests by tenant/category
5. Avoid dynamic elements (date, time) in the prefix
6. Test with before/after metrics

FAQ

Is prompt caching compatible with streaming?

Yes, prompt caching is fully compatible with streaming across all 3 providers. Caching acts on input tokens, not on generation (output). You can cache your prompt and stream the response normally.

What happens if my system prompt changes?

A change to the system prompt invalidates the cache. The first request after the change pays full price (+ write surcharge with Anthropic). Subsequent requests benefit from the new cache. Tip: version your prompts and deploy changes during off-peak hours.

Does caching work in multi-tenant setups?

Yes, but each tenant will have its own cache (since system prompts differ). The trick: structure the prompt with a portion common to all tenants (cached) and a tenant-specific portion (uncached or with its own cache via Anthropic breakpoints).

How long does the cache stay active?

With Anthropic: 5 minutes, renewed on each hit. With OpenAI: 5-10 minutes, not guaranteed. With Google: configurable up to 24h. For continuous traffic (> 1 request / 5 min), the cache practically never expires with Anthropic.

Is prompt caching useful for low volumes?

Beyond ~100 requests per day with a stable prompt, prompt caching is worthwhile. Below that, the cache may expire between requests (especially with OpenAI's short TTL). For very low volumes, focus first on model selection optimization.

---

Ready to slash your LLM bill? Create your Ailog account and benefit from an optimized RAG pipeline with built-in prompt caching, hosted in France.