Testing RAG Systems: The 5-Step Methodology Google Uses (And You Should Too)

TL;DR

Testing a RAG system is not just "does the answer look correct". Google, Anthropic, and the best AI teams follow a 5-step methodology: (1) build a golden dataset, (2) unit test retrieval, (3) unit test generation, (4) end-to-end evaluation with RAGAS, (5) A/B testing in production. This guide details each step with code, concrete metrics, and pitfalls to avoid.

Why 90% of RAG systems fail in production

Most teams test their RAG by asking a few questions manually. That is like testing a car by checking if it starts. The real problems appear after deployment:

The solution: a systematic, automated, and reproducible testing methodology.

Step 1: Build a Golden Dataset

The golden dataset is your source of truth. It is a set of question-answer-context pairs validated by human experts.

Golden dataset structure

DEVELOPERjson
{
  "id": "GD-001",
  "question": "What are the delivery times in France?",
  "expected_answer": "Delivery times in metropolitan France are 2-5 business days for standard shipping and 24h for express delivery.",
  "expected_contexts": [
    "doc_shipping_france.md#delivery-times",
    "faq_shipping.md#question-12"
  ],
  "category": "logistics",
  "difficulty": "easy",
  "metadata": {
    "created_by": "support_team",
    "created_at": "2026-01-15",
    "last_validated": "2026-03-01"
  }
}

How many pairs do you need?

Generate a golden dataset semi-automatically

DEVELOPERpython
from openai import OpenAI
import json

client = OpenAI()

def generate_golden_pairs(documents: list[dict], n_pairs: int = 5) -> list[dict]:
    """Generate Q&A pairs from real documents."""
    golden_pairs = []

    for doc in documents:
        prompt = f"""From the following document, generate {n_pairs} question-answer pairs.

Document: {doc['content'][:3000]}

Rules:
- Varied questions (factual, comparative, yes/no)
- Answers based ONLY on the document
- Include the exact source section

JSON format:
[{{"question": "...", "answer": "...", "source_section": "..."}}]"""

        response = client.chat.completions.create(
            model="gpt-4o",
            messages=[{"role": "user", "content": prompt}],
            response_format={"type": "json_object"}
        )

        pairs = json.loads(response.choices[0].message.content)
        for pair in pairs.get("pairs", []):
            golden_pairs.append({
                "question": pair["question"],
                "expected_answer": pair["answer"],
                "expected_contexts": [f"{doc['id']}#{pair['source_section']}"],
                "category": doc.get("category", "general"),
                "auto_generated": True,
                "validated": False  # Needs human review
            })

    return golden_pairs

# Usage
documents = [
    {"id": "doc_001", "content": "...", "category": "product"},
    {"id": "doc_002", "content": "...", "category": "support"},
]
golden = generate_golden_pairs(documents, n_pairs=5)
print(f"Generated {len(golden)} pairs - REVIEW REQUIRED")

Human validation: the rule of 3

Each golden dataset pair must be validated by at least 1 person. For critical systems (medical, legal), use the rule of 3: 3 independent validators, majority required.

Step 2: Unit Test Retrieval

Retrieval is the most critical component. If the wrong documents are retrieved, no LLM can generate a good answer.

Key metrics

Retrieval test code

DEVELOPERpython
import numpy as np
from dataclasses import dataclass

@dataclass
class RetrievalResult:
    query: str
    retrieved_docs: list[str]
    relevant_docs: list[str]

def recall_at_k(result: RetrievalResult, k: int = 10) -> float:
    """Recall@k - proportion of relevant documents retrieved."""
    retrieved_set = set(result.retrieved_docs[:k])
    relevant_set = set(result.relevant_docs)
    if not relevant_set:
        return 1.0
    return len(retrieved_set & relevant_set) / len(relevant_set)

def precision_at_k(result: RetrievalResult, k: int = 10) -> float:
    """Precision@k - proportion of retrieved documents that are relevant."""
    retrieved = result.retrieved_docs[:k]
    relevant_set = set(result.relevant_docs)
    if not retrieved:
        return 0.0
    return sum(1 for doc in retrieved if doc in relevant_set) / len(retrieved)

def mrr(result: RetrievalResult) -> float:
    """Mean Reciprocal Rank - inverse rank of first relevant result."""
    relevant_set = set(result.relevant_docs)
    for i, doc in enumerate(result.retrieved_docs):
        if doc in relevant_set:
            return 1.0 / (i + 1)
    return 0.0

def evaluate_retrieval(golden_dataset: list[dict], retriever) -> dict:
    """Evaluate retrieval on the complete golden dataset."""
    metrics = {"recall@5": [], "recall@10": [], "precision@5": [],
               "mrr": [], "hit_rate": []}

    for item in golden_dataset:
        retrieved = retriever.search(item["question"], top_k=10)
        result = RetrievalResult(
            query=item["question"],
            retrieved_docs=[doc.id for doc in retrieved],
            relevant_docs=item["expected_contexts"]
        )

        metrics["recall@5"].append(recall_at_k(result, 5))
        metrics["recall@10"].append(recall_at_k(result, 10))
        metrics["precision@5"].append(precision_at_k(result, 5))
        metrics["mrr"].append(mrr(result))
        metrics["hit_rate"].append(1.0 if recall_at_k(result, 10) > 0 else 0.0)

    return {k: np.mean(v) for k, v in metrics.items()}

Quality thresholds

Step 3: Unit Test Generation

Even with perfect retrieval, the LLM can hallucinate, ignore context, or generate off-topic responses.

Generation metrics

Faithfulness test with LLM-as-judge

DEVELOPERpython
from openai import OpenAI

client = OpenAI()

def evaluate_faithfulness(context: str, answer: str) -> dict:
    """Evaluate if the answer is faithful to the provided context."""
    prompt = f"""Evaluate if the following answer is faithful to the provided context.

CONTEXT:
{context}

ANSWER:
{answer}

Analyze each claim in the answer:
1. Is the claim supported by the context? (supported/unsupported)
2. Is there any fabricated information? (hallucination: yes/no)

Faithfulness score (0.0 to 1.0) = supported claims / total claims

Respond in JSON:
{{"claims": [{{"text": "...", "supported": true/false}}],
  "faithfulness_score": 0.X,
  "hallucinations": ["..."]}}"""

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}],
        response_format={"type": "json_object"}
    )

    return json.loads(response.choices[0].message.content)

# Example
result = evaluate_faithfulness(
    context="Ailog is a French RAG-as-a-Service platform, hosted in France.",
    answer="Ailog is an American RAG platform hosted in the United States."
)
# faithfulness_score: 0.0 (hallucination detected)

Relevancy test

DEVELOPERpython
def evaluate_relevancy(question: str, answer: str) -> dict:
    """Evaluate if the answer is relevant to the question."""
    prompt = f"""Evaluate the relevancy of this answer to the question.

QUESTION: {question}
ANSWER: {answer}

Criteria:
1. Does the answer directly address the question? (0-1)
2. Does the answer contain off-topic information? (0-1)
3. Is the answer complete? (0-1)

Relevancy score (0.0 to 1.0) = average of criteria.

Respond in JSON:
{{"directness": 0.X, "focus": 0.X, "completeness": 0.X,
  "relevancy_score": 0.X, "explanation": "..."}}"""

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}],
        response_format={"type": "json_object"}
    )

    return json.loads(response.choices[0].message.content)

Step 4: End-to-End Evaluation with RAGAS

RAGAS (Retrieval Augmented Generation Assessment) is the reference framework for evaluating a complete RAG pipeline.

Installation and usage

DEVELOPERpython
from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
    answer_correctness
)
from datasets import Dataset

# Prepare data
data = {
    "question": [],
    "answer": [],
    "contexts": [],
    "ground_truth": []
}

for item in golden_dataset:
    # Run the RAG pipeline
    rag_result = rag_pipeline.query(item["question"])

    data["question"].append(item["question"])
    data["answer"].append(rag_result.answer)
    data["contexts"].append(rag_result.contexts)
    data["ground_truth"].append(item["expected_answer"])

dataset = Dataset.from_dict(data)

# RAGAS evaluation
results = evaluate(
    dataset,
    metrics=[
        faithfulness,
        answer_relevancy,
        context_precision,
        context_recall,
        answer_correctness
    ]
)

print(results)
# {'faithfulness': 0.87, 'answer_relevancy': 0.91,
#  'context_precision': 0.78, 'context_recall': 0.85,
#  'answer_correctness': 0.82}

Evaluation framework comparison

RAGAS vs DeepEval: detailed comparison

DeepEval: solid alternative

DEVELOPERpython
from deepeval import evaluate
from deepeval.metrics import (
    FaithfulnessMetric,
    AnswerRelevancyMetric,
    ContextualRelevancyMetric
)
from deepeval.test_case import LLMTestCase

# Create test cases
test_cases = []
for item in golden_dataset:
    rag_result = rag_pipeline.query(item["question"])
    test_cases.append(
        LLMTestCase(
            input=item["question"],
            actual_output=rag_result.answer,
            expected_output=item["expected_answer"],
            retrieval_context=rag_result.contexts
        )
    )

# Metrics
faithfulness = FaithfulnessMetric(threshold=0.7)
relevancy = AnswerRelevancyMetric(threshold=0.7)
context = ContextualRelevancyMetric(threshold=0.7)

# Evaluation
results = evaluate(test_cases, [faithfulness, relevancy, context])

Step 5: A/B Testing in Production

Offline metrics are not enough. The real test is production.

Production metrics

A/B testing implementation

DEVELOPERpython
import random
import hashlib
from datetime import datetime

class RAGABTest:
    def __init__(self, variant_a, variant_b, traffic_split: float = 0.5):
        self.variant_a = variant_a  # RAG pipeline v1
        self.variant_b = variant_b  # RAG pipeline v2
        self.traffic_split = traffic_split
        self.results = {"a": [], "b": []}

    def get_variant(self, user_id: str) -> str:
        """Deterministic assignment based on user_id."""
        hash_val = int(hashlib.md5(user_id.encode()).hexdigest(), 16)
        return "a" if (hash_val % 100) < (self.traffic_split * 100) else "b"

    async def query(self, question: str, user_id: str) -> dict:
        variant = self.get_variant(user_id)
        pipeline = self.variant_a if variant == "a" else self.variant_b

        start_time = datetime.now()
        result = await pipeline.query(question)
        latency = (datetime.now() - start_time).total_seconds()

        # Log for analysis
        self.results[variant].append({
            "question": question,
            "answer": result.answer,
            "latency": latency,
            "variant": variant,
            "timestamp": datetime.now().isoformat()
        })

        return {
            "answer": result.answer,
            "sources": result.sources,
            "variant": variant  # For frontend tracking
        }

    def get_stats(self) -> dict:
        """Comparative statistics."""
        stats = {}
        for variant in ["a", "b"]:
            data = self.results[variant]
            if data:
                latencies = [d["latency"] for d in data]
                stats[variant] = {
                    "count": len(data),
                    "avg_latency": sum(latencies) / len(latencies),
                    "p95_latency": sorted(latencies)[int(len(latencies) * 0.95)]
                }
        return stats

Minimum A/B test duration

CI/CD Integration

GitHub Actions for RAG testing

DEVELOPERyaml
# .github/workflows/rag-tests.yml
name: RAG Quality Tests

on:
  push:
    branches: [develop, main]
    paths:
      - 'backend/rag/**'
      - 'backend/prompts/**'
      - 'tests/golden_dataset.json'

jobs:
  rag-evaluation:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: pip install ragas deepeval openai

      - name: Run retrieval tests
        run: python tests/test_retrieval.py
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          QDRANT_URL: ${{ secrets.QDRANT_TEST_URL }}

      - name: Run RAGAS evaluation
        run: python tests/test_ragas.py --threshold 0.80

      - name: Upload results
        uses: actions/upload-artifact@v4
        with:
          name: rag-evaluation-results
          path: tests/results/

      - name: Check thresholds
        run: |
          python -c "
          import json
          with open('tests/results/ragas_scores.json') as f:
              scores = json.load(f)
          assert scores['faithfulness'] >= 0.80, f'Faithfulness too low: {scores[\"faithfulness\"]}'
          assert scores['answer_relevancy'] >= 0.75, f'Relevancy too low: {scores[\"answer_relevancy\"]}'
          assert scores['context_recall'] >= 0.80, f'Context recall too low: {scores[\"context_recall\"]}'
          print('All RAG quality thresholds passed')
          "

Regression alerts

DEVELOPERpython
def check_regression(current_scores: dict, baseline_scores: dict,
                     tolerance: float = 0.05) -> list[str]:
    """Detect regressions against baseline."""
    alerts = []
    for metric, current in current_scores.items():
        baseline = baseline_scores.get(metric, 0)
        if current < baseline - tolerance:
            drop = baseline - current
            alerts.append(
                f"REGRESSION: {metric} dropped by {drop:.3f} "
                f"(baseline: {baseline:.3f}, current: {current:.3f})"
            )
    return alerts

The 7 most common pitfalls

Our approach at Ailog

At Ailog, every client RAG pipeline is tested automatically:

1. Golden dataset: semi-automatically generated from client documents, validated by our team
2. Retrieval tests: recall@10 > 0.85 required before deployment
3. RAGAS tests: faithfulness > 0.80, relevancy > 0.75
4. Production monitoring: automatic alerts if thumbs-down > 20%

Discover how we evaluate our RAG systems and how we monitor in production.

Conclusion

Testing a RAG system is not optional - it is what separates a prototype from a product. The 5-step methodology works at any scale:

1. Golden dataset: your source of truth
2. Retrieval tests: the most critical component
3. Generation tests: faithfulness and relevance
4. End-to-end RAGAS: the complete assessment
5. A/B testing: the final validation

Start small (50 pairs, basic RAGAS), then automate. The investment pays for itself with the first bug caught before production.

Want a RAG system that is tested and monitored automatically? Try Ailog - our pipelines include automatic evaluation and real-time monitoring.