Automated RAG Evaluation Pipeline: Catch Regressions Before Your Users Do

TL;DR

83% of RAG teams discover regressions through user complaints. This guide shows you how to build an automated evaluation pipeline that detects problems before production: managed golden datasets, automated RAGAS metrics (faithfulness, relevancy, context recall), alert thresholds, and GitHub Actions integration. With a well-configured pipeline, you catch 95% of regressions before they reach users.

Why an automated evaluation pipeline

The problem with manual evaluations

Causes of regression in RAG

┌────────────────────────────────────────────────────┐
│              RAG REGRESSION CAUSES                   │
├────────────────────────────────────────────────────┤
│                                                    │
│  Document changes                                  │
│  ├─ Poorly formatted new documents                 │
│  ├─ Deleted documents that were relevant           │
│  └─ Updates that break chunking                    │
│                                                    │
│  Configuration changes                             │
│  ├─ New embedding model                            │
│  ├─ Chunk size change                              │
│  ├─ Prompt modification                            │
│  └─ Reranker update                                │
│                                                    │
│  LLM model changes                                 │
│  ├─ New model version                              │
│  ├─ Provider change                                │
│  └─ Temperature/top-p modification                 │
│                                                    │
│  Silent degradation                                │
│  ├─ Data drift                                     │
│  ├─ Fragmented vector index                        │
│  └─ Embedding provider API change                  │
└────────────────────────────────────────────────────┘

Evaluation pipeline architecture

Overview

┌─────────────┐     ┌──────────────┐     ┌──────────────┐
│   Golden     │     │    RAG       │     │  Automated   │
│   Dataset    │────▶│   Pipeline   │────▶│  Evaluation  │
│              │     │              │     │              │
│ - Questions  │     │ - Retrieval  │     │ - RAGAS      │
│ - Answers    │     │ - Generation │     │ - DeepEval   │
│ - Contexts   │     │              │     │ - Custom     │
└─────────────┘     └──────────────┘     └──────┬───────┘
                                                 │
                                        ┌────────┴────────┐
                                        │                 │
                                  ┌─────┴─────┐    ┌──────┴──────┐
                                  │ Thresholds │    │  Dashboard  │
                                  │ & Alerts   │    │  Langfuse   │
                                  └─────┬─────┘    └─────────────┘
                                        │
                                  ┌─────┴─────┐
                                  │  CI/CD    │
                                  │  Gate     │
                                  └───────────┘

Step 1: Build the golden dataset

Golden dataset structure

DEVELOPERpython
from dataclasses import dataclass, field
from typing import List, Optional
import json

@dataclass
class GoldenExample:
    """A golden dataset example."""
    id: str
    question: str
    expected_answer: str
    expected_contexts: List[str]
    category: str  # "factual", "comparison", "multi-hop", "opinion"
    difficulty: str  # "easy", "medium", "hard"
    tags: List[str] = field(default_factory=list)
    metadata: dict = field(default_factory=dict)

@dataclass
class GoldenDataset:
    """Complete evaluation dataset."""
    version: str
    created_at: str
    examples: List[GoldenExample]

    def to_json(self, path: str):
        with open(path, "w") as f:
            json.dump(self.__dict__, f, indent=2, default=str)

    @classmethod
    def from_json(cls, path: str):
        with open(path) as f:
            data = json.load(f)
        examples = [GoldenExample(**ex) for ex in data["examples"]]
        return cls(
            version=data["version"],
            created_at=data["created_at"],
            examples=examples
        )

Auto-generate the golden dataset

DEVELOPERpython
import anthropic

client = anthropic.Anthropic()

def generate_golden_examples(
    documents: list,
    num_questions: int = 50,
    categories: list = None
) -> list:
    """Generate test examples from documents."""
    if categories is None:
        categories = ["factual", "comparison", "multi-hop"]

    examples = []

    for doc in documents:
        response = client.messages.create(
            model="claude-sonnet-4-20250514",
            max_tokens=2000,
            messages=[{
                "role": "user",
                "content": f"""From the following document, generate
{num_questions // len(documents)} question/answer pairs to
test a RAG system.

For each pair:
- question: natural, like a real user would ask
- expected_answer: the correct and complete answer
- context: the exact passage from the document containing the answer
- category: {', '.join(categories)}
- difficulty: easy, medium, or hard

Return JSON: [{{"question": "...", "expected_answer": "...",
"context": "...", "category": "...", "difficulty": "..."}}]

Document:
{doc['text'][:5000]}"""
            }]
        )

        batch = json.loads(response.content[0].text)
        for item in batch:
            examples.append(GoldenExample(
                id=f"gen_{len(examples)}",
                question=item["question"],
                expected_answer=item["expected_answer"],
                expected_contexts=[item["context"]],
                category=item["category"],
                difficulty=item["difficulty"],
                tags=[doc.get("source", "unknown")]
            ))

    return examples

Recommended golden dataset size

Step 2: Implement metrics

With RAGAS

DEVELOPERpython
from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
    answer_correctness,
)
from datasets import Dataset

def evaluate_rag_with_ragas(
    golden_dataset: GoldenDataset,
    rag_pipeline,
    metrics=None
):
    """Evaluate RAG pipeline with RAGAS."""
    if metrics is None:
        metrics = [
            faithfulness,
            answer_relevancy,
            context_precision,
            context_recall,
            answer_correctness,
        ]

    questions = []
    answers = []
    contexts = []
    ground_truths = []

    for example in golden_dataset.examples:
        result = rag_pipeline.query(example.question)

        questions.append(example.question)
        answers.append(result["answer"])
        contexts.append(result["retrieved_contexts"])
        ground_truths.append(example.expected_answer)

    eval_dataset = Dataset.from_dict({
        "question": questions,
        "answer": answers,
        "contexts": contexts,
        "ground_truth": ground_truths,
    })

    results = evaluate(
        eval_dataset,
        metrics=metrics,
    )

    return results

results = evaluate_rag_with_ragas(golden_dataset, my_rag_pipeline)
print(f"Faithfulness: {results['faithfulness']:.3f}")
print(f"Answer Relevancy: {results['answer_relevancy']:.3f}")
print(f"Context Recall: {results['context_recall']:.3f}")
print(f"Context Precision: {results['context_precision']:.3f}")

With DeepEval

DEVELOPERpython
from deepeval import evaluate
from deepeval.metrics import (
    FaithfulnessMetric,
    AnswerRelevancyMetric,
    ContextualRecallMetric,
    ContextualPrecisionMetric,
    HallucinationMetric,
)
from deepeval.test_case import LLMTestCase

def evaluate_with_deepeval(
    golden_dataset: GoldenDataset,
    rag_pipeline
):
    """Evaluate RAG pipeline with DeepEval."""
    test_cases = []

    for example in golden_dataset.examples:
        result = rag_pipeline.query(example.question)

        test_case = LLMTestCase(
            input=example.question,
            actual_output=result["answer"],
            expected_output=example.expected_answer,
            retrieval_context=result["retrieved_contexts"],
            context=example.expected_contexts,
        )
        test_cases.append(test_case)

    metrics = [
        FaithfulnessMetric(threshold=0.8),
        AnswerRelevancyMetric(threshold=0.7),
        ContextualRecallMetric(threshold=0.7),
        ContextualPrecisionMetric(threshold=0.7),
        HallucinationMetric(threshold=0.2),
    ]

    results = evaluate(test_cases, metrics)
    return results

Custom metrics with pytest

DEVELOPERpython
# tests/test_rag_quality.py
import pytest
import json

GOLDEN_DATASET = GoldenDataset.from_json("tests/golden_dataset.json")
RAG_PIPELINE = init_rag_pipeline()

THRESHOLDS = {
    "faithfulness": 0.85,
    "answer_relevancy": 0.80,
    "context_recall": 0.75,
    "context_precision": 0.70,
    "answer_correctness": 0.75,
    "latency_p95_ms": 3000,
    "hallucination_rate": 0.05,
}

class TestRAGQuality:
    """Automated RAG quality tests."""

    @pytest.fixture(autouse=True)
    def setup(self):
        self.results = []
        for example in GOLDEN_DATASET.examples:
            result = RAG_PIPELINE.query(example.question)
            self.results.append({
                "example": example,
                "result": result
            })

    def test_faithfulness_above_threshold(self):
        """Answers are faithful to sources."""
        scores = [
            compute_faithfulness(r["result"]["answer"],
                                 r["result"]["retrieved_contexts"])
            for r in self.results
        ]
        avg_score = sum(scores) / len(scores)
        assert avg_score >= THRESHOLDS["faithfulness"], \
            f"Faithfulness {avg_score:.3f} < {THRESHOLDS['faithfulness']}"

    def test_context_recall_above_threshold(self):
        """Correct documents are retrieved."""
        scores = []
        for r in self.results:
            retrieved = r["result"]["retrieved_contexts"]
            expected = r["example"].expected_contexts
            recall = compute_context_recall(retrieved, expected)
            scores.append(recall)

        avg_score = sum(scores) / len(scores)
        assert avg_score >= THRESHOLDS["context_recall"], \
            f"Context Recall {avg_score:.3f} < {THRESHOLDS['context_recall']}"

    def test_no_hallucination_spike(self):
        """No more than 5% hallucinations."""
        hallucination_count = sum(
            1 for r in self.results
            if is_hallucination(
                r["result"]["answer"],
                r["result"]["retrieved_contexts"]
            )
        )
        rate = hallucination_count / len(self.results)
        assert rate <= THRESHOLDS["hallucination_rate"], \
            f"Hallucination rate {rate:.3f} > {THRESHOLDS['hallucination_rate']}"

    def test_latency_within_bounds(self):
        """P95 latency under 3 seconds."""
        latencies = [r["result"]["latency_ms"] for r in self.results]
        latencies.sort()
        p95 = latencies[int(len(latencies) * 0.95)]
        assert p95 <= THRESHOLDS["latency_p95_ms"], \
            f"P95 latency {p95}ms > {THRESHOLDS['latency_p95_ms']}ms"

    def test_no_regression_by_category(self):
        """No regression per question category."""
        category_scores = {}
        for r in self.results:
            cat = r["example"].category
            score = compute_answer_quality(
                r["result"]["answer"],
                r["example"].expected_answer
            )
            category_scores.setdefault(cat, []).append(score)

        baseline = load_baseline_scores()
        for cat, scores in category_scores.items():
            avg = sum(scores) / len(scores)
            if cat in baseline:
                assert avg >= baseline[cat] - 0.05, \
                    f"Regression in {cat}: {avg:.3f} vs baseline {baseline[cat]:.3f}"

Step 3: CI/CD integration with GitHub Actions

GitHub Actions workflow

DEVELOPERyaml
# .github/workflows/rag-evaluation.yml
name: RAG Evaluation Pipeline

on:
  push:
    paths:
      - 'backend/rag/**'
      - 'backend/prompts/**'
      - 'backend/config/**'
  pull_request:
    branches: [main, develop]
  schedule:
    - cron: '0 6 * * 1'  # Every Monday at 6am

jobs:
  evaluate:
    runs-on: ubuntu-latest
    timeout-minutes: 30

    steps:
      - uses: actions/checkout@v4

      - name: Setup Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: |
          pip install ragas deepeval pytest langfuse
          pip install -r backend/requirements.txt

      - name: Run RAG evaluation
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          QDRANT_URL: ${{ secrets.QDRANT_URL }}
          LANGFUSE_SECRET_KEY: ${{ secrets.LANGFUSE_SECRET_KEY }}
        run: |
          pytest tests/test_rag_quality.py \
            --tb=long \
            --json-report \
            --json-report-file=eval_results.json \
            -v

      - name: Check thresholds
        run: |
          python scripts/check_eval_thresholds.py \
            --results eval_results.json \
            --baseline tests/baseline_scores.json

      - name: Upload results to Langfuse
        if: always()
        run: |
          python scripts/upload_eval_to_langfuse.py \
            --results eval_results.json

      - name: Comment PR with results
        if: github.event_name == 'pull_request'
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const results = JSON.parse(
              fs.readFileSync('eval_results.json', 'utf8')
            );
            const body = formatEvalResults(results);
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: body
            });

      - name: Alert on regression
        if: failure()
        run: |
          python scripts/send_regression_alert.py \
            --results eval_results.json \
            --channel slack \
            --webhook ${{ secrets.SLACK_WEBHOOK }}

Step 4: Dashboard and monitoring

Evaluation tools comparison

Langfuse integration

DEVELOPERpython
from langfuse import Langfuse

langfuse = Langfuse(
    secret_key="sk-lf-...",
    public_key="pk-lf-...",
    host="https://cloud.langfuse.com"
)

def log_evaluation_to_langfuse(
    results: dict,
    run_id: str,
    metadata: dict = None
):
    """Log evaluation results to Langfuse."""
    for example_result in results["examples"]:
        trace = langfuse.trace(
            name="rag-evaluation",
            metadata={
                "run_id": run_id,
                "category": example_result["category"],
                **(metadata or {})
            }
        )

        trace.span(
            name="retrieval",
            input=example_result["question"],
            output=example_result["retrieved_contexts"],
            metadata={
                "num_contexts": len(example_result["retrieved_contexts"]),
                "context_recall": example_result["context_recall"],
            }
        )

        generation = trace.generation(
            name="generation",
            input=example_result["question"],
            output=example_result["answer"],
            metadata={
                "faithfulness": example_result["faithfulness"],
                "relevancy": example_result["relevancy"],
            }
        )

        trace.score(name="faithfulness", value=example_result["faithfulness"])
        trace.score(name="answer_relevancy", value=example_result["relevancy"])
        trace.score(name="context_recall", value=example_result["context_recall"])

    langfuse.flush()

Dashboard metrics to track

Step 5: Advanced regression detection

Statistical detection

DEVELOPERpython
import numpy as np
from scipy import stats

def detect_regression(
    current_scores: list,
    baseline_scores: list,
    alpha: float = 0.05,
    min_delta: float = 0.03
) -> dict:
    """Detect statistically significant regressions."""
    current = np.array(current_scores)
    baseline = np.array(baseline_scores)

    # Welch's t-test (for unequal sample sizes)
    t_stat, p_value = stats.ttest_ind(
        baseline, current,
        equal_var=False,
        alternative='greater'  # baseline > current = regression
    )

    delta = np.mean(baseline) - np.mean(current)

    is_regression = (
        p_value < alpha and
        delta > min_delta
    )

    return {
        "is_regression": is_regression,
        "p_value": p_value,
        "delta": delta,
        "current_mean": np.mean(current),
        "baseline_mean": np.mean(baseline),
        "confidence": 1 - p_value,
        "effect_size": delta / np.std(baseline)  # Cohen's d
    }

Smart alerting

DEVELOPERpython
def evaluate_and_alert(rag_pipeline, golden_dataset, config):
    """Evaluate RAG and send alerts if needed."""
    results = evaluate_rag_with_ragas(golden_dataset, rag_pipeline)
    baseline = load_baseline()

    alerts = []

    for metric, value in results.items():
        threshold = config["thresholds"].get(metric)
        previous = baseline.get(metric)

        # Critical alert: below absolute threshold
        if threshold and value < threshold:
            alerts.append({
                "level": "critical",
                "metric": metric,
                "value": value,
                "threshold": threshold,
                "message": (
                    f"{metric} dropped to {value:.3f}, "
                    f"below critical threshold {threshold:.3f}"
                )
            })

        # Warning alert: regression from baseline
        elif previous and value < previous - 0.03:
            alerts.append({
                "level": "warning",
                "metric": metric,
                "value": value,
                "previous": previous,
                "message": (
                    f"{metric} decreased from {previous:.3f} to "
                    f"{value:.3f} (delta: {value - previous:+.3f})"
                )
            })

    if alerts:
        send_alerts(alerts, config)

    if not any(a["level"] == "critical" for a in alerts):
        update_baseline(results)

    return results, alerts

Best practices

Golden dataset management

1. Version the golden dataset in git (like code)
2. Update when documents change significantly
3. Stratify by category, difficulty, and source
4. Humanly validate a sample regularly
5. Never optimize for the golden dataset (overfitting)

Robust pipeline

Evaluation frequency

FAQ

How many examples in the golden dataset?

Minimum 30 for statistically significant results. Ideal is 100-200 examples stratified by question category. Beyond 500, evaluation costs increase without proportional confidence gain. Start with 50 and increase progressively. See our guide on RAG evaluation metrics for metric details.

RAGAS or DeepEval, which one to choose?

RAGAS is more mature, better documented, and the academic reference. DeepEval offers more metrics (14+), native CI/CD integration, and a dashboard. For a simple project, RAGAS is enough. For a production pipeline with monitoring, DeepEval + Langfuse is more complete. Both are open-source and compatible with pytest.

How to handle questions without a good answer?

Include in your golden dataset questions the RAG should not answer (out of scope, missing information). Add a "refusal accuracy" metric: the RAG should refuse to answer or say "I don't know" rather than hallucinate. This is often the hardest metric to maintain.

Is the evaluation pipeline expensive?

For 100 examples with RAGAS (which uses GPT-4o-mini by default for evaluation): from a few cents to a couple of dollars per run depending on the judge model. With DeepEval and a local model, it's nearly free. The main cost is run time (~5-15 minutes for 100 examples). This is negligible compared to the cost of a production regression. See our guide on RAG cost optimization.

How to test prompt changes?

Use A/B evaluation: run the golden dataset with the old and new prompt, then compare metrics. The new prompt must be statistically better (not just marginal) to justify the change. Integrate this comparison into your PR review. See our guide on RAG prompt engineering.

---

An automated RAG evaluation pipeline is your safety net against silent regressions. Investing a few hours in setup will save you days of debugging and protect your users' trust. Try Ailog to benefit from integrated quality monitoring on your RAG chatbot.