Build a Self-Improving RAG System With Automatic Evaluation and Feedback Loops

Build a Self-Improving RAG System With Automatic Evaluation and Feedback Loops

1. Clarifying Requirements

1.1 Functional Requirements

• The system serves RAG queries — users submit natural language questions and receive answers grounded in a document corpus with inline citations, exactly as in a standard RAG system

• The system continuously evaluates its own answer quality — after every response, an automated evaluation pipeline scores the answer across multiple dimensions: faithfulness (is every claim grounded?), answer relevance (does the answer address the question?), context precision (were the retrieved chunks actually useful?), and context recall (were relevant chunks missed?)

• The system identifies failure modes automatically — it classifies each evaluation result into a failure category: retrieval failure (right documents exist but weren't retrieved), chunking failure (retrieved documents have the relevant passage but it's split across chunk boundaries), embedding failure (the query vector doesn't find semantically similar chunks), ranking failure (relevant chunks were retrieved but ranked below the top-K cutoff), generation failure (retrieved context was correct but the LLM synthesised a wrong answer), and knowledge gap (the correct answer simply doesn't exist in the corpus)

• The system closes feedback loops automatically — based on identified failure modes, it triggers targeted remediation: re-chunking affected documents, re-embedding with improved queries, adding to a golden query cache, fine-tuning retrieval parameters, or flagging for corpus expansion

• The system maintains a golden evaluation set — a curated collection of (question, expected_answer, required_source_chunks) triples; this set grows automatically from high-confidence evaluated queries and from human-verified corrections; nightly regression tests run the full system against the golden set and alert on quality regressions

• The system performs A/B testing of pipeline variants — retrieval strategies (BM25, dense, hybrid), chunk sizes, embedding models, reranker configurations, and LLM prompts can be tested as variants; traffic is split between control and treatment; quality metrics determine automatic promotion or rollback

• The system exposes a feedback UI — users can rate answers (thumbs up/down), highlight incorrect passages, and suggest corrections; this explicit feedback is weighted more heavily than automated evaluation signals

• Admins can inspect the improvement history — a timeline of every automated change made to the pipeline (which parameter changed, what quality metric triggered it, what the before/after quality delta was) with full rollback capability

1.2 Non-Functional Requirements

• Query latency: p99 < 3s for standard queries; evaluation runs async and never blocks the user response

• Evaluation throughput: every query must be evaluated within 60 seconds of the answer being delivered — the feedback loop must close quickly enough to improve retrieval for subsequent similar queries

• Improvement cycle latency: a detected retrieval failure must result in a measurable quality improvement for similar queries within 24 hours — the system cannot wait for a weekly manual review cycle

• Regression safety: no automated change may be promoted if it degrades any golden set metric by more than 2% — the system must be incapable of silently degrading its own quality through miscalibrated feedback loops

• Evaluation accuracy: the automated evaluator must agree with human judges on answer quality at > 85% agreement rate — below this threshold, the evaluator itself needs recalibration

• A/B test statistical rigour: no pipeline variant is promoted without reaching p < 0.05 statistical significance on quality metrics with a minimum sample size of 500 queries — prevents premature promotion of variants that got lucky on a small sample

• Auditability: every automated decision (parameter change, variant promotion, chunk re-indexing) must be logged with the evidence that triggered it, the expected improvement, and the actual measured improvement post-change

• Graceful degradation: if the evaluation pipeline fails, queries continue to be served from the last known-good pipeline configuration — the evaluation system is not on the query critical path

2. Back-of-the-Envelope Estimation

Query and evaluation volume:

This is the system's central cost constraint: evaluating every query with an LLM-as-judge is economically infeasible at scale. The solution is a tiered evaluation strategy: lightweight heuristic evaluation for all queries (< 1ms), LLM-as-judge for a 1% sample (~50 QPS), and human evaluation for a 0.1% sample (5 QPS). The heuristic evaluator is calibrated against the LLM judge to maintain > 85% agreement.

Golden set and A/B test scale:

Improvement cycle compute:

3. Core APIs

Query API (user-facing, unchanged from standard RAG)

POST /v1/query
Body: { question: string, session_id: string, top_k: 5 }
Response: {
  answer_id: "ans_001",
  answer: "string with inline [1][2] citations",
  citations: [{ chunk_id, doc_id, title, excerpt, score }],
  confidence: 0.91,
  latency_ms: 1240
}

Feedback API (user-facing)

POST /v1/answers/{answer_id}/feedback
Body: {
  rating: "positive" | "negative",
  issues: ["wrong_answer", "missing_context", "wrong_source", "incomplete"],
  corrected_answer: "string",        // optional user correction
  relevant_doc_ids: ["doc_001"]      // optional: docs the user knows are relevant
}
Response: { feedback_id, improvement_queued: true }

Evaluation result API (internal)

GET /v1/evaluations/{answer_id}
Response: {
  answer_id,
  evaluation_tier: "heuristic" | "llm_judge" | "human",
  scores: {
    faithfulness: 0.94,              // every claim grounded in retrieved context?
    answer_relevance: 0.87,          // does the answer address the question?
    context_precision: 0.72,         // were retrieved chunks actually useful?
    context_recall: 0.65             // were relevant chunks missed?
  },
  failure_mode: "retrieval_failure" | "chunking_failure" | "embedding_failure" |
                "ranking_failure" | "generation_failure" | "knowledge_gap" | null,
  failure_evidence: {
    missed_chunk_ids: ["chunk_abc"],  // chunks that existed and were relevant but not retrieved
    false_positive_chunk_ids: [],     // chunks retrieved but not used in answer
    answer_groundedness_per_claim: [{ claim: "...", grounded: true, nli_score: 0.96 }]
  },
  improvement_actions: [
    { action: "re_embed_query", priority: "high" },
    { action: "add_to_golden_set", priority: "medium" }
  ]
}

A/B test management API (admin)

POST /v1/experiments
Body: {
  name: "hybrid_retrieval_vs_dense_only",
  control: { retrieval_strategy: "dense", top_k: 5 },
  treatment: { retrieval_strategy: "hybrid_rrf", top_k: 7 },
  traffic_split: 0.05,
  min_sample_size: 500,
  primary_metric: "context_recall",
  guardrail_metrics: ["faithfulness", "latency_p99"]
}
Response: { experiment_id, status: "running", expected_significance_date: "..." }

GET /v1/experiments/{id}/results
Response: {
  experiment_id, status: "running" | "significant" | "no_effect" | "guardrail_breached",
  control_metrics: { context_recall: 0.65, faithfulness: 0.94, latency_p99: 2.1 },
  treatment_metrics: { context_recall: 0.73, faithfulness: 0.93, latency_p99: 2.4 },
  p_value: 0.031,
  sample_sizes: { control: 9450, treatment: 498 },
  recommendation: "promote_treatment" | "reject" | "continue_testing",
  auto_promote_at: "2024-06-03T00:00:00Z"    // null if guardrail_breached
}

Pipeline configuration API (admin)

GET /v1/pipeline/current
Response: {
  version: "v23",
  config: { chunk_size: 512, chunk_overlap: 128, top_k: 5, retrieval_strategy: "hybrid_rrf",
            embedding_model: "text-embedding-3-small", reranker: "bge-reranker-v2",
            prompt_version: "v7" },
  promoted_at, promoted_by: "auto_experiment_e12" | "admin_user_id",
  quality_at_promotion: { faithfulness: 0.94, context_recall: 0.72 }
}

POST /v1/pipeline/rollback
Body: { target_version: "v22", reason: "quality_regression_on_golden_set" }

4. High-Level Design

4.1 Requirement 1: Tiered Evaluation — Scoring Every Query Without Bankrupting on LLM Calls

Components needed:

Every query must be evaluated, but LLM-as-judge at 5,000 QPS costs ~$4M/day. The tiered evaluation architecture makes comprehensive quality measurement economically viable.

Tier 1 — Heuristic evaluator (all queries, < 1ms): runs synchronously in the query response path, adding negligible latency. Computes five lightweight signals:

Citation coverage: fraction of answer sentences that contain a [N] citation. Low citation coverage (< 0.7) predicts low faithfulness. Computed by regex in < 0.1ms.

BM25 query-answer overlap: the BM25 score between the user's query and the generated answer. High overlap (answer directly addresses the question) predicts high answer relevance. Computed by sparse vector dot product in < 0.5ms.

Retrieval score gap: the difference between the top-1 and top-5 retrieval scores. A large gap (top-1 score 0.92, top-5 score 0.43) suggests the retrieval is polarised — either one chunk is extremely relevant and the others are noise, or the retrieval is uncertain. Computed from already-available retrieval scores in < 0.1ms.

Answer length ratio: ratio of answer token count to context token count. An answer significantly longer than its context is a hallucination signal (the model is generating content not in the context). Ratio > 1.5 is flagged. Computed by token counter in < 0.1ms.

NLI spot check: runs NLI (DeBERTa) on a single claim sampled from the answer — the most suspicious claim (the one with the lowest BM25 overlap with any retrieved chunk). This single NLI call takes ~20ms and provides a grounding signal without running full NLI on every sentence. The single-claim NLI score is the strongest individual predictor of full faithfulness score (Pearson r = 0.78 in calibration experiments).

The five heuristic signals are combined by a lightweight linear regression model (10 parameters, trained against the LLM judge's scores) into a heuristic_quality_score ∈ [0, 1]. This score is the primary signal for: routing to Tier 2 evaluation, triggering improvement actions, and feeding the feedback loop. Calibration target: > 85% agreement with LLM judge on high/medium/low quality classification.

Tier 2 — LLM-as-judge (1% sample + low heuristic score queries, ~50–200 QPS): triggered for: (a) 1% random sample of all queries (for calibration and unbiased quality measurement); (b) any query with heuristic_quality_score < 0.6 (the bottom 15% by heuristic assessment — highest-priority for improvement); (c) any query with user negative feedback; (d) any query from the golden set (every golden set query is evaluated by the LLM judge). The LLM judge runs a RAGAS-style evaluation: faithfulness (is every claim supported by the retrieved context?), answer relevance (does the answer address the question without irrelevant content?), context precision (what fraction of retrieved chunks contributed to the answer?), context recall (estimated — are there obvious relevant topics in the question not covered by retrieved chunks?). Total tokens: ~2,000 per evaluation. At 200 QPS × 2,000 tokens = 400K tokens/sec → 133 A100s. Economically viable.

Tier 3 — Human evaluation (0.1% sample + escalations, ~5 QPS): triggered for: (a) 0.1% random sample (human-in-the-loop quality anchor); (b) any query where Tier 2 LLM judge score strongly disagrees with Tier 1 heuristic score (disagreement > 0.3 on [0,1] scale — indicates the heuristic model needs recalibration); (c) any query with detailed user feedback (written correction or highlighted passage). Human evaluators see: the question, the retrieved chunks, the generated answer, and the Tier 1 and 2 scores. They provide: corrected scores, failure mode classification, and optionally a corrected answer. Human evaluation results feed back into heuristic model recalibration (if Tier 1 is drifting from Tier 3) and into the golden set (high-confidence human evaluations become golden set entries).

4.2 Requirement 2: Failure Mode Classification and Root Cause Analysis

Components needed:

Knowing that an answer scored 0.65 on context recall is not actionable. Knowing that context recall is low because chunk 482 (which contains the relevant passage) scored 0.41 on retrieval and fell below the top-5 cutoff, while a less relevant chunk 219 scored 0.68 and took its slot — that is actionable. The failure classifier converts evaluation scores into specific, targeted diagnoses.

Oracle retriever — the key component that makes root cause analysis possible. For every evaluated query, the oracle retriever asks: "Given this query and this ideal answer (from the LLM judge's evaluation), which chunks in the corpus should have been retrieved?" It runs a separate retrieval pass with: (1) the original query, (2) the LLM judge's correct answer text (embedded and used as an additional query vector), and (3) the user's question reformulated by an LLM into 3 alternative phrasings (for robustness). The union of top-K results across all query variants is the "oracle retrieved set" — the best possible retrieval given the current corpus and index. Comparing the oracle retrieved set to the actual retrieved set identifies:

Ranking failure: oracle chunks exist in the actual retrieval results (they were retrieved) but ranked below the top-K cutoff. Fix: tune top_k, adjust reranker parameters, or re-embed the query with a better query expansion.

Embedding failure: oracle chunks are not in the actual retrieval results at all (not found by vector ANN), but they exist in the corpus and their text clearly contains the answer. The embedding model produced a query vector too far from the relevant chunk vectors. Fix: re-embed with query expansion (HyDE), try a different embedding model for this domain.

Chunking failure: oracle retrieval finds a chunk adjacent to the relevant passage but not the passage itself — because the relevant passage was split across chunk boundaries. Fix: re-chunk the affected document with different boundaries, or increase chunk overlap.

Knowledge gap: oracle retrieval finds nothing — the correct answer genuinely does not exist in the corpus. No chunk, even using the ideal answer as a query, returns a relevant result. Fix: flag for corpus expansion; the knowledge gap is logged for content team review.

Generation failure: oracle chunks are retrieved (both actual and oracle retrieval agree on the relevant chunks), but the LLM did not use them correctly — the faithfulness score is low despite correct retrieval. Fix: improve the prompt, add stronger citation enforcement, or switch to a more instruction-following LLM.

Failure classifier — a decision tree (not an LLM — fast, deterministic, auditable) that takes the comparison between actual and oracle retrieval plus the evaluation scores and outputs the failure mode with evidence:

if oracle_chunks ∩ actual_top_k == ∅:
    if oracle_chunks ∩ actual_top_100 != ∅:
        return RANKING_FAILURE, evidence={missed_chunks, their_ranks}
    elif oracle_chunks exist in corpus:
        if relevant_passage_split_across_chunks(oracle_chunks):
            return CHUNKING_FAILURE, evidence={affected_doc, split_boundary}
        else:
            return EMBEDDING_FAILURE, evidence={query_embedding_distance_to_oracle_chunks}
    else:
        return KNOWLEDGE_GAP, evidence={best_available_chunk, best_score}
elif faithfulness < 0.7 and context_precision > 0.8:
    return GENERATION_FAILURE, evidence={unfaithful_claims, retrieved_context}
else:
    return RETRIEVAL_SUCCESS  # answer may still be low quality but retrieval is working

4.3 Requirement 3: Automated Remediation — Closing the Feedback Loop

Components needed:

Failure classification without automated remediation is just expensive monitoring. The remediation engine takes classified failures and triggers targeted, measurable improvements without human intervention.

Remediation action library — six types of automated remediation, each triggered by a specific failure mode:

Query expansion cache (for Embedding Failures): the query that failed is stored in the Query Expansion Cache alongside the oracle chunk IDs. When the same or similar query arrives (cosine similarity > 0.90 with a cached failed query), the system pre-loads the oracle chunks into the retrieval candidate pool before running ANN — guaranteeing they appear in the top-K regardless of embedding quality. This is a targeted short-term fix (operational within seconds of detection) while the embedding model improvement is the long-term fix.

Document re-chunking (for Chunking Failures): the affected document is re-chunked with adjusted parameters: if the failure shows a passage split across two 512-token chunks, the document is re-chunked with 800-token chunks and 200-token overlap for that document specifically. Per-document chunk configuration is stored in the Document Registry. The re-chunked document goes through the full embedding and indexing pipeline, replacing the old chunks. Per-document re-chunking means a failure in one document does not require re-chunking the entire corpus.

Synthetic query generation (for Knowledge Gap failures, corpus expansion): when a knowledge gap is detected, the system generates synthetic queries that would be answered by the missing knowledge. These synthetic queries are published to the Knowledge Gap Log with a priority score (how often similar queries appear). Content teams review the Knowledge Gap Log weekly — the most frequently asked unanswerable questions drive corpus expansion decisions.

Reranker score calibration (for Ranking Failures): when a ranking failure is detected, the (query, oracle_chunk, actual_rank, score) tuple is added to a Reranker Training Buffer. When the buffer reaches 1,000 examples, a reranker fine-tuning job is triggered — the reranker is trained to assign higher scores to oracle chunks for similar queries. Fine-tuned reranker candidates go through A/B testing before promotion.

Prompt mutation (for Generation Failures): the prompt version that produced the generation failure is flagged. The Prompt Optimiser (DSPy-style automatic prompt optimisation) mutates the prompt — adjusts citation instruction phrasing, adds explicit negative examples, tightens the "only use provided context" instruction — and evaluates the mutated prompt against the golden set. The mutated prompt variant is launched as an A/B test if it improves golden set faithfulness by > 2%.

Heuristic recalibration (when Tier 1 disagrees with Tier 3): when the heuristic evaluator's score disagrees with human evaluation by > 0.3, the (heuristic_features, human_score) pair is added to a recalibration buffer. When the buffer reaches 200 examples, the linear regression model is retrained. New model coefficients are validated against a held-out calibration set before deployment — the recalibrated heuristic model replaces the old one atomically.

Remediation priority queue — not all failures are equally important. The Remediation Scheduler prioritises actions based on: (1) impact — how many future queries would benefit from this fix? A retrieval failure for a query asked 500 times/day has 500× the impact of a unique query; (2) confidence — how certain is the failure classification? Low-confidence classifications get a lower priority; (3) cost — re-chunking a 500-page document costs more than adding a query expansion cache entry; (4) urgency — user-reported failures get priority boost. The scheduler runs remediation actions concurrently (up to 20 parallel remediation jobs) and logs every action to the Improvement Audit Log.

4.4 Requirement 4: A/B Testing Framework — Promoting Improvements Safely

Components needed:

Every automated improvement that affects the retrieval pipeline (new reranker, new chunk size, new embedding model, new prompt) must be validated through A/B testing before becoming the default. Unsafe promotion of an improvement that helps one query category but hurts another is the system's greatest risk.

Experiment registry — stores all active experiments with their configurations, traffic splits, and required sample sizes. The query router checks the registry on every request: if the query is assigned to an experiment's treatment group (by consistent hashing on session_id), the treatment pipeline configuration is used instead of the control. Consistent hashing on session_id ensures that a user always sees the same variant across multiple queries in the same session — preventing the jarring experience of receiving better and worse answers in alternating turns.

Traffic assignment — treatment traffic is set at 5% by default. 5% of 5,000 QPS = 250 treatment queries/sec. At minimum sample size of 500 queries, significance can be reached in 2 seconds of traffic — but we require at least 24 hours of traffic to account for time-of-day variation (morning queries may be qualitatively different from evening queries). The minimum 24-hour window prevents spurious significance from uneven traffic distributions.

Metric collection — for each query in an experiment, both the heuristic evaluation score (immediate, synchronous) and the LLM judge score (async, within 60 seconds) are tagged with the experiment_id and variant (control or treatment). Metric aggregation runs every 5 minutes: compute mean and standard deviation of each quality metric per variant, run a Welch's t-test for the primary metric, and check guardrail metrics.

Guardrail metrics — every experiment defines guardrail metrics that must not regress. If any guardrail metric degrades by > 2% in the treatment group, the experiment is immediately halted — treatment traffic is redirected to control — regardless of primary metric performance. This prevents the system from promoting a variant that improves context recall but breaks faithfulness. Guardrail checks run after every 5-minute metric aggregation, not just at experiment conclusion.

Automatic promotion — when an experiment reaches statistical significance (p < 0.05, sample size ≥ 500) with positive effect on the primary metric and no guardrail breaches, the Experiment Promoter schedules automatic promotion for the next daily maintenance window (02:00 UTC). The promotion: updates the pipeline configuration in etcd, writes an entry to the Improvement Audit Log, runs a smoke test against the golden set (aborts promotion if smoke test fails), and sends a notification to admins. The previous configuration is retained as pipeline_version - 1 for instant rollback.

Rollback mechanism — any promoted change can be rolled back instantly: POST /v1/pipeline/rollback restores the previous pipeline configuration from etcd. Rollback is also triggered automatically: if the nightly golden set regression test shows > 2% degradation on any metric compared to the baseline (the metrics at the most recent manual approval checkpoint), the system automatically rolls back to the last manually approved configuration and pages the on-call admin.

4.5 Requirement 5: The Golden Set — The Ground Truth for Quality

Components needed:

The golden set is the immune system of the self-improving RAG system. It is the unchanging benchmark against which every proposed improvement is validated. Without a growing, high-quality golden set, the feedback loop has no reliable north star.

Golden set composition — three sources:

Human-created entries: domain experts write (question, expected_answer, required_chunk_ids) triples directly. These are the highest-quality entries — they capture questions that humans know are important and answers that humans know are correct. Maintained in a Google Docs-like editing interface where domain experts can add, edit, and review entries.

Automatically promoted entries: queries that received the highest evaluation scores (heuristic > 0.92, LLM judge > 0.90, at least 3 positive user ratings, no negative feedback) are automatically nominated for golden set inclusion. A background job runs nightly, sampling the top-1% highest-quality evaluated queries and generating a (question, current_answer, retrieved_chunks) triple. These nominations are shown to a human reviewer who approves or rejects them (the human is shown the question and answer, not the evaluation scores — to avoid anchoring on machine-generated quality scores). Approved nominations enter the golden set.

Adversarial entries: deliberately challenging queries — questions where the answer requires combining information from multiple documents, questions phrased in a way that produces common embedding failures, questions where the obvious answer is wrong and requires careful reading of the source — are generated by an adversarial query generator (an LLM prompted to generate "questions that would fool a RAG system"). These entries test the system's robustness and prevent the golden set from being dominated by easy, high-recall queries that give falsely optimistic quality signals.

Golden set versioning — the golden set is version-controlled (Git-like, in PostgreSQL). Every addition, modification, and deletion is tracked. This enables: running the regression test against any past version of the golden set (to detect if quality improvements are genuine or if the golden set has drifted toward easier questions); diffing golden set versions to explain why regression scores changed; and rolling back accidental golden set corruptions.

Nightly regression test — runs every night at 00:00 UTC. Process: (1) fetch the current golden set; (2) run every golden set query through the current pipeline; (3) evaluate every answer with the Tier 2 LLM judge (not the heuristic — this is the authoritative regression result); (4) compare metrics to the baseline (the metrics at the last human-approved checkpoint); (5) if any metric degrades > 2%, trigger automatic rollback and page the on-call admin; (6) publish the regression report to the admin dashboard. Duration: ~28 minutes at 60-way parallelism for a 50K golden set.

4.6 Putting It All Together

The system operates as two interleaved loops — a fast loop and a slow loop. The fast loop (query → heuristic evaluation → failure classification → query expansion cache / escalation to Tier 2) closes within seconds. The slow loop (Tier 2 LLM evaluation → oracle retrieval → failure mode analysis → remediation action → A/B test → automatic promotion → nightly regression validation) closes within 24 hours. Both loops feed the same Improvement Audit Log and are governed by the same golden set. The fast loop handles the majority of traffic quickly; the slow loop handles the high-stakes decisions carefully.

<br>

Full architecture diagram:

5. Database Design

5.1 SQL vs NoSQL

PostgreSQL (primary operational store — evaluation results, experiments, golden set, audit log) — all four are highly relational and require ACID guarantees. Evaluation results join to queries join to experiments join to golden set entries — complex queries that span these tables are common (e.g. "show me all queries in experiment E1 that scored low on context recall and were classified as ranking failures"). PostgreSQL's JSONB column handles variable-schema payloads (failure_evidence, evaluation_scores, experiment_config) without sacrificing indexability. The improvement_audit_log table is INSERT-only — the application service account has no UPDATE or DELETE — providing tamper protection for the audit trail. The golden_set_entries table uses PostgreSQL's row-level locking to prevent concurrent edits to the same golden set entry from producing inconsistent results.

Redis (fast path — query expansion cache + experiment routing + heuristic score cache) — three critical sub-millisecond lookups. The query expansion cache: HGET query_expansion:{sha256(query_embedding_bucket)} oracle_chunk_ids — a hash lookup that pre-loads oracle chunks for known-bad queries. The experiment routing cache: GET experiment_assignment:{session_id} — consistent hashing result cached so re-hashing is not needed on every query. The heuristic score cache: SET heuristic_score:{answer_id} {score} EX 3600 — the Tier 1 score is written immediately and read by the Tier 2 evaluation trigger (which checks if Tier 2 is warranted). All three require sub-millisecond latency — Redis Cluster is the only viable choice.

Qdrant (vector stores — corpus ANN index + oracle retrieval index) — two separate Qdrant collections. The corpus collection serves live queries (standard ANN retrieval). The oracle collection is a separate, evaluation-only collection that stores the same vectors but with additional payload fields: oracle_score (how often this chunk was in the oracle set for evaluated queries) — used by the oracle retriever to bias toward historically useful chunks. The oracle collection is never queried at serving time — only by the evaluation pipeline. Separating them ensures oracle retrieval infrastructure never affects live query latency.

Elasticsearch (BM25 index — corpus keyword search + query-answer overlap scoring) — two uses. BM25 keyword retrieval (the BM25 component of hybrid RRF retrieval). The query-answer BM25 overlap scoring in Tier 1 evaluation — the BM25 score between query text and answer text is computed by querying the ES _explain API on a virtual document (the answer text). This reuses the already-deployed ES cluster rather than requiring a separate BM25 implementation.

Apache Kafka (evaluation queue + remediation job queue) — two topics. The evaluation-queue topic receives (answer_id, query, answer, retrieved_chunks, experiment_id) messages immediately after each query is served — it is the buffer between the synchronous query path and the async evaluation pipeline. The remediation-queue topic receives classified failures and triggers the remediation engine. Kafka's consumer group model allows multiple evaluation workers to consume the evaluation queue in parallel. Kafka's retention (7 days) means evaluation results for queries served on a Sunday are still available for the nightly Monday regression test even if the evaluation pipeline was briefly unavailable.

S3 (golden set archive + evaluation log bulk storage + model checkpoints) — golden set versions are stored as Parquet snapshots (golden_set/v{version}/{date}.parquet) alongside the PostgreSQL version-controlled entries. The Parquet format enables Spark-based bulk operations: running the nightly regression test as a parallel Spark job that reads the golden set from S3 and distributes queries across a Spark cluster. Evaluation logs older than the PostgreSQL retention window (30 days) are archived to S3 Parquet for long-term analysis. Fine-tuned model checkpoints (reranker fine-tunes, embedding model fine-tunes) are stored in S3 with version tags matching the experiment ID that produced them — enabling instant rollback by loading a previous checkpoint.

5.2 Database Schema

-- PostgreSQL

queries (
  query_id          UUID PRIMARY KEY,
  session_id        UUID NOT NULL,
  user_id           UUID,
  question_text     TEXT NOT NULL,
  question_hash     TEXT NOT NULL,        -- SHA-256; for dedup and grouping
  answer_id         UUID,
  experiment_id     UUID,                 -- null if not in an experiment
  variant           TEXT,                 -- 'control' | 'treatment' | null
  pipeline_version  TEXT NOT NULL,        -- which pipeline config served this query
  retrieved_chunk_ids UUID[],
  retrieval_scores  FLOAT[],
  latency_ms        INT,
  created_at        TIMESTAMP DEFAULT now()
)
CREATE INDEX idx_queries_experiment ON queries(experiment_id, variant, created_at);
CREATE INDEX idx_queries_question_hash ON queries(question_hash);  -- for dedup grouping

answers (
  answer_id         UUID PRIMARY KEY,
  query_id          UUID REFERENCES queries(query_id),
  answer_text       TEXT NOT NULL,
  answer_hash       TEXT NOT NULL,
  citation_count    INT,
  token_count       INT,
  created_at        TIMESTAMP DEFAULT now()
)

evaluations (
  eval_id           UUID PRIMARY KEY,
  answer_id         UUID REFERENCES answers(answer_id),
  evaluation_tier   TEXT NOT NULL,        -- 'heuristic'|'llm_judge'|'human'
  faithfulness      FLOAT,
  answer_relevance  FLOAT,
  context_precision FLOAT,
  context_recall    FLOAT,
  heuristic_score   FLOAT,               -- Tier 1 composite score
  failure_mode      TEXT,               -- 'ranking_failure'|'embedding_failure'|...
  failure_evidence  JSONB,               -- { missed_chunk_ids, oracle_chunks, ... }
  oracle_chunk_ids  UUID[],
  improvement_actions JSONB,             -- [{ action, priority }]
  evaluator_model   TEXT,               -- which LLM was the judge (for Tier 2)
  evaluated_at      TIMESTAMP DEFAULT now()
)
CREATE INDEX idx_eval_failure ON evaluations(failure_mode, evaluated_at);
CREATE INDEX idx_eval_tier_score ON evaluations(evaluation_tier, heuristic_score);

user_feedback (
  feedback_id       UUID PRIMARY KEY,
  answer_id         UUID REFERENCES answers(answer_id),
  user_id           UUID,
  rating            TEXT NOT NULL,        -- 'positive'|'negative'
  issues            TEXT[],
  corrected_answer  TEXT,
  relevant_doc_ids  TEXT[],
  weight            FLOAT DEFAULT 1.0,   -- increased for detailed feedback
  created_at        TIMESTAMP DEFAULT now()
)

golden_set_entries (
  entry_id          UUID PRIMARY KEY,
  version           INT NOT NULL DEFAULT 1,
  question          TEXT NOT NULL,
  expected_answer   TEXT NOT NULL,
  required_chunk_ids UUID[],
  source            TEXT NOT NULL,        -- 'human'|'auto_promoted'|'adversarial'
  difficulty        TEXT DEFAULT 'standard', -- 'easy'|'standard'|'hard'|'adversarial'
  domain_tags       TEXT[],
  created_by        TEXT NOT NULL,
  approved_at       TIMESTAMP,
  deprecated_at     TIMESTAMP,           -- null = active
  created_at        TIMESTAMP DEFAULT now()
)
-- Version-controlled via insert+deprecate pattern (never UPDATE existing entries)

regression_runs (
  run_id            UUID PRIMARY KEY,
  golden_set_version INT NOT NULL,
  pipeline_version  TEXT NOT NULL,
  started_at        TIMESTAMP NOT NULL,
  completed_at      TIMESTAMP,
  faithfulness_mean FLOAT,
  answer_relevance_mean FLOAT,
  context_precision_mean FLOAT,
  context_recall_mean FLOAT,
  baseline_faithfulness FLOAT,           -- value at last human-approved checkpoint
  regression_detected BOOLEAN DEFAULT false,
  rollback_triggered BOOLEAN DEFAULT false,
  run_status        TEXT DEFAULT 'running'
)

experiments (
  experiment_id     UUID PRIMARY KEY,
  name              TEXT NOT NULL,
  control_config    JSONB NOT NULL,
  treatment_config  JSONB NOT NULL,
  traffic_split     FLOAT NOT NULL DEFAULT 0.05,
  min_sample_size   INT DEFAULT 500,
  primary_metric    TEXT NOT NULL,
  guardrail_metrics TEXT[],
  status            TEXT DEFAULT 'running',
  p_value           FLOAT,
  control_sample    INT DEFAULT 0,
  treatment_sample  INT DEFAULT 0,
  recommendation    TEXT,
  promoted_at       TIMESTAMP,
  created_at        TIMESTAMP DEFAULT now()
)

improvement_audit_log (
  log_id            UUID PRIMARY KEY,
  action_type       TEXT NOT NULL,       -- 're_chunk'|'promote_experiment'|'recalibrate'|...
  triggered_by      TEXT NOT NULL,       -- failure_mode, experiment_id, regression_run_id
  evidence          JSONB NOT NULL,      -- what evaluation signal triggered this
  pipeline_version_before TEXT,
  pipeline_version_after  TEXT,
  expected_delta    JSONB,               -- { metric: expected_change }
  actual_delta      JSONB,               -- populated 24h after action
  created_at        TIMESTAMP DEFAULT now()
-- INSERT-only: no UPDATE or DELETE
)

6. Design Deep Dive

6.1 Calibrating the Heuristic Evaluator — Making 85% Agreement Achievable

The entire tiered evaluation strategy rests on one assumption: the Tier 1 heuristic evaluator agrees with the Tier 3 human evaluator at > 85% rate. If this assumption fails, the feedback loops are driven by a miscalibrated signal — the system optimises for the wrong objective and degrades quality while believing it is improving. Calibration is not a one-time setup; it must be maintained continuously.

The calibration dataset — built from 10,000 queries where all three tiers evaluated the same answer. For each query: the five Tier 1 heuristic features, the Tier 2 LLM judge scores, and the Tier 3 human scores are all available. The heuristic model is trained to predict the Tier 3 human score (not the Tier 2 LLM score — the human is the ground truth). The calibration dataset is stratified across: failure modes (ensure the model sees examples of each failure type), difficulty levels (easy questions score high across all signals; hard questions show more disagreement), and domains (technical, conversational, factual questions have different linguistic patterns).

The agreement metric — agreement is measured as exact match on a 3-way classification: high quality (score ≥ 0.80), medium quality (0.60–0.79), low quality (< 0.60). Continuous score regression has misleadingly high R² even for poor predictors — the 3-way classification better captures whether the heuristic correctly identifies which answers need remediation. The 85% agreement target means the heuristic misclassifies at most 1 in 7 answers — acceptable because miscalibration is detected and corrected by the ongoing Tier 1 vs Tier 3 comparison.

Recalibration trigger — every week, the system computes the rolling 7-day agreement rate between Tier 1 (heuristic) and Tier 3 (human) on the ~350 queries that received human evaluation that week. If agreement drops below 80% (5 percentage points below target), recalibration is triggered immediately. If agreement drops below 75%, the heuristic evaluator is put in "monitor mode" — its scores are recorded but no automated improvements are triggered until recalibration restores agreement above 85%. This prevents the system from making automated changes based on a demonstrably miscalibrated signal.

The recalibration procedure — the linear regression model is retrained on all available (heuristic_features, human_score) pairs (accumulated since the last recalibration). The retrained model is validated on a held-out set (20% of calibration data). If held-out agreement > 87% (2 points above target — requiring the retrained model to be meaningfully better, not just marginally), the retrained model replaces the current one. The model replacement is logged to the Improvement Audit Log — this is itself a pipeline change that must be tracked.

6.2 Oracle Retrieval — The Key to Actionable Failure Classification

Without the oracle retriever, failure classification is limited to "retrieval was bad" — a useless diagnosis. With the oracle retriever, the system knows precisely which chunks were relevant, where they ranked in the actual retrieval, and why they were missed. This specificity is what makes automated remediation possible.

Why ideal-answer-as-query works — the oracle retriever embeds the LLM judge's correct answer (the answer the system should have generated) and uses it as a query vector alongside the original question. This works because: the embedding of a correct, detailed answer is much closer in vector space to the relevant document chunks than the embedding of a brief, ambiguous user question. The user asked "what's the refund policy?" — the answer "Customers may return items within 30 days for a full refund, excluding digital goods" has much higher cosine similarity with the policy document chunk than the question itself. The oracle retriever is essentially running retrieval with perfect knowledge of what the answer should contain — it finds what should have been retrieved.

False positive rejection in the oracle set — not every chunk that the oracle retriever returns is genuinely oracle-relevant. Some high-scoring chunks may be retrieved because they contain common words from the answer, not because they contain the relevant information. The oracle set is filtered by a minimum oracle score threshold: only chunks with cosine similarity > 0.75 to the ideal answer embedding are included. Below this threshold, the retrieval similarity is too low to reliably indicate genuine relevance.

Oracle retrieval latency — oracle retrieval runs async in the evaluation pipeline, not on the query critical path. Its latency budget is 30 seconds (within the 60-second evaluation SLA). Running 4 query variants (original + ideal answer + 3 paraphrases) through Qdrant ANN in parallel takes ~4 × 30ms = 120ms of ANN time plus ~500ms for the paraphrase generation LLM call — total ~700ms. Well within the 30-second budget.

6.3 The A/B Testing Framework — Statistical Rigour at 5,000 QPS

At 5,000 QPS with 5% treatment traffic, treatment gets 250 queries/second. The minimum sample size of 500 queries is reached in 2 seconds. This creates a temptation to promote experiments immediately — but 2 seconds of traffic at 5,000 QPS is not a representative sample. Three constraints prevent premature promotion.

Minimum experiment duration (24 hours) — regardless of sample size, no experiment is evaluated before it has run for at least 24 hours. This captures time-of-day variation (morning queries about yesterday's news are qualitatively different from late-night queries about general knowledge), ensures the sample spans multiple user cohorts (different users at different times of day), and prevents lucky short windows from driving promotion.

Sequential testing with early stopping bounds — instead of checking significance only at the end of the experiment, the system uses a sequential probability ratio test (SPRT) that computes a valid p-value at every metric aggregation point (every 5 minutes). This enables both early stopping for harm (guardrail breach immediately terminates the experiment) and early stopping for success (overwhelming evidence of improvement can promote before the full planned duration). The SPRT correctly controls Type I error rate (false positive promotions) even with repeated testing — a crucial correctness property that standard t-tests do not have with optional stopping.

Practical significance threshold — statistical significance (p < 0.05) alone is not sufficient for promotion. The treatment must also show practical significance: the absolute improvement in the primary metric must exceed a minimum effect size. For context recall, the minimum practical effect size is Δ > 0.02 (2 percentage points). An experiment that reaches p = 0.03 with a 0.1 percentage point improvement is statistically significant but practically meaningless — it is rejected. The combined criterion (p < 0.05 AND |Δ| > minimum_effect_size) prevents the system from making pointless configuration changes for statistically detectable but operationally irrelevant improvements.

6.4 The Golden Set Growth Strategy — Quality vs Quantity

A golden set of 10 poorly chosen queries is worse than no golden set — it gives false confidence in a system that may be failing on real-world queries. A golden set of 50,000 well-chosen queries, however, is a powerful regression and benchmarking tool. The growth strategy must balance quality (every entry must be trustworthy) with coverage (the set must represent the full query distribution).

Coverage measurement — the existing golden set's coverage of the query space is measured weekly by comparing the distribution of golden set question embeddings to the distribution of all query embeddings (via a 2D UMAP projection and coverage metric). Regions of query space with few golden set entries are identified as coverage gaps. The adversarial query generator is directed toward these gaps — it generates questions in the underrepresented semantic regions, ensuring the golden set does not become dominated by the most common, easiest questions.

Difficulty stratification — the golden set is explicitly stratified by difficulty: 30% easy (unambiguous factual questions with single-chunk answers), 50% standard (questions requiring synthesis across 2–3 chunks), 20% hard/adversarial (questions requiring multi-document reasoning, temporal reasoning, or phrasing that commonly confuses the retriever). The difficulty distribution is checked monthly — if the distribution drifts toward too many easy questions (because easy questions get approved more readily), the adversarial generator is activated to restore the target distribution.

Anti-gaming prevention — a self-improving system that can both improve its pipeline and add entries to the golden set has an obvious failure mode: it promotes improvements that improve golden set scores but do not improve real-world quality, if the golden set is not independent of the training signal. Three safeguards prevent this: (1) Human approval is required for golden set entries — no automated process can add entries without a human seeing and approving the question-answer pair; (2) Golden set entries cannot be generated by querying the same pipeline being evaluated — the "expected answer" in a golden set entry must come from an independent source (human expert, different LLM, or authoritative document); (3) 20% of the golden set is held out from the nightly regression test and used only for periodic out-of-sample evaluation — this held-out set provides an uncontaminated quality signal.

6.5 Remediation Safety — Preventing Feedback Loop Instability

A feedback loop that improves quality monotonically is ideal. A feedback loop that oscillates (an improvement triggered by a signal causes a different signal to degrade, which triggers a remediation that undoes the original improvement) is a disaster. Three mechanisms prevent loop instability.

Remediation rate limiting — no more than 5 automated remediations per domain per day. This prevents a burst of failures from triggering a cascade of simultaneous remediations that collectively overshoot the optimum. When more than 5 failures are detected in a day, the highest-impact remediations are queued and the rest wait for the next day. The daily cap also provides a natural experiment window: if today's remediations cause a problem, the next nightly regression test catches it before more remediations are applied.

Remediation conflict detection — before any remediation is applied, the system checks whether a conflicting remediation is already in progress. A "conflicting remediation" is one that modifies the same pipeline component: two simultaneous reranker fine-tunes, or a re-chunking job while an embedding model change is being A/B tested. Conflicting remediations are serialised — the second waits for the first to complete and its quality impact to be measured before proceeding.

Post-remediation quality measurement — every remediation records an expected_delta (the expected improvement in quality metrics based on the failure analysis). 24 hours after a remediation is applied (allowing enough query traffic to generate a meaningful quality signal), the Remediation Auditor computes the actual_delta (the measured change in quality metrics on the affected query types). If actual_delta is negative (the remediation made things worse) or significantly lower than expected_delta (the remediation had much less effect than predicted), the Remediation Auditor: (1) rolls back the remediation if possible (e.g. re-chunking can be reverted; a fine-tuned model can be rolled back to the previous checkpoint); (2) increases the failure confidence threshold required to trigger this type of remediation in the future (adaptive conservatism — if re-chunking based on chunking failure classification is not working, require stronger evidence before triggering it again).

6.6 Evaluator Drift — The Meta-Problem of Self-Improving Systems

A self-improving RAG system faces a meta-problem: if the evaluator that guides improvements is itself drifting (because the LLM-as-judge models are being updated, because the definition of "good" changes over time, or because the evaluation prompts become subtly misaligned with user expectations), the system will optimise toward the evaluator's definition of quality rather than the user's. This is evaluation drift — the system gets better at satisfying its own evaluator while potentially getting worse for actual users.

Evaluator version pinning — the LLM-as-judge model is version-pinned: updates to the underlying model require an explicit re-calibration of the judge against the Tier 3 human evaluation corpus before being deployed. A new LLM-as-judge version must demonstrate ≥ 85% agreement with the existing human evaluation corpus (not just a new set of human evaluations) — this prevents the evaluator from drifting to a different definition of quality even if that definition is internally consistent.

User satisfaction as the north star — implicit user satisfaction signals (session length after receiving an answer, whether the user follows up with a refinement question, whether the user returns to ask questions in the same domain) are collected and tracked separately from evaluation scores. Periodically (monthly), the correlation between evaluation scores and user satisfaction signals is measured. If the correlation drops below 0.6 (the evaluator is increasingly diverging from what users find valuable), the evaluation rubric is reviewed and updated. This provides an external validity check on the entire evaluation framework — ensuring that what the system is optimising for remains aligned with what users actually want.

Summary