The outcome

Replace anecdotal RAG testing with repeatable evidence for relevance, faithfulness, latency, and cost.

Step by step

A workflow you can repeat.

  1. 01

    Create a representative dataset with queries, expected source IDs, reference answers or criteria, and difficult no-answer cases.

  2. 02

    Measure retrieval with appropriate ranking metrics and inspect false positives, false negatives, permission errors, and stale documents.

  3. 03

    Measure response faithfulness, relevance, correctness, citation support, refusal behavior, latency, and model or embedding cost.

  4. 04

    Trace loading, indexing, retrieval, reranking, prompt construction, and generation so each failed example has a diagnosable stage.

  5. 05

    Set release thresholds, compare changes against a fixed baseline, review evaluator disagreements, and monitor sampled production traffic safely.

Working standard

What good use looks like.

  • Keep a fixed regression set.
  • Separate retrieval and answer metrics.
  • Review automated evaluator failures manually.

Official references

Check the current product documentation.