The outcome

Replace ad hoc prompt edits with a reproducible offline and online evaluation loop and a fast rollback path.

Step by step

A workflow you can repeat.

  1. 01

    Turn representative production successes, failures, adversarial cases, and expected outputs into a versioned dataset with access controls.

  2. 02

    Create a prompt version with explicit variables and configuration, label environments deliberately, and link the prompt to generation traces.

  3. 03

    Run an experiment against the current production baseline using deterministic code checks, human review, or calibrated model judges as appropriate.

  4. 04

    Compare quality, safety, latency, tokens, cost, and subgroup failures; inspect disagreements and do not average away critical regressions.

  5. 05

    Promote the version only after thresholds pass, monitor linked online scores, add new failures to the dataset, and roll labels back if needed.

Working standard

What good use looks like.

  • Keep a production baseline in every experiment.
  • Use code checks for deterministic requirements.
  • Promote and roll back with labels.

Official references

Check the current product documentation.