持续迭代Continuous improvement

上线后基于真实调用持续迭代Improvement from real traffic after launch

真实调用和人工修改回流成新数据,触发再训练;新版本先影子对照、再灰度晋升。Real calls and human edits flow back as new data and trigger retraining. New versions run in shadow first, then earn traffic as a canary.

真实调用Real calls 验收与反馈Feedback 回流新数据New data 再训练Retrain 影子对照Shadow 灰度晋升Canary v3当前全量版本current version
0.78 0.93

同一份冻结测试集上,每一版的质量得分。每个新版本都要先赢过上一版,才能接管流量。Quality on the same frozen test set, version by version. Each new version has to beat the last before it takes traffic.

示例数据Sample data

再训练的触发条件When retraining is triggered

不按月定时,按信号触发;新版本必须先过回归集和金丝雀检查。Not on a calendar but on signals. Every new version must pass the regression set and a canary check first.

触发信号Triggers

  • 效果漂移Drift线上评分下降,或输入分布明显变化Live scores drop or inputs shift
  • 否决率上升More rejections人工修改和退回的比例变高Humans edit or reject more answers
  • 新偏好攒够Enough feedback积累约 500 组新的偏好对比About 500 new preference pairs collected
  • 底座升级A better base新开源底座发布,用存量数据和评分器重新训练A new open base ships; retrain on existing data and graders

安全上线Safe promotion

影子阶段只打分不返回;灰度从 10% 开始,回退率低于门槛才放大;任何时候一键回滚到上一版。In shadow, answers are scored and never returned. The canary starts at 10% and grows only while fallbacks stay under the bar, and one action returns to the last version at any time.

v4v3

建议从一个高频任务开始评估Start the evaluation with one frequent task

评估从数据体检开始:判断该任务能否训练、是否值得训练,并给出预期成本与回本周期。The evaluation starts with a data check-up: whether the task can be trained, whether it is worth it, and the expected cost and payback.