上线后基于真实调用持续迭代Improvement from real traffic after launch
真实调用和人工修改回流成新数据,触发再训练;新版本先影子对照、再灰度晋升。Real calls and human edits flow back as new data and trigger retraining. New versions run in shadow first, then earn traffic as a canary.
同一份冻结测试集上,每一版的质量得分。每个新版本都要先赢过上一版,才能接管流量。Quality on the same frozen test set, version by version. Each new version has to beat the last before it takes traffic.
示例数据Sample data再训练的触发条件When retraining is triggered
不按月定时,按信号触发;新版本必须先过回归集和金丝雀检查。Not on a calendar but on signals. Every new version must pass the regression set and a canary check first.
触发信号Triggers
- 效果漂移Drift线上评分下降,或输入分布明显变化Live scores drop or inputs shift
- 否决率上升More rejections人工修改和退回的比例变高Humans edit or reject more answers
- 新偏好攒够Enough feedback积累约 500 组新的偏好对比About 500 new preference pairs collected
- 底座升级A better base新开源底座发布,用存量数据和评分器重新训练A new open base ships; retrain on existing data and graders
安全上线Safe promotion
影子阶段只打分不返回;灰度从 10% 开始,回退率低于门槛才放大;任何时候一键回滚到上一版。In shadow, answers are scored and never returned. The canary starts at 10% and grows only while fallbacks stay under the bar, and one action returns to the last version at any time.
建议从一个高频任务开始评估Start the evaluation with one frequent task
评估从数据体检开始:判断该任务能否训练、是否值得训练,并给出预期成本与回本周期。The evaluation starts with a data check-up: whether the task can be trained, whether it is worth it, and the expected cost and payback.