跳转到内容

评测集格式

JSONL,每行一个样本。四种期望类型:精确匹配、规则、rubric 评分、代码测试。引擎会自动留出 20% 作为盲测切片,不参与进化,只用于最终验证。

# evals.jsonl — 每行一个样本;expected 可以是精确答案、规则或评分 rubric
{"id": "t-0001", "input": {"messages": [...]}, "expected": {"type": "exact", "value": "ORDER-8821 已退款"}}
{"id": "t-0002", "input": {"messages": [...]}, "expected": {"type": "rule", "must_call": "lookup_order", "must_not": ["转人工"]}}
{"id": "t-0003", "input": {"messages": [...]}, "expected": {"type": "rubric", "criteria": ["引用了正确的退货政策", "语气礼貌"], "judge": "ouro-judge-v1"}}
{"id": "t-0004", "input": {"messages": [...]}, "expected": {"type": "code", "tests": "tests/test_refund.py"}}