你有没有过这种经历:改了切分策略,感觉检索效果好了一点但说不上来好多少;换了个Embedding模型,有些问题答得更好了但有些变差了;上线之后用户反馈"感觉不如以前了",但你看代码什么都没改。

这就是没有评估体系的典型症状——优化全凭感觉,改了不知道好了没有,出了问题不知道哪里的问题。

RAG评估的本质就是给系统做"CT检查"——不是看整体"感觉好不好",而是逐环节扫描,精确到"检索的召回率是多少"“生成的忠实度是多少"“哪个环节拉了后腿”。有了这些数据,优化才有方向。

为什么需要专门的RAG评估?

传统的NLP评估指标(BLEU、ROUGE)不适合RAG,因为RAG不是简单的文本生成,而是检索+增强+生成的复合系统。你需要知道的不只是"答案好不好”,还有:

  • 检索到的文档跟问题相关吗?(上下文精确度)

  • 检索到的文档全吗?有没有漏掉关键信息?(上下文召回率)

  • 模型的回答忠于检索到的文档吗?有没有编造?(忠实度)

  • 回答直接解决了用户的问题吗?(回答相关性)

这四个指标分别对应RAG的不同环节,哪个环节出问题就优化哪个。

RAGAS:RAG评估的标准工具

RAGAS(RAG Assessment)是目前最主流的RAG评估框架,开源免费,提供标准化的评估指标。

四个核心指标

1. Context Precision(上下文精确度)

检索到的文档中,有多少是真正相关的?

  • 高分:检索到5个文档,5个都跟问题相关

  • 低分:检索到5个文档,只有1个相关,其余4个是噪音

2. Context Recall(上下文召回率)

回答问题所需的信息,检索到了吗?

  • 高分:标准答案中的所有关键信息都能在检索结果中找到

  • 低分:标准答案中的重要信息检索结果里没有

3. Faithfulness(忠实度)

模型的回答是否忠实于检索到的文档?

  • 高分:回答中的每个论断都能在检索结果中找到依据

  • 低分:模型编造了检索结果中没有的信息

4. Answer Relevancy(回答相关性)

回答是否直接解决了用户的问题?

  • 高分:回答紧扣问题,不跑题

  • 低分:答非所问,或回答了很多无关内容

评估数据集

评估需要四个字段的数据:

<span leaf="">eval_data = {</span><br><span leaf="">&nbsp; &nbsp;&nbsp;</span><span><span leaf="">"question"</span></span><span leaf="">: [</span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</span><span><span leaf="">"公司年假政策是什么?"</span></span><span leaf="">,</span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</span><span><span leaf="">"如何申请差旅报销?"</span></span><span leaf="">,</span><br><span leaf="">&nbsp; &nbsp; ],</span><br><span leaf="">&nbsp; &nbsp;&nbsp;</span><span><span leaf="">"contexts"</span></span><span leaf="">: [ &nbsp;</span><span><span leaf=""># 检索到的文档(自动获取或手动标注)</span></span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp; [</span><span><span leaf="">"公司年假政策:入职满1年可享5天带薪年假..."</span></span><span leaf="">,&nbsp;</span><span><span leaf="">"年假申请需提前3天在OA系统提交..."</span></span><span leaf="">],</span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp; [</span><span><span leaf="">"差旅报销流程:1. 填写报销单 2. 附上发票 3. 提交审批..."</span></span><span leaf="">],</span><br><span leaf="">&nbsp; &nbsp; ],</span><br><span leaf="">&nbsp; &nbsp;&nbsp;</span><span><span leaf="">"answer"</span></span><span leaf="">: [ &nbsp;</span><span><span leaf=""># RAG系统的实际回答</span></span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</span><span><span leaf="">"入职满1年可享5天带薪年假,需提前3天申请。"</span></span><span leaf="">,</span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</span><span><span leaf="">"填写报销单并附上发票后提交审批。"</span></span><span leaf="">,</span><br><span leaf="">&nbsp; &nbsp; ],</span><br><span leaf="">&nbsp; &nbsp;&nbsp;</span><span><span leaf="">"ground_truth"</span></span><span leaf="">: [ &nbsp;</span><span><span leaf=""># 标准答案(人工标注)</span></span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</span><span><span leaf="">"入职满1年可享5天带薪年假,需提前3天在OA系统提交申请。"</span></span><span leaf="">,</span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</span><span><span leaf="">"填写差旅报销单,附上原始发票,提交至直属上级审批。"</span></span><span leaf="">,</span><br><span leaf="">&nbsp; &nbsp; ],</span><br><span leaf="">}</span><br>

运行评估

<span><span leaf="">from</span></span><span leaf="">&nbsp;ragas&nbsp;</span><span><span leaf="">import</span></span><span leaf="">&nbsp;evaluate</span><br><span><span leaf="">from</span></span><span leaf="">&nbsp;ragas.metrics&nbsp;</span><span><span leaf="">import</span></span><span leaf="">&nbsp;(</span><br><span leaf="">&nbsp; &nbsp; context_precision,</span><br><span leaf="">&nbsp; &nbsp; context_recall,</span><br><span leaf="">&nbsp; &nbsp; faithfulness,</span><br><span leaf="">&nbsp; &nbsp; answer_relevancy,</span><br><span leaf="">)</span><br><span><span leaf="">from</span></span><span leaf="">&nbsp;datasets&nbsp;</span><span><span leaf="">import</span></span><span leaf="">&nbsp;Dataset</span><br><br><span><span leaf=""># 构建评估数据集</span></span><br><span leaf="">eval_dataset = Dataset.from_dict(eval_data)</span><br><br><span><span leaf=""># 运行评估</span></span><br><span leaf="">result = evaluate(</span><br><span leaf="">&nbsp; &nbsp; eval_dataset,</span><br><span leaf="">&nbsp; &nbsp; metrics=[context_precision, context_recall, faithfulness, answer_relevancy],</span><br><span leaf="">)</span><br><br><span leaf="">print(result)</span><br><span><span leaf=""># 输出示例:</span></span><br><span><span leaf=""># {</span></span><br><span><span leaf=""># &nbsp; 'context_precision': 0.78,</span></span><br><span><span leaf=""># &nbsp; 'context_recall': 0.65,</span></span><br><span><span leaf=""># &nbsp; 'faithfulness': 0.85,</span></span><br><span><span leaf=""># &nbsp; 'answer_relevancy': 0.72,</span></span><br><span><span leaf=""># }</span></span><br>

解读评估结果

| 指标低

|

说明

|

优化方向

Context Precision低

|

检索噪音多

|

加Rerank、调相似度阈值、优化chunk

| |

Context Recall低

|

检索遗漏多

|

换Embedding、加混合检索、查询改写

| |

Faithfulness低

|

模型在编

|

改提示词、换模型、加来源标注要求

| |

Answer Relevancy低

|

答非所问

|

改查询理解、提示词加约束

|

构建黄金测试集:评估的地基

评估结果的可靠性取决于测试集的质量。随便找几个问题测一下,结论不靠谱。

测试集构建方法

方法1:人工标注(最可靠)

从真实用户日志中抽取50-100个问题,人工标注标准答案:

<span><span leaf=""># 从日志中抽取真实问题</span></span><br><span leaf="">real_queries = [</span><br><span leaf="">&nbsp; &nbsp;&nbsp;</span><span><span leaf="">"年假可以跨年累积吗?"</span></span><span leaf="">,</span><br><span leaf="">&nbsp; &nbsp;&nbsp;</span><span><span leaf="">"出差补贴标准是多少?"</span></span><span leaf="">,</span><br><span leaf="">&nbsp; &nbsp;&nbsp;</span><span><span leaf="">"试用期多长时间?"</span></span><span leaf="">,</span><br><span leaf="">&nbsp; &nbsp;&nbsp;</span><span><span leaf=""># ...50-100个</span></span><br><span leaf="">]</span><br><br><span><span leaf=""># 人工标注标准答案</span></span><br><span leaf="">ground_truths = [</span><br><span leaf="">&nbsp; &nbsp;&nbsp;</span><span><span leaf="">"年假不可跨年累积,当年未使用的年假在12月31日自动清零。"</span></span><span leaf="">,</span><br><span leaf="">&nbsp; &nbsp;&nbsp;</span><span><span leaf="">"出差补贴分为住宿、交通、餐饮三项..."</span></span><span leaf="">,</span><br><span leaf="">&nbsp; &nbsp;&nbsp;</span><span><span leaf="">"试用期根据合同期限不同..."</span></span><span leaf="">,</span><br><span leaf="">]</span><br>

方法2:LLM生成(快速但有偏差)

用模型基于文档生成问答对:

<span><span leaf="">from</span></span><span leaf="">&nbsp;ragas.testset.generator&nbsp;</span><span><span leaf="">import</span></span><span leaf="">&nbsp;TestsetGenerator</span><br><span><span leaf="">from</span></span><span leaf="">&nbsp;ragas.testset.evolutions&nbsp;</span><span><span leaf="">import</span></span><span leaf="">&nbsp;simple, reasoning, multi_context</span><br><br><span leaf="">generator = TestsetGenerator.from_langchain(model)</span><br><br><span><span leaf=""># 基于文档自动生成测试集</span></span><br><span leaf="">testset = generator.generate_with_langchain_docs(</span><br><span leaf="">&nbsp; &nbsp; documents=docs,</span><br><span leaf="">&nbsp; &nbsp; test_size=</span><span><span leaf="">50</span></span><span leaf="">, &nbsp;</span><span><span leaf=""># 生成50个测试用例</span></span><br><span leaf="">&nbsp; &nbsp; distributions={simple:&nbsp;</span><span><span leaf="">0.5</span></span><span leaf="">, reasoning:&nbsp;</span><span><span leaf="">0.3</span></span><span leaf="">, multi_context:&nbsp;</span><span><span leaf="">0.2</span></span><span leaf="">},</span><br><span leaf="">&nbsp; &nbsp;&nbsp;</span><span><span leaf=""># 50%简单问题 + 30%推理问题 + 20%多文档问题</span></span><br><span leaf="">)</span><br>

A/B测试:量化对比不同配置

改了配置到底好了还是差了?不能凭感觉,要A/B测试:

<span><span><span leaf="">def</span></span><span leaf="">&nbsp;</span><span><span leaf="">run_ab_test</span></span><span><span leaf="">(config_a_name, config_a_retriever,&nbsp;</span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; config_b_name, config_b_retriever,&nbsp;</span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; test_dataset)</span></span><span leaf="">:</span></span><br><span leaf="">&nbsp; &nbsp;&nbsp;</span><span><span leaf="">"""A/B测试:对比两种检索配置的RAGAS分数"""</span></span><br><span leaf="">&nbsp; &nbsp;&nbsp;</span><br><span leaf="">&nbsp; &nbsp;&nbsp;</span><span><span><span leaf="">def</span></span><span leaf="">&nbsp;</span><span><span leaf="">run_rag</span></span><span><span leaf="">(retriever, question)</span></span><span leaf="">:</span></span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp; docs = retriever.invoke(question)</span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp; context =&nbsp;</span><span><span leaf="">"\n"</span></span><span leaf="">.join([d.page_content&nbsp;</span><span><span leaf="">for</span></span><span leaf="">&nbsp;d&nbsp;</span><span><span leaf="">in</span></span><span leaf="">&nbsp;docs])</span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp; prompt =&nbsp;</span><span><span leaf="">f"基于以下内容回答问题:\n</span><span><span leaf="">{context}</span></span><span leaf="">\n\n问题:</span><span><span leaf="">{question}</span></span><span leaf="">"</span></span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</span><span><span leaf="">return</span></span><span leaf="">&nbsp;model.invoke(prompt).content, [d.page_content&nbsp;</span><span><span leaf="">for</span></span><span leaf="">&nbsp;d&nbsp;</span><span><span leaf="">in</span></span><span leaf="">&nbsp;docs]</span><br><span leaf="">&nbsp; &nbsp;&nbsp;</span><br><span leaf="">&nbsp; &nbsp; results_a = {</span><span><span leaf="">"question"</span></span><span leaf="">: [],&nbsp;</span><span><span leaf="">"contexts"</span></span><span leaf="">: [],&nbsp;</span><span><span leaf="">"answer"</span></span><span leaf="">: [],&nbsp;</span><span><span leaf="">"ground_truth"</span></span><span leaf="">: []}</span><br><span leaf="">&nbsp; &nbsp; results_b = {</span><span><span leaf="">"question"</span></span><span leaf="">: [],&nbsp;</span><span><span leaf="">"contexts"</span></span><span leaf="">: [],&nbsp;</span><span><span leaf="">"answer"</span></span><span leaf="">: [],&nbsp;</span><span><span leaf="">"ground_truth"</span></span><span leaf="">: []}</span><br><span leaf="">&nbsp; &nbsp;&nbsp;</span><br><span leaf="">&nbsp; &nbsp;&nbsp;</span><span><span leaf="">for</span></span><span leaf="">&nbsp;item&nbsp;</span><span><span leaf="">in</span></span><span leaf="">&nbsp;test_dataset:</span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</span><span><span leaf=""># Config A</span></span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp; answer_a, contexts_a = run_rag(config_a_retriever, item[</span><span><span leaf="">"question"</span></span><span leaf="">])</span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp; results_a[</span><span><span leaf="">"question"</span></span><span leaf="">].append(item[</span><span><span leaf="">"question"</span></span><span leaf="">])</span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp; results_a[</span><span><span leaf="">"contexts"</span></span><span leaf="">].append(contexts_a)</span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp; results_a[</span><span><span leaf="">"answer"</span></span><span leaf="">].append(answer_a)</span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp; results_a[</span><span><span leaf="">"ground_truth"</span></span><span leaf="">].append(item[</span><span><span leaf="">"ground_truth"</span></span><span leaf="">])</span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</span><span><span leaf=""># Config B</span></span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp; answer_b, contexts_b = run_rag(config_b_retriever, item[</span><span><span leaf="">"question"</span></span><span leaf="">])</span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp; results_b[</span><span><span leaf="">"question"</span></span><span leaf="">].append(item[</span><span><span leaf="">"question"</span></span><span leaf="">])</span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp; results_b[</span><span><span leaf="">"contexts"</span></span><span leaf="">].append(contexts_b)</span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp; results_b[</span><span><span leaf="">"answer"</span></span><span leaf="">].append(answer_b)</span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp; results_b[</span><span><span leaf="">"ground_truth"</span></span><span leaf="">].append(item[</span><span><span leaf="">"ground_truth"</span></span><span leaf="">])</span><br><span leaf="">&nbsp; &nbsp;&nbsp;</span><br><span leaf="">&nbsp; &nbsp;&nbsp;</span><span><span leaf=""># 评估</span></span><br><span leaf="">&nbsp; &nbsp; score_a = evaluate(Dataset.from_dict(results_a), metrics=[...])</span><br><span leaf="">&nbsp; &nbsp; score_b = evaluate(Dataset.from_dict(results_b), metrics=[...])</span><br><span leaf="">&nbsp; &nbsp;&nbsp;</span><br><span leaf="">&nbsp; &nbsp; print(</span><span><span leaf="">f"Config A (</span><span><span leaf="">{config_a_name}</span></span><span leaf="">):&nbsp;</span><span><span leaf="">{score_a}</span></span><span leaf="">"</span></span><span leaf="">)</span><br><span leaf="">&nbsp; &nbsp; print(</span><span><span leaf="">f"Config B (</span><span><span leaf="">{config_b_name}</span></span><span leaf="">):&nbsp;</span><span><span leaf="">{score_b}</span></span><span leaf="">"</span></span><span leaf="">)</span><br>

实测案例:对比纯向量检索 vs 混合检索+Rerank

| 指标

|

纯向量

|

混合+Rerank

|

提升

Context Precision

|

0.72

|

0.88

|

+22%

| |

Context Recall

|

0.68

|

0.82

|

+21%

| |

Faithfulness

|

0.83

|

0.86

|

+4%

| |

Answer Relevancy

|

0.71

|

0.79

|

+11%

|

Bad Case分析:定位"谁拖了后腿"

整体分数只能告诉你"好不好",Bad Case分析能告诉你"为什么不好"。

<span><span><span leaf="">def</span></span><span leaf="">&nbsp;</span><span><span leaf="">bad_case_analysis</span></span><span><span leaf="">(eval_results, test_dataset, threshold=</span><span><span leaf="">0.5</span></span><span leaf="">)</span></span><span leaf="">:</span></span><br><span leaf="">&nbsp; &nbsp;&nbsp;</span><span><span leaf="">"""分析得分低的测试用例,定位瓶颈"""</span></span><br><span leaf="">&nbsp; &nbsp; bad_cases = []</span><br><span leaf="">&nbsp; &nbsp;&nbsp;</span><br><span leaf="">&nbsp; &nbsp;&nbsp;</span><span><span leaf="">for</span></span><span leaf="">&nbsp;i, scores&nbsp;</span><span><span leaf="">in</span></span><span leaf="">&nbsp;enumerate(eval_results):</span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</span><span><span leaf="">if</span></span><span leaf="">&nbsp;any(v &lt; threshold&nbsp;</span><span><span leaf="">for</span></span><span leaf="">&nbsp;v&nbsp;</span><span><span leaf="">in</span></span><span leaf="">&nbsp;scores.values()):</span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; case = {</span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</span><span><span leaf="">"question"</span></span><span leaf="">: test_dataset[i][</span><span><span leaf="">"question"</span></span><span leaf="">],</span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</span><span><span leaf="">"answer"</span></span><span leaf="">: test_dataset[i][</span><span><span leaf="">"answer"</span></span><span leaf="">],</span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</span><span><span leaf="">"scores"</span></span><span leaf="">: scores,</span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</span><span><span leaf="">"diagnosis"</span></span><span leaf="">:&nbsp;</span><span><span leaf="">""</span></span><span leaf="">,</span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; }</span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</span><span><span leaf=""># 诊断</span></span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</span><span><span leaf="">if</span></span><span leaf="">&nbsp;scores[</span><span><span leaf="">"context_precision"</span></span><span leaf="">] &lt; threshold:</span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; case[</span><span><span leaf="">"diagnosis"</span></span><span leaf="">] +=&nbsp;</span><span><span leaf="">"🔍 检索噪音多(Context Precision低)→ 优化检索/Rerank "</span></span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</span><span><span leaf="">if</span></span><span leaf="">&nbsp;scores[</span><span><span leaf="">"context_recall"</span></span><span leaf="">] &lt; threshold:</span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; case[</span><span><span leaf="">"diagnosis"</span></span><span leaf="">] +=&nbsp;</span><span><span leaf="">"📭 检索遗漏(Context Recall低)→ 优化Embedding/查询改写 "</span></span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</span><span><span leaf="">if</span></span><span leaf="">&nbsp;scores[</span><span><span leaf="">"faithfulness"</span></span><span leaf="">] &lt; threshold:</span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; case[</span><span><span leaf="">"diagnosis"</span></span><span leaf="">] +=&nbsp;</span><span><span leaf="">"🤥 模型编造(Faithfulness低)→ 优化提示词/换模型 "</span></span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</span><span><span leaf="">if</span></span><span leaf="">&nbsp;scores[</span><span><span leaf="">"answer_relevancy"</span></span><span leaf="">] &lt; threshold:</span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; case[</span><span><span leaf="">"diagnosis"</span></span><span leaf="">] +=&nbsp;</span><span><span leaf="">"↗️ 答非所问(Answer Relevancy低)→ 优化查询理解 "</span></span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; bad_cases.append(case)</span><br><span leaf="">&nbsp; &nbsp;&nbsp;</span><br><span leaf="">&nbsp; &nbsp;&nbsp;</span><span><span leaf="">return</span></span><span leaf="">&nbsp;bad_cases</span><br>

这是RAG调优最有效的方法:找到低分case→定位是哪个环节→针对性优化→重新评估→看分数有没有涨。

Java类比:这就像你用JUnit做单元测试——先跑一遍覆盖率报告,找到覆盖率最低的类,针对性补测试用例,再跑一遍看覆盖率涨了没。RAGAS评估就是RAG的JaCoCo+JUnit。

自动化评估流水线

把评估集成到CI/CD中,每次改代码自动跑评估:

<span><span leaf=""># eval_pipeline.py</span></span><br><span><span leaf="">import</span></span><span leaf="">&nbsp;json</span><br><span><span leaf="">from</span></span><span leaf="">&nbsp;ragas&nbsp;</span><span><span leaf="">import</span></span><span leaf="">&nbsp;evaluate</span><br><span><span leaf="">from</span></span><span leaf="">&nbsp;ragas.metrics&nbsp;</span><span><span leaf="">import</span></span><span leaf="">&nbsp;context_precision, context_recall, faithfulness, answer_relevancy</span><br><br><span><span><span leaf="">def</span></span><span leaf="">&nbsp;</span><span><span leaf="">run_evaluation</span></span><span><span leaf="">()</span></span><span leaf="">:</span></span><br><span leaf="">&nbsp; &nbsp;&nbsp;</span><span><span leaf=""># 1. 加载测试集</span></span><br><span leaf="">&nbsp; &nbsp; testset = json.load(open(</span><span><span leaf="">"golden_testset.json"</span></span><span leaf="">))</span><br><span leaf="">&nbsp; &nbsp;&nbsp;</span><br><span leaf="">&nbsp; &nbsp;&nbsp;</span><span><span leaf=""># 2. 运行RAG获取回答</span></span><br><span leaf="">&nbsp; &nbsp; results = run_rag_on_testset(testset)</span><br><span leaf="">&nbsp; &nbsp;&nbsp;</span><br><span leaf="">&nbsp; &nbsp;&nbsp;</span><span><span leaf=""># 3. 运行RAGAS评估</span></span><br><span leaf="">&nbsp; &nbsp; scores = evaluate(</span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp; Dataset.from_dict(results),</span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp; metrics=[context_precision, context_recall, faithfulness, answer_relevancy],</span><br><span leaf="">&nbsp; &nbsp; )</span><br><span leaf="">&nbsp; &nbsp;&nbsp;</span><br><span leaf="">&nbsp; &nbsp;&nbsp;</span><span><span leaf=""># 4. 与基线对比</span></span><br><span leaf="">&nbsp; &nbsp; baseline = json.load(open(</span><span><span leaf="">"baseline_scores.json"</span></span><span leaf="">))</span><br><span leaf="">&nbsp; &nbsp;&nbsp;</span><br><span leaf="">&nbsp; &nbsp; print(</span><span><span leaf="">"=== RAG评估报告 ==="</span></span><span leaf="">)</span><br><span leaf="">&nbsp; &nbsp;&nbsp;</span><span><span leaf="">for</span></span><span leaf="">&nbsp;metric, score&nbsp;</span><span><span leaf="">in</span></span><span leaf="">&nbsp;scores.items():</span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp; diff = score - baseline.get(metric,&nbsp;</span><span><span leaf="">0</span></span><span leaf="">)</span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp; arrow =&nbsp;</span><span><span leaf="">"↑"</span></span><span leaf="">&nbsp;</span><span><span leaf="">if</span></span><span leaf="">&nbsp;diff &gt;&nbsp;</span><span><span leaf="">0</span></span><span leaf="">&nbsp;</span><span><span leaf="">else</span></span><span leaf="">&nbsp;</span><span><span leaf="">"↓"</span></span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp; print(</span><span><span leaf="">f"</span><span><span leaf="">{metric}</span></span><span leaf="">:&nbsp;</span><span><span leaf="">{score:</span><span><span leaf="">.3</span></span><span leaf="">f}</span></span><span leaf="">&nbsp;(</span><span><span leaf="">{arrow}</span></span><span><span leaf="">{abs(diff):</span><span><span leaf="">.3</span></span><span leaf="">f}</span></span><span leaf="">)"</span></span><span leaf="">)</span><br><span leaf="">&nbsp; &nbsp;&nbsp;</span><br><span leaf="">&nbsp; &nbsp;&nbsp;</span><span><span leaf=""># 5. 分数低于阈值则告警</span></span><br><span leaf="">&nbsp; &nbsp;&nbsp;</span><span><span leaf="">if</span></span><span leaf="">&nbsp;scores[</span><span><span leaf="">"faithfulness"</span></span><span leaf="">] &lt;&nbsp;</span><span><span leaf="">0.7</span></span><span leaf="">:</span><br><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp; print(</span><span><span leaf="">"⚠️ Faithfulness低于0.7,模型可能在编造答案!"</span></span><span leaf="">)</span><br><span leaf="">&nbsp; &nbsp;&nbsp;</span><br><span leaf="">&nbsp; &nbsp;&nbsp;</span><span><span leaf=""># 6. 保存新基线</span></span><br><span leaf="">&nbsp; &nbsp; json.dump(dict(scores), open(</span><span><span leaf="">"baseline_scores.json"</span></span><span leaf="">,&nbsp;</span><span><span leaf="">"w"</span></span><span leaf="">))</span><br><br><span><span leaf="">if</span></span><span leaf="">&nbsp;__name__ ==&nbsp;</span><span><span leaf="">"__main__"</span></span><span leaf="">:</span><br><span leaf="">&nbsp; &nbsp; run_evaluation()</span><br>

每次改了RAG配置后跑一遍这个脚本,5分钟就能知道改好了还是改坏了。

我的使用建议

  1. 先建测试集再优化。没有测试集的优化就是盲人摸象。50个测试用例就够起步了。

  2. 先跑基线分数。在优化之前先跑一遍评估,记录基线。后面所有改动都跟基线对比。

  3. 每次只改一个变量。同时改切分+Embedding+Rerank,你永远不知道哪个改动有效。

  4. 关注Bad Case多于平均分。平均分80%但有几个0分的Bad Case,可能比平均分75%但所有case都及格更危险。

  5. Faithfulness是最重要的指标。模型编造答案比找不到答案更可怕——用户可能基于错误信息做决策。

本篇要点

| 要点

|

说明

四个核心指标

|

Context Precision/Recall + Faithfulness/Answer Relevancy

| |

RAGAS框架

|

开源RAG评估工具,自动化跑分

| |

黄金测试集

|

50-100个真实问题+人工标注答案

| |

A/B测试

|

每次只改一个变量,对比前后分数

| |

Bad Case分析

|

找低分case→定位瓶颈→针对性优化

| |

自动化流水线

|

集成到CI/CD,改了就跑评估

| |

最重要指标

|

Faithfulness > Recall > Precision > Relevancy

|

RAG优化速查checklist

| 症状

|

可能原因

|

优先尝试

Recall低

|

chunk太大/Embedding差/Top-K太小

|

调小chunk → 换BGE-m3 → 加BM25 → 调大K

| |

Precision低

|

没Rerank/Top-K太大/无过滤

|

加Rerank → 降K → 加元数据过滤

| |

Faithfulness低

|

Prompt没约束/模型太弱/温度高

|

加"仅基于上下文"约束 → temperature=0 → 换模型

| |

Relevancy低

|

查询理解差/检索到无关

|

查询改写 → 检查切分 → 换Prompt模板

| |

改了没效果

|

没有评估体系

|

先建50个测试用例,跑基线再调

|

下篇预告

最后一篇讲RAG的生产部署——增量索引策略、监控告警搭建、成本控制、多租户架构。把RAG从"能跑"变成"线上稳定运行"。


你有在做RAG评估吗?用什么工具?测试集怎么建的?评论区分享下经验。

觉得有用就点个在看,下一篇讲RAG生产部署——上线一周就翻车的事我见过太多了。