LLM Zoomcamp 第4模块实战RAG 系统离线评估与在线监控Cosine、LLM-as-a-Judge、Postgres Grafana【免费下载链接】llm-zoomcampLLM Zoomcamp - a free online course about real-life applications of LLMs. In 10 weeks you will learn how to build an AI system that answers questions about your knowledge base. Register here 项目地址: https://gitcode.com/GitHub_Trending/ll/llm-zoomcamp本文基于 LLM Zoomcamp 2024 课程第 4 模块Evaluation and Monitoring的模块文档 cohorts/2024/04-monitoring/README.md 展开完整覆盖该模块的两大主线上线前的离线 RAG 评估生成评估数据、A→Q→A′ 余弦相似度、ROUGE、LLM-as-a-Judge与上线后的在线监控Streamlit 应用收集用户反馈、Postgres 存储对话与指标、Grafana 可视化。读完后你可以掌握一套可复制的“评估 监控”工程方案知道用哪些数据集、哪些指标、哪些 SQL 与配置来量化 RAG 系统的答案质量并把监控面板真正跑起来。模块定位评估与监控分别解决什么问题原模块文档开篇给出了两条主线这也是全篇的骨架评估Evaluation在 RAG 系统上线goes live之前评估整个系统的质量监控Monitoring系统部署后持续收集、存储并可视化指标以衡量已部署 LLM 的答案质量同时收集对话历史chat history与用户反馈user feedback。模块的课时结构如下后文按此顺序展开课时主题核心产出4.1监控答案质量导论明确监控动机与边界4.2离线 vs 在线RAG评估模块回顾、指标选型4.3为离线 RAG 评估生成数据各模型答案 CSV4.4离线评估余弦相似度results-*-cosine.csv4.5离线评估LLM as a judgeevaluations-aqa.csv/evaluations-qa.csv4.6捕获用户反馈1/-1 按钮、Postgres、docker composeapp/完整代码4.6.2捕获用户反馈 Part 2加入向量检索与 OpenAIapp/完整代码4.7系统监控Grafana、Tokens 与成本、QA 相关性、反馈Grafana SQL、dashboard.json4.1 为什么要监控 LLM 系统课时 4.1对应文档中的视频 OWqinqemCmk覆盖四个问题为什么要监控 LLM 系统、如何监控答案质量、如何用用户反馈监控答案质量以及本模块没有覆盖的其他监控项。从仓库中最终的app/代码结构可以推断出课程给出的答案上线后你需要盯住的“质量信号”至少包括三类——答案质量本身用 LLM 在线判断每次问答的相关性relevance字段取值RELEVANT/PARTLY_RELEVANT/NON_RELEVANT资源消耗每次调用的prompt_tokens、completion_tokens、total_tokens与折算后的openai_cost用户体验响应时间response_time与用户显式反馈feedback1/-1。这三类信号最终都落库到 Postgres 的conversations和feedback两张表见 cohorts/2024/04-monitoring/app/db.py再交给 Grafana 可视化。模块文档也明确提示除答案质量外还有其他值得监控的维度如延迟分布、成本趋势这些正是 4.7 节监控面板的设计出发点。4.2 离线 vs 在线评估先回顾整个 RAG 链路课时 4.2视频 yTKGSqkhgI4先做了模块回顾再对比 online在线/部署后与 offline离线/上线前评估并引入离线评估指标。离线评估依赖一个完整的 RAG 链路。评估用 notebook cohorts/2024/04-monitoring/offline-rag-evaluation.ipynb 的开头部分把链路完整搭了出来关键代码与参数如下。加载带 ID 的文档与 ground truth数据来自第 3 模块的评估数据集import requests base_url https://github.com/DataTalksClub/llm-zoomcamp/blob/main relative_url 03-vector-search/eval/documents-with-ids.json docs_url f{base_url}/{relative_url}?raw1 documents requests.get(docs_url).json() import pandas as pd ground_truth_url f{base_url}/03-vector-search/eval/ground-truth-data.csv?raw1 df_ground_truth pd.read_csv(ground_truth_url) df_ground_truth df_ground_truth[df_ground_truth.course machine-learning-zoomcamp] ground_truth df_ground_truth.to_dict(orientrecords)建立 Elasticsearch 向量索引使用的嵌入模型是multi-qa-MiniLM-L6-cos-v1384 维cosine 相似度from sentence_transformers import SentenceTransformer model_name multi-qa-MiniLM-L6-cos-v1 model SentenceTransformer(model_name) from elasticsearch import Elasticsearch es_client Elasticsearch(http://localhost:9200) index_settings { settings: {number_of_shards: 1, number_of_replicas: 0}, mappings: { properties: { text: {type: text}, section: {type: text}, question: {type: text}, course: {type: keyword}, id: {type: keyword}, question_text_vector: { type: dense_vector, dims: 384, index: True, similarity: cosine }, } } } index_name course-questions es_client.indices.delete(indexindex_name, ignore_unavailableTrue) es_client.indices.create(indexindex_name, bodyindex_settings)注意索引时把question text拼接后编码成向量写入question_text_vector字段。kNN 检索k5num_candidates10000并按course做 term 过滤def elastic_search_knn(field, vector, course): knn { field: field, query_vector: vector, k: 5, num_candidates: 10000, filter: {term: {course: course}} } search_query { knn: knn, _source: [text, section, question, course, id] } es_results es_client.search(indexindex_name, bodysearch_query) return [hit[_source] for hit in es_results[hits][hits]] def question_text_vector_knn(q): question, course q[question], q[course] v_q model.encode(question) return elastic_search_knn(question_text_vector, v_q, course)RAG 主流程build_prompt把检索结果按section / question / answer拼成上下文rag()完成 检索 → 组 prompt → 调 LLM 三步def build_prompt(query, search_results): prompt_template Youre a course teaching assistant. Answer the QUESTION based on the CONTEXT from the FAQ database. Use only the facts from the CONTEXT when answering the QUESTION. QUESTION: {question} CONTEXT: {context} .strip() context for doc in search_results: context context fsection: {doc[section]}\nquestion: {doc[question]}\nanswer: {doc[text]}\n\n return prompt_template.format(questionquery, contextcontext).strip() from openai import OpenAI client OpenAI() def llm(prompt, modelgpt-4o): response client.chat.completions.create( modelmodel, messages[{role: user, content: prompt}] ) return response.choices[0].message.content def rag(query: dict, modelgpt-4o) - str: search_results question_text_vector_knn(query) prompt build_prompt(query[question], search_results) answer llm(prompt, modelmodel) return answer4.3 为离线评估生成数据课时 4.3视频 yTO5sRw6x78的任务是对 ground truth 中的每一条question分别用不同模型跑一遍 RAG得到answer_llm并保留原文档答案answer_orig取自doc_idx[rec[document]][text]形成“问题 → 原文档 A → LLM 生成的 A′”三元组。GPT-4o 版本串行即可answers {} for i, rec in enumerate(tqdm(ground_truth)): if i in answers: continue answer_llm rag(rec) doc_id rec[document] original_doc doc_idx[doc_id] answer_orig original_doc[text] answers[i] { answer_llm: answer_llm, answer_orig: answer_orig, document: doc_id, question: rec[question], course: rec[course], }GPT-3.5-Turbo 版本改用ThreadPoolExecutor(max_workers6)并行加速核心是通用的并行工具函数map_progressfrom concurrent.futures import ThreadPoolExecutor pool ThreadPoolExecutor(max_workers6) def map_progress(pool, seq, f): results [] with tqdm(totallen(seq)) as progress: futures [pool.submit(f, el) for el in seq] for f in futures: f.add_done_callback(lambda p: progress.update()) for future in futures: results.append(future.result()) return results def process_record(rec): model gpt-3.5-turbo answer_llm rag(rec, modelmodel) doc_id rec[document] answer_orig doc_idx[doc_id][text] return {answer_llm: answer_llm, answer_orig: answer_orig, document: doc_id, question: rec[question], course: rec[course]} results_gpt35 map_progress(pool, ground_truth, process_record)生成的结果保存为 CSV。原 README 链接指向模块内data/目录当前仓库中这批数据实际位于 2025 版课程同名的rag_evaluation/data/目录下各文件含义一致文件内容results-gpt4o.csvGPT-4o 的答案results-gpt35.csvGPT-3.5-Turbo 的答案results-gpt4o-mini.csvGPT-4o-mini 的答案results-gpt4o-cosine.csvGPT-4o 答案 已算的 cosineresults-gpt35-cosine.csvGPT-3.5-Turbo 答案 已算的 cosineresults-gpt4o-mini-cosine.csvGPT-4o-mini 答案 已算的 cosineevaluations-aqa.csvA→Q→A′ 的 LLM 评审结果evaluations-qa.csvQ→A 的 LLM 评审结果每行记录包含answer_llm、answer_orig、document文档 ID、question、course五个字段加 cosine 的版本在末列追加cosine。4.4 离线评估指标一A→Q→A′ 余弦相似度课时 4.4视频 LlXclbD3pms引入“答案到答案”的相似度评估范式记作 A → Q → A′原文档答案为 A它对应的问题为 QRAG 系统对 Q 生成 A′最终计算cosine(A, A′)。单条计算非常直接注意该 notebook 用的是multi-qa-MiniLM-L6-cos-v1其输出已归一化内积即余弦相似度answer_orig Yes, sessions are recorded if you miss one. ... answer_llm Everything is recorded, so you won’t miss anything. ... v_llm model.encode(answer_llm) v_orig model.encode(answer_orig) v_llm.dot(v_orig)批量计算并回写 CSVdef compute_similarity(record): v_llm model.encode(record[answer_llm]) v_orig model.encode(record[answer_orig]) return v_llm.dot(v_orig) similarity [compute_similarity(record) for record in tqdm(results_gpt4o)] df_gpt4o[cosine] similarity df_gpt4o[cosine].describe()notebook 中记录的 GPT-4o 分布统计可直接复现验证count 1830.000000 mean 0.679129 std 0.217995 min -0.153426 25% 0.591460 50% 0.734788 75% 0.835390 max 0.995339 Name: cosine, dtype: float64对 gpt-3.5-turbo 与 gpt-4o-mini 重复同一流程后notebook 用 seaborn 叠加绘制三个模型的分布图标题 “RAG LLM performance”横轴 “A-Q-A Cosine Similarity”用于直观对比不同 LLM 在同一个检索条件下的答案一致性。最后统一落盘df_gpt4o.to_csv(data/results-gpt4o-cosine.csv, indexFalse) df_gpt35.to_csv(data/results-gpt35-cosine.csv, indexFalse) df_gpt4o_mini.to_csv(data/results-gpt4o-mini-cosine.csv, indexFalse)值得对照的一点2024 年作业homework.md特意换用multi-qa-mpnet-base-dot-v1模型并指出该模型输出向量未归一化所以要先算范数再归一化v_norm v / np.sqrt((v * v).sum())才能得到落在 [-1, 1] 的余弦值。这说明“内积 余弦”只在归一化向量下成立是实践余弦指标时最容易踩的坑。作业答案可对照 cohorts/2024/04-monitoring/solution.ipynb。4.5 离线评估指标二LLM as a Judge课时 4.5视频 IB6jePK1s58用另一个模型当“评审”judge对 RAG 输出做三档分类NON_RELEVANT/PARTLY_RELEVANT/RELEVANT。notebook 中定义了两个评审 prompt分别对应两种评估路径路径一A→Q→A′对比原文档答案与生成答案prompt1_template You are an expert evaluator for a Retrieval-Augmented Generation (RAG) system. Your task is to analyze the relevance of the generated answer compared to the original answer provided. Based on the relevance and similarity of the generated answer to the original answer, you will classify it as NON_RELEVANT, PARTLY_RELEVANT, or RELEVANT. Here is the data for evaluation: Original Answer: {answer_orig} Generated Question: {question} Generated Answer: {answer_llm} Please analyze the content and context of the generated answer in relation to the original answer and provide your evaluation in parsable JSON without using code blocks: {{ Relevance: NON_RELEVANT | PARTLY_RELEVANT | RELEVANT, Explanation: [Provide a brief explanation for your evaluation] }} .strip()路径二Q→A只对比问题与生成答案不需要原文档答案prompt2_template You are an expert evaluator for a Retrieval-Augmented Generation (RAG) system. Your task is to analyze the relevance of the generated answer to the given question. Based on the relevance of the generated answer, you will classify it as NON_RELEVANT, PARTLY_RELEVANT, or RELEVANT. Here is the data for evaluation: Question: {question} Generated Answer: {answer_llm} ... 评审流程是从结果中随机抽样df_gpt4o_mini.sample(n150, random_state1)→ 逐条调llm(prompt, modelgpt-4o-mini)→ 把返回字符串json.loads解析为 DataFrame → 用value_counts()看三档分布再用df_evaluations[df_evaluations.Relevance NON_RELEVANT]抽查坏例。评审要求模型输出“可解析的 JSON 且不用代码块包裹”Explanation字段则用于人工复核判断依据。最终结果保存为 evaluations-aqa.csvA→Q→A′与 evaluations-qa.csvQ→A。从仓库结构看这套 Q→A 评审 prompt 后来被直接搬进了在线监控部署版应用 cohorts/2024/04-monitoring/app/assistant.py 的evaluate_relevance()L116-L144用几乎相同的模板对每次真实用户问答在线打分解析失败时回退为UNKNOWN, Failed to parse evaluation。也就是说离线用的评审方法成了在线质量信号的生产者——这正是“离线/在线两条线共用同一把尺子”的设计。4.6 捕获用户反馈Streamlit Postgres Docker Compose课时 4.6视频 XapKKBUMQ4M的目标是把监控落进产品给应用加 1 / -1 按钮用 Postgres 存对话与反馈最后用 docker compose 一键起全栈。完整可运行代码在 cohorts/2024/04-monitoring/app/ 目录中间过程含 prompt 与 Claude 的输出记录在 cohorts/2024/04-monitoring/code.md。反馈按钮与会话管理前端是 Streamlit 应用 app.py。核心逻辑用st.session_state.conversation_id str(uuid.uuid4())为每次会话生成 UUID每次提问完成后再生成新 ID保证反馈能对应到“上一条”对话选择项包括课程machine-learning-zoomcamp />CREATE TABLE conversations ( id TEXT PRIMARY KEY, question TEXT NOT NULL, answer TEXT NOT NULL, course TEXT NOT NULL, model_used TEXT NOT NULL, response_time FLOAT NOT NULL, relevance TEXT NOT NULL, relevance_explanation TEXT NOT NULL, prompt_tokens INTEGER NOT NULL, completion_tokens INTEGER NOT NULL, total_tokens INTEGER NOT NULL, eval_prompt_tokens INTEGER NOT NULL, eval_completion_tokens INTEGER NOT NULL, eval_total_tokens INTEGER NOT NULL, openai_cost FLOAT NOT NULL, timestamp TIMESTAMP WITH TIME ZONE NOT NULL ); CREATE TABLE feedback ( id SERIAL PRIMARY KEY, conversation_id TEXT REFERENCES conversations(id), feedback INTEGER NOT NULL, timestamp TIMESTAMP WITH TIME ZONE NOT NULL );注意feedback.conversation_id外键指向conversations.id这就是“反馈绑定到具体对话”的实现基础时间戳统一用带时区的TIMESTAMP WITH TIME ZONE代码中固定Europe/Berlin时区。查询侧提供get_recent_conversations(limit5, relevanceNone)LEFT JOIN feedback支持按相关性过滤和get_feedback_stats()SUM(CASE WHEN feedback 0 ...)统计点赞/点踩。模块文档给出的本地验证命令pip install pgcli pgcli -h localhost -U your_username -d course_assistant -W全栈编排docker-compose.yaml 编排了 5 个服务服务镜像说明elasticsearchdocker.elastic.co/elasticsearch/elasticsearch:8.4.3单节点、关闭安全数据卷持久化端口${ELASTIC_PORT:-9200}ollamaollama/ollama本地模型服务端口${OLLAMA_PORT:-11434}postgrespostgres:13库名/用户/密码均来自环境变量streamlit基于 Dockerfile 构建python:3.9-slimCMD [streamlit, run, app.py]端口${STREAMLIT_PORT:-8501}grafanagrafana/grafana:latest端口 3000管理密码默认adminstreamlit 容器通过环境变量拿到各服务地址ELASTIC_URLhttp://elasticsearch:9200、OLLAMA_URLhttp://ollama:11434/v1/、POSTGRES_HOSTpostgres等并depends_on其余三个服务。依赖见 requirements.txtstreamlit、elasticsearch8.14.0、psycopg2-binary2.9.9、openai1.35.7、sentence-transformers2.7.0以及 CPU 版torch2.3.1cpu通过 PyTorch 官方 find-links 源安装避免拉取 CUDA 大包。索引准备脚本 prep.py 与离线 notebook 使用同一套 ES mappingdims: 384、similarity: cosine从仓库拉取documents-with-ids.json建索引然后调用init_db()初始化表结构其中 re-index 部分可按注释选择性跳过只做建表。按 app/README.MD 的说明本地跑通需要pip install psycopg2-binary python-dotenv pip install pgcli docker-compose up -d docker-compose exec ollama ollama pull phi3 # 拉取本地 phi3 模型Part 2加入向量检索与 OpenAI课时 4.6.2视频 BG8MlbidatA把应用从“纯文本检索 本地模型”升级为“Text / Vector 双检索 本地/云端双模型”。assistant.py 的get_answer()L158-L185是整条在线链路def get_answer(query, course, model_choice, search_type): if search_type Vector: vector model.encode(query) search_results elastic_search_knn(question_text_vector, vector, course) else: search_results elastic_search_text(query, course) prompt build_prompt(query, search_results) answer, tokens, response_time llm(prompt, model_choice) relevance, explanation, eval_tokens evaluate_relevance(query, answer) openai_cost calculate_openai_cost(model_choice, tokens) return {answer: answer, response_time: response_time, relevance: relevance, ..., openai_cost: openai_cost}从源码结构看这一版应用的关键设计点有双检索elastic_search_text用multi_matchquestion^3 / text / sectionquestion 字段 3 倍加权elastic_search_knn与离线版完全一致k5, num_candidates10000, course 过滤双模型路由llm()按ollama/或openai/前缀分发到 Ollama 客户端OpenAI 兼容接口api_keyollama占位或 OpenAI 客户端并统一返回answer, tokens, response_timeresponse_time就是监控面板里的响应时间指标在线相关性评审evaluate_relevance()固定用openai/gpt-4o-mini跑 4.5 节同款的 Q→A 评审 prompt评审本身的 token 单独计为eval_*三列与业务回答的 token 分开便于核算“监控本身的开销”成本核算calculate_openai_cost()按单价折算gpt-3.5-turboinput $0.0015 / output $0.002 每千 tokengpt-4o 与 gpt-4o-mini 统一按 $0.03 / $0.06 计ollama 本地模型成本记 0。这些字段与conversations表的一一对应关系就是 4.7 节监控面板能查数的前提。4.7 系统监控Grafana 面板与 SQL课时 4.7视频 BQN0TOi2Rew搭监控面板覆盖文档列出的五类指标Grafana 部署、Tokens 与成本、QA 相关性、用户反馈、其他指标。监控侧代码与查询见 grafana.md 和 dashboard.json。dashboard.json标题 “Course assistant”定义了 7 个面板最近 5 条对话表格、1/-1 饼图、相关性仪表盘、OpenAI 成本时间序列、Token 时间序列、模型使用柱状图、响应时间时间序列。对应的原始 SQL摘自 grafana.md例如-- Response Time SELECT timestamp AS time, response_time FROM conversations ORDER BY timestamp; -- Relevance Distribution SELECT relevance, COUNT(*) as count FROM conversations GROUP BY relevance; -- Token Usage SELECT timestamp AS time, total_tokens FROM conversations ORDER BY timestamp; -- OpenAI Cost SELECT timestamp AS time, openai_cost FROM conversations WHERE openai_cost 0 ORDER BY timestamp; -- Feedback Statistics SELECT SUM(CASE WHEN feedback 0 THEN 1 ELSE 0 END) as thumbs_up, SUM(CASE WHEN feedback 0 THEN 1 ELSE 0 END) as thumbs_down FROM feedback;grafana.md 同时给出了修订版查询核心是引入 Grafana 内置变量让所有面板响应时间范围选择$__timeFrom()/$__timeTo()圈定时间窗$__timeGroup(timestamp, $__interval)按面板当前区间自动分桶。典型改写如 Token 面板SELECT $__timeGroup(timestamp, $__interval) AS time, AVG(total_tokens) AS avg_tokens FROM conversations WHERE timestamp BETWEEN $__timeFrom() AND $__timeTo() GROUP BY 1 ORDER BY 1成本面板则改为按区间求和SUM(openai_cost) AS total_cost ... AND openai_cost 0。这套“原始查询 → 加时间变量 → 聚合”的演进正是文档 4.7.2Grafana 变量、导出/导入 dashboard视频 qGFAX5ra1G8想传达的操作方法。合成数据让面板立刻有内容generate_data.py 提供了数据“预热”脚本先调用generate_synthetic_data()回填过去 6 小时的随机历史数据随机问题/答案、课程、模型、相关性按 1~15 分钟步长插入对话70% 概率附带 ±1 反馈再进入generate_live_data()死循环每秒插入一条新记录使 Grafana 时间序列面板持续滚动——这正是演示“在线监控”的完整闭环真实用户行为被替换为合成流量但入库路径、字段与真实路径完全相同。作业与延伸阅读模块作业homework.md解答 solution.ipynb让学员用前 300 条 gpt-4o-mini 评估数据亲手完成用multi-qa-mpnet-base-dot-v1生成嵌入并读首个分量 → 计算未归一化向量的点积及 75 分位 → 归一化后计算余弦相似度 → 用rouge包pip install rouge写作时最新版 1.0.1计算rouge-1 / rouge-2 / rouge-l的 precision/recall/F1。其中 ROUGE 一节补充了余弦之外的第二把离线尺子rouge-1看 unigram 重叠、rouge-2看 bigram、rouge-l看最长公共子序列。模块文档末尾的 Extra resources 还列出了模块总览视频Loom 分享与社区笔记入口课时 4.6 与 4.7 的中间实现过程完整记录在 code.md需要复现“从零到成品”路径时可对照阅读。小结一条可复用的“评估—监控”路线以本模块文档为骨架仓库代码为实现佐证可以提炼出一条完整的工程路线离线ground truth问题 文档 ID→ RAG 跑批生成answer_llm并行ThreadPoolExecutor→ 用multi-qa-MiniLM-L6-cos-v1计算 A→Q→A′ 余弦注意向量归一化→ 抽样用 LLM-as-a-Judge 输出三档分类 Explanation在线Streamlit 应用把每次问答的response_time / tokens / relevance / openai_cost落 Postgres1/-1 反馈经外键绑定到具体对话可视化Grafana 直连 Postgres7 个面板覆盖响应时间、相关性、模型使用、token、成本、对话流水与反馈统计查询统一套用$__timeFrom()/$__timeTo()时间变量冷启动用合成数据脚本回填历史并持续注入实时流量让监控链路在无真实用户时也能验证。适用前提Elasticsearch 8.xcompose 中为 8.4.3 镜像、Postgres 13、需要OPENAI_API_KEY环境变量gpt 系模型与在线评审均依赖Ollama 需手动pull模型如 phi3。仓库内 2025 年课程cohorts/2025/03-evaluation/对同一套离线评估数据与 notebook 做了延续与迭代可作为后续深入材料。【免费下载链接】llm-zoomcampLLM Zoomcamp - a free online course about real-life applications of LLMs. In 10 weeks you will learn how to build an AI system that answers questions about your knowledge base. Register here 项目地址: https://gitcode.com/GitHub_Trending/ll/llm-zoomcamp创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考 
长河君 · 2026/9/16 19:39:31 · 阅读 1.2k LLM Zoomcamp 第4模块实战：RAG 系统离线评估与在线监控（Cosine、LLM-as-a-Judge、Postgres + Grafana）
LLM Zoomcamp 第4模块实战RAG 系统离线评估与在线监控Cosine、LLM-as-a-Judge、Postgres Grafana【免费下载链接】llm-zoomcampLLM Zoomcamp - a free online course about real-life applications of LLMs. In 10 weeks you will learn how to build an AI system that answers questions about your knowledge base. Register here 项目地址: https://gitcode.com/GitHub_Trending/ll/llm-zoomcamp本文基于 LLM Zoomcamp 2024 课程第 4 模块Evaluation and Monitoring的模块文档 cohorts/2024/04-monitoring/README.md 展开完整覆盖该模块的两大主线上线前的离线 RAG 评估生成评估数据、A→Q→A′ 余弦相似度、ROUGE、LLM-as-a-Judge与上线后的在线监控Streamlit 应用收集用户反馈、Postgres 存储对话与指标、Grafana 可视化。读完后你可以掌握一套可复制的“评估 监控”工程方案知道用哪些数据集、哪些指标、哪些 SQL 与配置来量化 RAG 系统的答案质量并把监控面板真正跑起来。模块定位评估与监控分别解决什么问题原模块文档开篇给出了两条主线这也是全篇的骨架评估Evaluation在 RAG 系统上线goes live之前评估整个系统的质量监控Monitoring系统部署后持续收集、存储并可视化指标以衡量已部署 LLM 的答案质量同时收集对话历史chat history与用户反馈user feedback。模块的课时结构如下后文按此顺序展开课时主题核心产出4.1监控答案质量导论明确监控动机与边界4.2离线 vs 在线RAG评估模块回顾、指标选型4.3为离线 RAG 评估生成数据各模型答案 CSV4.4离线评估余弦相似度results-*-cosine.csv4.5离线评估LLM as a judgeevaluations-aqa.csv/evaluations-qa.csv4.6捕获用户反馈1/-1 按钮、Postgres、docker composeapp/完整代码4.6.2捕获用户反馈 Part 2加入向量检索与 OpenAIapp/完整代码4.7系统监控Grafana、Tokens 与成本、QA 相关性、反馈Grafana SQL、dashboard.json4.1 为什么要监控 LLM 系统课时 4.1对应文档中的视频 OWqinqemCmk覆盖四个问题为什么要监控 LLM 系统、如何监控答案质量、如何用用户反馈监控答案质量以及本模块没有覆盖的其他监控项。从仓库中最终的app/代码结构可以推断出课程给出的答案上线后你需要盯住的“质量信号”至少包括三类——答案质量本身用 LLM 在线判断每次问答的相关性relevance字段取值RELEVANT/PARTLY_RELEVANT/NON_RELEVANT资源消耗每次调用的prompt_tokens、completion_tokens、total_tokens与折算后的openai_cost用户体验响应时间response_time与用户显式反馈feedback1/-1。这三类信号最终都落库到 Postgres 的conversations和feedback两张表见 cohorts/2024/04-monitoring/app/db.py再交给 Grafana 可视化。模块文档也明确提示除答案质量外还有其他值得监控的维度如延迟分布、成本趋势这些正是 4.7 节监控面板的设计出发点。4.2 离线 vs 在线评估先回顾整个 RAG 链路课时 4.2视频 yTKGSqkhgI4先做了模块回顾再对比 online在线/部署后与 offline离线/上线前评估并引入离线评估指标。离线评估依赖一个完整的 RAG 链路。评估用 notebook cohorts/2024/04-monitoring/offline-rag-evaluation.ipynb 的开头部分把链路完整搭了出来关键代码与参数如下。加载带 ID 的文档与 ground truth数据来自第 3 模块的评估数据集import requests base_url https://github.com/DataTalksClub/llm-zoomcamp/blob/main relative_url 03-vector-search/eval/documents-with-ids.json docs_url f{base_url}/{relative_url}?raw1 documents requests.get(docs_url).json() import pandas as pd ground_truth_url f{base_url}/03-vector-search/eval/ground-truth-data.csv?raw1 df_ground_truth pd.read_csv(ground_truth_url) df_ground_truth df_ground_truth[df_ground_truth.course machine-learning-zoomcamp] ground_truth df_ground_truth.to_dict(orientrecords)建立 Elasticsearch 向量索引使用的嵌入模型是multi-qa-MiniLM-L6-cos-v1384 维cosine 相似度from sentence_transformers import SentenceTransformer model_name multi-qa-MiniLM-L6-cos-v1 model SentenceTransformer(model_name) from elasticsearch import Elasticsearch es_client Elasticsearch(http://localhost:9200) index_settings { settings: {number_of_shards: 1, number_of_replicas: 0}, mappings: { properties: { text: {type: text}, section: {type: text}, question: {type: text}, course: {type: keyword}, id: {type: keyword}, question_text_vector: { type: dense_vector, dims: 384, index: True, similarity: cosine }, } } } index_name course-questions es_client.indices.delete(indexindex_name, ignore_unavailableTrue) es_client.indices.create(indexindex_name, bodyindex_settings)注意索引时把question text拼接后编码成向量写入question_text_vector字段。kNN 检索k5num_candidates10000并按course做 term 过滤def elastic_search_knn(field, vector, course): knn { field: field, query_vector: vector, k: 5, num_candidates: 10000, filter: {term: {course: course}} } search_query { knn: knn, _source: [text, section, question, course, id] } es_results es_client.search(indexindex_name, bodysearch_query) return [hit[_source] for hit in es_results[hits][hits]] def question_text_vector_knn(q): question, course q[question], q[course] v_q model.encode(question) return elastic_search_knn(question_text_vector, v_q, course)RAG 主流程build_prompt把检索结果按section / question / answer拼成上下文rag()完成 检索 → 组 prompt → 调 LLM 三步def build_prompt(query, search_results): prompt_template Youre a course teaching assistant. Answer the QUESTION based on the CONTEXT from the FAQ database. Use only the facts from the CONTEXT when answering the QUESTION. QUESTION: {question} CONTEXT: {context} .strip() context for doc in search_results: context context fsection: {doc[section]}\nquestion: {doc[question]}\nanswer: {doc[text]}\n\n return prompt_template.format(questionquery, contextcontext).strip() from openai import OpenAI client OpenAI() def llm(prompt, modelgpt-4o): response client.chat.completions.create( modelmodel, messages[{role: user, content: prompt}] ) return response.choices[0].message.content def rag(query: dict, modelgpt-4o) - str: search_results question_text_vector_knn(query) prompt build_prompt(query[question], search_results) answer llm(prompt, modelmodel) return answer4.3 为离线评估生成数据课时 4.3视频 yTO5sRw6x78的任务是对 ground truth 中的每一条question分别用不同模型跑一遍 RAG得到answer_llm并保留原文档答案answer_orig取自doc_idx[rec[document]][text]形成“问题 → 原文档 A → LLM 生成的 A′”三元组。GPT-4o 版本串行即可answers {} for i, rec in enumerate(tqdm(ground_truth)): if i in answers: continue answer_llm rag(rec) doc_id rec[document] original_doc doc_idx[doc_id] answer_orig original_doc[text] answers[i] { answer_llm: answer_llm, answer_orig: answer_orig, document: doc_id, question: rec[question], course: rec[course], }GPT-3.5-Turbo 版本改用ThreadPoolExecutor(max_workers6)并行加速核心是通用的并行工具函数map_progressfrom concurrent.futures import ThreadPoolExecutor pool ThreadPoolExecutor(max_workers6) def map_progress(pool, seq, f): results [] with tqdm(totallen(seq)) as progress: futures [pool.submit(f, el) for el in seq] for f in futures: f.add_done_callback(lambda p: progress.update()) for future in futures: results.append(future.result()) return results def process_record(rec): model gpt-3.5-turbo answer_llm rag(rec, modelmodel) doc_id rec[document] answer_orig doc_idx[doc_id][text] return {answer_llm: answer_llm, answer_orig: answer_orig, document: doc_id, question: rec[question], course: rec[course]} results_gpt35 map_progress(pool, ground_truth, process_record)生成的结果保存为 CSV。原 README 链接指向模块内data/目录当前仓库中这批数据实际位于 2025 版课程同名的rag_evaluation/data/目录下各文件含义一致文件内容results-gpt4o.csvGPT-4o 的答案results-gpt35.csvGPT-3.5-Turbo 的答案results-gpt4o-mini.csvGPT-4o-mini 的答案results-gpt4o-cosine.csvGPT-4o 答案 已算的 cosineresults-gpt35-cosine.csvGPT-3.5-Turbo 答案 已算的 cosineresults-gpt4o-mini-cosine.csvGPT-4o-mini 答案 已算的 cosineevaluations-aqa.csvA→Q→A′ 的 LLM 评审结果evaluations-qa.csvQ→A 的 LLM 评审结果每行记录包含answer_llm、answer_orig、document文档 ID、question、course五个字段加 cosine 的版本在末列追加cosine。4.4 离线评估指标一A→Q→A′ 余弦相似度课时 4.4视频 LlXclbD3pms引入“答案到答案”的相似度评估范式记作 A → Q → A′原文档答案为 A它对应的问题为 QRAG 系统对 Q 生成 A′最终计算cosine(A, A′)。单条计算非常直接注意该 notebook 用的是multi-qa-MiniLM-L6-cos-v1其输出已归一化内积即余弦相似度answer_orig Yes, sessions are recorded if you miss one. ... answer_llm Everything is recorded, so you won’t miss anything. ... v_llm model.encode(answer_llm) v_orig model.encode(answer_orig) v_llm.dot(v_orig)批量计算并回写 CSVdef compute_similarity(record): v_llm model.encode(record[answer_llm]) v_orig model.encode(record[answer_orig]) return v_llm.dot(v_orig) similarity [compute_similarity(record) for record in tqdm(results_gpt4o)] df_gpt4o[cosine] similarity df_gpt4o[cosine].describe()notebook 中记录的 GPT-4o 分布统计可直接复现验证count 1830.000000 mean 0.679129 std 0.217995 min -0.153426 25% 0.591460 50% 0.734788 75% 0.835390 max 0.995339 Name: cosine, dtype: float64对 gpt-3.5-turbo 与 gpt-4o-mini 重复同一流程后notebook 用 seaborn 叠加绘制三个模型的分布图标题 “RAG LLM performance”横轴 “A-Q-A Cosine Similarity”用于直观对比不同 LLM 在同一个检索条件下的答案一致性。最后统一落盘df_gpt4o.to_csv(data/results-gpt4o-cosine.csv, indexFalse) df_gpt35.to_csv(data/results-gpt35-cosine.csv, indexFalse) df_gpt4o_mini.to_csv(data/results-gpt4o-mini-cosine.csv, indexFalse)值得对照的一点2024 年作业homework.md特意换用multi-qa-mpnet-base-dot-v1模型并指出该模型输出向量未归一化所以要先算范数再归一化v_norm v / np.sqrt((v * v).sum())才能得到落在 [-1, 1] 的余弦值。这说明“内积 余弦”只在归一化向量下成立是实践余弦指标时最容易踩的坑。作业答案可对照 cohorts/2024/04-monitoring/solution.ipynb。4.5 离线评估指标二LLM as a Judge课时 4.5视频 IB6jePK1s58用另一个模型当“评审”judge对 RAG 输出做三档分类NON_RELEVANT/PARTLY_RELEVANT/RELEVANT。notebook 中定义了两个评审 prompt分别对应两种评估路径路径一A→Q→A′对比原文档答案与生成答案prompt1_template You are an expert evaluator for a Retrieval-Augmented Generation (RAG) system. Your task is to analyze the relevance of the generated answer compared to the original answer provided. Based on the relevance and similarity of the generated answer to the original answer, you will classify it as NON_RELEVANT, PARTLY_RELEVANT, or RELEVANT. Here is the data for evaluation: Original Answer: {answer_orig} Generated Question: {question} Generated Answer: {answer_llm} Please analyze the content and context of the generated answer in relation to the original answer and provide your evaluation in parsable JSON without using code blocks: {{ Relevance: NON_RELEVANT | PARTLY_RELEVANT | RELEVANT, Explanation: [Provide a brief explanation for your evaluation] }} .strip()路径二Q→A只对比问题与生成答案不需要原文档答案prompt2_template You are an expert evaluator for a Retrieval-Augmented Generation (RAG) system. Your task is to analyze the relevance of the generated answer to the given question. Based on the relevance of the generated answer, you will classify it as NON_RELEVANT, PARTLY_RELEVANT, or RELEVANT. Here is the data for evaluation: Question: {question} Generated Answer: {answer_llm} ... 评审流程是从结果中随机抽样df_gpt4o_mini.sample(n150, random_state1)→ 逐条调llm(prompt, modelgpt-4o-mini)→ 把返回字符串json.loads解析为 DataFrame → 用value_counts()看三档分布再用df_evaluations[df_evaluations.Relevance NON_RELEVANT]抽查坏例。评审要求模型输出“可解析的 JSON 且不用代码块包裹”Explanation字段则用于人工复核判断依据。最终结果保存为 evaluations-aqa.csvA→Q→A′与 evaluations-qa.csvQ→A。从仓库结构看这套 Q→A 评审 prompt 后来被直接搬进了在线监控部署版应用 cohorts/2024/04-monitoring/app/assistant.py 的evaluate_relevance()L116-L144用几乎相同的模板对每次真实用户问答在线打分解析失败时回退为UNKNOWN, Failed to parse evaluation。也就是说离线用的评审方法成了在线质量信号的生产者——这正是“离线/在线两条线共用同一把尺子”的设计。4.6 捕获用户反馈Streamlit Postgres Docker Compose课时 4.6视频 XapKKBUMQ4M的目标是把监控落进产品给应用加 1 / -1 按钮用 Postgres 存对话与反馈最后用 docker compose 一键起全栈。完整可运行代码在 cohorts/2024/04-monitoring/app/ 目录中间过程含 prompt 与 Claude 的输出记录在 cohorts/2024/04-monitoring/code.md。反馈按钮与会话管理前端是 Streamlit 应用 app.py。核心逻辑用st.session_state.conversation_id str(uuid.uuid4())为每次会话生成 UUID每次提问完成后再生成新 ID保证反馈能对应到“上一条”对话选择项包括课程machine-learning-zoomcamp />CREATE TABLE conversations ( id TEXT PRIMARY KEY, question TEXT NOT NULL, answer TEXT NOT NULL, course TEXT NOT NULL, model_used TEXT NOT NULL, response_time FLOAT NOT NULL, relevance TEXT NOT NULL, relevance_explanation TEXT NOT NULL, prompt_tokens INTEGER NOT NULL, completion_tokens INTEGER NOT NULL, total_tokens INTEGER NOT NULL, eval_prompt_tokens INTEGER NOT NULL, eval_completion_tokens INTEGER NOT NULL, eval_total_tokens INTEGER NOT NULL, openai_cost FLOAT NOT NULL, timestamp TIMESTAMP WITH TIME ZONE NOT NULL ); CREATE TABLE feedback ( id SERIAL PRIMARY KEY, conversation_id TEXT REFERENCES conversations(id), feedback INTEGER NOT NULL, timestamp TIMESTAMP WITH TIME ZONE NOT NULL );注意feedback.conversation_id外键指向conversations.id这就是“反馈绑定到具体对话”的实现基础时间戳统一用带时区的TIMESTAMP WITH TIME ZONE代码中固定Europe/Berlin时区。查询侧提供get_recent_conversations(limit5, relevanceNone)LEFT JOIN feedback支持按相关性过滤和get_feedback_stats()SUM(CASE WHEN feedback 0 ...)统计点赞/点踩。模块文档给出的本地验证命令pip install pgcli pgcli -h localhost -U your_username -d course_assistant -W全栈编排docker-compose.yaml 编排了 5 个服务服务镜像说明elasticsearchdocker.elastic.co/elasticsearch/elasticsearch:8.4.3单节点、关闭安全数据卷持久化端口${ELASTIC_PORT:-9200}ollamaollama/ollama本地模型服务端口${OLLAMA_PORT:-11434}postgrespostgres:13库名/用户/密码均来自环境变量streamlit基于 Dockerfile 构建python:3.9-slimCMD [streamlit, run, app.py]端口${STREAMLIT_PORT:-8501}grafanagrafana/grafana:latest端口 3000管理密码默认adminstreamlit 容器通过环境变量拿到各服务地址ELASTIC_URLhttp://elasticsearch:9200、OLLAMA_URLhttp://ollama:11434/v1/、POSTGRES_HOSTpostgres等并depends_on其余三个服务。依赖见 requirements.txtstreamlit、elasticsearch8.14.0、psycopg2-binary2.9.9、openai1.35.7、sentence-transformers2.7.0以及 CPU 版torch2.3.1cpu通过 PyTorch 官方 find-links 源安装避免拉取 CUDA 大包。索引准备脚本 prep.py 与离线 notebook 使用同一套 ES mappingdims: 384、similarity: cosine从仓库拉取documents-with-ids.json建索引然后调用init_db()初始化表结构其中 re-index 部分可按注释选择性跳过只做建表。按 app/README.MD 的说明本地跑通需要pip install psycopg2-binary python-dotenv pip install pgcli docker-compose up -d docker-compose exec ollama ollama pull phi3 # 拉取本地 phi3 模型Part 2加入向量检索与 OpenAI课时 4.6.2视频 BG8MlbidatA把应用从“纯文本检索 本地模型”升级为“Text / Vector 双检索 本地/云端双模型”。assistant.py 的get_answer()L158-L185是整条在线链路def get_answer(query, course, model_choice, search_type): if search_type Vector: vector model.encode(query) search_results elastic_search_knn(question_text_vector, vector, course) else: search_results elastic_search_text(query, course) prompt build_prompt(query, search_results) answer, tokens, response_time llm(prompt, model_choice) relevance, explanation, eval_tokens evaluate_relevance(query, answer) openai_cost calculate_openai_cost(model_choice, tokens) return {answer: answer, response_time: response_time, relevance: relevance, ..., openai_cost: openai_cost}从源码结构看这一版应用的关键设计点有双检索elastic_search_text用multi_matchquestion^3 / text / sectionquestion 字段 3 倍加权elastic_search_knn与离线版完全一致k5, num_candidates10000, course 过滤双模型路由llm()按ollama/或openai/前缀分发到 Ollama 客户端OpenAI 兼容接口api_keyollama占位或 OpenAI 客户端并统一返回answer, tokens, response_timeresponse_time就是监控面板里的响应时间指标在线相关性评审evaluate_relevance()固定用openai/gpt-4o-mini跑 4.5 节同款的 Q→A 评审 prompt评审本身的 token 单独计为eval_*三列与业务回答的 token 分开便于核算“监控本身的开销”成本核算calculate_openai_cost()按单价折算gpt-3.5-turboinput $0.0015 / output $0.002 每千 tokengpt-4o 与 gpt-4o-mini 统一按 $0.03 / $0.06 计ollama 本地模型成本记 0。这些字段与conversations表的一一对应关系就是 4.7 节监控面板能查数的前提。4.7 系统监控Grafana 面板与 SQL课时 4.7视频 BQN0TOi2Rew搭监控面板覆盖文档列出的五类指标Grafana 部署、Tokens 与成本、QA 相关性、用户反馈、其他指标。监控侧代码与查询见 grafana.md 和 dashboard.json。dashboard.json标题 “Course assistant”定义了 7 个面板最近 5 条对话表格、1/-1 饼图、相关性仪表盘、OpenAI 成本时间序列、Token 时间序列、模型使用柱状图、响应时间时间序列。对应的原始 SQL摘自 grafana.md例如-- Response Time SELECT timestamp AS time, response_time FROM conversations ORDER BY timestamp; -- Relevance Distribution SELECT relevance, COUNT(*) as count FROM conversations GROUP BY relevance; -- Token Usage SELECT timestamp AS time, total_tokens FROM conversations ORDER BY timestamp; -- OpenAI Cost SELECT timestamp AS time, openai_cost FROM conversations WHERE openai_cost 0 ORDER BY timestamp; -- Feedback Statistics SELECT SUM(CASE WHEN feedback 0 THEN 1 ELSE 0 END) as thumbs_up, SUM(CASE WHEN feedback 0 THEN 1 ELSE 0 END) as thumbs_down FROM feedback;grafana.md 同时给出了修订版查询核心是引入 Grafana 内置变量让所有面板响应时间范围选择$__timeFrom()/$__timeTo()圈定时间窗$__timeGroup(timestamp, $__interval)按面板当前区间自动分桶。典型改写如 Token 面板SELECT $__timeGroup(timestamp, $__interval) AS time, AVG(total_tokens) AS avg_tokens FROM conversations WHERE timestamp BETWEEN $__timeFrom() AND $__timeTo() GROUP BY 1 ORDER BY 1成本面板则改为按区间求和SUM(openai_cost) AS total_cost ... AND openai_cost 0。这套“原始查询 → 加时间变量 → 聚合”的演进正是文档 4.7.2Grafana 变量、导出/导入 dashboard视频 qGFAX5ra1G8想传达的操作方法。合成数据让面板立刻有内容generate_data.py 提供了数据“预热”脚本先调用generate_synthetic_data()回填过去 6 小时的随机历史数据随机问题/答案、课程、模型、相关性按 1~15 分钟步长插入对话70% 概率附带 ±1 反馈再进入generate_live_data()死循环每秒插入一条新记录使 Grafana 时间序列面板持续滚动——这正是演示“在线监控”的完整闭环真实用户行为被替换为合成流量但入库路径、字段与真实路径完全相同。作业与延伸阅读模块作业homework.md解答 solution.ipynb让学员用前 300 条 gpt-4o-mini 评估数据亲手完成用multi-qa-mpnet-base-dot-v1生成嵌入并读首个分量 → 计算未归一化向量的点积及 75 分位 → 归一化后计算余弦相似度 → 用rouge包pip install rouge写作时最新版 1.0.1计算rouge-1 / rouge-2 / rouge-l的 precision/recall/F1。其中 ROUGE 一节补充了余弦之外的第二把离线尺子rouge-1看 unigram 重叠、rouge-2看 bigram、rouge-l看最长公共子序列。模块文档末尾的 Extra resources 还列出了模块总览视频Loom 分享与社区笔记入口课时 4.6 与 4.7 的中间实现过程完整记录在 code.md需要复现“从零到成品”路径时可对照阅读。小结一条可复用的“评估—监控”路线以本模块文档为骨架仓库代码为实现佐证可以提炼出一条完整的工程路线离线ground truth问题 文档 ID→ RAG 跑批生成answer_llm并行ThreadPoolExecutor→ 用multi-qa-MiniLM-L6-cos-v1计算 A→Q→A′ 余弦注意向量归一化→ 抽样用 LLM-as-a-Judge 输出三档分类 Explanation在线Streamlit 应用把每次问答的response_time / tokens / relevance / openai_cost落 Postgres1/-1 反馈经外键绑定到具体对话可视化Grafana 直连 Postgres7 个面板覆盖响应时间、相关性、模型使用、token、成本、对话流水与反馈统计查询统一套用$__timeFrom()/$__timeTo()时间变量冷启动用合成数据脚本回填历史并持续注入实时流量让监控链路在无真实用户时也能验证。适用前提Elasticsearch 8.xcompose 中为 8.4.3 镜像、Postgres 13、需要OPENAI_API_KEY环境变量gpt 系模型与在线评审均依赖Ollama 需手动pull模型如 phi3。仓库内 2025 年课程cohorts/2025/03-evaluation/对同一套离线评估数据与 notebook 做了延续与迭代可作为后续深入材料。【免费下载链接】llm-zoomcampLLM Zoomcamp - a free online course about real-life applications of LLMs. In 10 weeks you will learn how to build an AI system that answers questions about your knowledge base. Register here 项目地址: https://gitcode.com/GitHub_Trending/ll/llm-zoomcamp创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考