{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 27 部署与评估\n", "\n", "## 学习目标\n", "1. 了解智能体应用的部署方式，例如本地函数、本地 API、Docker 和云平台\n", "2. 掌握智能体效果评估的常见指标和方法\n", "3. 学会用测试集评估智能体的稳定性和准确性\n", "4. 理解日志、追踪和 LangSmith 在智能体项目中的作用\n", "5. 能够为一个智能体项目设计简单的上线检查流程\n", "\n", "前面我们已经完成了一个带交互界面的智能体项目。本节课要解决的问题是：**项目写完以后，怎么交付给别人使用？怎么判断它运行得好不好？**" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. 为什么要学习部署与评估\n", "\n", "很多初学者做智能体项目时，会停留在 notebook 能运行的阶段。\n", "\n", "但真实项目还需要回答三个问题：\n", "\n", "1. **别人怎么使用它？**\n", "2. **它回答得好不好？**\n", "3. **出问题时怎么排查？**\n", "\n", "这三个问题分别对应：\n", "\n", "| 问题 | 对应能力 |\n", "| --- | --- |\n", "| 怎么使用 | 部署 |\n", "| 好不好用 | 评估 |\n", "| 怎么排查 | 追踪与日志 |\n", "\n", "所以，智能体开发不是写完代码就结束。真正完整的流程应该是：\n", "\n", "```text\n", "开发 -> 测试 -> 评估 -> 部署 -> 监控 -> 迭代\n", "```\n", "\n", "本节会用通俗方式把这条链路讲清楚。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. 智能体常见部署方式\n", "\n", "部署可以理解为：**把只能自己运行的代码，变成别人也能使用的服务。**\n", "\n", "常见部署方式有下面几种：\n", "\n", "| 部署方式 | 适合场景 | 特点 |\n", "| --- | --- | --- |\n", "| 本地函数 | 学习、测试、内部脚本 | 最简单，但只能在代码里调用 |\n", "| 命令行程序 | 工具型项目、个人使用 | 可以通过终端交互 |\n", "| 本地 API | 前后端分离、系统集成 | 其他程序可以通过 HTTP 调用 |\n", "| Docker | 统一环境、方便迁移 | 避免“我电脑能跑，你电脑不能跑” |\n", "| 云平台 | 正式上线、多用户访问 | 稳定性更高，但配置更复杂 |\n", "\n", "初学时不要一上来就追求复杂部署。\n", "\n", "推荐顺序是：\n", "\n", "```text\n", "函数调用 -> 命令行 -> 本地 API -> Docker -> 云平台\n", "```\n", "\n", "每一步都是在上一层基础上增加一点工程化能力。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3. 准备一个可部署的智能体核心函数\n", "\n", "部署前最重要的一件事是：**把核心逻辑封装成一个清晰的函数。**\n", "\n", "不管后面是接命令行、Web API，还是云服务，最终都应该调用同一个核心函数。\n", "\n", "下面我们用一个简化版学习助手作为示例。" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def run_learning_assistant(user_input):\n", " if '计划' in user_input or '安排' in user_input:\n", " intent = 'plan'\n", " response = '建议先学习 State、Node、Edge，再学习条件边、工具调用和项目实战。'\n", " elif '解释' in user_input or '什么是' in user_input:\n", " intent = 'explain'\n", " response = 'LangGraph 可以理解为用图结构组织智能体流程的框架。'\n", " else:\n", " intent = 'chat'\n", " response = '我可以帮你制定学习计划、解释概念或梳理项目思路。'\n", "\n", " return {\n", " 'input': user_input,\n", " 'intent': intent,\n", " 'response': response\n", " }\n", "\n", "result = run_learning_assistant('帮我制定 LangGraph 学习计划')\n", "print(result)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 代码解释\n", "\n", "这段代码是一个最小可部署单元。\n", "\n", "#### `run_learning_assistant(user_input)`\n", "这个函数接收用户输入，然后返回一个结构化结果。\n", "\n", "它没有直接使用 `input()`，也没有直接打印最终答案。\n", "\n", "这样做是为了方便部署。因为不同界面拿到结果后，可以用自己的方式展示：\n", "\n", "- 命令行可以 `print`\n", "- Web API 可以返回 JSON\n", "- 前端页面可以渲染到聊天窗口\n", "\n", "#### 为什么返回字典\n", "函数返回：\n", "\n", "```python\n", "{\n", " 'input': user_input,\n", " 'intent': intent,\n", " 'response': response\n", "}\n", "```\n", "\n", "这比只返回一个字符串更适合工程项目。\n", "\n", "因为调用方不仅能拿到回复，还能看到系统识别出的意图，方便调试和评估。\n", "\n", "#### 部署前的关键原则\n", "无论项目内部多复杂，都建议对外提供一个简单入口。\n", "\n", "可以把它理解成智能体项目的“总开关”。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4. 本地 API 部署的基本思路\n", "\n", "API 部署的意思是：让其他程序通过 HTTP 请求调用你的智能体。\n", "\n", "例如前端页面可以发送：\n", "\n", "```text\n", "POST /chat\n", "{\"message\": \"帮我制定学习计划\"}\n", "```\n", "\n", "服务端返回：\n", "\n", "```text\n", "{\"response\": \"建议先学习 State、Node、Edge...\"}\n", "```\n", "\n", "下面先不用 FastAPI，直接用一个普通函数模拟 API 处理过程。" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def chat_api_handler(request_json):\n", " user_message = request_json.get('message', '')\n", "\n", " if not user_message.strip():\n", " return {\n", " 'ok': False,\n", " 'error': 'message 不能为空'\n", " }\n", "\n", " result = run_learning_assistant(user_message)\n", "\n", " return {\n", " 'ok': True,\n", " 'data': result\n", " }\n", "\n", "request = {'message': '解释一下什么是 LangGraph'}\n", "response = chat_api_handler(request)\n", "print(response)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 代码解释\n", "\n", "这段代码模拟了一个 API 接口。\n", "\n", "#### `request_json`\n", "它代表外部系统传来的请求数据。\n", "\n", "真实 API 中，这个数据通常来自 HTTP 请求体。\n", "\n", "#### `request_json.get('message', '')`\n", "这行代码从请求中取出用户消息。\n", "\n", "如果请求里没有 `message`，就默认使用空字符串。\n", "\n", "#### 空输入检查\n", "如果用户没有传有效内容，接口返回：\n", "\n", "```python\n", "{'ok': False, 'error': 'message 不能为空'}\n", "```\n", "\n", "这是部署时常见的边界检查。\n", "\n", "#### 调用核心函数\n", "真正处理智能体逻辑的是：\n", "\n", "```python\n", "result = run_learning_assistant(user_message)\n", "```\n", "\n", "API 层不应该写太多智能体逻辑，它只负责接收请求、调用核心函数、返回结果。\n", "\n", "#### 为什么这样分层\n", "分层后，项目会更容易维护：\n", "\n", "- 智能体逻辑改动，不影响 API 格式\n", "- API 部署方式改动，不影响智能体核心\n", "- 测试时可以单独测试核心函数，也可以测试接口函数" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5. FastAPI 部署示意\n", "\n", "真实项目中，经常会用 FastAPI 把智能体包装成 HTTP 服务。\n", "\n", "下面是示意代码。这里放在 Markdown 中展示，不要求你现在必须运行。\n", "\n", "```python\n", "from fastapi import FastAPI\n", "from pydantic import BaseModel\n", "\n", "app = FastAPI()\n", "\n", "class ChatRequest(BaseModel):\n", " message: str\n", "\n", "@app.post('/chat')\n", "def chat(request: ChatRequest):\n", " result = run_learning_assistant(request.message)\n", " return result\n", "```\n", "\n", "如果保存为 `app.py`，通常可以用下面命令启动：\n", "\n", "```bash\n", "uvicorn app:app --host 0.0.0.0 --port 8000\n", "```\n", "\n", "启动后，其他程序就可以通过接口调用智能体。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 6. Docker 和云平台部署怎么理解\n", "\n", "当项目只在自己电脑上运行时，问题通常不大。\n", "\n", "但交给别人运行时，经常会出现：\n", "\n", "- Python 版本不同\n", "- 依赖包没安装\n", "- 环境变量没配置\n", "- 系统路径不一致\n", "\n", "Docker 的作用就是把运行环境打包起来。\n", "\n", "可以把 Docker 镜像理解成一个“装好环境的盒子”。别人只要运行这个盒子，就能得到接近一致的环境。\n", "\n", "云平台部署则是在远程服务器上运行这个应用，让更多用户可以访问。\n", "\n", "一个常见上线链路是：\n", "\n", "```text\n", "本地开发 -> 写 requirements.txt -> 写 Dockerfile -> 构建镜像 -> 部署到云服务器\n", "```\n", "\n", "本课程重点是智能体开发，所以这里只需要理解部署思路，不要求一次掌握所有运维细节。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 7. 为什么要评估智能体\n", "\n", "部署解决的是“能不能给别人用”。\n", "\n", "评估解决的是“用得好不好”。\n", "\n", "智能体评估通常关注：\n", "\n", "| 指标 | 含义 | 示例 |\n", "| --- | --- | --- |\n", "| 正确性 | 回答是否正确 | 是否解释对了概念 |\n", "| 相关性 | 是否回答了用户问题 | 问学习计划，不要回复天气 |\n", "| 完整性 | 回答是否覆盖关键点 | 学习计划是否有步骤 |\n", "| 稳定性 | 多次运行是否表现一致 | 同类问题是否走同类分支 |\n", "| 成本 | 调用模型的费用和耗时 | token、延迟、API 成本 |\n", "| 安全性 | 是否避免危险输出 | 不泄露密钥，不执行高风险操作 |\n", "\n", "初学阶段可以先做最简单的评估：准备一批测试问题，看系统是否识别出正确意图。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 8. 构建一个简单评估集\n", "\n", "评估集就是提前准备好的测试样例。\n", "\n", "每条样例通常包含：\n", "\n", "- 输入问题\n", "- 期望结果\n", "\n", "下面我们先评估意图识别是否正确。" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "eval_dataset = [\n", " {'input': '帮我制定 LangGraph 学习计划', 'expected_intent': 'plan'},\n", " {'input': '请安排一个 AI 学习路线', 'expected_intent': 'plan'},\n", " {'input': '解释一下什么是条件边', 'expected_intent': 'explain'},\n", " {'input': '什么是智能体状态', 'expected_intent': 'explain'},\n", " {'input': '你好，你能做什么', 'expected_intent': 'chat'},\n", "]\n", "\n", "print('评估集数量：', len(eval_dataset))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 代码解释\n", "\n", "这段代码定义了一个小型评估集。\n", "\n", "#### `eval_dataset`\n", "它是一个列表，里面每个元素都是一个测试样例。\n", "\n", "每个样例有两个字段：\n", "\n", "- `input`：用户输入\n", "- `expected_intent`：期望识别出的意图\n", "\n", "#### 为什么先评估意图\n", "因为当前学习助手的核心流程是：\n", "\n", "```text\n", "输入 -> 意图识别 -> 分支处理 -> 回复\n", "```\n", "\n", "如果意图识别错了，后面的节点也会走错。\n", "\n", "所以意图识别是一个很适合入门评估的指标。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 9. 自动运行评估\n", "\n", "有了评估集后，就可以写代码自动跑一遍，并计算准确率。" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def evaluate_intent(dataset):\n", " records = []\n", " correct_count = 0\n", "\n", " for item in dataset:\n", " result = run_learning_assistant(item['input'])\n", " predicted = result['intent']\n", " expected = item['expected_intent']\n", " is_correct = predicted == expected\n", "\n", " if is_correct:\n", " correct_count += 1\n", "\n", " records.append({\n", " 'input': item['input'],\n", " 'expected': expected,\n", " 'predicted': predicted,\n", " 'correct': is_correct\n", " })\n", "\n", " accuracy = correct_count / len(dataset)\n", " return accuracy, records\n", "\n", "accuracy, records = evaluate_intent(eval_dataset)\n", "print('准确率：', accuracy)\n", "for record in records:\n", " print(record)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 代码解释\n", "\n", "这段代码完成了一个最小自动评估流程。\n", "\n", "#### `evaluate_intent(dataset)`\n", "这个函数接收评估集，然后逐条测试。\n", "\n", "#### `result = run_learning_assistant(item['input'])`\n", "每条测试样例都会调用一次智能体核心函数。\n", "\n", "这和真实用户调用项目的方式是一致的。\n", "\n", "#### `predicted` 和 `expected`\n", "- `predicted` 是系统实际识别出的意图\n", "- `expected` 是我们提前标注的正确意图\n", "\n", "如果两者相等，就认为这条样例通过。\n", "\n", "#### `accuracy`\n", "准确率计算公式是：\n", "\n", "```text\n", "准确率 = 正确数量 / 总样例数量\n", "```\n", "\n", "#### `records`\n", "除了给出总分，还要保存每条样例的详细结果。\n", "\n", "因为只有总分不够。我们还需要知道到底哪些问题错了，才能继续优化。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 10. 记录运行日志\n", "\n", "部署后，用户的问题可能非常多。\n", "\n", "如果没有日志，出问题时就很难排查。\n", "\n", "日志至少应该记录：\n", "\n", "- 用户输入\n", "- 系统输出\n", "- 识别出的意图\n", "- 是否成功\n", "- 运行时间\n", "\n", "下面用一个简单包装函数演示如何记录日志。" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import time\n", "\n", "run_logs = []\n", "\n", "def run_with_trace(user_input):\n", " start_time = time.time()\n", "\n", " try:\n", " result = run_learning_assistant(user_input)\n", " success = True\n", " error = ''\n", " except Exception as exc:\n", " result = {'input': user_input, 'intent': '', 'response': ''}\n", " success = False\n", " error = str(exc)\n", "\n", " latency = time.time() - start_time\n", "\n", " log = {\n", " 'input': user_input,\n", " 'intent': result['intent'],\n", " 'response': result['response'],\n", " 'success': success,\n", " 'error': error,\n", " 'latency_seconds': round(latency, 4)\n", " }\n", "\n", " run_logs.append(log)\n", " return result\n", "\n", "run_with_trace('解释一下什么是 LangGraph')\n", "run_with_trace('帮我安排学习计划')\n", "\n", "for log in run_logs:\n", " print(log)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 代码解释\n", "\n", "这段代码演示了最基础的追踪思路。\n", "\n", "#### `run_logs`\n", "这是一个列表，用来保存运行日志。\n", "\n", "真实项目中，日志通常会写入文件、数据库或日志平台。\n", "\n", "#### `start_time = time.time()`\n", "记录开始时间，用来计算本次调用耗时。\n", "\n", "#### `try ... except ...`\n", "部署后的系统不能因为一次错误就直接崩溃。\n", "\n", "所以这里用 `try ... except` 捕获异常，并把错误信息写入日志。\n", "\n", "#### `latency_seconds`\n", "表示运行耗时。\n", "\n", "对于真实大模型应用，耗时是非常重要的指标。\n", "\n", "用户不仅关心回答对不对，也关心等多久。\n", "\n", "#### 追踪的价值\n", "日志可以帮助我们回答：\n", "\n", "- 用户都问了什么？\n", "- 哪些问题经常失败？\n", "- 哪类意图识别不准？\n", "- 哪些请求耗时很长？\n", "\n", "这些信息是后续优化智能体的重要依据。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 11. LangSmith 是做什么的\n", "\n", "LangSmith 是 LangChain 生态中的调试、追踪和评估平台。\n", "\n", "你可以把它理解成智能体应用的“运行记录仪”。\n", "\n", "它可以帮助你查看：\n", "\n", "- 每次调用输入了什么\n", "- 调用了哪些链、工具或模型\n", "- 每一步输出是什么\n", "- 哪一步出错\n", "- 每次调用耗时多久\n", "- token 和成本情况\n", "\n", "本节我们不强制接入 LangSmith，因为它需要账号和环境变量配置。\n", "\n", "但你需要理解它解决的问题：**让智能体运行过程可观察、可追踪、可评估。**\n", "\n", "在真实项目中，如果智能体流程比较复杂，建议尽早接入类似追踪工具。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 12. 上线前检查清单\n", "\n", "智能体项目上线前，建议检查下面这些内容。\n", "\n", "| 检查项 | 说明 |\n", "| --- | --- |\n", "| 环境变量 | API Key、Base URL 是否配置正确 |\n", "| 依赖版本 | requirements 是否完整 |\n", "| 基础测试 | 核心函数是否能正常运行 |\n", "| 评估集 | 常见问题是否通过测试 |\n", "| 错误处理 | 空输入、异常情况是否能返回友好提示 |\n", "| 日志追踪 | 是否能记录输入、输出和错误 |\n", "| 安全检查 | 是否避免泄露密钥和执行危险操作 |\n", "| 成本控制 | 是否限制过长输入和频繁调用 |\n", "\n", "这个清单不复杂，但很实用。\n", "\n", "它能帮助你从“代码能跑”过渡到“项目能交付”。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 13. 本节小结\n", "\n", "本节最重要的内容有五点：\n", "\n", "1. **部署是把智能体从个人代码变成别人可使用的服务**\n", "2. **核心逻辑应该先封装成稳定函数，再接命令行、API 或 Web 界面**\n", "3. **评估集可以帮助我们判断智能体是否稳定、准确**\n", "4. **日志和追踪可以帮助我们排查问题、分析用户行为、持续优化系统**\n", "5. **LangSmith 这类工具可以让复杂智能体流程变得可观察、可评估**\n", "\n", "到这里，一个智能体项目就不只是能运行的 demo，而是开始具备工程化交付能力。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 14. 本节练习\n", "\n", "1. 给 `eval_dataset` 增加 5 条新的测试样例\n", "2. 修改 `run_learning_assistant`，增加一个 `summary` 意图\n", "3. 修改评估函数，同时评估 `intent` 和 `response` 中是否包含关键字\n", "4. 给 `run_with_trace` 增加一个 `timestamp` 字段，记录调用时间\n", "5. 思考：如果这个助手要部署给 100 个用户使用，还需要补充哪些能力？" ] } ], "metadata": { "kernelspec": { "display_name": ".venv", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.14.0" } }, "nbformat": 4, "nbformat_minor": 5 }