506 lines
17 KiB
Plaintext
506 lines
17 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"# 11 检索增强\n",
|
||
"\n",
|
||
"## 学习目标\n",
|
||
"1. 理解 RAG（Retrieval-Augmented Generation，检索增强生成）的核心原理\n",
|
||
"2. 掌握文档加载、文本分割、向量化的基本流程\n",
|
||
"3. 学会使用 LangChain 构建简单的 RAG 问答链\n",
|
||
"4. 理解检索器（Retriever）在智能体中的作用\n",
|
||
"5. 能够处理本地文本数据并基于其回答问题"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 1. 为什么需要 RAG\n",
|
||
"\n",
|
||
"大语言模型有两个明显局限：\n",
|
||
"\n",
|
||
"- **知识过时**：模型训练数据有截止日期，不知道最新信息\n",
|
||
"- **容易产生幻觉**：对未训练过的问题可能编造答案\n",
|
||
"\n",
|
||
"**RAG** 通过在回答前先从外部知识库中检索相关信息，把检索到的内容作为上下文喂给模型，从而：\n",
|
||
"\n",
|
||
"- 让模型基于最新、准确的资料回答\n",
|
||
"- 减少幻觉\n",
|
||
"- 实现私有知识库问答（如公司文档、个人笔记）"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 2. RAG 的核心流程\n",
|
||
"\n",
|
||
"一个完整的 RAG 系统通常包含以下步骤：\n",
|
||
"\n",
|
||
"```\n",
|
||
"1. 加载文档 -> DocumentLoader\n",
|
||
"2. 分割文档 -> TextSplitter\n",
|
||
"3. 文本向量化 -> Embeddings\n",
|
||
"4. 存入向量库 -> VectorStore\n",
|
||
"5. 用户提问 -> Query\n",
|
||
"6. 检索相关片段 -> Retriever\n",
|
||
"7. 拼接上下文 -> Context\n",
|
||
"8. 生成回答 -> LLM\n",
|
||
"```\n",
|
||
"\n",
|
||
"其中第 1-4 步是「离线」的索引阶段，第 5-8 步是「在线」的查询阶段。"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 3. 环境准备\n",
|
||
"\n",
|
||
"本节课会用到文档加载、文本分割组件。如果你的虚拟环境中还没有安装，请先执行：\n",
|
||
"\n",
|
||
"```powershell\n",
|
||
"pip install langchain-community langchain-text-splitters -i https://pypi.tuna.tsinghua.edu.cn/simple\n",
|
||
"```\n",
|
||
"\n",
|
||
"> 如果暂时不想安装，本课件也提供了用 Python 原生代码实现的备用方案。"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 4. 准备示例文档\n",
|
||
"\n",
|
||
"先创建一个示例文本文件，后面会用它来演示 RAG 流程。"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"sample_text = \"\"\"\n",
|
||
"人工智能（Artificial Intelligence，简称 AI）是计算机科学的一个分支，旨在创建能够执行通常需要人类智能才能完成的任务的系统。\n",
|
||
"\n",
|
||
"机器学习是人工智能的一个重要子领域。它通过数据训练模型，使计算机能够从经验中学习，而无需进行明确的编程。\n",
|
||
"\n",
|
||
"深度学习是机器学习的一个分支，使用多层神经网络来模拟人脑的工作方式。深度学习在图像识别、自然语言处理和语音识别等领域取得了显著成果。\n",
|
||
"\n",
|
||
"自然语言处理（Natural Language Processing，简称 NLP）是人工智能和语言学的交叉领域，研究如何让计算机理解、解释和生成人类语言。\n",
|
||
"\n",
|
||
"大语言模型（Large Language Model，简称 LLM）是基于深度学习的自然语言处理模型。它们通过在海量文本数据上进行预训练，能够生成连贯的文本、回答问题并完成多种语言任务。\n",
|
||
"\"\"\"\n",
|
||
"\n",
|
||
"with open('sample_rag.txt', 'w', encoding='utf-8') as f:\n",
|
||
" f.write(sample_text.strip())\n",
|
||
"\n",
|
||
"print('示例文档已创建：sample_rag.txt')"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 5.1 使用 LangChain 的 TextLoader（推荐）\n",
|
||
"\n",
|
||
"`TextLoader` 可以自动处理编码、元数据等问题。\n",
|
||
"\n",
|
||
"> 注意：`langchain-community` 在 LangChain 1.x 中已被标记为弃用，但仍然可用。代码中用 `warnings.filterwarnings('ignore', category=DeprecationWarning)` 过滤了警告。"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"import warnings\n",
|
||
"\n",
|
||
"# langchain-community 在 LangChain 1.x 中已被标记为弃用，但仍然可用\n",
|
||
"# 这里过滤掉 DeprecationWarning，避免输出中显示警告信息\n",
|
||
"warnings.filterwarnings('ignore', category=DeprecationWarning)\n",
|
||
"\n",
|
||
"from langchain_community.document_loaders import TextLoader\n",
|
||
"\n",
|
||
"loader = TextLoader('sample_rag.txt', encoding='utf-8')\n",
|
||
"documents = loader.load()\n",
|
||
"\n",
|
||
"print(f'加载了 {len(documents)} 个文档')\n",
|
||
"print(f'第一个文档长度：{len(documents[0].page_content)} 字符')\n",
|
||
"print(f'元数据：{documents[0].metadata}')"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 5.2 原生 Python 加载方式（备用）\n",
|
||
"\n",
|
||
"如果暂时不想安装 `langchain-community`，可以直接用 Python 读取文件。"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from langchain_core.documents import Document\n",
|
||
"\n",
|
||
"with open('sample_rag.txt', 'r', encoding='utf-8') as f:\n",
|
||
" text = f.read()\n",
|
||
"\n",
|
||
"documents = [Document(page_content=text, metadata={'source': 'sample_rag.txt'})]\n",
|
||
"print(f'加载了 {len(documents)} 个文档，长度：{len(documents[0].page_content)} 字符')"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 6. 文档分割\n",
|
||
"\n",
|
||
"大模型一次能处理的文本长度有限，而且过长的文档会影响检索精度。因此需要把长文档切分成小块。"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from langchain_text_splitters import RecursiveCharacterTextSplitter\n",
|
||
"\n",
|
||
"# 创建一个文本分割器\n",
|
||
"# chunk_size: 每个块的最大字符数\n",
|
||
"# chunk_overlap: 相邻块之间的重叠字符数，用于保持上下文连贯\n",
|
||
"splitter = RecursiveCharacterTextSplitter(\n",
|
||
" chunk_size=100,\n",
|
||
" chunk_overlap=20,\n",
|
||
" separators=['\\n\\n', '\\n', '。', '，', ' ', '']\n",
|
||
")\n",
|
||
"\n",
|
||
"chunks = splitter.split_documents(documents)\n",
|
||
"\n",
|
||
"print(f'分割成 {len(chunks)} 个文本块')\n",
|
||
"for i, chunk in enumerate(chunks[:3]):\n",
|
||
" print(f'\\n--- 第 {i+1} 块 ---')\n",
|
||
" print(chunk.page_content[:80] + '...')"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 代码解释\n",
|
||
"\n",
|
||
"- `chunk_size`：每个文本块的最大长度\n",
|
||
"- `chunk_overlap`：相邻块重叠的字符数，避免关键信息被切分断掉\n",
|
||
"- `separators`：切分优先级，先按段落切，再按句子切，最后按字符切\n",
|
||
"- `RecursiveCharacterTextSplitter` 会尽量在语义边界处切分"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 7. 文本向量化\n",
|
||
"\n",
|
||
"向量化是把文本转换成数值向量的过程。语义相近的文本，向量距离也更近。\n",
|
||
"\n",
|
||
"本课件使用 `OpenAIEmbeddings`。如果你的 API 不支持 `qwen3-embedding`，请把 `model` 参数改成你实际可用的 embedding 模型名。"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from langchain_openai import OpenAIEmbeddings\n",
|
||
"from dotenv import load_dotenv\n",
|
||
"\n",
|
||
"load_dotenv()\n",
|
||
"\n",
|
||
"\n",
|
||
"# 创建 embedding 模型\n",
|
||
"# 如果你的 API 提供的是其他 embedding 模型名，请修改 model 参数\n",
|
||
"embeddings = OpenAIEmbeddings(model='qwen3-embedding')\n",
|
||
"\n",
|
||
"# 测试向量化\n",
|
||
"test_text = '人工智能是计算机科学的分支'\n",
|
||
"vector = embeddings.embed_query(test_text)\n",
|
||
"\n",
|
||
"print(f'文本：{test_text}')\n",
|
||
"print(f'向量维度：{len(vector)}')\n",
|
||
"print(f'向量前5个值：{vector[:5]}')"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 备用方案：使用 FakeEmbeddings 测试流程\n",
|
||
"\n",
|
||
"如果你暂时没有可用的 embedding 接口，可以用 `FakeEmbeddings` 仅测试 RAG 流程。注意：FakeEmbeddings 生成的向量是随机的，检索结果没有实际语义意义。"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from langchain_core.embeddings import FakeEmbeddings\n",
|
||
"\n",
|
||
"# 仅用于测试流程，向量维度设为 384\n",
|
||
"fake_embeddings = FakeEmbeddings(size=384)\n",
|
||
"\n",
|
||
"fake_vector = fake_embeddings.embed_query('测试文本')\n",
|
||
"print(f'Fake 向量维度：{len(fake_vector)}')"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 8. 向量存储与检索\n",
|
||
"\n",
|
||
"把分割后的文本块存入向量数据库。这里使用 `InMemoryVectorStore`，它是一个内存型向量存储，适合学习和测试。"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from langchain_core.vectorstores import InMemoryVectorStore\n",
|
||
"\n",
|
||
"# 创建内存向量库\n",
|
||
"vectorstore = InMemoryVectorStore(embeddings)\n",
|
||
"\n",
|
||
"# 把文本块加入向量库\n",
|
||
"vectorstore.add_documents(chunks)\n",
|
||
"\n",
|
||
"# 创建检索器，k=2 表示返回最相关的 2 个片段\n",
|
||
"retriever = vectorstore.as_retriever(search_kwargs={'k': 2})\n",
|
||
"\n",
|
||
"# 测试检索\n",
|
||
"query = '什么是深度学习？'\n",
|
||
"retrieved_docs = retriever.invoke(query)\n",
|
||
"\n",
|
||
"print(f'检索到 {len(retrieved_docs)} 个相关片段：\\n')\n",
|
||
"for i, doc in enumerate(retrieved_docs):\n",
|
||
" print(f'--- 片段 {i+1} ---')\n",
|
||
" print(doc.page_content)\n",
|
||
" print(f'来源：{doc.metadata}\\n')"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 代码解释\n",
|
||
"\n",
|
||
"- `InMemoryVectorStore(embeddings)`：创建内存向量库，使用指定的 embedding 模型\n",
|
||
"- `add_documents(chunks)`：把文本块存入向量库\n",
|
||
"- `as_retriever(search_kwargs={'k': 2})`：把向量库转成检索器，每次返回前 2 个最相关结果\n",
|
||
"- `retriever.invoke(query)`：根据用户问题检索相关文本块"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 9. 构建 RAG 问答链\n",
|
||
"\n",
|
||
"把检索器和 LLM 组合起来，构建一个完整的 RAG 链。流程是：\n",
|
||
"\n",
|
||
"1. 接收用户问题\n",
|
||
"2. 用检索器找到相关文档\n",
|
||
"3. 把文档内容拼接到 Prompt 中\n",
|
||
"4. 让 LLM 基于上下文回答"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from langchain_openai import ChatOpenAI\n",
|
||
"from langchain_core.prompts import ChatPromptTemplate\n",
|
||
"from langchain_core.output_parsers import StrOutputParser\n",
|
||
"from langchain_core.runnables import RunnablePassthrough\n",
|
||
"\n",
|
||
"llm = ChatOpenAI(model='qwen3.6-35b-A3b', temperature=0.1)\n",
|
||
"\n",
|
||
"# 定义 RAG Prompt 模板\n",
|
||
"rag_prompt = ChatPromptTemplate.from_messages([\n",
|
||
" ('system', '你是一个知识库问答助手。请根据下面的上下文回答问题，如果上下文没有相关信息，请说「我不知道」。'),\n",
|
||
" ('user', '''上下文：\\n{context}\\n\\n问题：{question}''')\n",
|
||
"])\n",
|
||
"\n",
|
||
"# 把检索到的文档拼接成字符串\n",
|
||
"def format_docs(docs):\n",
|
||
" return '\\n\\n'.join([doc.page_content for doc in docs])\n",
|
||
"\n",
|
||
"# 构建 RAG 链\n",
|
||
"rag_chain = (\n",
|
||
" {'context': retriever | format_docs, 'question': RunnablePassthrough()}\n",
|
||
" | rag_prompt\n",
|
||
" | llm\n",
|
||
" | StrOutputParser()\n",
|
||
")\n",
|
||
"\n",
|
||
"# 测试\n",
|
||
"question = '深度学习和大语言模型有什么关系？'\n",
|
||
"answer = rag_chain.invoke(question)\n",
|
||
"print('问题：', question)\n",
|
||
"print('\\n回答：', answer)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 代码解释\n",
|
||
"\n",
|
||
"- `{'context': retriever | format_docs, 'question': RunnablePassthrough()}`：\n",
|
||
" - `context` 分支：用检索器找到相关文档，再用 `format_docs` 拼接成字符串\n",
|
||
" - `question` 分支：用 `RunnablePassthrough()` 把原始问题传递下去\n",
|
||
"- `rag_prompt`：把 context 和 question 填充到模板中\n",
|
||
"- `llm`：让模型基于上下文生成回答\n",
|
||
"- `StrOutputParser()`：把 AIMessage 转成字符串"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 10. 完整示例：基于私有文档的问答系统\n",
|
||
"\n",
|
||
"下面把前面的步骤整合成一个完整的 RAG 系统。"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from langchain_community.document_loaders import TextLoader\n",
|
||
"from langchain_text_splitters import RecursiveCharacterTextSplitter\n",
|
||
"from langchain_core.vectorstores import InMemoryVectorStore\n",
|
||
"from langchain_openai import ChatOpenAI, OpenAIEmbeddings\n",
|
||
"from langchain_core.prompts import ChatPromptTemplate\n",
|
||
"from langchain_core.output_parsers import StrOutputParser\n",
|
||
"from langchain_core.runnables import RunnablePassthrough\n",
|
||
"\n",
|
||
"# 1. 加载文档\n",
|
||
"loader = TextLoader('sample_rag.txt', encoding='utf-8')\n",
|
||
"documents = loader.load()\n",
|
||
"\n",
|
||
"# 2. 分割文档\n",
|
||
"splitter = RecursiveCharacterTextSplitter(chunk_size=100, chunk_overlap=20)\n",
|
||
"chunks = splitter.split_documents(documents)\n",
|
||
"\n",
|
||
"# 3. 创建 embedding 和向量库\n",
|
||
"embeddings = OpenAIEmbeddings(model='qwen3-embedding')\n",
|
||
"vectorstore = InMemoryVectorStore(embeddings)\n",
|
||
"vectorstore.add_documents(chunks)\n",
|
||
"\n",
|
||
"# 4. 创建检索器\n",
|
||
"retriever = vectorstore.as_retriever(search_kwargs={'k': 2})\n",
|
||
"\n",
|
||
"# 5. 创建 LLM 和 Prompt\n",
|
||
"llm = ChatOpenAI(model='qwen3.6-35b-A3b', temperature=0.1)\n",
|
||
"prompt = ChatPromptTemplate.from_messages([\n",
|
||
" ('system', '你是知识库问答助手，请严格根据上下文回答。'),\n",
|
||
" ('user', '''上下文：\\n{context}\\n\\n问题：{question}''')\n",
|
||
"])\n",
|
||
"\n",
|
||
"def format_docs(docs):\n",
|
||
" return '\\n\\n'.join([doc.page_content for doc in docs])\n",
|
||
"\n",
|
||
"# 6. 构建 RAG 链\n",
|
||
"qa_chain = (\n",
|
||
" {'context': retriever | format_docs, 'question': RunnablePassthrough()}\n",
|
||
" | prompt\n",
|
||
" | llm\n",
|
||
" | StrOutputParser()\n",
|
||
")\n",
|
||
"\n",
|
||
"# 7. 提问\n",
|
||
"questions = [\n",
|
||
" '机器学习是什么？',\n",
|
||
" '自然语言处理和大语言模型有什么关系？',\n",
|
||
" '苹果是什么颜色的？' # 这个问题不在文档中，观察模型是否说不知道\n",
|
||
"]\n",
|
||
"\n",
|
||
"for q in questions:\n",
|
||
" print(f'Q: {q}')\n",
|
||
" print(f'A: {qa_chain.invoke(q)}\\n')"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 11. RAG 系统的优化方向\n",
|
||
"\n",
|
||
"实际应用中，RAG 系统还有很多优化空间：\n",
|
||
"\n",
|
||
"| 优化方向 | 说明 |\n",
|
||
"| --- | --- |\n",
|
||
"| **更好的分块策略** | 按语义、按段落、按标题切分，避免切断关键信息 |\n",
|
||
"| **更优的 Embedding 模型** | 使用针对中文优化的模型，如 BGE、M3E 等 |\n",
|
||
"| **重排序（Rerank）** | 先检索多个候选，再用更精确的模型排序 |\n",
|
||
"| **查询改写** | 把用户问题改写成更适合检索的形式 |\n",
|
||
"| **持久化向量库** | 使用 Chroma、FAISS、Milvus 等持久化存储 |\n",
|
||
"| **元数据过滤** | 根据来源、时间、类别等元数据进行筛选 |"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 12. 本节课练习\n",
|
||
"\n",
|
||
"1. 创建一个新的文本文件，写入关于你感兴趣的主题（如 Python、篮球、电影），用 TextLoader 加载并分割\n",
|
||
"2. 调整 `chunk_size` 和 `chunk_overlap`，观察分割结果的变化\n",
|
||
"3. 用 `OpenAIEmbeddings` 或 `FakeEmbeddings` 把文本块存入 `InMemoryVectorStore`\n",
|
||
"4. 针对你的文档提出 3 个问题，观察 RAG 链的回答质量\n",
|
||
"5. 尝试问一个文档中没有的问题，观察模型是否按 Prompt 要求回答「我不知道」"
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"kernelspec": {
|
||
"display_name": ".venv",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3",
|
||
"version": "3.11.3"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 4
|
||
}
|