{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 11 检索增强\n",
    "\n",
    "## 学习目标\n",
    "1. 理解 RAG（Retrieval-Augmented Generation，检索增强生成）的核心原理\n",
    "2. 掌握文档加载、文本分割、向量化的基本流程\n",
    "3. 学会使用 LangChain 构建简单的 RAG 问答链\n",
    "4. 理解检索器（Retriever）在智能体中的作用\n",
    "5. 能够处理本地文本数据并基于其回答问题"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. 为什么需要 RAG\n",
    "\n",
    "大语言模型有两个明显局限：\n",
    "\n",
    "- **知识过时**：模型训练数据有截止日期，不知道最新信息\n",
    "- **容易产生幻觉**：对未训练过的问题可能编造答案\n",
    "\n",
    "**RAG** 通过在回答前先从外部知识库中检索相关信息，把检索到的内容作为上下文喂给模型，从而：\n",
    "\n",
    "- 让模型基于最新、准确的资料回答\n",
    "- 减少幻觉\n",
    "- 实现私有知识库问答（如公司文档、个人笔记）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. RAG 的核心流程\n",
    "\n",
    "一个完整的 RAG 系统通常包含以下步骤：\n",
    "\n",
    "```\n",
    "1. 加载文档     -> DocumentLoader\n",
    "2. 分割文档     -> TextSplitter\n",
    "3. 文本向量化   -> Embeddings\n",
    "4. 存入向量库   -> VectorStore\n",
    "5. 用户提问     -> Query\n",
    "6. 检索相关片段 -> Retriever\n",
    "7. 拼接上下文   -> Context\n",
    "8. 生成回答     -> LLM\n",
    "```\n",
    "\n",
    "其中第 1-4 步是「离线」的索引阶段，第 5-8 步是「在线」的查询阶段。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. 环境准备\n",
    "\n",
    "本节课会用到文档加载、文本分割组件。如果你的虚拟环境中还没有安装，请先执行：\n",
    "\n",
    "```powershell\n",
    "pip install langchain-community langchain-text-splitters -i https://pypi.tuna.tsinghua.edu.cn/simple\n",
    "```\n",
    "\n",
    "> 如果暂时不想安装，本课件也提供了用 Python 原生代码实现的备用方案。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. 准备示例文档\n",
    "\n",
    "先创建一个示例文本文件，后面会用它来演示 RAG 流程。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_text = \"\"\"\n",
    "人工智能（Artificial Intelligence，简称 AI）是计算机科学的一个分支，旨在创建能够执行通常需要人类智能才能完成的任务的系统。\n",
    "\n",
    "机器学习是人工智能的一个重要子领域。它通过数据训练模型，使计算机能够从经验中学习，而无需进行明确的编程。\n",
    "\n",
    "深度学习是机器学习的一个分支，使用多层神经网络来模拟人脑的工作方式。深度学习在图像识别、自然语言处理和语音识别等领域取得了显著成果。\n",
    "\n",
    "自然语言处理（Natural Language Processing，简称 NLP）是人工智能和语言学的交叉领域，研究如何让计算机理解、解释和生成人类语言。\n",
    "\n",
    "大语言模型（Large Language Model，简称 LLM）是基于深度学习的自然语言处理模型。它们通过在海量文本数据上进行预训练，能够生成连贯的文本、回答问题并完成多种语言任务。\n",
    "\"\"\"\n",
    "\n",
    "with open('sample_rag.txt', 'w', encoding='utf-8') as f:\n",
    "    f.write(sample_text.strip())\n",
    "\n",
    "print('示例文档已创建：sample_rag.txt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.1 使用 LangChain 的 TextLoader（推荐）\n",
    "\n",
    "`TextLoader` 可以自动处理编码、元数据等问题。\n",
    "\n",
    "> 注意：`langchain-community` 在 LangChain 1.x 中已被标记为弃用，但仍然可用。代码中用 `warnings.filterwarnings('ignore', category=DeprecationWarning)` 过滤了警告。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "\n",
    "# langchain-community 在 LangChain 1.x 中已被标记为弃用，但仍然可用\n",
    "# 这里过滤掉 DeprecationWarning，避免输出中显示警告信息\n",
    "warnings.filterwarnings('ignore', category=DeprecationWarning)\n",
    "\n",
    "from langchain_community.document_loaders import TextLoader\n",
    "\n",
    "loader = TextLoader('sample_rag.txt', encoding='utf-8')\n",
    "documents = loader.load()\n",
    "\n",
    "print(f'加载了 {len(documents)} 个文档')\n",
    "print(f'第一个文档长度：{len(documents[0].page_content)} 字符')\n",
    "print(f'元数据：{documents[0].metadata}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.2 原生 Python 加载方式（备用）\n",
    "\n",
    "如果暂时不想安装 `langchain-community`，可以直接用 Python 读取文件。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.documents import Document\n",
    "\n",
    "with open('sample_rag.txt', 'r', encoding='utf-8') as f:\n",
    "    text = f.read()\n",
    "\n",
    "documents = [Document(page_content=text, metadata={'source': 'sample_rag.txt'})]\n",
    "print(f'加载了 {len(documents)} 个文档，长度：{len(documents[0].page_content)} 字符')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. 文档分割\n",
    "\n",
    "大模型一次能处理的文本长度有限，而且过长的文档会影响检索精度。因此需要把长文档切分成小块。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_text_splitters import RecursiveCharacterTextSplitter\n",
    "\n",
    "# 创建一个文本分割器\n",
    "# chunk_size: 每个块的最大字符数\n",
    "# chunk_overlap: 相邻块之间的重叠字符数，用于保持上下文连贯\n",
    "splitter = RecursiveCharacterTextSplitter(\n",
    "    chunk_size=100,\n",
    "    chunk_overlap=20,\n",
    "    separators=['\\n\\n', '\\n', '。', '，', ' ', '']\n",
    ")\n",
    "\n",
    "chunks = splitter.split_documents(documents)\n",
    "\n",
    "print(f'分割成 {len(chunks)} 个文本块')\n",
    "for i, chunk in enumerate(chunks[:3]):\n",
    "    print(f'\\n--- 第 {i+1} 块 ---')\n",
    "    print(chunk.page_content[:80] + '...')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 代码解释\n",
    "\n",
    "- `chunk_size`：每个文本块的最大长度\n",
    "- `chunk_overlap`：相邻块重叠的字符数，避免关键信息被切分断掉\n",
    "- `separators`：切分优先级，先按段落切，再按句子切，最后按字符切\n",
    "- `RecursiveCharacterTextSplitter` 会尽量在语义边界处切分"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. 文本向量化\n",
    "\n",
    "向量化是把文本转换成数值向量的过程。语义相近的文本，向量距离也更近。\n",
    "\n",
    "本课件使用 `OpenAIEmbeddings`。如果你的 API 不支持 `qwen3-embedding`，请把 `model` 参数改成你实际可用的 embedding 模型名。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_openai import OpenAIEmbeddings\n",
    "from dotenv import load_dotenv\n",
    "\n",
    "load_dotenv()\n",
    "\n",
    "\n",
    "# 创建 embedding 模型\n",
    "# 如果你的 API 提供的是其他 embedding 模型名，请修改 model 参数\n",
    "embeddings = OpenAIEmbeddings(model='qwen3-embedding')\n",
    "\n",
    "# 测试向量化\n",
    "test_text = '人工智能是计算机科学的分支'\n",
    "vector = embeddings.embed_query(test_text)\n",
    "\n",
    "print(f'文本：{test_text}')\n",
    "print(f'向量维度：{len(vector)}')\n",
    "print(f'向量前5个值：{vector[:5]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 备用方案：使用 FakeEmbeddings 测试流程\n",
    "\n",
    "如果你暂时没有可用的 embedding 接口，可以用 `FakeEmbeddings` 仅测试 RAG 流程。注意：FakeEmbeddings 生成的向量是随机的，检索结果没有实际语义意义。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.embeddings import FakeEmbeddings\n",
    "\n",
    "# 仅用于测试流程，向量维度设为 384\n",
    "fake_embeddings = FakeEmbeddings(size=384)\n",
    "\n",
    "fake_vector = fake_embeddings.embed_query('测试文本')\n",
    "print(f'Fake 向量维度：{len(fake_vector)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8. 向量存储与检索\n",
    "\n",
    "把分割后的文本块存入向量数据库。这里使用 `InMemoryVectorStore`，它是一个内存型向量存储，适合学习和测试。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.vectorstores import InMemoryVectorStore\n",
    "\n",
    "# 创建内存向量库\n",
    "vectorstore = InMemoryVectorStore(embeddings)\n",
    "\n",
    "# 把文本块加入向量库\n",
    "vectorstore.add_documents(chunks)\n",
    "\n",
    "# 创建检索器，k=2 表示返回最相关的 2 个片段\n",
    "retriever = vectorstore.as_retriever(search_kwargs={'k': 2})\n",
    "\n",
    "# 测试检索\n",
    "query = '什么是深度学习？'\n",
    "retrieved_docs = retriever.invoke(query)\n",
    "\n",
    "print(f'检索到 {len(retrieved_docs)} 个相关片段：\\n')\n",
    "for i, doc in enumerate(retrieved_docs):\n",
    "    print(f'--- 片段 {i+1} ---')\n",
    "    print(doc.page_content)\n",
    "    print(f'来源：{doc.metadata}\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 代码解释\n",
    "\n",
    "- `InMemoryVectorStore(embeddings)`：创建内存向量库，使用指定的 embedding 模型\n",
    "- `add_documents(chunks)`：把文本块存入向量库\n",
    "- `as_retriever(search_kwargs={'k': 2})`：把向量库转成检索器，每次返回前 2 个最相关结果\n",
    "- `retriever.invoke(query)`：根据用户问题检索相关文本块"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9. 构建 RAG 问答链\n",
    "\n",
    "把检索器和 LLM 组合起来，构建一个完整的 RAG 链。流程是：\n",
    "\n",
    "1. 接收用户问题\n",
    "2. 用检索器找到相关文档\n",
    "3. 把文档内容拼接到 Prompt 中\n",
    "4. 让 LLM 基于上下文回答"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_openai import ChatOpenAI\n",
    "from langchain_core.prompts import ChatPromptTemplate\n",
    "from langchain_core.output_parsers import StrOutputParser\n",
    "from langchain_core.runnables import RunnablePassthrough\n",
    "\n",
    "llm = ChatOpenAI(model='qwen3.6-35b-A3b', temperature=0.1)\n",
    "\n",
    "# 定义 RAG Prompt 模板\n",
    "rag_prompt = ChatPromptTemplate.from_messages([\n",
    "    ('system', '你是一个知识库问答助手。请根据下面的上下文回答问题，如果上下文没有相关信息，请说「我不知道」。'),\n",
    "    ('user', '''上下文：\\n{context}\\n\\n问题：{question}''')\n",
    "])\n",
    "\n",
    "# 把检索到的文档拼接成字符串\n",
    "def format_docs(docs):\n",
    "    return '\\n\\n'.join([doc.page_content for doc in docs])\n",
    "\n",
    "# 构建 RAG 链\n",
    "rag_chain = (\n",
    "    {'context': retriever | format_docs, 'question': RunnablePassthrough()}\n",
    "    | rag_prompt\n",
    "    | llm\n",
    "    | StrOutputParser()\n",
    ")\n",
    "\n",
    "# 测试\n",
    "question = '深度学习和大语言模型有什么关系？'\n",
    "answer = rag_chain.invoke(question)\n",
    "print('问题：', question)\n",
    "print('\\n回答：', answer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 代码解释\n",
    "\n",
    "- `{'context': retriever | format_docs, 'question': RunnablePassthrough()}`：\n",
    "  - `context` 分支：用检索器找到相关文档，再用 `format_docs` 拼接成字符串\n",
    "  - `question` 分支：用 `RunnablePassthrough()` 把原始问题传递下去\n",
    "- `rag_prompt`：把 context 和 question 填充到模板中\n",
    "- `llm`：让模型基于上下文生成回答\n",
    "- `StrOutputParser()`：把 AIMessage 转成字符串"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 10. 完整示例：基于私有文档的问答系统\n",
    "\n",
    "下面把前面的步骤整合成一个完整的 RAG 系统。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.document_loaders import TextLoader\n",
    "from langchain_text_splitters import RecursiveCharacterTextSplitter\n",
    "from langchain_core.vectorstores import InMemoryVectorStore\n",
    "from langchain_openai import ChatOpenAI, OpenAIEmbeddings\n",
    "from langchain_core.prompts import ChatPromptTemplate\n",
    "from langchain_core.output_parsers import StrOutputParser\n",
    "from langchain_core.runnables import RunnablePassthrough\n",
    "\n",
    "# 1. 加载文档\n",
    "loader = TextLoader('sample_rag.txt', encoding='utf-8')\n",
    "documents = loader.load()\n",
    "\n",
    "# 2. 分割文档\n",
    "splitter = RecursiveCharacterTextSplitter(chunk_size=100, chunk_overlap=20)\n",
    "chunks = splitter.split_documents(documents)\n",
    "\n",
    "# 3. 创建 embedding 和向量库\n",
    "embeddings = OpenAIEmbeddings(model='qwen3-embedding')\n",
    "vectorstore = InMemoryVectorStore(embeddings)\n",
    "vectorstore.add_documents(chunks)\n",
    "\n",
    "# 4. 创建检索器\n",
    "retriever = vectorstore.as_retriever(search_kwargs={'k': 2})\n",
    "\n",
    "# 5. 创建 LLM 和 Prompt\n",
    "llm = ChatOpenAI(model='qwen3.6-35b-A3b', temperature=0.1)\n",
    "prompt = ChatPromptTemplate.from_messages([\n",
    "    ('system', '你是知识库问答助手，请严格根据上下文回答。'),\n",
    "    ('user', '''上下文：\\n{context}\\n\\n问题：{question}''')\n",
    "])\n",
    "\n",
    "def format_docs(docs):\n",
    "    return '\\n\\n'.join([doc.page_content for doc in docs])\n",
    "\n",
    "# 6. 构建 RAG 链\n",
    "qa_chain = (\n",
    "    {'context': retriever | format_docs, 'question': RunnablePassthrough()}\n",
    "    | prompt\n",
    "    | llm\n",
    "    | StrOutputParser()\n",
    ")\n",
    "\n",
    "# 7. 提问\n",
    "questions = [\n",
    "    '机器学习是什么？',\n",
    "    '自然语言处理和大语言模型有什么关系？',\n",
    "    '苹果是什么颜色的？'  # 这个问题不在文档中，观察模型是否说不知道\n",
    "]\n",
    "\n",
    "for q in questions:\n",
    "    print(f'Q: {q}')\n",
    "    print(f'A: {qa_chain.invoke(q)}\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 11. RAG 系统的优化方向\n",
    "\n",
    "实际应用中，RAG 系统还有很多优化空间：\n",
    "\n",
    "| 优化方向 | 说明 |\n",
    "| --- | --- |\n",
    "| **更好的分块策略** | 按语义、按段落、按标题切分，避免切断关键信息 |\n",
    "| **更优的 Embedding 模型** | 使用针对中文优化的模型，如 BGE、M3E 等 |\n",
    "| **重排序（Rerank）** | 先检索多个候选，再用更精确的模型排序 |\n",
    "| **查询改写** | 把用户问题改写成更适合检索的形式 |\n",
    "| **持久化向量库** | 使用 Chroma、FAISS、Milvus 等持久化存储 |\n",
    "| **元数据过滤** | 根据来源、时间、类别等元数据进行筛选 |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 12. 本节课练习\n",
    "\n",
    "1. 创建一个新的文本文件，写入关于你感兴趣的主题（如 Python、篮球、电影），用 TextLoader 加载并分割\n",
    "2. 调整 `chunk_size` 和 `chunk_overlap`，观察分割结果的变化\n",
    "3. 用 `OpenAIEmbeddings` 或 `FakeEmbeddings` 把文本块存入 `InMemoryVectorStore`\n",
    "4. 针对你的文档提出 3 个问题，观察 RAG 链的回答质量\n",
    "5. 尝试问一个文档中没有的问题，观察模型是否按 Prompt 要求回答「我不知道」"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}