上一篇我说过，80%的RAG项目问题出在检索和文档处理，而其中又有40%直接出在切分环节。你可能不信——切分不就是按字符数切嘛，能有多大事？

我之前帮一个团队做法律知识库，他们用的固定512字符切分，结果一份合同里的"甲方权利"和"甲方义务"被切到了两个chunk里。用户问"甲方有什么权利"，检索到的是"义务"那段，模型回答完全跑偏。切分不对，检索再准也没用——因为你存进去的东西就是残缺的。

今天这篇专门讲文档处理和切分，从PDF解析到6种切分策略，我把踩过的坑全掏出来。

先说PDF解析：你以为的文本提取可能是错的

如果你的知识库主要是PDF文档，先别急着切分——PDF解析本身就是个大坑。

常见问题

  • 表格丢失：很多PDF解析器把表格变成一行行文本，列关系全没了

  • 多栏排版混乱：学术论文的双栏排版，解析后左右两栏的文字交叉在一起

  • 图片中的文字：扫描件PDF，文字是图片不是文本，普通解析器提取不了

  • 页眉页脚混入正文：页码、章节标题等噪音被当作正文处理

解析工具选型

| 工具

|

特点

|

适用场景

PyPDFLoader

|

简单快速，纯文本PDF

|

结构简单的文档

| |

PyMuPDFLoader

|

保留更多格式信息

|

带表格、图片的PDF

| |

UnstructuredLoader

|

支持复杂布局、表格识别

|

复杂排版文档

| |

marker

|

开源OCR+布局识别

|

扫描件PDF

| |

docling

|

IBM开源，多格式支持

|

企业级文档处理

|

<span><span leaf=""># 简单PDF</span></span><span leaf=""><br></span><span><span leaf="">from</span></span><span leaf="">&nbsp;langchain_community.document_loaders&nbsp;</span><span><span leaf="">import</span></span><span leaf="">&nbsp;PyPDFLoader</span><span leaf=""><br></span><span leaf="">loader = PyPDFLoader(</span><span><span leaf="">"contract.pdf"</span></span><span leaf="">)</span><span leaf=""><br></span><span leaf="">docs = loader.load()</span><span leaf=""><br></span><span leaf=""><br></span><span><span leaf=""># 复杂PDF（带表格、多栏）</span></span><span leaf=""><br></span><span><span leaf="">from</span></span><span leaf="">&nbsp;langchain_community.document_loaders&nbsp;</span><span><span leaf="">import</span></span><span leaf="">&nbsp;UnstructuredLoader</span><span leaf=""><br></span><span leaf="">loader = UnstructuredLoader(</span><span><span leaf="">"report.pdf"</span></span><span leaf="">, mode=</span><span><span leaf="">"elements"</span></span><span leaf="">)</span><span leaf=""><br></span><span leaf="">docs = loader.load()</span><span leaf=""><br></span><span><span leaf=""># Unstructured会自动识别表格、标题、列表等元素</span></span><span leaf=""><br></span>

我的建议：别在PDF解析上省钱。解析质量直接决定后续所有环节的天花板。如果你的PDF很复杂（表格多、排版乱），用Unstructured或docling；简单的用PyMuPDFLoader就行。

6种切分策略：从粗暴到精细

策略1：固定长度切分（最基础）

按字符数或Token数切，最简单但问题最多：

<span><span leaf="">from</span></span><span leaf="">&nbsp;langchain_text_splitters&nbsp;</span><span><span leaf="">import</span></span><span leaf="">&nbsp;CharacterTextSplitter</span><span leaf=""><br></span><span leaf=""><br></span><span leaf="">splitter = CharacterTextSplitter(</span><span leaf=""><br></span><span leaf="">&nbsp; &nbsp; chunk_size=</span><span><span leaf="">500</span></span><span leaf="">,</span><span leaf=""><br></span><span leaf="">&nbsp; &nbsp; chunk_overlap=</span><span><span leaf="">50</span></span><span leaf="">,</span><span leaf=""><br></span><span leaf="">&nbsp; &nbsp; separator=</span><span><span leaf="">"\n\n"</span></span><span leaf="">, &nbsp;</span><span><span leaf=""># 优先在段落处切</span></span><span leaf=""><br></span><span leaf="">)</span><span leaf=""><br></span><span leaf="">chunks = splitter.split_documents(docs)</span><span leaf=""><br></span>

问题：可能在句子中间切断，破坏语义。适合对精度要求不高的场景。

策略2：递归字符切分（最常用）

LangChain的RecursiveCharacterTextSplitter，按优先级尝试不同的分隔符：

<span><span leaf="">from</span></span><span leaf="">&nbsp;langchain_text_splitters&nbsp;</span><span><span leaf="">import</span></span><span leaf="">&nbsp;RecursiveCharacterTextSplitter</span><span leaf=""><br></span><span leaf=""><br></span><span leaf="">splitter = RecursiveCharacterTextSplitter(</span><span leaf=""><br></span><span leaf="">&nbsp; &nbsp; chunk_size=</span><span><span leaf="">500</span></span><span leaf="">,</span><span leaf=""><br></span><span leaf="">&nbsp; &nbsp; chunk_overlap=</span><span><span leaf="">50</span></span><span leaf="">,</span><span leaf=""><br></span><span leaf="">&nbsp; &nbsp; separators=[</span><span><span leaf="">"\n\n"</span></span><span leaf="">,&nbsp;</span><span><span leaf="">"\n"</span></span><span leaf="">,&nbsp;</span><span><span leaf="">"。"</span></span><span leaf="">,&nbsp;</span><span><span leaf="">"！"</span></span><span leaf="">,&nbsp;</span><span><span leaf="">"？"</span></span><span leaf="">,&nbsp;</span><span><span leaf="">"，"</span></span><span leaf="">,&nbsp;</span><span><span leaf="">" "</span></span><span leaf="">,&nbsp;</span><span><span leaf="">""</span></span><span leaf="">],</span><span leaf=""><br></span><span leaf="">)</span><span leaf=""><br></span><span leaf="">chunks = splitter.split_documents(docs)</span><span leaf=""><br></span>

优先在段落→换行→句子→逗号→空格处切，实在不行才在字符中间切。这是大部分项目的默认选择，够用了。

策略3：语义切分（更精准）

根据语义相似度来决定切分点——如果相邻两句话的语义差异很大，就在那里切：

<span><span leaf="">from</span></span><span leaf="">&nbsp;langchain_experimental.text_splitter&nbsp;</span><span><span leaf="">import</span></span><span leaf="">&nbsp;SemanticChunker</span><span leaf=""><br></span><span><span leaf="">from</span></span><span leaf="">&nbsp;langchain_openai&nbsp;</span><span><span leaf="">import</span></span><span leaf="">&nbsp;OpenAIEmbeddings</span><span leaf=""><br></span><span leaf=""><br></span><span leaf="">splitter = SemanticChunker(</span><span leaf=""><br></span><span leaf="">&nbsp; &nbsp; OpenAIEmbeddings(),</span><span leaf=""><br></span><span leaf="">&nbsp; &nbsp; breakpoint_threshold_type=</span><span><span leaf="">"percentile"</span></span><span leaf="">, &nbsp;</span><span><span leaf=""># 用百分位数判断语义断点</span></span><span leaf=""><br></span><span leaf="">&nbsp; &nbsp; breakpoint_threshold_amount=</span><span><span leaf="">75</span></span><span leaf="">,</span><span leaf=""><br></span><span leaf="">)</span><span leaf=""><br></span><span leaf="">chunks = splitter.split_documents(docs)</span><span leaf=""><br></span>

优点：每个chunk内的语义是连贯的。缺点：慢（要算Embedding），贵（每次切分都调API）。

适合对检索精度要求高、文档量不太大的场景。

策略4：按文档结构切分（最推荐）

利用文档本身的标题、章节、段落结构来切分：

<span><span leaf="">from</span></span><span leaf="">&nbsp;langchain_text_splitters&nbsp;</span><span><span leaf="">import</span></span><span leaf="">&nbsp;MarkdownHeaderTextSplitter</span><span leaf=""><br></span><span leaf=""><br></span><span><span leaf=""># Markdown文档：按标题层级切分</span></span><span leaf=""><br></span><span leaf="">headers_to_split_on = [</span><span leaf=""><br></span><span leaf="">&nbsp; &nbsp; (</span><span><span leaf="">"#"</span></span><span leaf="">,&nbsp;</span><span><span leaf="">"Header 1"</span></span><span leaf="">),</span><span leaf=""><br></span><span leaf="">&nbsp; &nbsp; (</span><span><span leaf="">"##"</span></span><span leaf="">,&nbsp;</span><span><span leaf="">"Header 2"</span></span><span leaf="">),</span><span leaf=""><br></span><span leaf="">&nbsp; &nbsp; (</span><span><span leaf="">"###"</span></span><span leaf="">,&nbsp;</span><span><span leaf="">"Header 3"</span></span><span leaf="">),</span><span leaf=""><br></span><span leaf="">]</span><span leaf=""><br></span><span leaf="">splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers_to_split_on)</span><span leaf=""><br></span><span leaf="">chunks = splitter.split_text(markdown_text)</span><span leaf=""><br></span><span><span leaf=""># 每个chunk会自动携带标题元数据</span></span><span leaf=""><br></span>

<span><span leaf=""># HTML文档：按标签切分</span></span><span leaf=""><br></span><span><span leaf="">from</span></span><span leaf="">&nbsp;langchain_text_splitters&nbsp;</span><span><span leaf="">import</span></span><span leaf="">&nbsp;HTMLSectionSplitter</span><span leaf=""><br></span><span leaf="">sections_to_split_on = [(</span><span><span leaf="">"h1"</span></span><span leaf="">,&nbsp;</span><span><span leaf="">"Header 1"</span></span><span leaf="">), (</span><span><span leaf="">"h2"</span></span><span leaf="">,&nbsp;</span><span><span leaf="">"Header 2"</span></span><span leaf="">)]</span><span leaf=""><br></span><span leaf="">splitter = HTMLSectionSplitter(sections_to_split_on=sections_to_split_on)</span><span leaf=""><br></span>

这种方式最推荐，因为文档的标题层级天然就是语义划分。切出来的chunk语义完整，元数据也保留了，后续检索时可以用元数据做过滤。

策略5：父文档检索切分（大小块配合）

存两种粒度的chunk：小块用于精确检索，大块用于返回给模型：

<span><span leaf="">from</span></span><span leaf="">&nbsp;langchain.retrievers&nbsp;</span><span><span leaf="">import</span></span><span leaf="">&nbsp;ParentDocumentRetriever</span><span leaf=""><br></span><span><span leaf="">from</span></span><span leaf="">&nbsp;langchain_text_splitters&nbsp;</span><span><span leaf="">import</span></span><span leaf="">&nbsp;RecursiveCharacterTextSplitter</span><span leaf=""><br></span><span><span leaf="">from</span></span><span leaf="">&nbsp;langchain_community.storage&nbsp;</span><span><span leaf="">import</span></span><span leaf="">&nbsp;InMemoryStore</span><span leaf=""><br></span><span leaf=""><br></span><span><span leaf=""># 小块：用于检索</span></span><span leaf=""><br></span><span leaf="">child_splitter = RecursiveCharacterTextSplitter(chunk_size=</span><span><span leaf="">200</span></span><span leaf="">)</span><span leaf=""><br></span><span><span leaf=""># 大块：用于返回</span></span><span leaf=""><br></span><span leaf="">parent_splitter = RecursiveCharacterTextSplitter(chunk_size=</span><span><span leaf="">1000</span></span><span leaf="">)</span><span leaf=""><br></span><span leaf=""><br></span><span leaf="">retriever = ParentDocumentRetriever(</span><span leaf=""><br></span><span leaf="">&nbsp; &nbsp; vectorstore=vectorstore, &nbsp; &nbsp; &nbsp;</span><span><span leaf=""># 存小块的向量</span></span><span leaf=""><br></span><span leaf="">&nbsp; &nbsp; docstore=InMemoryStore(), &nbsp; &nbsp;&nbsp;</span><span><span leaf=""># 存大块的原文</span></span><span leaf=""><br></span><span leaf="">&nbsp; &nbsp; child_splitter=child_splitter,</span><span leaf=""><br></span><span leaf="">&nbsp; &nbsp; parent_splitter=parent_splitter,</span><span leaf=""><br></span><span leaf="">)</span><span leaf=""><br></span>

用户检索到小块，但返回给模型的是小块所属的大块。这样既保证了检索精度（小块更精准），又保证了上下文完整（大块不丢信息）。

策略6：自适应切分（最前沿）

根据内容类型自动选择切分策略——表格用表格切分、代码用代码切分、文本用语义切分：

<span><span leaf="">from</span></span><span leaf="">&nbsp;unstructured.partition.auto&nbsp;</span><span><span leaf="">import</span></span><span leaf="">&nbsp;partition</span><span leaf=""><br></span><span><span leaf="">from</span></span><span leaf="">&nbsp;unstructured.staging.base&nbsp;</span><span><span leaf="">import</span></span><span leaf="">&nbsp;elements_to_chunks</span><span leaf=""><br></span><span leaf=""><br></span><span><span leaf=""># 自动识别文档元素类型</span></span><span leaf=""><br></span><span leaf="">elements = partition(filename=</span><span><span leaf="">"report.pdf"</span></span><span leaf="">)</span><span leaf=""><br></span><span><span leaf=""># 表格、代码块、列表等不同元素用不同策略处理</span></span><span leaf=""><br></span><span leaf="">chunks = elements_to_chunks(elements)</span><span leaf=""><br></span>

复杂但效果最好，适合文档类型杂、质量要求高的生产场景。

chunk_size怎么选？一个实用的调参方法

chunk_size没有万能最优值，但有一个系统化的调参方法：

Step 1：从500字符开始试

500-800字符是大部分中文场景的起点。

Step 2：评估检索效果

用一组测试问题跑一遍，看两个指标：

  • 召回率：该找到的chunk找到了没？

  • 上下文完整率：找到的chunk信息够不够回答问题？

Step 3：根据结果调整

| 现象

|

原因

|

调整方向

找到了但信息不完整

|

chunk太小，上下文被切断

|

调大chunk_size

| |

找到了但噪音太多

|

chunk太大，混入无关内容

|

调小chunk_size

| |

根本找不到

|

chunk切断了关键段落

|

增加overlap或换切分策略

| |

找到了但答不对

|

不是切分问题，是检索或提示词问题

|

别调chunk了，查别的环节

|

Step 4：加入元数据

给每个chunk加元数据，后续检索时可以用元数据过滤：

<span><span leaf="">from</span></span><span leaf="">&nbsp;langchain_core.documents&nbsp;</span><span><span leaf="">import</span></span><span leaf="">&nbsp;Document</span><span leaf=""><br></span><span leaf=""><br></span><span leaf="">chunk = Document(</span><span leaf=""><br></span><span leaf="">&nbsp; &nbsp; page_content=</span><span><span leaf="">"公司年假政策：入职满1年可享受5天带薪年假..."</span></span><span leaf="">,</span><span leaf=""><br></span><span leaf="">&nbsp; &nbsp; metadata={</span><span leaf=""><br></span><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</span><span><span leaf="">"source"</span></span><span leaf="">:&nbsp;</span><span><span leaf="">"员工手册v3.2.pdf"</span></span><span leaf="">,</span><span leaf=""><br></span><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</span><span><span leaf="">"page"</span></span><span leaf="">:&nbsp;</span><span><span leaf="">42</span></span><span leaf="">,</span><span leaf=""><br></span><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</span><span><span leaf="">"section"</span></span><span leaf="">:&nbsp;</span><span><span leaf="">"假期管理"</span></span><span leaf="">,</span><span leaf=""><br></span><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</span><span><span leaf="">"doc_type"</span></span><span leaf="">:&nbsp;</span><span><span leaf="">"policy"</span></span><span leaf="">,</span><span leaf=""><br></span><span leaf="">&nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</span><span><span leaf="">"update_date"</span></span><span leaf="">:&nbsp;</span><span><span leaf="">"2025-03-15"</span></span><span leaf="">,</span><span leaf=""><br></span><span leaf="">&nbsp; &nbsp; }</span><span leaf=""><br></span><span leaf="">)</span><span leaf=""><br></span>

检索时可以过滤：只搜"假期管理"章节的、只搜2025年更新的文档。这样精确率直接上一个台阶。

不同场景的推荐配置

| 场景

|

推荐切分

|

chunk_size

|

overlap

|

特殊处理

法律合同

|

按条款切分

|

800-1200

|

100

|

保留条款编号

| |

技术文档

|

按标题结构

|

500-800

|

50

|

代码块单独切

| |

FAQ问答

|

按问答对

|

200-400

|

0

|

一个Q-A就是一个chunk

| |

论文

|

按章节

|

800-1000

|

100

|

摘要和结论单独存

| |

新闻资讯

|

递归字符

|

300-500

|

30

|

加时间元数据

|

我踩过的切分坑

1. chunk_overlap设太大

overlap设100意味着每个chunk有100字符是跟相邻chunk重复的。如果你存10万个chunk，相当于浪费了大量的存储和检索开销。overlap建议设chunk_size的5%-10% ，够防止关键信息被切断就行。

2. 中文分词没考虑

RecursiveCharacterTextSplitter的默认separators是英文优先的。中文文档建议加上"。""！""？""；“等中文标点：

<span leaf="">separators=[</span><span><span leaf="">"\n\n"</span></span><span leaf="">,&nbsp;</span><span><span leaf="">"\n"</span></span><span leaf="">,&nbsp;</span><span><span leaf="">"。"</span></span><span leaf="">,&nbsp;</span><span><span leaf="">"！"</span></span><span leaf="">,&nbsp;</span><span><span leaf="">"？"</span></span><span leaf="">,&nbsp;</span><span><span leaf="">"；"</span></span><span leaf="">,&nbsp;</span><span><span leaf="">"，"</span></span><span leaf="">,&nbsp;</span><span><span leaf="">" "</span></span><span leaf="">,&nbsp;</span><span><span leaf="">""</span></span><span leaf="">]</span><span leaf=""><br></span>

3. 元数据丢了

有些切分器会把原文档的元数据丢掉。切分后一定要检查：chunk.metadata里有没有source、page等信息。没有的话检索时就没法做过滤。

4. 表格被切碎

合同、财报里的表格，按字符切分后完全不可用。要么用能识别表格的解析器，要么把表格转成Markdown/HTML格式再存。

下篇预告

下一篇讲Embedding模型选型和向量数据库对比——你的知识库用哪种Embedding、用哪个向量库，直接决定了检索的天花板。我会对比OpenAI、BGE、Jina等主流Embedding，以及Chroma、FAISS、Milvus、Qdrant等向量库的选型建议。

你现在用的切分方式是哪种？chunk_size设的多少？效果怎么样？评论区聊聊。