图表、表格、PDF图文，多模态RAG全搞定
前面7篇RAG文章都在处理文本——把文档切分、向量化、检索、生成。但真实世界的文档长什么样？PDF里有图表，Word里有截图，PPT里有流程图，Excel里有数据透视表。这些非文本内容，纯文本RAG根本检索不到。
我之前做过一个项目，客户的PDF报告里有大量数据图表——柱状图、折线图、饼图。用户问"2025年Q3各区域销售额对比"，文本RAG找到的是报告里"如图3所示，Q3销售额增长15%“这句话，但图表里的具体数据（华北区300万、华东区450万）根本没检索到。因为图表是图片，文本切分时直接被跳过了。
多模态RAG解决的就是这个问题：让RAG不仅能读文字，还能看懂图片、理解表格、检索图表里的信息。
这篇把多模态RAG的三条技术路线拆透：多模态Embedding、视觉检索（ColPali）、表格抽取方案。每条路线适合不同场景，别搞混。
多模态RAG的三条路线
先搞清楚有哪些方案，再选适合你的：
| 路线
|
核心思路
|
优势
|
劣势
|
适合场景
文本描述法
|
先把图片/表格转成文字描述，再用文本RAG
|
最简单，复用现有RAG
|
信息有损，描述可能遗漏细节
|
图片有alt文本、表格有标题
| |
多模态Embedding
|
用CLIP等模型把图片和文本映射到同一向量空间
|
原生支持图文混合检索
|
Embedding维度大，精度不如纯文本
|
图片+文本混合知识库
| |
视觉检索（ColPali）
|
直接把PDF页面当图片，用视觉模型做检索
|
不丢任何信息，图表全保留
|
计算量大，需要GPU
|
PDF报告、扫描文档
|
选择建议：
-
你的文档图片有描述文字（如产品图配说明）→ 路线1，最省事
-
你的知识库是图片+文本混合（如电商商品图+描述）→ 路线2，CLIP方案
-
你的文档是PDF报告，大量图表无文字描述 → 路线3，ColPali方案
下面逐个拆。
路线一：文本描述法
最简单的方案：把图片和表格转成文字描述，然后纯走文本RAG的流程。
图片转文字描述
用多模态LLM（GPT-4o、Qwen-VL）给图片生成描述：
<span><span leaf="">from</span></span><span leaf=""> langchain_core.messages </span><span><span leaf="">import</span></span><span leaf=""> HumanMessage</span><span leaf=""><br></span><span><span leaf="">from</span></span><span leaf=""> langchain_openai </span><span><span leaf="">import</span></span><span leaf=""> ChatOpenAI</span><span leaf=""><br></span><span leaf=""><br></span><span><span><span leaf="">def</span></span><span leaf=""> </span><span><span leaf="">image_to_text_description</span></span><span><span leaf="">(image_path, vision_llm)</span></span><span leaf="">:</span></span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">"""用视觉大模型把图片转成文字描述</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> 这是最简单的方案：图片 → LLM看图说话 → 文字描述 → 文本RAG</span><span leaf=""><br></span><span leaf=""> 适合：图片本身有明确语义（产品图、流程图、截图）</span><span leaf=""><br></span><span leaf=""> 不适合：图片信息太复杂（密密麻麻的数据图表）</span><span leaf=""><br></span><span leaf=""> """</span></span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">with</span></span><span leaf=""> open(image_path, </span><span><span leaf="">"rb"</span></span><span leaf="">) </span><span><span leaf="">as</span></span><span leaf=""> f:</span><span leaf=""><br></span><span leaf=""> image_data = f.read()</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">import</span></span><span leaf=""> base64</span><span leaf=""><br></span><span leaf=""> image_b64 = base64.b64encode(image_data).decode()</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> message = HumanMessage(content=[</span><span leaf=""><br></span><span leaf=""> {</span><span><span leaf="">"type"</span></span><span leaf="">: </span><span><span leaf="">"text"</span></span><span leaf="">, </span><span><span leaf="">"text"</span></span><span leaf="">: </span><span><span leaf="">"""请详细描述这张图片的内容，要求：</span><span leaf=""><br></span><span leaf="">1. 描述图片的类型（柱状图/折线图/流程图/截图/照片等）</span><span leaf=""><br></span><span leaf="">2. 列出所有可见的文字、标签、数值</span><span leaf=""><br></span><span leaf="">3. 描述数据趋势或关键信息</span><span leaf=""><br></span><span leaf="">4. 用一段连贯的文字输出，不要用列表</span><span leaf=""><br></span><span leaf=""><br></span><span leaf="">描述："""</span></span><span leaf="">},</span><span leaf=""><br></span><span leaf=""> {</span><span><span leaf="">"type"</span></span><span leaf="">: </span><span><span leaf="">"image_url"</span></span><span leaf="">, </span><span><span leaf="">"image_url"</span></span><span leaf="">: {</span><span><span leaf="">"url"</span></span><span leaf="">: </span><span><span leaf="">f"data:image/png;base64,</span><span><span leaf="">{image_b64}</span></span><span leaf="">"</span></span><span leaf="">}},</span><span leaf=""><br></span><span leaf=""> ])</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> response = vision_llm.invoke([message])</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">return</span></span><span leaf=""> response.content</span><span leaf=""><br></span><span leaf=""><br></span><span leaf=""><br></span><span><span leaf=""># 批量处理PDF中的图片</span></span><span leaf=""><br></span><span><span><span leaf="">def</span></span><span leaf=""> </span><span><span leaf="">extract_images_from_pdf</span></span><span><span leaf="">(pdf_path, output_dir)</span></span><span leaf="">:</span></span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">"""从PDF中提取所有图片"""</span></span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">import</span></span><span leaf=""> fitz </span><span><span leaf=""># PyMuPDF</span></span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">import</span></span><span leaf=""> os</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> os.makedirs(output_dir, exist_ok=</span><span><span leaf="">True</span></span><span leaf="">)</span><span leaf=""><br></span><span leaf=""> doc = fitz.open(pdf_path)</span><span leaf=""><br></span><span leaf=""> image_paths = []</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">for</span></span><span leaf=""> page_num </span><span><span leaf="">in</span></span><span leaf=""> range(len(doc)):</span><span leaf=""><br></span><span leaf=""> page = doc[page_num]</span><span leaf=""><br></span><span leaf=""> images = page.get_images(full=</span><span><span leaf="">True</span></span><span leaf="">)</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">for</span></span><span leaf=""> img_idx, img </span><span><span leaf="">in</span></span><span leaf=""> enumerate(images):</span><span leaf=""><br></span><span leaf=""> xref = img[</span><span><span leaf="">0</span></span><span leaf="">]</span><span leaf=""><br></span><span leaf=""> base_image = doc.extract_image(xref)</span><span leaf=""><br></span><span leaf=""> image_bytes = base_image[</span><span><span leaf="">"image"</span></span><span leaf="">]</span><span leaf=""><br></span><span leaf=""> image_ext = base_image[</span><span><span leaf="">"ext"</span></span><span leaf="">]</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> img_path = </span><span><span leaf="">f"</span><span><span leaf="">{output_dir}</span></span><span leaf="">/page</span><span><span leaf="">{page_num}</span></span><span leaf="">_img</span><span><span leaf="">{img_idx}</span></span><span leaf="">.</span><span><span leaf="">{image_ext}</span></span><span leaf="">"</span></span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">with</span></span><span leaf=""> open(img_path, </span><span><span leaf="">"wb"</span></span><span leaf="">) </span><span><span leaf="">as</span></span><span leaf=""> f:</span><span leaf=""><br></span><span leaf=""> f.write(image_bytes)</span><span leaf=""><br></span><span leaf=""> image_paths.append((img_path, page_num))</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> doc.close()</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">return</span></span><span leaf=""> image_paths</span><span leaf=""><br></span>
表格转结构化文本
PDF里的表格用PyMuPDF或pdfplumber提取，转成Markdown格式：
<span><span leaf="">import</span></span><span leaf=""> pdfplumber</span><span leaf=""><br></span><span leaf=""><br></span><span><span><span leaf="">def</span></span><span leaf=""> </span><span><span leaf="">extract_tables_from_pdf</span></span><span><span leaf="">(pdf_path)</span></span><span leaf="">:</span></span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">"""从PDF中提取表格，转成Markdown格式</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> 表格转文本的关键：保持结构信息。</span><span leaf=""><br></span><span leaf=""> Markdown表格比纯文本好检索，因为保留了行列关系。</span><span leaf=""><br></span><span leaf=""> """</span></span><span leaf=""><br></span><span leaf=""> all_tables = []</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">with</span></span><span leaf=""> pdfplumber.open(pdf_path) </span><span><span leaf="">as</span></span><span leaf=""> pdf:</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">for</span></span><span leaf=""> page_num, page </span><span><span leaf="">in</span></span><span leaf=""> enumerate(pdf.pages):</span><span leaf=""><br></span><span leaf=""> tables = page.extract_tables()</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">for</span></span><span leaf=""> table_idx, table </span><span><span leaf="">in</span></span><span leaf=""> enumerate(tables):</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">if</span></span><span leaf=""> </span><span><span leaf="">not</span></span><span leaf=""> table </span><span><span leaf="">or</span></span><span leaf=""> len(table) < </span><span><span leaf="">2</span></span><span leaf="">:</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">continue</span></span><span leaf=""> </span><span><span leaf=""># 空表格或只有表头</span></span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> </span><span><span leaf=""># 转成Markdown表格</span></span><span leaf=""><br></span><span leaf=""> markdown_table = table_to_markdown(table)</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> all_tables.append({</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">"content"</span></span><span leaf="">: markdown_table,</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">"page"</span></span><span leaf="">: page_num,</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">"table_idx"</span></span><span leaf="">: table_idx,</span><span leaf=""><br></span><span leaf=""> })</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">return</span></span><span leaf=""> all_tables</span><span leaf=""><br></span><span leaf=""><br></span><span leaf=""><br></span><span><span><span leaf="">def</span></span><span leaf=""> </span><span><span leaf="">table_to_markdown</span></span><span><span leaf="">(table)</span></span><span leaf="">:</span></span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">"""把二维数组转成Markdown表格格式"""</span></span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">if</span></span><span leaf=""> </span><span><span leaf="">not</span></span><span leaf=""> table:</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">return</span></span><span leaf=""> </span><span><span leaf="">""</span></span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> </span><span><span leaf=""># 第一行是表头</span></span><span leaf=""><br></span><span leaf=""> header = table[</span><span><span leaf="">0</span></span><span leaf="">]</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf=""># 分隔行</span></span><span leaf=""><br></span><span leaf=""> separator = [</span><span><span leaf="">"---"</span></span><span leaf="">] * len(header)</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf=""># 数据行</span></span><span leaf=""><br></span><span leaf=""> rows = table[</span><span><span leaf="">1</span></span><span leaf="">:]</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> lines = []</span><span leaf=""><br></span><span leaf=""> lines.append(</span><span><span leaf="">"| "</span></span><span leaf=""> + </span><span><span leaf="">" | "</span></span><span leaf="">.join(str(cell </span><span><span leaf="">or</span></span><span leaf=""> </span><span><span leaf="">""</span></span><span leaf="">) </span><span><span leaf="">for</span></span><span leaf=""> cell </span><span><span leaf="">in</span></span><span leaf=""> header) + </span><span><span leaf="">" |"</span></span><span leaf="">)</span><span leaf=""><br></span><span leaf=""> lines.append(</span><span><span leaf="">"| "</span></span><span leaf=""> + </span><span><span leaf="">" | "</span></span><span leaf="">.join(separator) + </span><span><span leaf="">" |"</span></span><span leaf="">)</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">for</span></span><span leaf=""> row </span><span><span leaf="">in</span></span><span leaf=""> rows:</span><span leaf=""><br></span><span leaf=""> lines.append(</span><span><span leaf="">"| "</span></span><span leaf=""> + </span><span><span leaf="">" | "</span></span><span leaf="">.join(str(cell </span><span><span leaf="">or</span></span><span leaf=""> </span><span><span leaf="">""</span></span><span leaf="">) </span><span><span leaf="">for</span></span><span leaf=""> cell </span><span><span leaf="">in</span></span><span leaf=""> row) + </span><span><span leaf="">" |"</span></span><span leaf="">)</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">return</span></span><span leaf=""> </span><span><span leaf="">"\n"</span></span><span leaf="">.join(lines)</span><span leaf=""><br></span><span leaf=""><br></span><span leaf=""><br></span><span><span leaf=""># 示例输出：</span></span><span leaf=""><br></span><span><span leaf=""># | 区域 | Q3销售额 | 同比增长 |</span></span><span leaf=""><br></span><span><span leaf=""># | --- | --- | --- |</span></span><span leaf=""><br></span><span><span leaf=""># | 华北 | 300万 | 15% |</span></span><span leaf=""><br></span><span><span leaf=""># | 华东 | 450万 | 22% |</span></span><span leaf=""><br></span>
坑1：LLM看图说数据容易编。GPT-4o看柱状图读数值，经常读个大概——柱子高度差不多就给你四舍五入。如果你需要精确数值，别让LLM读图，用OCR或表格提取工具直接拿数据。LLM适合描述"图表画的是什么”，不适合精确读数。
坑2：PDF表格提取不完整。pdfplumber对合并单元格、跨页表格支持不好。如果你的PDF表格很复杂，考虑用Camelot（专门做PDF表格提取）或直接用LLM看页面截图提取。
路线二：多模态Embedding
文本描述法有信息损失——LLM的描述不一定能覆盖图片所有细节。多模态Embedding直接把图片和文本映射到同一个向量空间，实现"用文字搜图片"或"用图片搜图片"。
CLIP模型：图文共享向量空间
CLIP（Contrastive Language-Image Pre-training）是OpenAI的多模态模型，把图片和文本编码到同一个768维（或512维）向量空间。
<span><span leaf="">from</span></span><span leaf=""> transformers </span><span><span leaf="">import</span></span><span leaf=""> CLIPModel, CLIPProcessor, CLIPTokenizer</span><span leaf=""><br></span><span><span leaf="">import</span></span><span leaf=""> torch</span><span leaf=""><br></span><span><span leaf="">from</span></span><span leaf=""> PIL </span><span><span leaf="">import</span></span><span leaf=""> Image</span><span leaf=""><br></span><span leaf=""><br></span><span><span><span leaf="">class</span></span><span leaf=""> </span><span><span leaf="">CLIPEmbedder</span></span><span leaf="">:</span></span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">"""CLIP多模态Embedding：文本和图片共享向量空间"""</span></span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> </span><span><span><span leaf="">def</span></span><span leaf=""> </span><span><span leaf="">__init__</span></span><span><span leaf="">(self, model_name=</span><span><span leaf="">"openai/clip-vit-base-patch32"</span></span><span leaf="">)</span></span><span leaf="">:</span></span><span leaf=""><br></span><span leaf=""> self.model = CLIPModel.from_pretrained(model_name)</span><span leaf=""><br></span><span leaf=""> self.processor = CLIPProcessor.from_pretrained(model_name)</span><span leaf=""><br></span><span leaf=""> self.tokenizer = CLIPTokenizer.from_pretrained(model_name)</span><span leaf=""><br></span><span leaf=""> self.model.eval()</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span><span leaf=""> @torch.no_grad()</span></span><span leaf=""><br></span><span leaf=""> </span><span><span><span leaf="">def</span></span><span leaf=""> </span><span><span leaf="">embed_text</span></span><span><span leaf="">(self, text)</span></span><span leaf="">:</span></span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">"""文本 → 向量"""</span></span><span leaf=""><br></span><span leaf=""> inputs = self.tokenizer(text, return_tensors=</span><span><span leaf="">"pt"</span></span><span leaf="">, </span><span leaf=""><br></span><span leaf=""> padding=</span><span><span leaf="">True</span></span><span leaf="">, truncation=</span><span><span leaf="">True</span></span><span leaf="">, max_length=</span><span><span leaf="">77</span></span><span leaf="">)</span><span leaf=""><br></span><span leaf=""> outputs = self.model.get_text_features(**inputs)</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf=""># L2归一化，方便余弦相似度计算</span></span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">return</span></span><span leaf=""> outputs / outputs.norm(dim=</span><span><span leaf="">-1</span></span><span leaf="">, keepdim=</span><span><span leaf="">True</span></span><span leaf="">)</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span><span leaf=""> @torch.no_grad()</span></span><span leaf=""><br></span><span leaf=""> </span><span><span><span leaf="">def</span></span><span leaf=""> </span><span><span leaf="">embed_image</span></span><span><span leaf="">(self, image_path)</span></span><span leaf="">:</span></span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">"""图片 → 向量"""</span></span><span leaf=""><br></span><span leaf=""> image = Image.open(image_path).convert(</span><span><span leaf="">"RGB"</span></span><span leaf="">)</span><span leaf=""><br></span><span leaf=""> inputs = self.processor(images=image, return_tensors=</span><span><span leaf="">"pt"</span></span><span leaf="">)</span><span leaf=""><br></span><span leaf=""> outputs = self.model.get_image_features(**inputs)</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">return</span></span><span leaf=""> outputs / outputs.norm(dim=</span><span><span leaf="">-1</span></span><span leaf="">, keepdim=</span><span><span leaf="">True</span></span><span leaf="">)</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span><span leaf=""> @torch.no_grad()</span></span><span leaf=""><br></span><span leaf=""> </span><span><span><span leaf="">def</span></span><span leaf=""> </span><span><span leaf="">embed_text_batch</span></span><span><span leaf="">(self, texts)</span></span><span leaf="">:</span></span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">"""批量文本Embedding"""</span></span><span leaf=""><br></span><span leaf=""> inputs = self.tokenizer(texts, return_tensors=</span><span><span leaf="">"pt"</span></span><span leaf="">,</span><span leaf=""><br></span><span leaf=""> padding=</span><span><span leaf="">True</span></span><span leaf="">, truncation=</span><span><span leaf="">True</span></span><span leaf="">, max_length=</span><span><span leaf="">77</span></span><span leaf="">)</span><span leaf=""><br></span><span leaf=""> outputs = self.model.get_text_features(**inputs)</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">return</span></span><span leaf=""> outputs / outputs.norm(dim=</span><span><span leaf="">-1</span></span><span leaf="">, keepdim=</span><span><span leaf="">True</span></span><span leaf="">)</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span><span leaf=""> @torch.no_grad()</span></span><span leaf=""><br></span><span leaf=""> </span><span><span><span leaf="">def</span></span><span leaf=""> </span><span><span leaf="">embed_image_batch</span></span><span><span leaf="">(self, image_paths)</span></span><span leaf="">:</span></span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">"""批量图片Embedding"""</span></span><span leaf=""><br></span><span leaf=""> images = [Image.open(p).convert(</span><span><span leaf="">"RGB"</span></span><span leaf="">) </span><span><span leaf="">for</span></span><span leaf=""> p </span><span><span leaf="">in</span></span><span leaf=""> image_paths]</span><span leaf=""><br></span><span leaf=""> inputs = self.processor(images=images, return_tensors=</span><span><span leaf="">"pt"</span></span><span leaf="">)</span><span leaf=""><br></span><span leaf=""> outputs = self.model.get_image_features(**inputs)</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">return</span></span><span leaf=""> outputs / outputs.norm(dim=</span><span><span leaf="">-1</span></span><span leaf="">, keepdim=</span><span><span leaf="">True</span></span><span leaf="">)</span><span leaf=""><br></span><span leaf=""><br></span><span leaf=""><br></span><span><span leaf=""># 使用示例</span></span><span leaf=""><br></span><span leaf="">embedder = CLIPEmbedder()</span><span leaf=""><br></span><span leaf=""><br></span><span><span leaf=""># 用文字搜图片</span></span><span leaf=""><br></span><span leaf="">query_vec = embedder.embed_text(</span><span><span leaf="">"一张展示销售数据的柱状图"</span></span><span leaf="">)</span><span leaf=""><br></span><span leaf="">image_vecs = embedder.embed_image_batch([</span><span><span leaf="">"chart1.png"</span></span><span leaf="">, </span><span><span leaf="">"chart2.png"</span></span><span leaf="">, </span><span><span leaf="">"photo1.png"</span></span><span leaf="">])</span><span leaf=""><br></span><span leaf=""><br></span><span><span leaf=""># 计算相似度</span></span><span leaf=""><br></span><span leaf="">similarities = (query_vec @ image_vecs.T).squeeze(</span><span><span leaf="">0</span></span><span leaf="">)</span><span leaf=""><br></span><span><span leaf=""># → chart1.png相似度最高（如果它确实是柱状图）</span></span><span leaf=""><br></span>
多模态向量库：图文混合检索
<span><span leaf="">from</span></span><span leaf=""> langchain_core.documents </span><span><span leaf="">import</span></span><span leaf=""> Document</span><span leaf=""><br></span><span><span leaf="">from</span></span><span leaf=""> langchain_community.vectorstores </span><span><span leaf="">import</span></span><span leaf=""> Chroma</span><span leaf=""><br></span><span leaf=""><br></span><span><span><span leaf="">class</span></span><span leaf=""> </span><span><span leaf="">MultimodalVectorStore</span></span><span leaf="">:</span></span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">"""多模态向量库：文本和图片存在同一个ChromaDB里"""</span></span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> </span><span><span><span leaf="">def</span></span><span leaf=""> </span><span><span leaf="">__init__</span></span><span><span leaf="">(self, embedder)</span></span><span leaf="">:</span></span><span leaf=""><br></span><span leaf=""> self.embedder = embedder</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf=""># ChromaDB默认用文本embedding，这里需要自定义embedding函数</span></span><span leaf=""><br></span><span leaf=""> </span><span><span leaf=""># 简化版：用CLIP的文本embedding做query，图片embedding做索引</span></span><span leaf=""><br></span><span leaf=""> self.text_embeddings = {} </span><span><span leaf=""># 文本内容 → 向量</span></span><span leaf=""><br></span><span leaf=""> self.image_embeddings = {} </span><span><span leaf=""># 图片路径 → 向量</span></span><span leaf=""><br></span><span leaf=""> self.documents = [] </span><span><span leaf=""># 所有文档（文本+图片）</span></span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> </span><span><span><span leaf="">def</span></span><span leaf=""> </span><span><span leaf="">add_texts</span></span><span><span leaf="">(self, texts, metadatas=None)</span></span><span leaf="">:</span></span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">"""添加文本"""</span></span><span leaf=""><br></span><span leaf=""> embeddings = self.embedder.embed_text_batch(texts)</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">for</span></span><span leaf=""> i, (text, emb) </span><span><span leaf="">in</span></span><span leaf=""> enumerate(zip(texts, embeddings)):</span><span leaf=""><br></span><span leaf=""> doc_id = </span><span><span leaf="">f"text_</span><span><span leaf="">{len(self.documents)}</span></span><span leaf="">"</span></span><span leaf=""><br></span><span leaf=""> self.text_embeddings[doc_id] = emb</span><span leaf=""><br></span><span leaf=""> self.documents.append({</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">"id"</span></span><span leaf="">: doc_id,</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">"content"</span></span><span leaf="">: text,</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">"type"</span></span><span leaf="">: </span><span><span leaf="">"text"</span></span><span leaf="">,</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">"embedding"</span></span><span leaf="">: emb,</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">"metadata"</span></span><span leaf="">: metadatas[i] </span><span><span leaf="">if</span></span><span leaf=""> metadatas </span><span><span leaf="">else</span></span><span leaf=""> {},</span><span leaf=""><br></span><span leaf=""> })</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> </span><span><span><span leaf="">def</span></span><span leaf=""> </span><span><span leaf="">add_images</span></span><span><span leaf="">(self, image_paths, descriptions=None, metadatas=None)</span></span><span leaf="">:</span></span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">"""添加图片"""</span></span><span leaf=""><br></span><span leaf=""> embeddings = self.embedder.embed_image_batch(image_paths)</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">for</span></span><span leaf=""> i, (path, emb) </span><span><span leaf="">in</span></span><span leaf=""> enumerate(zip(image_paths, embeddings)):</span><span leaf=""><br></span><span leaf=""> doc_id = </span><span><span leaf="">f"image_</span><span><span leaf="">{len(self.documents)}</span></span><span leaf="">"</span></span><span leaf=""><br></span><span leaf=""> self.image_embeddings[doc_id] = emb</span><span leaf=""><br></span><span leaf=""> self.documents.append({</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">"id"</span></span><span leaf="">: doc_id,</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">"content"</span></span><span leaf="">: descriptions[i] </span><span><span leaf="">if</span></span><span leaf=""> descriptions </span><span><span leaf="">else</span></span><span leaf=""> </span><span><span leaf="">f"图片: </span><span><span leaf="">{path}</span></span><span leaf="">"</span></span><span leaf="">,</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">"type"</span></span><span leaf="">: </span><span><span leaf="">"image"</span></span><span leaf="">,</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">"path"</span></span><span leaf="">: path,</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">"embedding"</span></span><span leaf="">: emb,</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">"metadata"</span></span><span leaf="">: metadatas[i] </span><span><span leaf="">if</span></span><span leaf=""> metadatas </span><span><span leaf="">else</span></span><span leaf=""> {},</span><span leaf=""><br></span><span leaf=""> })</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> </span><span><span><span leaf="">def</span></span><span leaf=""> </span><span><span leaf="">search</span></span><span><span leaf="">(self, query, top_k=</span><span><span leaf="">5</span></span><span leaf="">)</span></span><span leaf="">:</span></span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">"""混合检索：用文本query同时搜文本和图片"""</span></span><span leaf=""><br></span><span leaf=""> query_vec = self.embedder.embed_text(query)</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> results = []</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">for</span></span><span leaf=""> doc </span><span><span leaf="">in</span></span><span leaf=""> self.documents:</span><span leaf=""><br></span><span leaf=""> sim = (query_vec @ doc[</span><span><span leaf="">"embedding"</span></span><span leaf="">]).item()</span><span leaf=""><br></span><span leaf=""> results.append((doc, sim))</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> results.sort(key=</span><span><span leaf="">lambda</span></span><span leaf=""> x: x[</span><span><span leaf="">1</span></span><span leaf="">], reverse=</span><span><span leaf="">True</span></span><span leaf="">)</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">return</span></span><span leaf=""> results[:top_k]</span><span leaf=""><br></span><span leaf=""><br></span><span leaf=""><br></span><span><span leaf=""># 使用示例</span></span><span leaf=""><br></span><span leaf="">store = MultimodalVectorStore(embedder)</span><span leaf=""><br></span><span leaf=""><br></span><span><span leaf=""># 添加文本和图片</span></span><span leaf=""><br></span><span leaf="">store.add_texts([</span><span><span leaf="">"公司2025年Q3销售额增长15%"</span></span><span leaf="">])</span><span leaf=""><br></span><span leaf="">store.add_images([</span><span><span leaf="">"sales_chart.png"</span></span><span leaf="">], descriptions=[</span><span><span leaf="">"Q3各区域销售额柱状图"</span></span><span leaf="">])</span><span leaf=""><br></span><span leaf=""><br></span><span><span leaf=""># 检索：文字搜图文</span></span><span leaf=""><br></span><span leaf="">results = store.search(</span><span><span leaf="">"Q3销售额数据"</span></span><span leaf="">)</span><span leaf=""><br></span><span><span leaf=""># → 同时返回文本片段和柱状图图片</span></span><span leaf=""><br></span>
坑3：CLIP的中文支持差。原版CLIP主要用英文数据训练，中文query的检索效果明显不如英文。解决：用Chinese-CLIP（阿里达摩院开源）或BGE-VL（BAAI的多模态Embedding），对中文支持好得多。
<span><span leaf=""># Chinese-CLIP：中文多模态Embedding</span></span><span leaf=""><br></span><span><span leaf=""># 模型名：OFA-Sys/chinese-clip-vit-base-patch16</span></span><span leaf=""><br></span><span><span leaf=""># 用法和CLIP一样，但中文检索效果好很多</span></span><span leaf=""><br></span>
坑4：CLIP对细粒度检索不行。CLIP擅长"找一张跟这个描述匹配的图"（如"找一只猫的图片"），但不擅长"找包含某个具体数值的图表"。因为CLIP的训练目标是图文匹配，不是OCR。如果你的图片是数据图表需要精确检索数值，回到路线一用LLM生成描述。
路线三：ColPali视觉检索
ColPali是2024年提出的方法，思路非常巧妙：不做任何文本提取和表格解析，直接把PDF每一页当图片，用视觉语言模型做检索。
为什么ColPali是革命性的
传统PDF RAG的流程：PDF → 文本提取 → 切分 → Embedding → 检索。每一步都有信息损失：文本提取可能丢图表，切分可能截断表格，Embedding可能丢失结构信息。
ColPali的流程：PDF → 页面截图 → 视觉Embedding → 检索。零信息损失——页面上所有内容（文字、图表、表格、布局）都保留在图片里。
<span><span leaf=""># ColPali需要安装：</span></span><span leaf=""><br></span><span><span leaf=""># pip install colpali-engine</span></span><span leaf=""><br></span><span leaf=""><br></span><span><span leaf="">from</span></span><span leaf=""> colpali_engine.models </span><span><span leaf="">import</span></span><span leaf=""> ColPali, ColPaliProcessor</span><span leaf=""><br></span><span><span leaf="">from</span></span><span leaf=""> colpali_engine.utils </span><span><span leaf="">import</span></span><span leaf=""> page_from_image_path</span><span leaf=""><br></span><span><span leaf="">import</span></span><span leaf=""> torch</span><span leaf=""><br></span><span leaf=""><br></span><span><span><span leaf="">class</span></span><span leaf=""> </span><span><span leaf="">ColPaliRetriever</span></span><span leaf="">:</span></span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">"""ColPali视觉检索：把PDF页面当图片做检索</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> 适合：PDF报告、扫描文档、图表密集型文档</span><span leaf=""><br></span><span leaf=""> 不适合：纯文本文档（杀鸡用牛刀）</span><span leaf=""><br></span><span leaf=""> """</span></span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> </span><span><span><span leaf="">def</span></span><span leaf=""> </span><span><span leaf="">__init__</span></span><span><span leaf="">(self, model_name=</span><span><span leaf="">"vidore/colpali-v1.2"</span></span><span leaf="">)</span></span><span leaf="">:</span></span><span leaf=""><br></span><span leaf=""> self.model = ColPali.from_pretrained(</span><span leaf=""><br></span><span leaf=""> model_name,</span><span leaf=""><br></span><span leaf=""> torch_dtype=torch.float32, </span><span><span leaf=""># CPU用float32，GPU可用float16</span></span><span leaf=""><br></span><span leaf=""> device_map=</span><span><span leaf="">"cpu"</span></span><span leaf="">, </span><span><span leaf=""># 没GPU用CPU，有GPU改"cuda"</span></span><span leaf=""><br></span><span leaf=""> )</span><span leaf=""><br></span><span leaf=""> self.processor = ColPaliProcessor.from_pretrained(model_name)</span><span leaf=""><br></span><span leaf=""> self.model.eval()</span><span leaf=""><br></span><span leaf=""> self.page_embeddings = [] </span><span><span leaf=""># 存储所有页面的embedding</span></span><span leaf=""><br></span><span leaf=""> self.page_info = [] </span><span><span leaf=""># 存储页面元信息</span></span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span><span leaf=""> @torch.no_grad()</span></span><span leaf=""><br></span><span leaf=""> </span><span><span><span leaf="">def</span></span><span leaf=""> </span><span><span leaf="">index_pdf</span></span><span><span leaf="">(self, pdf_path)</span></span><span leaf="">:</span></span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">"""把PDF每一页转成图片并建索引"""</span></span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">import</span></span><span leaf=""> fitz</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> doc = fitz.open(pdf_path)</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">for</span></span><span leaf=""> page_num </span><span><span leaf="">in</span></span><span leaf=""> range(len(doc)):</span><span leaf=""><br></span><span leaf=""> page = doc[page_num]</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> </span><span><span leaf=""># PDF页面 → 高清图片（150 DPI够用，200 DPI更清晰但更慢）</span></span><span leaf=""><br></span><span leaf=""> pix = page.get_pixmap(dpi=</span><span><span leaf="">150</span></span><span leaf="">)</span><span leaf=""><br></span><span leaf=""> img_path = </span><span><span leaf="">f"/tmp/page_</span><span><span leaf="">{page_num}</span></span><span leaf="">.png"</span></span><span leaf=""><br></span><span leaf=""> pix.save(img_path)</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">from</span></span><span leaf=""> PIL </span><span><span leaf="">import</span></span><span leaf=""> Image</span><span leaf=""><br></span><span leaf=""> image = Image.open(img_path)</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> </span><span><span leaf=""># 视觉Embedding</span></span><span leaf=""><br></span><span leaf=""> inputs = self.processor(images=[image], return_tensors=</span><span><span leaf="">"pt"</span></span><span leaf="">)</span><span leaf=""><br></span><span leaf=""> embeddings = self.model(**inputs)</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> </span><span><span leaf=""># ColPali输出的是多向量（每个patch一个向量），不是单向量</span></span><span leaf=""><br></span><span leaf=""> </span><span><span leaf=""># 存储所有patch向量，检索时用MaxSim计算</span></span><span leaf=""><br></span><span leaf=""> self.page_embeddings.append(embeddings)</span><span leaf=""><br></span><span leaf=""> self.page_info.append({</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">"pdf_path"</span></span><span leaf="">: pdf_path,</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">"page_num"</span></span><span leaf="">: page_num,</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">"image_path"</span></span><span leaf="">: img_path,</span><span leaf=""><br></span><span leaf=""> })</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> doc.close()</span><span leaf=""><br></span><span leaf=""> print(</span><span><span leaf="">f"索引完成：</span><span><span leaf="">{len(self.page_embeddings)}</span></span><span leaf="">页"</span></span><span leaf="">)</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span><span leaf=""> @torch.no_grad()</span></span><span leaf=""><br></span><span leaf=""> </span><span><span><span leaf="">def</span></span><span leaf=""> </span><span><span leaf="">search</span></span><span><span leaf="">(self, query, top_k=</span><span><span leaf="">3</span></span><span leaf="">)</span></span><span leaf="">:</span></span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">"""用文本query检索最相关的页面"""</span></span><span leaf=""><br></span><span leaf=""> </span><span><span leaf=""># 文本 → Embedding</span></span><span leaf=""><br></span><span leaf=""> inputs = self.processor(text=[query], return_tensors=</span><span><span leaf="">"pt"</span></span><span leaf="">)</span><span leaf=""><br></span><span leaf=""> query_embeddings = self.model(**inputs)</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> </span><span><span leaf=""># MaxSim计算：对query的每个patch向量，找页面中最相似的patch，求和</span></span><span leaf=""><br></span><span leaf=""> scores = []</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">for</span></span><span leaf=""> page_emb </span><span><span leaf="">in</span></span><span leaf=""> self.page_embeddings:</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf=""># query_embeddings: (1, num_query_tokens, dim)</span></span><span leaf=""><br></span><span leaf=""> </span><span><span leaf=""># page_emb: (1, num_page_tokens, dim)</span></span><span leaf=""><br></span><span leaf=""> </span><span><span leaf=""># MaxSim: query的每个token跟page所有token算相似度，取max，再求和</span></span><span leaf=""><br></span><span leaf=""> sim_matrix = query_embeddings[</span><span><span leaf="">0</span></span><span leaf="">] @ page_emb[</span><span><span leaf="">0</span></span><span leaf="">].T </span><span><span leaf=""># (num_query, num_page)</span></span><span leaf=""><br></span><span leaf=""> max_sim = sim_matrix.max(dim=</span><span><span leaf="">1</span></span><span leaf="">)[</span><span><span leaf="">0</span></span><span leaf="">].sum().item()</span><span leaf=""><br></span><span leaf=""> scores.append(max_sim)</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> </span><span><span leaf=""># 排序取TopK</span></span><span leaf=""><br></span><span leaf=""> ranked = sorted(enumerate(scores), key=</span><span><span leaf="">lambda</span></span><span leaf=""> x: x[</span><span><span leaf="">1</span></span><span leaf="">], reverse=</span><span><span leaf="">True</span></span><span leaf="">)</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">return</span></span><span leaf=""> [(self.page_info[i], score) </span><span><span leaf="">for</span></span><span leaf=""> i, score </span><span><span leaf="">in</span></span><span leaf=""> ranked[:top_k]]</span><span leaf=""><br></span>
坑5：ColPali的存储和计算量大。每个页面不是一个向量，而是几百个patch向量（一张A4页面约1000+个patch）。100页PDF的索引就有10万个向量。解决：用PQ（Product Quantization）压缩向量，或只保留TopK最相关的patch向量。
坑6：ColPali检索到页面后还要生成答案。ColPali只负责检索——告诉你"答案在第3页"，但第3页是图片，LLM怎么读？两个方案：一是把页面图片直接喂给多模态LLM（GPT-4o/Qwen-VL）让它看图回答；二是用OCR把页面转成文字再喂给文本LLM。前者效果更好但成本高。
多模态RAG的完整架构
把三条路线组合起来，一个生产级多模态RAG架构长这样：
<span><span><span leaf="">class</span></span><span leaf=""> </span><span><span leaf="">MultimodalRAG</span></span><span leaf="">:</span></span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">"""生产级多模态RAG：文本 + 图片 + 表格混合检索"""</span></span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> </span><span><span><span leaf="">def</span></span><span leaf=""> </span><span><span leaf="">__init__</span></span><span><span leaf="">(self, text_llm, vision_llm, text_embedder, clip_embedder)</span></span><span leaf="">:</span></span><span leaf=""><br></span><span leaf=""> self.text_llm = text_llm </span><span><span leaf=""># 文本生成LLM</span></span><span leaf=""><br></span><span leaf=""> self.vision_llm = vision_llm </span><span><span leaf=""># 视觉LLM（看图）</span></span><span leaf=""><br></span><span leaf=""> self.text_embedder = text_embedder </span><span><span leaf=""># 文本Embedding（BGE等）</span></span><span leaf=""><br></span><span leaf=""> self.clip_embedder = clip_embedder </span><span><span leaf=""># 多模态Embedding（CLIP）</span></span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> self.text_store = [] </span><span><span leaf=""># 文本向量库</span></span><span leaf=""><br></span><span leaf=""> self.image_store = [] </span><span><span leaf=""># 图片向量库</span></span><span leaf=""><br></span><span leaf=""> self.table_store = [] </span><span><span leaf=""># 表格文本库</span></span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> </span><span><span><span leaf="">def</span></span><span leaf=""> </span><span><span leaf="">index_document</span></span><span><span leaf="">(self, pdf_path)</span></span><span leaf="">:</span></span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">"""索引PDF文档：文本 + 图片 + 表格"""</span></span><span leaf=""><br></span><span leaf=""> </span><span><span leaf=""># 1. 文本提取和切分 → 文本向量库</span></span><span leaf=""><br></span><span leaf=""> text_chunks = self._extract_and_split_text(pdf_path)</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">for</span></span><span leaf=""> chunk </span><span><span leaf="">in</span></span><span leaf=""> text_chunks:</span><span leaf=""><br></span><span leaf=""> emb = self.text_embedder.embed_query(chunk)</span><span leaf=""><br></span><span leaf=""> self.text_store.append({</span><span><span leaf="">"content"</span></span><span leaf="">: chunk, </span><span><span leaf="">"embedding"</span></span><span leaf="">: emb})</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> </span><span><span leaf=""># 2. 图片提取 → LLM生成描述 → 描述存文本库 + 图片存CLIP库</span></span><span leaf=""><br></span><span leaf=""> images = extract_images_from_pdf(pdf_path, </span><span><span leaf="">"/tmp/images"</span></span><span leaf="">)</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">for</span></span><span leaf=""> img_path, page_num </span><span><span leaf="">in</span></span><span leaf=""> images:</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf=""># 图片描述（路线一）</span></span><span leaf=""><br></span><span leaf=""> description = image_to_text_description(img_path, self.vision_llm)</span><span leaf=""><br></span><span leaf=""> emb = self.text_embedder.embed_query(description)</span><span leaf=""><br></span><span leaf=""> self.text_store.append({</span><span><span leaf="">"content"</span></span><span leaf="">: description, </span><span><span leaf="">"embedding"</span></span><span leaf="">: emb, </span><span><span leaf="">"image"</span></span><span leaf="">: img_path})</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> </span><span><span leaf=""># 图片CLIP向量（路线二）</span></span><span leaf=""><br></span><span leaf=""> clip_emb = self.clip_embedder.embed_image(img_path)</span><span leaf=""><br></span><span leaf=""> self.image_store.append({</span><span><span leaf="">"path"</span></span><span leaf="">: img_path, </span><span><span leaf="">"embedding"</span></span><span leaf="">: clip_emb, </span><span><span leaf="">"description"</span></span><span leaf="">: description})</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> </span><span><span leaf=""># 3. 表格提取 → Markdown格式 → 存文本库</span></span><span leaf=""><br></span><span leaf=""> tables = extract_tables_from_pdf(pdf_path)</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">for</span></span><span leaf=""> table </span><span><span leaf="">in</span></span><span leaf=""> tables:</span><span leaf=""><br></span><span leaf=""> emb = self.text_embedder.embed_query(table[</span><span><span leaf="">"content"</span></span><span leaf="">])</span><span leaf=""><br></span><span leaf=""> self.table_store.append({</span><span><span leaf="">"content"</span></span><span leaf="">: table[</span><span><span leaf="">"content"</span></span><span leaf="">], </span><span><span leaf="">"embedding"</span></span><span leaf="">: emb, </span><span><span leaf="">"page"</span></span><span leaf="">: table[</span><span><span leaf="">"page"</span></span><span leaf="">]})</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> print(</span><span><span leaf="">f"索引完成：</span><span><span leaf="">{len(self.text_store)}</span></span><span leaf="">文本块, </span><span><span leaf="">{len(self.image_store)}</span></span><span leaf="">图片, </span><span><span leaf="">{len(self.table_store)}</span></span><span leaf="">表格"</span></span><span leaf="">)</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> </span><span><span><span leaf="">def</span></span><span leaf=""> </span><span><span leaf="">query</span></span><span><span leaf="">(self, question)</span></span><span leaf="">:</span></span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">"""混合检索 + 多模态生成"""</span></span><span leaf=""><br></span><span leaf=""> </span><span><span leaf=""># 1. 文本检索（包含图片描述和表格文本）</span></span><span leaf=""><br></span><span leaf=""> text_results = self._search_text(question, top_k=</span><span><span leaf="">5</span></span><span leaf="">)</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> </span><span><span leaf=""># 2. 图片检索（CLIP）</span></span><span leaf=""><br></span><span leaf=""> image_results = self._search_images(question, top_k=</span><span><span leaf="">3</span></span><span leaf="">)</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> </span><span><span leaf=""># 3. 组装上下文</span></span><span leaf=""><br></span><span leaf=""> context_parts = []</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> </span><span><span leaf=""># 文本上下文</span></span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">for</span></span><span leaf=""> result </span><span><span leaf="">in</span></span><span leaf=""> text_results:</span><span leaf=""><br></span><span leaf=""> context_parts.append(</span><span><span leaf="">f"[文本] </span><span><span leaf="">{result[</span><span><span leaf="">'content'</span></span><span leaf="">]}</span></span><span leaf="">"</span></span><span leaf="">)</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> </span><span><span leaf=""># 表格上下文</span></span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">for</span></span><span leaf=""> result </span><span><span leaf="">in</span></span><span leaf=""> text_results:</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">if</span></span><span leaf=""> </span><span><span leaf="">"content"</span></span><span leaf=""> </span><span><span leaf="">in</span></span><span leaf=""> result </span><span><span leaf="">and</span></span><span leaf=""> </span><span><span leaf="">"|"</span></span><span leaf=""> </span><span><span leaf="">in</span></span><span leaf=""> result.get(</span><span><span leaf="">"content"</span></span><span leaf="">, </span><span><span leaf="">""</span></span><span leaf="">):</span><span leaf=""><br></span><span leaf=""> context_parts.append(</span><span><span leaf="">f"[表格] </span><span><span leaf="">{result[</span><span><span leaf="">'content'</span></span><span leaf="">]}</span></span><span leaf="">"</span></span><span leaf="">)</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> </span><span><span leaf=""># 4. 生成答案</span></span><span leaf=""><br></span><span leaf=""> text_context = </span><span><span leaf="">"\n\n"</span></span><span leaf="">.join(context_parts)</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> </span><span><span leaf=""># 如果有相关图片，用视觉LLM看图</span></span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">if</span></span><span leaf=""> image_results:</span><span leaf=""><br></span><span leaf=""> best_image = image_results[</span><span><span leaf="">0</span></span><span leaf="">]</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf=""># 多模态LLM同时看文本上下文和图片</span></span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">import</span></span><span leaf=""> base64</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">with</span></span><span leaf=""> open(best_image[</span><span><span leaf="">"path"</span></span><span leaf="">], </span><span><span leaf="">"rb"</span></span><span leaf="">) </span><span><span leaf="">as</span></span><span leaf=""> f:</span><span leaf=""><br></span><span leaf=""> img_b64 = base64.b64encode(f.read()).decode()</span><span leaf=""><br></span><span leaf=""> </span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">from</span></span><span leaf=""> langchain_core.messages </span><span><span leaf="">import</span></span><span leaf=""> HumanMessage</span><span leaf=""><br></span><span leaf=""> msg = HumanMessage(content=[</span><span leaf=""><br></span><span leaf=""> {</span><span><span leaf="">"type"</span></span><span leaf="">: </span><span><span leaf="">"text"</span></span><span leaf="">, </span><span><span leaf="">"text"</span></span><span leaf="">: </span><span><span leaf="">f"""基于以下文本上下文和图片回答问题。</span><span leaf=""><br></span><span leaf=""><br></span><span leaf="">文本上下文：</span><span leaf=""><br></span><span><span leaf="">{text_context}</span></span><span leaf=""><br></span><span leaf=""><br></span><span leaf="">问题：</span><span><span leaf="">{question}</span></span><span leaf=""><br></span><span leaf=""><br></span><span leaf="">回答："""</span></span><span leaf="">},</span><span leaf=""><br></span><span leaf=""> {</span><span><span leaf="">"type"</span></span><span leaf="">: </span><span><span leaf="">"image_url"</span></span><span leaf="">, </span><span><span leaf="">"image_url"</span></span><span leaf="">: {</span><span><span leaf="">"url"</span></span><span leaf="">: </span><span><span leaf="">f"data:image/png;base64,</span><span><span leaf="">{img_b64}</span></span><span leaf="">"</span></span><span leaf="">}},</span><span leaf=""><br></span><span leaf=""> ])</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">return</span></span><span leaf=""> self.vision_llm.invoke([msg]).content</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">else</span></span><span leaf="">:</span><span leaf=""><br></span><span leaf=""> </span><span><span leaf=""># 纯文本生成</span></span><span leaf=""><br></span><span leaf=""> prompt = </span><span><span leaf="">f"基于以下上下文回答问题。\n\n上下文：\n</span><span><span leaf="">{text_context}</span></span><span leaf="">\n\n问题：</span><span><span leaf="">{question}</span></span><span leaf="">\n\n回答："</span></span><span leaf=""><br></span><span leaf=""> </span><span><span leaf="">return</span></span><span leaf=""> self.text_llm.invoke(prompt).content</span><span leaf=""><br></span>
Java类比：这就像你做了一个全文搜索引擎，除了索引文字还索引了图片和表格。用户搜索时，引擎从三个索引库（文本/图片/表格）同时检索，把结果汇总后交给"大脑"（LLM）生成答案。跟Java里多数据源聚合查询是一个思路——Solr/Elasticsearch做多字段索引，查询时并行检索再合并结果。
本篇要点
| 要点
|
说明
三条路线
|
文本描述法（最简单）、多模态Embedding（CLIP）、视觉检索（ColPali）
| |
文本描述法
|
LLM看图说话 + 表格转Markdown，信息有损但工程最简单
| |
多模态Embedding
|
CLIP把图文映射到同一向量空间，实现文字搜图片
| |
ColPali
|
PDF页面直接当图片检索，零信息损失，但计算量大
| |
表格处理
|
pdfplumber提取 + Markdown格式保留行列结构
| |
混合架构
|
三条路线组合使用，文本+图片+表格同时检索
| |
中文CLIP
|
用Chinese-CLIP或BGE-VL，别用原版CLIP
|
踩坑清单
-
LLM看图读数值不准：GPT-4o读柱状图数值会四舍五入。需要精确数值用OCR或表格提取，别让LLM猜。
-
PDF表格提取不完整：合并单元格和跨页表格是重灾区。pdfplumber搞不定就上Camelot或LLM看截图。
-
CLIP中文支持差：原版CLIP用英文训练，中文query效果差。换Chinese-CLIP或BGE-VL。
-
CLIP不适合细粒度检索：CLIP擅长找"猫的图片"，不擅长找"包含300万这个数值的图表"。细粒度用文本描述法。
-
ColPali存储量大：每页1000+个patch向量，100页就是10万向量。用PQ压缩或只保留TopK patch。
-
多模态LLM成本高：每张图片都要调用GPT-4o看图，100张图就是$5-10。批量处理用缓存，或用本地视觉模型（Qwen-VL-7B）。
-
图片描述质量不稳定：同一个图表让LLM描述两次，结果可能不一样。对关键图表，人工校验描述质量。
三条路线选择速查
| 你的场景
|
推荐路线
|
原因
图片有alt文本/表格有标题
|
文本描述法
|
最简单，复用现有RAG
| |
电商商品图+描述混合
|
CLIP多模态Embedding
|
原生图文混合检索
| |
PDF报告含大量图表无描述
|
ColPali视觉检索
|
零信息损失
| |
需要精确读图表数值
|
表格提取(pdfplumber)
|
别让LLM猜数字
| |
中文图文混合检索
|
Chinese-CLIP/BGE-VL
|
别用原版CLIP
| |
预算有限
|
文本描述法
|
成本最低
|
下篇预告
下一篇讲Agentic RAG——让RAG从固定流水线变成能自主思考的智能体。Corrective RAG、Self-RAG、多步推理策略，以及怎么在LangGraph里实现一个生产可用的Agentic RAG。
你的RAG项目有需要检索图片和表格的场景吗？用的什么方案？评论区说说。
觉得有用就点个在看，下一篇讲Agentic RAG——让RAG学会自己纠错。
- 原文作者：知识铺
- 原文链接：https://index.zshipu.com/ai002/post/20260822/%E5%9B%BE%E8%A1%A8%E8%A1%A8%E6%A0%BCPDF%E5%9B%BE%E6%96%87%E5%A4%9A%E6%A8%A1%E6%80%81RAG%E5%85%A8%E6%90%9E%E5%AE%9A/
- 版权声明：本作品采用知识共享署名-非商业性使用-禁止演绎 4.0 国际许可协议进行许可，非商业转载请注明出处（作者，原文链接），商业转载请联系作者获得授权。
- 免责声明：本页面内容均来源于站内编辑发布，部分信息来源互联网，并不意味着本站赞同其观点或者证实其内容的真实性，如涉及版权等问题，请立即联系客服进行更改或删除，保证您的合法权益。转载请注明来源，欢迎对文章中的引用来源进行考证，欢迎指出任何有错误或不够清晰的表达。也可以邮件至 sblig@126.com
