kb-rag — Local Literature Knowledge-Base RAG (DSH Plugin)

Ingest once, search forever. Only the most relevant few sentences ever reach the LLM — and every claim carries exact provenance.

Who it's for

Graduate students and PhD researchers. An idea strikes, and you know it's somewhere in your library — but which paper said it, and where? kb-rag makes the whole pile queryable: hybrid retrieval + reranking associate the right passages, every answer lands on a clickable DOI (or the exact file), and the reply tells you what your library is still missing. Think it → find it → cite it.

kb-rag is a lightweight local database-RAG plugin for DSH (DeepSeek Harness): it turns PDF/Zotero literature into a SQLite knowledge base with section structure and vector indexes, providing the full hybrid search + rerank + cited-QA workflow. All indexing, embedding, and reranking run locally — zero API cost, zero upload.

核心卖点

1. 检索准 — BM25 + bge-small 向量 + bge-reranker 三级混合检索，精排后命中相关性 0.99+（实测）；章节感知权重让"找机制"不会翻到致谢里。
2. 引文联动 — 每个命中都是可点击坐标：DOI 一键跳原文，无 DOI 给可复制的 Scholar 搜索串；自动关联同作者/同期刊/主题相近的文献；正文引用的图自动挂图注坐标；答案末尾提示"库里还缺哪些文献"。
3. 本地零成本 — 全本地嵌入与重排，零 API 费用、零上传，dsh.so 安全扫描 passed。

Features

• 9 model tools: kb_ingest (file/folder ingest), kb_zotero (Zotero migration), kb_search (hybrid search), kb_rag (cited QA), kb_scope (scope/strict mode), kb_dedup (dedup), kb_clear (wipe), kb_stats (stats), kb_fetch (DOI/arXiv PDF download)
• Structured chunking: paper section recognition (abstract ×1.5, methods ×1.2 weights), inline-heading detection, abstract auto-promotion, caption blocks; paragraph fallback for non-papers
• Hybrid retrieval: keyword BM25 (CJK-bigram friendly) + bge-small vector cosine, RRF fusion, × section weights
• Reranking: bge-reranker-base Cross-Encoder, Top-20 → Top-3 (auto-fallback to bge-large-en bi-encoder if missing)
• Incremental & dedup: sha256 incremental skip (40× faster reruns), cross-path duplicate interception, kb_dedup for existing stores
• Query cache: same query+filters never recompute; any ingest invalidates it
• Citation standard: with DOI → markdown link; without DOI → [authors, year, filename]
• Scope & strict mode: closed-KB / KB+web / web-only; strict mode forbids outside-knowledge extrapolation
• Related literature: every search also returns associated papers (same authors / same journal / nearby year / thematically similar), so one query surfaces the surrounding literature — and the answer's "suggested additions" cites them
• Engine daemon: models load once, sub-second hot queries; crash self-heal; auto-reclaim on plugin stop

Design Principles

• Deliberately zero UI: every operation and inspection happens through conversation and tool returns (search results render with clickable DOI links); no management panel, no frontend state, no client dependencies — a positioning choice, not a gap. DSH's interface is conversation, and a plugin's interface is tool calls; "panels" belong to scenarios that need direct human administration.
• Vertical on academic literature: section-aware chunking (abstract/methods weighting), native Zotero migration, DOI citation standards — not a general-purpose KB manager, but "papers, out of the box".
• Stay in the sweet spot: at 20k chunks, brute-force BM25 + IndexFlatIP is optimal; simple implementation plus measured numbers beats feature-stacking.

Architecture

DSH model ──tool call──▶ plugin Host (thin JS) ──JSON-lines──▶ kb_engine.py (resident serve)
                                                             ├─ ingest: hash skip → PyMuPDF extract → section chunking → bge-small encode
                                                             ├─ search: SQL prefilter → BM25+vector dual path → RRF fuse → reranker → snippet+source
                                                             └─ storage: workspace/.kb/kb.sqlite (docs/chunks/vecs/cache)

Data flow: raw PDF → verbatim extraction + section chunking → chunks into the DB (with metadata and vectors) → hybrid search + rerank on query → Top-N verbatim snippets (with DOI/file/section/score) → the current conversation model answers with citations.

Quick Start

最新版本 v1.6.0 — 下载：npm install dsh-kb-rag@latest　安装：dsh plugin --profile web add dsh-kb-rag@latest

推荐：npx 一键装环境 + 激活（无需先安装包）

# ✅ 正确：--package dsh-kb-rag 指明命令来自哪个包
npx --yes --package dsh-kb-rag -c "dsh-kb-rag-install --profile web"

⚠️ 常见坑：裸写 npx dsh-kb-rag-install 会失败（npx 会去找一个名为 dsh-kb-rag-install 的包，注册表里不存在 → E404）。必须带 --package dsh-kb-rag 和 -c。Windows 下 bash 风格参数（--profile/--models/--dry-run）会被自动翻译，全平台通用；--dry-run 可先演练。

安装器一条链完成：Python 依赖（--mirror pip 镜像）→ 引擎冒烟 → Node/pnpm 检查（缺 pnpm 自动装）→ dsh plugin add 激活（--profile <name>）→ 可选 --models 预下载模型（尊重 HF_ENDPOINT）。

也可从源码一键安装：

git clone https://github.com/Breeze136/dsh-kb-rag.git && cd dsh-kb-rag
./npm-package/scripts/install.sh        # Windows: install.cmd（或 npm-package\scripts\install.ps1）

手动三步（老式动态插件，一般用户用上面两条即可）：

1. 安装 Python 依赖（见 requirements.txt）
2. 把 kb_engine.py 放到 DSH 会话工作区根目录
3. 通过 cordis_define 加载 plugin/host.js 与 plugin/client.js，运行后直接对话（首次检索会询问查询范围）

装完务必重启 DSH 并开新会话（工具在会话创建时注入，老会话不会自动获得）。升级旧版：在 profile 目录 npm install dsh-kb-rag@latest，旧 .kb 库自动迁移（schema 版本化，见 docs/MIGRATION.md）。更多常见坑见 npm-package/README.md 的 Troubleshooting 表。

npm Static Package (for other Harness users)

Published to npm: dsh-kb-rag (npmjs.com/package/dsh-kb-rag), and indexed on the dsh.so registry (security scan: passed).

最新版本 v1.6.0 — 下载安装：dsh plugin --profile web add dsh-kb-rag@latest（或 npm install dsh-kb-rag@latest）

Option 1 — one command (recommended, DSH profiles)

The package declares dsh.bundle, so dsh plugin add installs and activates it in one step:

dsh plugin --profile web add dsh-kb-rag

Requires pnpm on PATH (the official DSH plugin flow uses pnpm). Python dependencies are then handled two ways:

• Zero-config: set KB_AUTO_PIP=1 in the host environment and restart DSH — the plugin pip-installs missing packages itself (fixed argv, off by default; normally it only logs the command).
• One-shot installer: npx --yes --package dsh-kb-rag -c "dsh-kb-rag-install --profile web" runs the bundled scripts/install.ps1 / scripts/install.sh (node_modules/dsh-kb-rag/scripts/) — Python deps, engine smoke test, pnpm, plugin activation, optional model pre-download in one shot. (⚠️ 裸 npx dsh-kb-rag-install 会失败，必须带 --package dsh-kb-rag，见上方 Quick Start 坑提示。)

Then restart DSH and open a new session — the 9 tools register automatically.

Option 2 — plugin marketplace (no terminal)

Install dsh-plugin-registry once; its Settings "plugin marketplace" panel lists kb-rag (we are in the curated awesome-dsh-plugin list) with one-click install.

Option 3 — manual

1. npm install dsh-kb-rag in the deployment/profile directory
2. Activate it: add "dsh-kb-rag" to dsh.profile.bundles in the profile's package.json (or copy the bundled cordis.patch.yml insert into your own patch layer)
3. Restart DSH and open a new session

Notes: the DSH plugin loader resolves package names from the deployment's node_modules and does not auto-download missing packages. The package ships its own kb_engine.py (no manual placement needed). On startup it auto-checks Python dependencies and reports the complete missing list (importlib find_spec probe); by default it prints the pip install command to the host log, with KB_AUTO_PIP=1 set it installs them itself, and tool calls return an actionable error (with the exact fix) instead of an opaque engine crash while deps are missing. The npx one-liner and the bundled scripts/install.ps1 / scripts/install.sh do the whole chain in one shot. See npm-package/README.md for full details.

Tool Reference

Benchmarks (measured)

Citation Style (answer format)

Configuration

Repository Layout

kb-rag/
├─ install.cmd             # Windows one-click entry (double-click)
├─ kb_engine.py          # Python search engine (CLI + serve protocol)
├─ plugin/
│  ├─ host.js            # DSH plugin Host half (9 tools + daemon + RPC)
│  └─ client.js          # DSH plugin Client half (tool source cards, optional)
├─ npm-package/          # npm static package dsh-kb-rag
│  ├─ lib/index.js       # Host plugin (9 tools + dep probe / KB_AUTO_PIP)
│  ├─ install.mjs        # npm bin: dsh-kb-rag-install (npx entry)
│  ├─ scripts/           # one-click installers (install.ps1 / install.sh)
│  └─ kb_engine.py
├─ docs/DESIGN.md        # Design doc (chunking/search/protocol details)
├─ QUICKSTART.md         # Five-minute start
├─ CHANGELOG.md
├─ requirements.txt
└─ LICENSE

Known Limitations & Roadmap

• Metadata year: scraped from text when PDF metadata is missing, may mis-pick (Zotero metadata can override)
• Search performance: keyword scan is an in-memory implementation; beyond a few hundred thousand chunks consider FAISS HNSW / SQLite FTS5
• Roadmap: zh→en query translation (local opus-mt model), caption OCR, citation-network graph

Uninstall

See UNINSTALL.md — stop the plugin, delete only the index/kb.sqlite it generated, and keep your PDFs and Zotero library untouched.

License

MIT — see LICENSE