mirror of
https://github.com/pewdiepie-archdaemon/odysseus.git
synced 2026-08-09 06:58:41 -04:00
Joeseph Grey
4c9a8ca115
* fix(rag): skip hidden and junk directories when indexing (#5559) index_personal_documents walked the whole tree with no pruning, so pointing RAG at a real-world folder silently swept in .obsidian/ plugin JS, .git/ internals, node_modules/, and __pycache__/ — multiplying indexing time and polluting retrieval with junk chunks. Prune hidden directories and well-known junk directories from the walk, and skip hidden files. The explicitly passed root is exempt, so a user who deliberately indexes a hidden directory still gets its contents. * fix(rag): prune hidden/junk dirs in the keyword index too, via a shared helper The #5559 fix pruned only VectorRAG.index_personal_documents (the vector index). The parallel keyword index built by PersonalDocsManager.refresh_index -> load_personal_index walked the same tree unpruned, so .obsidian/, .git/, node_modules/ etc. still swept into keyword retrieval and the file listing — the 'end-to-end' guarantee was only half true. Single-source the pruning policy in src/index_walk (prune_index_dirs + is_indexable_file) and use it from both walkers so they cannot drift again. The junk-dir match is now case-insensitive, so a Node_Modules on a case-insensitive filesystem is pruned too. Tests: keyword-path regressions covering hidden/junk dirs, hidden files, junk at depth (not just top level), case-insensitive junk, and the explicit-hidden- root exemption. The existing vector tests still pass against the shared helper.
36 lines
1.6 KiB
Python
36 lines
1.6 KiB
Python
"""Shared directory-walk pruning for personal-document indexing (#5559).
|
|
|
|
Single source of the hidden-dir / junk-dir / hidden-file skip so the vector
|
|
index (``rag_vector.index_personal_documents``) and the keyword index
|
|
(``personal_docs.load_personal_index``) apply the exact same policy and cannot
|
|
drift — the drift is what left the keyword path sweeping in `.obsidian/`,
|
|
`.git/`, and `node_modules/` after the vector path was fixed.
|
|
"""
|
|
from typing import List, Set
|
|
|
|
# Well-known non-hidden junk directories to skip. Matched case-insensitively so
|
|
# a `Node_Modules` on a case-insensitive filesystem (macOS default) is still
|
|
# pruned. Hidden directories (dot-prefixed) are pruned separately. Kept
|
|
# deliberately small: over-pruning would silently drop a user's real content
|
|
# (e.g. a notes directory legitimately named "build").
|
|
EXCLUDED_DIR_NAMES: Set[str] = {'node_modules', '__pycache__', 'venv'}
|
|
|
|
|
|
def prune_index_dirs(dirs: List[str]) -> None:
|
|
"""In-place ``os.walk`` (topdown) directory prune: drop hidden and known
|
|
junk directories so the walk never descends into them.
|
|
|
|
The explicitly-targeted walk root is never a member of ``dirs`` (it is the
|
|
``dirpath`` argument), so it stays exempt — a user who deliberately points
|
|
indexing at a hidden directory gets its contents, minus nested junk.
|
|
"""
|
|
dirs[:] = [
|
|
d for d in dirs
|
|
if not d.startswith('.') and d.lower() not in EXCLUDED_DIR_NAMES
|
|
]
|
|
|
|
|
|
def is_indexable_file(name: str) -> bool:
|
|
"""A file is indexable only if it is not hidden (dot-prefixed)."""
|
|
return not name.startswith('.')
|