fix(indexer): restore module-level embedding/batch constants
PR Checks / hadolint (pull_request) Successful in 12s
Docker Build and Push / test (pull_request) Successful in 2m17s
Docker Build and Push / integration-test (pull_request) Successful in 2m38s
PR Checks / pytest (pull_request) Successful in 2m19s
Docker Build and Push / build (pull_request) Successful in 2m39s
PR Checks / build-test (pull_request) Successful in 3m5s

The "make indexer importable without env" refactor (e32806a, merged via PR
#52 on 2026-10-09) removed EMBEDDING_MODEL_NAME, BATCH_SIZE and
EMBEDDING_BATCH_SIZE from the module scope while main() kept referencing
them, so every indexer run died before reaching Qdrant:

    File "/app/src/indexer.py", line 264, in main
        print(f"Loading embedding model: {EMBEDDING_MODEL_NAME}...")
    NameError: name 'EMBEDDING_MODEL_NAME' is not defined

CI never caught it because no test executes main(). Both *-indexer cronjobs
have been failing (BackoffLimitExceeded) since 2026-10-16, i.e. the first
run after the image pin moved to v0.0.22-latest.

- restore the three constants with their pre-refactor defaults
- cover them in tests/test_indexer.py and cross-check that the indexer
  embeds with the same model as server.py (vectors must stay comparable)
- add a ruff F821 gate to the unit job so undefined names fail CI

Fixes mcp-maildir-cabilloteu-indexer and mcp-maildir-gmail-indexer.
This commit is contained in:
Sagent committed 2026-10-17 21:04:34 +00:00
1 parent 4d9bd4c470
commit 3b575bbf83
3 files changed
+35 -1

No files matched your search

+3 -1
View File
@@ -22,7 +22,9 @@ jobs:
with: with:
python-version: "3.14" python-version: "3.14"
- name: Install dependencies - name: Install dependencies
run: pip install -r requirements.txt pytest run: pip install -r requirements.txt pytest ruff==0.17.1
- name: Lint (undefined names)
run: ruff check src/ tests/ --select F821
- name: Run unit tests - name: Run unit tests
run: pytest tests/ -v run: pytest tests/ -v
env: env:
+7
View File
@@ -29,6 +29,13 @@ MAILDIR_FOLDERS = os.environ.get("MAILDIR_FOLDERS", "")
QDRANT_URL = os.environ.get("QDRANT_URL", "") QDRANT_URL = os.environ.get("QDRANT_URL", "")
COLLECTION_NAME = os.environ.get("COLLECTION_NAME", "") COLLECTION_NAME = os.environ.get("COLLECTION_NAME", "")
# Embedding model and batching. EMBEDDING_MODEL_NAME MUST stay in sync with the
# one used by server.py: indexer and server have to embed with the same model,
# otherwise the vectors stored in Qdrant are not comparable with query vectors.
EMBEDDING_MODEL_NAME = os.environ.get("EMBEDDING_MODEL_NAME", "BAAI/bge-small-en-v1.5")
BATCH_SIZE = int(os.environ.get("BATCH_SIZE", "100"))
EMBEDDING_BATCH_SIZE = int(os.environ.get("EMBEDDING_BATCH_SIZE", "64"))
METADATA_COLLECTION = "mcp_indexer_metadata" METADATA_COLLECTION = "mcp_indexer_metadata"
INCREMENTAL_DAYS = int(os.environ.get("INCREMENTAL_DAYS", "7")) INCREMENTAL_DAYS = int(os.environ.get("INCREMENTAL_DAYS", "7"))
FORCE_REINDEX = os.environ.get("FORCE_REINDEX", "").lower() in ("1", "true", "yes") FORCE_REINDEX = os.environ.get("FORCE_REINDEX", "").lower() in ("1", "true", "yes")
+25
View File
@@ -21,6 +21,9 @@ from indexer import (
parse_email_message, parse_email_message,
get_recent_keys, get_recent_keys,
METADATA_COLLECTION, METADATA_COLLECTION,
EMBEDDING_MODEL_NAME,
BATCH_SIZE,
EMBEDDING_BATCH_SIZE,
) )
@@ -227,3 +230,25 @@ class TestGetRecentKeys:
assert key.split(":")[0] in keys assert key.split(":")[0] in keys
for k in keys: for k in keys:
assert ":" not in k assert ":" not in k
# ---------------------------------------------------------------------------
# Regression: module-level constants consumed by main()
# ---------------------------------------------------------------------------
def test_indexer_module_constants_are_defined():
"""Regression (e32806a): the "make indexer importable without env" refactor
dropped these three module-level definitions while main() kept using them,
so every indexer run died with NameError before touching Qdrant."""
assert isinstance(EMBEDDING_MODEL_NAME, str) and EMBEDDING_MODEL_NAME
assert isinstance(BATCH_SIZE, int) and BATCH_SIZE > 0
assert isinstance(EMBEDDING_BATCH_SIZE, int) and EMBEDDING_BATCH_SIZE > 0
def test_embedding_model_is_the_same_as_the_server_one():
"""Indexer and server must embed with the same model, otherwise the vectors
stored by the indexer are not comparable with the query vectors."""
from server import EMBEDDING_MODEL_NAME as server_embedding_model
assert EMBEDDING_MODEL_NAME == server_embedding_model