Orchestrator/bahn/teamlandkarte-mcp/openspec/changes/add-bm25-rrf-competence-matching/tasks.md

Implementation Tasks: BM25 + RRF Competence Matching

Status: Implemented

Summary

This change introduces an optional BM25-based competence matching strategy with optional LLM-based Auto-Tagging as an alternative to the existing embedding-based approach:

1. Restore LLM client: AzureOpenAIClient.chat_completion() restored; AzureOpenAIConfig.chat_deployment + llm_api_key added.
2. New auto_tagger.py module: AutoTagger class with expand_competences() — ephemeral LLM-driven competence expansion.
3. New bm25.py module: Bm25Index dataclass + bm25_rank_competences() convenience function.
4. New rrf.py module: reciprocal_rank_fusion() fuses ranked lists into a normalized score dict.
5. similarity.py extended: _bm25_rrf_similarity() internal method + auto-tag branch + flag routing in compute_competence_similarity().
6. SimilarityConfig extended: new use_bm25_search: bool = False and use_auto_tagging: bool = False fields.
7. config.py parser extended: reads use_bm25_search, use_auto_tagging, chat_deployment, and AZURE_OPENAI_LLM_API_KEY.
8. config.toml.example updated: documents all new keys.
9. mcp_server.py updated: constructs AutoTagger when use_auto_tagging=True; passes flags to SimilarityEngine.
10. Legacy debug files removed: 4 root-level files no longer part of the module.
11. Tests added: tests/test_bm25.py, tests/test_rrf.py, tests/test_auto_tagger.py, extensions to tests/test_similarity.py and tests/test_matcher_integration.py.
12. Documentation updated: architecture, tradeoffs doc, assistant prompt, README.

Clarifications / Decisions

• BM25 scope: BM25 + RRF replaces competence similarity only. Role similarity is always embedding-based, regardless of use_bm25_search.
• Auto-tagging scope: use_auto_tagging only applies when use_bm25_search = true. When use_auto_tagging = true, AutoTagger.expand_competences() is called once per candidate per compute_competence_similarity() invocation. Result is ephemeral.
• Auto-tagging works in both directions: both find_matching_capacities and find_matching_tasks pass through compute_competence_similarity() and benefit.
• Graceful degradation: if the LLM call in AutoTagger fails for any reason, the original candidate list is used unchanged. BM25 scoring continues.
• LLM client restored: AzureOpenAIClient.chat_completion() is re-added, scoped exclusively to auto-tagging. The embedding-related API surface is unchanged.
• Zero-out rule: If all BM25 scores for a required competence are 0.0 (no token overlap at all), the RRF score for that required competence is also 0.0.
• Global index (revised): Originally the Bm25Index was to be built fresh per compute_competence_similarity() call. This was identified as an IDF pathology: with only one person's competences as the corpus, BM25 IDF is always 0 or negative for any term that appears in every document (and with N=1 all terms do). The fix (Phase 9) builds one Bm25Index over the union of all filtered candidates' competences before the loop and passes it in as global_index.
• Single ranked list: Initial implementation fuses one BM25 ranked list per required competence. The reciprocal_rank_fusion() signature accepts multiple lists for future hybrid extension.
• Output shape unchanged: _bm25_rrf_similarity() returns the same dict[str, dict[str, object]] shape as _per_skill_similarity(). Matcher, scorer, and all output formatting remain unchanged.
• rank-bm25 dependency: Added to pyproject.toml [project.dependencies].
• All new code uses Google Docstrings per project convention.
• Default false: Existing deployments are unaffected until opt-in.

Estimated Effort

• Phase 0 (Restore LLM client): 1 hour
• Phase 1 (New modules — bm25, rrf, auto_tagger): 3–4 hours
• Phase 2 (Config + wiring): 1–1.5 hours
• Phase 3 (Similarity integration): 2 hours
• Phase 4 (Legacy cleanup): 0.5 hour
• Phase 5 (Tests): 4–5 hours
• Phase 6 (Documentation): 2 hours
• Phase 7 (Dependency): 0.5 hour
• Phase 8 (Quality gates): 0.5 hour
• Total: 15–17 hours

---

Phase 0: Restore LLM Client

Task 0.1: Extend AzureOpenAIConfig

File: src/teamlandkarte_mcp/config.py

Subtasks:

• Add chat_deployment: str = "" field to AzureOpenAIConfig dataclass
• Add Google Docstring field comment: """Azure deployment name for the chat model used in auto-tagging (e.g. 'gpt-4.1'). Required when use_auto_tagging = true."""
• Update the TOML parser to read azure_openai.chat_deployment with default ""
• Read AZURE_OPENAI_LLM_API_KEY from environment (store as llm_api_key: str = "" on AzureOpenAIConfig)
• When use_auto_tagging = true and llm_api_key is empty, raise ConfigurationError at startup

Acceptance:

• AzureOpenAIConfig() instantiates with chat_deployment="" and llm_api_key=""
• TOML azure_openai.chat_deployment = "gpt-4.1" is read correctly
• Missing AZURE_OPENAI_LLM_API_KEY with use_auto_tagging=true raises at startup

---

Task 0.2: Add chat_completion() to AzureOpenAIClient

File: src/teamlandkarte_mcp/azure/openai_client.py

Subtasks:

• Add chat_deployment: str and llm_api_key: str constructor parameters
• Implement chat_completion(system: str, user: str) -> str method:
  • Calls Azure OpenAI chat completions API with response_format={"type": "json_object"}
  • Uses self.chat_deployment as the model/deployment
  • Uses self.llm_api_key for auth (separate from embedding key)
  • Returns the raw JSON string from the first choice's message content
  • Raises AzureOpenAIError on API errors (consistent with existing error handling)
• Add Google Docstring to chat_completion()

Acceptance:

• chat_completion(system="...", user="...") returns a JSON string
• API errors surface as AzureOpenAIError
• chat_deployment="" + chat_completion() call raises ConfigurationError (misconfigured guard)

---

Phase 1: New Modules

Task 1.1: Add bm25.py

File: src/teamlandkarte_mcp/matching/bm25.py (new)

Subtasks:

• Create bm25.py with _tokenize(text: str) -> list[str] helper (lowercase, split on [\W_]+, filter empty)
• Implement Bm25Index dataclass:
  • corpus: list[str] field
  • Internal _bm25: BM25Okapi | None built in __post_init__
  • rank(query: str) -> list[tuple[str, float]] method:
    • returns [] for empty corpus
    • returns (corpus_text, bm25_score) list sorted by score descending
    • score 0.0 entries are included (caller decides cutoff)
• Implement bm25_rank_competences(required: str, candidates: list[str]) -> list[tuple[str, float]] convenience function
• Add Google Docstrings to all public symbols

Acceptance:

• Bm25Index(corpus=[]).rank("Python") → []
• Bm25Index(corpus=["Python"]).rank("Python") → [("Python", score > 0)]
• Bm25Index(corpus=["JavaScript"]).rank("Python") → [("JavaScript", 0.0)]
• Bm25Index(corpus=["Machine Learning"]).rank("machine learning") → score > 0 (case-insensitive)

---

Task 1.2: Add rrf.py

File: src/teamlandkarte_mcp/matching/rrf.py (new)

Subtasks:

• Implement reciprocal_rank_fusion(ranked_lists, *, k=60) -> dict[str, float]:
  • Skip any list where all scores are 0.0 (no token-overlap signal)
  • For each candidate in each non-zero list, accumulate 1 / (k + rank) (1-based rank)
  • Normalize so the highest-scoring candidate maps to 1.0
  • Return {} if all lists are empty or all-zero
• Add Google Docstring including mathematical formula, zero-out rule, and normalization description

Acceptance:

• Single list, one candidate → {"A": 1.0}
• Single list [("A", 1.0), ("B", 0.5)] → {"A": 1.0, "B": ...} with 0 < B < 1
• Single all-zero list [("A", 0.0), ("B", 0.0)] → {} (empty, no signal)
• k=60 default: rank-1 score = 1/61, rank-2 score = 1/62, normalized → 61/62 ≈ 0.984

---

Task 1.3: Add auto_tagger.py

File: src/teamlandkarte_mcp/matching/auto_tagger.py (new)

Subtasks:

• Implement AutoTagger class:
  • Constructor: __init__(self, client: AzureOpenAIClient) -> None
  • Stores the client reference
• Implement expand_competences(self, required: list[str], existing: list[str]) -> list[str]:
  • Build system prompt: instructs LLM to return only required-competence names already covered by existing entries (synonym/abbreviation/cross-language), as JSON {"additions": ["...", ...]}
  • Build user prompt: REQUIRED: {required}\nEXISTING: {existing}
  • Call self._client.chat_completion(system=..., user=...) → JSON string
  • Parse JSON, extract additions list
  • Validate: keep only items present in required (guard against LLM hallucination)
  • Return existing + [a for a in additions if a not in existing]
  • On any exception (API error, JSON parse, key error): log warning, return existing unchanged
• Add Google Docstrings to class and all public methods

Acceptance:

• LLM returns {"additions": ["Machine Learning"]} for existing ["ML"], required ["Machine Learning"] → returns ["ML", "Machine Learning"]
• LLM returns {"additions": []} → returns existing unchanged
• LLM raises exception → returns existing unchanged (no exception propagation)
• Addition not in required is silently dropped (hallucination guard)
• No duplicate entries if existing already contains the addition

---

Phase 2: Configuration Extension

Task 2.1: Extend SimilarityConfig

File: src/teamlandkarte_mcp/config.py

Subtasks:

• Add use_bm25_search: bool = False field to SimilarityConfig dataclass
• Add use_auto_tagging: bool = False field to SimilarityConfig dataclass
• Add Google Docstring comments to both fields
• Update the TOML parser to read matching.similarity.use_bm25_search (default False)
• Update the TOML parser to read matching.similarity.use_auto_tagging (default False)

Acceptance:

• SimilarityConfig() instantiates with both flags False
• SimilarityConfig(use_bm25_search=True, use_auto_tagging=True) works
• Parser reads both flags from TOML correctly

---

Task 2.2: Update config.toml.example

File: config.toml.example

Subtasks:

• Add use_bm25_search = false under [matching.similarity] with inline comment
• Add use_auto_tagging = false under [matching.similarity] with inline comment:

  # When true, BM25 + RRF replaces embedding-based competence similarity.
  # BM25 assigns 0 to candidates with no token overlap, eliminating false positives.
  # Role similarity is always embedding-based. Default: false.
  use_bm25_search = false
  
  # When true (and use_bm25_search = true), an LLM pre-expands each candidate's
  # competence list with canonical equivalents of required competences it already covers
  # (synonyms, abbreviations, cross-language). The expansion is ephemeral. Default: false.
  use_auto_tagging = false
  

• Add chat_deployment = "" under [azure_openai] with inline comment:

  # Azure deployment name for the chat model used in auto-tagging (e.g. "gpt-4.1").
  # Required when use_auto_tagging = true.
  chat_deployment = ""
  

Acceptance:

• Config example is self-documenting for all three new keys
• All new defaults are false / ""

---

Task 2.3: Wire use_bm25_search, use_auto_tagging, and AutoTagger in mcp_server.py

File: src/teamlandkarte_mcp/mcp_server.py

Subtasks:

• When use_auto_tagging=True: construct AzureOpenAIClient with chat_deployment and llm_api_key; construct AutoTagger(client=llm_client)
• Pass auto_tagger (or None) and both flags to SimilarityEngine constructor
• When use_auto_tagging=False: auto_tagger=None — no LLM client constructed

Acceptance:

• Setting use_auto_tagging = true in config.toml activates auto-tagging at runtime
• Setting use_auto_tagging = false (default) does not construct any LLM client

---

Phase 3: Similarity Engine Integration

Task 3.1: Add _bm25_rrf_similarity() to SimilarityEngine

File: src/teamlandkarte_mcp/matching/similarity.py

Subtasks:

• Import bm25_rank_competences from .bm25, reciprocal_rank_fusion from .rrf, and AutoTagger from .auto_tagger
• Implement _bm25_rrf_similarity(self, required: list[str], candidate: list[str]) -> dict[str, dict[str, object]]:
  • For each req in required:
    1. Call bm25_rank_competences(req, candidate) → ranked
    2. Call reciprocal_rank_fusion([ranked]) → fused: dict[str, float]
    3. If fused is empty or all scores are 0.0:
       • score = 0.0, best_match = None, rationale = "BM25: no token overlap with any candidate competence."
    4. Otherwise:
       • best_match = max(fused, key=fused.get)
       • score = fused[best_match]
       • rationale = f"BM25+RRF: best match '{best_match}' (score {score:.3f})."
  • Returns dict shaped identically to _per_skill_similarity() output
• Add Google Docstring to _bm25_rrf_similarity()

---

Task 3.2: Route compute_competence_similarity() by flags

File: src/teamlandkarte_mcp/matching/similarity.py

Subtasks:

• Ensure SimilarityEngine.__init__ accepts and stores use_bm25_search, use_auto_tagging, and auto_tagger: AutoTagger | None = None
• In compute_competence_similarity():

  if self.config.use_bm25_search:
      working = candidate
      if self.config.use_auto_tagging and self._auto_tagger is not None:
          working = self._auto_tagger.expand_competences(required, candidate)
      return self._bm25_rrf_similarity(required, working)
  # else: existing strategy dispatch (unchanged)
  

Acceptance:

• use_bm25_search=False: behavior bit-for-bit identical to before
• use_bm25_search=True, use_auto_tagging=False: BM25 on raw candidate list
• use_bm25_search=True, use_auto_tagging=True: BM25 on LLM-expanded list
• use_bm25_search=True + required=["Python"], candidate=["JavaScript"]: score == 0.0
• use_bm25_search=True + required=["Python"], candidate=["Python"]: score == 1.0

---

Phase 4: Legacy Cleanup

Task 4.1: Remove legacy debug files

Files to delete:

• debug_similarity_analysis.py
• debug_categorization.py
• SEMANTIC_SIMILARITY_ANALYSIS.md
• SCORE_BUG_FIX.md

Subtasks:

• Verify none of these files are imported or referenced in source code or tests
• Delete debug_similarity_analysis.py
• Delete debug_categorization.py
• Delete SEMANTIC_SIMILARITY_ANALYSIS.md
• Delete SCORE_BUG_FIX.md

Acceptance:

• git status shows 4 deleted files
• No import errors in existing code

---

Phase 5: Tests

Task 5.1: Unit tests for bm25.py

File: tests/test_bm25.py (new)

Subtasks:

• Test Bm25Index with empty corpus → rank() returns []
• Test exact match (single token) → score > 0
• Test no token overlap (query "Python", corpus "JavaScript") → score == 0.0
• Test partial token overlap multi-token query ("Machine Learning") against ("Deep Learning") → partial score > 0
• Test case-insensitivity: rank("Python") matches "python" in corpus
• Test tokenization edge cases:
  • "Progressive Web App (PWA)" → ["progressive", "web", "app", "pwa"]
  • "CI/CD Pipeline" → ["ci", "cd", "pipeline"]
  • "React.js" → ["react", "js"]
• Test bm25_rank_competences() convenience wrapper produces same result as Bm25Index.rank()
• Add Google Docstrings to test module

Task 5.2: Unit tests for rrf.py

File: tests/test_rrf.py (new)

Subtasks:

• Test single list, single candidate → {"A": 1.0}
• Test single list multiple candidates: top candidate maps to 1.0, others < 1.0 and > 0.0
• Test all-zero list → returns {} (empty)
• Test mixed: one non-zero list + one all-zero list → only non-zero list contributes
• Test k parameter: higher k flattens differences (rank-1 and rank-2 scores closer together)
• Test score ordering is preserved (rank 1 > rank 2 > rank 3 in output)
• Add Google Docstrings to test module

Task 5.3: Unit tests for auto_tagger.py

File: tests/test_auto_tagger.py (new)

Subtasks:

• Mock AzureOpenAIClient.chat_completion() for all tests (no real LLM calls)
• Test: LLM returns {"additions": ["Machine Learning"]} for existing ["ML"], required ["Machine Learning"] → result is ["ML", "Machine Learning"]
• Test: LLM returns {"additions": []} → result is identical to existing
• Test: LLM chat_completion raises exception → result is identical to existing, no exception propagated
• Test: LLM returns malformed JSON (not parseable) → result is existing unchanged
• Test: LLM returns addition not in required (hallucination) → addition is dropped
• Test: LLM returns addition already in existing → no duplicate in result
• Test: empty required list → expand_competences() returns existing unchanged (no LLM call needed)
• Add Google Docstrings to test module

Task 5.4: Extend tests/test_similarity.py

File: tests/test_similarity.py (extended)

Subtasks:

• Add tests with use_bm25_search=True:
  • Required ["Python"], candidate ["JavaScript"] → competence_score == 0.0
  • Required ["Python"], candidate ["Python"] → competence_score == 1.0
  • Required ["Python", "Machine Learning"], candidate ["JavaScript", "TypeScript"] → both individual scores 0.0, overall competence_score == 0.0
  • Required ["Python", "Machine Learning"], candidate ["Python", "Machine Learning"] → both scores 1.0
• Add tests with use_bm25_search=True, use_auto_tagging=True (mocked AutoTagger):
  • Mocked AutoTagger.expand_competences injects "Machine Learning" into ["ML"]
  • Required ["Machine Learning"], candidate ["ML"] (after expansion) → score > 0
• Confirm that with use_bm25_search=False, behavior is unchanged (existing tests still pass)

Task 5.5: Extend tests/test_matcher_integration.py

File: tests/test_matcher_integration.py (extended)

Subtasks:

• Add integration test with use_bm25_search=True:
  • Task requires: ["Python", "Machine Learning"]
  • Candidate A has: ["Python", "Machine Learning", "Pandas"] → high score
  • Candidate B has: ["JavaScript", "TypeScript", "Node.js", "Vue.js"] → score 0.0
  • Assert: candidate A competence_score > candidate B competence_score
  • Assert: candidate B competence_score == 0.0 (false-positive eliminated)
• Add integration test with use_bm25_search=True, use_auto_tagging=True (mocked LLM via mocked AutoTagger):
  • Candidate has ["ML", "Python"]; auto-tag adds "Machine Learning" to working list
  • Required ["Machine Learning", "Python"] → both scores > 0
• These tests explicitly document the false-positive scenario from production

---

Phase 6: Documentation

Task 6.1: Update docs/architecture.md

File: docs/architecture.md

Subtasks:

• Add section "BM25 + RRF + Auto-Tagging Competence Matching" under the matching/similarity section:
  • Describe the three new modules (auto_tagger.py, bm25.py, rrf.py)
  • Describe the auto-tag → BM25 → RRF pipeline and routing in compute_competence_similarity()
  • Note that role similarity is unaffected
  • Reference use_bm25_search and use_auto_tagging config keys

Task 6.2: Update docs/semantic_similarity_tradeoffs.md

File: docs/semantic_similarity_tradeoffs.md

Subtasks:

• Add BM25+RRF row to the strategy comparison table:
  • Strengths: exact/lexical match, no false positives for non-overlapping skills
  • Weaknesses: misses synonyms and cross-language variants (mitigated by use_auto_tagging)
  • When to use: when skill names in the dataset are mostly canonical English terms
• Add BM25+RRF+AutoTag row:
  • Strengths: exact match + synonym/abbreviation/cross-language bridging via LLM
  • Weaknesses: LLM latency per candidate, requires AZURE_OPENAI_LLM_API_KEY
  • When to use: mixed-language or abbreviation-heavy skill datasets

Task 6.3: Update docs/assistant_system_prompt.md

File: docs/assistant_system_prompt.md

Subtasks:

• Add note on score interpretation when use_bm25_search=true:
  • A competence_score of 0.0 means no lexical token overlap between required and candidate competences
  • The score is not a semantic similarity — it is a lexical matching score
  • Users should be informed when BM25 mode is active so they understand why cross-language synonyms may not match
• Add note on use_auto_tagging=true:
  • Auto-tagging bridges common synonym/abbreviation gaps (e.g. "ML" → "Machine Learning")
  • A positive competence_score with auto-tagging enabled may reflect an LLM-inferred equivalence, not a literal token match; the rationale field will say "BM25+RRF" regardless

Task 6.4: Update README.md

File: README.md

Subtasks:

• Add all three new config keys to the config reference section:
  • matching.similarity.use_bm25_search (boolean, default false): Enables BM25+RRF lexical competence matching
  • matching.similarity.use_auto_tagging (boolean, default false): Enables LLM pre-expansion before BM25
  • azure_openai.chat_deployment (string, default ""): Chat model deployment name for auto-tagging
• Document AZURE_OPENAI_LLM_API_KEY environment variable in the env-vars section

---

Phase 7: Dependency

Task 7.1: Add rank-bm25 to project dependencies

File: pyproject.toml

Subtasks:

• Add rank-bm25 to [project.dependencies] in pyproject.toml
• Run uv lock (or equivalent) to update the lockfile
• Verify from rank_bm25 import BM25Okapi works in the environment

Acceptance:

• uv run python -c "from rank_bm25 import BM25Okapi; print('ok')" exits 0

---

Phase 8: Quality Gates

Task 8.1: Run unit tests

• uv run pytest tests/test_bm25.py tests/test_rrf.py tests/test_auto_tagger.py -v
• uv run pytest -m "not integration" -q
• All tests pass

Task 8.2: Run linters

• uv run ruff check src/ tests/
• uv run mypy src/teamlandkarte_mcp/matching/bm25.py src/teamlandkarte_mcp/matching/rrf.py src/teamlandkarte_mcp/matching/auto_tagger.py
• Zero errors

Task 8.3: OpenSpec validation

• openspec validate add-bm25-rrf-competence-matching --strict
• All checks pass

Task 8.4: Confirm legacy files are removed

• ls debug_*.py → no such files
• ls SEMANTIC_SIMILARITY_ANALYSIS.md SCORE_BUG_FIX.md → no such files

---

Phase 9: Architectural Fix — Global BM25 Index

Problem identified post-spec: When Bm25Index was built per candidate (N = number of that person's competences), IDF values were always 0 or negative for every term — the library rank_bm25 computes log((N - df + 0.5) / (df + 0.5)), and with N=1 every term has df=1, yielding log(0) = negative. All BM25 scores were therefore ≤ 0, making the max(0.0, raw) clamp produce a flat 0.0 for everything. The fix builds one global corpus from the union of all filtered candidates' competences, ensuring N is large enough for IDF to be meaningful.

Task 9.1: Remove 1e-10 clamp in bm25.py

File: src/teamlandkarte_mcp/matching/bm25.py

Subtasks:

• Remove the special-case block that returned 1e-10 for zero/negative BM25 scores when token overlap existed (a band-aid for the IDF pathology)
• Replace with max(0.0, float(raw)) — ubiquitous terms (negative IDF) simply score 0.0
• Update Bm25Index and bm25_rank_competences() docstrings to reflect the change

Acceptance:

• A term present in every document of a sufficiently large corpus scores 0.0, not 1e-10

---

Task 9.2: Add use_bm25_search property to SimilarityEngine

File: src/teamlandkarte_mcp/matching/similarity.py

Subtasks:

• Expose use_bm25_search: bool as a read-only property (or attribute) on SimilarityEngine so Matcher can check it without accessing internal config directly

Acceptance:

• engine.use_bm25_search returns the value of engine.config.use_bm25_search

---

Task 9.3: Thread global_index parameter through similarity layer

File: src/teamlandkarte_mcp/matching/similarity.py

Subtasks:

• Add global_index: Bm25Index | None = None parameter to compute_competence_similarity()
• Pass it through to _bm25_rrf_similarity()
• In _bm25_rrf_similarity(): when global_index is not None, call global_index.rank(req) and filter the result list to only entries whose text is in set(candidate); otherwise fall back to bm25_rank_competences(req, candidate)

Acceptance:

• global_index.rank(req) is called once per required competence, not once per candidate
• Results are correctly filtered to the current candidate's competence set
• Fall-back path (global_index=None) still works unchanged

---

Task 9.4: Build global corpus in Matcher before per-candidate loop

File: src/teamlandkarte_mcp/matching/matcher.py

Subtasks:

• Before the for cap in filtered loop: if self._sim.use_bm25_search and filtered is non-empty, collect the deduplicated union of all candidates' competences into a list global_corpus, instantiate Bm25Index(corpus=global_corpus)
• Pass global_index to every compute_competence_similarity() call inside the loop
• When use_bm25_search=False (or filtered is empty): global_bm25_index = None (no-op, existing behaviour)

Acceptance:

• A corpus of N candidates' competences is built once, not N times
• Candidates with no overlap with required competences still score 0.0
• Candidates with overlap score > 0.0 (IDF is meaningful because N >> 1 in the global corpus)

---

Task 9.5: Update tests for global index

Files: tests/test_bm25.py, tests/test_similarity.py, tests/test_matcher_integration.py, tests/test_matching_refinements.py

Subtasks:

• test_bm25.py: update small-corpus tests to use ≥ 5-doc corpora so IDF is positive; add test_bm25_index_rank_filters_to_candidate_subset and test_bm25_ubiquitous_term_clamped_to_zero
• test_similarity.py: update 4 BM25 tests to pass a 5-doc global_index; add test_use_bm25_search_property_reflects_flag and test_bm25_global_index_filters_to_candidate_subset
• test_matcher_integration.py: add use_bm25_search = False to _FakeSimilarityEngine, add global_index=None param to compute_competence_similarity; update test_bm25_auto_tag_candidate_matches_after_expansion to use a 5-doc global corpus; add test_matcher_builds_global_bm25_index_across_all_candidates
• test_matching_refinements.py: add use_bm25_search = False and global_index=None to local _FakeSim stub

Acceptance:

• All 140 tests pass, 2 skipped

---

Task 9.6: Update docs/architecture.md

File: docs/architecture.md

Subtasks:

• Replace the incorrect "Per-call index" bullet in §9.3 with a "Global BM25 index" bullet describing the pre-loop construction and per-candidate filtering
• Update the §9.1 module table row for bm25.py to remove the 1e-10 clamp description and replace with max(0.0, raw) description
• Update the §9.2 data flow diagram to show match_candidates building the global index before the per-candidate loop

Acceptance:

• No mention of "per-call index" or 1e-10 clamp remains in the docs