Orchestrator/bahn/teamlandkarte-mcp/openspec/changes/add-bm25-rrf-competence-matching/tasks.md

# Implementation Tasks: BM25 + RRF Competence Matching

## Status: Implemented

## Summary

This change introduces an optional BM25-based competence matching strategy with optional
LLM-based Auto-Tagging as an alternative to the existing embedding-based approach:

1. **Restore LLM client**: `AzureOpenAIClient.chat_completion()` restored; `AzureOpenAIConfig.chat_deployment` + `llm_api_key` added.
2. **New `auto_tagger.py` module**: `AutoTagger` class with `expand_competences()` — ephemeral LLM-driven competence expansion.
3. **New `bm25.py` module**: `Bm25Index` dataclass + `bm25_rank_competences()` convenience function.
4. **New `rrf.py` module**: `reciprocal_rank_fusion()` fuses ranked lists into a normalized score dict.
5. **`similarity.py` extended**: `_bm25_rrf_similarity()` internal method + auto-tag branch + flag routing in `compute_competence_similarity()`.
6. **`SimilarityConfig` extended**: new `use_bm25_search: bool = False` and `use_auto_tagging: bool = False` fields.
7. **`config.py` parser extended**: reads `use_bm25_search`, `use_auto_tagging`, `chat_deployment`, and `AZURE_OPENAI_LLM_API_KEY`.
8. **`config.toml.example` updated**: documents all new keys.
9. **`mcp_server.py` updated**: constructs `AutoTagger` when `use_auto_tagging=True`; passes flags to `SimilarityEngine`.
10. **Legacy debug files removed**: 4 root-level files no longer part of the module.
11. **Tests added**: `tests/test_bm25.py`, `tests/test_rrf.py`, `tests/test_auto_tagger.py`, extensions to `tests/test_similarity.py` and `tests/test_matcher_integration.py`.
12. **Documentation updated**: architecture, tradeoffs doc, assistant prompt, README.

## Clarifications / Decisions

- **BM25 scope**: BM25 + RRF replaces **competence** similarity only. Role similarity is
  always embedding-based, regardless of `use_bm25_search`.
- **Auto-tagging scope**: `use_auto_tagging` only applies when `use_bm25_search = true`.
  When `use_auto_tagging = true`, `AutoTagger.expand_competences()` is called once per
  candidate per `compute_competence_similarity()` invocation. Result is ephemeral.
- **Auto-tagging works in both directions**: both `find_matching_capacities` and
  `find_matching_tasks` pass through `compute_competence_similarity()` and benefit.
- **Graceful degradation**: if the LLM call in `AutoTagger` fails for any reason, the
  original candidate list is used unchanged. BM25 scoring continues.
- **LLM client restored**: `AzureOpenAIClient.chat_completion()` is re-added, scoped
  exclusively to auto-tagging. The embedding-related API surface is unchanged.
- **Zero-out rule**: If all BM25 scores for a required competence are 0.0 (no token
  overlap at all), the RRF score for that required competence is also 0.0.
- **Global index** *(revised)*: Originally the `Bm25Index` was to be built fresh per
  `compute_competence_similarity()` call. This was identified as an IDF pathology: with
  only one person's competences as the corpus, BM25 IDF is always 0 or negative for any
  term that appears in every document (and with N=1 all terms do). The fix (Phase 9) builds
  one `Bm25Index` over the union of all filtered candidates' competences before the loop
  and passes it in as `global_index`.
- **Single ranked list**: Initial implementation fuses one BM25 ranked list per required
  competence. The `reciprocal_rank_fusion()` signature accepts multiple lists for future
  hybrid extension.
- **Output shape unchanged**: `_bm25_rrf_similarity()` returns the same
  `dict[str, dict[str, object]]` shape as `_per_skill_similarity()`. Matcher, scorer,
  and all output formatting remain unchanged.
- **`rank-bm25` dependency**: Added to `pyproject.toml` `[project.dependencies]`.
- **All new code uses Google Docstrings** per project convention.
- **Default `false`**: Existing deployments are unaffected until opt-in.

## Estimated Effort

- **Phase 0 (Restore LLM client)**: 1 hour
- **Phase 1 (New modules — bm25, rrf, auto_tagger)**: 3–4 hours
- **Phase 2 (Config + wiring)**: 1–1.5 hours
- **Phase 3 (Similarity integration)**: 2 hours
- **Phase 4 (Legacy cleanup)**: 0.5 hour
- **Phase 5 (Tests)**: 4–5 hours
- **Phase 6 (Documentation)**: 2 hours
- **Phase 7 (Dependency)**: 0.5 hour
- **Phase 8 (Quality gates)**: 0.5 hour
- **Total**: 15–17 hours

---

## Phase 0: Restore LLM Client

### Task 0.1: Extend `AzureOpenAIConfig`
**File:** `src/teamlandkarte_mcp/config.py`

**Subtasks:**
- [x] Add `chat_deployment: str = ""` field to `AzureOpenAIConfig` dataclass
- [x] Add Google Docstring field comment:
  `"""Azure deployment name for the chat model used in auto-tagging (e.g. 'gpt-4.1'). Required when use_auto_tagging = true."""`
- [x] Update the TOML parser to read `azure_openai.chat_deployment` with default `""`
- [x] Read `AZURE_OPENAI_LLM_API_KEY` from environment (store as `llm_api_key: str = ""` on `AzureOpenAIConfig`)
- [x] When `use_auto_tagging = true` and `llm_api_key` is empty, raise `ConfigurationError` at startup

**Acceptance:**
- `AzureOpenAIConfig()` instantiates with `chat_deployment=""` and `llm_api_key=""`
- TOML `azure_openai.chat_deployment = "gpt-4.1"` is read correctly
- Missing `AZURE_OPENAI_LLM_API_KEY` with `use_auto_tagging=true` raises at startup

---

### Task 0.2: Add `chat_completion()` to `AzureOpenAIClient`
**File:** `src/teamlandkarte_mcp/azure/openai_client.py`

**Subtasks:**
- [x] Add `chat_deployment: str` and `llm_api_key: str` constructor parameters
- [x] Implement `chat_completion(system: str, user: str) -> str` method:
  - Calls Azure OpenAI chat completions API with `response_format={"type": "json_object"}`
  - Uses `self.chat_deployment` as the model/deployment
  - Uses `self.llm_api_key` for auth (separate from embedding key)
  - Returns the raw JSON string from the first choice's message content
  - Raises `AzureOpenAIError` on API errors (consistent with existing error handling)
- [x] Add Google Docstring to `chat_completion()`

**Acceptance:**
- `chat_completion(system="...", user="...")` returns a JSON string
- API errors surface as `AzureOpenAIError`
- `chat_deployment=""` + `chat_completion()` call raises `ConfigurationError` (misconfigured guard)

---

## Phase 1: New Modules

### Task 1.1: Add `bm25.py`
**File:** `src/teamlandkarte_mcp/matching/bm25.py` (**new**)

**Subtasks:**
- [x] Create `bm25.py` with `_tokenize(text: str) -> list[str]` helper (lowercase, split on `[\W_]+`, filter empty)
- [x] Implement `Bm25Index` dataclass:
  - `corpus: list[str]` field
  - Internal `_bm25: BM25Okapi | None` built in `__post_init__`
  - `rank(query: str) -> list[tuple[str, float]]` method:
    - returns `[]` for empty corpus
    - returns `(corpus_text, bm25_score)` list sorted by score descending
    - score 0.0 entries are included (caller decides cutoff)
- [x] Implement `bm25_rank_competences(required: str, candidates: list[str]) -> list[tuple[str, float]]` convenience function
- [x] Add Google Docstrings to all public symbols

**Acceptance:**
- `Bm25Index(corpus=[]).rank("Python")` → `[]`
- `Bm25Index(corpus=["Python"]).rank("Python")` → `[("Python", score > 0)]`
- `Bm25Index(corpus=["JavaScript"]).rank("Python")` → `[("JavaScript", 0.0)]`
- `Bm25Index(corpus=["Machine Learning"]).rank("machine learning")` → score > 0 (case-insensitive)

---

### Task 1.2: Add `rrf.py`
**File:** `src/teamlandkarte_mcp/matching/rrf.py` (**new**)

**Subtasks:**
- [x] Implement `reciprocal_rank_fusion(ranked_lists, *, k=60) -> dict[str, float]`:
  - Skip any list where all scores are 0.0 (no token-overlap signal)
  - For each candidate in each non-zero list, accumulate `1 / (k + rank)` (1-based rank)
  - Normalize so the highest-scoring candidate maps to 1.0
  - Return `{}` if all lists are empty or all-zero
- [x] Add Google Docstring including mathematical formula, zero-out rule, and normalization description

**Acceptance:**
- Single list, one candidate → `{"A": 1.0}`
- Single list `[("A", 1.0), ("B", 0.5)]` → `{"A": 1.0, "B": ...}` with `0 < B < 1`
- Single all-zero list `[("A", 0.0), ("B", 0.0)]` → `{}` (empty, no signal)
- `k=60` default: rank-1 score = `1/61`, rank-2 score = `1/62`, normalized → `61/62 ≈ 0.984`

---

### Task 1.3: Add `auto_tagger.py`
**File:** `src/teamlandkarte_mcp/matching/auto_tagger.py` (**new**)

**Subtasks:**
- [x] Implement `AutoTagger` class:
  - Constructor: `__init__(self, client: AzureOpenAIClient) -> None`
  - Stores the client reference
- [x] Implement `expand_competences(self, required: list[str], existing: list[str]) -> list[str]`:
  - Build system prompt: instructs LLM to return only required-competence names already
    covered by existing entries (synonym/abbreviation/cross-language), as JSON
    `{"additions": ["...", ...]}`
  - Build user prompt: `REQUIRED: {required}\nEXISTING: {existing}`
  - Call `self._client.chat_completion(system=..., user=...)` → JSON string
  - Parse JSON, extract `additions` list
  - Validate: keep only items present in `required` (guard against LLM hallucination)
  - Return `existing + [a for a in additions if a not in existing]`
  - On any exception (API error, JSON parse, key error): log warning, return `existing` unchanged
- [x] Add Google Docstrings to class and all public methods

**Acceptance:**
- LLM returns `{"additions": ["Machine Learning"]}` for existing `["ML"]`, required `["Machine Learning"]`
  → returns `["ML", "Machine Learning"]`
- LLM returns `{"additions": []}` → returns `existing` unchanged
- LLM raises exception → returns `existing` unchanged (no exception propagation)
- Addition not in `required` is silently dropped (hallucination guard)
- No duplicate entries if `existing` already contains the addition

---

## Phase 2: Configuration Extension

### Task 2.1: Extend `SimilarityConfig`
**File:** `src/teamlandkarte_mcp/config.py`

**Subtasks:**
- [x] Add `use_bm25_search: bool = False` field to `SimilarityConfig` dataclass
- [x] Add `use_auto_tagging: bool = False` field to `SimilarityConfig` dataclass
- [x] Add Google Docstring comments to both fields
- [x] Update the TOML parser to read `matching.similarity.use_bm25_search` (default `False`)
- [x] Update the TOML parser to read `matching.similarity.use_auto_tagging` (default `False`)

**Acceptance:**
- `SimilarityConfig()` instantiates with both flags `False`
- `SimilarityConfig(use_bm25_search=True, use_auto_tagging=True)` works
- Parser reads both flags from TOML correctly

---

### Task 2.2: Update `config.toml.example`
**File:** `config.toml.example`

**Subtasks:**
- [x] Add `use_bm25_search = false` under `[matching.similarity]` with inline comment
- [x] Add `use_auto_tagging = false` under `[matching.similarity]` with inline comment:
  ```toml
  # When true, BM25 + RRF replaces embedding-based competence similarity.
  # BM25 assigns 0 to candidates with no token overlap, eliminating false positives.
  # Role similarity is always embedding-based. Default: false.
  use_bm25_search = false

  # When true (and use_bm25_search = true), an LLM pre-expands each candidate's
  # competence list with canonical equivalents of required competences it already covers
  # (synonyms, abbreviations, cross-language). The expansion is ephemeral. Default: false.
  use_auto_tagging = false
  ```
- [x] Add `chat_deployment = ""` under `[azure_openai]` with inline comment:
  ```toml
  # Azure deployment name for the chat model used in auto-tagging (e.g. "gpt-4.1").
  # Required when use_auto_tagging = true.
  chat_deployment = ""
  ```

**Acceptance:**
- Config example is self-documenting for all three new keys
- All new defaults are `false` / `""`

---

### Task 2.3: Wire `use_bm25_search`, `use_auto_tagging`, and `AutoTagger` in `mcp_server.py`
**File:** `src/teamlandkarte_mcp/mcp_server.py`

**Subtasks:**
- [x] When `use_auto_tagging=True`: construct `AzureOpenAIClient` with `chat_deployment`
  and `llm_api_key`; construct `AutoTagger(client=llm_client)`
- [x] Pass `auto_tagger` (or `None`) and both flags to `SimilarityEngine` constructor
- [x] When `use_auto_tagging=False`: `auto_tagger=None` — no LLM client constructed

**Acceptance:**
- Setting `use_auto_tagging = true` in `config.toml` activates auto-tagging at runtime
- Setting `use_auto_tagging = false` (default) does not construct any LLM client

---

## Phase 3: Similarity Engine Integration

### Task 3.1: Add `_bm25_rrf_similarity()` to `SimilarityEngine`
**File:** `src/teamlandkarte_mcp/matching/similarity.py`

**Subtasks:**
- [x] Import `bm25_rank_competences` from `.bm25`, `reciprocal_rank_fusion` from `.rrf`,
  and `AutoTagger` from `.auto_tagger`
- [x] Implement `_bm25_rrf_similarity(self, required: list[str], candidate: list[str]) -> dict[str, dict[str, object]]`:
  - For each `req` in `required`:
    1. Call `bm25_rank_competences(req, candidate)` → `ranked`
    2. Call `reciprocal_rank_fusion([ranked])` → `fused: dict[str, float]`
    3. If `fused` is empty or all scores are 0.0:
       - `score = 0.0`, `best_match = None`, `rationale = "BM25: no token overlap with any candidate competence."`
    4. Otherwise:
       - `best_match = max(fused, key=fused.get)`
       - `score = fused[best_match]`
       - `rationale = f"BM25+RRF: best match '{best_match}' (score {score:.3f})."`
  - Returns dict shaped identically to `_per_skill_similarity()` output
- [x] Add Google Docstring to `_bm25_rrf_similarity()`

---

### Task 3.2: Route `compute_competence_similarity()` by flags
**File:** `src/teamlandkarte_mcp/matching/similarity.py`

**Subtasks:**
- [x] Ensure `SimilarityEngine.__init__` accepts and stores `use_bm25_search`,
  `use_auto_tagging`, and `auto_tagger: AutoTagger | None = None`
- [x] In `compute_competence_similarity()`:
  ```python
  if self.config.use_bm25_search:
      working = candidate
      if self.config.use_auto_tagging and self._auto_tagger is not None:
          working = self._auto_tagger.expand_competences(required, candidate)
      return self._bm25_rrf_similarity(required, working)
  # else: existing strategy dispatch (unchanged)
  ```

**Acceptance:**
- `use_bm25_search=False`: behavior bit-for-bit identical to before
- `use_bm25_search=True, use_auto_tagging=False`: BM25 on raw candidate list
- `use_bm25_search=True, use_auto_tagging=True`: BM25 on LLM-expanded list
- `use_bm25_search=True` + required=`["Python"]`, candidate=`["JavaScript"]`: `score == 0.0`
- `use_bm25_search=True` + required=`["Python"]`, candidate=`["Python"]`: `score == 1.0`

---

## Phase 4: Legacy Cleanup

### Task 4.1: Remove legacy debug files
**Files to delete:**
- `debug_similarity_analysis.py`
- `debug_categorization.py`
- `SEMANTIC_SIMILARITY_ANALYSIS.md`
- `SCORE_BUG_FIX.md`

**Subtasks:**
- [x] Verify none of these files are imported or referenced in source code or tests
- [x] Delete `debug_similarity_analysis.py`
- [x] Delete `debug_categorization.py`
- [x] Delete `SEMANTIC_SIMILARITY_ANALYSIS.md`
- [x] Delete `SCORE_BUG_FIX.md`

**Acceptance:**
- `git status` shows 4 deleted files
- No import errors in existing code

---

## Phase 5: Tests

### Task 5.1: Unit tests for `bm25.py`
**File:** `tests/test_bm25.py` (**new**)

**Subtasks:**
- [x] Test `Bm25Index` with empty corpus → `rank()` returns `[]`
- [x] Test exact match (single token) → score > 0
- [x] Test no token overlap (query "Python", corpus "JavaScript") → score == 0.0
- [x] Test partial token overlap multi-token query ("Machine Learning") against ("Deep Learning") → partial score > 0
- [x] Test case-insensitivity: `rank("Python")` matches "python" in corpus
- [x] Test tokenization edge cases:
  - "Progressive Web App (PWA)" → `["progressive", "web", "app", "pwa"]`
  - "CI/CD Pipeline" → `["ci", "cd", "pipeline"]`
  - "React.js" → `["react", "js"]`
- [x] Test `bm25_rank_competences()` convenience wrapper produces same result as `Bm25Index.rank()`
- [x] Add Google Docstrings to test module

### Task 5.2: Unit tests for `rrf.py`
**File:** `tests/test_rrf.py` (**new**)

**Subtasks:**
- [x] Test single list, single candidate → `{"A": 1.0}`
- [x] Test single list multiple candidates: top candidate maps to 1.0, others < 1.0 and > 0.0
- [x] Test all-zero list → returns `{}` (empty)
- [x] Test mixed: one non-zero list + one all-zero list → only non-zero list contributes
- [x] Test `k` parameter: higher `k` flattens differences (rank-1 and rank-2 scores closer together)
- [x] Test score ordering is preserved (rank 1 > rank 2 > rank 3 in output)
- [x] Add Google Docstrings to test module

### Task 5.3: Unit tests for `auto_tagger.py`
**File:** `tests/test_auto_tagger.py` (**new**)

**Subtasks:**
- [x] Mock `AzureOpenAIClient.chat_completion()` for all tests (no real LLM calls)
- [x] Test: LLM returns `{"additions": ["Machine Learning"]}` for existing `["ML"]`,
  required `["Machine Learning"]` → result is `["ML", "Machine Learning"]`
- [x] Test: LLM returns `{"additions": []}` → result is identical to `existing`
- [x] Test: LLM `chat_completion` raises exception → result is identical to `existing`,
  no exception propagated
- [x] Test: LLM returns malformed JSON (not parseable) → result is `existing` unchanged
- [x] Test: LLM returns addition not in `required` (hallucination) → addition is dropped
- [x] Test: LLM returns addition already in `existing` → no duplicate in result
- [x] Test: empty `required` list → `expand_competences()` returns `existing` unchanged
  (no LLM call needed)
- [x] Add Google Docstrings to test module

### Task 5.4: Extend `tests/test_similarity.py`
**File:** `tests/test_similarity.py` (extended)

**Subtasks:**
- [x] Add tests with `use_bm25_search=True`:
  - Required `["Python"]`, candidate `["JavaScript"]` → `competence_score == 0.0`
  - Required `["Python"]`, candidate `["Python"]` → `competence_score == 1.0`
  - Required `["Python", "Machine Learning"]`, candidate `["JavaScript", "TypeScript"]` → both individual scores 0.0, overall `competence_score == 0.0`
  - Required `["Python", "Machine Learning"]`, candidate `["Python", "Machine Learning"]` → both scores 1.0
- [x] Add tests with `use_bm25_search=True, use_auto_tagging=True` (mocked `AutoTagger`):
  - Mocked `AutoTagger.expand_competences` injects `"Machine Learning"` into `["ML"]`
  - Required `["Machine Learning"]`, candidate `["ML"]` (after expansion) → `score > 0`
- [x] Confirm that with `use_bm25_search=False`, behavior is unchanged (existing tests still pass)

### Task 5.5: Extend `tests/test_matcher_integration.py`
**File:** `tests/test_matcher_integration.py` (extended)

**Subtasks:**
- [x] Add integration test with `use_bm25_search=True`:
  - Task requires: `["Python", "Machine Learning"]`
  - Candidate A has: `["Python", "Machine Learning", "Pandas"]` → high score
  - Candidate B has: `["JavaScript", "TypeScript", "Node.js", "Vue.js"]` → score 0.0
  - Assert: candidate A `competence_score` > candidate B `competence_score`
  - Assert: candidate B `competence_score == 0.0` (false-positive eliminated)
- [x] Add integration test with `use_bm25_search=True, use_auto_tagging=True`
  (mocked LLM via mocked `AutoTagger`):
  - Candidate has `["ML", "Python"]`; auto-tag adds `"Machine Learning"` to working list
  - Required `["Machine Learning", "Python"]` → both scores > 0
- [x] These tests explicitly document the false-positive scenario from production

---

## Phase 6: Documentation

### Task 6.1: Update `docs/architecture.md`
**File:** `docs/architecture.md`

**Subtasks:**
- [x] Add section "BM25 + RRF + Auto-Tagging Competence Matching" under the matching/similarity section:
  - Describe the three new modules (`auto_tagger.py`, `bm25.py`, `rrf.py`)
  - Describe the auto-tag → BM25 → RRF pipeline and routing in `compute_competence_similarity()`
  - Note that role similarity is unaffected
  - Reference `use_bm25_search` and `use_auto_tagging` config keys

### Task 6.2: Update `docs/semantic_similarity_tradeoffs.md`
**File:** `docs/semantic_similarity_tradeoffs.md`

**Subtasks:**
- [x] Add BM25+RRF row to the strategy comparison table:
  - Strengths: exact/lexical match, no false positives for non-overlapping skills
  - Weaknesses: misses synonyms and cross-language variants (mitigated by `use_auto_tagging`)
  - When to use: when skill names in the dataset are mostly canonical English terms
- [x] Add BM25+RRF+AutoTag row:
  - Strengths: exact match + synonym/abbreviation/cross-language bridging via LLM
  - Weaknesses: LLM latency per candidate, requires `AZURE_OPENAI_LLM_API_KEY`
  - When to use: mixed-language or abbreviation-heavy skill datasets

### Task 6.3: Update `docs/assistant_system_prompt.md`
**File:** `docs/assistant_system_prompt.md`

**Subtasks:**
- [x] Add note on score interpretation when `use_bm25_search=true`:
  - A `competence_score` of 0.0 means **no lexical token overlap** between required and candidate competences
  - The score is not a semantic similarity — it is a lexical matching score
  - Users should be informed when BM25 mode is active so they understand why cross-language synonyms may not match
- [x] Add note on `use_auto_tagging=true`:
  - Auto-tagging bridges common synonym/abbreviation gaps (e.g. "ML" → "Machine Learning")
  - A positive `competence_score` with auto-tagging enabled may reflect an LLM-inferred equivalence,
    not a literal token match; the rationale field will say "BM25+RRF" regardless

### Task 6.4: Update `README.md`
**File:** `README.md`

**Subtasks:**
- [x] Add all three new config keys to the config reference section:
  - `matching.similarity.use_bm25_search` (boolean, default `false`): Enables BM25+RRF lexical competence matching
  - `matching.similarity.use_auto_tagging` (boolean, default `false`): Enables LLM pre-expansion before BM25
  - `azure_openai.chat_deployment` (string, default `""`): Chat model deployment name for auto-tagging
- [x] Document `AZURE_OPENAI_LLM_API_KEY` environment variable in the env-vars section

---

## Phase 7: Dependency

### Task 7.1: Add `rank-bm25` to project dependencies
**File:** `pyproject.toml`

**Subtasks:**
- [x] Add `rank-bm25` to `[project.dependencies]` in `pyproject.toml`
- [x] Run `uv lock` (or equivalent) to update the lockfile
- [x] Verify `from rank_bm25 import BM25Okapi` works in the environment

**Acceptance:**
- `uv run python -c "from rank_bm25 import BM25Okapi; print('ok')"` exits 0

---

## Phase 8: Quality Gates

### Task 8.1: Run unit tests
- [x] `uv run pytest tests/test_bm25.py tests/test_rrf.py tests/test_auto_tagger.py -v`
- [x] `uv run pytest -m "not integration" -q`
- [x] All tests pass

### Task 8.2: Run linters
- [x] `uv run ruff check src/ tests/`
- [x] `uv run mypy src/teamlandkarte_mcp/matching/bm25.py src/teamlandkarte_mcp/matching/rrf.py src/teamlandkarte_mcp/matching/auto_tagger.py`
- [x] Zero errors

### Task 8.3: OpenSpec validation
- [x] `openspec validate add-bm25-rrf-competence-matching --strict`
- [x] All checks pass

### Task 8.4: Confirm legacy files are removed
- [x] `ls debug_*.py` → no such files
- [x] `ls SEMANTIC_SIMILARITY_ANALYSIS.md SCORE_BUG_FIX.md` → no such files

---

## Phase 9: Architectural Fix — Global BM25 Index

**Problem identified post-spec**: When `Bm25Index` was built per candidate (N = number of that
person's competences), IDF values were always 0 or negative for every term — the library
`rank_bm25` computes `log((N - df + 0.5) / (df + 0.5))`, and with N=1 every term has df=1,
yielding `log(0)` = negative. All BM25 scores were therefore ≤ 0, making the `max(0.0, raw)`
clamp produce a flat 0.0 for everything. The fix builds one global corpus from the union of
all filtered candidates' competences, ensuring N is large enough for IDF to be meaningful.

### Task 9.1: Remove `1e-10` clamp in `bm25.py`
**File:** `src/teamlandkarte_mcp/matching/bm25.py`

**Subtasks:**
- [x] Remove the special-case block that returned `1e-10` for zero/negative BM25 scores
  when token overlap existed (a band-aid for the IDF pathology)
- [x] Replace with `max(0.0, float(raw))` — ubiquitous terms (negative IDF) simply score 0.0
- [x] Update `Bm25Index` and `bm25_rank_competences()` docstrings to reflect the change

**Acceptance:**
- A term present in every document of a sufficiently large corpus scores 0.0, not 1e-10

---

### Task 9.2: Add `use_bm25_search` property to `SimilarityEngine`
**File:** `src/teamlandkarte_mcp/matching/similarity.py`

**Subtasks:**
- [x] Expose `use_bm25_search: bool` as a read-only property (or attribute) on `SimilarityEngine`
  so `Matcher` can check it without accessing internal config directly

**Acceptance:**
- `engine.use_bm25_search` returns the value of `engine.config.use_bm25_search`

---

### Task 9.3: Thread `global_index` parameter through similarity layer
**File:** `src/teamlandkarte_mcp/matching/similarity.py`

**Subtasks:**
- [x] Add `global_index: Bm25Index | None = None` parameter to `compute_competence_similarity()`
- [x] Pass it through to `_bm25_rrf_similarity()`
- [x] In `_bm25_rrf_similarity()`: when `global_index is not None`, call
  `global_index.rank(req)` and filter the result list to only entries whose text is in
  `set(candidate)`; otherwise fall back to `bm25_rank_competences(req, candidate)`

**Acceptance:**
- `global_index.rank(req)` is called once per required competence, not once per candidate
- Results are correctly filtered to the current candidate's competence set
- Fall-back path (`global_index=None`) still works unchanged

---

### Task 9.4: Build global corpus in `Matcher` before per-candidate loop
**File:** `src/teamlandkarte_mcp/matching/matcher.py`

**Subtasks:**
- [x] Before the `for cap in filtered` loop: if `self._sim.use_bm25_search` and `filtered`
  is non-empty, collect the deduplicated union of all candidates' competences into a list
  `global_corpus`, instantiate `Bm25Index(corpus=global_corpus)`
- [x] Pass `global_index` to every `compute_competence_similarity()` call inside the loop
- [x] When `use_bm25_search=False` (or `filtered` is empty): `global_bm25_index = None`
  (no-op, existing behaviour)

**Acceptance:**
- A corpus of N candidates' competences is built once, not N times
- Candidates with no overlap with required competences still score 0.0
- Candidates with overlap score > 0.0 (IDF is meaningful because N >> 1 in the global corpus)

---

### Task 9.5: Update tests for global index
**Files:** `tests/test_bm25.py`, `tests/test_similarity.py`, `tests/test_matcher_integration.py`, `tests/test_matching_refinements.py`

**Subtasks:**
- [x] `test_bm25.py`: update small-corpus tests to use ≥ 5-doc corpora so IDF is positive;
  add `test_bm25_index_rank_filters_to_candidate_subset` and
  `test_bm25_ubiquitous_term_clamped_to_zero`
- [x] `test_similarity.py`: update 4 BM25 tests to pass a 5-doc `global_index`; add
  `test_use_bm25_search_property_reflects_flag` and `test_bm25_global_index_filters_to_candidate_subset`
- [x] `test_matcher_integration.py`: add `use_bm25_search = False` to `_FakeSimilarityEngine`,
  add `global_index=None` param to `compute_competence_similarity`; update
  `test_bm25_auto_tag_candidate_matches_after_expansion` to use a 5-doc global corpus;
  add `test_matcher_builds_global_bm25_index_across_all_candidates`
- [x] `test_matching_refinements.py`: add `use_bm25_search = False` and `global_index=None`
  to local `_FakeSim` stub

**Acceptance:**
- All 140 tests pass, 2 skipped

---

### Task 9.6: Update `docs/architecture.md`
**File:** `docs/architecture.md`

**Subtasks:**
- [x] Replace the incorrect "Per-call index" bullet in §9.3 with a "Global BM25 index" bullet
  describing the pre-loop construction and per-candidate filtering
- [x] Update the §9.1 module table row for `bm25.py` to remove the `1e-10` clamp description
  and replace with `max(0.0, raw)` description
- [x] Update the §9.2 data flow diagram to show `match_candidates` building the global index
  before the per-candidate loop

**Acceptance:**
- No mention of "per-call index" or `1e-10` clamp remains in the docs