Migrate all repos into monorepo context folders

Bahn: aisupport, Analyse-O2C-C2S, awesome-bahn-mcp-servers, beam-mcp,
      Confluence_Bot, db-planet-mcp-server, O2C-Harness, project-audit,
      Projekt-KIQ-HP, teamlandkarte-mcp
Dhive: Jury-Voting
Privat: CV, NoteGraph (NOTE: NoteGraph needs complete redo after consolidation)
Shared: AI-Orchestrator, OrgMyLife, power_skills_and_more
Shared/references: symphony (read-only)

Bahn repos remain available as independent remotes - this monorepo
pulls them in via subtree, the originals are untouched.

bahn/teamlandkarte-mcp/openspec/changes/add-bm25-rrf-competence-matching/tasks.md

@@ -0,0 +1,586 @@
+# Implementation Tasks: BM25 + RRF Competence Matching
+
+## Status: Implemented
+
+## Summary
+
+This change introduces an optional BM25-based competence matching strategy with optional
+LLM-based Auto-Tagging as an alternative to the existing embedding-based approach:
+
+1. **Restore LLM client**: `AzureOpenAIClient.chat_completion()` restored; `AzureOpenAIConfig.chat_deployment` + `llm_api_key` added.
+2. **New `auto_tagger.py` module**: `AutoTagger` class with `expand_competences()` — ephemeral LLM-driven competence expansion.
+3. **New `bm25.py` module**: `Bm25Index` dataclass + `bm25_rank_competences()` convenience function.
+4. **New `rrf.py` module**: `reciprocal_rank_fusion()` fuses ranked lists into a normalized score dict.
+5. **`similarity.py` extended**: `_bm25_rrf_similarity()` internal method + auto-tag branch + flag routing in `compute_competence_similarity()`.
+6. **`SimilarityConfig` extended**: new `use_bm25_search: bool = False` and `use_auto_tagging: bool = False` fields.
+7. **`config.py` parser extended**: reads `use_bm25_search`, `use_auto_tagging`, `chat_deployment`, and `AZURE_OPENAI_LLM_API_KEY`.
+8. **`config.toml.example` updated**: documents all new keys.
+9. **`mcp_server.py` updated**: constructs `AutoTagger` when `use_auto_tagging=True`; passes flags to `SimilarityEngine`.
+10. **Legacy debug files removed**: 4 root-level files no longer part of the module.
+11. **Tests added**: `tests/test_bm25.py`, `tests/test_rrf.py`, `tests/test_auto_tagger.py`, extensions to `tests/test_similarity.py` and `tests/test_matcher_integration.py`.
+12. **Documentation updated**: architecture, tradeoffs doc, assistant prompt, README.
+
+## Clarifications / Decisions
+
+- **BM25 scope**: BM25 + RRF replaces **competence** similarity only. Role similarity is
+  always embedding-based, regardless of `use_bm25_search`.
+- **Auto-tagging scope**: `use_auto_tagging` only applies when `use_bm25_search = true`.
+  When `use_auto_tagging = true`, `AutoTagger.expand_competences()` is called once per
+  candidate per `compute_competence_similarity()` invocation. Result is ephemeral.
+- **Auto-tagging works in both directions**: both `find_matching_capacities` and
+  `find_matching_tasks` pass through `compute_competence_similarity()` and benefit.
+- **Graceful degradation**: if the LLM call in `AutoTagger` fails for any reason, the
+  original candidate list is used unchanged. BM25 scoring continues.
+- **LLM client restored**: `AzureOpenAIClient.chat_completion()` is re-added, scoped
+  exclusively to auto-tagging. The embedding-related API surface is unchanged.
+- **Zero-out rule**: If all BM25 scores for a required competence are 0.0 (no token
+  overlap at all), the RRF score for that required competence is also 0.0.
+- **Global index** *(revised)*: Originally the `Bm25Index` was to be built fresh per
+  `compute_competence_similarity()` call. This was identified as an IDF pathology: with
+  only one person's competences as the corpus, BM25 IDF is always 0 or negative for any
+  term that appears in every document (and with N=1 all terms do). The fix (Phase 9) builds
+  one `Bm25Index` over the union of all filtered candidates' competences before the loop
+  and passes it in as `global_index`.
+- **Single ranked list**: Initial implementation fuses one BM25 ranked list per required
+  competence. The `reciprocal_rank_fusion()` signature accepts multiple lists for future
+  hybrid extension.
+- **Output shape unchanged**: `_bm25_rrf_similarity()` returns the same
+  `dict[str, dict[str, object]]` shape as `_per_skill_similarity()`. Matcher, scorer,
+  and all output formatting remain unchanged.
+- **`rank-bm25` dependency**: Added to `pyproject.toml` `[project.dependencies]`.
+- **All new code uses Google Docstrings** per project convention.
+- **Default `false`**: Existing deployments are unaffected until opt-in.
+
+## Estimated Effort
+
+- **Phase 0 (Restore LLM client)**: 1 hour
+- **Phase 1 (New modules — bm25, rrf, auto_tagger)**: 3–4 hours
+- **Phase 2 (Config + wiring)**: 1–1.5 hours
+- **Phase 3 (Similarity integration)**: 2 hours
+- **Phase 4 (Legacy cleanup)**: 0.5 hour
+- **Phase 5 (Tests)**: 4–5 hours
+- **Phase 6 (Documentation)**: 2 hours
+- **Phase 7 (Dependency)**: 0.5 hour
+- **Phase 8 (Quality gates)**: 0.5 hour
+- **Total**: 15–17 hours
+
+---
+
+## Phase 0: Restore LLM Client
+
+### Task 0.1: Extend `AzureOpenAIConfig`
+**File:** `src/teamlandkarte_mcp/config.py`
+
+**Subtasks:**
+- [x] Add `chat_deployment: str = ""` field to `AzureOpenAIConfig` dataclass
+- [x] Add Google Docstring field comment:
+  `"""Azure deployment name for the chat model used in auto-tagging (e.g. 'gpt-4.1'). Required when use_auto_tagging = true."""`
+- [x] Update the TOML parser to read `azure_openai.chat_deployment` with default `""`
+- [x] Read `AZURE_OPENAI_LLM_API_KEY` from environment (store as `llm_api_key: str = ""` on `AzureOpenAIConfig`)
+- [x] When `use_auto_tagging = true` and `llm_api_key` is empty, raise `ConfigurationError` at startup
+
+**Acceptance:**
+- `AzureOpenAIConfig()` instantiates with `chat_deployment=""` and `llm_api_key=""`
+- TOML `azure_openai.chat_deployment = "gpt-4.1"` is read correctly
+- Missing `AZURE_OPENAI_LLM_API_KEY` with `use_auto_tagging=true` raises at startup
+
+---
+
+### Task 0.2: Add `chat_completion()` to `AzureOpenAIClient`
+**File:** `src/teamlandkarte_mcp/azure/openai_client.py`
+
+**Subtasks:**
+- [x] Add `chat_deployment: str` and `llm_api_key: str` constructor parameters
+- [x] Implement `chat_completion(system: str, user: str) -> str` method:
+  - Calls Azure OpenAI chat completions API with `response_format={"type": "json_object"}`
+  - Uses `self.chat_deployment` as the model/deployment
+  - Uses `self.llm_api_key` for auth (separate from embedding key)
+  - Returns the raw JSON string from the first choice's message content
+  - Raises `AzureOpenAIError` on API errors (consistent with existing error handling)
+- [x] Add Google Docstring to `chat_completion()`
+
+**Acceptance:**
+- `chat_completion(system="...", user="...")` returns a JSON string
+- API errors surface as `AzureOpenAIError`
+- `chat_deployment=""` + `chat_completion()` call raises `ConfigurationError` (misconfigured guard)
+
+---
+
+## Phase 1: New Modules
+
+### Task 1.1: Add `bm25.py`
+**File:** `src/teamlandkarte_mcp/matching/bm25.py` (**new**)
+
+**Subtasks:**
+- [x] Create `bm25.py` with `_tokenize(text: str) -> list[str]` helper (lowercase, split on `[\W_]+`, filter empty)
+- [x] Implement `Bm25Index` dataclass:
+  - `corpus: list[str]` field
+  - Internal `_bm25: BM25Okapi | None` built in `__post_init__`
+  - `rank(query: str) -> list[tuple[str, float]]` method:
+    - returns `[]` for empty corpus
+    - returns `(corpus_text, bm25_score)` list sorted by score descending
+    - score 0.0 entries are included (caller decides cutoff)
+- [x] Implement `bm25_rank_competences(required: str, candidates: list[str]) -> list[tuple[str, float]]` convenience function
+- [x] Add Google Docstrings to all public symbols
+
+**Acceptance:**
+- `Bm25Index(corpus=[]).rank("Python")` → `[]`
+- `Bm25Index(corpus=["Python"]).rank("Python")` → `[("Python", score > 0)]`
+- `Bm25Index(corpus=["JavaScript"]).rank("Python")` → `[("JavaScript", 0.0)]`
+- `Bm25Index(corpus=["Machine Learning"]).rank("machine learning")` → score > 0 (case-insensitive)
+
+---
+
+### Task 1.2: Add `rrf.py`
+**File:** `src/teamlandkarte_mcp/matching/rrf.py` (**new**)
+
+**Subtasks:**
+- [x] Implement `reciprocal_rank_fusion(ranked_lists, *, k=60) -> dict[str, float]`:
+  - Skip any list where all scores are 0.0 (no token-overlap signal)
+  - For each candidate in each non-zero list, accumulate `1 / (k + rank)` (1-based rank)
+  - Normalize so the highest-scoring candidate maps to 1.0
+  - Return `{}` if all lists are empty or all-zero
+- [x] Add Google Docstring including mathematical formula, zero-out rule, and normalization description
+
+**Acceptance:**
+- Single list, one candidate → `{"A": 1.0}`
+- Single list `[("A", 1.0), ("B", 0.5)]` → `{"A": 1.0, "B": ...}` with `0 < B < 1`
+- Single all-zero list `[("A", 0.0), ("B", 0.0)]` → `{}` (empty, no signal)
+- `k=60` default: rank-1 score = `1/61`, rank-2 score = `1/62`, normalized → `61/62 ≈ 0.984`
+
+---
+
+### Task 1.3: Add `auto_tagger.py`
+**File:** `src/teamlandkarte_mcp/matching/auto_tagger.py` (**new**)
+
+**Subtasks:**
+- [x] Implement `AutoTagger` class:
+  - Constructor: `__init__(self, client: AzureOpenAIClient) -> None`
+  - Stores the client reference
+- [x] Implement `expand_competences(self, required: list[str], existing: list[str]) -> list[str]`:
+  - Build system prompt: instructs LLM to return only required-competence names already
+    covered by existing entries (synonym/abbreviation/cross-language), as JSON
+    `{"additions": ["...", ...]}`
+  - Build user prompt: `REQUIRED: {required}\nEXISTING: {existing}`
+  - Call `self._client.chat_completion(system=..., user=...)` → JSON string
+  - Parse JSON, extract `additions` list
+  - Validate: keep only items present in `required` (guard against LLM hallucination)
+  - Return `existing + [a for a in additions if a not in existing]`
+  - On any exception (API error, JSON parse, key error): log warning, return `existing` unchanged
+- [x] Add Google Docstrings to class and all public methods
+
+**Acceptance:**
+- LLM returns `{"additions": ["Machine Learning"]}` for existing `["ML"]`, required `["Machine Learning"]`
+  → returns `["ML", "Machine Learning"]`
+- LLM returns `{"additions": []}` → returns `existing` unchanged
+- LLM raises exception → returns `existing` unchanged (no exception propagation)
+- Addition not in `required` is silently dropped (hallucination guard)
+- No duplicate entries if `existing` already contains the addition
+
+---
+
+## Phase 2: Configuration Extension
+
+### Task 2.1: Extend `SimilarityConfig`
+**File:** `src/teamlandkarte_mcp/config.py`
+
+**Subtasks:**
+- [x] Add `use_bm25_search: bool = False` field to `SimilarityConfig` dataclass
+- [x] Add `use_auto_tagging: bool = False` field to `SimilarityConfig` dataclass
+- [x] Add Google Docstring comments to both fields
+- [x] Update the TOML parser to read `matching.similarity.use_bm25_search` (default `False`)
+- [x] Update the TOML parser to read `matching.similarity.use_auto_tagging` (default `False`)
+
+**Acceptance:**
+- `SimilarityConfig()` instantiates with both flags `False`
+- `SimilarityConfig(use_bm25_search=True, use_auto_tagging=True)` works
+- Parser reads both flags from TOML correctly
+
+---
+
+### Task 2.2: Update `config.toml.example`
+**File:** `config.toml.example`
+
+**Subtasks:**
+- [x] Add `use_bm25_search = false` under `[matching.similarity]` with inline comment
+- [x] Add `use_auto_tagging = false` under `[matching.similarity]` with inline comment:
+  ```toml
+  # When true, BM25 + RRF replaces embedding-based competence similarity.
+  # BM25 assigns 0 to candidates with no token overlap, eliminating false positives.
+  # Role similarity is always embedding-based. Default: false.
+  use_bm25_search = false
+
+  # When true (and use_bm25_search = true), an LLM pre-expands each candidate's
+  # competence list with canonical equivalents of required competences it already covers
+  # (synonyms, abbreviations, cross-language). The expansion is ephemeral. Default: false.
+  use_auto_tagging = false
+  ```
+- [x] Add `chat_deployment = ""` under `[azure_openai]` with inline comment:
+  ```toml
+  # Azure deployment name for the chat model used in auto-tagging (e.g. "gpt-4.1").
+  # Required when use_auto_tagging = true.
+  chat_deployment = ""
+  ```
+
+**Acceptance:**
+- Config example is self-documenting for all three new keys
+- All new defaults are `false` / `""`
+
+---
+
+### Task 2.3: Wire `use_bm25_search`, `use_auto_tagging`, and `AutoTagger` in `mcp_server.py`
+**File:** `src/teamlandkarte_mcp/mcp_server.py`
+
+**Subtasks:**
+- [x] When `use_auto_tagging=True`: construct `AzureOpenAIClient` with `chat_deployment`
+  and `llm_api_key`; construct `AutoTagger(client=llm_client)`
+- [x] Pass `auto_tagger` (or `None`) and both flags to `SimilarityEngine` constructor
+- [x] When `use_auto_tagging=False`: `auto_tagger=None` — no LLM client constructed
+
+**Acceptance:**
+- Setting `use_auto_tagging = true` in `config.toml` activates auto-tagging at runtime
+- Setting `use_auto_tagging = false` (default) does not construct any LLM client
+
+---
+
+## Phase 3: Similarity Engine Integration
+
+### Task 3.1: Add `_bm25_rrf_similarity()` to `SimilarityEngine`
+**File:** `src/teamlandkarte_mcp/matching/similarity.py`
+
+**Subtasks:**
+- [x] Import `bm25_rank_competences` from `.bm25`, `reciprocal_rank_fusion` from `.rrf`,
+  and `AutoTagger` from `.auto_tagger`
+- [x] Implement `_bm25_rrf_similarity(self, required: list[str], candidate: list[str]) -> dict[str, dict[str, object]]`:
+  - For each `req` in `required`:
+    1. Call `bm25_rank_competences(req, candidate)` → `ranked`
+    2. Call `reciprocal_rank_fusion([ranked])` → `fused: dict[str, float]`
+    3. If `fused` is empty or all scores are 0.0:
+       - `score = 0.0`, `best_match = None`, `rationale = "BM25: no token overlap with any candidate competence."`
+    4. Otherwise:
+       - `best_match = max(fused, key=fused.get)`
+       - `score = fused[best_match]`
+       - `rationale = f"BM25+RRF: best match '{best_match}' (score {score:.3f})."`
+  - Returns dict shaped identically to `_per_skill_similarity()` output
+- [x] Add Google Docstring to `_bm25_rrf_similarity()`
+
+---
+
+### Task 3.2: Route `compute_competence_similarity()` by flags
+**File:** `src/teamlandkarte_mcp/matching/similarity.py`
+
+**Subtasks:**
+- [x] Ensure `SimilarityEngine.__init__` accepts and stores `use_bm25_search`,
+  `use_auto_tagging`, and `auto_tagger: AutoTagger | None = None`
+- [x] In `compute_competence_similarity()`:
+  ```python
+  if self.config.use_bm25_search:
+      working = candidate
+      if self.config.use_auto_tagging and self._auto_tagger is not None:
+          working = self._auto_tagger.expand_competences(required, candidate)
+      return self._bm25_rrf_similarity(required, working)
+  # else: existing strategy dispatch (unchanged)
+  ```
+
+**Acceptance:**
+- `use_bm25_search=False`: behavior bit-for-bit identical to before
+- `use_bm25_search=True, use_auto_tagging=False`: BM25 on raw candidate list
+- `use_bm25_search=True, use_auto_tagging=True`: BM25 on LLM-expanded list
+- `use_bm25_search=True` + required=`["Python"]`, candidate=`["JavaScript"]`: `score == 0.0`
+- `use_bm25_search=True` + required=`["Python"]`, candidate=`["Python"]`: `score == 1.0`
+
+---
+
+## Phase 4: Legacy Cleanup
+
+### Task 4.1: Remove legacy debug files
+**Files to delete:**
+- `debug_similarity_analysis.py`
+- `debug_categorization.py`
+- `SEMANTIC_SIMILARITY_ANALYSIS.md`
+- `SCORE_BUG_FIX.md`
+
+**Subtasks:**
+- [x] Verify none of these files are imported or referenced in source code or tests
+- [x] Delete `debug_similarity_analysis.py`
+- [x] Delete `debug_categorization.py`
+- [x] Delete `SEMANTIC_SIMILARITY_ANALYSIS.md`
+- [x] Delete `SCORE_BUG_FIX.md`
+
+**Acceptance:**
+- `git status` shows 4 deleted files
+- No import errors in existing code
+
+---
+
+## Phase 5: Tests
+
+### Task 5.1: Unit tests for `bm25.py`
+**File:** `tests/test_bm25.py` (**new**)
+
+**Subtasks:**
+- [x] Test `Bm25Index` with empty corpus → `rank()` returns `[]`
+- [x] Test exact match (single token) → score > 0
+- [x] Test no token overlap (query "Python", corpus "JavaScript") → score == 0.0
+- [x] Test partial token overlap multi-token query ("Machine Learning") against ("Deep Learning") → partial score > 0
+- [x] Test case-insensitivity: `rank("Python")` matches "python" in corpus
+- [x] Test tokenization edge cases:
+  - "Progressive Web App (PWA)" → `["progressive", "web", "app", "pwa"]`
+  - "CI/CD Pipeline" → `["ci", "cd", "pipeline"]`
+  - "React.js" → `["react", "js"]`
+- [x] Test `bm25_rank_competences()` convenience wrapper produces same result as `Bm25Index.rank()`
+- [x] Add Google Docstrings to test module
+
+### Task 5.2: Unit tests for `rrf.py`
+**File:** `tests/test_rrf.py` (**new**)
+
+**Subtasks:**
+- [x] Test single list, single candidate → `{"A": 1.0}`
+- [x] Test single list multiple candidates: top candidate maps to 1.0, others < 1.0 and > 0.0
+- [x] Test all-zero list → returns `{}` (empty)
+- [x] Test mixed: one non-zero list + one all-zero list → only non-zero list contributes
+- [x] Test `k` parameter: higher `k` flattens differences (rank-1 and rank-2 scores closer together)
+- [x] Test score ordering is preserved (rank 1 > rank 2 > rank 3 in output)
+- [x] Add Google Docstrings to test module
+
+### Task 5.3: Unit tests for `auto_tagger.py`
+**File:** `tests/test_auto_tagger.py` (**new**)
+
+**Subtasks:**
+- [x] Mock `AzureOpenAIClient.chat_completion()` for all tests (no real LLM calls)
+- [x] Test: LLM returns `{"additions": ["Machine Learning"]}` for existing `["ML"]`,
+  required `["Machine Learning"]` → result is `["ML", "Machine Learning"]`
+- [x] Test: LLM returns `{"additions": []}` → result is identical to `existing`
+- [x] Test: LLM `chat_completion` raises exception → result is identical to `existing`,
+  no exception propagated
+- [x] Test: LLM returns malformed JSON (not parseable) → result is `existing` unchanged
+- [x] Test: LLM returns addition not in `required` (hallucination) → addition is dropped
+- [x] Test: LLM returns addition already in `existing` → no duplicate in result
+- [x] Test: empty `required` list → `expand_competences()` returns `existing` unchanged
+  (no LLM call needed)
+- [x] Add Google Docstrings to test module
+
+### Task 5.4: Extend `tests/test_similarity.py`
+**File:** `tests/test_similarity.py` (extended)
+
+**Subtasks:**
+- [x] Add tests with `use_bm25_search=True`:
+  - Required `["Python"]`, candidate `["JavaScript"]` → `competence_score == 0.0`
+  - Required `["Python"]`, candidate `["Python"]` → `competence_score == 1.0`
+  - Required `["Python", "Machine Learning"]`, candidate `["JavaScript", "TypeScript"]` → both individual scores 0.0, overall `competence_score == 0.0`
+  - Required `["Python", "Machine Learning"]`, candidate `["Python", "Machine Learning"]` → both scores 1.0
+- [x] Add tests with `use_bm25_search=True, use_auto_tagging=True` (mocked `AutoTagger`):
+  - Mocked `AutoTagger.expand_competences` injects `"Machine Learning"` into `["ML"]`
+  - Required `["Machine Learning"]`, candidate `["ML"]` (after expansion) → `score > 0`
+- [x] Confirm that with `use_bm25_search=False`, behavior is unchanged (existing tests still pass)
+
+### Task 5.5: Extend `tests/test_matcher_integration.py`
+**File:** `tests/test_matcher_integration.py` (extended)
+
+**Subtasks:**
+- [x] Add integration test with `use_bm25_search=True`:
+  - Task requires: `["Python", "Machine Learning"]`
+  - Candidate A has: `["Python", "Machine Learning", "Pandas"]` → high score
+  - Candidate B has: `["JavaScript", "TypeScript", "Node.js", "Vue.js"]` → score 0.0
+  - Assert: candidate A `competence_score` > candidate B `competence_score`
+  - Assert: candidate B `competence_score == 0.0` (false-positive eliminated)
+- [x] Add integration test with `use_bm25_search=True, use_auto_tagging=True`
+  (mocked LLM via mocked `AutoTagger`):
+  - Candidate has `["ML", "Python"]`; auto-tag adds `"Machine Learning"` to working list
+  - Required `["Machine Learning", "Python"]` → both scores > 0
+- [x] These tests explicitly document the false-positive scenario from production
+
+---
+
+## Phase 6: Documentation
+
+### Task 6.1: Update `docs/architecture.md`
+**File:** `docs/architecture.md`
+
+**Subtasks:**
+- [x] Add section "BM25 + RRF + Auto-Tagging Competence Matching" under the matching/similarity section:
+  - Describe the three new modules (`auto_tagger.py`, `bm25.py`, `rrf.py`)
+  - Describe the auto-tag → BM25 → RRF pipeline and routing in `compute_competence_similarity()`
+  - Note that role similarity is unaffected
+  - Reference `use_bm25_search` and `use_auto_tagging` config keys
+
+### Task 6.2: Update `docs/semantic_similarity_tradeoffs.md`
+**File:** `docs/semantic_similarity_tradeoffs.md`
+
+**Subtasks:**
+- [x] Add BM25+RRF row to the strategy comparison table:
+  - Strengths: exact/lexical match, no false positives for non-overlapping skills
+  - Weaknesses: misses synonyms and cross-language variants (mitigated by `use_auto_tagging`)
+  - When to use: when skill names in the dataset are mostly canonical English terms
+- [x] Add BM25+RRF+AutoTag row:
+  - Strengths: exact match + synonym/abbreviation/cross-language bridging via LLM
+  - Weaknesses: LLM latency per candidate, requires `AZURE_OPENAI_LLM_API_KEY`
+  - When to use: mixed-language or abbreviation-heavy skill datasets
+
+### Task 6.3: Update `docs/assistant_system_prompt.md`
+**File:** `docs/assistant_system_prompt.md`
+
+**Subtasks:**
+- [x] Add note on score interpretation when `use_bm25_search=true`:
+  - A `competence_score` of 0.0 means **no lexical token overlap** between required and candidate competences
+  - The score is not a semantic similarity — it is a lexical matching score
+  - Users should be informed when BM25 mode is active so they understand why cross-language synonyms may not match
+- [x] Add note on `use_auto_tagging=true`:
+  - Auto-tagging bridges common synonym/abbreviation gaps (e.g. "ML" → "Machine Learning")
+  - A positive `competence_score` with auto-tagging enabled may reflect an LLM-inferred equivalence,
+    not a literal token match; the rationale field will say "BM25+RRF" regardless
+
+### Task 6.4: Update `README.md`
+**File:** `README.md`
+
+**Subtasks:**
+- [x] Add all three new config keys to the config reference section:
+  - `matching.similarity.use_bm25_search` (boolean, default `false`): Enables BM25+RRF lexical competence matching
+  - `matching.similarity.use_auto_tagging` (boolean, default `false`): Enables LLM pre-expansion before BM25
+  - `azure_openai.chat_deployment` (string, default `""`): Chat model deployment name for auto-tagging
+- [x] Document `AZURE_OPENAI_LLM_API_KEY` environment variable in the env-vars section
+
+---
+
+## Phase 7: Dependency
+
+### Task 7.1: Add `rank-bm25` to project dependencies
+**File:** `pyproject.toml`
+
+**Subtasks:**
+- [x] Add `rank-bm25` to `[project.dependencies]` in `pyproject.toml`
+- [x] Run `uv lock` (or equivalent) to update the lockfile
+- [x] Verify `from rank_bm25 import BM25Okapi` works in the environment
+
+**Acceptance:**
+- `uv run python -c "from rank_bm25 import BM25Okapi; print('ok')"` exits 0
+
+---
+
+## Phase 8: Quality Gates
+
+### Task 8.1: Run unit tests
+- [x] `uv run pytest tests/test_bm25.py tests/test_rrf.py tests/test_auto_tagger.py -v`
+- [x] `uv run pytest -m "not integration" -q`
+- [x] All tests pass
+
+### Task 8.2: Run linters
+- [x] `uv run ruff check src/ tests/`
+- [x] `uv run mypy src/teamlandkarte_mcp/matching/bm25.py src/teamlandkarte_mcp/matching/rrf.py src/teamlandkarte_mcp/matching/auto_tagger.py`
+- [x] Zero errors
+
+### Task 8.3: OpenSpec validation
+- [x] `openspec validate add-bm25-rrf-competence-matching --strict`
+- [x] All checks pass
+
+### Task 8.4: Confirm legacy files are removed
+- [x] `ls debug_*.py` → no such files
+- [x] `ls SEMANTIC_SIMILARITY_ANALYSIS.md SCORE_BUG_FIX.md` → no such files
+
+---
+
+## Phase 9: Architectural Fix — Global BM25 Index
+
+**Problem identified post-spec**: When `Bm25Index` was built per candidate (N = number of that
+person's competences), IDF values were always 0 or negative for every term — the library
+`rank_bm25` computes `log((N - df + 0.5) / (df + 0.5))`, and with N=1 every term has df=1,
+yielding `log(0)` = negative. All BM25 scores were therefore ≤ 0, making the `max(0.0, raw)`
+clamp produce a flat 0.0 for everything. The fix builds one global corpus from the union of
+all filtered candidates' competences, ensuring N is large enough for IDF to be meaningful.
+
+### Task 9.1: Remove `1e-10` clamp in `bm25.py`
+**File:** `src/teamlandkarte_mcp/matching/bm25.py`
+
+**Subtasks:**
+- [x] Remove the special-case block that returned `1e-10` for zero/negative BM25 scores
+  when token overlap existed (a band-aid for the IDF pathology)
+- [x] Replace with `max(0.0, float(raw))` — ubiquitous terms (negative IDF) simply score 0.0
+- [x] Update `Bm25Index` and `bm25_rank_competences()` docstrings to reflect the change
+
+**Acceptance:**
+- A term present in every document of a sufficiently large corpus scores 0.0, not 1e-10
+
+---
+
+### Task 9.2: Add `use_bm25_search` property to `SimilarityEngine`
+**File:** `src/teamlandkarte_mcp/matching/similarity.py`
+
+**Subtasks:**
+- [x] Expose `use_bm25_search: bool` as a read-only property (or attribute) on `SimilarityEngine`
+  so `Matcher` can check it without accessing internal config directly
+
+**Acceptance:**
+- `engine.use_bm25_search` returns the value of `engine.config.use_bm25_search`
+
+---
+
+### Task 9.3: Thread `global_index` parameter through similarity layer
+**File:** `src/teamlandkarte_mcp/matching/similarity.py`
+
+**Subtasks:**
+- [x] Add `global_index: Bm25Index | None = None` parameter to `compute_competence_similarity()`
+- [x] Pass it through to `_bm25_rrf_similarity()`
+- [x] In `_bm25_rrf_similarity()`: when `global_index is not None`, call
+  `global_index.rank(req)` and filter the result list to only entries whose text is in
+  `set(candidate)`; otherwise fall back to `bm25_rank_competences(req, candidate)`
+
+**Acceptance:**
+- `global_index.rank(req)` is called once per required competence, not once per candidate
+- Results are correctly filtered to the current candidate's competence set
+- Fall-back path (`global_index=None`) still works unchanged
+
+---
+
+### Task 9.4: Build global corpus in `Matcher` before per-candidate loop
+**File:** `src/teamlandkarte_mcp/matching/matcher.py`
+
+**Subtasks:**
+- [x] Before the `for cap in filtered` loop: if `self._sim.use_bm25_search` and `filtered`
+  is non-empty, collect the deduplicated union of all candidates' competences into a list
+  `global_corpus`, instantiate `Bm25Index(corpus=global_corpus)`
+- [x] Pass `global_index` to every `compute_competence_similarity()` call inside the loop
+- [x] When `use_bm25_search=False` (or `filtered` is empty): `global_bm25_index = None`
+  (no-op, existing behaviour)
+
+**Acceptance:**
+- A corpus of N candidates' competences is built once, not N times
+- Candidates with no overlap with required competences still score 0.0
+- Candidates with overlap score > 0.0 (IDF is meaningful because N >> 1 in the global corpus)
+
+---
+
+### Task 9.5: Update tests for global index
+**Files:** `tests/test_bm25.py`, `tests/test_similarity.py`, `tests/test_matcher_integration.py`, `tests/test_matching_refinements.py`
+
+**Subtasks:**
+- [x] `test_bm25.py`: update small-corpus tests to use ≥ 5-doc corpora so IDF is positive;
+  add `test_bm25_index_rank_filters_to_candidate_subset` and
+  `test_bm25_ubiquitous_term_clamped_to_zero`
+- [x] `test_similarity.py`: update 4 BM25 tests to pass a 5-doc `global_index`; add
+  `test_use_bm25_search_property_reflects_flag` and `test_bm25_global_index_filters_to_candidate_subset`
+- [x] `test_matcher_integration.py`: add `use_bm25_search = False` to `_FakeSimilarityEngine`,
+  add `global_index=None` param to `compute_competence_similarity`; update
+  `test_bm25_auto_tag_candidate_matches_after_expansion` to use a 5-doc global corpus;
+  add `test_matcher_builds_global_bm25_index_across_all_candidates`
+- [x] `test_matching_refinements.py`: add `use_bm25_search = False` and `global_index=None`
+  to local `_FakeSim` stub
+
+**Acceptance:**
+- All 140 tests pass, 2 skipped
+
+---
+
+### Task 9.6: Update `docs/architecture.md`
+**File:** `docs/architecture.md`
+
+**Subtasks:**
+- [x] Replace the incorrect "Per-call index" bullet in §9.3 with a "Global BM25 index" bullet
+  describing the pre-loop construction and per-candidate filtering
+- [x] Update the §9.1 module table row for `bm25.py` to remove the `1e-10` clamp description
+  and replace with `max(0.0, raw)` description
+- [x] Update the §9.2 data flow diagram to show `match_candidates` building the global index
+  before the per-candidate loop
+
+**Acceptance:**
+- No mention of "per-call index" or `1e-10` clamp remains in the docs
+