ankn
a5f8fb49ab
Bahn: aisupport, Analyse-O2C-C2S, awesome-bahn-mcp-servers, beam-mcp,
Confluence_Bot, db-planet-mcp-server, O2C-Harness, project-audit,
Projekt-KIQ-HP, teamlandkarte-mcp
Dhive: Jury-Voting
Privat: CV, NoteGraph (NOTE: NoteGraph needs complete redo after consolidation)
Shared: AI-Orchestrator, OrgMyLife, power_skills_and_more
Shared/references: symphony (read-only)
Bahn repos remain available as independent remotes - this monorepo
pulls them in via subtree, the originals are untouched.
289 lines
13 KiB
Markdown
289 lines
13 KiB
Markdown
# Change Proposal: BM25 + RRF Competence Matching
|
||
|
||
- **Change ID**: `add-bm25-rrf-competence-matching`
|
||
- **Status**: Proposed
|
||
- **Target**: `teamlandkarte-mcp`
|
||
- **Author**: Thomas Handke
|
||
- **Date**: 2026-02-26
|
||
|
||
## Summary
|
||
|
||
Add an optional BM25-based competence matching strategy as an alternative to the existing
|
||
embedding-based similarity approach. When enabled, the following pipeline replaces
|
||
embedding-based competence similarity:
|
||
|
||
1. **Auto-Tagging (LLM):** Before BM25 scoring, an LLM compares the required competences
|
||
against each candidate's existing competences and suggests canonical additions
|
||
(synonyms, abbreviations, cross-language equivalents). The expanded competence list
|
||
is ephemeral — it is used only for the current scoring call and is never persisted.
|
||
2. **BM25 Ranking:** The (expanded) candidate competences are indexed with BM25. Each
|
||
required competence is used as a query to rank the candidates.
|
||
3. **RRF Fusion:** Reciprocal Rank Fusion normalizes the BM25 ranked list into a score
|
||
in [0, 1] per required competence.
|
||
|
||
The feature is controlled by two new booleans in `config.toml`:
|
||
- `use_bm25_search` (default `false`): enables BM25 + RRF.
|
||
- `use_auto_tagging` (default `false`): enables LLM-based competence expansion before
|
||
BM25 scoring. Strongly recommended when `use_bm25_search = true`.
|
||
|
||
## Why
|
||
|
||
The current embedding-based (`per_skill` / `aggregate`) strategy has a known weakness:
|
||
it assigns non-trivial scores to candidates whose competences are only _semantically
|
||
adjacent_ to the required skills but not actually matching.
|
||
|
||
**Example (observed in production):**
|
||
Searching for **Python** + **Machine Learning**, a candidate with only
|
||
**JavaScript, TypeScript, Node.js, Vue.js, Web Architekturen** received a score of **0.392**
|
||
because the embedding model considers all programming languages semantically related.
|
||
|
||
BM25 (Best Match 25) is a classical lexical ranking algorithm that scores documents by
|
||
**term overlap** between query and document. It is immune to the semantic drift problem
|
||
because it only counts tokens that literally occur in the candidate text. BM25 will assign
|
||
**0** to "JavaScript" when the query is "Python".
|
||
|
||
### When BM25 outperforms embeddings
|
||
|
||
| Scenario | Embeddings | BM25 |
|
||
|----------|-----------|------|
|
||
| Query "Python", candidate has "Python" | High ✓ | High ✓ |
|
||
| Query "Python", candidate has "JavaScript" | Medium ✗ | Zero ✓ |
|
||
| Query "Machine Learning", candidate has "ML" | High ✓ | Low ✗ |
|
||
| Query "Softwarearchitektur", candidate has "software architecture" | High ✓ | Low ✗ |
|
||
| Query "React", candidate has "React.js" | High ✓ | Medium ~ |
|
||
|
||
BM25 excels at exact or near-exact skill names. Embeddings excel at synonyms and
|
||
cross-language variants (e.g., German/English). The optional `use_auto_tagging` feature
|
||
bridges BM25's synonym and cross-language gap by having an LLM pre-expand the candidate
|
||
competence list — making the combination of BM25 + Auto-Tagging competitive with
|
||
embeddings while still eliminating false positives from semantic drift.
|
||
|
||
### Why RRF for fusion
|
||
|
||
Reciprocal Rank Fusion merges multiple ranked lists into one without requiring score
|
||
normalization. For each required competence query, we obtain a ranked list of candidate
|
||
competences. RRF then assigns each candidate a fused score based on its (1-based) rank in
|
||
one or more lists.
|
||
|
||
RRF fused score formula (Cormack et al. 2009):
|
||
|
||
```
|
||
rrf_score(candidate) = Σ_i 1 / (k + rank_i(candidate))
|
||
```
|
||
|
||
RRF is robust to score scale differences and has been widely validated in information
|
||
retrieval research as a high-quality fusion method.
|
||
|
||
## What Changes
|
||
|
||
### 1. New config parameter `matching.similarity.use_bm25_search` (boolean, default `false`)
|
||
|
||
- When `false` (default): existing embedding-based strategy is used unchanged.
|
||
- When `true`: BM25 + RRF replaces embedding-based competence similarity.
|
||
Role similarity is **always** embedding-based (unchanged).
|
||
|
||
### 2. New config parameter `matching.similarity.use_auto_tagging` (boolean, default `false`)
|
||
|
||
- Only applies when `use_bm25_search = true`. When `false`, BM25 runs on the raw
|
||
candidate competence list (no LLM expansion).
|
||
- When `true`: before each BM25 scoring call, an LLM expands the candidate's competence
|
||
list with canonical forms of any required competences that are already covered by the
|
||
candidate's existing entries (via synonym, abbreviation, or cross-language equivalence).
|
||
- The expanded list is ephemeral — it is not written back to the database and does not
|
||
affect any other operation.
|
||
|
||
### 3. New config parameter `azure_openai.chat_deployment` (string)
|
||
|
||
- Azure deployment name of the chat model used for auto-tagging (e.g. `"gpt-4.1"`).
|
||
- Required only when `use_auto_tagging = true`.
|
||
- API key is read from environment variable `AZURE_OPENAI_LLM_API_KEY`.
|
||
|
||
### 4. `AzureOpenAIClient` extended: LLM chat capability restored
|
||
|
||
- Add `chat_completion(system: str, user: str) -> str` method supporting structured JSON
|
||
responses via Azure OpenAI chat completions API.
|
||
- Add `chat_deployment` and `llm_api_key` constructor parameters.
|
||
- This restores LLM capability removed in a previous change, now scoped exclusively
|
||
to auto-tagging.
|
||
|
||
### 5. New module `src/teamlandkarte_mcp/matching/auto_tagger.py`
|
||
|
||
Implements:
|
||
- `AutoTagger`: class that wraps the LLM client and executes the tagging prompt.
|
||
- `expand_competences()`: given `required: list[str]` and `existing: list[str]`,
|
||
returns `list[str]` — the combined list `existing + additions`, where `additions`
|
||
are the canonical required-competence names that the LLM identifies as already
|
||
covered by `existing` entries.
|
||
- Structured JSON output schema enforced via the Azure OpenAI `response_format` parameter.
|
||
- Google Docstrings on all public symbols.
|
||
|
||
### 6. New module `src/teamlandkarte_mcp/matching/bm25.py`
|
||
|
||
Implements:
|
||
- `Bm25Index`: builds an in-memory BM25 index over a corpus of competence strings.
|
||
Designed to be instantiated once over the global candidate pool (see item 8a below)
|
||
so IDF values are meaningful. Raw scores are clamped to `max(0.0, raw)` — terms with
|
||
negative IDF produce 0.0 rather than a spurious tiny positive value.
|
||
- `bm25_rank_competences()`: for a single required competence query, returns a ranked list
|
||
of `(candidate_text, score)` tuples. Convenience wrapper for standalone/test use.
|
||
- Pure Python, no external runtime dependencies beyond `rank_bm25` from PyPI.
|
||
|
||
### 7. New module `src/teamlandkarte_mcp/matching/rrf.py`
|
||
|
||
Implements:
|
||
- `reciprocal_rank_fusion()`: merges one or more ranked lists into a final score dict.
|
||
- `k` constant (default `60`, standard RRF value from Cormack et al. 2009).
|
||
- Returns `dict[str, float]` mapping candidate text → fused score in `(0, 1]`
|
||
(top-ranked candidate receives exactly `1.0`).
|
||
|
||
### 8. `SimilarityEngine.compute_competence_similarity()` extended
|
||
|
||
- Reads `use_bm25_search` and `use_auto_tagging` flags.
|
||
- Accepts an optional `global_index: Bm25Index | None` parameter (passed in by `Matcher`).
|
||
- When `use_bm25_search=True`:
|
||
1. If `use_auto_tagging=True`: calls `AutoTagger.expand_competences()` to build the
|
||
expanded candidate list.
|
||
2. Delegates to `_bm25_rrf_similarity()` with the (optionally expanded) candidate list
|
||
and the global index.
|
||
- `_bm25_rrf_similarity()` uses `global_index.rank(req)` filtered to the candidate's
|
||
competence set when `global_index` is provided, otherwise falls back to
|
||
`bm25_rank_competences(req, candidate)`.
|
||
- Output shape is **identical** to existing strategies:
|
||
`dict[required_competence → {score, best_match, rationale}]`
|
||
- Preserves the `Matcher` and scorer contracts unchanged.
|
||
|
||
### 8a. `Matcher.match()` builds a global `Bm25Index` before the per-candidate loop
|
||
|
||
When `use_bm25_search=True` and the filtered candidate pool is non-empty, `Matcher.match()`
|
||
collects the deduplicated union of all filtered candidates' competences and builds one
|
||
`Bm25Index` over that global corpus before the loop. The same index is passed to every
|
||
`compute_competence_similarity()` call.
|
||
|
||
**Why this matters:** With a per-person corpus (N ≈ 5–30 entries), BM25 IDF is 0 or
|
||
negative for every term — all raw scores collapse to ≤ 0, and `max(0.0, raw)` clamping
|
||
makes every match appear as 0.0. A global corpus ensures N is large enough for IDF to
|
||
carry real signal.
|
||
|
||
### 9. `SimilarityConfig` dataclass extended
|
||
|
||
New fields:
|
||
- `use_bm25_search: bool = False`
|
||
- `use_auto_tagging: bool = False`
|
||
|
||
### 10. `AzureOpenAIConfig` dataclass extended
|
||
|
||
New fields:
|
||
- `chat_deployment: str = ""`
|
||
- (API key via `AZURE_OPENAI_LLM_API_KEY` env var)
|
||
|
||
### 11. `config.py` parser extended
|
||
|
||
Reads `matching.similarity.use_bm25_search`, `matching.similarity.use_auto_tagging`,
|
||
and `azure_openai.chat_deployment` from TOML. Reads `AZURE_OPENAI_LLM_API_KEY` from
|
||
environment when `use_auto_tagging = true`.
|
||
|
||
### 12. `config.toml.example` updated
|
||
|
||
Documents all three new keys with inline comments.
|
||
|
||
### 13. Server initialization in `mcp_server.py`
|
||
|
||
- Passes `use_bm25_search` and `use_auto_tagging` flags when constructing `SimilarityEngine`.
|
||
- When `use_auto_tagging=True`: constructs `AutoTagger` with the LLM-capable
|
||
`AzureOpenAIClient` and passes it to `SimilarityEngine`.
|
||
|
||
### 14. Documentation updated
|
||
|
||
- `docs/architecture.md`: new section on BM25/RRF/Auto-Tagging strategy.
|
||
- `docs/semantic_similarity_tradeoffs.md`: comparison table updated with BM25+AutoTag entry.
|
||
- `docs/assistant_system_prompt.md`: notes on score interpretation with BM25 and auto-tagging.
|
||
- `README.md`: all new config parameters documented.
|
||
|
||
### 15. Legacy/deprecated code removed
|
||
|
||
- Remove `debug_similarity_analysis.py`
|
||
- Remove `debug_categorization.py`
|
||
- Remove `SEMANTIC_SIMILARITY_ANALYSIS.md`
|
||
- Remove `SCORE_BUG_FIX.md`
|
||
|
||
### 16. Tests
|
||
|
||
- `tests/test_bm25.py`: unit tests for `Bm25Index` and `bm25_rank_competences()`.
|
||
- `tests/test_rrf.py`: unit tests for `reciprocal_rank_fusion()`.
|
||
- `tests/test_auto_tagger.py`: unit tests for `AutoTagger.expand_competences()` (mocked LLM).
|
||
- `tests/test_similarity.py`: extend with BM25/RRF and auto-tagging strategy tests.
|
||
- `tests/test_matcher_integration.py`: integration test for the full pipeline
|
||
(auto-tag → BM25 → RRF) in both matching directions.
|
||
|
||
## Impact
|
||
|
||
### Affected specs
|
||
|
||
- `matching-tools` (competence scoring path)
|
||
|
||
### Affected code
|
||
|
||
| File | Change |
|
||
|------|--------|
|
||
| `src/teamlandkarte_mcp/matching/similarity.py` | Add `_bm25_rrf_similarity()`, `global_index` param, auto-tag branch, read flags |
|
||
| `src/teamlandkarte_mcp/matching/matcher.py` | Build global `Bm25Index` before per-candidate loop; pass as `global_index` |
|
||
| `src/teamlandkarte_mcp/matching/bm25.py` | **New** |
|
||
| `src/teamlandkarte_mcp/matching/rrf.py` | **New** |
|
||
| `src/teamlandkarte_mcp/matching/auto_tagger.py` | **New** |
|
||
| `src/teamlandkarte_mcp/azure/openai_client.py` | Restore `chat_completion()`; add `chat_deployment` + `llm_api_key` params |
|
||
| `src/teamlandkarte_mcp/config.py` | `SimilarityConfig.use_bm25_search`, `use_auto_tagging`; `AzureOpenAIConfig.chat_deployment` |
|
||
| `src/teamlandkarte_mcp/mcp_server.py` | Pass flags + `AutoTagger` to `SimilarityEngine` |
|
||
| `config.toml.example` | Document all new keys |
|
||
| `docs/architecture.md` | BM25/RRF/Auto-Tagging section |
|
||
| `docs/semantic_similarity_tradeoffs.md` | Updated comparison with BM25+AutoTag entry |
|
||
| `docs/assistant_system_prompt.md` | Score notes for BM25 and auto-tagging modes |
|
||
| `README.md` | All new config parameters |
|
||
| `tests/test_bm25.py` | **New** |
|
||
| `tests/test_rrf.py` | **New** |
|
||
| `tests/test_auto_tagger.py` | **New** |
|
||
| `tests/test_similarity.py` | Extended |
|
||
| `tests/test_matcher_integration.py` | Extended |
|
||
| `debug_similarity_analysis.py` | **Removed** |
|
||
| `debug_categorization.py` | **Removed** |
|
||
| `SEMANTIC_SIMILARITY_ANALYSIS.md` | **Removed** |
|
||
| `SCORE_BUG_FIX.md` | **Removed** |
|
||
|
||
### Breaking changes
|
||
|
||
None. The `use_bm25_search = false` and `use_auto_tagging = false` defaults preserve all
|
||
existing behavior. The `overall_score` computation (weighted mean of `role_score` and
|
||
`competence_score`) is unchanged.
|
||
|
||
### New environment variables
|
||
|
||
| Variable | Required when | Description |
|
||
|----------|--------------|-------------|
|
||
| `AZURE_OPENAI_LLM_API_KEY` | `use_auto_tagging = true` | Azure OpenAI API key for the chat model used in auto-tagging. The server raises a configuration error at startup if this key is absent and `use_auto_tagging = true`. |
|
||
|
||
### New dependencies
|
||
|
||
- `rank-bm25` (PyPI): pure-Python BM25 implementation, no native code, no external services.
|
||
Adds to `pyproject.toml` `dependencies`.
|
||
|
||
## Alternatives Considered
|
||
|
||
### A. Hard-code exact-match bonus in per_skill
|
||
|
||
Simpler but not principled: would require special-casing the scorer instead of a clean
|
||
strategy abstraction.
|
||
|
||
### B. Hybrid embedding + BM25 in a single `_bm25_rrf_similarity()` call
|
||
|
||
Combining both signals via RRF is architecturally clean and naturally extensible. However,
|
||
to keep this change focused and reviewable, the initial implementation uses BM25-only RRF.
|
||
A hybrid mode can be added as a follow-up change.
|
||
|
||
### C. Replace embeddings entirely
|
||
|
||
Not desirable. Embeddings are essential for role inference and handle multilingual /
|
||
synonymous skill names (e.g., "Softwarearchitektur" ↔ "Software Architecture").
|
||
|
||
## Open Questions
|
||
|
||
None. The design is sufficiently clear to proceed.
|