Orchestrator/bahn/teamlandkarte-mcp/openspec/changes/add-bm25-rrf-competence-matching/proposal.md

Change Proposal: BM25 + RRF Competence Matching

• Change ID: add-bm25-rrf-competence-matching
• Status: Proposed
• Target: teamlandkarte-mcp
• Author: Thomas Handke
• Date: 2026-02-26

Summary

Add an optional BM25-based competence matching strategy as an alternative to the existing embedding-based similarity approach. When enabled, the following pipeline replaces embedding-based competence similarity:

1. Auto-Tagging (LLM): Before BM25 scoring, an LLM compares the required competences against each candidate's existing competences and suggests canonical additions (synonyms, abbreviations, cross-language equivalents). The expanded competence list is ephemeral — it is used only for the current scoring call and is never persisted.
2. BM25 Ranking: The (expanded) candidate competences are indexed with BM25. Each required competence is used as a query to rank the candidates.
3. RRF Fusion: Reciprocal Rank Fusion normalizes the BM25 ranked list into a score in [0, 1] per required competence.

The feature is controlled by two new booleans in config.toml:

• use_bm25_search (default false): enables BM25 + RRF.
• use_auto_tagging (default false): enables LLM-based competence expansion before BM25 scoring. Strongly recommended when use_bm25_search = true.

Why

The current embedding-based (per_skill / aggregate) strategy has a known weakness: it assigns non-trivial scores to candidates whose competences are only semantically adjacent to the required skills but not actually matching.

Example (observed in production): Searching for Python + Machine Learning, a candidate with only JavaScript, TypeScript, Node.js, Vue.js, Web Architekturen received a score of 0.392 because the embedding model considers all programming languages semantically related.

BM25 (Best Match 25) is a classical lexical ranking algorithm that scores documents by term overlap between query and document. It is immune to the semantic drift problem because it only counts tokens that literally occur in the candidate text. BM25 will assign 0 to "JavaScript" when the query is "Python".

When BM25 outperforms embeddings

Scenario	Embeddings	BM25
Query "Python", candidate has "Python"	High ✓	High ✓
Query "Python", candidate has "JavaScript"	Medium ✗	Zero ✓
Query "Machine Learning", candidate has "ML"	High ✓	Low ✗
Query "Softwarearchitektur", candidate has "software architecture"	High ✓	Low ✗
Query "React", candidate has "React.js"	High ✓	Medium ~

BM25 excels at exact or near-exact skill names. Embeddings excel at synonyms and cross-language variants (e.g., German/English). The optional use_auto_tagging feature bridges BM25's synonym and cross-language gap by having an LLM pre-expand the candidate competence list — making the combination of BM25 + Auto-Tagging competitive with embeddings while still eliminating false positives from semantic drift.

Why RRF for fusion

Reciprocal Rank Fusion merges multiple ranked lists into one without requiring score normalization. For each required competence query, we obtain a ranked list of candidate competences. RRF then assigns each candidate a fused score based on its (1-based) rank in one or more lists.

RRF fused score formula (Cormack et al. 2009):

rrf_score(candidate) = Σ_i  1 / (k + rank_i(candidate))

RRF is robust to score scale differences and has been widely validated in information retrieval research as a high-quality fusion method.

What Changes

1. New config parameter matching.similarity.use_bm25_search (boolean, default false)

• When false (default): existing embedding-based strategy is used unchanged.
• When true: BM25 + RRF replaces embedding-based competence similarity. Role similarity is always embedding-based (unchanged).

2. New config parameter matching.similarity.use_auto_tagging (boolean, default false)

• Only applies when use_bm25_search = true. When false, BM25 runs on the raw candidate competence list (no LLM expansion).
• When true: before each BM25 scoring call, an LLM expands the candidate's competence list with canonical forms of any required competences that are already covered by the candidate's existing entries (via synonym, abbreviation, or cross-language equivalence).
• The expanded list is ephemeral — it is not written back to the database and does not affect any other operation.

3. New config parameter azure_openai.chat_deployment (string)

• Azure deployment name of the chat model used for auto-tagging (e.g. "gpt-4.1").
• Required only when use_auto_tagging = true.
• API key is read from environment variable AZURE_OPENAI_LLM_API_KEY.

4. AzureOpenAIClient extended: LLM chat capability restored

• Add chat_completion(system: str, user: str) -> str method supporting structured JSON responses via Azure OpenAI chat completions API.
• Add chat_deployment and llm_api_key constructor parameters.
• This restores LLM capability removed in a previous change, now scoped exclusively to auto-tagging.

5. New module src/teamlandkarte_mcp/matching/auto_tagger.py

Implements:

• AutoTagger: class that wraps the LLM client and executes the tagging prompt.
• expand_competences(): given required: list[str] and existing: list[str], returns list[str] — the combined list existing + additions, where additions are the canonical required-competence names that the LLM identifies as already covered by existing entries.
• Structured JSON output schema enforced via the Azure OpenAI response_format parameter.
• Google Docstrings on all public symbols.

6. New module src/teamlandkarte_mcp/matching/bm25.py

Implements:

• Bm25Index: builds an in-memory BM25 index over a corpus of competence strings. Designed to be instantiated once over the global candidate pool (see item 8a below) so IDF values are meaningful. Raw scores are clamped to max(0.0, raw) — terms with negative IDF produce 0.0 rather than a spurious tiny positive value.
• bm25_rank_competences(): for a single required competence query, returns a ranked list of (candidate_text, score) tuples. Convenience wrapper for standalone/test use.
• Pure Python, no external runtime dependencies beyond rank_bm25 from PyPI.

7. New module src/teamlandkarte_mcp/matching/rrf.py

Implements:

• reciprocal_rank_fusion(): merges one or more ranked lists into a final score dict.
• k constant (default 60, standard RRF value from Cormack et al. 2009).
• Returns dict[str, float] mapping candidate text → fused score in (0, 1] (top-ranked candidate receives exactly 1.0).

8. SimilarityEngine.compute_competence_similarity() extended

• Reads use_bm25_search and use_auto_tagging flags.
• Accepts an optional global_index: Bm25Index | None parameter (passed in by Matcher).
• When use_bm25_search=True:
  1. If use_auto_tagging=True: calls AutoTagger.expand_competences() to build the expanded candidate list.
  2. Delegates to _bm25_rrf_similarity() with the (optionally expanded) candidate list and the global index.
• _bm25_rrf_similarity() uses global_index.rank(req) filtered to the candidate's competence set when global_index is provided, otherwise falls back to bm25_rank_competences(req, candidate).
• Output shape is identical to existing strategies: dict[required_competence → {score, best_match, rationale}]
• Preserves the Matcher and scorer contracts unchanged.

8a. Matcher.match() builds a global Bm25Index before the per-candidate loop

When use_bm25_search=True and the filtered candidate pool is non-empty, Matcher.match() collects the deduplicated union of all filtered candidates' competences and builds one Bm25Index over that global corpus before the loop. The same index is passed to every compute_competence_similarity() call.

Why this matters: With a per-person corpus (N ≈ 5–30 entries), BM25 IDF is 0 or negative for every term — all raw scores collapse to ≤ 0, and max(0.0, raw) clamping makes every match appear as 0.0. A global corpus ensures N is large enough for IDF to carry real signal.

9. SimilarityConfig dataclass extended

New fields:

• use_bm25_search: bool = False
• use_auto_tagging: bool = False

10. AzureOpenAIConfig dataclass extended

New fields:

• chat_deployment: str = ""
• (API key via AZURE_OPENAI_LLM_API_KEY env var)

11. config.py parser extended

Reads matching.similarity.use_bm25_search, matching.similarity.use_auto_tagging, and azure_openai.chat_deployment from TOML. Reads AZURE_OPENAI_LLM_API_KEY from environment when use_auto_tagging = true.

12. config.toml.example updated

Documents all three new keys with inline comments.

13. Server initialization in mcp_server.py

• Passes use_bm25_search and use_auto_tagging flags when constructing SimilarityEngine.
• When use_auto_tagging=True: constructs AutoTagger with the LLM-capable AzureOpenAIClient and passes it to SimilarityEngine.

14. Documentation updated

• docs/architecture.md: new section on BM25/RRF/Auto-Tagging strategy.
• docs/semantic_similarity_tradeoffs.md: comparison table updated with BM25+AutoTag entry.
• docs/assistant_system_prompt.md: notes on score interpretation with BM25 and auto-tagging.
• README.md: all new config parameters documented.

15. Legacy/deprecated code removed

• Remove debug_similarity_analysis.py
• Remove debug_categorization.py
• Remove SEMANTIC_SIMILARITY_ANALYSIS.md
• Remove SCORE_BUG_FIX.md

16. Tests

• tests/test_bm25.py: unit tests for Bm25Index and bm25_rank_competences().
• tests/test_rrf.py: unit tests for reciprocal_rank_fusion().
• tests/test_auto_tagger.py: unit tests for AutoTagger.expand_competences() (mocked LLM).
• tests/test_similarity.py: extend with BM25/RRF and auto-tagging strategy tests.
• tests/test_matcher_integration.py: integration test for the full pipeline (auto-tag → BM25 → RRF) in both matching directions.

Impact

Affected specs

• matching-tools (competence scoring path)

Affected code

File	Change
src/teamlandkarte_mcp/matching/similarity.py	Add _bm25_rrf_similarity(), global_index param, auto-tag branch, read flags
src/teamlandkarte_mcp/matching/matcher.py	Build global Bm25Index before per-candidate loop; pass as global_index
src/teamlandkarte_mcp/matching/bm25.py	New
src/teamlandkarte_mcp/matching/rrf.py	New
src/teamlandkarte_mcp/matching/auto_tagger.py	New
src/teamlandkarte_mcp/azure/openai_client.py	Restore chat_completion(); add chat_deployment + llm_api_key params
src/teamlandkarte_mcp/config.py	SimilarityConfig.use_bm25_search, use_auto_tagging; AzureOpenAIConfig.chat_deployment
src/teamlandkarte_mcp/mcp_server.py	Pass flags + AutoTagger to SimilarityEngine
config.toml.example	Document all new keys
docs/architecture.md	BM25/RRF/Auto-Tagging section
docs/semantic_similarity_tradeoffs.md	Updated comparison with BM25+AutoTag entry
docs/assistant_system_prompt.md	Score notes for BM25 and auto-tagging modes
README.md	All new config parameters
tests/test_bm25.py	New
tests/test_rrf.py	New
tests/test_auto_tagger.py	New
tests/test_similarity.py	Extended
tests/test_matcher_integration.py	Extended
debug_similarity_analysis.py	Removed
debug_categorization.py	Removed
SEMANTIC_SIMILARITY_ANALYSIS.md	Removed
SCORE_BUG_FIX.md	Removed

Breaking changes

None. The use_bm25_search = false and use_auto_tagging = false defaults preserve all existing behavior. The overall_score computation (weighted mean of role_score and competence_score) is unchanged.

New environment variables

Variable	Required when	Description
AZURE_OPENAI_LLM_API_KEY	use_auto_tagging = true	Azure OpenAI API key for the chat model used in auto-tagging. The server raises a configuration error at startup if this key is absent and use_auto_tagging = true.

New dependencies

• rank-bm25 (PyPI): pure-Python BM25 implementation, no native code, no external services. Adds to pyproject.toml dependencies.

Alternatives Considered

A. Hard-code exact-match bonus in per_skill

Simpler but not principled: would require special-casing the scorer instead of a clean strategy abstraction.

B. Hybrid embedding + BM25 in a single _bm25_rrf_similarity() call

Combining both signals via RRF is architecturally clean and naturally extensible. However, to keep this change focused and reviewable, the initial implementation uses BM25-only RRF. A hybrid mode can be added as a follow-up change.

C. Replace embeddings entirely

Not desirable. Embeddings are essential for role inference and handle multilingual / synonymous skill names (e.g., "Softwarearchitektur" ↔ "Software Architecture").

Open Questions

None. The design is sufficiently clear to proceed.