Migrate all repos into monorepo context folders
Bahn: aisupport, Analyse-O2C-C2S, awesome-bahn-mcp-servers, beam-mcp,
Confluence_Bot, db-planet-mcp-server, O2C-Harness, project-audit,
Projekt-KIQ-HP, teamlandkarte-mcp
Dhive: Jury-Voting
Privat: CV, NoteGraph (NOTE: NoteGraph needs complete redo after consolidation)
Shared: AI-Orchestrator, OrgMyLife, power_skills_and_more
Shared/references: symphony (read-only)
Bahn repos remain available as independent remotes - this monorepo
pulls them in via subtree, the originals are untouched.
This commit is contained in:
@@ -0,0 +1,666 @@
|
||||
# Design: BM25 + RRF Competence Matching
|
||||
|
||||
## Context
|
||||
|
||||
This design specifies the internal architecture for the BM25 + Reciprocal Rank Fusion
|
||||
(RRF) competence matching strategy with optional LLM-based Auto-Tagging, controlled by
|
||||
`matching.similarity.use_bm25_search` and `matching.similarity.use_auto_tagging`.
|
||||
|
||||
The goal is to provide a lexical alternative to embedding-based similarity that assigns
|
||||
near-zero scores when candidate competences do not literally overlap with the required
|
||||
competences, while still producing a normalized score compatible with the existing
|
||||
`Matcher` and `scorer` contracts.
|
||||
|
||||
Auto-Tagging optionally extends BM25 by having an LLM pre-expand each candidate's
|
||||
competence list with canonical forms of any required competences it already covers
|
||||
(synonyms, abbreviations, cross-language equivalents). This bridges the lexical gap for
|
||||
known skill aliases without persisting any data.
|
||||
|
||||
---
|
||||
|
||||
## Key Decisions
|
||||
|
||||
### 1. BM25 is per-skill only, not aggregate
|
||||
|
||||
**Decision:** `_bm25_rrf_similarity()` operates per required competence (like the existing
|
||||
`per_skill` strategy), not as an aggregate over all required competences.
|
||||
|
||||
**Rationale:**
|
||||
- Aggregate-style BM25 would require concatenating all required competences into a single
|
||||
query document, which loses the per-skill granularity needed for `matched_competences`
|
||||
and `missing_competences` diagnostic output.
|
||||
- Per-skill BM25 + RRF preserves the same output shape as `_per_skill_similarity()`,
|
||||
making it a drop-in replacement.
|
||||
|
||||
---
|
||||
|
||||
### 2. BM25 index is built once globally over all filtered candidates
|
||||
|
||||
**Decision:** One `Bm25Index` is built over the **union of all filtered candidates'
|
||||
competences** before the per-candidate scoring loop in `Matcher.match()`, not fresh per
|
||||
`compute_competence_similarity()` call. The index is passed in as `global_index` and each
|
||||
per-candidate call filters BM25 results down to that candidate's own competence set.
|
||||
|
||||
**Rationale:**
|
||||
- **IDF pathology fix:** With a per-person corpus (N = that person's competence count),
|
||||
BM25 IDF is `log((N − df + 0.5) / (df + 0.5))`. For N = 5–30 and any term that appears
|
||||
in most or all documents, this is 0 or negative. `rank_bm25` therefore returns zero or
|
||||
negative raw scores for typical real-world skill lists — making every BM25 match appear
|
||||
as zero before `max(0.0, raw)` clamping. A global corpus of all candidates' competences
|
||||
ensures N is large enough for IDF to carry meaningful signal.
|
||||
- **Still cheap:** Building one index over the deduplicated union of all candidates'
|
||||
competences is O(M) where M is the total unique competence count — still milliseconds.
|
||||
- **Correctness over caching:** The global index is not cached across MCP tool calls; it is
|
||||
rebuilt fresh for each `match_candidates` invocation, so stale state is impossible.
|
||||
|
||||
---
|
||||
|
||||
### 3. RRF with single ranked list (BM25 only)
|
||||
|
||||
**Decision:** For the initial implementation, `reciprocal_rank_fusion()` fuses a single
|
||||
ranked list (BM25 ranks only), effectively normalizing BM25 scores via the RRF formula.
|
||||
|
||||
**Formula (single list):**
|
||||
|
||||
```
|
||||
rrf_score(candidate) = 1 / (k + rank(candidate))
|
||||
```
|
||||
|
||||
where `k = 60` (standard constant, Cormack et al. 2009), and `rank` is 1-based.
|
||||
|
||||
**Output normalization:**
|
||||
The raw RRF scores sum to a value > 1 across all candidates. To map to [0.0, 1.0]:
|
||||
|
||||
```
|
||||
normalized = rrf_score(best_candidate) / rrf_score_at_rank_1
|
||||
= 1 / (k + rank) * (k + 1)
|
||||
= (k + 1) / (k + rank)
|
||||
```
|
||||
|
||||
For `k=60`, rank 1 → 1.0, rank 2 → 61/62 ≈ 0.984, rank 10 → 61/70 ≈ 0.871.
|
||||
|
||||
This means BM25 scores do NOT inherently drop to near-zero for adjacent candidates.
|
||||
To address this, RRF scores are **zeroed** if the underlying BM25 score is 0.0, because
|
||||
a BM25 score of 0 means **no token overlap at all**. This is the critical property that
|
||||
fixes the false-positive problem.
|
||||
|
||||
**Implementation:**
|
||||
```python
|
||||
def reciprocal_rank_fusion(
|
||||
ranked_lists: list[list[tuple[str, float]]],
|
||||
*,
|
||||
k: int = 60,
|
||||
) -> dict[str, float]:
|
||||
...
|
||||
# Skip lists with no BM25 signal at all
|
||||
if all(score == 0.0 for _, score in ranked_list):
|
||||
continue
|
||||
|
||||
# IMPORTANT: Keep (candidate, 0.0) entries in the ranked list for transparency,
|
||||
# but treat them as "no signal" for fusion. I.e., candidates with BM25==0.0 MUST
|
||||
# NOT receive a positive fused score just because they have a (low) rank at the
|
||||
# end of the list.
|
||||
for rank, (candidate, score) in enumerate(ranked_list, start=1):
|
||||
if score == 0.0:
|
||||
continue
|
||||
fused[candidate] += 1 / (k + rank)
|
||||
...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4. `best_match` reported per required competence
|
||||
|
||||
**Decision:** `best_match` in the output dict is the candidate with the highest BM25 score
|
||||
(rank 1 in the BM25 list). If no candidate has any token overlap (BM25 = 0 for all),
|
||||
`best_match = None` and `score = 0.0`.
|
||||
|
||||
---
|
||||
|
||||
### 5. Role similarity unchanged
|
||||
|
||||
**Decision:** `compute_role_similarity()` is always embedding-based, regardless of
|
||||
`use_bm25_search`.
|
||||
|
||||
**Rationale:**
|
||||
- Role names are short, single-phrase labels ("Backend Developer", "Data Engineer").
|
||||
- Embeddings handle multilingual and synonym variants better for roles.
|
||||
- BM25's exact-match strength is less valuable for role inference.
|
||||
|
||||
---
|
||||
|
||||
### 6. `rank_bm25` as the BM25 library
|
||||
|
||||
**Decision:** Use the `rank-bm25` PyPI package (`BM25Okapi` class).
|
||||
|
||||
**Rationale:**
|
||||
- Pure Python, no C extensions or native code.
|
||||
- Actively maintained, MIT licensed.
|
||||
- Implements BM25 Okapi (the standard variant) with configurable `k1` and `b` parameters.
|
||||
- Minimal footprint, no additional build steps.
|
||||
|
||||
**Tokenization:**
|
||||
Candidate and query texts are tokenized by lowercasing and splitting on non-word
|
||||
characters (whitespace, hyphens, slashes, parentheses, dots):
|
||||
|
||||
```python
|
||||
import re
|
||||
|
||||
def _tokenize(text: str) -> list[str]:
|
||||
return [t for t in re.split(r"[\W_]+", text.lower()) if t]
|
||||
```
|
||||
|
||||
This ensures "Progressive Web App (PWA)" tokenizes to
|
||||
`["progressive", "web", "app", "pwa"]`, and "CI/CD Pipeline" becomes
|
||||
`["ci", "cd", "pipeline"]`.
|
||||
|
||||
---
|
||||
|
||||
### 7. No changes to `Matcher`, `scorer`, or output format
|
||||
|
||||
**Decision:** `_bm25_rrf_similarity()` returns `dict[str, dict[str, object]]` in the
|
||||
identical shape to `_per_skill_similarity()`:
|
||||
|
||||
```python
|
||||
{
|
||||
"Python": {
|
||||
"score": 0.0,
|
||||
"best_match": None,
|
||||
"rationale": "BM25: no token overlap with any candidate competence.",
|
||||
},
|
||||
"Machine Learning": {
|
||||
"score": 0.0,
|
||||
"best_match": None,
|
||||
"rationale": "BM25: no token overlap with any candidate competence.",
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
The `Matcher.match()` loop, the `ScoredCapacity` model, the `compute_overall()` scorer,
|
||||
and all MCP tool output formatting remain unchanged.
|
||||
|
||||
---
|
||||
|
||||
### 8. LLM Auto-Tagging: ephemeral competence expansion
|
||||
|
||||
**Decision:** When `use_auto_tagging = true`, the `AutoTagger` issues a single LLM
|
||||
chat-completion call per candidate per scoring invocation to identify which required
|
||||
competences are already covered by the candidate's existing competences (via synonym,
|
||||
abbreviation, or cross-language equivalence). The response is a JSON list of canonical
|
||||
required-competence names to add to the candidate's working list. The expanded list is
|
||||
used only for that BM25 scoring call and is **never persisted**.
|
||||
|
||||
**Rationale:**
|
||||
- Auto-tagging directly addresses BM25's known limitation with lexical disjoint synonyms
|
||||
(e.g., "ML" for "Machine Learning", "Softwarearchitektur" for "Software Architecture").
|
||||
- Making the expansion ephemeral keeps the data model clean and audit-friendly.
|
||||
- Restricting additions to "names already covered by the candidate's existing entries"
|
||||
means the LLM cannot invent skills — it can only surface canonical equivalents.
|
||||
- LLM call cost is bounded: one call per candidate per `compute_competence_similarity()`
|
||||
invocation, with a compact prompt (required list + candidate list).
|
||||
|
||||
**LLM prompt schema:**
|
||||
```
|
||||
System: You are a skill-taxonomy assistant. Given a list of REQUIRED competences and a
|
||||
candidate's EXISTING competences, output ONLY the canonical names from REQUIRED
|
||||
that are already covered by one or more entries in EXISTING (via synonym,
|
||||
abbreviation, or cross-language equivalence). Do not invent new skills.
|
||||
|
||||
User: REQUIRED: ["Machine Learning", "Python", "CI/CD"]
|
||||
EXISTING: ["ML", "Python", "Jenkins", "Continuous Integration"]
|
||||
|
||||
Response (JSON): {"additions": ["Machine Learning", "CI/CD"]}
|
||||
```
|
||||
|
||||
**Azure OpenAI `response_format`:** `{"type": "json_object"}` is used to enforce
|
||||
structured JSON output, eliminating parsing failures.
|
||||
|
||||
**Error handling:** If the LLM call fails (timeout, API error, malformed JSON), the
|
||||
`AutoTagger` returns the original unmodified candidate list. BM25 scoring continues with
|
||||
no expansion — graceful degradation.
|
||||
|
||||
---
|
||||
|
||||
## Data Flow
|
||||
|
||||
```
|
||||
find_matching_capacities(...) / find_matching_tasks(...)
|
||||
│
|
||||
└─► Matcher.match(capacities, requirements)
|
||||
│
|
||||
├─ [use_bm25_search=True]
|
||||
│ Build ONE Bm25Index over all filtered candidates' competences (global corpus)
|
||||
│ → global_bm25_index (passed into every compute_competence_similarity call)
|
||||
│
|
||||
└─► for each candidate cap in filtered:
|
||||
SimilarityEngine.compute_competence_similarity(
|
||||
required=requirements.competences,
|
||||
candidate=cap.competences,
|
||||
global_index=global_bm25_index,
|
||||
)
|
||||
│
|
||||
├─ [use_bm25_search=False]
|
||||
│ _per_skill_similarity() / _aggregate_similarity()
|
||||
│ (embedding-based, unchanged)
|
||||
│
|
||||
└─ [use_bm25_search=True]
|
||||
│
|
||||
├─ [use_auto_tagging=True]
|
||||
│ AutoTagger.expand_competences(
|
||||
│ required=required,
|
||||
│ existing=candidate,
|
||||
│ )
|
||||
│ → expanded_candidate (ephemeral, not persisted)
|
||||
│
|
||||
└─► _bm25_rrf_similarity(required, expanded_candidate, global_index)
|
||||
│
|
||||
├─► global_index.rank(query=required_comp)
|
||||
│ → filter to candidate_set
|
||||
│ → list[(candidate_text, bm25_score)]
|
||||
│ (falls back to bm25_rank_competences() if no global_index)
|
||||
│
|
||||
└─► reciprocal_rank_fusion(ranked_lists)
|
||||
→ dict[required_comp → rrf_score]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Module Structure
|
||||
|
||||
```
|
||||
src/teamlandkarte_mcp/matching/
|
||||
├── auto_tagger.py # AutoTagger class + expand_competences()
|
||||
├── bm25.py # Bm25Index dataclass + bm25_rank_competences()
|
||||
├── rrf.py # reciprocal_rank_fusion()
|
||||
├── matcher.py # unchanged
|
||||
├── scorer.py # unchanged
|
||||
└── similarity.py # extended: _bm25_rrf_similarity(), auto-tag branch, flag routing
|
||||
|
||||
src/teamlandkarte_mcp/azure/
|
||||
└── openai_client.py # extended: chat_completion(), chat_deployment, llm_api_key
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## `bm25.py` — Public Interface
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class Bm25Index:
|
||||
"""In-memory BM25 index over a corpus of competence strings.
|
||||
|
||||
Intended to be built once over the union of all candidates' competences
|
||||
(the global corpus) so that IDF values are meaningful. Each per-candidate
|
||||
scoring call queries the global index and filters results to that candidate's
|
||||
own competence set.
|
||||
|
||||
Args:
|
||||
corpus: List of competence strings to index.
|
||||
|
||||
Notes:
|
||||
Tokenization lowercases and splits on non-word characters.
|
||||
An empty corpus produces an index that scores all queries as 0.0.
|
||||
Raw BM25 scores are clamped to ``max(0.0, raw)`` — ubiquitous terms
|
||||
whose IDF is negative simply produce 0.0.
|
||||
"""
|
||||
corpus: list[str]
|
||||
|
||||
def rank(self, query: str) -> list[tuple[str, float]]:
|
||||
"""Rank corpus documents for a single query string.
|
||||
|
||||
Args:
|
||||
query: Required competence text used as the BM25 query.
|
||||
|
||||
Returns:
|
||||
List of (candidate_text, bm25_score) sorted by score descending.
|
||||
Entries with score 0.0 are included (caller decides cutoff).
|
||||
Scores are clamped to max(0.0, raw); negative IDF yields 0.0.
|
||||
"""
|
||||
...
|
||||
|
||||
|
||||
def bm25_rank_competences(
|
||||
required: str,
|
||||
candidates: list[str],
|
||||
) -> list[tuple[str, float]]:
|
||||
"""Rank candidate competences against a single required competence.
|
||||
|
||||
Convenience wrapper: builds a temporary ``Bm25Index`` and ranks the query.
|
||||
Suitable for standalone use and tests; in production the global index built
|
||||
by ``Matcher`` is preferred (see ``Bm25Index`` notes on IDF correctness).
|
||||
|
||||
Args:
|
||||
required: The required competence text (BM25 query).
|
||||
candidates: Candidate competence strings to rank.
|
||||
|
||||
Returns:
|
||||
Ranked list of (candidate_text, bm25_score), descending by score.
|
||||
"""
|
||||
...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## `rrf.py` — Public Interface
|
||||
|
||||
```python
|
||||
def reciprocal_rank_fusion(
|
||||
ranked_lists: list[list[tuple[str, float]]],
|
||||
*,
|
||||
k: int = 60,
|
||||
) -> dict[str, float]:
|
||||
"""Fuse multiple ranked lists using Reciprocal Rank Fusion.
|
||||
|
||||
For each candidate present in any list, the fused score is:
|
||||
|
||||
score(c) = Σ_i 1 / (k + rank_i(c))
|
||||
|
||||
where rank_i(c) is the 1-based rank of candidate c in list i.
|
||||
|
||||
Candidates with zero BM25 score in ALL contributing lists receive
|
||||
a fused score of 0.0 (no token overlap signal).
|
||||
|
||||
Scores are normalized so the top-ranked candidate receives 1.0.
|
||||
|
||||
Args:
|
||||
ranked_lists: One or more ranked lists of (text, score) pairs.
|
||||
Each list MUST be sorted by score descending.
|
||||
Lists with all-zero scores are skipped (no signal).
|
||||
k: RRF smoothing constant (default 60, Cormack et al. 2009).
|
||||
Higher k flattens rank differences; lower k amplifies them.
|
||||
|
||||
Returns:
|
||||
Mapping of candidate text → normalized fused score in [0.0, 1.0].
|
||||
Returns empty dict if all input lists are empty or all-zero.
|
||||
"""
|
||||
...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## `auto_tagger.py` — Public Interface
|
||||
|
||||
```python
|
||||
class AutoTagger:
|
||||
"""LLM-based competence expander for BM25 pre-processing.
|
||||
|
||||
Calls an Azure OpenAI chat model to identify which required competences
|
||||
are already covered by the candidate's existing competences (via synonym,
|
||||
abbreviation, or cross-language equivalence) and returns the canonical
|
||||
required-competence names as additions.
|
||||
|
||||
All LLM calls use ``response_format={"type": "json_object"}`` to ensure
|
||||
parseable output. On any LLM or parsing error, the original candidate list
|
||||
is returned unchanged (graceful degradation).
|
||||
|
||||
Args:
|
||||
client: An ``AzureOpenAIClient`` instance configured with a chat
|
||||
deployment (``chat_deployment`` and ``llm_api_key``).
|
||||
"""
|
||||
|
||||
def __init__(self, client: AzureOpenAIClient) -> None: ...
|
||||
|
||||
def expand_competences(
|
||||
self,
|
||||
required: list[str],
|
||||
existing: list[str],
|
||||
) -> list[str]:
|
||||
"""Expand a candidate's competence list with covered required terms.
|
||||
|
||||
Sends a prompt to the LLM asking: given ``required`` competences and
|
||||
the candidate's ``existing`` competences, which required competences
|
||||
are already implicitly covered by the existing entries?
|
||||
|
||||
The returned list is ``existing + additions`` where ``additions``
|
||||
are the canonical required-competence names identified by the LLM.
|
||||
|
||||
The result is **ephemeral**: it is used only for the current BM25
|
||||
scoring call and is never written back to the database.
|
||||
|
||||
Args:
|
||||
required: List of required competence names (BM25 query terms).
|
||||
existing: Candidate's current competence list.
|
||||
|
||||
Returns:
|
||||
Combined list ``existing + additions``. If the LLM call fails or
|
||||
returns no additions, returns ``existing`` unchanged.
|
||||
"""
|
||||
...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
### `SimilarityConfig` (updated)
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class SimilarityConfig:
|
||||
"""Similarity engine configuration."""
|
||||
|
||||
embedding_model: str = "text-embedding-3-large"
|
||||
embedding_dimensions: int = 3072
|
||||
strategy: str = "per_skill"
|
||||
use_bm25_search: bool = False
|
||||
"""When True, BM25 + RRF replaces embedding-based competence similarity.
|
||||
Role similarity is always embedding-based.
|
||||
Default: False (preserves existing behavior).
|
||||
"""
|
||||
use_auto_tagging: bool = False
|
||||
"""When True (and use_bm25_search is also True), calls the LLM AutoTagger
|
||||
before each BM25 scoring call to expand the candidate's competence list
|
||||
with canonical equivalents of required competences it already covers.
|
||||
The expansion is ephemeral and never persisted.
|
||||
Default: False.
|
||||
"""
|
||||
```
|
||||
|
||||
### `AzureOpenAIConfig` (updated)
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class AzureOpenAIConfig:
|
||||
"""Azure OpenAI client configuration."""
|
||||
|
||||
# ... existing fields ...
|
||||
chat_deployment: str = ""
|
||||
"""Azure deployment name of the chat model used for auto-tagging
|
||||
(e.g. ``"gpt-4.1"``). Required only when ``use_auto_tagging = true``.
|
||||
"""
|
||||
llm_api_key: str = ""
|
||||
"""Azure OpenAI API key for the chat deployment. Read from the
|
||||
``AZURE_OPENAI_LLM_API_KEY`` environment variable.
|
||||
Required when ``use_auto_tagging = true``; raises ``ConfigurationError``
|
||||
at startup if absent.
|
||||
"""
|
||||
```
|
||||
|
||||
### Config parser (`config.toml` keys)
|
||||
|
||||
```toml
|
||||
[matching.similarity]
|
||||
strategy = "per_skill"
|
||||
use_bm25_search = false # new — enables BM25+RRF competence matching
|
||||
use_auto_tagging = false # new — enables LLM pre-expansion before BM25
|
||||
|
||||
[azure_openai]
|
||||
# ... existing keys ...
|
||||
chat_deployment = "gpt-4.1" # new — required when use_auto_tagging = true
|
||||
```
|
||||
|
||||
### Environment variables
|
||||
|
||||
| Variable | Required when | Description |
|
||||
|----------|--------------|-------------|
|
||||
| `AZURE_OPENAI_LLM_API_KEY` | `use_auto_tagging = true` | API key for the Azure OpenAI chat deployment used in auto-tagging. Server raises `ConfigurationError` at startup if absent and `use_auto_tagging = true`. |
|
||||
|
||||
---
|
||||
|
||||
## BM25 Score Behavior
|
||||
|
||||
### Example: Query "Python" against candidate competences
|
||||
|
||||
| Candidate | Tokens | BM25 Score | RRF Score (k=60) |
|
||||
|-----------|--------|------------|-----------------|
|
||||
| Python | ["python"] | ~3.5 | 1.000 |
|
||||
| Python (advanced) | ["python", "advanced"] | ~2.1 | 0.984 |
|
||||
| JavaScript | ["javascript"] | 0.0 | **0.000** |
|
||||
| TypeScript | ["typescript"] | 0.0 | **0.000** |
|
||||
| Node.js | ["node", "js"] | 0.0 | **0.000** |
|
||||
|
||||
Candidates without any shared token receive **0.0** — the false-positive problem is
|
||||
eliminated.
|
||||
|
||||
### Example: Query "Machine Learning" against candidate competences
|
||||
|
||||
| Candidate | Tokens | BM25 Score | RRF Score |
|
||||
|-----------|--------|------------|-----------|
|
||||
| Machine Learning | ["machine", "learning"] | ~4.0 | 1.000 |
|
||||
| Deep Learning | ["deep", "learning"] | ~1.8 | 0.984 |
|
||||
| Machine Learning (Python) | ["machine", "learning", "python"] | ~3.2 | 0.492 |
|
||||
| JavaScript | ["javascript"] | 0.0 | **0.000** |
|
||||
|
||||
---
|
||||
|
||||
## Known Limitations
|
||||
|
||||
### 1. Conceptual synonyms with zero token overlap score 0.0
|
||||
|
||||
BM25 requires at least one shared token between query and document. Skill pairs that are
|
||||
semantically equivalent but lexically disjoint will always score 0.0:
|
||||
|
||||
| Required competence | Candidate competence | Shared tokens | BM25 score |
|
||||
|---------------------|----------------------|---------------|------------|
|
||||
| Data Science | Machine Learning | none | **0.000** |
|
||||
| Data Science | Artificial Intelligence | none | **0.000** |
|
||||
| Frontend Development | UI Engineering | none | **0.000** |
|
||||
| Agile | Scrum | none | **0.000** |
|
||||
| Softwarearchitektur | Software Architecture | none | **0.000** |
|
||||
|
||||
These are **false negatives** — real skill matches that BM25 misses entirely. Embeddings
|
||||
handle all of these correctly via semantic similarity.
|
||||
|
||||
**Consequence:** BM25 mode is best suited to datasets where competence names are
|
||||
**canonical, consistent, and in a single language**. If profiles mix German and English
|
||||
skill names, or use informal abbreviations ("ML" for "Machine Learning"), BM25 will
|
||||
produce false negatives that embeddings would have caught.
|
||||
|
||||
**Mitigation:** Enable `use_auto_tagging = true`. The LLM will bridge synonym and
|
||||
cross-language gaps by pre-expanding the candidate list before BM25 scoring. This
|
||||
addresses limitations 1, 2, and 3 at the cost of one LLM call per candidate per scoring
|
||||
invocation.
|
||||
|
||||
---
|
||||
|
||||
### 2. Abbreviations and acronyms
|
||||
|
||||
Short forms that do not share tokens with the expanded form score 0.0:
|
||||
|
||||
| Required | Candidate | BM25 score |
|
||||
|----------|-----------|------------|
|
||||
| Machine Learning | ML | **0.000** |
|
||||
| Continuous Integration | CI | **0.000** |
|
||||
| Natural Language Processing | NLP | **0.000** |
|
||||
|
||||
Partial exception: "CI/CD" tokenizes to `["ci", "cd"]`, so a query "CI/CD Pipeline"
|
||||
would find partial overlap with a candidate "CI/CD", but not with "Continuous
|
||||
Integration/Continuous Delivery".
|
||||
|
||||
**Mitigation:** `use_auto_tagging = true` (see limitation 1).
|
||||
|
||||
---
|
||||
|
||||
### 3. Cross-language skill names
|
||||
|
||||
A required competence in German and a candidate competence in English (or vice versa)
|
||||
score 0.0 even when they describe the same skill:
|
||||
|
||||
| Required | Candidate | BM25 score |
|
||||
|----------|-----------|------------|
|
||||
| Softwarearchitektur | Software Architecture | **0.000** |
|
||||
| Maschinelles Lernen | Machine Learning | **0.000** |
|
||||
| Datenwissenschaft | Data Science | **0.000** |
|
||||
|
||||
This is a significant limitation for German-language skill databases. Embeddings handle
|
||||
cross-language pairs well; BM25 does not.
|
||||
|
||||
**Mitigation:** `use_auto_tagging = true` (see limitation 1).
|
||||
|
||||
---
|
||||
|
||||
### 4. RRF score compression near 1.0
|
||||
|
||||
Because the normalization formula is `(k+1) / (k+rank)` with `k=60`, ranks 1–5 all
|
||||
compress into the range `[0.871, 1.0]`. There is little score differentiation between
|
||||
the top-ranked candidates. This is intentional (rank differences matter less than
|
||||
presence/absence of token overlap) but means BM25 mode should not be used to produce
|
||||
fine-grained competence similarity scores — only to separate matching (score > 0) from
|
||||
non-matching (score = 0) candidates.
|
||||
|
||||
---
|
||||
|
||||
### 5. Small per-person corpus causes IDF pathology — mitigated by global index
|
||||
|
||||
BM25's IDF term rewards tokens that are rare across the corpus. When the BM25 index was
|
||||
built per candidate (N = that person's competence count ≈ 5–30), any term appearing in
|
||||
most documents had IDF ≤ 0, causing `rank_bm25` to return negative raw scores. After
|
||||
`max(0.0, raw)` clamping, all candidates scored 0.0 — making the BM25 signal useless.
|
||||
|
||||
**Fix (implemented):** The `Bm25Index` is now built once over the **union of all filtered
|
||||
candidates' competences** in `Matcher.match()` before the per-candidate loop (see Key
|
||||
Decision #2). This makes N equal to the total unique competence count across all
|
||||
candidates, giving IDF proper signal. Per-candidate scoring filters the global results to
|
||||
only that candidate's competence set.
|
||||
|
||||
**Residual limitation:** If the filtered candidate pool itself is very small (e.g., 2–3
|
||||
candidates with few unique competences), IDF may still be weak. In practice this is
|
||||
negligible since matching runs are always invoked against a meaningful pool of candidates.
|
||||
|
||||
---
|
||||
|
||||
## Test Strategy
|
||||
|
||||
### Unit tests
|
||||
|
||||
1. **`tests/test_bm25.py`**
|
||||
- `Bm25Index` with empty corpus → all scores 0.0
|
||||
- Exact match → score > 0
|
||||
- No token overlap → score == 0.0
|
||||
- Partial token overlap (multi-token queries) → partial score
|
||||
- Case-insensitive matching ("Python" == "python")
|
||||
- Tokenization edge cases: hyphens, parentheses, slashes
|
||||
|
||||
2. **`tests/test_rrf.py`**
|
||||
- Single list, single candidate → score 1.0
|
||||
- Single list, multiple candidates → scores normalized to [0, 1]
|
||||
- All-zero list → returns empty / all zeros
|
||||
- `k` parameter effect on score distribution
|
||||
|
||||
3. **`tests/test_auto_tagger.py`** (new)
|
||||
- LLM returns valid additions → `expand_competences()` appends them to existing list
|
||||
- LLM returns empty additions → original list returned unchanged
|
||||
- LLM call raises exception → original list returned unchanged (graceful degradation)
|
||||
- LLM returns malformed JSON → original list returned unchanged
|
||||
- Additions from LLM that are NOT in `required` are ignored
|
||||
- `expand_competences()` never duplicates entries already in `existing`
|
||||
- All tests use a mocked `AzureOpenAIClient` (no real LLM calls)
|
||||
|
||||
4. **`tests/test_similarity.py`** (extended)
|
||||
- `compute_competence_similarity()` with `use_bm25_search=True`
|
||||
- Required "Python", candidate ["JavaScript"] → score 0.0
|
||||
- Required "Python", candidate ["Python"] → score 1.0
|
||||
- Required "Python" + "Machine Learning", candidate ["JavaScript", "TypeScript"]
|
||||
→ both scores 0.0, overall competence_score 0.0
|
||||
- With `use_bm25_search=True, use_auto_tagging=True`: mocked `AutoTagger` injects
|
||||
"Machine Learning" into candidate ["ML"] → score > 0
|
||||
|
||||
### Integration test
|
||||
|
||||
5. **`tests/test_matcher_integration.py`** (extended)
|
||||
- With `use_bm25_search=True`: candidate with exact skills scores higher than
|
||||
candidate with only semantically adjacent skills.
|
||||
- The false-positive scenario (Python/ML vs JavaScript/TypeScript) is explicitly
|
||||
tested.
|
||||
- With `use_bm25_search=True, use_auto_tagging=True` (mocked LLM): candidate with
|
||||
"ML" matches required "Machine Learning" after auto-tag expansion.
|
||||
@@ -0,0 +1,288 @@
|
||||
# Change Proposal: BM25 + RRF Competence Matching
|
||||
|
||||
- **Change ID**: `add-bm25-rrf-competence-matching`
|
||||
- **Status**: Proposed
|
||||
- **Target**: `teamlandkarte-mcp`
|
||||
- **Author**: Thomas Handke
|
||||
- **Date**: 2026-02-26
|
||||
|
||||
## Summary
|
||||
|
||||
Add an optional BM25-based competence matching strategy as an alternative to the existing
|
||||
embedding-based similarity approach. When enabled, the following pipeline replaces
|
||||
embedding-based competence similarity:
|
||||
|
||||
1. **Auto-Tagging (LLM):** Before BM25 scoring, an LLM compares the required competences
|
||||
against each candidate's existing competences and suggests canonical additions
|
||||
(synonyms, abbreviations, cross-language equivalents). The expanded competence list
|
||||
is ephemeral — it is used only for the current scoring call and is never persisted.
|
||||
2. **BM25 Ranking:** The (expanded) candidate competences are indexed with BM25. Each
|
||||
required competence is used as a query to rank the candidates.
|
||||
3. **RRF Fusion:** Reciprocal Rank Fusion normalizes the BM25 ranked list into a score
|
||||
in [0, 1] per required competence.
|
||||
|
||||
The feature is controlled by two new booleans in `config.toml`:
|
||||
- `use_bm25_search` (default `false`): enables BM25 + RRF.
|
||||
- `use_auto_tagging` (default `false`): enables LLM-based competence expansion before
|
||||
BM25 scoring. Strongly recommended when `use_bm25_search = true`.
|
||||
|
||||
## Why
|
||||
|
||||
The current embedding-based (`per_skill` / `aggregate`) strategy has a known weakness:
|
||||
it assigns non-trivial scores to candidates whose competences are only _semantically
|
||||
adjacent_ to the required skills but not actually matching.
|
||||
|
||||
**Example (observed in production):**
|
||||
Searching for **Python** + **Machine Learning**, a candidate with only
|
||||
**JavaScript, TypeScript, Node.js, Vue.js, Web Architekturen** received a score of **0.392**
|
||||
because the embedding model considers all programming languages semantically related.
|
||||
|
||||
BM25 (Best Match 25) is a classical lexical ranking algorithm that scores documents by
|
||||
**term overlap** between query and document. It is immune to the semantic drift problem
|
||||
because it only counts tokens that literally occur in the candidate text. BM25 will assign
|
||||
**0** to "JavaScript" when the query is "Python".
|
||||
|
||||
### When BM25 outperforms embeddings
|
||||
|
||||
| Scenario | Embeddings | BM25 |
|
||||
|----------|-----------|------|
|
||||
| Query "Python", candidate has "Python" | High ✓ | High ✓ |
|
||||
| Query "Python", candidate has "JavaScript" | Medium ✗ | Zero ✓ |
|
||||
| Query "Machine Learning", candidate has "ML" | High ✓ | Low ✗ |
|
||||
| Query "Softwarearchitektur", candidate has "software architecture" | High ✓ | Low ✗ |
|
||||
| Query "React", candidate has "React.js" | High ✓ | Medium ~ |
|
||||
|
||||
BM25 excels at exact or near-exact skill names. Embeddings excel at synonyms and
|
||||
cross-language variants (e.g., German/English). The optional `use_auto_tagging` feature
|
||||
bridges BM25's synonym and cross-language gap by having an LLM pre-expand the candidate
|
||||
competence list — making the combination of BM25 + Auto-Tagging competitive with
|
||||
embeddings while still eliminating false positives from semantic drift.
|
||||
|
||||
### Why RRF for fusion
|
||||
|
||||
Reciprocal Rank Fusion merges multiple ranked lists into one without requiring score
|
||||
normalization. For each required competence query, we obtain a ranked list of candidate
|
||||
competences. RRF then assigns each candidate a fused score based on its (1-based) rank in
|
||||
one or more lists.
|
||||
|
||||
RRF fused score formula (Cormack et al. 2009):
|
||||
|
||||
```
|
||||
rrf_score(candidate) = Σ_i 1 / (k + rank_i(candidate))
|
||||
```
|
||||
|
||||
RRF is robust to score scale differences and has been widely validated in information
|
||||
retrieval research as a high-quality fusion method.
|
||||
|
||||
## What Changes
|
||||
|
||||
### 1. New config parameter `matching.similarity.use_bm25_search` (boolean, default `false`)
|
||||
|
||||
- When `false` (default): existing embedding-based strategy is used unchanged.
|
||||
- When `true`: BM25 + RRF replaces embedding-based competence similarity.
|
||||
Role similarity is **always** embedding-based (unchanged).
|
||||
|
||||
### 2. New config parameter `matching.similarity.use_auto_tagging` (boolean, default `false`)
|
||||
|
||||
- Only applies when `use_bm25_search = true`. When `false`, BM25 runs on the raw
|
||||
candidate competence list (no LLM expansion).
|
||||
- When `true`: before each BM25 scoring call, an LLM expands the candidate's competence
|
||||
list with canonical forms of any required competences that are already covered by the
|
||||
candidate's existing entries (via synonym, abbreviation, or cross-language equivalence).
|
||||
- The expanded list is ephemeral — it is not written back to the database and does not
|
||||
affect any other operation.
|
||||
|
||||
### 3. New config parameter `azure_openai.chat_deployment` (string)
|
||||
|
||||
- Azure deployment name of the chat model used for auto-tagging (e.g. `"gpt-4.1"`).
|
||||
- Required only when `use_auto_tagging = true`.
|
||||
- API key is read from environment variable `AZURE_OPENAI_LLM_API_KEY`.
|
||||
|
||||
### 4. `AzureOpenAIClient` extended: LLM chat capability restored
|
||||
|
||||
- Add `chat_completion(system: str, user: str) -> str` method supporting structured JSON
|
||||
responses via Azure OpenAI chat completions API.
|
||||
- Add `chat_deployment` and `llm_api_key` constructor parameters.
|
||||
- This restores LLM capability removed in a previous change, now scoped exclusively
|
||||
to auto-tagging.
|
||||
|
||||
### 5. New module `src/teamlandkarte_mcp/matching/auto_tagger.py`
|
||||
|
||||
Implements:
|
||||
- `AutoTagger`: class that wraps the LLM client and executes the tagging prompt.
|
||||
- `expand_competences()`: given `required: list[str]` and `existing: list[str]`,
|
||||
returns `list[str]` — the combined list `existing + additions`, where `additions`
|
||||
are the canonical required-competence names that the LLM identifies as already
|
||||
covered by `existing` entries.
|
||||
- Structured JSON output schema enforced via the Azure OpenAI `response_format` parameter.
|
||||
- Google Docstrings on all public symbols.
|
||||
|
||||
### 6. New module `src/teamlandkarte_mcp/matching/bm25.py`
|
||||
|
||||
Implements:
|
||||
- `Bm25Index`: builds an in-memory BM25 index over a corpus of competence strings.
|
||||
Designed to be instantiated once over the global candidate pool (see item 8a below)
|
||||
so IDF values are meaningful. Raw scores are clamped to `max(0.0, raw)` — terms with
|
||||
negative IDF produce 0.0 rather than a spurious tiny positive value.
|
||||
- `bm25_rank_competences()`: for a single required competence query, returns a ranked list
|
||||
of `(candidate_text, score)` tuples. Convenience wrapper for standalone/test use.
|
||||
- Pure Python, no external runtime dependencies beyond `rank_bm25` from PyPI.
|
||||
|
||||
### 7. New module `src/teamlandkarte_mcp/matching/rrf.py`
|
||||
|
||||
Implements:
|
||||
- `reciprocal_rank_fusion()`: merges one or more ranked lists into a final score dict.
|
||||
- `k` constant (default `60`, standard RRF value from Cormack et al. 2009).
|
||||
- Returns `dict[str, float]` mapping candidate text → fused score in `(0, 1]`
|
||||
(top-ranked candidate receives exactly `1.0`).
|
||||
|
||||
### 8. `SimilarityEngine.compute_competence_similarity()` extended
|
||||
|
||||
- Reads `use_bm25_search` and `use_auto_tagging` flags.
|
||||
- Accepts an optional `global_index: Bm25Index | None` parameter (passed in by `Matcher`).
|
||||
- When `use_bm25_search=True`:
|
||||
1. If `use_auto_tagging=True`: calls `AutoTagger.expand_competences()` to build the
|
||||
expanded candidate list.
|
||||
2. Delegates to `_bm25_rrf_similarity()` with the (optionally expanded) candidate list
|
||||
and the global index.
|
||||
- `_bm25_rrf_similarity()` uses `global_index.rank(req)` filtered to the candidate's
|
||||
competence set when `global_index` is provided, otherwise falls back to
|
||||
`bm25_rank_competences(req, candidate)`.
|
||||
- Output shape is **identical** to existing strategies:
|
||||
`dict[required_competence → {score, best_match, rationale}]`
|
||||
- Preserves the `Matcher` and scorer contracts unchanged.
|
||||
|
||||
### 8a. `Matcher.match()` builds a global `Bm25Index` before the per-candidate loop
|
||||
|
||||
When `use_bm25_search=True` and the filtered candidate pool is non-empty, `Matcher.match()`
|
||||
collects the deduplicated union of all filtered candidates' competences and builds one
|
||||
`Bm25Index` over that global corpus before the loop. The same index is passed to every
|
||||
`compute_competence_similarity()` call.
|
||||
|
||||
**Why this matters:** With a per-person corpus (N ≈ 5–30 entries), BM25 IDF is 0 or
|
||||
negative for every term — all raw scores collapse to ≤ 0, and `max(0.0, raw)` clamping
|
||||
makes every match appear as 0.0. A global corpus ensures N is large enough for IDF to
|
||||
carry real signal.
|
||||
|
||||
### 9. `SimilarityConfig` dataclass extended
|
||||
|
||||
New fields:
|
||||
- `use_bm25_search: bool = False`
|
||||
- `use_auto_tagging: bool = False`
|
||||
|
||||
### 10. `AzureOpenAIConfig` dataclass extended
|
||||
|
||||
New fields:
|
||||
- `chat_deployment: str = ""`
|
||||
- (API key via `AZURE_OPENAI_LLM_API_KEY` env var)
|
||||
|
||||
### 11. `config.py` parser extended
|
||||
|
||||
Reads `matching.similarity.use_bm25_search`, `matching.similarity.use_auto_tagging`,
|
||||
and `azure_openai.chat_deployment` from TOML. Reads `AZURE_OPENAI_LLM_API_KEY` from
|
||||
environment when `use_auto_tagging = true`.
|
||||
|
||||
### 12. `config.toml.example` updated
|
||||
|
||||
Documents all three new keys with inline comments.
|
||||
|
||||
### 13. Server initialization in `mcp_server.py`
|
||||
|
||||
- Passes `use_bm25_search` and `use_auto_tagging` flags when constructing `SimilarityEngine`.
|
||||
- When `use_auto_tagging=True`: constructs `AutoTagger` with the LLM-capable
|
||||
`AzureOpenAIClient` and passes it to `SimilarityEngine`.
|
||||
|
||||
### 14. Documentation updated
|
||||
|
||||
- `docs/architecture.md`: new section on BM25/RRF/Auto-Tagging strategy.
|
||||
- `docs/semantic_similarity_tradeoffs.md`: comparison table updated with BM25+AutoTag entry.
|
||||
- `docs/assistant_system_prompt.md`: notes on score interpretation with BM25 and auto-tagging.
|
||||
- `README.md`: all new config parameters documented.
|
||||
|
||||
### 15. Legacy/deprecated code removed
|
||||
|
||||
- Remove `debug_similarity_analysis.py`
|
||||
- Remove `debug_categorization.py`
|
||||
- Remove `SEMANTIC_SIMILARITY_ANALYSIS.md`
|
||||
- Remove `SCORE_BUG_FIX.md`
|
||||
|
||||
### 16. Tests
|
||||
|
||||
- `tests/test_bm25.py`: unit tests for `Bm25Index` and `bm25_rank_competences()`.
|
||||
- `tests/test_rrf.py`: unit tests for `reciprocal_rank_fusion()`.
|
||||
- `tests/test_auto_tagger.py`: unit tests for `AutoTagger.expand_competences()` (mocked LLM).
|
||||
- `tests/test_similarity.py`: extend with BM25/RRF and auto-tagging strategy tests.
|
||||
- `tests/test_matcher_integration.py`: integration test for the full pipeline
|
||||
(auto-tag → BM25 → RRF) in both matching directions.
|
||||
|
||||
## Impact
|
||||
|
||||
### Affected specs
|
||||
|
||||
- `matching-tools` (competence scoring path)
|
||||
|
||||
### Affected code
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `src/teamlandkarte_mcp/matching/similarity.py` | Add `_bm25_rrf_similarity()`, `global_index` param, auto-tag branch, read flags |
|
||||
| `src/teamlandkarte_mcp/matching/matcher.py` | Build global `Bm25Index` before per-candidate loop; pass as `global_index` |
|
||||
| `src/teamlandkarte_mcp/matching/bm25.py` | **New** |
|
||||
| `src/teamlandkarte_mcp/matching/rrf.py` | **New** |
|
||||
| `src/teamlandkarte_mcp/matching/auto_tagger.py` | **New** |
|
||||
| `src/teamlandkarte_mcp/azure/openai_client.py` | Restore `chat_completion()`; add `chat_deployment` + `llm_api_key` params |
|
||||
| `src/teamlandkarte_mcp/config.py` | `SimilarityConfig.use_bm25_search`, `use_auto_tagging`; `AzureOpenAIConfig.chat_deployment` |
|
||||
| `src/teamlandkarte_mcp/mcp_server.py` | Pass flags + `AutoTagger` to `SimilarityEngine` |
|
||||
| `config.toml.example` | Document all new keys |
|
||||
| `docs/architecture.md` | BM25/RRF/Auto-Tagging section |
|
||||
| `docs/semantic_similarity_tradeoffs.md` | Updated comparison with BM25+AutoTag entry |
|
||||
| `docs/assistant_system_prompt.md` | Score notes for BM25 and auto-tagging modes |
|
||||
| `README.md` | All new config parameters |
|
||||
| `tests/test_bm25.py` | **New** |
|
||||
| `tests/test_rrf.py` | **New** |
|
||||
| `tests/test_auto_tagger.py` | **New** |
|
||||
| `tests/test_similarity.py` | Extended |
|
||||
| `tests/test_matcher_integration.py` | Extended |
|
||||
| `debug_similarity_analysis.py` | **Removed** |
|
||||
| `debug_categorization.py` | **Removed** |
|
||||
| `SEMANTIC_SIMILARITY_ANALYSIS.md` | **Removed** |
|
||||
| `SCORE_BUG_FIX.md` | **Removed** |
|
||||
|
||||
### Breaking changes
|
||||
|
||||
None. The `use_bm25_search = false` and `use_auto_tagging = false` defaults preserve all
|
||||
existing behavior. The `overall_score` computation (weighted mean of `role_score` and
|
||||
`competence_score`) is unchanged.
|
||||
|
||||
### New environment variables
|
||||
|
||||
| Variable | Required when | Description |
|
||||
|----------|--------------|-------------|
|
||||
| `AZURE_OPENAI_LLM_API_KEY` | `use_auto_tagging = true` | Azure OpenAI API key for the chat model used in auto-tagging. The server raises a configuration error at startup if this key is absent and `use_auto_tagging = true`. |
|
||||
|
||||
### New dependencies
|
||||
|
||||
- `rank-bm25` (PyPI): pure-Python BM25 implementation, no native code, no external services.
|
||||
Adds to `pyproject.toml` `dependencies`.
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
### A. Hard-code exact-match bonus in per_skill
|
||||
|
||||
Simpler but not principled: would require special-casing the scorer instead of a clean
|
||||
strategy abstraction.
|
||||
|
||||
### B. Hybrid embedding + BM25 in a single `_bm25_rrf_similarity()` call
|
||||
|
||||
Combining both signals via RRF is architecturally clean and naturally extensible. However,
|
||||
to keep this change focused and reviewable, the initial implementation uses BM25-only RRF.
|
||||
A hybrid mode can be added as a follow-up change.
|
||||
|
||||
### C. Replace embeddings entirely
|
||||
|
||||
Not desirable. Embeddings are essential for role inference and handle multilingual /
|
||||
synonymous skill names (e.g., "Softwarearchitektur" ↔ "Software Architecture").
|
||||
|
||||
## Open Questions
|
||||
|
||||
None. The design is sufficiently clear to proceed.
|
||||
+266
@@ -0,0 +1,266 @@
|
||||
# Spec Delta: BM25 + RRF Competence Matching
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: BM25 + RRF optional competence matching strategy
|
||||
|
||||
The system SHALL support an optional BM25-based competence matching strategy controlled
|
||||
by `matching.similarity.use_bm25_search` in `config.toml`. When enabled, BM25 + RRF
|
||||
replaces embedding-based competence similarity for `find_matching_capacities` and
|
||||
`find_matching_tasks`. Role similarity SHALL always remain embedding-based.
|
||||
|
||||
#### Scenario: Candidate without matching skills receives zero competence score
|
||||
|
||||
- **GIVEN** `use_bm25_search = true` in `config.toml`
|
||||
- **AND** a task requires competences `["Python", "Machine Learning"]`
|
||||
- **AND** a candidate has competences `["JavaScript", "TypeScript", "Node.js", "Vue.js"]`
|
||||
- **WHEN** the system computes `competence_score` for that candidate
|
||||
- **THEN** `competence_score == 0.0`
|
||||
- **AND** the candidate is categorized as "Irrelevant" or "Low" based on `overall_score`
|
||||
|
||||
#### Scenario: Candidate with exact matching skills receives high competence score
|
||||
|
||||
- **GIVEN** `use_bm25_search = true` in `config.toml`
|
||||
- **AND** a task requires competences `["Python", "Machine Learning"]`
|
||||
- **AND** a candidate has competences `["Python", "Machine Learning", "Pandas"]`
|
||||
- **WHEN** the system computes `competence_score` for that candidate
|
||||
- **THEN** `competence_score > 0.8`
|
||||
- **AND** the candidate scores significantly higher than a candidate with no token overlap
|
||||
|
||||
#### Scenario: BM25 mode is disabled by default
|
||||
|
||||
- **GIVEN** no `use_bm25_search` key in `config.toml` (or `use_bm25_search = false`)
|
||||
- **WHEN** the system computes competence similarity
|
||||
- **THEN** the existing embedding-based strategy is used
|
||||
- **AND** behavior is bit-for-bit identical to the pre-change implementation
|
||||
|
||||
### Requirement: BM25 index ranks candidates by token overlap per required competence
|
||||
|
||||
The `Bm25Index` SHALL build an in-memory BM25 index over a corpus of competence strings
|
||||
and rank them for each required competence query. The index is built once over the global
|
||||
candidate pool per matching run (see "Global BM25 index" requirement below) and is queried
|
||||
per candidate with results filtered to that candidate's competence set.
|
||||
|
||||
#### Scenario: BM25 index ranks exact token match highest
|
||||
|
||||
- **GIVEN** a candidate corpus `["Python", "JavaScript", "TypeScript"]`
|
||||
- **AND** query `"Python"`
|
||||
- **WHEN** the `Bm25Index.rank()` method is called
|
||||
- **THEN** `"Python"` is ranked first with the highest BM25 score
|
||||
- **AND** `"JavaScript"` and `"TypeScript"` have BM25 score 0.0 (no shared tokens)
|
||||
|
||||
#### Scenario: BM25 index handles empty corpus without error
|
||||
|
||||
- **GIVEN** a candidate with no competences (empty corpus)
|
||||
- **WHEN** the system computes `competence_score`
|
||||
- **THEN** `competence_score == 0.0` for every required competence
|
||||
- **AND** no exception is raised
|
||||
|
||||
#### Scenario: Tokenization is case-insensitive and splits on non-word characters
|
||||
|
||||
- **GIVEN** a candidate corpus `["Progressive Web App (PWA)", "CI/CD Pipeline"]`
|
||||
- **AND** query `"ci cd"`
|
||||
- **WHEN** the `Bm25Index.rank()` method is called
|
||||
- **THEN** `"CI/CD Pipeline"` receives a score > 0
|
||||
- **AND** `"Progressive Web App (PWA)"` receives score 0.0
|
||||
|
||||
### Requirement: Reciprocal Rank Fusion normalizes BM25 ranks into scores in [0, 1]
|
||||
|
||||
The `reciprocal_rank_fusion()` function SHALL fuse ranked lists into a normalized score
|
||||
dict. Candidates with zero BM25 score in all contributing lists SHALL receive a fused
|
||||
score of 0.0. The top-ranked candidate SHALL receive score 1.0.
|
||||
|
||||
#### Scenario: Single-list fusion normalizes top candidate to 1.0
|
||||
|
||||
- **GIVEN** a single ranked list `[("Python", 3.5), ("Python (advanced)", 2.1)]`
|
||||
- **WHEN** `reciprocal_rank_fusion([ranked_list], k=60)` is called
|
||||
- **THEN** `"Python"` receives normalized score `1.0`
|
||||
- **AND** `"Python (advanced)"` receives a score in `(0, 1)`
|
||||
|
||||
#### Scenario: All-zero BM25 list produces empty result
|
||||
|
||||
- **GIVEN** a ranked list `[("JavaScript", 0.0), ("TypeScript", 0.0)]`
|
||||
- **WHEN** `reciprocal_rank_fusion([ranked_list])` is called
|
||||
- **THEN** the function returns an empty dict `{}`
|
||||
- **AND** the caller maps this to `score = 0.0` and `best_match = None`
|
||||
|
||||
### Requirement: Global BM25 index is built once over all filtered candidates
|
||||
|
||||
When `use_bm25_search = true`, the `Matcher` SHALL build one `Bm25Index` over the
|
||||
deduplicated union of all filtered candidates' competences before the per-candidate
|
||||
scoring loop. The same index SHALL be passed into every `compute_competence_similarity()`
|
||||
call and results SHALL be filtered to the current candidate's competence set. This ensures
|
||||
BM25 IDF values reflect the full candidate pool rather than a single person's small corpus.
|
||||
|
||||
#### Scenario: Candidate with unique skill scores above zero in global index
|
||||
|
||||
- **GIVEN** `use_bm25_search = true` in `config.toml`
|
||||
- **AND** two candidates: one with `["Python"]` only, another with 6 unrelated skills
|
||||
- **AND** the task requires `["Python"]`
|
||||
- **WHEN** the system computes `competence_score` for each candidate
|
||||
- **THEN** the candidate with `["Python"]` receives `competence_score > 0.0`
|
||||
- **AND** the candidate with no Python token receives `competence_score == 0.0`
|
||||
|
||||
#### Scenario: Ubiquitous term present in every candidate scores 0.0
|
||||
|
||||
- **GIVEN** a global corpus where a token appears in every candidate's competences
|
||||
- **WHEN** `Bm25Index.rank()` is called with that token as the query
|
||||
- **THEN** the raw BM25 score is clamped to `max(0.0, raw)` (IDF is negative → score is 0.0)
|
||||
- **AND** no exception is raised
|
||||
|
||||
#### Scenario: Global index filters results to current candidate's competence set
|
||||
|
||||
- **GIVEN** a global index built over candidates A (`["Python", "Django"]`) and B (`["Java", "Spring"]`)
|
||||
- **AND** the system scores candidate A for required `["Python"]`
|
||||
- **WHEN** `_bm25_rrf_similarity()` is called with candidate A's competence set
|
||||
- **THEN** only `"Python"` and `"Django"` appear in the filtered results
|
||||
- **AND** `"Java"` and `"Spring"` do NOT appear in candidate A's result
|
||||
|
||||
### Requirement: Optional LLM-based Auto-Tagging expands candidate competences before BM25
|
||||
|
||||
When `use_auto_tagging = true` (and `use_bm25_search = true`), the system SHALL call the
|
||||
`AutoTagger` before each BM25 scoring call to identify which required competences are
|
||||
already covered by the candidate's existing entries via synonym, abbreviation, or
|
||||
cross-language equivalence. The LLM returns canonical required-competence names as
|
||||
additions. The expanded list is ephemeral — it SHALL be used only for the current BM25
|
||||
scoring call and SHALL NOT be written back to any database or persisted between calls.
|
||||
|
||||
#### Scenario: Auto-tagging bridges synonym gap so candidate matches required competence
|
||||
|
||||
- **GIVEN** `use_bm25_search = true` and `use_auto_tagging = true` in `config.toml`
|
||||
- **AND** a task requires competences `["Machine Learning"]`
|
||||
- **AND** a candidate has competences `["ML", "Python"]`
|
||||
- **AND** the LLM identifies `"ML"` as covering `"Machine Learning"`
|
||||
- **WHEN** the system computes `competence_score`
|
||||
- **THEN** `competence_score > 0.0` (auto-tagging added `"Machine Learning"` to working list)
|
||||
- **AND** the candidate's stored competences remain `["ML", "Python"]` (not mutated)
|
||||
|
||||
#### Scenario: Auto-tagging bridges cross-language gap
|
||||
|
||||
- **GIVEN** `use_bm25_search = true` and `use_auto_tagging = true` in `config.toml`
|
||||
- **AND** a task requires competences `["Software Architecture"]`
|
||||
- **AND** a candidate has competences `["Softwarearchitektur"]`
|
||||
- **AND** the LLM identifies `"Softwarearchitektur"` as covering `"Software Architecture"`
|
||||
- **WHEN** the system computes `competence_score`
|
||||
- **THEN** `competence_score > 0.0`
|
||||
|
||||
#### Scenario: Auto-tagging graceful degradation on LLM failure
|
||||
|
||||
- **GIVEN** `use_bm25_search = true` and `use_auto_tagging = true` in `config.toml`
|
||||
- **AND** the LLM call raises an exception (timeout, API error, or malformed JSON)
|
||||
- **WHEN** the system computes `competence_score`
|
||||
- **THEN** no exception is propagated to the caller
|
||||
- **AND** BM25 scoring continues on the original (unexpanded) candidate list
|
||||
- **AND** a warning is logged
|
||||
|
||||
#### Scenario: Auto-tagging LLM cannot invent skills not in required list
|
||||
|
||||
- **GIVEN** `use_auto_tagging = true`
|
||||
- **AND** the LLM response includes an addition that is NOT in the `required` list
|
||||
- **WHEN** `AutoTagger.expand_competences()` processes the response
|
||||
- **THEN** the hallucinated addition is silently dropped
|
||||
- **AND** only valid required-competence names are appended to the working list
|
||||
|
||||
#### Scenario: Auto-tagging is no-op when use_auto_tagging is false
|
||||
|
||||
- **GIVEN** `use_bm25_search = true` and `use_auto_tagging = false` (default) in `config.toml`
|
||||
- **WHEN** the system computes `competence_score`
|
||||
- **THEN** no LLM call is made
|
||||
- **AND** BM25 runs on the raw candidate competence list without expansion
|
||||
|
||||
### Requirement: `use_auto_tagging` configuration key with default false
|
||||
|
||||
`config.toml` SHALL support a `matching.similarity.use_auto_tagging` boolean key.
|
||||
The default value SHALL be `false`. The key SHALL be documented in `config.toml.example`.
|
||||
`use_auto_tagging = true` is only effective when `use_bm25_search = true`.
|
||||
|
||||
#### Scenario: Config key defaults to false when omitted
|
||||
|
||||
- **GIVEN** a `config.toml` that does not contain `use_auto_tagging`
|
||||
- **WHEN** the server starts and loads configuration
|
||||
- **THEN** `SimilarityConfig.use_auto_tagging == False`
|
||||
- **AND** no `AutoTagger` is constructed
|
||||
- **AND** no LLM client is constructed for auto-tagging
|
||||
|
||||
#### Scenario: Config key enables auto-tagging when set to true
|
||||
|
||||
- **GIVEN** `config.toml` contains `use_auto_tagging = true` under `[matching.similarity]`
|
||||
- **AND** `chat_deployment` is set and `AZURE_OPENAI_LLM_API_KEY` is present
|
||||
- **WHEN** the server starts and loads configuration
|
||||
- **THEN** `SimilarityConfig.use_auto_tagging == True`
|
||||
- **AND** an `AutoTagger` is constructed and passed to `SimilarityEngine`
|
||||
|
||||
### Requirement: `azure_openai.chat_deployment` configuration key for auto-tagging
|
||||
|
||||
`config.toml` SHALL support an `azure_openai.chat_deployment` string key naming the Azure
|
||||
OpenAI chat model deployment used by the `AutoTagger`. This key is required when
|
||||
`use_auto_tagging = true`. The `AZURE_OPENAI_LLM_API_KEY` environment variable SHALL
|
||||
supply the API key for that deployment. The server SHALL raise a `ConfigurationError` at
|
||||
startup if `use_auto_tagging = true` and `AZURE_OPENAI_LLM_API_KEY` is absent.
|
||||
|
||||
#### Scenario: Server raises ConfigurationError when API key missing with auto-tagging enabled
|
||||
|
||||
- **GIVEN** `use_auto_tagging = true` in `config.toml`
|
||||
- **AND** the environment variable `AZURE_OPENAI_LLM_API_KEY` is not set
|
||||
- **WHEN** the server starts and loads configuration
|
||||
- **THEN** a `ConfigurationError` is raised before any MCP tool is registered
|
||||
- **AND** the error message indicates the missing environment variable
|
||||
|
||||
#### Scenario: chat_deployment key is read from config when auto-tagging enabled
|
||||
|
||||
- **GIVEN** `config.toml` contains `chat_deployment = "gpt-4.1"` under `[azure_openai]`
|
||||
- **AND** `use_auto_tagging = true` and `AZURE_OPENAI_LLM_API_KEY` is set
|
||||
- **WHEN** the server starts
|
||||
- **THEN** the `AzureOpenAIClient` for auto-tagging is initialized with deployment `"gpt-4.1"`
|
||||
|
||||
### Requirement: `use_bm25_search` configuration key with default false
|
||||
|
||||
`config.toml` SHALL support a `matching.similarity.use_bm25_search` boolean key.
|
||||
The default value SHALL be `false`. The key SHALL be documented in `config.toml.example`.
|
||||
|
||||
#### Scenario: Config key defaults to false when omitted
|
||||
|
||||
- **GIVEN** a `config.toml` that does not contain `use_bm25_search`
|
||||
- **WHEN** the server starts and loads configuration
|
||||
- **THEN** `SimilarityConfig.use_bm25_search == False`
|
||||
- **AND** the embedding-based strategy is used unchanged
|
||||
|
||||
#### Scenario: Config key enables BM25 mode when set to true
|
||||
|
||||
- **GIVEN** `config.toml` contains `use_bm25_search = true` under `[matching.similarity]`
|
||||
- **WHEN** the server starts and loads configuration
|
||||
- **THEN** `SimilarityConfig.use_bm25_search == True`
|
||||
- **AND** `compute_competence_similarity()` delegates to `_bm25_rrf_similarity()`
|
||||
|
||||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: Role similarity is always embedding-based regardless of use_bm25_search
|
||||
|
||||
Role similarity computation SHALL remain embedding-based even when `use_bm25_search = true`.
|
||||
The BM25 strategy applies only to competence similarity.
|
||||
|
||||
#### Scenario: Role scoring unaffected by use_bm25_search flag
|
||||
|
||||
- **GIVEN** `use_bm25_search = true` in `config.toml`
|
||||
- **AND** a task requiring role "Backend Developer"
|
||||
- **AND** a capacity with role "Backend Developer"
|
||||
- **WHEN** the system computes `role_score`
|
||||
- **THEN** `role_score` is computed via embedding similarity (unchanged behavior)
|
||||
- **AND** embedding cache and Azure OpenAI calls for role embeddings proceed as before
|
||||
|
||||
## REMOVED Requirements
|
||||
|
||||
### Requirement: Legacy debug scripts are removed from the repository root
|
||||
|
||||
The files `debug_similarity_analysis.py`, `debug_categorization.py`,
|
||||
`SEMANTIC_SIMILARITY_ANALYSIS.md`, and `SCORE_BUG_FIX.md` SHALL be removed from the
|
||||
repository root. They are superseded by the test suite and `docs/semantic_similarity_tradeoffs.md`.
|
||||
|
||||
#### Scenario: Legacy files do not exist after the change is applied
|
||||
|
||||
- **GIVEN** the change `add-bm25-rrf-competence-matching` is fully applied
|
||||
- **WHEN** the repository root is listed
|
||||
- **THEN** `debug_similarity_analysis.py` does not exist
|
||||
- **AND** `debug_categorization.py` does not exist
|
||||
- **AND** `SEMANTIC_SIMILARITY_ANALYSIS.md` does not exist
|
||||
- **AND** `SCORE_BUG_FIX.md` does not exist
|
||||
@@ -0,0 +1,586 @@
|
||||
# Implementation Tasks: BM25 + RRF Competence Matching
|
||||
|
||||
## Status: Implemented
|
||||
|
||||
## Summary
|
||||
|
||||
This change introduces an optional BM25-based competence matching strategy with optional
|
||||
LLM-based Auto-Tagging as an alternative to the existing embedding-based approach:
|
||||
|
||||
1. **Restore LLM client**: `AzureOpenAIClient.chat_completion()` restored; `AzureOpenAIConfig.chat_deployment` + `llm_api_key` added.
|
||||
2. **New `auto_tagger.py` module**: `AutoTagger` class with `expand_competences()` — ephemeral LLM-driven competence expansion.
|
||||
3. **New `bm25.py` module**: `Bm25Index` dataclass + `bm25_rank_competences()` convenience function.
|
||||
4. **New `rrf.py` module**: `reciprocal_rank_fusion()` fuses ranked lists into a normalized score dict.
|
||||
5. **`similarity.py` extended**: `_bm25_rrf_similarity()` internal method + auto-tag branch + flag routing in `compute_competence_similarity()`.
|
||||
6. **`SimilarityConfig` extended**: new `use_bm25_search: bool = False` and `use_auto_tagging: bool = False` fields.
|
||||
7. **`config.py` parser extended**: reads `use_bm25_search`, `use_auto_tagging`, `chat_deployment`, and `AZURE_OPENAI_LLM_API_KEY`.
|
||||
8. **`config.toml.example` updated**: documents all new keys.
|
||||
9. **`mcp_server.py` updated**: constructs `AutoTagger` when `use_auto_tagging=True`; passes flags to `SimilarityEngine`.
|
||||
10. **Legacy debug files removed**: 4 root-level files no longer part of the module.
|
||||
11. **Tests added**: `tests/test_bm25.py`, `tests/test_rrf.py`, `tests/test_auto_tagger.py`, extensions to `tests/test_similarity.py` and `tests/test_matcher_integration.py`.
|
||||
12. **Documentation updated**: architecture, tradeoffs doc, assistant prompt, README.
|
||||
|
||||
## Clarifications / Decisions
|
||||
|
||||
- **BM25 scope**: BM25 + RRF replaces **competence** similarity only. Role similarity is
|
||||
always embedding-based, regardless of `use_bm25_search`.
|
||||
- **Auto-tagging scope**: `use_auto_tagging` only applies when `use_bm25_search = true`.
|
||||
When `use_auto_tagging = true`, `AutoTagger.expand_competences()` is called once per
|
||||
candidate per `compute_competence_similarity()` invocation. Result is ephemeral.
|
||||
- **Auto-tagging works in both directions**: both `find_matching_capacities` and
|
||||
`find_matching_tasks` pass through `compute_competence_similarity()` and benefit.
|
||||
- **Graceful degradation**: if the LLM call in `AutoTagger` fails for any reason, the
|
||||
original candidate list is used unchanged. BM25 scoring continues.
|
||||
- **LLM client restored**: `AzureOpenAIClient.chat_completion()` is re-added, scoped
|
||||
exclusively to auto-tagging. The embedding-related API surface is unchanged.
|
||||
- **Zero-out rule**: If all BM25 scores for a required competence are 0.0 (no token
|
||||
overlap at all), the RRF score for that required competence is also 0.0.
|
||||
- **Global index** *(revised)*: Originally the `Bm25Index` was to be built fresh per
|
||||
`compute_competence_similarity()` call. This was identified as an IDF pathology: with
|
||||
only one person's competences as the corpus, BM25 IDF is always 0 or negative for any
|
||||
term that appears in every document (and with N=1 all terms do). The fix (Phase 9) builds
|
||||
one `Bm25Index` over the union of all filtered candidates' competences before the loop
|
||||
and passes it in as `global_index`.
|
||||
- **Single ranked list**: Initial implementation fuses one BM25 ranked list per required
|
||||
competence. The `reciprocal_rank_fusion()` signature accepts multiple lists for future
|
||||
hybrid extension.
|
||||
- **Output shape unchanged**: `_bm25_rrf_similarity()` returns the same
|
||||
`dict[str, dict[str, object]]` shape as `_per_skill_similarity()`. Matcher, scorer,
|
||||
and all output formatting remain unchanged.
|
||||
- **`rank-bm25` dependency**: Added to `pyproject.toml` `[project.dependencies]`.
|
||||
- **All new code uses Google Docstrings** per project convention.
|
||||
- **Default `false`**: Existing deployments are unaffected until opt-in.
|
||||
|
||||
## Estimated Effort
|
||||
|
||||
- **Phase 0 (Restore LLM client)**: 1 hour
|
||||
- **Phase 1 (New modules — bm25, rrf, auto_tagger)**: 3–4 hours
|
||||
- **Phase 2 (Config + wiring)**: 1–1.5 hours
|
||||
- **Phase 3 (Similarity integration)**: 2 hours
|
||||
- **Phase 4 (Legacy cleanup)**: 0.5 hour
|
||||
- **Phase 5 (Tests)**: 4–5 hours
|
||||
- **Phase 6 (Documentation)**: 2 hours
|
||||
- **Phase 7 (Dependency)**: 0.5 hour
|
||||
- **Phase 8 (Quality gates)**: 0.5 hour
|
||||
- **Total**: 15–17 hours
|
||||
|
||||
---
|
||||
|
||||
## Phase 0: Restore LLM Client
|
||||
|
||||
### Task 0.1: Extend `AzureOpenAIConfig`
|
||||
**File:** `src/teamlandkarte_mcp/config.py`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Add `chat_deployment: str = ""` field to `AzureOpenAIConfig` dataclass
|
||||
- [x] Add Google Docstring field comment:
|
||||
`"""Azure deployment name for the chat model used in auto-tagging (e.g. 'gpt-4.1'). Required when use_auto_tagging = true."""`
|
||||
- [x] Update the TOML parser to read `azure_openai.chat_deployment` with default `""`
|
||||
- [x] Read `AZURE_OPENAI_LLM_API_KEY` from environment (store as `llm_api_key: str = ""` on `AzureOpenAIConfig`)
|
||||
- [x] When `use_auto_tagging = true` and `llm_api_key` is empty, raise `ConfigurationError` at startup
|
||||
|
||||
**Acceptance:**
|
||||
- `AzureOpenAIConfig()` instantiates with `chat_deployment=""` and `llm_api_key=""`
|
||||
- TOML `azure_openai.chat_deployment = "gpt-4.1"` is read correctly
|
||||
- Missing `AZURE_OPENAI_LLM_API_KEY` with `use_auto_tagging=true` raises at startup
|
||||
|
||||
---
|
||||
|
||||
### Task 0.2: Add `chat_completion()` to `AzureOpenAIClient`
|
||||
**File:** `src/teamlandkarte_mcp/azure/openai_client.py`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Add `chat_deployment: str` and `llm_api_key: str` constructor parameters
|
||||
- [x] Implement `chat_completion(system: str, user: str) -> str` method:
|
||||
- Calls Azure OpenAI chat completions API with `response_format={"type": "json_object"}`
|
||||
- Uses `self.chat_deployment` as the model/deployment
|
||||
- Uses `self.llm_api_key` for auth (separate from embedding key)
|
||||
- Returns the raw JSON string from the first choice's message content
|
||||
- Raises `AzureOpenAIError` on API errors (consistent with existing error handling)
|
||||
- [x] Add Google Docstring to `chat_completion()`
|
||||
|
||||
**Acceptance:**
|
||||
- `chat_completion(system="...", user="...")` returns a JSON string
|
||||
- API errors surface as `AzureOpenAIError`
|
||||
- `chat_deployment=""` + `chat_completion()` call raises `ConfigurationError` (misconfigured guard)
|
||||
|
||||
---
|
||||
|
||||
## Phase 1: New Modules
|
||||
|
||||
### Task 1.1: Add `bm25.py`
|
||||
**File:** `src/teamlandkarte_mcp/matching/bm25.py` (**new**)
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Create `bm25.py` with `_tokenize(text: str) -> list[str]` helper (lowercase, split on `[\W_]+`, filter empty)
|
||||
- [x] Implement `Bm25Index` dataclass:
|
||||
- `corpus: list[str]` field
|
||||
- Internal `_bm25: BM25Okapi | None` built in `__post_init__`
|
||||
- `rank(query: str) -> list[tuple[str, float]]` method:
|
||||
- returns `[]` for empty corpus
|
||||
- returns `(corpus_text, bm25_score)` list sorted by score descending
|
||||
- score 0.0 entries are included (caller decides cutoff)
|
||||
- [x] Implement `bm25_rank_competences(required: str, candidates: list[str]) -> list[tuple[str, float]]` convenience function
|
||||
- [x] Add Google Docstrings to all public symbols
|
||||
|
||||
**Acceptance:**
|
||||
- `Bm25Index(corpus=[]).rank("Python")` → `[]`
|
||||
- `Bm25Index(corpus=["Python"]).rank("Python")` → `[("Python", score > 0)]`
|
||||
- `Bm25Index(corpus=["JavaScript"]).rank("Python")` → `[("JavaScript", 0.0)]`
|
||||
- `Bm25Index(corpus=["Machine Learning"]).rank("machine learning")` → score > 0 (case-insensitive)
|
||||
|
||||
---
|
||||
|
||||
### Task 1.2: Add `rrf.py`
|
||||
**File:** `src/teamlandkarte_mcp/matching/rrf.py` (**new**)
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Implement `reciprocal_rank_fusion(ranked_lists, *, k=60) -> dict[str, float]`:
|
||||
- Skip any list where all scores are 0.0 (no token-overlap signal)
|
||||
- For each candidate in each non-zero list, accumulate `1 / (k + rank)` (1-based rank)
|
||||
- Normalize so the highest-scoring candidate maps to 1.0
|
||||
- Return `{}` if all lists are empty or all-zero
|
||||
- [x] Add Google Docstring including mathematical formula, zero-out rule, and normalization description
|
||||
|
||||
**Acceptance:**
|
||||
- Single list, one candidate → `{"A": 1.0}`
|
||||
- Single list `[("A", 1.0), ("B", 0.5)]` → `{"A": 1.0, "B": ...}` with `0 < B < 1`
|
||||
- Single all-zero list `[("A", 0.0), ("B", 0.0)]` → `{}` (empty, no signal)
|
||||
- `k=60` default: rank-1 score = `1/61`, rank-2 score = `1/62`, normalized → `61/62 ≈ 0.984`
|
||||
|
||||
---
|
||||
|
||||
### Task 1.3: Add `auto_tagger.py`
|
||||
**File:** `src/teamlandkarte_mcp/matching/auto_tagger.py` (**new**)
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Implement `AutoTagger` class:
|
||||
- Constructor: `__init__(self, client: AzureOpenAIClient) -> None`
|
||||
- Stores the client reference
|
||||
- [x] Implement `expand_competences(self, required: list[str], existing: list[str]) -> list[str]`:
|
||||
- Build system prompt: instructs LLM to return only required-competence names already
|
||||
covered by existing entries (synonym/abbreviation/cross-language), as JSON
|
||||
`{"additions": ["...", ...]}`
|
||||
- Build user prompt: `REQUIRED: {required}\nEXISTING: {existing}`
|
||||
- Call `self._client.chat_completion(system=..., user=...)` → JSON string
|
||||
- Parse JSON, extract `additions` list
|
||||
- Validate: keep only items present in `required` (guard against LLM hallucination)
|
||||
- Return `existing + [a for a in additions if a not in existing]`
|
||||
- On any exception (API error, JSON parse, key error): log warning, return `existing` unchanged
|
||||
- [x] Add Google Docstrings to class and all public methods
|
||||
|
||||
**Acceptance:**
|
||||
- LLM returns `{"additions": ["Machine Learning"]}` for existing `["ML"]`, required `["Machine Learning"]`
|
||||
→ returns `["ML", "Machine Learning"]`
|
||||
- LLM returns `{"additions": []}` → returns `existing` unchanged
|
||||
- LLM raises exception → returns `existing` unchanged (no exception propagation)
|
||||
- Addition not in `required` is silently dropped (hallucination guard)
|
||||
- No duplicate entries if `existing` already contains the addition
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Configuration Extension
|
||||
|
||||
### Task 2.1: Extend `SimilarityConfig`
|
||||
**File:** `src/teamlandkarte_mcp/config.py`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Add `use_bm25_search: bool = False` field to `SimilarityConfig` dataclass
|
||||
- [x] Add `use_auto_tagging: bool = False` field to `SimilarityConfig` dataclass
|
||||
- [x] Add Google Docstring comments to both fields
|
||||
- [x] Update the TOML parser to read `matching.similarity.use_bm25_search` (default `False`)
|
||||
- [x] Update the TOML parser to read `matching.similarity.use_auto_tagging` (default `False`)
|
||||
|
||||
**Acceptance:**
|
||||
- `SimilarityConfig()` instantiates with both flags `False`
|
||||
- `SimilarityConfig(use_bm25_search=True, use_auto_tagging=True)` works
|
||||
- Parser reads both flags from TOML correctly
|
||||
|
||||
---
|
||||
|
||||
### Task 2.2: Update `config.toml.example`
|
||||
**File:** `config.toml.example`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Add `use_bm25_search = false` under `[matching.similarity]` with inline comment
|
||||
- [x] Add `use_auto_tagging = false` under `[matching.similarity]` with inline comment:
|
||||
```toml
|
||||
# When true, BM25 + RRF replaces embedding-based competence similarity.
|
||||
# BM25 assigns 0 to candidates with no token overlap, eliminating false positives.
|
||||
# Role similarity is always embedding-based. Default: false.
|
||||
use_bm25_search = false
|
||||
|
||||
# When true (and use_bm25_search = true), an LLM pre-expands each candidate's
|
||||
# competence list with canonical equivalents of required competences it already covers
|
||||
# (synonyms, abbreviations, cross-language). The expansion is ephemeral. Default: false.
|
||||
use_auto_tagging = false
|
||||
```
|
||||
- [x] Add `chat_deployment = ""` under `[azure_openai]` with inline comment:
|
||||
```toml
|
||||
# Azure deployment name for the chat model used in auto-tagging (e.g. "gpt-4.1").
|
||||
# Required when use_auto_tagging = true.
|
||||
chat_deployment = ""
|
||||
```
|
||||
|
||||
**Acceptance:**
|
||||
- Config example is self-documenting for all three new keys
|
||||
- All new defaults are `false` / `""`
|
||||
|
||||
---
|
||||
|
||||
### Task 2.3: Wire `use_bm25_search`, `use_auto_tagging`, and `AutoTagger` in `mcp_server.py`
|
||||
**File:** `src/teamlandkarte_mcp/mcp_server.py`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] When `use_auto_tagging=True`: construct `AzureOpenAIClient` with `chat_deployment`
|
||||
and `llm_api_key`; construct `AutoTagger(client=llm_client)`
|
||||
- [x] Pass `auto_tagger` (or `None`) and both flags to `SimilarityEngine` constructor
|
||||
- [x] When `use_auto_tagging=False`: `auto_tagger=None` — no LLM client constructed
|
||||
|
||||
**Acceptance:**
|
||||
- Setting `use_auto_tagging = true` in `config.toml` activates auto-tagging at runtime
|
||||
- Setting `use_auto_tagging = false` (default) does not construct any LLM client
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: Similarity Engine Integration
|
||||
|
||||
### Task 3.1: Add `_bm25_rrf_similarity()` to `SimilarityEngine`
|
||||
**File:** `src/teamlandkarte_mcp/matching/similarity.py`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Import `bm25_rank_competences` from `.bm25`, `reciprocal_rank_fusion` from `.rrf`,
|
||||
and `AutoTagger` from `.auto_tagger`
|
||||
- [x] Implement `_bm25_rrf_similarity(self, required: list[str], candidate: list[str]) -> dict[str, dict[str, object]]`:
|
||||
- For each `req` in `required`:
|
||||
1. Call `bm25_rank_competences(req, candidate)` → `ranked`
|
||||
2. Call `reciprocal_rank_fusion([ranked])` → `fused: dict[str, float]`
|
||||
3. If `fused` is empty or all scores are 0.0:
|
||||
- `score = 0.0`, `best_match = None`, `rationale = "BM25: no token overlap with any candidate competence."`
|
||||
4. Otherwise:
|
||||
- `best_match = max(fused, key=fused.get)`
|
||||
- `score = fused[best_match]`
|
||||
- `rationale = f"BM25+RRF: best match '{best_match}' (score {score:.3f})."`
|
||||
- Returns dict shaped identically to `_per_skill_similarity()` output
|
||||
- [x] Add Google Docstring to `_bm25_rrf_similarity()`
|
||||
|
||||
---
|
||||
|
||||
### Task 3.2: Route `compute_competence_similarity()` by flags
|
||||
**File:** `src/teamlandkarte_mcp/matching/similarity.py`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Ensure `SimilarityEngine.__init__` accepts and stores `use_bm25_search`,
|
||||
`use_auto_tagging`, and `auto_tagger: AutoTagger | None = None`
|
||||
- [x] In `compute_competence_similarity()`:
|
||||
```python
|
||||
if self.config.use_bm25_search:
|
||||
working = candidate
|
||||
if self.config.use_auto_tagging and self._auto_tagger is not None:
|
||||
working = self._auto_tagger.expand_competences(required, candidate)
|
||||
return self._bm25_rrf_similarity(required, working)
|
||||
# else: existing strategy dispatch (unchanged)
|
||||
```
|
||||
|
||||
**Acceptance:**
|
||||
- `use_bm25_search=False`: behavior bit-for-bit identical to before
|
||||
- `use_bm25_search=True, use_auto_tagging=False`: BM25 on raw candidate list
|
||||
- `use_bm25_search=True, use_auto_tagging=True`: BM25 on LLM-expanded list
|
||||
- `use_bm25_search=True` + required=`["Python"]`, candidate=`["JavaScript"]`: `score == 0.0`
|
||||
- `use_bm25_search=True` + required=`["Python"]`, candidate=`["Python"]`: `score == 1.0`
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: Legacy Cleanup
|
||||
|
||||
### Task 4.1: Remove legacy debug files
|
||||
**Files to delete:**
|
||||
- `debug_similarity_analysis.py`
|
||||
- `debug_categorization.py`
|
||||
- `SEMANTIC_SIMILARITY_ANALYSIS.md`
|
||||
- `SCORE_BUG_FIX.md`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Verify none of these files are imported or referenced in source code or tests
|
||||
- [x] Delete `debug_similarity_analysis.py`
|
||||
- [x] Delete `debug_categorization.py`
|
||||
- [x] Delete `SEMANTIC_SIMILARITY_ANALYSIS.md`
|
||||
- [x] Delete `SCORE_BUG_FIX.md`
|
||||
|
||||
**Acceptance:**
|
||||
- `git status` shows 4 deleted files
|
||||
- No import errors in existing code
|
||||
|
||||
---
|
||||
|
||||
## Phase 5: Tests
|
||||
|
||||
### Task 5.1: Unit tests for `bm25.py`
|
||||
**File:** `tests/test_bm25.py` (**new**)
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Test `Bm25Index` with empty corpus → `rank()` returns `[]`
|
||||
- [x] Test exact match (single token) → score > 0
|
||||
- [x] Test no token overlap (query "Python", corpus "JavaScript") → score == 0.0
|
||||
- [x] Test partial token overlap multi-token query ("Machine Learning") against ("Deep Learning") → partial score > 0
|
||||
- [x] Test case-insensitivity: `rank("Python")` matches "python" in corpus
|
||||
- [x] Test tokenization edge cases:
|
||||
- "Progressive Web App (PWA)" → `["progressive", "web", "app", "pwa"]`
|
||||
- "CI/CD Pipeline" → `["ci", "cd", "pipeline"]`
|
||||
- "React.js" → `["react", "js"]`
|
||||
- [x] Test `bm25_rank_competences()` convenience wrapper produces same result as `Bm25Index.rank()`
|
||||
- [x] Add Google Docstrings to test module
|
||||
|
||||
### Task 5.2: Unit tests for `rrf.py`
|
||||
**File:** `tests/test_rrf.py` (**new**)
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Test single list, single candidate → `{"A": 1.0}`
|
||||
- [x] Test single list multiple candidates: top candidate maps to 1.0, others < 1.0 and > 0.0
|
||||
- [x] Test all-zero list → returns `{}` (empty)
|
||||
- [x] Test mixed: one non-zero list + one all-zero list → only non-zero list contributes
|
||||
- [x] Test `k` parameter: higher `k` flattens differences (rank-1 and rank-2 scores closer together)
|
||||
- [x] Test score ordering is preserved (rank 1 > rank 2 > rank 3 in output)
|
||||
- [x] Add Google Docstrings to test module
|
||||
|
||||
### Task 5.3: Unit tests for `auto_tagger.py`
|
||||
**File:** `tests/test_auto_tagger.py` (**new**)
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Mock `AzureOpenAIClient.chat_completion()` for all tests (no real LLM calls)
|
||||
- [x] Test: LLM returns `{"additions": ["Machine Learning"]}` for existing `["ML"]`,
|
||||
required `["Machine Learning"]` → result is `["ML", "Machine Learning"]`
|
||||
- [x] Test: LLM returns `{"additions": []}` → result is identical to `existing`
|
||||
- [x] Test: LLM `chat_completion` raises exception → result is identical to `existing`,
|
||||
no exception propagated
|
||||
- [x] Test: LLM returns malformed JSON (not parseable) → result is `existing` unchanged
|
||||
- [x] Test: LLM returns addition not in `required` (hallucination) → addition is dropped
|
||||
- [x] Test: LLM returns addition already in `existing` → no duplicate in result
|
||||
- [x] Test: empty `required` list → `expand_competences()` returns `existing` unchanged
|
||||
(no LLM call needed)
|
||||
- [x] Add Google Docstrings to test module
|
||||
|
||||
### Task 5.4: Extend `tests/test_similarity.py`
|
||||
**File:** `tests/test_similarity.py` (extended)
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Add tests with `use_bm25_search=True`:
|
||||
- Required `["Python"]`, candidate `["JavaScript"]` → `competence_score == 0.0`
|
||||
- Required `["Python"]`, candidate `["Python"]` → `competence_score == 1.0`
|
||||
- Required `["Python", "Machine Learning"]`, candidate `["JavaScript", "TypeScript"]` → both individual scores 0.0, overall `competence_score == 0.0`
|
||||
- Required `["Python", "Machine Learning"]`, candidate `["Python", "Machine Learning"]` → both scores 1.0
|
||||
- [x] Add tests with `use_bm25_search=True, use_auto_tagging=True` (mocked `AutoTagger`):
|
||||
- Mocked `AutoTagger.expand_competences` injects `"Machine Learning"` into `["ML"]`
|
||||
- Required `["Machine Learning"]`, candidate `["ML"]` (after expansion) → `score > 0`
|
||||
- [x] Confirm that with `use_bm25_search=False`, behavior is unchanged (existing tests still pass)
|
||||
|
||||
### Task 5.5: Extend `tests/test_matcher_integration.py`
|
||||
**File:** `tests/test_matcher_integration.py` (extended)
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Add integration test with `use_bm25_search=True`:
|
||||
- Task requires: `["Python", "Machine Learning"]`
|
||||
- Candidate A has: `["Python", "Machine Learning", "Pandas"]` → high score
|
||||
- Candidate B has: `["JavaScript", "TypeScript", "Node.js", "Vue.js"]` → score 0.0
|
||||
- Assert: candidate A `competence_score` > candidate B `competence_score`
|
||||
- Assert: candidate B `competence_score == 0.0` (false-positive eliminated)
|
||||
- [x] Add integration test with `use_bm25_search=True, use_auto_tagging=True`
|
||||
(mocked LLM via mocked `AutoTagger`):
|
||||
- Candidate has `["ML", "Python"]`; auto-tag adds `"Machine Learning"` to working list
|
||||
- Required `["Machine Learning", "Python"]` → both scores > 0
|
||||
- [x] These tests explicitly document the false-positive scenario from production
|
||||
|
||||
---
|
||||
|
||||
## Phase 6: Documentation
|
||||
|
||||
### Task 6.1: Update `docs/architecture.md`
|
||||
**File:** `docs/architecture.md`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Add section "BM25 + RRF + Auto-Tagging Competence Matching" under the matching/similarity section:
|
||||
- Describe the three new modules (`auto_tagger.py`, `bm25.py`, `rrf.py`)
|
||||
- Describe the auto-tag → BM25 → RRF pipeline and routing in `compute_competence_similarity()`
|
||||
- Note that role similarity is unaffected
|
||||
- Reference `use_bm25_search` and `use_auto_tagging` config keys
|
||||
|
||||
### Task 6.2: Update `docs/semantic_similarity_tradeoffs.md`
|
||||
**File:** `docs/semantic_similarity_tradeoffs.md`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Add BM25+RRF row to the strategy comparison table:
|
||||
- Strengths: exact/lexical match, no false positives for non-overlapping skills
|
||||
- Weaknesses: misses synonyms and cross-language variants (mitigated by `use_auto_tagging`)
|
||||
- When to use: when skill names in the dataset are mostly canonical English terms
|
||||
- [x] Add BM25+RRF+AutoTag row:
|
||||
- Strengths: exact match + synonym/abbreviation/cross-language bridging via LLM
|
||||
- Weaknesses: LLM latency per candidate, requires `AZURE_OPENAI_LLM_API_KEY`
|
||||
- When to use: mixed-language or abbreviation-heavy skill datasets
|
||||
|
||||
### Task 6.3: Update `docs/assistant_system_prompt.md`
|
||||
**File:** `docs/assistant_system_prompt.md`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Add note on score interpretation when `use_bm25_search=true`:
|
||||
- A `competence_score` of 0.0 means **no lexical token overlap** between required and candidate competences
|
||||
- The score is not a semantic similarity — it is a lexical matching score
|
||||
- Users should be informed when BM25 mode is active so they understand why cross-language synonyms may not match
|
||||
- [x] Add note on `use_auto_tagging=true`:
|
||||
- Auto-tagging bridges common synonym/abbreviation gaps (e.g. "ML" → "Machine Learning")
|
||||
- A positive `competence_score` with auto-tagging enabled may reflect an LLM-inferred equivalence,
|
||||
not a literal token match; the rationale field will say "BM25+RRF" regardless
|
||||
|
||||
### Task 6.4: Update `README.md`
|
||||
**File:** `README.md`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Add all three new config keys to the config reference section:
|
||||
- `matching.similarity.use_bm25_search` (boolean, default `false`): Enables BM25+RRF lexical competence matching
|
||||
- `matching.similarity.use_auto_tagging` (boolean, default `false`): Enables LLM pre-expansion before BM25
|
||||
- `azure_openai.chat_deployment` (string, default `""`): Chat model deployment name for auto-tagging
|
||||
- [x] Document `AZURE_OPENAI_LLM_API_KEY` environment variable in the env-vars section
|
||||
|
||||
---
|
||||
|
||||
## Phase 7: Dependency
|
||||
|
||||
### Task 7.1: Add `rank-bm25` to project dependencies
|
||||
**File:** `pyproject.toml`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Add `rank-bm25` to `[project.dependencies]` in `pyproject.toml`
|
||||
- [x] Run `uv lock` (or equivalent) to update the lockfile
|
||||
- [x] Verify `from rank_bm25 import BM25Okapi` works in the environment
|
||||
|
||||
**Acceptance:**
|
||||
- `uv run python -c "from rank_bm25 import BM25Okapi; print('ok')"` exits 0
|
||||
|
||||
---
|
||||
|
||||
## Phase 8: Quality Gates
|
||||
|
||||
### Task 8.1: Run unit tests
|
||||
- [x] `uv run pytest tests/test_bm25.py tests/test_rrf.py tests/test_auto_tagger.py -v`
|
||||
- [x] `uv run pytest -m "not integration" -q`
|
||||
- [x] All tests pass
|
||||
|
||||
### Task 8.2: Run linters
|
||||
- [x] `uv run ruff check src/ tests/`
|
||||
- [x] `uv run mypy src/teamlandkarte_mcp/matching/bm25.py src/teamlandkarte_mcp/matching/rrf.py src/teamlandkarte_mcp/matching/auto_tagger.py`
|
||||
- [x] Zero errors
|
||||
|
||||
### Task 8.3: OpenSpec validation
|
||||
- [x] `openspec validate add-bm25-rrf-competence-matching --strict`
|
||||
- [x] All checks pass
|
||||
|
||||
### Task 8.4: Confirm legacy files are removed
|
||||
- [x] `ls debug_*.py` → no such files
|
||||
- [x] `ls SEMANTIC_SIMILARITY_ANALYSIS.md SCORE_BUG_FIX.md` → no such files
|
||||
|
||||
---
|
||||
|
||||
## Phase 9: Architectural Fix — Global BM25 Index
|
||||
|
||||
**Problem identified post-spec**: When `Bm25Index` was built per candidate (N = number of that
|
||||
person's competences), IDF values were always 0 or negative for every term — the library
|
||||
`rank_bm25` computes `log((N - df + 0.5) / (df + 0.5))`, and with N=1 every term has df=1,
|
||||
yielding `log(0)` = negative. All BM25 scores were therefore ≤ 0, making the `max(0.0, raw)`
|
||||
clamp produce a flat 0.0 for everything. The fix builds one global corpus from the union of
|
||||
all filtered candidates' competences, ensuring N is large enough for IDF to be meaningful.
|
||||
|
||||
### Task 9.1: Remove `1e-10` clamp in `bm25.py`
|
||||
**File:** `src/teamlandkarte_mcp/matching/bm25.py`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Remove the special-case block that returned `1e-10` for zero/negative BM25 scores
|
||||
when token overlap existed (a band-aid for the IDF pathology)
|
||||
- [x] Replace with `max(0.0, float(raw))` — ubiquitous terms (negative IDF) simply score 0.0
|
||||
- [x] Update `Bm25Index` and `bm25_rank_competences()` docstrings to reflect the change
|
||||
|
||||
**Acceptance:**
|
||||
- A term present in every document of a sufficiently large corpus scores 0.0, not 1e-10
|
||||
|
||||
---
|
||||
|
||||
### Task 9.2: Add `use_bm25_search` property to `SimilarityEngine`
|
||||
**File:** `src/teamlandkarte_mcp/matching/similarity.py`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Expose `use_bm25_search: bool` as a read-only property (or attribute) on `SimilarityEngine`
|
||||
so `Matcher` can check it without accessing internal config directly
|
||||
|
||||
**Acceptance:**
|
||||
- `engine.use_bm25_search` returns the value of `engine.config.use_bm25_search`
|
||||
|
||||
---
|
||||
|
||||
### Task 9.3: Thread `global_index` parameter through similarity layer
|
||||
**File:** `src/teamlandkarte_mcp/matching/similarity.py`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Add `global_index: Bm25Index | None = None` parameter to `compute_competence_similarity()`
|
||||
- [x] Pass it through to `_bm25_rrf_similarity()`
|
||||
- [x] In `_bm25_rrf_similarity()`: when `global_index is not None`, call
|
||||
`global_index.rank(req)` and filter the result list to only entries whose text is in
|
||||
`set(candidate)`; otherwise fall back to `bm25_rank_competences(req, candidate)`
|
||||
|
||||
**Acceptance:**
|
||||
- `global_index.rank(req)` is called once per required competence, not once per candidate
|
||||
- Results are correctly filtered to the current candidate's competence set
|
||||
- Fall-back path (`global_index=None`) still works unchanged
|
||||
|
||||
---
|
||||
|
||||
### Task 9.4: Build global corpus in `Matcher` before per-candidate loop
|
||||
**File:** `src/teamlandkarte_mcp/matching/matcher.py`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Before the `for cap in filtered` loop: if `self._sim.use_bm25_search` and `filtered`
|
||||
is non-empty, collect the deduplicated union of all candidates' competences into a list
|
||||
`global_corpus`, instantiate `Bm25Index(corpus=global_corpus)`
|
||||
- [x] Pass `global_index` to every `compute_competence_similarity()` call inside the loop
|
||||
- [x] When `use_bm25_search=False` (or `filtered` is empty): `global_bm25_index = None`
|
||||
(no-op, existing behaviour)
|
||||
|
||||
**Acceptance:**
|
||||
- A corpus of N candidates' competences is built once, not N times
|
||||
- Candidates with no overlap with required competences still score 0.0
|
||||
- Candidates with overlap score > 0.0 (IDF is meaningful because N >> 1 in the global corpus)
|
||||
|
||||
---
|
||||
|
||||
### Task 9.5: Update tests for global index
|
||||
**Files:** `tests/test_bm25.py`, `tests/test_similarity.py`, `tests/test_matcher_integration.py`, `tests/test_matching_refinements.py`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] `test_bm25.py`: update small-corpus tests to use ≥ 5-doc corpora so IDF is positive;
|
||||
add `test_bm25_index_rank_filters_to_candidate_subset` and
|
||||
`test_bm25_ubiquitous_term_clamped_to_zero`
|
||||
- [x] `test_similarity.py`: update 4 BM25 tests to pass a 5-doc `global_index`; add
|
||||
`test_use_bm25_search_property_reflects_flag` and `test_bm25_global_index_filters_to_candidate_subset`
|
||||
- [x] `test_matcher_integration.py`: add `use_bm25_search = False` to `_FakeSimilarityEngine`,
|
||||
add `global_index=None` param to `compute_competence_similarity`; update
|
||||
`test_bm25_auto_tag_candidate_matches_after_expansion` to use a 5-doc global corpus;
|
||||
add `test_matcher_builds_global_bm25_index_across_all_candidates`
|
||||
- [x] `test_matching_refinements.py`: add `use_bm25_search = False` and `global_index=None`
|
||||
to local `_FakeSim` stub
|
||||
|
||||
**Acceptance:**
|
||||
- All 140 tests pass, 2 skipped
|
||||
|
||||
---
|
||||
|
||||
### Task 9.6: Update `docs/architecture.md`
|
||||
**File:** `docs/architecture.md`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Replace the incorrect "Per-call index" bullet in §9.3 with a "Global BM25 index" bullet
|
||||
describing the pre-loop construction and per-candidate filtering
|
||||
- [x] Update the §9.1 module table row for `bm25.py` to remove the `1e-10` clamp description
|
||||
and replace with `max(0.0, raw)` description
|
||||
- [x] Update the §9.2 data flow diagram to show `match_candidates` building the global index
|
||||
before the per-candidate loop
|
||||
|
||||
**Acceptance:**
|
||||
- No mention of "per-call index" or `1e-10` clamp remains in the docs
|
||||
|
||||
Reference in New Issue
Block a user