Migrate all repos into monorepo context folders
Bahn: aisupport, Analyse-O2C-C2S, awesome-bahn-mcp-servers, beam-mcp,
Confluence_Bot, db-planet-mcp-server, O2C-Harness, project-audit,
Projekt-KIQ-HP, teamlandkarte-mcp
Dhive: Jury-Voting
Privat: CV, NoteGraph (NOTE: NoteGraph needs complete redo after consolidation)
Shared: AI-Orchestrator, OrgMyLife, power_skills_and_more
Shared/references: symphony (read-only)
Bahn repos remain available as independent remotes - this monorepo
pulls them in via subtree, the originals are untouched.
This commit is contained in:
@@ -0,0 +1,677 @@
|
||||
# Design: Entfernung Embedding-basierter Similarity – Umstellung auf BM25 + LLM
|
||||
|
||||
## Übersicht
|
||||
|
||||
Dieses Design beschreibt die vollständige Entfernung der Embedding-Infrastruktur aus dem Teamlandkarte-MCP-System. Das System wird von einem hybriden Embedding/BM25-Ansatz auf eine reine BM25 + LLM-Architektur umgestellt:
|
||||
|
||||
- **Kompetenz-Matching**: Ausschließlich BM25+RRF (bereits implementiert, nur Conditional-Logik entfernen)
|
||||
- **Rollen-Similarity**: LLM Chat Completion (neu, ersetzt Embedding-Cosine-Similarity)
|
||||
- **Rollen-Inferenz**: LLM Chat Completion (neu, ersetzt Embedding-basierte Nearest-Neighbor-Suche)
|
||||
- **AutoTagger**: Bleibt unverändert (bereits LLM-basiert)
|
||||
|
||||
Entfernt werden: `EmbeddingCache`, `_embed`, `prefetch_embeddings`, `_per_skill_similarity`, `_aggregate_similarity`, Embedding-API-Methoden im `AzureOpenAIClient`, alle Embedding-Konfigurationsfelder, und der Startup-Preload.
|
||||
|
||||
## Architektur
|
||||
|
||||
### Vorher (Hybrid)
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[MCP Server Startup] --> B[Preload Embeddings]
|
||||
B --> C[VocabularyCache: Role + Competence Embeddings]
|
||||
B --> D[Task Embeddings]
|
||||
|
||||
E[Matching Request] --> F{use_bm25_search?}
|
||||
F -->|true| G[BM25+RRF Competence Similarity]
|
||||
F -->|false| H[Embedding Cosine Competence Similarity]
|
||||
|
||||
E --> I[Embedding Cosine Role Similarity]
|
||||
|
||||
J[infer_primary_role] --> K[Task Embedding → Cosine vs Role Vocab]
|
||||
```
|
||||
|
||||
### Nachher (BM25 + LLM)
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[MCP Server Startup] --> B[DB Schema Verification]
|
||||
A --> C[AzureOpenAIClient: nur Chat Completion]
|
||||
|
||||
E[Matching Request] --> G[BM25+RRF Competence Similarity]
|
||||
E --> I[LLM Role Similarity mit In-Run Cache]
|
||||
|
||||
J[infer_primary_role] --> K[LLM Chat Completion: Task Text → Role Selection]
|
||||
```
|
||||
|
||||
### Datenfluss Matching (Nachher)
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Client
|
||||
participant Matcher
|
||||
participant SimilarityEngine
|
||||
participant BM25
|
||||
participant LLM as AzureOpenAIClient (Chat)
|
||||
|
||||
Client->>Matcher: match(capacities, requirements)
|
||||
Matcher->>Matcher: Filter by availability
|
||||
Matcher->>BM25: Build global index (all candidate competences)
|
||||
|
||||
loop Per Candidate
|
||||
Matcher->>SimilarityEngine: compute_competence_similarity(required, candidate, global_index)
|
||||
SimilarityEngine->>BM25: rank per required competence
|
||||
SimilarityEngine-->>Matcher: {req → {score, best_match, rationale}}
|
||||
|
||||
Matcher->>SimilarityEngine: compute_role_similarity(required_role, candidate_role)
|
||||
SimilarityEngine->>SimilarityEngine: Check in-run cache
|
||||
alt Cache Miss
|
||||
SimilarityEngine->>LLM: chat_completion(role_similarity_prompt)
|
||||
LLM-->>SimilarityEngine: {"similarity": 0.85}
|
||||
SimilarityEngine->>SimilarityEngine: Cache result
|
||||
end
|
||||
SimilarityEngine-->>Matcher: float score
|
||||
end
|
||||
|
||||
Matcher-->>Client: MatchResult
|
||||
```
|
||||
|
||||
## Komponenten und Schnittstellen
|
||||
|
||||
### 1. SimilarityEngine (refactored)
|
||||
|
||||
**Datei:** `src/teamlandkarte_mcp/matching/similarity.py`
|
||||
|
||||
**Entfernte Methoden:**
|
||||
- `prefetch_embeddings`
|
||||
- `_embed`
|
||||
- `get_embedding_for_cache_key`
|
||||
- `get_embeddings_for_cache_keys`
|
||||
- `_aggregate_similarity`
|
||||
- `_per_skill_similarity`
|
||||
|
||||
**Entfernte Properties:**
|
||||
- `embedding_model`
|
||||
- `embedding_dimensions`
|
||||
- `use_bm25_search`
|
||||
|
||||
**Entfernte Constructor-Parameter:**
|
||||
- `cache` (EmbeddingCache)
|
||||
- `embedding_model`
|
||||
- `embedding_dimensions`
|
||||
- `strategy`
|
||||
- `use_bm25_search`
|
||||
|
||||
**Neuer Constructor:**
|
||||
|
||||
```python
|
||||
class SimilarityEngine:
|
||||
"""Compute similarity scores using BM25+RRF and LLM."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
client: AzureOpenAIClient,
|
||||
cost_tracker: CostTracker | None = None,
|
||||
use_auto_tagging: bool = False,
|
||||
auto_tagger: AutoTagger | None = None,
|
||||
) -> None:
|
||||
self._client = client
|
||||
self._cost_tracker = cost_tracker
|
||||
self._use_auto_tagging = use_auto_tagging
|
||||
self._auto_tagger = auto_tagger
|
||||
# Per-run cache for LLM role similarity: (role_a, role_b) → score
|
||||
self._role_similarity_cache: dict[tuple[str, str], float] = {}
|
||||
```
|
||||
|
||||
**Geänderte Methode `compute_competence_similarity`:**
|
||||
|
||||
```python
|
||||
async def compute_competence_similarity(
|
||||
self,
|
||||
required: list[str],
|
||||
candidate: list[str],
|
||||
global_index: Bm25Index | None = None,
|
||||
) -> dict[str, dict[str, object]]:
|
||||
"""BM25+RRF competence similarity (einziger Pfad)."""
|
||||
working = list(candidate)
|
||||
if self._use_auto_tagging and self._auto_tagger is not None:
|
||||
working = await self._auto_tagger.expand_competences(required, candidate)
|
||||
return self._bm25_rrf_similarity(required, working, global_index=global_index)
|
||||
```
|
||||
|
||||
**Neue Methode `compute_role_similarity` (LLM-basiert):**
|
||||
|
||||
```python
|
||||
_ROLE_SIMILARITY_SYSTEM_PROMPT = (
|
||||
"You are a job-role similarity expert. Given two role names, "
|
||||
"determine their semantic similarity on a scale from 0.0 to 1.0. "
|
||||
"0.0 means completely unrelated roles, 1.0 means identical or "
|
||||
"interchangeable roles. Consider synonyms, hierarchy, and domain overlap. "
|
||||
'Respond with a JSON object: {"similarity": <float>}'
|
||||
)
|
||||
|
||||
async def compute_role_similarity(
|
||||
self,
|
||||
required_role: str | None,
|
||||
candidate_role: str | None,
|
||||
) -> float:
|
||||
"""LLM-basierte Rollen-Similarity."""
|
||||
if self._is_bad_role(required_role) or self._is_bad_role(candidate_role):
|
||||
return 0.0
|
||||
|
||||
req_norm = str(required_role).strip().lower()
|
||||
cand_norm = str(candidate_role).strip().lower()
|
||||
|
||||
if req_norm == cand_norm:
|
||||
return 1.0
|
||||
|
||||
# Symmetrischer Cache-Key
|
||||
cache_key = tuple(sorted((req_norm, cand_norm)))
|
||||
if cache_key in self._role_similarity_cache:
|
||||
return self._role_similarity_cache[cache_key]
|
||||
|
||||
try:
|
||||
raw = await self._client.chat_completion(
|
||||
system=_ROLE_SIMILARITY_SYSTEM_PROMPT,
|
||||
user=f"Role A: {required_role}\nRole B: {candidate_role}",
|
||||
)
|
||||
payload = json.loads(raw)
|
||||
score = float(payload.get("similarity", 0.0))
|
||||
score = max(0.0, min(1.0, score))
|
||||
except Exception:
|
||||
score = 0.0
|
||||
|
||||
self._role_similarity_cache[cache_key] = score
|
||||
return score
|
||||
```
|
||||
|
||||
**Neue Methode `clear_role_cache`:**
|
||||
|
||||
```python
|
||||
def clear_role_cache(self) -> None:
|
||||
"""Cache zwischen Matching-Runs leeren."""
|
||||
self._role_similarity_cache.clear()
|
||||
```
|
||||
|
||||
### 2. VocabularyCache (refactored)
|
||||
|
||||
**Datei:** `src/teamlandkarte_mcp/matching/vocabulary.py`
|
||||
|
||||
**Entfernte Methoden:**
|
||||
- `preload`
|
||||
- `_preload_role_vocab`
|
||||
- `_preload_competence_vocab`
|
||||
- `ensure_task_embedding`
|
||||
- `infer_primary_role` (alte Signatur mit `task_embedding`)
|
||||
- `infer_competences`
|
||||
|
||||
**Entfernte Properties:**
|
||||
- `roles`
|
||||
- `competences`
|
||||
|
||||
**Neuer Constructor:**
|
||||
|
||||
```python
|
||||
class VocabularyCache:
|
||||
"""LLM-basierte Rollen-Inferenz aus Task-Text."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
db: DBClient,
|
||||
client: AzureOpenAIClient,
|
||||
) -> None:
|
||||
self._db = db
|
||||
self._client = client
|
||||
```
|
||||
|
||||
**Neue Methode `infer_primary_role` (LLM-basiert):**
|
||||
|
||||
```python
|
||||
_ROLE_INFERENCE_SYSTEM_PROMPT = (
|
||||
"You are a role classification expert. Given a task description and a list "
|
||||
"of available roles, select the single most appropriate role for the task. "
|
||||
"You MUST select exactly one role from the provided list. "
|
||||
"Respond with a JSON object: "
|
||||
'{"role": "<selected role name>", "confidence": <float 0.0-1.0>}'
|
||||
)
|
||||
|
||||
async def infer_primary_role(
|
||||
self,
|
||||
*,
|
||||
task_text: str,
|
||||
) -> tuple[str, float] | None:
|
||||
"""Inferiere die passendste Rolle für einen Task-Text via LLM."""
|
||||
role_names = self._db.get_all_role_names()
|
||||
if not role_names:
|
||||
return None
|
||||
|
||||
text = (task_text or "").strip()
|
||||
if not text:
|
||||
return None
|
||||
|
||||
user_prompt = (
|
||||
f"Task: {text}\n\n"
|
||||
f"Available roles: {json.dumps(list(role_names), ensure_ascii=False)}"
|
||||
)
|
||||
|
||||
try:
|
||||
raw = await self._client.chat_completion(
|
||||
system=_ROLE_INFERENCE_SYSTEM_PROMPT,
|
||||
user=user_prompt,
|
||||
)
|
||||
payload = json.loads(raw)
|
||||
role = str(payload.get("role", "")).strip()
|
||||
confidence = float(payload.get("confidence", 0.0))
|
||||
confidence = max(0.0, min(1.0, confidence))
|
||||
|
||||
# Validierung: Rolle muss in der DB-Liste existieren
|
||||
if role not in set(role_names):
|
||||
return None
|
||||
|
||||
return role, confidence
|
||||
except Exception:
|
||||
return None
|
||||
```
|
||||
|
||||
### 3. AzureOpenAIClient (vereinfacht)
|
||||
|
||||
**Datei:** `src/teamlandkarte_mcp/azure/openai_client.py`
|
||||
|
||||
**Entfernte Methoden:**
|
||||
- `embeddings`
|
||||
- `get_embeddings_batch`
|
||||
|
||||
**Entfernte Constructor-Parameter:**
|
||||
- `embedding_api_key`
|
||||
- `embedding_deployment`
|
||||
- `embedding_batch_size`
|
||||
|
||||
**Neuer Constructor:**
|
||||
|
||||
```python
|
||||
class AzureOpenAIClient:
|
||||
"""Azure OpenAI Chat Completion Client."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
endpoint: str,
|
||||
api_version: str,
|
||||
chat_deployment: str,
|
||||
llm_api_key: str,
|
||||
timeout_s: float = 30.0,
|
||||
max_retries: int = 5,
|
||||
cost_tracker: CostTracker | None = None,
|
||||
) -> None:
|
||||
self._chat_deployment = chat_deployment
|
||||
self._max_retries = max_retries
|
||||
self._timeout_s = timeout_s
|
||||
self._cost_tracker = cost_tracker
|
||||
|
||||
self._chat = AsyncAzureOpenAI(
|
||||
api_key=llm_api_key,
|
||||
azure_endpoint=endpoint,
|
||||
api_version=api_version,
|
||||
)
|
||||
```
|
||||
|
||||
Die `chat_completion`-Methode bleibt unverändert, außer dass die Guard-Clause für fehlende Konfiguration entfällt (Chat ist jetzt immer konfiguriert).
|
||||
|
||||
### 4. Matcher (minimal geändert)
|
||||
|
||||
**Datei:** `src/teamlandkarte_mcp/matching/matcher.py`
|
||||
|
||||
**Änderungen:**
|
||||
- Entfernung der `if self._sim.use_bm25_search`-Bedingung beim BM25-Index-Aufbau
|
||||
- Der globale BM25-Index wird **immer** gebaut (unconditional)
|
||||
|
||||
```python
|
||||
# Vorher:
|
||||
global_bm25_index: Bm25Index | None = None
|
||||
if self._sim.use_bm25_search and filtered:
|
||||
global_corpus = list(...)
|
||||
global_bm25_index = Bm25Index(corpus=global_corpus)
|
||||
|
||||
# Nachher:
|
||||
global_bm25_index: Bm25Index | None = None
|
||||
if filtered:
|
||||
global_corpus = list(
|
||||
{c for cap in filtered for c in cap.competences if c.strip()}
|
||||
)
|
||||
global_bm25_index = Bm25Index(corpus=global_corpus)
|
||||
```
|
||||
|
||||
### 5. MCP Server (vereinfacht)
|
||||
|
||||
**Datei:** `src/teamlandkarte_mcp/mcp_server.py`
|
||||
|
||||
**Entfernte Elemente:**
|
||||
- `_startup_preload_embeddings` Funktion
|
||||
- `_deferred_preload` Funktion und `_preload_started` Flag
|
||||
- `_ensure_preloaded` Funktion
|
||||
- `EmbeddingCache`-Instanziierung
|
||||
- `emb_cache`-Variable
|
||||
- Alle `await _ensure_preloaded()`-Aufrufe in Tools
|
||||
- Import von `EmbeddingCache`
|
||||
- Import von `VocabularyCache` (wird direkt mit `AzureOpenAIClient` konstruiert)
|
||||
|
||||
**Geänderte Konstruktion:**
|
||||
|
||||
```python
|
||||
# Vorher: Embedding-Client + separater LLM-Client
|
||||
azure_client = AzureOpenAIClient(
|
||||
endpoint=..., embedding_api_key=..., embedding_deployment=..., ...
|
||||
)
|
||||
|
||||
# Nachher: Nur noch ein LLM-Client
|
||||
azure_client = AzureOpenAIClient(
|
||||
endpoint=cfg.azure_openai.endpoint,
|
||||
api_version=cfg.azure_openai.api_version,
|
||||
chat_deployment=cfg.azure_openai.chat_deployment,
|
||||
llm_api_key=cfg.azure_openai.llm_api_key,
|
||||
cost_tracker=cost_tracker,
|
||||
)
|
||||
|
||||
similarity = SimilarityEngine(
|
||||
client=azure_client,
|
||||
cost_tracker=cost_tracker,
|
||||
use_auto_tagging=cfg.similarity.use_auto_tagging,
|
||||
auto_tagger=auto_tagger,
|
||||
)
|
||||
|
||||
vocab_cache = VocabularyCache(
|
||||
db=db_client,
|
||||
client=azure_client,
|
||||
)
|
||||
```
|
||||
|
||||
**Geändertes `infer_primary_role`-Tool:**
|
||||
|
||||
```python
|
||||
@mcp.tool()
|
||||
async def infer_primary_role(
|
||||
task_id: Optional[str] = None,
|
||||
task_text: Optional[str] = None,
|
||||
) -> str:
|
||||
"""Infer the single closest role from either a DB task or free text."""
|
||||
if bool(task_id) == bool(task_text):
|
||||
return "Provide exactly one of task_id or task_text."
|
||||
|
||||
text = (task_text or "").strip()
|
||||
if task_id:
|
||||
_ensure_db()
|
||||
task = db_client.get_task_by_id(task_id)
|
||||
if task is None:
|
||||
return f"Task not found or not published: {task_id}"
|
||||
title = (task.title or "").strip()
|
||||
desc = (task.description or "").strip()
|
||||
text = (title + "\n\n" + desc).strip() if title else desc
|
||||
|
||||
if not text:
|
||||
return "Task text is empty."
|
||||
|
||||
best = await vocab_cache.infer_primary_role(task_text=text)
|
||||
if best is None:
|
||||
rows = [["", ""]]
|
||||
else:
|
||||
role, score = best
|
||||
rows = [[str(role), f"{float(score):.3f}"]]
|
||||
|
||||
return md_table(["Role", "Confidence"], rows)
|
||||
```
|
||||
|
||||
**Entferntes Tool `validate_task_requirements`:**
|
||||
Dieses Tool basiert vollständig auf Embedding-Inferenz (`infer_competences`, `ensure_task_embedding`). Es wird entfernt oder durch eine vereinfachte Version ersetzt, die nur DB-Felder anzeigt.
|
||||
|
||||
### 6. Config (vereinfacht)
|
||||
|
||||
**Datei:** `src/teamlandkarte_mcp/config.py`
|
||||
|
||||
**Entfernte Dataclasses:**
|
||||
- `EmbeddingCacheConfig`
|
||||
- `InferenceConfig`
|
||||
|
||||
**Geänderte Dataclasses:**
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class AzureOpenAIConfig:
|
||||
"""Azure OpenAI configuration (nur Chat Completion)."""
|
||||
endpoint: str
|
||||
api_version: str = "2024-02-15-preview"
|
||||
chat_deployment: str = ""
|
||||
llm_api_key: str = ""
|
||||
show_costs_in_output: bool = False
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SimilarityConfig:
|
||||
"""Similarity engine configuration."""
|
||||
use_auto_tagging: bool = False
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class MatchingConfig:
|
||||
"""Matching weights and thresholds."""
|
||||
competence_weight: float = 0.8
|
||||
role_weight: float = 0.2
|
||||
require_confirmation: bool = True
|
||||
thresholds: MatchingThresholds = MatchingThresholds()
|
||||
fuzzy: FuzzyConfig = FuzzyConfig()
|
||||
# inference entfällt
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AppConfig:
|
||||
"""Top-level application configuration."""
|
||||
database: DatabaseConfig
|
||||
matching: MatchingConfig
|
||||
cache: CacheConfig
|
||||
azure_openai: AzureOpenAIConfig
|
||||
similarity: SimilarityConfig
|
||||
# embedding_cache entfällt
|
||||
```
|
||||
|
||||
**Entfernte Felder aus `SimilarityConfig`:**
|
||||
- `embedding_model`
|
||||
- `embedding_dimensions`
|
||||
- `strategy`
|
||||
- `use_bm25_search`
|
||||
|
||||
**Entfernte Felder aus `AzureOpenAIConfig`:**
|
||||
- `embedding_deployment`
|
||||
- `embedding_batch_size`
|
||||
|
||||
**Entfernte Validierungen in `load_config`:**
|
||||
- `AZURE_OPENAI_EMBEDDING_API_KEY` Prüfung
|
||||
- `embedding_dimensions == 3072` Prüfung
|
||||
- `strategy` Validierung
|
||||
- `inference` Parsing
|
||||
|
||||
**Neue Validierung:**
|
||||
- `AZURE_OPENAI_LLM_API_KEY` ist jetzt immer erforderlich (nicht nur bei auto_tagging)
|
||||
- `chat_deployment` ist jetzt erforderlich
|
||||
|
||||
### 7. Entfernte Dateien
|
||||
|
||||
| Datei | Grund |
|
||||
|-------|-------|
|
||||
| `src/teamlandkarte_mcp/cache/embedding_cache.py` | Keine Embeddings mehr |
|
||||
| Zugehörige Tests für EmbeddingCache | Keine Embeddings mehr |
|
||||
|
||||
### 8. config.toml Änderungen
|
||||
|
||||
**Entfernte Sektionen:**
|
||||
- `[embedding_cache]` komplett
|
||||
|
||||
**Entfernte Felder aus `[matching.similarity]`:**
|
||||
- `embedding_model`
|
||||
- `embedding_dimensions`
|
||||
- `strategy`
|
||||
- `use_bm25_search`
|
||||
|
||||
**Entfernte Felder aus `[azure_openai]`:**
|
||||
- `embedding_deployment`
|
||||
- `embedding_batch_size`
|
||||
|
||||
**Entfernte Felder aus `[matching]`:**
|
||||
- `[matching.inference]` komplett
|
||||
|
||||
**Neue Pflichtfelder in `[azure_openai]`:**
|
||||
- `chat_deployment` (bereits vorhanden als optionales Feld, wird Pflicht)
|
||||
|
||||
**Neue Umgebungsvariable (Pflicht):**
|
||||
- `AZURE_OPENAI_LLM_API_KEY` (ersetzt `AZURE_OPENAI_EMBEDDING_API_KEY`)
|
||||
|
||||
**Entfernte Umgebungsvariable:**
|
||||
- `AZURE_OPENAI_EMBEDDING_API_KEY`
|
||||
|
||||
## Datenmodelle
|
||||
|
||||
### LLM Role Similarity Request/Response
|
||||
|
||||
```json
|
||||
// System Prompt: _ROLE_SIMILARITY_SYSTEM_PROMPT
|
||||
// User Message:
|
||||
"Role A: Software Engineer\nRole B: Backend Developer"
|
||||
|
||||
// Expected Response:
|
||||
{"similarity": 0.75}
|
||||
```
|
||||
|
||||
### LLM Role Inference Request/Response
|
||||
|
||||
```json
|
||||
// System Prompt: _ROLE_INFERENCE_SYSTEM_PROMPT
|
||||
// User Message:
|
||||
"Task: Implementierung einer REST-API für Benutzerverwaltung\n\nAvailable roles: [\"Backend Developer\", \"Frontend Developer\", \"DevOps Engineer\", \"Data Engineer\"]"
|
||||
|
||||
// Expected Response:
|
||||
{"role": "Backend Developer", "confidence": 0.92}
|
||||
```
|
||||
|
||||
### In-Run Role Similarity Cache
|
||||
|
||||
```python
|
||||
# Symmetrischer Cache innerhalb eines Matching-Runs
|
||||
# Key: tuple(sorted((role_a_lower, role_b_lower)))
|
||||
# Value: float (0.0 - 1.0)
|
||||
_role_similarity_cache: dict[tuple[str, str], float] = {}
|
||||
```
|
||||
|
||||
Der Cache wird pro `SimilarityEngine`-Instanz gehalten und lebt für die Dauer des Server-Prozesses. Da Rollennamen stabil sind (aus der DB), ist kein TTL nötig.
|
||||
|
||||
|
||||
## Correctness Properties
|
||||
|
||||
*A property is a characteristic or behavior that should hold true across all valid executions of a system — essentially, a formal statement about what the system should do. Properties serve as the bridge between human-readable specifications and machine-verifiable correctness guarantees.*
|
||||
|
||||
### Property 1: BM25 Zero-Score für fehlenden Token-Overlap
|
||||
|
||||
*For any* set of required competences and candidate competences where no candidate shares any token with a required competence, `compute_competence_similarity` shall return a score of 0.0 for that required competence.
|
||||
|
||||
**Validates: Requirements 1.1**
|
||||
|
||||
### Property 2: Matcher baut BM25-Index bedingungslos
|
||||
|
||||
*For any* non-empty list of filtered candidates with competences, the Matcher shall always build a global BM25 index and use it for competence scoring, producing results where candidates with no token overlap receive score 0.0.
|
||||
|
||||
**Validates: Requirements 2.6**
|
||||
|
||||
### Property 3: infer_primary_role Ausgabe-Validität
|
||||
|
||||
*For any* non-empty task text and non-empty role list from the database, if `infer_primary_role` returns a non-None result, the returned role name must be an element of the database role list and the confidence must be a float in [0.0, 1.0].
|
||||
|
||||
**Validates: Requirements 3.1, 3.4**
|
||||
|
||||
### Property 4: Graceful Degradation bei LLM-Fehler
|
||||
|
||||
*For any* LLM call that raises an exception, `compute_role_similarity` shall return 0.0 and `infer_primary_role` shall return None, without propagating the exception to the caller.
|
||||
|
||||
**Validates: Requirements 3.5, 5.6**
|
||||
|
||||
### Property 5: compute_role_similarity Wertebereich
|
||||
|
||||
*For any* two non-empty, non-"(unknown)" role names, `compute_role_similarity` shall return a float in the closed interval [0.0, 1.0]. For identical role names (case-insensitive), it shall return 1.0.
|
||||
|
||||
**Validates: Requirements 5.1**
|
||||
|
||||
### Property 6: Rollen-Similarity-Cache ist symmetrisch und idempotent
|
||||
|
||||
*For any* role pair (A, B), calling `compute_role_similarity(A, B)` and then `compute_role_similarity(B, A)` shall return the same score, and the second call shall not invoke the LLM (cache hit). Repeated calls with the same pair shall always return the same cached value.
|
||||
|
||||
**Validates: Requirements 5.8**
|
||||
|
||||
## Error Handling
|
||||
|
||||
### LLM-Fehler in compute_role_similarity
|
||||
|
||||
- Bei jeder Exception (Timeout, API-Fehler, JSON-Parse-Fehler) wird `0.0` zurückgegeben
|
||||
- Fehler wird geloggt (LOGGER.warning)
|
||||
- Kein Eintrag im Cache für fehlgeschlagene Aufrufe (Retry bei nächstem Aufruf möglich)
|
||||
|
||||
### LLM-Fehler in infer_primary_role
|
||||
|
||||
- Bei jeder Exception wird `None` zurückgegeben
|
||||
- Fehler wird geloggt (LOGGER.warning)
|
||||
- Caller (MCP-Tool) zeigt leere Tabelle an
|
||||
|
||||
### Ungültige LLM-Antworten
|
||||
|
||||
- **Role Similarity**: Wenn `similarity` nicht im JSON oder nicht parsebar → 0.0
|
||||
- **Role Inference**: Wenn `role` nicht in der DB-Liste → None
|
||||
- **Role Inference**: Wenn `confidence` nicht parsebar → 0.0 (aber Rolle wird trotzdem zurückgegeben wenn valide)
|
||||
|
||||
### Leere/Ungültige Eingaben
|
||||
|
||||
- `compute_role_similarity` mit None/leer/"(unknown)" → 0.0 (kein LLM-Aufruf)
|
||||
- `infer_primary_role` mit leerem Text → None (kein LLM-Aufruf)
|
||||
- `infer_primary_role` mit leerer Rollenliste aus DB → None (kein LLM-Aufruf)
|
||||
- `compute_competence_similarity` mit leerer Required-Liste → leeres Dict
|
||||
- `compute_competence_similarity` mit leerer Candidate-Liste → alle Scores 0.0
|
||||
|
||||
### Konfigurationsfehler
|
||||
|
||||
- Fehlende `AZURE_OPENAI_LLM_API_KEY` → `ConfigError` beim Laden (fail-fast)
|
||||
- Fehlender `chat_deployment` → `ConfigError` beim Laden (fail-fast)
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Property-Based Tests (fast-check / Hypothesis)
|
||||
|
||||
Bibliothek: **Hypothesis** (Python PBT-Standard)
|
||||
|
||||
Konfiguration: Mindestens 100 Iterationen pro Property-Test.
|
||||
|
||||
Jeder Property-Test wird mit einem Kommentar getaggt:
|
||||
```
|
||||
# Feature: remove-embedding-competence-similarity, Property {N}: {title}
|
||||
```
|
||||
|
||||
| Property | Test-Ansatz | Generator |
|
||||
|----------|-------------|-----------|
|
||||
| 1: BM25 Zero-Score | Generiere disjunkte Token-Sets für required/candidate, prüfe Score == 0.0 | `st.lists(st.text(alphabet=st.characters(whitelist_categories=('L',)), min_size=3))` |
|
||||
| 2: Matcher BM25 unconditional | Generiere Capacities + Requirements, prüfe dass Ergebnis BM25-Charakteristik hat (0.0 bei no-overlap) | Custom Capacity/Requirements strategies |
|
||||
| 3: infer_primary_role Validität | Generiere Task-Texte + Role-Listen, mocke LLM mit zufälliger valider Antwort, prüfe Output-Constraints | `st.text(min_size=1)`, `st.lists(st.text(min_size=1), min_size=1)` |
|
||||
| 4: Graceful Degradation | Generiere zufällige Exceptions, prüfe dass 0.0/None zurückkommt | `st.sampled_from([TimeoutError, RuntimeError, ValueError, json.JSONDecodeError])` |
|
||||
| 5: Role Similarity Wertebereich | Generiere Rollenpaare, mocke LLM mit zufälligem Score, prüfe [0.0, 1.0] und Identitäts-Case | `st.text(min_size=1, max_size=50)` |
|
||||
| 6: Cache Symmetrie | Generiere Rollenpaare, rufe in beiden Reihenfolgen auf, prüfe gleichen Score + nur 1 LLM-Call | `st.text(min_size=1, max_size=50)` |
|
||||
|
||||
### Unit Tests
|
||||
|
||||
Unit Tests fokussieren auf:
|
||||
|
||||
- **Spezifische Beispiele**: Bekannte Rollenpaare (z.B. "Backend Developer" vs "Software Engineer") mit gemocktem LLM
|
||||
- **Edge Cases**: Leere Strings, None-Werte, "(unknown)", Whitespace-only
|
||||
- **Integration**: Config-Loading ohne Embedding-Felder, Server-Startup ohne Preload
|
||||
- **Regressions**: Sicherstellen dass BM25+RRF-Ergebnisse identisch zum bisherigen `use_bm25_search=True`-Pfad sind
|
||||
|
||||
### Integrationstests
|
||||
|
||||
- End-to-End Matching-Run mit gemocktem LLM-Client
|
||||
- Config-Loading aus minimaler TOML-Datei (ohne Embedding-Sektionen)
|
||||
- Server-Startup ohne `AZURE_OPENAI_EMBEDDING_API_KEY` (darf nicht mehr geprüft werden)
|
||||
|
||||
### Nicht getestet (bewusst ausgelassen)
|
||||
|
||||
- Prompt-Qualität (subjektiv, erfordert manuelles Review)
|
||||
- LLM-Antwort-Genauigkeit (abhängig vom Modell, nicht deterministisch)
|
||||
- Startup-Performance-Verbesserung (Benchmark, kein Unit-Test)
|
||||
Reference in New Issue
Block a user