# Implementation Plan: Entfernung Embedding-basierter Similarity – Umstellung auf BM25 + LLM ## Overview Incremental removal of all embedding infrastructure and replacement with BM25 + LLM. Tasks are ordered to avoid breaking the system mid-way: first add new LLM-based methods, then remove embedding code paths, then clean up config and delete dead modules. ## Tasks - [x] 1. Simplify AzureOpenAIClient to remove embedding methods - [x] 1.1 Remove `embeddings` and `get_embeddings_batch` methods from `AzureOpenAIClient` - Remove the `self._emb` AsyncAzureOpenAI client instance - Remove constructor parameters: `embedding_api_key`, `embedding_deployment`, `embedding_batch_size` - Keep only `chat_completion` method and its supporting `self._chat` client - Make `self._chat` the primary client (no longer optional/conditional) - Remove the guard clause in `chat_completion` that checks for missing config - _Requirements: 6.7_ - [x] 1.2 Update `AzureOpenAIClient` constructor signature - New required params: `endpoint`, `api_version`, `chat_deployment`, `llm_api_key` - Keep optional: `timeout_s`, `max_retries`, `cost_tracker` - Remove cost tracker calls for embedding requests (`log_embedding_request`, `log_embedding_batch_request`) - _Requirements: 6.7_ - [-] 2. Refactor SimilarityEngine to remove embedding methods and add LLM role similarity - [x] 2.1 Remove embedding-related methods and properties from `SimilarityEngine` - Remove methods: `prefetch_embeddings`, `_embed`, `get_embedding_for_cache_key`, `get_embeddings_for_cache_keys`, `_aggregate_similarity`, `_per_skill_similarity` - Remove properties: `embedding_model`, `embedding_dimensions`, `use_bm25_search` - Remove constructor parameters: `cache`, `embedding_model`, `embedding_dimensions`, `strategy`, `use_bm25_search` - Remove module-level helpers: `_cache_key`, `cosine_similarity` - Update constructor to accept only: `client`, `cost_tracker`, `use_auto_tagging`, `auto_tagger` - Add `_role_similarity_cache: dict[tuple[str, str], float]` to constructor - _Requirements: 1.3, 1.4, 1.5, 1.6, 1.7, 6.1, 6.2_ - [x] 2.2 Simplify `compute_competence_similarity` to always use BM25+RRF - Remove any strategy/conditional branching - Always call `_bm25_rrf_similarity` directly (with optional auto-tagging expansion) - _Requirements: 1.1, 1.2_ - [x] 2.3 Implement LLM-based `compute_role_similarity` - Replace embedding cosine similarity with LLM chat completion - Add `_ROLE_SIMILARITY_SYSTEM_PROMPT` constant - Implement symmetric cache key: `tuple(sorted((role_a_lower, role_b_lower)))` - Return 0.0 for bad/empty/None/"(unknown)" roles without LLM call - Return 1.0 for identical roles (case-insensitive) without LLM call - Parse JSON response, clamp score to [0.0, 1.0] - On any exception: return 0.0, do not cache failed results - _Requirements: 5.1, 5.2, 5.3, 5.4, 5.5, 5.6, 5.8_ - [x] 2.4 Add `clear_role_cache` method - Clears `_role_similarity_cache` dict - _Requirements: 5.8_ - [x] 2.5 Write property test for BM25 zero-score on disjoint tokens - **Property 1: BM25 Zero-Score für fehlenden Token-Overlap** - **Validates: Requirements 1.1** - [x] 2.6 Write property test for role similarity value range and identity - **Property 5: compute_role_similarity Wertebereich** - **Validates: Requirements 5.1** - [x] 2.7 Write property test for role similarity cache symmetry - **Property 6: Rollen-Similarity-Cache ist symmetrisch und idempotent** - **Validates: Requirements 5.8** - [-] 3. Refactor VocabularyCache to use LLM-based role inference - [x] 3.1 Rewrite `VocabularyCache` class - Remove all embedding-related methods: `preload`, `_preload_role_vocab`, `_preload_competence_vocab`, `ensure_task_embedding`, `infer_competences` - Remove old `infer_primary_role` (embedding-based) - Remove properties: `roles`, `competences` - Remove constructor dependencies on `SimilarityEngine` and `EmbeddingCache` - New constructor accepts only: `db: DBClient`, `client: AzureOpenAIClient` - Remove module-level helpers: `role_vocab_cache_key`, `competence_vocab_cache_key`, `task_id_cache_key`, `VocabItem` - _Requirements: 4.3, 6.3, 6.4, 6.5_ - [x] 3.2 Implement new LLM-based `infer_primary_role` - Accept `task_text: str` keyword argument (no more `task_embedding`) - Fetch all role names from DB via `self._db.get_all_role_names()` - Build LLM prompt with task text and available roles list - Parse JSON response: `{"role": "...", "confidence": 0.0-1.0}` - Validate returned role exists in DB role list - Return `None` on empty text, empty role list, or any exception - _Requirements: 3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 3.7, 3.8_ - [x] 3.3 Write property test for infer_primary_role output validity - **Property 3: infer_primary_role Ausgabe-Validität** - **Validates: Requirements 3.1, 3.4** - [x] 3.4 Write property test for graceful degradation on LLM failure - **Property 4: Graceful Degradation bei LLM-Fehler** - **Validates: Requirements 3.5, 5.6** - [x] 4. Checkpoint - Ensure all tests pass - Ensure all tests pass, ask the user if questions arise. - [-] 5. Update Matcher to unconditionally build BM25 index - [x] 5.1 Remove `use_bm25_search` conditional in `Matcher.match` - Remove `if self._sim.use_bm25_search` guard around BM25 index construction - Always build global BM25 index when `filtered` is non-empty - _Requirements: 2.6_ - [x] 5.2 Write property test for unconditional BM25 index building - **Property 2: Matcher baut BM25-Index bedingungslos** - **Validates: Requirements 2.6** - [x] 6. Update MCP Server to remove embedding preload and wire new components - [x] 6.1 Remove embedding preload infrastructure from `mcp_server.py` - Remove `_startup_preload_embeddings` function - Remove `_deferred_preload` function and `_preload_started` flag - Remove `_ensure_preloaded` function - Remove all `await _ensure_preloaded()` calls in tool handlers - Remove `EmbeddingCache` import and instantiation - _Requirements: 4.1, 4.2, 4.4, 4.5, 6.12_ - [x] 6.2 Update component wiring in `build_server` - Construct `AzureOpenAIClient` with new simplified signature (no embedding params) - Construct `SimilarityEngine` with new signature (client, cost_tracker, use_auto_tagging, auto_tagger) - Construct `VocabularyCache` with new signature (db, client) - _Requirements: 6.7_ - [x] 6.3 Update `infer_primary_role` tool handler - Remove embedding generation step - Call `vocab_cache.infer_primary_role(task_text=text)` directly with task text - _Requirements: 3.9_ - [x] 6.4 Remove or simplify `validate_task_requirements` tool if it depends on embeddings - Remove embedding-based `infer_competences` usage - Either remove the tool entirely or replace with a DB-field-only version - _Requirements: 6.5_ - [x] 7. Simplify config.py and config.toml - [x] 7.1 Remove embedding-related dataclasses and fields from `config.py` - Delete `EmbeddingCacheConfig` dataclass - Delete `InferenceConfig` dataclass - Remove from `AzureOpenAIConfig`: `embedding_deployment`, `embedding_batch_size` - Remove from `SimilarityConfig`: `embedding_model`, `embedding_dimensions`, `strategy`, `use_bm25_search` - Remove `inference` field from `MatchingConfig` - Remove `embedding_cache` field from `AppConfig` - _Requirements: 2.1, 2.2, 6.8, 6.9, 6.10_ - [x] 7.2 Update `load_config` / `_parse_azure_openai` validation - Remove `AZURE_OPENAI_EMBEDDING_API_KEY` environment variable check - Make `AZURE_OPENAI_LLM_API_KEY` always required - Make `chat_deployment` required - Remove `embedding_dimensions == 3072` validation - Remove `strategy` validation - Remove `[matching.inference]` parsing - Remove `[embedding_cache]` parsing - _Requirements: 2.3, 2.4_ - [x] 7.3 Update `config.toml` and `config.toml.example` - Remove `[embedding_cache]` section - Remove from `[matching.similarity]`: `embedding_model`, `embedding_dimensions`, `strategy`, `use_bm25_search` - Remove from `[azure_openai]`: `embedding_deployment`, `embedding_batch_size` - Remove `[matching.inference]` section - _Requirements: 6.11_ - [x] 8. Delete EmbeddingCache module and related tests - [x] 8.1 Delete `src/teamlandkarte_mcp/cache/embedding_cache.py` - _Requirements: 6.6_ - [x] 8.2 Remove or update tests that reference embedding functionality - Delete tests for `EmbeddingCache` - Update tests for `SimilarityEngine` to use new constructor - Update tests for `VocabularyCache` to use new constructor - Update tests for `AzureOpenAIClient` to use new constructor - Remove any test fixtures that create embedding mocks - _Requirements: 6.6_ - [x] 9. Final checkpoint - Ensure all tests pass - Ensure all tests pass, ask the user if questions arise. ## Notes - Tasks marked with `*` are optional and can be skipped for faster MVP - Each task references specific requirements for traceability - Checkpoints ensure incremental validation - Property tests validate universal correctness properties from the design document - Order ensures no broken intermediate states: new LLM methods added before old embedding paths removed