Orchestrator/bahn/teamlandkarte-mcp/.kiro/specs/remove-embedding-competence-similarity/tasks.md

Implementation Plan: Entfernung Embedding-basierter Similarity – Umstellung auf BM25 + LLM

Overview

Incremental removal of all embedding infrastructure and replacement with BM25 + LLM. Tasks are ordered to avoid breaking the system mid-way: first add new LLM-based methods, then remove embedding code paths, then clean up config and delete dead modules.

Tasks

• 1. Simplify AzureOpenAIClient to remove embedding methods

  • 1.1 Remove embeddings and get_embeddings_batch methods from AzureOpenAIClient

    • Remove the self._emb AsyncAzureOpenAI client instance
    • Remove constructor parameters: embedding_api_key, embedding_deployment, embedding_batch_size
    • Keep only chat_completion method and its supporting self._chat client
    • Make self._chat the primary client (no longer optional/conditional)
    • Remove the guard clause in chat_completion that checks for missing config
    • Requirements: 6.7

  • 1.2 Update AzureOpenAIClient constructor signature

    • New required params: endpoint, api_version, chat_deployment, llm_api_key
    • Keep optional: timeout_s, max_retries, cost_tracker
    • Remove cost tracker calls for embedding requests (log_embedding_request, log_embedding_batch_request)
    • Requirements: 6.7

• [-] 2. Refactor SimilarityEngine to remove embedding methods and add LLM role similarity

  • 2.1 Remove embedding-related methods and properties from SimilarityEngine

    • Remove methods: prefetch_embeddings, _embed, get_embedding_for_cache_key, get_embeddings_for_cache_keys, _aggregate_similarity, _per_skill_similarity
    • Remove properties: embedding_model, embedding_dimensions, use_bm25_search
    • Remove constructor parameters: cache, embedding_model, embedding_dimensions, strategy, use_bm25_search
    • Remove module-level helpers: _cache_key, cosine_similarity
    • Update constructor to accept only: client, cost_tracker, use_auto_tagging, auto_tagger
    • Add _role_similarity_cache: dict[tuple[str, str], float] to constructor
    • Requirements: 1.3, 1.4, 1.5, 1.6, 1.7, 6.1, 6.2

  • 2.2 Simplify compute_competence_similarity to always use BM25+RRF

    • Remove any strategy/conditional branching
    • Always call _bm25_rrf_similarity directly (with optional auto-tagging expansion)
    • Requirements: 1.1, 1.2

  • 2.3 Implement LLM-based compute_role_similarity

    • Replace embedding cosine similarity with LLM chat completion
    • Add _ROLE_SIMILARITY_SYSTEM_PROMPT constant
    • Implement symmetric cache key: tuple(sorted((role_a_lower, role_b_lower)))
    • Return 0.0 for bad/empty/None/"(unknown)" roles without LLM call
    • Return 1.0 for identical roles (case-insensitive) without LLM call
    • Parse JSON response, clamp score to [0.0, 1.0]
    • On any exception: return 0.0, do not cache failed results
    • Requirements: 5.1, 5.2, 5.3, 5.4, 5.5, 5.6, 5.8

  • 2.4 Add clear_role_cache method

    • Clears _role_similarity_cache dict
    • Requirements: 5.8

  • 2.5 Write property test for BM25 zero-score on disjoint tokens

    • Property 1: BM25 Zero-Score für fehlenden Token-Overlap
    • Validates: Requirements 1.1

  • 2.6 Write property test for role similarity value range and identity

    • Property 5: compute_role_similarity Wertebereich
    • Validates: Requirements 5.1

  • 2.7 Write property test for role similarity cache symmetry

    • Property 6: Rollen-Similarity-Cache ist symmetrisch und idempotent
    • Validates: Requirements 5.8

• [-] 3. Refactor VocabularyCache to use LLM-based role inference

  • 3.1 Rewrite VocabularyCache class

    • Remove all embedding-related methods: preload, _preload_role_vocab, _preload_competence_vocab, ensure_task_embedding, infer_competences
    • Remove old infer_primary_role (embedding-based)
    • Remove properties: roles, competences
    • Remove constructor dependencies on SimilarityEngine and EmbeddingCache
    • New constructor accepts only: db: DBClient, client: AzureOpenAIClient
    • Remove module-level helpers: role_vocab_cache_key, competence_vocab_cache_key, task_id_cache_key, VocabItem
    • Requirements: 4.3, 6.3, 6.4, 6.5

  • 3.2 Implement new LLM-based infer_primary_role

    • Accept task_text: str keyword argument (no more task_embedding)
    • Fetch all role names from DB via self._db.get_all_role_names()
    • Build LLM prompt with task text and available roles list
    • Parse JSON response: {"role": "...", "confidence": 0.0-1.0}
    • Validate returned role exists in DB role list
    • Return None on empty text, empty role list, or any exception
    • Requirements: 3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 3.7, 3.8

  • 3.3 Write property test for infer_primary_role output validity

    • Property 3: infer_primary_role Ausgabe-Validität
    • Validates: Requirements 3.1, 3.4

  • 3.4 Write property test for graceful degradation on LLM failure

    • Property 4: Graceful Degradation bei LLM-Fehler
    • Validates: Requirements 3.5, 5.6

• 4. Checkpoint - Ensure all tests pass

  • Ensure all tests pass, ask the user if questions arise.

• [-] 5. Update Matcher to unconditionally build BM25 index

  • 5.1 Remove use_bm25_search conditional in Matcher.match

    • Remove if self._sim.use_bm25_search guard around BM25 index construction
    • Always build global BM25 index when filtered is non-empty
    • Requirements: 2.6

  • 5.2 Write property test for unconditional BM25 index building

    • Property 2: Matcher baut BM25-Index bedingungslos
    • Validates: Requirements 2.6

• 6. Update MCP Server to remove embedding preload and wire new components

  • 6.1 Remove embedding preload infrastructure from mcp_server.py

    • Remove _startup_preload_embeddings function
    • Remove _deferred_preload function and _preload_started flag
    • Remove _ensure_preloaded function
    • Remove all await _ensure_preloaded() calls in tool handlers
    • Remove EmbeddingCache import and instantiation
    • Requirements: 4.1, 4.2, 4.4, 4.5, 6.12

  • 6.2 Update component wiring in build_server

    • Construct AzureOpenAIClient with new simplified signature (no embedding params)
    • Construct SimilarityEngine with new signature (client, cost_tracker, use_auto_tagging, auto_tagger)
    • Construct VocabularyCache with new signature (db, client)
    • Requirements: 6.7

  • 6.3 Update infer_primary_role tool handler

    • Remove embedding generation step
    • Call vocab_cache.infer_primary_role(task_text=text) directly with task text
    • Requirements: 3.9

  • 6.4 Remove or simplify validate_task_requirements tool if it depends on embeddings

    • Remove embedding-based infer_competences usage
    • Either remove the tool entirely or replace with a DB-field-only version
    • Requirements: 6.5

• 7. Simplify config.py and config.toml

  • 7.1 Remove embedding-related dataclasses and fields from config.py

    • Delete EmbeddingCacheConfig dataclass
    • Delete InferenceConfig dataclass
    • Remove from AzureOpenAIConfig: embedding_deployment, embedding_batch_size
    • Remove from SimilarityConfig: embedding_model, embedding_dimensions, strategy, use_bm25_search
    • Remove inference field from MatchingConfig
    • Remove embedding_cache field from AppConfig
    • Requirements: 2.1, 2.2, 6.8, 6.9, 6.10

  • 7.2 Update load_config / _parse_azure_openai validation

    • Remove AZURE_OPENAI_EMBEDDING_API_KEY environment variable check
    • Make AZURE_OPENAI_LLM_API_KEY always required
    • Make chat_deployment required
    • Remove embedding_dimensions == 3072 validation
    • Remove strategy validation
    • Remove [matching.inference] parsing
    • Remove [embedding_cache] parsing
    • Requirements: 2.3, 2.4

  • 7.3 Update config.toml and config.toml.example

    • Remove [embedding_cache] section
    • Remove from [matching.similarity]: embedding_model, embedding_dimensions, strategy, use_bm25_search
    • Remove from [azure_openai]: embedding_deployment, embedding_batch_size
    • Remove [matching.inference] section
    • Requirements: 6.11

• 8. Delete EmbeddingCache module and related tests

  • 8.1 Delete src/teamlandkarte_mcp/cache/embedding_cache.py

    • Requirements: 6.6

  • 8.2 Remove or update tests that reference embedding functionality

    • Delete tests for EmbeddingCache
    • Update tests for SimilarityEngine to use new constructor
    • Update tests for VocabularyCache to use new constructor
    • Update tests for AzureOpenAIClient to use new constructor
    • Remove any test fixtures that create embedding mocks
    • Requirements: 6.6

• 9. Final checkpoint - Ensure all tests pass

  • Ensure all tests pass, ask the user if questions arise.

Notes

• Tasks marked with * are optional and can be skipped for faster MVP
• Each task references specific requirements for traceability
• Checkpoints ensure incremental validation
• Property tests validate universal correctness properties from the design document
• Order ensures no broken intermediate states: new LLM methods added before old embedding paths removed