Orchestrator/bahn/teamlandkarte-mcp/openspec/changes/accelerate-embedding-similarity/design.md

Design: Accelerate Embedding Similarity

Context

The server uses Azure OpenAI embeddings for competence and role similarity. The matching pipeline can be slow because embeddings are currently requested on-demand and repeatedly from inner loops.

The system already has a persistent SQLite embedding cache (EmbeddingCache). However, without bulk prefetch and without a run-scoped memoization layer, the similarity engine still:

• repeatedly normalizes and hashes text keys
• performs many repeated SQLite reads
• performs sequential Azure calls for cache misses

Goals

• Bulk prefetch embeddings for a matching run.
• Maintain deterministic behavior and strict error semantics.
• Preserve existing similarity contracts.

Architecture Changes

Components impacted

• SimilarityEngine (src/teamlandkarte_mcp/matching/similarity.py)
  • Add instance variable for in-memory cache (lives with the engine; not created/cleared per invocation)
  • Add public method prefetch_embeddings(required_texts, candidate_texts, required_roles, candidate_roles) -> dict[str, list[float]]
• AzureOpenAIClient (src/teamlandkarte_mcp/azure/openai_client.py)
  • Modify get_embeddings_batch() to use true batch API with input=[...]
• CostTracker (src/teamlandkarte_mcp/azure/cost_tracker.py)
  • May need adjustment to log batch requests (log once per chunk with count)
• EmbeddingCache (no schema changes expected)

Data flow (proposed)

1. Collect all texts that will be embedded in the current run.
   • This is a single global collection pass over the inputs and all free capacities (roles + competences), not a per-person loop that embeds incrementally.
2. Normalize + deduplicate using existing _normalize_text() logic.
   • Skip empty/whitespace-only texts
   • Emit a Python logger warning if all candidate competences normalize to empty
3. Resolve embeddings via:
   • in-memory cache (instance variable in SimilarityEngine)
   • SQLite cache (EmbeddingCache)
   • Azure embeddings API (batch only the cache-missing texts, chunked by batch size)
4. Compute cosine similarities locally using the prefetched mapping.

Normalization and keys

Current cache key behavior:

• normalization: trim + collapse whitespace
• key: SHA256 of model|dims|normalized(text).lower()

This proposal keeps the same normalization and key derivation to avoid invalidating the on-disk cache.

Batch embeddings API

The OpenAI/Azure embeddings API supports embedding multiple inputs per request (input=[...]).

Implementation requirements:

• preserve stable mapping from input texts → returned embedding vectors
  • rely on index field in API response if available
  • otherwise assume stable ordering
• chunk large requests to avoid request size limits (configurable batch size, default 128)
• strict failure semantics:
  • entire matching operation fails immediately on error
  • log the failing chunk input list via Python logging to aid diagnosis
• cost tracking: log once per batch chunk with a count of embeddings requested

Trade-offs

• Batch calls reduce network requests dramatically but may increase blast radius: one failing request could affect multiple inputs.
  • Mitigation: keep retry logic at the chunk level.
• Prefetch requires holding larger embedding dictionaries in memory.
  • Mitigation: scope to a single invocation, and only store vectors required for that run.

Test Strategy

• Unit test prefetch/dedup: ensure the Azure client is called once per unique normalized text (only for cache misses).
• Unit test cache layering: ensure hits are served from in-memory cache (first), then SQLite, and do not call Azure.
• Unit test chunking: ensure multiple Azure calls are made when len(cache_missing_texts) > batch_size.
• Unit test empty input handling: verify warning when all candidate competences normalize to empty; verify prefetch skips empty texts.
• Unit test error identification: verify that batch failures report which specific text(s) caused the error.
• Integration test (manual/gated): validate real Azure batch API behavior with live credentials.