Orchestrator/bahn/teamlandkarte-mcp/openspec/changes/accelerate-embedding-similarity/proposal.md

Change Proposal: Accelerate Embedding Prefetch and Similarity Computation

• Change ID: accelerate-embedding-similarity
• Status: Proposed
• Target: teamlandkarte-mcp
• Author: Thomas Handke
• Date: 2026-02-14

Summary

Speed up find_matching_capacities(...) by reducing embedding API calls and repeated cache lookups.

This change introduces three complementary optimizations:

1. Dedup + bulk prefetch of all embedding texts needed in a matching run (required competences, candidate competences, and roles).
2. A per-run in-memory cache layered on top of the existing SQLite embedding cache to avoid repeated SQLite reads within a single run.
3. Support for true Azure Embeddings batch requests (single request with multiple inputs), including chunking, while preserving deterministic error semantics.

Motivation

The current implementation calls embeddings repeatedly inside inner loops. While a persistent SQLite cache exists, repeated calls still incur overhead:

• many redundant cache.get(...) reads
• repeated normalization/keying for identical strings
• unnecessary Azure round-trips when multiple missing terms are encountered and requests are performed sequentially

The result is that find_matching_capacities(...) can be slow, and while the second run is faster (due to cache hits), it can still take noticeable time.

Goals

1. Ensure each unique text is embedded at most once per run (network and cache).
2. Prefetch embeddings before similarity computation so inner loops only perform vector math.
3. Enable Azure embeddings API batch calls to reduce network round-trips.
4. Preserve current similarity output contracts and scoring behavior.
5. Maintain explicit, deterministic failure semantics (no heuristics).

Non-Goals

• Changing match scoring weights or category thresholds.
• Changing the similarity strategy semantics ("per_skill" vs "aggregate").
• Changing the embedding model or dimensionality.
• Introducing new external dependencies unless necessary.

Proposed Changes

1) Bulk prefetch & dedup in SimilarityEngine

Current behavior (problem)

SimilarityEngine currently embeds terms on demand via _embed(...). For large candidate lists, this forces a high number of calls to _embed, and repeated SQLite reads.

New behavior

Add a prefetch step that:

• collects all texts required to compute similarity for a run (competences and roles)
  • Important: this collection is performed across the entire candidate set (i.e., roles and competences of all free capacities at once), not person-by-person.
• normalizes for cache key stability
• deduplicates texts
• fetches all embeddings from caches/API in bulk
• returns a mapping from normalized text → embedding vector

Then similarity computation uses the prefetched mapping (pure Python vector math).

2) Per-run in-memory cache

Add a lightweight in-memory dict for the duration of a single matching invocation.

• key: (model, dimensions, normalized_text) or reuse the computed SHA256 cache key
• value: list[float]

This prevents repeated SQLite lookups for frequently used items (common competences, repeated role strings, etc.).

3) True Azure embeddings batch calls

Change AzureOpenAIClient.get_embeddings_batch(...) to use the Azure embeddings API with input=[...] to embed multiple texts in a single request.

Key details:

• supports chunking with a configurable batch size
• preserves stable output ordering and deterministic error semantics
• continues to log costs via CostTracker

If any item fails, the batch call should raise AzureAPIError (no fallback).

Decisions (resolved)

• Scope: bulk prefetch includes competences and roles.
• Batching: Azure embeddings requests are made in chunks with a configurable batch size.
• Normalization: keep the current text normalization and cache-key behavior.
• Concurrency: process chunks sequentially (no parallel calls).
• In-memory cache lifecycle: implemented as a new instance variable in SimilarityEngine and not created/cleaned per matching invocation.
• Prefetch API: new public method prefetch_embeddings(required_texts, candidate_texts, required_roles, candidate_roles) that returns a mapping from normalized text → embedding vector.
• Azure batch response ordering: rely on the index field in the response if available, otherwise assume stable ordering.
• Error handling: entire matching operation fails immediately; attempt to identify and report which specific text(s) caused the failure.
• Cost tracking: log once per batch chunk with a count.
• Empty inputs:
  • emit a Python logger warning when all candidate competences normalize to empty strings
  • prefetch skips empty/whitespace-only texts
• Cache miss batching: batch-request only the missing embeddings (after resolving cache hits from SQLite).

Configuration

Introduce (or reuse if already present) one configuration knob:

• azure_openai.embedding_batch_size (integer)
  • default: 128 (conservative)
  • no enforced maximum value
  • used to chunk input=[...] for Azure embeddings batch requests
  • if not specified in config.toml, the default value is used

Success Criteria

• A single find_matching_capacities(...) run should perform at most one embedding generation per unique term (after cache resolution).
• Only cache-missing embeddings are batch-requested from Azure (cache hits are not re-requested).
• For warmed cache runs, no Azure embedding calls should occur.
• Unit tests cover dedup/prefetch, cache layering, and batch request behavior.
• Integration tests (manual/gated) verify real Azure batch API behavior.

Rollout / Compatibility

• No configuration changes are strictly required; the baseline refactor should work with existing config.toml.
• If new config knobs are added (e.g., batch size), defaults should preserve behavior.