# Change Proposal: Accelerate Embedding Prefetch and Similarity Computation

- **Change ID**: `accelerate-embedding-similarity`
- **Status**: Proposed
- **Target**: `teamlandkarte-mcp`
- **Author**: Thomas Handke
- **Date**: 2026-02-14

## Summary

Speed up `find_matching_capacities(...)` by reducing embedding API calls and repeated cache lookups.

This change introduces three complementary optimizations:

1. **Dedup + bulk prefetch** of all embedding texts needed in a matching run (required competences, candidate competences, and roles).
2. A **per-run in-memory cache** layered on top of the existing SQLite embedding cache to avoid repeated SQLite reads within a single run.
3. Support for **true Azure Embeddings batch requests** (single request with multiple inputs), including chunking, while preserving deterministic error semantics.

## Motivation

The current implementation calls embeddings repeatedly inside inner loops. While a persistent SQLite cache exists, repeated calls still incur overhead:

- many redundant `cache.get(...)` reads
- repeated normalization/keying for identical strings
- unnecessary Azure round-trips when multiple missing terms are encountered and requests are performed sequentially

The result is that `find_matching_capacities(...)` can be slow, and while the second run is faster (due to cache hits), it can still take noticeable time.

## Goals

1. Ensure each unique text is embedded at most once per run (network and cache).
2. Prefetch embeddings **before** similarity computation so inner loops only perform vector math.
3. Enable Azure embeddings API batch calls to reduce network round-trips.
4. Preserve current similarity output contracts and scoring behavior.
5. Maintain explicit, deterministic failure semantics (no heuristics).

## Non-Goals

- Changing match scoring weights or category thresholds.
- Changing the similarity strategy semantics ("per_skill" vs "aggregate").
- Changing the embedding model or dimensionality.
- Introducing new external dependencies unless necessary.

## Proposed Changes

### 1) Bulk prefetch & dedup in `SimilarityEngine`

#### Current behavior (problem)

`SimilarityEngine` currently embeds terms on demand via `_embed(...)`. For large candidate lists, this forces a high number of calls to `_embed`, and repeated SQLite reads.

#### New behavior

Add a prefetch step that:

- collects all texts required to compute similarity for a run (competences **and roles**)
  - **Important:** this collection is performed **across the entire candidate set** (i.e., roles and competences of **all free capacities at once**), not person-by-person.
- normalizes for cache key stability
- deduplicates texts
- fetches all embeddings from caches/API in bulk
- returns a mapping from normalized text → embedding vector

Then similarity computation uses the prefetched mapping (pure Python vector math).

### 2) Per-run in-memory cache

Add a lightweight in-memory dict for the duration of a single matching invocation.

- key: `(model, dimensions, normalized_text)` or reuse the computed SHA256 cache key
- value: `list[float]`

This prevents repeated SQLite lookups for frequently used items (common competences, repeated role strings, etc.).

### 3) True Azure embeddings batch calls

Change `AzureOpenAIClient.get_embeddings_batch(...)` to use the Azure embeddings API with `input=[...]` to embed multiple texts in a single request.

Key details:

- supports chunking with a **configurable** batch size
- preserves stable output ordering and deterministic error semantics
- continues to log costs via `CostTracker`

If any item fails, the batch call should raise `AzureAPIError` (no fallback).

## Decisions (resolved)

- **Scope**: bulk prefetch includes **competences and roles**.
- **Batching**: Azure embeddings requests are made in chunks with a **configurable** batch size.
- **Normalization**: keep the current text normalization and cache-key behavior.
- **Concurrency**: process chunks **sequentially** (no parallel calls).
- **In-memory cache lifecycle**: implemented as a new instance variable in `SimilarityEngine` and **not** created/cleaned per matching invocation.
- **Prefetch API**: new public method `prefetch_embeddings(required_texts, candidate_texts, required_roles, candidate_roles)` that returns a mapping from normalized text → embedding vector.
- **Azure batch response ordering**: rely on the `index` field in the response if available, otherwise assume stable ordering.
- **Error handling**: entire matching operation fails immediately; attempt to identify and report which specific text(s) caused the failure.
- **Cost tracking**: log once per batch chunk with a count.
- **Empty inputs**:
  - emit a **Python logger warning** when all candidate competences normalize to empty strings
  - prefetch skips empty/whitespace-only texts
- **Cache miss batching**: batch-request only the missing embeddings (after resolving cache hits from SQLite).

## Configuration

Introduce (or reuse if already present) one configuration knob:

- `azure_openai.embedding_batch_size` (integer)
  - default: `128` (conservative)
  - no enforced maximum value
  - used to chunk `input=[...]` for Azure embeddings batch requests
  - if not specified in `config.toml`, the default value is used

## Success Criteria

- A single `find_matching_capacities(...)` run should perform at most one embedding generation per unique term (after cache resolution).
- Only cache-missing embeddings are batch-requested from Azure (cache hits are not re-requested).
- For warmed cache runs, no Azure embedding calls should occur.
- Unit tests cover dedup/prefetch, cache layering, and batch request behavior.
- Integration tests (manual/gated) verify real Azure batch API behavior.

## Rollout / Compatibility

- No configuration changes are strictly required; the baseline refactor should work with existing `config.toml`.
- If new config knobs are added (e.g., batch size), defaults should preserve behavior.