# Tasks: Accelerate Embedding Similarity

> Change: `accelerate-embedding-similarity`
>
> Status: **Approved / Implemented**

## Overview

Reduce runtime of `find_matching_capacities(...)` by implementing:

- embedding text deduplication + **global** bulk prefetch (roles + competences for **all free capacities at once**, not person-by-person)
- in-memory embedding memoization
- true Azure embeddings batch requests with chunking

## Task Breakdown

### Phase 1: Design & Validation Plan

- [x] 1.1 Identify hot paths during matching (where `_embed(...)` is called most).
- [x] 1.2 Confirm normalization + cache key rules are unchanged and documented.
- [x] 1.3 Define expected call-count behavior (upper bounds):
  - per run: max 1 Azure embedding per unique normalized text (subject to chunking, after cache resolution)
  - per run: max 1 SQLite read per unique normalized text (after in-memory cache check)

### Phase 2: Global Bulk Prefetch in `SimilarityEngine`

- [x] 2.1 Add a new instance variable `_run_cache: dict[str, list[float]]` to `SimilarityEngine` (in-memory cache; **not** created/cleared per matching invocation).
- [x] 2.2 Implement a public method `prefetch_embeddings(required_texts: list[str], candidate_texts: list[str], required_roles: list[str], candidate_roles: list[str]) -> dict[str, list[float]]`:
  - collect all input texts (required competences, **required roles**, all candidate competences, **all candidate roles**)
  - normalize + deduplicate using existing `_normalize_text()` logic
  - skip empty/whitespace-only texts
  - emit a **Python logger warning** if all candidate competences normalize to empty
- [x] 2.3 Implement prefetch resolution logic:
  1) check in-memory cache (`_run_cache`)
  2) check SQLite embedding cache
  3) batch-request only cache-missing texts from Azure API (chunked by batch size)
- [x] 2.4 Refactor `_per_skill_similarity` to use prefetched embeddings only (no embedding calls inside inner loops).
- [x] 2.5 Refactor `_aggregate_similarity` to use prefetched embeddings only.
- [x] 2.6 Ensure output format is unchanged and remains deterministic.

### Phase 3: In-memory Run Cache

- [x] 3.1 Verify `_run_cache` is an instance-level cache that does not require per-invocation clearing.
- [x] 3.2 Add tests ensuring repeated embedding lookups for the same text:
  - are served from `_run_cache` after first lookup
  - do not call SQLite/API multiple times for the same text

### Phase 4: Azure Batch Embeddings Support

- [x] 4.1 Modify `AzureOpenAIClient.get_embeddings_batch()` to use Azure embeddings API with `input=[...]`.
- [x] 4.2 Implement chunking using `azure_openai.embedding_batch_size` (default: 128, configurable, no enforced maximum).
- [x] 4.3 Preserve ordering using `index` field in API response if available, otherwise assume stable ordering.
- [x] 4.4 Ensure strict error semantics:
  - entire matching operation fails immediately on error
  - log the failing **chunk input list** via Python logging
- [x] 4.5 Update `CostTracker` to log batch requests: log once per chunk with count of embeddings requested.

### Phase 5: Configuration

- [x] 5.1 Extend config model to include `azure_openai.embedding_batch_size` (integer, default: 128, no enforced maximum).
- [x] 5.2 Update `config.toml.example` with the new setting and comments.
- [x] 5.3 Ensure fallback: if `embedding_batch_size` is not specified in `config.toml`, use the default value (128).

### Phase 6: Tests

- [x] 6.1 Unit tests: global prefetch/dedup
  - verify dedup across required + all candidates (incl. roles)
  - verify embedding client is called once per unique normalized text (only for cache misses)
  - verify empty/whitespace-only texts are skipped
  - verify a Python logger warning is emitted when all candidate competences normalize to empty
- [x] 6.2 Unit tests: cache layering
  - verify in-memory `_run_cache` prevents repeated SQLite reads within a run
  - verify SQLite hits prevent Azure calls
- [x] 6.3 Unit tests: Azure batch request behavior (mocked)
  - verifies batch call for N texts when `N <= batch_size`
  - verifies chunking when `N > batch_size` (multiple batch calls)
  - verifies output ordering is stable (using `index` field or assuming stable order)
  - verifies strict error propagation
  - verifies the failing chunk input list is logged
  - verifies cost tracking logs once per chunk with count
- [x] 6.4 Integration test (manual/gated only): real Azure batch embeddings
  - confirm request uses batch `input=[...]` semantics
  - ensure behavior matches expected semantics for a representative set of texts

### Phase 7: Documentation + Assistant Prompt Updates

- [x] 7.1 Update `README.md` to describe the optimization at a high level and mention the new batch size knob.
- [x] 7.2 Update `docs/assistant_system_prompt.md` to reflect:
  - embeddings are prefetched globally per run
  - batching/chunking is used and configurable
  - failure semantics remain strict (no heuristics)
- [x] 7.3 Update any Azure setup docs if needed (e.g., `docs/azure_openai_setup.md`) to mention embedding batch behavior and tuning.

### Phase 8: Quality Gates / Release Hygiene

- [x] 8.1 Run unit tests: `uv run pytest -m "not integration" -q`
- [x] 8.2 Run Ruff and mypy via `uv`.
- [ ] 8.3 (Manual/gated only) Run integration tests (Azure) to validate real batch calls.
- [x] 8.4 Update change docs status/checklists and ensure OpenSpec validates (`openspec validate accelerate-embedding-similarity --strict`).