Orchestrator/bahn/teamlandkarte-mcp/openspec/changes/accelerate-embedding-similarity/tasks.md

Tasks: Accelerate Embedding Similarity

Change: accelerate-embedding-similarity

Status: Approved / Implemented

Overview

Reduce runtime of find_matching_capacities(...) by implementing:

• embedding text deduplication + global bulk prefetch (roles + competences for all free capacities at once, not person-by-person)
• in-memory embedding memoization
• true Azure embeddings batch requests with chunking

Task Breakdown

Phase 1: Design & Validation Plan

• 1.1 Identify hot paths during matching (where _embed(...) is called most).
• 1.2 Confirm normalization + cache key rules are unchanged and documented.
• 1.3 Define expected call-count behavior (upper bounds):
  • per run: max 1 Azure embedding per unique normalized text (subject to chunking, after cache resolution)
  • per run: max 1 SQLite read per unique normalized text (after in-memory cache check)

Phase 2: Global Bulk Prefetch in SimilarityEngine

• 2.1 Add a new instance variable _run_cache: dict[str, list[float]] to SimilarityEngine (in-memory cache; not created/cleared per matching invocation).
• 2.2 Implement a public method prefetch_embeddings(required_texts: list[str], candidate_texts: list[str], required_roles: list[str], candidate_roles: list[str]) -> dict[str, list[float]]:
  • collect all input texts (required competences, required roles, all candidate competences, all candidate roles)
  • normalize + deduplicate using existing _normalize_text() logic
  • skip empty/whitespace-only texts
  • emit a Python logger warning if all candidate competences normalize to empty
• 2.3 Implement prefetch resolution logic:
  1. check in-memory cache (_run_cache)
  2. check SQLite embedding cache
  3. batch-request only cache-missing texts from Azure API (chunked by batch size)
• 2.4 Refactor _per_skill_similarity to use prefetched embeddings only (no embedding calls inside inner loops).
• 2.5 Refactor _aggregate_similarity to use prefetched embeddings only.
• 2.6 Ensure output format is unchanged and remains deterministic.

Phase 3: In-memory Run Cache

• 3.1 Verify _run_cache is an instance-level cache that does not require per-invocation clearing.
• 3.2 Add tests ensuring repeated embedding lookups for the same text:
  • are served from _run_cache after first lookup
  • do not call SQLite/API multiple times for the same text

Phase 4: Azure Batch Embeddings Support

• 4.1 Modify AzureOpenAIClient.get_embeddings_batch() to use Azure embeddings API with input=[...].
• 4.2 Implement chunking using azure_openai.embedding_batch_size (default: 128, configurable, no enforced maximum).
• 4.3 Preserve ordering using index field in API response if available, otherwise assume stable ordering.
• 4.4 Ensure strict error semantics:
  • entire matching operation fails immediately on error
  • log the failing chunk input list via Python logging
• 4.5 Update CostTracker to log batch requests: log once per chunk with count of embeddings requested.

Phase 5: Configuration

• 5.1 Extend config model to include azure_openai.embedding_batch_size (integer, default: 128, no enforced maximum).
• 5.2 Update config.toml.example with the new setting and comments.
• 5.3 Ensure fallback: if embedding_batch_size is not specified in config.toml, use the default value (128).

Phase 6: Tests

• 6.1 Unit tests: global prefetch/dedup
  • verify dedup across required + all candidates (incl. roles)
  • verify embedding client is called once per unique normalized text (only for cache misses)
  • verify empty/whitespace-only texts are skipped
  • verify a Python logger warning is emitted when all candidate competences normalize to empty
• 6.2 Unit tests: cache layering
  • verify in-memory _run_cache prevents repeated SQLite reads within a run
  • verify SQLite hits prevent Azure calls
• 6.3 Unit tests: Azure batch request behavior (mocked)
  • verifies batch call for N texts when N <= batch_size
  • verifies chunking when N > batch_size (multiple batch calls)
  • verifies output ordering is stable (using index field or assuming stable order)
  • verifies strict error propagation
  • verifies the failing chunk input list is logged
  • verifies cost tracking logs once per chunk with count
• 6.4 Integration test (manual/gated only): real Azure batch embeddings
  • confirm request uses batch input=[...] semantics
  • ensure behavior matches expected semantics for a representative set of texts

Phase 7: Documentation + Assistant Prompt Updates

• 7.1 Update README.md to describe the optimization at a high level and mention the new batch size knob.
• 7.2 Update docs/assistant_system_prompt.md to reflect:
  • embeddings are prefetched globally per run
  • batching/chunking is used and configurable
  • failure semantics remain strict (no heuristics)
• 7.3 Update any Azure setup docs if needed (e.g., docs/azure_openai_setup.md) to mention embedding batch behavior and tuning.

Phase 8: Quality Gates / Release Hygiene

• 8.1 Run unit tests: uv run pytest -m "not integration" -q
• 8.2 Run Ruff and mypy via uv.
• 8.3 (Manual/gated only) Run integration tests (Azure) to validate real batch calls.
• 8.4 Update change docs status/checklists and ensure OpenSpec validates (openspec validate accelerate-embedding-similarity --strict).