ankn
a5f8fb49ab
Bahn: aisupport, Analyse-O2C-C2S, awesome-bahn-mcp-servers, beam-mcp,
Confluence_Bot, db-planet-mcp-server, O2C-Harness, project-audit,
Projekt-KIQ-HP, teamlandkarte-mcp
Dhive: Jury-Voting
Privat: CV, NoteGraph (NOTE: NoteGraph needs complete redo after consolidation)
Shared: AI-Orchestrator, OrgMyLife, power_skills_and_more
Shared/references: symphony (read-only)
Bahn repos remain available as independent remotes - this monorepo
pulls them in via subtree, the originals are untouched.
85 lines
4.1 KiB
Markdown
85 lines
4.1 KiB
Markdown
# Design: Accelerate Embedding Similarity
|
|
|
|
## Context
|
|
|
|
The server uses Azure OpenAI embeddings for competence and role similarity. The matching pipeline can be slow because embeddings are currently requested on-demand and repeatedly from inner loops.
|
|
|
|
The system already has a persistent SQLite embedding cache (`EmbeddingCache`). However, without bulk prefetch and without a run-scoped memoization layer, the similarity engine still:
|
|
|
|
- repeatedly normalizes and hashes text keys
|
|
- performs many repeated SQLite reads
|
|
- performs sequential Azure calls for cache misses
|
|
|
|
## Goals
|
|
|
|
- Bulk prefetch embeddings for a matching run.
|
|
- Maintain deterministic behavior and strict error semantics.
|
|
- Preserve existing similarity contracts.
|
|
|
|
## Architecture Changes
|
|
|
|
### Components impacted
|
|
|
|
- `SimilarityEngine` (`src/teamlandkarte_mcp/matching/similarity.py`)
|
|
- Add instance variable for in-memory cache (lives with the engine; not created/cleared per invocation)
|
|
- Add public method `prefetch_embeddings(required_texts, candidate_texts, required_roles, candidate_roles) -> dict[str, list[float]]`
|
|
- `AzureOpenAIClient` (`src/teamlandkarte_mcp/azure/openai_client.py`)
|
|
- Modify `get_embeddings_batch()` to use true batch API with `input=[...]`
|
|
- `CostTracker` (`src/teamlandkarte_mcp/azure/cost_tracker.py`)
|
|
- May need adjustment to log batch requests (log once per chunk with count)
|
|
- `EmbeddingCache` (no schema changes expected)
|
|
|
|
### Data flow (proposed)
|
|
|
|
1. **Collect** all texts that will be embedded in the current run.
|
|
- This is a **single global collection pass** over the inputs and **all free capacities** (roles + competences), not a per-person loop that embeds incrementally.
|
|
2. **Normalize + deduplicate** using existing `_normalize_text()` logic.
|
|
- Skip empty/whitespace-only texts
|
|
- Emit a **Python logger warning** if all candidate competences normalize to empty
|
|
3. **Resolve embeddings** via:
|
|
- in-memory cache (instance variable in `SimilarityEngine`)
|
|
- SQLite cache (`EmbeddingCache`)
|
|
- Azure embeddings API (**batch only the cache-missing texts**, **chunked** by batch size)
|
|
4. **Compute** cosine similarities locally using the prefetched mapping.
|
|
|
|
### Normalization and keys
|
|
|
|
Current cache key behavior:
|
|
|
|
- normalization: trim + collapse whitespace
|
|
- key: SHA256 of `model|dims|normalized(text).lower()`
|
|
|
|
This proposal keeps the same normalization and key derivation to avoid invalidating the on-disk cache.
|
|
|
|
## Batch embeddings API
|
|
|
|
The OpenAI/Azure embeddings API supports embedding multiple inputs per request (`input=[...]`).
|
|
|
|
Implementation requirements:
|
|
|
|
- preserve stable mapping from input texts → returned embedding vectors
|
|
- rely on `index` field in API response if available
|
|
- otherwise assume stable ordering
|
|
- chunk large requests to avoid request size limits (configurable batch size, default 128)
|
|
- strict failure semantics:
|
|
- entire matching operation fails immediately on error
|
|
- log the failing **chunk input list** via Python logging to aid diagnosis
|
|
- cost tracking: log once per batch chunk with a count of embeddings requested
|
|
|
|
## Trade-offs
|
|
|
|
- Batch calls reduce network requests dramatically but may increase blast radius: one failing request could affect multiple inputs.
|
|
- Mitigation: keep retry logic at the chunk level.
|
|
- Prefetch requires holding larger embedding dictionaries in memory.
|
|
- Mitigation: scope to a single invocation, and only store vectors required for that run.
|
|
|
|
## Test Strategy
|
|
|
|
- Unit test prefetch/dedup: ensure the Azure client is called once per unique normalized text (only for cache misses).
|
|
- Unit test cache layering: ensure hits are served from in-memory cache (first), then SQLite, and do not call Azure.
|
|
- Unit test chunking: ensure multiple Azure calls are made when `len(cache_missing_texts) > batch_size`.
|
|
- Unit test empty input handling: verify warning when all candidate competences normalize to empty; verify prefetch skips empty texts.
|
|
- Unit test error identification: verify that batch failures report which specific text(s) caused the error.
|
|
- Integration test (manual/gated): validate real Azure batch API behavior with live credentials.
|
|
|