# Design: Capacity→Task Matching and Embedding-based Inference

## Context

The system was originally optimized for task→capacity matching and relied on Azure chat completions for role/requirements extraction.

To enable a symmetrical workflow and reduce latency/cost, this change:

- adds tools to browse capacities and match a capacity against open tasks
- reworks role and competence inference to use embeddings only (Data Lake vocabularies)
- preloads and caches role/competence vocab embeddings on startup (with runtime on-demand fallback)
- preloads and caches task text embeddings by `task_id`

## Key decisions

### A. Vocabulary-driven inference (roles + competences)

- Maintain two vocabularies sourced from the Data Lake:
  - roles: `teamlandkarte_v_capacity_roles_latest` filtered by `active=true` and `staffing_board_relevant=true`
  - competences: unique values from `beschaffungstool_kmp_skill_latest.skillname__c`
- Generate embeddings for these vocabularies and store them in the local SQLite embedding cache.

### B. Task text embedding stored by task_id

To minimize embedding transaction costs for repeated analysis of the same task:

- build `task_text = f"{title}\n\n{description}"` (title optional)
- embed `task_text` once
- store that embedding under a stable key derived from `task_id` (and embedding model/dims)

**Explicit cache key formats**:

```python
# Role vocabulary entries
key = f"role_vocab:{model}:{dims}:{normalized_role_name}"

# Competence vocabulary entries
key = f"comp_vocab:{model}:{dims}:{normalized_competence_name}"

# Task text embeddings by task_id
key = f"task_id:{model}:{dims}:{task_id}"
```

Where:
- `model` = embedding model name (e.g., `text-embedding-3-large`)
- `dims` = embedding dimensions (e.g., `3072`)
- `normalized_role_name` = normalized role text (trimmed, whitespace-collapsed, lowercased)
- `normalized_competence_name` = normalized competence text (trimmed, whitespace-collapsed, lowercased)
- `task_id` = database task ID (string or integer, converted to string)

### C. Inference logic (embedding similarity)

- Role inference:
  - compute cosine similarity between `task_text_embedding` and all role vocabulary embeddings
  - return the single best matching role with similarity score as a Markdown table: `Role | Similarity`
- Competence inference (for validation and display):
  - compute cosine similarity between `task_text_embedding` and all competence vocabulary embeddings
  - return top-N competences with similarity scores as a Markdown table: `Competence | Similarity` (descending order)

**Configuration** (under `[matching.inference]`):
```toml
[matching.inference]
max_competences = 8  # default: 8; omit to disable limit (no max)
min_similarity = 0.4  # default: 0.4; omit to disable threshold filtering
```

**Configuration rules**:
- If `max_competences` is not specified: no limit on number of competences returned.
- If `min_similarity` is not specified: no threshold filtering applied (include all).
- At least one of `max_competences` or `min_similarity` MUST be specified in config to avoid unbounded results.
- Selection rule: if both are set, return top `max_competences` among those with similarity `>= min_similarity`.

### D. Component structure for vocabulary preload

**Component: VocabularyCache**
- **Location**: `src/teamlandkarte_mcp/matching/vocabulary.py` (new module)
- **Responsibilities**:
  - Fetch role + competence vocabularies from Data Lake via database client
  - Normalize and deduplicate vocabulary entries
  - Embed missing entries using batch + chunking via `SimilarityEngine.prefetch_embeddings(...)`
  - Store embeddings in SQLite cache with stable keys
  - Provide in-memory memoization for fast repeated inference during tool execution
- **Initialization**: Called from `build_server()` after DB client + similarity engine initialization
- **Refresh strategy**: On startup only; no TTL-based refresh during runtime

**Task embedding preload**:
- Preload **all open tasks** on startup (no limit).
- Build `task_text = title + "\n\n" + description` for each task (title optional).
- Embed only cache-missing `task_id` vectors (check SQLite cache first).
- Store into SQLite embedding cache with key format: `task_id:{model}:{dims}:{task_id}`.
- Runtime fallback: if a task embedding is missing during tool execution, embed on-demand (strict semantics).

### E. Determinism and strict failure semantics

- Runs are deterministic given fixed models and cached embeddings.
- Azure embedding failures fail the operation immediately.
- The existing global embedding improvements (dedup + batch + layered caches) should be reused.

## Data flow

### 1) Vocabulary refresh

1. fetch role list + competence list from Data Lake
2. normalize/deduplicate
3. embed missing entries via Azure batch API
4. store embeddings in SQLite cache

### 2) Task validation

1. fetch task (title+description) + DB skills
2. compute/retrieve `task_text_embedding` by task_id
3. infer:
   - top-K roles
   - top-K competences
4. compare:
   - DB skills vs inferred competences (plus similarity thresholds)
   - dates (unchanged)

### 3) Capacity→task matching

1. load the target capacity (role + competences + availability)
2. list open tasks
3. compute/retrieve `task_text_embedding` for each task
4. compute overall similarity (role + competences) between the capacity and each task
5. categorize + return table-first results (same output conventions as existing matching tools)

## Alternatives considered

- Keep chat API for role inference: rejected (latency/cost + redundant).
- Use raw string matching for vocab mapping: rejected (less robust than embeddings).

## Risks

- Vocabulary size (competences) may be large → requires batch embedding + caching and careful refresh strategy.
- Model updates can change nearest neighbors → mitigate with pinned model/deployment and cache key including model/dims.