Migrate all repos into monorepo context folders
Bahn: aisupport, Analyse-O2C-C2S, awesome-bahn-mcp-servers, beam-mcp,
Confluence_Bot, db-planet-mcp-server, O2C-Harness, project-audit,
Projekt-KIQ-HP, teamlandkarte-mcp
Dhive: Jury-Voting
Privat: CV, NoteGraph (NOTE: NoteGraph needs complete redo after consolidation)
Shared: AI-Orchestrator, OrgMyLife, power_skills_and_more
Shared/references: symphony (read-only)
Bahn repos remain available as independent remotes - this monorepo
pulls them in via subtree, the originals are untouched.
This commit is contained in:
+135
@@ -0,0 +1,135 @@
|
||||
# Design: Capacity→Task Matching and Embedding-based Inference
|
||||
|
||||
## Context
|
||||
|
||||
The system was originally optimized for task→capacity matching and relied on Azure chat completions for role/requirements extraction.
|
||||
|
||||
To enable a symmetrical workflow and reduce latency/cost, this change:
|
||||
|
||||
- adds tools to browse capacities and match a capacity against open tasks
|
||||
- reworks role and competence inference to use embeddings only (Data Lake vocabularies)
|
||||
- preloads and caches role/competence vocab embeddings on startup (with runtime on-demand fallback)
|
||||
- preloads and caches task text embeddings by `task_id`
|
||||
|
||||
## Key decisions
|
||||
|
||||
### A. Vocabulary-driven inference (roles + competences)
|
||||
|
||||
- Maintain two vocabularies sourced from the Data Lake:
|
||||
- roles: `teamlandkarte_v_capacity_roles_latest` filtered by `active=true` and `staffing_board_relevant=true`
|
||||
- competences: unique values from `beschaffungstool_kmp_skill_latest.skillname__c`
|
||||
- Generate embeddings for these vocabularies and store them in the local SQLite embedding cache.
|
||||
|
||||
### B. Task text embedding stored by task_id
|
||||
|
||||
To minimize embedding transaction costs for repeated analysis of the same task:
|
||||
|
||||
- build `task_text = f"{title}\n\n{description}"` (title optional)
|
||||
- embed `task_text` once
|
||||
- store that embedding under a stable key derived from `task_id` (and embedding model/dims)
|
||||
|
||||
**Explicit cache key formats**:
|
||||
|
||||
```python
|
||||
# Role vocabulary entries
|
||||
key = f"role_vocab:{model}:{dims}:{normalized_role_name}"
|
||||
|
||||
# Competence vocabulary entries
|
||||
key = f"comp_vocab:{model}:{dims}:{normalized_competence_name}"
|
||||
|
||||
# Task text embeddings by task_id
|
||||
key = f"task_id:{model}:{dims}:{task_id}"
|
||||
```
|
||||
|
||||
Where:
|
||||
- `model` = embedding model name (e.g., `text-embedding-3-large`)
|
||||
- `dims` = embedding dimensions (e.g., `3072`)
|
||||
- `normalized_role_name` = normalized role text (trimmed, whitespace-collapsed, lowercased)
|
||||
- `normalized_competence_name` = normalized competence text (trimmed, whitespace-collapsed, lowercased)
|
||||
- `task_id` = database task ID (string or integer, converted to string)
|
||||
|
||||
### C. Inference logic (embedding similarity)
|
||||
|
||||
- Role inference:
|
||||
- compute cosine similarity between `task_text_embedding` and all role vocabulary embeddings
|
||||
- return the single best matching role with similarity score as a Markdown table: `Role | Similarity`
|
||||
- Competence inference (for validation and display):
|
||||
- compute cosine similarity between `task_text_embedding` and all competence vocabulary embeddings
|
||||
- return top-N competences with similarity scores as a Markdown table: `Competence | Similarity` (descending order)
|
||||
|
||||
**Configuration** (under `[matching.inference]`):
|
||||
```toml
|
||||
[matching.inference]
|
||||
max_competences = 8 # default: 8; omit to disable limit (no max)
|
||||
min_similarity = 0.4 # default: 0.4; omit to disable threshold filtering
|
||||
```
|
||||
|
||||
**Configuration rules**:
|
||||
- If `max_competences` is not specified: no limit on number of competences returned.
|
||||
- If `min_similarity` is not specified: no threshold filtering applied (include all).
|
||||
- At least one of `max_competences` or `min_similarity` MUST be specified in config to avoid unbounded results.
|
||||
- Selection rule: if both are set, return top `max_competences` among those with similarity `>= min_similarity`.
|
||||
|
||||
### D. Component structure for vocabulary preload
|
||||
|
||||
**Component: VocabularyCache**
|
||||
- **Location**: `src/teamlandkarte_mcp/matching/vocabulary.py` (new module)
|
||||
- **Responsibilities**:
|
||||
- Fetch role + competence vocabularies from Data Lake via database client
|
||||
- Normalize and deduplicate vocabulary entries
|
||||
- Embed missing entries using batch + chunking via `SimilarityEngine.prefetch_embeddings(...)`
|
||||
- Store embeddings in SQLite cache with stable keys
|
||||
- Provide in-memory memoization for fast repeated inference during tool execution
|
||||
- **Initialization**: Called from `build_server()` after DB client + similarity engine initialization
|
||||
- **Refresh strategy**: On startup only; no TTL-based refresh during runtime
|
||||
|
||||
**Task embedding preload**:
|
||||
- Preload **all open tasks** on startup (no limit).
|
||||
- Build `task_text = title + "\n\n" + description` for each task (title optional).
|
||||
- Embed only cache-missing `task_id` vectors (check SQLite cache first).
|
||||
- Store into SQLite embedding cache with key format: `task_id:{model}:{dims}:{task_id}`.
|
||||
- Runtime fallback: if a task embedding is missing during tool execution, embed on-demand (strict semantics).
|
||||
|
||||
### E. Determinism and strict failure semantics
|
||||
|
||||
- Runs are deterministic given fixed models and cached embeddings.
|
||||
- Azure embedding failures fail the operation immediately.
|
||||
- The existing global embedding improvements (dedup + batch + layered caches) should be reused.
|
||||
|
||||
## Data flow
|
||||
|
||||
### 1) Vocabulary refresh
|
||||
|
||||
1. fetch role list + competence list from Data Lake
|
||||
2. normalize/deduplicate
|
||||
3. embed missing entries via Azure batch API
|
||||
4. store embeddings in SQLite cache
|
||||
|
||||
### 2) Task validation
|
||||
|
||||
1. fetch task (title+description) + DB skills
|
||||
2. compute/retrieve `task_text_embedding` by task_id
|
||||
3. infer:
|
||||
- top-K roles
|
||||
- top-K competences
|
||||
4. compare:
|
||||
- DB skills vs inferred competences (plus similarity thresholds)
|
||||
- dates (unchanged)
|
||||
|
||||
### 3) Capacity→task matching
|
||||
|
||||
1. load the target capacity (role + competences + availability)
|
||||
2. list open tasks
|
||||
3. compute/retrieve `task_text_embedding` for each task
|
||||
4. compute overall similarity (role + competences) between the capacity and each task
|
||||
5. categorize + return table-first results (same output conventions as existing matching tools)
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- Keep chat API for role inference: rejected (latency/cost + redundant).
|
||||
- Use raw string matching for vocab mapping: rejected (less robust than embeddings).
|
||||
|
||||
## Risks
|
||||
|
||||
- Vocabulary size (competences) may be large → requires batch embedding + caching and careful refresh strategy.
|
||||
- Model updates can change nearest neighbors → mitigate with pinned model/deployment and cache key including model/dims.
|
||||
Reference in New Issue
Block a user