Migrate all repos into monorepo context folders

Bahn: aisupport, Analyse-O2C-C2S, awesome-bahn-mcp-servers, beam-mcp,
      Confluence_Bot, db-planet-mcp-server, O2C-Harness, project-audit,
      Projekt-KIQ-HP, teamlandkarte-mcp
Dhive: Jury-Voting
Privat: CV, NoteGraph (NOTE: NoteGraph needs complete redo after consolidation)
Shared: AI-Orchestrator, OrgMyLife, power_skills_and_more
Shared/references: symphony (read-only)

Bahn repos remain available as independent remotes - this monorepo
pulls them in via subtree, the originals are untouched.
This commit is contained in:
2026-06-30 20:39:52 +02:00
parent 2f2b295531
commit a5f8fb49ab
1717 changed files with 447332 additions and 0 deletions
@@ -0,0 +1,135 @@
# Design: Capacity→Task Matching and Embedding-based Inference
## Context
The system was originally optimized for task→capacity matching and relied on Azure chat completions for role/requirements extraction.
To enable a symmetrical workflow and reduce latency/cost, this change:
- adds tools to browse capacities and match a capacity against open tasks
- reworks role and competence inference to use embeddings only (Data Lake vocabularies)
- preloads and caches role/competence vocab embeddings on startup (with runtime on-demand fallback)
- preloads and caches task text embeddings by `task_id`
## Key decisions
### A. Vocabulary-driven inference (roles + competences)
- Maintain two vocabularies sourced from the Data Lake:
- roles: `teamlandkarte_v_capacity_roles_latest` filtered by `active=true` and `staffing_board_relevant=true`
- competences: unique values from `beschaffungstool_kmp_skill_latest.skillname__c`
- Generate embeddings for these vocabularies and store them in the local SQLite embedding cache.
### B. Task text embedding stored by task_id
To minimize embedding transaction costs for repeated analysis of the same task:
- build `task_text = f"{title}\n\n{description}"` (title optional)
- embed `task_text` once
- store that embedding under a stable key derived from `task_id` (and embedding model/dims)
**Explicit cache key formats**:
```python
# Role vocabulary entries
key = f"role_vocab:{model}:{dims}:{normalized_role_name}"
# Competence vocabulary entries
key = f"comp_vocab:{model}:{dims}:{normalized_competence_name}"
# Task text embeddings by task_id
key = f"task_id:{model}:{dims}:{task_id}"
```
Where:
- `model` = embedding model name (e.g., `text-embedding-3-large`)
- `dims` = embedding dimensions (e.g., `3072`)
- `normalized_role_name` = normalized role text (trimmed, whitespace-collapsed, lowercased)
- `normalized_competence_name` = normalized competence text (trimmed, whitespace-collapsed, lowercased)
- `task_id` = database task ID (string or integer, converted to string)
### C. Inference logic (embedding similarity)
- Role inference:
- compute cosine similarity between `task_text_embedding` and all role vocabulary embeddings
- return the single best matching role with similarity score as a Markdown table: `Role | Similarity`
- Competence inference (for validation and display):
- compute cosine similarity between `task_text_embedding` and all competence vocabulary embeddings
- return top-N competences with similarity scores as a Markdown table: `Competence | Similarity` (descending order)
**Configuration** (under `[matching.inference]`):
```toml
[matching.inference]
max_competences = 8 # default: 8; omit to disable limit (no max)
min_similarity = 0.4 # default: 0.4; omit to disable threshold filtering
```
**Configuration rules**:
- If `max_competences` is not specified: no limit on number of competences returned.
- If `min_similarity` is not specified: no threshold filtering applied (include all).
- At least one of `max_competences` or `min_similarity` MUST be specified in config to avoid unbounded results.
- Selection rule: if both are set, return top `max_competences` among those with similarity `>= min_similarity`.
### D. Component structure for vocabulary preload
**Component: VocabularyCache**
- **Location**: `src/teamlandkarte_mcp/matching/vocabulary.py` (new module)
- **Responsibilities**:
- Fetch role + competence vocabularies from Data Lake via database client
- Normalize and deduplicate vocabulary entries
- Embed missing entries using batch + chunking via `SimilarityEngine.prefetch_embeddings(...)`
- Store embeddings in SQLite cache with stable keys
- Provide in-memory memoization for fast repeated inference during tool execution
- **Initialization**: Called from `build_server()` after DB client + similarity engine initialization
- **Refresh strategy**: On startup only; no TTL-based refresh during runtime
**Task embedding preload**:
- Preload **all open tasks** on startup (no limit).
- Build `task_text = title + "\n\n" + description` for each task (title optional).
- Embed only cache-missing `task_id` vectors (check SQLite cache first).
- Store into SQLite embedding cache with key format: `task_id:{model}:{dims}:{task_id}`.
- Runtime fallback: if a task embedding is missing during tool execution, embed on-demand (strict semantics).
### E. Determinism and strict failure semantics
- Runs are deterministic given fixed models and cached embeddings.
- Azure embedding failures fail the operation immediately.
- The existing global embedding improvements (dedup + batch + layered caches) should be reused.
## Data flow
### 1) Vocabulary refresh
1. fetch role list + competence list from Data Lake
2. normalize/deduplicate
3. embed missing entries via Azure batch API
4. store embeddings in SQLite cache
### 2) Task validation
1. fetch task (title+description) + DB skills
2. compute/retrieve `task_text_embedding` by task_id
3. infer:
- top-K roles
- top-K competences
4. compare:
- DB skills vs inferred competences (plus similarity thresholds)
- dates (unchanged)
### 3) Capacity→task matching
1. load the target capacity (role + competences + availability)
2. list open tasks
3. compute/retrieve `task_text_embedding` for each task
4. compute overall similarity (role + competences) between the capacity and each task
5. categorize + return table-first results (same output conventions as existing matching tools)
## Alternatives considered
- Keep chat API for role inference: rejected (latency/cost + redundant).
- Use raw string matching for vocab mapping: rejected (less robust than embeddings).
## Risks
- Vocabulary size (competences) may be large → requires batch embedding + caching and careful refresh strategy.
- Model updates can change nearest neighbors → mitigate with pinned model/deployment and cache key including model/dims.