ankn/Orchestrator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Migrate all repos into monorepo context folders

Browse Source 
Bahn: aisupport, Analyse-O2C-C2S, awesome-bahn-mcp-servers, beam-mcp,
      Confluence_Bot, db-planet-mcp-server, O2C-Harness, project-audit,
      Projekt-KIQ-HP, teamlandkarte-mcp
Dhive: Jury-Voting
Privat: CV, NoteGraph (NOTE: NoteGraph needs complete redo after consolidation)
Shared: AI-Orchestrator, OrgMyLife, power_skills_and_more
Shared/references: symphony (read-only)

Bahn repos remain available as independent remotes - this monorepo
pulls them in via subtree, the originals are untouched.
This commit is contained in:
ankn
2026-06-30 20:39:52 +02:00
parent 2f2b295531
commit a5f8fb49ab
1717 changed files with 447332 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
bahn/teamlandkarte-mcp/docs/architecture.md
+720
View File
@@ -0,0 +1,720 @@
# Architecture Documentation (Arc42)
## 1. Introduction and Goals
### 1.1 Requirements Overview
The Teamlandkarte MCP Server enables AI assistants to match **tasks** to DB Systel employees with **free work capacity**, and to match a **specific capacity** to relevant **tasks**.
The server:
- queries the DB Systel Open Data Lake (Trino) read-only
- performs **BM25+RRF lexical matching** for competence similarity
- uses **LLM chat completions** (Azure OpenAI) for role similarity and role inference
- returns deterministic, table-first Markdown outputs suitable for chat UIs
Availability is displayed and may be applied as an optional date-range filter. It does **not** influence similarity scoring.
**Key Features:**
- MCP workflow for DB tasks and ad-hoc matching
- LLM-based role inference (tool: `infer_primary_role`) using Data Lake role vocabulary
- BM25+RRF competence matching with optional LLM-based auto-tagging pre-expansion
- **Two matching methods** selectable per call via `matching_method`:
- `score` (default): BM25+RRF competence + LLM role similarity → numeric scores + category
- `llm_fulltext`: LLM-based full-text comparison of complete profiles → category + rationale (no numeric scores)
- Capacity browsing + capacity-to-task matching tools:
- `list_free_capacities`, `get_capacity_details`, `find_matching_tasks`
- **Hard confirmation gate** before matching runs by default (`matching.require_confirmation = true`)
- Interactive result exploration with search session management (search_id + filter_id)
- Multi-level caching (DB query cache + search result cache)
- **Strict DB schema verification at startup** (fail-fast) against required view columns
- Fuzzy filtering on search results (role, competence) via `fuzzywuzzy`
### 1.2 Quality Goals
| Priority | Quality Goal | Motivation |
|----------|-------------|------------|
| 1 | **Security** | Read-only database access, secure credential management, no data leakage |
| 2 | **Performance** | Fast response times through caching, connection pooling |
| 3 | **Usability** | Simple MCP interface, clear result presentation, deterministic tool outputs |
| 4 | **Maintainability** | Modular architecture, clear separation of concerns, testable components |
| 5 | **Reliability** | Fail-fast schema checks, graceful degradation on LLM failures |
#### Reliability
- The server uses Azure OpenAI for **chat completions** (role similarity, role inference, auto-tagging).
- There is **no embedding dependency** -- competence matching is purely lexical (BM25+RRF).
- LLM failures in role similarity return 0.0 (graceful degradation).
- LLM failures in auto-tagging fall back to the unmodified candidate list.
- LLM failures in role inference return `None` (surfaced to the user).
### 1.3 Stakeholders
| Role | Contact | Expectations |
|------|---------|--------------|
| Product Owner | Thomas Handke | Feature delivery, quality, timeline |
| End Users | DB Systel Employees | Fast, accurate matching |
| Database Team | DB Systel IT | Minimal database load, read-only access |
| Security Team | DB Systel Security | Credential protection, audit logging |
---
## 2. Architecture Constraints
### 2.1 Technical Constraints
| Constraint | Background |
|------------|------------|
| Python 3.13+ | Required for latest language features and type hints |
| Trino client | Required for Trino/Presto connectivity to the Open Data Lake |
| MCP Protocol | Must comply with Model Context Protocol specification (FastMCP SDK) |
| Azure OpenAI chat completions | Required for role similarity, role inference, and optional auto-tagging |
| Read-only database | No write operations allowed on production database |
| Strict schema verification | Server fails fast if required view columns are missing |
### 2.2 Organizational Constraints
| Constraint | Background |
|------------|------------|
| Internal DB Infrastructure | Hosted on Deutsche Bahn Systel GitLab and data lake |
| Credential Management | Stored in environment variables / `.env`, not in version control |
| OpenSpec Workflow | All significant changes require spec proposals |
### 2.3 Conventions
| Convention | Details |
|------------|---------|
| Code Style | PEP 8, type hints, dataclasses for DTOs, ruff for linting/formatting |
| Configuration | TOML format (`config.toml`) for all configuration |
| API Design | MCP tools with clear parameter schemas, Markdown table outputs |
| Documentation | Inline docstrings, OpenSpec for architecture decisions |
| Testing | pytest + pytest-asyncio + hypothesis |
---
## 3. System Scope and Context
### 3.1 Business Context
```text
End User (DB Systel Employee)
| natural language
AI Assistant (MCP client)
| MCP (tool calls over stdio)
Teamlandkarte MCP Server
| SQL (read-only)
DB Systel Open Data Lake (Trino)
Teamlandkarte MCP Server
| HTTPS (chat completions)
Azure OpenAI
```
**External Entities:**
| Entity | Role | Interface |
|--------|------|-----------|
| AI Assistant (Claude, GPT, etc.) | Interprets user requests and calls MCP tools | MCP Protocol over stdio |
| DB Systel Open Data Lake | Stores employee capacity, competence, task, and role data | Trino/Presto SQL |
| Azure OpenAI | Provides chat completions for role similarity + inference + auto-tagging | HTTPS (Azure OpenAI REST API) |
| End User | Requests matching through natural conversation | Natural language via AI assistant |
### 3.2 Technical Context
```text
MCP Client (assistant)
<-> MCP protocol (tool calls, stdio JSON-RPC)
Teamlandkarte MCP Server (Python)
<-> Trino (read-only SQL, connection pool)
<-> Azure OpenAI (chat completions)
<-> Local caches (DB query cache + search session cache)
```
---
## 4. Solution Strategy
### 4.1 Architecture Approach
**Layered Architecture** with clear separation:
1. **MCP Interface Layer** (`mcp_server.py`): Exposes tools to client, formats Markdown output
2. **Business Logic Layer** (`matching/`): Matcher, scorer, SimilarityEngine, VocabularyCache, BM25, RRF, AutoTagger, **LlmFulltextMatcher** (LLM-based full-text matching with rationale)
3. **Data Access Layer** (`database/`): TrinoClient, connection pool, query logger, schema verification, read-only guard
4. **External Integration Layer** (`azure/`): AzureOpenAIClient, CostTracker
5. **Infrastructure** (`cache/`, `utils/`, `config.py`): QueryCache, SearchCache, date/markdown utilities, configuration loading
### 4.2 Key Design Decisions
| Decision | Rationale |
|----------|-----------|
| **BM25+RRF competence matching** | Lexical matching eliminates embedding false-positives; zero-out rule ensures no token overlap = score 0.0 |
| **LLM-based role similarity** | Semantic role comparison requires understanding synonyms and hierarchy; cached per-run |
| **LLM-based role inference** | Selects from DB vocabulary; constrained to known roles only |
| **LLM-based full-text matching (alternative method)** | `llm_fulltext` mode compares complete capacity/task profiles (description, references, certificates, skills) via Azure OpenAI Chat Completion and returns a category + 1–2 sentence rationale per item; no numeric scores |
| **Optional LLM auto-tagging** | Bridges lexical gap for BM25 by expanding candidate competences with covered synonyms |
| **Global BM25 index** | One index over all candidates' competences ensures stable IDF weights across the pool |
| **Hard confirmation gating (default)** | Prevents accidental matching runs; user must confirm requirements |
| **Markdown table output** | Better rendering in LLM clients; table-first outputs for robust parsing |
| **Connection pooling** | Bounded pool (default 4) for Trino connections; thread-safe acquire/release |
| **Strict startup schema verification** | Fail-fast when upstream views change columns |
| **Two-tier caching** | DB query cache (TTL) + search session cache (TTL) for interactive exploration |
| **Fuzzy filtering** | `fuzzywuzzy` for post-hoc filtering of search results by role/competence text |
### 4.3 Technology Stack
| Layer | Technology | Justification |
|-------|------------|---------------|
| Runtime | Python 3.13 | Modern features, strong typing |
| MCP SDK | `mcp` (FastMCP) | MCP server runtime, stdio transport |
| Database Client | `trino` | Trino/Presto SQL connectivity |
| Caching | `cachetools` (TTLCache) | In-memory TTL caches for DB queries and search sessions |
| Configuration | TOML (`tomllib`) | Human-readable, validated at startup |
| Competence Matching | `rank-bm25` (BM25Okapi) | Lexical ranking, no native extensions |
| Fuzzy Matching | `fuzzywuzzy` + `python-Levenshtein` | Fast fuzzy string matching for filters |
| LLM Integration | `openai` (AsyncAzureOpenAI) | Azure OpenAI chat completions |
| Environment | `python-dotenv` | Load credentials from `.env` |
---
## 5. Building Block View
### 5.1 Level 0: System Context
```
+-------------------------------------------+
| Teamlandkarte MCP Server |
| (Task <-> Capacity Matching) |
+-------------------------------------------+
```
### 5.2 Level 1: Container View
```
+------------------------------------------------------------+
| Teamlandkarte MCP Server |
| |
| +-------------------------------------------------------+ |
| | MCP Interface Layer (mcp_server.py) | |
| | - task browsing + details | |
| | - requirement capture + confirmation gate | |
| | - task->capacity matching + search refinement | |
| | - capacity browsing + capacity->task matching | |
| +----------------------------+---------------------------+ |
| | |
| +----------------------------v---------------------------+ |
| | Business Logic Layer (matching/) | |
| | - Matcher (orchestrates matching runs) | |
| | - SimilarityEngine (BM25+RRF competence, LLM role) | |
| | - VocabularyCache (LLM role inference from DB vocab) | |
| | - Scorer (weighted overall score + categorization) | |
| | - Bm25Index + bm25_rank_competences | |
| | - reciprocal_rank_fusion (RRF normalization) | |
| | - AutoTagger (optional LLM competence expansion) | |
| | - LlmFulltextMatcher (LLM full-text profile matching) | |
| +----------------------------+---------------------------+ |
| | |
| +----------------------------v---------------------------+ |
| | Data Access Layer (database/) | |
| | - TrinoClient (SQL queries, implements DBClient) | |
| | - ConnectionPool (bounded, thread-safe) | |
| | - SchemaVerifier (startup fail-fast) | |
| | - ReadOnly guard (SELECT/WITH only) | |
| | - QueryLogger (redacted SQL logging) | |
| +----------------------------+---------------------------+ |
| | |
| +----------------------------v---------------------------+ |
| | External Integration (azure/) | |
| | - AzureOpenAIClient (chat completions, retry/backoff) | |
| | - CostTracker (session-level token accounting) | |
| +--------------------------------------------------------+ |
| |
| +--------------------------------------------------------+ |
| | Infrastructure (cache/, utils/, config.py) | |
| | - QueryCache (TTL, DB results) | |
| | - SearchCache (TTL, search sessions + filters) | |
| | - dates.py (ISO parsing, overlap checks) | |
| | - markdown.py (table rendering) | |
| | - config.py (TOML loading + validation) | |
| +--------------------------------------------------------+ |
+------------------------------------------------------------+
| |
v v
+-----------------+ +-----------------+
| Open Data Lake | | Azure OpenAI |
| (Trino) | | (Chat API) |
+-----------------+ +-----------------+
```
### 5.3 Level 2: Source Module Map
```
src/teamlandkarte_mcp/
├── __init__.py
├── __main__.py # CLI entry point, arg parsing, logging setup
├── config.py # TOML config loading + validation (AppConfig)
├── logging_config.py # Logging to stderr (MCP stdio safety)
├── mcp_server.py # FastMCP server, all tool definitions, SessionState
├── models.py # Frozen dataclasses: Task, Capacity, Requirements, ScoredCapacity
├── azure/
│ ├── openai_client.py # AsyncAzureOpenAI wrapper (chat completions, retry)
│ └── cost_tracker.py # Session-level token/cost accounting
├── cache/
│ ├── query_cache.py # Generic TTLCache wrapper for DB queries
│ └── search_cache.py # Search session store (search_id, filter_id, TTL)
├── database/
│ ├── types.py # DBClient Protocol definition
│ ├── db_client.py # Factory: create_db_client() -> TrinoClient
│ ├── trino_client.py # TrinoClient (implements DBClient, uses pool)
│ ├── pool.py # ConnectionPool (bounded, thread-safe, LIFO)
│ ├── schema_verifier.py # Startup column verification
│ ├── read_only.py # SQL guard (SELECT/WITH only)
│ ├── query_logger.py # Redacted query logging
│ └── hive_client.py # (empty, reserved)
├── matching/
│ ├── matcher.py # Matcher: orchestrates matching runs
│ ├── scorer.py # compute_overall(), categorize()
│ ├── similarity.py # SimilarityEngine (BM25+RRF competence, LLM role)
│ ├── vocabulary.py # VocabularyCache (LLM role inference)
│ ├── bm25.py # Bm25Index, bm25_rank_competences, _tokenize
│ ├── rrf.py # reciprocal_rank_fusion (zero-out rule)
│ ├── auto_tagger.py # AutoTagger (LLM competence expansion)
│ ├── llm_fulltext_matcher.py # LlmFulltextMatcher (LLM full-text profile matching)
│ ├── profiles.py # CapacityProfile, TaskProfile + serializers
│ ├── task_analyzer.py # (deprecated stub)
│ └── task_helpers.py # (deprecated stub)
└── utils/
├── dates.py # parse_iso_date, availability_overlaps
└── markdown.py # md_table (GFM table rendering)
```
### 5.4 Tool Surface (MCP Tools)
#### Discovery (DB tasks)
| Tool | Responsibility | Dependencies |
|------|----------------|--------------|
| `list_open_tasks` | List newest published tasks | DB |
| `get_task_details` | Table-first task fields + inferred role | DB, LLM (role inference) |
| `validate_task_requirements` | Compare DB fields vs LLM-inferred role + competences | DB, LLM |
| `infer_primary_role` | Infer closest role from task id or free text | DB vocab, LLM |
#### Requirement capture + confirmation gate
| Tool | Responsibility | Dependencies |
|------|----------------|--------------|
| `extract_requirements` | LLM-based inference for role + competences from free text | DB vocab, LLM |
| `collect_structured_requirement_data` | Accept user-supplied fields and stage requirements | session state |
| `update_requirements` | (deprecated stub, directs to `collect_structured_requirement_data`) | -- |
| `start_guided_capture` | Start step-by-step guided capture | session state |
| `guided_set_description` | Set description (step 1/4) | session state |
| `guided_set_role` | Set role (step 2/4) | session state |
| `guided_set_time_range` | Set date range (step 3/4) | session state |
| `guided_set_competences` | Set competences (step 4/4) | session state |
| `show_pending_requirements` | Show review table for pending requirements | session state |
| `request_requirements_confirmation` | Mark that assistant asked user to confirm | session state |
| `confirm_requirements` | Confirm or reject staged requirements | session state |
#### Task -> capacity matching
| Tool | Responsibility | Dependencies |
|------|----------------|--------------|
| `find_matching_capacities` | Compute matches and store a search session (`search_id`). Accepts `matching_method` (`"score"` \| `"llm_fulltext"`). | DB, BM25, LLM (role) or LlmFulltextMatcher, caches |
| `filter_search_results` | Refine stored results (fuzzy role/competence, availability, min_similarity). `min_similarity` is ignored in `llm_fulltext` mode and surfaced as a hint in the Applied Filters table. | SearchCache |
| `get_results_by_category` | Page through stored results by category. Renders `Begründung` column instead of score columns when the search was run in `llm_fulltext` mode. | SearchCache |
#### Capacity browsing + capacity -> task matching
| Tool | Responsibility | Dependencies |
|------|----------------|--------------|
| `list_free_capacities` | List recently created free capacities | DB |
| `get_capacity_details` | Table-first capacity fields + next steps | DB |
| `find_matching_tasks` | Find tasks that match a specific capacity. Accepts `matching_method` (`"score"` \| `"llm_fulltext"`). | DB, BM25, LLM (role) or LlmFulltextMatcher, caches |
#### Team browsing + task -> team matching
| Tool | Responsibility | Dependencies |
|------|----------------|--------------|
| `list_teams` | List the first *N* teams (default 20) as a Markdown table (`Team Id`, `Team Name`, `Schwerpunkt`, `Anzahl Kompetenzen`, `Anzahl Referenzen`). | DB |
| `get_team_details` | Table-first team fields plus sections `## Über uns`, `## Leistungen`, `## Interessen`, `## Kompetenzen` (with `(Top)` marker), `## Referenzen` (Partner_Name + Projekte) and `## Next steps`. | DB |
| `find_matching_teams` | Match a task against team profiles and store a search session (`search_id`). Accepts `matching_method` (`"score"` \| `"llm_fulltext"`). Persists `search_type = "team_search"` and the chosen `matching_method` in `SearchCache` and the META JSON. | DB, BM25, LLM (role) or LlmFulltextMatcher, caches |
#### Profile_Type parameter
| Parameter | Allowed values | Default | Selection mechanism | Affected tools |
|-----------|---------------|---------|---------------------|----------------|
| `Profile_Type` | `"capacity"`, `"team"` | implicit per tool name | Tool selection (`find_matching_capacities` ⇒ `capacity`, `find_matching_teams` ⇒ `team`); persisted as `search_type` (`"capacity_search"` / `"team_search"`) in `SearchCache` and META JSON | `find_matching_capacities`, `find_matching_teams`, `list_teams`, `get_team_details` |
- `Profile_Type = "capacity"` (existing behavior): match a task against employee capacity profiles. `find_matching_capacities` is the entry point; availability filters apply.
- `Profile_Type = "team"` (new): match a task against aggregated team profiles. `find_matching_teams` is the entry point; `list_teams` and `get_team_details` browse and inspect team profiles independently of a matching run. Availability filters are **not** applied for team searches and are surfaced as not-effective hints in the `Applied Filters` table.
#### Matching method parameter
| Parameter | Allowed values | Default | Affected tools |
|-----------|---------------|---------|----------------|
| `matching_method` | `"score"`, `"llm_fulltext"` | `[matching].default_method` (TOML, default `"score"`) | `find_matching_capacities`, `find_matching_tasks`, `find_matching_teams` |
- `"score"`: existing BM25+RRF competence + LLM role similarity. Output includes `Role Score`, `Competence Score`, `Overall Score`, `Category`. For team searches, the team's `focus_name` is used as the role stand-in and top competences are weighted by `matching.team.top_competency_weight` (default `1.5`).
- `"llm_fulltext"`: LLM-based full-text comparison via `LlmFulltextMatcher`. Output replaces all score columns with a single `Begründung` column (1–2 sentence rationale from the LLM). The persisted META JSON contains `matching_method` so downstream tools (`get_results_by_category`, `filter_search_results`) know which schema to render.
For team searches, the result table columns are:
| Mode | Columns |
|------|---------|
| `team_search` × `score` | `Team Name`, `Schwerpunkt`, `Top-Kompetenzen`, `Role Score`, `Competence Score`, `Overall Score`, `Category` |
| `team_search` × `llm_fulltext` | `Team Name`, `Schwerpunkt`, `Top-Kompetenzen`, `Category`, `Begründung` |
### 5.5 LlmFulltextMatcher (component)
The `LlmFulltextMatcher` (in `matching/llm_fulltext_matcher.py`) is the business-logic component behind `matching_method = "llm_fulltext"`.
| Aspect | Details |
|--------|---------|
| **Inputs** | A `TaskProfile` (id, title, description, skills) for `match_capacities`; a `CapacityProfile` (id, owner_name, role_name, competences, description, references with `partner_name`/`projects`, certificates) for `match_tasks`. The candidate list (`capacities` or `tasks`) is already prefiltered by the MCP tool (same prefilter as in `score` mode, e.g. availability). |
| **Profile sources** | Built from in-memory `Capacity`/`Task` objects plus extras loaded from the DB via `DBClient.batch_get_capacity_descriptions`, `batch_get_capacity_certificates`, and `batch_get_capacity_references` (with the partner LEFT JOIN inside the same SQL query). |
| **Outputs** | `LlmFulltextResult` with `by_category: dict[str, list[LlmFulltextItem]]` (categories `Top`/`Good`/`Partial`/`Low`/`Irrelevant`; each item carries `category`, ungekürzte `rationale`, raw fields) and `errors: list[LlmFulltextError]` for items where the LLM call/parse failed. |
| **External dependency** | Azure OpenAI Chat Completion via `AzureOpenAIClient.chat_completion(...)` with `response_format=json_object`, deterministic German system prompt, JSON schema `{"category": "...", "rationale": "..."}`. One LLM call per candidate. |
| **Determinism** | Profile serialization is field-stable; results within a category are sorted lexicographically by `item_id`; invalid LLM categories map to `Irrelevant` with a hint appended to the rationale. |
| **Persistence** | Items are stored in `SearchCache` with `category` + ungekürzte `rationale` (no numeric score fields); `errors` are persisted alongside; META JSON carries `matching_method`. |
---
## 6. Runtime View (Key Flows)
### 6.1 Server startup (fail-fast)
1. Parse CLI arguments (`--config`, `--log-level`)
2. Configure logging (stderr only, MCP stdio safety)
3. Load and validate `config.toml` + environment variables
4. Create DB client (TrinoClient with connection pool)
5. Verify required DB view columns (schema verification -> fail-fast on mismatch)
6. Initialize caches (QueryCache, SearchCache)
7. Initialize Azure OpenAI client + CostTracker
8. Optionally construct AutoTagger (if `use_auto_tagging = true`)
9. Initialize SimilarityEngine, VocabularyCache, Matcher
10. Register all MCP tools
11. Start MCP server (stdio transport via `mcp.run()`)
### 6.2 Task -> capacity matching
1. Capture requirements (guided capture or `extract_requirements` or `collect_structured_requirement_data`)
2. (If `require_confirmation = true`) confirmation gate:
- `show_pending_requirements` -> user reviews
- `request_requirements_confirmation` -> assistant marks confirmation requested
- `confirm_requirements(true)` -> user confirms
3. Run `find_matching_capacities(role_name, competences, date_start?, date_end?, matching_method?)`:
- Resolve `matching_method` (parameter > `[matching].default_method` > `"score"`)
- Query all capacities + competences from DB (cached)
- Filter by date overlap (same prefilter for both methods)
- Branch by `matching_method`:
- `"score"`: build global BM25 index, run BM25+RRF competence similarity + LLM role similarity, compute weighted overall score + categorize
- `"llm_fulltext"`: batch-load `description`, `certificates`, `references` (with partner LEFT JOIN) for the filtered capacities; build `TaskProfile` + `CapacityProfile`; one Azure OpenAI Chat Completion per capacity (`response_format=json_object`); LLM returns `{category, rationale}`; invalid categories → `Irrelevant` with hint; LLM/JSON failures recorded in a separate `errors` list
- Store results in SearchCache (payload includes `matching_method`; LLM mode stores `category` + ungekürzte `rationale` per item, no score fields) -> return `search_id`
4. Browse/filter results:
- `get_results_by_category(search_id, category, page, page_size)` (renders `Begründung` column in LLM mode)
- `filter_search_results(search_id, ...)` -> returns `filter_id`. In LLM mode, `min_similarity` is ignored and surfaced as a hint; sorting falls back to `(category_rank, item_id)`.
### 6.3 Capacity -> task matching
1. Browse capacities (`list_free_capacities`)
2. Inspect (`get_capacity_details`)
3. Run `find_matching_tasks(capacity_id, matching_method?)`:
- Resolve `matching_method` (parameter > `[matching].default_method` > `"score"`)
- Query capacity details + competences from DB
- Query open tasks from DB (same prefilter for both methods)
- Branch by `matching_method`:
- `"score"`: for each task infer role + competences, compute BM25+RRF + LLM role similarity
- `"llm_fulltext"`: load `description`, `certificates`, `references` (with partner LEFT JOIN) for the capacity; build `CapacityProfile` once; for each task build a `TaskProfile` and run one Azure OpenAI Chat Completion; same error/category-normalization handling as in 6.2
- Store results as a search session (`matching_method` persisted) -> return `search_id`
4. Browse/filter results (same tools as task->capacity)
### 6.4 Task -> team matching
1. Capture requirements as in 6.2 (guided capture, `extract_requirements`, or `collect_structured_requirement_data`)
2. Optional confirmation gate (same as in 6.2): `show_pending_requirements` -> `request_requirements_confirmation` -> `confirm_requirements(true)`
3. Run `find_matching_teams(role_name, competences, matching_method?)`:
- Resolve `matching_method` (parameter > `[matching].default_method` > `"score"`); reject unknown values with an error listing the allowed values, without any DB or LLM call
- Validate minimum requirements (`competences` non-empty)
- Load all teams from the DB-backed cache `_get_teams_cached()`. The DB layer joins `teamlandkarte_v_teams_latest` with `teamlandkarte_v_teammeter_organizational_units_latest` (INNER JOIN over `team_id = id` for the team name), and aggregates competences (`teamlandkarte_v_teammeter_team_competences_latest` joined to `teamlandkarte_v_competences_latest` over `competence_id = id`) and references (`teamlandkarte_v_team_references_latest` LEFT JOIN `teamlandkarte_v_partners_latest` over `partner_id = id`, exposing `name` as `Partner_Name`). Joins between team master data, competences, and references run over `ouid`.
- **No availability prefilter**: team profiles do not carry an availability range; any provided date range is ignored and surfaced as a not-effective hint in the `Applied Filters` table.
- Branch by `matching_method`:
- `"score"`: build the requirements competence list, derive each team's competence list and parallel top-competence set from `Team.competences`, and compute competence similarity via the existing `SimilarityEngine`. Top competences are upweighted by `matching.team.top_competency_weight` (default `1.5`); the per-required score is clamped to `[0, 1]`. Role similarity uses `Team.focus_name` as the role stand-in (`compute_role_similarity(req.role_name, team.focus_name)`). Overall score and category use the same `compute_overall(...)` / `categorize(...)` thresholds as for capacities.
- `"llm_fulltext"`: build a `TaskProfile` from the requirements and a `TeamProfile` per team via `build_team_profile` + `serialize_team_profile`. The user prompt is `=== Aufgabe ===` + `=== Team ===` (with an `ID:` header). The system prompt and JSON schema (`{"category": ..., "rationale": ...}`) are reused from the capacity LLM path; invalid categories are normalized to `Irrelevant` with a hint suffix in the rationale; LLM/JSON errors land in a separate `errors` list.
- Store results in `SearchCache` with `search_type = "team_search"` and `matching_method = ...` -> return `search_id`. The META JSON exposes both `search_type` and `matching_method` so `get_results_by_category` / `filter_search_results` render the team-specific columns.
4. Browse/filter results:
- `get_results_by_category(search_id, category, page, page_size)` (renders `Begründung` column in `llm_fulltext` mode)
- `filter_search_results(search_id, ...)`. `role_filter` matches against `Team_Focus_Name`; `competence_filter` matches the competence names of the team and supports a `(Top)` suffix that restricts the filter to top competences. `availability_date_start`, `availability_date_end`, and `is_fully_available` are ignored for `team_search` and surfaced as `team_search` not-effective hints in the `Applied Filters` table.
### 6.5 Role inference flow
1. `infer_primary_role(task_id=... | task_text=...)`
2. Fetch all role names from DB vocabulary view
3. Send task text + role list to LLM (JSON response format)
4. Validate returned role exists in DB vocabulary
5. Return `(role_name, confidence)` or `None` on failure
---
## 7. Deployment View
```
+-------------------------------------------------+
| Developer Machine / CI |
| |
| +--------------------------------------------+ |
| | MCP Client (IDE / AI Assistant) | |
| | <-> stdio (JSON-RPC) | |
| | teamlandkarte-mcp process | |
| | (Python 3.13, single process) | |
| +--------------------------------------------+ |
| |
| config.toml + .env (credentials) |
+-------------------------------------------------+
| |
v v
+-----------------+ +-----------------+
| Trino cluster | | Azure OpenAI |
| (Data Lake) | | (Chat API) |
+-----------------+ +-----------------+
```
**Requirements:**
- Python 3.13+ with project dependencies installed
- Trino connectivity (host/port via `config.toml`, credentials via env vars)
- Azure OpenAI endpoint + chat deployment + API key (env var `AZURE_OPENAI_LLM_API_KEY`)
- Environment variables: `DATA_LAKE_USERNAME`, `DATA_LAKE_PASSWORD`, `AZURE_OPENAI_LLM_API_KEY`
---
## 8. Cross-cutting Concepts
### 8.1 Deterministic outputs
- Tools produce **table-first Markdown output** (GFM tables via `md_table()`).
- Search tools emit deterministic headers with `SEARCH_ID=...` / `FILTER_ID=...` for robust client parsing.
- Scoring formula is deterministic: `overall = competence_weight * competence_score + role_weight * role_score`.
### 8.2 Caching strategy
| Cache | Implementation | TTL | Purpose |
|-------|---------------|-----|---------|
| DB query cache | `QueryCache` (cachetools TTLCache) | Configurable (`db_ttl_hours`, default 12h) | Avoid repeated expensive Trino queries |
| Search session cache | `SearchCache` (cachetools TTLCache) | Configurable (`search_ttl_minutes`, default 60min) | Enable interactive exploration of results |
| LLM role similarity cache | In-memory dict on `SimilarityEngine` | Per matching run (cleared between runs) | Avoid duplicate LLM calls for same role pair |
### 8.3 Security
- **Read-only DB access**: `ensure_select_only()` guard rejects non-SELECT/WITH queries
- **No write operations**: Server never modifies the Data Lake
- **Credentials via environment**: `DATA_LAKE_USERNAME`, `DATA_LAKE_PASSWORD`, `AZURE_OPENAI_LLM_API_KEY` -- never in config files
- **Query logging**: Parameters are redacted before logging (`query_logger.py`)
- **Logging to stderr**: MCP stdio transport requires stdout to remain clean JSON-RPC
### 8.4 Error handling
- **Schema verification failure**: Server refuses to start (fail-fast)
- **DB connectivity**: Lazy validation on first DB-backed tool call
- **LLM failures (role similarity)**: Return 0.0, no caching of failed result
- **LLM failures (auto-tagging)**: Return unmodified candidate list (graceful degradation)
- **LLM failures (role inference)**: Return `None`, logged as warning
- **Connection pool exhaustion**: `PoolExhaustedError` raised immediately (no blocking wait)
- **Config errors**: `ConfigError` raised at startup with descriptive message
### 8.5 Connection pooling
- `ConnectionPool` (generic, thread-safe, LIFO queue)
- Bounded at `pool_size` (default 4) connections
- Connections created lazily on demand
- `closeall()` for graceful shutdown
---
## 9. Scoring and Matching Algorithm
### 9.1 Overall scoring formula
```
overall_score = competence_weight * competence_score + role_weight * role_score
```
Default weights: `competence_weight = 0.8`, `role_weight = 0.2` (must sum to 1.0).
### 9.2 Competence scoring (BM25 + RRF)
For each required competence:
1. Query the global BM25 index (built over all filtered candidates' competences)
2. Filter ranked results to the current candidate's competence set
3. Apply Reciprocal Rank Fusion (k=60) to normalize scores to [0.0, 1.0]
4. **Zero-out rule**: No token overlap -> score 0.0 (eliminates false positives)
Aggregate: `competence_score = mean(per_required_competence_scores)`
Threshold for "matched": score >= 0.5.
### 9.3 Role scoring (LLM)
- LLM chat completion compares two role names semantically -> similarity in [0.0, 1.0]
- Identical roles (case-insensitive) -> 1.0 (short-circuit, no LLM call)
- Empty/unknown roles -> 0.0 (short-circuit)
- Results cached symmetrically per matching run
### 9.4 Categorization
| Category | Threshold |
|----------|-----------|
| Top | overall >= configured `top` (default 0.8) |
| Good | overall >= configured `good` (default 0.65) |
| Partial | overall >= configured `partial` (default 0.5) |
| Low | overall >= configured `low` (default 0.3) |
| Irrelevant | overall < configured `low` |
(Thresholds are configurable via `[matching.thresholds]`.)
### 9.5 Optional: Auto-Tagging (LLM competence expansion)
When `matching.similarity.use_auto_tagging = true`:
1. Before BM25 scoring, the AutoTagger asks the LLM which required competences are already covered by the candidate's existing competences (via synonym, abbreviation, cross-language equivalence)
2. Covered required-competence names are appended to the candidate's working list
3. Expansion is **ephemeral** -- never persisted to DB
4. On LLM failure, scoring proceeds with the unmodified list
---
## 10. Configuration Reference
### 10.1 `config.toml` structure
```toml
[database]
backend = "trino" # Only "trino" supported
host = "..."
port = 443
http_scheme = "https"
verify_ssl = true
catalog = "hive"
schema = "tier1_open_lake"
connect_timeout = 10
pool_size = 4
[matching]
competence_weight = 0.8 # Must sum to 1.0 with role_weight
role_weight = 0.2
require_confirmation = true # Hard gate before matching
default_method = "score" # Default for matching_method when callers omit it.
# Allowed: "score" | "llm_fulltext".
[matching.thresholds]
top = 0.8
good = 0.65
partial = 0.5
low = 0.3
[matching.fuzzy]
min_similarity = 0.7 # Fuzzy filter threshold
[matching.similarity]
use_auto_tagging = false # LLM pre-expansion for BM25
[cache]
db_ttl_hours = 12
search_ttl_minutes = 60
max_size = 100
[azure_openai]
endpoint = "https://..."
api_version = "2024-02-15-preview"
chat_deployment = "gpt-4.1"
show_costs_in_output = false
```
### 10.2 Required environment variables
| Variable | Purpose |
|----------|---------|
| `DATA_LAKE_USERNAME` | Trino login username |
| `DATA_LAKE_PASSWORD` | Trino login password |
| `AZURE_OPENAI_LLM_API_KEY` | Azure OpenAI API key for chat completions |
---
## 11. Data Model
### 11.1 Core domain objects (frozen dataclasses)
| Class | Fields | Purpose |
|-------|--------|---------|
| `Task` | id, name, title, description, start_date, end_date, created_date, skills | Published task from DB |
| `Capacity` | id, owner_name, role_name, role_level, begin_date, end_date, competences | Employee capacity entry |
| `Requirements` | role_name, competences, date_start, date_end, description | Structured matching input |
| `ScoredCapacity` | capacity, competence_score, role_score, overall_score, category, matched_competences, missing_competences | Matching result |
| `Team` | team_id, ouid, team_name, focus_name, about_us, offerings, interests, competences, references | Aggregated team master data |
| `TeamCompetence` | name, top_competency | Single team competence with top marker |
| `TeamReference` | partner_name, projects | Single team reference; `partner_name` may be empty |
| `ScoredTeam` | team, competence_score, role_score, overall_score, category, matched_competences, missing_competences | Team matching result (score mode) |
### 11.2 Team Profile
A **Team Profile** is the second profile type next to the existing capacity profile. It aggregates fields from four read-only views into a single immutable `Team` instance and is used both for browsing (`list_teams`, `get_team_details`) and for matching (`find_matching_teams` in both `score` and `llm_fulltext` modes).
| Field | Type | Source | Notes |
|-------|------|--------|-------|
| `team_id` | `str` | `teamlandkarte_v_teams_latest.team_id` | Stable identifier; persisted in `SearchCache`. |
| `ouid` | `str` | `teamlandkarte_v_teams_latest.ouid` | Join key for competences and references. |
| `team_name` | `str` | `teamlandkarte_v_teammeter_organizational_units_latest.name` | Resolved via INNER JOIN `teams_latest.team_id = organizational_units_latest.id`. Teams without an OU match are excluded. |
| `focus_name` | `str` | `teamlandkarte_v_teams_latest.focus_name` | NULL → `""`. Used as role stand-in in `score` mode. |
| `about_us` | `str` | `teamlandkarte_v_teams_latest.about_us` | NULL → `""`. |
| `offerings` | `str` | `teamlandkarte_v_teams_latest.offerings` | NULL → `""`. |
| `interests` | `str` | `teamlandkarte_v_teams_latest.interests` | NULL → `""`. |
| `competences` | `list[TeamCompetence]` | `teamlandkarte_v_teammeter_team_competences_latest` joined to `teamlandkarte_v_competences_latest` over `competence_id = id` | Joined on `ouid`. `top_competency = COALESCE(top_competency, FALSE)`. Order: top competences first, then name ascending. |
| `references` | `list[TeamReference]` | `teamlandkarte_v_team_references_latest` LEFT JOIN `teamlandkarte_v_partners_latest` over `partner_id = id` | Joined on `ouid`. Partner_Name from `p.name` (`COALESCE(..., '')`). Whitespace-only `projects` are filtered. Order: partner_name asc, projects asc. |
**Profile serialization** (`build_team_profile` + `serialize_team_profile` in `matching/profiles.py`) emits a deterministic, German-headed text block in fixed order: `Teamname:`, `Schwerpunkt:`, `Über uns:`, `Leistungen:`, `Interessen:`, `Kompetenzen:`, `Referenzen:`. Top competences are suffixed with `(Top)`. Reference rows render as `Partner: <partner_name> – Projekte: <projects>` when the partner is known, and as `Projekte: <projects>` (no placeholder) when the partner is empty. Empty lists render as `Kompetenzen: (keine)` / `Referenzen: (keine)`. List ordering matches the DB-provided order.
### 11.3 Database views (Trino)
The server queries read-only views in the `tier1_open_lake` schema. Schema verification at startup checks:
- `teamlandkarte_v_capacity_roles_latest`: requires columns `name`, `active`, `staffing_board_relevant`
- `teamlandkarte_v_capacities_latest`: requires column `creation_date`
- `teamlandkarte_v_teams_latest`: requires columns `team_id`, `ouid`, `about_us`, `offerings`, `interests`, `focus_name`
- `teamlandkarte_v_teammeter_organizational_units_latest`: requires columns `id`, `name`
- `teamlandkarte_v_teammeter_team_competences_latest`: requires columns `ouid`, `competence_id`, `top_competency`
- `teamlandkarte_v_team_references_latest`: requires columns `ouid`, `partner_id`, `projects`
#### Additional views read in `llm_fulltext` mode
The following columns/views are not part of the startup schema verification but are read at runtime when `matching_method = "llm_fulltext"`:
| View | Columns used | Used for |
|------|--------------|----------|
| `teamlandkarte_v_capacities_latest` | `description` | `CapacityProfile.description` |
| `teamlandkarte_v_capacity_certificates_latest` | `capacity_id`, `description` | `CapacityProfile.certificates` (1:n via `capacity_id`) |
| `teamlandkarte_v_capacity_references_latest` | `capacity_id`, `partner_id`, `projects` | `CapacityProfile.references[].projects` (1:n via `capacity_id`) |
| `teamlandkarte_v_partners_latest` | `id`, `name` | Partner name per reference (`CapacityProfile.references[].partner_name`) |
**Reference ↔ Partner join:** `teamlandkarte_v_capacity_references_latest.partner_id = teamlandkarte_v_partners_latest.id`. The partner `name` column is exposed as `partner_name` (via `COALESCE(p.name, '')`) inside the same SQL query that loads the references, so no extra round-trip is required. When `partner_id` is `NULL` or no partner matches, `partner_name` is the empty string and the reference is still returned with its `projects` text. The LEFT JOIN keeps the reference list complete even for unknown partners.
#### Views read for team profiles
The following views back the `Team` data type and are loaded by `find_matching_teams`, `list_teams`, and `get_team_details`:
| View | Columns used | Used for |
|------|--------------|----------|
| `teamlandkarte_v_teams_latest` | `team_id`, `ouid`, `about_us`, `offerings`, `interests`, `focus_name` | Team master data and free-text fields |
| `teamlandkarte_v_teammeter_organizational_units_latest` | `id`, `name` | `Team.team_name` via INNER JOIN over `teams_latest.team_id = organizational_units_latest.id` |
| `teamlandkarte_v_teammeter_team_competences_latest` | `ouid`, `competence_id`, `top_competency` | Team competences with top markers (joined to `teamlandkarte_v_competences_latest` over `competence_id = id` to resolve the competence name) |
| `teamlandkarte_v_team_references_latest` | `ouid`, `partner_id`, `projects` | Team references with partner names |
| `teamlandkarte_v_partners_latest` | `id`, `name` | `Partner_Name` per team reference |
**Team_Name join (INNER JOIN):** `teamlandkarte_v_teams_latest.team_id = teamlandkarte_v_teammeter_organizational_units_latest.id`. Teams without a matching OU row are excluded from `get_all_teams` and `get_team_by_id`.
**Competence and reference joins:** Both `teamlandkarte_v_teammeter_team_competences_latest` and `teamlandkarte_v_team_references_latest` are joined to the team master data over the shared `ouid` column.
**Reference ↔ Partner join (LEFT JOIN):** `teamlandkarte_v_team_references_latest.partner_id = teamlandkarte_v_partners_latest.id`. The `name` column from `teamlandkarte_v_partners_latest` is exposed as `Partner_Name` via `COALESCE(p.name, '')` inside the same SQL query that loads the references. When `partner_id` is `NULL` or no partner matches, `Partner_Name` is the empty string and the reference is still returned with its `projects` text. Whitespace-only `projects` are filtered out in the DB layer.
---
## 12. Testing
- **Unit tests**: pytest + pytest-asyncio
- **Property-based tests**: hypothesis (for scoring, BM25, RRF invariants)
- **Type checking**: mypy (strict)
- **Linting/formatting**: ruff
- **No integration test infrastructure**: DB and Azure OpenAI are mocked in tests