Orchestrator/bahn/teamlandkarte-mcp/docs/architecture.md

# Architecture Documentation (Arc42)

## 1. Introduction and Goals

### 1.1 Requirements Overview

The Teamlandkarte MCP Server enables AI assistants to match **tasks** to DB Systel employees with **free work capacity**, and to match a **specific capacity** to relevant **tasks**.

The server:
- queries the DB Systel Open Data Lake (Trino) read-only
- performs **BM25+RRF lexical matching** for competence similarity
- uses **LLM chat completions** (Azure OpenAI) for role similarity and role inference
- returns deterministic, table-first Markdown outputs suitable for chat UIs

Availability is displayed and may be applied as an optional date-range filter. It does **not** influence similarity scoring.

**Key Features:**
- MCP workflow for DB tasks and ad-hoc matching
- LLM-based role inference (tool: `infer_primary_role`) using Data Lake role vocabulary
- BM25+RRF competence matching with optional LLM-based auto-tagging pre-expansion
- **Two matching methods** selectable per call via `matching_method`:
  - `score` (default): BM25+RRF competence + LLM role similarity → numeric scores + category
  - `llm_fulltext`: LLM-based full-text comparison of complete profiles → category + rationale (no numeric scores)
- Capacity browsing + capacity-to-task matching tools:
  - `list_free_capacities`, `get_capacity_details`, `find_matching_tasks`
- **Hard confirmation gate** before matching runs by default (`matching.require_confirmation = true`)
- Interactive result exploration with search session management (search_id + filter_id)
- Multi-level caching (DB query cache + search result cache)
- **Strict DB schema verification at startup** (fail-fast) against required view columns
- Fuzzy filtering on search results (role, competence) via `fuzzywuzzy`

### 1.2 Quality Goals

| Priority | Quality Goal | Motivation |
|----------|-------------|------------|
| 1 | **Security** | Read-only database access, secure credential management, no data leakage |
| 2 | **Performance** | Fast response times through caching, connection pooling |
| 3 | **Usability** | Simple MCP interface, clear result presentation, deterministic tool outputs |
| 4 | **Maintainability** | Modular architecture, clear separation of concerns, testable components |
| 5 | **Reliability** | Fail-fast schema checks, graceful degradation on LLM failures |

#### Reliability

- The server uses Azure OpenAI for **chat completions** (role similarity, role inference, auto-tagging).
- There is **no embedding dependency** -- competence matching is purely lexical (BM25+RRF).
- LLM failures in role similarity return 0.0 (graceful degradation).
- LLM failures in auto-tagging fall back to the unmodified candidate list.
- LLM failures in role inference return `None` (surfaced to the user).

### 1.3 Stakeholders

| Role | Contact | Expectations |
|------|---------|--------------|
| Product Owner | Thomas Handke | Feature delivery, quality, timeline |
| End Users | DB Systel Employees | Fast, accurate matching |
| Database Team | DB Systel IT | Minimal database load, read-only access |
| Security Team | DB Systel Security | Credential protection, audit logging |

---

## 2. Architecture Constraints

### 2.1 Technical Constraints

| Constraint | Background |
|------------|------------|
| Python 3.13+ | Required for latest language features and type hints |
| Trino client | Required for Trino/Presto connectivity to the Open Data Lake |
| MCP Protocol | Must comply with Model Context Protocol specification (FastMCP SDK) |
| Azure OpenAI chat completions | Required for role similarity, role inference, and optional auto-tagging |
| Read-only database | No write operations allowed on production database |
| Strict schema verification | Server fails fast if required view columns are missing |

### 2.2 Organizational Constraints

| Constraint | Background |
|------------|------------|
| Internal DB Infrastructure | Hosted on Deutsche Bahn Systel GitLab and data lake |
| Credential Management | Stored in environment variables / `.env`, not in version control |
| OpenSpec Workflow | All significant changes require spec proposals |

### 2.3 Conventions

| Convention | Details |
|------------|---------|
| Code Style | PEP 8, type hints, dataclasses for DTOs, ruff for linting/formatting |
| Configuration | TOML format (`config.toml`) for all configuration |
| API Design | MCP tools with clear parameter schemas, Markdown table outputs |
| Documentation | Inline docstrings, OpenSpec for architecture decisions |
| Testing | pytest + pytest-asyncio + hypothesis |

---

## 3. System Scope and Context

### 3.1 Business Context

```text
End User (DB Systel Employee)
  | natural language
AI Assistant (MCP client)
  | MCP (tool calls over stdio)
Teamlandkarte MCP Server
  | SQL (read-only)
DB Systel Open Data Lake (Trino)

Teamlandkarte MCP Server
  | HTTPS (chat completions)
Azure OpenAI
```

**External Entities:**

| Entity | Role | Interface |
|--------|------|-----------|
| AI Assistant (Claude, GPT, etc.) | Interprets user requests and calls MCP tools | MCP Protocol over stdio |
| DB Systel Open Data Lake | Stores employee capacity, competence, task, and role data | Trino/Presto SQL |
| Azure OpenAI | Provides chat completions for role similarity + inference + auto-tagging | HTTPS (Azure OpenAI REST API) |
| End User | Requests matching through natural conversation | Natural language via AI assistant |

### 3.2 Technical Context

```text
MCP Client (assistant)
  <-> MCP protocol (tool calls, stdio JSON-RPC)
Teamlandkarte MCP Server (Python)
  <-> Trino (read-only SQL, connection pool)
  <-> Azure OpenAI (chat completions)
  <-> Local caches (DB query cache + search session cache)
```

---

## 4. Solution Strategy

### 4.1 Architecture Approach

**Layered Architecture** with clear separation:
1. **MCP Interface Layer** (`mcp_server.py`): Exposes tools to client, formats Markdown output
2. **Business Logic Layer** (`matching/`): Matcher, scorer, SimilarityEngine, VocabularyCache, BM25, RRF, AutoTagger, **LlmFulltextMatcher** (LLM-based full-text matching with rationale)
3. **Data Access Layer** (`database/`): TrinoClient, connection pool, query logger, schema verification, read-only guard
4. **External Integration Layer** (`azure/`): AzureOpenAIClient, CostTracker
5. **Infrastructure** (`cache/`, `utils/`, `config.py`): QueryCache, SearchCache, date/markdown utilities, configuration loading

### 4.2 Key Design Decisions

| Decision | Rationale |
|----------|-----------|
| **BM25+RRF competence matching** | Lexical matching eliminates embedding false-positives; zero-out rule ensures no token overlap = score 0.0 |
| **LLM-based role similarity** | Semantic role comparison requires understanding synonyms and hierarchy; cached per-run |
| **LLM-based role inference** | Selects from DB vocabulary; constrained to known roles only |
| **LLM-based full-text matching (alternative method)** | `llm_fulltext` mode compares complete capacity/task profiles (description, references, certificates, skills) via Azure OpenAI Chat Completion and returns a category + 1–2 sentence rationale per item; no numeric scores |
| **Optional LLM auto-tagging** | Bridges lexical gap for BM25 by expanding candidate competences with covered synonyms |
| **Global BM25 index** | One index over all candidates' competences ensures stable IDF weights across the pool |
| **Hard confirmation gating (default)** | Prevents accidental matching runs; user must confirm requirements |
| **Markdown table output** | Better rendering in LLM clients; table-first outputs for robust parsing |
| **Connection pooling** | Bounded pool (default 4) for Trino connections; thread-safe acquire/release |
| **Strict startup schema verification** | Fail-fast when upstream views change columns |
| **Two-tier caching** | DB query cache (TTL) + search session cache (TTL) for interactive exploration |
| **Fuzzy filtering** | `fuzzywuzzy` for post-hoc filtering of search results by role/competence text |

### 4.3 Technology Stack

| Layer | Technology | Justification |
|-------|------------|---------------|
| Runtime | Python 3.13 | Modern features, strong typing |
| MCP SDK | `mcp` (FastMCP) | MCP server runtime, stdio transport |
| Database Client | `trino` | Trino/Presto SQL connectivity |
| Caching | `cachetools` (TTLCache) | In-memory TTL caches for DB queries and search sessions |
| Configuration | TOML (`tomllib`) | Human-readable, validated at startup |
| Competence Matching | `rank-bm25` (BM25Okapi) | Lexical ranking, no native extensions |
| Fuzzy Matching | `fuzzywuzzy` + `python-Levenshtein` | Fast fuzzy string matching for filters |
| LLM Integration | `openai` (AsyncAzureOpenAI) | Azure OpenAI chat completions |
| Environment | `python-dotenv` | Load credentials from `.env` |

---

## 5. Building Block View

### 5.1 Level 0: System Context

```
+-------------------------------------------+
|       Teamlandkarte MCP Server            |
|       (Task <-> Capacity Matching)        |
+-------------------------------------------+
```

### 5.2 Level 1: Container View

```
+------------------------------------------------------------+
|                  Teamlandkarte MCP Server                   |
|                                                            |
|  +-------------------------------------------------------+ |
|  | MCP Interface Layer (mcp_server.py)                    | |
|  | - task browsing + details                              | |
|  | - requirement capture + confirmation gate              | |
|  | - task->capacity matching + search refinement          | |
|  | - capacity browsing + capacity->task matching          | |
|  +----------------------------+---------------------------+ |
|                               |                             |
|  +----------------------------v---------------------------+ |
|  | Business Logic Layer (matching/)                        | |
|  | - Matcher (orchestrates matching runs)                  | |
|  | - SimilarityEngine (BM25+RRF competence, LLM role)     | |
|  | - VocabularyCache (LLM role inference from DB vocab)    | |
|  | - Scorer (weighted overall score + categorization)      | |
|  | - Bm25Index + bm25_rank_competences                    | |
|  | - reciprocal_rank_fusion (RRF normalization)            | |
|  | - AutoTagger (optional LLM competence expansion)        | |
|  | - LlmFulltextMatcher (LLM full-text profile matching)   | |
|  +----------------------------+---------------------------+ |
|                               |                             |
|  +----------------------------v---------------------------+ |
|  | Data Access Layer (database/)                           | |
|  | - TrinoClient (SQL queries, implements DBClient)        | |
|  | - ConnectionPool (bounded, thread-safe)                 | |
|  | - SchemaVerifier (startup fail-fast)                    | |
|  | - ReadOnly guard (SELECT/WITH only)                     | |
|  | - QueryLogger (redacted SQL logging)                    | |
|  +----------------------------+---------------------------+ |
|                               |                             |
|  +----------------------------v---------------------------+ |
|  | External Integration (azure/)                           | |
|  | - AzureOpenAIClient (chat completions, retry/backoff)   | |
|  | - CostTracker (session-level token accounting)          | |
|  +--------------------------------------------------------+ |
|                                                            |
|  +--------------------------------------------------------+ |
|  | Infrastructure (cache/, utils/, config.py)              | |
|  | - QueryCache (TTL, DB results)                          | |
|  | - SearchCache (TTL, search sessions + filters)          | |
|  | - dates.py (ISO parsing, overlap checks)                | |
|  | - markdown.py (table rendering)                         | |
|  | - config.py (TOML loading + validation)                 | |
|  +--------------------------------------------------------+ |
+------------------------------------------------------------+
         |                          |
         v                          v
+-----------------+       +-----------------+
|  Open Data Lake |       |  Azure OpenAI   |
|  (Trino)        |       |  (Chat API)     |
+-----------------+       +-----------------+
```

### 5.3 Level 2: Source Module Map

```
src/teamlandkarte_mcp/
├── __init__.py
├── __main__.py          # CLI entry point, arg parsing, logging setup
├── config.py            # TOML config loading + validation (AppConfig)
├── logging_config.py    # Logging to stderr (MCP stdio safety)
├── mcp_server.py        # FastMCP server, all tool definitions, SessionState
├── models.py            # Frozen dataclasses: Task, Capacity, Requirements, ScoredCapacity
├── azure/
│   ├── openai_client.py # AsyncAzureOpenAI wrapper (chat completions, retry)
│   └── cost_tracker.py  # Session-level token/cost accounting
├── cache/
│   ├── query_cache.py   # Generic TTLCache wrapper for DB queries
│   └── search_cache.py  # Search session store (search_id, filter_id, TTL)
├── database/
│   ├── types.py         # DBClient Protocol definition
│   ├── db_client.py     # Factory: create_db_client() -> TrinoClient
│   ├── trino_client.py  # TrinoClient (implements DBClient, uses pool)
│   ├── pool.py          # ConnectionPool (bounded, thread-safe, LIFO)
│   ├── schema_verifier.py # Startup column verification
│   ├── read_only.py     # SQL guard (SELECT/WITH only)
│   ├── query_logger.py  # Redacted query logging
│   └── hive_client.py   # (empty, reserved)
├── matching/
│   ├── matcher.py       # Matcher: orchestrates matching runs
│   ├── scorer.py        # compute_overall(), categorize()
│   ├── similarity.py    # SimilarityEngine (BM25+RRF competence, LLM role)
│   ├── vocabulary.py    # VocabularyCache (LLM role inference)
│   ├── bm25.py          # Bm25Index, bm25_rank_competences, _tokenize
│   ├── rrf.py           # reciprocal_rank_fusion (zero-out rule)
│   ├── auto_tagger.py   # AutoTagger (LLM competence expansion)
│   ├── llm_fulltext_matcher.py  # LlmFulltextMatcher (LLM full-text profile matching)
│   ├── profiles.py      # CapacityProfile, TaskProfile + serializers
│   ├── task_analyzer.py # (deprecated stub)
│   └── task_helpers.py  # (deprecated stub)
└── utils/
    ├── dates.py         # parse_iso_date, availability_overlaps
    └── markdown.py      # md_table (GFM table rendering)
```

### 5.4 Tool Surface (MCP Tools)

#### Discovery (DB tasks)

| Tool | Responsibility | Dependencies |
|------|----------------|--------------|
| `list_open_tasks` | List newest published tasks | DB |
| `get_task_details` | Table-first task fields + inferred role | DB, LLM (role inference) |
| `validate_task_requirements` | Compare DB fields vs LLM-inferred role + competences | DB, LLM |
| `infer_primary_role` | Infer closest role from task id or free text | DB vocab, LLM |

#### Requirement capture + confirmation gate

| Tool | Responsibility | Dependencies |
|------|----------------|--------------|
| `extract_requirements` | LLM-based inference for role + competences from free text | DB vocab, LLM |
| `collect_structured_requirement_data` | Accept user-supplied fields and stage requirements | session state |
| `update_requirements` | (deprecated stub, directs to `collect_structured_requirement_data`) | -- |
| `start_guided_capture` | Start step-by-step guided capture | session state |
| `guided_set_description` | Set description (step 1/4) | session state |
| `guided_set_role` | Set role (step 2/4) | session state |
| `guided_set_time_range` | Set date range (step 3/4) | session state |
| `guided_set_competences` | Set competences (step 4/4) | session state |
| `show_pending_requirements` | Show review table for pending requirements | session state |
| `request_requirements_confirmation` | Mark that assistant asked user to confirm | session state |
| `confirm_requirements` | Confirm or reject staged requirements | session state |

#### Task -> capacity matching

| Tool | Responsibility | Dependencies |
|------|----------------|--------------|
| `find_matching_capacities` | Compute matches and store a search session (`search_id`). Accepts `matching_method` (`"score"` \| `"llm_fulltext"`). | DB, BM25, LLM (role) or LlmFulltextMatcher, caches |
| `filter_search_results` | Refine stored results (fuzzy role/competence, availability, min_similarity). `min_similarity` is ignored in `llm_fulltext` mode and surfaced as a hint in the Applied Filters table. | SearchCache |
| `get_results_by_category` | Page through stored results by category. Renders `Begründung` column instead of score columns when the search was run in `llm_fulltext` mode. | SearchCache |

#### Capacity browsing + capacity -> task matching

| Tool | Responsibility | Dependencies |
|------|----------------|--------------|
| `list_free_capacities` | List recently created free capacities | DB |
| `get_capacity_details` | Table-first capacity fields + next steps | DB |
| `find_matching_tasks` | Find tasks that match a specific capacity. Accepts `matching_method` (`"score"` \| `"llm_fulltext"`). | DB, BM25, LLM (role) or LlmFulltextMatcher, caches |

#### Team browsing + task -> team matching

| Tool | Responsibility | Dependencies |
|------|----------------|--------------|
| `list_teams` | List the first *N* teams (default 20) as a Markdown table (`Team Id`, `Team Name`, `Schwerpunkt`, `Anzahl Kompetenzen`, `Anzahl Referenzen`). | DB |
| `get_team_details` | Table-first team fields plus sections `## Über uns`, `## Leistungen`, `## Interessen`, `## Kompetenzen` (with `(Top)` marker), `## Referenzen` (Partner_Name + Projekte) and `## Next steps`. | DB |
| `find_matching_teams` | Match a task against team profiles and store a search session (`search_id`). Accepts `matching_method` (`"score"` \| `"llm_fulltext"`). Persists `search_type = "team_search"` and the chosen `matching_method` in `SearchCache` and the META JSON. | DB, BM25, LLM (role) or LlmFulltextMatcher, caches |

#### Profile_Type parameter

| Parameter | Allowed values | Default | Selection mechanism | Affected tools |
|-----------|---------------|---------|---------------------|----------------|
| `Profile_Type` | `"capacity"`, `"team"` | implicit per tool name | Tool selection (`find_matching_capacities` ⇒ `capacity`, `find_matching_teams` ⇒ `team`); persisted as `search_type` (`"capacity_search"` / `"team_search"`) in `SearchCache` and META JSON | `find_matching_capacities`, `find_matching_teams`, `list_teams`, `get_team_details` |

- `Profile_Type = "capacity"` (existing behavior): match a task against employee capacity profiles. `find_matching_capacities` is the entry point; availability filters apply.
- `Profile_Type = "team"` (new): match a task against aggregated team profiles. `find_matching_teams` is the entry point; `list_teams` and `get_team_details` browse and inspect team profiles independently of a matching run. Availability filters are **not** applied for team searches and are surfaced as not-effective hints in the `Applied Filters` table.

#### Matching method parameter

| Parameter | Allowed values | Default | Affected tools |
|-----------|---------------|---------|----------------|
| `matching_method` | `"score"`, `"llm_fulltext"` | `[matching].default_method` (TOML, default `"score"`) | `find_matching_capacities`, `find_matching_tasks`, `find_matching_teams` |

- `"score"`: existing BM25+RRF competence + LLM role similarity. Output includes `Role Score`, `Competence Score`, `Overall Score`, `Category`. For team searches, the team's `focus_name` is used as the role stand-in and top competences are weighted by `matching.team.top_competency_weight` (default `1.5`).
- `"llm_fulltext"`: LLM-based full-text comparison via `LlmFulltextMatcher`. Output replaces all score columns with a single `Begründung` column (1–2 sentence rationale from the LLM). The persisted META JSON contains `matching_method` so downstream tools (`get_results_by_category`, `filter_search_results`) know which schema to render.

For team searches, the result table columns are:

| Mode | Columns |
|------|---------|
| `team_search` × `score` | `Team Name`, `Schwerpunkt`, `Top-Kompetenzen`, `Role Score`, `Competence Score`, `Overall Score`, `Category` |
| `team_search` × `llm_fulltext` | `Team Name`, `Schwerpunkt`, `Top-Kompetenzen`, `Category`, `Begründung` |

### 5.5 LlmFulltextMatcher (component)

The `LlmFulltextMatcher` (in `matching/llm_fulltext_matcher.py`) is the business-logic component behind `matching_method = "llm_fulltext"`.

| Aspect | Details |
|--------|---------|
| **Inputs** | A `TaskProfile` (id, title, description, skills) for `match_capacities`; a `CapacityProfile` (id, owner_name, role_name, competences, description, references with `partner_name`/`projects`, certificates) for `match_tasks`. The candidate list (`capacities` or `tasks`) is already prefiltered by the MCP tool (same prefilter as in `score` mode, e.g. availability). |
| **Profile sources** | Built from in-memory `Capacity`/`Task` objects plus extras loaded from the DB via `DBClient.batch_get_capacity_descriptions`, `batch_get_capacity_certificates`, and `batch_get_capacity_references` (with the partner LEFT JOIN inside the same SQL query). |
| **Outputs** | `LlmFulltextResult` with `by_category: dict[str, list[LlmFulltextItem]]` (categories `Top`/`Good`/`Partial`/`Low`/`Irrelevant`; each item carries `category`, ungekürzte `rationale`, raw fields) and `errors: list[LlmFulltextError]` for items where the LLM call/parse failed. |
| **External dependency** | Azure OpenAI Chat Completion via `AzureOpenAIClient.chat_completion(...)` with `response_format=json_object`, deterministic German system prompt, JSON schema `{"category": "...", "rationale": "..."}`. One LLM call per candidate. |
| **Determinism** | Profile serialization is field-stable; results within a category are sorted lexicographically by `item_id`; invalid LLM categories map to `Irrelevant` with a hint appended to the rationale. |
| **Persistence** | Items are stored in `SearchCache` with `category` + ungekürzte `rationale` (no numeric score fields); `errors` are persisted alongside; META JSON carries `matching_method`. |

---

## 6. Runtime View (Key Flows)

### 6.1 Server startup (fail-fast)

1. Parse CLI arguments (`--config`, `--log-level`)
2. Configure logging (stderr only, MCP stdio safety)
3. Load and validate `config.toml` + environment variables
4. Create DB client (TrinoClient with connection pool)
5. Verify required DB view columns (schema verification -> fail-fast on mismatch)
6. Initialize caches (QueryCache, SearchCache)
7. Initialize Azure OpenAI client + CostTracker
8. Optionally construct AutoTagger (if `use_auto_tagging = true`)
9. Initialize SimilarityEngine, VocabularyCache, Matcher
10. Register all MCP tools
11. Start MCP server (stdio transport via `mcp.run()`)

### 6.2 Task -> capacity matching

1. Capture requirements (guided capture or `extract_requirements` or `collect_structured_requirement_data`)
2. (If `require_confirmation = true`) confirmation gate:
   - `show_pending_requirements` -> user reviews
   - `request_requirements_confirmation` -> assistant marks confirmation requested
   - `confirm_requirements(true)` -> user confirms
3. Run `find_matching_capacities(role_name, competences, date_start?, date_end?, matching_method?)`:
   - Resolve `matching_method` (parameter > `[matching].default_method` > `"score"`)
   - Query all capacities + competences from DB (cached)
   - Filter by date overlap (same prefilter for both methods)
   - Branch by `matching_method`:
     - `"score"`: build global BM25 index, run BM25+RRF competence similarity + LLM role similarity, compute weighted overall score + categorize
     - `"llm_fulltext"`: batch-load `description`, `certificates`, `references` (with partner LEFT JOIN) for the filtered capacities; build `TaskProfile` + `CapacityProfile`; one Azure OpenAI Chat Completion per capacity (`response_format=json_object`); LLM returns `{category, rationale}`; invalid categories → `Irrelevant` with hint; LLM/JSON failures recorded in a separate `errors` list
   - Store results in SearchCache (payload includes `matching_method`; LLM mode stores `category` + ungekürzte `rationale` per item, no score fields) -> return `search_id`
4. Browse/filter results:
   - `get_results_by_category(search_id, category, page, page_size)` (renders `Begründung` column in LLM mode)
   - `filter_search_results(search_id, ...)` -> returns `filter_id`. In LLM mode, `min_similarity` is ignored and surfaced as a hint; sorting falls back to `(category_rank, item_id)`.

### 6.3 Capacity -> task matching

1. Browse capacities (`list_free_capacities`)
2. Inspect (`get_capacity_details`)
3. Run `find_matching_tasks(capacity_id, matching_method?)`:
   - Resolve `matching_method` (parameter > `[matching].default_method` > `"score"`)
   - Query capacity details + competences from DB
   - Query open tasks from DB (same prefilter for both methods)
   - Branch by `matching_method`:
     - `"score"`: for each task infer role + competences, compute BM25+RRF + LLM role similarity
     - `"llm_fulltext"`: load `description`, `certificates`, `references` (with partner LEFT JOIN) for the capacity; build `CapacityProfile` once; for each task build a `TaskProfile` and run one Azure OpenAI Chat Completion; same error/category-normalization handling as in 6.2
   - Store results as a search session (`matching_method` persisted) -> return `search_id`
4. Browse/filter results (same tools as task->capacity)

### 6.4 Task -> team matching

1. Capture requirements as in 6.2 (guided capture, `extract_requirements`, or `collect_structured_requirement_data`)
2. Optional confirmation gate (same as in 6.2): `show_pending_requirements` -> `request_requirements_confirmation` -> `confirm_requirements(true)`
3. Run `find_matching_teams(role_name, competences, matching_method?)`:
   - Resolve `matching_method` (parameter > `[matching].default_method` > `"score"`); reject unknown values with an error listing the allowed values, without any DB or LLM call
   - Validate minimum requirements (`competences` non-empty)
   - Load all teams from the DB-backed cache `_get_teams_cached()`. The DB layer joins `teamlandkarte_v_teams_latest` with `teamlandkarte_v_teammeter_organizational_units_latest` (INNER JOIN over `team_id = id` for the team name), and aggregates competences (`teamlandkarte_v_teammeter_team_competences_latest` joined to `teamlandkarte_v_competences_latest` over `competence_id = id`) and references (`teamlandkarte_v_team_references_latest` LEFT JOIN `teamlandkarte_v_partners_latest` over `partner_id = id`, exposing `name` as `Partner_Name`). Joins between team master data, competences, and references run over `ouid`.
   - **No availability prefilter**: team profiles do not carry an availability range; any provided date range is ignored and surfaced as a not-effective hint in the `Applied Filters` table.
   - Branch by `matching_method`:
     - `"score"`: build the requirements competence list, derive each team's competence list and parallel top-competence set from `Team.competences`, and compute competence similarity via the existing `SimilarityEngine`. Top competences are upweighted by `matching.team.top_competency_weight` (default `1.5`); the per-required score is clamped to `[0, 1]`. Role similarity uses `Team.focus_name` as the role stand-in (`compute_role_similarity(req.role_name, team.focus_name)`). Overall score and category use the same `compute_overall(...)` / `categorize(...)` thresholds as for capacities.
     - `"llm_fulltext"`: build a `TaskProfile` from the requirements and a `TeamProfile` per team via `build_team_profile` + `serialize_team_profile`. The user prompt is `=== Aufgabe ===` + `=== Team ===` (with an `ID:` header). The system prompt and JSON schema (`{"category": ..., "rationale": ...}`) are reused from the capacity LLM path; invalid categories are normalized to `Irrelevant` with a hint suffix in the rationale; LLM/JSON errors land in a separate `errors` list.
   - Store results in `SearchCache` with `search_type = "team_search"` and `matching_method = ...` -> return `search_id`. The META JSON exposes both `search_type` and `matching_method` so `get_results_by_category` / `filter_search_results` render the team-specific columns.
4. Browse/filter results:
   - `get_results_by_category(search_id, category, page, page_size)` (renders `Begründung` column in `llm_fulltext` mode)
   - `filter_search_results(search_id, ...)`. `role_filter` matches against `Team_Focus_Name`; `competence_filter` matches the competence names of the team and supports a `(Top)` suffix that restricts the filter to top competences. `availability_date_start`, `availability_date_end`, and `is_fully_available` are ignored for `team_search` and surfaced as `team_search` not-effective hints in the `Applied Filters` table.

### 6.5 Role inference flow

1. `infer_primary_role(task_id=... | task_text=...)`
2. Fetch all role names from DB vocabulary view
3. Send task text + role list to LLM (JSON response format)
4. Validate returned role exists in DB vocabulary
5. Return `(role_name, confidence)` or `None` on failure

---

## 7. Deployment View

```
+-------------------------------------------------+
|  Developer Machine / CI                          |
|                                                  |
|  +--------------------------------------------+ |
|  |  MCP Client (IDE / AI Assistant)           | |
|  |  <-> stdio (JSON-RPC)                      | |
|  |  teamlandkarte-mcp process                 | |
|  |  (Python 3.13, single process)             | |
|  +--------------------------------------------+ |
|                                                  |
|  config.toml + .env (credentials)                |
+-------------------------------------------------+
         |                          |
         v                          v
+-----------------+       +-----------------+
|  Trino cluster  |       |  Azure OpenAI   |
|  (Data Lake)    |       |  (Chat API)     |
+-----------------+       +-----------------+
```

**Requirements:**
- Python 3.13+ with project dependencies installed
- Trino connectivity (host/port via `config.toml`, credentials via env vars)
- Azure OpenAI endpoint + chat deployment + API key (env var `AZURE_OPENAI_LLM_API_KEY`)
- Environment variables: `DATA_LAKE_USERNAME`, `DATA_LAKE_PASSWORD`, `AZURE_OPENAI_LLM_API_KEY`

---

## 8. Cross-cutting Concepts

### 8.1 Deterministic outputs

- Tools produce **table-first Markdown output** (GFM tables via `md_table()`).
- Search tools emit deterministic headers with `SEARCH_ID=...` / `FILTER_ID=...` for robust client parsing.
- Scoring formula is deterministic: `overall = competence_weight * competence_score + role_weight * role_score`.

### 8.2 Caching strategy

| Cache | Implementation | TTL | Purpose |
|-------|---------------|-----|---------|
| DB query cache | `QueryCache` (cachetools TTLCache) | Configurable (`db_ttl_hours`, default 12h) | Avoid repeated expensive Trino queries |
| Search session cache | `SearchCache` (cachetools TTLCache) | Configurable (`search_ttl_minutes`, default 60min) | Enable interactive exploration of results |
| LLM role similarity cache | In-memory dict on `SimilarityEngine` | Per matching run (cleared between runs) | Avoid duplicate LLM calls for same role pair |

### 8.3 Security

- **Read-only DB access**: `ensure_select_only()` guard rejects non-SELECT/WITH queries
- **No write operations**: Server never modifies the Data Lake
- **Credentials via environment**: `DATA_LAKE_USERNAME`, `DATA_LAKE_PASSWORD`, `AZURE_OPENAI_LLM_API_KEY` -- never in config files
- **Query logging**: Parameters are redacted before logging (`query_logger.py`)
- **Logging to stderr**: MCP stdio transport requires stdout to remain clean JSON-RPC

### 8.4 Error handling

- **Schema verification failure**: Server refuses to start (fail-fast)
- **DB connectivity**: Lazy validation on first DB-backed tool call
- **LLM failures (role similarity)**: Return 0.0, no caching of failed result
- **LLM failures (auto-tagging)**: Return unmodified candidate list (graceful degradation)
- **LLM failures (role inference)**: Return `None`, logged as warning
- **Connection pool exhaustion**: `PoolExhaustedError` raised immediately (no blocking wait)
- **Config errors**: `ConfigError` raised at startup with descriptive message

### 8.5 Connection pooling

- `ConnectionPool` (generic, thread-safe, LIFO queue)
- Bounded at `pool_size` (default 4) connections
- Connections created lazily on demand
- `closeall()` for graceful shutdown

---

## 9. Scoring and Matching Algorithm

### 9.1 Overall scoring formula

```
overall_score = competence_weight * competence_score + role_weight * role_score
```

Default weights: `competence_weight = 0.8`, `role_weight = 0.2` (must sum to 1.0).

### 9.2 Competence scoring (BM25 + RRF)

For each required competence:
1. Query the global BM25 index (built over all filtered candidates' competences)
2. Filter ranked results to the current candidate's competence set
3. Apply Reciprocal Rank Fusion (k=60) to normalize scores to [0.0, 1.0]
4. **Zero-out rule**: No token overlap -> score 0.0 (eliminates false positives)

Aggregate: `competence_score = mean(per_required_competence_scores)`

Threshold for "matched": score >= 0.5.

### 9.3 Role scoring (LLM)

- LLM chat completion compares two role names semantically -> similarity in [0.0, 1.0]
- Identical roles (case-insensitive) -> 1.0 (short-circuit, no LLM call)
- Empty/unknown roles -> 0.0 (short-circuit)
- Results cached symmetrically per matching run

### 9.4 Categorization

| Category | Threshold |
|----------|-----------|
| Top | overall >= configured `top` (default 0.8) |
| Good | overall >= configured `good` (default 0.65) |
| Partial | overall >= configured `partial` (default 0.5) |
| Low | overall >= configured `low` (default 0.3) |
| Irrelevant | overall < configured `low` |

(Thresholds are configurable via `[matching.thresholds]`.)

### 9.5 Optional: Auto-Tagging (LLM competence expansion)

When `matching.similarity.use_auto_tagging = true`:
1. Before BM25 scoring, the AutoTagger asks the LLM which required competences are already covered by the candidate's existing competences (via synonym, abbreviation, cross-language equivalence)
2. Covered required-competence names are appended to the candidate's working list
3. Expansion is **ephemeral** -- never persisted to DB
4. On LLM failure, scoring proceeds with the unmodified list

---

## 10. Configuration Reference

### 10.1 `config.toml` structure

```toml
[database]
backend = "trino"           # Only "trino" supported
host = "..."
port = 443
http_scheme = "https"
verify_ssl = true
catalog = "hive"
schema = "tier1_open_lake"
connect_timeout = 10
pool_size = 4

[matching]
competence_weight = 0.8     # Must sum to 1.0 with role_weight
role_weight = 0.2
require_confirmation = true # Hard gate before matching
default_method = "score"    # Default for matching_method when callers omit it.
                            # Allowed: "score" | "llm_fulltext".

[matching.thresholds]
top = 0.8
good = 0.65
partial = 0.5
low = 0.3

[matching.fuzzy]
min_similarity = 0.7       # Fuzzy filter threshold

[matching.similarity]
use_auto_tagging = false   # LLM pre-expansion for BM25

[cache]
db_ttl_hours = 12
search_ttl_minutes = 60
max_size = 100

[azure_openai]
endpoint = "https://..."
api_version = "2024-02-15-preview"
chat_deployment = "gpt-4.1"
show_costs_in_output = false
```

### 10.2 Required environment variables

| Variable | Purpose |
|----------|---------|
| `DATA_LAKE_USERNAME` | Trino login username |
| `DATA_LAKE_PASSWORD` | Trino login password |
| `AZURE_OPENAI_LLM_API_KEY` | Azure OpenAI API key for chat completions |

---

## 11. Data Model

### 11.1 Core domain objects (frozen dataclasses)

| Class | Fields | Purpose |
|-------|--------|---------|
| `Task` | id, name, title, description, start_date, end_date, created_date, skills | Published task from DB |
| `Capacity` | id, owner_name, role_name, role_level, begin_date, end_date, competences | Employee capacity entry |
| `Requirements` | role_name, competences, date_start, date_end, description | Structured matching input |
| `ScoredCapacity` | capacity, competence_score, role_score, overall_score, category, matched_competences, missing_competences | Matching result |
| `Team` | team_id, ouid, team_name, focus_name, about_us, offerings, interests, competences, references | Aggregated team master data |
| `TeamCompetence` | name, top_competency | Single team competence with top marker |
| `TeamReference` | partner_name, projects | Single team reference; `partner_name` may be empty |
| `ScoredTeam` | team, competence_score, role_score, overall_score, category, matched_competences, missing_competences | Team matching result (score mode) |

### 11.2 Team Profile

A **Team Profile** is the second profile type next to the existing capacity profile. It aggregates fields from four read-only views into a single immutable `Team` instance and is used both for browsing (`list_teams`, `get_team_details`) and for matching (`find_matching_teams` in both `score` and `llm_fulltext` modes).

| Field | Type | Source | Notes |
|-------|------|--------|-------|
| `team_id` | `str` | `teamlandkarte_v_teams_latest.team_id` | Stable identifier; persisted in `SearchCache`. |
| `ouid` | `str` | `teamlandkarte_v_teams_latest.ouid` | Join key for competences and references. |
| `team_name` | `str` | `teamlandkarte_v_teammeter_organizational_units_latest.name` | Resolved via INNER JOIN `teams_latest.team_id = organizational_units_latest.id`. Teams without an OU match are excluded. |
| `focus_name` | `str` | `teamlandkarte_v_teams_latest.focus_name` | NULL → `""`. Used as role stand-in in `score` mode. |
| `about_us` | `str` | `teamlandkarte_v_teams_latest.about_us` | NULL → `""`. |
| `offerings` | `str` | `teamlandkarte_v_teams_latest.offerings` | NULL → `""`. |
| `interests` | `str` | `teamlandkarte_v_teams_latest.interests` | NULL → `""`. |
| `competences` | `list[TeamCompetence]` | `teamlandkarte_v_teammeter_team_competences_latest` joined to `teamlandkarte_v_competences_latest` over `competence_id = id` | Joined on `ouid`. `top_competency = COALESCE(top_competency, FALSE)`. Order: top competences first, then name ascending. |
| `references` | `list[TeamReference]` | `teamlandkarte_v_team_references_latest` LEFT JOIN `teamlandkarte_v_partners_latest` over `partner_id = id` | Joined on `ouid`. Partner_Name from `p.name` (`COALESCE(..., '')`). Whitespace-only `projects` are filtered. Order: partner_name asc, projects asc. |

**Profile serialization** (`build_team_profile` + `serialize_team_profile` in `matching/profiles.py`) emits a deterministic, German-headed text block in fixed order: `Teamname:`, `Schwerpunkt:`, `Über uns:`, `Leistungen:`, `Interessen:`, `Kompetenzen:`, `Referenzen:`. Top competences are suffixed with `(Top)`. Reference rows render as `Partner: <partner_name> – Projekte: <projects>` when the partner is known, and as `Projekte: <projects>` (no placeholder) when the partner is empty. Empty lists render as `Kompetenzen: (keine)` / `Referenzen: (keine)`. List ordering matches the DB-provided order.

### 11.3 Database views (Trino)

The server queries read-only views in the `tier1_open_lake` schema. Schema verification at startup checks:
- `teamlandkarte_v_capacity_roles_latest`: requires columns `name`, `active`, `staffing_board_relevant`
- `teamlandkarte_v_capacities_latest`: requires column `creation_date`
- `teamlandkarte_v_teams_latest`: requires columns `team_id`, `ouid`, `about_us`, `offerings`, `interests`, `focus_name`
- `teamlandkarte_v_teammeter_organizational_units_latest`: requires columns `id`, `name`
- `teamlandkarte_v_teammeter_team_competences_latest`: requires columns `ouid`, `competence_id`, `top_competency`
- `teamlandkarte_v_team_references_latest`: requires columns `ouid`, `partner_id`, `projects`

#### Additional views read in `llm_fulltext` mode

The following columns/views are not part of the startup schema verification but are read at runtime when `matching_method = "llm_fulltext"`:

| View | Columns used | Used for |
|------|--------------|----------|
| `teamlandkarte_v_capacities_latest` | `description` | `CapacityProfile.description` |
| `teamlandkarte_v_capacity_certificates_latest` | `capacity_id`, `description` | `CapacityProfile.certificates` (1:n via `capacity_id`) |
| `teamlandkarte_v_capacity_references_latest` | `capacity_id`, `partner_id`, `projects` | `CapacityProfile.references[].projects` (1:n via `capacity_id`) |
| `teamlandkarte_v_partners_latest` | `id`, `name` | Partner name per reference (`CapacityProfile.references[].partner_name`) |

**Reference ↔ Partner join:** `teamlandkarte_v_capacity_references_latest.partner_id = teamlandkarte_v_partners_latest.id`. The partner `name` column is exposed as `partner_name` (via `COALESCE(p.name, '')`) inside the same SQL query that loads the references, so no extra round-trip is required. When `partner_id` is `NULL` or no partner matches, `partner_name` is the empty string and the reference is still returned with its `projects` text. The LEFT JOIN keeps the reference list complete even for unknown partners.

#### Views read for team profiles

The following views back the `Team` data type and are loaded by `find_matching_teams`, `list_teams`, and `get_team_details`:

| View | Columns used | Used for |
|------|--------------|----------|
| `teamlandkarte_v_teams_latest` | `team_id`, `ouid`, `about_us`, `offerings`, `interests`, `focus_name` | Team master data and free-text fields |
| `teamlandkarte_v_teammeter_organizational_units_latest` | `id`, `name` | `Team.team_name` via INNER JOIN over `teams_latest.team_id = organizational_units_latest.id` |
| `teamlandkarte_v_teammeter_team_competences_latest` | `ouid`, `competence_id`, `top_competency` | Team competences with top markers (joined to `teamlandkarte_v_competences_latest` over `competence_id = id` to resolve the competence name) |
| `teamlandkarte_v_team_references_latest` | `ouid`, `partner_id`, `projects` | Team references with partner names |
| `teamlandkarte_v_partners_latest` | `id`, `name` | `Partner_Name` per team reference |

**Team_Name join (INNER JOIN):** `teamlandkarte_v_teams_latest.team_id = teamlandkarte_v_teammeter_organizational_units_latest.id`. Teams without a matching OU row are excluded from `get_all_teams` and `get_team_by_id`.

**Competence and reference joins:** Both `teamlandkarte_v_teammeter_team_competences_latest` and `teamlandkarte_v_team_references_latest` are joined to the team master data over the shared `ouid` column.

**Reference ↔ Partner join (LEFT JOIN):** `teamlandkarte_v_team_references_latest.partner_id = teamlandkarte_v_partners_latest.id`. The `name` column from `teamlandkarte_v_partners_latest` is exposed as `Partner_Name` via `COALESCE(p.name, '')` inside the same SQL query that loads the references. When `partner_id` is `NULL` or no partner matches, `Partner_Name` is the empty string and the reference is still returned with its `projects` text. Whitespace-only `projects` are filtered out in the DB layer.

---

## 12. Testing

- **Unit tests**: pytest + pytest-asyncio
- **Property-based tests**: hypothesis (for scoring, BM25, RRF invariants)
- **Type checking**: mypy (strict)
- **Linting/formatting**: ruff
- **No integration test infrastructure**: DB and Azure OpenAI are mocked in tests