Orchestrator/bahn/teamlandkarte-mcp/openspec/changes/add-capacity-matching-mcp-server/proposal.md

# Change: Add Capacity Matching MCP Server (Stage 1)

## Why
DB Systel employees have free work capacities stored in the Open Data Lake, but there's no efficient way for AI assistants to match these capacities with tasks. Tasks are also stored in the database, but finding suitable capacities for them requires manual work. This MCP server will enable AI clients to automatically:
1. Browse and search published tasks from the database
2. Validate task requirements (comparing DB skills vs description-extracted skills)
3. Find the best-suited employees for both database tasks and ad-hoc requirements
4. Match capacities based on competences and role, with optional availability (date-range) filtering

## What Changes
- Add MCP server implementation with **10-tool workflow** (task management, requirement gathering, search, and refinement)
- Add **database task management** (4 new tools): list tasks, view details, validate requirements, automated search
- Add **task database integration**: Query published tasks with skills from Beschaffungstool KMP tables
- Add **requirement validation**: Compare database skills vs LLM-extracted skills from description
- Add **common extraction function**: DRY principle with `_extract_requirements_from_description()`
- Add **internal helper functions**: `extract_requirements_from_task()`, `extract_roles_from_task()`
- Add database connection layer for Trino/Presto with single join query
- Requirement extraction and semantic matching are implemented via **Azure OpenAI**:
  - Chat completions for role/requirement extraction
  - Embeddings + cosine similarity for semantic similarity scoring
  - Errors are deterministic (no heuristic fallbacks)

> **Historical note (2026-02)**: The initial Stage-1 proposal assumed only
> client-side LLM access via MCP sampling and used heuristic fallbacks because
> sampling was not available in the runtime. The current implementation has been
> migrated to an Azure OpenAI-only integration (see change
> `replace-heuristics-with-azure-openai`).

- Add **session state management** for requirement tracking across tool calls
- Add configurable scoring system with categorical results (Top/Good/Partial/Low)
- Add **two-tier caching**: database cache (12h TTL) and search results cache (60min TTL) with task_id tracking
- Emit deterministic, machine-readable headers in search-related tool outputs so MCP
  clients can reliably capture `search_id` / `filter_id`:
  - first line: `Using SEARCH_ID=<uuid>`
  - plus `SEARCH_ID=<uuid>`, `FILTER_ID=<uuid>` (if applicable), `META=<json>`
- Reject non-UUID `search_id` values early with a clear message
- Track the latest search id internally for debugging/consistency, but do **not** expose a "last search id" tool (multi-user/session unsafe)
- Add **requirement workflow**: extract from free text, collect structured, or update requirements
- Add **explicit confirmation** before search execution
- Remove availability from weighted scoring (availability is displayed and can be used as a filter)
- Update default weights: competence 0.8, role 0.2
- Keep date range as an optional availability filter (when provided)
- Add interactive result exploration with **search session management** (search_id)
- Add **fuzzy filtering** for post-search refinement (role and/or competences as list)
- Add **filter tracking** with filter_id (no search_id proliferation)
- Add **pagination** for result browsing (20 results per page)
- Add database configuration in `database.toml` (Trino)
- Add **Arc42 architecture documentation** with ADRs

### Strict two-step confirmation (Review → Ask → Confirm)

When `matching.require_confirmation = true` (default), the server enforces a
strict, user-driven confirmation sequence before any matching run:

1. Requirements are captured/updated into **pending** state (e.g. via
   `extract_requirements`, `collect_structured_requirement_data`, `update_requirements`,
   or guided capture tools).
2. The assistant must **review** pending requirements using
   `show_pending_requirements()` (preferred) or, if requirements were already
   shown to the user in chat, call `request_requirements_confirmation()` as a
   marker.
3. The assistant must **ask** the user explicitly for Yes/No confirmation.
4. Only after user approval, the assistant calls `confirm_requirements(confirm=true)`.
   If the user says No, it calls `confirm_requirements(confirm=false)` and
   continues requirement capture.

The server refuses to confirm requirements unless step 2 has happened first
(prevents auto-confirm in the same turn as requirement capture).

**Stage 1 Scope:**
This is the first development stage focusing on basic matching functionality and database task integration. Future stages will add more sophisticated features (to be specified later).

**Key Architectural Decisions:**
- **Azure OpenAI integration**: Server calls Azure OpenAI directly for extraction
  and embeddings (no sampling dependency)
- **10-tool workflow**: Separate tools for task management (4), requirement gathering (3), search execution (1), and result refinement (2)
- **Database task integration**: Direct access to published tasks with skills from Beschaffungstool KMP
- **Validation-only approach**: Validation compares DB vs description but always uses DB skills for search
- **Common extraction function**: `_extract_requirements_from_description()` as single source of truth (DRY principle)
- **Automatic task workflow**: `find_capacities_for_task()` fully automated with validation display
- **Task reference tracking**: Search cache includes `task_id` field (null for ad-hoc, task ID for DB tasks)
- **Explicit confirmation**: Requirements returned for user confirmation before search
- **Session state**: In-memory storage of requirements for update workflow
- **Filter tracking**: Filters stored within search cache using filter_id (not new search_id)
- **Search sessions**: Complete results stored server-side, accessed via search_id (60min expiry)
- **Single query pattern**: One JOIN query fetches all data, processed in Python
- **Markdown output**: All tables formatted as markdown for better client rendering
- **No truncation**: Full competence lists always shown
- **No availability scoring**: Availability is displayed and can be used as a date-range filter, but does not influence the score
- **Weights**: competence 0.8, role 0.2
- **Fuzzy matching**: Filter tool uses fuzzywuzzy for flexible result refinement
- **LLM integration**: Azure OpenAI (`gpt-4.1` chat + `text-embedding-3-large`)
  with embedding cache; no heuristic fallback

## Impact
- **New capability**: `capacity-matching` - Core matching functionality with database task integration
- **New files**:
  - `database.toml` - Database connection configuration
  - `architecture.md` - Arc42 architecture documentation with ADRs
  - `src/teamlandkarte_mcp/mcp_server.py` - MCP server implementation (10 tools)
  - `src/teamlandkarte_mcp/database/trino_client.py` - Trino client (capacity + task queries)
  - `src/teamlandkarte_mcp/matching/task_analyzer.py` - Azure-based task analysis
  - `src/matching/matcher.py` - Matching algorithm with common extraction function
  - `src/matching/scorer.py` - Scoring logic
  - `src/cache/query_cache.py` - Database query caching (12h TTL)
  - `src/cache/search_cache.py` - Search results caching (60min TTL) with task_id tracking
  - `src/config.py` - Configuration management
  - `requirements.txt` or updated `pyproject.toml` - Dependencies
- **Modified files**:
  - `main.py` - Launch MCP server
  - `pyproject.toml` - Add dependencies
  - `README.md` - Update with task management features, validation workflow
- **Dependencies added**:
  - `mcp` - Model Context Protocol SDK
  - `trino` - Trino/Presto database client
  - `cachetools` - Caching implementation
  - `fuzzywuzzy` - Fuzzy string matching for filters

## Security & Constraints
- **Read-only operations**: Server must NEVER modify database data
- **Credential storage**: Credentials are provided via environment variables
  (recommended: local `.env`): `DATA_LAKE_USERNAME`, `DATA_LAKE_PASSWORD`.
  `database.toml` contains non-secret connection settings and must not be
  committed.
- **Network security**: Trino/Presto connection over port 8446
- **Data filtering**: 
  - Capacities: Only consider where `deletion_reason IS NULL`
  - Tasks: Only consider where `status__c = "Veröffentlicht"`
- **LLM access**: Not available in the current server runtime. If reintroduced,
  it must use a supported MCP API.
- **SQL injection prevention**: All queries use proper parameterization
- **Validation scope**: Comparison only (skills and dates), role not validated (not in DB yet)

## Non-Goals (Future Stages)
- Advanced filtering options beyond fuzzy matching
- Capacity booking/reservation
- Task modification or status updates
- Historical analysis
- Integration with other systems
- Multi-language support
- Role storage in database (currently extracted from description)