Orchestrator/bahn/teamlandkarte-mcp/openspec/changes/replace-heuristics-with-azure-openai/tasks.md

# Tasks: Replace Heuristics with Azure OpenAI

> Change: `replace-heuristics-with-azure-openai`
> 
> Status: **Approved / Implemented**

## Overview

Replace legacy client-side sampling/heuristic logic with direct Azure OpenAI integration:
- Embeddings (`text-embedding-3-large`) for competence/role similarity
- LLM (`gpt-4.1`) for requirements extraction and validation
- Persistent SQLite embedding cache
- Two configurable similarity strategies

# Removed: migration note about database.toml rename (now a completed internal detail).

## Estimated Effort

**Total: 28-35 hours** (1 week full-time or 2 weeks part-time)

## Task Breakdown

### Phase 1: Configuration & Infrastructure (3-4 hours)

- [x] 1.1 Rename `database.toml` → `config.toml`
- [x] 1.2 Rename `database.toml.example` → `config.toml.example`
- [x] 1.3 Update all file references in code (`README.md`, `docs/`, `src/config.py`, etc.)
- [x] 1.4 Extend `src/config.py`: Add `AzureOpenAIConfig` model
  - `endpoint`, `api_version_embeddings`, `api_version_llm`
  - `embeddings.model`, `embeddings.deployment` (deployment must equal model name)
  - `llm.model`, `llm.deployment` (deployment must equal model name), `llm.temperature`, `llm.max_tokens`
- [x] 1.5 Extend `src/config.py`: Add `EmbeddingCacheConfig` model
  - `enabled`, `db_path`, `ttl_days`
- [x] 1.6 Extend `src/config.py`: Add `matching.similarity.strategy` field (`"per_skill"` or `"aggregate"`)
- [x] 1.7 Update `config.toml.example` with new `[azure_openai]`, `[embedding_cache]`, `[matching.similarity]` sections
- [x] 1.8 Load `AZURE_OPENAI_EMBEDDING_API_KEY` from environment in `load_config()`
- [x] 1.9 Load `AZURE_OPENAI_LLM_API_KEY` from environment in `load_config()`
- [x] 1.10 Add `openai` package to `pyproject.toml` dependencies
- [x] 1.11 Run `uv sync` to install dependencies
- [x] 1.12 Add `config.toml` to `.gitignore` (if not already there)
- [x] 1.13 Configure pytest integration marker in `pytest.ini` (if not already present):
  - `integration: marks tests that call real Azure OpenAI API (deselect with '-m "not integration"')`

### Phase 2: Embedding Cache (2-3 hours)

- [x] 2.1 Create `src/teamlandkarte_mcp/cache/embedding_cache.py`
- [x] 2.2 Implement `EmbeddingCache` class:
  - `__init__(db_path: str, ttl_days: int)`
  - Create SQLite database and schema if not exists:
    ```sql
    CREATE TABLE IF NOT EXISTS embeddings (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
        text TEXT UNIQUE NOT NULL,
        model TEXT NOT NULL,
        embedding BLOB NOT NULL,
        created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
    )
    CREATE INDEX IF NOT EXISTS idx_text_model ON embeddings(text, model)
    ```
- [x] 2.3 Implement `EmbeddingCache.get(text: str, model: str) -> Optional[list[float]]`
  - Query by text + model
  - Check if `created_at` is within TTL
  - Deserialize BLOB to `list[float]`
  - Return `None` if missing or expired
- [x] 2.4 Implement `EmbeddingCache.put(text: str, model: str, embedding: list[float]) -> None`
  - Serialize embedding as UTF-8 bytes of JSON: `json.dumps(embedding).encode("utf-8")`
  - INSERT OR REPLACE into database
- [x] 2.5 Implement `EmbeddingCache.cleanup_expired() -> int`
  - DELETE embeddings older than TTL
  - Return count of deleted rows
- [x] 2.6 Add graceful handling for concurrent access (SQLite locks)
- [x] 2.7 Create `tests/test_embedding_cache.py`
  - Test cache hit/miss
  - Test TTL expiration
  - Test cleanup
  - Test concurrent access
- [x] 2.8 Add `embeddings_cache.db` to `.gitignore`

### Phase 3: Azure OpenAI Client (3-4 hours)

- [x] 3.1 Create `src/teamlandkarte_mcp/azure/__init__.py`
- [x] 3.2 Create `src/teamlandkarte_mcp/azure/openai_client.py`
- [x] 3.3 Define `AzureAPIError(RuntimeError)` exception class
- [x] 3.4 Implement `AzureOpenAIClient` class:
  - `__init__(config: AzureOpenAIConfig, embedding_cache: Optional[EmbeddingCache])`
  - Initialize two `AzureOpenAI` clients (embeddings + LLM) using `AzureKeyCredential`
- [x] 3.5 Implement `AzureOpenAIClient.get_embedding(text: str) -> list[float]`
  - Check cache first (if enabled)
  - If cache miss: call Azure API
  - Store in cache (if enabled)
  - Raise `AzureAPIError` on failure (no fallback)
- [x] 3.6 Implement `AzureOpenAIClient.get_embeddings_batch(texts: list[str]) -> list[list[float]]`
  - Call `get_embedding()` for each text individually (per user requirement #3)
  - Return list of embeddings in same order
- [x] 3.7 Implement `AzureOpenAIClient.chat_completion(system: str, user: str, response_format: Optional[dict] = None) -> str`
  - Call Azure OpenAI chat completions API
  - Use `gpt-4.1` model with configured temperature/max_tokens
  - Support `response_format={"type": "json_object"}` for structured JSON
  - Implement retry logic (3 attempts, exponential backoff)
  - Raise `AzureAPIError` on final failure (no fallback)
- [x] 3.8 Add logging for API calls (cache hit/miss, request/response sizes)
- [x] 3.9 Create `tests/test_azure_client.py`
  - Mock Azure API with `unittest.mock`
  - Test embedding retrieval with cache hit/miss
  - Test LLM completion
  - Test error handling and retries
- [x] 3.10 Create `tests/test_azure_client_integration.py` (marked `@pytest.mark.integration`)
  - Real API calls (requires valid credentials)
  - Test embedding generation
  - Test LLM structured JSON response

### Phase 4: Similarity Engine (4-5 hours)

- [x] 4.1 Create `src/teamlandkarte_mcp/matching/similarity.py`
- [x] 4.2 Implement `cosine_similarity(vec_a: list[float], vec_b: list[float]) -> float`
  - Compute dot product and magnitudes
  - Return cosine similarity in [0.0, 1.0]
  - Handle zero vectors gracefully
- [x] 4.3 Implement `SimilarityEngine` class:
  - `__init__(client: AzureOpenAIClient, strategy: str = "per_skill")`
  - Validate strategy is `"per_skill"` or `"aggregate"`
- [x] 4.4 Implement `SimilarityEngine._per_skill_similarity(required: list[str], candidate: list[str]) -> dict[str, dict[str, Any]]`
  - For each required skill:
    - Get embedding
    - Compare with all candidate skill embeddings
    - Find best match (highest cosine similarity)
    - Return `{required: {best_match: str|None, score: float, rationale: str}}`
  - Rationale example: `"Cosine similarity: 0.92 (best match: 'React.js')"`
- [x] 4.5 Implement `SimilarityEngine._aggregate_similarity(required: list[str], candidate: list[str]) -> dict[str, dict[str, Any]]`
  - Compute average embedding of all required skills
  - Compute average embedding of all candidate skills
  - Compute cosine similarity between averages
  - Return same format as `_per_skill_similarity` (each required skill gets same aggregate score)
  - Rationale example: `"Aggregate embedding similarity: 0.85"`
- [x] 4.6 Implement `SimilarityEngine.compute_competence_similarity(required: list[str], candidate: list[str]) -> dict[str, dict[str, Any]]`
  - Route to `_per_skill_similarity()` or `_aggregate_similarity()` based on configured strategy
  - Maintain same output format as current `TaskAnalyzer.semantic_competence_similarity()`
  - Guarantee `rationale` is always present (even when best_match is None)
  - Edge cases:
    - `required == []` → return `{}`
    - `candidate == []` → return per-required entry with `best_match=None`, `score=0.0`, and rationale
- [x] 4.7 Implement `SimilarityEngine.compute_role_similarity(required_role: str, candidate_role: str) -> float`
  - Edge cases:
    - If either role is empty/None → return 0.0
    - If either role is "(unknown)" (case-insensitive, trimmed) → return 0.0
  - Otherwise compute cosine similarity of role embeddings
- [x] 4.8 Create `tests/test_similarity.py`
  - Test `cosine_similarity()` with known vectors
  - Mock embeddings and test both strategies
  - Test edge cases (empty lists, identical skills, no matches)
- [x] 4.9 Create `tests/test_similarity_integration.py` (marked `@pytest.mark.integration`)
  - Compare both strategies with real embeddings
  - Validate scores are reasonable (e.g., "Python" vs "Python 3" → high similarity)

### Phase 5: TaskAnalyzer Refactoring (3-4 hours)

- [x] 5.1 Update `src/teamlandkarte_mcp/matching/task_analyzer.py`
- [x] 5.2 Change `TaskAnalyzer.__init__` signature:
  - Remove: `__init__(self, mcp: FastMCP)`
  - Add: `__init__(self, azure_client: AzureOpenAIClient)`
- [x] 5.3 **Delete** `_sample_json()` method completely
- [x] 5.4 **Delete** `_heuristic_competences_from_text()` static method completely
- [x] 5.5 Refactor `extract_ranked_roles(description: str, limit: int = 5) -> list[RankedRole]`:
  - Remove all heuristic fallback code
  - Build prompt for structured JSON response:
    ```
    system: "Extract a ranked list of possible roles from a task description. Return STRICT JSON: {roles:[{rank:int, role:str, rationale:str}]}"
    user: "Limit: {limit}\n\nTask description:\n{description}"
    ```
  - Call `self._client.chat_completion(system, user, response_format={"type": "json_object"})`
  - Parse JSON response and build `list[RankedRole]`
  - Raise `AzureAPIError` on failure (no fallback)
- [x] 5.6 Refactor `_extract_requirements_from_description(description: str) -> ExtractedRequirements`:
  - Remove all heuristic fallback code
  - Build prompt for structured JSON:
    ```
    system: "Extract structured requirements from a task description. Return STRICT JSON with keys: competences (list[str]), date_start (YYYY-MM-DD|null), date_end (YYYY-MM-DD|null), roles ([{rank, role, rationale}])."
    user: "Task description:\n{description}"
    ```
  - Call `self._client.chat_completion(system, user, response_format={"type": "json_object"})`
  - Parse JSON and build `ExtractedRequirements`
  - Raise `AzureAPIError` on failure (no fallback)
- [x] 5.7 Refactor `apply_requirement_update(current: Requirements, change_description: str) -> Requirements`:
  - Remove all heuristic fallback code
  - Build prompt with current requirements + change request
  - Call `self._client.chat_completion(system, user, response_format={"type": "json_object"})`
  - Parse JSON and build updated `Requirements`
  - Raise `AzureAPIError` on failure (no fallback)
- [x] 5.8 **Delete** `semantic_competence_similarity()` method completely (replaced by `SimilarityEngine`)
- [x] 5.9 Update `ExtractedRequirements` dataclass docstring (no longer mentions heuristics)
- [x] 5.10 Update `TaskAnalyzer` class docstring (remove FastMCP sampling references, mention Azure OpenAI)
- [x] 5.11 Remove unused imports (`FastMCP`, `random`, `asyncio` if no longer needed)
- [x] 5.12 Update all `tests/test_task_analyzer.py` (or similar):
  - Mock `AzureOpenAIClient` instead of `FastMCP`
  - Remove heuristic fallback tests
  - Add tests for Azure API error propagation
- [x] 5.13 Update `AnalysisError` semantics (or rename/remove if no longer needed):
  - Previously: heuristic fallback masked missing sampling
  - Now: Azure failures must surface as exceptions (no heuristics)

### Phase 6: Matcher & Scorer Integration (2-3 hours)

- [x] 6.1 Identify where `Matcher` or scoring logic calls `TaskAnalyzer.semantic_competence_similarity()`
- [x] 6.2 Update `Matcher.__init__` to accept `SimilarityEngine` as parameter
- [x] 6.3 Replace `analyzer.semantic_competence_similarity(required, candidate)` with `similarity_engine.compute_competence_similarity(required, candidate)`
- [x] 6.4 Update role matching to use `similarity_engine.compute_role_similarity(required_role, candidate_role)`
- [x] 6.5 Ensure scorer aggregation logic works with new similarity output format (should be identical)
- [x] 6.6 Update `tests/test_scorer.py` (and/or add `tests/test_matcher.py` if introduced):
  - Mock `SimilarityEngine` instead of `TaskAnalyzer.semantic_competence_similarity`
  - Test that both similarity strategies produce valid scores
- [x] 6.7 Update matcher/scorer tests to reflect new edge-case contracts:
  - role similarity returns 0.0 for empty/"(unknown)" roles
  - competence similarity returns 0.0 when candidate list is empty

### Phase 7: Server Initialization (2 hours)

- [x] 7.1 Update `build_server()` in `src/teamlandkarte_mcp/mcp_server.py`
- [x] 7.2 Update server build to use top-level embedding cache config (not nested under azure):
  - `cfg.embedding_cache.enabled/db_path/ttl_days`
- [x] 7.3 Initialize `EmbeddingCache` (if enabled) using `cfg.embedding_cache`
- [x] 7.4 Initialize `AzureOpenAIClient` using `cfg.azure_openai`
- [x] 7.5 Initialize `SimilarityEngine` using `cfg.matching.similarity.strategy`
- [x] 7.6 Update `TaskAnalyzer` instantiation
- [x] 7.7 Update `Matcher` instantiation
- [x] 7.8 Add startup logging (stderr):
  - Log Azure OpenAI endpoint
  - Log embedding cache status (enabled/disabled, path)
  - Log similarity strategy
- [x] 7.9 Add graceful error handling if Azure credentials missing:
  - Log error to stderr
  - Exit with clear message: "Azure OpenAI credentials not found in environment"

### Phase 8: Cost Tracking & Monitoring (2 hours)

- [x] 8.1 Create `src/teamlandkarte_mcp/azure/cost_tracker.py`
- [x] 8.2 Implement `CostTracker` class:
  - Class-level constants for pricing
  - `log_embedding_request(text: str, cached: bool)`
  - `log_llm_request(input_tokens: int, output_tokens: int)`
  - `get_session_costs() -> dict` (return totals)
- [x] 8.3 Integrate `CostTracker` into `AzureOpenAIClient`:
  - Track embedding requests (cache hit/miss)
  - Track LLM requests (token counts from API response)
- [x] 8.4 Add periodic cost logging to stderr:
  - Every 10 API calls or every 5 minutes
  - Log: `"Azure OpenAI costs this session: embeddings=$X.XX, llm=$Y.YY, total=$Z.ZZ"`

### Phase 9: Documentation (2-3 hours)

- [x] 9.1 Update `README.md`:
  - Add "Azure OpenAI Setup" section with prerequisites
  - Document environment variables (`AZURE_OPENAI_EMBEDDING_API_KEY`, `AZURE_OPENAI_LLM_API_KEY`)
  - Document `config.toml` Azure sections
  - Add cost estimation table
  - Add embedding cache explanation
  - Add similarity strategy comparison
- [x] 9.2 Add migration guide to `README.md`:
  - Step 1: Rename `database.toml` → `config.toml`
  - Step 2: Add Azure sections to config
  - Step 3: Add Azure API keys to `.env`
  - Step 4: Run `uv sync`
  - Step 5: Start server (embedding cache will be created automatically)
- [x] 9.3 Update `docs/troubleshooting.md`:
  - Add section: "Azure OpenAI API Errors"
  - Document common errors (rate limits, invalid credentials, network issues)
  - Document `AzureAPIError` and how to debug
- [x] 9.4 Update `config.toml.example`:
  - Add comprehensive comments for all Azure sections
  - Document both similarity strategies with examples
  - Document embedding cache TTL trade-offs
- [x] 9.5 Update OpenSpec architecture docs:
  - `openspec/changes/add-capacity-matching-mcp-server/architecture.md`
  - `openspec/changes/add-capacity-matching-mcp-server/design.md`
  - Update diagrams to show Azure OpenAI dependency
  - Update component descriptions
- [x] 9.6 Create `docs/azure_openai_setup.md` (detailed setup guide):
  - Prerequisites (Azure subscription, OpenAI resource)
  - API key generation steps
  - Cost monitoring in Azure portal
  - Troubleshooting deployment/model availability

### Phase 10: Testing & Validation (3-4 hours)

- [x] 10.1 Run fast unit tests in CI mode: `pytest -m "not integration" -q`
- [x] 10.2 Run integration tests with real Azure API: `pytest -m integration`
- [x] 10.3 Test embedding cache behavior:
  - Start server, run search → check cache DB created
  - Restart server, run same search → verify cache hit
  - Wait past TTL, run cleanup → verify expired entries removed
- [x] 10.4 Test both similarity strategies:
  - Set `matching.similarity.strategy = "per_skill"` → run search
  - Set `matching.similarity.strategy = "aggregate"` → run search
  - Compare results and validate scoring differences
- [x] 10.5 Test cost tracking:
  - Run several searches
  - Check stderr for cost logs
  - Validate cost estimates are reasonable
- [x] 10.6 Test error handling:
  - Comment out Azure API keys → verify graceful startup error
  - Mock API failure → verify `AzureAPIError` propagates correctly
  - Mock rate limit error → verify retry logic works
- [x] 10.7 Performance benchmark:
  - Cold cache: measure time for first search (100 candidates)
  - Warm cache: measure time for repeated search
  - Document speedup ratio
- [x] 10.8 Manual validation in Cherry Studio:
  - Browse tasks
  - Extract requirements from complex task description
  - Run matching workflow
  - Validate similarity scores look reasonable
  - Test guided capture with Azure LLM
  - Test update_requirements

### Phase 11: Cleanup & Final Review (2-3 hours)

- [x] 11.1 Final sweep: ensure there are no remaining FastMCP sampling references
- [x] 11.2 Final sweep: ensure no heuristic fallback paths remain
- [x] 11.3 Tighten type hints, docstrings, and error semantics
- [x] 11.4 Run `ruff check` (via `uv`) and fix all issues
- [x] 11.5 Run `mypy` (via `uv`) and fix all issues
- [x] 11.6 Final code review: validate all modules, remove stray prints/debug logs
- [x] 11.7 Git commit review: ensure no secrets (config/.env/cache DB) are tracked
- [x] 11.7a Cleanup: remove obsolete `scripts/trino_smoke_check.py`
- [x] 11.7b Dependency hygiene: keep `trino` as a required runtime dependency
- [ ] 11.8 Merge to main and tag release

## Risk Mitigation

**Risk**: Azure API downtime → System unusable  
**Mitigation**: Document SLA expectations; consider adding a simple health check tool/endpoint and a clear user-facing error message (`AzureAPIError`).

**Risk**: Unexpected high costs  
**Mitigation**: Enable embedding cache (default); monitor costs weekly; set Azure budget alerts.

**Risk**: Embedding cache grows too large  
**Mitigation**: Run periodic cleanup; set reasonable TTL (default 30 days); ensure cache DB files stay ignored.

**Risk**: Secrets accidentally committed (`config.toml`, `.env`, caches)  
**Mitigation**: Keep `config.toml` and `.env` ignored; review `git status` before commit; run a secret scan locally (e.g. `git grep` for key patterns) before merging.

**Risk**: Migration breaks existing deployments  
**Mitigation**: Provide clear migration guide; test migration path manually; run integration tests against Azure in at least one real environment.

## Success Criteria

- [x] Unit tests pass (non-integration)
- [ ] Integration tests pass (`pytest -m integration`) in an environment with valid Azure credentials
- [ ] Server starts successfully with Azure credentials
- [ ] Matching results show improved semantic understanding vs. heuristics
- [ ] Embedding cache demonstrates significant performance improvement on repeated searches
- [ ] Cost tracking shows accurate estimates
- [ ] Documentation complete and clear
- [ ] Repo clean: no secrets or local config files tracked (verify before merge)

---

**Total estimated effort**: 28-35 hours  
**Priority**: Medium-High (significant quality improvement, but breaking change)  
**Approval status**: ✅ Approved