Migrate all repos into monorepo context folders
Bahn: aisupport, Analyse-O2C-C2S, awesome-bahn-mcp-servers, beam-mcp,
Confluence_Bot, db-planet-mcp-server, O2C-Harness, project-audit,
Projekt-KIQ-HP, teamlandkarte-mcp
Dhive: Jury-Voting
Privat: CV, NoteGraph (NOTE: NoteGraph needs complete redo after consolidation)
Shared: AI-Orchestrator, OrgMyLife, power_skills_and_more
Shared/references: symphony (read-only)
Bahn repos remain available as independent remotes - this monorepo
pulls them in via subtree, the originals are untouched.
This commit is contained in:
+8
@@ -0,0 +1,8 @@
|
||||
+# Changelog
|
||||
+
|
||||
+## replace-heuristics-with-azure-openai
|
||||
+
|
||||
+- Removed heuristic/sampling-based similarity and replaced role similarity with Azure embedding cosine similarity.
|
||||
+- Tightened docstrings and typing for `TaskAnalyzer` and `SimilarityEngine`.
|
||||
+- Removed stray debug output and ensured stderr-safe logging.
|
||||
+
|
||||
+752
@@ -0,0 +1,752 @@
|
||||
# Change Proposal: Replace Heuristics with Azure OpenAI Embeddings and LLM
|
||||
|
||||
- **Change ID**: `replace-heuristics-with-azure-openai`
|
||||
- **Status**: Implemented
|
||||
- **Target**: `teamlandkarte-mcp`
|
||||
- **Author**: Thomas Handke
|
||||
- **Date**: 2026-02-13
|
||||
|
||||
## Summary
|
||||
|
||||
Replace the current FastMCP sampling (unavailable) and deterministic heuristic fallbacks with direct Azure OpenAI API integration for:
|
||||
|
||||
1. **Competence and role similarity matching** via embeddings (`text-embedding-3-large`, 3072 dimensions)
|
||||
2. **Requirements extraction and validation** via LLM (`gpt-4.1`)
|
||||
|
||||
Additionally:
|
||||
- Rename `database.toml` → `config.toml` (and template accordingly)
|
||||
- Implement persistent embedding cache using a local SQLite database
|
||||
- Support two similarity scoring strategies (per-skill vs. aggregate) via configuration
|
||||
- Remove all FastMCP sampling code and heuristic fallbacks completely
|
||||
|
||||
## Motivation
|
||||
|
||||
### Current State Problems
|
||||
|
||||
1. **FastMCP sampling is unavailable**: The `mcp` package does not expose a `FastMCP.sampling` API, forcing the system to rely entirely on weak deterministic heuristics.
|
||||
|
||||
2. **Heuristic limitations**:
|
||||
- Role inference: simple keyword matching (`"cloud" → "Cloud Engineer"`)
|
||||
- Competence extraction: small hardcoded keyword list (~15 terms)
|
||||
- Competence similarity: exact normalized match (1.0) or substring (0.7), otherwise 0.0
|
||||
- No true semantic understanding
|
||||
|
||||
3. **Poor matching quality**:
|
||||
- Synonyms not recognized (e.g., "React.js" vs "ReactJS")
|
||||
- Related skills missed (e.g., "FastAPI" vs "REST API Development")
|
||||
- Role extraction frequently returns `"(unknown)"`
|
||||
|
||||
4. **Maintenance burden**: Heuristic keyword lists require manual updates
|
||||
|
||||
### Proposed Benefits
|
||||
|
||||
1. **Semantic matching**: True similarity via embeddings (e.g., "Kubernetes" ↔ "K8s", "Python" ↔ "Python 3")
|
||||
2. **Robust extraction**: LLM-backed role and competence extraction from free text
|
||||
3. **Better user experience**: More accurate Top/Good/Partial/Low categorization
|
||||
4. **Reduced maintenance**: No manual keyword list updates
|
||||
5. **Production-ready**: Direct API integration, no dependency on MCP sampling feature
|
||||
|
||||
## Goals
|
||||
|
||||
1. Replace `TaskAnalyzer.semantic_competence_similarity` with embedding-based similarity
|
||||
2. Replace `TaskAnalyzer.extract_ranked_roles` with Azure OpenAI LLM
|
||||
3. Replace `TaskAnalyzer._extract_requirements_from_description` with Azure OpenAI LLM
|
||||
4. Replace `TaskAnalyzer.apply_requirement_update` with Azure OpenAI LLM
|
||||
5. Implement persistent embedding cache (SQLite)
|
||||
6. Support two similarity strategies via config
|
||||
7. Remove all FastMCP sampling and heuristic code
|
||||
8. Rename `database.toml` → `config.toml`
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Changing the overall matching workflow (confirmation gate, guided capture, etc.)
|
||||
- Changing the scoring weights (competence 0.8, role 0.2)
|
||||
- Changing the categorical thresholds (Top/Good/Partial/Low)
|
||||
- Supporting multiple embedding models
|
||||
- Supporting other LLM providers (only Azure OpenAI)
|
||||
|
||||
## Proposed Changes
|
||||
|
||||
### 1. Configuration Changes
|
||||
|
||||
#### 1.1 Rename configuration file
|
||||
- `database.toml` → `config.toml`
|
||||
- `database.toml.example` → `config.toml.example`
|
||||
- Update all references in code, docs, README
|
||||
|
||||
#### 1.2 New configuration sections in `config.toml`
|
||||
|
||||
```toml
|
||||
[azure_openai]
|
||||
endpoint = "https://aiservice-ca00361106.cognitiveservices.azure.com/"
|
||||
api_version_embeddings = "2024-02-01"
|
||||
api_version_llm = "2024-12-01-preview"
|
||||
|
||||
[azure_openai.embeddings]
|
||||
model = "text-embedding-3-large"
|
||||
# Azure deployment name is equal to model name
|
||||
deployment = "text-embedding-3-large"
|
||||
|
||||
[azure_openai.llm]
|
||||
model = "gpt-4.1"
|
||||
# Azure deployment name is equal to model name
|
||||
deployment = "gpt-4.1"
|
||||
temperature = 0.2
|
||||
max_tokens = 2000
|
||||
|
||||
[embedding_cache]
|
||||
enabled = true
|
||||
db_path = "embeddings_cache.db"
|
||||
ttl_days = 30
|
||||
|
||||
[matching.similarity]
|
||||
# Strategy: "per_skill" or "aggregate"
|
||||
# - per_skill: Match each required skill to best candidate skill (current behavior)
|
||||
# - aggregate: Compare average embedding of all required vs all candidate skills
|
||||
strategy = "per_skill"
|
||||
# Chat model/deployment name (Azure OpenAI)
|
||||
chat_model = "gpt-4.1"
|
||||
```
|
||||
|
||||
#### 1.3 Environment variables (`.env`)
|
||||
```bash
|
||||
AZURE_OPENAI_EMBEDDING_API_KEY="..."
|
||||
AZURE_OPENAI_LLM_API_KEY="..."
|
||||
```
|
||||
|
||||
### 2. Embedding Cache Implementation
|
||||
|
||||
Create `src/teamlandkarte_mcp/cache/embedding_cache.py`:
|
||||
|
||||
```python
|
||||
class EmbeddingCache:
|
||||
"""Persistent embedding cache using SQLite.
|
||||
|
||||
Schema:
|
||||
embeddings(
|
||||
id INTEGER PRIMARY KEY,
|
||||
text TEXT UNIQUE NOT NULL,
|
||||
model TEXT NOT NULL,
|
||||
embedding BLOB NOT NULL, -- UTF-8 bytes of JSON-serialized list[float]
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||
)
|
||||
|
||||
Notes:
|
||||
- Store embeddings as UTF-8 encoded JSON in a BLOB column for portability.
|
||||
- Serialize with: json.dumps(embedding).encode("utf-8")
|
||||
- Deserialize with: json.loads(blob.decode("utf-8"))
|
||||
"""
|
||||
|
||||
def __init__(self, db_path: str, ttl_days: int = 30):
|
||||
"""Initialize cache, create DB if not exists."""
|
||||
|
||||
def get(self, text: str, model: str) -> Optional[list[float]]:
|
||||
"""Retrieve cached embedding, return None if missing/expired."""
|
||||
|
||||
def put(self, text: str, model: str, embedding: list[float]) -> None:
|
||||
"""Store embedding in cache."""
|
||||
|
||||
def cleanup_expired(self) -> int:
|
||||
"""Remove embeddings older than TTL, return count deleted."""
|
||||
```
|
||||
|
||||
### 3. Azure OpenAI Client Layer
|
||||
|
||||
Create `src/teamlandkarte_mcp/azure/openai_client.py`:
|
||||
|
||||
```python
|
||||
class AzureOpenAIClient:
|
||||
"""Wrapper for Azure OpenAI API calls using AsyncAzureOpenAI client."""
|
||||
|
||||
def __init__(self, config: AzureOpenAIConfig, embedding_cache: EmbeddingCache):
|
||||
"""Initialize AsyncAzureOpenAI clients for embeddings and LLM.
|
||||
|
||||
Uses openai.AsyncAzureOpenAI for all async API calls.
|
||||
"""
|
||||
|
||||
async def get_embedding(self, text: str) -> list[float]:
|
||||
"""Get embedding for text, using cache if available.
|
||||
|
||||
Raises:
|
||||
AzureAPIError: If API call fails (no fallback).
|
||||
"""
|
||||
|
||||
async def get_embeddings_batch(self, texts: list[str]) -> list[list[float]]:
|
||||
"""Get embeddings for multiple texts (not batched per user requirement)."""
|
||||
|
||||
async def chat_completion(
|
||||
self,
|
||||
system: str,
|
||||
user: str,
|
||||
response_format: dict = None
|
||||
) -> str:
|
||||
"""LLM completion with structured JSON response.
|
||||
|
||||
Raises:
|
||||
AzureAPIError: If API call fails (no fallback).
|
||||
"""
|
||||
```
|
||||
|
||||
### 4. Similarity Computation
|
||||
|
||||
Create `src/teamlandkarte_mcp/matching/similarity.py`:
|
||||
|
||||
```python
|
||||
def cosine_similarity(vec_a: list[float], vec_b: list[float]) -> float:
|
||||
"""Compute cosine similarity between two vectors."""
|
||||
|
||||
class SimilarityEngine:
|
||||
"""Compute semantic similarity using embeddings."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
client: AzureOpenAIClient,
|
||||
strategy: str = "per_skill"
|
||||
):
|
||||
"""Initialize with Azure client and strategy."""
|
||||
|
||||
async def compute_competence_similarity(
|
||||
self,
|
||||
required: list[str],
|
||||
candidate: list[str],
|
||||
) -> dict[str, dict[str, Any]]:
|
||||
"""Compute similarity using configured strategy.
|
||||
|
||||
Returns same format as current semantic_competence_similarity:
|
||||
{required_comp: {best_match: str|null, score: float, rationale: str}}
|
||||
|
||||
Behavioral guarantees:
|
||||
- `rationale` is always included (even when best_match is null).
|
||||
- If `required` is empty, return {}.
|
||||
- If `candidate` is empty, return an entry per required competence with:
|
||||
best_match=null, score=0.0, and a short rationale.
|
||||
"""
|
||||
|
||||
async def _per_skill_similarity(
|
||||
self, required: list[str], candidate: list[str]
|
||||
) -> dict[str, dict[str, Any]]:
|
||||
"""For each required skill, find best matching candidate skill."""
|
||||
|
||||
async def _aggregate_similarity(
|
||||
self, required: list[str], candidate: list[str]
|
||||
) -> dict[str, dict[str, Any]]:
|
||||
"""Compare average embeddings of required vs candidate skills."""
|
||||
|
||||
async def compute_role_similarity(
|
||||
self, required_role: str, candidate_role: str
|
||||
) -> float:
|
||||
"""Compute role similarity (0.0 to 1.0).
|
||||
|
||||
Behavioral guarantees:
|
||||
- If either role is empty/None, return 0.0.
|
||||
- If either role is "(unknown)" (case-insensitive, trimmed), return 0.0.
|
||||
- Otherwise compute cosine similarity of the role embeddings.
|
||||
|
||||
Note:
|
||||
- Role similarity returns only a float score. Rationales are provided by
|
||||
`compute_competence_similarity()` results.
|
||||
"""
|
||||
```
|
||||
|
||||
### 5. TaskAnalyzer Refactoring
|
||||
|
||||
**Remove**:
|
||||
- `_sample_json()` method
|
||||
- `_heuristic_competences_from_text()` static method
|
||||
- All heuristic fallback code in:
|
||||
- `extract_ranked_roles()`
|
||||
- `_extract_requirements_from_description()`
|
||||
- `apply_requirement_update()`
|
||||
- `semantic_competence_similarity()`
|
||||
|
||||
**Replace with**:
|
||||
|
||||
```python
|
||||
class TaskAnalyzer:
|
||||
"""Task text analyzer using Azure OpenAI."""
|
||||
|
||||
def __init__(self, azure_client: AzureOpenAIClient):
|
||||
self._client = azure_client
|
||||
|
||||
async def extract_ranked_roles(
|
||||
self, description: str, limit: int = 5
|
||||
) -> list[RankedRole]:
|
||||
"""Extract ranked roles using Azure OpenAI LLM.
|
||||
|
||||
Raises:
|
||||
AzureAPIError: If API call fails.
|
||||
"""
|
||||
|
||||
async def _extract_requirements_from_description(
|
||||
self, description: str
|
||||
) -> ExtractedRequirements:
|
||||
"""Extract requirements using Azure OpenAI LLM.
|
||||
|
||||
Raises:
|
||||
AzureAPIError: If API call fails.
|
||||
"""
|
||||
|
||||
async def apply_requirement_update(
|
||||
self, current: Requirements, change_description: str
|
||||
) -> Requirements:
|
||||
"""Update requirements using Azure OpenAI LLM.
|
||||
|
||||
Raises:
|
||||
AzureAPIError: If API call fails.
|
||||
"""
|
||||
```
|
||||
|
||||
**Additional behavior change**:
|
||||
- The current `TaskAnalyzer` raises `AnalysisError` only when MCP sampling is present but fails/returns invalid JSON, and otherwise silently falls back to heuristics.
|
||||
- After this change, heuristics are removed. Any Azure OpenAI failure (HTTP error, timeout after retries, invalid JSON/schema) will result in an analysis exception (e.g. `AzureAPIError` and/or a narrower `AnalysisError`) and should be surfaced to callers.
|
||||
|
||||
**Testing impact**:
|
||||
- Existing tests that currently assert heuristic fallback behavior in `TaskAnalyzer` must be updated.
|
||||
- Replace “heuristic fallback expected” assertions with either:
|
||||
- mocked Azure responses (unit tests), or
|
||||
- explicit error expectations when Azure is unavailable.
|
||||
|
||||
### 6. Matcher & Scorer Integration
|
||||
|
||||
Update `src/teamlandkarte_mcp/matching/matcher.py`:
|
||||
|
||||
**Key Changes**:
|
||||
1. Accept `SimilarityEngine` in constructor
|
||||
2. Replace `TaskAnalyzer.semantic_competence_similarity()` calls with `SimilarityEngine.compute_competence_similarity()`
|
||||
3. Replace the existing `_role_similarity()` function with `SimilarityEngine.compute_role_similarity()`
|
||||
4. Maintain current scoring weights (competence 0.8, role 0.2)
|
||||
|
||||
**Details**:
|
||||
- The `Matcher` currently uses `TaskAnalyzer.semantic_competence_similarity()` for competence matching (per-skill, best-match strategy)
|
||||
- It also has a standalone `_role_similarity()` function for role matching (currently uses simple string comparison)
|
||||
- Both will be replaced by `SimilarityEngine` methods, which provide semantic similarity via embeddings
|
||||
- The `SimilarityEngine.compute_competence_similarity()` returns the same dict structure as current `semantic_competence_similarity()`, ensuring drop-in compatibility
|
||||
- The `SimilarityEngine.compute_role_similarity()` returns a float (0.0 to 1.0), matching current `_role_similarity()` signature
|
||||
|
||||
```python
|
||||
class Matcher:
|
||||
"""Capacity matcher with semantic similarity."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
analyzer: TaskAnalyzer,
|
||||
similarity_engine: SimilarityEngine,
|
||||
config: MatchingConfig
|
||||
):
|
||||
"""Initialize matcher with analyzer, similarity engine, and config."""
|
||||
self._analyzer = analyzer
|
||||
self._similarity = similarity_engine
|
||||
self._config = config
|
||||
|
||||
async def _compute_match_score(
|
||||
self,
|
||||
required: Requirements,
|
||||
capacity: Capacity
|
||||
) -> MatchScore:
|
||||
"""Compute match score using SimilarityEngine.
|
||||
|
||||
Competence matching:
|
||||
comp_sim = await self._similarity.compute_competence_similarity(
|
||||
required.competences, capacity.competences
|
||||
)
|
||||
|
||||
Role matching:
|
||||
role_score = await self._similarity.compute_role_similarity(
|
||||
required.role, capacity.role
|
||||
)
|
||||
|
||||
Final score: (0.8 * avg_comp_score) + (0.2 * role_score)
|
||||
"""
|
||||
```
|
||||
|
||||
**Owning module note**:
|
||||
- The end-to-end match scoring aggregation currently lives in `src/teamlandkarte_mcp/matching/matcher.py` (with supporting score shaping asserted by `tests/test_scorer.py`).
|
||||
|
||||
### 7. Server Initialization Changes
|
||||
|
||||
Update `src/teamlandkarte_mcp/mcp_server.py`:
|
||||
|
||||
```python
|
||||
# Initialize Azure OpenAI components
|
||||
embedding_cache = EmbeddingCache(
|
||||
db_path=cfg.embedding_cache.db_path,
|
||||
ttl_days=cfg.embedding_cache.ttl_days,
|
||||
) if cfg.embedding_cache.enabled else None
|
||||
|
||||
azure_client = AzureOpenAIClient(cfg.azure_openai, embedding_cache)
|
||||
similarity_engine = SimilarityEngine(
|
||||
azure_client,
|
||||
strategy=cfg.matching.similarity.strategy,
|
||||
)
|
||||
|
||||
# Initialize analyzer with Azure client
|
||||
analyzer = TaskAnalyzer(azure_client)
|
||||
|
||||
# Initialize matcher with similarity engine
|
||||
matcher = Matcher(analyzer, similarity_engine, cfg.matching)
|
||||
```
|
||||
|
||||
### 8. Cost Estimation & Monitoring
|
||||
|
||||
Add cost tracking module `src/teamlandkarte_mcp/azure/cost_tracker.py`:
|
||||
|
||||
```python
|
||||
class CostTracker:
|
||||
"""Track and estimate Azure OpenAI API costs."""
|
||||
|
||||
# Pricing (as of 2026-02, may change)
|
||||
EMBEDDING_COST_PER_1K_TOKENS = 0.00013 # text-embedding-3-large
|
||||
LLM_INPUT_COST_PER_1K_TOKENS = 0.03 # gpt-4.1
|
||||
LLM_OUTPUT_COST_PER_1K_TOKENS = 0.06 # gpt-4.1
|
||||
|
||||
def log_embedding_request(self, text: str, cached: bool):
|
||||
"""Log embedding request for cost tracking."""
|
||||
|
||||
def log_llm_request(self, input_tokens: int, output_tokens: int):
|
||||
"""Log LLM request for cost tracking."""
|
||||
|
||||
def get_session_costs(self) -> dict:
|
||||
"""Return estimated costs for current session."""
|
||||
```
|
||||
|
||||
Add to `README.md`:
|
||||
|
||||
```markdown
|
||||
## Azure OpenAI Cost Estimation
|
||||
|
||||
Typical matching workflow costs (estimated):
|
||||
|
||||
| Operation | API Calls | Estimated Cost |
|
||||
|-----------|-----------|----------------|
|
||||
| Single capacity match (100 candidates) | ~200 embeddings (mostly cached), 0 LLM | $0.001 - $0.01 |
|
||||
| Task requirements extraction | 0 embeddings, 1 LLM call (~1000 tokens) | $0.03 - $0.06 |
|
||||
| Role inference | 0 embeddings, 1 LLM call (~500 tokens) | $0.015 - $0.03 |
|
||||
|
||||
With embedding caching enabled (default), repeated searches are significantly cheaper.
|
||||
|
||||
**Monthly cost estimate** (100 searches/day, 20 days):
|
||||
- Without cache: ~$60-120/month
|
||||
- With cache (90% hit rate): ~$10-20/month
|
||||
```
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests (Mocked Azure)
|
||||
|
||||
All unit tests should mock Azure OpenAI API responses to avoid API costs and ensure deterministic behavior:
|
||||
|
||||
**Embedding Mocks**:
|
||||
- Use fixed-dimension vectors matching `text-embedding-3-large` (3072 dimensions)
|
||||
- Create realistic patterns:
|
||||
- Similar terms: `[0.9, 0.8, 0.1, ..., 0.0]` and `[0.85, 0.82, 0.15, ..., 0.0]` → high cosine similarity (~0.95)
|
||||
- Different terms: `[0.9, 0.1, 0.0, ..., 0.0]` and `[0.1, 0.9, 0.0, ..., 0.0]` → low cosine similarity (~0.1)
|
||||
- Use `unittest.mock.AsyncMock` or `pytest-mock` for `AsyncAzureOpenAI` client
|
||||
|
||||
**LLM Mocks**:
|
||||
- Mock `chat.completions.create()` to return structured JSON responses
|
||||
- Example: `{"roles": [{"role": "Backend Developer", "confidence": 0.9, "rationale": "..."}], ...}`
|
||||
|
||||
**Example Mock Pattern**:
|
||||
```python
|
||||
@pytest.mark.asyncio
|
||||
async def test_compute_competence_similarity():
|
||||
mock_client = AsyncMock()
|
||||
mock_client.get_embedding.side_effect = [
|
||||
[0.9, 0.8, 0.1] + [0.0] * 3069, # "Python"
|
||||
[0.85, 0.82, 0.15] + [0.0] * 3069, # "Python 3"
|
||||
]
|
||||
engine = SimilarityEngine(mock_client, strategy="per_skill")
|
||||
result = await engine.compute_competence_similarity(["Python"], ["Python 3"])
|
||||
assert result["Python"]["score"] > 0.9
|
||||
```
|
||||
|
||||
### Integration Tests (Real Azure)
|
||||
|
||||
- Mark all integration tests with `@pytest.mark.integration`
|
||||
- Configure `pytest.ini`:
|
||||
```ini
|
||||
[pytest]
|
||||
markers =
|
||||
integration: marks tests that call real Azure OpenAI API (deselect with '-m "not integration"')
|
||||
```
|
||||
- Run integration tests manually before deployment: `pytest -m integration`
|
||||
- Run fast tests in CI: `pytest -m "not integration"`
|
||||
- Integration tests should use small, cheap prompts to minimize costs
|
||||
|
||||
### Test Coverage Goals
|
||||
|
||||
- **Unit tests**: ≥90% coverage for all new code (cache, client, similarity, analyzer)
|
||||
- **Integration tests**: Cover happy path + common error scenarios (API timeout, invalid credentials)
|
||||
- **Manual tests**: Full end-to-end workflow in Cherry Studio (search → filter → confirm)
|
||||
|
||||
## Implementation Task List
|
||||
|
||||
### Phase 1: Configuration & Infrastructure (3-4 hours)
|
||||
|
||||
- [ ] 1.1 Rename `database.toml` → `config.toml` and template
|
||||
- [ ] 1.2 Update all file references in code
|
||||
- [ ] 1.3 Add `[azure_openai]` section to config model (`src/config.py`)
|
||||
- [ ] 1.4 Add `[embedding_cache]` section to config model
|
||||
- [ ] 1.5 Add `[matching.similarity]` section to config model
|
||||
- [ ] 1.6 Update `config.toml.example` with new sections
|
||||
- [ ] 1.7 Load `AZURE_OPENAI_EMBEDDING_API_KEY` from `.env`
|
||||
- [ ] 1.8 Load `AZURE_OPENAI_LLM_API_KEY` from `.env`
|
||||
- [ ] 1.9 Add `openai` package to `pyproject.toml` dependencies
|
||||
|
||||
### Phase 2: Embedding Cache (2-3 hours)
|
||||
|
||||
- [ ] 2.1 Create `src/teamlandkarte_mcp/cache/embedding_cache.py`
|
||||
- [ ] 2.2 Implement `EmbeddingCache.__init__` (create DB/schema if not exists)
|
||||
- [ ] 2.3 Implement `EmbeddingCache.get()` (with TTL check)
|
||||
- [ ] 2.4 Implement `EmbeddingCache.put()`
|
||||
- [ ] 2.5 Implement `EmbeddingCache.cleanup_expired()`
|
||||
- [ ] 2.6 Add SQLite schema migration support (if DB exists but schema old)
|
||||
- [ ] 2.7 Add unit tests for `EmbeddingCache`
|
||||
- [ ] 2.8 Add `.gitignore` entry for `embeddings_cache.db`
|
||||
|
||||
### Phase 3: Azure OpenAI Client (3-4 hours)
|
||||
|
||||
**Testing Strategy**:
|
||||
- Unit tests should mock Azure API responses with realistic embedding vectors (3072 dimensions for text-embedding-3-large)
|
||||
- Use fixed seed embeddings for deterministic test behavior (e.g., `[0.1, 0.2, ..., 0.0]` patterns)
|
||||
- Integration tests marked with `@pytest.mark.integration` use real API calls
|
||||
|
||||
- [ ] 3.1 Create `src/teamlandkarte_mcp/azure/__init__.py`
|
||||
- [ ] 3.2 Create `src/teamlandkarte_mcp/azure/openai_client.py`
|
||||
- [ ] 3.3 Implement `AzureOpenAIClient.__init__`
|
||||
- [ ] 3.4 Implement `AzureOpenAIClient.get_embedding()` with cache integration
|
||||
- [ ] 3.5 Implement `AzureOpenAIClient.get_embeddings_batch()` (individual calls)
|
||||
- [ ] 3.6 Implement `AzureOpenAIClient.chat_completion()` with retry logic
|
||||
- [ ] 3.7 Add `AzureAPIError` exception class
|
||||
- [ ] 3.8 Add unit tests for client (mocked API)
|
||||
- [ ] 3.9 Add integration test with real API (marked as `@pytest.mark.integration`)
|
||||
|
||||
### Phase 4: Similarity Engine (4-5 hours)
|
||||
|
||||
- [ ] 4.1 Create `src/teamlandkarte_mcp/matching/similarity.py`
|
||||
- [ ] 4.2 Implement `cosine_similarity()` function
|
||||
- [ ] 4.3 Implement `SimilarityEngine.__init__`
|
||||
- [ ] 4.4 Implement `SimilarityEngine._per_skill_similarity()`
|
||||
- [ ] 4.5 Implement `SimilarityEngine._aggregate_similarity()`
|
||||
- [ ] 4.6 Implement `SimilarityEngine.compute_competence_similarity()` (router)
|
||||
- [ ] 4.7 Implement `SimilarityEngine.compute_role_similarity()`
|
||||
- [ ] 4.8 Add unit tests for similarity computation (mocked embeddings)
|
||||
- [ ] 4.9 Add integration test comparing both strategies
|
||||
|
||||
### Phase 5: TaskAnalyzer Refactoring (3-4 hours)
|
||||
|
||||
- [ ] 5.1 Remove `TaskAnalyzer.__init__(self, mcp: FastMCP)` signature
|
||||
- [ ] 5.2 Add new `TaskAnalyzer.__init__(self, azure_client: AzureOpenAIClient)`
|
||||
- [ ] 5.3 Remove `_sample_json()` method completely
|
||||
- [ ] 5.4 Remove `_heuristic_competences_from_text()` method completely
|
||||
- [ ] 5.5 Refactor `extract_ranked_roles()` to use `azure_client.chat_completion()`
|
||||
- [ ] 5.6 Remove heuristic fallback from `extract_ranked_roles()`
|
||||
- [ ] 5.7 Refactor `_extract_requirements_from_description()` to use LLM
|
||||
- [ ] 5.8 Remove heuristic fallback from `_extract_requirements_from_description()`
|
||||
- [ ] 5.9 Refactor `apply_requirement_update()` to use LLM
|
||||
- [ ] 5.10 Remove heuristic fallback from `apply_requirement_update()`
|
||||
- [ ] 5.11 Remove `semantic_competence_similarity()` method (replaced by `SimilarityEngine`)
|
||||
- [ ] 5.12 Update all `TaskAnalyzer` unit tests
|
||||
|
||||
### Phase 6: Matcher & Scorer Integration (2-3 hours)
|
||||
|
||||
- [ ] 6.1 Update `Matcher.__init__` to accept `SimilarityEngine`
|
||||
- [ ] 6.2 Replace competence similarity calls with `SimilarityEngine.compute_competence_similarity()`
|
||||
- [ ] 6.3 Replace role similarity calls with `SimilarityEngine.compute_role_similarity()`
|
||||
- [ ] 6.4 Update scorer aggregation logic if needed
|
||||
- [ ] 6.5 Update integration tests for matching pipeline
|
||||
|
||||
### Phase 7: Server Initialization (2 hours)
|
||||
|
||||
- [ ] 7.1 Update `build_server()` in `mcp_server.py` to load Azure config
|
||||
- [ ] 7.2 Initialize `EmbeddingCache` instance
|
||||
- [ ] 7.3 Initialize `AzureOpenAIClient` instance
|
||||
- [ ] 7.4 Initialize `SimilarityEngine` instance
|
||||
- [ ] 7.5 Update `TaskAnalyzer` instantiation
|
||||
- [ ] 7.6 Update `Matcher` instantiation
|
||||
- [ ] 7.7 Add startup logging for Azure OpenAI connection
|
||||
- [ ] 7.8 Add graceful error handling if Azure credentials missing
|
||||
|
||||
### Phase 8: Cost Tracking & Monitoring (2 hours)
|
||||
|
||||
- [ ] 8.1 Create `src/teamlandkarte_mcp/azure/cost_tracker.py`
|
||||
- [ ] 8.2 Implement `CostTracker` class with logging methods
|
||||
- [ ] 8.3 Integrate cost tracking into `AzureOpenAIClient`
|
||||
- [ ] 8.4 Add periodic cost report logging (stderr)
|
||||
- [ ] 8.5 Add cost summary to tool outputs (optional, via config)
|
||||
|
||||
### Phase 9: Documentation (2-3 hours)
|
||||
|
||||
- [ ] 9.1 Update `README.md` with Azure OpenAI setup instructions
|
||||
- [ ] 9.2 Add Azure cost estimation section to `README.md`
|
||||
- [ ] 9.3 Update `docs/troubleshooting.md` with Azure API error guidance
|
||||
- [ ] 9.4 Document similarity strategies (`per_skill` vs `aggregate`)
|
||||
- [ ] 9.5 Update `config.toml.example` with comprehensive comments
|
||||
- [ ] 9.6 Update OpenSpec architecture docs
|
||||
- [ ] 9.7 Add migration guide from old `database.toml` to new `config.toml`
|
||||
|
||||
### Phase 10: Testing & Validation (3-4 hours)
|
||||
|
||||
- [ ] 10.1 Configure `pytest.ini` with integration marker: `markers = integration: marks tests that call real Azure OpenAI API (deselect with '-m "not integration"')`
|
||||
- [ ] 10.2 Run full test suite after refactoring (`pytest -m "not integration"` for fast CI)
|
||||
- [ ] 10.3 Add new integration tests for Azure API paths (marked with `@pytest.mark.integration`)
|
||||
- [ ] 10.4 Test embedding cache persistence across server restarts
|
||||
- [ ] 10.5 Test both similarity strategies with real data
|
||||
- [ ] 10.6 Validate cost tracking accuracy
|
||||
- [ ] 10.7 Test error handling when Azure API unavailable
|
||||
- [ ] 10.8 Performance benchmark: compare cache hit/miss scenarios
|
||||
- [ ] 10.9 Manual validation in Cherry Studio
|
||||
|
||||
### Phase 11: Cleanup (1 hour)
|
||||
|
||||
- [ ] 11.1 Remove all commented-out FastMCP sampling code
|
||||
- [ ] 11.2 Remove unused imports (`from mcp.server.fastmcp import FastMCP` from `TaskAnalyzer`)
|
||||
- [ ] 11.3 Update type hints and docstrings
|
||||
- [ ] 11.4 Run linter and fix style issues
|
||||
- [ ] 11.5 Final code review
|
||||
|
||||
## Impact Analysis
|
||||
|
||||
### Changed Files
|
||||
|
||||
**New files**:
|
||||
- `config.toml` (renamed from `database.toml`)
|
||||
- `config.toml.example` (renamed from `database.toml.example`)
|
||||
- `src/teamlandkarte_mcp/cache/embedding_cache.py`
|
||||
- `src/teamlandkarte_mcp/azure/__init__.py`
|
||||
- `src/teamlandkarte_mcp/azure/openai_client.py`
|
||||
- `src/teamlandkarte_mcp/azure/cost_tracker.py`
|
||||
- `src/teamlandkarte_mcp/matching/similarity.py`
|
||||
- `embeddings_cache.db` (generated at runtime, gitignored)
|
||||
|
||||
**Modified files**:
|
||||
- `src/teamlandkarte_mcp/config.py` (new config sections)
|
||||
- `src/teamlandkarte_mcp/matching/task_analyzer.py` (complete refactor)
|
||||
- `src/teamlandkarte_mcp/matching/matcher.py` (similarity engine integration)
|
||||
- `src/teamlandkarte_mcp/mcp_server.py` (initialization changes)
|
||||
- `README.md` (setup instructions, cost docs)
|
||||
- `docs/troubleshooting.md` (Azure error guidance)
|
||||
- `pyproject.toml` (add `openai` dependency)
|
||||
- `.gitignore` (add `embeddings_cache.db`, `config.toml`)
|
||||
- All OpenSpec architecture/design docs
|
||||
|
||||
**Deleted code**:
|
||||
- All FastMCP sampling code in `TaskAnalyzer`
|
||||
- All heuristic fallback code in `TaskAnalyzer`
|
||||
- `_sample_json()`, `_heuristic_competences_from_text()` methods
|
||||
|
||||
### Breaking Changes
|
||||
|
||||
1. **Configuration file renamed**: Users must rename `database.toml` → `config.toml`
|
||||
2. **New required environment variables**: `AZURE_OPENAI_EMBEDDING_API_KEY`, `AZURE_OPENAI_LLM_API_KEY`
|
||||
3. **Hard dependency on Azure OpenAI**: No offline/fallback mode
|
||||
4. **New dependency**: `openai` Python package
|
||||
5. **TaskAnalyzer constructor signature changed**: `TaskAnalyzer.__init__(self, mcp: FastMCP)` → `TaskAnalyzer.__init__(self, azure_client: AzureOpenAIClient)`
|
||||
- This affects any code that instantiates `TaskAnalyzer` directly
|
||||
- Server initialization in `mcp_server.py` must be updated
|
||||
6. **Matcher constructor signature changed**: `Matcher.__init__(self, analyzer: TaskAnalyzer, config: MatchingConfig)` → `Matcher.__init__(self, analyzer: TaskAnalyzer, similarity_engine: SimilarityEngine, config: MatchingConfig)`
|
||||
- Adds new required `similarity_engine` parameter
|
||||
- Server initialization must pass `SimilarityEngine` instance
|
||||
|
||||
### Migration Path
|
||||
|
||||
1. Rename `database.toml` → `config.toml`
|
||||
2. Add new Azure OpenAI sections to `config.toml`
|
||||
3. Add Azure API keys to `.env`
|
||||
4. Install updated dependencies: `uv sync`
|
||||
5. First run will create `embeddings_cache.db` automatically
|
||||
|
||||
## Security & Constraints
|
||||
|
||||
### Security
|
||||
|
||||
- **API keys in `.env`**: Never commit `.env` or `config.toml` with credentials
|
||||
- **Embedding cache**: Contains only embeddings, not raw capacity data (safe to persist)
|
||||
- **Network**: All API calls over HTTPS to Azure OpenAI endpoint
|
||||
- **Error messages**: Do not log API keys in error messages
|
||||
|
||||
### Operational Constraints
|
||||
|
||||
- **Azure OpenAI dependency**: System will fail if Azure API unavailable (no fallback)
|
||||
- **API rate limits**: Azure OpenAI enforces rate limits (TPM/RPM); implement retry with backoff
|
||||
- **Cost**: Real monetary cost per API call (mitigated by caching)
|
||||
- **Latency**: First search for a capacity will be slower (embedding generation); subsequent searches fast (cached)
|
||||
|
||||
### Configuration Defaults
|
||||
|
||||
```toml
|
||||
[embedding_cache]
|
||||
enabled = true
|
||||
db_path = "embeddings_cache.db"
|
||||
ttl_days = 30
|
||||
|
||||
[matching.similarity]
|
||||
strategy = "per_skill" # Maintains current behavior
|
||||
```
|
||||
|
||||
## Cost Estimation Details
|
||||
|
||||
### API Pricing (as of 2026-02)
|
||||
|
||||
| Service | Model | Cost |
|
||||
|---------|-------|------|
|
||||
| Embeddings | text-embedding-3-large | $0.00013 per 1K tokens |
|
||||
| LLM (input) | gpt-4.1 | $0.03 per 1K tokens |
|
||||
| LLM (output) | gpt-4.1 | $0.06 per 1K tokens |
|
||||
|
||||
### Typical Workflow Costs
|
||||
|
||||
**Scenario 1: Task requirements extraction**
|
||||
- Input: ~500 tokens (task description)
|
||||
- Output: ~300 tokens (JSON with roles/competences/dates)
|
||||
- Cost: (500 × $0.03 / 1000) + (300 × $0.06 / 1000) = **$0.033**
|
||||
|
||||
**Scenario 2: Capacity matching (100 candidates, 5 required skills)**
|
||||
- First run (cold cache):
|
||||
- Required skills: 5 × ~10 tokens = 50 tokens → ~$0.0000065
|
||||
- Candidate skills: 100 candidates × 3 skills avg × 10 tokens = 3000 tokens → ~$0.00039
|
||||
- Total: **~$0.0004** (negligible)
|
||||
- Subsequent runs (warm cache): **$0** (all cached)
|
||||
|
||||
**Scenario 3: Role inference**
|
||||
- Input: ~400 tokens
|
||||
- Output: ~200 tokens
|
||||
- Cost: **~$0.024**
|
||||
|
||||
### Monthly Estimates
|
||||
|
||||
**Light usage** (20 searches/month, 10 extractions):
|
||||
- Embeddings: ~$0.05
|
||||
- LLM: ~$0.50
|
||||
- **Total: ~$0.55/month**
|
||||
|
||||
**Moderate usage** (100 searches/month, 50 extractions):
|
||||
- Embeddings: ~$0.20 (with 90% cache hit rate)
|
||||
- LLM: ~$2.50
|
||||
- **Total: ~$2.70/month**
|
||||
|
||||
**Heavy usage** (500 searches/month, 200 extractions):
|
||||
- Embeddings: ~$1.00 (with 90% cache hit rate)
|
||||
- LLM: ~$10.00
|
||||
- **Total: ~$11/month**
|
||||
|
||||
## Open Questions
|
||||
|
||||
None (all clarifications provided by user).
|
||||
|
||||
## Approval & Timeline
|
||||
|
||||
- **Estimated effort**: 28-35 hours (1 week full-time or 2 weeks part-time)
|
||||
- **Risk level**: Medium (API dependency, cost implications, major refactor)
|
||||
- **Approval required**: Yes (architecture change, new external dependency)
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. Review and approve this proposal
|
||||
2. Create detailed implementation branch
|
||||
3. Implement phases 1-11 sequentially
|
||||
4. Conduct thorough testing (unit + integration + manual)
|
||||
5. Document migration guide
|
||||
6. Deploy to staging environment
|
||||
7. Monitor costs and performance
|
||||
8. Deploy to production
|
||||
+359
@@ -0,0 +1,359 @@
|
||||
# Tasks: Replace Heuristics with Azure OpenAI
|
||||
|
||||
> Change: `replace-heuristics-with-azure-openai`
|
||||
>
|
||||
> Status: **Approved / Implemented**
|
||||
|
||||
## Overview
|
||||
|
||||
Replace legacy client-side sampling/heuristic logic with direct Azure OpenAI integration:
|
||||
- Embeddings (`text-embedding-3-large`) for competence/role similarity
|
||||
- LLM (`gpt-4.1`) for requirements extraction and validation
|
||||
- Persistent SQLite embedding cache
|
||||
- Two configurable similarity strategies
|
||||
|
||||
# Removed: migration note about database.toml rename (now a completed internal detail).
|
||||
|
||||
## Estimated Effort
|
||||
|
||||
**Total: 28-35 hours** (1 week full-time or 2 weeks part-time)
|
||||
|
||||
## Task Breakdown
|
||||
|
||||
### Phase 1: Configuration & Infrastructure (3-4 hours)
|
||||
|
||||
- [x] 1.1 Rename `database.toml` → `config.toml`
|
||||
- [x] 1.2 Rename `database.toml.example` → `config.toml.example`
|
||||
- [x] 1.3 Update all file references in code (`README.md`, `docs/`, `src/config.py`, etc.)
|
||||
- [x] 1.4 Extend `src/config.py`: Add `AzureOpenAIConfig` model
|
||||
- `endpoint`, `api_version_embeddings`, `api_version_llm`
|
||||
- `embeddings.model`, `embeddings.deployment` (deployment must equal model name)
|
||||
- `llm.model`, `llm.deployment` (deployment must equal model name), `llm.temperature`, `llm.max_tokens`
|
||||
- [x] 1.5 Extend `src/config.py`: Add `EmbeddingCacheConfig` model
|
||||
- `enabled`, `db_path`, `ttl_days`
|
||||
- [x] 1.6 Extend `src/config.py`: Add `matching.similarity.strategy` field (`"per_skill"` or `"aggregate"`)
|
||||
- [x] 1.7 Update `config.toml.example` with new `[azure_openai]`, `[embedding_cache]`, `[matching.similarity]` sections
|
||||
- [x] 1.8 Load `AZURE_OPENAI_EMBEDDING_API_KEY` from environment in `load_config()`
|
||||
- [x] 1.9 Load `AZURE_OPENAI_LLM_API_KEY` from environment in `load_config()`
|
||||
- [x] 1.10 Add `openai` package to `pyproject.toml` dependencies
|
||||
- [x] 1.11 Run `uv sync` to install dependencies
|
||||
- [x] 1.12 Add `config.toml` to `.gitignore` (if not already there)
|
||||
- [x] 1.13 Configure pytest integration marker in `pytest.ini` (if not already present):
|
||||
- `integration: marks tests that call real Azure OpenAI API (deselect with '-m "not integration"')`
|
||||
|
||||
### Phase 2: Embedding Cache (2-3 hours)
|
||||
|
||||
- [x] 2.1 Create `src/teamlandkarte_mcp/cache/embedding_cache.py`
|
||||
- [x] 2.2 Implement `EmbeddingCache` class:
|
||||
- `__init__(db_path: str, ttl_days: int)`
|
||||
- Create SQLite database and schema if not exists:
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS embeddings (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
text TEXT UNIQUE NOT NULL,
|
||||
model TEXT NOT NULL,
|
||||
embedding BLOB NOT NULL,
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||
)
|
||||
CREATE INDEX IF NOT EXISTS idx_text_model ON embeddings(text, model)
|
||||
```
|
||||
- [x] 2.3 Implement `EmbeddingCache.get(text: str, model: str) -> Optional[list[float]]`
|
||||
- Query by text + model
|
||||
- Check if `created_at` is within TTL
|
||||
- Deserialize BLOB to `list[float]`
|
||||
- Return `None` if missing or expired
|
||||
- [x] 2.4 Implement `EmbeddingCache.put(text: str, model: str, embedding: list[float]) -> None`
|
||||
- Serialize embedding as UTF-8 bytes of JSON: `json.dumps(embedding).encode("utf-8")`
|
||||
- INSERT OR REPLACE into database
|
||||
- [x] 2.5 Implement `EmbeddingCache.cleanup_expired() -> int`
|
||||
- DELETE embeddings older than TTL
|
||||
- Return count of deleted rows
|
||||
- [x] 2.6 Add graceful handling for concurrent access (SQLite locks)
|
||||
- [x] 2.7 Create `tests/test_embedding_cache.py`
|
||||
- Test cache hit/miss
|
||||
- Test TTL expiration
|
||||
- Test cleanup
|
||||
- Test concurrent access
|
||||
- [x] 2.8 Add `embeddings_cache.db` to `.gitignore`
|
||||
|
||||
### Phase 3: Azure OpenAI Client (3-4 hours)
|
||||
|
||||
- [x] 3.1 Create `src/teamlandkarte_mcp/azure/__init__.py`
|
||||
- [x] 3.2 Create `src/teamlandkarte_mcp/azure/openai_client.py`
|
||||
- [x] 3.3 Define `AzureAPIError(RuntimeError)` exception class
|
||||
- [x] 3.4 Implement `AzureOpenAIClient` class:
|
||||
- `__init__(config: AzureOpenAIConfig, embedding_cache: Optional[EmbeddingCache])`
|
||||
- Initialize two `AzureOpenAI` clients (embeddings + LLM) using `AzureKeyCredential`
|
||||
- [x] 3.5 Implement `AzureOpenAIClient.get_embedding(text: str) -> list[float]`
|
||||
- Check cache first (if enabled)
|
||||
- If cache miss: call Azure API
|
||||
- Store in cache (if enabled)
|
||||
- Raise `AzureAPIError` on failure (no fallback)
|
||||
- [x] 3.6 Implement `AzureOpenAIClient.get_embeddings_batch(texts: list[str]) -> list[list[float]]`
|
||||
- Call `get_embedding()` for each text individually (per user requirement #3)
|
||||
- Return list of embeddings in same order
|
||||
- [x] 3.7 Implement `AzureOpenAIClient.chat_completion(system: str, user: str, response_format: Optional[dict] = None) -> str`
|
||||
- Call Azure OpenAI chat completions API
|
||||
- Use `gpt-4.1` model with configured temperature/max_tokens
|
||||
- Support `response_format={"type": "json_object"}` for structured JSON
|
||||
- Implement retry logic (3 attempts, exponential backoff)
|
||||
- Raise `AzureAPIError` on final failure (no fallback)
|
||||
- [x] 3.8 Add logging for API calls (cache hit/miss, request/response sizes)
|
||||
- [x] 3.9 Create `tests/test_azure_client.py`
|
||||
- Mock Azure API with `unittest.mock`
|
||||
- Test embedding retrieval with cache hit/miss
|
||||
- Test LLM completion
|
||||
- Test error handling and retries
|
||||
- [x] 3.10 Create `tests/test_azure_client_integration.py` (marked `@pytest.mark.integration`)
|
||||
- Real API calls (requires valid credentials)
|
||||
- Test embedding generation
|
||||
- Test LLM structured JSON response
|
||||
|
||||
### Phase 4: Similarity Engine (4-5 hours)
|
||||
|
||||
- [x] 4.1 Create `src/teamlandkarte_mcp/matching/similarity.py`
|
||||
- [x] 4.2 Implement `cosine_similarity(vec_a: list[float], vec_b: list[float]) -> float`
|
||||
- Compute dot product and magnitudes
|
||||
- Return cosine similarity in [0.0, 1.0]
|
||||
- Handle zero vectors gracefully
|
||||
- [x] 4.3 Implement `SimilarityEngine` class:
|
||||
- `__init__(client: AzureOpenAIClient, strategy: str = "per_skill")`
|
||||
- Validate strategy is `"per_skill"` or `"aggregate"`
|
||||
- [x] 4.4 Implement `SimilarityEngine._per_skill_similarity(required: list[str], candidate: list[str]) -> dict[str, dict[str, Any]]`
|
||||
- For each required skill:
|
||||
- Get embedding
|
||||
- Compare with all candidate skill embeddings
|
||||
- Find best match (highest cosine similarity)
|
||||
- Return `{required: {best_match: str|None, score: float, rationale: str}}`
|
||||
- Rationale example: `"Cosine similarity: 0.92 (best match: 'React.js')"`
|
||||
- [x] 4.5 Implement `SimilarityEngine._aggregate_similarity(required: list[str], candidate: list[str]) -> dict[str, dict[str, Any]]`
|
||||
- Compute average embedding of all required skills
|
||||
- Compute average embedding of all candidate skills
|
||||
- Compute cosine similarity between averages
|
||||
- Return same format as `_per_skill_similarity` (each required skill gets same aggregate score)
|
||||
- Rationale example: `"Aggregate embedding similarity: 0.85"`
|
||||
- [x] 4.6 Implement `SimilarityEngine.compute_competence_similarity(required: list[str], candidate: list[str]) -> dict[str, dict[str, Any]]`
|
||||
- Route to `_per_skill_similarity()` or `_aggregate_similarity()` based on configured strategy
|
||||
- Maintain same output format as current `TaskAnalyzer.semantic_competence_similarity()`
|
||||
- Guarantee `rationale` is always present (even when best_match is None)
|
||||
- Edge cases:
|
||||
- `required == []` → return `{}`
|
||||
- `candidate == []` → return per-required entry with `best_match=None`, `score=0.0`, and rationale
|
||||
- [x] 4.7 Implement `SimilarityEngine.compute_role_similarity(required_role: str, candidate_role: str) -> float`
|
||||
- Edge cases:
|
||||
- If either role is empty/None → return 0.0
|
||||
- If either role is "(unknown)" (case-insensitive, trimmed) → return 0.0
|
||||
- Otherwise compute cosine similarity of role embeddings
|
||||
- [x] 4.8 Create `tests/test_similarity.py`
|
||||
- Test `cosine_similarity()` with known vectors
|
||||
- Mock embeddings and test both strategies
|
||||
- Test edge cases (empty lists, identical skills, no matches)
|
||||
- [x] 4.9 Create `tests/test_similarity_integration.py` (marked `@pytest.mark.integration`)
|
||||
- Compare both strategies with real embeddings
|
||||
- Validate scores are reasonable (e.g., "Python" vs "Python 3" → high similarity)
|
||||
|
||||
### Phase 5: TaskAnalyzer Refactoring (3-4 hours)
|
||||
|
||||
- [x] 5.1 Update `src/teamlandkarte_mcp/matching/task_analyzer.py`
|
||||
- [x] 5.2 Change `TaskAnalyzer.__init__` signature:
|
||||
- Remove: `__init__(self, mcp: FastMCP)`
|
||||
- Add: `__init__(self, azure_client: AzureOpenAIClient)`
|
||||
- [x] 5.3 **Delete** `_sample_json()` method completely
|
||||
- [x] 5.4 **Delete** `_heuristic_competences_from_text()` static method completely
|
||||
- [x] 5.5 Refactor `extract_ranked_roles(description: str, limit: int = 5) -> list[RankedRole]`:
|
||||
- Remove all heuristic fallback code
|
||||
- Build prompt for structured JSON response:
|
||||
```
|
||||
system: "Extract a ranked list of possible roles from a task description. Return STRICT JSON: {roles:[{rank:int, role:str, rationale:str}]}"
|
||||
user: "Limit: {limit}\n\nTask description:\n{description}"
|
||||
```
|
||||
- Call `self._client.chat_completion(system, user, response_format={"type": "json_object"})`
|
||||
- Parse JSON response and build `list[RankedRole]`
|
||||
- Raise `AzureAPIError` on failure (no fallback)
|
||||
- [x] 5.6 Refactor `_extract_requirements_from_description(description: str) -> ExtractedRequirements`:
|
||||
- Remove all heuristic fallback code
|
||||
- Build prompt for structured JSON:
|
||||
```
|
||||
system: "Extract structured requirements from a task description. Return STRICT JSON with keys: competences (list[str]), date_start (YYYY-MM-DD|null), date_end (YYYY-MM-DD|null), roles ([{rank, role, rationale}])."
|
||||
user: "Task description:\n{description}"
|
||||
```
|
||||
- Call `self._client.chat_completion(system, user, response_format={"type": "json_object"})`
|
||||
- Parse JSON and build `ExtractedRequirements`
|
||||
- Raise `AzureAPIError` on failure (no fallback)
|
||||
- [x] 5.7 Refactor `apply_requirement_update(current: Requirements, change_description: str) -> Requirements`:
|
||||
- Remove all heuristic fallback code
|
||||
- Build prompt with current requirements + change request
|
||||
- Call `self._client.chat_completion(system, user, response_format={"type": "json_object"})`
|
||||
- Parse JSON and build updated `Requirements`
|
||||
- Raise `AzureAPIError` on failure (no fallback)
|
||||
- [x] 5.8 **Delete** `semantic_competence_similarity()` method completely (replaced by `SimilarityEngine`)
|
||||
- [x] 5.9 Update `ExtractedRequirements` dataclass docstring (no longer mentions heuristics)
|
||||
- [x] 5.10 Update `TaskAnalyzer` class docstring (remove FastMCP sampling references, mention Azure OpenAI)
|
||||
- [x] 5.11 Remove unused imports (`FastMCP`, `random`, `asyncio` if no longer needed)
|
||||
- [x] 5.12 Update all `tests/test_task_analyzer.py` (or similar):
|
||||
- Mock `AzureOpenAIClient` instead of `FastMCP`
|
||||
- Remove heuristic fallback tests
|
||||
- Add tests for Azure API error propagation
|
||||
- [x] 5.13 Update `AnalysisError` semantics (or rename/remove if no longer needed):
|
||||
- Previously: heuristic fallback masked missing sampling
|
||||
- Now: Azure failures must surface as exceptions (no heuristics)
|
||||
|
||||
### Phase 6: Matcher & Scorer Integration (2-3 hours)
|
||||
|
||||
- [x] 6.1 Identify where `Matcher` or scoring logic calls `TaskAnalyzer.semantic_competence_similarity()`
|
||||
- [x] 6.2 Update `Matcher.__init__` to accept `SimilarityEngine` as parameter
|
||||
- [x] 6.3 Replace `analyzer.semantic_competence_similarity(required, candidate)` with `similarity_engine.compute_competence_similarity(required, candidate)`
|
||||
- [x] 6.4 Update role matching to use `similarity_engine.compute_role_similarity(required_role, candidate_role)`
|
||||
- [x] 6.5 Ensure scorer aggregation logic works with new similarity output format (should be identical)
|
||||
- [x] 6.6 Update `tests/test_scorer.py` (and/or add `tests/test_matcher.py` if introduced):
|
||||
- Mock `SimilarityEngine` instead of `TaskAnalyzer.semantic_competence_similarity`
|
||||
- Test that both similarity strategies produce valid scores
|
||||
- [x] 6.7 Update matcher/scorer tests to reflect new edge-case contracts:
|
||||
- role similarity returns 0.0 for empty/"(unknown)" roles
|
||||
- competence similarity returns 0.0 when candidate list is empty
|
||||
|
||||
### Phase 7: Server Initialization (2 hours)
|
||||
|
||||
- [x] 7.1 Update `build_server()` in `src/teamlandkarte_mcp/mcp_server.py`
|
||||
- [x] 7.2 Update server build to use top-level embedding cache config (not nested under azure):
|
||||
- `cfg.embedding_cache.enabled/db_path/ttl_days`
|
||||
- [x] 7.3 Initialize `EmbeddingCache` (if enabled) using `cfg.embedding_cache`
|
||||
- [x] 7.4 Initialize `AzureOpenAIClient` using `cfg.azure_openai`
|
||||
- [x] 7.5 Initialize `SimilarityEngine` using `cfg.matching.similarity.strategy`
|
||||
- [x] 7.6 Update `TaskAnalyzer` instantiation
|
||||
- [x] 7.7 Update `Matcher` instantiation
|
||||
- [x] 7.8 Add startup logging (stderr):
|
||||
- Log Azure OpenAI endpoint
|
||||
- Log embedding cache status (enabled/disabled, path)
|
||||
- Log similarity strategy
|
||||
- [x] 7.9 Add graceful error handling if Azure credentials missing:
|
||||
- Log error to stderr
|
||||
- Exit with clear message: "Azure OpenAI credentials not found in environment"
|
||||
|
||||
### Phase 8: Cost Tracking & Monitoring (2 hours)
|
||||
|
||||
- [x] 8.1 Create `src/teamlandkarte_mcp/azure/cost_tracker.py`
|
||||
- [x] 8.2 Implement `CostTracker` class:
|
||||
- Class-level constants for pricing
|
||||
- `log_embedding_request(text: str, cached: bool)`
|
||||
- `log_llm_request(input_tokens: int, output_tokens: int)`
|
||||
- `get_session_costs() -> dict` (return totals)
|
||||
- [x] 8.3 Integrate `CostTracker` into `AzureOpenAIClient`:
|
||||
- Track embedding requests (cache hit/miss)
|
||||
- Track LLM requests (token counts from API response)
|
||||
- [x] 8.4 Add periodic cost logging to stderr:
|
||||
- Every 10 API calls or every 5 minutes
|
||||
- Log: `"Azure OpenAI costs this session: embeddings=$X.XX, llm=$Y.YY, total=$Z.ZZ"`
|
||||
|
||||
### Phase 9: Documentation (2-3 hours)
|
||||
|
||||
- [x] 9.1 Update `README.md`:
|
||||
- Add "Azure OpenAI Setup" section with prerequisites
|
||||
- Document environment variables (`AZURE_OPENAI_EMBEDDING_API_KEY`, `AZURE_OPENAI_LLM_API_KEY`)
|
||||
- Document `config.toml` Azure sections
|
||||
- Add cost estimation table
|
||||
- Add embedding cache explanation
|
||||
- Add similarity strategy comparison
|
||||
- [x] 9.2 Add migration guide to `README.md`:
|
||||
- Step 1: Rename `database.toml` → `config.toml`
|
||||
- Step 2: Add Azure sections to config
|
||||
- Step 3: Add Azure API keys to `.env`
|
||||
- Step 4: Run `uv sync`
|
||||
- Step 5: Start server (embedding cache will be created automatically)
|
||||
- [x] 9.3 Update `docs/troubleshooting.md`:
|
||||
- Add section: "Azure OpenAI API Errors"
|
||||
- Document common errors (rate limits, invalid credentials, network issues)
|
||||
- Document `AzureAPIError` and how to debug
|
||||
- [x] 9.4 Update `config.toml.example`:
|
||||
- Add comprehensive comments for all Azure sections
|
||||
- Document both similarity strategies with examples
|
||||
- Document embedding cache TTL trade-offs
|
||||
- [x] 9.5 Update OpenSpec architecture docs:
|
||||
- `openspec/changes/add-capacity-matching-mcp-server/architecture.md`
|
||||
- `openspec/changes/add-capacity-matching-mcp-server/design.md`
|
||||
- Update diagrams to show Azure OpenAI dependency
|
||||
- Update component descriptions
|
||||
- [x] 9.6 Create `docs/azure_openai_setup.md` (detailed setup guide):
|
||||
- Prerequisites (Azure subscription, OpenAI resource)
|
||||
- API key generation steps
|
||||
- Cost monitoring in Azure portal
|
||||
- Troubleshooting deployment/model availability
|
||||
|
||||
### Phase 10: Testing & Validation (3-4 hours)
|
||||
|
||||
- [x] 10.1 Run fast unit tests in CI mode: `pytest -m "not integration" -q`
|
||||
- [x] 10.2 Run integration tests with real Azure API: `pytest -m integration`
|
||||
- [x] 10.3 Test embedding cache behavior:
|
||||
- Start server, run search → check cache DB created
|
||||
- Restart server, run same search → verify cache hit
|
||||
- Wait past TTL, run cleanup → verify expired entries removed
|
||||
- [x] 10.4 Test both similarity strategies:
|
||||
- Set `matching.similarity.strategy = "per_skill"` → run search
|
||||
- Set `matching.similarity.strategy = "aggregate"` → run search
|
||||
- Compare results and validate scoring differences
|
||||
- [x] 10.5 Test cost tracking:
|
||||
- Run several searches
|
||||
- Check stderr for cost logs
|
||||
- Validate cost estimates are reasonable
|
||||
- [x] 10.6 Test error handling:
|
||||
- Comment out Azure API keys → verify graceful startup error
|
||||
- Mock API failure → verify `AzureAPIError` propagates correctly
|
||||
- Mock rate limit error → verify retry logic works
|
||||
- [x] 10.7 Performance benchmark:
|
||||
- Cold cache: measure time for first search (100 candidates)
|
||||
- Warm cache: measure time for repeated search
|
||||
- Document speedup ratio
|
||||
- [x] 10.8 Manual validation in Cherry Studio:
|
||||
- Browse tasks
|
||||
- Extract requirements from complex task description
|
||||
- Run matching workflow
|
||||
- Validate similarity scores look reasonable
|
||||
- Test guided capture with Azure LLM
|
||||
- Test update_requirements
|
||||
|
||||
### Phase 11: Cleanup & Final Review (2-3 hours)
|
||||
|
||||
- [x] 11.1 Final sweep: ensure there are no remaining FastMCP sampling references
|
||||
- [x] 11.2 Final sweep: ensure no heuristic fallback paths remain
|
||||
- [x] 11.3 Tighten type hints, docstrings, and error semantics
|
||||
- [x] 11.4 Run `ruff check` (via `uv`) and fix all issues
|
||||
- [x] 11.5 Run `mypy` (via `uv`) and fix all issues
|
||||
- [x] 11.6 Final code review: validate all modules, remove stray prints/debug logs
|
||||
- [x] 11.7 Git commit review: ensure no secrets (config/.env/cache DB) are tracked
|
||||
- [x] 11.7a Cleanup: remove obsolete `scripts/trino_smoke_check.py`
|
||||
- [x] 11.7b Dependency hygiene: keep `trino` as a required runtime dependency
|
||||
- [ ] 11.8 Merge to main and tag release
|
||||
|
||||
## Risk Mitigation
|
||||
|
||||
**Risk**: Azure API downtime → System unusable
|
||||
**Mitigation**: Document SLA expectations; consider adding a simple health check tool/endpoint and a clear user-facing error message (`AzureAPIError`).
|
||||
|
||||
**Risk**: Unexpected high costs
|
||||
**Mitigation**: Enable embedding cache (default); monitor costs weekly; set Azure budget alerts.
|
||||
|
||||
**Risk**: Embedding cache grows too large
|
||||
**Mitigation**: Run periodic cleanup; set reasonable TTL (default 30 days); ensure cache DB files stay ignored.
|
||||
|
||||
**Risk**: Secrets accidentally committed (`config.toml`, `.env`, caches)
|
||||
**Mitigation**: Keep `config.toml` and `.env` ignored; review `git status` before commit; run a secret scan locally (e.g. `git grep` for key patterns) before merging.
|
||||
|
||||
**Risk**: Migration breaks existing deployments
|
||||
**Mitigation**: Provide clear migration guide; test migration path manually; run integration tests against Azure in at least one real environment.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [x] Unit tests pass (non-integration)
|
||||
- [ ] Integration tests pass (`pytest -m integration`) in an environment with valid Azure credentials
|
||||
- [ ] Server starts successfully with Azure credentials
|
||||
- [ ] Matching results show improved semantic understanding vs. heuristics
|
||||
- [ ] Embedding cache demonstrates significant performance improvement on repeated searches
|
||||
- [ ] Cost tracking shows accurate estimates
|
||||
- [ ] Documentation complete and clear
|
||||
- [ ] Repo clean: no secrets or local config files tracked (verify before merge)
|
||||
|
||||
---
|
||||
|
||||
**Total estimated effort**: 28-35 hours
|
||||
**Priority**: Medium-High (significant quality improvement, but breaking change)
|
||||
**Approval status**: ✅ Approved
|
||||
Reference in New Issue
Block a user