Migrate all repos into monorepo context folders
Bahn: aisupport, Analyse-O2C-C2S, awesome-bahn-mcp-servers, beam-mcp,
Confluence_Bot, db-planet-mcp-server, O2C-Harness, project-audit,
Projekt-KIQ-HP, teamlandkarte-mcp
Dhive: Jury-Voting
Privat: CV, NoteGraph (NOTE: NoteGraph needs complete redo after consolidation)
Shared: AI-Orchestrator, OrgMyLife, power_skills_and_more
Shared/references: symphony (read-only)
Bahn repos remain available as independent remotes - this monorepo
pulls them in via subtree, the originals are untouched.
This commit is contained in:
@@ -0,0 +1,456 @@
|
||||
# OpenSpec Instructions
|
||||
|
||||
Instructions for AI coding assistants using OpenSpec for spec-driven development.
|
||||
|
||||
## TL;DR Quick Checklist
|
||||
|
||||
- Search existing work: `openspec spec list --long`, `openspec list` (use `rg` only for full-text search)
|
||||
- Decide scope: new capability vs modify existing capability
|
||||
- Pick a unique `change-id`: kebab-case, verb-led (`add-`, `update-`, `remove-`, `refactor-`)
|
||||
- Scaffold: `proposal.md`, `tasks.md`, `design.md` (only if needed), and delta specs per affected capability
|
||||
- Write deltas: use `## ADDED|MODIFIED|REMOVED|RENAMED Requirements`; include at least one `#### Scenario:` per requirement
|
||||
- Validate: `openspec validate [change-id] --strict` and fix issues
|
||||
- Request approval: Do not start implementation until proposal is approved
|
||||
|
||||
## Three-Stage Workflow
|
||||
|
||||
### Stage 1: Creating Changes
|
||||
Create proposal when you need to:
|
||||
- Add features or functionality
|
||||
- Make breaking changes (API, schema)
|
||||
- Change architecture or patterns
|
||||
- Optimize performance (changes behavior)
|
||||
- Update security patterns
|
||||
|
||||
Triggers (examples):
|
||||
- "Help me create a change proposal"
|
||||
- "Help me plan a change"
|
||||
- "Help me create a proposal"
|
||||
- "I want to create a spec proposal"
|
||||
- "I want to create a spec"
|
||||
|
||||
Loose matching guidance:
|
||||
- Contains one of: `proposal`, `change`, `spec`
|
||||
- With one of: `create`, `plan`, `make`, `start`, `help`
|
||||
|
||||
Skip proposal for:
|
||||
- Bug fixes (restore intended behavior)
|
||||
- Typos, formatting, comments
|
||||
- Dependency updates (non-breaking)
|
||||
- Configuration changes
|
||||
- Tests for existing behavior
|
||||
|
||||
**Workflow**
|
||||
1. Review `openspec/project.md`, `openspec list`, and `openspec list --specs` to understand current context.
|
||||
2. Choose a unique verb-led `change-id` and scaffold `proposal.md`, `tasks.md`, optional `design.md`, and spec deltas under `openspec/changes/<id>/`.
|
||||
3. Draft spec deltas using `## ADDED|MODIFIED|REMOVED Requirements` with at least one `#### Scenario:` per requirement.
|
||||
4. Run `openspec validate <id> --strict` and resolve any issues before sharing the proposal.
|
||||
|
||||
### Stage 2: Implementing Changes
|
||||
Track these steps as TODOs and complete them one by one.
|
||||
1. **Read proposal.md** - Understand what's being built
|
||||
2. **Read design.md** (if exists) - Review technical decisions
|
||||
3. **Read tasks.md** - Get implementation checklist
|
||||
4. **Implement tasks sequentially** - Complete in order
|
||||
5. **Confirm completion** - Ensure every item in `tasks.md` is finished before updating statuses
|
||||
6. **Update checklist** - After all work is done, set every task to `- [x]` so the list reflects reality
|
||||
7. **Approval gate** - Do not start implementation until the proposal is reviewed and approved
|
||||
|
||||
### Stage 3: Archiving Changes
|
||||
After deployment, create separate PR to:
|
||||
- Move `changes/[name]/` → `changes/archive/YYYY-MM-DD-[name]/`
|
||||
- Update `specs/` if capabilities changed
|
||||
- Use `openspec archive <change-id> --skip-specs --yes` for tooling-only changes (always pass the change ID explicitly)
|
||||
- Run `openspec validate --strict` to confirm the archived change passes checks
|
||||
|
||||
## Before Any Task
|
||||
|
||||
**Context Checklist:**
|
||||
- [ ] Read relevant specs in `specs/[capability]/spec.md`
|
||||
- [ ] Check pending changes in `changes/` for conflicts
|
||||
- [ ] Read `openspec/project.md` for conventions
|
||||
- [ ] Run `openspec list` to see active changes
|
||||
- [ ] Run `openspec list --specs` to see existing capabilities
|
||||
|
||||
**Before Creating Specs:**
|
||||
- Always check if capability already exists
|
||||
- Prefer modifying existing specs over creating duplicates
|
||||
- Use `openspec show [spec]` to review current state
|
||||
- If request is ambiguous, ask 1–2 clarifying questions before scaffolding
|
||||
|
||||
### Search Guidance
|
||||
- Enumerate specs: `openspec spec list --long` (or `--json` for scripts)
|
||||
- Enumerate changes: `openspec list` (or `openspec change list --json` - deprecated but available)
|
||||
- Show details:
|
||||
- Spec: `openspec show <spec-id> --type spec` (use `--json` for filters)
|
||||
- Change: `openspec show <change-id> --json --deltas-only`
|
||||
- Full-text search (use ripgrep): `rg -n "Requirement:|Scenario:" openspec/specs`
|
||||
|
||||
## Quick Start
|
||||
|
||||
### CLI Commands
|
||||
|
||||
```bash
|
||||
# Essential commands
|
||||
openspec list # List active changes
|
||||
openspec list --specs # List specifications
|
||||
openspec show [item] # Display change or spec
|
||||
openspec validate [item] # Validate changes or specs
|
||||
openspec archive <change-id> [--yes|-y] # Archive after deployment (add --yes for non-interactive runs)
|
||||
|
||||
# Project management
|
||||
openspec init [path] # Initialize OpenSpec
|
||||
openspec update [path] # Update instruction files
|
||||
|
||||
# Interactive mode
|
||||
openspec show # Prompts for selection
|
||||
openspec validate # Bulk validation mode
|
||||
|
||||
# Debugging
|
||||
openspec show [change] --json --deltas-only
|
||||
openspec validate [change] --strict
|
||||
```
|
||||
|
||||
### Command Flags
|
||||
|
||||
- `--json` - Machine-readable output
|
||||
- `--type change|spec` - Disambiguate items
|
||||
- `--strict` - Comprehensive validation
|
||||
- `--no-interactive` - Disable prompts
|
||||
- `--skip-specs` - Archive without spec updates
|
||||
- `--yes`/`-y` - Skip confirmation prompts (non-interactive archive)
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
openspec/
|
||||
├── project.md # Project conventions
|
||||
├── specs/ # Current truth - what IS built
|
||||
│ └── [capability]/ # Single focused capability
|
||||
│ ├── spec.md # Requirements and scenarios
|
||||
│ └── design.md # Technical patterns
|
||||
├── changes/ # Proposals - what SHOULD change
|
||||
│ ├── [change-name]/
|
||||
│ │ ├── proposal.md # Why, what, impact
|
||||
│ │ ├── tasks.md # Implementation checklist
|
||||
│ │ ├── design.md # Technical decisions (optional; see criteria)
|
||||
│ │ └── specs/ # Delta changes
|
||||
│ │ └── [capability]/
|
||||
│ │ └── spec.md # ADDED/MODIFIED/REMOVED
|
||||
│ └── archive/ # Completed changes
|
||||
```
|
||||
|
||||
## Creating Change Proposals
|
||||
|
||||
### Decision Tree
|
||||
|
||||
```
|
||||
New request?
|
||||
├─ Bug fix restoring spec behavior? → Fix directly
|
||||
├─ Typo/format/comment? → Fix directly
|
||||
├─ New feature/capability? → Create proposal
|
||||
├─ Breaking change? → Create proposal
|
||||
├─ Architecture change? → Create proposal
|
||||
└─ Unclear? → Create proposal (safer)
|
||||
```
|
||||
|
||||
### Proposal Structure
|
||||
|
||||
1. **Create directory:** `changes/[change-id]/` (kebab-case, verb-led, unique)
|
||||
|
||||
2. **Write proposal.md:**
|
||||
```markdown
|
||||
# Change: [Brief description of change]
|
||||
|
||||
## Why
|
||||
[1-2 sentences on problem/opportunity]
|
||||
|
||||
## What Changes
|
||||
- [Bullet list of changes]
|
||||
- [Mark breaking changes with **BREAKING**]
|
||||
|
||||
## Impact
|
||||
- Affected specs: [list capabilities]
|
||||
- Affected code: [key files/systems]
|
||||
```
|
||||
|
||||
3. **Create spec deltas:** `specs/[capability]/spec.md`
|
||||
```markdown
|
||||
## ADDED Requirements
|
||||
### Requirement: New Feature
|
||||
The system SHALL provide...
|
||||
|
||||
#### Scenario: Success case
|
||||
- **WHEN** user performs action
|
||||
- **THEN** expected result
|
||||
|
||||
## MODIFIED Requirements
|
||||
### Requirement: Existing Feature
|
||||
[Complete modified requirement]
|
||||
|
||||
## REMOVED Requirements
|
||||
### Requirement: Old Feature
|
||||
**Reason**: [Why removing]
|
||||
**Migration**: [How to handle]
|
||||
```
|
||||
If multiple capabilities are affected, create multiple delta files under `changes/[change-id]/specs/<capability>/spec.md`—one per capability.
|
||||
|
||||
4. **Create tasks.md:**
|
||||
```markdown
|
||||
## 1. Implementation
|
||||
- [ ] 1.1 Create database schema
|
||||
- [ ] 1.2 Implement API endpoint
|
||||
- [ ] 1.3 Add frontend component
|
||||
- [ ] 1.4 Write tests
|
||||
```
|
||||
|
||||
5. **Create design.md when needed:**
|
||||
Create `design.md` if any of the following apply; otherwise omit it:
|
||||
- Cross-cutting change (multiple services/modules) or a new architectural pattern
|
||||
- New external dependency or significant data model changes
|
||||
- Security, performance, or migration complexity
|
||||
- Ambiguity that benefits from technical decisions before coding
|
||||
|
||||
Minimal `design.md` skeleton:
|
||||
```markdown
|
||||
## Context
|
||||
[Background, constraints, stakeholders]
|
||||
|
||||
## Goals / Non-Goals
|
||||
- Goals: [...]
|
||||
- Non-Goals: [...]
|
||||
|
||||
## Decisions
|
||||
- Decision: [What and why]
|
||||
- Alternatives considered: [Options + rationale]
|
||||
|
||||
## Risks / Trade-offs
|
||||
- [Risk] → Mitigation
|
||||
|
||||
## Migration Plan
|
||||
[Steps, rollback]
|
||||
|
||||
## Open Questions
|
||||
- [...]
|
||||
```
|
||||
|
||||
## Spec File Format
|
||||
|
||||
### Critical: Scenario Formatting
|
||||
|
||||
**CORRECT** (use #### headers):
|
||||
```markdown
|
||||
#### Scenario: User login success
|
||||
- **WHEN** valid credentials provided
|
||||
- **THEN** return JWT token
|
||||
```
|
||||
|
||||
**WRONG** (don't use bullets or bold):
|
||||
```markdown
|
||||
- **Scenario: User login** ❌
|
||||
**Scenario**: User login ❌
|
||||
### Scenario: User login ❌
|
||||
```
|
||||
|
||||
Every requirement MUST have at least one scenario.
|
||||
|
||||
### Requirement Wording
|
||||
- Use SHALL/MUST for normative requirements (avoid should/may unless intentionally non-normative)
|
||||
|
||||
### Delta Operations
|
||||
|
||||
- `## ADDED Requirements` - New capabilities
|
||||
- `## MODIFIED Requirements` - Changed behavior
|
||||
- `## REMOVED Requirements` - Deprecated features
|
||||
- `## RENAMED Requirements` - Name changes
|
||||
|
||||
Headers matched with `trim(header)` - whitespace ignored.
|
||||
|
||||
#### When to use ADDED vs MODIFIED
|
||||
- ADDED: Introduces a new capability or sub-capability that can stand alone as a requirement. Prefer ADDED when the change is orthogonal (e.g., adding "Slash Command Configuration") rather than altering the semantics of an existing requirement.
|
||||
- MODIFIED: Changes the behavior, scope, or acceptance criteria of an existing requirement. Always paste the full, updated requirement content (header + all scenarios). The archiver will replace the entire requirement with what you provide here; partial deltas will drop previous details.
|
||||
- RENAMED: Use when only the name changes. If you also change behavior, use RENAMED (name) plus MODIFIED (content) referencing the new name.
|
||||
|
||||
Common pitfall: Using MODIFIED to add a new concern without including the previous text. This causes loss of detail at archive time. If you aren’t explicitly changing the existing requirement, add a new requirement under ADDED instead.
|
||||
|
||||
Authoring a MODIFIED requirement correctly:
|
||||
1) Locate the existing requirement in `openspec/specs/<capability>/spec.md`.
|
||||
2) Copy the entire requirement block (from `### Requirement: ...` through its scenarios).
|
||||
3) Paste it under `## MODIFIED Requirements` and edit to reflect the new behavior.
|
||||
4) Ensure the header text matches exactly (whitespace-insensitive) and keep at least one `#### Scenario:`.
|
||||
|
||||
Example for RENAMED:
|
||||
```markdown
|
||||
## RENAMED Requirements
|
||||
- FROM: `### Requirement: Login`
|
||||
- TO: `### Requirement: User Authentication`
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Errors
|
||||
|
||||
**"Change must have at least one delta"**
|
||||
- Check `changes/[name]/specs/` exists with .md files
|
||||
- Verify files have operation prefixes (## ADDED Requirements)
|
||||
|
||||
**"Requirement must have at least one scenario"**
|
||||
- Check scenarios use `#### Scenario:` format (4 hashtags)
|
||||
- Don't use bullet points or bold for scenario headers
|
||||
|
||||
**Silent scenario parsing failures**
|
||||
- Exact format required: `#### Scenario: Name`
|
||||
- Debug with: `openspec show [change] --json --deltas-only`
|
||||
|
||||
### Validation Tips
|
||||
|
||||
```bash
|
||||
# Always use strict mode for comprehensive checks
|
||||
openspec validate [change] --strict
|
||||
|
||||
# Debug delta parsing
|
||||
openspec show [change] --json | jq '.deltas'
|
||||
|
||||
# Check specific requirement
|
||||
openspec show [spec] --json -r 1
|
||||
```
|
||||
|
||||
## Happy Path Script
|
||||
|
||||
```bash
|
||||
# 1) Explore current state
|
||||
openspec spec list --long
|
||||
openspec list
|
||||
# Optional full-text search:
|
||||
# rg -n "Requirement:|Scenario:" openspec/specs
|
||||
# rg -n "^#|Requirement:" openspec/changes
|
||||
|
||||
# 2) Choose change id and scaffold
|
||||
CHANGE=add-two-factor-auth
|
||||
mkdir -p openspec/changes/$CHANGE/{specs/auth}
|
||||
printf "## Why\n...\n\n## What Changes\n- ...\n\n## Impact\n- ...\n" > openspec/changes/$CHANGE/proposal.md
|
||||
printf "## 1. Implementation\n- [ ] 1.1 ...\n" > openspec/changes/$CHANGE/tasks.md
|
||||
|
||||
# 3) Add deltas (example)
|
||||
cat > openspec/changes/$CHANGE/specs/auth/spec.md << 'EOF'
|
||||
## ADDED Requirements
|
||||
### Requirement: Two-Factor Authentication
|
||||
Users MUST provide a second factor during login.
|
||||
|
||||
#### Scenario: OTP required
|
||||
- **WHEN** valid credentials are provided
|
||||
- **THEN** an OTP challenge is required
|
||||
EOF
|
||||
|
||||
# 4) Validate
|
||||
openspec validate $CHANGE --strict
|
||||
```
|
||||
|
||||
## Multi-Capability Example
|
||||
|
||||
```
|
||||
openspec/changes/add-2fa-notify/
|
||||
├── proposal.md
|
||||
├── tasks.md
|
||||
└── specs/
|
||||
├── auth/
|
||||
│ └── spec.md # ADDED: Two-Factor Authentication
|
||||
└── notifications/
|
||||
└── spec.md # ADDED: OTP email notification
|
||||
```
|
||||
|
||||
auth/spec.md
|
||||
```markdown
|
||||
## ADDED Requirements
|
||||
### Requirement: Two-Factor Authentication
|
||||
...
|
||||
```
|
||||
|
||||
notifications/spec.md
|
||||
```markdown
|
||||
## ADDED Requirements
|
||||
### Requirement: OTP Email Notification
|
||||
...
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Simplicity First
|
||||
- Default to <100 lines of new code
|
||||
- Single-file implementations until proven insufficient
|
||||
- Avoid frameworks without clear justification
|
||||
- Choose boring, proven patterns
|
||||
|
||||
### Complexity Triggers
|
||||
Only add complexity with:
|
||||
- Performance data showing current solution too slow
|
||||
- Concrete scale requirements (>1000 users, >100MB data)
|
||||
- Multiple proven use cases requiring abstraction
|
||||
|
||||
### Clear References
|
||||
- Use `file.ts:42` format for code locations
|
||||
- Reference specs as `specs/auth/spec.md`
|
||||
- Link related changes and PRs
|
||||
|
||||
### Capability Naming
|
||||
- Use verb-noun: `user-auth`, `payment-capture`
|
||||
- Single purpose per capability
|
||||
- 10-minute understandability rule
|
||||
- Split if description needs "AND"
|
||||
|
||||
### Change ID Naming
|
||||
- Use kebab-case, short and descriptive: `add-two-factor-auth`
|
||||
- Prefer verb-led prefixes: `add-`, `update-`, `remove-`, `refactor-`
|
||||
- Ensure uniqueness; if taken, append `-2`, `-3`, etc.
|
||||
|
||||
## Tool Selection Guide
|
||||
|
||||
| Task | Tool | Why |
|
||||
|------|------|-----|
|
||||
| Find files by pattern | Glob | Fast pattern matching |
|
||||
| Search code content | Grep | Optimized regex search |
|
||||
| Read specific files | Read | Direct file access |
|
||||
| Explore unknown scope | Task | Multi-step investigation |
|
||||
|
||||
## Error Recovery
|
||||
|
||||
### Change Conflicts
|
||||
1. Run `openspec list` to see active changes
|
||||
2. Check for overlapping specs
|
||||
3. Coordinate with change owners
|
||||
4. Consider combining proposals
|
||||
|
||||
### Validation Failures
|
||||
1. Run with `--strict` flag
|
||||
2. Check JSON output for details
|
||||
3. Verify spec file format
|
||||
4. Ensure scenarios properly formatted
|
||||
|
||||
### Missing Context
|
||||
1. Read project.md first
|
||||
2. Check related specs
|
||||
3. Review recent archives
|
||||
4. Ask for clarification
|
||||
|
||||
## Quick Reference
|
||||
|
||||
### Stage Indicators
|
||||
- `changes/` - Proposed, not yet built
|
||||
- `specs/` - Built and deployed
|
||||
- `archive/` - Completed changes
|
||||
|
||||
### File Purposes
|
||||
- `proposal.md` - Why and what
|
||||
- `tasks.md` - Implementation steps
|
||||
- `design.md` - Technical decisions
|
||||
- `spec.md` - Requirements and behavior
|
||||
|
||||
### CLI Essentials
|
||||
```bash
|
||||
openspec list # What's in progress?
|
||||
openspec show [item] # View details
|
||||
openspec validate --strict # Is it correct?
|
||||
openspec archive <change-id> [--yes|-y] # Mark complete (add --yes for automation)
|
||||
```
|
||||
|
||||
Remember: Specs are truth. Changes are proposals. Keep them in sync.
|
||||
@@ -0,0 +1 @@
|
||||
This folder intentionally left blank. (OpenSpec scaffolding will add proposal/tasks/design.)
|
||||
@@ -0,0 +1,84 @@
|
||||
# Design: Accelerate Embedding Similarity
|
||||
|
||||
## Context
|
||||
|
||||
The server uses Azure OpenAI embeddings for competence and role similarity. The matching pipeline can be slow because embeddings are currently requested on-demand and repeatedly from inner loops.
|
||||
|
||||
The system already has a persistent SQLite embedding cache (`EmbeddingCache`). However, without bulk prefetch and without a run-scoped memoization layer, the similarity engine still:
|
||||
|
||||
- repeatedly normalizes and hashes text keys
|
||||
- performs many repeated SQLite reads
|
||||
- performs sequential Azure calls for cache misses
|
||||
|
||||
## Goals
|
||||
|
||||
- Bulk prefetch embeddings for a matching run.
|
||||
- Maintain deterministic behavior and strict error semantics.
|
||||
- Preserve existing similarity contracts.
|
||||
|
||||
## Architecture Changes
|
||||
|
||||
### Components impacted
|
||||
|
||||
- `SimilarityEngine` (`src/teamlandkarte_mcp/matching/similarity.py`)
|
||||
- Add instance variable for in-memory cache (lives with the engine; not created/cleared per invocation)
|
||||
- Add public method `prefetch_embeddings(required_texts, candidate_texts, required_roles, candidate_roles) -> dict[str, list[float]]`
|
||||
- `AzureOpenAIClient` (`src/teamlandkarte_mcp/azure/openai_client.py`)
|
||||
- Modify `get_embeddings_batch()` to use true batch API with `input=[...]`
|
||||
- `CostTracker` (`src/teamlandkarte_mcp/azure/cost_tracker.py`)
|
||||
- May need adjustment to log batch requests (log once per chunk with count)
|
||||
- `EmbeddingCache` (no schema changes expected)
|
||||
|
||||
### Data flow (proposed)
|
||||
|
||||
1. **Collect** all texts that will be embedded in the current run.
|
||||
- This is a **single global collection pass** over the inputs and **all free capacities** (roles + competences), not a per-person loop that embeds incrementally.
|
||||
2. **Normalize + deduplicate** using existing `_normalize_text()` logic.
|
||||
- Skip empty/whitespace-only texts
|
||||
- Emit a **Python logger warning** if all candidate competences normalize to empty
|
||||
3. **Resolve embeddings** via:
|
||||
- in-memory cache (instance variable in `SimilarityEngine`)
|
||||
- SQLite cache (`EmbeddingCache`)
|
||||
- Azure embeddings API (**batch only the cache-missing texts**, **chunked** by batch size)
|
||||
4. **Compute** cosine similarities locally using the prefetched mapping.
|
||||
|
||||
### Normalization and keys
|
||||
|
||||
Current cache key behavior:
|
||||
|
||||
- normalization: trim + collapse whitespace
|
||||
- key: SHA256 of `model|dims|normalized(text).lower()`
|
||||
|
||||
This proposal keeps the same normalization and key derivation to avoid invalidating the on-disk cache.
|
||||
|
||||
## Batch embeddings API
|
||||
|
||||
The OpenAI/Azure embeddings API supports embedding multiple inputs per request (`input=[...]`).
|
||||
|
||||
Implementation requirements:
|
||||
|
||||
- preserve stable mapping from input texts → returned embedding vectors
|
||||
- rely on `index` field in API response if available
|
||||
- otherwise assume stable ordering
|
||||
- chunk large requests to avoid request size limits (configurable batch size, default 128)
|
||||
- strict failure semantics:
|
||||
- entire matching operation fails immediately on error
|
||||
- log the failing **chunk input list** via Python logging to aid diagnosis
|
||||
- cost tracking: log once per batch chunk with a count of embeddings requested
|
||||
|
||||
## Trade-offs
|
||||
|
||||
- Batch calls reduce network requests dramatically but may increase blast radius: one failing request could affect multiple inputs.
|
||||
- Mitigation: keep retry logic at the chunk level.
|
||||
- Prefetch requires holding larger embedding dictionaries in memory.
|
||||
- Mitigation: scope to a single invocation, and only store vectors required for that run.
|
||||
|
||||
## Test Strategy
|
||||
|
||||
- Unit test prefetch/dedup: ensure the Azure client is called once per unique normalized text (only for cache misses).
|
||||
- Unit test cache layering: ensure hits are served from in-memory cache (first), then SQLite, and do not call Azure.
|
||||
- Unit test chunking: ensure multiple Azure calls are made when `len(cache_missing_texts) > batch_size`.
|
||||
- Unit test empty input handling: verify warning when all candidate competences normalize to empty; verify prefetch skips empty texts.
|
||||
- Unit test error identification: verify that batch failures report which specific text(s) caused the error.
|
||||
- Integration test (manual/gated): validate real Azure batch API behavior with live credentials.
|
||||
|
||||
@@ -0,0 +1,124 @@
|
||||
# Change Proposal: Accelerate Embedding Prefetch and Similarity Computation
|
||||
|
||||
- **Change ID**: `accelerate-embedding-similarity`
|
||||
- **Status**: Proposed
|
||||
- **Target**: `teamlandkarte-mcp`
|
||||
- **Author**: Thomas Handke
|
||||
- **Date**: 2026-02-14
|
||||
|
||||
## Summary
|
||||
|
||||
Speed up `find_matching_capacities(...)` by reducing embedding API calls and repeated cache lookups.
|
||||
|
||||
This change introduces three complementary optimizations:
|
||||
|
||||
1. **Dedup + bulk prefetch** of all embedding texts needed in a matching run (required competences, candidate competences, and roles).
|
||||
2. A **per-run in-memory cache** layered on top of the existing SQLite embedding cache to avoid repeated SQLite reads within a single run.
|
||||
3. Support for **true Azure Embeddings batch requests** (single request with multiple inputs), including chunking, while preserving deterministic error semantics.
|
||||
|
||||
## Motivation
|
||||
|
||||
The current implementation calls embeddings repeatedly inside inner loops. While a persistent SQLite cache exists, repeated calls still incur overhead:
|
||||
|
||||
- many redundant `cache.get(...)` reads
|
||||
- repeated normalization/keying for identical strings
|
||||
- unnecessary Azure round-trips when multiple missing terms are encountered and requests are performed sequentially
|
||||
|
||||
The result is that `find_matching_capacities(...)` can be slow, and while the second run is faster (due to cache hits), it can still take noticeable time.
|
||||
|
||||
## Goals
|
||||
|
||||
1. Ensure each unique text is embedded at most once per run (network and cache).
|
||||
2. Prefetch embeddings **before** similarity computation so inner loops only perform vector math.
|
||||
3. Enable Azure embeddings API batch calls to reduce network round-trips.
|
||||
4. Preserve current similarity output contracts and scoring behavior.
|
||||
5. Maintain explicit, deterministic failure semantics (no heuristics).
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Changing match scoring weights or category thresholds.
|
||||
- Changing the similarity strategy semantics ("per_skill" vs "aggregate").
|
||||
- Changing the embedding model or dimensionality.
|
||||
- Introducing new external dependencies unless necessary.
|
||||
|
||||
## Proposed Changes
|
||||
|
||||
### 1) Bulk prefetch & dedup in `SimilarityEngine`
|
||||
|
||||
#### Current behavior (problem)
|
||||
|
||||
`SimilarityEngine` currently embeds terms on demand via `_embed(...)`. For large candidate lists, this forces a high number of calls to `_embed`, and repeated SQLite reads.
|
||||
|
||||
#### New behavior
|
||||
|
||||
Add a prefetch step that:
|
||||
|
||||
- collects all texts required to compute similarity for a run (competences **and roles**)
|
||||
- **Important:** this collection is performed **across the entire candidate set** (i.e., roles and competences of **all free capacities at once**), not person-by-person.
|
||||
- normalizes for cache key stability
|
||||
- deduplicates texts
|
||||
- fetches all embeddings from caches/API in bulk
|
||||
- returns a mapping from normalized text → embedding vector
|
||||
|
||||
Then similarity computation uses the prefetched mapping (pure Python vector math).
|
||||
|
||||
### 2) Per-run in-memory cache
|
||||
|
||||
Add a lightweight in-memory dict for the duration of a single matching invocation.
|
||||
|
||||
- key: `(model, dimensions, normalized_text)` or reuse the computed SHA256 cache key
|
||||
- value: `list[float]`
|
||||
|
||||
This prevents repeated SQLite lookups for frequently used items (common competences, repeated role strings, etc.).
|
||||
|
||||
### 3) True Azure embeddings batch calls
|
||||
|
||||
Change `AzureOpenAIClient.get_embeddings_batch(...)` to use the Azure embeddings API with `input=[...]` to embed multiple texts in a single request.
|
||||
|
||||
Key details:
|
||||
|
||||
- supports chunking with a **configurable** batch size
|
||||
- preserves stable output ordering and deterministic error semantics
|
||||
- continues to log costs via `CostTracker`
|
||||
|
||||
If any item fails, the batch call should raise `AzureAPIError` (no fallback).
|
||||
|
||||
## Decisions (resolved)
|
||||
|
||||
- **Scope**: bulk prefetch includes **competences and roles**.
|
||||
- **Batching**: Azure embeddings requests are made in chunks with a **configurable** batch size.
|
||||
- **Normalization**: keep the current text normalization and cache-key behavior.
|
||||
- **Concurrency**: process chunks **sequentially** (no parallel calls).
|
||||
- **In-memory cache lifecycle**: implemented as a new instance variable in `SimilarityEngine` and **not** created/cleaned per matching invocation.
|
||||
- **Prefetch API**: new public method `prefetch_embeddings(required_texts, candidate_texts, required_roles, candidate_roles)` that returns a mapping from normalized text → embedding vector.
|
||||
- **Azure batch response ordering**: rely on the `index` field in the response if available, otherwise assume stable ordering.
|
||||
- **Error handling**: entire matching operation fails immediately; attempt to identify and report which specific text(s) caused the failure.
|
||||
- **Cost tracking**: log once per batch chunk with a count.
|
||||
- **Empty inputs**:
|
||||
- emit a **Python logger warning** when all candidate competences normalize to empty strings
|
||||
- prefetch skips empty/whitespace-only texts
|
||||
- **Cache miss batching**: batch-request only the missing embeddings (after resolving cache hits from SQLite).
|
||||
|
||||
## Configuration
|
||||
|
||||
Introduce (or reuse if already present) one configuration knob:
|
||||
|
||||
- `azure_openai.embedding_batch_size` (integer)
|
||||
- default: `128` (conservative)
|
||||
- no enforced maximum value
|
||||
- used to chunk `input=[...]` for Azure embeddings batch requests
|
||||
- if not specified in `config.toml`, the default value is used
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- A single `find_matching_capacities(...)` run should perform at most one embedding generation per unique term (after cache resolution).
|
||||
- Only cache-missing embeddings are batch-requested from Azure (cache hits are not re-requested).
|
||||
- For warmed cache runs, no Azure embedding calls should occur.
|
||||
- Unit tests cover dedup/prefetch, cache layering, and batch request behavior.
|
||||
- Integration tests (manual/gated) verify real Azure batch API behavior.
|
||||
|
||||
## Rollout / Compatibility
|
||||
|
||||
- No configuration changes are strictly required; the baseline refactor should work with existing `config.toml`.
|
||||
- If new config knobs are added (e.g., batch size), defaults should preserve behavior.
|
||||
|
||||
+65
@@ -0,0 +1,65 @@
|
||||
# Capability: Embedding Prefetch & Batch Similarity
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Bulk prefetch and deduplicate embedding inputs
|
||||
|
||||
The server MUST collect all texts that require embeddings for a matching run, normalize them, deduplicate them, and prefetch their embeddings before computing similarity scores.
|
||||
|
||||
#### Scenario: Matching run with repeated competences
|
||||
|
||||
- Given a matching run where the required competences contain duplicates (e.g. "Python", "Python")
|
||||
- And a candidate list contains overlapping competences (e.g. multiple candidates list "Python")
|
||||
- When the similarity computation starts
|
||||
- Then the embedding for "Python" is requested at most once from the Azure embeddings API
|
||||
- And all uses of "Python" reuse the same embedding vector for similarity computation
|
||||
|
||||
### Requirement: Compute similarity using prefetched vectors (no inner-loop embedding calls)
|
||||
|
||||
The similarity computation MUST use prefetched embedding vectors and MUST NOT request embeddings from inside per-competence or per-candidate inner loops.
|
||||
|
||||
#### Scenario: Per-skill similarity on large candidate set
|
||||
|
||||
- Given required competences and a large candidate competence list
|
||||
- When `per_skill` similarity is computed
|
||||
- Then all embeddings are resolved prior to the nested loops
|
||||
- And the nested loops perform only cosine similarity calculations
|
||||
|
||||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: Batch embedding API support in Azure client
|
||||
|
||||
`AzureOpenAIClient.get_embeddings_batch(...)` MUST support true request-level batching using the Azure embeddings API (`input=[...]`) and MUST preserve input ordering.
|
||||
|
||||
#### Scenario: Batch embedding input list
|
||||
|
||||
- Given a list of N texts to embed
|
||||
- When `get_embeddings_batch(texts)` is called
|
||||
- Then the client makes as few Azure embedding requests as possible (subject to chunking)
|
||||
- And the returned list of embeddings aligns with `texts` order
|
||||
|
||||
### Requirement: Run-scoped in-memory embedding cache
|
||||
|
||||
The server MUST maintain a run-scoped in-memory cache to prevent repeated SQLite cache reads for identical texts during a single matching operation.
|
||||
|
||||
#### Scenario: Frequent repeated cache hits within one run
|
||||
|
||||
- Given cached embeddings are present on disk
|
||||
- And a matching run needs the same text embedding multiple times
|
||||
- When the similarity engine resolves embeddings
|
||||
- Then repeated lookups for the same text are served from an in-memory cache after the first lookup
|
||||
- And no additional SQLite reads are required for that text during the run
|
||||
|
||||
## REMOVED Requirements
|
||||
|
||||
### Requirement: Sequential batch behavior requirement
|
||||
|
||||
The previous constraint that `get_embeddings_batch` must call the embeddings API sequentially per text is removed.
|
||||
|
||||
#### Scenario: Legacy sequential constraint
|
||||
|
||||
- Given earlier requirements constrained `get_embeddings_batch` to per-text calls
|
||||
- When the system is upgraded for performance
|
||||
- Then this constraint no longer applies
|
||||
- And request-level batching is allowed
|
||||
|
||||
@@ -0,0 +1,100 @@
|
||||
# Tasks: Accelerate Embedding Similarity
|
||||
|
||||
> Change: `accelerate-embedding-similarity`
|
||||
>
|
||||
> Status: **Approved / Implemented**
|
||||
|
||||
## Overview
|
||||
|
||||
Reduce runtime of `find_matching_capacities(...)` by implementing:
|
||||
|
||||
- embedding text deduplication + **global** bulk prefetch (roles + competences for **all free capacities at once**, not person-by-person)
|
||||
- in-memory embedding memoization
|
||||
- true Azure embeddings batch requests with chunking
|
||||
|
||||
## Task Breakdown
|
||||
|
||||
### Phase 1: Design & Validation Plan
|
||||
|
||||
- [x] 1.1 Identify hot paths during matching (where `_embed(...)` is called most).
|
||||
- [x] 1.2 Confirm normalization + cache key rules are unchanged and documented.
|
||||
- [x] 1.3 Define expected call-count behavior (upper bounds):
|
||||
- per run: max 1 Azure embedding per unique normalized text (subject to chunking, after cache resolution)
|
||||
- per run: max 1 SQLite read per unique normalized text (after in-memory cache check)
|
||||
|
||||
### Phase 2: Global Bulk Prefetch in `SimilarityEngine`
|
||||
|
||||
- [x] 2.1 Add a new instance variable `_run_cache: dict[str, list[float]]` to `SimilarityEngine` (in-memory cache; **not** created/cleared per matching invocation).
|
||||
- [x] 2.2 Implement a public method `prefetch_embeddings(required_texts: list[str], candidate_texts: list[str], required_roles: list[str], candidate_roles: list[str]) -> dict[str, list[float]]`:
|
||||
- collect all input texts (required competences, **required roles**, all candidate competences, **all candidate roles**)
|
||||
- normalize + deduplicate using existing `_normalize_text()` logic
|
||||
- skip empty/whitespace-only texts
|
||||
- emit a **Python logger warning** if all candidate competences normalize to empty
|
||||
- [x] 2.3 Implement prefetch resolution logic:
|
||||
1) check in-memory cache (`_run_cache`)
|
||||
2) check SQLite embedding cache
|
||||
3) batch-request only cache-missing texts from Azure API (chunked by batch size)
|
||||
- [x] 2.4 Refactor `_per_skill_similarity` to use prefetched embeddings only (no embedding calls inside inner loops).
|
||||
- [x] 2.5 Refactor `_aggregate_similarity` to use prefetched embeddings only.
|
||||
- [x] 2.6 Ensure output format is unchanged and remains deterministic.
|
||||
|
||||
### Phase 3: In-memory Run Cache
|
||||
|
||||
- [x] 3.1 Verify `_run_cache` is an instance-level cache that does not require per-invocation clearing.
|
||||
- [x] 3.2 Add tests ensuring repeated embedding lookups for the same text:
|
||||
- are served from `_run_cache` after first lookup
|
||||
- do not call SQLite/API multiple times for the same text
|
||||
|
||||
### Phase 4: Azure Batch Embeddings Support
|
||||
|
||||
- [x] 4.1 Modify `AzureOpenAIClient.get_embeddings_batch()` to use Azure embeddings API with `input=[...]`.
|
||||
- [x] 4.2 Implement chunking using `azure_openai.embedding_batch_size` (default: 128, configurable, no enforced maximum).
|
||||
- [x] 4.3 Preserve ordering using `index` field in API response if available, otherwise assume stable ordering.
|
||||
- [x] 4.4 Ensure strict error semantics:
|
||||
- entire matching operation fails immediately on error
|
||||
- log the failing **chunk input list** via Python logging
|
||||
- [x] 4.5 Update `CostTracker` to log batch requests: log once per chunk with count of embeddings requested.
|
||||
|
||||
### Phase 5: Configuration
|
||||
|
||||
- [x] 5.1 Extend config model to include `azure_openai.embedding_batch_size` (integer, default: 128, no enforced maximum).
|
||||
- [x] 5.2 Update `config.toml.example` with the new setting and comments.
|
||||
- [x] 5.3 Ensure fallback: if `embedding_batch_size` is not specified in `config.toml`, use the default value (128).
|
||||
|
||||
### Phase 6: Tests
|
||||
|
||||
- [x] 6.1 Unit tests: global prefetch/dedup
|
||||
- verify dedup across required + all candidates (incl. roles)
|
||||
- verify embedding client is called once per unique normalized text (only for cache misses)
|
||||
- verify empty/whitespace-only texts are skipped
|
||||
- verify a Python logger warning is emitted when all candidate competences normalize to empty
|
||||
- [x] 6.2 Unit tests: cache layering
|
||||
- verify in-memory `_run_cache` prevents repeated SQLite reads within a run
|
||||
- verify SQLite hits prevent Azure calls
|
||||
- [x] 6.3 Unit tests: Azure batch request behavior (mocked)
|
||||
- verifies batch call for N texts when `N <= batch_size`
|
||||
- verifies chunking when `N > batch_size` (multiple batch calls)
|
||||
- verifies output ordering is stable (using `index` field or assuming stable order)
|
||||
- verifies strict error propagation
|
||||
- verifies the failing chunk input list is logged
|
||||
- verifies cost tracking logs once per chunk with count
|
||||
- [x] 6.4 Integration test (manual/gated only): real Azure batch embeddings
|
||||
- confirm request uses batch `input=[...]` semantics
|
||||
- ensure behavior matches expected semantics for a representative set of texts
|
||||
|
||||
### Phase 7: Documentation + Assistant Prompt Updates
|
||||
|
||||
- [x] 7.1 Update `README.md` to describe the optimization at a high level and mention the new batch size knob.
|
||||
- [x] 7.2 Update `docs/assistant_system_prompt.md` to reflect:
|
||||
- embeddings are prefetched globally per run
|
||||
- batching/chunking is used and configurable
|
||||
- failure semantics remain strict (no heuristics)
|
||||
- [x] 7.3 Update any Azure setup docs if needed (e.g., `docs/azure_openai_setup.md`) to mention embedding batch behavior and tuning.
|
||||
|
||||
### Phase 8: Quality Gates / Release Hygiene
|
||||
|
||||
- [x] 8.1 Run unit tests: `uv run pytest -m "not integration" -q`
|
||||
- [x] 8.2 Run Ruff and mypy via `uv`.
|
||||
- [ ] 8.3 (Manual/gated only) Run integration tests (Azure) to validate real batch calls.
|
||||
- [x] 8.4 Update change docs status/checklists and ensure OpenSpec validates (`openspec validate accelerate-embedding-similarity --strict`).
|
||||
|
||||
@@ -0,0 +1,666 @@
|
||||
# Design: BM25 + RRF Competence Matching
|
||||
|
||||
## Context
|
||||
|
||||
This design specifies the internal architecture for the BM25 + Reciprocal Rank Fusion
|
||||
(RRF) competence matching strategy with optional LLM-based Auto-Tagging, controlled by
|
||||
`matching.similarity.use_bm25_search` and `matching.similarity.use_auto_tagging`.
|
||||
|
||||
The goal is to provide a lexical alternative to embedding-based similarity that assigns
|
||||
near-zero scores when candidate competences do not literally overlap with the required
|
||||
competences, while still producing a normalized score compatible with the existing
|
||||
`Matcher` and `scorer` contracts.
|
||||
|
||||
Auto-Tagging optionally extends BM25 by having an LLM pre-expand each candidate's
|
||||
competence list with canonical forms of any required competences it already covers
|
||||
(synonyms, abbreviations, cross-language equivalents). This bridges the lexical gap for
|
||||
known skill aliases without persisting any data.
|
||||
|
||||
---
|
||||
|
||||
## Key Decisions
|
||||
|
||||
### 1. BM25 is per-skill only, not aggregate
|
||||
|
||||
**Decision:** `_bm25_rrf_similarity()` operates per required competence (like the existing
|
||||
`per_skill` strategy), not as an aggregate over all required competences.
|
||||
|
||||
**Rationale:**
|
||||
- Aggregate-style BM25 would require concatenating all required competences into a single
|
||||
query document, which loses the per-skill granularity needed for `matched_competences`
|
||||
and `missing_competences` diagnostic output.
|
||||
- Per-skill BM25 + RRF preserves the same output shape as `_per_skill_similarity()`,
|
||||
making it a drop-in replacement.
|
||||
|
||||
---
|
||||
|
||||
### 2. BM25 index is built once globally over all filtered candidates
|
||||
|
||||
**Decision:** One `Bm25Index` is built over the **union of all filtered candidates'
|
||||
competences** before the per-candidate scoring loop in `Matcher.match()`, not fresh per
|
||||
`compute_competence_similarity()` call. The index is passed in as `global_index` and each
|
||||
per-candidate call filters BM25 results down to that candidate's own competence set.
|
||||
|
||||
**Rationale:**
|
||||
- **IDF pathology fix:** With a per-person corpus (N = that person's competence count),
|
||||
BM25 IDF is `log((N − df + 0.5) / (df + 0.5))`. For N = 5–30 and any term that appears
|
||||
in most or all documents, this is 0 or negative. `rank_bm25` therefore returns zero or
|
||||
negative raw scores for typical real-world skill lists — making every BM25 match appear
|
||||
as zero before `max(0.0, raw)` clamping. A global corpus of all candidates' competences
|
||||
ensures N is large enough for IDF to carry meaningful signal.
|
||||
- **Still cheap:** Building one index over the deduplicated union of all candidates'
|
||||
competences is O(M) where M is the total unique competence count — still milliseconds.
|
||||
- **Correctness over caching:** The global index is not cached across MCP tool calls; it is
|
||||
rebuilt fresh for each `match_candidates` invocation, so stale state is impossible.
|
||||
|
||||
---
|
||||
|
||||
### 3. RRF with single ranked list (BM25 only)
|
||||
|
||||
**Decision:** For the initial implementation, `reciprocal_rank_fusion()` fuses a single
|
||||
ranked list (BM25 ranks only), effectively normalizing BM25 scores via the RRF formula.
|
||||
|
||||
**Formula (single list):**
|
||||
|
||||
```
|
||||
rrf_score(candidate) = 1 / (k + rank(candidate))
|
||||
```
|
||||
|
||||
where `k = 60` (standard constant, Cormack et al. 2009), and `rank` is 1-based.
|
||||
|
||||
**Output normalization:**
|
||||
The raw RRF scores sum to a value > 1 across all candidates. To map to [0.0, 1.0]:
|
||||
|
||||
```
|
||||
normalized = rrf_score(best_candidate) / rrf_score_at_rank_1
|
||||
= 1 / (k + rank) * (k + 1)
|
||||
= (k + 1) / (k + rank)
|
||||
```
|
||||
|
||||
For `k=60`, rank 1 → 1.0, rank 2 → 61/62 ≈ 0.984, rank 10 → 61/70 ≈ 0.871.
|
||||
|
||||
This means BM25 scores do NOT inherently drop to near-zero for adjacent candidates.
|
||||
To address this, RRF scores are **zeroed** if the underlying BM25 score is 0.0, because
|
||||
a BM25 score of 0 means **no token overlap at all**. This is the critical property that
|
||||
fixes the false-positive problem.
|
||||
|
||||
**Implementation:**
|
||||
```python
|
||||
def reciprocal_rank_fusion(
|
||||
ranked_lists: list[list[tuple[str, float]]],
|
||||
*,
|
||||
k: int = 60,
|
||||
) -> dict[str, float]:
|
||||
...
|
||||
# Skip lists with no BM25 signal at all
|
||||
if all(score == 0.0 for _, score in ranked_list):
|
||||
continue
|
||||
|
||||
# IMPORTANT: Keep (candidate, 0.0) entries in the ranked list for transparency,
|
||||
# but treat them as "no signal" for fusion. I.e., candidates with BM25==0.0 MUST
|
||||
# NOT receive a positive fused score just because they have a (low) rank at the
|
||||
# end of the list.
|
||||
for rank, (candidate, score) in enumerate(ranked_list, start=1):
|
||||
if score == 0.0:
|
||||
continue
|
||||
fused[candidate] += 1 / (k + rank)
|
||||
...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4. `best_match` reported per required competence
|
||||
|
||||
**Decision:** `best_match` in the output dict is the candidate with the highest BM25 score
|
||||
(rank 1 in the BM25 list). If no candidate has any token overlap (BM25 = 0 for all),
|
||||
`best_match = None` and `score = 0.0`.
|
||||
|
||||
---
|
||||
|
||||
### 5. Role similarity unchanged
|
||||
|
||||
**Decision:** `compute_role_similarity()` is always embedding-based, regardless of
|
||||
`use_bm25_search`.
|
||||
|
||||
**Rationale:**
|
||||
- Role names are short, single-phrase labels ("Backend Developer", "Data Engineer").
|
||||
- Embeddings handle multilingual and synonym variants better for roles.
|
||||
- BM25's exact-match strength is less valuable for role inference.
|
||||
|
||||
---
|
||||
|
||||
### 6. `rank_bm25` as the BM25 library
|
||||
|
||||
**Decision:** Use the `rank-bm25` PyPI package (`BM25Okapi` class).
|
||||
|
||||
**Rationale:**
|
||||
- Pure Python, no C extensions or native code.
|
||||
- Actively maintained, MIT licensed.
|
||||
- Implements BM25 Okapi (the standard variant) with configurable `k1` and `b` parameters.
|
||||
- Minimal footprint, no additional build steps.
|
||||
|
||||
**Tokenization:**
|
||||
Candidate and query texts are tokenized by lowercasing and splitting on non-word
|
||||
characters (whitespace, hyphens, slashes, parentheses, dots):
|
||||
|
||||
```python
|
||||
import re
|
||||
|
||||
def _tokenize(text: str) -> list[str]:
|
||||
return [t for t in re.split(r"[\W_]+", text.lower()) if t]
|
||||
```
|
||||
|
||||
This ensures "Progressive Web App (PWA)" tokenizes to
|
||||
`["progressive", "web", "app", "pwa"]`, and "CI/CD Pipeline" becomes
|
||||
`["ci", "cd", "pipeline"]`.
|
||||
|
||||
---
|
||||
|
||||
### 7. No changes to `Matcher`, `scorer`, or output format
|
||||
|
||||
**Decision:** `_bm25_rrf_similarity()` returns `dict[str, dict[str, object]]` in the
|
||||
identical shape to `_per_skill_similarity()`:
|
||||
|
||||
```python
|
||||
{
|
||||
"Python": {
|
||||
"score": 0.0,
|
||||
"best_match": None,
|
||||
"rationale": "BM25: no token overlap with any candidate competence.",
|
||||
},
|
||||
"Machine Learning": {
|
||||
"score": 0.0,
|
||||
"best_match": None,
|
||||
"rationale": "BM25: no token overlap with any candidate competence.",
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
The `Matcher.match()` loop, the `ScoredCapacity` model, the `compute_overall()` scorer,
|
||||
and all MCP tool output formatting remain unchanged.
|
||||
|
||||
---
|
||||
|
||||
### 8. LLM Auto-Tagging: ephemeral competence expansion
|
||||
|
||||
**Decision:** When `use_auto_tagging = true`, the `AutoTagger` issues a single LLM
|
||||
chat-completion call per candidate per scoring invocation to identify which required
|
||||
competences are already covered by the candidate's existing competences (via synonym,
|
||||
abbreviation, or cross-language equivalence). The response is a JSON list of canonical
|
||||
required-competence names to add to the candidate's working list. The expanded list is
|
||||
used only for that BM25 scoring call and is **never persisted**.
|
||||
|
||||
**Rationale:**
|
||||
- Auto-tagging directly addresses BM25's known limitation with lexical disjoint synonyms
|
||||
(e.g., "ML" for "Machine Learning", "Softwarearchitektur" for "Software Architecture").
|
||||
- Making the expansion ephemeral keeps the data model clean and audit-friendly.
|
||||
- Restricting additions to "names already covered by the candidate's existing entries"
|
||||
means the LLM cannot invent skills — it can only surface canonical equivalents.
|
||||
- LLM call cost is bounded: one call per candidate per `compute_competence_similarity()`
|
||||
invocation, with a compact prompt (required list + candidate list).
|
||||
|
||||
**LLM prompt schema:**
|
||||
```
|
||||
System: You are a skill-taxonomy assistant. Given a list of REQUIRED competences and a
|
||||
candidate's EXISTING competences, output ONLY the canonical names from REQUIRED
|
||||
that are already covered by one or more entries in EXISTING (via synonym,
|
||||
abbreviation, or cross-language equivalence). Do not invent new skills.
|
||||
|
||||
User: REQUIRED: ["Machine Learning", "Python", "CI/CD"]
|
||||
EXISTING: ["ML", "Python", "Jenkins", "Continuous Integration"]
|
||||
|
||||
Response (JSON): {"additions": ["Machine Learning", "CI/CD"]}
|
||||
```
|
||||
|
||||
**Azure OpenAI `response_format`:** `{"type": "json_object"}` is used to enforce
|
||||
structured JSON output, eliminating parsing failures.
|
||||
|
||||
**Error handling:** If the LLM call fails (timeout, API error, malformed JSON), the
|
||||
`AutoTagger` returns the original unmodified candidate list. BM25 scoring continues with
|
||||
no expansion — graceful degradation.
|
||||
|
||||
---
|
||||
|
||||
## Data Flow
|
||||
|
||||
```
|
||||
find_matching_capacities(...) / find_matching_tasks(...)
|
||||
│
|
||||
└─► Matcher.match(capacities, requirements)
|
||||
│
|
||||
├─ [use_bm25_search=True]
|
||||
│ Build ONE Bm25Index over all filtered candidates' competences (global corpus)
|
||||
│ → global_bm25_index (passed into every compute_competence_similarity call)
|
||||
│
|
||||
└─► for each candidate cap in filtered:
|
||||
SimilarityEngine.compute_competence_similarity(
|
||||
required=requirements.competences,
|
||||
candidate=cap.competences,
|
||||
global_index=global_bm25_index,
|
||||
)
|
||||
│
|
||||
├─ [use_bm25_search=False]
|
||||
│ _per_skill_similarity() / _aggregate_similarity()
|
||||
│ (embedding-based, unchanged)
|
||||
│
|
||||
└─ [use_bm25_search=True]
|
||||
│
|
||||
├─ [use_auto_tagging=True]
|
||||
│ AutoTagger.expand_competences(
|
||||
│ required=required,
|
||||
│ existing=candidate,
|
||||
│ )
|
||||
│ → expanded_candidate (ephemeral, not persisted)
|
||||
│
|
||||
└─► _bm25_rrf_similarity(required, expanded_candidate, global_index)
|
||||
│
|
||||
├─► global_index.rank(query=required_comp)
|
||||
│ → filter to candidate_set
|
||||
│ → list[(candidate_text, bm25_score)]
|
||||
│ (falls back to bm25_rank_competences() if no global_index)
|
||||
│
|
||||
└─► reciprocal_rank_fusion(ranked_lists)
|
||||
→ dict[required_comp → rrf_score]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Module Structure
|
||||
|
||||
```
|
||||
src/teamlandkarte_mcp/matching/
|
||||
├── auto_tagger.py # AutoTagger class + expand_competences()
|
||||
├── bm25.py # Bm25Index dataclass + bm25_rank_competences()
|
||||
├── rrf.py # reciprocal_rank_fusion()
|
||||
├── matcher.py # unchanged
|
||||
├── scorer.py # unchanged
|
||||
└── similarity.py # extended: _bm25_rrf_similarity(), auto-tag branch, flag routing
|
||||
|
||||
src/teamlandkarte_mcp/azure/
|
||||
└── openai_client.py # extended: chat_completion(), chat_deployment, llm_api_key
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## `bm25.py` — Public Interface
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class Bm25Index:
|
||||
"""In-memory BM25 index over a corpus of competence strings.
|
||||
|
||||
Intended to be built once over the union of all candidates' competences
|
||||
(the global corpus) so that IDF values are meaningful. Each per-candidate
|
||||
scoring call queries the global index and filters results to that candidate's
|
||||
own competence set.
|
||||
|
||||
Args:
|
||||
corpus: List of competence strings to index.
|
||||
|
||||
Notes:
|
||||
Tokenization lowercases and splits on non-word characters.
|
||||
An empty corpus produces an index that scores all queries as 0.0.
|
||||
Raw BM25 scores are clamped to ``max(0.0, raw)`` — ubiquitous terms
|
||||
whose IDF is negative simply produce 0.0.
|
||||
"""
|
||||
corpus: list[str]
|
||||
|
||||
def rank(self, query: str) -> list[tuple[str, float]]:
|
||||
"""Rank corpus documents for a single query string.
|
||||
|
||||
Args:
|
||||
query: Required competence text used as the BM25 query.
|
||||
|
||||
Returns:
|
||||
List of (candidate_text, bm25_score) sorted by score descending.
|
||||
Entries with score 0.0 are included (caller decides cutoff).
|
||||
Scores are clamped to max(0.0, raw); negative IDF yields 0.0.
|
||||
"""
|
||||
...
|
||||
|
||||
|
||||
def bm25_rank_competences(
|
||||
required: str,
|
||||
candidates: list[str],
|
||||
) -> list[tuple[str, float]]:
|
||||
"""Rank candidate competences against a single required competence.
|
||||
|
||||
Convenience wrapper: builds a temporary ``Bm25Index`` and ranks the query.
|
||||
Suitable for standalone use and tests; in production the global index built
|
||||
by ``Matcher`` is preferred (see ``Bm25Index`` notes on IDF correctness).
|
||||
|
||||
Args:
|
||||
required: The required competence text (BM25 query).
|
||||
candidates: Candidate competence strings to rank.
|
||||
|
||||
Returns:
|
||||
Ranked list of (candidate_text, bm25_score), descending by score.
|
||||
"""
|
||||
...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## `rrf.py` — Public Interface
|
||||
|
||||
```python
|
||||
def reciprocal_rank_fusion(
|
||||
ranked_lists: list[list[tuple[str, float]]],
|
||||
*,
|
||||
k: int = 60,
|
||||
) -> dict[str, float]:
|
||||
"""Fuse multiple ranked lists using Reciprocal Rank Fusion.
|
||||
|
||||
For each candidate present in any list, the fused score is:
|
||||
|
||||
score(c) = Σ_i 1 / (k + rank_i(c))
|
||||
|
||||
where rank_i(c) is the 1-based rank of candidate c in list i.
|
||||
|
||||
Candidates with zero BM25 score in ALL contributing lists receive
|
||||
a fused score of 0.0 (no token overlap signal).
|
||||
|
||||
Scores are normalized so the top-ranked candidate receives 1.0.
|
||||
|
||||
Args:
|
||||
ranked_lists: One or more ranked lists of (text, score) pairs.
|
||||
Each list MUST be sorted by score descending.
|
||||
Lists with all-zero scores are skipped (no signal).
|
||||
k: RRF smoothing constant (default 60, Cormack et al. 2009).
|
||||
Higher k flattens rank differences; lower k amplifies them.
|
||||
|
||||
Returns:
|
||||
Mapping of candidate text → normalized fused score in [0.0, 1.0].
|
||||
Returns empty dict if all input lists are empty or all-zero.
|
||||
"""
|
||||
...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## `auto_tagger.py` — Public Interface
|
||||
|
||||
```python
|
||||
class AutoTagger:
|
||||
"""LLM-based competence expander for BM25 pre-processing.
|
||||
|
||||
Calls an Azure OpenAI chat model to identify which required competences
|
||||
are already covered by the candidate's existing competences (via synonym,
|
||||
abbreviation, or cross-language equivalence) and returns the canonical
|
||||
required-competence names as additions.
|
||||
|
||||
All LLM calls use ``response_format={"type": "json_object"}`` to ensure
|
||||
parseable output. On any LLM or parsing error, the original candidate list
|
||||
is returned unchanged (graceful degradation).
|
||||
|
||||
Args:
|
||||
client: An ``AzureOpenAIClient`` instance configured with a chat
|
||||
deployment (``chat_deployment`` and ``llm_api_key``).
|
||||
"""
|
||||
|
||||
def __init__(self, client: AzureOpenAIClient) -> None: ...
|
||||
|
||||
def expand_competences(
|
||||
self,
|
||||
required: list[str],
|
||||
existing: list[str],
|
||||
) -> list[str]:
|
||||
"""Expand a candidate's competence list with covered required terms.
|
||||
|
||||
Sends a prompt to the LLM asking: given ``required`` competences and
|
||||
the candidate's ``existing`` competences, which required competences
|
||||
are already implicitly covered by the existing entries?
|
||||
|
||||
The returned list is ``existing + additions`` where ``additions``
|
||||
are the canonical required-competence names identified by the LLM.
|
||||
|
||||
The result is **ephemeral**: it is used only for the current BM25
|
||||
scoring call and is never written back to the database.
|
||||
|
||||
Args:
|
||||
required: List of required competence names (BM25 query terms).
|
||||
existing: Candidate's current competence list.
|
||||
|
||||
Returns:
|
||||
Combined list ``existing + additions``. If the LLM call fails or
|
||||
returns no additions, returns ``existing`` unchanged.
|
||||
"""
|
||||
...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
### `SimilarityConfig` (updated)
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class SimilarityConfig:
|
||||
"""Similarity engine configuration."""
|
||||
|
||||
embedding_model: str = "text-embedding-3-large"
|
||||
embedding_dimensions: int = 3072
|
||||
strategy: str = "per_skill"
|
||||
use_bm25_search: bool = False
|
||||
"""When True, BM25 + RRF replaces embedding-based competence similarity.
|
||||
Role similarity is always embedding-based.
|
||||
Default: False (preserves existing behavior).
|
||||
"""
|
||||
use_auto_tagging: bool = False
|
||||
"""When True (and use_bm25_search is also True), calls the LLM AutoTagger
|
||||
before each BM25 scoring call to expand the candidate's competence list
|
||||
with canonical equivalents of required competences it already covers.
|
||||
The expansion is ephemeral and never persisted.
|
||||
Default: False.
|
||||
"""
|
||||
```
|
||||
|
||||
### `AzureOpenAIConfig` (updated)
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class AzureOpenAIConfig:
|
||||
"""Azure OpenAI client configuration."""
|
||||
|
||||
# ... existing fields ...
|
||||
chat_deployment: str = ""
|
||||
"""Azure deployment name of the chat model used for auto-tagging
|
||||
(e.g. ``"gpt-4.1"``). Required only when ``use_auto_tagging = true``.
|
||||
"""
|
||||
llm_api_key: str = ""
|
||||
"""Azure OpenAI API key for the chat deployment. Read from the
|
||||
``AZURE_OPENAI_LLM_API_KEY`` environment variable.
|
||||
Required when ``use_auto_tagging = true``; raises ``ConfigurationError``
|
||||
at startup if absent.
|
||||
"""
|
||||
```
|
||||
|
||||
### Config parser (`config.toml` keys)
|
||||
|
||||
```toml
|
||||
[matching.similarity]
|
||||
strategy = "per_skill"
|
||||
use_bm25_search = false # new — enables BM25+RRF competence matching
|
||||
use_auto_tagging = false # new — enables LLM pre-expansion before BM25
|
||||
|
||||
[azure_openai]
|
||||
# ... existing keys ...
|
||||
chat_deployment = "gpt-4.1" # new — required when use_auto_tagging = true
|
||||
```
|
||||
|
||||
### Environment variables
|
||||
|
||||
| Variable | Required when | Description |
|
||||
|----------|--------------|-------------|
|
||||
| `AZURE_OPENAI_LLM_API_KEY` | `use_auto_tagging = true` | API key for the Azure OpenAI chat deployment used in auto-tagging. Server raises `ConfigurationError` at startup if absent and `use_auto_tagging = true`. |
|
||||
|
||||
---
|
||||
|
||||
## BM25 Score Behavior
|
||||
|
||||
### Example: Query "Python" against candidate competences
|
||||
|
||||
| Candidate | Tokens | BM25 Score | RRF Score (k=60) |
|
||||
|-----------|--------|------------|-----------------|
|
||||
| Python | ["python"] | ~3.5 | 1.000 |
|
||||
| Python (advanced) | ["python", "advanced"] | ~2.1 | 0.984 |
|
||||
| JavaScript | ["javascript"] | 0.0 | **0.000** |
|
||||
| TypeScript | ["typescript"] | 0.0 | **0.000** |
|
||||
| Node.js | ["node", "js"] | 0.0 | **0.000** |
|
||||
|
||||
Candidates without any shared token receive **0.0** — the false-positive problem is
|
||||
eliminated.
|
||||
|
||||
### Example: Query "Machine Learning" against candidate competences
|
||||
|
||||
| Candidate | Tokens | BM25 Score | RRF Score |
|
||||
|-----------|--------|------------|-----------|
|
||||
| Machine Learning | ["machine", "learning"] | ~4.0 | 1.000 |
|
||||
| Deep Learning | ["deep", "learning"] | ~1.8 | 0.984 |
|
||||
| Machine Learning (Python) | ["machine", "learning", "python"] | ~3.2 | 0.492 |
|
||||
| JavaScript | ["javascript"] | 0.0 | **0.000** |
|
||||
|
||||
---
|
||||
|
||||
## Known Limitations
|
||||
|
||||
### 1. Conceptual synonyms with zero token overlap score 0.0
|
||||
|
||||
BM25 requires at least one shared token between query and document. Skill pairs that are
|
||||
semantically equivalent but lexically disjoint will always score 0.0:
|
||||
|
||||
| Required competence | Candidate competence | Shared tokens | BM25 score |
|
||||
|---------------------|----------------------|---------------|------------|
|
||||
| Data Science | Machine Learning | none | **0.000** |
|
||||
| Data Science | Artificial Intelligence | none | **0.000** |
|
||||
| Frontend Development | UI Engineering | none | **0.000** |
|
||||
| Agile | Scrum | none | **0.000** |
|
||||
| Softwarearchitektur | Software Architecture | none | **0.000** |
|
||||
|
||||
These are **false negatives** — real skill matches that BM25 misses entirely. Embeddings
|
||||
handle all of these correctly via semantic similarity.
|
||||
|
||||
**Consequence:** BM25 mode is best suited to datasets where competence names are
|
||||
**canonical, consistent, and in a single language**. If profiles mix German and English
|
||||
skill names, or use informal abbreviations ("ML" for "Machine Learning"), BM25 will
|
||||
produce false negatives that embeddings would have caught.
|
||||
|
||||
**Mitigation:** Enable `use_auto_tagging = true`. The LLM will bridge synonym and
|
||||
cross-language gaps by pre-expanding the candidate list before BM25 scoring. This
|
||||
addresses limitations 1, 2, and 3 at the cost of one LLM call per candidate per scoring
|
||||
invocation.
|
||||
|
||||
---
|
||||
|
||||
### 2. Abbreviations and acronyms
|
||||
|
||||
Short forms that do not share tokens with the expanded form score 0.0:
|
||||
|
||||
| Required | Candidate | BM25 score |
|
||||
|----------|-----------|------------|
|
||||
| Machine Learning | ML | **0.000** |
|
||||
| Continuous Integration | CI | **0.000** |
|
||||
| Natural Language Processing | NLP | **0.000** |
|
||||
|
||||
Partial exception: "CI/CD" tokenizes to `["ci", "cd"]`, so a query "CI/CD Pipeline"
|
||||
would find partial overlap with a candidate "CI/CD", but not with "Continuous
|
||||
Integration/Continuous Delivery".
|
||||
|
||||
**Mitigation:** `use_auto_tagging = true` (see limitation 1).
|
||||
|
||||
---
|
||||
|
||||
### 3. Cross-language skill names
|
||||
|
||||
A required competence in German and a candidate competence in English (or vice versa)
|
||||
score 0.0 even when they describe the same skill:
|
||||
|
||||
| Required | Candidate | BM25 score |
|
||||
|----------|-----------|------------|
|
||||
| Softwarearchitektur | Software Architecture | **0.000** |
|
||||
| Maschinelles Lernen | Machine Learning | **0.000** |
|
||||
| Datenwissenschaft | Data Science | **0.000** |
|
||||
|
||||
This is a significant limitation for German-language skill databases. Embeddings handle
|
||||
cross-language pairs well; BM25 does not.
|
||||
|
||||
**Mitigation:** `use_auto_tagging = true` (see limitation 1).
|
||||
|
||||
---
|
||||
|
||||
### 4. RRF score compression near 1.0
|
||||
|
||||
Because the normalization formula is `(k+1) / (k+rank)` with `k=60`, ranks 1–5 all
|
||||
compress into the range `[0.871, 1.0]`. There is little score differentiation between
|
||||
the top-ranked candidates. This is intentional (rank differences matter less than
|
||||
presence/absence of token overlap) but means BM25 mode should not be used to produce
|
||||
fine-grained competence similarity scores — only to separate matching (score > 0) from
|
||||
non-matching (score = 0) candidates.
|
||||
|
||||
---
|
||||
|
||||
### 5. Small per-person corpus causes IDF pathology — mitigated by global index
|
||||
|
||||
BM25's IDF term rewards tokens that are rare across the corpus. When the BM25 index was
|
||||
built per candidate (N = that person's competence count ≈ 5–30), any term appearing in
|
||||
most documents had IDF ≤ 0, causing `rank_bm25` to return negative raw scores. After
|
||||
`max(0.0, raw)` clamping, all candidates scored 0.0 — making the BM25 signal useless.
|
||||
|
||||
**Fix (implemented):** The `Bm25Index` is now built once over the **union of all filtered
|
||||
candidates' competences** in `Matcher.match()` before the per-candidate loop (see Key
|
||||
Decision #2). This makes N equal to the total unique competence count across all
|
||||
candidates, giving IDF proper signal. Per-candidate scoring filters the global results to
|
||||
only that candidate's competence set.
|
||||
|
||||
**Residual limitation:** If the filtered candidate pool itself is very small (e.g., 2–3
|
||||
candidates with few unique competences), IDF may still be weak. In practice this is
|
||||
negligible since matching runs are always invoked against a meaningful pool of candidates.
|
||||
|
||||
---
|
||||
|
||||
## Test Strategy
|
||||
|
||||
### Unit tests
|
||||
|
||||
1. **`tests/test_bm25.py`**
|
||||
- `Bm25Index` with empty corpus → all scores 0.0
|
||||
- Exact match → score > 0
|
||||
- No token overlap → score == 0.0
|
||||
- Partial token overlap (multi-token queries) → partial score
|
||||
- Case-insensitive matching ("Python" == "python")
|
||||
- Tokenization edge cases: hyphens, parentheses, slashes
|
||||
|
||||
2. **`tests/test_rrf.py`**
|
||||
- Single list, single candidate → score 1.0
|
||||
- Single list, multiple candidates → scores normalized to [0, 1]
|
||||
- All-zero list → returns empty / all zeros
|
||||
- `k` parameter effect on score distribution
|
||||
|
||||
3. **`tests/test_auto_tagger.py`** (new)
|
||||
- LLM returns valid additions → `expand_competences()` appends them to existing list
|
||||
- LLM returns empty additions → original list returned unchanged
|
||||
- LLM call raises exception → original list returned unchanged (graceful degradation)
|
||||
- LLM returns malformed JSON → original list returned unchanged
|
||||
- Additions from LLM that are NOT in `required` are ignored
|
||||
- `expand_competences()` never duplicates entries already in `existing`
|
||||
- All tests use a mocked `AzureOpenAIClient` (no real LLM calls)
|
||||
|
||||
4. **`tests/test_similarity.py`** (extended)
|
||||
- `compute_competence_similarity()` with `use_bm25_search=True`
|
||||
- Required "Python", candidate ["JavaScript"] → score 0.0
|
||||
- Required "Python", candidate ["Python"] → score 1.0
|
||||
- Required "Python" + "Machine Learning", candidate ["JavaScript", "TypeScript"]
|
||||
→ both scores 0.0, overall competence_score 0.0
|
||||
- With `use_bm25_search=True, use_auto_tagging=True`: mocked `AutoTagger` injects
|
||||
"Machine Learning" into candidate ["ML"] → score > 0
|
||||
|
||||
### Integration test
|
||||
|
||||
5. **`tests/test_matcher_integration.py`** (extended)
|
||||
- With `use_bm25_search=True`: candidate with exact skills scores higher than
|
||||
candidate with only semantically adjacent skills.
|
||||
- The false-positive scenario (Python/ML vs JavaScript/TypeScript) is explicitly
|
||||
tested.
|
||||
- With `use_bm25_search=True, use_auto_tagging=True` (mocked LLM): candidate with
|
||||
"ML" matches required "Machine Learning" after auto-tag expansion.
|
||||
@@ -0,0 +1,288 @@
|
||||
# Change Proposal: BM25 + RRF Competence Matching
|
||||
|
||||
- **Change ID**: `add-bm25-rrf-competence-matching`
|
||||
- **Status**: Proposed
|
||||
- **Target**: `teamlandkarte-mcp`
|
||||
- **Author**: Thomas Handke
|
||||
- **Date**: 2026-02-26
|
||||
|
||||
## Summary
|
||||
|
||||
Add an optional BM25-based competence matching strategy as an alternative to the existing
|
||||
embedding-based similarity approach. When enabled, the following pipeline replaces
|
||||
embedding-based competence similarity:
|
||||
|
||||
1. **Auto-Tagging (LLM):** Before BM25 scoring, an LLM compares the required competences
|
||||
against each candidate's existing competences and suggests canonical additions
|
||||
(synonyms, abbreviations, cross-language equivalents). The expanded competence list
|
||||
is ephemeral — it is used only for the current scoring call and is never persisted.
|
||||
2. **BM25 Ranking:** The (expanded) candidate competences are indexed with BM25. Each
|
||||
required competence is used as a query to rank the candidates.
|
||||
3. **RRF Fusion:** Reciprocal Rank Fusion normalizes the BM25 ranked list into a score
|
||||
in [0, 1] per required competence.
|
||||
|
||||
The feature is controlled by two new booleans in `config.toml`:
|
||||
- `use_bm25_search` (default `false`): enables BM25 + RRF.
|
||||
- `use_auto_tagging` (default `false`): enables LLM-based competence expansion before
|
||||
BM25 scoring. Strongly recommended when `use_bm25_search = true`.
|
||||
|
||||
## Why
|
||||
|
||||
The current embedding-based (`per_skill` / `aggregate`) strategy has a known weakness:
|
||||
it assigns non-trivial scores to candidates whose competences are only _semantically
|
||||
adjacent_ to the required skills but not actually matching.
|
||||
|
||||
**Example (observed in production):**
|
||||
Searching for **Python** + **Machine Learning**, a candidate with only
|
||||
**JavaScript, TypeScript, Node.js, Vue.js, Web Architekturen** received a score of **0.392**
|
||||
because the embedding model considers all programming languages semantically related.
|
||||
|
||||
BM25 (Best Match 25) is a classical lexical ranking algorithm that scores documents by
|
||||
**term overlap** between query and document. It is immune to the semantic drift problem
|
||||
because it only counts tokens that literally occur in the candidate text. BM25 will assign
|
||||
**0** to "JavaScript" when the query is "Python".
|
||||
|
||||
### When BM25 outperforms embeddings
|
||||
|
||||
| Scenario | Embeddings | BM25 |
|
||||
|----------|-----------|------|
|
||||
| Query "Python", candidate has "Python" | High ✓ | High ✓ |
|
||||
| Query "Python", candidate has "JavaScript" | Medium ✗ | Zero ✓ |
|
||||
| Query "Machine Learning", candidate has "ML" | High ✓ | Low ✗ |
|
||||
| Query "Softwarearchitektur", candidate has "software architecture" | High ✓ | Low ✗ |
|
||||
| Query "React", candidate has "React.js" | High ✓ | Medium ~ |
|
||||
|
||||
BM25 excels at exact or near-exact skill names. Embeddings excel at synonyms and
|
||||
cross-language variants (e.g., German/English). The optional `use_auto_tagging` feature
|
||||
bridges BM25's synonym and cross-language gap by having an LLM pre-expand the candidate
|
||||
competence list — making the combination of BM25 + Auto-Tagging competitive with
|
||||
embeddings while still eliminating false positives from semantic drift.
|
||||
|
||||
### Why RRF for fusion
|
||||
|
||||
Reciprocal Rank Fusion merges multiple ranked lists into one without requiring score
|
||||
normalization. For each required competence query, we obtain a ranked list of candidate
|
||||
competences. RRF then assigns each candidate a fused score based on its (1-based) rank in
|
||||
one or more lists.
|
||||
|
||||
RRF fused score formula (Cormack et al. 2009):
|
||||
|
||||
```
|
||||
rrf_score(candidate) = Σ_i 1 / (k + rank_i(candidate))
|
||||
```
|
||||
|
||||
RRF is robust to score scale differences and has been widely validated in information
|
||||
retrieval research as a high-quality fusion method.
|
||||
|
||||
## What Changes
|
||||
|
||||
### 1. New config parameter `matching.similarity.use_bm25_search` (boolean, default `false`)
|
||||
|
||||
- When `false` (default): existing embedding-based strategy is used unchanged.
|
||||
- When `true`: BM25 + RRF replaces embedding-based competence similarity.
|
||||
Role similarity is **always** embedding-based (unchanged).
|
||||
|
||||
### 2. New config parameter `matching.similarity.use_auto_tagging` (boolean, default `false`)
|
||||
|
||||
- Only applies when `use_bm25_search = true`. When `false`, BM25 runs on the raw
|
||||
candidate competence list (no LLM expansion).
|
||||
- When `true`: before each BM25 scoring call, an LLM expands the candidate's competence
|
||||
list with canonical forms of any required competences that are already covered by the
|
||||
candidate's existing entries (via synonym, abbreviation, or cross-language equivalence).
|
||||
- The expanded list is ephemeral — it is not written back to the database and does not
|
||||
affect any other operation.
|
||||
|
||||
### 3. New config parameter `azure_openai.chat_deployment` (string)
|
||||
|
||||
- Azure deployment name of the chat model used for auto-tagging (e.g. `"gpt-4.1"`).
|
||||
- Required only when `use_auto_tagging = true`.
|
||||
- API key is read from environment variable `AZURE_OPENAI_LLM_API_KEY`.
|
||||
|
||||
### 4. `AzureOpenAIClient` extended: LLM chat capability restored
|
||||
|
||||
- Add `chat_completion(system: str, user: str) -> str` method supporting structured JSON
|
||||
responses via Azure OpenAI chat completions API.
|
||||
- Add `chat_deployment` and `llm_api_key` constructor parameters.
|
||||
- This restores LLM capability removed in a previous change, now scoped exclusively
|
||||
to auto-tagging.
|
||||
|
||||
### 5. New module `src/teamlandkarte_mcp/matching/auto_tagger.py`
|
||||
|
||||
Implements:
|
||||
- `AutoTagger`: class that wraps the LLM client and executes the tagging prompt.
|
||||
- `expand_competences()`: given `required: list[str]` and `existing: list[str]`,
|
||||
returns `list[str]` — the combined list `existing + additions`, where `additions`
|
||||
are the canonical required-competence names that the LLM identifies as already
|
||||
covered by `existing` entries.
|
||||
- Structured JSON output schema enforced via the Azure OpenAI `response_format` parameter.
|
||||
- Google Docstrings on all public symbols.
|
||||
|
||||
### 6. New module `src/teamlandkarte_mcp/matching/bm25.py`
|
||||
|
||||
Implements:
|
||||
- `Bm25Index`: builds an in-memory BM25 index over a corpus of competence strings.
|
||||
Designed to be instantiated once over the global candidate pool (see item 8a below)
|
||||
so IDF values are meaningful. Raw scores are clamped to `max(0.0, raw)` — terms with
|
||||
negative IDF produce 0.0 rather than a spurious tiny positive value.
|
||||
- `bm25_rank_competences()`: for a single required competence query, returns a ranked list
|
||||
of `(candidate_text, score)` tuples. Convenience wrapper for standalone/test use.
|
||||
- Pure Python, no external runtime dependencies beyond `rank_bm25` from PyPI.
|
||||
|
||||
### 7. New module `src/teamlandkarte_mcp/matching/rrf.py`
|
||||
|
||||
Implements:
|
||||
- `reciprocal_rank_fusion()`: merges one or more ranked lists into a final score dict.
|
||||
- `k` constant (default `60`, standard RRF value from Cormack et al. 2009).
|
||||
- Returns `dict[str, float]` mapping candidate text → fused score in `(0, 1]`
|
||||
(top-ranked candidate receives exactly `1.0`).
|
||||
|
||||
### 8. `SimilarityEngine.compute_competence_similarity()` extended
|
||||
|
||||
- Reads `use_bm25_search` and `use_auto_tagging` flags.
|
||||
- Accepts an optional `global_index: Bm25Index | None` parameter (passed in by `Matcher`).
|
||||
- When `use_bm25_search=True`:
|
||||
1. If `use_auto_tagging=True`: calls `AutoTagger.expand_competences()` to build the
|
||||
expanded candidate list.
|
||||
2. Delegates to `_bm25_rrf_similarity()` with the (optionally expanded) candidate list
|
||||
and the global index.
|
||||
- `_bm25_rrf_similarity()` uses `global_index.rank(req)` filtered to the candidate's
|
||||
competence set when `global_index` is provided, otherwise falls back to
|
||||
`bm25_rank_competences(req, candidate)`.
|
||||
- Output shape is **identical** to existing strategies:
|
||||
`dict[required_competence → {score, best_match, rationale}]`
|
||||
- Preserves the `Matcher` and scorer contracts unchanged.
|
||||
|
||||
### 8a. `Matcher.match()` builds a global `Bm25Index` before the per-candidate loop
|
||||
|
||||
When `use_bm25_search=True` and the filtered candidate pool is non-empty, `Matcher.match()`
|
||||
collects the deduplicated union of all filtered candidates' competences and builds one
|
||||
`Bm25Index` over that global corpus before the loop. The same index is passed to every
|
||||
`compute_competence_similarity()` call.
|
||||
|
||||
**Why this matters:** With a per-person corpus (N ≈ 5–30 entries), BM25 IDF is 0 or
|
||||
negative for every term — all raw scores collapse to ≤ 0, and `max(0.0, raw)` clamping
|
||||
makes every match appear as 0.0. A global corpus ensures N is large enough for IDF to
|
||||
carry real signal.
|
||||
|
||||
### 9. `SimilarityConfig` dataclass extended
|
||||
|
||||
New fields:
|
||||
- `use_bm25_search: bool = False`
|
||||
- `use_auto_tagging: bool = False`
|
||||
|
||||
### 10. `AzureOpenAIConfig` dataclass extended
|
||||
|
||||
New fields:
|
||||
- `chat_deployment: str = ""`
|
||||
- (API key via `AZURE_OPENAI_LLM_API_KEY` env var)
|
||||
|
||||
### 11. `config.py` parser extended
|
||||
|
||||
Reads `matching.similarity.use_bm25_search`, `matching.similarity.use_auto_tagging`,
|
||||
and `azure_openai.chat_deployment` from TOML. Reads `AZURE_OPENAI_LLM_API_KEY` from
|
||||
environment when `use_auto_tagging = true`.
|
||||
|
||||
### 12. `config.toml.example` updated
|
||||
|
||||
Documents all three new keys with inline comments.
|
||||
|
||||
### 13. Server initialization in `mcp_server.py`
|
||||
|
||||
- Passes `use_bm25_search` and `use_auto_tagging` flags when constructing `SimilarityEngine`.
|
||||
- When `use_auto_tagging=True`: constructs `AutoTagger` with the LLM-capable
|
||||
`AzureOpenAIClient` and passes it to `SimilarityEngine`.
|
||||
|
||||
### 14. Documentation updated
|
||||
|
||||
- `docs/architecture.md`: new section on BM25/RRF/Auto-Tagging strategy.
|
||||
- `docs/semantic_similarity_tradeoffs.md`: comparison table updated with BM25+AutoTag entry.
|
||||
- `docs/assistant_system_prompt.md`: notes on score interpretation with BM25 and auto-tagging.
|
||||
- `README.md`: all new config parameters documented.
|
||||
|
||||
### 15. Legacy/deprecated code removed
|
||||
|
||||
- Remove `debug_similarity_analysis.py`
|
||||
- Remove `debug_categorization.py`
|
||||
- Remove `SEMANTIC_SIMILARITY_ANALYSIS.md`
|
||||
- Remove `SCORE_BUG_FIX.md`
|
||||
|
||||
### 16. Tests
|
||||
|
||||
- `tests/test_bm25.py`: unit tests for `Bm25Index` and `bm25_rank_competences()`.
|
||||
- `tests/test_rrf.py`: unit tests for `reciprocal_rank_fusion()`.
|
||||
- `tests/test_auto_tagger.py`: unit tests for `AutoTagger.expand_competences()` (mocked LLM).
|
||||
- `tests/test_similarity.py`: extend with BM25/RRF and auto-tagging strategy tests.
|
||||
- `tests/test_matcher_integration.py`: integration test for the full pipeline
|
||||
(auto-tag → BM25 → RRF) in both matching directions.
|
||||
|
||||
## Impact
|
||||
|
||||
### Affected specs
|
||||
|
||||
- `matching-tools` (competence scoring path)
|
||||
|
||||
### Affected code
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `src/teamlandkarte_mcp/matching/similarity.py` | Add `_bm25_rrf_similarity()`, `global_index` param, auto-tag branch, read flags |
|
||||
| `src/teamlandkarte_mcp/matching/matcher.py` | Build global `Bm25Index` before per-candidate loop; pass as `global_index` |
|
||||
| `src/teamlandkarte_mcp/matching/bm25.py` | **New** |
|
||||
| `src/teamlandkarte_mcp/matching/rrf.py` | **New** |
|
||||
| `src/teamlandkarte_mcp/matching/auto_tagger.py` | **New** |
|
||||
| `src/teamlandkarte_mcp/azure/openai_client.py` | Restore `chat_completion()`; add `chat_deployment` + `llm_api_key` params |
|
||||
| `src/teamlandkarte_mcp/config.py` | `SimilarityConfig.use_bm25_search`, `use_auto_tagging`; `AzureOpenAIConfig.chat_deployment` |
|
||||
| `src/teamlandkarte_mcp/mcp_server.py` | Pass flags + `AutoTagger` to `SimilarityEngine` |
|
||||
| `config.toml.example` | Document all new keys |
|
||||
| `docs/architecture.md` | BM25/RRF/Auto-Tagging section |
|
||||
| `docs/semantic_similarity_tradeoffs.md` | Updated comparison with BM25+AutoTag entry |
|
||||
| `docs/assistant_system_prompt.md` | Score notes for BM25 and auto-tagging modes |
|
||||
| `README.md` | All new config parameters |
|
||||
| `tests/test_bm25.py` | **New** |
|
||||
| `tests/test_rrf.py` | **New** |
|
||||
| `tests/test_auto_tagger.py` | **New** |
|
||||
| `tests/test_similarity.py` | Extended |
|
||||
| `tests/test_matcher_integration.py` | Extended |
|
||||
| `debug_similarity_analysis.py` | **Removed** |
|
||||
| `debug_categorization.py` | **Removed** |
|
||||
| `SEMANTIC_SIMILARITY_ANALYSIS.md` | **Removed** |
|
||||
| `SCORE_BUG_FIX.md` | **Removed** |
|
||||
|
||||
### Breaking changes
|
||||
|
||||
None. The `use_bm25_search = false` and `use_auto_tagging = false` defaults preserve all
|
||||
existing behavior. The `overall_score` computation (weighted mean of `role_score` and
|
||||
`competence_score`) is unchanged.
|
||||
|
||||
### New environment variables
|
||||
|
||||
| Variable | Required when | Description |
|
||||
|----------|--------------|-------------|
|
||||
| `AZURE_OPENAI_LLM_API_KEY` | `use_auto_tagging = true` | Azure OpenAI API key for the chat model used in auto-tagging. The server raises a configuration error at startup if this key is absent and `use_auto_tagging = true`. |
|
||||
|
||||
### New dependencies
|
||||
|
||||
- `rank-bm25` (PyPI): pure-Python BM25 implementation, no native code, no external services.
|
||||
Adds to `pyproject.toml` `dependencies`.
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
### A. Hard-code exact-match bonus in per_skill
|
||||
|
||||
Simpler but not principled: would require special-casing the scorer instead of a clean
|
||||
strategy abstraction.
|
||||
|
||||
### B. Hybrid embedding + BM25 in a single `_bm25_rrf_similarity()` call
|
||||
|
||||
Combining both signals via RRF is architecturally clean and naturally extensible. However,
|
||||
to keep this change focused and reviewable, the initial implementation uses BM25-only RRF.
|
||||
A hybrid mode can be added as a follow-up change.
|
||||
|
||||
### C. Replace embeddings entirely
|
||||
|
||||
Not desirable. Embeddings are essential for role inference and handle multilingual /
|
||||
synonymous skill names (e.g., "Softwarearchitektur" ↔ "Software Architecture").
|
||||
|
||||
## Open Questions
|
||||
|
||||
None. The design is sufficiently clear to proceed.
|
||||
+266
@@ -0,0 +1,266 @@
|
||||
# Spec Delta: BM25 + RRF Competence Matching
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: BM25 + RRF optional competence matching strategy
|
||||
|
||||
The system SHALL support an optional BM25-based competence matching strategy controlled
|
||||
by `matching.similarity.use_bm25_search` in `config.toml`. When enabled, BM25 + RRF
|
||||
replaces embedding-based competence similarity for `find_matching_capacities` and
|
||||
`find_matching_tasks`. Role similarity SHALL always remain embedding-based.
|
||||
|
||||
#### Scenario: Candidate without matching skills receives zero competence score
|
||||
|
||||
- **GIVEN** `use_bm25_search = true` in `config.toml`
|
||||
- **AND** a task requires competences `["Python", "Machine Learning"]`
|
||||
- **AND** a candidate has competences `["JavaScript", "TypeScript", "Node.js", "Vue.js"]`
|
||||
- **WHEN** the system computes `competence_score` for that candidate
|
||||
- **THEN** `competence_score == 0.0`
|
||||
- **AND** the candidate is categorized as "Irrelevant" or "Low" based on `overall_score`
|
||||
|
||||
#### Scenario: Candidate with exact matching skills receives high competence score
|
||||
|
||||
- **GIVEN** `use_bm25_search = true` in `config.toml`
|
||||
- **AND** a task requires competences `["Python", "Machine Learning"]`
|
||||
- **AND** a candidate has competences `["Python", "Machine Learning", "Pandas"]`
|
||||
- **WHEN** the system computes `competence_score` for that candidate
|
||||
- **THEN** `competence_score > 0.8`
|
||||
- **AND** the candidate scores significantly higher than a candidate with no token overlap
|
||||
|
||||
#### Scenario: BM25 mode is disabled by default
|
||||
|
||||
- **GIVEN** no `use_bm25_search` key in `config.toml` (or `use_bm25_search = false`)
|
||||
- **WHEN** the system computes competence similarity
|
||||
- **THEN** the existing embedding-based strategy is used
|
||||
- **AND** behavior is bit-for-bit identical to the pre-change implementation
|
||||
|
||||
### Requirement: BM25 index ranks candidates by token overlap per required competence
|
||||
|
||||
The `Bm25Index` SHALL build an in-memory BM25 index over a corpus of competence strings
|
||||
and rank them for each required competence query. The index is built once over the global
|
||||
candidate pool per matching run (see "Global BM25 index" requirement below) and is queried
|
||||
per candidate with results filtered to that candidate's competence set.
|
||||
|
||||
#### Scenario: BM25 index ranks exact token match highest
|
||||
|
||||
- **GIVEN** a candidate corpus `["Python", "JavaScript", "TypeScript"]`
|
||||
- **AND** query `"Python"`
|
||||
- **WHEN** the `Bm25Index.rank()` method is called
|
||||
- **THEN** `"Python"` is ranked first with the highest BM25 score
|
||||
- **AND** `"JavaScript"` and `"TypeScript"` have BM25 score 0.0 (no shared tokens)
|
||||
|
||||
#### Scenario: BM25 index handles empty corpus without error
|
||||
|
||||
- **GIVEN** a candidate with no competences (empty corpus)
|
||||
- **WHEN** the system computes `competence_score`
|
||||
- **THEN** `competence_score == 0.0` for every required competence
|
||||
- **AND** no exception is raised
|
||||
|
||||
#### Scenario: Tokenization is case-insensitive and splits on non-word characters
|
||||
|
||||
- **GIVEN** a candidate corpus `["Progressive Web App (PWA)", "CI/CD Pipeline"]`
|
||||
- **AND** query `"ci cd"`
|
||||
- **WHEN** the `Bm25Index.rank()` method is called
|
||||
- **THEN** `"CI/CD Pipeline"` receives a score > 0
|
||||
- **AND** `"Progressive Web App (PWA)"` receives score 0.0
|
||||
|
||||
### Requirement: Reciprocal Rank Fusion normalizes BM25 ranks into scores in [0, 1]
|
||||
|
||||
The `reciprocal_rank_fusion()` function SHALL fuse ranked lists into a normalized score
|
||||
dict. Candidates with zero BM25 score in all contributing lists SHALL receive a fused
|
||||
score of 0.0. The top-ranked candidate SHALL receive score 1.0.
|
||||
|
||||
#### Scenario: Single-list fusion normalizes top candidate to 1.0
|
||||
|
||||
- **GIVEN** a single ranked list `[("Python", 3.5), ("Python (advanced)", 2.1)]`
|
||||
- **WHEN** `reciprocal_rank_fusion([ranked_list], k=60)` is called
|
||||
- **THEN** `"Python"` receives normalized score `1.0`
|
||||
- **AND** `"Python (advanced)"` receives a score in `(0, 1)`
|
||||
|
||||
#### Scenario: All-zero BM25 list produces empty result
|
||||
|
||||
- **GIVEN** a ranked list `[("JavaScript", 0.0), ("TypeScript", 0.0)]`
|
||||
- **WHEN** `reciprocal_rank_fusion([ranked_list])` is called
|
||||
- **THEN** the function returns an empty dict `{}`
|
||||
- **AND** the caller maps this to `score = 0.0` and `best_match = None`
|
||||
|
||||
### Requirement: Global BM25 index is built once over all filtered candidates
|
||||
|
||||
When `use_bm25_search = true`, the `Matcher` SHALL build one `Bm25Index` over the
|
||||
deduplicated union of all filtered candidates' competences before the per-candidate
|
||||
scoring loop. The same index SHALL be passed into every `compute_competence_similarity()`
|
||||
call and results SHALL be filtered to the current candidate's competence set. This ensures
|
||||
BM25 IDF values reflect the full candidate pool rather than a single person's small corpus.
|
||||
|
||||
#### Scenario: Candidate with unique skill scores above zero in global index
|
||||
|
||||
- **GIVEN** `use_bm25_search = true` in `config.toml`
|
||||
- **AND** two candidates: one with `["Python"]` only, another with 6 unrelated skills
|
||||
- **AND** the task requires `["Python"]`
|
||||
- **WHEN** the system computes `competence_score` for each candidate
|
||||
- **THEN** the candidate with `["Python"]` receives `competence_score > 0.0`
|
||||
- **AND** the candidate with no Python token receives `competence_score == 0.0`
|
||||
|
||||
#### Scenario: Ubiquitous term present in every candidate scores 0.0
|
||||
|
||||
- **GIVEN** a global corpus where a token appears in every candidate's competences
|
||||
- **WHEN** `Bm25Index.rank()` is called with that token as the query
|
||||
- **THEN** the raw BM25 score is clamped to `max(0.0, raw)` (IDF is negative → score is 0.0)
|
||||
- **AND** no exception is raised
|
||||
|
||||
#### Scenario: Global index filters results to current candidate's competence set
|
||||
|
||||
- **GIVEN** a global index built over candidates A (`["Python", "Django"]`) and B (`["Java", "Spring"]`)
|
||||
- **AND** the system scores candidate A for required `["Python"]`
|
||||
- **WHEN** `_bm25_rrf_similarity()` is called with candidate A's competence set
|
||||
- **THEN** only `"Python"` and `"Django"` appear in the filtered results
|
||||
- **AND** `"Java"` and `"Spring"` do NOT appear in candidate A's result
|
||||
|
||||
### Requirement: Optional LLM-based Auto-Tagging expands candidate competences before BM25
|
||||
|
||||
When `use_auto_tagging = true` (and `use_bm25_search = true`), the system SHALL call the
|
||||
`AutoTagger` before each BM25 scoring call to identify which required competences are
|
||||
already covered by the candidate's existing entries via synonym, abbreviation, or
|
||||
cross-language equivalence. The LLM returns canonical required-competence names as
|
||||
additions. The expanded list is ephemeral — it SHALL be used only for the current BM25
|
||||
scoring call and SHALL NOT be written back to any database or persisted between calls.
|
||||
|
||||
#### Scenario: Auto-tagging bridges synonym gap so candidate matches required competence
|
||||
|
||||
- **GIVEN** `use_bm25_search = true` and `use_auto_tagging = true` in `config.toml`
|
||||
- **AND** a task requires competences `["Machine Learning"]`
|
||||
- **AND** a candidate has competences `["ML", "Python"]`
|
||||
- **AND** the LLM identifies `"ML"` as covering `"Machine Learning"`
|
||||
- **WHEN** the system computes `competence_score`
|
||||
- **THEN** `competence_score > 0.0` (auto-tagging added `"Machine Learning"` to working list)
|
||||
- **AND** the candidate's stored competences remain `["ML", "Python"]` (not mutated)
|
||||
|
||||
#### Scenario: Auto-tagging bridges cross-language gap
|
||||
|
||||
- **GIVEN** `use_bm25_search = true` and `use_auto_tagging = true` in `config.toml`
|
||||
- **AND** a task requires competences `["Software Architecture"]`
|
||||
- **AND** a candidate has competences `["Softwarearchitektur"]`
|
||||
- **AND** the LLM identifies `"Softwarearchitektur"` as covering `"Software Architecture"`
|
||||
- **WHEN** the system computes `competence_score`
|
||||
- **THEN** `competence_score > 0.0`
|
||||
|
||||
#### Scenario: Auto-tagging graceful degradation on LLM failure
|
||||
|
||||
- **GIVEN** `use_bm25_search = true` and `use_auto_tagging = true` in `config.toml`
|
||||
- **AND** the LLM call raises an exception (timeout, API error, or malformed JSON)
|
||||
- **WHEN** the system computes `competence_score`
|
||||
- **THEN** no exception is propagated to the caller
|
||||
- **AND** BM25 scoring continues on the original (unexpanded) candidate list
|
||||
- **AND** a warning is logged
|
||||
|
||||
#### Scenario: Auto-tagging LLM cannot invent skills not in required list
|
||||
|
||||
- **GIVEN** `use_auto_tagging = true`
|
||||
- **AND** the LLM response includes an addition that is NOT in the `required` list
|
||||
- **WHEN** `AutoTagger.expand_competences()` processes the response
|
||||
- **THEN** the hallucinated addition is silently dropped
|
||||
- **AND** only valid required-competence names are appended to the working list
|
||||
|
||||
#### Scenario: Auto-tagging is no-op when use_auto_tagging is false
|
||||
|
||||
- **GIVEN** `use_bm25_search = true` and `use_auto_tagging = false` (default) in `config.toml`
|
||||
- **WHEN** the system computes `competence_score`
|
||||
- **THEN** no LLM call is made
|
||||
- **AND** BM25 runs on the raw candidate competence list without expansion
|
||||
|
||||
### Requirement: `use_auto_tagging` configuration key with default false
|
||||
|
||||
`config.toml` SHALL support a `matching.similarity.use_auto_tagging` boolean key.
|
||||
The default value SHALL be `false`. The key SHALL be documented in `config.toml.example`.
|
||||
`use_auto_tagging = true` is only effective when `use_bm25_search = true`.
|
||||
|
||||
#### Scenario: Config key defaults to false when omitted
|
||||
|
||||
- **GIVEN** a `config.toml` that does not contain `use_auto_tagging`
|
||||
- **WHEN** the server starts and loads configuration
|
||||
- **THEN** `SimilarityConfig.use_auto_tagging == False`
|
||||
- **AND** no `AutoTagger` is constructed
|
||||
- **AND** no LLM client is constructed for auto-tagging
|
||||
|
||||
#### Scenario: Config key enables auto-tagging when set to true
|
||||
|
||||
- **GIVEN** `config.toml` contains `use_auto_tagging = true` under `[matching.similarity]`
|
||||
- **AND** `chat_deployment` is set and `AZURE_OPENAI_LLM_API_KEY` is present
|
||||
- **WHEN** the server starts and loads configuration
|
||||
- **THEN** `SimilarityConfig.use_auto_tagging == True`
|
||||
- **AND** an `AutoTagger` is constructed and passed to `SimilarityEngine`
|
||||
|
||||
### Requirement: `azure_openai.chat_deployment` configuration key for auto-tagging
|
||||
|
||||
`config.toml` SHALL support an `azure_openai.chat_deployment` string key naming the Azure
|
||||
OpenAI chat model deployment used by the `AutoTagger`. This key is required when
|
||||
`use_auto_tagging = true`. The `AZURE_OPENAI_LLM_API_KEY` environment variable SHALL
|
||||
supply the API key for that deployment. The server SHALL raise a `ConfigurationError` at
|
||||
startup if `use_auto_tagging = true` and `AZURE_OPENAI_LLM_API_KEY` is absent.
|
||||
|
||||
#### Scenario: Server raises ConfigurationError when API key missing with auto-tagging enabled
|
||||
|
||||
- **GIVEN** `use_auto_tagging = true` in `config.toml`
|
||||
- **AND** the environment variable `AZURE_OPENAI_LLM_API_KEY` is not set
|
||||
- **WHEN** the server starts and loads configuration
|
||||
- **THEN** a `ConfigurationError` is raised before any MCP tool is registered
|
||||
- **AND** the error message indicates the missing environment variable
|
||||
|
||||
#### Scenario: chat_deployment key is read from config when auto-tagging enabled
|
||||
|
||||
- **GIVEN** `config.toml` contains `chat_deployment = "gpt-4.1"` under `[azure_openai]`
|
||||
- **AND** `use_auto_tagging = true` and `AZURE_OPENAI_LLM_API_KEY` is set
|
||||
- **WHEN** the server starts
|
||||
- **THEN** the `AzureOpenAIClient` for auto-tagging is initialized with deployment `"gpt-4.1"`
|
||||
|
||||
### Requirement: `use_bm25_search` configuration key with default false
|
||||
|
||||
`config.toml` SHALL support a `matching.similarity.use_bm25_search` boolean key.
|
||||
The default value SHALL be `false`. The key SHALL be documented in `config.toml.example`.
|
||||
|
||||
#### Scenario: Config key defaults to false when omitted
|
||||
|
||||
- **GIVEN** a `config.toml` that does not contain `use_bm25_search`
|
||||
- **WHEN** the server starts and loads configuration
|
||||
- **THEN** `SimilarityConfig.use_bm25_search == False`
|
||||
- **AND** the embedding-based strategy is used unchanged
|
||||
|
||||
#### Scenario: Config key enables BM25 mode when set to true
|
||||
|
||||
- **GIVEN** `config.toml` contains `use_bm25_search = true` under `[matching.similarity]`
|
||||
- **WHEN** the server starts and loads configuration
|
||||
- **THEN** `SimilarityConfig.use_bm25_search == True`
|
||||
- **AND** `compute_competence_similarity()` delegates to `_bm25_rrf_similarity()`
|
||||
|
||||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: Role similarity is always embedding-based regardless of use_bm25_search
|
||||
|
||||
Role similarity computation SHALL remain embedding-based even when `use_bm25_search = true`.
|
||||
The BM25 strategy applies only to competence similarity.
|
||||
|
||||
#### Scenario: Role scoring unaffected by use_bm25_search flag
|
||||
|
||||
- **GIVEN** `use_bm25_search = true` in `config.toml`
|
||||
- **AND** a task requiring role "Backend Developer"
|
||||
- **AND** a capacity with role "Backend Developer"
|
||||
- **WHEN** the system computes `role_score`
|
||||
- **THEN** `role_score` is computed via embedding similarity (unchanged behavior)
|
||||
- **AND** embedding cache and Azure OpenAI calls for role embeddings proceed as before
|
||||
|
||||
## REMOVED Requirements
|
||||
|
||||
### Requirement: Legacy debug scripts are removed from the repository root
|
||||
|
||||
The files `debug_similarity_analysis.py`, `debug_categorization.py`,
|
||||
`SEMANTIC_SIMILARITY_ANALYSIS.md`, and `SCORE_BUG_FIX.md` SHALL be removed from the
|
||||
repository root. They are superseded by the test suite and `docs/semantic_similarity_tradeoffs.md`.
|
||||
|
||||
#### Scenario: Legacy files do not exist after the change is applied
|
||||
|
||||
- **GIVEN** the change `add-bm25-rrf-competence-matching` is fully applied
|
||||
- **WHEN** the repository root is listed
|
||||
- **THEN** `debug_similarity_analysis.py` does not exist
|
||||
- **AND** `debug_categorization.py` does not exist
|
||||
- **AND** `SEMANTIC_SIMILARITY_ANALYSIS.md` does not exist
|
||||
- **AND** `SCORE_BUG_FIX.md` does not exist
|
||||
@@ -0,0 +1,586 @@
|
||||
# Implementation Tasks: BM25 + RRF Competence Matching
|
||||
|
||||
## Status: Implemented
|
||||
|
||||
## Summary
|
||||
|
||||
This change introduces an optional BM25-based competence matching strategy with optional
|
||||
LLM-based Auto-Tagging as an alternative to the existing embedding-based approach:
|
||||
|
||||
1. **Restore LLM client**: `AzureOpenAIClient.chat_completion()` restored; `AzureOpenAIConfig.chat_deployment` + `llm_api_key` added.
|
||||
2. **New `auto_tagger.py` module**: `AutoTagger` class with `expand_competences()` — ephemeral LLM-driven competence expansion.
|
||||
3. **New `bm25.py` module**: `Bm25Index` dataclass + `bm25_rank_competences()` convenience function.
|
||||
4. **New `rrf.py` module**: `reciprocal_rank_fusion()` fuses ranked lists into a normalized score dict.
|
||||
5. **`similarity.py` extended**: `_bm25_rrf_similarity()` internal method + auto-tag branch + flag routing in `compute_competence_similarity()`.
|
||||
6. **`SimilarityConfig` extended**: new `use_bm25_search: bool = False` and `use_auto_tagging: bool = False` fields.
|
||||
7. **`config.py` parser extended**: reads `use_bm25_search`, `use_auto_tagging`, `chat_deployment`, and `AZURE_OPENAI_LLM_API_KEY`.
|
||||
8. **`config.toml.example` updated**: documents all new keys.
|
||||
9. **`mcp_server.py` updated**: constructs `AutoTagger` when `use_auto_tagging=True`; passes flags to `SimilarityEngine`.
|
||||
10. **Legacy debug files removed**: 4 root-level files no longer part of the module.
|
||||
11. **Tests added**: `tests/test_bm25.py`, `tests/test_rrf.py`, `tests/test_auto_tagger.py`, extensions to `tests/test_similarity.py` and `tests/test_matcher_integration.py`.
|
||||
12. **Documentation updated**: architecture, tradeoffs doc, assistant prompt, README.
|
||||
|
||||
## Clarifications / Decisions
|
||||
|
||||
- **BM25 scope**: BM25 + RRF replaces **competence** similarity only. Role similarity is
|
||||
always embedding-based, regardless of `use_bm25_search`.
|
||||
- **Auto-tagging scope**: `use_auto_tagging` only applies when `use_bm25_search = true`.
|
||||
When `use_auto_tagging = true`, `AutoTagger.expand_competences()` is called once per
|
||||
candidate per `compute_competence_similarity()` invocation. Result is ephemeral.
|
||||
- **Auto-tagging works in both directions**: both `find_matching_capacities` and
|
||||
`find_matching_tasks` pass through `compute_competence_similarity()` and benefit.
|
||||
- **Graceful degradation**: if the LLM call in `AutoTagger` fails for any reason, the
|
||||
original candidate list is used unchanged. BM25 scoring continues.
|
||||
- **LLM client restored**: `AzureOpenAIClient.chat_completion()` is re-added, scoped
|
||||
exclusively to auto-tagging. The embedding-related API surface is unchanged.
|
||||
- **Zero-out rule**: If all BM25 scores for a required competence are 0.0 (no token
|
||||
overlap at all), the RRF score for that required competence is also 0.0.
|
||||
- **Global index** *(revised)*: Originally the `Bm25Index` was to be built fresh per
|
||||
`compute_competence_similarity()` call. This was identified as an IDF pathology: with
|
||||
only one person's competences as the corpus, BM25 IDF is always 0 or negative for any
|
||||
term that appears in every document (and with N=1 all terms do). The fix (Phase 9) builds
|
||||
one `Bm25Index` over the union of all filtered candidates' competences before the loop
|
||||
and passes it in as `global_index`.
|
||||
- **Single ranked list**: Initial implementation fuses one BM25 ranked list per required
|
||||
competence. The `reciprocal_rank_fusion()` signature accepts multiple lists for future
|
||||
hybrid extension.
|
||||
- **Output shape unchanged**: `_bm25_rrf_similarity()` returns the same
|
||||
`dict[str, dict[str, object]]` shape as `_per_skill_similarity()`. Matcher, scorer,
|
||||
and all output formatting remain unchanged.
|
||||
- **`rank-bm25` dependency**: Added to `pyproject.toml` `[project.dependencies]`.
|
||||
- **All new code uses Google Docstrings** per project convention.
|
||||
- **Default `false`**: Existing deployments are unaffected until opt-in.
|
||||
|
||||
## Estimated Effort
|
||||
|
||||
- **Phase 0 (Restore LLM client)**: 1 hour
|
||||
- **Phase 1 (New modules — bm25, rrf, auto_tagger)**: 3–4 hours
|
||||
- **Phase 2 (Config + wiring)**: 1–1.5 hours
|
||||
- **Phase 3 (Similarity integration)**: 2 hours
|
||||
- **Phase 4 (Legacy cleanup)**: 0.5 hour
|
||||
- **Phase 5 (Tests)**: 4–5 hours
|
||||
- **Phase 6 (Documentation)**: 2 hours
|
||||
- **Phase 7 (Dependency)**: 0.5 hour
|
||||
- **Phase 8 (Quality gates)**: 0.5 hour
|
||||
- **Total**: 15–17 hours
|
||||
|
||||
---
|
||||
|
||||
## Phase 0: Restore LLM Client
|
||||
|
||||
### Task 0.1: Extend `AzureOpenAIConfig`
|
||||
**File:** `src/teamlandkarte_mcp/config.py`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Add `chat_deployment: str = ""` field to `AzureOpenAIConfig` dataclass
|
||||
- [x] Add Google Docstring field comment:
|
||||
`"""Azure deployment name for the chat model used in auto-tagging (e.g. 'gpt-4.1'). Required when use_auto_tagging = true."""`
|
||||
- [x] Update the TOML parser to read `azure_openai.chat_deployment` with default `""`
|
||||
- [x] Read `AZURE_OPENAI_LLM_API_KEY` from environment (store as `llm_api_key: str = ""` on `AzureOpenAIConfig`)
|
||||
- [x] When `use_auto_tagging = true` and `llm_api_key` is empty, raise `ConfigurationError` at startup
|
||||
|
||||
**Acceptance:**
|
||||
- `AzureOpenAIConfig()` instantiates with `chat_deployment=""` and `llm_api_key=""`
|
||||
- TOML `azure_openai.chat_deployment = "gpt-4.1"` is read correctly
|
||||
- Missing `AZURE_OPENAI_LLM_API_KEY` with `use_auto_tagging=true` raises at startup
|
||||
|
||||
---
|
||||
|
||||
### Task 0.2: Add `chat_completion()` to `AzureOpenAIClient`
|
||||
**File:** `src/teamlandkarte_mcp/azure/openai_client.py`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Add `chat_deployment: str` and `llm_api_key: str` constructor parameters
|
||||
- [x] Implement `chat_completion(system: str, user: str) -> str` method:
|
||||
- Calls Azure OpenAI chat completions API with `response_format={"type": "json_object"}`
|
||||
- Uses `self.chat_deployment` as the model/deployment
|
||||
- Uses `self.llm_api_key` for auth (separate from embedding key)
|
||||
- Returns the raw JSON string from the first choice's message content
|
||||
- Raises `AzureOpenAIError` on API errors (consistent with existing error handling)
|
||||
- [x] Add Google Docstring to `chat_completion()`
|
||||
|
||||
**Acceptance:**
|
||||
- `chat_completion(system="...", user="...")` returns a JSON string
|
||||
- API errors surface as `AzureOpenAIError`
|
||||
- `chat_deployment=""` + `chat_completion()` call raises `ConfigurationError` (misconfigured guard)
|
||||
|
||||
---
|
||||
|
||||
## Phase 1: New Modules
|
||||
|
||||
### Task 1.1: Add `bm25.py`
|
||||
**File:** `src/teamlandkarte_mcp/matching/bm25.py` (**new**)
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Create `bm25.py` with `_tokenize(text: str) -> list[str]` helper (lowercase, split on `[\W_]+`, filter empty)
|
||||
- [x] Implement `Bm25Index` dataclass:
|
||||
- `corpus: list[str]` field
|
||||
- Internal `_bm25: BM25Okapi | None` built in `__post_init__`
|
||||
- `rank(query: str) -> list[tuple[str, float]]` method:
|
||||
- returns `[]` for empty corpus
|
||||
- returns `(corpus_text, bm25_score)` list sorted by score descending
|
||||
- score 0.0 entries are included (caller decides cutoff)
|
||||
- [x] Implement `bm25_rank_competences(required: str, candidates: list[str]) -> list[tuple[str, float]]` convenience function
|
||||
- [x] Add Google Docstrings to all public symbols
|
||||
|
||||
**Acceptance:**
|
||||
- `Bm25Index(corpus=[]).rank("Python")` → `[]`
|
||||
- `Bm25Index(corpus=["Python"]).rank("Python")` → `[("Python", score > 0)]`
|
||||
- `Bm25Index(corpus=["JavaScript"]).rank("Python")` → `[("JavaScript", 0.0)]`
|
||||
- `Bm25Index(corpus=["Machine Learning"]).rank("machine learning")` → score > 0 (case-insensitive)
|
||||
|
||||
---
|
||||
|
||||
### Task 1.2: Add `rrf.py`
|
||||
**File:** `src/teamlandkarte_mcp/matching/rrf.py` (**new**)
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Implement `reciprocal_rank_fusion(ranked_lists, *, k=60) -> dict[str, float]`:
|
||||
- Skip any list where all scores are 0.0 (no token-overlap signal)
|
||||
- For each candidate in each non-zero list, accumulate `1 / (k + rank)` (1-based rank)
|
||||
- Normalize so the highest-scoring candidate maps to 1.0
|
||||
- Return `{}` if all lists are empty or all-zero
|
||||
- [x] Add Google Docstring including mathematical formula, zero-out rule, and normalization description
|
||||
|
||||
**Acceptance:**
|
||||
- Single list, one candidate → `{"A": 1.0}`
|
||||
- Single list `[("A", 1.0), ("B", 0.5)]` → `{"A": 1.0, "B": ...}` with `0 < B < 1`
|
||||
- Single all-zero list `[("A", 0.0), ("B", 0.0)]` → `{}` (empty, no signal)
|
||||
- `k=60` default: rank-1 score = `1/61`, rank-2 score = `1/62`, normalized → `61/62 ≈ 0.984`
|
||||
|
||||
---
|
||||
|
||||
### Task 1.3: Add `auto_tagger.py`
|
||||
**File:** `src/teamlandkarte_mcp/matching/auto_tagger.py` (**new**)
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Implement `AutoTagger` class:
|
||||
- Constructor: `__init__(self, client: AzureOpenAIClient) -> None`
|
||||
- Stores the client reference
|
||||
- [x] Implement `expand_competences(self, required: list[str], existing: list[str]) -> list[str]`:
|
||||
- Build system prompt: instructs LLM to return only required-competence names already
|
||||
covered by existing entries (synonym/abbreviation/cross-language), as JSON
|
||||
`{"additions": ["...", ...]}`
|
||||
- Build user prompt: `REQUIRED: {required}\nEXISTING: {existing}`
|
||||
- Call `self._client.chat_completion(system=..., user=...)` → JSON string
|
||||
- Parse JSON, extract `additions` list
|
||||
- Validate: keep only items present in `required` (guard against LLM hallucination)
|
||||
- Return `existing + [a for a in additions if a not in existing]`
|
||||
- On any exception (API error, JSON parse, key error): log warning, return `existing` unchanged
|
||||
- [x] Add Google Docstrings to class and all public methods
|
||||
|
||||
**Acceptance:**
|
||||
- LLM returns `{"additions": ["Machine Learning"]}` for existing `["ML"]`, required `["Machine Learning"]`
|
||||
→ returns `["ML", "Machine Learning"]`
|
||||
- LLM returns `{"additions": []}` → returns `existing` unchanged
|
||||
- LLM raises exception → returns `existing` unchanged (no exception propagation)
|
||||
- Addition not in `required` is silently dropped (hallucination guard)
|
||||
- No duplicate entries if `existing` already contains the addition
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Configuration Extension
|
||||
|
||||
### Task 2.1: Extend `SimilarityConfig`
|
||||
**File:** `src/teamlandkarte_mcp/config.py`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Add `use_bm25_search: bool = False` field to `SimilarityConfig` dataclass
|
||||
- [x] Add `use_auto_tagging: bool = False` field to `SimilarityConfig` dataclass
|
||||
- [x] Add Google Docstring comments to both fields
|
||||
- [x] Update the TOML parser to read `matching.similarity.use_bm25_search` (default `False`)
|
||||
- [x] Update the TOML parser to read `matching.similarity.use_auto_tagging` (default `False`)
|
||||
|
||||
**Acceptance:**
|
||||
- `SimilarityConfig()` instantiates with both flags `False`
|
||||
- `SimilarityConfig(use_bm25_search=True, use_auto_tagging=True)` works
|
||||
- Parser reads both flags from TOML correctly
|
||||
|
||||
---
|
||||
|
||||
### Task 2.2: Update `config.toml.example`
|
||||
**File:** `config.toml.example`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Add `use_bm25_search = false` under `[matching.similarity]` with inline comment
|
||||
- [x] Add `use_auto_tagging = false` under `[matching.similarity]` with inline comment:
|
||||
```toml
|
||||
# When true, BM25 + RRF replaces embedding-based competence similarity.
|
||||
# BM25 assigns 0 to candidates with no token overlap, eliminating false positives.
|
||||
# Role similarity is always embedding-based. Default: false.
|
||||
use_bm25_search = false
|
||||
|
||||
# When true (and use_bm25_search = true), an LLM pre-expands each candidate's
|
||||
# competence list with canonical equivalents of required competences it already covers
|
||||
# (synonyms, abbreviations, cross-language). The expansion is ephemeral. Default: false.
|
||||
use_auto_tagging = false
|
||||
```
|
||||
- [x] Add `chat_deployment = ""` under `[azure_openai]` with inline comment:
|
||||
```toml
|
||||
# Azure deployment name for the chat model used in auto-tagging (e.g. "gpt-4.1").
|
||||
# Required when use_auto_tagging = true.
|
||||
chat_deployment = ""
|
||||
```
|
||||
|
||||
**Acceptance:**
|
||||
- Config example is self-documenting for all three new keys
|
||||
- All new defaults are `false` / `""`
|
||||
|
||||
---
|
||||
|
||||
### Task 2.3: Wire `use_bm25_search`, `use_auto_tagging`, and `AutoTagger` in `mcp_server.py`
|
||||
**File:** `src/teamlandkarte_mcp/mcp_server.py`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] When `use_auto_tagging=True`: construct `AzureOpenAIClient` with `chat_deployment`
|
||||
and `llm_api_key`; construct `AutoTagger(client=llm_client)`
|
||||
- [x] Pass `auto_tagger` (or `None`) and both flags to `SimilarityEngine` constructor
|
||||
- [x] When `use_auto_tagging=False`: `auto_tagger=None` — no LLM client constructed
|
||||
|
||||
**Acceptance:**
|
||||
- Setting `use_auto_tagging = true` in `config.toml` activates auto-tagging at runtime
|
||||
- Setting `use_auto_tagging = false` (default) does not construct any LLM client
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: Similarity Engine Integration
|
||||
|
||||
### Task 3.1: Add `_bm25_rrf_similarity()` to `SimilarityEngine`
|
||||
**File:** `src/teamlandkarte_mcp/matching/similarity.py`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Import `bm25_rank_competences` from `.bm25`, `reciprocal_rank_fusion` from `.rrf`,
|
||||
and `AutoTagger` from `.auto_tagger`
|
||||
- [x] Implement `_bm25_rrf_similarity(self, required: list[str], candidate: list[str]) -> dict[str, dict[str, object]]`:
|
||||
- For each `req` in `required`:
|
||||
1. Call `bm25_rank_competences(req, candidate)` → `ranked`
|
||||
2. Call `reciprocal_rank_fusion([ranked])` → `fused: dict[str, float]`
|
||||
3. If `fused` is empty or all scores are 0.0:
|
||||
- `score = 0.0`, `best_match = None`, `rationale = "BM25: no token overlap with any candidate competence."`
|
||||
4. Otherwise:
|
||||
- `best_match = max(fused, key=fused.get)`
|
||||
- `score = fused[best_match]`
|
||||
- `rationale = f"BM25+RRF: best match '{best_match}' (score {score:.3f})."`
|
||||
- Returns dict shaped identically to `_per_skill_similarity()` output
|
||||
- [x] Add Google Docstring to `_bm25_rrf_similarity()`
|
||||
|
||||
---
|
||||
|
||||
### Task 3.2: Route `compute_competence_similarity()` by flags
|
||||
**File:** `src/teamlandkarte_mcp/matching/similarity.py`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Ensure `SimilarityEngine.__init__` accepts and stores `use_bm25_search`,
|
||||
`use_auto_tagging`, and `auto_tagger: AutoTagger | None = None`
|
||||
- [x] In `compute_competence_similarity()`:
|
||||
```python
|
||||
if self.config.use_bm25_search:
|
||||
working = candidate
|
||||
if self.config.use_auto_tagging and self._auto_tagger is not None:
|
||||
working = self._auto_tagger.expand_competences(required, candidate)
|
||||
return self._bm25_rrf_similarity(required, working)
|
||||
# else: existing strategy dispatch (unchanged)
|
||||
```
|
||||
|
||||
**Acceptance:**
|
||||
- `use_bm25_search=False`: behavior bit-for-bit identical to before
|
||||
- `use_bm25_search=True, use_auto_tagging=False`: BM25 on raw candidate list
|
||||
- `use_bm25_search=True, use_auto_tagging=True`: BM25 on LLM-expanded list
|
||||
- `use_bm25_search=True` + required=`["Python"]`, candidate=`["JavaScript"]`: `score == 0.0`
|
||||
- `use_bm25_search=True` + required=`["Python"]`, candidate=`["Python"]`: `score == 1.0`
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: Legacy Cleanup
|
||||
|
||||
### Task 4.1: Remove legacy debug files
|
||||
**Files to delete:**
|
||||
- `debug_similarity_analysis.py`
|
||||
- `debug_categorization.py`
|
||||
- `SEMANTIC_SIMILARITY_ANALYSIS.md`
|
||||
- `SCORE_BUG_FIX.md`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Verify none of these files are imported or referenced in source code or tests
|
||||
- [x] Delete `debug_similarity_analysis.py`
|
||||
- [x] Delete `debug_categorization.py`
|
||||
- [x] Delete `SEMANTIC_SIMILARITY_ANALYSIS.md`
|
||||
- [x] Delete `SCORE_BUG_FIX.md`
|
||||
|
||||
**Acceptance:**
|
||||
- `git status` shows 4 deleted files
|
||||
- No import errors in existing code
|
||||
|
||||
---
|
||||
|
||||
## Phase 5: Tests
|
||||
|
||||
### Task 5.1: Unit tests for `bm25.py`
|
||||
**File:** `tests/test_bm25.py` (**new**)
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Test `Bm25Index` with empty corpus → `rank()` returns `[]`
|
||||
- [x] Test exact match (single token) → score > 0
|
||||
- [x] Test no token overlap (query "Python", corpus "JavaScript") → score == 0.0
|
||||
- [x] Test partial token overlap multi-token query ("Machine Learning") against ("Deep Learning") → partial score > 0
|
||||
- [x] Test case-insensitivity: `rank("Python")` matches "python" in corpus
|
||||
- [x] Test tokenization edge cases:
|
||||
- "Progressive Web App (PWA)" → `["progressive", "web", "app", "pwa"]`
|
||||
- "CI/CD Pipeline" → `["ci", "cd", "pipeline"]`
|
||||
- "React.js" → `["react", "js"]`
|
||||
- [x] Test `bm25_rank_competences()` convenience wrapper produces same result as `Bm25Index.rank()`
|
||||
- [x] Add Google Docstrings to test module
|
||||
|
||||
### Task 5.2: Unit tests for `rrf.py`
|
||||
**File:** `tests/test_rrf.py` (**new**)
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Test single list, single candidate → `{"A": 1.0}`
|
||||
- [x] Test single list multiple candidates: top candidate maps to 1.0, others < 1.0 and > 0.0
|
||||
- [x] Test all-zero list → returns `{}` (empty)
|
||||
- [x] Test mixed: one non-zero list + one all-zero list → only non-zero list contributes
|
||||
- [x] Test `k` parameter: higher `k` flattens differences (rank-1 and rank-2 scores closer together)
|
||||
- [x] Test score ordering is preserved (rank 1 > rank 2 > rank 3 in output)
|
||||
- [x] Add Google Docstrings to test module
|
||||
|
||||
### Task 5.3: Unit tests for `auto_tagger.py`
|
||||
**File:** `tests/test_auto_tagger.py` (**new**)
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Mock `AzureOpenAIClient.chat_completion()` for all tests (no real LLM calls)
|
||||
- [x] Test: LLM returns `{"additions": ["Machine Learning"]}` for existing `["ML"]`,
|
||||
required `["Machine Learning"]` → result is `["ML", "Machine Learning"]`
|
||||
- [x] Test: LLM returns `{"additions": []}` → result is identical to `existing`
|
||||
- [x] Test: LLM `chat_completion` raises exception → result is identical to `existing`,
|
||||
no exception propagated
|
||||
- [x] Test: LLM returns malformed JSON (not parseable) → result is `existing` unchanged
|
||||
- [x] Test: LLM returns addition not in `required` (hallucination) → addition is dropped
|
||||
- [x] Test: LLM returns addition already in `existing` → no duplicate in result
|
||||
- [x] Test: empty `required` list → `expand_competences()` returns `existing` unchanged
|
||||
(no LLM call needed)
|
||||
- [x] Add Google Docstrings to test module
|
||||
|
||||
### Task 5.4: Extend `tests/test_similarity.py`
|
||||
**File:** `tests/test_similarity.py` (extended)
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Add tests with `use_bm25_search=True`:
|
||||
- Required `["Python"]`, candidate `["JavaScript"]` → `competence_score == 0.0`
|
||||
- Required `["Python"]`, candidate `["Python"]` → `competence_score == 1.0`
|
||||
- Required `["Python", "Machine Learning"]`, candidate `["JavaScript", "TypeScript"]` → both individual scores 0.0, overall `competence_score == 0.0`
|
||||
- Required `["Python", "Machine Learning"]`, candidate `["Python", "Machine Learning"]` → both scores 1.0
|
||||
- [x] Add tests with `use_bm25_search=True, use_auto_tagging=True` (mocked `AutoTagger`):
|
||||
- Mocked `AutoTagger.expand_competences` injects `"Machine Learning"` into `["ML"]`
|
||||
- Required `["Machine Learning"]`, candidate `["ML"]` (after expansion) → `score > 0`
|
||||
- [x] Confirm that with `use_bm25_search=False`, behavior is unchanged (existing tests still pass)
|
||||
|
||||
### Task 5.5: Extend `tests/test_matcher_integration.py`
|
||||
**File:** `tests/test_matcher_integration.py` (extended)
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Add integration test with `use_bm25_search=True`:
|
||||
- Task requires: `["Python", "Machine Learning"]`
|
||||
- Candidate A has: `["Python", "Machine Learning", "Pandas"]` → high score
|
||||
- Candidate B has: `["JavaScript", "TypeScript", "Node.js", "Vue.js"]` → score 0.0
|
||||
- Assert: candidate A `competence_score` > candidate B `competence_score`
|
||||
- Assert: candidate B `competence_score == 0.0` (false-positive eliminated)
|
||||
- [x] Add integration test with `use_bm25_search=True, use_auto_tagging=True`
|
||||
(mocked LLM via mocked `AutoTagger`):
|
||||
- Candidate has `["ML", "Python"]`; auto-tag adds `"Machine Learning"` to working list
|
||||
- Required `["Machine Learning", "Python"]` → both scores > 0
|
||||
- [x] These tests explicitly document the false-positive scenario from production
|
||||
|
||||
---
|
||||
|
||||
## Phase 6: Documentation
|
||||
|
||||
### Task 6.1: Update `docs/architecture.md`
|
||||
**File:** `docs/architecture.md`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Add section "BM25 + RRF + Auto-Tagging Competence Matching" under the matching/similarity section:
|
||||
- Describe the three new modules (`auto_tagger.py`, `bm25.py`, `rrf.py`)
|
||||
- Describe the auto-tag → BM25 → RRF pipeline and routing in `compute_competence_similarity()`
|
||||
- Note that role similarity is unaffected
|
||||
- Reference `use_bm25_search` and `use_auto_tagging` config keys
|
||||
|
||||
### Task 6.2: Update `docs/semantic_similarity_tradeoffs.md`
|
||||
**File:** `docs/semantic_similarity_tradeoffs.md`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Add BM25+RRF row to the strategy comparison table:
|
||||
- Strengths: exact/lexical match, no false positives for non-overlapping skills
|
||||
- Weaknesses: misses synonyms and cross-language variants (mitigated by `use_auto_tagging`)
|
||||
- When to use: when skill names in the dataset are mostly canonical English terms
|
||||
- [x] Add BM25+RRF+AutoTag row:
|
||||
- Strengths: exact match + synonym/abbreviation/cross-language bridging via LLM
|
||||
- Weaknesses: LLM latency per candidate, requires `AZURE_OPENAI_LLM_API_KEY`
|
||||
- When to use: mixed-language or abbreviation-heavy skill datasets
|
||||
|
||||
### Task 6.3: Update `docs/assistant_system_prompt.md`
|
||||
**File:** `docs/assistant_system_prompt.md`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Add note on score interpretation when `use_bm25_search=true`:
|
||||
- A `competence_score` of 0.0 means **no lexical token overlap** between required and candidate competences
|
||||
- The score is not a semantic similarity — it is a lexical matching score
|
||||
- Users should be informed when BM25 mode is active so they understand why cross-language synonyms may not match
|
||||
- [x] Add note on `use_auto_tagging=true`:
|
||||
- Auto-tagging bridges common synonym/abbreviation gaps (e.g. "ML" → "Machine Learning")
|
||||
- A positive `competence_score` with auto-tagging enabled may reflect an LLM-inferred equivalence,
|
||||
not a literal token match; the rationale field will say "BM25+RRF" regardless
|
||||
|
||||
### Task 6.4: Update `README.md`
|
||||
**File:** `README.md`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Add all three new config keys to the config reference section:
|
||||
- `matching.similarity.use_bm25_search` (boolean, default `false`): Enables BM25+RRF lexical competence matching
|
||||
- `matching.similarity.use_auto_tagging` (boolean, default `false`): Enables LLM pre-expansion before BM25
|
||||
- `azure_openai.chat_deployment` (string, default `""`): Chat model deployment name for auto-tagging
|
||||
- [x] Document `AZURE_OPENAI_LLM_API_KEY` environment variable in the env-vars section
|
||||
|
||||
---
|
||||
|
||||
## Phase 7: Dependency
|
||||
|
||||
### Task 7.1: Add `rank-bm25` to project dependencies
|
||||
**File:** `pyproject.toml`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Add `rank-bm25` to `[project.dependencies]` in `pyproject.toml`
|
||||
- [x] Run `uv lock` (or equivalent) to update the lockfile
|
||||
- [x] Verify `from rank_bm25 import BM25Okapi` works in the environment
|
||||
|
||||
**Acceptance:**
|
||||
- `uv run python -c "from rank_bm25 import BM25Okapi; print('ok')"` exits 0
|
||||
|
||||
---
|
||||
|
||||
## Phase 8: Quality Gates
|
||||
|
||||
### Task 8.1: Run unit tests
|
||||
- [x] `uv run pytest tests/test_bm25.py tests/test_rrf.py tests/test_auto_tagger.py -v`
|
||||
- [x] `uv run pytest -m "not integration" -q`
|
||||
- [x] All tests pass
|
||||
|
||||
### Task 8.2: Run linters
|
||||
- [x] `uv run ruff check src/ tests/`
|
||||
- [x] `uv run mypy src/teamlandkarte_mcp/matching/bm25.py src/teamlandkarte_mcp/matching/rrf.py src/teamlandkarte_mcp/matching/auto_tagger.py`
|
||||
- [x] Zero errors
|
||||
|
||||
### Task 8.3: OpenSpec validation
|
||||
- [x] `openspec validate add-bm25-rrf-competence-matching --strict`
|
||||
- [x] All checks pass
|
||||
|
||||
### Task 8.4: Confirm legacy files are removed
|
||||
- [x] `ls debug_*.py` → no such files
|
||||
- [x] `ls SEMANTIC_SIMILARITY_ANALYSIS.md SCORE_BUG_FIX.md` → no such files
|
||||
|
||||
---
|
||||
|
||||
## Phase 9: Architectural Fix — Global BM25 Index
|
||||
|
||||
**Problem identified post-spec**: When `Bm25Index` was built per candidate (N = number of that
|
||||
person's competences), IDF values were always 0 or negative for every term — the library
|
||||
`rank_bm25` computes `log((N - df + 0.5) / (df + 0.5))`, and with N=1 every term has df=1,
|
||||
yielding `log(0)` = negative. All BM25 scores were therefore ≤ 0, making the `max(0.0, raw)`
|
||||
clamp produce a flat 0.0 for everything. The fix builds one global corpus from the union of
|
||||
all filtered candidates' competences, ensuring N is large enough for IDF to be meaningful.
|
||||
|
||||
### Task 9.1: Remove `1e-10` clamp in `bm25.py`
|
||||
**File:** `src/teamlandkarte_mcp/matching/bm25.py`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Remove the special-case block that returned `1e-10` for zero/negative BM25 scores
|
||||
when token overlap existed (a band-aid for the IDF pathology)
|
||||
- [x] Replace with `max(0.0, float(raw))` — ubiquitous terms (negative IDF) simply score 0.0
|
||||
- [x] Update `Bm25Index` and `bm25_rank_competences()` docstrings to reflect the change
|
||||
|
||||
**Acceptance:**
|
||||
- A term present in every document of a sufficiently large corpus scores 0.0, not 1e-10
|
||||
|
||||
---
|
||||
|
||||
### Task 9.2: Add `use_bm25_search` property to `SimilarityEngine`
|
||||
**File:** `src/teamlandkarte_mcp/matching/similarity.py`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Expose `use_bm25_search: bool` as a read-only property (or attribute) on `SimilarityEngine`
|
||||
so `Matcher` can check it without accessing internal config directly
|
||||
|
||||
**Acceptance:**
|
||||
- `engine.use_bm25_search` returns the value of `engine.config.use_bm25_search`
|
||||
|
||||
---
|
||||
|
||||
### Task 9.3: Thread `global_index` parameter through similarity layer
|
||||
**File:** `src/teamlandkarte_mcp/matching/similarity.py`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Add `global_index: Bm25Index | None = None` parameter to `compute_competence_similarity()`
|
||||
- [x] Pass it through to `_bm25_rrf_similarity()`
|
||||
- [x] In `_bm25_rrf_similarity()`: when `global_index is not None`, call
|
||||
`global_index.rank(req)` and filter the result list to only entries whose text is in
|
||||
`set(candidate)`; otherwise fall back to `bm25_rank_competences(req, candidate)`
|
||||
|
||||
**Acceptance:**
|
||||
- `global_index.rank(req)` is called once per required competence, not once per candidate
|
||||
- Results are correctly filtered to the current candidate's competence set
|
||||
- Fall-back path (`global_index=None`) still works unchanged
|
||||
|
||||
---
|
||||
|
||||
### Task 9.4: Build global corpus in `Matcher` before per-candidate loop
|
||||
**File:** `src/teamlandkarte_mcp/matching/matcher.py`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Before the `for cap in filtered` loop: if `self._sim.use_bm25_search` and `filtered`
|
||||
is non-empty, collect the deduplicated union of all candidates' competences into a list
|
||||
`global_corpus`, instantiate `Bm25Index(corpus=global_corpus)`
|
||||
- [x] Pass `global_index` to every `compute_competence_similarity()` call inside the loop
|
||||
- [x] When `use_bm25_search=False` (or `filtered` is empty): `global_bm25_index = None`
|
||||
(no-op, existing behaviour)
|
||||
|
||||
**Acceptance:**
|
||||
- A corpus of N candidates' competences is built once, not N times
|
||||
- Candidates with no overlap with required competences still score 0.0
|
||||
- Candidates with overlap score > 0.0 (IDF is meaningful because N >> 1 in the global corpus)
|
||||
|
||||
---
|
||||
|
||||
### Task 9.5: Update tests for global index
|
||||
**Files:** `tests/test_bm25.py`, `tests/test_similarity.py`, `tests/test_matcher_integration.py`, `tests/test_matching_refinements.py`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] `test_bm25.py`: update small-corpus tests to use ≥ 5-doc corpora so IDF is positive;
|
||||
add `test_bm25_index_rank_filters_to_candidate_subset` and
|
||||
`test_bm25_ubiquitous_term_clamped_to_zero`
|
||||
- [x] `test_similarity.py`: update 4 BM25 tests to pass a 5-doc `global_index`; add
|
||||
`test_use_bm25_search_property_reflects_flag` and `test_bm25_global_index_filters_to_candidate_subset`
|
||||
- [x] `test_matcher_integration.py`: add `use_bm25_search = False` to `_FakeSimilarityEngine`,
|
||||
add `global_index=None` param to `compute_competence_similarity`; update
|
||||
`test_bm25_auto_tag_candidate_matches_after_expansion` to use a 5-doc global corpus;
|
||||
add `test_matcher_builds_global_bm25_index_across_all_candidates`
|
||||
- [x] `test_matching_refinements.py`: add `use_bm25_search = False` and `global_index=None`
|
||||
to local `_FakeSim` stub
|
||||
|
||||
**Acceptance:**
|
||||
- All 140 tests pass, 2 skipped
|
||||
|
||||
---
|
||||
|
||||
### Task 9.6: Update `docs/architecture.md`
|
||||
**File:** `docs/architecture.md`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Replace the incorrect "Per-call index" bullet in §9.3 with a "Global BM25 index" bullet
|
||||
describing the pre-loop construction and per-candidate filtering
|
||||
- [x] Update the §9.1 module table row for `bm25.py` to remove the `1e-10` clamp description
|
||||
and replace with `max(0.0, raw)` description
|
||||
- [x] Update the §9.2 data flow diagram to show `match_candidates` building the global index
|
||||
before the per-candidate loop
|
||||
|
||||
**Acceptance:**
|
||||
- No mention of "per-call index" or `1e-10` clamp remains in the docs
|
||||
|
||||
@@ -0,0 +1,607 @@
|
||||
# Design: Capacity Matching MCP Server
|
||||
|
||||
> **Note**: For comprehensive architecture documentation including ADRs, quality requirements, and deployment view, see [architecture.md](./architecture.md).
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
> **Update (2026-02)**: The server uses **Azure OpenAI** for role/requirements
|
||||
> extraction and similarity scoring via embeddings.
|
||||
> There are **no heuristic fallbacks** for these features.
|
||||
>
|
||||
> The embedding model is `text-embedding-3-large` with **3072 dimensions** and
|
||||
> results are cached in a local SQLite embedding cache.
|
||||
|
||||
## Embedding similarity acceleration (global prefetch + batch embeddings)
|
||||
|
||||
> **Update (2026-02)**: Embedding similarity is optimized via **global bulk prefetch**
|
||||
> plus **true Azure embeddings batch requests**. This is primarily implemented in
|
||||
> `SimilarityEngine` and `AzureOpenAIClient`.
|
||||
|
||||
### Motivation
|
||||
|
||||
The matching pipeline can be slow if embeddings are requested on-demand from inner
|
||||
loops. While the system already uses a persistent SQLite embedding cache
|
||||
(`EmbeddingCache`), additional optimization is necessary to avoid repeated:
|
||||
|
||||
- normalization + cache-key computation
|
||||
- SQLite reads
|
||||
- sequential Azure embedding calls for cache misses
|
||||
|
||||
### Goals
|
||||
|
||||
- Bulk prefetch embeddings for a matching run (roles + competences), across **all free
|
||||
capacities at once** (not person-by-person / incremental embedding).
|
||||
- Maintain deterministic behavior and strict error semantics.
|
||||
- Preserve existing similarity contracts (`per_skill` / `aggregate`) and tool outputs.
|
||||
|
||||
### Data flow
|
||||
|
||||
1. **Collect** all texts that will be embedded in the current run:
|
||||
- required competences
|
||||
- candidate competences (from **all free capacities**)
|
||||
- required roles
|
||||
- candidate roles (from **all free capacities**)
|
||||
2. **Normalize + deduplicate** using existing `_normalize_text()` logic:
|
||||
- skip empty/whitespace-only texts
|
||||
- emit a **Python logger warning** if all candidate competences normalize to empty
|
||||
3. **Resolve embeddings** using a layered cache approach:
|
||||
1) in-memory cache (instance-level, attached to `SimilarityEngine`; not created/cleared per invocation)
|
||||
2) persistent SQLite embedding cache (`EmbeddingCache`)
|
||||
3) Azure embeddings API (batch only the cache-missing texts; chunked)
|
||||
4. **Compute** cosine similarities locally using the prefetched mapping.
|
||||
|
||||
### Normalization and cache keys
|
||||
|
||||
Cache key behavior remains unchanged to avoid invalidating the on-disk cache:
|
||||
|
||||
- normalization: trim + collapse whitespace
|
||||
- key: SHA256 of `model|dims|normalized(text).lower()`
|
||||
|
||||
### Azure embeddings batch API + chunking
|
||||
|
||||
The Azure embeddings API supports embedding multiple inputs per request via
|
||||
`input=[...]`. The implementation requirements are:
|
||||
|
||||
- preserve stable mapping from input texts → returned vectors
|
||||
- rely on the `index` field in the API response if available
|
||||
- otherwise assume stable ordering
|
||||
- chunk large requests using `azure_openai.embedding_batch_size` (default: 128)
|
||||
- strict failure semantics:
|
||||
- the matching operation fails immediately on error
|
||||
- log the failing **chunk input list** via Python logging
|
||||
- cost tracking: log once per chunk with a count of embeddings requested
|
||||
|
||||
### Trade-offs
|
||||
|
||||
- Batch calls reduce network requests dramatically but increase blast radius: one
|
||||
failing request affects multiple inputs.
|
||||
- mitigation: keep retry logic at the chunk level
|
||||
- Prefetch holds a larger in-memory mapping for the current run.
|
||||
- mitigation: only store vectors required for that run (after dedup)
|
||||
|
||||
### Tests
|
||||
|
||||
- prefetch/dedup: client is called once per unique normalized text (cache misses only)
|
||||
- cache layering: in-memory first, then SQLite, then Azure; no duplicate calls
|
||||
- chunking: multiple Azure requests when `len(missing_texts) > batch_size`
|
||||
- empty input handling: warning when all candidate competences normalize to empty
|
||||
- strict error propagation: batch failures identify/log which chunk failed
|
||||
|
||||
```text
|
||||
(diagram below is historical; “sampling” arrows represent the client’s LLM runtime.
|
||||
The MCP server itself does not rely on MCP “sampling” for Azure OpenAI calls.)
|
||||
```
|
||||
|
||||
```
|
||||
┌──────────────┐
|
||||
│ MCP Client │
|
||||
│ (with LLM) │
|
||||
└──────┬───────┘
|
||||
│ MCP protocol (tool calls)
|
||||
▼
|
||||
┌───────────────────────────────────────────────────────────────────────┐
|
||||
│ MCP Server │
|
||||
│ │
|
||||
│ Task Management Tools: │
|
||||
│ - list_open_tasks │
|
||||
│ - get_task_details (table-first) │
|
||||
│ - validate_task_requirements (table-first) │
|
||||
│ - find_capacities_for_task (requires confirm) │
|
||||
│ - infer_roles │
|
||||
│ │
|
||||
│ Requirement Gathering Tools: │
|
||||
│ - extract_requirements (writes pending) │
|
||||
│ - update_requirements (writes pending) │
|
||||
│ - collect_structured_requirement_data │
|
||||
│ (writes pending) │
|
||||
│ - start_guided_capture │
|
||||
│ - guided_set_description │
|
||||
│ - guided_set_role │
|
||||
│ - guided_set_time_range (open-ended) │
|
||||
│ - guided_set_competences (writes pending) │
|
||||
│ - confirm_requirements │
|
||||
│ │
|
||||
│ Search Execution & Refinement: │
|
||||
│ - find_matching_capacities (hard-gated) │
|
||||
│ - filter_search_results │
|
||||
│ - get_results_by_category │
|
||||
│ │
|
||||
│ Business Logic: Matcher / Scorer / Caches │
|
||||
│ │
|
||||
│ Integrations: │
|
||||
│ - Trino / Open Data Lake (read-only SQL) │
|
||||
│ - Azure OpenAI (embeddings only) │
|
||||
└───────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌────────────────────────────┐
|
||||
│ Open Data Lake (Trino) │
|
||||
└────────────────────────────┘
|
||||
```
|
||||
|
||||
## Component Design
|
||||
|
||||
### 1. Configuration Layer (`src/teamlandkarte_mcp/config.py`)
|
||||
|
||||
**Purpose**: Load and validate configuration from `config.toml` and environment.
|
||||
|
||||
**Important**:
|
||||
- `config.toml` contains **non-secret** settings only (endpoints, model/deployment names, cache settings).
|
||||
- credentials come from environment (recommended: `.env`) and must not be committed:
|
||||
- `DATA_LAKE_USERNAME`
|
||||
- `DATA_LAKE_PASSWORD`
|
||||
- `AZURE_OPENAI_EMBEDDING_API_KEY`
|
||||
|
||||
**config.toml structure (excerpt)**:
|
||||
```toml
|
||||
[database]
|
||||
host = "..."
|
||||
port = 8446
|
||||
backend = "trino"
|
||||
http_scheme = "https"
|
||||
verify_ssl = true
|
||||
catalog = "hive"
|
||||
schema = "tier1_open_lake"
|
||||
|
||||
[azure_openai]
|
||||
endpoint = "https://<resource>.openai.azure.com"
|
||||
api_version = "2024-02-15-preview"
|
||||
embedding_deployment = "text-embedding-3-large"
|
||||
|
||||
[embedding_cache]
|
||||
path = ".cache/embeddings.sqlite3"
|
||||
ttl_days = 30
|
||||
|
||||
[matching.similarity]
|
||||
# `per_skill` = best-match per required competence
|
||||
# `aggregate` = mean(required) vs mean(candidate)
|
||||
strategy = "per_skill"
|
||||
```
|
||||
|
||||
### Roundtrip examples (tool-level sequences)
|
||||
|
||||
These examples describe the intended end-to-end client/server interaction.
|
||||
They explicitly include the strict **Review → Ask → Confirm** step before any
|
||||
matching execution when `matching.require_confirmation = true`.
|
||||
|
||||
#### Roundtrip A: Ad-hoc search (underspecified) using guided capture
|
||||
|
||||
1. `start_guided_capture()`
|
||||
2. `guided_set_description("...")` (must capture concrete scope/goal; skill-only is insufficient)
|
||||
3. `guided_set_role("Backend Developer")`
|
||||
4. `guided_set_time_range(date_start="2026-04-01", date_end="2026-06-30")`
|
||||
5. `guided_set_competences(["Python", "FastAPI", "Docker"])`
|
||||
6. **Review**: `show_pending_requirements()` (server returns a single review table)
|
||||
7. **Ask** (assistant → user): "Soll ich diese Anforderungen so übernehmen und die Suche starten?" (Ja/Nein)
|
||||
8. **Confirm** (only on "Ja"): `confirm_requirements(confirm=true)`
|
||||
9. Execute: `find_matching_capacities(role_name="Backend Developer", competences=[...], date_start="2026-04-01", date_end="2026-06-30")`
|
||||
- Server returns deterministic headers (`Using SEARCH_ID=...`, `SEARCH_ID=...`, `META=...`) plus `## Summary` and Top results.
|
||||
|
||||
If the user answers "Nein" in step 7, call `confirm_requirements(confirm=false)`
|
||||
and continue capturing/updating requirements.
|
||||
|
||||
#### Roundtrip B: DB task workflow (task_id-based)
|
||||
|
||||
1. `list_open_tasks(limit=...)`
|
||||
2. `get_task_details(task_id)`
|
||||
3. Optional helpers:
|
||||
- `infer_primary_role(task_id=...)` (if role unclear)
|
||||
- `validate_task_requirements(task_id)` (**only if the user explicitly requests validation**; validation is independent from matching)
|
||||
4. Capture/update requirements as needed (e.g. from task details)
|
||||
5. **Review**: `show_pending_requirements()`
|
||||
6. **Ask** (assistant → user): confirm Yes/No
|
||||
7. **Confirm** (only on "Yes"): `confirm_requirements(confirm=true)`
|
||||
8. Execute matching (depending on the client UX):
|
||||
- `find_capacities_for_task(task_id)` OR
|
||||
- `find_matching_capacities(...)`
|
||||
|
||||
> Alternative marker: If the assistant already displayed requirements to the user
|
||||
> outside of the server review table, it MAY call `request_requirements_confirmation()`
|
||||
> instead of `show_pending_requirements()`. The actual confirmation is still
|
||||
> performed exclusively via `confirm_requirements(confirm=true)` after user approval.
|
||||
|
||||
### 2. Database Layer (`src/teamlandkarte_mcp/database/trino_client.py`)
|
||||
|
||||
**Purpose**: Manage Trino/Presto connectivity to the Open Data Lake and execute read-only queries.
|
||||
|
||||
**Database Schema**:
|
||||
|
||||
**Capacity Tables** (existing):
|
||||
- `teamlandkarte_v_capacities_latest`
|
||||
- Fields: `id`, `owner_name`, `role_name`, `role_level`, `begin_date`, `end_date`, `deletion_reason`
|
||||
- Filter: `deletion_reason IS NULL` (active capacities only)
|
||||
- `teamlandkarte_v_capacity_competences_latest`
|
||||
- Fields: `capacity_id` (FK), `competence_id` (FK)
|
||||
- JOIN: Links capacities to competences
|
||||
- `teamlandkarte_v_competences_latest`
|
||||
- Fields: `id`, `name`
|
||||
- Stores competence/skill names
|
||||
|
||||
**Task Tables** (new for Round 5):
|
||||
- `beschaffungstool_kmp_task_latest`
|
||||
- Fields: `id`, `title__c`, `description__c`, `startdate__c`, `enddate__c`, `createddate`, `status__c`
|
||||
- Filter: `status__c = "Veröffentlicht"` (published tasks only)
|
||||
- Note: Role is NOT stored in DB, extracted from description
|
||||
- `beschaffungstool_kmp_skill_latest`
|
||||
- Fields: `task__c` (FK to task.id), `skillname__c`
|
||||
- JOIN: Links tasks to their required skills
|
||||
- Multiple rows per task (one per skill)
|
||||
|
||||
> Note: Trino catalog/schema are configured via `config.toml` (`catalog` / `schema`).
|
||||
> The queries in code use unqualified view names.
|
||||
|
||||
**Key Classes**:
|
||||
```python
|
||||
class TrinoClient:
|
||||
def __init__(self, config: DatabaseConfig):
|
||||
# Initialize Trino DB-API connection
|
||||
|
||||
def get_all_capacities_with_competences(self) -> List[Capacity]:
|
||||
"""Fetch all active capacities with their competences in a single query"""
|
||||
query = """
|
||||
SELECT
|
||||
cap.id,
|
||||
cap.owner_name,
|
||||
cap.role_name,
|
||||
cap.role_level,
|
||||
cap.begin_date,
|
||||
cap.end_date,
|
||||
comp.name as competence_name
|
||||
FROM teamlandkarte_v_capacities_latest cap
|
||||
LEFT JOIN teamlandkarte_v_capacity_competences_latest cap_comp
|
||||
ON cap.id = cap_comp.capacity_id
|
||||
LEFT JOIN teamlandkarte_v_competences_latest comp
|
||||
ON cap_comp.competence_id = comp.id
|
||||
WHERE cap.deletion_reason IS NULL
|
||||
ORDER BY cap.id, comp.name
|
||||
"""
|
||||
# Execute query with proper parameterization
|
||||
# Group results by capacity_id in Python
|
||||
# Return List[Capacity] with competences populated
|
||||
|
||||
def get_open_tasks(self, limit: int = 20) -> List[Task]:
|
||||
"""Fetch latest open tasks with their skills."""
|
||||
|
||||
# Apply LIMIT to tasks before joining skills, otherwise one task with
|
||||
# many skill rows can consume the full LIMIT.
|
||||
query = """
|
||||
WITH latest_tasks AS (
|
||||
SELECT
|
||||
t.id,
|
||||
t.title__c,
|
||||
t.description__c,
|
||||
t.startdate__c,
|
||||
t.enddate__c,
|
||||
t.createddate
|
||||
FROM beschaffungstool_kmp_task_latest t
|
||||
WHERE t.status__c = ?
|
||||
ORDER BY t.createddate DESC
|
||||
LIMIT ?
|
||||
)
|
||||
SELECT
|
||||
t.id,
|
||||
t.title__c,
|
||||
t.description__c,
|
||||
t.startdate__c,
|
||||
t.enddate__c,
|
||||
t.createddate,
|
||||
s.skillname__c
|
||||
FROM latest_tasks t
|
||||
LEFT JOIN beschaffungstool_kmp_skill_latest s
|
||||
ON t.id = s.task__c
|
||||
ORDER BY t.createddate DESC
|
||||
"""
|
||||
# Execute with limit parameter
|
||||
# Group results by task_id (multiple rows per task for skills)
|
||||
# Return List[Task] with skills populated
|
||||
|
||||
def get_task_by_id(self, task_id: str) -> Optional[Task]:
|
||||
"""Fetch a single task with its skills by ID"""
|
||||
query = """
|
||||
SELECT
|
||||
t.id,
|
||||
t.title__c,
|
||||
t.description__c,
|
||||
t.startdate__c,
|
||||
t.enddate__c,
|
||||
t.createddate,
|
||||
s.skillname__c
|
||||
FROM beschaffungstool_kmp_task_latest t
|
||||
LEFT JOIN beschaffungstool_kmp_skill_latest s
|
||||
ON t.id = s.task__c
|
||||
WHERE t.id = ? AND t.status__c = 'Veröffentlicht'
|
||||
"""
|
||||
# Execute with task_id parameter
|
||||
# Group skill rows into single Task object
|
||||
# Return Task or None if not found
|
||||
|
||||
@dataclass
|
||||
class Capacity:
|
||||
id: int
|
||||
owner_name: str
|
||||
role_name: str
|
||||
role_level: str
|
||||
begin_date: date
|
||||
end_date: date
|
||||
competences: List[str] = field(default_factory=list)
|
||||
|
||||
@dataclass
|
||||
class Task:
|
||||
id: str
|
||||
title: str
|
||||
description: str
|
||||
start_date: Optional[date]
|
||||
end_date: Optional[date]
|
||||
created_date: datetime
|
||||
skills: List[str] = field(default_factory.list)
|
||||
```
|
||||
|
||||
**Safety**:
|
||||
- Connection uses read-only user (enforced by DB permissions)
|
||||
- Query validation to prevent any INSERT/UPDATE/DELETE
|
||||
- Parameterized queries to prevent SQL injection
|
||||
- Single join query reduces round-trips and complexity
|
||||
|
||||
**Implementation Note**:
|
||||
The query returns multiple rows per capacity (one per competence). Python code groups these rows by capacity_id to create the final List[Capacity] structure.
|
||||
|
||||
### 3. Cache Layer (`src/cache/`)
|
||||
|
||||
**Purpose**: Provide two-tier caching for database queries and search results.
|
||||
|
||||
**Search session behavior**:
|
||||
- Matching creates a new `search_id` (UUID) and stores full results in-memory.
|
||||
- Filtering creates a new `filter_id` under the same `search_id`.
|
||||
- Search-related tools emit a JSON block so clients can reliably capture IDs.
|
||||
- Tools reject non-UUID `search_id` early to avoid confusing cache-miss flows.
|
||||
|
||||
**Implementation**:
|
||||
```python
|
||||
from cachetools import TTLCache
|
||||
from datetime import timedelta
|
||||
import json
|
||||
import uuid
|
||||
|
||||
class QueryCache:
|
||||
"""Database query cache with 12-hour TTL"""
|
||||
def __init__(self, ttl_hours: int = 12, max_size: int = 100):
|
||||
self.cache = TTLCache(
|
||||
maxsize=max_size,
|
||||
ttl=ttl_hours * 3600
|
||||
)
|
||||
self.stats = {"hits": 0, "misses": 0}
|
||||
|
||||
def get_or_fetch(self, key: str, fetch_fn: callable) -> Any:
|
||||
if key in self.cache:
|
||||
self.stats["hits"] += 1
|
||||
return self.cache[key]
|
||||
|
||||
self.stats["misses"] += 1
|
||||
result = fetch_fn()
|
||||
self.cache[key] = result
|
||||
return result
|
||||
|
||||
class SearchCache:
|
||||
"""Search results cache with 60-minute TTL, keyed by search_id"""
|
||||
def __init__(self, ttl_minutes: int = 60, max_size: int = 100):
|
||||
self.cache = TTLCache(
|
||||
maxsize=max_size,
|
||||
ttl=ttl_minutes * 60
|
||||
)
|
||||
self.stats = {"hits": 0, "misses": 0}
|
||||
|
||||
def store_search(self, search_data: dict) -> str:
|
||||
"""Store search results and return search_id"""
|
||||
search_id = str(uuid.uuid4())
|
||||
self.cache[search_id] = json.dumps(search_data)
|
||||
return search_id
|
||||
|
||||
def get_search(self, search_id: str) -> Optional[dict]:
|
||||
"""Retrieve search results by search_id"""
|
||||
if search_id in self.cache:
|
||||
self.stats["hits"] += 1
|
||||
return json.loads(self.cache[search_id])
|
||||
|
||||
self.stats["misses"] += 1
|
||||
return None
|
||||
|
||||
def update_search(self, search_id: str, search_data: dict) -> bool:
|
||||
"""Update existing search results (e.g., add filters)"""
|
||||
if search_id in self.cache:
|
||||
self.cache[search_id] = json.dumps(search_data)
|
||||
return True
|
||||
return False
|
||||
```
|
||||
|
||||
**Cache Keys**:
|
||||
|
||||
**QueryCache**:
|
||||
- `"all_capacities_with_competences"` - Complete dataset from database
|
||||
|
||||
**SearchCache**:
|
||||
- `<search_id>` (UUID) - Stores complete search results and associated filters as JSON:
|
||||
```json
|
||||
{
|
||||
"task_id": "T-12345", // NEW: Reference to database task (null for ad-hoc searches)
|
||||
"requirements": {
|
||||
"required_competences": ["Python", "FastAPI"],
|
||||
"preferred_role": "Backend Developer",
|
||||
"candidate_roles": ["Backend Developer", "Software Engineer"],
|
||||
"date_start": "2026-04-01",
|
||||
"date_end": "2026-06-30"
|
||||
},
|
||||
"results_by_category": {
|
||||
"Top": [...],
|
||||
"Good": [...],
|
||||
"Partial": [...],
|
||||
"Low": [...]
|
||||
},
|
||||
"filters": {
|
||||
"filter-1": {
|
||||
"criteria": {
|
||||
"role": "Developer",
|
||||
"competences": ["Python", "Docker"],
|
||||
"min_similarity": 0.7
|
||||
},
|
||||
"results_by_category": {
|
||||
"Top": [...],
|
||||
"Good": [...],
|
||||
"Partial": [...],
|
||||
"Low": [...]
|
||||
},
|
||||
"timestamp": "2026-02-11T14:35:00Z"
|
||||
},
|
||||
"filter-2": {
|
||||
"criteria": {
|
||||
"competences": ["Cloud"],
|
||||
"min_similarity": 0.7
|
||||
},
|
||||
"results_by_category": {...},
|
||||
"timestamp": "2026-02-11T14:40:00Z"
|
||||
}
|
||||
},
|
||||
"timestamp": "2026-02-11T14:30:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
**Notes**:
|
||||
- `task_id` field is `null` for ad-hoc searches (via `find_matching_capacities`)
|
||||
- `task_id` field contains database task ID for task-based searches (via `find_capacities_for_task`)
|
||||
- This allows tracking the source of each search for audit and debugging purposes
|
||||
|
||||
**Session State** (in-memory, per client session):
|
||||
- `"last_requirements"` - Stores TaskRequirements object for use with `update_requirements`
|
||||
- Cleared when new search is executed via `find_matching_capacities`
|
||||
|
||||
### 4. Matching Engine (`src/matching/matcher.py`)
|
||||
|
||||
**Purpose**: Analyze tasks and match capacities based on requirements.
|
||||
|
||||
**Key Components**:
|
||||
|
||||
#### Matching Logic
|
||||
```python
|
||||
# NOTE (2026-02): This section is historical pseudocode. The current
|
||||
# implementation uses Azure OpenAI via `AzureOpenAIClient` (chat + embeddings)
|
||||
# and does not rely on MCP sampling.
|
||||
|
||||
class CapacityMatcher:
|
||||
def __init__(
|
||||
self,
|
||||
config: MatchingConfig,
|
||||
similarity_engine,
|
||||
):
|
||||
self.config = config
|
||||
self.similarity_engine = similarity_engine
|
||||
|
||||
async def _match_competences(
|
||||
self,
|
||||
capacity_comps: List[str],
|
||||
required_comps: List[str],
|
||||
) -> float:
|
||||
"""Compute similarity via embeddings + cosine similarity."""
|
||||
|
||||
details = await self.similarity_engine.compute_competence_similarity(
|
||||
required=required_comps,
|
||||
candidate=capacity_comps,
|
||||
)
|
||||
if not details:
|
||||
return 0.0
|
||||
return sum(d["score"] for d in details.values()) / len(details)
|
||||
```
|
||||
|
||||
### 5. Scoring System (`src/matching/scorer.py`)
|
||||
|
||||
**Purpose**: Categorize match results into Top/Good/Partial/Low.
|
||||
|
||||
```python
|
||||
class ScoreCategory(Enum):
|
||||
TOP = "Top"
|
||||
GOOD = "Good"
|
||||
PARTIAL = "Partial"
|
||||
LOW = "Low"
|
||||
|
||||
class Scorer:
|
||||
def __init__(self, config: MatchingConfig):
|
||||
self.thresholds = {
|
||||
ScoreCategory.TOP: config.top_threshold,
|
||||
ScoreCategory.GOOD: config.good_threshold,
|
||||
ScoreCategory.PARTIAL: config.partial_threshold
|
||||
}
|
||||
|
||||
def categorize(self, score: float) -> ScoreCategory:
|
||||
if score >= self.thresholds[ScoreCategory.TOP]:
|
||||
return ScoreCategory.TOP
|
||||
elif score >= self.thresholds[ScoreCategory.GOOD]:
|
||||
return ScoreCategory.GOOD
|
||||
elif score >= self.thresholds[ScoreCategory.PARTIAL]:
|
||||
return ScoreCategory.PARTIAL
|
||||
else:
|
||||
return ScoreCategory.LOW
|
||||
|
||||
def group_by_category(self, results: List[MatchResult]) -> Dict[ScoreCategory, List[MatchResult]]:
|
||||
"""Group results by score category"""
|
||||
grouped = {cat: [] for cat in ScoreCategory}
|
||||
for result in results:
|
||||
category = self.categorize(result.overall_score)
|
||||
result.category = category
|
||||
grouped[category].append(result)
|
||||
return grouped
|
||||
```
|
||||
|
||||
### 6. MCP Server (`src/mcp_server.py`)
|
||||
|
||||
**Purpose**: Expose capacity matching as MCP tools.
|
||||
|
||||
**Workflow Design**:
|
||||
|
||||
The server provides a **multi-step workflow** with separate tools for each phase:
|
||||
|
||||
0. **Task Management Phase** (NEW in Round 5):
|
||||
- `list_open_tasks()` - Browse published tasks from database
|
||||
- `get_task_details()` - View task details with extracted role
|
||||
- `validate_task_requirements()` - Compare DB skills vs description-extracted skills
|
||||
- `find_capacities_for_task()` - Automated capacity search for database task
|
||||
|
||||
1. **Requirement Gathering Phase** (Ad-hoc):
|
||||
- `extract_requirements()` - Extract from natural language task description
|
||||
- `collect_structured_requirement_data()` - Interactive step-by-step collection
|
||||
- `update_requirements()` - Modify previously extracted requirements
|
||||
|
||||
2. **Search Execution Phase**:
|
||||
- `find_matching_capacities()` - Execute search with confirmed requirements
|
||||
|
||||
3. **Result Refinement Phase**:
|
||||
- `filter_search_results()` - Apply fuzzy filters to results
|
||||
- `get_results_by_category()` - Browse results by score category
|
||||
|
||||
**Key Design Principles**:
|
||||
- **Separation of Concerns**: Each tool has a single, clear responsibility
|
||||
- **LLM State Management**: The client-side LLM tracks requirement state across tool calls
|
||||
- **Explicit Confirmation**: Requirements are returned for user confirmation before search execution
|
||||
- **Filter Chain Tracking**: Filters use `filter_id` to maintain navigation hierarchy
|
||||
- **Database Task Integration**: Seamless workflow for published tasks with automatic validation
|
||||
- **Common Extraction Logic**: DRY principle with shared `_extract_requirements_from_description()` function
|
||||
- **Availability as filter**: Availability is displayed and can be used as a date-range filter (not part of scoring)
|
||||
|
||||
**MCP Tools**:
|
||||
@@ -0,0 +1,141 @@
|
||||
# Change: Add Capacity Matching MCP Server (Stage 1)
|
||||
|
||||
## Why
|
||||
DB Systel employees have free work capacities stored in the Open Data Lake, but there's no efficient way for AI assistants to match these capacities with tasks. Tasks are also stored in the database, but finding suitable capacities for them requires manual work. This MCP server will enable AI clients to automatically:
|
||||
1. Browse and search published tasks from the database
|
||||
2. Validate task requirements (comparing DB skills vs description-extracted skills)
|
||||
3. Find the best-suited employees for both database tasks and ad-hoc requirements
|
||||
4. Match capacities based on competences and role, with optional availability (date-range) filtering
|
||||
|
||||
## What Changes
|
||||
- Add MCP server implementation with **10-tool workflow** (task management, requirement gathering, search, and refinement)
|
||||
- Add **database task management** (4 new tools): list tasks, view details, validate requirements, automated search
|
||||
- Add **task database integration**: Query published tasks with skills from Beschaffungstool KMP tables
|
||||
- Add **requirement validation**: Compare database skills vs LLM-extracted skills from description
|
||||
- Add **common extraction function**: DRY principle with `_extract_requirements_from_description()`
|
||||
- Add **internal helper functions**: `extract_requirements_from_task()`, `extract_roles_from_task()`
|
||||
- Add database connection layer for Trino/Presto with single join query
|
||||
- Requirement extraction and semantic matching are implemented via **Azure OpenAI**:
|
||||
- Chat completions for role/requirement extraction
|
||||
- Embeddings + cosine similarity for semantic similarity scoring
|
||||
- Errors are deterministic (no heuristic fallbacks)
|
||||
|
||||
> **Historical note (2026-02)**: The initial Stage-1 proposal assumed only
|
||||
> client-side LLM access via MCP sampling and used heuristic fallbacks because
|
||||
> sampling was not available in the runtime. The current implementation has been
|
||||
> migrated to an Azure OpenAI-only integration (see change
|
||||
> `replace-heuristics-with-azure-openai`).
|
||||
|
||||
- Add **session state management** for requirement tracking across tool calls
|
||||
- Add configurable scoring system with categorical results (Top/Good/Partial/Low)
|
||||
- Add **two-tier caching**: database cache (12h TTL) and search results cache (60min TTL) with task_id tracking
|
||||
- Emit deterministic, machine-readable headers in search-related tool outputs so MCP
|
||||
clients can reliably capture `search_id` / `filter_id`:
|
||||
- first line: `Using SEARCH_ID=<uuid>`
|
||||
- plus `SEARCH_ID=<uuid>`, `FILTER_ID=<uuid>` (if applicable), `META=<json>`
|
||||
- Reject non-UUID `search_id` values early with a clear message
|
||||
- Track the latest search id internally for debugging/consistency, but do **not** expose a "last search id" tool (multi-user/session unsafe)
|
||||
- Add **requirement workflow**: extract from free text, collect structured, or update requirements
|
||||
- Add **explicit confirmation** before search execution
|
||||
- Remove availability from weighted scoring (availability is displayed and can be used as a filter)
|
||||
- Update default weights: competence 0.8, role 0.2
|
||||
- Keep date range as an optional availability filter (when provided)
|
||||
- Add interactive result exploration with **search session management** (search_id)
|
||||
- Add **fuzzy filtering** for post-search refinement (role and/or competences as list)
|
||||
- Add **filter tracking** with filter_id (no search_id proliferation)
|
||||
- Add **pagination** for result browsing (20 results per page)
|
||||
- Add database configuration in `database.toml` (Trino)
|
||||
- Add **Arc42 architecture documentation** with ADRs
|
||||
|
||||
### Strict two-step confirmation (Review → Ask → Confirm)
|
||||
|
||||
When `matching.require_confirmation = true` (default), the server enforces a
|
||||
strict, user-driven confirmation sequence before any matching run:
|
||||
|
||||
1. Requirements are captured/updated into **pending** state (e.g. via
|
||||
`extract_requirements`, `collect_structured_requirement_data`, `update_requirements`,
|
||||
or guided capture tools).
|
||||
2. The assistant must **review** pending requirements using
|
||||
`show_pending_requirements()` (preferred) or, if requirements were already
|
||||
shown to the user in chat, call `request_requirements_confirmation()` as a
|
||||
marker.
|
||||
3. The assistant must **ask** the user explicitly for Yes/No confirmation.
|
||||
4. Only after user approval, the assistant calls `confirm_requirements(confirm=true)`.
|
||||
If the user says No, it calls `confirm_requirements(confirm=false)` and
|
||||
continues requirement capture.
|
||||
|
||||
The server refuses to confirm requirements unless step 2 has happened first
|
||||
(prevents auto-confirm in the same turn as requirement capture).
|
||||
|
||||
**Stage 1 Scope:**
|
||||
This is the first development stage focusing on basic matching functionality and database task integration. Future stages will add more sophisticated features (to be specified later).
|
||||
|
||||
**Key Architectural Decisions:**
|
||||
- **Azure OpenAI integration**: Server calls Azure OpenAI directly for extraction
|
||||
and embeddings (no sampling dependency)
|
||||
- **10-tool workflow**: Separate tools for task management (4), requirement gathering (3), search execution (1), and result refinement (2)
|
||||
- **Database task integration**: Direct access to published tasks with skills from Beschaffungstool KMP
|
||||
- **Validation-only approach**: Validation compares DB vs description but always uses DB skills for search
|
||||
- **Common extraction function**: `_extract_requirements_from_description()` as single source of truth (DRY principle)
|
||||
- **Automatic task workflow**: `find_capacities_for_task()` fully automated with validation display
|
||||
- **Task reference tracking**: Search cache includes `task_id` field (null for ad-hoc, task ID for DB tasks)
|
||||
- **Explicit confirmation**: Requirements returned for user confirmation before search
|
||||
- **Session state**: In-memory storage of requirements for update workflow
|
||||
- **Filter tracking**: Filters stored within search cache using filter_id (not new search_id)
|
||||
- **Search sessions**: Complete results stored server-side, accessed via search_id (60min expiry)
|
||||
- **Single query pattern**: One JOIN query fetches all data, processed in Python
|
||||
- **Markdown output**: All tables formatted as markdown for better client rendering
|
||||
- **No truncation**: Full competence lists always shown
|
||||
- **No availability scoring**: Availability is displayed and can be used as a date-range filter, but does not influence the score
|
||||
- **Weights**: competence 0.8, role 0.2
|
||||
- **Fuzzy matching**: Filter tool uses fuzzywuzzy for flexible result refinement
|
||||
- **LLM integration**: Azure OpenAI (`gpt-4.1` chat + `text-embedding-3-large`)
|
||||
with embedding cache; no heuristic fallback
|
||||
|
||||
## Impact
|
||||
- **New capability**: `capacity-matching` - Core matching functionality with database task integration
|
||||
- **New files**:
|
||||
- `database.toml` - Database connection configuration
|
||||
- `architecture.md` - Arc42 architecture documentation with ADRs
|
||||
- `src/teamlandkarte_mcp/mcp_server.py` - MCP server implementation (10 tools)
|
||||
- `src/teamlandkarte_mcp/database/trino_client.py` - Trino client (capacity + task queries)
|
||||
- `src/teamlandkarte_mcp/matching/task_analyzer.py` - Azure-based task analysis
|
||||
- `src/matching/matcher.py` - Matching algorithm with common extraction function
|
||||
- `src/matching/scorer.py` - Scoring logic
|
||||
- `src/cache/query_cache.py` - Database query caching (12h TTL)
|
||||
- `src/cache/search_cache.py` - Search results caching (60min TTL) with task_id tracking
|
||||
- `src/config.py` - Configuration management
|
||||
- `requirements.txt` or updated `pyproject.toml` - Dependencies
|
||||
- **Modified files**:
|
||||
- `main.py` - Launch MCP server
|
||||
- `pyproject.toml` - Add dependencies
|
||||
- `README.md` - Update with task management features, validation workflow
|
||||
- **Dependencies added**:
|
||||
- `mcp` - Model Context Protocol SDK
|
||||
- `trino` - Trino/Presto database client
|
||||
- `cachetools` - Caching implementation
|
||||
- `fuzzywuzzy` - Fuzzy string matching for filters
|
||||
|
||||
## Security & Constraints
|
||||
- **Read-only operations**: Server must NEVER modify database data
|
||||
- **Credential storage**: Credentials are provided via environment variables
|
||||
(recommended: local `.env`): `DATA_LAKE_USERNAME`, `DATA_LAKE_PASSWORD`.
|
||||
`database.toml` contains non-secret connection settings and must not be
|
||||
committed.
|
||||
- **Network security**: Trino/Presto connection over port 8446
|
||||
- **Data filtering**:
|
||||
- Capacities: Only consider where `deletion_reason IS NULL`
|
||||
- Tasks: Only consider where `status__c = "Veröffentlicht"`
|
||||
- **LLM access**: Not available in the current server runtime. If reintroduced,
|
||||
it must use a supported MCP API.
|
||||
- **SQL injection prevention**: All queries use proper parameterization
|
||||
- **Validation scope**: Comparison only (skills and dates), role not validated (not in DB yet)
|
||||
|
||||
## Non-Goals (Future Stages)
|
||||
- Advanced filtering options beyond fuzzy matching
|
||||
- Capacity booking/reservation
|
||||
- Task modification or status updates
|
||||
- Historical analysis
|
||||
- Integration with other systems
|
||||
- Multi-language support
|
||||
- Role storage in database (currently extracted from description)
|
||||
+99
@@ -0,0 +1,99 @@
|
||||
# Capacity Matching Specification
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Workflow overview (Round 6)
|
||||
|
||||
The system SHALL support two primary workflows:
|
||||
|
||||
1. **Database task workflow (Phase 0)**: Users browse published tasks from the database, inspect details, optionally validate, and run a capacity search for an existing task.
|
||||
2. **Ad-hoc workflow (Phases 1-3)**: Users gather requirements (free-text extraction, structured input, or guided capture), optionally update requirements, explicitly confirm, execute search, and refine results.
|
||||
|
||||
The system SHALL expose the MCP tools required to support the workflows below. The implementation MAY add additional tools over time.
|
||||
|
||||
#### Scenario: Tool surface includes both DB-task and ad-hoc workflows
|
||||
|
||||
**Phase 0 – Task Management (Database tasks)**
|
||||
- `list_open_tasks(limit=20)`
|
||||
- `get_task_details(task_id)`
|
||||
- `validate_task_requirements(task_id)`
|
||||
- `find_capacities_for_task(task_id)`
|
||||
- `infer_roles(task_id? | task_description?, limit=5)`
|
||||
|
||||
**Phase 1 – Requirement Gathering (Ad-hoc)**
|
||||
- `extract_requirements(task_description)` → writes pending requirements
|
||||
- `collect_structured_requirement_data(role_name?, competences?, date_start?, date_end?, description?)` → writes pending requirements
|
||||
- `update_requirements(change_description)` → writes pending requirements
|
||||
- Guided capture tools:
|
||||
- `start_guided_capture()`
|
||||
- `guided_set_description(description)`
|
||||
- `guided_set_role(role_name)`
|
||||
- `guided_set_time_range(date_start?, date_end?)` (open-ended allowed)
|
||||
- `guided_set_competences(competences)` → writes pending requirements
|
||||
- Review/ask tools:
|
||||
- `show_pending_requirements()` → returns a review table and marks that user confirmation was requested
|
||||
- `request_requirements_confirmation()` → marks that the assistant asked the user to confirm (no table)
|
||||
- `confirm_requirements(confirm: bool = True)`
|
||||
|
||||
**Phase 2 – Search Execution (Ad-hoc)**
|
||||
- `find_matching_capacities(role_name, competences, date_start?, date_end?)`
|
||||
|
||||
**Phase 3 – Result Refinement**
|
||||
- `filter_search_results(search_id, role_filter?, competence_filter?, availability_date_start?, availability_date_end?, min_similarity=0.7)` → returns `filter_id`
|
||||
- `get_results_by_category(search_id, filter_id?, category="Top", page=1, page_size=20)`
|
||||
|
||||
### Requirement: Confirmation gating
|
||||
|
||||
The system SHALL ensure that matching execution is confirmation-gated by default and behaves deterministically based on configuration.
|
||||
|
||||
#### Scenario: Matching is refused until requirements are confirmed (default)
|
||||
|
||||
- Confirmation gating:
|
||||
- The server's confirmation state is stored in in-memory session state and is **not** safe as a multi-client / multi-tenant source of truth.
|
||||
Therefore, treat confirmation gating primarily as a **client-side UX rule**: clients should only trigger matching after an explicit user Yes/No.
|
||||
- If `matching.require_confirmation = true` (default), matching tools MUST refuse to run unless requirements have been explicitly confirmed.
|
||||
- If `matching.require_confirmation = false`, clients MUST NOT ask for confirmation and MUST NOT call confirmation tools; matching MUST proceed once requirements are complete.
|
||||
- The confirmation flow MUST be two-step:
|
||||
1) `show_pending_requirements()` (preferred) OR `request_requirements_confirmation()`
|
||||
2) user explicitly confirms in chat
|
||||
3) `confirm_requirements(confirm=true)`
|
||||
- Servers MAY refuse `confirm_requirements(confirm=true)` if step (1) did not occur.
|
||||
|
||||
### Requirement: Tool outputs and result headers
|
||||
|
||||
The system SHALL emit table-first tool output formats and deterministic, machine-readable headers for search-session tools.
|
||||
|
||||
#### Scenario: Tool results are table-first and include deterministic header metadata
|
||||
|
||||
- Tool outputs:
|
||||
- `get_task_details` MUST be table-first (summary table + description below)
|
||||
- `validate_task_requirements` MUST be table-first (single comparison table + short written summary)
|
||||
- `infer_roles` MUST return a ranked table: `Rank | Role | Rationale`
|
||||
|
||||
- Search session output headers (robust client parsing):
|
||||
- The tools `find_matching_capacities`, `filter_search_results`, and `get_results_by_category` MUST emit deterministic, machine-readable header lines.
|
||||
- On success, the first line MUST be exactly:
|
||||
- `Using SEARCH_ID=<uuid>`
|
||||
- The output MUST also contain:
|
||||
- `SEARCH_ID=<uuid>`
|
||||
- `FILTER_ID=<uuid>` (only if a filter was created/used)
|
||||
- `META=<json>` (MUST include at least `{ "search_id": "...", "status": "ok"|... }`)
|
||||
|
||||
### Requirement: Search session validation and table guarantees
|
||||
|
||||
The system SHALL validate search session identifiers deterministically and MUST always return well-formed Markdown tables for search result tools.
|
||||
|
||||
#### Scenario: Invalid or expired search_id inputs produce deterministic errors and tables are always present
|
||||
|
||||
- Search session validation:
|
||||
- `search_id` inputs MUST be validated as UUIDs.
|
||||
- If the `search_id` is not a UUID, the tool MUST return `META.status=invalid_search_id_format` and guidance to copy the id from tool output.
|
||||
- If the `search_id` is unknown/expired, the tool MUST return `META.status=unknown_or_expired` and guidance to re-run the matching tool.
|
||||
|
||||
- Table-first guarantees for search results:
|
||||
- `find_matching_capacities` MUST include the `## Summary` table.
|
||||
- `find_matching_capacities` MUST include a results table for the **first non-empty category** in this order: Top → Good → Partial → Low.
|
||||
- If all categories are empty, it MUST still emit a valid Markdown table (a single dummy row is acceptable).
|
||||
- `filter_search_results` and `get_results_by_category` MUST NOT repeat the summary counts table.
|
||||
- `filter_search_results` MUST output a flat results table (not split by category) and MUST include a `Category` column **separate** from the numeric `Score`.
|
||||
- `filter_search_results` MUST always return the results as a valid Markdown table (even for exactly one match, and also when there are zero matches).
|
||||
@@ -0,0 +1,230 @@
|
||||
## 1. Project Setup
|
||||
- [x] 1.1 Update `pyproject.toml` with required dependencies (mcp, trino, cachetools)
|
||||
- [x] 1.2 Create `database.toml` with connection configuration structure (template only, no real credentials)
|
||||
- [x] 1.3 Create project structure: `src/`, `src/database/`, `src/matching/`, `src/cache/`
|
||||
- [x] 1.4 Add `.gitignore` entry for `database.toml` to prevent credential exposure
|
||||
|
||||
## 2. Configuration Management
|
||||
- [x] 2.1 Create `src/config.py` to load database and matching configuration
|
||||
- [x] 2.2 Implement configuration validation (ensure required fields present)
|
||||
- [x] 2.3 Add configurable scoring thresholds (Top: ≥80%, Good: 60-79%, Partial: 40-59%, Low: <40%)
|
||||
- [x] 2.4 Add configurable matching weights (competence: 0.8, role: 0.2)
|
||||
- [x] 2.5 Validate that weights sum to 1.0
|
||||
- [x] 2.6 Support two cache TTL settings: db_ttl_hours (12h), search_ttl_minutes (60min)
|
||||
- [x] 2.7 Add fuzzy matching configuration: min_similarity (default: 0.7)
|
||||
|
||||
## 3. Database Layer
|
||||
- [x] 3.1 Create Trino client with connection handling
|
||||
- [x] 3.2 Implement connection pooling/management
|
||||
- [x] 3.3 **VERIFY SCHEMA**: Confirm database schema and field names with actual Open Data Lake tables (via Trino)
|
||||
- [x] 3.4 Implement single join query method:
|
||||
- [x] 3.4.1 `get_all_capacities_with_competences()` - Single query joining all 3 tables
|
||||
- [x] 3.4.2 Filter: deletion_reason IS NULL
|
||||
- [x] 3.4.3 Use proper parameterized queries (no string interpolation)
|
||||
- [x] 3.4.4 Group results by capacity_id in Python to build List[Capacity]
|
||||
- [x] 3.5 Ensure read-only operations (no INSERT/UPDATE/DELETE queries)
|
||||
- [x] 3.6 Add error handling for connection failures with retry logic (3x, exponential backoff)
|
||||
- [x] 3.7 Add query logging for debugging (without sensitive data)
|
||||
- [x] 3.8 **NEW**: Implement task database queries:
|
||||
- [x] 3.8.1 Create `Task` dataclass with fields: id, title__c, description__c, startdate__c, enddate__c, createddate, skills
|
||||
- [x] 3.8.2 `get_open_tasks(limit)` - Fetch published tasks sorted by createddate DESC
|
||||
- [x] 3.8.3 JOIN task table with task skills table (task.id = skill.task__c)
|
||||
- [x] 3.8.4 Filter: status__c = "Veröffentlicht"
|
||||
- [x] 3.8.5 Group results by task_id in Python to build List[Task]
|
||||
- [x] 3.8.6 `get_task_by_id(task_id)` - Fetch single task with skills by ID
|
||||
- [x] 3.8.7 Use parameterized queries for task_id parameter
|
||||
|
||||
## 4. Caching Layer
|
||||
- [x] 4.1 Create `src/cache/query_cache.py` for database cache (12h TTL)
|
||||
- [x] 4.2 Create `src/cache/search_cache.py` for search results cache (60min TTL)
|
||||
- [x] 4.3 Implement QueryCache with cachetools.TTLCache:
|
||||
- [x] 4.3.1 Key: "all_capacities_with_competences"
|
||||
- [x] 4.3.2 TTL: 12 hours
|
||||
- [x] 4.3.3 Cache statistics (hits/misses)
|
||||
- [x] 4.4 Implement SearchCache with cachetools.TTLCache:
|
||||
- [x] 4.4.1 Key: search_id (UUID)
|
||||
- [x] 4.4.2 Value: JSON-serialized search results with task_id (null or task ID), filters dict
|
||||
- [x] 4.4.3 TTL: 60 minutes
|
||||
- [x] 4.4.4 Methods: store_search() returns search_id, get_search(search_id) returns dict, update_search() for adding filters
|
||||
- [x] 4.5 Ensure thread-safe cache operations
|
||||
|
||||
## 5. Matching Engine
|
||||
- [x] 5.1 Create `src/matching/matcher.py` for matching logic
|
||||
- [x] 5.2 Implement task analysis using Azure OpenAI (chat completion):
|
||||
- [x] 5.2.1 **NEW**: Create common internal function `_extract_requirements_from_description(description)` as single source of truth
|
||||
- [x] 5.2.2 Free-text mode: `analyze_free_text()` delegates to common extraction function
|
||||
- [x] 5.2.3 Structured mode: Parse provided parameters directly
|
||||
- [x] 5.2.4 Generate and return task analysis summary for free-text mode
|
||||
- [x] 5.2.5 Handle Azure API errors (timeouts, invalid JSON) with retry logic
|
||||
- [x] 5.2.6 **NEW**: Implement internal helper `extract_requirements_from_task(task_id)` - fetches task from DB and uses common extraction
|
||||
- [x] 5.2.7 **UPDATED**: Implement internal helper `extract_roles_from_task(description)` - returns ranked role list; primary role is roles[0] (if any)
|
||||
- [x] 5.3 Implement matching algorithm:
|
||||
- [x] 5.3.1 Semantic competence matching via Azure OpenAI embeddings
|
||||
- [x] 5.3.2 Role name matching (exact and similarity)
|
||||
- [x] 5.3.3 Availability date range filtering (open-ended capacities: missing end_date means available without limit)
|
||||
- [x] 5.3.4 Availability is NOT scored. If date_start/date_end are provided, availability is used as a filter only.
|
||||
- [x] 5.3.5 Apply configured weights (competence + role) to calculate overall score
|
||||
- [x] 5.4 Create `src/matching/scorer.py` for scoring logic
|
||||
- [x] 5.5 Implement categorical scoring (Top/Good/Partial/Low) with configurable thresholds
|
||||
- [x] 5.6 Generate match score and category for each capacity
|
||||
- [x] 5.7 Group results by category for storage
|
||||
|
||||
## 6. MCP Server Implementation
|
||||
- [x] 6.1 Create `src/mcp_server.py` with MCP SDK integration
|
||||
- [x] 6.2 Integrate Azure OpenAI client for LLM access (chat + embeddings)
|
||||
- [x] 6.3 Implement session state management (in-memory, per client session)
|
||||
- [x] 6.3.1 Store `last_requirements` for use with update_requirements
|
||||
- [x] 6.3.2 Clear session state when new search is executed
|
||||
- [x] 6.4 **NEW**: Implement MCP tool: `list_open_tasks` (Phase 0)
|
||||
- [x] 6.4.1 Tool parameters: limit (default: 20)
|
||||
- [x] 6.4.2 Query database for published tasks (status = "Veröffentlicht")
|
||||
- [x] 6.4.3 Sort by createddate DESC (newest first)
|
||||
- [x] 6.4.4 Format as markdown table: Task ID, Title, Created Date, Start/End Dates, Skills Count
|
||||
- [x] 6.4.5 Return table with next steps hints
|
||||
- [x] 6.5 **NEW**: Implement MCP tool: `get_task_details` (Phase 0)
|
||||
- [x] 6.5.1 Tool parameters: task_id (required)
|
||||
- [x] 6.5.2 Fetch task from database by ID
|
||||
- [x] 6.5.3 Extract ranked roles from description using `extract_roles_from_task()` helper; assign primary role as roles[0]
|
||||
- [x] 6.5.4 Format task details with metadata, description, DB skills, extracted role
|
||||
- [x] 6.5.5 Return formatted details with next steps hints
|
||||
- [x] 6.6 **NEW**: Implement MCP tool: `validate_task_requirements` (Phase 0)
|
||||
- [x] 6.6.1 Tool parameters: task_id (required)
|
||||
- [x] 6.6.2 Fetch task from database
|
||||
- [x] 6.6.3 Extract requirements from description using `extract_requirements_from_task()` helper
|
||||
- [x] 6.6.4 Compare DB skills vs extracted skills (set operations)
|
||||
- [x] 6.6.5 Compare DB dates vs extracted dates
|
||||
- [x] 6.6.6 Build comparison table: Type, In Description, In Database, Status
|
||||
- [x] 6.6.7 Generate summary with matched/missing counts
|
||||
- [x] 6.6.8 Return validation table (informational only, no modifications)
|
||||
- [x] 6.7 **NEW**: Implement MCP tool: `find_capacities_for_task` (Phase 0)
|
||||
- [x] 6.7.1 Tool parameters: task_id (required)
|
||||
- [x] 6.7.2 Fetch task from database
|
||||
- [x] 6.7.3 Extract requirements for validation display
|
||||
- [x] 6.7.4 Run validation internally (build comparison table)
|
||||
- [x] 6.7.5 Build search requirements using DB skills, extracted role, DB dates
|
||||
- [x] 6.7.6 Execute capacity search (same as find_matching_capacities)
|
||||
- [x] 6.7.7 Store in search cache with task_id field set to task ID
|
||||
- [x] 6.7.8 Return validation table + search results (combined response)
|
||||
- [x] 6.8 **EXISTING**: Implement MCP tool: `extract_requirements` (Phase 1)
|
||||
- [x] 6.8.1 Tool parameters: task_description (required), confirm_requirements (default: True)
|
||||
- [x] 6.8.2 Use Azure OpenAI (chat completion) to extract structured requirements from free text
|
||||
- [x] 6.8.3 Store extracted requirements in session state
|
||||
- [x] 6.8.4 Return JSON-formatted requirements
|
||||
- [x] 6.8.5 Include confirmation prompt if confirm_requirements=True
|
||||
- [x] 6.9 **EXISTING**: Implement MCP tool: `collect_structured_requirement_data` (Phase 1)
|
||||
- [x] 6.9.1 Tool parameters: role_name, competences, date_start, date_end (all optional), confirm_requirements (default: True)
|
||||
- [x] 6.9.2 Check completeness: require at least role_name AND competences
|
||||
- [x] 6.9.3 Prompt for missing required parameters with examples
|
||||
- [x] 6.9.4 Store collected requirements in session state
|
||||
- [x] 6.9.5 Return JSON-formatted requirements with confirmation prompt
|
||||
- [x] 6.10 **EXISTING**: Implement MCP tool: `update_requirements` (Phase 1)
|
||||
- [x] 6.10.1 Tool parameters: change_description (required), confirm_requirements (default: True)
|
||||
- [x] 6.10.2 Check if requirements exist in session state
|
||||
- [x] 6.10.3 Use Azure OpenAI (chat completion) to interpret change and update requirements
|
||||
- [x] 6.10.4 Update session state with modified requirements
|
||||
- [x] 6.10.5 Return updated JSON-formatted requirements with confirmation prompt
|
||||
- [x] 6.11 **EXISTING**: Implement MCP tool: `find_matching_capacities` (Phase 2)
|
||||
- [x] 6.11.1 Tool parameters: role_name (required), competences (required), date_start (optional), date_end (optional)
|
||||
- [x] 6.11.2 **NO extraction or interpretation** - accept only explicit structured parameters
|
||||
- [x] 6.11.3 Execute matching algorithm (uses Azure OpenAI embeddings for competence matching; score uses competences+role only; availability is filter-only)
|
||||
- [x] 6.11.4 Generate summary table with count by score category (markdown format)
|
||||
- [x] 6.11.5 Store complete results in SearchCache with task_id=null, empty filters dict
|
||||
- [x] 6.11.6 Return search_id + summary + Top category results (markdown table)
|
||||
- [x] 6.11.7 Clear session state after successful search
|
||||
- [x] 6.12 **EXISTING**: Implement MCP tool: `filter_search_results` (Phase 3)
|
||||
- [x] 6.12.1 Ensure fuzzywuzzy dependency is installed
|
||||
- [x] 6.12.2 Tool parameters: search_id (required), role_filter (optional), competence_filter (optional List[str]), availability_date_start (optional), availability_date_end (optional), min_similarity (default: 0.7)
|
||||
- [x] 6.12.3 **ALWAYS use original search_id as input** (not filtered search_id)
|
||||
- [x] 6.12.4 Validate at least one filter is provided
|
||||
- [x] 6.12.5 Retrieve original search results from SearchCache
|
||||
- [x] 6.12.6 Collect all results across all categories
|
||||
- [x] 6.12.7 Apply fuzzy matching: role_filter (fuzzy match on role_name), competence_filter (ALL competences must match at least one capacity competence)
|
||||
- [x] 6.12.8 Combine filters with AND logic
|
||||
- [x] 6.12.9 Re-categorize filtered results using Scorer
|
||||
- [x] 6.12.10 Generate unique filter_id (e.g., "filter-1", "filter-2")
|
||||
- [x] 6.12.11 Store filtered results under search_data["filters"][filter_id]
|
||||
- [x] 6.12.12 Update search cache with filter data
|
||||
- [x] 6.12.13 Return filter_id + search_id + filter summary + Top category results
|
||||
- [x] 6.13 **EXISTING**: Implement MCP tool: `get_results_by_category` (Phase 3)
|
||||
- [x] 6.13.1 Tool parameters: search_id (required), filter_id (optional), category (default: "Top"), page (default: 1), page_size (default: 20)
|
||||
- [x] 6.13.2 Retrieve search results from SearchCache by search_id
|
||||
- [x] 6.13.3 If filter_id provided, retrieve filtered results from search_data["filters"][filter_id]
|
||||
- [x] 6.13.4 If filter_id not provided, use original search results
|
||||
- [x] 6.13.5 Extract category results, apply pagination
|
||||
- [x] 6.13.6 Format as markdown table with full competence lists
|
||||
- [x] 6.13.7 Return paginated results with page info and next page hint
|
||||
- [x] 6.14 Format results as markdown tables with fields: id, owner_name, role_name, role_level, begin_date, end_date, competences
|
||||
- [x] 6.15 Show full competence lists (no truncation)
|
||||
- [x] 6.16 Add tool descriptions and parameter schemas for MCP client
|
||||
|
||||
> **Update (2026-02)**: Items previously described as "via MCP sampling" are now
|
||||
> implemented with **Azure OpenAI** (chat + embeddings) as part of the change
|
||||
> `replace-heuristics-with-azure-openai`.
|
||||
>
|
||||
> There is **no runtime MCP sampling dependency** and **no heuristic fallback**
|
||||
> path in the current implementation.
|
||||
|
||||
## 7. Integration & Entry Point
|
||||
- [x] 7.1 Update `main.py` to launch MCP server
|
||||
- [x] 7.2 Add command-line arguments for server configuration (port, debug mode, etc.)
|
||||
- [x] 7.3 Implement graceful startup and shutdown
|
||||
- [x] 7.4 Add logging configuration
|
||||
|
||||
## 8. Testing & Validation
|
||||
- [x] 8.1 Manual testing with sample task descriptions
|
||||
- [x] 8.2 Verify database connection and single join query execution
|
||||
- [x] 8.3 Test both input modes (free-text via Azure OpenAI and structured)
|
||||
- [x] 8.4 **NEW**: Test incremental structured mode (partial parameters with prompts)
|
||||
- [x] 8.5 Validate scoring and categorization with updated weights (0.8/0.2)
|
||||
- [x] 8.6 Test that availability does not affect score and is only applied as an optional filter
|
||||
- [x] 8.7 Test two-tier caching behavior (DB cache 12h, search cache 60min)
|
||||
- [x] 8.8 Verify read-only constraint (no writes to database)
|
||||
- [x] 8.9 Test error handling (database unavailable, Azure OpenAI timeout, invalid search_id)
|
||||
- [x] 8.10 Test search session management (search_id creation and retrieval)
|
||||
- [x] 8.11 Test pagination in get_results_by_category
|
||||
- [x] 8.12 Verify markdown table formatting and full competence display
|
||||
- [x] 8.13 **NEW**: Test filter_search_results with role filter only
|
||||
- [x] 8.14 **NEW**: Test filter_search_results with competence filter only
|
||||
- [x] 8.15 **NEW**: Test filter_search_results with combined filters
|
||||
- [x] 8.16 **NEW**: Test multi-step filtering (filter a filtered search)
|
||||
- [x] 8.17 **NEW**: Test fuzzy matching with various similarity thresholds
|
||||
|
||||
## 9. Documentation
|
||||
- [x] 9.1 Update `README.md` with project description and setup instructions
|
||||
- [x] 9.2 Document `database.toml` structure and required fields
|
||||
- [x] 9.3 Document three MCP tools and their parameters (find_matching_capacities, get_results_by_category, filter_search_results)
|
||||
- [x] 9.4 Add usage examples for both input modes (free-text and incremental structured)
|
||||
- [x] 9.5 Document configuration options (weights, thresholds, cache TTLs, fuzzy matching)
|
||||
- [x] 9.6 Add troubleshooting guide
|
||||
- [x] 9.7 Document Azure OpenAI architecture and client-side LLM requirement
|
||||
- [x] 9.8 Document pagination and search session workflow
|
||||
- [x] 9.9 Document availability behavior (display + date-range filtering; open-ended capacities)
|
||||
- [x] 9.10 **NEW**: Document fuzzy filtering workflow and multi-step filtering
|
||||
- [x] 9.11 **NEW**: Add examples of filter_search_results usage
|
||||
|
||||
## 10. Security & Configuration
|
||||
- [x] 10.1 Ensure `database.toml` is in `.gitignore`
|
||||
- [x] 10.2 Create `database.toml.example` as template (already done)
|
||||
- [x] 10.3 Validate secure credential handling
|
||||
- [x] 10.4 Document security best practices
|
||||
- [x] 10.5 Add warning about credential rotation if database.toml was committed
|
||||
|
||||
## 11. Architecture Documentation
|
||||
- [x] 11.1 Review architecture.md (Arc42 format) - already created with 9 ADRs
|
||||
- [x] 11.2 Validate ADRs (Architecture Decision Records) align with implementation
|
||||
- [x] 11.3 Update component diagrams if needed during implementation
|
||||
- [x] 11.4 Verify ADR-007 (Availability is a Filter, Not a Scoring Criterion) implementation
|
||||
- [x] 11.5 **NEW**: Verify ADR-008 (Incremental Structured Mode) implementation
|
||||
- [x] 11.6 **NEW**: Verify ADR-009 (Fuzzy Filtering) implementation
|
||||
- [x] 11.7 Document any additional architectural decisions made during development
|
||||
|
||||
## Notes
|
||||
- All database operations must be read-only
|
||||
- LLM access only via Azure OpenAI (chat + embeddings)
|
||||
- Two separate caches with different TTLs (DB: 12h, Search: 60min)
|
||||
- Single join query for all data retrieval
|
||||
- Markdown tables for all output
|
||||
- Full competence lists (no truncation)
|
||||
- Search sessions via search_id (60min expiry)
|
||||
- Initial implementation focuses on core functionality; refinements in future stages
|
||||
- Total: tracked via `openspec list` (this markdown count is not authoritative)
|
||||
+135
@@ -0,0 +1,135 @@
|
||||
# Design: Capacity→Task Matching and Embedding-based Inference
|
||||
|
||||
## Context
|
||||
|
||||
The system was originally optimized for task→capacity matching and relied on Azure chat completions for role/requirements extraction.
|
||||
|
||||
To enable a symmetrical workflow and reduce latency/cost, this change:
|
||||
|
||||
- adds tools to browse capacities and match a capacity against open tasks
|
||||
- reworks role and competence inference to use embeddings only (Data Lake vocabularies)
|
||||
- preloads and caches role/competence vocab embeddings on startup (with runtime on-demand fallback)
|
||||
- preloads and caches task text embeddings by `task_id`
|
||||
|
||||
## Key decisions
|
||||
|
||||
### A. Vocabulary-driven inference (roles + competences)
|
||||
|
||||
- Maintain two vocabularies sourced from the Data Lake:
|
||||
- roles: `teamlandkarte_v_capacity_roles_latest` filtered by `active=true` and `staffing_board_relevant=true`
|
||||
- competences: unique values from `beschaffungstool_kmp_skill_latest.skillname__c`
|
||||
- Generate embeddings for these vocabularies and store them in the local SQLite embedding cache.
|
||||
|
||||
### B. Task text embedding stored by task_id
|
||||
|
||||
To minimize embedding transaction costs for repeated analysis of the same task:
|
||||
|
||||
- build `task_text = f"{title}\n\n{description}"` (title optional)
|
||||
- embed `task_text` once
|
||||
- store that embedding under a stable key derived from `task_id` (and embedding model/dims)
|
||||
|
||||
**Explicit cache key formats**:
|
||||
|
||||
```python
|
||||
# Role vocabulary entries
|
||||
key = f"role_vocab:{model}:{dims}:{normalized_role_name}"
|
||||
|
||||
# Competence vocabulary entries
|
||||
key = f"comp_vocab:{model}:{dims}:{normalized_competence_name}"
|
||||
|
||||
# Task text embeddings by task_id
|
||||
key = f"task_id:{model}:{dims}:{task_id}"
|
||||
```
|
||||
|
||||
Where:
|
||||
- `model` = embedding model name (e.g., `text-embedding-3-large`)
|
||||
- `dims` = embedding dimensions (e.g., `3072`)
|
||||
- `normalized_role_name` = normalized role text (trimmed, whitespace-collapsed, lowercased)
|
||||
- `normalized_competence_name` = normalized competence text (trimmed, whitespace-collapsed, lowercased)
|
||||
- `task_id` = database task ID (string or integer, converted to string)
|
||||
|
||||
### C. Inference logic (embedding similarity)
|
||||
|
||||
- Role inference:
|
||||
- compute cosine similarity between `task_text_embedding` and all role vocabulary embeddings
|
||||
- return the single best matching role with similarity score as a Markdown table: `Role | Similarity`
|
||||
- Competence inference (for validation and display):
|
||||
- compute cosine similarity between `task_text_embedding` and all competence vocabulary embeddings
|
||||
- return top-N competences with similarity scores as a Markdown table: `Competence | Similarity` (descending order)
|
||||
|
||||
**Configuration** (under `[matching.inference]`):
|
||||
```toml
|
||||
[matching.inference]
|
||||
max_competences = 8 # default: 8; omit to disable limit (no max)
|
||||
min_similarity = 0.4 # default: 0.4; omit to disable threshold filtering
|
||||
```
|
||||
|
||||
**Configuration rules**:
|
||||
- If `max_competences` is not specified: no limit on number of competences returned.
|
||||
- If `min_similarity` is not specified: no threshold filtering applied (include all).
|
||||
- At least one of `max_competences` or `min_similarity` MUST be specified in config to avoid unbounded results.
|
||||
- Selection rule: if both are set, return top `max_competences` among those with similarity `>= min_similarity`.
|
||||
|
||||
### D. Component structure for vocabulary preload
|
||||
|
||||
**Component: VocabularyCache**
|
||||
- **Location**: `src/teamlandkarte_mcp/matching/vocabulary.py` (new module)
|
||||
- **Responsibilities**:
|
||||
- Fetch role + competence vocabularies from Data Lake via database client
|
||||
- Normalize and deduplicate vocabulary entries
|
||||
- Embed missing entries using batch + chunking via `SimilarityEngine.prefetch_embeddings(...)`
|
||||
- Store embeddings in SQLite cache with stable keys
|
||||
- Provide in-memory memoization for fast repeated inference during tool execution
|
||||
- **Initialization**: Called from `build_server()` after DB client + similarity engine initialization
|
||||
- **Refresh strategy**: On startup only; no TTL-based refresh during runtime
|
||||
|
||||
**Task embedding preload**:
|
||||
- Preload **all open tasks** on startup (no limit).
|
||||
- Build `task_text = title + "\n\n" + description` for each task (title optional).
|
||||
- Embed only cache-missing `task_id` vectors (check SQLite cache first).
|
||||
- Store into SQLite embedding cache with key format: `task_id:{model}:{dims}:{task_id}`.
|
||||
- Runtime fallback: if a task embedding is missing during tool execution, embed on-demand (strict semantics).
|
||||
|
||||
### E. Determinism and strict failure semantics
|
||||
|
||||
- Runs are deterministic given fixed models and cached embeddings.
|
||||
- Azure embedding failures fail the operation immediately.
|
||||
- The existing global embedding improvements (dedup + batch + layered caches) should be reused.
|
||||
|
||||
## Data flow
|
||||
|
||||
### 1) Vocabulary refresh
|
||||
|
||||
1. fetch role list + competence list from Data Lake
|
||||
2. normalize/deduplicate
|
||||
3. embed missing entries via Azure batch API
|
||||
4. store embeddings in SQLite cache
|
||||
|
||||
### 2) Task validation
|
||||
|
||||
1. fetch task (title+description) + DB skills
|
||||
2. compute/retrieve `task_text_embedding` by task_id
|
||||
3. infer:
|
||||
- top-K roles
|
||||
- top-K competences
|
||||
4. compare:
|
||||
- DB skills vs inferred competences (plus similarity thresholds)
|
||||
- dates (unchanged)
|
||||
|
||||
### 3) Capacity→task matching
|
||||
|
||||
1. load the target capacity (role + competences + availability)
|
||||
2. list open tasks
|
||||
3. compute/retrieve `task_text_embedding` for each task
|
||||
4. compute overall similarity (role + competences) between the capacity and each task
|
||||
5. categorize + return table-first results (same output conventions as existing matching tools)
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- Keep chat API for role inference: rejected (latency/cost + redundant).
|
||||
- Use raw string matching for vocab mapping: rejected (less robust than embeddings).
|
||||
|
||||
## Risks
|
||||
|
||||
- Vocabulary size (competences) may be large → requires batch embedding + caching and careful refresh strategy.
|
||||
- Model updates can change nearest neighbors → mitigate with pinned model/deployment and cache key including model/dims.
|
||||
+82
@@ -0,0 +1,82 @@
|
||||
# Change: Add capacity→task matching tools and embedding-based inference
|
||||
|
||||
## Why
|
||||
|
||||
The server currently supports **task→capacity matching** (`list_open_tasks`, `get_task_details`, `find_matching_capacities`, `find_capacities_for_task`). There is no symmetrical workflow to start from a capacity/person and discover relevant tasks.
|
||||
|
||||
Additionally, role/requirements inference and task validation still rely on the Azure chat API. This introduces latency/cost, complicates failure semantics, and is redundant given the system already uses embeddings + cosine similarity for matching.
|
||||
|
||||
## What changes
|
||||
|
||||
### New MCP tools
|
||||
|
||||
- `list_free_capacities(limit=20)`
|
||||
- Show the X most recent active capacities in a table-first Markdown output.
|
||||
- `get_capacity_details(capacity_id)`
|
||||
- Show a single capacity/person entry (same fields as the list, table-first).
|
||||
- `find_matching_tasks(capacity_id)`
|
||||
- Match a capacity against all open tasks based on role + competences.
|
||||
- Returns paginated Markdown table with task_id, title, required competences, availability, score, and category.
|
||||
|
||||
### Rework role + competence inference (remove chat API)
|
||||
|
||||
- Replace chat-based role inference with **embedding-based role inference**:
|
||||
- infer roles from a combined text: **task title (if present) + task description**
|
||||
- embed this combined text once and match against a pre-embedded **role vocabulary** from the Data Lake.
|
||||
- Replace chat-based competence extraction/validation with **embedding-based competence matching**:
|
||||
- embed the task text and match against a pre-embedded **competence vocabulary** from the Data Lake.
|
||||
- Store the embedding for `task_id` (combined title+description text) in the local SQLite embedding cache to minimize repeated embedding costs.
|
||||
|
||||
### Data Lake sourcing
|
||||
|
||||
- Competence vocabulary: `hive.tier1_open_lake.beschaffungstool_kmp_skill_latest` (existing source)
|
||||
- Role vocabulary: `teamlandkarte_v_capacity_roles_latest` via columns:
|
||||
- role name column: `name`
|
||||
- filters: `active=true` and `staffing_board_relevant=true`
|
||||
|
||||
### Azure OpenAI usage
|
||||
|
||||
- Azure embeddings remain required.
|
||||
- Azure chat API usage is removed completely once all dependent features are migrated.
|
||||
|
||||
## Impact
|
||||
|
||||
- **MCP API surface**: Adds 3 tools (capacity browsing + capacity→task matching).
|
||||
- **Business logic**: TaskAnalyzer and validation logic migrate to embeddings-only.
|
||||
- **Data access**: Requires new read queries to fetch role/competence vocabularies.
|
||||
- **Caching**: Reuses existing SQLite embedding cache; adds stable keys for task_id text embeddings.
|
||||
- **Docs**: README, assistant prompt, and architecture must be updated.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Changing scoring weights or category thresholds.
|
||||
- Building a full-text search for tasks/capacities.
|
||||
|
||||
## Open questions (need confirmation)
|
||||
|
||||
1. **“Most recent capacities”**: should ordering be by `begin_date` descending, by `end_date` descending, or by a DB “created/updated” timestamp (if available)?
|
||||
2. `list_free_capacities` “owner/team”: do you want **owner_name only**, or both `owner_name` and an explicit `team` field (if available in the view)?
|
||||
3. Do we keep the existing `infer_roles(...)` tool name/shape, but change its implementation to embeddings-only (recommended), or introduce a new tool and deprecate the old one?
|
||||
4. For embedding-based competence inference for validation: should the output be a **ranked list with similarity scores**, or a **binary chosen set** (top-N)? If top-N, what default N?
|
||||
|
||||
---
|
||||
|
||||
## Confirmed decisions
|
||||
|
||||
- `list_free_capacities`: order by **most recent `creation_date`**.
|
||||
- Capacity “Owner/Team”: use the `owner_name` column (contains the **team name**).
|
||||
- Role inference tool:
|
||||
- introduce a **new tool** and deprecate `infer_roles`
|
||||
- the new tool returns only the **single closest role** (no rank/rationale)
|
||||
- Competence inference:
|
||||
- return a maximum of **X** competences (**configurable**, default: 8)
|
||||
- apply a configurable **similarity score threshold** (default: 0.4)
|
||||
- if `max_competences` is not specified: no limit on number of competences
|
||||
- if `min_similarity` is not specified: no threshold filtering
|
||||
- at least one parameter must be specified to avoid unbounded results
|
||||
- if both are set: select **top X** among those **>= threshold**
|
||||
- Refresh strategy:
|
||||
- on startup only
|
||||
- only cache-missing roles/competences are embedded
|
||||
- embed **all open tasks** (combined title+description) on startup
|
||||
- if something is still missing at runtime, embed on-demand (strict semantics)
|
||||
+150
@@ -0,0 +1,150 @@
|
||||
# Spec Delta: Capacity→Task Matc### Requirement: Match tasks### Requirement: Role inference tool is embe### Requirement: Task validation uses embedding-based role and competence inference
|
||||
|
||||
Task validation SHALL use embedding similarity against role and competence vocabularies and SHALL NOT call the Azure chat API. Output SHALL include Markdown tables for inferred roles and competences with similarity scores.
|
||||
|
||||
#### Scenario: Validate task requirements without chat calls
|
||||
|
||||
- **WHEN** the client calls `validate_task_requirements(task_id=...)`
|
||||
- **THEN** the server infers the task's primary role via the role vocabulary (embeddings)
|
||||
- **AND** the output includes a Markdown table with columns `Role | Similarity` (single row)
|
||||
- **AND** the server infers competences via the competence vocabulary (embeddings) honoring configured max/threshold
|
||||
- **AND** the output includes a Markdown table with columns `Competence | Similarity` in descending order by similarity
|
||||
- **AND** the output remains table-first and includes inferred role/competences with similarity scoresy and returns a single closest role
|
||||
|
||||
Role inference SHALL be based on embedding similarity between a task's combined text and the Data Lake role vocabulary and SHALL return only the single closest role as a Markdown table.
|
||||
|
||||
#### Scenario: Infer primary role from task_id or task_text
|
||||
|
||||
- **WHEN** the client calls `infer_primary_role(task_id=...)`
|
||||
- **THEN** the system builds a combined text consisting of the task title (if present) and the task description
|
||||
- **AND** the combined text is embedded and compared against the available role list
|
||||
- **AND** the output is a Markdown table with columns `Role | Similarity` containing the single best matching role
|
||||
|
||||
- **WHEN** the client calls `infer_primary_role(task_text=...)`
|
||||
- **THEN** the system embeds the provided text and compares it against the available role list
|
||||
- **AND** the output is a Markdown table with columns `Role | Similarity` containing the single best matching role
|
||||
|
||||
- **AND** exactly one of `task_id` or `task_text` MUST be provided
|
||||
- **AND** the legacy `infer_roles(...)` tool MUST be removed completelyThe system SHALL expose a tool `find_matching_tasks(capacity_id)` that matches a capacity against all published tasks using role + competence similarity.
|
||||
|
||||
#### Scenario: Match tasks for a capacity using role + competence similarity
|
||||
|
||||
- **GIVEN** a capacity has a role name and a competence list
|
||||
- **AND** at least one open/published task exists in the Data Lake
|
||||
- **WHEN** the client calls `find_matching_tasks(capacity_id=<id>)`
|
||||
- **THEN** the server infers each task's primary role before computing similarity
|
||||
- **AND** the server infers each task's competences using embedding similarity with configured max/threshold
|
||||
- **AND** the server returns a Markdown table with columns: `task_id`, `Title`, `Required Competences`, `Availability`, `Score`, `Category`
|
||||
- **AND** results are sorted by similarity score in descending order
|
||||
- **AND** the output structure matches `find_matching_capacities` (summary counts table + results table)
|
||||
- **AND** the server returns categorized results (Top/Good/Partial/Low) using the same conventions as `find_matching_capacities`+ Embedding-based Inference
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: List free capacities
|
||||
|
||||
The system SHALL expose a tool `list_free_capacities(limit=20)` to list the most recent active capacities in a table-first Markdown format.
|
||||
|
||||
#### Scenario: List free capacities ordered by creation_date
|
||||
|
||||
- **WHEN** the client calls `list_free_capacities(limit=3)`
|
||||
- **THEN** the server returns a Markdown table with columns:
|
||||
- `capacity_id`
|
||||
- `Owner/Team` (sourced from the capacity’s `owner_name` column)
|
||||
- `Role`
|
||||
- `Competences`
|
||||
- `Availability`
|
||||
- **AND** the entries are ordered by `creation_date` descending
|
||||
- **AND** `Competences` is rendered as a comma-separated list (or `(none)`)
|
||||
- **AND** `Availability` is rendered as `begin_date .. end_date` (using `(open)` when the end date is missing)
|
||||
- **AND** when there are zero results, the output still contains a valid Markdown table (a dummy row is acceptable)
|
||||
|
||||
### Requirement: Get capacity details
|
||||
|
||||
The system SHALL expose a tool `get_capacity_details(capacity_id)` that returns a single capacity entry in the same table-first format as `list_free_capacities`.
|
||||
|
||||
#### Scenario: Get capacity details for a known capacity_id
|
||||
|
||||
- **WHEN** the client calls `get_capacity_details(capacity_id=<id>)`
|
||||
- **THEN** the server returns a table-first Markdown output containing the same fields as `list_free_capacities`
|
||||
- **AND** the output MAY include a “Next steps” section pointing to `find_matching_tasks(capacity_id=...)`
|
||||
|
||||
### Requirement: Match tasks for a capacity
|
||||
|
||||
The system SHALL expose a tool `find_matching_tasks(capacity_id, ...)` that matches a capacity against all published tasks using role + competence similarity.
|
||||
|
||||
#### Scenario: Match tasks for a capacity using role + competence similarity
|
||||
|
||||
- **GIVEN** a capacity has a role name and a competence list
|
||||
- **AND** at least one open/published task exists in the Data Lake
|
||||
- **WHEN** the client calls `find_matching_tasks(capacity_id=<id>)`
|
||||
- **THEN** the server infers each task’s primary role before computing similarity
|
||||
- **AND** the server returns categorized results (Top/Good/Partial/Low) using the same conventions as `find_matching_capacities`
|
||||
|
||||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: Role inference tool is embeddings-only and returns a single closest role
|
||||
|
||||
Role inference SHALL be based on embedding similarity between a task’s combined text and the Data Lake role vocabulary and SHALL return only the single closest role.
|
||||
|
||||
#### Scenario: Infer primary role from task_id or task_text
|
||||
|
||||
- **WHEN** the client calls `infer_primary_role(task_id=...)`
|
||||
- **THEN** the system builds a combined text consisting of the task title (if present) and the task description
|
||||
- **AND** the combined text is embedded and compared against the available role list
|
||||
- **AND** the output is the single best matching role
|
||||
|
||||
- **WHEN** the client calls `infer_primary_role(task_text=...)`
|
||||
- **THEN** the system embeds the provided text and compares it against the available role list
|
||||
- **AND** the output is the single best matching role
|
||||
|
||||
- **AND** exactly one of `task_id` or `task_text` MUST be provided
|
||||
- **AND** the legacy `infer_roles(...)` tool MUST be removed completely
|
||||
|
||||
### Requirement: Competence inference is configurable by max count and threshold
|
||||
|
||||
Competence inference for validation SHALL be derived via embedding similarity against the competence vocabulary and MUST support limiting by both maximum count and a minimum similarity threshold. Output SHALL be a Markdown table.
|
||||
|
||||
#### Scenario: Select top X competences above threshold
|
||||
|
||||
- **GIVEN** `max_competences = X` and `min_similarity = T` are configured
|
||||
- **WHEN** competences are inferred for a task
|
||||
- **THEN** only competences with similarity `>= T` are considered
|
||||
- **AND** from those, the top `X` competences by similarity are returned
|
||||
- **AND** the output is a Markdown table with columns `Competence | Similarity` in descending order by similarity
|
||||
|
||||
### Requirement: Task validation uses embedding-based role and competence inference
|
||||
|
||||
Task validation SHALL use embedding similarity against role and competence vocabularies and SHALL NOT call the Azure chat API.
|
||||
|
||||
#### Scenario: Validate task requirements without chat calls
|
||||
|
||||
- **WHEN** the client calls `validate_task_requirements(task_id=...)`
|
||||
- **THEN** the server infers the task’s primary role via the role vocabulary (embeddings)
|
||||
- **AND** the server infers competences via the competence vocabulary (embeddings) honoring configured max/threshold
|
||||
- **AND** the output remains table-first and includes inferred role/competences with similarity scores
|
||||
|
||||
### Requirement: Startup-only embedding preload with runtime fallback
|
||||
|
||||
The server SHOULD preload embeddings on startup to reduce runtime Azure calls, but MUST still embed on-demand for any cache misses encountered during tool execution.
|
||||
|
||||
#### Scenario: Preload only cache-missing embeddings and fall back at runtime
|
||||
|
||||
- **WHEN** the server starts
|
||||
- **THEN** it fetches role vocabulary + competence vocabulary from the Data Lake
|
||||
- **AND** it embeds only cache-missing role/competence vocabulary entries
|
||||
- **AND** it fetches **all open tasks** (no limit)
|
||||
- **AND** it builds combined task text (`title + "\n\n" + description`) and embeds only cache-missing task text embeddings keyed by task_id
|
||||
- **AND** during runtime, if an embedding is still missing, the server retrieves it on-demand (strict semantics)
|
||||
|
||||
## REMOVED Requirements
|
||||
|
||||
### Requirement: Chat-based extraction for roles and competences
|
||||
|
||||
The system SHALL NOT require Azure chat completions for role inference or task validation once this change is implemented.
|
||||
|
||||
#### Scenario: Azure chat API is not required for matching and validation
|
||||
|
||||
- **WHEN** a matching or validation tool is executed
|
||||
- **THEN** no Azure chat completion request is made
|
||||
- **AND** only Azure embeddings are used where LLM intelligence is required
|
||||
@@ -0,0 +1,149 @@
|
||||
# Tasks: Add capacity→task matching tools and embedding-based inference
|
||||
|
||||
> Change: `add-capacity-to-task-matching-tools`
|
||||
>
|
||||
> Status: **Implemented**
|
||||
|
||||
## Phase 1: Requirements + Configuration decisions
|
||||
|
||||
- [x] 1.1 Define `list_free_capacities` ordering precisely:
|
||||
- order by most recent `creation_date` (descending)
|
||||
- [x] 1.2 Confirm “Owner/Team” display:
|
||||
- use `owner_name` (contains the team name)
|
||||
- [x] 1.3 Introduce a new role inference tool and remove the old:
|
||||
- add `infer_primary_role(task_id? | task_text?)` that returns the single closest role as a Markdown table with `Role | Similarity` columns
|
||||
- remove `infer_roles(...)` MCP tool completely (no deprecation period)
|
||||
- remove `TaskAnalyzer.extract_ranked_roles()` method completely
|
||||
- update `extract_requirements()` and other callers to use new primary-role inference
|
||||
- [x] 1.4 Add configuration for competence inference:
|
||||
- `matching.inference.max_competences` (int, default: 8; if not specified, no limit)
|
||||
- `matching.inference.min_similarity` (float in [0,1], default: 0.4; if not specified, no threshold)
|
||||
- at least one parameter MUST be specified to avoid unbounded results
|
||||
- selection rule: pick top X among those >= threshold when both are set
|
||||
- [x] 1.5 Define startup-only refresh behavior:
|
||||
- on startup embed only cache-missing role/competence vocab entries
|
||||
- on startup embed **all open tasks** (title+description by task_id)
|
||||
- at runtime still embed on-demand for misses (strict failure semantics)
|
||||
|
||||
## Phase 2: Data Lake queries (roles, competences, capacities)
|
||||
|
||||
- [x] 2.0 Verify Data Lake schema:
|
||||
- confirm `teamlandkarte_v_capacity_roles_latest` has columns: `name`, `active`, `staffing_board_relevant`
|
||||
- confirm capacities view has `creation_date` column
|
||||
- [x] 2.1 Add `TrinoClient.get_recent_free_capacities(limit)` query returning:
|
||||
- `capacity_id`, `owner_name` (team), `role_name`, `creation_date`, `begin_date`, `end_date`, `competences[]`
|
||||
- ordering: `creation_date DESC`
|
||||
- [x] 2.2 Add `TrinoClient.get_capacity_by_id(capacity_id)` query returning the same fields.
|
||||
- [x] 2.3 Add `TrinoClient.get_all_role_names()`:
|
||||
- source: `teamlandkarte_v_capacity_roles_latest`
|
||||
- filter: `active=true AND staffing_board_relevant=true`
|
||||
- return unique, non-empty role names
|
||||
- [x] 2.4 Add `TrinoClient.get_all_competence_names()`:
|
||||
- source: `beschaffungstool_kmp_skill_latest.skillname__c`
|
||||
- return unique, non-empty competence names
|
||||
- [x] 2.5 Add unit tests for SQL read-only guard + query shape (where applicable).
|
||||
|
||||
## Phase 3: Startup embedding preload (vocabularies + tasks)
|
||||
|
||||
- [x] 3.1 Define stable cache-key scheme (explicit format as documented in design.md):
|
||||
- role vocab entries: `f"role_vocab:{model}:{dims}:{normalized_role_name}"`
|
||||
- competence vocab entries: `f"comp_vocab:{model}:{dims}:{normalized_competence_name}"`
|
||||
- task text embeddings by task_id: `f"task_id:{model}:{dims}:{task_id}"`
|
||||
- [x] 3.2 Implement `VocabularyCache` component (`src/teamlandkarte_mcp/matching/vocabulary.py`):
|
||||
- fetch role vocabulary + competence vocabulary from Trino
|
||||
- normalize/deduplicate
|
||||
- embed only cache-missing entries (batch + chunking via `SimilarityEngine.prefetch_embeddings(...)`)
|
||||
- store into SQLite embedding cache
|
||||
- keep in-memory memoization for fast repeated inference
|
||||
- invoked from `build_server()` after DB + similarity engine init
|
||||
- [x] 3.3 Implement a “task embedding preload” routine executed on startup:
|
||||
- fetch **all open tasks** (no limit)
|
||||
- build `task_text = title + "\n\n" + description` (title optional)
|
||||
- embed only cache-missing `task_id` vectors
|
||||
- store into SQLite embedding cache
|
||||
- [x] 3.4 Runtime fallback:
|
||||
- if a role/competence/task embedding is missing during a tool call, embed on-demand (strict semantics)
|
||||
- [x] 3.5 Tests:
|
||||
- preload embeds only cache misses
|
||||
- preload writes to SQLite
|
||||
- missing-at-runtime triggers on-demand embedding
|
||||
|
||||
## Phase 4: Embedding-only inference (roles + competences)
|
||||
|
||||
- [x] 4.1 Implement primary role inference against role vocabulary:
|
||||
- input: combined `title+description` (or description-only for ad-hoc)
|
||||
- output: single closest role with similarity score as Markdown table: `Role | Similarity`
|
||||
- [x] 4.2 Add a new MCP tool:
|
||||
- `infer_primary_role(task_id? | task_text?)`
|
||||
- returns: Markdown table `Role | Similarity` (single best matching role)
|
||||
- validation: exactly one of `task_id` or `task_text` must be provided
|
||||
- [x] 4.3 Remove deprecated tools and methods:
|
||||
- remove `infer_roles(...)` MCP tool completely
|
||||
- remove `TaskAnalyzer.extract_ranked_roles()` method completely
|
||||
- [x] 4.4 Update callers to use new primary-role inference:
|
||||
- update `get_task_details` to use `infer_primary_role`
|
||||
- update `extract_requirements()` to use new primary-role inference
|
||||
- update any internal helpers relying on `TaskAnalyzer.extract_ranked_roles()`
|
||||
|
||||
## Phase 5: Rework task validation (embedding-only)
|
||||
|
||||
- [x] 5.1 Replace chat-based extraction in `validate_task_requirements` / `extract_requirements_from_task` with embedding-based inference:
|
||||
- roles: infer primary role via role vocabulary (Markdown table output)
|
||||
- competences: infer competences via competence vocabulary with max X and threshold (Markdown table output)
|
||||
- [x] 5.2 Update output format:
|
||||
- keep current table-first display
|
||||
- add inferred primary role as Markdown table: `Role | Similarity` (single row)
|
||||
- add inferred competences as Markdown table: `Competence | Similarity` (descending order by similarity)
|
||||
- [x] 5.3 Tests:
|
||||
- validation uses embeddings path only (no chat calls)
|
||||
- respects max X and threshold rules
|
||||
- output includes role and competence similarity tables
|
||||
|
||||
## Phase 6: New MCP tools (capacity browsing + capacity→task matching)
|
||||
|
||||
- [x] 6.1 Implement `list_free_capacities(limit=20)`:
|
||||
- ordering: `creation_date DESC`
|
||||
- Markdown table columns: `capacity_id`, `Owner/Team`, `Role`, `Competences`, `Availability`
|
||||
- stable empty-table handling (dummy row)
|
||||
- [x] 6.2 Implement `get_capacity_details(capacity_id)`:
|
||||
- table-first, same fields as list
|
||||
- include a “Next steps” pointer to `find_matching_tasks(capacity_id=...)`
|
||||
- [x] 6.3 Implement `find_matching_tasks(capacity_id)`:
|
||||
- signature: takes only `capacity_id` (no optional params)
|
||||
- fetch capacity + all open tasks
|
||||
- infer each task’s primary role via embedding similarity (title+description)
|
||||
- infer each task’s competences via embedding similarity (title+description) with max/threshold
|
||||
- compute a similarity score (role + competences) and categorize results
|
||||
- return output as Markdown table with columns:
|
||||
- `task_id`, `Title`, `Required Competences`, `Availability`, `Score`, `Category`
|
||||
- results are sorted by similarity score (descending)
|
||||
- output structure matches `find_matching_capacities` (includes summary counts table + results table for first non-empty category)
|
||||
- paginated via same search session conventions used by `find_matching_capacities`
|
||||
- [x] 6.4 Tests for new tools:
|
||||
- `list_free_capacities`: ordering by `creation_date DESC`, empty-table handling
|
||||
- `get_capacity_details`: not-found handling, "Next steps" section
|
||||
- `find_matching_tasks`: categorization (Top/Good/Partial/Low), search session conventions, table structure, score sorting
|
||||
|
||||
## Phase 7: Remove Azure chat API
|
||||
|
||||
- [x] 7.1 Remove chat config fields and environment variables if no longer needed (`AZURE_OPENAI_LLM_API_KEY`, chat deployment, chat codepaths).
|
||||
- [x] 7.2 Delete/retire chat client methods and `TaskAnalyzer` chat-based methods.
|
||||
- [x] 7.3 Update docs to remove chat setup instructions.
|
||||
- [x] 7.4 Ensure all gates pass (ruff/mypy/pytest).
|
||||
|
||||
## Phase 8: Documentation + assistant prompt + architecture
|
||||
|
||||
- [x] 8.1 Update `README.md`:
|
||||
- describe new tools and capacity→task workflow
|
||||
- document startup embedding preload and inference config knobs
|
||||
- [x] 8.2 Update `docs/assistant_system_prompt.md`:
|
||||
- add guidance for new tools and matching direction
|
||||
- update role/competence inference description (embeddings-only)
|
||||
- [x] 8.3 Update architecture docs (Arc42 / design docs):
|
||||
- reflect removal of chat API
|
||||
- describe vocab/task embedding preload and cache keys
|
||||
|
||||
## Phase 9: OpenSpec hygiene
|
||||
|
||||
- [x] 9.1 Update spec deltas to reflect confirmed tool shapes + config.
|
||||
- [x] 9.2 Run `openspec validate add-capacity-to-task-matching-tools --strict`.
|
||||
+262
@@ -0,0 +1,262 @@
|
||||
# Change Proposal: Improve Task Details, Role Inference, Guided Requirements Capture, and Search Reliability
|
||||
|
||||
- **Change ID**: `improve-task-details-role-inference-and-guided-flow`
|
||||
- **Status**: Implemented
|
||||
- **Target**: `teamlandkarte-mcp`
|
||||
- **Author**: (to fill)
|
||||
- **Date**: 2026-02-12
|
||||
|
||||
## Summary
|
||||
|
||||
This proposal refines several MCP tools and interaction flows to improve usability in Cherry Studio and other MCP clients:
|
||||
|
||||
1. `get_task_details` should return a concise markdown summary (table) of the task plus description below.
|
||||
2. Introduce a dedicated tool `infer_roles` for role inference from either a DB task ID or free-text description.
|
||||
3. `validate_task_requirements` should return a single comparison table plus a short written summary.
|
||||
4. When a user requests a person search without providing project/task details, switch to a **step-by-step guided capture** (no "all questions at once").
|
||||
5. Enforce a **mandatory confirmation step** before triggering matching, for both free-text and guided capture / structured entry.
|
||||
6. Fix a suspected competence matching regression (cloud expert + AWS scoring ends up `Low`), likely caused by extraction/matching issues.
|
||||
7. Improve search session robustness for pagination/filtering:
|
||||
- search tools emit a JSON block containing `search_id`/`filter_id`
|
||||
- non-UUID `search_id` inputs are rejected early with a clear message
|
||||
- internal tracking of the latest search id (session state) for diagnostics only; **no** "get last search id" tool is provided
|
||||
8. Update documentation, architecture, and design markdown accordingly.
|
||||
|
||||
## Motivation
|
||||
|
||||
- Current outputs are verbose and inconsistent across tools.
|
||||
- Role inference exists but is not clearly available as an explicit tool.
|
||||
- Guided capture and confirmation are required to prevent accidental matching runs.
|
||||
- Cache invalidation/expiry breaks multi-step flows (`filter_search_results`, pagination, etc.).
|
||||
|
||||
## Goals
|
||||
|
||||
- Produce stable, markdown-first outputs that are easy to consume in chat UIs.
|
||||
- Make role inference available as an explicit, reusable capability for both DB-backed tasks and free-text.
|
||||
- Improve correctness and robustness of role/requirements extraction and semantic matching via the server’s Azure OpenAI integration.
|
||||
- Ensure multi-step search workflows remain functional for realistic time spans.
|
||||
- Remove artificial constraints on the number of exposed MCP tools.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Re-introducing Hive/PyHive support.
|
||||
- Changing the underlying DB schema.
|
||||
- Replacing LLM-based role inference with a numeric-only scorer (categorical matching output should remain the main UX).
|
||||
|
||||
## Proposed Changes
|
||||
|
||||
### 1) `get_task_details` output format
|
||||
|
||||
Change `get_task_details(task_id)` to return:
|
||||
|
||||
1. A markdown table with columns:
|
||||
- Task ID
|
||||
- Title
|
||||
- Created (date only, no time)
|
||||
- Start
|
||||
- End
|
||||
- Competences (comma-separated, no numbering)
|
||||
- Inferred Role
|
||||
2. Below the table:
|
||||
- The task description (plain markdown, no table)
|
||||
|
||||
Notes:
|
||||
- "Inferred Role" is derived using `infer_roles` (see section 2). If no role is inferred, display `(none)`.
|
||||
|
||||
### 2) Dedicated role inference tool: `infer_roles`
|
||||
|
||||
Add a tool to infer roles from either:
|
||||
|
||||
- A DB task via `task_id`, or
|
||||
- A free-text `task_description`
|
||||
|
||||
Signature:
|
||||
|
||||
- `infer_roles(task_id: Optional[str] = None, task_description: Optional[str] = None, limit: int = 5) -> str`
|
||||
|
||||
Output:
|
||||
- A markdown table containing all recognized roles, each with:
|
||||
- Rank
|
||||
- Role
|
||||
- Rationale
|
||||
|
||||
Behavior:
|
||||
- Exactly one of `task_id` or `task_description` must be provided.
|
||||
- Uses Azure OpenAI chat completion for role inference (no heuristic fallback).
|
||||
|
||||
### 3) `validate_task_requirements` output format
|
||||
|
||||
Change output to:
|
||||
|
||||
1. A single markdown comparison table that contrasts DB vs extracted information.
|
||||
2. Table must include:
|
||||
- Task ID
|
||||
- Title
|
||||
- Time ranges (DB vs extracted)
|
||||
- Competences (DB vs extracted)
|
||||
3. Below the table, a short written summary, including:
|
||||
- Skill overlap and notable differences
|
||||
- Any missing DB fields (dates/skills)
|
||||
|
||||
### 4) Step-by-step guided capture mode (step-specific tools)
|
||||
|
||||
Introduce a guided capture workflow for the case:
|
||||
|
||||
> User requests people/capacity search but provides insufficient task/project details.
|
||||
|
||||
The guided flow should collect input sequentially using step-specific tools (to avoid free-text ambiguity):
|
||||
|
||||
1. Task description
|
||||
2. Role
|
||||
3. Time range
|
||||
4. Competences
|
||||
|
||||
Implementation idea:
|
||||
- Add a server-side state machine stored in `SessionState`.
|
||||
- Add tools such as:
|
||||
- `start_guided_capture()`
|
||||
- `guided_set_description(description: str)`
|
||||
- `guided_set_role(role_name: str)`
|
||||
- `guided_set_time_range(date_start: Optional[str] = None, date_end: Optional[str] = None)`
|
||||
- `guided_set_competences(competences: list[str])`
|
||||
|
||||
### 5) Enforce confirmation before matching (shared mechanism, hard-gated)
|
||||
|
||||
`extract_requirements` and structured/guided capture must not directly trigger matching.
|
||||
|
||||
Instead, both should:
|
||||
- produce a "pending" `Requirements` object, and
|
||||
- require a confirmation step **before matching tools can execute**.
|
||||
|
||||
Hard enforcement:
|
||||
- `find_matching_capacities` (and any DB-backed matching tool) must refuse to run unless requirements have been explicitly confirmed **or** confirmation is auto-skipped by configuration (see below).
|
||||
|
||||
Config key (name + location):
|
||||
- Add a matching section flag in `database.toml`:
|
||||
|
||||
```toml
|
||||
[matching]
|
||||
require_confirmation = true
|
||||
```
|
||||
|
||||
- Default: `true` (strict).
|
||||
- When `false`: pending requirements are treated as auto-confirmed (skip confirmation).
|
||||
|
||||
Implementation idea:
|
||||
- Keep `confirm_requirements` parameters for backwards compatibility, but route the behavior through a shared internal helper and a shared tool:
|
||||
- internal: `_set_pending_requirements(req)`
|
||||
- tool: `confirm_requirements(confirm: bool = True)`
|
||||
|
||||
### 6) Fix competence matching producing incorrect `Low` results
|
||||
|
||||
Investigate and fix why "Cloud expert" + "AWS" competences yields `Low` across all capacities.
|
||||
|
||||
Hypotheses:
|
||||
- Requirements extraction produces empty or wrong competences.
|
||||
- Competence normalization / matching thresholds could be misapplied.
|
||||
|
||||
Deliverables:
|
||||
- Add a regression test reproducing the AWS case.
|
||||
- Ensure competence scoring uses the extracted competences correctly.
|
||||
|
||||
Capacity scoring UX note:
|
||||
- Keep numeric scores (`overall_score`, `competence_score`, `role_score`) visible.
|
||||
- Emphasize categorical results (`Top`/`Good`/`Partial`/`Low`) as the primary interpretation in docs and tables.
|
||||
|
||||
### 7) Fix search cache expiry / invalid IDs in multi-step tools
|
||||
|
||||
Observed:
|
||||
- `filter_search_results` and `get_results_by_category` sometimes return
|
||||
"Unknown or expired search_id".
|
||||
|
||||
Additional observed behavior in chat clients:
|
||||
- Some clients shorten, reformat, or accidentally reuse a previous `search_id`
|
||||
when the user requests another category (e.g. "show Good results").
|
||||
|
||||
Assumption:
|
||||
- The server process is continuous (no restart between tool calls).
|
||||
|
||||
Likely causes to verify (not only TTL):
|
||||
- **Eviction due to `[cache].max_size`** (entry removed even within TTL).
|
||||
- Search IDs **not stored** on all paths (serialization error, early return, etc.).
|
||||
- **ID formatting/copy issues** (backticks/whitespace) in client UIs.
|
||||
- Hidden **process restarts / multiple instances** in the client environment.
|
||||
|
||||
Proposed fixes:
|
||||
- Verify `SearchCache` TTL configuration and access.
|
||||
- Verify and test behavior under max-size eviction.
|
||||
- Ensure IDs are stored on every `store_search` call.
|
||||
- Add troubleshooting guidance to copy IDs without surrounding formatting.
|
||||
- Ensure search tools return a machine-readable JSON block containing IDs:
|
||||
- `find_matching_capacities` returns `{search_id, filter_id=null}`
|
||||
- `filter_search_results` returns `{search_id, filter_id}`
|
||||
- `get_results_by_category` echoes `{search_id, filter_id, category, page, ...}`
|
||||
- Reject non-UUID `search_id` values early (clear message: expected UUID).
|
||||
- Track latest `search_id` server-side (session state) for diagnostics only; **no** "get last search id" tool is provided.
|
||||
|
||||
### 8) Documentation updates
|
||||
|
||||
Update:
|
||||
- `README.md`
|
||||
- `docs/troubleshooting.md`
|
||||
- OpenSpec design/architecture docs
|
||||
|
||||
…to reflect:
|
||||
- new/changed tool outputs
|
||||
- `infer_roles`
|
||||
- guided capture step tools
|
||||
- confirmation requirement
|
||||
- role inference and capacity scoring UX (categorical)
|
||||
- cache behavior expectations
|
||||
|
||||
## Implementation Task List
|
||||
|
||||
### A. Tool output refactors
|
||||
|
||||
1. Update `get_task_details` to produce the new table-first format.
|
||||
2. Implement `infer_roles` and reuse it from `get_task_details`.
|
||||
3. Update `validate_task_requirements` output as specified.
|
||||
4. Add/adjust markdown table helpers if needed (column wrapping, newline handling).
|
||||
|
||||
### B. Role inference and extraction robustness
|
||||
|
||||
5. Review role/requirements extraction prompts and output validation.
|
||||
6. Add regression tests for:
|
||||
- role inference
|
||||
- AWS/cloud expert competence matching not collapsing to `Low`
|
||||
|
||||
### C. Guided capture + confirmation flow
|
||||
|
||||
7. Design a minimal state machine for guided capture in `SessionState`.
|
||||
8. Implement step-specific guided tools:
|
||||
- `start_guided_capture`
|
||||
- `guided_set_description`
|
||||
- `guided_set_role`
|
||||
- `guided_set_time_range`
|
||||
- `guided_set_competences`
|
||||
9. Add `matching.require_confirmation` to the configuration model + loader (default: `true`).
|
||||
10. Implement shared confirmation tool `confirm_requirements(confirm: bool = True)`.
|
||||
11. Refactor `extract_requirements` and structured/guided collection tools to set pending requirements and instruct confirmation.
|
||||
12. Refactor matching tools to hard-enforce confirmation unless `matching.require_confirmation = false`.
|
||||
|
||||
### D. Search cache reliability
|
||||
|
||||
13. Diagnose and fix `SearchCache` invalidation/expiry issues.
|
||||
14. Add tests covering:
|
||||
- storing search results
|
||||
- subsequent filtering
|
||||
- pagination
|
||||
- TTL behavior
|
||||
- max-size eviction behavior
|
||||
15. Add (optional) diagnostic support for identifying server instance continuity.
|
||||
|
||||
### E. Documentation and specs
|
||||
|
||||
16. Update `README.md` for the new workflow and new tools.
|
||||
17. Update `docs/troubleshooting.md` with Cherry Studio guidance for multi-step flows (including ID copy hygiene and cache eviction notes).
|
||||
18. Update OpenSpec design and architecture notes for guided capture, confirmation, and categorical-first scoring (numeric scores remain visible).
|
||||
|
||||
## Open Questions / Clarifications Needed
|
||||
|
||||
- (resolved) `infer_roles` output should be `Rank | Role | Rationale` only (no numeric score).
|
||||
- (resolved) Guided time ranges should allow open-ended ranges (either start or end may be omitted).
|
||||
+128
@@ -0,0 +1,128 @@
|
||||
# Tasks: Improve Task Details, Role Inference, Guided Requirements Capture, and Search Reliability
|
||||
|
||||
> Change: `improve-task-details-role-inference-and-guided-flow`
|
||||
>
|
||||
> Notes:
|
||||
> - All code/doc/spec content stays in English.
|
||||
> - Tool count is **not** limited.
|
||||
> - Confirmation is **hard-gated** by default; can be auto-skipped via config.
|
||||
|
||||
## Status
|
||||
|
||||
- ✅ **Implemented** (core behavior + UX contracts shipped).
|
||||
- 📝 Some follow-up items remain for additional hardening/coverage (diagnostics/logging and a few missing tests), but they do not block the change being considered implemented.
|
||||
|
||||
## 1. Proposal Alignment & Baseline
|
||||
|
||||
- [x] 1.1 Re-scan current tools in `src/teamlandkarte_mcp/mcp_server.py` and confirm required changes vs. current behavior.
|
||||
- [ ] 1.2 Confirm existing `SearchCache` semantics (`ttl_minutes`, `max_size`) and how eviction is handled.
|
||||
- [x] 1.3 Add/update a short architectural note: numeric scores remain visible, categories are primary.
|
||||
|
||||
## 2. Configuration: hard confirmation gate (+ optional auto-skip)
|
||||
|
||||
- [x] 2.1 Extend matching config model in `src/teamlandkarte_mcp/config.py`:
|
||||
- add `require_confirmation: bool = True` under `[matching]`.
|
||||
- [x] 2.2 Update TOML loading/validation to accept `[matching].require_confirmation`.
|
||||
- [x] 2.3 Update `database.toml.example` to document `require_confirmation` (default `true`).
|
||||
- [ ] 2.4 Add tests for config parsing (default true, explicit false).
|
||||
|
||||
## 3. Session State: pending + confirmed requirements
|
||||
|
||||
- [x] 3.1 Extend `SessionState` in `src/teamlandkarte_mcp/mcp_server.py`:
|
||||
- pending requirements object
|
||||
- confirmed requirements object (or a confirmed flag/version)
|
||||
- guided capture state (see section 6)
|
||||
- [x] 3.2 Implement internal helper(s):
|
||||
- `_set_pending_requirements(req: Requirements)`
|
||||
- `_require_confirmed_requirements_or_throw()` (implemented as `_require_confirmed_or_auto`)
|
||||
- [x] 3.3 Implement tool: `confirm_requirements(confirm: bool = True)`.
|
||||
- [x] 3.4 Update `extract_requirements`, `collect_structured_requirement_data`, `update_requirements`:
|
||||
- always write **pending** requirements
|
||||
- never implicitly run matching
|
||||
- include next-step guidance to call `confirm_requirements()`
|
||||
- [x] 3.5 Update `find_matching_capacities` (and DB-backed matching tools) to **hard-enforce** confirmation when `matching.require_confirmation = true`.
|
||||
- [x] 3.6 Auto-skip behavior:
|
||||
- when `matching.require_confirmation = false`, treat pending requirements as confirmed (no refusal).
|
||||
- [x] 3.7 Add unit tests for confirmation gate behavior.
|
||||
|
||||
## 4. Tool: `infer_roles` (ranked roles, no score column)
|
||||
|
||||
- [x] 4.1 Add tool `infer_roles(task_id: Optional[str] = None, task_description: Optional[str] = None, limit: int = 5)`.
|
||||
- [x] 4.2 Enforce input rules:
|
||||
- exactly one of `task_id` or `task_description` must be provided.
|
||||
- [x] 4.3 If `task_id` provided:
|
||||
- fetch task from DB, use its description.
|
||||
- [x] 4.4 Use `TaskAnalyzer.extract_ranked_roles(description, limit=...)`.
|
||||
- [x] 4.5 Output a markdown table with columns:
|
||||
- `Rank | Role | Rationale`
|
||||
- [ ] 4.6 Add tests for both modes (DB id mocked + free-text).
|
||||
|
||||
## 5. Tool output refactors
|
||||
|
||||
### 5.1 `get_task_details`
|
||||
|
||||
- [x] 5.1.1 Replace current multi-section output with:
|
||||
- a single markdown table: `Task ID | Title | Created (date only) | Start | End | Competences | Inferred Role`
|
||||
- below: task description
|
||||
- [x] 5.1.2 Created date formatting: date-only (YYYY-MM-DD).
|
||||
- [x] 5.1.3 Competences formatting: comma-separated, no numbering.
|
||||
- [x] 5.1.4 Inferred role:
|
||||
- derive from role inference (reuse same analyzer call used by `infer_roles`)
|
||||
- show `(none)` if empty
|
||||
- [ ] 5.1.5 Add/update tests asserting output format.
|
||||
|
||||
### 5.2 `validate_task_requirements`
|
||||
|
||||
- [x] 5.2.1 Replace current multi-table output with a **single** comparison table.
|
||||
- [x] 5.2.2 Add a short textual summary below the table.
|
||||
- [ ] 5.2.3 Add/update tests asserting table shape + summary presence.
|
||||
|
||||
## 6. Guided capture: step-specific tools
|
||||
|
||||
- [x] 6.1 Define guided capture state model (internal only).
|
||||
- [x] 6.2 Implement tool `start_guided_capture()`.
|
||||
- [x] 6.3 Implement tool `guided_set_description(description: str)`.
|
||||
- [x] 6.4 Implement tool `guided_set_role(role_name: str)`.
|
||||
- [x] 6.5 Implement tool `guided_set_time_range(date_start: Optional[str] = None, date_end: Optional[str] = None)`.
|
||||
- [x] 6.6 Implement tool `guided_set_competences(competences: list[str])`.
|
||||
- [ ] 6.7 Add tests covering the step transitions and open-ended ranges.
|
||||
|
||||
## 7. Competence matching regression (AWS/cloud)
|
||||
|
||||
- [x] 7.1 Reproduce reported case with a focused test (added deterministic similarity fallback to avoid collapse).
|
||||
- [x] 7.2 Inspect `TaskAnalyzer` heuristic competence extraction and normalization.
|
||||
- [x] 7.3 Inspect matching/scoring pipeline.
|
||||
- [x] 7.4 Fix competence similarity computation so obvious matches ("AWS" vs "AWS") do not collapse.
|
||||
- [ ] 7.5 Add regression test(s) to prevent reintroduction.
|
||||
|
||||
## 8. Search cache reliability
|
||||
|
||||
- [ ] 8.1 Add debug-level logging (stderr-safe) around:
|
||||
- `SearchCache.store_search` creation
|
||||
- `SearchCache.get` misses (include age/ttl if available)
|
||||
- eviction events (max_size)
|
||||
- [ ] 8.2 Verify `max_size` eviction behavior in `SearchCache` implementation.
|
||||
- [ ] 8.3 Add tests:
|
||||
- store -> filter -> paginate within TTL
|
||||
- behavior under forced max-size eviction
|
||||
- [ ] 8.4 (Optional) Add a diagnostic tool returning `server_instance_id` to confirm process continuity during client tests.
|
||||
- [x] 8.5 Update troubleshooting docs with:
|
||||
- ID copy hygiene (avoid extra backticks/whitespace)
|
||||
- max_size eviction symptom/fix
|
||||
|
||||
## 9. Documentation updates
|
||||
|
||||
- [x] 9.1 Update `README.md`.
|
||||
- [x] 9.2 Update `docs/troubleshooting.md`.
|
||||
- [x] 9.3 Update OpenSpec design/architecture docs under:
|
||||
- `openspec/changes/add-capacity-matching-mcp-server/` (historical baseline)
|
||||
|
||||
## 10. QA / Validation
|
||||
|
||||
- [x] 10.1 Run full test suite.
|
||||
- [ ] 10.2 Manually validate in Cherry Studio (guide):
|
||||
- guided capture step tools
|
||||
- confirmation gating (on/off via config)
|
||||
- infer_roles output
|
||||
- get_task_details formatting
|
||||
- search cache persistence across multi-step tools
|
||||
+99
@@ -0,0 +1,99 @@
|
||||
# CHANGES_APPLIED — Refine Matching and Filtering
|
||||
|
||||
This file tracks which items from `openspec/changes/refine-matching-and-filtering/` have been implemented in the codebase and documentation.
|
||||
|
||||
## Status
|
||||
|
||||
- Change proposal: **Implemented**
|
||||
- Tests: **Passing** (`uv run pytest -q`)
|
||||
- Lint: **Passing** (`uv run ruff check src/teamlandkarte_mcp`)
|
||||
|
||||
## Implemented requirements (high level)
|
||||
|
||||
1. **Title-first role inference with description fallback**
|
||||
- Role inference prefers task title text; if title missing/empty, falls back to full task text.
|
||||
- Competence inference continues to use the full task text (title + description).
|
||||
|
||||
2. **Separate embedding cache keys for role vs competence**
|
||||
- Embeddings are cached separately for:
|
||||
- task role embeddings vs task competence embeddings
|
||||
- capacity role embeddings vs capacity competence embeddings
|
||||
|
||||
3. **Filtering improvements**
|
||||
- `filter_search_results(...)` works symmetrically for both directions:
|
||||
- task → capacity (“capacity_search”)
|
||||
- capacity → task (“task_search”)
|
||||
- Added full-coverage availability filtering via `is_fully_available`.
|
||||
- Added task-search specific filters:
|
||||
- `task_text_filter`
|
||||
- `task_competence_filter`
|
||||
|
||||
4. **Pagination and category browsing**
|
||||
- `get_results_by_category(...)` supports both search types.
|
||||
- Added new category bucket: **Irrelevant** (`overall_score < matching.thresholds.low`).
|
||||
|
||||
5. **Availability display refinement**
|
||||
- Availability is displayed as **overlap percentage** with the reference range.
|
||||
|
||||
6. **Result table column consistency**
|
||||
- Matching outputs show **Role Score / Competence Score / Overall Score** consistently.
|
||||
- Removed `Level` column from capacity-search matching result tables (Level remains in listing/inspection tools).
|
||||
|
||||
7. **Bug fixes**
|
||||
- Fixed task-search score display bug by standardizing the stored score key to `overall_score`.
|
||||
|
||||
## Code changes (by area)
|
||||
|
||||
### Matching + scoring
|
||||
|
||||
- `src/teamlandkarte_mcp/matching/scorer.py`
|
||||
- Added **Irrelevant** categorization for scores below `matching.thresholds.low`.
|
||||
|
||||
- `src/teamlandkarte_mcp/matching/matcher.py`
|
||||
- Added **Irrelevant** bucket to `by_category`.
|
||||
|
||||
### Embedding cache separation
|
||||
|
||||
- `src/teamlandkarte_mcp/matching/vocabulary.py`
|
||||
- Added stable cache keys and separate ensure methods:
|
||||
- `ensure_task_role_embedding` / `ensure_task_competence_embedding`
|
||||
- `ensure_capacity_role_embedding` / `ensure_capacity_competence_embedding`
|
||||
- Retained legacy `ensure_task_embedding` as deprecated.
|
||||
|
||||
### MCP tools + search payload + filtering
|
||||
|
||||
- `src/teamlandkarte_mcp/mcp_server.py`
|
||||
- Restored MCP tool `infer_primary_role` (title-first behavior).
|
||||
- Role inference uses title-first text; competence inference uses full text.
|
||||
- Matching payload stores:
|
||||
- `search_type: "capacity_search" | "task_search"`
|
||||
- reference dates
|
||||
- `role_score`, `competence_score`, `overall_score`
|
||||
- category buckets including `Irrelevant`
|
||||
- `filter_search_results(...)`:
|
||||
- symmetric for both search types
|
||||
- supports `is_fully_available`
|
||||
- supports `task_text_filter` / `task_competence_filter` for task search
|
||||
- `get_results_by_category(...)`:
|
||||
- supports `Irrelevant`
|
||||
- consistent score columns
|
||||
- availability shown as overlap %
|
||||
- no `Level` in matching result tables
|
||||
|
||||
## Documentation updates
|
||||
|
||||
- `README.md`
|
||||
- Documented refined matching semantics, Irrelevant category, overlap % availability, and symmetric filtering.
|
||||
|
||||
- `docs/assistant_system_prompt.md`
|
||||
- Updated guidance to include Irrelevant, overlap %, score breakdown columns, and new task-search filters.
|
||||
|
||||
- OpenSpec docs updated to match final behavior:
|
||||
- `openspec/changes/refine-matching-and-filtering/tasks.md`
|
||||
- `openspec/changes/refine-matching-and-filtering/proposal.md`
|
||||
- `openspec/changes/refine-matching-and-filtering/design.md`
|
||||
- `openspec/changes/refine-matching-and-filtering/specs/matching-tools/spec.md`
|
||||
|
||||
## Known follow-ups / non-code items
|
||||
|
||||
- Editor diagnostics still appear to enforce a 79-char line limit despite Ruff being configured for 110. This is likely an editor/extension setting mismatch (not a repo lint failure).
|
||||
@@ -0,0 +1,231 @@
|
||||
# Design: Refine Matching and Filtering
|
||||
|
||||
## Context
|
||||
|
||||
This change improves matching accuracy and filtering capabilities across five areas: role inference precision, validation completeness, availability filtering, task search result management, and result categorization.
|
||||
|
||||
## Key Decisions
|
||||
|
||||
### 1. Title-first Role Inference (with fallback)
|
||||
|
||||
**Decision:** Use task **title first** for role inference; if title is missing/empty, fall back to full task text (title + description).
|
||||
|
||||
**Rationale:**
|
||||
- Task titles typically contain clear role indicators (e.g., "Senior Backend Developer")
|
||||
- Descriptions contain detailed requirements/context that dilute role signals
|
||||
- Fallback ensures robustness when titles are absent
|
||||
|
||||
**Implementation (final):**
|
||||
- Role inference text: `title.strip()` if present else `full_text`
|
||||
- Competence inference text: always uses `full_text`
|
||||
|
||||
### 2. Separate cache keys for role vs competence embeddings
|
||||
|
||||
**Decision:** Use separate embedding cache keys for role inference vs competence inference.
|
||||
|
||||
**Rationale:**
|
||||
- The same entity (task/capacity) is embedded with different texts depending on purpose.
|
||||
- Sharing the same cache key would cause collisions (role embedding overwriting competence embedding or vice versa).
|
||||
|
||||
**Implementation (final):**
|
||||
- Task:
|
||||
- role embedding cache key
|
||||
- competence embedding cache key
|
||||
- Capacity:
|
||||
- role embedding cache key
|
||||
- competence embedding cache key
|
||||
|
||||
### 2. Validation Displays All Returned Competences
|
||||
|
||||
**Decision:** Keep `max_competences` config active in validation and display all competences returned by inference (i.e., do not apply any additional output truncation).
|
||||
|
||||
**Rationale:**
|
||||
- `max_competences` is a deliberate, configurable guardrail
|
||||
- Validation should still show the complete inferred set produced under that configuration
|
||||
- The problem to fix is an *extra* display truncation, not the config itself
|
||||
|
||||
### C. Full-coverage availability filter
|
||||
|
||||
**Decision:** Add `is_fully_available` boolean to `filter_search_results(...)`.
|
||||
|
||||
**Semantics:**
|
||||
- Overlap (default): candidate overlaps the reference period.
|
||||
- Full coverage (`is_fully_available=true`): candidate fully covers the reference period.
|
||||
|
||||
**Reference period source:**
|
||||
- If `availability_date_start` / `availability_date_end` is provided in the filter call, use those.
|
||||
- Otherwise, use reference dates stored in the search payload.
|
||||
|
||||
### D. Task search filtering
|
||||
|
||||
**Decision:** Extend `filter_search_results(...)` with task-specific parameters.
|
||||
|
||||
**Parameters:**
|
||||
- `task_competence_filter` (list[str]): Filter tasks that require at least one of the specified competences (OR semantics)
|
||||
- `task_text_filter` (str): Case-insensitive substring match in title + description
|
||||
|
||||
### E. Irrelevant category
|
||||
|
||||
**Decision:** Add "Irrelevant" category for scores below the configured `matching.thresholds.low`.
|
||||
|
||||
**Thresholds (final):**
|
||||
```python
|
||||
if score >= top: "Top"
|
||||
elif score >= good: "Good"
|
||||
elif score >= partial:"Partial"
|
||||
elif score >= low: "Low"
|
||||
else: "Irrelevant"
|
||||
```
|
||||
|
||||
**Config:**
|
||||
- `matching.thresholds.low` in `config.toml` controls the Low vs Irrelevant boundary.
|
||||
|
||||
### F. Availability as percentage overlap
|
||||
|
||||
**Decision:** Display availability as **overlap percentage**.
|
||||
|
||||
**Interpretation:**
|
||||
- Capacity search (task → capacity): overlap % is relative to the **task** period.
|
||||
- Task search (capacity → task): overlap % is relative to the **capacity** period.
|
||||
|
||||
**Display:** integer percentage `0–100%`.
|
||||
|
||||
### G. Remove Level column from capacity search
|
||||
|
||||
**Decision:** Remove "Level" column from capacity-search matching result tables.
|
||||
|
||||
**Rationale:**
|
||||
- Level is not used in matching and clutters output.
|
||||
- Level remains visible in capacity listing/inspection tools.
|
||||
|
||||
### H. Fix task-search score display bug
|
||||
|
||||
**Root cause:** Field name mismatch in stored search results (using `score` vs `overall_score`).
|
||||
|
||||
**Fix:** Standardize stored key to `overall_score` in both directions.
|
||||
|
||||
### I. Separate role and competence scores
|
||||
|
||||
**Decision:** Display `role_score` and `competence_score` alongside `overall_score` in all matching result tables.
|
||||
|
||||
## Data Flow Changes
|
||||
|
||||
### Startup preload (modified)
|
||||
|
||||
```
|
||||
1. Fetch all open tasks
|
||||
2. For each task:
|
||||
- Build task_text = title.strip() # Changed: was title + "\n\n" + description
|
||||
- Embed with key f"task_id:{model}:{dims}:{task_id}"
|
||||
- Store in SQLite cache
|
||||
```
|
||||
|
||||
### Role inference (modified)
|
||||
|
||||
```
|
||||
1. Get task title (not full text)
|
||||
2. Retrieve/compute title embedding
|
||||
3. Compare against role vocabulary embeddings
|
||||
4. Return best match
|
||||
```
|
||||
|
||||
### Validation (modified)
|
||||
|
||||
```
|
||||
1. Get task (title + description + DB skills)
|
||||
2. Infer role from title only
|
||||
3. Infer competences from title+description (respect max_competences, threshold only)
|
||||
4. Display:
|
||||
- DB requirements table
|
||||
- Inferred role table (1 row)
|
||||
- Inferred competences table (all rows returned by inference)
|
||||
```
|
||||
|
||||
### Task filtering (new)
|
||||
|
||||
```
|
||||
1. Retrieve search results from SearchCache
|
||||
2. Detect search_type from payload
|
||||
3. Apply filters based on type:
|
||||
- capacity_search: role, competence, availability, similarity
|
||||
- task_search: task_competence, task_text, availability
|
||||
4. Store filtered results with new filter_id
|
||||
5. Return table with applied filters summary
|
||||
```
|
||||
|
||||
### Result display (modified)
|
||||
|
||||
```
|
||||
1. Fetch search results from SearchCache
|
||||
2. Calculate overlap percentage for each result
|
||||
3. Format table with columns:
|
||||
- Entity fields (ID, Owner/Title, Role, Competences)
|
||||
- Availability (percentage)
|
||||
- Role Score, Competence Score, Overall Score
|
||||
4. Return Markdown table
|
||||
```
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
### Alternative: Keep title+description for role inference
|
||||
**Rejected:** Descriptions contain too much noise for role matching. Titles are more precise.
|
||||
|
||||
### Alternative: Add separate validation tool for unlimited competences
|
||||
**Rejected:** Better to fix the existing tool than add complexity.
|
||||
|
||||
### Alternative: Separate tools for task filtering
|
||||
**Rejected:** Agent UX is better with unified tools. Shared tools reduce learning curve.
|
||||
|
||||
### Alternative: "None" or "No Match" instead of "Irrelevant"
|
||||
**Rejected:** "Irrelevant" is clearer and more user-friendly.
|
||||
|
||||
## Risks and Mitigations
|
||||
|
||||
### Risk: Role inference quality degrades with short titles
|
||||
**Mitigation:**
|
||||
- Monitor role inference accuracy
|
||||
- Consider fallback to description if title is too short (<3 words)
|
||||
- Document best practices for task title format
|
||||
|
||||
### Risk: Free-text search is too simple
|
||||
**Mitigation:**
|
||||
- Start with case-insensitive substring match
|
||||
- Can enhance later if needed (fuzzy, tokenization, etc.)
|
||||
|
||||
### Risk: Cache invalidation when changing embedded text
|
||||
**Mitigation:**
|
||||
- Accept that old entries will age out naturally (30-day TTL)
|
||||
- Or clear cache on deployment
|
||||
- Document in migration notes
|
||||
|
||||
### Risk: Too many filter parameters confuse agents
|
||||
**Mitigation:**
|
||||
- Clear docstrings with examples
|
||||
- Update assistant prompt with filtering patterns
|
||||
- Test with actual agent interactions
|
||||
|
||||
### Risk: Overlap percentage calculation edge cases
|
||||
**Mitigation:**
|
||||
- Handle NULL/missing dates gracefully (show "Open" or "N/A")
|
||||
- Round to nearest integer percentage
|
||||
- Test with various date combinations
|
||||
- Document calculation logic in code comments
|
||||
|
||||
### Risk: Breaking change for clients parsing tables
|
||||
**Mitigation:**
|
||||
- Document table format changes in release notes
|
||||
- Most clients use semantic parsing (LLMs), not brittle column parsing
|
||||
- Provide migration guide if needed
|
||||
|
||||
## Success Metrics
|
||||
|
||||
- Role inference accuracy improves (measure manually on sample tasks)
|
||||
- Validation always shows complete competence list
|
||||
- Full-coverage filtering works reliably in both directions
|
||||
- Task filtering works as documented
|
||||
- "Irrelevant" category appears for low-score matches
|
||||
- Availability percentages are accurate and intuitive
|
||||
- Task search results show correct non-zero scores
|
||||
- Separate scores help users understand match quality
|
||||
- All tests pass
|
||||
- No performance degradation
|
||||
@@ -0,0 +1,169 @@
|
||||
# Change Proposal: Refine Matching and Filtering
|
||||
|
||||
## Summary
|
||||
|
||||
Improve matching accuracy and filtering capabilities by refining role inference, competence validation display, availability filtering, task search result management, and scoring categorization.
|
||||
|
||||
## Why
|
||||
|
||||
The current matching and filtering implementation has several areas where behavior can be improved:
|
||||
|
||||
1. **Role inference uses full task text:** Using both title and description for role inference can introduce noise, as descriptions often contain detailed requirements rather than role indicators. Titles typically contain clearer role signals.
|
||||
|
||||
2. **Validation truncates competence list:** The current validation output respects the global `max_competences` limit, which hides potentially relevant competences from the user during inspection.
|
||||
|
||||
3. **No full-coverage availability filter:** Users cannot filter for capacities that are available for the *entire* task duration—only overlap filtering is supported.
|
||||
|
||||
4. **Task search results lack refinement tools:** The capacity→task matching flow stores search results but provides no tools to filter or refine them, unlike the task→capacity direction.
|
||||
|
||||
5. **No category for irrelevant results:** All matches fall into Top/Good/Partial/Low even when there is zero overlap, making it harder to identify truly irrelevant results.
|
||||
|
||||
6. **Availability display shows exact dates:** Search results show full date ranges (e.g., "2025-03-01 .. 2025-06-30"), which clutters the output. A percentage-based overlap indicator would be more concise.
|
||||
|
||||
7. **Capacity search shows unnecessary Level column:** When matching capacities to a task, the "Level" column appears in results but adds no value in this context.
|
||||
|
||||
8. **Bug: Zero scores displayed in task search results:** When matching tasks to a capacity, all results show a score of 0.0 even though higher scores are calculated internally. This is a display formatting bug.
|
||||
|
||||
9. **Missing score breakdown:** Users cannot see how role match and competence match contribute to the overall score, making it harder to understand why a match scored a certain way.
|
||||
|
||||
## What changes
|
||||
|
||||
### 1. Use task title first for role inference (with description fallback)
|
||||
|
||||
**Change:** Role inference should use **task title first**. If title is missing/empty, fall back to full task text (title + description).
|
||||
|
||||
**Scope:**
|
||||
- `validate_task_requirements(task_id)` tool
|
||||
- `find_matching_capacities(...)` tool (internal role inference)
|
||||
- `find_matching_tasks(capacity_id)` tool (internal role inference)
|
||||
- `infer_primary_role(task_id=...|task_text=...)` tool
|
||||
|
||||
**Exclusions:**
|
||||
- `extract_requirements(task_description, ...)` continues using full text for competence inference.
|
||||
|
||||
**Implementation notes:**
|
||||
- Use separate cache keys for task role embeddings vs task competence embeddings.
|
||||
|
||||
### 2. Show all inferred competences in validation
|
||||
|
||||
**Change:** The `validate_task_requirements(task_id)` tool should display **all competences returned by inference** that meet the `min_similarity` threshold, up to the configured `max_competences`.
|
||||
|
||||
**Scope:**
|
||||
- `validate_task_requirements(task_id)` tool only
|
||||
|
||||
**Implementation notes:**
|
||||
- Keep calling competence inference with `max_competences=cfg.matching.inference.max_competences`.
|
||||
- Ensure table rendering shows all returned rows (no extra hard-coded truncation).
|
||||
|
||||
### 3. Add full-coverage availability filter
|
||||
|
||||
**Change:** Add a boolean parameter `is_fully_available` to `filter_search_results(...)` that filters results to only those fully covering the reference time range.
|
||||
|
||||
**Scope:**
|
||||
- `filter_search_results(...)` tool
|
||||
- Applies to both task→capacity and capacity→task directions
|
||||
|
||||
### 4. Extend search result tools for task direction
|
||||
|
||||
**Change:** Make `get_results_by_category(...)` and `filter_search_results(...)` work seamlessly for **both** task→capacity and capacity→task search directions.
|
||||
|
||||
**Additions:**
|
||||
- Store `search_type` in the cached search payload: `"capacity_search"` or `"task_search"`.
|
||||
- Add task-search filters:
|
||||
- `task_competence_filter` (list[str])
|
||||
- `task_text_filter` (str)
|
||||
|
||||
### 5. Add "Irrelevant" category
|
||||
|
||||
**Change:** Introduce a new scoring category **"Irrelevant"** for matches with `overall_score` below the configured `matching.thresholds.low` value.
|
||||
|
||||
**Config:**
|
||||
- Add `matching.thresholds.low` to `config.toml`.
|
||||
- Scores `>= low` are categorized as "Low".
|
||||
- Scores `< low` are categorized as "Irrelevant".
|
||||
|
||||
### 6. Update tool docstrings
|
||||
|
||||
**Change:** Update docstrings for all affected tools with:
|
||||
- Clear parameter descriptions
|
||||
- Examples showing both directions (where applicable)
|
||||
- Notes about filtering semantics
|
||||
|
||||
### 7. Show availability as percentage overlap
|
||||
|
||||
**Change:** Replace exact date ranges in search result tables with an overlap percentage indicator.
|
||||
|
||||
**Interpretation:**
|
||||
- task→capacity: overlap % relative to the task period
|
||||
- capacity→task: overlap % relative to the capacity period
|
||||
|
||||
### 8. Remove Level column from capacity search results
|
||||
|
||||
**Change:** Remove the "Level" column from result tables when matching capacities to a task.
|
||||
|
||||
### 9. Fix zero score display bug in task search
|
||||
|
||||
**Fix:** Standardize on `'overall_score'` field name for stored results in both directions.
|
||||
|
||||
### 10. Show separate role and competence scores
|
||||
|
||||
**Change:** Display separate scores for role match and competence match in addition to overall score:
|
||||
- `role_score`
|
||||
- `competence_score`
|
||||
- `overall_score`
|
||||
|
||||
## Impact
|
||||
|
||||
### API surface
|
||||
- **Breaking:**
|
||||
- Table column changes (removed "Level" from capacity search, added score columns, changed "Availability" format)
|
||||
- Output format changes may affect clients parsing Markdown tables
|
||||
- **New parameters:** `is_fully_available`, `task_competence_filter`, `task_text_filter`
|
||||
- **New category:** "Irrelevant"
|
||||
- **Bug fixes:** Zero score display in task search results
|
||||
|
||||
### Business logic
|
||||
- Role inference behavior changes (title-only)
|
||||
- Validation shows more competences
|
||||
- New filtering options
|
||||
- Availability display changed from dates to percentage
|
||||
|
||||
### Data access
|
||||
- May need separate cache keys for title-only embeddings
|
||||
- Search payloads now store `role_score` and `competence_score` separately
|
||||
|
||||
### Performance
|
||||
- Minimal impact (same number of embeddings, just different text)
|
||||
- Overlap percentage calculation adds negligible overhead
|
||||
|
||||
### Documentation
|
||||
- README.md: document new filters, category, and output format changes
|
||||
- assistant_system_prompt.md: update category list, add filtering examples, describe new table format
|
||||
- architecture.md: update scoring thresholds and output format
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Changing scoring weights or thresholds for existing categories (Top/Good/Partial/Low)
|
||||
- Adding fuzzy/approximate text search (simple substring match is sufficient)
|
||||
- Changing the fundamental matching algorithm
|
||||
- Showing individual competence match percentages (only overall competence score)
|
||||
- Changing the date overlap logic itself (only the display format)
|
||||
|
||||
## Open Questions
|
||||
|
||||
None—all design decisions have been clarified.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- Role inference uses task title only (verified by tests)
|
||||
- Validation shows all competences above threshold
|
||||
- `is_fully_available` filter works in both directions
|
||||
- Task search results can be filtered by competences and text
|
||||
- "Irrelevant" category appears when `overall_score < matching.thresholds.low`
|
||||
- Availability shown as percentage overlap in all result tables
|
||||
- Level column removed from capacity search results
|
||||
- Task search results show correct (non-zero) scores
|
||||
- All result tables show Role Score, Competence Score, and Overall Score columns
|
||||
- All tool docstrings are clear and include examples
|
||||
- All tests pass
|
||||
- `openspec validate refine-matching-and-filtering --strict` passes
|
||||
+197
@@ -0,0 +1,197 @@
|
||||
# Spec Delta: Matching Tools Refinements
|
||||
|
||||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: Role inference uses task title only
|
||||
|
||||
Role inference for validation and matching tools SHALL use **only task title** for embedding and matching against role vocabulary. The `extract_requirements` tool continues to use combined text.
|
||||
|
||||
#### Scenario: Validate task requirements with title-only role inference
|
||||
|
||||
- **GIVEN** task ID `T-123` with title "Senior Frontend Developer" and description containing multiple competences
|
||||
- **AND** role vocabulary includes "Frontend Developer" and "Backend Developer"
|
||||
- **WHEN** the client calls `validate_task_requirements(task_id="T-123")`
|
||||
- **THEN** the system embeds **only** the title "Senior Frontend Developer"
|
||||
- **AND** the system matches against role vocabulary
|
||||
- **AND** the system returns inferred role based on title embedding only
|
||||
- **AND** the system still infers competences from full task text (title + description)
|
||||
|
||||
#### Scenario: Find matching capacities with title-only role inference
|
||||
|
||||
- **GIVEN** task with title "DevOps Engineer" and description "Experience with Kubernetes, Docker, CI/CD"
|
||||
- **WHEN** the client calls `find_matching_capacities(...)`
|
||||
- **THEN** the system infers task role from title "DevOps Engineer" only
|
||||
- **AND** the system infers task competences from full text (title + description)
|
||||
|
||||
#### Scenario: Find matching tasks with title-only role inference
|
||||
|
||||
- **GIVEN** capacity with role "Backend Developer"
|
||||
- **AND** task with title "Java Backend Developer" and description "Spring Boot, PostgreSQL"
|
||||
- **WHEN** the client calls `find_matching_tasks(capacity_id=...)`
|
||||
- **THEN** the system infers task role from title "Java Backend Developer" only
|
||||
- **AND** the system matches capacity role against task role
|
||||
|
||||
### Requirement: Validation shows all inferred competences returned by inference
|
||||
|
||||
Task validation SHALL display all inferred competences meeting `min_similarity` threshold, up to the configured `max_competences` limit. The output MUST NOT apply any additional hard-coded truncation (e.g., always showing only the top 8 rows).
|
||||
|
||||
#### Scenario: Validation displays up to max_competences competences
|
||||
|
||||
- **GIVEN** task with description mentioning many competences
|
||||
- **AND** config: `matching.inference.max_competences = 16`, `matching.inference.min_similarity = 0.4`
|
||||
- **AND** at least 16 competences have similarity >= 0.4
|
||||
- **WHEN** the client calls `validate_task_requirements(task_id="T-456")`
|
||||
- **THEN** the system returns and displays 16 competences (not fewer)
|
||||
- **AND** the output is NOT additionally limited to 8 competences
|
||||
- **AND** results are sorted by similarity score in descending order
|
||||
|
||||
### Requirement: Full availability filter parameter
|
||||
|
||||
Search result filtering SHALL support `is_fully_available` boolean parameter for date-coverage filtering in both directions.
|
||||
|
||||
#### Scenario: Filter capacities fully available for task duration
|
||||
|
||||
- **GIVEN** capacity search with 5 results
|
||||
- **AND** task period: 2025-03-01 to 2025-06-30
|
||||
- **AND** capacities: C1 (2025-02-01 to 2025-07-31), C2 (2025-03-15 to 2025-06-15), C3 (2025-01-01 to 2025-12-31)
|
||||
- **WHEN** the client calls `filter_search_results(search_id="...", availability_date_start="2025-03-01", availability_date_end="2025-06-30", is_fully_available=true)`
|
||||
- **THEN** the system returns filtered results containing only C1 and C3
|
||||
- **AND** applied filters table shows: `is_fully_available = true`
|
||||
|
||||
#### Scenario: Filter tasks within capacity availability window
|
||||
|
||||
- **GIVEN** task search with 4 results
|
||||
- **AND** capacity period: 2025-04-01 to 2025-08-31
|
||||
- **AND** tasks: T1 (2025-03-01 to 2025-05-31), T2 (2025-05-01 to 2025-07-31), T3 (2025-06-01 to 2025-09-30)
|
||||
- **WHEN** the client calls `filter_search_results(search_id="...", availability_date_start="2025-04-01", availability_date_end="2025-08-31", is_fully_available=true)`
|
||||
- **THEN** the system returns filtered results containing only T2
|
||||
- **AND** T1 excluded (starts before capacity)
|
||||
- **AND** T3 excluded (ends after capacity)
|
||||
|
||||
### Requirement: Task-specific search filters
|
||||
|
||||
Search result filtering SHALL support task-specific filters `task_competence_filter` and `task_text_filter` when filtering task search results.
|
||||
|
||||
#### Scenario: Filter tasks by required competences
|
||||
|
||||
- **GIVEN** task search with 6 results
|
||||
- **AND** tasks requiring various competences including Python, React, Docker
|
||||
- **WHEN** the client calls `filter_search_results(search_id="...", task_competence_filter=["Python", "Docker"])`
|
||||
- **THEN** the system returns only tasks that require either Python OR Docker
|
||||
- **AND** applied filters table shows: `task_competence_filter = ["Python", "Docker"]`
|
||||
|
||||
#### Scenario: Filter tasks by text search
|
||||
|
||||
- **GIVEN** task search with 5 results
|
||||
- **AND** task T1: title "Frontend Developer", description "React and TypeScript"
|
||||
- **AND** task T2: title "Backend Engineer", description "Python and PostgreSQL"
|
||||
- **AND** task T3: title "Full-Stack Developer", description "React, Node.js, TypeScript"
|
||||
- **WHEN** the client calls `filter_search_results(search_id="...", task_text_filter="typescript")`
|
||||
- **THEN** the system returns T1 and T3 (case-insensitive match)
|
||||
- **AND** applied filters table shows: `task_text_filter = "typescript"`
|
||||
|
||||
### Requirement: Irrelevant match category
|
||||
|
||||
Matching tools SHALL categorize matches with `overall_score` below `matching.thresholds.low` as "Irrelevant" and include them in summary tables and result displays.
|
||||
|
||||
#### Scenario: Match is categorized as Irrelevant
|
||||
|
||||
- **GIVEN** capacity with role "Data Scientist" and competences ["Python", "Machine Learning"]
|
||||
- **AND** task requiring role "Frontend Developer" and competences ["React", "TypeScript"]
|
||||
- **AND** computed match score: 0.05
|
||||
- **WHEN** the system categorizes the match
|
||||
- **THEN** the match is categorized as "Irrelevant"
|
||||
- **AND** summary table includes "Irrelevant" category with count
|
||||
- **AND** results include Irrelevant section if such matches exist
|
||||
|
||||
### Requirement: Enhanced tool docstrings with examples
|
||||
|
||||
All affected matching and filtering tools SHALL include updated docstrings with usage examples demonstrating new features.
|
||||
|
||||
#### Scenario: Tool docstrings show usage examples
|
||||
|
||||
- **GIVEN** tools with new parameters and features
|
||||
- **WHEN** a client inspects tool descriptions
|
||||
- **THEN** each tool docstring includes clear parameter descriptions
|
||||
- **AND** usage examples for new features
|
||||
- **AND** guidance on when to use each feature
|
||||
- **AND** cross-references to related tools
|
||||
|
||||
### Requirement: Availability displayed as percentage overlap
|
||||
|
||||
Search result tables SHALL display availability as percentage overlap instead of exact date ranges.
|
||||
|
||||
#### Scenario: Capacity search shows percentage overlap
|
||||
|
||||
- **GIVEN** task with period 2025-03-01 to 2025-06-30 (122 days)
|
||||
- **AND** capacity with period 2025-03-15 to 2025-06-30 (108 days overlap)
|
||||
- **WHEN** the client views capacity search results
|
||||
- **THEN** the Availability column shows "88%" (108/122 rounded)
|
||||
- **AND** the Availability column is positioned left of score columns
|
||||
|
||||
#### Scenario: Task search shows percentage overlap
|
||||
|
||||
- **GIVEN** capacity with period 2025-04-01 to 2025-08-31 (153 days)
|
||||
- **AND** task with period 2025-05-01 to 2025-07-31 (92 days overlap)
|
||||
- **WHEN** the client views task search results
|
||||
- **THEN** the Availability column shows "60%" (92/153 rounded)
|
||||
|
||||
#### Scenario: Open-ended availability period
|
||||
|
||||
- **GIVEN** capacity with open end date (end_date is NULL)
|
||||
- **WHEN** calculating overlap percentage
|
||||
- **THEN** the system displays "Open" or "∞" instead of a percentage
|
||||
|
||||
### Requirement: Level column removed from capacity search
|
||||
|
||||
Capacity search result tables SHALL NOT include the "Level" column.
|
||||
|
||||
#### Scenario: Find matching capacities output excludes Level
|
||||
|
||||
- **GIVEN** user searches for matching capacities
|
||||
- **WHEN** the client calls find_matching_capacities(...)
|
||||
- **THEN** the result table includes columns: ID, Owner, Role, Competences, Availability, Role Score, Competence Score, Overall Score
|
||||
- **AND** the table does NOT include a Level column
|
||||
|
||||
### Requirement: Task search score display bug fixed
|
||||
|
||||
The system SHALL correctly display non-zero scores for task search results in get_results_by_category.
|
||||
|
||||
#### Scenario: Task search results show correct scores
|
||||
|
||||
- **GIVEN** capacity with role "Backend Developer" and competences ["Python", "PostgreSQL"]
|
||||
- **AND** task requiring role "Backend Developer" and competences ["Python", "Django"]
|
||||
- **AND** computed overall score is 0.85
|
||||
- **WHEN** the client calls find_matching_tasks(capacity_id=...)
|
||||
- **THEN** the result table shows Overall Score as "0.850"
|
||||
- **WHEN** the client calls get_results_by_category(search_id=..., category="Top")
|
||||
- **THEN** the result table shows Overall Score as "0.850" (not "0.000")
|
||||
|
||||
### Requirement: Separate role and competence scores displayed
|
||||
|
||||
All matching result tables SHALL display separate Role Score, Competence Score, and Overall Score columns.
|
||||
|
||||
#### Scenario: Capacity search shows separate scores
|
||||
|
||||
- **GIVEN** task requiring role "Frontend Developer" and competences ["React", "TypeScript", "CSS"]
|
||||
- **AND** capacity with role "Frontend Developer" and competences ["React", "TypeScript"]
|
||||
- **AND** computed role_score = 1.0, competence_score = 0.667, overall_score = 0.750
|
||||
- **WHEN** the client views capacity search results
|
||||
- **THEN** the result table includes columns: Role Score, Competence Score, Overall Score (rightmost)
|
||||
- **AND** the row shows: 1.000, 0.667, 0.750
|
||||
|
||||
#### Scenario: Task search shows separate scores
|
||||
|
||||
- **GIVEN** capacity with role "DevOps Engineer" and competences ["Kubernetes", "Docker"]
|
||||
- **AND** task requiring role "DevOps Engineer" and competences ["Kubernetes", "Terraform", "AWS"]
|
||||
- **AND** computed role_score = 1.0, competence_score = 0.333, overall_score = 0.500
|
||||
- **WHEN** the client views task search results
|
||||
- **THEN** the result table includes columns: Role Score, Competence Score, Overall Score (rightmost)
|
||||
- **AND** the row shows: 1.000, 0.333, 0.500
|
||||
|
||||
#### Scenario: Categorization uses overall score only
|
||||
|
||||
- **GIVEN** a match with role_score = 1.0, competence_score = 0.2, overall_score = 0.45
|
||||
- **WHEN** the system categorizes the match
|
||||
- **THEN** the match is categorized as "Partial" (based on overall_score >= 0.4)
|
||||
- **AND** role_score and competence_score are NOT used for categorization
|
||||
@@ -0,0 +1,259 @@
|
||||
# Implementation Tasks: Refine Matching and Filtering
|
||||
|
||||
## Status: Completed
|
||||
|
||||
## Summary
|
||||
|
||||
This change implements 10 improvements to matching and filtering:
|
||||
|
||||
1. **Title-only role inference**: Use task title (not description) for role matching
|
||||
2. **Unlimited validation**: Show all competences above threshold in validation
|
||||
3. **Full-coverage filter**: Add `is_fully_available` for complete date coverage filtering
|
||||
4. **Task filtering**: Add competence and text filters for task search results
|
||||
5. **Irrelevant category**: Add an "Irrelevant" category for scores below `matching.thresholds.low`
|
||||
6. **Enhanced docstrings**: Comprehensive examples for all tools
|
||||
7. **Percentage availability**: Show overlap as percentage instead of date ranges
|
||||
8. **Remove Level column**: Clean up capacity search output
|
||||
9. **Fix score bug**: Correct zero score display in task search results
|
||||
10. **Separate scores**: Display Role Score, Competence Score, Overall Score
|
||||
|
||||
## Clarifications / Decisions (added during review)
|
||||
|
||||
- **Separate embedding keys (MUST):** Roles and competences MUST use separate embedding cache keys/paths for both tasks and capacities.
|
||||
- If such separation does not exist yet, it MUST be introduced.
|
||||
- For task-role inference specifically, we do NOT need separate keys for title-only vs title+description; the role embedding key can stay stable while input text changes per fallback logic.
|
||||
- **Role inference fallback:** If task title is missing/too short/generic, infer role from task description instead. If that still yields no role above threshold (or no vocab match), role_score MUST be `0.0`.
|
||||
- **`is_fully_available` without explicit date params:** `filter_search_results(..., is_fully_available=true)` MUST work even when `availability_date_start/end` are omitted by using reference dates stored in the search context.
|
||||
- **Availability % with open-ended other_end:** If the reference period is fully defined and `other_end is None`, availability percentage MUST still be computed (overlap is well-defined until ref_end).
|
||||
- **Overall score formula (fixed):** `overall_score` MUST be a weighted mean of role_score and competence_score using config weights:
|
||||
- `overall_score = role_weight * role_score + competence_weight * competence_score`
|
||||
- Weights are read from `cfg.matching.role_weight` and `cfg.matching.competence_weight`.
|
||||
- **Irrelevant always displayed:** The "Irrelevant" category MUST always be shown in summary tables and category lists.
|
||||
- **Task filters are case-insensitive:** task competence and text filtering MUST be case-insensitive.
|
||||
- `required_competences` are structured fields from task details and MUST be present in task-search results.
|
||||
|
||||
## Estimated Effort
|
||||
|
||||
- **Phase 1 (Core changes)**: 6-8 hours
|
||||
- **Phase 2 (Testing)**: 3-4 hours
|
||||
- **Phase 3 (Documentation)**: 2-3 hours
|
||||
- **Phase 4 (Validation)**: 1 hour
|
||||
- **Total**: 12-16 hours
|
||||
|
||||
## Phase 1: Core Changes
|
||||
|
||||
### Task 1.1: Title-only role inference ✅
|
||||
**Files:** `src/teamlandkarte_mcp/matching/vocabulary.py`, `src/teamlandkarte_mcp/mcp_server.py`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Identify where task text is built for role inference (startup preload, validation, matching)
|
||||
- [x] Modify role inference code paths to use title only (not title + description):
|
||||
- `startup_preload()` for task embeddings
|
||||
- `validate_task_requirements()` for role inference
|
||||
- `find_matching_capacities()` for task role inference
|
||||
- `find_matching_tasks()` for task role inference
|
||||
- [x] **Important**: Competence inference still uses title + description (unchanged)
|
||||
- [x] Verify cache key strategy: reuse stable role cache key; competence embeddings use separate key
|
||||
- [x] Document: role embeddings now use title-first text with fallback; competence embeddings use full text
|
||||
- [x] Add fallback when title is empty
|
||||
|
||||
**Acceptance:**
|
||||
- Role inference uses title-first for role embedding
|
||||
- Competence inference still uses title + description
|
||||
- Cache separation prevents collisions
|
||||
- Fallback handles empty titles gracefully
|
||||
|
||||
---
|
||||
|
||||
### Task 1.2: Unlimited competences in validation ✅
|
||||
**Files:** `src/teamlandkarte_mcp/mcp_server.py`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Keep using `cfg.matching.inference.max_competences` in `validate_task_requirements()` (do NOT override it)
|
||||
- [x] Ensure the output table displays **all competences returned by inference**
|
||||
- [x] Ensure the output is not hard-limited to 8 competences anywhere in formatting
|
||||
- [x] Keep threshold behavior (`min_similarity`) unchanged
|
||||
- [x] Update docstring to clarify: validation shows up to `max_competences` competences meeting threshold
|
||||
|
||||
**Acceptance:**
|
||||
- Validation displays all inferred competences returned by inference (up to configured `max_competences`)
|
||||
- No additional hard-coded limit is applied in output formatting
|
||||
- Threshold still applies (`min_similarity` config)
|
||||
|
||||
---
|
||||
|
||||
### Task 1.3: Add `is_fully_available` filter ✅
|
||||
**Files:** `src/teamlandkarte_mcp/mcp_server.py`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Add `search_type` field to search payloads:
|
||||
- Update `find_matching_capacities()` to store `"capacity_search"`
|
||||
- Update `find_matching_tasks()` to store `"task_search"`
|
||||
- [x] Add `is_fully_available` parameter to `filter_search_results()`
|
||||
- [x] Detect `search_type` from search payload metadata
|
||||
- [x] Implement full-coverage logic for `capacity_search`
|
||||
- [x] Implement full-coverage logic for `task_search`
|
||||
- [x] Use stored reference dates when explicit availability_date_* are omitted
|
||||
|
||||
**Acceptance:**
|
||||
- Search payloads include `search_type` field
|
||||
- `is_fully_available=True` filters for full coverage in both directions
|
||||
- Default keeps overlap behavior
|
||||
- Edge cases handled (missing dates)
|
||||
|
||||
---
|
||||
|
||||
### Task 1.4: Add task filtering parameters ✅
|
||||
**Files:** `src/teamlandkarte_mcp/mcp_server.py`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Add `task_competence_filter` parameter to `filter_search_results()`
|
||||
- [x] Add `task_text_filter` parameter to `filter_search_results()`
|
||||
- [x] Implement competence filtering for tasks
|
||||
- [x] Implement text filtering for tasks (case-insensitive)
|
||||
- [x] Apply filters only when `search_type == "task_search"`
|
||||
|
||||
**Acceptance:**
|
||||
- Task competence filter works for capacity→task searches
|
||||
- Task text filter works for capacity→task searches
|
||||
- Filters are ignored for task→capacity searches
|
||||
|
||||
---
|
||||
|
||||
### Task 1.5: Add "Irrelevant" category ✅
|
||||
**Files:** `src/teamlandkarte_mcp/matching/scorer.py`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Modify categorization to add `score < matching.thresholds.low` → "Irrelevant"
|
||||
- [x] Update category order: Top, Good, Partial, Low, Irrelevant
|
||||
- [x] Ensure "Irrelevant" appears in summary tables and category lists
|
||||
|
||||
**Acceptance:**
|
||||
- Scores below `matching.thresholds.low` are categorized as "Irrelevant"
|
||||
- All tools display consistent categories
|
||||
|
||||
---
|
||||
|
||||
### Task 1.6: Update tool docstrings ✅
|
||||
**Files:** `src/teamlandkarte_mcp/mcp_server.py`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Update `filter_search_results()` docstring with bidirectional examples and new params
|
||||
- [x] Update `get_results_by_category()` docstring with both search types examples
|
||||
- [x] Update `find_matching_capacities()` docstring with title-first role inference note
|
||||
- [x] Update `find_matching_tasks()` docstring with title-first role inference note
|
||||
- [x] Update `validate_task_requirements()` docstring with title-first + max_competences note
|
||||
|
||||
**Acceptance:**
|
||||
- All tool docstrings are clear and comprehensive
|
||||
- Examples demonstrate both search directions
|
||||
|
||||
---
|
||||
|
||||
### Task 1.7: Show availability as percentage overlap ✅
|
||||
**Files:** `src/teamlandkarte_mcp/mcp_server.py`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Create helper function `_calculate_overlap_percentage(...)`
|
||||
- [x] Update `find_matching_capacities()` and `find_matching_tasks()` output tables
|
||||
- [x] Update `get_results_by_category()` and `filter_search_results()` preview tables
|
||||
|
||||
**Acceptance:**
|
||||
- All result tables show availability as percentage
|
||||
- Open-ended periods handled
|
||||
|
||||
---
|
||||
|
||||
### Task 1.8: Remove Level column from capacity search ✅
|
||||
**Files:** `src/teamlandkarte_mcp/mcp_server.py`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Remove "Level" from capacity-search matching result tables
|
||||
- [x] Keep Level in `list_free_capacities()` (inspection)
|
||||
|
||||
**Acceptance:**
|
||||
- Capacity search matching outputs do NOT show "Level"
|
||||
- Listing/inspection still shows Level
|
||||
|
||||
---
|
||||
|
||||
### Task 1.9: Fix zero score display bug ✅
|
||||
**Files:** `src/teamlandkarte_mcp/mcp_server.py`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Standardize on `overall_score` in task-search stored payloads
|
||||
- [x] Verify display in `get_results_by_category()`
|
||||
|
||||
**Acceptance:**
|
||||
- Task search results show correct non-zero scores
|
||||
|
||||
---
|
||||
|
||||
### Task 1.10: Display separate role and competence scores ✅
|
||||
**Files:** `src/teamlandkarte_mcp/mcp_server.py`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Store and display `role_score`, `competence_score`, `overall_score` for both searches
|
||||
- [x] Ensure consistent formatting in result tables
|
||||
|
||||
**Acceptance:**
|
||||
- All result tables show Role/Competence/Overall score columns
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Testing
|
||||
|
||||
### Task 2.1: Unit tests ✅
|
||||
**Files:** `tests/test_matching_refinements.py` (new)
|
||||
|
||||
**Subtasks:**
|
||||
- [x] Test title-first role inference helpers and fallback behavior
|
||||
- [x] Test "Irrelevant" category threshold
|
||||
- [x] Test availability percentage calculation incl. open-ended end date
|
||||
|
||||
**Acceptance:**
|
||||
- Unit tests cover key refinements
|
||||
- Tests pass with `pytest -q`
|
||||
|
||||
---
|
||||
|
||||
### Task 2.2: Integration tests ✅
|
||||
**Files:** `tests/test_integration_matching.py`
|
||||
|
||||
**Subtasks:**
|
||||
- [x] End-to-end test: capacity search with new filters and display
|
||||
- [x] End-to-end test: task search with new filters and display
|
||||
- [x] Test filtering with multiple parameters
|
||||
- [x] Test pagination with new table format
|
||||
- [x] Verify Markdown table format consistency
|
||||
|
||||
**Acceptance:**
|
||||
- Integration tests demonstrate full workflows
|
||||
- All tests pass
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: Documentation
|
||||
|
||||
### Task 3.1: Update README ✅
|
||||
**Files:** `README.md`
|
||||
|
||||
### Task 3.2: Update assistant prompt ✅
|
||||
**Files:** `docs/assistant_system_prompt.md`
|
||||
|
||||
### Task 3.3: Update architecture docs ✅
|
||||
**Files:** `docs/architecture.md`
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: Validation and Completion
|
||||
|
||||
### Task 4.1: OpenSpec validation ✅
|
||||
|
||||
- [x] All tasks in this change marked as completed
|
||||
- [x] Unit + integration tests passing (`uv run pytest -q`)
|
||||
|
||||
### Task 4.2: Final testing and deployment ✅
|
||||
|
||||
- [x] Final local test run completed
|
||||
- [x] No external deployment steps are automated in this repository
|
||||
+8
@@ -0,0 +1,8 @@
|
||||
+# Changelog
|
||||
+
|
||||
+## replace-heuristics-with-azure-openai
|
||||
+
|
||||
+- Removed heuristic/sampling-based similarity and replaced role similarity with Azure embedding cosine similarity.
|
||||
+- Tightened docstrings and typing for `TaskAnalyzer` and `SimilarityEngine`.
|
||||
+- Removed stray debug output and ensured stderr-safe logging.
|
||||
+
|
||||
+752
@@ -0,0 +1,752 @@
|
||||
# Change Proposal: Replace Heuristics with Azure OpenAI Embeddings and LLM
|
||||
|
||||
- **Change ID**: `replace-heuristics-with-azure-openai`
|
||||
- **Status**: Implemented
|
||||
- **Target**: `teamlandkarte-mcp`
|
||||
- **Author**: Thomas Handke
|
||||
- **Date**: 2026-02-13
|
||||
|
||||
## Summary
|
||||
|
||||
Replace the current FastMCP sampling (unavailable) and deterministic heuristic fallbacks with direct Azure OpenAI API integration for:
|
||||
|
||||
1. **Competence and role similarity matching** via embeddings (`text-embedding-3-large`, 3072 dimensions)
|
||||
2. **Requirements extraction and validation** via LLM (`gpt-4.1`)
|
||||
|
||||
Additionally:
|
||||
- Rename `database.toml` → `config.toml` (and template accordingly)
|
||||
- Implement persistent embedding cache using a local SQLite database
|
||||
- Support two similarity scoring strategies (per-skill vs. aggregate) via configuration
|
||||
- Remove all FastMCP sampling code and heuristic fallbacks completely
|
||||
|
||||
## Motivation
|
||||
|
||||
### Current State Problems
|
||||
|
||||
1. **FastMCP sampling is unavailable**: The `mcp` package does not expose a `FastMCP.sampling` API, forcing the system to rely entirely on weak deterministic heuristics.
|
||||
|
||||
2. **Heuristic limitations**:
|
||||
- Role inference: simple keyword matching (`"cloud" → "Cloud Engineer"`)
|
||||
- Competence extraction: small hardcoded keyword list (~15 terms)
|
||||
- Competence similarity: exact normalized match (1.0) or substring (0.7), otherwise 0.0
|
||||
- No true semantic understanding
|
||||
|
||||
3. **Poor matching quality**:
|
||||
- Synonyms not recognized (e.g., "React.js" vs "ReactJS")
|
||||
- Related skills missed (e.g., "FastAPI" vs "REST API Development")
|
||||
- Role extraction frequently returns `"(unknown)"`
|
||||
|
||||
4. **Maintenance burden**: Heuristic keyword lists require manual updates
|
||||
|
||||
### Proposed Benefits
|
||||
|
||||
1. **Semantic matching**: True similarity via embeddings (e.g., "Kubernetes" ↔ "K8s", "Python" ↔ "Python 3")
|
||||
2. **Robust extraction**: LLM-backed role and competence extraction from free text
|
||||
3. **Better user experience**: More accurate Top/Good/Partial/Low categorization
|
||||
4. **Reduced maintenance**: No manual keyword list updates
|
||||
5. **Production-ready**: Direct API integration, no dependency on MCP sampling feature
|
||||
|
||||
## Goals
|
||||
|
||||
1. Replace `TaskAnalyzer.semantic_competence_similarity` with embedding-based similarity
|
||||
2. Replace `TaskAnalyzer.extract_ranked_roles` with Azure OpenAI LLM
|
||||
3. Replace `TaskAnalyzer._extract_requirements_from_description` with Azure OpenAI LLM
|
||||
4. Replace `TaskAnalyzer.apply_requirement_update` with Azure OpenAI LLM
|
||||
5. Implement persistent embedding cache (SQLite)
|
||||
6. Support two similarity strategies via config
|
||||
7. Remove all FastMCP sampling and heuristic code
|
||||
8. Rename `database.toml` → `config.toml`
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Changing the overall matching workflow (confirmation gate, guided capture, etc.)
|
||||
- Changing the scoring weights (competence 0.8, role 0.2)
|
||||
- Changing the categorical thresholds (Top/Good/Partial/Low)
|
||||
- Supporting multiple embedding models
|
||||
- Supporting other LLM providers (only Azure OpenAI)
|
||||
|
||||
## Proposed Changes
|
||||
|
||||
### 1. Configuration Changes
|
||||
|
||||
#### 1.1 Rename configuration file
|
||||
- `database.toml` → `config.toml`
|
||||
- `database.toml.example` → `config.toml.example`
|
||||
- Update all references in code, docs, README
|
||||
|
||||
#### 1.2 New configuration sections in `config.toml`
|
||||
|
||||
```toml
|
||||
[azure_openai]
|
||||
endpoint = "https://aiservice-ca00361106.cognitiveservices.azure.com/"
|
||||
api_version_embeddings = "2024-02-01"
|
||||
api_version_llm = "2024-12-01-preview"
|
||||
|
||||
[azure_openai.embeddings]
|
||||
model = "text-embedding-3-large"
|
||||
# Azure deployment name is equal to model name
|
||||
deployment = "text-embedding-3-large"
|
||||
|
||||
[azure_openai.llm]
|
||||
model = "gpt-4.1"
|
||||
# Azure deployment name is equal to model name
|
||||
deployment = "gpt-4.1"
|
||||
temperature = 0.2
|
||||
max_tokens = 2000
|
||||
|
||||
[embedding_cache]
|
||||
enabled = true
|
||||
db_path = "embeddings_cache.db"
|
||||
ttl_days = 30
|
||||
|
||||
[matching.similarity]
|
||||
# Strategy: "per_skill" or "aggregate"
|
||||
# - per_skill: Match each required skill to best candidate skill (current behavior)
|
||||
# - aggregate: Compare average embedding of all required vs all candidate skills
|
||||
strategy = "per_skill"
|
||||
# Chat model/deployment name (Azure OpenAI)
|
||||
chat_model = "gpt-4.1"
|
||||
```
|
||||
|
||||
#### 1.3 Environment variables (`.env`)
|
||||
```bash
|
||||
AZURE_OPENAI_EMBEDDING_API_KEY="..."
|
||||
AZURE_OPENAI_LLM_API_KEY="..."
|
||||
```
|
||||
|
||||
### 2. Embedding Cache Implementation
|
||||
|
||||
Create `src/teamlandkarte_mcp/cache/embedding_cache.py`:
|
||||
|
||||
```python
|
||||
class EmbeddingCache:
|
||||
"""Persistent embedding cache using SQLite.
|
||||
|
||||
Schema:
|
||||
embeddings(
|
||||
id INTEGER PRIMARY KEY,
|
||||
text TEXT UNIQUE NOT NULL,
|
||||
model TEXT NOT NULL,
|
||||
embedding BLOB NOT NULL, -- UTF-8 bytes of JSON-serialized list[float]
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||
)
|
||||
|
||||
Notes:
|
||||
- Store embeddings as UTF-8 encoded JSON in a BLOB column for portability.
|
||||
- Serialize with: json.dumps(embedding).encode("utf-8")
|
||||
- Deserialize with: json.loads(blob.decode("utf-8"))
|
||||
"""
|
||||
|
||||
def __init__(self, db_path: str, ttl_days: int = 30):
|
||||
"""Initialize cache, create DB if not exists."""
|
||||
|
||||
def get(self, text: str, model: str) -> Optional[list[float]]:
|
||||
"""Retrieve cached embedding, return None if missing/expired."""
|
||||
|
||||
def put(self, text: str, model: str, embedding: list[float]) -> None:
|
||||
"""Store embedding in cache."""
|
||||
|
||||
def cleanup_expired(self) -> int:
|
||||
"""Remove embeddings older than TTL, return count deleted."""
|
||||
```
|
||||
|
||||
### 3. Azure OpenAI Client Layer
|
||||
|
||||
Create `src/teamlandkarte_mcp/azure/openai_client.py`:
|
||||
|
||||
```python
|
||||
class AzureOpenAIClient:
|
||||
"""Wrapper for Azure OpenAI API calls using AsyncAzureOpenAI client."""
|
||||
|
||||
def __init__(self, config: AzureOpenAIConfig, embedding_cache: EmbeddingCache):
|
||||
"""Initialize AsyncAzureOpenAI clients for embeddings and LLM.
|
||||
|
||||
Uses openai.AsyncAzureOpenAI for all async API calls.
|
||||
"""
|
||||
|
||||
async def get_embedding(self, text: str) -> list[float]:
|
||||
"""Get embedding for text, using cache if available.
|
||||
|
||||
Raises:
|
||||
AzureAPIError: If API call fails (no fallback).
|
||||
"""
|
||||
|
||||
async def get_embeddings_batch(self, texts: list[str]) -> list[list[float]]:
|
||||
"""Get embeddings for multiple texts (not batched per user requirement)."""
|
||||
|
||||
async def chat_completion(
|
||||
self,
|
||||
system: str,
|
||||
user: str,
|
||||
response_format: dict = None
|
||||
) -> str:
|
||||
"""LLM completion with structured JSON response.
|
||||
|
||||
Raises:
|
||||
AzureAPIError: If API call fails (no fallback).
|
||||
"""
|
||||
```
|
||||
|
||||
### 4. Similarity Computation
|
||||
|
||||
Create `src/teamlandkarte_mcp/matching/similarity.py`:
|
||||
|
||||
```python
|
||||
def cosine_similarity(vec_a: list[float], vec_b: list[float]) -> float:
|
||||
"""Compute cosine similarity between two vectors."""
|
||||
|
||||
class SimilarityEngine:
|
||||
"""Compute semantic similarity using embeddings."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
client: AzureOpenAIClient,
|
||||
strategy: str = "per_skill"
|
||||
):
|
||||
"""Initialize with Azure client and strategy."""
|
||||
|
||||
async def compute_competence_similarity(
|
||||
self,
|
||||
required: list[str],
|
||||
candidate: list[str],
|
||||
) -> dict[str, dict[str, Any]]:
|
||||
"""Compute similarity using configured strategy.
|
||||
|
||||
Returns same format as current semantic_competence_similarity:
|
||||
{required_comp: {best_match: str|null, score: float, rationale: str}}
|
||||
|
||||
Behavioral guarantees:
|
||||
- `rationale` is always included (even when best_match is null).
|
||||
- If `required` is empty, return {}.
|
||||
- If `candidate` is empty, return an entry per required competence with:
|
||||
best_match=null, score=0.0, and a short rationale.
|
||||
"""
|
||||
|
||||
async def _per_skill_similarity(
|
||||
self, required: list[str], candidate: list[str]
|
||||
) -> dict[str, dict[str, Any]]:
|
||||
"""For each required skill, find best matching candidate skill."""
|
||||
|
||||
async def _aggregate_similarity(
|
||||
self, required: list[str], candidate: list[str]
|
||||
) -> dict[str, dict[str, Any]]:
|
||||
"""Compare average embeddings of required vs candidate skills."""
|
||||
|
||||
async def compute_role_similarity(
|
||||
self, required_role: str, candidate_role: str
|
||||
) -> float:
|
||||
"""Compute role similarity (0.0 to 1.0).
|
||||
|
||||
Behavioral guarantees:
|
||||
- If either role is empty/None, return 0.0.
|
||||
- If either role is "(unknown)" (case-insensitive, trimmed), return 0.0.
|
||||
- Otherwise compute cosine similarity of the role embeddings.
|
||||
|
||||
Note:
|
||||
- Role similarity returns only a float score. Rationales are provided by
|
||||
`compute_competence_similarity()` results.
|
||||
"""
|
||||
```
|
||||
|
||||
### 5. TaskAnalyzer Refactoring
|
||||
|
||||
**Remove**:
|
||||
- `_sample_json()` method
|
||||
- `_heuristic_competences_from_text()` static method
|
||||
- All heuristic fallback code in:
|
||||
- `extract_ranked_roles()`
|
||||
- `_extract_requirements_from_description()`
|
||||
- `apply_requirement_update()`
|
||||
- `semantic_competence_similarity()`
|
||||
|
||||
**Replace with**:
|
||||
|
||||
```python
|
||||
class TaskAnalyzer:
|
||||
"""Task text analyzer using Azure OpenAI."""
|
||||
|
||||
def __init__(self, azure_client: AzureOpenAIClient):
|
||||
self._client = azure_client
|
||||
|
||||
async def extract_ranked_roles(
|
||||
self, description: str, limit: int = 5
|
||||
) -> list[RankedRole]:
|
||||
"""Extract ranked roles using Azure OpenAI LLM.
|
||||
|
||||
Raises:
|
||||
AzureAPIError: If API call fails.
|
||||
"""
|
||||
|
||||
async def _extract_requirements_from_description(
|
||||
self, description: str
|
||||
) -> ExtractedRequirements:
|
||||
"""Extract requirements using Azure OpenAI LLM.
|
||||
|
||||
Raises:
|
||||
AzureAPIError: If API call fails.
|
||||
"""
|
||||
|
||||
async def apply_requirement_update(
|
||||
self, current: Requirements, change_description: str
|
||||
) -> Requirements:
|
||||
"""Update requirements using Azure OpenAI LLM.
|
||||
|
||||
Raises:
|
||||
AzureAPIError: If API call fails.
|
||||
"""
|
||||
```
|
||||
|
||||
**Additional behavior change**:
|
||||
- The current `TaskAnalyzer` raises `AnalysisError` only when MCP sampling is present but fails/returns invalid JSON, and otherwise silently falls back to heuristics.
|
||||
- After this change, heuristics are removed. Any Azure OpenAI failure (HTTP error, timeout after retries, invalid JSON/schema) will result in an analysis exception (e.g. `AzureAPIError` and/or a narrower `AnalysisError`) and should be surfaced to callers.
|
||||
|
||||
**Testing impact**:
|
||||
- Existing tests that currently assert heuristic fallback behavior in `TaskAnalyzer` must be updated.
|
||||
- Replace “heuristic fallback expected” assertions with either:
|
||||
- mocked Azure responses (unit tests), or
|
||||
- explicit error expectations when Azure is unavailable.
|
||||
|
||||
### 6. Matcher & Scorer Integration
|
||||
|
||||
Update `src/teamlandkarte_mcp/matching/matcher.py`:
|
||||
|
||||
**Key Changes**:
|
||||
1. Accept `SimilarityEngine` in constructor
|
||||
2. Replace `TaskAnalyzer.semantic_competence_similarity()` calls with `SimilarityEngine.compute_competence_similarity()`
|
||||
3. Replace the existing `_role_similarity()` function with `SimilarityEngine.compute_role_similarity()`
|
||||
4. Maintain current scoring weights (competence 0.8, role 0.2)
|
||||
|
||||
**Details**:
|
||||
- The `Matcher` currently uses `TaskAnalyzer.semantic_competence_similarity()` for competence matching (per-skill, best-match strategy)
|
||||
- It also has a standalone `_role_similarity()` function for role matching (currently uses simple string comparison)
|
||||
- Both will be replaced by `SimilarityEngine` methods, which provide semantic similarity via embeddings
|
||||
- The `SimilarityEngine.compute_competence_similarity()` returns the same dict structure as current `semantic_competence_similarity()`, ensuring drop-in compatibility
|
||||
- The `SimilarityEngine.compute_role_similarity()` returns a float (0.0 to 1.0), matching current `_role_similarity()` signature
|
||||
|
||||
```python
|
||||
class Matcher:
|
||||
"""Capacity matcher with semantic similarity."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
analyzer: TaskAnalyzer,
|
||||
similarity_engine: SimilarityEngine,
|
||||
config: MatchingConfig
|
||||
):
|
||||
"""Initialize matcher with analyzer, similarity engine, and config."""
|
||||
self._analyzer = analyzer
|
||||
self._similarity = similarity_engine
|
||||
self._config = config
|
||||
|
||||
async def _compute_match_score(
|
||||
self,
|
||||
required: Requirements,
|
||||
capacity: Capacity
|
||||
) -> MatchScore:
|
||||
"""Compute match score using SimilarityEngine.
|
||||
|
||||
Competence matching:
|
||||
comp_sim = await self._similarity.compute_competence_similarity(
|
||||
required.competences, capacity.competences
|
||||
)
|
||||
|
||||
Role matching:
|
||||
role_score = await self._similarity.compute_role_similarity(
|
||||
required.role, capacity.role
|
||||
)
|
||||
|
||||
Final score: (0.8 * avg_comp_score) + (0.2 * role_score)
|
||||
"""
|
||||
```
|
||||
|
||||
**Owning module note**:
|
||||
- The end-to-end match scoring aggregation currently lives in `src/teamlandkarte_mcp/matching/matcher.py` (with supporting score shaping asserted by `tests/test_scorer.py`).
|
||||
|
||||
### 7. Server Initialization Changes
|
||||
|
||||
Update `src/teamlandkarte_mcp/mcp_server.py`:
|
||||
|
||||
```python
|
||||
# Initialize Azure OpenAI components
|
||||
embedding_cache = EmbeddingCache(
|
||||
db_path=cfg.embedding_cache.db_path,
|
||||
ttl_days=cfg.embedding_cache.ttl_days,
|
||||
) if cfg.embedding_cache.enabled else None
|
||||
|
||||
azure_client = AzureOpenAIClient(cfg.azure_openai, embedding_cache)
|
||||
similarity_engine = SimilarityEngine(
|
||||
azure_client,
|
||||
strategy=cfg.matching.similarity.strategy,
|
||||
)
|
||||
|
||||
# Initialize analyzer with Azure client
|
||||
analyzer = TaskAnalyzer(azure_client)
|
||||
|
||||
# Initialize matcher with similarity engine
|
||||
matcher = Matcher(analyzer, similarity_engine, cfg.matching)
|
||||
```
|
||||
|
||||
### 8. Cost Estimation & Monitoring
|
||||
|
||||
Add cost tracking module `src/teamlandkarte_mcp/azure/cost_tracker.py`:
|
||||
|
||||
```python
|
||||
class CostTracker:
|
||||
"""Track and estimate Azure OpenAI API costs."""
|
||||
|
||||
# Pricing (as of 2026-02, may change)
|
||||
EMBEDDING_COST_PER_1K_TOKENS = 0.00013 # text-embedding-3-large
|
||||
LLM_INPUT_COST_PER_1K_TOKENS = 0.03 # gpt-4.1
|
||||
LLM_OUTPUT_COST_PER_1K_TOKENS = 0.06 # gpt-4.1
|
||||
|
||||
def log_embedding_request(self, text: str, cached: bool):
|
||||
"""Log embedding request for cost tracking."""
|
||||
|
||||
def log_llm_request(self, input_tokens: int, output_tokens: int):
|
||||
"""Log LLM request for cost tracking."""
|
||||
|
||||
def get_session_costs(self) -> dict:
|
||||
"""Return estimated costs for current session."""
|
||||
```
|
||||
|
||||
Add to `README.md`:
|
||||
|
||||
```markdown
|
||||
## Azure OpenAI Cost Estimation
|
||||
|
||||
Typical matching workflow costs (estimated):
|
||||
|
||||
| Operation | API Calls | Estimated Cost |
|
||||
|-----------|-----------|----------------|
|
||||
| Single capacity match (100 candidates) | ~200 embeddings (mostly cached), 0 LLM | $0.001 - $0.01 |
|
||||
| Task requirements extraction | 0 embeddings, 1 LLM call (~1000 tokens) | $0.03 - $0.06 |
|
||||
| Role inference | 0 embeddings, 1 LLM call (~500 tokens) | $0.015 - $0.03 |
|
||||
|
||||
With embedding caching enabled (default), repeated searches are significantly cheaper.
|
||||
|
||||
**Monthly cost estimate** (100 searches/day, 20 days):
|
||||
- Without cache: ~$60-120/month
|
||||
- With cache (90% hit rate): ~$10-20/month
|
||||
```
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests (Mocked Azure)
|
||||
|
||||
All unit tests should mock Azure OpenAI API responses to avoid API costs and ensure deterministic behavior:
|
||||
|
||||
**Embedding Mocks**:
|
||||
- Use fixed-dimension vectors matching `text-embedding-3-large` (3072 dimensions)
|
||||
- Create realistic patterns:
|
||||
- Similar terms: `[0.9, 0.8, 0.1, ..., 0.0]` and `[0.85, 0.82, 0.15, ..., 0.0]` → high cosine similarity (~0.95)
|
||||
- Different terms: `[0.9, 0.1, 0.0, ..., 0.0]` and `[0.1, 0.9, 0.0, ..., 0.0]` → low cosine similarity (~0.1)
|
||||
- Use `unittest.mock.AsyncMock` or `pytest-mock` for `AsyncAzureOpenAI` client
|
||||
|
||||
**LLM Mocks**:
|
||||
- Mock `chat.completions.create()` to return structured JSON responses
|
||||
- Example: `{"roles": [{"role": "Backend Developer", "confidence": 0.9, "rationale": "..."}], ...}`
|
||||
|
||||
**Example Mock Pattern**:
|
||||
```python
|
||||
@pytest.mark.asyncio
|
||||
async def test_compute_competence_similarity():
|
||||
mock_client = AsyncMock()
|
||||
mock_client.get_embedding.side_effect = [
|
||||
[0.9, 0.8, 0.1] + [0.0] * 3069, # "Python"
|
||||
[0.85, 0.82, 0.15] + [0.0] * 3069, # "Python 3"
|
||||
]
|
||||
engine = SimilarityEngine(mock_client, strategy="per_skill")
|
||||
result = await engine.compute_competence_similarity(["Python"], ["Python 3"])
|
||||
assert result["Python"]["score"] > 0.9
|
||||
```
|
||||
|
||||
### Integration Tests (Real Azure)
|
||||
|
||||
- Mark all integration tests with `@pytest.mark.integration`
|
||||
- Configure `pytest.ini`:
|
||||
```ini
|
||||
[pytest]
|
||||
markers =
|
||||
integration: marks tests that call real Azure OpenAI API (deselect with '-m "not integration"')
|
||||
```
|
||||
- Run integration tests manually before deployment: `pytest -m integration`
|
||||
- Run fast tests in CI: `pytest -m "not integration"`
|
||||
- Integration tests should use small, cheap prompts to minimize costs
|
||||
|
||||
### Test Coverage Goals
|
||||
|
||||
- **Unit tests**: ≥90% coverage for all new code (cache, client, similarity, analyzer)
|
||||
- **Integration tests**: Cover happy path + common error scenarios (API timeout, invalid credentials)
|
||||
- **Manual tests**: Full end-to-end workflow in Cherry Studio (search → filter → confirm)
|
||||
|
||||
## Implementation Task List
|
||||
|
||||
### Phase 1: Configuration & Infrastructure (3-4 hours)
|
||||
|
||||
- [ ] 1.1 Rename `database.toml` → `config.toml` and template
|
||||
- [ ] 1.2 Update all file references in code
|
||||
- [ ] 1.3 Add `[azure_openai]` section to config model (`src/config.py`)
|
||||
- [ ] 1.4 Add `[embedding_cache]` section to config model
|
||||
- [ ] 1.5 Add `[matching.similarity]` section to config model
|
||||
- [ ] 1.6 Update `config.toml.example` with new sections
|
||||
- [ ] 1.7 Load `AZURE_OPENAI_EMBEDDING_API_KEY` from `.env`
|
||||
- [ ] 1.8 Load `AZURE_OPENAI_LLM_API_KEY` from `.env`
|
||||
- [ ] 1.9 Add `openai` package to `pyproject.toml` dependencies
|
||||
|
||||
### Phase 2: Embedding Cache (2-3 hours)
|
||||
|
||||
- [ ] 2.1 Create `src/teamlandkarte_mcp/cache/embedding_cache.py`
|
||||
- [ ] 2.2 Implement `EmbeddingCache.__init__` (create DB/schema if not exists)
|
||||
- [ ] 2.3 Implement `EmbeddingCache.get()` (with TTL check)
|
||||
- [ ] 2.4 Implement `EmbeddingCache.put()`
|
||||
- [ ] 2.5 Implement `EmbeddingCache.cleanup_expired()`
|
||||
- [ ] 2.6 Add SQLite schema migration support (if DB exists but schema old)
|
||||
- [ ] 2.7 Add unit tests for `EmbeddingCache`
|
||||
- [ ] 2.8 Add `.gitignore` entry for `embeddings_cache.db`
|
||||
|
||||
### Phase 3: Azure OpenAI Client (3-4 hours)
|
||||
|
||||
**Testing Strategy**:
|
||||
- Unit tests should mock Azure API responses with realistic embedding vectors (3072 dimensions for text-embedding-3-large)
|
||||
- Use fixed seed embeddings for deterministic test behavior (e.g., `[0.1, 0.2, ..., 0.0]` patterns)
|
||||
- Integration tests marked with `@pytest.mark.integration` use real API calls
|
||||
|
||||
- [ ] 3.1 Create `src/teamlandkarte_mcp/azure/__init__.py`
|
||||
- [ ] 3.2 Create `src/teamlandkarte_mcp/azure/openai_client.py`
|
||||
- [ ] 3.3 Implement `AzureOpenAIClient.__init__`
|
||||
- [ ] 3.4 Implement `AzureOpenAIClient.get_embedding()` with cache integration
|
||||
- [ ] 3.5 Implement `AzureOpenAIClient.get_embeddings_batch()` (individual calls)
|
||||
- [ ] 3.6 Implement `AzureOpenAIClient.chat_completion()` with retry logic
|
||||
- [ ] 3.7 Add `AzureAPIError` exception class
|
||||
- [ ] 3.8 Add unit tests for client (mocked API)
|
||||
- [ ] 3.9 Add integration test with real API (marked as `@pytest.mark.integration`)
|
||||
|
||||
### Phase 4: Similarity Engine (4-5 hours)
|
||||
|
||||
- [ ] 4.1 Create `src/teamlandkarte_mcp/matching/similarity.py`
|
||||
- [ ] 4.2 Implement `cosine_similarity()` function
|
||||
- [ ] 4.3 Implement `SimilarityEngine.__init__`
|
||||
- [ ] 4.4 Implement `SimilarityEngine._per_skill_similarity()`
|
||||
- [ ] 4.5 Implement `SimilarityEngine._aggregate_similarity()`
|
||||
- [ ] 4.6 Implement `SimilarityEngine.compute_competence_similarity()` (router)
|
||||
- [ ] 4.7 Implement `SimilarityEngine.compute_role_similarity()`
|
||||
- [ ] 4.8 Add unit tests for similarity computation (mocked embeddings)
|
||||
- [ ] 4.9 Add integration test comparing both strategies
|
||||
|
||||
### Phase 5: TaskAnalyzer Refactoring (3-4 hours)
|
||||
|
||||
- [ ] 5.1 Remove `TaskAnalyzer.__init__(self, mcp: FastMCP)` signature
|
||||
- [ ] 5.2 Add new `TaskAnalyzer.__init__(self, azure_client: AzureOpenAIClient)`
|
||||
- [ ] 5.3 Remove `_sample_json()` method completely
|
||||
- [ ] 5.4 Remove `_heuristic_competences_from_text()` method completely
|
||||
- [ ] 5.5 Refactor `extract_ranked_roles()` to use `azure_client.chat_completion()`
|
||||
- [ ] 5.6 Remove heuristic fallback from `extract_ranked_roles()`
|
||||
- [ ] 5.7 Refactor `_extract_requirements_from_description()` to use LLM
|
||||
- [ ] 5.8 Remove heuristic fallback from `_extract_requirements_from_description()`
|
||||
- [ ] 5.9 Refactor `apply_requirement_update()` to use LLM
|
||||
- [ ] 5.10 Remove heuristic fallback from `apply_requirement_update()`
|
||||
- [ ] 5.11 Remove `semantic_competence_similarity()` method (replaced by `SimilarityEngine`)
|
||||
- [ ] 5.12 Update all `TaskAnalyzer` unit tests
|
||||
|
||||
### Phase 6: Matcher & Scorer Integration (2-3 hours)
|
||||
|
||||
- [ ] 6.1 Update `Matcher.__init__` to accept `SimilarityEngine`
|
||||
- [ ] 6.2 Replace competence similarity calls with `SimilarityEngine.compute_competence_similarity()`
|
||||
- [ ] 6.3 Replace role similarity calls with `SimilarityEngine.compute_role_similarity()`
|
||||
- [ ] 6.4 Update scorer aggregation logic if needed
|
||||
- [ ] 6.5 Update integration tests for matching pipeline
|
||||
|
||||
### Phase 7: Server Initialization (2 hours)
|
||||
|
||||
- [ ] 7.1 Update `build_server()` in `mcp_server.py` to load Azure config
|
||||
- [ ] 7.2 Initialize `EmbeddingCache` instance
|
||||
- [ ] 7.3 Initialize `AzureOpenAIClient` instance
|
||||
- [ ] 7.4 Initialize `SimilarityEngine` instance
|
||||
- [ ] 7.5 Update `TaskAnalyzer` instantiation
|
||||
- [ ] 7.6 Update `Matcher` instantiation
|
||||
- [ ] 7.7 Add startup logging for Azure OpenAI connection
|
||||
- [ ] 7.8 Add graceful error handling if Azure credentials missing
|
||||
|
||||
### Phase 8: Cost Tracking & Monitoring (2 hours)
|
||||
|
||||
- [ ] 8.1 Create `src/teamlandkarte_mcp/azure/cost_tracker.py`
|
||||
- [ ] 8.2 Implement `CostTracker` class with logging methods
|
||||
- [ ] 8.3 Integrate cost tracking into `AzureOpenAIClient`
|
||||
- [ ] 8.4 Add periodic cost report logging (stderr)
|
||||
- [ ] 8.5 Add cost summary to tool outputs (optional, via config)
|
||||
|
||||
### Phase 9: Documentation (2-3 hours)
|
||||
|
||||
- [ ] 9.1 Update `README.md` with Azure OpenAI setup instructions
|
||||
- [ ] 9.2 Add Azure cost estimation section to `README.md`
|
||||
- [ ] 9.3 Update `docs/troubleshooting.md` with Azure API error guidance
|
||||
- [ ] 9.4 Document similarity strategies (`per_skill` vs `aggregate`)
|
||||
- [ ] 9.5 Update `config.toml.example` with comprehensive comments
|
||||
- [ ] 9.6 Update OpenSpec architecture docs
|
||||
- [ ] 9.7 Add migration guide from old `database.toml` to new `config.toml`
|
||||
|
||||
### Phase 10: Testing & Validation (3-4 hours)
|
||||
|
||||
- [ ] 10.1 Configure `pytest.ini` with integration marker: `markers = integration: marks tests that call real Azure OpenAI API (deselect with '-m "not integration"')`
|
||||
- [ ] 10.2 Run full test suite after refactoring (`pytest -m "not integration"` for fast CI)
|
||||
- [ ] 10.3 Add new integration tests for Azure API paths (marked with `@pytest.mark.integration`)
|
||||
- [ ] 10.4 Test embedding cache persistence across server restarts
|
||||
- [ ] 10.5 Test both similarity strategies with real data
|
||||
- [ ] 10.6 Validate cost tracking accuracy
|
||||
- [ ] 10.7 Test error handling when Azure API unavailable
|
||||
- [ ] 10.8 Performance benchmark: compare cache hit/miss scenarios
|
||||
- [ ] 10.9 Manual validation in Cherry Studio
|
||||
|
||||
### Phase 11: Cleanup (1 hour)
|
||||
|
||||
- [ ] 11.1 Remove all commented-out FastMCP sampling code
|
||||
- [ ] 11.2 Remove unused imports (`from mcp.server.fastmcp import FastMCP` from `TaskAnalyzer`)
|
||||
- [ ] 11.3 Update type hints and docstrings
|
||||
- [ ] 11.4 Run linter and fix style issues
|
||||
- [ ] 11.5 Final code review
|
||||
|
||||
## Impact Analysis
|
||||
|
||||
### Changed Files
|
||||
|
||||
**New files**:
|
||||
- `config.toml` (renamed from `database.toml`)
|
||||
- `config.toml.example` (renamed from `database.toml.example`)
|
||||
- `src/teamlandkarte_mcp/cache/embedding_cache.py`
|
||||
- `src/teamlandkarte_mcp/azure/__init__.py`
|
||||
- `src/teamlandkarte_mcp/azure/openai_client.py`
|
||||
- `src/teamlandkarte_mcp/azure/cost_tracker.py`
|
||||
- `src/teamlandkarte_mcp/matching/similarity.py`
|
||||
- `embeddings_cache.db` (generated at runtime, gitignored)
|
||||
|
||||
**Modified files**:
|
||||
- `src/teamlandkarte_mcp/config.py` (new config sections)
|
||||
- `src/teamlandkarte_mcp/matching/task_analyzer.py` (complete refactor)
|
||||
- `src/teamlandkarte_mcp/matching/matcher.py` (similarity engine integration)
|
||||
- `src/teamlandkarte_mcp/mcp_server.py` (initialization changes)
|
||||
- `README.md` (setup instructions, cost docs)
|
||||
- `docs/troubleshooting.md` (Azure error guidance)
|
||||
- `pyproject.toml` (add `openai` dependency)
|
||||
- `.gitignore` (add `embeddings_cache.db`, `config.toml`)
|
||||
- All OpenSpec architecture/design docs
|
||||
|
||||
**Deleted code**:
|
||||
- All FastMCP sampling code in `TaskAnalyzer`
|
||||
- All heuristic fallback code in `TaskAnalyzer`
|
||||
- `_sample_json()`, `_heuristic_competences_from_text()` methods
|
||||
|
||||
### Breaking Changes
|
||||
|
||||
1. **Configuration file renamed**: Users must rename `database.toml` → `config.toml`
|
||||
2. **New required environment variables**: `AZURE_OPENAI_EMBEDDING_API_KEY`, `AZURE_OPENAI_LLM_API_KEY`
|
||||
3. **Hard dependency on Azure OpenAI**: No offline/fallback mode
|
||||
4. **New dependency**: `openai` Python package
|
||||
5. **TaskAnalyzer constructor signature changed**: `TaskAnalyzer.__init__(self, mcp: FastMCP)` → `TaskAnalyzer.__init__(self, azure_client: AzureOpenAIClient)`
|
||||
- This affects any code that instantiates `TaskAnalyzer` directly
|
||||
- Server initialization in `mcp_server.py` must be updated
|
||||
6. **Matcher constructor signature changed**: `Matcher.__init__(self, analyzer: TaskAnalyzer, config: MatchingConfig)` → `Matcher.__init__(self, analyzer: TaskAnalyzer, similarity_engine: SimilarityEngine, config: MatchingConfig)`
|
||||
- Adds new required `similarity_engine` parameter
|
||||
- Server initialization must pass `SimilarityEngine` instance
|
||||
|
||||
### Migration Path
|
||||
|
||||
1. Rename `database.toml` → `config.toml`
|
||||
2. Add new Azure OpenAI sections to `config.toml`
|
||||
3. Add Azure API keys to `.env`
|
||||
4. Install updated dependencies: `uv sync`
|
||||
5. First run will create `embeddings_cache.db` automatically
|
||||
|
||||
## Security & Constraints
|
||||
|
||||
### Security
|
||||
|
||||
- **API keys in `.env`**: Never commit `.env` or `config.toml` with credentials
|
||||
- **Embedding cache**: Contains only embeddings, not raw capacity data (safe to persist)
|
||||
- **Network**: All API calls over HTTPS to Azure OpenAI endpoint
|
||||
- **Error messages**: Do not log API keys in error messages
|
||||
|
||||
### Operational Constraints
|
||||
|
||||
- **Azure OpenAI dependency**: System will fail if Azure API unavailable (no fallback)
|
||||
- **API rate limits**: Azure OpenAI enforces rate limits (TPM/RPM); implement retry with backoff
|
||||
- **Cost**: Real monetary cost per API call (mitigated by caching)
|
||||
- **Latency**: First search for a capacity will be slower (embedding generation); subsequent searches fast (cached)
|
||||
|
||||
### Configuration Defaults
|
||||
|
||||
```toml
|
||||
[embedding_cache]
|
||||
enabled = true
|
||||
db_path = "embeddings_cache.db"
|
||||
ttl_days = 30
|
||||
|
||||
[matching.similarity]
|
||||
strategy = "per_skill" # Maintains current behavior
|
||||
```
|
||||
|
||||
## Cost Estimation Details
|
||||
|
||||
### API Pricing (as of 2026-02)
|
||||
|
||||
| Service | Model | Cost |
|
||||
|---------|-------|------|
|
||||
| Embeddings | text-embedding-3-large | $0.00013 per 1K tokens |
|
||||
| LLM (input) | gpt-4.1 | $0.03 per 1K tokens |
|
||||
| LLM (output) | gpt-4.1 | $0.06 per 1K tokens |
|
||||
|
||||
### Typical Workflow Costs
|
||||
|
||||
**Scenario 1: Task requirements extraction**
|
||||
- Input: ~500 tokens (task description)
|
||||
- Output: ~300 tokens (JSON with roles/competences/dates)
|
||||
- Cost: (500 × $0.03 / 1000) + (300 × $0.06 / 1000) = **$0.033**
|
||||
|
||||
**Scenario 2: Capacity matching (100 candidates, 5 required skills)**
|
||||
- First run (cold cache):
|
||||
- Required skills: 5 × ~10 tokens = 50 tokens → ~$0.0000065
|
||||
- Candidate skills: 100 candidates × 3 skills avg × 10 tokens = 3000 tokens → ~$0.00039
|
||||
- Total: **~$0.0004** (negligible)
|
||||
- Subsequent runs (warm cache): **$0** (all cached)
|
||||
|
||||
**Scenario 3: Role inference**
|
||||
- Input: ~400 tokens
|
||||
- Output: ~200 tokens
|
||||
- Cost: **~$0.024**
|
||||
|
||||
### Monthly Estimates
|
||||
|
||||
**Light usage** (20 searches/month, 10 extractions):
|
||||
- Embeddings: ~$0.05
|
||||
- LLM: ~$0.50
|
||||
- **Total: ~$0.55/month**
|
||||
|
||||
**Moderate usage** (100 searches/month, 50 extractions):
|
||||
- Embeddings: ~$0.20 (with 90% cache hit rate)
|
||||
- LLM: ~$2.50
|
||||
- **Total: ~$2.70/month**
|
||||
|
||||
**Heavy usage** (500 searches/month, 200 extractions):
|
||||
- Embeddings: ~$1.00 (with 90% cache hit rate)
|
||||
- LLM: ~$10.00
|
||||
- **Total: ~$11/month**
|
||||
|
||||
## Open Questions
|
||||
|
||||
None (all clarifications provided by user).
|
||||
|
||||
## Approval & Timeline
|
||||
|
||||
- **Estimated effort**: 28-35 hours (1 week full-time or 2 weeks part-time)
|
||||
- **Risk level**: Medium (API dependency, cost implications, major refactor)
|
||||
- **Approval required**: Yes (architecture change, new external dependency)
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. Review and approve this proposal
|
||||
2. Create detailed implementation branch
|
||||
3. Implement phases 1-11 sequentially
|
||||
4. Conduct thorough testing (unit + integration + manual)
|
||||
5. Document migration guide
|
||||
6. Deploy to staging environment
|
||||
7. Monitor costs and performance
|
||||
8. Deploy to production
|
||||
+359
@@ -0,0 +1,359 @@
|
||||
# Tasks: Replace Heuristics with Azure OpenAI
|
||||
|
||||
> Change: `replace-heuristics-with-azure-openai`
|
||||
>
|
||||
> Status: **Approved / Implemented**
|
||||
|
||||
## Overview
|
||||
|
||||
Replace legacy client-side sampling/heuristic logic with direct Azure OpenAI integration:
|
||||
- Embeddings (`text-embedding-3-large`) for competence/role similarity
|
||||
- LLM (`gpt-4.1`) for requirements extraction and validation
|
||||
- Persistent SQLite embedding cache
|
||||
- Two configurable similarity strategies
|
||||
|
||||
# Removed: migration note about database.toml rename (now a completed internal detail).
|
||||
|
||||
## Estimated Effort
|
||||
|
||||
**Total: 28-35 hours** (1 week full-time or 2 weeks part-time)
|
||||
|
||||
## Task Breakdown
|
||||
|
||||
### Phase 1: Configuration & Infrastructure (3-4 hours)
|
||||
|
||||
- [x] 1.1 Rename `database.toml` → `config.toml`
|
||||
- [x] 1.2 Rename `database.toml.example` → `config.toml.example`
|
||||
- [x] 1.3 Update all file references in code (`README.md`, `docs/`, `src/config.py`, etc.)
|
||||
- [x] 1.4 Extend `src/config.py`: Add `AzureOpenAIConfig` model
|
||||
- `endpoint`, `api_version_embeddings`, `api_version_llm`
|
||||
- `embeddings.model`, `embeddings.deployment` (deployment must equal model name)
|
||||
- `llm.model`, `llm.deployment` (deployment must equal model name), `llm.temperature`, `llm.max_tokens`
|
||||
- [x] 1.5 Extend `src/config.py`: Add `EmbeddingCacheConfig` model
|
||||
- `enabled`, `db_path`, `ttl_days`
|
||||
- [x] 1.6 Extend `src/config.py`: Add `matching.similarity.strategy` field (`"per_skill"` or `"aggregate"`)
|
||||
- [x] 1.7 Update `config.toml.example` with new `[azure_openai]`, `[embedding_cache]`, `[matching.similarity]` sections
|
||||
- [x] 1.8 Load `AZURE_OPENAI_EMBEDDING_API_KEY` from environment in `load_config()`
|
||||
- [x] 1.9 Load `AZURE_OPENAI_LLM_API_KEY` from environment in `load_config()`
|
||||
- [x] 1.10 Add `openai` package to `pyproject.toml` dependencies
|
||||
- [x] 1.11 Run `uv sync` to install dependencies
|
||||
- [x] 1.12 Add `config.toml` to `.gitignore` (if not already there)
|
||||
- [x] 1.13 Configure pytest integration marker in `pytest.ini` (if not already present):
|
||||
- `integration: marks tests that call real Azure OpenAI API (deselect with '-m "not integration"')`
|
||||
|
||||
### Phase 2: Embedding Cache (2-3 hours)
|
||||
|
||||
- [x] 2.1 Create `src/teamlandkarte_mcp/cache/embedding_cache.py`
|
||||
- [x] 2.2 Implement `EmbeddingCache` class:
|
||||
- `__init__(db_path: str, ttl_days: int)`
|
||||
- Create SQLite database and schema if not exists:
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS embeddings (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
text TEXT UNIQUE NOT NULL,
|
||||
model TEXT NOT NULL,
|
||||
embedding BLOB NOT NULL,
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||
)
|
||||
CREATE INDEX IF NOT EXISTS idx_text_model ON embeddings(text, model)
|
||||
```
|
||||
- [x] 2.3 Implement `EmbeddingCache.get(text: str, model: str) -> Optional[list[float]]`
|
||||
- Query by text + model
|
||||
- Check if `created_at` is within TTL
|
||||
- Deserialize BLOB to `list[float]`
|
||||
- Return `None` if missing or expired
|
||||
- [x] 2.4 Implement `EmbeddingCache.put(text: str, model: str, embedding: list[float]) -> None`
|
||||
- Serialize embedding as UTF-8 bytes of JSON: `json.dumps(embedding).encode("utf-8")`
|
||||
- INSERT OR REPLACE into database
|
||||
- [x] 2.5 Implement `EmbeddingCache.cleanup_expired() -> int`
|
||||
- DELETE embeddings older than TTL
|
||||
- Return count of deleted rows
|
||||
- [x] 2.6 Add graceful handling for concurrent access (SQLite locks)
|
||||
- [x] 2.7 Create `tests/test_embedding_cache.py`
|
||||
- Test cache hit/miss
|
||||
- Test TTL expiration
|
||||
- Test cleanup
|
||||
- Test concurrent access
|
||||
- [x] 2.8 Add `embeddings_cache.db` to `.gitignore`
|
||||
|
||||
### Phase 3: Azure OpenAI Client (3-4 hours)
|
||||
|
||||
- [x] 3.1 Create `src/teamlandkarte_mcp/azure/__init__.py`
|
||||
- [x] 3.2 Create `src/teamlandkarte_mcp/azure/openai_client.py`
|
||||
- [x] 3.3 Define `AzureAPIError(RuntimeError)` exception class
|
||||
- [x] 3.4 Implement `AzureOpenAIClient` class:
|
||||
- `__init__(config: AzureOpenAIConfig, embedding_cache: Optional[EmbeddingCache])`
|
||||
- Initialize two `AzureOpenAI` clients (embeddings + LLM) using `AzureKeyCredential`
|
||||
- [x] 3.5 Implement `AzureOpenAIClient.get_embedding(text: str) -> list[float]`
|
||||
- Check cache first (if enabled)
|
||||
- If cache miss: call Azure API
|
||||
- Store in cache (if enabled)
|
||||
- Raise `AzureAPIError` on failure (no fallback)
|
||||
- [x] 3.6 Implement `AzureOpenAIClient.get_embeddings_batch(texts: list[str]) -> list[list[float]]`
|
||||
- Call `get_embedding()` for each text individually (per user requirement #3)
|
||||
- Return list of embeddings in same order
|
||||
- [x] 3.7 Implement `AzureOpenAIClient.chat_completion(system: str, user: str, response_format: Optional[dict] = None) -> str`
|
||||
- Call Azure OpenAI chat completions API
|
||||
- Use `gpt-4.1` model with configured temperature/max_tokens
|
||||
- Support `response_format={"type": "json_object"}` for structured JSON
|
||||
- Implement retry logic (3 attempts, exponential backoff)
|
||||
- Raise `AzureAPIError` on final failure (no fallback)
|
||||
- [x] 3.8 Add logging for API calls (cache hit/miss, request/response sizes)
|
||||
- [x] 3.9 Create `tests/test_azure_client.py`
|
||||
- Mock Azure API with `unittest.mock`
|
||||
- Test embedding retrieval with cache hit/miss
|
||||
- Test LLM completion
|
||||
- Test error handling and retries
|
||||
- [x] 3.10 Create `tests/test_azure_client_integration.py` (marked `@pytest.mark.integration`)
|
||||
- Real API calls (requires valid credentials)
|
||||
- Test embedding generation
|
||||
- Test LLM structured JSON response
|
||||
|
||||
### Phase 4: Similarity Engine (4-5 hours)
|
||||
|
||||
- [x] 4.1 Create `src/teamlandkarte_mcp/matching/similarity.py`
|
||||
- [x] 4.2 Implement `cosine_similarity(vec_a: list[float], vec_b: list[float]) -> float`
|
||||
- Compute dot product and magnitudes
|
||||
- Return cosine similarity in [0.0, 1.0]
|
||||
- Handle zero vectors gracefully
|
||||
- [x] 4.3 Implement `SimilarityEngine` class:
|
||||
- `__init__(client: AzureOpenAIClient, strategy: str = "per_skill")`
|
||||
- Validate strategy is `"per_skill"` or `"aggregate"`
|
||||
- [x] 4.4 Implement `SimilarityEngine._per_skill_similarity(required: list[str], candidate: list[str]) -> dict[str, dict[str, Any]]`
|
||||
- For each required skill:
|
||||
- Get embedding
|
||||
- Compare with all candidate skill embeddings
|
||||
- Find best match (highest cosine similarity)
|
||||
- Return `{required: {best_match: str|None, score: float, rationale: str}}`
|
||||
- Rationale example: `"Cosine similarity: 0.92 (best match: 'React.js')"`
|
||||
- [x] 4.5 Implement `SimilarityEngine._aggregate_similarity(required: list[str], candidate: list[str]) -> dict[str, dict[str, Any]]`
|
||||
- Compute average embedding of all required skills
|
||||
- Compute average embedding of all candidate skills
|
||||
- Compute cosine similarity between averages
|
||||
- Return same format as `_per_skill_similarity` (each required skill gets same aggregate score)
|
||||
- Rationale example: `"Aggregate embedding similarity: 0.85"`
|
||||
- [x] 4.6 Implement `SimilarityEngine.compute_competence_similarity(required: list[str], candidate: list[str]) -> dict[str, dict[str, Any]]`
|
||||
- Route to `_per_skill_similarity()` or `_aggregate_similarity()` based on configured strategy
|
||||
- Maintain same output format as current `TaskAnalyzer.semantic_competence_similarity()`
|
||||
- Guarantee `rationale` is always present (even when best_match is None)
|
||||
- Edge cases:
|
||||
- `required == []` → return `{}`
|
||||
- `candidate == []` → return per-required entry with `best_match=None`, `score=0.0`, and rationale
|
||||
- [x] 4.7 Implement `SimilarityEngine.compute_role_similarity(required_role: str, candidate_role: str) -> float`
|
||||
- Edge cases:
|
||||
- If either role is empty/None → return 0.0
|
||||
- If either role is "(unknown)" (case-insensitive, trimmed) → return 0.0
|
||||
- Otherwise compute cosine similarity of role embeddings
|
||||
- [x] 4.8 Create `tests/test_similarity.py`
|
||||
- Test `cosine_similarity()` with known vectors
|
||||
- Mock embeddings and test both strategies
|
||||
- Test edge cases (empty lists, identical skills, no matches)
|
||||
- [x] 4.9 Create `tests/test_similarity_integration.py` (marked `@pytest.mark.integration`)
|
||||
- Compare both strategies with real embeddings
|
||||
- Validate scores are reasonable (e.g., "Python" vs "Python 3" → high similarity)
|
||||
|
||||
### Phase 5: TaskAnalyzer Refactoring (3-4 hours)
|
||||
|
||||
- [x] 5.1 Update `src/teamlandkarte_mcp/matching/task_analyzer.py`
|
||||
- [x] 5.2 Change `TaskAnalyzer.__init__` signature:
|
||||
- Remove: `__init__(self, mcp: FastMCP)`
|
||||
- Add: `__init__(self, azure_client: AzureOpenAIClient)`
|
||||
- [x] 5.3 **Delete** `_sample_json()` method completely
|
||||
- [x] 5.4 **Delete** `_heuristic_competences_from_text()` static method completely
|
||||
- [x] 5.5 Refactor `extract_ranked_roles(description: str, limit: int = 5) -> list[RankedRole]`:
|
||||
- Remove all heuristic fallback code
|
||||
- Build prompt for structured JSON response:
|
||||
```
|
||||
system: "Extract a ranked list of possible roles from a task description. Return STRICT JSON: {roles:[{rank:int, role:str, rationale:str}]}"
|
||||
user: "Limit: {limit}\n\nTask description:\n{description}"
|
||||
```
|
||||
- Call `self._client.chat_completion(system, user, response_format={"type": "json_object"})`
|
||||
- Parse JSON response and build `list[RankedRole]`
|
||||
- Raise `AzureAPIError` on failure (no fallback)
|
||||
- [x] 5.6 Refactor `_extract_requirements_from_description(description: str) -> ExtractedRequirements`:
|
||||
- Remove all heuristic fallback code
|
||||
- Build prompt for structured JSON:
|
||||
```
|
||||
system: "Extract structured requirements from a task description. Return STRICT JSON with keys: competences (list[str]), date_start (YYYY-MM-DD|null), date_end (YYYY-MM-DD|null), roles ([{rank, role, rationale}])."
|
||||
user: "Task description:\n{description}"
|
||||
```
|
||||
- Call `self._client.chat_completion(system, user, response_format={"type": "json_object"})`
|
||||
- Parse JSON and build `ExtractedRequirements`
|
||||
- Raise `AzureAPIError` on failure (no fallback)
|
||||
- [x] 5.7 Refactor `apply_requirement_update(current: Requirements, change_description: str) -> Requirements`:
|
||||
- Remove all heuristic fallback code
|
||||
- Build prompt with current requirements + change request
|
||||
- Call `self._client.chat_completion(system, user, response_format={"type": "json_object"})`
|
||||
- Parse JSON and build updated `Requirements`
|
||||
- Raise `AzureAPIError` on failure (no fallback)
|
||||
- [x] 5.8 **Delete** `semantic_competence_similarity()` method completely (replaced by `SimilarityEngine`)
|
||||
- [x] 5.9 Update `ExtractedRequirements` dataclass docstring (no longer mentions heuristics)
|
||||
- [x] 5.10 Update `TaskAnalyzer` class docstring (remove FastMCP sampling references, mention Azure OpenAI)
|
||||
- [x] 5.11 Remove unused imports (`FastMCP`, `random`, `asyncio` if no longer needed)
|
||||
- [x] 5.12 Update all `tests/test_task_analyzer.py` (or similar):
|
||||
- Mock `AzureOpenAIClient` instead of `FastMCP`
|
||||
- Remove heuristic fallback tests
|
||||
- Add tests for Azure API error propagation
|
||||
- [x] 5.13 Update `AnalysisError` semantics (or rename/remove if no longer needed):
|
||||
- Previously: heuristic fallback masked missing sampling
|
||||
- Now: Azure failures must surface as exceptions (no heuristics)
|
||||
|
||||
### Phase 6: Matcher & Scorer Integration (2-3 hours)
|
||||
|
||||
- [x] 6.1 Identify where `Matcher` or scoring logic calls `TaskAnalyzer.semantic_competence_similarity()`
|
||||
- [x] 6.2 Update `Matcher.__init__` to accept `SimilarityEngine` as parameter
|
||||
- [x] 6.3 Replace `analyzer.semantic_competence_similarity(required, candidate)` with `similarity_engine.compute_competence_similarity(required, candidate)`
|
||||
- [x] 6.4 Update role matching to use `similarity_engine.compute_role_similarity(required_role, candidate_role)`
|
||||
- [x] 6.5 Ensure scorer aggregation logic works with new similarity output format (should be identical)
|
||||
- [x] 6.6 Update `tests/test_scorer.py` (and/or add `tests/test_matcher.py` if introduced):
|
||||
- Mock `SimilarityEngine` instead of `TaskAnalyzer.semantic_competence_similarity`
|
||||
- Test that both similarity strategies produce valid scores
|
||||
- [x] 6.7 Update matcher/scorer tests to reflect new edge-case contracts:
|
||||
- role similarity returns 0.0 for empty/"(unknown)" roles
|
||||
- competence similarity returns 0.0 when candidate list is empty
|
||||
|
||||
### Phase 7: Server Initialization (2 hours)
|
||||
|
||||
- [x] 7.1 Update `build_server()` in `src/teamlandkarte_mcp/mcp_server.py`
|
||||
- [x] 7.2 Update server build to use top-level embedding cache config (not nested under azure):
|
||||
- `cfg.embedding_cache.enabled/db_path/ttl_days`
|
||||
- [x] 7.3 Initialize `EmbeddingCache` (if enabled) using `cfg.embedding_cache`
|
||||
- [x] 7.4 Initialize `AzureOpenAIClient` using `cfg.azure_openai`
|
||||
- [x] 7.5 Initialize `SimilarityEngine` using `cfg.matching.similarity.strategy`
|
||||
- [x] 7.6 Update `TaskAnalyzer` instantiation
|
||||
- [x] 7.7 Update `Matcher` instantiation
|
||||
- [x] 7.8 Add startup logging (stderr):
|
||||
- Log Azure OpenAI endpoint
|
||||
- Log embedding cache status (enabled/disabled, path)
|
||||
- Log similarity strategy
|
||||
- [x] 7.9 Add graceful error handling if Azure credentials missing:
|
||||
- Log error to stderr
|
||||
- Exit with clear message: "Azure OpenAI credentials not found in environment"
|
||||
|
||||
### Phase 8: Cost Tracking & Monitoring (2 hours)
|
||||
|
||||
- [x] 8.1 Create `src/teamlandkarte_mcp/azure/cost_tracker.py`
|
||||
- [x] 8.2 Implement `CostTracker` class:
|
||||
- Class-level constants for pricing
|
||||
- `log_embedding_request(text: str, cached: bool)`
|
||||
- `log_llm_request(input_tokens: int, output_tokens: int)`
|
||||
- `get_session_costs() -> dict` (return totals)
|
||||
- [x] 8.3 Integrate `CostTracker` into `AzureOpenAIClient`:
|
||||
- Track embedding requests (cache hit/miss)
|
||||
- Track LLM requests (token counts from API response)
|
||||
- [x] 8.4 Add periodic cost logging to stderr:
|
||||
- Every 10 API calls or every 5 minutes
|
||||
- Log: `"Azure OpenAI costs this session: embeddings=$X.XX, llm=$Y.YY, total=$Z.ZZ"`
|
||||
|
||||
### Phase 9: Documentation (2-3 hours)
|
||||
|
||||
- [x] 9.1 Update `README.md`:
|
||||
- Add "Azure OpenAI Setup" section with prerequisites
|
||||
- Document environment variables (`AZURE_OPENAI_EMBEDDING_API_KEY`, `AZURE_OPENAI_LLM_API_KEY`)
|
||||
- Document `config.toml` Azure sections
|
||||
- Add cost estimation table
|
||||
- Add embedding cache explanation
|
||||
- Add similarity strategy comparison
|
||||
- [x] 9.2 Add migration guide to `README.md`:
|
||||
- Step 1: Rename `database.toml` → `config.toml`
|
||||
- Step 2: Add Azure sections to config
|
||||
- Step 3: Add Azure API keys to `.env`
|
||||
- Step 4: Run `uv sync`
|
||||
- Step 5: Start server (embedding cache will be created automatically)
|
||||
- [x] 9.3 Update `docs/troubleshooting.md`:
|
||||
- Add section: "Azure OpenAI API Errors"
|
||||
- Document common errors (rate limits, invalid credentials, network issues)
|
||||
- Document `AzureAPIError` and how to debug
|
||||
- [x] 9.4 Update `config.toml.example`:
|
||||
- Add comprehensive comments for all Azure sections
|
||||
- Document both similarity strategies with examples
|
||||
- Document embedding cache TTL trade-offs
|
||||
- [x] 9.5 Update OpenSpec architecture docs:
|
||||
- `openspec/changes/add-capacity-matching-mcp-server/architecture.md`
|
||||
- `openspec/changes/add-capacity-matching-mcp-server/design.md`
|
||||
- Update diagrams to show Azure OpenAI dependency
|
||||
- Update component descriptions
|
||||
- [x] 9.6 Create `docs/azure_openai_setup.md` (detailed setup guide):
|
||||
- Prerequisites (Azure subscription, OpenAI resource)
|
||||
- API key generation steps
|
||||
- Cost monitoring in Azure portal
|
||||
- Troubleshooting deployment/model availability
|
||||
|
||||
### Phase 10: Testing & Validation (3-4 hours)
|
||||
|
||||
- [x] 10.1 Run fast unit tests in CI mode: `pytest -m "not integration" -q`
|
||||
- [x] 10.2 Run integration tests with real Azure API: `pytest -m integration`
|
||||
- [x] 10.3 Test embedding cache behavior:
|
||||
- Start server, run search → check cache DB created
|
||||
- Restart server, run same search → verify cache hit
|
||||
- Wait past TTL, run cleanup → verify expired entries removed
|
||||
- [x] 10.4 Test both similarity strategies:
|
||||
- Set `matching.similarity.strategy = "per_skill"` → run search
|
||||
- Set `matching.similarity.strategy = "aggregate"` → run search
|
||||
- Compare results and validate scoring differences
|
||||
- [x] 10.5 Test cost tracking:
|
||||
- Run several searches
|
||||
- Check stderr for cost logs
|
||||
- Validate cost estimates are reasonable
|
||||
- [x] 10.6 Test error handling:
|
||||
- Comment out Azure API keys → verify graceful startup error
|
||||
- Mock API failure → verify `AzureAPIError` propagates correctly
|
||||
- Mock rate limit error → verify retry logic works
|
||||
- [x] 10.7 Performance benchmark:
|
||||
- Cold cache: measure time for first search (100 candidates)
|
||||
- Warm cache: measure time for repeated search
|
||||
- Document speedup ratio
|
||||
- [x] 10.8 Manual validation in Cherry Studio:
|
||||
- Browse tasks
|
||||
- Extract requirements from complex task description
|
||||
- Run matching workflow
|
||||
- Validate similarity scores look reasonable
|
||||
- Test guided capture with Azure LLM
|
||||
- Test update_requirements
|
||||
|
||||
### Phase 11: Cleanup & Final Review (2-3 hours)
|
||||
|
||||
- [x] 11.1 Final sweep: ensure there are no remaining FastMCP sampling references
|
||||
- [x] 11.2 Final sweep: ensure no heuristic fallback paths remain
|
||||
- [x] 11.3 Tighten type hints, docstrings, and error semantics
|
||||
- [x] 11.4 Run `ruff check` (via `uv`) and fix all issues
|
||||
- [x] 11.5 Run `mypy` (via `uv`) and fix all issues
|
||||
- [x] 11.6 Final code review: validate all modules, remove stray prints/debug logs
|
||||
- [x] 11.7 Git commit review: ensure no secrets (config/.env/cache DB) are tracked
|
||||
- [x] 11.7a Cleanup: remove obsolete `scripts/trino_smoke_check.py`
|
||||
- [x] 11.7b Dependency hygiene: keep `trino` as a required runtime dependency
|
||||
- [ ] 11.8 Merge to main and tag release
|
||||
|
||||
## Risk Mitigation
|
||||
|
||||
**Risk**: Azure API downtime → System unusable
|
||||
**Mitigation**: Document SLA expectations; consider adding a simple health check tool/endpoint and a clear user-facing error message (`AzureAPIError`).
|
||||
|
||||
**Risk**: Unexpected high costs
|
||||
**Mitigation**: Enable embedding cache (default); monitor costs weekly; set Azure budget alerts.
|
||||
|
||||
**Risk**: Embedding cache grows too large
|
||||
**Mitigation**: Run periodic cleanup; set reasonable TTL (default 30 days); ensure cache DB files stay ignored.
|
||||
|
||||
**Risk**: Secrets accidentally committed (`config.toml`, `.env`, caches)
|
||||
**Mitigation**: Keep `config.toml` and `.env` ignored; review `git status` before commit; run a secret scan locally (e.g. `git grep` for key patterns) before merging.
|
||||
|
||||
**Risk**: Migration breaks existing deployments
|
||||
**Mitigation**: Provide clear migration guide; test migration path manually; run integration tests against Azure in at least one real environment.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [x] Unit tests pass (non-integration)
|
||||
- [ ] Integration tests pass (`pytest -m integration`) in an environment with valid Azure credentials
|
||||
- [ ] Server starts successfully with Azure credentials
|
||||
- [ ] Matching results show improved semantic understanding vs. heuristics
|
||||
- [ ] Embedding cache demonstrates significant performance improvement on repeated searches
|
||||
- [ ] Cost tracking shows accurate estimates
|
||||
- [ ] Documentation complete and clear
|
||||
- [ ] Repo clean: no secrets or local config files tracked (verify before merge)
|
||||
|
||||
---
|
||||
|
||||
**Total estimated effort**: 28-35 hours
|
||||
**Priority**: Medium-High (significant quality improvement, but breaking change)
|
||||
**Approval status**: ✅ Approved
|
||||
@@ -0,0 +1,55 @@
|
||||
# Project Context
|
||||
|
||||
## Purpose
|
||||
This project is an MCP (Model Context Protocol) server for "Teamlandkarte" (Team Map). The server provides AI assistants with tools to browse tasks and match DB Systel employees with free capacity to task requirements via a confirmation-gated, table-first workflow.
|
||||
|
||||
## Tech Stack
|
||||
- **Language**: Python 3.13
|
||||
- **Package Manager**: uv (based on pyproject.toml)
|
||||
- **Configuration**: TOML format (pyproject.toml, config.toml)
|
||||
- **Architecture**: MCP Server (Model Context Protocol) exposing table-first tools and search sessions (`search_id` / `filter_id`)
|
||||
- **Version Control**: Git (hosted on GitLab - git.tech.rz.db.de)
|
||||
|
||||
## Project Conventions
|
||||
|
||||
### Code Style
|
||||
- Python 3.13+ syntax and features
|
||||
- Follow PEP 8 style guidelines
|
||||
- Use type hints where appropriate
|
||||
- Entry point: `main.py` with standard `if __name__ == "__main__"` pattern
|
||||
- Configuration files use TOML format
|
||||
|
||||
### Architecture Patterns
|
||||
- MCP Server architecture for AI assistant integration
|
||||
- OpenSpec workflow for managing change proposals and specifications
|
||||
- Modular structure with dedicated `openspec/` directory for specs and planning
|
||||
- Entry point in `main.py` for simple execution
|
||||
|
||||
### Testing Strategy
|
||||
- Test suite uses **pytest** (see `tests/`)
|
||||
- Focus areas:
|
||||
- confirmation gating
|
||||
- search id robustness and error handling
|
||||
- filtering and pagination workflows
|
||||
- read-only DB guard
|
||||
|
||||
### Git Workflow
|
||||
- Main branch: `main`
|
||||
- Hosted on internal GitLab instance (git.tech.rz.db.de)
|
||||
- Follow GitLab merge request workflow
|
||||
- Use OpenSpec for planning significant changes before implementation
|
||||
|
||||
## Domain Context
|
||||
- **Teamlandkarte** (Team Map): A team mapping or visualization tool
|
||||
- **MCP (Model Context Protocol)**: A standardized protocol for AI assistants to interact with external tools and data sources
|
||||
- The server acts as a bridge between AI assistants and team mapping functionality
|
||||
|
||||
## Important Constraints
|
||||
- Python version must be >= 3.13
|
||||
- Internal project hosted on Deutsche Bahn GitLab infrastructure
|
||||
- Must comply with MCP protocol specifications
|
||||
|
||||
## External Dependencies
|
||||
- MCP SDK/libraries (to be added as project develops)
|
||||
- Database connectivity (config.toml suggests DB integration)
|
||||
- Other dependencies to be defined in pyproject.toml as needed
|
||||
Reference in New Issue
Block a user