863 lines
33 KiB
Markdown
863 lines
33 KiB
Markdown
# Design Document: Knowledge Management System
|
||
|
||
## Overview
|
||
|
||
Das Knowledge Management System erweitert den bestehenden Monorepo-CLI um ein vollständiges persönliches Wissensmanagement. Es baut auf dem existierenden Knowledge Store (ETL-Pipeline, YAML-Index, Scope-basierte Suche) und der Wissensdatenbank (Confluence, Web, PDF-Ingestion) auf und vereint sie zu einer integrierten Lösung.
|
||
|
||
**Kernarchitektur:**
|
||
- Kontextlokale `knowledge/`-Ordner in bahn, dhive, privat (Google OKF-Struktur)
|
||
- Gemeinsame Ingestion-Pipeline in `shared/tools/monorepo-cli/src/monorepo/knowledge/ingestion/`
|
||
- Pluggable Source-Strategien (Markdown, PDF, OCR, Confluence, Jira, E-Mail, Chat, Links)
|
||
- LLM-basierter Enrichment Agent (Kiro als primärer Provider, LiteLLM als spätere Alternative)
|
||
- OrgMyLife-Integration für Action-Item-Extraktion
|
||
- CLI-Schnittstelle als `monorepo knowledge`-Subcommand
|
||
|
||
**Design-Entscheidungen:**
|
||
1. **Erweiterung statt Neubau:** Die bestehenden Module `knowledge/artifact.py`, `knowledge/etl.py`, `knowledge/index.py`, `knowledge/store.py` werden wiederverwendet und erweitert.
|
||
2. **Source-Strategie-Pattern:** Neue Quellen implementieren ein gemeinsames `SourceStrategy`-Interface (analog zu `MarkdownSource`).
|
||
3. **Separation of Concerns:** Ingestion (Extract/Transform/Load), Enrichment (LLM), Routing (Kontext-Zuordnung) und CLI sind separate Module.
|
||
4. **Wissensdatenbank als Read-Only-Upstream:** Die bestehende ETL-Pipeline in `bahn/wissensdatenbank/` wird referenziert, nicht modifiziert.
|
||
5. **Kiro-first für LLM und OCR:** Der AI-Orchestrator (Kiro) dient als primärer Provider für Enrichment und OCR – keine externe Abhängigkeit (LiteLLM, Tesseract) nötig für den Start. Alternativen können später für CI/Batch-Szenarien ergänzt werden.
|
||
|
||
## Architecture
|
||
|
||
### High-Level System Architecture
|
||
|
||
```mermaid
|
||
graph TB
|
||
subgraph "CLI Layer"
|
||
CLI[monorepo knowledge CLI]
|
||
end
|
||
|
||
subgraph "Ingestion Pipeline (shared/tools/monorepo-cli/src/monorepo/knowledge/ingestion/)"
|
||
Router[Context Router]
|
||
Pipeline[Ingestion Pipeline]
|
||
Enrichment[Enrichment Agent<br/>Kiro / LiteLLM]
|
||
end
|
||
|
||
subgraph "Source Strategies"
|
||
MD[Markdown Source]
|
||
PDF[PDF Source]
|
||
OCR[OCR Engine]
|
||
CONF[Confluence Source]
|
||
JIRA[Jira Source]
|
||
EMAIL[Email Source]
|
||
CHAT[Chat Source]
|
||
LINK[Link Source]
|
||
WDB[Wissensdatenbank Source]
|
||
end
|
||
|
||
subgraph "Context Knowledge Folders"
|
||
BK[bahn/knowledge/]
|
||
DK[dhive/knowledge/]
|
||
PK[privat/knowledge/]
|
||
end
|
||
|
||
subgraph "Integrations"
|
||
OML[OrgMyLife API]
|
||
GIT[Git Auto-Commit]
|
||
FED[Federation Sync]
|
||
end
|
||
|
||
subgraph "Search & Retrieval"
|
||
IDX[YAML Index<br/>per Context]
|
||
SEARCH[Cross-Context Search]
|
||
INJECT[Agent Context Injection]
|
||
end
|
||
|
||
CLI --> Router
|
||
CLI --> SEARCH
|
||
Router --> Pipeline
|
||
Pipeline --> Enrichment
|
||
Pipeline --> MD & PDF & OCR & CONF & JIRA & EMAIL & CHAT & LINK & WDB
|
||
|
||
Pipeline --> BK & DK & PK
|
||
Pipeline --> IDX
|
||
Enrichment --> OML
|
||
Pipeline --> GIT
|
||
BK & DK & PK --> FED
|
||
IDX --> SEARCH
|
||
SEARCH --> INJECT
|
||
```
|
||
|
||
### Data Flow
|
||
|
||
```mermaid
|
||
sequenceDiagram
|
||
participant U as User/CLI
|
||
participant R as Context Router
|
||
participant P as Ingestion Pipeline
|
||
participant S as Source Strategy
|
||
participant E as Enrichment Agent
|
||
participant K as Knowledge Folder
|
||
participant I as YAML Index
|
||
participant G as Git
|
||
|
||
U->>R: knowledge ingest --context bahn
|
||
R->>P: load sources.yaml, start pipeline
|
||
P->>S: extract(source_config)
|
||
S-->>P: raw artifacts
|
||
P->>E: enrich(raw_artifact)
|
||
E-->>P: enriched artifact (tags, category, entities, action-items)
|
||
P->>K: write artifact to {context}/knowledge/{category}/
|
||
P->>I: update _index.yaml
|
||
P->>G: git commit "knowledge(bahn): ingest N artifacts from source"
|
||
```
|
||
|
||
### Module Layout
|
||
|
||
```
|
||
shared/tools/monorepo-cli/src/monorepo/knowledge/
|
||
├── __init__.py # Existing exports + new ones
|
||
├── artifact.py # Existing: ArtifactMetadata, parse/write
|
||
├── etl.py # Existing: ETLPipeline (enhanced)
|
||
├── index.py # Existing: YAMLIndex, IndexEntry
|
||
├── store.py # Existing: KnowledgeStore, search
|
||
├── migration.py # Existing: migration utilities
|
||
├── ingestion/ # NEW: Unified ingestion pipeline
|
||
│ ├── __init__.py
|
||
│ ├── pipeline.py # IngestionPipeline orchestrator
|
||
│ ├── router.py # ContextRouter
|
||
│ ├── enrichment.py # EnrichmentAgent (Kiro-first, LiteLLM als Fallback)
|
||
│ ├── capture.py # QuickCapture logic
|
||
│ ├── config.py # SourceConfig, sources.yaml parsing
|
||
│ └── git_integration.py # Auto-commit logic
|
||
├── sources/ # Source strategies
|
||
│ ├── __init__.py
|
||
│ ├── base.py # NEW: SourceStrategy ABC
|
||
│ ├── markdown.py # Existing (enhanced)
|
||
│ ├── pdf.py # NEW: PDF + OCR (Kiro-Vision als primärer OCR-Provider)
|
||
│ ├── confluence.py # NEW: Confluence REST API
|
||
│ ├── jira.py # NEW: Jira JQL
|
||
│ ├── email.py # NEW: .eml/.msg parsing
|
||
│ ├── chat.py # NEW: Chat export parsing
|
||
│ ├── link.py # NEW: Link registry
|
||
│ └── wissensdatenbank.py # NEW: bahn/wissensdatenbank/ adapter
|
||
├── integrations/ # External integrations
|
||
│ ├── __init__.py
|
||
│ ├── orgmylife.py # OrgMyLife task creation
|
||
│ ├── kiro_client.py # Kiro AI-Agent als LLM-Provider (primär)
|
||
│ └── litellm_client.py # LiteLLM wrapper (optional, spätere Alternative)
|
||
└── cli.py # NEW: knowledge subcommand group
|
||
```
|
||
|
||
## Components and Interfaces
|
||
|
||
### 1. SourceStrategy (Abstract Base Class)
|
||
|
||
```python
|
||
from abc import ABC, abstractmethod
|
||
from dataclasses import dataclass, field
|
||
from pathlib import Path
|
||
from typing import Any
|
||
|
||
from monorepo.knowledge.artifact import Artifact
|
||
|
||
|
||
@dataclass
|
||
class SourceConfig:
|
||
"""Configuration for a single source in sources.yaml."""
|
||
type: str # confluence, jira, email, file, link, wissensdatenbank
|
||
name: str # human-readable identifier
|
||
params: dict[str, Any] # type-specific connection/filter params
|
||
target_folder: str = "" # override target subfolder
|
||
sync_frequency: str = "manual" # on-push, hourly, daily, manual
|
||
enabled: bool = True
|
||
|
||
|
||
@dataclass
|
||
class ExtractionResult:
|
||
"""Result from a source strategy extraction."""
|
||
artifacts: list[Artifact] = field(default_factory=list)
|
||
errors: list[SourceError] = field(default_factory=list)
|
||
skipped: int = 0
|
||
|
||
|
||
class SourceStrategy(ABC):
|
||
"""Abstract base class for all source strategies."""
|
||
|
||
@abstractmethod
|
||
def extract(self, config: SourceConfig, context: str) -> ExtractionResult:
|
||
"""Extract raw artifacts from the configured source."""
|
||
...
|
||
|
||
@abstractmethod
|
||
def supports_incremental(self) -> bool:
|
||
"""Whether this strategy supports incremental processing."""
|
||
...
|
||
```
|
||
|
||
### 2. ContextRouter
|
||
|
||
```python
|
||
@dataclass
|
||
class RoutingDecision:
|
||
"""Result of context routing."""
|
||
context: str # bahn, dhive, privat
|
||
target_folder: str # knowledge subfolder (meetings, decisions, etc.)
|
||
artifact_path: Path # full path for the artifact
|
||
|
||
|
||
class ContextRouter:
|
||
"""Routes incoming data to the correct context knowledge folder."""
|
||
|
||
def __init__(self, monorepo_root: Path) -> None: ...
|
||
|
||
def determine_context(self, source_config: SourceConfig | None = None,
|
||
cwd: Path | None = None) -> str:
|
||
"""Determine target context from source config or working directory."""
|
||
...
|
||
|
||
def resolve_target_path(self, context: str, category: str,
|
||
filename: str) -> Path:
|
||
"""Compute full path: {root}/{context}/knowledge/{category}/{filename}"""
|
||
...
|
||
```
|
||
|
||
### 3. IngestionPipeline
|
||
|
||
```python
|
||
@dataclass
|
||
class IngestionResult:
|
||
"""Result of a full ingestion run."""
|
||
context: str
|
||
processed: int = 0
|
||
updated: int = 0
|
||
skipped: int = 0
|
||
errors: list[IngestError] = field(default_factory=list)
|
||
tasks_created: int = 0
|
||
commit_sha: str = ""
|
||
|
||
|
||
class IngestionPipeline:
|
||
"""Orchestrates the full ingestion workflow."""
|
||
|
||
def __init__(self, monorepo_root: Path, context: str,
|
||
dry_run: bool = False, no_commit: bool = False) -> None: ...
|
||
|
||
def run(self) -> IngestionResult:
|
||
"""Execute full pipeline: load config → extract → enrich → store → index → commit."""
|
||
...
|
||
|
||
def process_inbox(self) -> IngestionResult:
|
||
"""Process all items in {context}/knowledge/inbox/."""
|
||
...
|
||
```
|
||
|
||
### 4. EnrichmentAgent
|
||
|
||
```python
|
||
@dataclass
|
||
class EnrichmentResult:
|
||
"""Output from LLM-based enrichment."""
|
||
title: str
|
||
category: str # meeting, decision, project, reference, link, inbox
|
||
tags: list[str]
|
||
people: list[str] # detected person names
|
||
projects: list[str] # detected project references
|
||
action_items: list[ActionItem]
|
||
confidence: float # overall confidence score
|
||
|
||
|
||
@dataclass
|
||
class ActionItem:
|
||
"""An extracted action item from content."""
|
||
description: str
|
||
assignee: str | None = None
|
||
deadline: str | None = None # ISO date string
|
||
hash: str = "" # for dedup: sha256 of description
|
||
|
||
|
||
class EnrichmentAgent:
|
||
"""LLM-powered content analysis and enrichment.
|
||
|
||
Provider-Strategie (Priorität):
|
||
1. Kiro AI-Agent (primär) – nutzt den laufenden Orchestrator-Agenten direkt,
|
||
keine externe Abhängigkeit, kein API-Key nötig
|
||
2. LiteLLM (optional, spätere Alternative) – für Batch-Verarbeitung oder
|
||
wenn Kiro nicht verfügbar ist (z.B. CI-Pipeline)
|
||
|
||
OCR-Strategie (Priorität):
|
||
1. Kiro Vision – Bilder werden direkt an den Agenten gesendet,
|
||
der den Text extrahiert (keine Tesseract-Installation nötig)
|
||
2. Tesseract (optional, spätere Alternative) – für Offline-/CI-Betrieb
|
||
"""
|
||
|
||
def __init__(self, provider: str = "kiro", model: str = "") -> None:
|
||
"""Initialize with LLM config. Defaults to 'kiro' provider.
|
||
|
||
Providers: 'kiro' (default), 'litellm' (future alternative)
|
||
Falls provider='litellm': config from env vars LLM_PROVIDER, LLM_MODEL.
|
||
"""
|
||
...
|
||
|
||
def enrich(self, content: str, source_type: str = "") -> EnrichmentResult:
|
||
"""Analyze content and generate metadata."""
|
||
...
|
||
|
||
def classify_relevance(self, content: str) -> float:
|
||
"""Score content relevance (0.0 = irrelevant, 1.0 = highly relevant)."""
|
||
...
|
||
```
|
||
|
||
### 5. QuickCapture
|
||
|
||
```python
|
||
class QuickCapture:
|
||
"""Fast knowledge capture to inbox."""
|
||
|
||
def __init__(self, monorepo_root: Path) -> None: ...
|
||
|
||
def capture(self, input_text: str, context: str | None = None,
|
||
tags: list[str] | None = None) -> Path:
|
||
"""Capture text/URL/filepath to inbox. Returns path of created artifact."""
|
||
...
|
||
|
||
def _detect_input_type(self, input_text: str) -> str:
|
||
"""Detect if input is URL, file path, or plain text."""
|
||
...
|
||
|
||
def _generate_filename(self, title: str) -> str:
|
||
"""Generate YYYY-MM-DD-HH-MM-title-slug.md filename."""
|
||
...
|
||
```
|
||
|
||
### 6. LinkRegistry
|
||
|
||
```python
|
||
class LinkRegistry:
|
||
"""Manages web links as knowledge artifacts."""
|
||
|
||
def __init__(self, context_path: Path) -> None: ...
|
||
|
||
def save_link(self, url: str, title: str | None = None,
|
||
tags: list[str] | None = None) -> Path:
|
||
"""Save or update a link artifact. Deduplicates by URL."""
|
||
...
|
||
|
||
def search_links(self, query: str) -> list[Artifact]:
|
||
"""Search links by URL, title, or tags."""
|
||
...
|
||
|
||
def _fetch_metadata(self, url: str) -> tuple[str, str]:
|
||
"""Fetch page title and description from URL."""
|
||
...
|
||
|
||
def _find_existing(self, url: str) -> Path | None:
|
||
"""Check for duplicate URL in context."""
|
||
...
|
||
```
|
||
|
||
### 7. OrgMyLife Integration
|
||
|
||
```python
|
||
@dataclass
|
||
class TaskMapping:
|
||
"""Maps artifact action items to OrgMyLife task IDs."""
|
||
artifact_id: str
|
||
action_item_hash: str
|
||
task_id: int
|
||
created: str # ISO date
|
||
|
||
|
||
class OrgMyLifeIntegration:
|
||
"""Creates tasks in OrgMyLife from detected action items."""
|
||
|
||
def __init__(self, context_path: Path, enabled: bool = True) -> None: ...
|
||
|
||
def create_tasks(self, artifact_id: str,
|
||
action_items: list[ActionItem],
|
||
source_path: str) -> list[TaskMapping]:
|
||
"""Create OrgMyLife tasks for new action items (dedup via mapping file)."""
|
||
...
|
||
|
||
def _load_mapping(self) -> dict[str, TaskMapping]: ...
|
||
def _save_mapping(self, mappings: dict[str, TaskMapping]) -> None: ...
|
||
```
|
||
|
||
### 8. CLI Interface
|
||
|
||
```python
|
||
# Registered as subcommand group under ctx-guard / monorepo CLI
|
||
# Commands:
|
||
# knowledge capture <text|url|path> [--context CTX] [--tags TAG,TAG]
|
||
# knowledge ingest [--context CTX] [--dry-run] [--verbose] [--no-commit]
|
||
# knowledge search <query> [--context CTX] [--limit N]
|
||
# knowledge status [--context CTX]
|
||
# knowledge link <url> [--title TITLE] [--tags TAG,TAG] [--context CTX]
|
||
```
|
||
|
||
## Data Models
|
||
|
||
### Extended ArtifactMetadata (backward-compatible extension)
|
||
|
||
The existing `ArtifactMetadata` dataclass is extended with optional fields:
|
||
|
||
```python
|
||
@dataclass
|
||
class ArtifactMetadata:
|
||
"""Extended metadata supporting knowledge management features."""
|
||
# Existing fields (unchanged)
|
||
type: str
|
||
title: str
|
||
tags: list[str] = field(default_factory=list)
|
||
source_context: str = ""
|
||
created: date | None = None
|
||
updated: date | None = None
|
||
shareable: bool = False
|
||
links: list[ArtifactLink] = field(default_factory=list)
|
||
content_hash: str = ""
|
||
|
||
# NEW: Knowledge Management extensions
|
||
source: dict[str, str] = field(default_factory=dict)
|
||
# e.g. {"type": "confluence", "url": "...", "space": "...", "file": "..."}
|
||
|
||
category: str = "" # meeting, decision, project, reference, link, inbox
|
||
people: list[str] = field(default_factory=list) # [[people/name]] refs
|
||
projects: list[str] = field(default_factory=list) # [[projects/slug]] refs
|
||
enrichment_pending: bool = False
|
||
ocr_failed: bool = False
|
||
fetch_failed: bool = False
|
||
```
|
||
|
||
### sources.yaml Schema
|
||
|
||
```yaml
|
||
# {context}/knowledge/sources.yaml
|
||
version: "1.0"
|
||
sources:
|
||
- name: "Team Confluence"
|
||
type: confluence
|
||
enabled: true
|
||
sync_frequency: daily
|
||
params:
|
||
base_url: "https://confluence.bahn.de"
|
||
api_key: "${CONFLUENCE_API_TOKEN}"
|
||
spaces: ["ACV2", "EINFACHBAHN"]
|
||
tree_roots: ["12345678"]
|
||
target_folder: references
|
||
|
||
- name: "Sprint Issues"
|
||
type: jira
|
||
enabled: true
|
||
sync_frequency: daily
|
||
params:
|
||
base_url: "https://jira.bahn.de"
|
||
api_key: "${JIRA_API_TOKEN}"
|
||
jql: "project = ACV2 AND updatedDate > -7d"
|
||
target_folder: projects
|
||
|
||
- name: "Wissensdatenbank"
|
||
type: wissensdatenbank
|
||
enabled: true
|
||
sync_frequency: manual
|
||
params:
|
||
output_path: "../../wissensdatenbank/output/"
|
||
target_folder: references
|
||
|
||
settings:
|
||
create_tasks: true
|
||
enrichment:
|
||
provider: "kiro" # Primär: Kiro AI-Agent (kein API-Key nötig)
|
||
# provider: "${LLM_PROVIDER}" # Alternative: LiteLLM (für CI/Batch)
|
||
# model: "${LLM_MODEL}"
|
||
confidence_threshold: 0.7
|
||
ocr:
|
||
provider: "kiro" # Primär: Kiro Vision (kein Tesseract nötig)
|
||
# provider: "tesseract" # Alternative: lokales Tesseract
|
||
# languages: "deu+eng"
|
||
git:
|
||
auto_commit: true
|
||
```
|
||
|
||
### Knowledge Folder Structure (Google OKF)
|
||
|
||
```
|
||
{context}/knowledge/
|
||
├── _index.yaml # YAML Index (Progressive Disclosure Layer 1)
|
||
├── sources.yaml # Source configuration
|
||
├── .task-mapping.yaml # OrgMyLife action-item → task ID mapping
|
||
├── inbox/ # Unprocessed captures and imports
|
||
│ └── index.md # Auto-generated directory listing
|
||
├── meetings/
|
||
│ └── index.md
|
||
├── decisions/
|
||
│ └── index.md
|
||
├── projects/
|
||
│ └── index.md
|
||
├── people/
|
||
│ └── index.md
|
||
├── references/
|
||
│ └── index.md
|
||
└── links/
|
||
└── index.md
|
||
```
|
||
|
||
### Artifact Frontmatter Example (Google OKF)
|
||
|
||
```yaml
|
||
---
|
||
type: meeting
|
||
title: "Sprint Planning ACV2 W24"
|
||
tags: [acv2, sprint, planning]
|
||
source_context: bahn
|
||
created: 2025-07-01
|
||
updated: 2025-07-01
|
||
shareable: true
|
||
category: meeting
|
||
source:
|
||
type: confluence
|
||
url: "https://confluence.bahn.de/pages/viewpage.action?pageId=123456"
|
||
space: ACV2
|
||
people:
|
||
- "[[people/max-mustermann]]"
|
||
- "[[people/anna-schmidt]]"
|
||
projects:
|
||
- "[[projects/acv2-migration]]"
|
||
links:
|
||
- target: "decisions/api-redesign"
|
||
relation: "discussed_in"
|
||
- target: "../references/inb-2026-kapitel-3"
|
||
relation: "references"
|
||
content_hash: "sha256:abc123..."
|
||
---
|
||
|
||
# Sprint Planning ACV2 W24
|
||
|
||
## Agenda
|
||
...
|
||
|
||
## Action Items
|
||
|
||
- [ ] @max-mustermann: API-Konzept bis Freitag finalisieren (Deadline: 2025-07-04)
|
||
- [ ] @anna-schmidt: Testdaten bereitstellen
|
||
```
|
||
|
||
### _index.yaml Entry Structure
|
||
|
||
```yaml
|
||
version: "1.0"
|
||
last_updated: "2025-07-01T14:30:00Z"
|
||
artifacts:
|
||
- id: "bahn/meeting/sprint-planning-acv2-w24"
|
||
title: "Sprint Planning ACV2 W24"
|
||
type: meeting
|
||
tags: [acv2, sprint, planning]
|
||
scope: bahn
|
||
summary: "Sprint Planning für ACV2 Migration, Woche 24. API-Redesign besprochen."
|
||
path: "bahn/knowledge/meetings/sprint-planning-acv2-w24.md"
|
||
content_hash: "sha256:abc123..."
|
||
links: ["bahn/decision/api-redesign"]
|
||
source: wissensdatenbank # or: personal, confluence, jira
|
||
```
|
||
|
||
### Git Commit Message Format
|
||
|
||
```
|
||
knowledge({context}): {action} {count} artifacts from {source}
|
||
knowledge(bahn): ingest 3 artifacts from confluence
|
||
knowledge(privat): capture "Quick idea about X"
|
||
knowledge(dhive): ingest 1 artifacts from jira
|
||
```
|
||
|
||
## Correctness Properties
|
||
|
||
*A property is a characteristic or behavior that should hold true across all valid executions of a system — essentially, a formal statement about what the system should do. Properties serve as the bridge between human-readable specifications and machine-verifiable correctness guarantees.*
|
||
|
||
### Property 1: Artifact Serialization Round-Trip
|
||
|
||
*For any* valid `ArtifactMetadata` object with content, writing it to a Markdown file with YAML frontmatter and then reading it back SHALL produce an identical `ArtifactMetadata` object (fields, types, and values match).
|
||
|
||
**Validates: Requirements 16.6, 1.5**
|
||
|
||
### Property 2: Context Derivation from Path
|
||
|
||
*For any* file path within the monorepo that resides under a context folder (`bahn/`, `dhive/`, `privat/`), the context derivation function SHALL return the correct context string matching the top-level folder name. This applies both to artifact paths and working directory resolution.
|
||
|
||
**Validates: Requirements 1.4, 3.2**
|
||
|
||
### Property 3: Idempotent Ingestion via Content Hash
|
||
|
||
*For any* artifact that has already been ingested and whose content has not changed, running the ingestion pipeline again SHALL skip re-processing (updated count = 0) and the YAML index entry SHALL remain unchanged.
|
||
|
||
**Validates: Requirements 2.4**
|
||
|
||
### Property 4: Error Resilience — Partial Source Failure
|
||
|
||
*For any* set of N configured sources where 1 ≤ K < N sources fail during extraction, the ingestion pipeline SHALL still successfully process all non-failing sources and produce artifacts for them. The IngestResult.errors list SHALL contain exactly the K failed sources.
|
||
|
||
**Validates: Requirements 2.5**
|
||
|
||
### Property 5: Index Completeness After Ingestion
|
||
|
||
*For any* set of artifacts successfully processed by the ingestion pipeline, the YAML index SHALL contain an entry for each processed artifact, with matching id, title, type, and content_hash.
|
||
|
||
**Validates: Requirements 2.6**
|
||
|
||
### Property 6: Context Routing Determinism
|
||
|
||
*For any* valid `SourceConfig` with an assigned context, the `ContextRouter` SHALL always route to that context. For any working directory within a context folder, the router SHALL derive the same context as the path-based derivation.
|
||
|
||
**Validates: Requirements 2.3**
|
||
|
||
### Property 7: Filename Pattern from Title
|
||
|
||
*For any* non-empty title string, the Quick Capture filename generator SHALL produce a filename matching the pattern `\d{4}-\d{2}-\d{2}-\d{2}-\d{2}-[a-z0-9-]+\.md` (date-time prefix + slugified title + .md extension).
|
||
|
||
**Validates: Requirements 3.3**
|
||
|
||
### Property 8: Category to Folder Mapping
|
||
|
||
*For any* valid category string from the set {meeting, decision, project, reference, link, inbox}, the folder mapping function SHALL return the corresponding plural folder name (meetings/, decisions/, projects/, references/, links/, inbox/). For any unknown category, it SHALL default to inbox/.
|
||
|
||
**Validates: Requirements 3.5, 9.6**
|
||
|
||
### Property 9: File Extension Format Detection
|
||
|
||
*For any* file path with a supported extension (`.md`, `.txt`, `.pdf`, `.png`, `.jpg`, `.jpeg`, `.docx`), the format detection function SHALL return the correct format type. For unsupported extensions, it SHALL raise an appropriate error or return a fallback type.
|
||
|
||
**Validates: Requirements 4.7**
|
||
|
||
### Property 10: Image Source File Reference Preservation
|
||
|
||
*For any* artifact created from an image source (PNG, JPG, JPEG), the resulting artifact's frontmatter SHALL contain a `source.file` field referencing the original image file path.
|
||
|
||
**Validates: Requirements 4.5**
|
||
|
||
### Property 11: URL Detection in Quick Capture
|
||
|
||
*For any* string that is a valid HTTP or HTTPS URL, the Quick Capture input type detector SHALL classify it as type "link". For any string that is not a valid HTTP(S) URL, it SHALL NOT classify it as "link".
|
||
|
||
**Validates: Requirements 8.3**
|
||
|
||
### Property 12: Link Artifact Structure
|
||
|
||
*For any* URL saved via the Link Registry, the resulting artifact SHALL have `type: link`, be stored in the `links/` subfolder, and its frontmatter SHALL contain the fields: url, title (non-empty), tags (list), and created (valid date).
|
||
|
||
**Validates: Requirements 8.1, 8.2**
|
||
|
||
### Property 13: Link Deduplication with Tag Merging
|
||
|
||
*For any* URL that is saved twice to the same context Link Registry (possibly with different tags), only one artifact file SHALL exist for that URL, and its tags SHALL be the union of all tags provided across both save operations.
|
||
|
||
**Validates: Requirements 8.7**
|
||
|
||
### Property 14: Link Search Completeness
|
||
|
||
*For any* stored link artifact and any substring query that matches its URL, title, or any of its tags, the Link Registry search SHALL return that artifact in its results.
|
||
|
||
**Validates: Requirements 8.6**
|
||
|
||
### Property 15: Wiki-Link Format for Entities
|
||
|
||
*For any* person name detected by the Enrichment Agent, the generated wiki-link SHALL match the format `[[people/{slugified-name}]]`. For any project name, it SHALL match `[[projects/{slugified-name}]]`. Slugification SHALL produce lowercase, hyphen-separated strings.
|
||
|
||
**Validates: Requirements 9.2**
|
||
|
||
### Property 16: Confidence Threshold Filtering
|
||
|
||
*For any* set of detected entities with varying confidence scores, only entities with confidence ≥ 0.7 SHALL appear in the final enrichment output. Entities below 0.7 SHALL be excluded.
|
||
|
||
**Validates: Requirements 9.5**
|
||
|
||
### Property 17: Action Items Section Generation
|
||
|
||
*For any* non-empty list of `ActionItem` objects, the generated markdown SHALL contain a `## Action Items` section header followed by a markdown list item for each action item's description.
|
||
|
||
**Validates: Requirements 9.4**
|
||
|
||
### Property 18: Task Deduplication via Mapping
|
||
|
||
*For any* action item whose hash already exists in the `.task-mapping.yaml` file, the OrgMyLife integration SHALL NOT create a new task. Only action items with hashes not present in the mapping SHALL trigger task creation.
|
||
|
||
**Validates: Requirements 10.5, 10.6**
|
||
|
||
### Property 19: Scope-Filtered Search Isolation
|
||
|
||
*For any* search query and set of allowed scopes, the search results SHALL contain ONLY artifacts whose scope is in the allowed set. No artifact from an unauthorized scope SHALL appear in results, and its existence SHALL NOT be revealed.
|
||
|
||
**Validates: Requirements 11.5**
|
||
|
||
### Property 20: Context Injection Result Limit
|
||
|
||
*For any* search query that would return more than 10 results, the context injection function SHALL return at most 10 artifact paths, sorted by relevance score in descending order.
|
||
|
||
**Validates: Requirements 11.6**
|
||
|
||
### Property 21: Environment Variable Resolution in Config
|
||
|
||
*For any* string containing `${VAR_NAME}` patterns where the environment variable exists, the resolution function SHALL substitute the pattern with the variable's value. For any `${VAR_NAME}` where the variable is NOT set, the validation SHALL report an error.
|
||
|
||
**Validates: Requirements 12.4, 12.6**
|
||
|
||
### Property 22: Federation Sync Exclusion
|
||
|
||
*For any* set of artifacts where some have `shareable: false` in their frontmatter, the federation sync file list SHALL exclude ALL artifacts with `shareable: false`. Only artifacts with `shareable: true` (or shareable not explicitly set to false) SHALL be included.
|
||
|
||
**Validates: Requirements 14.4**
|
||
|
||
### Property 23: Git Commit Message Format
|
||
|
||
*For any* valid combination of context (bahn|dhive|privat), action (ingest|capture), count (positive integer), and source name, the generated commit message SHALL match the pattern `knowledge({context}): {action} {count} artifacts from {source}`.
|
||
|
||
**Validates: Requirements 16.2**
|
||
|
||
### Property 24: Ingestion Produces Valid Artifacts
|
||
|
||
*For any* raw text content processed by the ingestion pipeline, the resulting artifact SHALL have a non-empty title, a valid category from the allowed set, a non-empty content_hash in sha256 format, and a valid source_context.
|
||
|
||
**Validates: Requirements 2.2**
|
||
|
||
|
||
## Error Handling
|
||
|
||
### Strategy: Fail-Forward with Logging
|
||
|
||
The system uses a "fail-forward" approach: individual failures are logged and skipped, while the overall pipeline continues processing. This matches the existing ETL pipeline's error resilience pattern.
|
||
|
||
### Error Categories
|
||
|
||
| Category | Handling | Recovery |
|
||
|----------|----------|----------|
|
||
| Source unavailable (API timeout, auth failure) | Log error with source identifier, skip source, continue with remaining | Retry on next scheduled run |
|
||
| OCR failure (unreadable image) | Create artifact with empty content, set `ocr_failed: true` | Manual review via `knowledge status` |
|
||
| LLM/Kiro provider unavailable | Store artifact with minimal metadata, set `enrichment_pending: true` | Re-enrich on next run with `--re-enrich` flag |
|
||
| OrgMyLife API failure | Log warning, continue without task creation | Tasks created on next successful run (dedup handles this) |
|
||
| Git commit failure | Log warning, keep created files, skip commit | User commits manually or runs `--no-commit` |
|
||
| Invalid source config (missing fields, bad env vars) | Reject entire config with validation errors BEFORE starting pipeline | User fixes sources.yaml |
|
||
| File write failure (permissions, disk full) | Log error, skip artifact, continue | User resolves filesystem issue |
|
||
| Duplicate URL (Link Registry) | Not an error — merge tags with existing artifact | N/A (expected behavior) |
|
||
|
||
### Error Propagation
|
||
|
||
```python
|
||
@dataclass
|
||
class PipelineError:
|
||
"""Structured error for pipeline operations."""
|
||
source: str # which source/component failed
|
||
error_type: str # "connection", "auth", "parse", "write", "llm"
|
||
message: str # human-readable description
|
||
timestamp: str # ISO 8601
|
||
retry: bool # whether retry is sensible
|
||
artifact_path: str = "" # affected artifact, if applicable
|
||
```
|
||
|
||
### Graceful Degradation Levels
|
||
|
||
1. **Full operation:** All sources accessible, Kiro/LLM available → enriched artifacts with tasks
|
||
2. **No Kiro/LLM:** Sources accessible, Kiro session not active and no LiteLLM configured → artifacts with minimal metadata (`enrichment_pending: true`)
|
||
3. **Partial sources:** Some sources down → process available sources, log failures
|
||
4. **No Git:** Git unavailable → files created but not committed (warning)
|
||
5. **Read-only fallback:** No write access → search and read still work via existing index
|
||
|
||
### Logging
|
||
|
||
- Uses Python `logging` module (consistent with existing codebase)
|
||
- Log level from `--verbose` flag or `KNOWLEDGE_LOG_LEVEL` env var
|
||
- Pipeline run summary always printed to stdout (processed/updated/skipped/errors)
|
||
- Detailed errors to stderr
|
||
|
||
## Testing Strategy
|
||
|
||
### Dual Testing Approach
|
||
|
||
The testing strategy combines property-based tests (PBT) for universal correctness properties with example-based unit tests for specific scenarios and integration tests for external service interactions.
|
||
|
||
### Property-Based Tests (Hypothesis)
|
||
|
||
**Library:** [Hypothesis](https://hypothesis.readthedocs.io/) (already in dev dependencies)
|
||
|
||
**Configuration:**
|
||
- Minimum 100 iterations per property (`@settings(max_examples=100)`)
|
||
- Each property test is tagged with its design document property reference
|
||
|
||
**Tag format:** `Feature: knowledge-management, Property {number}: {property_text}`
|
||
|
||
**Properties to implement:**
|
||
|
||
| Property | Module Under Test | Generator Strategy |
|
||
|----------|-------------------|-------------------|
|
||
| 1: Round-Trip | `artifact.py` | Generate random `ArtifactMetadata` with all field combinations |
|
||
| 2: Context from Path | `router.py` | Generate paths like `{context}/knowledge/{subdir}/{filename}` |
|
||
| 3: Idempotent Ingestion | `pipeline.py` | Generate artifacts, ingest twice, verify skip |
|
||
| 4: Error Resilience | `pipeline.py` | Generate source lists with injected failures |
|
||
| 5: Index Completeness | `pipeline.py` + `index.py` | Generate artifact batches |
|
||
| 6: Context Routing | `router.py` | Generate SourceConfig objects |
|
||
| 7: Filename Pattern | `capture.py` | Generate random Unicode titles |
|
||
| 8: Category Mapping | `router.py` | Generate from category enum + random strings |
|
||
| 9: Extension Detection | `pipeline.py` | Generate file paths with various extensions |
|
||
| 10: Image Source Ref | `sources/pdf.py` | Generate image-sourced artifacts |
|
||
| 11: URL Detection | `capture.py` | Generate valid URLs + non-URL strings |
|
||
| 12: Link Structure | `sources/link.py` | Generate random URLs and metadata |
|
||
| 13: Link Dedup | `sources/link.py` | Generate duplicate URL scenarios |
|
||
| 14: Link Search | `sources/link.py` | Generate links + substring queries |
|
||
| 15: Wiki-Link Format | `enrichment.py` | Generate person/project name strings |
|
||
| 16: Confidence Filter | `enrichment.py` | Generate entity lists with random confidences |
|
||
| 17: Action Items Markdown | `enrichment.py` | Generate ActionItem lists |
|
||
| 18: Task Dedup | `integrations/orgmylife.py` | Generate action items + existing mappings |
|
||
| 19: Scope Isolation | `store.py` | Generate multi-scope data + restricted scope lists |
|
||
| 20: Result Limit | `store.py` | Generate large artifact sets + queries |
|
||
| 21: Env Var Resolution | `config.py` | Generate strings with `${VAR}` patterns + env dicts |
|
||
| 22: Sync Exclusion | `federation` integration | Generate artifacts with mixed shareable flags |
|
||
| 23: Commit Message | `git_integration.py` | Generate context/action/count/source combinations |
|
||
| 24: Valid Artifact Output | `pipeline.py` | Generate raw text content |
|
||
|
||
### Example-Based Unit Tests
|
||
|
||
- CLI argument parsing and dispatch
|
||
- Specific Confluence page parsing scenarios
|
||
- Specific Jira issue formatting
|
||
- Email header extraction
|
||
- OKF structure validation examples
|
||
- sources.yaml parsing with real-world examples
|
||
|
||
### Integration Tests
|
||
|
||
- Confluence API interaction (mocked `httpx` responses)
|
||
- Jira API interaction (mocked responses)
|
||
- OrgMyLife API task creation (mocked)
|
||
- OCR via Kiro Vision (mocked agent responses); optional: Tesseract with sample images
|
||
- PDF text extraction with sample documents
|
||
- Git commit operations (using tmp repos)
|
||
- Full pipeline end-to-end with fixture data
|
||
|
||
### Test Organization
|
||
|
||
```
|
||
shared/tools/monorepo-cli/tests/
|
||
├── knowledge/
|
||
│ ├── test_artifact_roundtrip.py # Property 1
|
||
│ ├── test_context_routing.py # Properties 2, 6, 8
|
||
│ ├── test_ingestion_pipeline.py # Properties 3, 4, 5, 24
|
||
│ ├── test_capture.py # Properties 7, 11
|
||
│ ├── test_format_detection.py # Property 9
|
||
│ ├── test_link_registry.py # Properties 12, 13, 14
|
||
│ ├── test_enrichment.py # Properties 15, 16, 17
|
||
│ ├── test_orgmylife_integration.py # Property 18
|
||
│ ├── test_search.py # Properties 19, 20
|
||
│ ├── test_config.py # Property 21
|
||
│ ├── test_federation_filter.py # Property 22
|
||
│ ├── test_git_integration.py # Property 23
|
||
│ ├── test_image_source.py # Property 10
|
||
│ └── integration/
|
||
│ ├── test_confluence_source.py
|
||
│ ├── test_jira_source.py
|
||
│ ├── test_email_source.py
|
||
│ ├── test_ocr_engine.py
|
||
│ └── test_full_pipeline.py
|
||
```
|
||
|
||
### Running Tests
|
||
|
||
```powershell
|
||
# Quick property tests (fast)
|
||
cd shared\tools\monorepo-cli
|
||
python -m pytest tests/knowledge/ -x --tb=short -q -k "not integration"
|
||
|
||
# Full test suite including integration
|
||
python -m pytest tests/knowledge/ --tb=short
|
||
|
||
# Specific property
|
||
python -m pytest tests/knowledge/test_artifact_roundtrip.py -v
|
||
```
|