Orchestrator/.kiro/specs/notegraph-ingestion/design.md

Design Document: NoteGraph Ingestion & Auto-Reference

Overview

The NoteGraph Ingestion service is a Python package (NoteGraph/ingestion/) that provides a CLI-driven pipeline for importing documents (PDFs, images, markdown, text, DOCX) into the NoteGraph knowledge base. The pipeline extracts text (via OCR or direct extraction), enriches it using an LLM to detect entities (people, projects, dates, action items), generates wiki-links, and outputs structured markdown files with YAML frontmatter into the appropriate NoteGraph folder.

The system is designed around three principles:

1. Model-agnostic — A unified LLM interface (via LiteLLM) supports OpenAI, Anthropic, Google, Mistral, and local Ollama without code changes.
2. CLI-first — All operations are accessible via python -m ingestion.cli with subcommands for bulk import and continuous watching.
3. Markdown-native — Output is always SilverBullet-compatible markdown with YAML frontmatter and wiki-links.

Architecture

graph TD
    subgraph Input
        CLI[CLI Interface]
        WATCH[Drop-Folder Watcher]
    end

    subgraph Pipeline
        DISC[File Discovery]
        EXT[Text Extraction]
        OCR[OCR Engine]
        ENR[Enrichment Agent]
        LINK[Reference Linker]
        RENDER[Markdown Renderer]
    end

    subgraph Output
        MD[Markdown Files]
        STUB[Stub Pages]
        GIT[Git Commit]
        TASK[OrgMyLife Tasks]
    end

    subgraph External
        LLM[LLM Provider]
        ORG[OrgMyLife API]
    end

    CLI --> DISC
    WATCH --> DISC
    DISC --> EXT
    EXT --> OCR
    EXT --> ENR
    ENR --> LLM
    ENR --> LINK
    LINK --> RENDER
    RENDER --> MD
    RENDER --> STUB
    MD --> GIT
    STUB --> GIT
    ENR --> TASK
    TASK --> ORG

Data Flow

Input File → File Discovery → Text Extraction (OCR if needed)
    → Enrichment Agent (LLM call) → Entity Detection
    → Reference Linker (wiki-links) → Markdown Renderer (frontmatter + body)
    → File Placement (folder routing) → Git Commit
    → (optional) OrgMyLife Task Creation

Components and Interfaces

Package Structure

NoteGraph/ingestion/
├── __init__.py
├── __main__.py              # python -m ingestion entry point
├── cli.py                   # Click-based CLI (import, watch subcommands)
├── config.py                # Configuration from .env
├── pipeline.py              # Main orchestration pipeline
├── discovery.py             # File discovery and filtering
├── extraction/
│   ├── __init__.py
│   ├── base.py              # Extractor protocol
│   ├── text.py              # Plain text / markdown extractor
│   ├── pdf.py               # PDF text extraction (PyPDF2/pdfplumber)
│   ├── ocr.py               # OCR via Tesseract (pytesseract)
│   └── docx.py              # DOCX extraction (python-docx)
├── enrichment/
│   ├── __init__.py
│   ├── agent.py             # Enrichment orchestration
│   ├── prompts.py           # Structured prompts for entity detection
│   └── models.py            # Pydantic models for enrichment response
├── linking/
│   ├── __init__.py
│   ├── linker.py            # Reference linker (wiki-link generation)
│   └── stubs.py             # Stub page creation
├── output/
│   ├── __init__.py
│   ├── renderer.py          # Markdown + frontmatter rendering
│   ├── router.py            # Folder routing logic
│   └── naming.py            # Filename generation with collision avoidance
├── integrations/
│   ├── __init__.py
│   ├── llm.py               # LiteLLM wrapper (model-agnostic)
│   ├── orgmylife.py         # OrgMyLife API client
│   └── git.py               # Git commit operations
├── watcher.py               # Drop-folder watcher (watchdog)
└── tests/
    ├── __init__.py
    ├── conftest.py
    ├── test_discovery.py
    ├── test_extraction.py
    ├── test_enrichment.py
    ├── test_linking.py
    ├── test_output.py
    ├── test_pipeline.py
    └── test_properties.py   # Property-based tests

Core Interfaces

# ingestion/extraction/base.py
from typing import Protocol

class TextExtractor(Protocol):
    """Protocol for all text extractors."""
    
    def can_handle(self, file_path: Path) -> bool:
        """Return True if this extractor handles the given file type."""
        ...
    
    def extract(self, file_path: Path) -> ExtractionResult:
        """Extract text content from the file."""
        ...

@dataclass
class ExtractionResult:
    text: str
    source_file: Path
    extraction_method: str  # "direct", "ocr", "mixed"
    metadata: dict[str, Any] = field(default_factory=dict)

# ingestion/enrichment/models.py
from pydantic import BaseModel, Field

class Entity(BaseModel):
    type: Literal["person", "project", "date", "action_item"]
    value: str
    confidence: float = Field(ge=0.0, le=1.0)
    position: int | None = None  # character offset in source text

class ActionItem(BaseModel):
    description: str
    assignee: str | None = None
    deadline: str | None = None  # ISO 8601

class EnrichmentResult(BaseModel):
    title: str
    category: Literal["meeting", "project", "decision", "inbox"]
    tags: list[str]
    entities: list[Entity]
    action_items: list[ActionItem]
    summary: str | None = None

# ingestion/integrations/llm.py
from litellm import completion

class LLMClient:
    """Model-agnostic LLM client using LiteLLM."""
    
    def __init__(self, config: IngestionConfig):
        self.model = config.llm_model_string  # e.g. "openai/gpt-4o"
    
    def complete(self, messages: list[dict], response_format: type[BaseModel]) -> BaseModel:
        """Send a structured completion request."""
        ...

# ingestion/integrations/orgmylife.py
import httpx

class OrgMyLifeClient:
    """HTTP client for OrgMyLife task creation."""
    
    def __init__(self, base_url: str, api_key: str):
        self.base_url = base_url
        self.api_key = api_key
    
    def create_task(self, title: str, source_url: str, 
                    description: str = "", priority: int = 4) -> dict:
        """Create a task via POST /api/tasks."""
        ...

Configuration

# ingestion/config.py
from pydantic_settings import BaseSettings

class IngestionConfig(BaseSettings):
    # LLM Provider
    llm_provider: str = "openai"  # openai, anthropic, google, mistral, ollama
    llm_model: str = "gpt-4o"
    openai_api_key: str = ""
    anthropic_api_key: str = ""
    google_api_key: str = ""
    mistral_api_key: str = ""
    ollama_base_url: str = "http://localhost:11434"
    
    # OrgMyLife Integration
    orgmylife_api_url: str = "https://api.andreknie.de"
    orgmylife_api_key: str = ""
    
    # Paths
    notes_dir: str = "./notes"
    inbox_dir: str = "./ingestion/inbox"
    
    # OCR
    ocr_enabled: bool = True
    ocr_language: str = "deu+eng"  # German + English
    
    # Behavior
    confidence_threshold: float = 0.7
    auto_commit: bool = True
    
    class Config:
        env_file = ".env"
        env_prefix = ""
    
    @property
    def llm_model_string(self) -> str:
        """LiteLLM model string (e.g., 'openai/gpt-4o')."""
        if self.llm_provider == "ollama":
            return f"ollama/{self.llm_model}"
        return f"{self.llm_provider}/{self.llm_model}"
    
    def validate_api_key(self) -> None:
        """Raise if required API key is missing."""
        ...

Data Models

Enrichment Prompt (Structured Output)

The LLM receives extracted text and a system prompt requesting structured JSON output:

ENRICHMENT_SYSTEM_PROMPT = """You are a knowledge management assistant. Analyze the following text 
extracted from a document and return structured metadata.

Return a JSON object with:
- title: A concise, descriptive title for this note (max 80 chars)
- category: One of "meeting", "project", "decision", "inbox"
- tags: List of relevant topic tags (lowercase, no #)
- entities: List of detected entities, each with:
  - type: "person", "project", "date", or "action_item"
  - value: The extracted text
  - confidence: Float 0.0-1.0
  - position: Character offset where entity appears (approximate)
- action_items: List of action items, each with:
  - description: What needs to be done
  - assignee: Person responsible (if mentioned)
  - deadline: Date in ISO 8601 format (if mentioned)
- summary: Optional 1-2 sentence summary

Context: This is a personal knowledge base for a project manager working across 
multiple business projects. Common projects include: {known_projects}. 
Known people: {known_people}.
"""

Output Markdown Format

---
title: "Meeting with Beier GmbH about Druckluft project"
date: 2024-03-15
tags:
  - meeting
  - druckluft
  - customer
people:
  - André Knierim
  - Thomas Beier
projects:
  - Beier GmbH
  - Druckluft
source:
  file: "Beier GmbH.pdf"
  imported: "2024-03-20T14:30:00"
---

# Meeting with Beier GmbH about Druckluft project

Discussion about the [[projects/druckluft]] project with [[people/thomas-beier]].

Key points:
- Delivery timeline confirmed for Q2
- Budget approved at €45,000
- Next review meeting scheduled for April 10

## Action Items

- [ ] Send updated proposal to Thomas Beier (deadline: 2024-03-22)
- [ ] Schedule follow-up meeting for April 10
- [ ] Update project timeline in planning tool

Entity-to-WikiLink Mapping

Entity Type	Wiki-Link Format	Example
Person	[[people/firstname-lastname]]	[[people/thomas-beier]]
Project	[[projects/project-slug]]	[[projects/druckluft]]
Date	Frontmatter date field	2024-03-15
Action Item	## Action Items section	- [ ] Send proposal

File Routing Rules

Category	Target Folder	Condition
Meeting	notes/meetings/	Category = "meeting"
Project	notes/projects/	Category = "project" with single dominant project
Decision	notes/decisions/	Category = "decision"
Default	notes/inbox/	No clear category or mixed

Correctness Properties

A property is a characteristic or behavior that should hold true across all valid executions of a system — essentially, a formal statement about what the system should do. Properties serve as the bridge between human-readable specifications and machine-verifiable correctness guarantees.

Property 1: File discovery returns only supported extensions

For any directory tree containing files with arbitrary extensions, the discovery function SHALL return exactly and only those files whose extensions are in the supported set (.pdf, .jpg, .jpeg, .png, .md, .txt, .docx).

Validates: Requirements 1.1, 1.6

Property 2: Bulk import resilience — failed files do not block remaining processing

For any batch of N files where a subset F fails during processing, the pipeline SHALL successfully process all N-F non-failing files and the summary SHALL report success count = N-F and failed count = |F|.

Validates: Requirements 1.4, 1.5

Property 3: Text extraction preserves content for text-based formats

For any valid UTF-8 string written to a markdown or plain text file, extracting it via the Text_Extractor SHALL return the identical string content.

Validates: Requirements 3.1, 3.2, 3.4

Property 4: Entity parsing extracts all entities with valid structure

For any valid enrichment response JSON containing entities, each parsed entity SHALL have a non-empty type (one of person/project/date/action_item), a non-empty value, and a confidence score in the range [0.0, 1.0].

Validates: Requirements 4.1, 4.2, 4.3, 4.4, 4.5

Property 5: Confidence threshold filtering

For any list of entities with arbitrary confidence scores, the filtered output SHALL contain exactly those entities with confidence >= 0.7 and no others.

Validates: Requirements 4.6

Property 6: Wiki-link generation follows correct format per entity type

For any person name, the generated wiki-link SHALL match the pattern [[people/slugified-name]]. For any project name, the generated wiki-link SHALL match the pattern [[projects/slugified-name]].

Validates: Requirements 5.1, 5.2

Property 7: Only first entity occurrence is linked

For any text containing N >= 2 occurrences of the same entity, the Reference_Linker SHALL convert exactly the first occurrence to a wiki-link and leave all subsequent occurrences as plain text.

Validates: Requirements 5.6

Property 8: Frontmatter round-trip validity

For any valid EnrichmentResult, rendering it to markdown with YAML frontmatter and then parsing the frontmatter back SHALL produce a valid YAML block containing all required fields (title, date, tags, people, projects, source).

Validates: Requirements 6.1, 6.3, 6.4, 6.6, 6.7

Property 9: Provider routing from configuration

For any valid provider name from the supported set (openai, anthropic, google, mistral, ollama), configuring LLM_PROVIDER to that value SHALL cause the LLM client to route requests to the corresponding provider.

Validates: Requirements 7.3, 8.1

Property 10: Missing API key produces descriptive error

For any provider that requires an API key, when the corresponding environment variable is unset, the configuration validation SHALL raise an error that names the missing variable.

Validates: Requirements 7.4, 8.6

Property 11: Action items appear in output markdown section

For any list of extracted action items, the rendered markdown SHALL contain a ## Action Items section listing every action item description.

Validates: Requirements 9.2

Property 12: OrgMyLife task payload correctness

For any action item description and note file path, the task creation payload SHALL set title to the action item description and source_url to a URL referencing the note path.

Validates: Requirements 9.4

Property 13: Category-based folder routing

For any enrichment result with category "meeting", the output path SHALL be within notes/meetings/. For any result with category "project", the path SHALL be within notes/projects/. For any result with no clear category, the path SHALL be within notes/inbox/.

Validates: Requirements 10.1, 10.2, 10.3

Property 14: Output filename follows date-slug pattern

For any date and title string, the generated filename SHALL match the pattern YYYY-MM-DD-slugified-title.md where the slug contains only lowercase alphanumeric characters and hyphens.

Validates: Requirements 10.5

Property 15: Filename collision avoidance

For any set of existing files, when a new file would have the same name as an existing file, the system SHALL append a numeric suffix (-2, -3, etc.) producing a unique filename.

Validates: Requirements 10.6

Property 16: Temporary file filtering

For any filename, the watcher ignore logic SHALL return true if and only if the filename starts with . or ends with .tmp.

Validates: Requirements 11.5

Property 17: Dry-run produces no file writes

For any input processed with the --dry-run flag, the system SHALL create zero files on disk.

Validates: Requirements 13.6

Property 18: Git commit message format

For any count N and source type string, the generated commit message SHALL match the format ingestion: import N notes from [source-type].

Validates: Requirements 12.2

Error Handling

Error Categories and Strategies

Error Type	Strategy	User Impact
File read failure	Log + skip file, continue batch	Warning in summary
OCR failure (no text)	Create note with empty body + source ref	Note created, flagged
Encoding detection failure	Log + skip file	Warning in summary
LLM API error	Retry 3x with exponential backoff, then fail file	Warning per file
LLM response parse error	Retry once with stricter prompt, then use defaults	Degraded metadata
OrgMyLife API unreachable	Log warning, continue without task creation	Tasks not created
Git commit failure	Log warning, files still written	No version history
Disk full / write failure	Fail immediately with clear error	Process stops
Invalid config (missing key)	Exit with descriptive error before processing	Immediate feedback

Retry Policy

RETRY_CONFIG = {
    "max_retries": 3,
    "base_delay_seconds": 1.0,
    "max_delay_seconds": 30.0,
    "backoff_factor": 2.0,
    "retryable_errors": [
        "rate_limit",
        "timeout", 
        "server_error",  # 5xx
    ]
}

Logging

• Level INFO: File processing progress, summary counts
• Level WARNING: Skipped files, API failures (non-fatal), git failures
• Level ERROR: Configuration errors, unrecoverable failures
• Level DEBUG: LLM prompts/responses, entity detection details

All logs include timestamps and structured context (file path, provider, operation).

Testing Strategy

Property-Based Tests (Hypothesis)

The project uses Hypothesis (Python's standard PBT library) for property-based testing. Each property test runs a minimum of 100 iterations.

Property tests cover the core transformation logic:

• File discovery filtering (Property 1)
• Text extraction identity (Property 3)
• Entity parsing and filtering (Properties 4, 5)
• Wiki-link generation (Properties 6, 7)
• Frontmatter round-trip (Property 8)
• Folder routing (Property 13)
• Filename generation and collision avoidance (Properties 14, 15)
• Temporary file filtering (Property 16)

Each property test is tagged with:

# Feature: notegraph-ingestion, Property N: [property text]

Configuration: minimum 100 examples per test via @settings(max_examples=100).

Unit Tests (pytest)

Example-based tests for:

• CLI argument parsing and subcommand routing
• Configuration loading from environment variables
• Specific enrichment response scenarios
• OrgMyLife client request formatting
• Git commit message generation
• Dry-run mode behavior

Integration Tests

Integration tests (marked with @pytest.mark.integration) for:

• OCR extraction with sample images
• PDF text extraction with sample PDFs
• DOCX extraction with sample files
• LLM API calls (with real provider, gated by env var)
• OrgMyLife API task creation (against test instance)
• Drop-folder watcher file detection
• Git operations in a test repository

Test Dependencies

[project.optional-dependencies]
dev = [
    "pytest>=8.0",
    "pytest-asyncio>=0.23",
    "hypothesis>=6.100",
    "pytest-mock>=3.12",
    "respx>=0.21",  # httpx mocking
]