# Design Document: NoteGraph Ingestion & Auto-Reference

## Overview

The NoteGraph Ingestion service is a Python package (`NoteGraph/ingestion/`) that provides a CLI-driven pipeline for importing documents (PDFs, images, markdown, text, DOCX) into the NoteGraph knowledge base. The pipeline extracts text (via OCR or direct extraction), enriches it using an LLM to detect entities (people, projects, dates, action items), generates wiki-links, and outputs structured markdown files with YAML frontmatter into the appropriate NoteGraph folder.

The system is designed around three principles:
1. **Model-agnostic** — A unified LLM interface (via LiteLLM) supports OpenAI, Anthropic, Google, Mistral, and local Ollama without code changes.
2. **CLI-first** — All operations are accessible via `python -m ingestion.cli` with subcommands for bulk import and continuous watching.
3. **Markdown-native** — Output is always SilverBullet-compatible markdown with YAML frontmatter and wiki-links.

## Architecture

```mermaid
graph TD
    subgraph Input
        CLI[CLI Interface]
        WATCH[Drop-Folder Watcher]
    end

    subgraph Pipeline
        DISC[File Discovery]
        EXT[Text Extraction]
        OCR[OCR Engine]
        ENR[Enrichment Agent]
        LINK[Reference Linker]
        RENDER[Markdown Renderer]
    end

    subgraph Output
        MD[Markdown Files]
        STUB[Stub Pages]
        GIT[Git Commit]
        TASK[OrgMyLife Tasks]
    end

    subgraph External
        LLM[LLM Provider]
        ORG[OrgMyLife API]
    end

    CLI --> DISC
    WATCH --> DISC
    DISC --> EXT
    EXT --> OCR
    EXT --> ENR
    ENR --> LLM
    ENR --> LINK
    LINK --> RENDER
    RENDER --> MD
    RENDER --> STUB
    MD --> GIT
    STUB --> GIT
    ENR --> TASK
    TASK --> ORG
```

### Data Flow

```
Input File → File Discovery → Text Extraction (OCR if needed)
    → Enrichment Agent (LLM call) → Entity Detection
    → Reference Linker (wiki-links) → Markdown Renderer (frontmatter + body)
    → File Placement (folder routing) → Git Commit
    → (optional) OrgMyLife Task Creation
```

## Components and Interfaces

### Package Structure

```
NoteGraph/ingestion/
├── __init__.py
├── __main__.py              # python -m ingestion entry point
├── cli.py                   # Click-based CLI (import, watch subcommands)
├── config.py                # Configuration from .env
├── pipeline.py              # Main orchestration pipeline
├── discovery.py             # File discovery and filtering
├── extraction/
│   ├── __init__.py
│   ├── base.py              # Extractor protocol
│   ├── text.py              # Plain text / markdown extractor
│   ├── pdf.py               # PDF text extraction (PyPDF2/pdfplumber)
│   ├── ocr.py               # OCR via Tesseract (pytesseract)
│   └── docx.py              # DOCX extraction (python-docx)
├── enrichment/
│   ├── __init__.py
│   ├── agent.py             # Enrichment orchestration
│   ├── prompts.py           # Structured prompts for entity detection
│   └── models.py            # Pydantic models for enrichment response
├── linking/
│   ├── __init__.py
│   ├── linker.py            # Reference linker (wiki-link generation)
│   └── stubs.py             # Stub page creation
├── output/
│   ├── __init__.py
│   ├── renderer.py          # Markdown + frontmatter rendering
│   ├── router.py            # Folder routing logic
│   └── naming.py            # Filename generation with collision avoidance
├── integrations/
│   ├── __init__.py
│   ├── llm.py               # LiteLLM wrapper (model-agnostic)
│   ├── orgmylife.py         # OrgMyLife API client
│   └── git.py               # Git commit operations
├── watcher.py               # Drop-folder watcher (watchdog)
└── tests/
    ├── __init__.py
    ├── conftest.py
    ├── test_discovery.py
    ├── test_extraction.py
    ├── test_enrichment.py
    ├── test_linking.py
    ├── test_output.py
    ├── test_pipeline.py
    └── test_properties.py   # Property-based tests
```

### Core Interfaces

```python
# ingestion/extraction/base.py
from typing import Protocol

class TextExtractor(Protocol):
    """Protocol for all text extractors."""
    
    def can_handle(self, file_path: Path) -> bool:
        """Return True if this extractor handles the given file type."""
        ...
    
    def extract(self, file_path: Path) -> ExtractionResult:
        """Extract text content from the file."""
        ...

@dataclass
class ExtractionResult:
    text: str
    source_file: Path
    extraction_method: str  # "direct", "ocr", "mixed"
    metadata: dict[str, Any] = field(default_factory=dict)
```

```python
# ingestion/enrichment/models.py
from pydantic import BaseModel, Field

class Entity(BaseModel):
    type: Literal["person", "project", "date", "action_item"]
    value: str
    confidence: float = Field(ge=0.0, le=1.0)
    position: int | None = None  # character offset in source text

class ActionItem(BaseModel):
    description: str
    assignee: str | None = None
    deadline: str | None = None  # ISO 8601

class EnrichmentResult(BaseModel):
    title: str
    category: Literal["meeting", "project", "decision", "inbox"]
    tags: list[str]
    entities: list[Entity]
    action_items: list[ActionItem]
    summary: str | None = None
```

```python
# ingestion/integrations/llm.py
from litellm import completion

class LLMClient:
    """Model-agnostic LLM client using LiteLLM."""
    
    def __init__(self, config: IngestionConfig):
        self.model = config.llm_model_string  # e.g. "openai/gpt-4o"
    
    def complete(self, messages: list[dict], response_format: type[BaseModel]) -> BaseModel:
        """Send a structured completion request."""
        ...
```

```python
# ingestion/integrations/orgmylife.py
import httpx

class OrgMyLifeClient:
    """HTTP client for OrgMyLife task creation."""
    
    def __init__(self, base_url: str, api_key: str):
        self.base_url = base_url
        self.api_key = api_key
    
    def create_task(self, title: str, source_url: str, 
                    description: str = "", priority: int = 4) -> dict:
        """Create a task via POST /api/tasks."""
        ...
```

### Configuration

```python
# ingestion/config.py
from pydantic_settings import BaseSettings

class IngestionConfig(BaseSettings):
    # LLM Provider
    llm_provider: str = "openai"  # openai, anthropic, google, mistral, ollama
    llm_model: str = "gpt-4o"
    openai_api_key: str = ""
    anthropic_api_key: str = ""
    google_api_key: str = ""
    mistral_api_key: str = ""
    ollama_base_url: str = "http://localhost:11434"
    
    # OrgMyLife Integration
    orgmylife_api_url: str = "https://api.andreknie.de"
    orgmylife_api_key: str = ""
    
    # Paths
    notes_dir: str = "./notes"
    inbox_dir: str = "./ingestion/inbox"
    
    # OCR
    ocr_enabled: bool = True
    ocr_language: str = "deu+eng"  # German + English
    
    # Behavior
    confidence_threshold: float = 0.7
    auto_commit: bool = True
    
    class Config:
        env_file = ".env"
        env_prefix = ""
    
    @property
    def llm_model_string(self) -> str:
        """LiteLLM model string (e.g., 'openai/gpt-4o')."""
        if self.llm_provider == "ollama":
            return f"ollama/{self.llm_model}"
        return f"{self.llm_provider}/{self.llm_model}"
    
    def validate_api_key(self) -> None:
        """Raise if required API key is missing."""
        ...
```

## Data Models

### Enrichment Prompt (Structured Output)

The LLM receives extracted text and a system prompt requesting structured JSON output:

```python
ENRICHMENT_SYSTEM_PROMPT = """You are a knowledge management assistant. Analyze the following text 
extracted from a document and return structured metadata.

Return a JSON object with:
- title: A concise, descriptive title for this note (max 80 chars)
- category: One of "meeting", "project", "decision", "inbox"
- tags: List of relevant topic tags (lowercase, no #)
- entities: List of detected entities, each with:
  - type: "person", "project", "date", or "action_item"
  - value: The extracted text
  - confidence: Float 0.0-1.0
  - position: Character offset where entity appears (approximate)
- action_items: List of action items, each with:
  - description: What needs to be done
  - assignee: Person responsible (if mentioned)
  - deadline: Date in ISO 8601 format (if mentioned)
- summary: Optional 1-2 sentence summary

Context: This is a personal knowledge base for a project manager working across 
multiple business projects. Common projects include: {known_projects}. 
Known people: {known_people}.
"""
```

### Output Markdown Format

```yaml
---
title: "Meeting with Beier GmbH about Druckluft project"
date: 2024-03-15
tags:
  - meeting
  - druckluft
  - customer
people:
  - André Knierim
  - Thomas Beier
projects:
  - Beier GmbH
  - Druckluft
source:
  file: "Beier GmbH.pdf"
  imported: "2024-03-20T14:30:00"
---

# Meeting with Beier GmbH about Druckluft project

Discussion about the [[projects/druckluft]] project with [[people/thomas-beier]].

Key points:
- Delivery timeline confirmed for Q2
- Budget approved at €45,000
- Next review meeting scheduled for April 10

## Action Items

- [ ] Send updated proposal to Thomas Beier (deadline: 2024-03-22)
- [ ] Schedule follow-up meeting for April 10
- [ ] Update project timeline in planning tool
```

### Entity-to-WikiLink Mapping

| Entity Type | Wiki-Link Format | Example |
|-------------|-----------------|---------|
| Person | `[[people/firstname-lastname]]` | `[[people/thomas-beier]]` |
| Project | `[[projects/project-slug]]` | `[[projects/druckluft]]` |
| Date | Frontmatter `date` field | `2024-03-15` |
| Action Item | `## Action Items` section | `- [ ] Send proposal` |

### File Routing Rules

| Category | Target Folder | Condition |
|----------|--------------|-----------|
| Meeting | `notes/meetings/` | Category = "meeting" |
| Project | `notes/projects/` | Category = "project" with single dominant project |
| Decision | `notes/decisions/` | Category = "decision" |
| Default | `notes/inbox/` | No clear category or mixed |

## Correctness Properties

*A property is a characteristic or behavior that should hold true across all valid executions of a system — essentially, a formal statement about what the system should do. Properties serve as the bridge between human-readable specifications and machine-verifiable correctness guarantees.*

### Property 1: File discovery returns only supported extensions

*For any* directory tree containing files with arbitrary extensions, the discovery function SHALL return exactly and only those files whose extensions are in the supported set (`.pdf`, `.jpg`, `.jpeg`, `.png`, `.md`, `.txt`, `.docx`).

**Validates: Requirements 1.1, 1.6**

### Property 2: Bulk import resilience — failed files do not block remaining processing

*For any* batch of N files where a subset F fails during processing, the pipeline SHALL successfully process all N-F non-failing files and the summary SHALL report success count = N-F and failed count = |F|.

**Validates: Requirements 1.4, 1.5**

### Property 3: Text extraction preserves content for text-based formats

*For any* valid UTF-8 string written to a markdown or plain text file, extracting it via the Text_Extractor SHALL return the identical string content.

**Validates: Requirements 3.1, 3.2, 3.4**

### Property 4: Entity parsing extracts all entities with valid structure

*For any* valid enrichment response JSON containing entities, each parsed entity SHALL have a non-empty `type` (one of person/project/date/action_item), a non-empty `value`, and a `confidence` score in the range [0.0, 1.0].

**Validates: Requirements 4.1, 4.2, 4.3, 4.4, 4.5**

### Property 5: Confidence threshold filtering

*For any* list of entities with arbitrary confidence scores, the filtered output SHALL contain exactly those entities with confidence >= 0.7 and no others.

**Validates: Requirements 4.6**

### Property 6: Wiki-link generation follows correct format per entity type

*For any* person name, the generated wiki-link SHALL match the pattern `[[people/slugified-name]]`. *For any* project name, the generated wiki-link SHALL match the pattern `[[projects/slugified-name]]`.

**Validates: Requirements 5.1, 5.2**

### Property 7: Only first entity occurrence is linked

*For any* text containing N >= 2 occurrences of the same entity, the Reference_Linker SHALL convert exactly the first occurrence to a wiki-link and leave all subsequent occurrences as plain text.

**Validates: Requirements 5.6**

### Property 8: Frontmatter round-trip validity

*For any* valid `EnrichmentResult`, rendering it to markdown with YAML frontmatter and then parsing the frontmatter back SHALL produce a valid YAML block containing all required fields (title, date, tags, people, projects, source).

**Validates: Requirements 6.1, 6.3, 6.4, 6.6, 6.7**

### Property 9: Provider routing from configuration

*For any* valid provider name from the supported set (openai, anthropic, google, mistral, ollama), configuring `LLM_PROVIDER` to that value SHALL cause the LLM client to route requests to the corresponding provider.

**Validates: Requirements 7.3, 8.1**

### Property 10: Missing API key produces descriptive error

*For any* provider that requires an API key, when the corresponding environment variable is unset, the configuration validation SHALL raise an error that names the missing variable.

**Validates: Requirements 7.4, 8.6**

### Property 11: Action items appear in output markdown section

*For any* list of extracted action items, the rendered markdown SHALL contain a `## Action Items` section listing every action item description.

**Validates: Requirements 9.2**

### Property 12: OrgMyLife task payload correctness

*For any* action item description and note file path, the task creation payload SHALL set `title` to the action item description and `source_url` to a URL referencing the note path.

**Validates: Requirements 9.4**

### Property 13: Category-based folder routing

*For any* enrichment result with category "meeting", the output path SHALL be within `notes/meetings/`. *For any* result with category "project", the path SHALL be within `notes/projects/`. *For any* result with no clear category, the path SHALL be within `notes/inbox/`.

**Validates: Requirements 10.1, 10.2, 10.3**

### Property 14: Output filename follows date-slug pattern

*For any* date and title string, the generated filename SHALL match the pattern `YYYY-MM-DD-slugified-title.md` where the slug contains only lowercase alphanumeric characters and hyphens.

**Validates: Requirements 10.5**

### Property 15: Filename collision avoidance

*For any* set of existing files, when a new file would have the same name as an existing file, the system SHALL append a numeric suffix (`-2`, `-3`, etc.) producing a unique filename.

**Validates: Requirements 10.6**

### Property 16: Temporary file filtering

*For any* filename, the watcher ignore logic SHALL return true if and only if the filename starts with `.` or ends with `.tmp`.

**Validates: Requirements 11.5**

### Property 17: Dry-run produces no file writes

*For any* input processed with the `--dry-run` flag, the system SHALL create zero files on disk.

**Validates: Requirements 13.6**

### Property 18: Git commit message format

*For any* count N and source type string, the generated commit message SHALL match the format `ingestion: import N notes from [source-type]`.

**Validates: Requirements 12.2**

## Error Handling

### Error Categories and Strategies

| Error Type | Strategy | User Impact |
|-----------|----------|-------------|
| File read failure | Log + skip file, continue batch | Warning in summary |
| OCR failure (no text) | Create note with empty body + source ref | Note created, flagged |
| Encoding detection failure | Log + skip file | Warning in summary |
| LLM API error | Retry 3x with exponential backoff, then fail file | Warning per file |
| LLM response parse error | Retry once with stricter prompt, then use defaults | Degraded metadata |
| OrgMyLife API unreachable | Log warning, continue without task creation | Tasks not created |
| Git commit failure | Log warning, files still written | No version history |
| Disk full / write failure | Fail immediately with clear error | Process stops |
| Invalid config (missing key) | Exit with descriptive error before processing | Immediate feedback |

### Retry Policy

```python
RETRY_CONFIG = {
    "max_retries": 3,
    "base_delay_seconds": 1.0,
    "max_delay_seconds": 30.0,
    "backoff_factor": 2.0,
    "retryable_errors": [
        "rate_limit",
        "timeout", 
        "server_error",  # 5xx
    ]
}
```

### Logging

- **Level INFO**: File processing progress, summary counts
- **Level WARNING**: Skipped files, API failures (non-fatal), git failures
- **Level ERROR**: Configuration errors, unrecoverable failures
- **Level DEBUG**: LLM prompts/responses, entity detection details

All logs include timestamps and structured context (file path, provider, operation).

## Testing Strategy

### Property-Based Tests (Hypothesis)

The project uses **Hypothesis** (Python's standard PBT library) for property-based testing. Each property test runs a minimum of 100 iterations.

Property tests cover the core transformation logic:
- File discovery filtering (Property 1)
- Text extraction identity (Property 3)
- Entity parsing and filtering (Properties 4, 5)
- Wiki-link generation (Properties 6, 7)
- Frontmatter round-trip (Property 8)
- Folder routing (Property 13)
- Filename generation and collision avoidance (Properties 14, 15)
- Temporary file filtering (Property 16)

Each property test is tagged with:
```python
# Feature: notegraph-ingestion, Property N: [property text]
```

Configuration: minimum 100 examples per test via `@settings(max_examples=100)`.

### Unit Tests (pytest)

Example-based tests for:
- CLI argument parsing and subcommand routing
- Configuration loading from environment variables
- Specific enrichment response scenarios
- OrgMyLife client request formatting
- Git commit message generation
- Dry-run mode behavior

### Integration Tests

Integration tests (marked with `@pytest.mark.integration`) for:
- OCR extraction with sample images
- PDF text extraction with sample PDFs
- DOCX extraction with sample files
- LLM API calls (with real provider, gated by env var)
- OrgMyLife API task creation (against test instance)
- Drop-folder watcher file detection
- Git operations in a test repository

### Test Dependencies

```toml
[project.optional-dependencies]
dev = [
    "pytest>=8.0",
    "pytest-asyncio>=0.23",
    "hypothesis>=6.100",
    "pytest-mock>=3.12",
    "respx>=0.21",  # httpx mocking
]
```