Initial monorepo structure
This commit is contained in:
@@ -0,0 +1 @@
|
||||
{"specId": "e33fa6e5-89e0-4d8d-ad9e-d43d6bf23daa", "workflowType": "requirements-first", "specType": "feature"}
|
||||
@@ -0,0 +1,542 @@
|
||||
# Design Document: NoteGraph Ingestion & Auto-Reference
|
||||
|
||||
## Overview
|
||||
|
||||
The NoteGraph Ingestion service is a Python package (`NoteGraph/ingestion/`) that provides a CLI-driven pipeline for importing documents (PDFs, images, markdown, text, DOCX) into the NoteGraph knowledge base. The pipeline extracts text (via OCR or direct extraction), enriches it using an LLM to detect entities (people, projects, dates, action items), generates wiki-links, and outputs structured markdown files with YAML frontmatter into the appropriate NoteGraph folder.
|
||||
|
||||
The system is designed around three principles:
|
||||
1. **Model-agnostic** — A unified LLM interface (via LiteLLM) supports OpenAI, Anthropic, Google, Mistral, and local Ollama without code changes.
|
||||
2. **CLI-first** — All operations are accessible via `python -m ingestion.cli` with subcommands for bulk import and continuous watching.
|
||||
3. **Markdown-native** — Output is always SilverBullet-compatible markdown with YAML frontmatter and wiki-links.
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph Input
|
||||
CLI[CLI Interface]
|
||||
WATCH[Drop-Folder Watcher]
|
||||
end
|
||||
|
||||
subgraph Pipeline
|
||||
DISC[File Discovery]
|
||||
EXT[Text Extraction]
|
||||
OCR[OCR Engine]
|
||||
ENR[Enrichment Agent]
|
||||
LINK[Reference Linker]
|
||||
RENDER[Markdown Renderer]
|
||||
end
|
||||
|
||||
subgraph Output
|
||||
MD[Markdown Files]
|
||||
STUB[Stub Pages]
|
||||
GIT[Git Commit]
|
||||
TASK[OrgMyLife Tasks]
|
||||
end
|
||||
|
||||
subgraph External
|
||||
LLM[LLM Provider]
|
||||
ORG[OrgMyLife API]
|
||||
end
|
||||
|
||||
CLI --> DISC
|
||||
WATCH --> DISC
|
||||
DISC --> EXT
|
||||
EXT --> OCR
|
||||
EXT --> ENR
|
||||
ENR --> LLM
|
||||
ENR --> LINK
|
||||
LINK --> RENDER
|
||||
RENDER --> MD
|
||||
RENDER --> STUB
|
||||
MD --> GIT
|
||||
STUB --> GIT
|
||||
ENR --> TASK
|
||||
TASK --> ORG
|
||||
```
|
||||
|
||||
### Data Flow
|
||||
|
||||
```
|
||||
Input File → File Discovery → Text Extraction (OCR if needed)
|
||||
→ Enrichment Agent (LLM call) → Entity Detection
|
||||
→ Reference Linker (wiki-links) → Markdown Renderer (frontmatter + body)
|
||||
→ File Placement (folder routing) → Git Commit
|
||||
→ (optional) OrgMyLife Task Creation
|
||||
```
|
||||
|
||||
## Components and Interfaces
|
||||
|
||||
### Package Structure
|
||||
|
||||
```
|
||||
NoteGraph/ingestion/
|
||||
├── __init__.py
|
||||
├── __main__.py # python -m ingestion entry point
|
||||
├── cli.py # Click-based CLI (import, watch subcommands)
|
||||
├── config.py # Configuration from .env
|
||||
├── pipeline.py # Main orchestration pipeline
|
||||
├── discovery.py # File discovery and filtering
|
||||
├── extraction/
|
||||
│ ├── __init__.py
|
||||
│ ├── base.py # Extractor protocol
|
||||
│ ├── text.py # Plain text / markdown extractor
|
||||
│ ├── pdf.py # PDF text extraction (PyPDF2/pdfplumber)
|
||||
│ ├── ocr.py # OCR via Tesseract (pytesseract)
|
||||
│ └── docx.py # DOCX extraction (python-docx)
|
||||
├── enrichment/
|
||||
│ ├── __init__.py
|
||||
│ ├── agent.py # Enrichment orchestration
|
||||
│ ├── prompts.py # Structured prompts for entity detection
|
||||
│ └── models.py # Pydantic models for enrichment response
|
||||
├── linking/
|
||||
│ ├── __init__.py
|
||||
│ ├── linker.py # Reference linker (wiki-link generation)
|
||||
│ └── stubs.py # Stub page creation
|
||||
├── output/
|
||||
│ ├── __init__.py
|
||||
│ ├── renderer.py # Markdown + frontmatter rendering
|
||||
│ ├── router.py # Folder routing logic
|
||||
│ └── naming.py # Filename generation with collision avoidance
|
||||
├── integrations/
|
||||
│ ├── __init__.py
|
||||
│ ├── llm.py # LiteLLM wrapper (model-agnostic)
|
||||
│ ├── orgmylife.py # OrgMyLife API client
|
||||
│ └── git.py # Git commit operations
|
||||
├── watcher.py # Drop-folder watcher (watchdog)
|
||||
└── tests/
|
||||
├── __init__.py
|
||||
├── conftest.py
|
||||
├── test_discovery.py
|
||||
├── test_extraction.py
|
||||
├── test_enrichment.py
|
||||
├── test_linking.py
|
||||
├── test_output.py
|
||||
├── test_pipeline.py
|
||||
└── test_properties.py # Property-based tests
|
||||
```
|
||||
|
||||
### Core Interfaces
|
||||
|
||||
```python
|
||||
# ingestion/extraction/base.py
|
||||
from typing import Protocol
|
||||
|
||||
class TextExtractor(Protocol):
|
||||
"""Protocol for all text extractors."""
|
||||
|
||||
def can_handle(self, file_path: Path) -> bool:
|
||||
"""Return True if this extractor handles the given file type."""
|
||||
...
|
||||
|
||||
def extract(self, file_path: Path) -> ExtractionResult:
|
||||
"""Extract text content from the file."""
|
||||
...
|
||||
|
||||
@dataclass
|
||||
class ExtractionResult:
|
||||
text: str
|
||||
source_file: Path
|
||||
extraction_method: str # "direct", "ocr", "mixed"
|
||||
metadata: dict[str, Any] = field(default_factory=dict)
|
||||
```
|
||||
|
||||
```python
|
||||
# ingestion/enrichment/models.py
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
class Entity(BaseModel):
|
||||
type: Literal["person", "project", "date", "action_item"]
|
||||
value: str
|
||||
confidence: float = Field(ge=0.0, le=1.0)
|
||||
position: int | None = None # character offset in source text
|
||||
|
||||
class ActionItem(BaseModel):
|
||||
description: str
|
||||
assignee: str | None = None
|
||||
deadline: str | None = None # ISO 8601
|
||||
|
||||
class EnrichmentResult(BaseModel):
|
||||
title: str
|
||||
category: Literal["meeting", "project", "decision", "inbox"]
|
||||
tags: list[str]
|
||||
entities: list[Entity]
|
||||
action_items: list[ActionItem]
|
||||
summary: str | None = None
|
||||
```
|
||||
|
||||
```python
|
||||
# ingestion/integrations/llm.py
|
||||
from litellm import completion
|
||||
|
||||
class LLMClient:
|
||||
"""Model-agnostic LLM client using LiteLLM."""
|
||||
|
||||
def __init__(self, config: IngestionConfig):
|
||||
self.model = config.llm_model_string # e.g. "openai/gpt-4o"
|
||||
|
||||
def complete(self, messages: list[dict], response_format: type[BaseModel]) -> BaseModel:
|
||||
"""Send a structured completion request."""
|
||||
...
|
||||
```
|
||||
|
||||
```python
|
||||
# ingestion/integrations/orgmylife.py
|
||||
import httpx
|
||||
|
||||
class OrgMyLifeClient:
|
||||
"""HTTP client for OrgMyLife task creation."""
|
||||
|
||||
def __init__(self, base_url: str, api_key: str):
|
||||
self.base_url = base_url
|
||||
self.api_key = api_key
|
||||
|
||||
def create_task(self, title: str, source_url: str,
|
||||
description: str = "", priority: int = 4) -> dict:
|
||||
"""Create a task via POST /api/tasks."""
|
||||
...
|
||||
```
|
||||
|
||||
### Configuration
|
||||
|
||||
```python
|
||||
# ingestion/config.py
|
||||
from pydantic_settings import BaseSettings
|
||||
|
||||
class IngestionConfig(BaseSettings):
|
||||
# LLM Provider
|
||||
llm_provider: str = "openai" # openai, anthropic, google, mistral, ollama
|
||||
llm_model: str = "gpt-4o"
|
||||
openai_api_key: str = ""
|
||||
anthropic_api_key: str = ""
|
||||
google_api_key: str = ""
|
||||
mistral_api_key: str = ""
|
||||
ollama_base_url: str = "http://localhost:11434"
|
||||
|
||||
# OrgMyLife Integration
|
||||
orgmylife_api_url: str = "https://api.andreknie.de"
|
||||
orgmylife_api_key: str = ""
|
||||
|
||||
# Paths
|
||||
notes_dir: str = "./notes"
|
||||
inbox_dir: str = "./ingestion/inbox"
|
||||
|
||||
# OCR
|
||||
ocr_enabled: bool = True
|
||||
ocr_language: str = "deu+eng" # German + English
|
||||
|
||||
# Behavior
|
||||
confidence_threshold: float = 0.7
|
||||
auto_commit: bool = True
|
||||
|
||||
class Config:
|
||||
env_file = ".env"
|
||||
env_prefix = ""
|
||||
|
||||
@property
|
||||
def llm_model_string(self) -> str:
|
||||
"""LiteLLM model string (e.g., 'openai/gpt-4o')."""
|
||||
if self.llm_provider == "ollama":
|
||||
return f"ollama/{self.llm_model}"
|
||||
return f"{self.llm_provider}/{self.llm_model}"
|
||||
|
||||
def validate_api_key(self) -> None:
|
||||
"""Raise if required API key is missing."""
|
||||
...
|
||||
```
|
||||
|
||||
## Data Models
|
||||
|
||||
### Enrichment Prompt (Structured Output)
|
||||
|
||||
The LLM receives extracted text and a system prompt requesting structured JSON output:
|
||||
|
||||
```python
|
||||
ENRICHMENT_SYSTEM_PROMPT = """You are a knowledge management assistant. Analyze the following text
|
||||
extracted from a document and return structured metadata.
|
||||
|
||||
Return a JSON object with:
|
||||
- title: A concise, descriptive title for this note (max 80 chars)
|
||||
- category: One of "meeting", "project", "decision", "inbox"
|
||||
- tags: List of relevant topic tags (lowercase, no #)
|
||||
- entities: List of detected entities, each with:
|
||||
- type: "person", "project", "date", or "action_item"
|
||||
- value: The extracted text
|
||||
- confidence: Float 0.0-1.0
|
||||
- position: Character offset where entity appears (approximate)
|
||||
- action_items: List of action items, each with:
|
||||
- description: What needs to be done
|
||||
- assignee: Person responsible (if mentioned)
|
||||
- deadline: Date in ISO 8601 format (if mentioned)
|
||||
- summary: Optional 1-2 sentence summary
|
||||
|
||||
Context: This is a personal knowledge base for a project manager working across
|
||||
multiple business projects. Common projects include: {known_projects}.
|
||||
Known people: {known_people}.
|
||||
"""
|
||||
```
|
||||
|
||||
### Output Markdown Format
|
||||
|
||||
```yaml
|
||||
---
|
||||
title: "Meeting with Beier GmbH about Druckluft project"
|
||||
date: 2024-03-15
|
||||
tags:
|
||||
- meeting
|
||||
- druckluft
|
||||
- customer
|
||||
people:
|
||||
- André Knierim
|
||||
- Thomas Beier
|
||||
projects:
|
||||
- Beier GmbH
|
||||
- Druckluft
|
||||
source:
|
||||
file: "Beier GmbH.pdf"
|
||||
imported: "2024-03-20T14:30:00"
|
||||
---
|
||||
|
||||
# Meeting with Beier GmbH about Druckluft project
|
||||
|
||||
Discussion about the [[projects/druckluft]] project with [[people/thomas-beier]].
|
||||
|
||||
Key points:
|
||||
- Delivery timeline confirmed for Q2
|
||||
- Budget approved at €45,000
|
||||
- Next review meeting scheduled for April 10
|
||||
|
||||
## Action Items
|
||||
|
||||
- [ ] Send updated proposal to Thomas Beier (deadline: 2024-03-22)
|
||||
- [ ] Schedule follow-up meeting for April 10
|
||||
- [ ] Update project timeline in planning tool
|
||||
```
|
||||
|
||||
### Entity-to-WikiLink Mapping
|
||||
|
||||
| Entity Type | Wiki-Link Format | Example |
|
||||
|-------------|-----------------|---------|
|
||||
| Person | `[[people/firstname-lastname]]` | `[[people/thomas-beier]]` |
|
||||
| Project | `[[projects/project-slug]]` | `[[projects/druckluft]]` |
|
||||
| Date | Frontmatter `date` field | `2024-03-15` |
|
||||
| Action Item | `## Action Items` section | `- [ ] Send proposal` |
|
||||
|
||||
### File Routing Rules
|
||||
|
||||
| Category | Target Folder | Condition |
|
||||
|----------|--------------|-----------|
|
||||
| Meeting | `notes/meetings/` | Category = "meeting" |
|
||||
| Project | `notes/projects/` | Category = "project" with single dominant project |
|
||||
| Decision | `notes/decisions/` | Category = "decision" |
|
||||
| Default | `notes/inbox/` | No clear category or mixed |
|
||||
|
||||
## Correctness Properties
|
||||
|
||||
*A property is a characteristic or behavior that should hold true across all valid executions of a system — essentially, a formal statement about what the system should do. Properties serve as the bridge between human-readable specifications and machine-verifiable correctness guarantees.*
|
||||
|
||||
### Property 1: File discovery returns only supported extensions
|
||||
|
||||
*For any* directory tree containing files with arbitrary extensions, the discovery function SHALL return exactly and only those files whose extensions are in the supported set (`.pdf`, `.jpg`, `.jpeg`, `.png`, `.md`, `.txt`, `.docx`).
|
||||
|
||||
**Validates: Requirements 1.1, 1.6**
|
||||
|
||||
### Property 2: Bulk import resilience — failed files do not block remaining processing
|
||||
|
||||
*For any* batch of N files where a subset F fails during processing, the pipeline SHALL successfully process all N-F non-failing files and the summary SHALL report success count = N-F and failed count = |F|.
|
||||
|
||||
**Validates: Requirements 1.4, 1.5**
|
||||
|
||||
### Property 3: Text extraction preserves content for text-based formats
|
||||
|
||||
*For any* valid UTF-8 string written to a markdown or plain text file, extracting it via the Text_Extractor SHALL return the identical string content.
|
||||
|
||||
**Validates: Requirements 3.1, 3.2, 3.4**
|
||||
|
||||
### Property 4: Entity parsing extracts all entities with valid structure
|
||||
|
||||
*For any* valid enrichment response JSON containing entities, each parsed entity SHALL have a non-empty `type` (one of person/project/date/action_item), a non-empty `value`, and a `confidence` score in the range [0.0, 1.0].
|
||||
|
||||
**Validates: Requirements 4.1, 4.2, 4.3, 4.4, 4.5**
|
||||
|
||||
### Property 5: Confidence threshold filtering
|
||||
|
||||
*For any* list of entities with arbitrary confidence scores, the filtered output SHALL contain exactly those entities with confidence >= 0.7 and no others.
|
||||
|
||||
**Validates: Requirements 4.6**
|
||||
|
||||
### Property 6: Wiki-link generation follows correct format per entity type
|
||||
|
||||
*For any* person name, the generated wiki-link SHALL match the pattern `[[people/slugified-name]]`. *For any* project name, the generated wiki-link SHALL match the pattern `[[projects/slugified-name]]`.
|
||||
|
||||
**Validates: Requirements 5.1, 5.2**
|
||||
|
||||
### Property 7: Only first entity occurrence is linked
|
||||
|
||||
*For any* text containing N >= 2 occurrences of the same entity, the Reference_Linker SHALL convert exactly the first occurrence to a wiki-link and leave all subsequent occurrences as plain text.
|
||||
|
||||
**Validates: Requirements 5.6**
|
||||
|
||||
### Property 8: Frontmatter round-trip validity
|
||||
|
||||
*For any* valid `EnrichmentResult`, rendering it to markdown with YAML frontmatter and then parsing the frontmatter back SHALL produce a valid YAML block containing all required fields (title, date, tags, people, projects, source).
|
||||
|
||||
**Validates: Requirements 6.1, 6.3, 6.4, 6.6, 6.7**
|
||||
|
||||
### Property 9: Provider routing from configuration
|
||||
|
||||
*For any* valid provider name from the supported set (openai, anthropic, google, mistral, ollama), configuring `LLM_PROVIDER` to that value SHALL cause the LLM client to route requests to the corresponding provider.
|
||||
|
||||
**Validates: Requirements 7.3, 8.1**
|
||||
|
||||
### Property 10: Missing API key produces descriptive error
|
||||
|
||||
*For any* provider that requires an API key, when the corresponding environment variable is unset, the configuration validation SHALL raise an error that names the missing variable.
|
||||
|
||||
**Validates: Requirements 7.4, 8.6**
|
||||
|
||||
### Property 11: Action items appear in output markdown section
|
||||
|
||||
*For any* list of extracted action items, the rendered markdown SHALL contain a `## Action Items` section listing every action item description.
|
||||
|
||||
**Validates: Requirements 9.2**
|
||||
|
||||
### Property 12: OrgMyLife task payload correctness
|
||||
|
||||
*For any* action item description and note file path, the task creation payload SHALL set `title` to the action item description and `source_url` to a URL referencing the note path.
|
||||
|
||||
**Validates: Requirements 9.4**
|
||||
|
||||
### Property 13: Category-based folder routing
|
||||
|
||||
*For any* enrichment result with category "meeting", the output path SHALL be within `notes/meetings/`. *For any* result with category "project", the path SHALL be within `notes/projects/`. *For any* result with no clear category, the path SHALL be within `notes/inbox/`.
|
||||
|
||||
**Validates: Requirements 10.1, 10.2, 10.3**
|
||||
|
||||
### Property 14: Output filename follows date-slug pattern
|
||||
|
||||
*For any* date and title string, the generated filename SHALL match the pattern `YYYY-MM-DD-slugified-title.md` where the slug contains only lowercase alphanumeric characters and hyphens.
|
||||
|
||||
**Validates: Requirements 10.5**
|
||||
|
||||
### Property 15: Filename collision avoidance
|
||||
|
||||
*For any* set of existing files, when a new file would have the same name as an existing file, the system SHALL append a numeric suffix (`-2`, `-3`, etc.) producing a unique filename.
|
||||
|
||||
**Validates: Requirements 10.6**
|
||||
|
||||
### Property 16: Temporary file filtering
|
||||
|
||||
*For any* filename, the watcher ignore logic SHALL return true if and only if the filename starts with `.` or ends with `.tmp`.
|
||||
|
||||
**Validates: Requirements 11.5**
|
||||
|
||||
### Property 17: Dry-run produces no file writes
|
||||
|
||||
*For any* input processed with the `--dry-run` flag, the system SHALL create zero files on disk.
|
||||
|
||||
**Validates: Requirements 13.6**
|
||||
|
||||
### Property 18: Git commit message format
|
||||
|
||||
*For any* count N and source type string, the generated commit message SHALL match the format `ingestion: import N notes from [source-type]`.
|
||||
|
||||
**Validates: Requirements 12.2**
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Error Categories and Strategies
|
||||
|
||||
| Error Type | Strategy | User Impact |
|
||||
|-----------|----------|-------------|
|
||||
| File read failure | Log + skip file, continue batch | Warning in summary |
|
||||
| OCR failure (no text) | Create note with empty body + source ref | Note created, flagged |
|
||||
| Encoding detection failure | Log + skip file | Warning in summary |
|
||||
| LLM API error | Retry 3x with exponential backoff, then fail file | Warning per file |
|
||||
| LLM response parse error | Retry once with stricter prompt, then use defaults | Degraded metadata |
|
||||
| OrgMyLife API unreachable | Log warning, continue without task creation | Tasks not created |
|
||||
| Git commit failure | Log warning, files still written | No version history |
|
||||
| Disk full / write failure | Fail immediately with clear error | Process stops |
|
||||
| Invalid config (missing key) | Exit with descriptive error before processing | Immediate feedback |
|
||||
|
||||
### Retry Policy
|
||||
|
||||
```python
|
||||
RETRY_CONFIG = {
|
||||
"max_retries": 3,
|
||||
"base_delay_seconds": 1.0,
|
||||
"max_delay_seconds": 30.0,
|
||||
"backoff_factor": 2.0,
|
||||
"retryable_errors": [
|
||||
"rate_limit",
|
||||
"timeout",
|
||||
"server_error", # 5xx
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Logging
|
||||
|
||||
- **Level INFO**: File processing progress, summary counts
|
||||
- **Level WARNING**: Skipped files, API failures (non-fatal), git failures
|
||||
- **Level ERROR**: Configuration errors, unrecoverable failures
|
||||
- **Level DEBUG**: LLM prompts/responses, entity detection details
|
||||
|
||||
All logs include timestamps and structured context (file path, provider, operation).
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Property-Based Tests (Hypothesis)
|
||||
|
||||
The project uses **Hypothesis** (Python's standard PBT library) for property-based testing. Each property test runs a minimum of 100 iterations.
|
||||
|
||||
Property tests cover the core transformation logic:
|
||||
- File discovery filtering (Property 1)
|
||||
- Text extraction identity (Property 3)
|
||||
- Entity parsing and filtering (Properties 4, 5)
|
||||
- Wiki-link generation (Properties 6, 7)
|
||||
- Frontmatter round-trip (Property 8)
|
||||
- Folder routing (Property 13)
|
||||
- Filename generation and collision avoidance (Properties 14, 15)
|
||||
- Temporary file filtering (Property 16)
|
||||
|
||||
Each property test is tagged with:
|
||||
```python
|
||||
# Feature: notegraph-ingestion, Property N: [property text]
|
||||
```
|
||||
|
||||
Configuration: minimum 100 examples per test via `@settings(max_examples=100)`.
|
||||
|
||||
### Unit Tests (pytest)
|
||||
|
||||
Example-based tests for:
|
||||
- CLI argument parsing and subcommand routing
|
||||
- Configuration loading from environment variables
|
||||
- Specific enrichment response scenarios
|
||||
- OrgMyLife client request formatting
|
||||
- Git commit message generation
|
||||
- Dry-run mode behavior
|
||||
|
||||
### Integration Tests
|
||||
|
||||
Integration tests (marked with `@pytest.mark.integration`) for:
|
||||
- OCR extraction with sample images
|
||||
- PDF text extraction with sample PDFs
|
||||
- DOCX extraction with sample files
|
||||
- LLM API calls (with real provider, gated by env var)
|
||||
- OrgMyLife API task creation (against test instance)
|
||||
- Drop-folder watcher file detection
|
||||
- Git operations in a test repository
|
||||
|
||||
### Test Dependencies
|
||||
|
||||
```toml
|
||||
[project.optional-dependencies]
|
||||
dev = [
|
||||
"pytest>=8.0",
|
||||
"pytest-asyncio>=0.23",
|
||||
"hypothesis>=6.100",
|
||||
"pytest-mock>=3.12",
|
||||
"respx>=0.21", # httpx mocking
|
||||
]
|
||||
```
|
||||
@@ -0,0 +1,195 @@
|
||||
# Requirements Document
|
||||
|
||||
## Introduction
|
||||
|
||||
The NoteGraph Ingestion & Auto-Reference feature adds a document import pipeline and AI-powered entity linking system to NoteGraph. It enables bulk import of PDF files (from reMarkable tablet), images (photos of sticky notes, whiteboards, handwritten notes), and legacy text files (from OneNote, Android Notes, Obsidian, scattered file systems). Imported content is OCR-processed, enriched by an LLM to detect entities (people, projects, dates, TODOs), and output as structured markdown with wiki-links and frontmatter into the appropriate NoteGraph folder. The system uses a model-agnostic LLM abstraction layer supporting multiple providers via simple `.env` configuration.
|
||||
|
||||
## Glossary
|
||||
|
||||
- **Ingestion_Service**: The Python service responsible for orchestrating the full import pipeline from file input through OCR, enrichment, and markdown output.
|
||||
- **OCR_Engine**: The component that extracts raw text from image files (JPG, PNG) and scanned PDF pages using optical character recognition.
|
||||
- **Text_Extractor**: The component that extracts text content from text-based PDFs and legacy file formats (markdown, plain text, OneNote exports).
|
||||
- **LLM_Provider**: An abstraction layer that routes enrichment requests to a configured language model provider (OpenAI, Anthropic, Google, Mistral, Ollama/Gemma) without coupling to any specific API.
|
||||
- **Enrichment_Agent**: The AI-powered component that processes raw extracted text to detect entities, generate metadata, create wiki-links, and structure the output as markdown.
|
||||
- **Entity_Detector**: The sub-component of the Enrichment_Agent responsible for identifying person names, project references, dates, and action items in extracted text.
|
||||
- **Reference_Linker**: The sub-component that converts detected entities into SilverBullet-compatible wiki-links (e.g., `[[people/name]]`, `[[projects/name]]`).
|
||||
- **Drop_Folder_Watcher**: A background process that monitors a designated inbox directory for new files and triggers the Ingestion_Service automatically.
|
||||
- **Inbox_Folder**: The designated directory where files are placed for automatic ingestion (default: `NoteGraph/ingestion/inbox/`).
|
||||
- **Output_Folder**: The target directory within `NoteGraph/notes/` where processed markdown files are placed.
|
||||
- **Provider_Config**: The `.env`-based configuration that specifies which LLM_Provider to use and supplies the corresponding API keys.
|
||||
- **OrgMyLife_Client**: The HTTP client component that creates tasks in the OrgMyLife API from extracted action items.
|
||||
- **Frontmatter**: YAML metadata block at the top of a markdown file containing structured fields (title, date, tags, people, projects).
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement 1: Bulk File Import via CLI
|
||||
|
||||
**User Story:** As a user, I want to import many files at once from a folder using a CLI command, so that I can quickly ingest large collections of notes from various sources.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. WHEN the user runs the CLI import command with a directory path, THE Ingestion_Service SHALL recursively discover all supported files (PDF, JPG, PNG, MD, TXT) in that directory.
|
||||
2. WHEN the user runs the CLI import command with a single file path, THE Ingestion_Service SHALL process that individual file.
|
||||
3. WHEN processing multiple files, THE Ingestion_Service SHALL report progress showing the current file number, total count, and file name.
|
||||
4. WHEN a file fails to process during bulk import, THE Ingestion_Service SHALL log the error with the file path and reason, skip the file, and continue processing remaining files.
|
||||
5. WHEN bulk import completes, THE Ingestion_Service SHALL output a summary showing total files processed, successful count, and failed count.
|
||||
6. THE Ingestion_Service SHALL support the following file extensions: `.pdf`, `.jpg`, `.jpeg`, `.png`, `.md`, `.txt`, `.docx`.
|
||||
|
||||
### Requirement 2: OCR and Text Extraction from Images
|
||||
|
||||
**User Story:** As a user, I want photos of handwritten notes and whiteboards converted to text, so that I can search and reference them in my knowledge base.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. WHEN a JPG or PNG file is provided, THE OCR_Engine SHALL extract text content from the image and return it as a UTF-8 string.
|
||||
2. WHEN a PDF file contains scanned pages (image-based content without embedded text), THE OCR_Engine SHALL apply OCR to extract text from those pages.
|
||||
3. WHEN a PDF file contains embedded text, THE Text_Extractor SHALL extract the text directly without OCR.
|
||||
4. WHEN a PDF file contains a mix of text pages and scanned pages, THE Ingestion_Service SHALL apply the appropriate extraction method per page and concatenate results.
|
||||
5. IF the OCR_Engine cannot extract any readable text from an image, THEN THE Ingestion_Service SHALL create the output note with an empty body and attach the original file reference in the frontmatter.
|
||||
6. WHEN processing handwritten content, THE OCR_Engine SHALL preserve paragraph structure by detecting line breaks and groupings.
|
||||
|
||||
### Requirement 3: Legacy File Format Import
|
||||
|
||||
**User Story:** As a user, I want to import notes from old tools like OneNote, Obsidian, and scattered text files, so that I can consolidate all my knowledge in one place.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. WHEN a markdown file is provided, THE Text_Extractor SHALL read the file content preserving existing formatting, links, and structure.
|
||||
2. WHEN a plain text file is provided, THE Text_Extractor SHALL read the file content as UTF-8 text.
|
||||
3. WHEN a DOCX file is provided, THE Text_Extractor SHALL extract the text content preserving heading structure and paragraph breaks.
|
||||
4. WHEN an imported markdown file contains Obsidian-style wiki-links (`[[page name]]`), THE Text_Extractor SHALL preserve them for later resolution by the Reference_Linker.
|
||||
5. WHEN a file uses a non-UTF-8 encoding, THE Text_Extractor SHALL attempt to detect the encoding and convert to UTF-8.
|
||||
6. IF a file cannot be decoded to text, THEN THE Ingestion_Service SHALL log a warning and skip the file.
|
||||
|
||||
### Requirement 4: AI-Powered Entity Detection
|
||||
|
||||
**User Story:** As a user, I want the system to automatically detect people, projects, dates, and action items in my notes, so that I do not have to manually tag and link everything.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. WHEN extracted text is provided to the Enrichment_Agent, THE Entity_Detector SHALL identify person names mentioned in the text.
|
||||
2. WHEN extracted text is provided to the Enrichment_Agent, THE Entity_Detector SHALL identify project references mentioned in the text.
|
||||
3. WHEN extracted text is provided to the Enrichment_Agent, THE Entity_Detector SHALL identify dates and temporal references in the text and normalize them to ISO 8601 format.
|
||||
4. WHEN extracted text is provided to the Enrichment_Agent, THE Entity_Detector SHALL identify action items and TODO statements in the text.
|
||||
5. WHEN the Entity_Detector identifies entities, THE Entity_Detector SHALL return each entity with its type, extracted value, and confidence score between 0.0 and 1.0.
|
||||
6. THE Entity_Detector SHALL only include entities with a confidence score of 0.7 or higher in the final output.
|
||||
|
||||
### Requirement 5: Auto-Reference Wiki-Link Generation
|
||||
|
||||
**User Story:** As a user, I want detected entities automatically converted to wiki-links, so that my imported notes are immediately connected to my knowledge graph.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. WHEN a person entity is detected, THE Reference_Linker SHALL generate a wiki-link in the format `[[people/firstname-lastname]]`.
|
||||
2. WHEN a project entity is detected, THE Reference_Linker SHALL generate a wiki-link in the format `[[projects/project-name]]`.
|
||||
3. WHEN a date entity is detected, THE Reference_Linker SHALL add the normalized date to the frontmatter `date` field.
|
||||
4. WHEN the Reference_Linker generates a wiki-link for a person or project that does not yet exist in NoteGraph, THE Reference_Linker SHALL create a stub page with the entity name as title.
|
||||
5. WHEN the Reference_Linker generates a wiki-link, THE Reference_Linker SHALL insert the link inline in the note body at the position where the entity was detected.
|
||||
6. WHEN multiple references to the same entity exist in one note, THE Reference_Linker SHALL link only the first occurrence and leave subsequent mentions as plain text.
|
||||
|
||||
### Requirement 6: Structured Markdown Output with Frontmatter
|
||||
|
||||
**User Story:** As a user, I want imported notes output as structured markdown with proper metadata, so that they integrate seamlessly with SilverBullet's features.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE Ingestion_Service SHALL output each processed file as a markdown file with YAML frontmatter containing: title, date, tags, people, projects, and source fields.
|
||||
2. WHEN the Enrichment_Agent generates a title from the content, THE Ingestion_Service SHALL use it as the frontmatter `title` field.
|
||||
3. WHEN entities are detected, THE Ingestion_Service SHALL populate the frontmatter `people` field as a list of detected person names.
|
||||
4. WHEN entities are detected, THE Ingestion_Service SHALL populate the frontmatter `projects` field as a list of detected project names.
|
||||
5. WHEN entities are detected, THE Ingestion_Service SHALL populate the frontmatter `tags` field with relevant topic tags derived from the content.
|
||||
6. THE Ingestion_Service SHALL include a `source` field in the frontmatter containing the original file name and import timestamp.
|
||||
7. FOR ALL valid extracted text inputs, processing through the Enrichment_Agent and then rendering to markdown and then parsing the frontmatter SHALL produce a valid YAML frontmatter block (round-trip property).
|
||||
|
||||
### Requirement 7: Model-Agnostic LLM Provider Layer
|
||||
|
||||
**User Story:** As a user, I want to easily switch between LLM providers and try different models, so that I can compare behavior and avoid vendor lock-in.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE LLM_Provider SHALL expose a unified interface for text completion requests regardless of the underlying provider.
|
||||
2. THE LLM_Provider SHALL support at minimum the following providers: OpenAI, Anthropic, Google Gemini, Mistral, Gemma (local), and Ollama (local).
|
||||
3. WHEN a provider is specified in the Provider_Config, THE LLM_Provider SHALL route requests to that provider using the configured API key.
|
||||
4. WHEN the configured provider is unavailable or returns an error, THE LLM_Provider SHALL raise a descriptive exception including the provider name and error details.
|
||||
5. THE LLM_Provider SHALL read provider selection and API keys from environment variables defined in a `.env` file.
|
||||
6. THE LLM_Provider SHALL support configuring the model name per provider (e.g., `gpt-4o`, `claude-sonnet-4-20250514`, `gemini-pro`).
|
||||
7. WHEN switching providers, THE LLM_Provider SHALL require only a change to environment variables without code modifications.
|
||||
|
||||
### Requirement 8: Provider Configuration via Environment Variables
|
||||
|
||||
**User Story:** As a user, I want simple `.env`-based configuration for LLM providers, so that I can set up and switch providers without editing code.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE Provider_Config SHALL use the environment variable `LLM_PROVIDER` to select the active provider (values: `openai`, `anthropic`, `google`, `mistral`, `ollama`).
|
||||
2. THE Provider_Config SHALL use the environment variable `LLM_MODEL` to specify the model name for the active provider.
|
||||
3. THE Provider_Config SHALL use provider-specific environment variables for API keys: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GOOGLE_API_KEY`, `MISTRAL_API_KEY`.
|
||||
4. WHERE the Ollama provider is selected, THE Provider_Config SHALL use the environment variable `OLLAMA_BASE_URL` to specify the local Ollama endpoint (default: `http://localhost:11434`). Ollama SHALL support running Google Gemma models locally.
|
||||
5. THE Provider_Config SHALL NOT store API keys or secrets directly in git-tracked files. The `.env` file SHALL be listed in `.gitignore` and a `.env.example` file SHALL document all variables without actual secret values.
|
||||
6. WHEN a required API key is missing for the selected provider, THE Ingestion_Service SHALL exit with a clear error message naming the missing variable.
|
||||
7. THE Ingestion_Service SHALL provide a `.env.example` file documenting all supported environment variables with descriptions.
|
||||
|
||||
### Requirement 9: TODO Extraction and OrgMyLife Integration
|
||||
|
||||
**User Story:** As a user, I want action items extracted from my notes and optionally created as tasks in OrgMyLife, so that I do not lose track of commitments found in imported notes.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. WHEN the Entity_Detector identifies action items in the text, THE Enrichment_Agent SHALL extract each action item with its description and any associated person or deadline.
|
||||
2. WHEN action items are extracted, THE Ingestion_Service SHALL list them in a `## Action Items` section at the end of the output markdown.
|
||||
3. WHEN the `--create-tasks` flag is provided on the CLI, THE OrgMyLife_Client SHALL create a task in OrgMyLife for each extracted action item via POST to `/api/tasks`.
|
||||
4. WHEN creating a task in OrgMyLife, THE OrgMyLife_Client SHALL set the task title to the action item description and include a `source_url` linking back to the imported note.
|
||||
5. IF the OrgMyLife API is unreachable when `--create-tasks` is specified, THEN THE Ingestion_Service SHALL log a warning and continue processing without creating tasks.
|
||||
6. THE Provider_Config SHALL use the environment variable `ORGMYLIFE_API_URL` for the OrgMyLife endpoint and `ORGMYLIFE_API_KEY` for authentication.
|
||||
|
||||
### Requirement 10: File Placement and Folder Routing
|
||||
|
||||
**User Story:** As a user, I want imported notes automatically placed in the correct NoteGraph folder based on their content, so that my knowledge base stays organized.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. WHEN the Enrichment_Agent detects meeting-related content, THE Ingestion_Service SHALL place the output file in `NoteGraph/notes/meetings/`.
|
||||
2. WHEN the Enrichment_Agent detects project-specific content with a single dominant project, THE Ingestion_Service SHALL place the output file in `NoteGraph/notes/projects/`.
|
||||
3. WHEN the Enrichment_Agent cannot determine a specific category, THE Ingestion_Service SHALL place the output file in `NoteGraph/notes/inbox/`.
|
||||
4. WHEN the `--output-dir` flag is provided on the CLI, THE Ingestion_Service SHALL place all output files in the specified directory regardless of content analysis.
|
||||
5. THE Ingestion_Service SHALL generate output file names using the pattern `YYYY-MM-DD-slugified-title.md` based on the detected or current date and the generated title.
|
||||
6. IF an output file with the same name already exists, THEN THE Ingestion_Service SHALL append a numeric suffix (e.g., `-2`, `-3`) to avoid overwriting.
|
||||
|
||||
### Requirement 11: Drop-Folder Watcher for Continuous Import
|
||||
|
||||
**User Story:** As a user, I want to drop files into a folder and have them automatically processed, so that I can import notes without running CLI commands.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. WHEN the watcher mode is started, THE Drop_Folder_Watcher SHALL monitor the Inbox_Folder for new files.
|
||||
2. WHEN a new file appears in the Inbox_Folder, THE Drop_Folder_Watcher SHALL trigger the Ingestion_Service to process that file.
|
||||
3. WHEN a file is successfully processed, THE Drop_Folder_Watcher SHALL move the original file to an `archive/` subdirectory within the Inbox_Folder.
|
||||
4. IF a file fails to process, THEN THE Drop_Folder_Watcher SHALL move the file to a `failed/` subdirectory and log the error.
|
||||
5. THE Drop_Folder_Watcher SHALL ignore temporary files (names starting with `.` or ending with `.tmp`).
|
||||
6. WHEN the watcher mode is started, THE Drop_Folder_Watcher SHALL process any files already present in the Inbox_Folder before entering watch mode.
|
||||
|
||||
### Requirement 12: Git Auto-Commit for Imported Notes
|
||||
|
||||
**User Story:** As a user, I want imported notes automatically committed to git, so that my knowledge base history is preserved without manual intervention.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. WHEN one or more files are successfully imported, THE Ingestion_Service SHALL stage the new markdown files and commit them to the NoteGraph git repository.
|
||||
2. THE Ingestion_Service SHALL use the commit message format: `ingestion: import N notes from [source-type]` where N is the count and source-type describes the input (e.g., "pdf", "images", "bulk").
|
||||
3. WHEN stub pages are created by the Reference_Linker, THE Ingestion_Service SHALL include them in the same git commit.
|
||||
4. WHEN the `--no-commit` flag is provided on the CLI, THE Ingestion_Service SHALL skip the git commit step.
|
||||
5. IF the git commit fails, THEN THE Ingestion_Service SHALL log a warning but not fail the overall import process.
|
||||
|
||||
### Requirement 13: CLI Interface Design
|
||||
|
||||
**User Story:** As a user, I want a clear and consistent CLI interface, so that I can easily run imports with different options.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE Ingestion_Service SHALL provide a CLI entry point via `python -m ingestion.cli`.
|
||||
2. THE Ingestion_Service SHALL support the subcommand `import` accepting a path argument (file or directory).
|
||||
3. THE Ingestion_Service SHALL support the subcommand `watch` to start the Drop_Folder_Watcher.
|
||||
4. THE Ingestion_Service SHALL support the following global flags: `--verbose` for detailed logging, `--dry-run` to preview without writing files, `--no-commit` to skip git commits.
|
||||
5. THE Ingestion_Service SHALL support the following import flags: `--output-dir` to override output location, `--create-tasks` to enable OrgMyLife task creation, `--provider` to override the LLM provider for this run.
|
||||
6. WHEN the `--dry-run` flag is provided, THE Ingestion_Service SHALL display what would be created (file paths, detected entities, proposed links) without writing any files.
|
||||
7. WHEN invoked without arguments, THE Ingestion_Service SHALL display usage help with examples.
|
||||
@@ -0,0 +1,318 @@
|
||||
# Implementation Plan: NoteGraph Ingestion & Auto-Reference
|
||||
|
||||
## Overview
|
||||
|
||||
This plan implements the NoteGraph ingestion pipeline as a Python package at `NoteGraph/ingestion/`. Tasks are ordered so each builds on the previous: scaffolding → extraction → LLM → enrichment → linking → output → CLI → web upload → Nextcloud sync → OrgMyLife → git → Docker/deploy → tests. Property-based tests use Hypothesis and are placed close to the code they validate.
|
||||
|
||||
## Tasks
|
||||
|
||||
- [x] 1. Project scaffolding and configuration
|
||||
- [x] 1.1 Create `NoteGraph/ingestion/` package structure with `__init__.py`, `__main__.py`, and all subpackages (`extraction/`, `enrichment/`, `linking/`, `output/`, `integrations/`, `tests/`)
|
||||
- Create directory tree matching the design's package structure
|
||||
- Add empty `__init__.py` files in each subpackage
|
||||
- Create `__main__.py` with `from ingestion.cli import cli; cli()` entry point
|
||||
- _Requirements: 13.1_
|
||||
|
||||
- [x] 1.2 Create `NoteGraph/ingestion/pyproject.toml` with all dependencies
|
||||
- Include: click, watchdog, pytesseract, PyPDF2, pdfplumber, python-docx, litellm, httpx, pydantic, pydantic-settings, fastapi, uvicorn, python-multipart
|
||||
- Add dev dependencies: pytest, pytest-asyncio, hypothesis, pytest-mock, respx
|
||||
- Configure package metadata and entry points
|
||||
- _Requirements: 7.1, 8.7_
|
||||
|
||||
- [x] 1.3 Implement `NoteGraph/ingestion/config.py` with `IngestionConfig` (pydantic-settings)
|
||||
- Define all env vars: `LLM_PROVIDER`, `LLM_MODEL`, provider API keys, `ORGMYLIFE_API_URL`, `ORGMYLIFE_API_KEY`, `NEXTCLOUD_INBOX_URL`, `NEXTCLOUD_INBOX_USER`, `NEXTCLOUD_INBOX_PASSWORD`, `SB_USER`, paths, OCR settings, confidence threshold
|
||||
- Implement `llm_model_string` property for LiteLLM routing
|
||||
- Implement `validate_api_key()` that exits with clear error naming the missing variable
|
||||
- _Requirements: 8.1, 8.2, 8.3, 8.4, 8.5, 8.6_
|
||||
|
||||
- [x] 1.4 Create `NoteGraph/.env.example` with all supported environment variables and descriptions
|
||||
- Document every variable with comments explaining purpose and valid values
|
||||
- Include placeholders for: LLM keys, OrgMyLife, Nextcloud inbox, SB_USER
|
||||
- Ensure `.env` is in `.gitignore`
|
||||
- _Requirements: 8.5, 8.7_
|
||||
|
||||
- [x] 2. Text extraction layer
|
||||
- [x] 2.1 Implement `extraction/base.py` with `TextExtractor` protocol and `ExtractionResult` dataclass
|
||||
- Define `can_handle(file_path: Path) -> bool` and `extract(file_path: Path) -> ExtractionResult`
|
||||
- `ExtractionResult` fields: text, source_file, extraction_method, metadata
|
||||
- _Requirements: 2.1, 3.1_
|
||||
|
||||
- [x] 2.2 Implement `extraction/text.py` for plain text and markdown files
|
||||
- Handle `.md` and `.txt` extensions
|
||||
- Read as UTF-8, attempt encoding detection for non-UTF-8 files (chardet/charset-normalizer)
|
||||
- Preserve existing formatting, links, and Obsidian-style wiki-links
|
||||
- _Requirements: 3.1, 3.2, 3.4, 3.5, 3.6_
|
||||
|
||||
- [x] 2.3 Implement `extraction/pdf.py` for PDF text extraction
|
||||
- Use pdfplumber for text-based PDFs
|
||||
- Detect scanned pages (pages with no extractable text) and delegate to OCR
|
||||
- Handle mixed PDFs (text + scanned pages) by concatenating results
|
||||
- _Requirements: 2.2, 2.3, 2.4_
|
||||
|
||||
- [x] 2.4 Implement `extraction/ocr.py` for image OCR via pytesseract
|
||||
- Handle `.jpg`, `.jpeg`, `.png` files
|
||||
- Configure language as `deu+eng` (German + English)
|
||||
- Preserve paragraph structure by detecting line breaks
|
||||
- Return empty text (not error) when OCR cannot extract readable content
|
||||
- _Requirements: 2.1, 2.2, 2.5, 2.6_
|
||||
|
||||
- [x] 2.5 Implement `extraction/docx.py` for DOCX files
|
||||
- Use python-docx to extract text preserving heading structure and paragraph breaks
|
||||
- Handle `.docx` extension
|
||||
- _Requirements: 3.3_
|
||||
|
||||
- [ ]* 2.6 Write property test for text extraction identity (Property 3)
|
||||
- **Property 3: Text extraction preserves content for text-based formats**
|
||||
- For any valid UTF-8 string written to a .md or .txt file, extraction returns identical content
|
||||
- **Validates: Requirements 3.1, 3.2, 3.4**
|
||||
|
||||
- [x] 3. File discovery
|
||||
- [x] 3.1 Implement `discovery.py` with recursive file discovery and extension filtering
|
||||
- Support extensions: `.pdf`, `.jpg`, `.jpeg`, `.png`, `.md`, `.txt`, `.docx`
|
||||
- Accept both single file path and directory path
|
||||
- Recursive directory traversal
|
||||
- _Requirements: 1.1, 1.2, 1.6_
|
||||
|
||||
- [ ]* 3.2 Write property test for file discovery filtering (Property 1)
|
||||
- **Property 1: File discovery returns only supported extensions**
|
||||
- For any directory tree with arbitrary extensions, discovery returns exactly those with supported extensions
|
||||
- **Validates: Requirements 1.1, 1.6**
|
||||
|
||||
- [ ] 4. LLM provider layer
|
||||
- [x] 4.1 Implement `integrations/llm.py` with LiteLLM wrapper
|
||||
- `LLMClient` class with `complete(messages, response_format)` method
|
||||
- Route to provider based on `config.llm_model_string` (e.g., `openai/gpt-4o`, `anthropic/claude-sonnet-4-20250514`)
|
||||
- Implement retry logic: 3 retries with exponential backoff for rate limits, timeouts, 5xx errors
|
||||
- Raise descriptive exception on failure including provider name and error details
|
||||
- _Requirements: 7.1, 7.2, 7.3, 7.4, 7.5, 7.6, 7.7_
|
||||
|
||||
- [ ]* 4.2 Write property test for provider routing (Property 9)
|
||||
- **Property 9: Provider routing from configuration**
|
||||
- For any valid provider name, configuring `LLM_PROVIDER` causes the LLM client to route to the corresponding provider
|
||||
- **Validates: Requirements 7.3, 8.1**
|
||||
|
||||
- [ ]* 4.3 Write property test for missing API key error (Property 10)
|
||||
- **Property 10: Missing API key produces descriptive error**
|
||||
- For any provider requiring an API key, when the env var is unset, validation raises an error naming the missing variable
|
||||
- **Validates: Requirements 7.4, 8.6**
|
||||
|
||||
- [x] 5. Enrichment agent
|
||||
- [x] 5.1 Implement `enrichment/models.py` with Pydantic models
|
||||
- `Entity` model: type (person/project/date/action_item), value, confidence (0.0-1.0), position
|
||||
- `ActionItem` model: description, assignee, deadline (ISO 8601)
|
||||
- `EnrichmentResult` model: title, category, tags, entities, action_items, summary
|
||||
- _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5_
|
||||
|
||||
- [x] 5.2 Implement `enrichment/prompts.py` with structured LLM prompts
|
||||
- System prompt requesting structured JSON output for entity detection
|
||||
- Include context about known projects and people (read from existing NoteGraph pages)
|
||||
- Request confidence scores for each entity
|
||||
- _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5_
|
||||
|
||||
- [x] 5.3 Implement `enrichment/agent.py` with enrichment orchestration
|
||||
- Call LLM with extracted text and structured prompt
|
||||
- Parse response into `EnrichmentResult`
|
||||
- Filter entities by confidence threshold (>= 0.7)
|
||||
- Handle LLM parse errors: retry once with stricter prompt, then use defaults
|
||||
- _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5, 4.6_
|
||||
|
||||
- [ ]* 5.4 Write property tests for entity parsing (Properties 4, 5)
|
||||
- **Property 4: Entity parsing extracts all entities with valid structure**
|
||||
- For any valid enrichment response JSON, each entity has valid type, non-empty value, confidence in [0.0, 1.0]
|
||||
- **Property 5: Confidence threshold filtering**
|
||||
- For any list of entities, filtered output contains exactly those with confidence >= 0.7
|
||||
- **Validates: Requirements 4.1, 4.2, 4.3, 4.4, 4.5, 4.6**
|
||||
|
||||
- [x] 6. Checkpoint - Ensure all tests pass
|
||||
- Ensure all tests pass, ask the user if questions arise.
|
||||
|
||||
- [x] 7. Reference linker
|
||||
- [x] 7.1 Implement `linking/linker.py` with wiki-link generation
|
||||
- Person entities → `[[people/firstname-lastname]]` (slugified)
|
||||
- Project entities → `[[projects/project-slug]]` (slugified)
|
||||
- Insert wiki-link inline at entity position (first occurrence only)
|
||||
- Leave subsequent mentions of the same entity as plain text
|
||||
- Add dates to frontmatter `date` field
|
||||
- _Requirements: 5.1, 5.2, 5.3, 5.5, 5.6_
|
||||
|
||||
- [x] 7.2 Implement `linking/stubs.py` for stub page creation
|
||||
- Check if target page exists in NoteGraph notes directory
|
||||
- Create stub page with entity name as title if it doesn't exist
|
||||
- Stub pages go in `notes/people/` or `notes/projects/` as appropriate
|
||||
- _Requirements: 5.4_
|
||||
|
||||
- [ ]* 7.3 Write property tests for wiki-link generation (Properties 6, 7)
|
||||
- **Property 6: Wiki-link generation follows correct format per entity type**
|
||||
- For any person name → `[[people/slugified-name]]`; for any project name → `[[projects/slugified-name]]`
|
||||
- **Property 7: Only first entity occurrence is linked**
|
||||
- For any text with N >= 2 occurrences of the same entity, only the first is converted to a wiki-link
|
||||
- **Validates: Requirements 5.1, 5.2, 5.6**
|
||||
|
||||
- [x] 8. Output renderer and folder routing
|
||||
- [x] 8.1 Implement `output/renderer.py` for markdown + YAML frontmatter rendering
|
||||
- Render frontmatter with: title, date, tags, people, projects, source (file + timestamp)
|
||||
- Render body with wiki-links inserted
|
||||
- Render `## Action Items` section at the end with extracted action items
|
||||
- _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5, 6.6, 9.2_
|
||||
|
||||
- [x] 8.2 Implement `output/router.py` for category-based folder routing
|
||||
- "meeting" → `notes/meetings/`
|
||||
- "project" → `notes/projects/`
|
||||
- "decision" → `notes/decisions/`
|
||||
- Default/inbox → `notes/inbox/`
|
||||
- Support `--output-dir` override
|
||||
- _Requirements: 10.1, 10.2, 10.3, 10.4_
|
||||
|
||||
- [x] 8.3 Implement `output/naming.py` for filename generation with collision avoidance
|
||||
- Pattern: `YYYY-MM-DD-slugified-title.md`
|
||||
- Slug: lowercase alphanumeric + hyphens only
|
||||
- Append numeric suffix (`-2`, `-3`) if file already exists
|
||||
- _Requirements: 10.5, 10.6_
|
||||
|
||||
- [ ]* 8.4 Write property tests for output (Properties 8, 11, 13, 14, 15)
|
||||
- **Property 8: Frontmatter round-trip validity**
|
||||
- For any valid EnrichmentResult, render to markdown then parse frontmatter → valid YAML with all required fields
|
||||
- **Property 11: Action items appear in output markdown section**
|
||||
- For any list of action items, rendered markdown contains `## Action Items` section listing every item
|
||||
- **Property 13: Category-based folder routing**
|
||||
- For any category, output path is within the correct folder
|
||||
- **Property 14: Output filename follows date-slug pattern**
|
||||
- For any date and title, filename matches `YYYY-MM-DD-slugified-title.md`
|
||||
- **Property 15: Filename collision avoidance**
|
||||
- When a file with the same name exists, a numeric suffix is appended
|
||||
- **Validates: Requirements 6.1, 6.3, 6.4, 6.6, 6.7, 9.2, 10.1, 10.2, 10.3, 10.5, 10.6**
|
||||
|
||||
- [x] 9. Pipeline orchestration
|
||||
- [x] 9.1 Implement `pipeline.py` with main processing pipeline
|
||||
- Orchestrate: discovery → extraction → enrichment → linking → rendering → file write → git commit
|
||||
- Track progress: current file number, total count, file name
|
||||
- Handle errors per file: log + skip + continue
|
||||
- Output summary: total processed, success count, failed count
|
||||
- Support `--dry-run` mode (display what would be created without writing)
|
||||
- _Requirements: 1.3, 1.4, 1.5, 13.6_
|
||||
|
||||
- [ ]* 9.2 Write property tests for pipeline resilience and dry-run (Properties 2, 17)
|
||||
- **Property 2: Bulk import resilience — failed files do not block remaining processing**
|
||||
- For any batch where a subset fails, remaining files are processed successfully
|
||||
- **Property 17: Dry-run produces no file writes**
|
||||
- For any input with `--dry-run`, zero files are created on disk
|
||||
- **Validates: Requirements 1.4, 1.5, 13.6**
|
||||
|
||||
- [x] 10. CLI interface
|
||||
- [x] 10.1 Implement `cli.py` with Click-based CLI
|
||||
- `import` subcommand: accepts path argument (file or directory)
|
||||
- `watch` subcommand: starts drop-folder watcher
|
||||
- Global flags: `--verbose`, `--dry-run`, `--no-commit`
|
||||
- Import flags: `--output-dir`, `--create-tasks`, `--provider`
|
||||
- Display usage help with examples when invoked without arguments
|
||||
- _Requirements: 13.1, 13.2, 13.3, 13.4, 13.5, 13.6, 13.7_
|
||||
|
||||
- [x] 11. Checkpoint - Ensure all tests pass
|
||||
- Ensure all tests pass, ask the user if questions arise.
|
||||
|
||||
- [ ] 12. Web upload endpoint (FastAPI)
|
||||
- [x] 12.1 Implement `web/upload.py` with FastAPI app for file upload
|
||||
- HTML form with file upload accepting PDF, JPG, PNG
|
||||
- Mobile-friendly responsive HTML page (inline CSS, no external dependencies)
|
||||
- Basic auth protection using `SB_USER` env var (same credentials as SilverBullet)
|
||||
- On upload: save file to inbox folder, trigger processing pipeline
|
||||
- Return success/error response to the user
|
||||
- _Requirements: 11.2 (extends with web input)_
|
||||
|
||||
- [x] 12.2 Create `web/templates/upload.html` mobile-friendly upload page
|
||||
- Simple, clean form with drag-and-drop support
|
||||
- Responsive design for Android "Share to URL" workflow
|
||||
- Show upload progress and result feedback
|
||||
- _Requirements: (additional requirement: web upload for mobile)_
|
||||
|
||||
- [ ] 13. Nextcloud WebDAV inbox sync
|
||||
- [x] 13.1 Implement `integrations/nextcloud.py` with WebDAV folder polling
|
||||
- Poll `NEXTCLOUD_INBOX_URL` for new files using httpx WebDAV (PROPFIND/GET)
|
||||
- Download new files to local inbox for processing
|
||||
- After successful processing, move files to a `processed/` subfolder on Nextcloud (MOVE request)
|
||||
- Auth via `NEXTCLOUD_INBOX_USER` and `NEXTCLOUD_INBOX_PASSWORD` env vars
|
||||
- _Requirements: (additional requirement: Nextcloud WebDAV sync)_
|
||||
|
||||
- [x] 13.2 Add `sync` subcommand to CLI for Nextcloud polling
|
||||
- One-shot sync: poll once, process new files, exit
|
||||
- Can be called from cron or systemd timer
|
||||
- _Requirements: (additional requirement: Nextcloud WebDAV sync)_
|
||||
|
||||
- [x] 14. Drop-folder watcher
|
||||
- [x] 14.1 Implement `watcher.py` with watchdog-based folder monitoring
|
||||
- Monitor inbox folder for new files
|
||||
- Trigger pipeline on new file detection
|
||||
- Move processed files to `archive/` subdirectory
|
||||
- Move failed files to `failed/` subdirectory with error log
|
||||
- Ignore temporary files (names starting with `.` or ending with `.tmp`)
|
||||
- Process existing files in inbox on startup before entering watch mode
|
||||
- _Requirements: 11.1, 11.2, 11.3, 11.4, 11.5, 11.6_
|
||||
|
||||
- [ ]* 14.2 Write property test for temporary file filtering (Property 16)
|
||||
- **Property 16: Temporary file filtering**
|
||||
- For any filename, ignore logic returns true iff name starts with `.` or ends with `.tmp`
|
||||
- **Validates: Requirements 11.5**
|
||||
|
||||
- [x] 15. OrgMyLife integration
|
||||
- [x] 15.1 Implement `integrations/orgmylife.py` with HTTP client for task creation
|
||||
- POST to `/api/tasks` with title, source_url, description
|
||||
- Auth via `ORGMYLIFE_API_KEY` header
|
||||
- Handle unreachable API gracefully: log warning, continue without creating tasks
|
||||
- _Requirements: 9.3, 9.4, 9.5, 9.6_
|
||||
|
||||
- [ ]* 15.2 Write property test for OrgMyLife task payload (Property 12)
|
||||
- **Property 12: OrgMyLife task payload correctness**
|
||||
- For any action item description and note path, payload has correct title and source_url
|
||||
- **Validates: Requirements 9.4**
|
||||
|
||||
- [x] 16. Git auto-commit
|
||||
- [x] 16.1 Implement `integrations/git.py` for git commit operations
|
||||
- Stage new markdown files and stub pages
|
||||
- Commit with message format: `ingestion: import N notes from [source-type]`
|
||||
- Support `--no-commit` flag to skip
|
||||
- Handle git failures gracefully: log warning, don't fail the import
|
||||
- _Requirements: 12.1, 12.2, 12.3, 12.4, 12.5_
|
||||
|
||||
- [ ]* 16.2 Write property test for git commit message format (Property 18)
|
||||
- **Property 18: Git commit message format**
|
||||
- For any count N and source type, message matches `ingestion: import N notes from [source-type]`
|
||||
- **Validates: Requirements 12.2**
|
||||
|
||||
- [x] 17. Checkpoint - Ensure all tests pass
|
||||
- Ensure all tests pass, ask the user if questions arise.
|
||||
|
||||
- [x] 18. Docker integration and deployment
|
||||
- [x] 18.1 Create `NoteGraph/ingestion/Dockerfile` for the ingestion service
|
||||
- Python 3.11+ base image
|
||||
- Install system dependencies: tesseract-ocr, tesseract-ocr-deu
|
||||
- Install Python package with all dependencies
|
||||
- Run FastAPI upload server (uvicorn) as default command
|
||||
- Also support CLI entry point for import/watch/sync commands
|
||||
- _Requirements: 13.1_
|
||||
|
||||
- [x] 18.2 Update `NoteGraph/docker-compose.yml` to add ingestion service
|
||||
- Add `ingestion` service with build context `./ingestion`
|
||||
- Mount `./notes` volume for output
|
||||
- Mount `./ingestion/inbox` for drop-folder input
|
||||
- Expose upload endpoint port (8001)
|
||||
- Pass all env vars from `.env`
|
||||
- Add watcher mode as default or separate service
|
||||
- _Requirements: 11.1_
|
||||
|
||||
- [x] 18.3 Update `NoteGraph/.github/workflows/deploy.yml` with all secrets
|
||||
- Write ALL secrets from GitHub secrets into `.env` on VPS: LLM keys (OpenAI, Anthropic, Google, Mistral), Nextcloud inbox creds, OrgMyLife API key, SB_USER
|
||||
- Follow same pattern as OrgMyLife deploy workflow (ENV_CONTENT with printf)
|
||||
- Add `docker compose up -d --build` to rebuild with ingestion service
|
||||
- _Requirements: 8.5 (additional requirement: secrets via GitHub Actions)_
|
||||
|
||||
- [x] 19. Final checkpoint - Ensure all tests pass
|
||||
- Ensure all tests pass, ask the user if questions arise.
|
||||
|
||||
## Notes
|
||||
|
||||
- Tasks marked with `*` are optional property-based tests and can be skipped for faster MVP
|
||||
- Each task references specific requirements for traceability
|
||||
- Checkpoints ensure incremental validation
|
||||
- Property tests use Hypothesis with `@settings(max_examples=100)`
|
||||
- The web upload endpoint and Nextcloud sync are additional input methods beyond the original CLI/watcher
|
||||
- All secrets are managed via GitHub Actions → `.env` on VPS, never committed to git
|
||||
Reference in New Issue
Block a user