Implementation Plan: NoteGraph Ingestion & Auto-Reference

Overview

This plan implements the NoteGraph ingestion pipeline as a Python package at NoteGraph/ingestion/. Tasks are ordered so each builds on the previous: scaffolding → extraction → LLM → enrichment → linking → output → CLI → web upload → Nextcloud sync → OrgMyLife → git → Docker/deploy → tests. Property-based tests use Hypothesis and are placed close to the code they validate.

Tasks

1. Project scaffolding and configuration
- 1.1 Create NoteGraph/ingestion/ package structure with __init__.py, __main__.py, and all subpackages (extraction/, enrichment/, linking/, output/, integrations/, tests/)
  - Create directory tree matching the design's package structure
  - Add empty __init__.py files in each subpackage
  - Create __main__.py with from ingestion.cli import cli; cli() entry point
  - Requirements: 13.1
- 1.2 Create NoteGraph/ingestion/pyproject.toml with all dependencies
  - Include: click, watchdog, pytesseract, PyPDF2, pdfplumber, python-docx, litellm, httpx, pydantic, pydantic-settings, fastapi, uvicorn, python-multipart
  - Add dev dependencies: pytest, pytest-asyncio, hypothesis, pytest-mock, respx
  - Configure package metadata and entry points
  - Requirements: 7.1, 8.7
- 1.3 Implement NoteGraph/ingestion/config.py with IngestionConfig (pydantic-settings)
  - Define all env vars: LLM_PROVIDER, LLM_MODEL, provider API keys, ORGMYLIFE_API_URL, ORGMYLIFE_API_KEY, NEXTCLOUD_INBOX_URL, NEXTCLOUD_INBOX_USER, NEXTCLOUD_INBOX_PASSWORD, SB_USER, paths, OCR settings, confidence threshold
  - Implement llm_model_string property for LiteLLM routing
  - Implement validate_api_key() that exits with clear error naming the missing variable
  - Requirements: 8.1, 8.2, 8.3, 8.4, 8.5, 8.6
- 1.4 Create NoteGraph/.env.example with all supported environment variables and descriptions
  - Document every variable with comments explaining purpose and valid values
  - Include placeholders for: LLM keys, OrgMyLife, Nextcloud inbox, SB_USER
  - Ensure .env is in .gitignore
  - Requirements: 8.5, 8.7
2. Text extraction layer
- 2.1 Implement extraction/base.py with TextExtractor protocol and ExtractionResult dataclass
  - Define can_handle(file_path: Path) -> bool and extract(file_path: Path) -> ExtractionResult
  - ExtractionResult fields: text, source_file, extraction_method, metadata
  - Requirements: 2.1, 3.1
- 2.2 Implement extraction/text.py for plain text and markdown files
  - Handle .md and .txt extensions
  - Read as UTF-8, attempt encoding detection for non-UTF-8 files (chardet/charset-normalizer)
  - Preserve existing formatting, links, and Obsidian-style wiki-links
  - Requirements: 3.1, 3.2, 3.4, 3.5, 3.6
- 2.3 Implement extraction/pdf.py for PDF text extraction
  - Use pdfplumber for text-based PDFs
  - Detect scanned pages (pages with no extractable text) and delegate to OCR
  - Handle mixed PDFs (text + scanned pages) by concatenating results
  - Requirements: 2.2, 2.3, 2.4
- 2.4 Implement extraction/ocr.py for image OCR via pytesseract
  - Handle .jpg, .jpeg, .png files
  - Configure language as deu+eng (German + English)
  - Preserve paragraph structure by detecting line breaks
  - Return empty text (not error) when OCR cannot extract readable content
  - Requirements: 2.1, 2.2, 2.5, 2.6
- 2.5 Implement extraction/docx.py for DOCX files
  - Use python-docx to extract text preserving heading structure and paragraph breaks
  - Handle .docx extension
  - Requirements: 3.3
- * 2.6 Write property test for text extraction identity (Property 3)
  - Property 3: Text extraction preserves content for text-based formats
  - For any valid UTF-8 string written to a .md or .txt file, extraction returns identical content
  - Validates: Requirements 3.1, 3.2, 3.4
3. File discovery
- 3.1 Implement discovery.py with recursive file discovery and extension filtering
  - Support extensions: .pdf, .jpg, .jpeg, .png, .md, .txt, .docx
  - Accept both single file path and directory path
  - Recursive directory traversal
  - Requirements: 1.1, 1.2, 1.6
- * 3.2 Write property test for file discovery filtering (Property 1)
  - Property 1: File discovery returns only supported extensions
  - For any directory tree with arbitrary extensions, discovery returns exactly those with supported extensions
  - Validates: Requirements 1.1, 1.6
4. LLM provider layer
- 4.1 Implement integrations/llm.py with LiteLLM wrapper
  - LLMClient class with complete(messages, response_format) method
  - Route to provider based on config.llm_model_string (e.g., openai/gpt-4o, anthropic/claude-sonnet-4-20250514)
  - Implement retry logic: 3 retries with exponential backoff for rate limits, timeouts, 5xx errors
  - Raise descriptive exception on failure including provider name and error details
  - Requirements: 7.1, 7.2, 7.3, 7.4, 7.5, 7.6, 7.7
- * 4.2 Write property test for provider routing (Property 9)
  - Property 9: Provider routing from configuration
  - For any valid provider name, configuring LLM_PROVIDER causes the LLM client to route to the corresponding provider
  - Validates: Requirements 7.3, 8.1
- * 4.3 Write property test for missing API key error (Property 10)
  - Property 10: Missing API key produces descriptive error
  - For any provider requiring an API key, when the env var is unset, validation raises an error naming the missing variable
  - Validates: Requirements 7.4, 8.6
5. Enrichment agent
- 5.1 Implement enrichment/models.py with Pydantic models
  - Entity model: type (person/project/date/action_item), value, confidence (0.0-1.0), position
  - ActionItem model: description, assignee, deadline (ISO 8601)
  - EnrichmentResult model: title, category, tags, entities, action_items, summary
  - Requirements: 4.1, 4.2, 4.3, 4.4, 4.5
- 5.2 Implement enrichment/prompts.py with structured LLM prompts
  - System prompt requesting structured JSON output for entity detection
  - Include context about known projects and people (read from existing NoteGraph pages)
  - Request confidence scores for each entity
  - Requirements: 4.1, 4.2, 4.3, 4.4, 4.5
- 5.3 Implement enrichment/agent.py with enrichment orchestration
  - Call LLM with extracted text and structured prompt
  - Parse response into EnrichmentResult
  - Filter entities by confidence threshold (>= 0.7)
  - Handle LLM parse errors: retry once with stricter prompt, then use defaults
  - Requirements: 4.1, 4.2, 4.3, 4.4, 4.5, 4.6
- * 5.4 Write property tests for entity parsing (Properties 4, 5)
  - Property 4: Entity parsing extracts all entities with valid structure
  - For any valid enrichment response JSON, each entity has valid type, non-empty value, confidence in [0.0, 1.0]
  - Property 5: Confidence threshold filtering
  - For any list of entities, filtered output contains exactly those with confidence >= 0.7
  - Validates: Requirements 4.1, 4.2, 4.3, 4.4, 4.5, 4.6
6. Checkpoint - Ensure all tests pass
- Ensure all tests pass, ask the user if questions arise.
7. Reference linker
- 7.1 Implement linking/linker.py with wiki-link generation
  - Person entities → [[people/firstname-lastname]] (slugified)
  - Project entities → [[projects/project-slug]] (slugified)
  - Insert wiki-link inline at entity position (first occurrence only)
  - Leave subsequent mentions of the same entity as plain text
  - Add dates to frontmatter date field
  - Requirements: 5.1, 5.2, 5.3, 5.5, 5.6
- 7.2 Implement linking/stubs.py for stub page creation
  - Check if target page exists in NoteGraph notes directory
  - Create stub page with entity name as title if it doesn't exist
  - Stub pages go in notes/people/ or notes/projects/ as appropriate
  - Requirements: 5.4
- * 7.3 Write property tests for wiki-link generation (Properties 6, 7)
  - Property 6: Wiki-link generation follows correct format per entity type
  - For any person name → [[people/slugified-name]]; for any project name → [[projects/slugified-name]]
  - Property 7: Only first entity occurrence is linked
  - For any text with N >= 2 occurrences of the same entity, only the first is converted to a wiki-link
  - Validates: Requirements 5.1, 5.2, 5.6
8. Output renderer and folder routing
- 8.1 Implement output/renderer.py for markdown + YAML frontmatter rendering
  - Render frontmatter with: title, date, tags, people, projects, source (file + timestamp)
  - Render body with wiki-links inserted
  - Render ## Action Items section at the end with extracted action items
  - Requirements: 6.1, 6.2, 6.3, 6.4, 6.5, 6.6, 9.2
- 8.2 Implement output/router.py for category-based folder routing
  - "meeting" → notes/meetings/
  - "project" → notes/projects/
  - "decision" → notes/decisions/
  - Default/inbox → notes/inbox/
  - Support --output-dir override
  - Requirements: 10.1, 10.2, 10.3, 10.4
- 8.3 Implement output/naming.py for filename generation with collision avoidance
  - Pattern: YYYY-MM-DD-slugified-title.md
  - Slug: lowercase alphanumeric + hyphens only
  - Append numeric suffix (-2, -3) if file already exists
  - Requirements: 10.5, 10.6
- * 8.4 Write property tests for output (Properties 8, 11, 13, 14, 15)
  - Property 8: Frontmatter round-trip validity
  - For any valid EnrichmentResult, render to markdown then parse frontmatter → valid YAML with all required fields
  - Property 11: Action items appear in output markdown section
  - For any list of action items, rendered markdown contains ## Action Items section listing every item
  - Property 13: Category-based folder routing
  - For any category, output path is within the correct folder
  - Property 14: Output filename follows date-slug pattern
  - For any date and title, filename matches YYYY-MM-DD-slugified-title.md
  - Property 15: Filename collision avoidance
  - When a file with the same name exists, a numeric suffix is appended
  - Validates: Requirements 6.1, 6.3, 6.4, 6.6, 6.7, 9.2, 10.1, 10.2, 10.3, 10.5, 10.6
9. Pipeline orchestration
- 9.1 Implement pipeline.py with main processing pipeline
  - Orchestrate: discovery → extraction → enrichment → linking → rendering → file write → git commit
  - Track progress: current file number, total count, file name
  - Handle errors per file: log + skip + continue
  - Output summary: total processed, success count, failed count
  - Support --dry-run mode (display what would be created without writing)
  - Requirements: 1.3, 1.4, 1.5, 13.6
- * 9.2 Write property tests for pipeline resilience and dry-run (Properties 2, 17)
  - Property 2: Bulk import resilience — failed files do not block remaining processing
  - For any batch where a subset fails, remaining files are processed successfully
  - Property 17: Dry-run produces no file writes
  - For any input with --dry-run, zero files are created on disk
  - Validates: Requirements 1.4, 1.5, 13.6
10. CLI interface
- 10.1 Implement cli.py with Click-based CLI
  - import subcommand: accepts path argument (file or directory)
  - watch subcommand: starts drop-folder watcher
  - Global flags: --verbose, --dry-run, --no-commit
  - Import flags: --output-dir, --create-tasks, --provider
  - Display usage help with examples when invoked without arguments
  - Requirements: 13.1, 13.2, 13.3, 13.4, 13.5, 13.6, 13.7
11. Checkpoint - Ensure all tests pass
- Ensure all tests pass, ask the user if questions arise.
12. Web upload endpoint (FastAPI)
- 12.1 Implement web/upload.py with FastAPI app for file upload
  - HTML form with file upload accepting PDF, JPG, PNG
  - Mobile-friendly responsive HTML page (inline CSS, no external dependencies)
  - Basic auth protection using SB_USER env var (same credentials as SilverBullet)
  - On upload: save file to inbox folder, trigger processing pipeline
  - Return success/error response to the user
  - Requirements: 11.2 (extends with web input)
- 12.2 Create web/templates/upload.html mobile-friendly upload page
  - Simple, clean form with drag-and-drop support
  - Responsive design for Android "Share to URL" workflow
  - Show upload progress and result feedback
  - Requirements: (additional requirement: web upload for mobile)
13. Nextcloud WebDAV inbox sync
- 13.1 Implement integrations/nextcloud.py with WebDAV folder polling
  - Poll NEXTCLOUD_INBOX_URL for new files using httpx WebDAV (PROPFIND/GET)
  - Download new files to local inbox for processing
  - After successful processing, move files to a processed/ subfolder on Nextcloud (MOVE request)
  - Auth via NEXTCLOUD_INBOX_USER and NEXTCLOUD_INBOX_PASSWORD env vars
  - Requirements: (additional requirement: Nextcloud WebDAV sync)
- 13.2 Add sync subcommand to CLI for Nextcloud polling
  - One-shot sync: poll once, process new files, exit
  - Can be called from cron or systemd timer
  - Requirements: (additional requirement: Nextcloud WebDAV sync)
14. Drop-folder watcher
- 14.1 Implement watcher.py with watchdog-based folder monitoring
  - Monitor inbox folder for new files
  - Trigger pipeline on new file detection
  - Move processed files to archive/ subdirectory
  - Move failed files to failed/ subdirectory with error log
  - Ignore temporary files (names starting with . or ending with .tmp)
  - Process existing files in inbox on startup before entering watch mode
  - Requirements: 11.1, 11.2, 11.3, 11.4, 11.5, 11.6
- * 14.2 Write property test for temporary file filtering (Property 16)
  - Property 16: Temporary file filtering
  - For any filename, ignore logic returns true iff name starts with . or ends with .tmp
  - Validates: Requirements 11.5
15. OrgMyLife integration
- 15.1 Implement integrations/orgmylife.py with HTTP client for task creation
  - POST to /api/tasks with title, source_url, description
  - Auth via ORGMYLIFE_API_KEY header
  - Handle unreachable API gracefully: log warning, continue without creating tasks
  - Requirements: 9.3, 9.4, 9.5, 9.6
- * 15.2 Write property test for OrgMyLife task payload (Property 12)
  - Property 12: OrgMyLife task payload correctness
  - For any action item description and note path, payload has correct title and source_url
  - Validates: Requirements 9.4
16. Git auto-commit
- 16.1 Implement integrations/git.py for git commit operations
  - Stage new markdown files and stub pages
  - Commit with message format: ingestion: import N notes from [source-type]
  - Support --no-commit flag to skip
  - Handle git failures gracefully: log warning, don't fail the import
  - Requirements: 12.1, 12.2, 12.3, 12.4, 12.5
- * 16.2 Write property test for git commit message format (Property 18)
  - Property 18: Git commit message format
  - For any count N and source type, message matches ingestion: import N notes from [source-type]
  - Validates: Requirements 12.2
17. Checkpoint - Ensure all tests pass
- Ensure all tests pass, ask the user if questions arise.
18. Docker integration and deployment
- 18.1 Create NoteGraph/ingestion/Dockerfile for the ingestion service
  - Python 3.11+ base image
  - Install system dependencies: tesseract-ocr, tesseract-ocr-deu
  - Install Python package with all dependencies
  - Run FastAPI upload server (uvicorn) as default command
  - Also support CLI entry point for import/watch/sync commands
  - Requirements: 13.1
- 18.2 Update NoteGraph/docker-compose.yml to add ingestion service
  - Add ingestion service with build context ./ingestion
  - Mount ./notes volume for output
  - Mount ./ingestion/inbox for drop-folder input
  - Expose upload endpoint port (8001)
  - Pass all env vars from .env
  - Add watcher mode as default or separate service
  - Requirements: 11.1
- 18.3 Update NoteGraph/.github/workflows/deploy.yml with all secrets
  - Write ALL secrets from GitHub secrets into .env on VPS: LLM keys (OpenAI, Anthropic, Google, Mistral), Nextcloud inbox creds, OrgMyLife API key, SB_USER
  - Follow same pattern as OrgMyLife deploy workflow (ENV_CONTENT with printf)
  - Add docker compose up -d --build to rebuild with ingestion service
  - Requirements: 8.5 (additional requirement: secrets via GitHub Actions)
19. Final checkpoint - Ensure all tests pass
- Ensure all tests pass, ask the user if questions arise.

Notes

Tasks marked with * are optional property-based tests and can be skipped for faster MVP
Each task references specific requirements for traceability
Checkpoints ensure incremental validation
Property tests use Hypothesis with @settings(max_examples=100)
The web upload endpoint and Nextcloud sync are additional input methods beyond the original CLI/watcher
All secrets are managed via GitHub Actions → .env on VPS, never committed to git

17 KiB Raw Blame History

Implementation Plan: NoteGraph Ingestion & Auto-Reference

Overview

Tasks

Notes

17 KiB

Raw Blame History