319 lines
17 KiB
Markdown
319 lines
17 KiB
Markdown
# Implementation Plan: NoteGraph Ingestion & Auto-Reference
|
|
|
|
## Overview
|
|
|
|
This plan implements the NoteGraph ingestion pipeline as a Python package at `NoteGraph/ingestion/`. Tasks are ordered so each builds on the previous: scaffolding → extraction → LLM → enrichment → linking → output → CLI → web upload → Nextcloud sync → OrgMyLife → git → Docker/deploy → tests. Property-based tests use Hypothesis and are placed close to the code they validate.
|
|
|
|
## Tasks
|
|
|
|
- [x] 1. Project scaffolding and configuration
|
|
- [x] 1.1 Create `NoteGraph/ingestion/` package structure with `__init__.py`, `__main__.py`, and all subpackages (`extraction/`, `enrichment/`, `linking/`, `output/`, `integrations/`, `tests/`)
|
|
- Create directory tree matching the design's package structure
|
|
- Add empty `__init__.py` files in each subpackage
|
|
- Create `__main__.py` with `from ingestion.cli import cli; cli()` entry point
|
|
- _Requirements: 13.1_
|
|
|
|
- [x] 1.2 Create `NoteGraph/ingestion/pyproject.toml` with all dependencies
|
|
- Include: click, watchdog, pytesseract, PyPDF2, pdfplumber, python-docx, litellm, httpx, pydantic, pydantic-settings, fastapi, uvicorn, python-multipart
|
|
- Add dev dependencies: pytest, pytest-asyncio, hypothesis, pytest-mock, respx
|
|
- Configure package metadata and entry points
|
|
- _Requirements: 7.1, 8.7_
|
|
|
|
- [x] 1.3 Implement `NoteGraph/ingestion/config.py` with `IngestionConfig` (pydantic-settings)
|
|
- Define all env vars: `LLM_PROVIDER`, `LLM_MODEL`, provider API keys, `ORGMYLIFE_API_URL`, `ORGMYLIFE_API_KEY`, `NEXTCLOUD_INBOX_URL`, `NEXTCLOUD_INBOX_USER`, `NEXTCLOUD_INBOX_PASSWORD`, `SB_USER`, paths, OCR settings, confidence threshold
|
|
- Implement `llm_model_string` property for LiteLLM routing
|
|
- Implement `validate_api_key()` that exits with clear error naming the missing variable
|
|
- _Requirements: 8.1, 8.2, 8.3, 8.4, 8.5, 8.6_
|
|
|
|
- [x] 1.4 Create `NoteGraph/.env.example` with all supported environment variables and descriptions
|
|
- Document every variable with comments explaining purpose and valid values
|
|
- Include placeholders for: LLM keys, OrgMyLife, Nextcloud inbox, SB_USER
|
|
- Ensure `.env` is in `.gitignore`
|
|
- _Requirements: 8.5, 8.7_
|
|
|
|
- [x] 2. Text extraction layer
|
|
- [x] 2.1 Implement `extraction/base.py` with `TextExtractor` protocol and `ExtractionResult` dataclass
|
|
- Define `can_handle(file_path: Path) -> bool` and `extract(file_path: Path) -> ExtractionResult`
|
|
- `ExtractionResult` fields: text, source_file, extraction_method, metadata
|
|
- _Requirements: 2.1, 3.1_
|
|
|
|
- [x] 2.2 Implement `extraction/text.py` for plain text and markdown files
|
|
- Handle `.md` and `.txt` extensions
|
|
- Read as UTF-8, attempt encoding detection for non-UTF-8 files (chardet/charset-normalizer)
|
|
- Preserve existing formatting, links, and Obsidian-style wiki-links
|
|
- _Requirements: 3.1, 3.2, 3.4, 3.5, 3.6_
|
|
|
|
- [x] 2.3 Implement `extraction/pdf.py` for PDF text extraction
|
|
- Use pdfplumber for text-based PDFs
|
|
- Detect scanned pages (pages with no extractable text) and delegate to OCR
|
|
- Handle mixed PDFs (text + scanned pages) by concatenating results
|
|
- _Requirements: 2.2, 2.3, 2.4_
|
|
|
|
- [x] 2.4 Implement `extraction/ocr.py` for image OCR via pytesseract
|
|
- Handle `.jpg`, `.jpeg`, `.png` files
|
|
- Configure language as `deu+eng` (German + English)
|
|
- Preserve paragraph structure by detecting line breaks
|
|
- Return empty text (not error) when OCR cannot extract readable content
|
|
- _Requirements: 2.1, 2.2, 2.5, 2.6_
|
|
|
|
- [x] 2.5 Implement `extraction/docx.py` for DOCX files
|
|
- Use python-docx to extract text preserving heading structure and paragraph breaks
|
|
- Handle `.docx` extension
|
|
- _Requirements: 3.3_
|
|
|
|
- [ ]* 2.6 Write property test for text extraction identity (Property 3)
|
|
- **Property 3: Text extraction preserves content for text-based formats**
|
|
- For any valid UTF-8 string written to a .md or .txt file, extraction returns identical content
|
|
- **Validates: Requirements 3.1, 3.2, 3.4**
|
|
|
|
- [x] 3. File discovery
|
|
- [x] 3.1 Implement `discovery.py` with recursive file discovery and extension filtering
|
|
- Support extensions: `.pdf`, `.jpg`, `.jpeg`, `.png`, `.md`, `.txt`, `.docx`
|
|
- Accept both single file path and directory path
|
|
- Recursive directory traversal
|
|
- _Requirements: 1.1, 1.2, 1.6_
|
|
|
|
- [ ]* 3.2 Write property test for file discovery filtering (Property 1)
|
|
- **Property 1: File discovery returns only supported extensions**
|
|
- For any directory tree with arbitrary extensions, discovery returns exactly those with supported extensions
|
|
- **Validates: Requirements 1.1, 1.6**
|
|
|
|
- [ ] 4. LLM provider layer
|
|
- [x] 4.1 Implement `integrations/llm.py` with LiteLLM wrapper
|
|
- `LLMClient` class with `complete(messages, response_format)` method
|
|
- Route to provider based on `config.llm_model_string` (e.g., `openai/gpt-4o`, `anthropic/claude-sonnet-4-20250514`)
|
|
- Implement retry logic: 3 retries with exponential backoff for rate limits, timeouts, 5xx errors
|
|
- Raise descriptive exception on failure including provider name and error details
|
|
- _Requirements: 7.1, 7.2, 7.3, 7.4, 7.5, 7.6, 7.7_
|
|
|
|
- [ ]* 4.2 Write property test for provider routing (Property 9)
|
|
- **Property 9: Provider routing from configuration**
|
|
- For any valid provider name, configuring `LLM_PROVIDER` causes the LLM client to route to the corresponding provider
|
|
- **Validates: Requirements 7.3, 8.1**
|
|
|
|
- [ ]* 4.3 Write property test for missing API key error (Property 10)
|
|
- **Property 10: Missing API key produces descriptive error**
|
|
- For any provider requiring an API key, when the env var is unset, validation raises an error naming the missing variable
|
|
- **Validates: Requirements 7.4, 8.6**
|
|
|
|
- [x] 5. Enrichment agent
|
|
- [x] 5.1 Implement `enrichment/models.py` with Pydantic models
|
|
- `Entity` model: type (person/project/date/action_item), value, confidence (0.0-1.0), position
|
|
- `ActionItem` model: description, assignee, deadline (ISO 8601)
|
|
- `EnrichmentResult` model: title, category, tags, entities, action_items, summary
|
|
- _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5_
|
|
|
|
- [x] 5.2 Implement `enrichment/prompts.py` with structured LLM prompts
|
|
- System prompt requesting structured JSON output for entity detection
|
|
- Include context about known projects and people (read from existing NoteGraph pages)
|
|
- Request confidence scores for each entity
|
|
- _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5_
|
|
|
|
- [x] 5.3 Implement `enrichment/agent.py` with enrichment orchestration
|
|
- Call LLM with extracted text and structured prompt
|
|
- Parse response into `EnrichmentResult`
|
|
- Filter entities by confidence threshold (>= 0.7)
|
|
- Handle LLM parse errors: retry once with stricter prompt, then use defaults
|
|
- _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5, 4.6_
|
|
|
|
- [ ]* 5.4 Write property tests for entity parsing (Properties 4, 5)
|
|
- **Property 4: Entity parsing extracts all entities with valid structure**
|
|
- For any valid enrichment response JSON, each entity has valid type, non-empty value, confidence in [0.0, 1.0]
|
|
- **Property 5: Confidence threshold filtering**
|
|
- For any list of entities, filtered output contains exactly those with confidence >= 0.7
|
|
- **Validates: Requirements 4.1, 4.2, 4.3, 4.4, 4.5, 4.6**
|
|
|
|
- [x] 6. Checkpoint - Ensure all tests pass
|
|
- Ensure all tests pass, ask the user if questions arise.
|
|
|
|
- [x] 7. Reference linker
|
|
- [x] 7.1 Implement `linking/linker.py` with wiki-link generation
|
|
- Person entities → `[[people/firstname-lastname]]` (slugified)
|
|
- Project entities → `[[projects/project-slug]]` (slugified)
|
|
- Insert wiki-link inline at entity position (first occurrence only)
|
|
- Leave subsequent mentions of the same entity as plain text
|
|
- Add dates to frontmatter `date` field
|
|
- _Requirements: 5.1, 5.2, 5.3, 5.5, 5.6_
|
|
|
|
- [x] 7.2 Implement `linking/stubs.py` for stub page creation
|
|
- Check if target page exists in NoteGraph notes directory
|
|
- Create stub page with entity name as title if it doesn't exist
|
|
- Stub pages go in `notes/people/` or `notes/projects/` as appropriate
|
|
- _Requirements: 5.4_
|
|
|
|
- [ ]* 7.3 Write property tests for wiki-link generation (Properties 6, 7)
|
|
- **Property 6: Wiki-link generation follows correct format per entity type**
|
|
- For any person name → `[[people/slugified-name]]`; for any project name → `[[projects/slugified-name]]`
|
|
- **Property 7: Only first entity occurrence is linked**
|
|
- For any text with N >= 2 occurrences of the same entity, only the first is converted to a wiki-link
|
|
- **Validates: Requirements 5.1, 5.2, 5.6**
|
|
|
|
- [x] 8. Output renderer and folder routing
|
|
- [x] 8.1 Implement `output/renderer.py` for markdown + YAML frontmatter rendering
|
|
- Render frontmatter with: title, date, tags, people, projects, source (file + timestamp)
|
|
- Render body with wiki-links inserted
|
|
- Render `## Action Items` section at the end with extracted action items
|
|
- _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5, 6.6, 9.2_
|
|
|
|
- [x] 8.2 Implement `output/router.py` for category-based folder routing
|
|
- "meeting" → `notes/meetings/`
|
|
- "project" → `notes/projects/`
|
|
- "decision" → `notes/decisions/`
|
|
- Default/inbox → `notes/inbox/`
|
|
- Support `--output-dir` override
|
|
- _Requirements: 10.1, 10.2, 10.3, 10.4_
|
|
|
|
- [x] 8.3 Implement `output/naming.py` for filename generation with collision avoidance
|
|
- Pattern: `YYYY-MM-DD-slugified-title.md`
|
|
- Slug: lowercase alphanumeric + hyphens only
|
|
- Append numeric suffix (`-2`, `-3`) if file already exists
|
|
- _Requirements: 10.5, 10.6_
|
|
|
|
- [ ]* 8.4 Write property tests for output (Properties 8, 11, 13, 14, 15)
|
|
- **Property 8: Frontmatter round-trip validity**
|
|
- For any valid EnrichmentResult, render to markdown then parse frontmatter → valid YAML with all required fields
|
|
- **Property 11: Action items appear in output markdown section**
|
|
- For any list of action items, rendered markdown contains `## Action Items` section listing every item
|
|
- **Property 13: Category-based folder routing**
|
|
- For any category, output path is within the correct folder
|
|
- **Property 14: Output filename follows date-slug pattern**
|
|
- For any date and title, filename matches `YYYY-MM-DD-slugified-title.md`
|
|
- **Property 15: Filename collision avoidance**
|
|
- When a file with the same name exists, a numeric suffix is appended
|
|
- **Validates: Requirements 6.1, 6.3, 6.4, 6.6, 6.7, 9.2, 10.1, 10.2, 10.3, 10.5, 10.6**
|
|
|
|
- [x] 9. Pipeline orchestration
|
|
- [x] 9.1 Implement `pipeline.py` with main processing pipeline
|
|
- Orchestrate: discovery → extraction → enrichment → linking → rendering → file write → git commit
|
|
- Track progress: current file number, total count, file name
|
|
- Handle errors per file: log + skip + continue
|
|
- Output summary: total processed, success count, failed count
|
|
- Support `--dry-run` mode (display what would be created without writing)
|
|
- _Requirements: 1.3, 1.4, 1.5, 13.6_
|
|
|
|
- [ ]* 9.2 Write property tests for pipeline resilience and dry-run (Properties 2, 17)
|
|
- **Property 2: Bulk import resilience — failed files do not block remaining processing**
|
|
- For any batch where a subset fails, remaining files are processed successfully
|
|
- **Property 17: Dry-run produces no file writes**
|
|
- For any input with `--dry-run`, zero files are created on disk
|
|
- **Validates: Requirements 1.4, 1.5, 13.6**
|
|
|
|
- [x] 10. CLI interface
|
|
- [x] 10.1 Implement `cli.py` with Click-based CLI
|
|
- `import` subcommand: accepts path argument (file or directory)
|
|
- `watch` subcommand: starts drop-folder watcher
|
|
- Global flags: `--verbose`, `--dry-run`, `--no-commit`
|
|
- Import flags: `--output-dir`, `--create-tasks`, `--provider`
|
|
- Display usage help with examples when invoked without arguments
|
|
- _Requirements: 13.1, 13.2, 13.3, 13.4, 13.5, 13.6, 13.7_
|
|
|
|
- [x] 11. Checkpoint - Ensure all tests pass
|
|
- Ensure all tests pass, ask the user if questions arise.
|
|
|
|
- [ ] 12. Web upload endpoint (FastAPI)
|
|
- [x] 12.1 Implement `web/upload.py` with FastAPI app for file upload
|
|
- HTML form with file upload accepting PDF, JPG, PNG
|
|
- Mobile-friendly responsive HTML page (inline CSS, no external dependencies)
|
|
- Basic auth protection using `SB_USER` env var (same credentials as SilverBullet)
|
|
- On upload: save file to inbox folder, trigger processing pipeline
|
|
- Return success/error response to the user
|
|
- _Requirements: 11.2 (extends with web input)_
|
|
|
|
- [x] 12.2 Create `web/templates/upload.html` mobile-friendly upload page
|
|
- Simple, clean form with drag-and-drop support
|
|
- Responsive design for Android "Share to URL" workflow
|
|
- Show upload progress and result feedback
|
|
- _Requirements: (additional requirement: web upload for mobile)_
|
|
|
|
- [ ] 13. Nextcloud WebDAV inbox sync
|
|
- [x] 13.1 Implement `integrations/nextcloud.py` with WebDAV folder polling
|
|
- Poll `NEXTCLOUD_INBOX_URL` for new files using httpx WebDAV (PROPFIND/GET)
|
|
- Download new files to local inbox for processing
|
|
- After successful processing, move files to a `processed/` subfolder on Nextcloud (MOVE request)
|
|
- Auth via `NEXTCLOUD_INBOX_USER` and `NEXTCLOUD_INBOX_PASSWORD` env vars
|
|
- _Requirements: (additional requirement: Nextcloud WebDAV sync)_
|
|
|
|
- [x] 13.2 Add `sync` subcommand to CLI for Nextcloud polling
|
|
- One-shot sync: poll once, process new files, exit
|
|
- Can be called from cron or systemd timer
|
|
- _Requirements: (additional requirement: Nextcloud WebDAV sync)_
|
|
|
|
- [x] 14. Drop-folder watcher
|
|
- [x] 14.1 Implement `watcher.py` with watchdog-based folder monitoring
|
|
- Monitor inbox folder for new files
|
|
- Trigger pipeline on new file detection
|
|
- Move processed files to `archive/` subdirectory
|
|
- Move failed files to `failed/` subdirectory with error log
|
|
- Ignore temporary files (names starting with `.` or ending with `.tmp`)
|
|
- Process existing files in inbox on startup before entering watch mode
|
|
- _Requirements: 11.1, 11.2, 11.3, 11.4, 11.5, 11.6_
|
|
|
|
- [ ]* 14.2 Write property test for temporary file filtering (Property 16)
|
|
- **Property 16: Temporary file filtering**
|
|
- For any filename, ignore logic returns true iff name starts with `.` or ends with `.tmp`
|
|
- **Validates: Requirements 11.5**
|
|
|
|
- [x] 15. OrgMyLife integration
|
|
- [x] 15.1 Implement `integrations/orgmylife.py` with HTTP client for task creation
|
|
- POST to `/api/tasks` with title, source_url, description
|
|
- Auth via `ORGMYLIFE_API_KEY` header
|
|
- Handle unreachable API gracefully: log warning, continue without creating tasks
|
|
- _Requirements: 9.3, 9.4, 9.5, 9.6_
|
|
|
|
- [ ]* 15.2 Write property test for OrgMyLife task payload (Property 12)
|
|
- **Property 12: OrgMyLife task payload correctness**
|
|
- For any action item description and note path, payload has correct title and source_url
|
|
- **Validates: Requirements 9.4**
|
|
|
|
- [x] 16. Git auto-commit
|
|
- [x] 16.1 Implement `integrations/git.py` for git commit operations
|
|
- Stage new markdown files and stub pages
|
|
- Commit with message format: `ingestion: import N notes from [source-type]`
|
|
- Support `--no-commit` flag to skip
|
|
- Handle git failures gracefully: log warning, don't fail the import
|
|
- _Requirements: 12.1, 12.2, 12.3, 12.4, 12.5_
|
|
|
|
- [ ]* 16.2 Write property test for git commit message format (Property 18)
|
|
- **Property 18: Git commit message format**
|
|
- For any count N and source type, message matches `ingestion: import N notes from [source-type]`
|
|
- **Validates: Requirements 12.2**
|
|
|
|
- [x] 17. Checkpoint - Ensure all tests pass
|
|
- Ensure all tests pass, ask the user if questions arise.
|
|
|
|
- [x] 18. Docker integration and deployment
|
|
- [x] 18.1 Create `NoteGraph/ingestion/Dockerfile` for the ingestion service
|
|
- Python 3.11+ base image
|
|
- Install system dependencies: tesseract-ocr, tesseract-ocr-deu
|
|
- Install Python package with all dependencies
|
|
- Run FastAPI upload server (uvicorn) as default command
|
|
- Also support CLI entry point for import/watch/sync commands
|
|
- _Requirements: 13.1_
|
|
|
|
- [x] 18.2 Update `NoteGraph/docker-compose.yml` to add ingestion service
|
|
- Add `ingestion` service with build context `./ingestion`
|
|
- Mount `./notes` volume for output
|
|
- Mount `./ingestion/inbox` for drop-folder input
|
|
- Expose upload endpoint port (8001)
|
|
- Pass all env vars from `.env`
|
|
- Add watcher mode as default or separate service
|
|
- _Requirements: 11.1_
|
|
|
|
- [x] 18.3 Update `NoteGraph/.github/workflows/deploy.yml` with all secrets
|
|
- Write ALL secrets from GitHub secrets into `.env` on VPS: LLM keys (OpenAI, Anthropic, Google, Mistral), Nextcloud inbox creds, OrgMyLife API key, SB_USER
|
|
- Follow same pattern as OrgMyLife deploy workflow (ENV_CONTENT with printf)
|
|
- Add `docker compose up -d --build` to rebuild with ingestion service
|
|
- _Requirements: 8.5 (additional requirement: secrets via GitHub Actions)_
|
|
|
|
- [x] 19. Final checkpoint - Ensure all tests pass
|
|
- Ensure all tests pass, ask the user if questions arise.
|
|
|
|
## Notes
|
|
|
|
- Tasks marked with `*` are optional property-based tests and can be skipped for faster MVP
|
|
- Each task references specific requirements for traceability
|
|
- Checkpoints ensure incremental validation
|
|
- Property tests use Hypothesis with `@settings(max_examples=100)`
|
|
- The web upload endpoint and Nextcloud sync are additional input methods beyond the original CLI/watcher
|
|
- All secrets are managed via GitHub Actions → `.env` on VPS, never committed to git
|