ankn/Orchestrator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial monorepo structure

Browse Source
This commit is contained in:
ankn
2026-06-30 20:37:40 +02:00
commit 2f2b295531
121 changed files with 39171 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.kiro/specs/notegraph-ingestion/tasks.md
+318
View File
@@ -0,0 +1,318 @@
# Implementation Plan: NoteGraph Ingestion & Auto-Reference
## Overview
This plan implements the NoteGraph ingestion pipeline as a Python package at `NoteGraph/ingestion/`. Tasks are ordered so each builds on the previous: scaffolding → extraction → LLM → enrichment → linking → output → CLI → web upload → Nextcloud sync → OrgMyLife → git → Docker/deploy → tests. Property-based tests use Hypothesis and are placed close to the code they validate.
## Tasks
- [x] 1. Project scaffolding and configuration
- [x] 1.1 Create `NoteGraph/ingestion/` package structure with `__init__.py`, `__main__.py`, and all subpackages (`extraction/`, `enrichment/`, `linking/`, `output/`, `integrations/`, `tests/`)
- Create directory tree matching the design's package structure
- Add empty `__init__.py` files in each subpackage
- Create `__main__.py` with `from ingestion.cli import cli; cli()` entry point
- _Requirements: 13.1_
- [x] 1.2 Create `NoteGraph/ingestion/pyproject.toml` with all dependencies
- Include: click, watchdog, pytesseract, PyPDF2, pdfplumber, python-docx, litellm, httpx, pydantic, pydantic-settings, fastapi, uvicorn, python-multipart
- Add dev dependencies: pytest, pytest-asyncio, hypothesis, pytest-mock, respx
- Configure package metadata and entry points
- _Requirements: 7.1, 8.7_
- [x] 1.3 Implement `NoteGraph/ingestion/config.py` with `IngestionConfig` (pydantic-settings)
- Define all env vars: `LLM_PROVIDER`, `LLM_MODEL`, provider API keys, `ORGMYLIFE_API_URL`, `ORGMYLIFE_API_KEY`, `NEXTCLOUD_INBOX_URL`, `NEXTCLOUD_INBOX_USER`, `NEXTCLOUD_INBOX_PASSWORD`, `SB_USER`, paths, OCR settings, confidence threshold
- Implement `llm_model_string` property for LiteLLM routing
- Implement `validate_api_key()` that exits with clear error naming the missing variable
- _Requirements: 8.1, 8.2, 8.3, 8.4, 8.5, 8.6_
- [x] 1.4 Create `NoteGraph/.env.example` with all supported environment variables and descriptions
- Document every variable with comments explaining purpose and valid values
- Include placeholders for: LLM keys, OrgMyLife, Nextcloud inbox, SB_USER
- Ensure `.env` is in `.gitignore`
- _Requirements: 8.5, 8.7_
- [x] 2. Text extraction layer
- [x] 2.1 Implement `extraction/base.py` with `TextExtractor` protocol and `ExtractionResult` dataclass
- Define `can_handle(file_path: Path) -> bool` and `extract(file_path: Path) -> ExtractionResult`
- `ExtractionResult` fields: text, source_file, extraction_method, metadata
- _Requirements: 2.1, 3.1_
- [x] 2.2 Implement `extraction/text.py` for plain text and markdown files
- Handle `.md` and `.txt` extensions
- Read as UTF-8, attempt encoding detection for non-UTF-8 files (chardet/charset-normalizer)
- Preserve existing formatting, links, and Obsidian-style wiki-links
- _Requirements: 3.1, 3.2, 3.4, 3.5, 3.6_
- [x] 2.3 Implement `extraction/pdf.py` for PDF text extraction
- Use pdfplumber for text-based PDFs
- Detect scanned pages (pages with no extractable text) and delegate to OCR
- Handle mixed PDFs (text + scanned pages) by concatenating results
- _Requirements: 2.2, 2.3, 2.4_
- [x] 2.4 Implement `extraction/ocr.py` for image OCR via pytesseract
- Handle `.jpg`, `.jpeg`, `.png` files
- Configure language as `deu+eng` (German + English)
- Preserve paragraph structure by detecting line breaks
- Return empty text (not error) when OCR cannot extract readable content
- _Requirements: 2.1, 2.2, 2.5, 2.6_
- [x] 2.5 Implement `extraction/docx.py` for DOCX files
- Use python-docx to extract text preserving heading structure and paragraph breaks
- Handle `.docx` extension
- _Requirements: 3.3_
- [ ]* 2.6 Write property test for text extraction identity (Property 3)
- **Property 3: Text extraction preserves content for text-based formats**
- For any valid UTF-8 string written to a .md or .txt file, extraction returns identical content
- **Validates: Requirements 3.1, 3.2, 3.4**
- [x] 3. File discovery
- [x] 3.1 Implement `discovery.py` with recursive file discovery and extension filtering
- Support extensions: `.pdf`, `.jpg`, `.jpeg`, `.png`, `.md`, `.txt`, `.docx`
- Accept both single file path and directory path
- Recursive directory traversal
- _Requirements: 1.1, 1.2, 1.6_
- [ ]* 3.2 Write property test for file discovery filtering (Property 1)
- **Property 1: File discovery returns only supported extensions**
- For any directory tree with arbitrary extensions, discovery returns exactly those with supported extensions
- **Validates: Requirements 1.1, 1.6**
- [ ] 4. LLM provider layer
- [x] 4.1 Implement `integrations/llm.py` with LiteLLM wrapper
- `LLMClient` class with `complete(messages, response_format)` method
- Route to provider based on `config.llm_model_string` (e.g., `openai/gpt-4o`, `anthropic/claude-sonnet-4-20250514`)
- Implement retry logic: 3 retries with exponential backoff for rate limits, timeouts, 5xx errors
- Raise descriptive exception on failure including provider name and error details
- _Requirements: 7.1, 7.2, 7.3, 7.4, 7.5, 7.6, 7.7_
- [ ]* 4.2 Write property test for provider routing (Property 9)
- **Property 9: Provider routing from configuration**
- For any valid provider name, configuring `LLM_PROVIDER` causes the LLM client to route to the corresponding provider
- **Validates: Requirements 7.3, 8.1**
- [ ]* 4.3 Write property test for missing API key error (Property 10)
- **Property 10: Missing API key produces descriptive error**
- For any provider requiring an API key, when the env var is unset, validation raises an error naming the missing variable
- **Validates: Requirements 7.4, 8.6**
- [x] 5. Enrichment agent
- [x] 5.1 Implement `enrichment/models.py` with Pydantic models
- `Entity` model: type (person/project/date/action_item), value, confidence (0.0-1.0), position
- `ActionItem` model: description, assignee, deadline (ISO 8601)
- `EnrichmentResult` model: title, category, tags, entities, action_items, summary
- _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5_
- [x] 5.2 Implement `enrichment/prompts.py` with structured LLM prompts
- System prompt requesting structured JSON output for entity detection
- Include context about known projects and people (read from existing NoteGraph pages)
- Request confidence scores for each entity
- _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5_
- [x] 5.3 Implement `enrichment/agent.py` with enrichment orchestration
- Call LLM with extracted text and structured prompt
- Parse response into `EnrichmentResult`
- Filter entities by confidence threshold (>= 0.7)
- Handle LLM parse errors: retry once with stricter prompt, then use defaults
- _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5, 4.6_
- [ ]* 5.4 Write property tests for entity parsing (Properties 4, 5)
- **Property 4: Entity parsing extracts all entities with valid structure**
- For any valid enrichment response JSON, each entity has valid type, non-empty value, confidence in [0.0, 1.0]
- **Property 5: Confidence threshold filtering**
- For any list of entities, filtered output contains exactly those with confidence >= 0.7
- **Validates: Requirements 4.1, 4.2, 4.3, 4.4, 4.5, 4.6**
- [x] 6. Checkpoint - Ensure all tests pass
- Ensure all tests pass, ask the user if questions arise.
- [x] 7. Reference linker
- [x] 7.1 Implement `linking/linker.py` with wiki-link generation
- Person entities → `[[people/firstname-lastname]]` (slugified)
- Project entities → `[[projects/project-slug]]` (slugified)
- Insert wiki-link inline at entity position (first occurrence only)
- Leave subsequent mentions of the same entity as plain text
- Add dates to frontmatter `date` field
- _Requirements: 5.1, 5.2, 5.3, 5.5, 5.6_
- [x] 7.2 Implement `linking/stubs.py` for stub page creation
- Check if target page exists in NoteGraph notes directory
- Create stub page with entity name as title if it doesn't exist
- Stub pages go in `notes/people/` or `notes/projects/` as appropriate
- _Requirements: 5.4_
- [ ]* 7.3 Write property tests for wiki-link generation (Properties 6, 7)
- **Property 6: Wiki-link generation follows correct format per entity type**
- For any person name → `[[people/slugified-name]]`; for any project name → `[[projects/slugified-name]]`
- **Property 7: Only first entity occurrence is linked**
- For any text with N >= 2 occurrences of the same entity, only the first is converted to a wiki-link
- **Validates: Requirements 5.1, 5.2, 5.6**
- [x] 8. Output renderer and folder routing
- [x] 8.1 Implement `output/renderer.py` for markdown + YAML frontmatter rendering
- Render frontmatter with: title, date, tags, people, projects, source (file + timestamp)
- Render body with wiki-links inserted
- Render `## Action Items` section at the end with extracted action items
- _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5, 6.6, 9.2_
- [x] 8.2 Implement `output/router.py` for category-based folder routing
- "meeting" → `notes/meetings/`
- "project" → `notes/projects/`
- "decision" → `notes/decisions/`
- Default/inbox → `notes/inbox/`
- Support `--output-dir` override
- _Requirements: 10.1, 10.2, 10.3, 10.4_
- [x] 8.3 Implement `output/naming.py` for filename generation with collision avoidance
- Pattern: `YYYY-MM-DD-slugified-title.md`
- Slug: lowercase alphanumeric + hyphens only
- Append numeric suffix (`-2`, `-3`) if file already exists
- _Requirements: 10.5, 10.6_
- [ ]* 8.4 Write property tests for output (Properties 8, 11, 13, 14, 15)
- **Property 8: Frontmatter round-trip validity**
- For any valid EnrichmentResult, render to markdown then parse frontmatter → valid YAML with all required fields
- **Property 11: Action items appear in output markdown section**
- For any list of action items, rendered markdown contains `## Action Items` section listing every item
- **Property 13: Category-based folder routing**
- For any category, output path is within the correct folder
- **Property 14: Output filename follows date-slug pattern**
- For any date and title, filename matches `YYYY-MM-DD-slugified-title.md`
- **Property 15: Filename collision avoidance**
- When a file with the same name exists, a numeric suffix is appended
- **Validates: Requirements 6.1, 6.3, 6.4, 6.6, 6.7, 9.2, 10.1, 10.2, 10.3, 10.5, 10.6**
- [x] 9. Pipeline orchestration
- [x] 9.1 Implement `pipeline.py` with main processing pipeline
- Orchestrate: discovery → extraction → enrichment → linking → rendering → file write → git commit
- Track progress: current file number, total count, file name
- Handle errors per file: log + skip + continue
- Output summary: total processed, success count, failed count
- Support `--dry-run` mode (display what would be created without writing)
- _Requirements: 1.3, 1.4, 1.5, 13.6_
- [ ]* 9.2 Write property tests for pipeline resilience and dry-run (Properties 2, 17)
- **Property 2: Bulk import resilience — failed files do not block remaining processing**
- For any batch where a subset fails, remaining files are processed successfully
- **Property 17: Dry-run produces no file writes**
- For any input with `--dry-run`, zero files are created on disk
- **Validates: Requirements 1.4, 1.5, 13.6**
- [x] 10. CLI interface
- [x] 10.1 Implement `cli.py` with Click-based CLI
- `import` subcommand: accepts path argument (file or directory)
- `watch` subcommand: starts drop-folder watcher
- Global flags: `--verbose`, `--dry-run`, `--no-commit`
- Import flags: `--output-dir`, `--create-tasks`, `--provider`
- Display usage help with examples when invoked without arguments
- _Requirements: 13.1, 13.2, 13.3, 13.4, 13.5, 13.6, 13.7_
- [x] 11. Checkpoint - Ensure all tests pass
- Ensure all tests pass, ask the user if questions arise.
- [ ] 12. Web upload endpoint (FastAPI)
- [x] 12.1 Implement `web/upload.py` with FastAPI app for file upload
- HTML form with file upload accepting PDF, JPG, PNG
- Mobile-friendly responsive HTML page (inline CSS, no external dependencies)
- Basic auth protection using `SB_USER` env var (same credentials as SilverBullet)
- On upload: save file to inbox folder, trigger processing pipeline
- Return success/error response to the user
- _Requirements: 11.2 (extends with web input)_
- [x] 12.2 Create `web/templates/upload.html` mobile-friendly upload page
- Simple, clean form with drag-and-drop support
- Responsive design for Android "Share to URL" workflow
- Show upload progress and result feedback
- _Requirements: (additional requirement: web upload for mobile)_
- [ ] 13. Nextcloud WebDAV inbox sync
- [x] 13.1 Implement `integrations/nextcloud.py` with WebDAV folder polling
- Poll `NEXTCLOUD_INBOX_URL` for new files using httpx WebDAV (PROPFIND/GET)
- Download new files to local inbox for processing
- After successful processing, move files to a `processed/` subfolder on Nextcloud (MOVE request)
- Auth via `NEXTCLOUD_INBOX_USER` and `NEXTCLOUD_INBOX_PASSWORD` env vars
- _Requirements: (additional requirement: Nextcloud WebDAV sync)_
- [x] 13.2 Add `sync` subcommand to CLI for Nextcloud polling
- One-shot sync: poll once, process new files, exit
- Can be called from cron or systemd timer
- _Requirements: (additional requirement: Nextcloud WebDAV sync)_
- [x] 14. Drop-folder watcher
- [x] 14.1 Implement `watcher.py` with watchdog-based folder monitoring
- Monitor inbox folder for new files
- Trigger pipeline on new file detection
- Move processed files to `archive/` subdirectory
- Move failed files to `failed/` subdirectory with error log
- Ignore temporary files (names starting with `.` or ending with `.tmp`)
- Process existing files in inbox on startup before entering watch mode
- _Requirements: 11.1, 11.2, 11.3, 11.4, 11.5, 11.6_
- [ ]* 14.2 Write property test for temporary file filtering (Property 16)
- **Property 16: Temporary file filtering**
- For any filename, ignore logic returns true iff name starts with `.` or ends with `.tmp`
- **Validates: Requirements 11.5**
- [x] 15. OrgMyLife integration
- [x] 15.1 Implement `integrations/orgmylife.py` with HTTP client for task creation
- POST to `/api/tasks` with title, source_url, description
- Auth via `ORGMYLIFE_API_KEY` header
- Handle unreachable API gracefully: log warning, continue without creating tasks
- _Requirements: 9.3, 9.4, 9.5, 9.6_
- [ ]* 15.2 Write property test for OrgMyLife task payload (Property 12)
- **Property 12: OrgMyLife task payload correctness**
- For any action item description and note path, payload has correct title and source_url
- **Validates: Requirements 9.4**
- [x] 16. Git auto-commit
- [x] 16.1 Implement `integrations/git.py` for git commit operations
- Stage new markdown files and stub pages
- Commit with message format: `ingestion: import N notes from [source-type]`
- Support `--no-commit` flag to skip
- Handle git failures gracefully: log warning, don't fail the import
- _Requirements: 12.1, 12.2, 12.3, 12.4, 12.5_
- [ ]* 16.2 Write property test for git commit message format (Property 18)
- **Property 18: Git commit message format**
- For any count N and source type, message matches `ingestion: import N notes from [source-type]`
- **Validates: Requirements 12.2**
- [x] 17. Checkpoint - Ensure all tests pass
- Ensure all tests pass, ask the user if questions arise.
- [x] 18. Docker integration and deployment
- [x] 18.1 Create `NoteGraph/ingestion/Dockerfile` for the ingestion service
- Python 3.11+ base image
- Install system dependencies: tesseract-ocr, tesseract-ocr-deu
- Install Python package with all dependencies
- Run FastAPI upload server (uvicorn) as default command
- Also support CLI entry point for import/watch/sync commands
- _Requirements: 13.1_
- [x] 18.2 Update `NoteGraph/docker-compose.yml` to add ingestion service
- Add `ingestion` service with build context `./ingestion`
- Mount `./notes` volume for output
- Mount `./ingestion/inbox` for drop-folder input
- Expose upload endpoint port (8001)
- Pass all env vars from `.env`
- Add watcher mode as default or separate service
- _Requirements: 11.1_
- [x] 18.3 Update `NoteGraph/.github/workflows/deploy.yml` with all secrets
- Write ALL secrets from GitHub secrets into `.env` on VPS: LLM keys (OpenAI, Anthropic, Google, Mistral), Nextcloud inbox creds, OrgMyLife API key, SB_USER
- Follow same pattern as OrgMyLife deploy workflow (ENV_CONTENT with printf)
- Add `docker compose up -d --build` to rebuild with ingestion service
- _Requirements: 8.5 (additional requirement: secrets via GitHub Actions)_
- [x] 19. Final checkpoint - Ensure all tests pass
- Ensure all tests pass, ask the user if questions arise.
## Notes
- Tasks marked with `*` are optional property-based tests and can be skipped for faster MVP
- Each task references specific requirements for traceability
- Checkpoints ensure incremental validation
- Property tests use Hypothesis with `@settings(max_examples=100)`
- The web upload endpoint and Nextcloud sync are additional input methods beyond the original CLI/watcher
- All secrets are managed via GitHub Actions → `.env` on VPS, never committed to git