Orchestrator/.kiro/specs/notegraph-ingestion/requirements.md

Requirements Document

Introduction

The NoteGraph Ingestion & Auto-Reference feature adds a document import pipeline and AI-powered entity linking system to NoteGraph. It enables bulk import of PDF files (from reMarkable tablet), images (photos of sticky notes, whiteboards, handwritten notes), and legacy text files (from OneNote, Android Notes, Obsidian, scattered file systems). Imported content is OCR-processed, enriched by an LLM to detect entities (people, projects, dates, TODOs), and output as structured markdown with wiki-links and frontmatter into the appropriate NoteGraph folder. The system uses a model-agnostic LLM abstraction layer supporting multiple providers via simple .env configuration.

Glossary

• Ingestion_Service: The Python service responsible for orchestrating the full import pipeline from file input through OCR, enrichment, and markdown output.
• OCR_Engine: The component that extracts raw text from image files (JPG, PNG) and scanned PDF pages using optical character recognition.
• Text_Extractor: The component that extracts text content from text-based PDFs and legacy file formats (markdown, plain text, OneNote exports).
• LLM_Provider: An abstraction layer that routes enrichment requests to a configured language model provider (OpenAI, Anthropic, Google, Mistral, Ollama/Gemma) without coupling to any specific API.
• Enrichment_Agent: The AI-powered component that processes raw extracted text to detect entities, generate metadata, create wiki-links, and structure the output as markdown.
• Entity_Detector: The sub-component of the Enrichment_Agent responsible for identifying person names, project references, dates, and action items in extracted text.
• Reference_Linker: The sub-component that converts detected entities into SilverBullet-compatible wiki-links (e.g., [[people/name]], [[projects/name]]).
• Drop_Folder_Watcher: A background process that monitors a designated inbox directory for new files and triggers the Ingestion_Service automatically.
• Inbox_Folder: The designated directory where files are placed for automatic ingestion (default: NoteGraph/ingestion/inbox/).
• Output_Folder: The target directory within NoteGraph/notes/ where processed markdown files are placed.
• Provider_Config: The .env-based configuration that specifies which LLM_Provider to use and supplies the corresponding API keys.
• OrgMyLife_Client: The HTTP client component that creates tasks in the OrgMyLife API from extracted action items.
• Frontmatter: YAML metadata block at the top of a markdown file containing structured fields (title, date, tags, people, projects).

Requirements

Requirement 1: Bulk File Import via CLI

User Story: As a user, I want to import many files at once from a folder using a CLI command, so that I can quickly ingest large collections of notes from various sources.

Acceptance Criteria

1. WHEN the user runs the CLI import command with a directory path, THE Ingestion_Service SHALL recursively discover all supported files (PDF, JPG, PNG, MD, TXT) in that directory.
2. WHEN the user runs the CLI import command with a single file path, THE Ingestion_Service SHALL process that individual file.
3. WHEN processing multiple files, THE Ingestion_Service SHALL report progress showing the current file number, total count, and file name.
4. WHEN a file fails to process during bulk import, THE Ingestion_Service SHALL log the error with the file path and reason, skip the file, and continue processing remaining files.
5. WHEN bulk import completes, THE Ingestion_Service SHALL output a summary showing total files processed, successful count, and failed count.
6. THE Ingestion_Service SHALL support the following file extensions: .pdf, .jpg, .jpeg, .png, .md, .txt, .docx.

Requirement 2: OCR and Text Extraction from Images

User Story: As a user, I want photos of handwritten notes and whiteboards converted to text, so that I can search and reference them in my knowledge base.

Acceptance Criteria

1. WHEN a JPG or PNG file is provided, THE OCR_Engine SHALL extract text content from the image and return it as a UTF-8 string.
2. WHEN a PDF file contains scanned pages (image-based content without embedded text), THE OCR_Engine SHALL apply OCR to extract text from those pages.
3. WHEN a PDF file contains embedded text, THE Text_Extractor SHALL extract the text directly without OCR.
4. WHEN a PDF file contains a mix of text pages and scanned pages, THE Ingestion_Service SHALL apply the appropriate extraction method per page and concatenate results.
5. IF the OCR_Engine cannot extract any readable text from an image, THEN THE Ingestion_Service SHALL create the output note with an empty body and attach the original file reference in the frontmatter.
6. WHEN processing handwritten content, THE OCR_Engine SHALL preserve paragraph structure by detecting line breaks and groupings.

Requirement 3: Legacy File Format Import

User Story: As a user, I want to import notes from old tools like OneNote, Obsidian, and scattered text files, so that I can consolidate all my knowledge in one place.

Acceptance Criteria

1. WHEN a markdown file is provided, THE Text_Extractor SHALL read the file content preserving existing formatting, links, and structure.
2. WHEN a plain text file is provided, THE Text_Extractor SHALL read the file content as UTF-8 text.
3. WHEN a DOCX file is provided, THE Text_Extractor SHALL extract the text content preserving heading structure and paragraph breaks.
4. WHEN an imported markdown file contains Obsidian-style wiki-links ([[page name]]), THE Text_Extractor SHALL preserve them for later resolution by the Reference_Linker.
5. WHEN a file uses a non-UTF-8 encoding, THE Text_Extractor SHALL attempt to detect the encoding and convert to UTF-8.
6. IF a file cannot be decoded to text, THEN THE Ingestion_Service SHALL log a warning and skip the file.

Requirement 4: AI-Powered Entity Detection

User Story: As a user, I want the system to automatically detect people, projects, dates, and action items in my notes, so that I do not have to manually tag and link everything.

Acceptance Criteria

1. WHEN extracted text is provided to the Enrichment_Agent, THE Entity_Detector SHALL identify person names mentioned in the text.
2. WHEN extracted text is provided to the Enrichment_Agent, THE Entity_Detector SHALL identify project references mentioned in the text.
3. WHEN extracted text is provided to the Enrichment_Agent, THE Entity_Detector SHALL identify dates and temporal references in the text and normalize them to ISO 8601 format.
4. WHEN extracted text is provided to the Enrichment_Agent, THE Entity_Detector SHALL identify action items and TODO statements in the text.
5. WHEN the Entity_Detector identifies entities, THE Entity_Detector SHALL return each entity with its type, extracted value, and confidence score between 0.0 and 1.0.
6. THE Entity_Detector SHALL only include entities with a confidence score of 0.7 or higher in the final output.

Requirement 5: Auto-Reference Wiki-Link Generation

User Story: As a user, I want detected entities automatically converted to wiki-links, so that my imported notes are immediately connected to my knowledge graph.

Acceptance Criteria

1. WHEN a person entity is detected, THE Reference_Linker SHALL generate a wiki-link in the format [[people/firstname-lastname]].
2. WHEN a project entity is detected, THE Reference_Linker SHALL generate a wiki-link in the format [[projects/project-name]].
3. WHEN a date entity is detected, THE Reference_Linker SHALL add the normalized date to the frontmatter date field.
4. WHEN the Reference_Linker generates a wiki-link for a person or project that does not yet exist in NoteGraph, THE Reference_Linker SHALL create a stub page with the entity name as title.
5. WHEN the Reference_Linker generates a wiki-link, THE Reference_Linker SHALL insert the link inline in the note body at the position where the entity was detected.
6. WHEN multiple references to the same entity exist in one note, THE Reference_Linker SHALL link only the first occurrence and leave subsequent mentions as plain text.

Requirement 6: Structured Markdown Output with Frontmatter

User Story: As a user, I want imported notes output as structured markdown with proper metadata, so that they integrate seamlessly with SilverBullet's features.

Acceptance Criteria

1. THE Ingestion_Service SHALL output each processed file as a markdown file with YAML frontmatter containing: title, date, tags, people, projects, and source fields.
2. WHEN the Enrichment_Agent generates a title from the content, THE Ingestion_Service SHALL use it as the frontmatter title field.
3. WHEN entities are detected, THE Ingestion_Service SHALL populate the frontmatter people field as a list of detected person names.
4. WHEN entities are detected, THE Ingestion_Service SHALL populate the frontmatter projects field as a list of detected project names.
5. WHEN entities are detected, THE Ingestion_Service SHALL populate the frontmatter tags field with relevant topic tags derived from the content.
6. THE Ingestion_Service SHALL include a source field in the frontmatter containing the original file name and import timestamp.
7. FOR ALL valid extracted text inputs, processing through the Enrichment_Agent and then rendering to markdown and then parsing the frontmatter SHALL produce a valid YAML frontmatter block (round-trip property).

Requirement 7: Model-Agnostic LLM Provider Layer

User Story: As a user, I want to easily switch between LLM providers and try different models, so that I can compare behavior and avoid vendor lock-in.

Acceptance Criteria

1. THE LLM_Provider SHALL expose a unified interface for text completion requests regardless of the underlying provider.
2. THE LLM_Provider SHALL support at minimum the following providers: OpenAI, Anthropic, Google Gemini, Mistral, Gemma (local), and Ollama (local).
3. WHEN a provider is specified in the Provider_Config, THE LLM_Provider SHALL route requests to that provider using the configured API key.
4. WHEN the configured provider is unavailable or returns an error, THE LLM_Provider SHALL raise a descriptive exception including the provider name and error details.
5. THE LLM_Provider SHALL read provider selection and API keys from environment variables defined in a .env file.
6. THE LLM_Provider SHALL support configuring the model name per provider (e.g., gpt-4o, claude-sonnet-4-20250514, gemini-pro).
7. WHEN switching providers, THE LLM_Provider SHALL require only a change to environment variables without code modifications.

Requirement 8: Provider Configuration via Environment Variables

User Story: As a user, I want simple .env-based configuration for LLM providers, so that I can set up and switch providers without editing code.

Acceptance Criteria

1. THE Provider_Config SHALL use the environment variable LLM_PROVIDER to select the active provider (values: openai, anthropic, google, mistral, ollama).
2. THE Provider_Config SHALL use the environment variable LLM_MODEL to specify the model name for the active provider.
3. THE Provider_Config SHALL use provider-specific environment variables for API keys: OPENAI_API_KEY, ANTHROPIC_API_KEY, GOOGLE_API_KEY, MISTRAL_API_KEY.
4. WHERE the Ollama provider is selected, THE Provider_Config SHALL use the environment variable OLLAMA_BASE_URL to specify the local Ollama endpoint (default: http://localhost:11434). Ollama SHALL support running Google Gemma models locally.
5. THE Provider_Config SHALL NOT store API keys or secrets directly in git-tracked files. The .env file SHALL be listed in .gitignore and a .env.example file SHALL document all variables without actual secret values.
6. WHEN a required API key is missing for the selected provider, THE Ingestion_Service SHALL exit with a clear error message naming the missing variable.
7. THE Ingestion_Service SHALL provide a .env.example file documenting all supported environment variables with descriptions.

Requirement 9: TODO Extraction and OrgMyLife Integration

User Story: As a user, I want action items extracted from my notes and optionally created as tasks in OrgMyLife, so that I do not lose track of commitments found in imported notes.

Acceptance Criteria

1. WHEN the Entity_Detector identifies action items in the text, THE Enrichment_Agent SHALL extract each action item with its description and any associated person or deadline.
2. WHEN action items are extracted, THE Ingestion_Service SHALL list them in a ## Action Items section at the end of the output markdown.
3. WHEN the --create-tasks flag is provided on the CLI, THE OrgMyLife_Client SHALL create a task in OrgMyLife for each extracted action item via POST to /api/tasks.
4. WHEN creating a task in OrgMyLife, THE OrgMyLife_Client SHALL set the task title to the action item description and include a source_url linking back to the imported note.
5. IF the OrgMyLife API is unreachable when --create-tasks is specified, THEN THE Ingestion_Service SHALL log a warning and continue processing without creating tasks.
6. THE Provider_Config SHALL use the environment variable ORGMYLIFE_API_URL for the OrgMyLife endpoint and ORGMYLIFE_API_KEY for authentication.

Requirement 10: File Placement and Folder Routing

User Story: As a user, I want imported notes automatically placed in the correct NoteGraph folder based on their content, so that my knowledge base stays organized.

Acceptance Criteria

1. WHEN the Enrichment_Agent detects meeting-related content, THE Ingestion_Service SHALL place the output file in NoteGraph/notes/meetings/.
2. WHEN the Enrichment_Agent detects project-specific content with a single dominant project, THE Ingestion_Service SHALL place the output file in NoteGraph/notes/projects/.
3. WHEN the Enrichment_Agent cannot determine a specific category, THE Ingestion_Service SHALL place the output file in NoteGraph/notes/inbox/.
4. WHEN the --output-dir flag is provided on the CLI, THE Ingestion_Service SHALL place all output files in the specified directory regardless of content analysis.
5. THE Ingestion_Service SHALL generate output file names using the pattern YYYY-MM-DD-slugified-title.md based on the detected or current date and the generated title.
6. IF an output file with the same name already exists, THEN THE Ingestion_Service SHALL append a numeric suffix (e.g., -2, -3) to avoid overwriting.

Requirement 11: Drop-Folder Watcher for Continuous Import

User Story: As a user, I want to drop files into a folder and have them automatically processed, so that I can import notes without running CLI commands.

Acceptance Criteria

1. WHEN the watcher mode is started, THE Drop_Folder_Watcher SHALL monitor the Inbox_Folder for new files.
2. WHEN a new file appears in the Inbox_Folder, THE Drop_Folder_Watcher SHALL trigger the Ingestion_Service to process that file.
3. WHEN a file is successfully processed, THE Drop_Folder_Watcher SHALL move the original file to an archive/ subdirectory within the Inbox_Folder.
4. IF a file fails to process, THEN THE Drop_Folder_Watcher SHALL move the file to a failed/ subdirectory and log the error.
5. THE Drop_Folder_Watcher SHALL ignore temporary files (names starting with . or ending with .tmp).
6. WHEN the watcher mode is started, THE Drop_Folder_Watcher SHALL process any files already present in the Inbox_Folder before entering watch mode.

Requirement 12: Git Auto-Commit for Imported Notes

User Story: As a user, I want imported notes automatically committed to git, so that my knowledge base history is preserved without manual intervention.

Acceptance Criteria

1. WHEN one or more files are successfully imported, THE Ingestion_Service SHALL stage the new markdown files and commit them to the NoteGraph git repository.
2. THE Ingestion_Service SHALL use the commit message format: ingestion: import N notes from [source-type] where N is the count and source-type describes the input (e.g., "pdf", "images", "bulk").
3. WHEN stub pages are created by the Reference_Linker, THE Ingestion_Service SHALL include them in the same git commit.
4. WHEN the --no-commit flag is provided on the CLI, THE Ingestion_Service SHALL skip the git commit step.
5. IF the git commit fails, THEN THE Ingestion_Service SHALL log a warning but not fail the overall import process.

Requirement 13: CLI Interface Design

User Story: As a user, I want a clear and consistent CLI interface, so that I can easily run imports with different options.

Acceptance Criteria

1. THE Ingestion_Service SHALL provide a CLI entry point via python -m ingestion.cli.
2. THE Ingestion_Service SHALL support the subcommand import accepting a path argument (file or directory).
3. THE Ingestion_Service SHALL support the subcommand watch to start the Drop_Folder_Watcher.
4. THE Ingestion_Service SHALL support the following global flags: --verbose for detailed logging, --dry-run to preview without writing files, --no-commit to skip git commits.
5. THE Ingestion_Service SHALL support the following import flags: --output-dir to override output location, --create-tasks to enable OrgMyLife task creation, --provider to override the LLM provider for this run.
6. WHEN the --dry-run flag is provided, THE Ingestion_Service SHALL display what would be created (file paths, detected entities, proposed links) without writing any files.
7. WHEN invoked without arguments, THE Ingestion_Service SHALL display usage help with examples.