Cleanup: remove clutter, gitignore secrets, update README
- Remove .secrets from git (real tokens! now gitignored) - Remove orgmylife.db, remarkable_token.json from tracking - Remove SPEC-NoteGraph.md (superseded by monorepo knowledge store) - Remove empty placeholder dirs (shared/powers, shared/mcp-servers, shared/knowledge-store) - Update .gitignore with secrets patterns - Replace old Wissensdatenbank README with monorepo README
This commit is contained in:
@@ -71,3 +71,9 @@ yarn-error.log*
|
||||
|
||||
# Misc
|
||||
.cache/
|
||||
|
||||
# Secrets (NICHT im Klartext committen!)
|
||||
.secrets
|
||||
*.token.json
|
||||
remarkable_token.json
|
||||
orgmylife.db
|
||||
|
||||
@@ -1,21 +0,0 @@
|
||||
# Tokens fuer API-Zugriffe (NICHT committen!)
|
||||
# Diese Datei steht in .gitignore
|
||||
|
||||
# GitLab: Bestellsystem-Repos (read_api fuer Inventar, POM-Analyse etc.)
|
||||
# https://git.tech.rz.db.de/bestellsystem1
|
||||
GITLAB_TOKEN_BESTELLSYSTEM=glpat-4hLnqiONvqxLOGB8AvgerW86MQp1OnpuOAk.01.0z0kt1fdf
|
||||
|
||||
# GitLab: Dieses Audit-Projekt (write fuer Push)
|
||||
# https://git.tech.rz.db.de/AndreKnie/project-audit
|
||||
GITLAB_TOKEN_AUDIT=glpat-my1E6DY1HykjnMuSm5mMYG86MQp1OjFnNTcK.01.1006v7db9
|
||||
|
||||
# Jira
|
||||
JIRA_TOKEN=NTMwMjQ0MzY2NzQ0OkpIZBEiwwYfES+c80xqOpsjVdpx
|
||||
JIRA_TOKEN_SAB=ODc5OTA4NzEwOTA3OtYXcOwp6DF0w2jIqUtpZy08wP6L
|
||||
|
||||
# Confluence
|
||||
CONFLUENCE_TOKEN=MjM0Mzc4ODEyNTkxOlFySnuOOXbzsYdRjrXrSVFR4fRK
|
||||
CONFLUENCE_SAB=NTcwMTk3NjY0MzQ5Ov5N/ZnglAroOMcDmrlLQGrsyrae
|
||||
|
||||
# OrgMyLife
|
||||
ORGMYLIFE_API_SECRET !Z5l5vUcm!&qfhUuE=Nf
|
||||
@@ -1,293 +1,40 @@
|
||||
# Wissensdatenbank ETL-Pipeline
|
||||
# Monorepo
|
||||
|
||||
## Worum geht es? (in einfach)
|
||||
Zentrales Repository für alle Projekte, organisiert nach Arbeitskontext.
|
||||
|
||||
Dies ist die **zentrale Wissenssammlung** fuer den 1st-Level-Support und Chatbots rund um
|
||||
DB InfraGO. Wir **sammeln hier das Wissen** aus vielen Quellen (Confluence, Webseiten,
|
||||
PDFs), pruefen es und legen es **versioniert im Git-Repo** ab. Damit ist dieses Repo die
|
||||
**Single Source of Truth**: Wenn etwas hier steht und freigegeben ist, gilt es - und genau
|
||||
das lesen die Chatbots.
|
||||
|
||||
**Was ist „ETL"?** Drei einfache Schritte, die bei jeder Aufnahme passieren:
|
||||
|
||||
1. **Extract = Sammeln.** Wissen aus den Quellen holen (Confluence-Seiten, Webseiten, PDFs).
|
||||
2. **Transform = Aufbereiten.** In einheitliches Markdown umwandeln, mit Schlagworten/Scope
|
||||
versehen und vertrauliche Inhalte herausfiltern.
|
||||
3. **Load = Ablegen.** Das geprueefte Wissen versioniert nach `data/processed/` schreiben -
|
||||
die zentrale Wahrheit, aus der Chatbots/Vektor-DB lesen.
|
||||
|
||||
Kurz: **Quellen rein → aufbereiten/pruefen → als Single Source of Truth ablegen.**
|
||||
|
||||
## Ziel
|
||||
|
||||
Klar definieren, **welches Wissen in Chatbots darf und welches nicht.** Wissen wird bei
|
||||
der Aufnahme speziell verarbeitet und bereitgestellt, sodass Chatbots es nutzen koennen.
|
||||
Klassifikation nach Scope:
|
||||
|
||||
- **allgemein** – darf intern **und** extern genutzt werden; nicht toolspezifisch
|
||||
(z.B. Regulierung/INB, Kundeninformationen).
|
||||
- **intern** – nur DB InfraGO intern.
|
||||
- **extern** – public im Internet bzw. fuer EVUs/EIUs.
|
||||
|
||||
Technisch eine **ETL-Pipeline fuer LLM/RAG**: holt Wissen aus Confluence, Webseiten und
|
||||
PDFs, klassifiziert es nach Scope, filtert vertrauliche Inhalte und gibt nur geprueftes
|
||||
Wissen frei (`data/processed/<scope>/...` = Feed fuer die Vektor-DB).
|
||||
|
||||
> Architektur & vollstaendiger Wissensfluss inkl. Diagramme:
|
||||
> [`.kiro/steering/architecture.md`](.kiro/steering/architecture.md)
|
||||
> · Manuelle Einrichtungsschritte: [`docs/SETUP.md`](docs/SETUP.md)
|
||||
|
||||
## Denkmodell
|
||||
## Struktur
|
||||
|
||||
```
|
||||
Domaene -> Scope (intern|extern|allgemein) -> Tool -> Dokument(e)
|
||||
bahn/ ← DB InfraGO Projekte (10 Repos, untouched remotes)
|
||||
dhive/ ← dhive GmbH Projekte
|
||||
privat/ ← Persönliche Projekte
|
||||
shared/ ← Kontextübergreifende Tools und Infrastruktur
|
||||
```
|
||||
|
||||
- **Domaene**: fachlicher Bereich (jedes Tool ist eigene Domaene; `regulierung`, `kundeninfo` fuer allgemeines Wissen).
|
||||
- **Scope**: `intern` | `extern` | `allgemein` (allgemein gilt fuer intern UND extern).
|
||||
- **Tool**: eine Fachanwendung aus dem 1st-Level-Support (Katalog automatisch aus Seite 428909879).
|
||||
- **Strategie pro Quelle** – siehe Tabelle unten.
|
||||
## Kontexte
|
||||
|
||||
### Strategien (Verarbeitung pro Quelle)
|
||||
| Kontext | Inhalt | Repos |
|
||||
|---------|--------|-------|
|
||||
| **bahn** | DB InfraGO | aisupport, Analyse-O2C-C2S, awesome-bahn-mcp-servers, beam-mcp, Confluence_Bot, db-planet-mcp-server, O2C-Harness, project-audit, Projekt-KIQ-HP, teamlandkarte-mcp |
|
||||
| **dhive** | dhive GmbH | Jury-Voting |
|
||||
| **privat** | Persönlich | CV, NoteGraph (→ Redo geplant) |
|
||||
| **shared** | Übergreifend | AI-Orchestrator, OrgMyLife, power_skills_and_more, tools/monorepo-cli, references/symphony |
|
||||
|
||||
| Strategie | Was sie tut | Wichtige Options |
|
||||
|-----------|-------------|------------------|
|
||||
| `confluence_page` | genau diese eine Seite | `scope` (auch `intern,extern`), `incremental`, `attachments` |
|
||||
| `confluence_tree` | Seite inkl. Unterseiten | `max_depth` (-1=alle, 0=nur Seite), `incremental`, `attachments` |
|
||||
| `confluence_faq` | FAQs aus Tabellen | `incremental` |
|
||||
| `crawler` | Webseite: Index → Detailseiten | `detail_pattern`, `selector`, `max_pages` |
|
||||
| `sitemap` | Detailseiten aus (gz-)Sitemap | `sitemap_url`, `url_pattern`, `limit`, `incremental` |
|
||||
| `gitlab_md` | Markdown-Dateien aus einem GitLab-Repo | `ref`, `path`, `max_depth` (Token: `GITLAB_TOKEN`) |
|
||||
| `file` | Dateien (md/pdf) aus dem Repo-Ordner `files/` | `max_depth` |
|
||||
| `pdf` | PDFs einer Seite → Markdown (1→n) | `split: headings\|pages`, `heading_pattern`, `keep_raw`, `redact` |
|
||||
|
||||
### Confluence-Anhaenge (eingebundene PDFs)
|
||||
|
||||
Bei `confluence_page`/`confluence_tree` werden **eingebundene PDF-Anhaenge**
|
||||
(`view-file`/`viewpdf`-Makros) automatisch geparst und als **eigene Dokumente** im
|
||||
selben Feed abgelegt (Frontmatter `kind: attachment`, `parent_url` zeigt auf die Seite).
|
||||
Scope/Domain/Owner werden von der Seite geerbt. Inkrementell ueber die Attachment-Version
|
||||
(unveraenderter Anhang -> kein Re-Parse). Default an, pro Quelle ueber
|
||||
`options: { attachments: false }` abschaltbar.
|
||||
|
||||
Bilder werden nicht binaer uebernommen; der **alt-Text/Dateiname** bleibt als
|
||||
`[Bild: ...]` im Markdown - gibt dem RAG Kontext ohne Volumen. Datei-Embeds ohne
|
||||
Parsing-Pfad (z.B. Office-Docs) tauchen als `[Anhang: ...]`-Marker im Seitentext auf.
|
||||
|
||||
### Mehrere Quellen pro Tool
|
||||
|
||||
Ein Tool kann beliebig viele, gemischte Quellen haben – einfach weitere Eintraege
|
||||
unter `sources:` anhaengen (z.B. mehrere Confluence-Quellen mit `max_depth`, eine
|
||||
interne und eine externe Seite, oder zusaetzlich eine FAQ-Seite). Siehe die Tools
|
||||
`pathos` (mehrere Quellen inkl. `confluence_faq` und `intern,extern`) und `nur`.
|
||||
|
||||
### intern vs. extern – wie wird klassifiziert?
|
||||
|
||||
Es gibt **keine Inhaltstrennung innerhalb einer Seite** mehr. Klassifiziert wird pro
|
||||
Tool und pro Quelle:
|
||||
|
||||
**Tool-`scope`** (in `config/tools.yaml`):
|
||||
|
||||
| Tool-`scope` | Bedeutung |
|
||||
|--------------|-----------|
|
||||
| `intern` | alles intern (Default, restriktiv – z.B. Salesforce, NuR) |
|
||||
| `extern` | alles extern |
|
||||
| `allgemein` | gilt fuer intern UND extern |
|
||||
| `mixed` | Tool hat Quellen mit **unterschiedlichem** Scope (z.B. eine interne und eine externe Seite) |
|
||||
|
||||
**Source-`scope`** (pro Quelle, gilt fuer die ganze Seite):
|
||||
- `intern` | `extern` | `allgemein`
|
||||
- `"intern,extern"` → die Seite wird fuer **beide** Scopes genutzt (dupliziert).
|
||||
|
||||
Quellen ohne eigenen `scope` erben den Tool-Scope (bei `mixed` => intern).
|
||||
**Wichtig:** Hat ein Tool gemischtes Wissen, braucht es im Zweifel **zwei Seiten**
|
||||
(eine interne, eine externe) als zwei Quellen – statt einer gemischten Seite.
|
||||
|
||||
Allgemeines, tool-uebergreifendes Wissen steht separat in **`config/general.yaml`**.
|
||||
|
||||
Der Tool-Katalog wird **initial** mit `scripts/bootstrap_tools.py` erzeugt und danach
|
||||
**manuell** in `config/tools.yaml` gepflegt. Uebersicht aller Quellen: GitLab-Pages-Seite
|
||||
**„Wissensquellen"** (`config.html`, geparst aus tools.yaml/general.yaml/approvals.yaml).
|
||||
|
||||
## Neues Wissen hinzufuegen (Workflow)
|
||||
|
||||
Jedes Wissen hat **owners** (intern Verantwortliche, Accountability) und einen
|
||||
**contact** (herausgebbare Kontaktadresse, Default `einfachbahn@deutschebahn.com`).
|
||||
Beide stehen im Frontmatter jedes Dokuments und auf der Pages-Seite, damit klar ist,
|
||||
wer verantwortlich ist und an wen man sich wenden darf.
|
||||
|
||||
1. **Issue anlegen** mit der Vorlage „Neues Wissen" (`.gitlab/issue_templates/`):
|
||||
Tool/Domaene, **Quellen je Zeile** (`URL | Strategie | Scope`), **Verantwortliche**
|
||||
(`owners`) und optional **Kontakt** (Default `einfachbahn@deutschebahn.com`).
|
||||
2. **Reviewer uebernimmt** die Angaben nach `config/tools.yaml` bzw. `config/general.yaml`
|
||||
(neuer Tool-/Quellen-Eintrag inkl. `owners:`) – per GitLab Web-IDE (Variante A) oder
|
||||
manuell (Variante B).
|
||||
3. **Merge Request** öffnen. Die **MR-Freigabe ist das Quality Gate** (4-Augen, via
|
||||
`CODEOWNERS` + Protected Branch).
|
||||
4. **Vorschau** im MR: den `preview`-Job manuell starten und `PREVIEW_ONLY` auf
|
||||
Tool-Id/Domaene/URL der neuen Quelle setzen. Er verarbeitet nur diese Quelle und
|
||||
stellt die erzeugten Markdowns als Job-Artefakt bereit. Lokal alternativ
|
||||
`python -m src.main --only "<tool|domaene|url>" --data preview`.
|
||||
5. **Merge** → der ETL-Schedule verarbeitet die Quelle (stuendlich Mo–Fr, inkrementell) und
|
||||
committet das Wissen; die Vektor-DB liest direkt aus `data/processed/<scope>/...`.
|
||||
|
||||
## Freigabe-Prozess
|
||||
|
||||
```
|
||||
MR mergen (Mensch) -> ETL: Extract -> Transform -> Auto-Filter -> data/processed/ -> Vektor-DB
|
||||
```
|
||||
|
||||
- **Freigabe = Merge Request (Mensch, das eigentliche Gate):** Wer eine Quelle
|
||||
(Link + Strategie + Scope) in `config/tools.yaml`/`general.yaml` eintraegt, stellt einen
|
||||
MR. Beim **Merge** (4-Augen via `CODEOWNERS` + Protected Branch) wird entschieden,
|
||||
welche Quelle mit welchem Scope aufgenommen wird. Danach laeuft der ETL automatisch.
|
||||
- **Auto-Filter im ETL (Sicherheitsnetz, kein Freigabe-Knopf):** setzt pro Dokument
|
||||
`review_status` – standardmaessig **approved**:
|
||||
- `approved` -> `data/processed/...` (**direkt live**, sobald gemergt + ETL gelaufen)
|
||||
- `pending` -> `data/staging/pending/` (Blacklist-Treffer, Scope-Warnung, oder
|
||||
Inhalt < 50 Zeichen – Grund im Frontmatter `review_notes`)
|
||||
- `trusted: true` (z.B. INB) bleibt trotz Treffer approved; `redact: false` schaltet
|
||||
die Redaction pro Quelle ab.
|
||||
- **`config/approvals.yaml` (nachtraegliche Korrektur):** hebt einzelne
|
||||
`pending`-Dokumente per URL oder `hash:<content_hash>` doch auf `approved`.
|
||||
Kein Schritt, den jedes Dokument durchlaeuft. Den Hash zeigt die Pages-Seite (Detail-Fenster).
|
||||
- **Vektor-DB liest direkt aus `data/processed/<scope>/<domaene>/`** (implementiert):
|
||||
Nur freigegebenes Wissen liegt dort, der Scope steckt im Ordnerpfad **und** im
|
||||
Frontmatter (inkl. `owners`/`contact`). Kein separater Export-/Ingestion-Schritt –
|
||||
`processed/` IST der Feed. Der externe Index nimmt nur `extern/` + `allgemein/`.
|
||||
- **`data/_meta.json`**: globale Statusdatei fuer nachgelagerte Systeme mit
|
||||
`last_run` (letzter ETL-Lauf), `last_change` (wann sich der Bestand zuletzt
|
||||
inhaltlich/metadatenseitig geaendert hat), `documents`, `by_scope` und einer
|
||||
`content_signature`. Ein Index-/RAG-Consumer kann daran erkennen, ob ein erneutes
|
||||
Einlesen ueberhaupt noetig ist.
|
||||
- **`data/_index.json`**: dokument-genauer Katalog des gesamten Bestands.
|
||||
Pro Voll-Dokument: `domain`/`tool`/`scope`, `url`, `path`, `content_hash`,
|
||||
`last_updated` und - falls vorhanden - die zugehoerigen `chunks` (Anzahl + Pfad)
|
||||
und `attachments` (Anhang-Dokumente: Name + Pfad). Anhang-Dokumente haben zudem
|
||||
`kind: "attachment"` und `parent_url`. Aggregate: `documents_total`,
|
||||
`chunks_total`, `attachments_total`, `by_domain` (mit `documents`/`chunks`/`attachments`),
|
||||
`by_scope`. So sieht ein Anschliesser auf einen Blick, was pro Domaene/Tool wo liegt
|
||||
und welche Dokumente zusaetzlich als Chunks oder Anhaenge vorliegen (z.B. INB =
|
||||
Voll-Dokument **und** Chunks; pathOS-Seite = Voll-Dokument **und** Anhang-PDF).
|
||||
Deterministisch (kein Zeitstempel) -> aendert sich nur bei echten Bestandsaenderungen.
|
||||
- **`data/run_log.jsonl`**: append-only **Lauf-Historie** (1 JSON-Zeile je ETL-Lauf):
|
||||
Zeitstempel, verarbeitete Dokumente, Status-Counts und **Fehler je Quelle**
|
||||
(z.B. fehlgeschlagene Confluence-Abrufe). Auf die letzten 500 Laeufe gekappt. So sieht
|
||||
man - auch historisch - ob ein Lauf sauber durchlief, ohne die fluechtigen CI-Job-Logs
|
||||
zu durchsuchen. Die Uebersichtsseite zeigt zusaetzlich eine kompakte Health-Zeile
|
||||
(„N Dok verarbeitet, M Quellen mit Fehler") in der Fusszeile.
|
||||
|
||||
### Wo liegt das final freigegebene Wissen?
|
||||
|
||||
Ausschliesslich unter **`data/processed/<scope>/<domaene>/[<tool>/]`** im Repo
|
||||
(versioniert, = Single Source of Truth). `pending`-Dokumente liegen in
|
||||
`data/staging/pending/` und werden zur **Transparenz ebenfalls committet** (auf der
|
||||
Pages-Seite einsehbar). Der Audit-Trail steht in `data/review_report.json` und in
|
||||
der Git-Historie.
|
||||
|
||||
## Lokal testen
|
||||
## Tools
|
||||
|
||||
```bash
|
||||
python3 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
pip install -r requirements.txt
|
||||
|
||||
ruff check src tests # Lint
|
||||
python -m pytest -q # Offline-Tests
|
||||
python -m src.main --config config/tools.yaml --data data # Live-Lauf
|
||||
python -m src.chunk --data data # Chunks erzeugen (offline, optional)
|
||||
# CLI-Tool für Monorepo-Verwaltung
|
||||
cd shared/tools/monorepo-cli
|
||||
pip install -e .
|
||||
ctx-guard --help
|
||||
```
|
||||
|
||||
## Confluence aktivieren
|
||||
## Sicherheit
|
||||
|
||||
```bash
|
||||
export CONFLUENCE_URL="https://arija-confluence.jaas.service.deutschebahn.com"
|
||||
export CONFLUENCE_TOKEN="<PAT>" # Bearer-PAT (Server/DC)
|
||||
pip install atlassian-python-api
|
||||
```
|
||||
- Secrets werden via git-crypt verschlüsselt (pro Kontext eigener Schlüssel)
|
||||
- `.secrets` ist in `.gitignore` — niemals committen
|
||||
- Zugriffskontrolle via `shared/config/access-config.yaml`
|
||||
|
||||
Ohne Credentials werden Confluence-Quellen sauber uebersprungen.
|
||||
## Föderation
|
||||
|
||||
## Betrieb (GitLab CI)
|
||||
|
||||
- `quality`-Stage: `ruff` (Lint) + `gitleaks` (Secret-Scan) bei jedem Push/MR.
|
||||
- `etl`-Stage: **stuendlich Mo–Fr 8–17 Uhr** (Cron `0 8-17 * * 1-5`), baut Wissen,
|
||||
committet `data/` nach `main` (per `GIT_PUSH_TOKEN`); der Commit loest automatisch
|
||||
einen `pages`-Refresh aus.
|
||||
- `pages`-Stage: baut `public/` (Uebersicht, Hilfe, Chatbot-Anschluss, Wissensquellen)
|
||||
fuer GitLab Pages auf `main`.
|
||||
- `preview`-Job (MR, manuell): verarbeitet eine neue/geaenderte Quelle via
|
||||
`--only` und stellt die erzeugten Markdowns als Job-Artefakt bereit (zum Pruefen
|
||||
vor dem Merge). `PREVIEW_ONLY` beim Start auf Tool-Id/Domaene/URL setzen.
|
||||
- **Kundeninfos inkrementell**: `incremental: true` crawlt nur NEUE Detailseiten und
|
||||
ueberschreibt vorhandene nicht – Re-Runs bleiben schnell.
|
||||
- **Confluence inkrementell**: `options: { incremental: true }` prueft pro Seite die
|
||||
Confluence-Versionsnummer (`source_version` im Frontmatter). Unveraenderte Seiten
|
||||
werden uebersprungen, nur geaenderte/neue Seiten neu verarbeitet – spart API-Last.
|
||||
- **Re-Tagging bei Metadaten-Aenderung**: Aenderst du in `tools.yaml`/`general.yaml`
|
||||
nur `tags`/`owners`/`contact` einer Quelle, weicht deren `meta_fingerprint`
|
||||
(im Frontmatter) vom gespeicherten ab. Der naechste inkrementelle Lauf passt dann
|
||||
die bestehenden Dateien dieser Quelle in-place an (nur Frontmatter, kein erneuter
|
||||
Download/API-Call).
|
||||
- **Scope-Wechsel**: Aenderst du den `scope` einer Quelle, wird die Seite neu
|
||||
verarbeitet und an `data/processed/<neuer-scope>/...` geschrieben; die alte Ablage
|
||||
wird nach erfolgreichem Schreiben **automatisch geloescht** (domaenen-intern, damit
|
||||
eine URL, die legitim unter mehreren Domaenen liegt – z.B. eine pathOS-Seite auch
|
||||
unter `web` – nicht versehentlich entfernt wird).
|
||||
- **Domain-Umbenennung**: Die neue Ablage entsteht automatisch, die alte Domain-Ablage
|
||||
bleibt aber stehen (domaenenuebergreifendes Loeschen waere wegen geteilter URLs
|
||||
unsicher). Den alten `data/processed/<scope>/<alte-domain>/`-Ordner daher bei einer
|
||||
Domain-Umbenennung manuell loeschen.
|
||||
- CI/CD-Variablen: `GIT_PUSH_TOKEN`, `CONFLUENCE_URL`, `CONFLUENCE_TOKEN` (Masked).
|
||||
- `scm-info.yaml`: Pflichtdatei fuer DB-GitLab-Compliance.
|
||||
|
||||
## Konfiguration
|
||||
|
||||
- `config/tools.yaml` – Tools, Domaenen, Quellen, Strategien, Scopes, Tags
|
||||
- `config/filter_rules.json` – Blacklist-Keywords, Regex-Redaction
|
||||
- `config/chunking.yaml` – gemeinsame Chunk-Defaults (pro Quelle ueber `options.chunk` ueberschreibbar)
|
||||
|
||||
### Chunking (optional, Default aus)
|
||||
|
||||
Grosse, stark gegliederte Dokumente (v.a. **INB**) werden zusaetzlich zur Voll-Datei in
|
||||
**Chunks entlang der Ueberschriften** zerlegt – ein **abgeleitetes, jederzeit neu
|
||||
erzeugbares** Artefakt unter `data/chunks/<scope>/<domaene>/<docslug>/`. Das Voll-Dokument
|
||||
in `data/processed/` bleibt unangetastet (Parent-Document-Muster): ein Anschliesser waehlt
|
||||
Voll-Dokument, Chunks oder beides. Es werden bewusst nur **deterministische, embedding-freie**
|
||||
Strategien angeboten (`headings | faq | recursive`); semantisches Chunking gehoert in die
|
||||
Vektor-DB des Anschliessers. Jeder Chunk bekommt eine kurze **Contextual-Retrieval**-Zeile
|
||||
(`> Kontext: <Dokument> > <Abschnitt>`). Aktivierung pro Quelle via `options: { chunk: headings }`.
|
||||
|
||||
**Inkrementell (kein Voll-Rebuild):** Chunking laeuft am Ende jedes ETL-Laufs automatisch
|
||||
mit (`src.main`) und ist Teil von `data/`. Pro Dokument wird nur dann neu gechunkt, wenn
|
||||
sich der **Inhalt** (`parent_hash`) ODER die **wirksamen Optionen/Strategie**
|
||||
(`chunk_fingerprint`) geaendert haben - sonst wird das Dokument uebersprungen (keine
|
||||
Datei-Aenderung, kein Git-Churn). Aenderst du also bei einer Quelle z.B. `chunk_level`
|
||||
oder von `headings` auf `recursive`, werden **nur die Dokumente dieser Ablage** neu
|
||||
erzeugt. Wird `chunk` wieder auf `off` gestellt oder ein Voll-Dokument geloescht, werden
|
||||
die zugehoerigen Chunks automatisch entfernt.
|
||||
|
||||
**Feinsteuerung (Default aus, pro Quelle ueberschreibbar):**
|
||||
|
||||
- `chunk_min_doc_chars`: Dokumente unter dieser Zeichenzahl bleiben **ganz** (kleine
|
||||
FAQ-/How-to-Seiten muessen nicht zwingend gechunkt werden).
|
||||
- `chunk_kind`: chunkt nur Dokumente mit passendem `kind` (`""` = alle; `attachment` =
|
||||
nur Anhang-PDFs; `document` = nur Voll-Seiten).
|
||||
|
||||
Beispiel **pathOS** (an der Tool-Tree-Quelle in `config/tools.yaml`): grosse Anhang-PDFs
|
||||
(z.B. EVU-Schnittstellen-Doku, 2,8 MB) werden gechunkt, Seiten und FAQs bleiben ganz.
|
||||
|
||||
```yaml
|
||||
options:
|
||||
incremental: true
|
||||
chunk: headings
|
||||
chunk_kind: attachment # nur Anhang-PDFs chunken
|
||||
chunk_min_doc_chars: 3000 # < 3k Zeichen bleiben ganz (~750 Token)
|
||||
```
|
||||
|
||||
## Version & Changelog
|
||||
|
||||
Aktuelle Version steht in **`VERSION`** (SemVer), die Historie in **`CHANGELOG.md`**
|
||||
(Format „Keep a Changelog"). Beides wird auf der GitLab-Pages-Seite **„Changelog"**
|
||||
angezeigt (Version auch in der Navigationsleiste). Pflege-Regeln:
|
||||
`.kiro/steering/changelog-versioning.md`.
|
||||
Bahn-Team, dhive-Team und private Repos können als eigenständige Team-Repos synchronisiert werden (Hub-and-Spoke). Konfiguration: `shared/config/team-repos.yaml`.
|
||||
|
||||
@@ -1,402 +0,0 @@
|
||||
SPEC.md
|
||||
1. Executive Summary
|
||||
Product / Initiative Name
|
||||
|
||||
Note-Graph
|
||||
|
||||
Interviewed Stakeholder Role
|
||||
|
||||
Owner, sponsor, project lead, and implementation orchestrator (“AI conductor”)
|
||||
|
||||
Product Type
|
||||
|
||||
New product
|
||||
|
||||
Goal in One Sentence
|
||||
|
||||
Create a personal knowledge infrastructure that enables reliable access to all notes, thoughts, decisions, and contextual information at any time in order to improve memory, prioritization, and project continuity.
|
||||
|
||||
Short Business Context
|
||||
|
||||
The stakeholder operates across many parallel projects, domains, meetings, and customer contexts. Information currently exists across handwritten notes, PDFs, meeting artifacts, task systems, Confluence spaces, and AI-generated summaries. The current process relies heavily on memory and chronological note-taking, causing information loss, weak follow-up tracking, and fragmented context retrieval.
|
||||
|
||||
Primary Target Users
|
||||
Primary: Single knowledge worker / project orchestrator
|
||||
Future potential: Extended collaborative usage (currently out of scope)
|
||||
Core Functional Scope Summary
|
||||
|
||||
The system shall:
|
||||
|
||||
consolidate notes and knowledge artifacts from multiple formats and sources,
|
||||
support flexible tagging and contextual relationships,
|
||||
preserve historical context and traceability,
|
||||
enable fast contextual retrieval,
|
||||
connect notes with tasks, people, projects, and decisions,
|
||||
support gradual AI-assisted enrichment,
|
||||
maintain portability and data ownership.
|
||||
In Scope
|
||||
Personal knowledge management
|
||||
Cross-project contextual search
|
||||
Flexible labels/tags
|
||||
Relationships between notes, projects, people, tasks, and decisions
|
||||
Integration of handwritten notes and document-based sources
|
||||
Offline-capable note capture
|
||||
Mobile access
|
||||
Traceability between tasks and source context
|
||||
Export and publication workflows
|
||||
AI-assisted enrichment and retrieval
|
||||
Out of Scope
|
||||
Full project management platform
|
||||
CRM functionality
|
||||
Team collaboration platform
|
||||
Chat/messaging system
|
||||
Calendar replacement
|
||||
Complex automation workflows
|
||||
Autonomous AI agents
|
||||
Advanced graph visualization in first phase
|
||||
Semantic contradiction detection in first phase
|
||||
Main Open Questions
|
||||
Degree of automation vs manual curation
|
||||
How granular note segmentation should become in practice
|
||||
How historical relationship changes should be surfaced to users
|
||||
Scope of AI-generated suggestions in early versions
|
||||
Exact publication/export workflows
|
||||
Main Acceptance Indicators
|
||||
Relevant information can be retrieved during meetings.
|
||||
Meeting preparation becomes significantly easier.
|
||||
Contextual information can be found reliably across projects and time.
|
||||
Tasks remain traceable to their originating notes and decisions.
|
||||
2. Goal
|
||||
One-Sentence Goal
|
||||
|
||||
Provide a durable, searchable, context-aware personal knowledge system that acts as an external memory and decision-support environment.
|
||||
|
||||
Detailed Goal
|
||||
|
||||
The system shall centralize fragmented knowledge sources and allow the stakeholder to:
|
||||
|
||||
capture ideas quickly,
|
||||
retain project continuity,
|
||||
retrieve relevant context rapidly,
|
||||
preserve historical relationships,
|
||||
connect operational tasks with their originating discussions and decisions,
|
||||
maintain structure despite frequent context switching.
|
||||
Business Value
|
||||
Reduced information loss
|
||||
Better prioritization
|
||||
Faster meeting preparation
|
||||
Improved continuity across projects
|
||||
Reduced cognitive load
|
||||
Better follow-through on decisions and tasks
|
||||
Long-term retention of institutional and personal knowledge
|
||||
Intended Outcome for Users and Organization
|
||||
|
||||
Users should be able to trust the system as a reliable external memory and operational context assistant.
|
||||
|
||||
3. Stakeholder Context
|
||||
Respondent Role
|
||||
Owner
|
||||
Sponsor
|
||||
Project lead
|
||||
Implementation orchestrator (“AI conductor”)
|
||||
Perspective Represented
|
||||
|
||||
Single stakeholder representing both business and implementation direction.
|
||||
|
||||
Relevant Organizational or Customer Context
|
||||
|
||||
The stakeholder works across multiple projects, domains, customers, and meeting contexts with heterogeneous documentation ecosystems.
|
||||
|
||||
Whether This Spec Reflects One Interview Only
|
||||
|
||||
Yes. This specification reflects a single interview.
|
||||
|
||||
4. Target Users
|
||||
User Groups
|
||||
Primary User
|
||||
|
||||
Knowledge worker managing many parallel projects and information streams.
|
||||
|
||||
Potential Future Users
|
||||
|
||||
Additional collaborators or project participants (currently out of scope).
|
||||
|
||||
User Context
|
||||
|
||||
Users frequently:
|
||||
|
||||
switch contexts,
|
||||
attend meetings,
|
||||
create handwritten and digital notes,
|
||||
consume large document collections,
|
||||
derive tasks from discussions,
|
||||
revisit historical decisions.
|
||||
User Needs
|
||||
|
||||
Users need to:
|
||||
|
||||
retrieve information quickly,
|
||||
preserve context,
|
||||
avoid forgetting commitments,
|
||||
connect ideas across projects,
|
||||
maintain operational continuity,
|
||||
understand historical decisions.
|
||||
Pain Points
|
||||
Media breaks between tools and formats
|
||||
Difficulty finding information later
|
||||
Fragmented storage locations
|
||||
Reliance on memory
|
||||
Missing conversion from notes to tasks
|
||||
Weak cross-project traceability
|
||||
Loss of contextual continuity
|
||||
Important Differences Between Groups
|
||||
|
||||
No differentiated user groups are currently defined.
|
||||
|
||||
5. Current Situation / Current Process
|
||||
Current Process
|
||||
|
||||
Information currently exists across:
|
||||
|
||||
handwritten meeting notes,
|
||||
PDFs,
|
||||
Confluence spaces,
|
||||
task management tools,
|
||||
AI-assisted document search tools,
|
||||
digital note collections.
|
||||
|
||||
Meeting notes are often incrementally extended over time and searched manually using:
|
||||
|
||||
project names,
|
||||
dates,
|
||||
people,
|
||||
remembered context.
|
||||
Existing Workarounds
|
||||
Manual search
|
||||
AI-assisted document retrieval
|
||||
Chronological note accumulation
|
||||
Personal memory and contextual recall
|
||||
Main Gaps, Bottlenecks, and Failure Points
|
||||
Media discontinuity
|
||||
Information fragmentation
|
||||
Weak traceability
|
||||
Missing task extraction
|
||||
Incomplete prioritization
|
||||
Context loss across projects
|
||||
Inability to reliably recover ideas generated in unrelated contexts
|
||||
What Should Remain Unchanged
|
||||
Existing working style should remain largely compatible.
|
||||
Notes may continue to originate from different tools and formats.
|
||||
6. Functional Scope
|
||||
Main Capabilities
|
||||
|
||||
The system shall:
|
||||
|
||||
ingest and organize knowledge artifacts,
|
||||
support contextual relationships,
|
||||
preserve historical information,
|
||||
enable retrieval across projects and time,
|
||||
support flexible tagging and labeling,
|
||||
connect operational tasks to source knowledge,
|
||||
support gradual AI-assisted enrichment.
|
||||
Key User Interactions
|
||||
|
||||
Users shall be able to:
|
||||
|
||||
capture notes quickly,
|
||||
associate notes with multiple projects,
|
||||
associate notes with people and topics,
|
||||
split notes into smaller contextual fragments,
|
||||
search by content, project, person, or time,
|
||||
review historical decisions,
|
||||
export or publish derived content,
|
||||
revisit unresolved items.
|
||||
Major Functional Areas
|
||||
Knowledge capture
|
||||
Knowledge organization
|
||||
Search and retrieval
|
||||
Relationship management
|
||||
Task traceability
|
||||
Historical context management
|
||||
AI-assisted enrichment
|
||||
Export/publication support
|
||||
Role-Based Differences in Usage
|
||||
|
||||
No differentiated role behavior currently defined.
|
||||
|
||||
Relevant Triggers, Inputs, Outputs, and Outcomes
|
||||
Inputs
|
||||
Handwritten notes
|
||||
PDFs
|
||||
Markdown files
|
||||
Office documents
|
||||
Meeting artifacts
|
||||
AI-generated summaries
|
||||
Outputs
|
||||
Search results
|
||||
Structured contextual views
|
||||
Exported content
|
||||
Task references
|
||||
Meeting preparation material
|
||||
Outcomes
|
||||
Faster information access
|
||||
Improved continuity
|
||||
Better prioritization
|
||||
Reduced forgotten tasks and decisions
|
||||
7. Business Rules
|
||||
Notes may belong to multiple projects simultaneously.
|
||||
Relationships between entities may evolve over time.
|
||||
Historical assignments and timestamps must remain preserved.
|
||||
Original notes and source information must not be lost.
|
||||
Tasks must remain traceable to originating notes or decisions.
|
||||
Structured relationships and free-form associations must coexist.
|
||||
Offline capture capability is required.
|
||||
Mobile access is required.
|
||||
Users must retain control over AI-generated enrichments or suggestions.
|
||||
Relationship interpretation may change over time without overwriting historical truth.
|
||||
8. Business Objects / Functional Data Objects
|
||||
Note
|
||||
|
||||
Represents a captured thought, meeting artifact, observation, or written content.
|
||||
|
||||
Thought Fragment
|
||||
|
||||
Represents a smaller extracted or separable unit derived from a larger note.
|
||||
|
||||
Project
|
||||
|
||||
Represents a business or operational context grouping related knowledge.
|
||||
|
||||
Person
|
||||
|
||||
Represents individuals connected to notes, tasks, meetings, or decisions.
|
||||
|
||||
Task
|
||||
|
||||
Represents actionable follow-up work derived from notes or decisions.
|
||||
|
||||
Decision
|
||||
|
||||
Represents an explicitly recorded outcome or conclusion.
|
||||
|
||||
Source Document
|
||||
|
||||
Represents imported or referenced material such as PDFs, Office documents, or AI-generated summaries.
|
||||
|
||||
Label / Tag
|
||||
|
||||
Represents flexible classification and contextual grouping metadata.
|
||||
|
||||
Relationship
|
||||
|
||||
Represents contextual connections between business objects.
|
||||
|
||||
Time Context
|
||||
|
||||
Represents historical assignment and chronological relevance.
|
||||
|
||||
9. In Scope
|
||||
Personal knowledge graph functionality
|
||||
Multi-project note association
|
||||
Flexible tagging and labeling
|
||||
Contextual relationships
|
||||
Historical traceability
|
||||
Search by text, project, person, and time
|
||||
Mobile access
|
||||
Offline usage
|
||||
Handwritten note integration
|
||||
AI-assisted retrieval and enrichment
|
||||
Export and publishing workflows
|
||||
Task traceability to source context
|
||||
Gradual enrichment through review/question workflows
|
||||
10. Out of Scope
|
||||
Enterprise collaboration platform
|
||||
Full project management suite
|
||||
CRM capabilities
|
||||
Messaging/chat platform
|
||||
Calendar replacement
|
||||
Autonomous AI workflows
|
||||
Complex process automation
|
||||
Advanced graph visualizations in first release
|
||||
Automatic contradiction analysis in first release
|
||||
Fully automated prioritization
|
||||
11. Requirements
|
||||
Knowledge Capture
|
||||
The system shall allow users to capture notes quickly.
|
||||
The system shall support offline note capture.
|
||||
The system shall support mobile access.
|
||||
The system shall support handwritten note integration.
|
||||
The system shall support importing external document sources.
|
||||
Knowledge Organization
|
||||
The system shall allow notes to belong to multiple projects simultaneously.
|
||||
The system shall support labels and tags.
|
||||
The system shall support both structured and free-form relationships.
|
||||
The system shall allow users to split notes into smaller contextual fragments.
|
||||
The system shall preserve relationships between fragmented content and original source notes.
|
||||
Search and Retrieval
|
||||
The system shall provide full-text search.
|
||||
The system shall support contextual retrieval across projects and relationships.
|
||||
The system shall support search by people.
|
||||
The system shall support search by project.
|
||||
The system shall support time-based navigation and retrieval.
|
||||
The system shall allow users to retrieve meeting-relevant information quickly.
|
||||
Task Traceability
|
||||
The system shall support linking tasks to source notes or decisions.
|
||||
The system shall preserve the origin context of tasks.
|
||||
The system shall support identifying missing task metadata such as priorities or deadlines.
|
||||
Historical Integrity
|
||||
The system shall preserve original notes and timestamps.
|
||||
The system shall preserve historical relationship assignments over time.
|
||||
The system shall allow contextual relationships to evolve without losing historical information.
|
||||
AI Assistance
|
||||
The system shall support AI-assisted search and summarization.
|
||||
The system shall support AI-assisted enrichment and categorization.
|
||||
The system shall support gradual future expansion of AI capabilities.
|
||||
The system shall support review or quiz-style interactions for knowledge enrichment.
|
||||
Portability and Ownership
|
||||
The system shall support export of information and derived content.
|
||||
The system shall minimize dependency on proprietary structures.
|
||||
The system shall preserve user data ownership.
|
||||
The system shall support long-term portability and migration.
|
||||
12. Open Questions / Items to Clarify
|
||||
Desired level of automatic relationship generation
|
||||
Exact workflow for exporting/publishing to external systems
|
||||
Preferred granularity for note fragmentation
|
||||
Scope of AI-generated task extraction
|
||||
Desired review cadence for enrichment workflows
|
||||
Long-term collaboration requirements
|
||||
Definition of archival versus active information
|
||||
Rules for conflict resolution between changing contextual assignments
|
||||
13. Risks and Ambiguities
|
||||
Excessive maintenance effort may reduce adoption.
|
||||
Weak search quality could undermine trust in the system.
|
||||
Too many manual steps may interrupt workflows.
|
||||
Incorrect AI-generated relationships may reduce confidence.
|
||||
Information overload may reduce usability.
|
||||
Overly rigid structures may conflict with real-world usage patterns.
|
||||
Insufficient offline support may limit operational usefulness.
|
||||
Lack of clear contextual boundaries may create organizational confusion.
|
||||
Future migration requirements may conflict with short-term optimizations.
|
||||
14. Acceptance Perspective
|
||||
|
||||
The product will be considered successful if:
|
||||
|
||||
users can reliably retrieve relevant information during meetings,
|
||||
meeting preparation becomes significantly easier,
|
||||
contextual information can be recovered across projects and time,
|
||||
tasks remain traceable to their origin,
|
||||
users trust the system as an external memory,
|
||||
the system reduces forgotten follow-ups and lost context,
|
||||
users can maintain organization despite frequent context switching,
|
||||
knowledge remains portable and under user control.
|
||||
15. Glossary
|
||||
Original Term English Explanation Notes / Context
|
||||
Notiz-Graph Note graph / knowledge graph system Working title of the product
|
||||
KI-Dirigent AI orchestrator Stakeholder role coordinating AI-supported workflows
|
||||
Confluence Knowledge collaboration platform Existing documentation source
|
||||
NotebookLM AI-supported document knowledge tool Existing AI-generated summary source
|
||||
Labels Context labels Used for structured assignment
|
||||
Tags Flexible metadata tags Used for contextual classification
|
||||
Todo-App Task management application External task system
|
||||
Medienbruch Media discontinuity Fragmentation between formats/tools
|
||||
Gedächtnisstütze Memory aid Contextual reminder mechanism
|
||||
Binary file not shown.
@@ -1 +0,0 @@
|
||||
{"device_id": "b34feb7e-2c5f-4308-a657-a20a24547171", "device_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdXRoMC11c2VyaWQiOiJhdXRoMHw2NjBkNzM2NWEyZDYzZjBmMmIyOGIwNGIiLCJkZXZpY2UtZGVzYyI6ImRlc2t0b3AtbGludXgiLCJkZXZpY2UtaWQiOiJiMzRmZWI3ZS0yYzVmLTQzMDgtYTY1Ny1hMjBhMjQ1NDcxNzEiLCJpYXQiOjE3NzkzNDc1OTEsImlzcyI6InJNIFdlYkFwcCIsImp0aSI6ImNrMHRUUkZvVXhzPSIsIm5iZiI6MTc3OTM0NzU5MSwic3ViIjoick0gRGV2aWNlIFRva2VuIn0.29K8KBvJynjJyDRpnd36LvcG8-wXwA7MBweaXE0ocmY"}
|
||||
Reference in New Issue
Block a user