Cleanup: remove clutter, gitignore secrets, update README
- Remove .secrets from git (real tokens! now gitignored) - Remove orgmylife.db, remarkable_token.json from tracking - Remove SPEC-NoteGraph.md (superseded by monorepo knowledge store) - Remove empty placeholder dirs (shared/powers, shared/mcp-servers, shared/knowledge-store) - Update .gitignore with secrets patterns - Replace old Wissensdatenbank README with monorepo README
This commit is contained in:
@@ -71,3 +71,9 @@ yarn-error.log*
|
|||||||
|
|
||||||
# Misc
|
# Misc
|
||||||
.cache/
|
.cache/
|
||||||
|
|
||||||
|
# Secrets (NICHT im Klartext committen!)
|
||||||
|
.secrets
|
||||||
|
*.token.json
|
||||||
|
remarkable_token.json
|
||||||
|
orgmylife.db
|
||||||
|
|||||||
@@ -1,21 +0,0 @@
|
|||||||
# Tokens fuer API-Zugriffe (NICHT committen!)
|
|
||||||
# Diese Datei steht in .gitignore
|
|
||||||
|
|
||||||
# GitLab: Bestellsystem-Repos (read_api fuer Inventar, POM-Analyse etc.)
|
|
||||||
# https://git.tech.rz.db.de/bestellsystem1
|
|
||||||
GITLAB_TOKEN_BESTELLSYSTEM=glpat-4hLnqiONvqxLOGB8AvgerW86MQp1OnpuOAk.01.0z0kt1fdf
|
|
||||||
|
|
||||||
# GitLab: Dieses Audit-Projekt (write fuer Push)
|
|
||||||
# https://git.tech.rz.db.de/AndreKnie/project-audit
|
|
||||||
GITLAB_TOKEN_AUDIT=glpat-my1E6DY1HykjnMuSm5mMYG86MQp1OjFnNTcK.01.1006v7db9
|
|
||||||
|
|
||||||
# Jira
|
|
||||||
JIRA_TOKEN=NTMwMjQ0MzY2NzQ0OkpIZBEiwwYfES+c80xqOpsjVdpx
|
|
||||||
JIRA_TOKEN_SAB=ODc5OTA4NzEwOTA3OtYXcOwp6DF0w2jIqUtpZy08wP6L
|
|
||||||
|
|
||||||
# Confluence
|
|
||||||
CONFLUENCE_TOKEN=MjM0Mzc4ODEyNTkxOlFySnuOOXbzsYdRjrXrSVFR4fRK
|
|
||||||
CONFLUENCE_SAB=NTcwMTk3NjY0MzQ5Ov5N/ZnglAroOMcDmrlLQGrsyrae
|
|
||||||
|
|
||||||
# OrgMyLife
|
|
||||||
ORGMYLIFE_API_SECRET !Z5l5vUcm!&qfhUuE=Nf
|
|
||||||
@@ -1,293 +1,40 @@
|
|||||||
# Wissensdatenbank ETL-Pipeline
|
# Monorepo
|
||||||
|
|
||||||
## Worum geht es? (in einfach)
|
Zentrales Repository für alle Projekte, organisiert nach Arbeitskontext.
|
||||||
|
|
||||||
Dies ist die **zentrale Wissenssammlung** fuer den 1st-Level-Support und Chatbots rund um
|
## Struktur
|
||||||
DB InfraGO. Wir **sammeln hier das Wissen** aus vielen Quellen (Confluence, Webseiten,
|
|
||||||
PDFs), pruefen es und legen es **versioniert im Git-Repo** ab. Damit ist dieses Repo die
|
|
||||||
**Single Source of Truth**: Wenn etwas hier steht und freigegeben ist, gilt es - und genau
|
|
||||||
das lesen die Chatbots.
|
|
||||||
|
|
||||||
**Was ist „ETL"?** Drei einfache Schritte, die bei jeder Aufnahme passieren:
|
|
||||||
|
|
||||||
1. **Extract = Sammeln.** Wissen aus den Quellen holen (Confluence-Seiten, Webseiten, PDFs).
|
|
||||||
2. **Transform = Aufbereiten.** In einheitliches Markdown umwandeln, mit Schlagworten/Scope
|
|
||||||
versehen und vertrauliche Inhalte herausfiltern.
|
|
||||||
3. **Load = Ablegen.** Das geprueefte Wissen versioniert nach `data/processed/` schreiben -
|
|
||||||
die zentrale Wahrheit, aus der Chatbots/Vektor-DB lesen.
|
|
||||||
|
|
||||||
Kurz: **Quellen rein → aufbereiten/pruefen → als Single Source of Truth ablegen.**
|
|
||||||
|
|
||||||
## Ziel
|
|
||||||
|
|
||||||
Klar definieren, **welches Wissen in Chatbots darf und welches nicht.** Wissen wird bei
|
|
||||||
der Aufnahme speziell verarbeitet und bereitgestellt, sodass Chatbots es nutzen koennen.
|
|
||||||
Klassifikation nach Scope:
|
|
||||||
|
|
||||||
- **allgemein** – darf intern **und** extern genutzt werden; nicht toolspezifisch
|
|
||||||
(z.B. Regulierung/INB, Kundeninformationen).
|
|
||||||
- **intern** – nur DB InfraGO intern.
|
|
||||||
- **extern** – public im Internet bzw. fuer EVUs/EIUs.
|
|
||||||
|
|
||||||
Technisch eine **ETL-Pipeline fuer LLM/RAG**: holt Wissen aus Confluence, Webseiten und
|
|
||||||
PDFs, klassifiziert es nach Scope, filtert vertrauliche Inhalte und gibt nur geprueftes
|
|
||||||
Wissen frei (`data/processed/<scope>/...` = Feed fuer die Vektor-DB).
|
|
||||||
|
|
||||||
> Architektur & vollstaendiger Wissensfluss inkl. Diagramme:
|
|
||||||
> [`.kiro/steering/architecture.md`](.kiro/steering/architecture.md)
|
|
||||||
> · Manuelle Einrichtungsschritte: [`docs/SETUP.md`](docs/SETUP.md)
|
|
||||||
|
|
||||||
## Denkmodell
|
|
||||||
|
|
||||||
```
|
```
|
||||||
Domaene -> Scope (intern|extern|allgemein) -> Tool -> Dokument(e)
|
bahn/ ← DB InfraGO Projekte (10 Repos, untouched remotes)
|
||||||
|
dhive/ ← dhive GmbH Projekte
|
||||||
|
privat/ ← Persönliche Projekte
|
||||||
|
shared/ ← Kontextübergreifende Tools und Infrastruktur
|
||||||
```
|
```
|
||||||
|
|
||||||
- **Domaene**: fachlicher Bereich (jedes Tool ist eigene Domaene; `regulierung`, `kundeninfo` fuer allgemeines Wissen).
|
## Kontexte
|
||||||
- **Scope**: `intern` | `extern` | `allgemein` (allgemein gilt fuer intern UND extern).
|
|
||||||
- **Tool**: eine Fachanwendung aus dem 1st-Level-Support (Katalog automatisch aus Seite 428909879).
|
|
||||||
- **Strategie pro Quelle** – siehe Tabelle unten.
|
|
||||||
|
|
||||||
### Strategien (Verarbeitung pro Quelle)
|
| Kontext | Inhalt | Repos |
|
||||||
|
|---------|--------|-------|
|
||||||
|
| **bahn** | DB InfraGO | aisupport, Analyse-O2C-C2S, awesome-bahn-mcp-servers, beam-mcp, Confluence_Bot, db-planet-mcp-server, O2C-Harness, project-audit, Projekt-KIQ-HP, teamlandkarte-mcp |
|
||||||
|
| **dhive** | dhive GmbH | Jury-Voting |
|
||||||
|
| **privat** | Persönlich | CV, NoteGraph (→ Redo geplant) |
|
||||||
|
| **shared** | Übergreifend | AI-Orchestrator, OrgMyLife, power_skills_and_more, tools/monorepo-cli, references/symphony |
|
||||||
|
|
||||||
| Strategie | Was sie tut | Wichtige Options |
|
## Tools
|
||||||
|-----------|-------------|------------------|
|
|
||||||
| `confluence_page` | genau diese eine Seite | `scope` (auch `intern,extern`), `incremental`, `attachments` |
|
|
||||||
| `confluence_tree` | Seite inkl. Unterseiten | `max_depth` (-1=alle, 0=nur Seite), `incremental`, `attachments` |
|
|
||||||
| `confluence_faq` | FAQs aus Tabellen | `incremental` |
|
|
||||||
| `crawler` | Webseite: Index → Detailseiten | `detail_pattern`, `selector`, `max_pages` |
|
|
||||||
| `sitemap` | Detailseiten aus (gz-)Sitemap | `sitemap_url`, `url_pattern`, `limit`, `incremental` |
|
|
||||||
| `gitlab_md` | Markdown-Dateien aus einem GitLab-Repo | `ref`, `path`, `max_depth` (Token: `GITLAB_TOKEN`) |
|
|
||||||
| `file` | Dateien (md/pdf) aus dem Repo-Ordner `files/` | `max_depth` |
|
|
||||||
| `pdf` | PDFs einer Seite → Markdown (1→n) | `split: headings\|pages`, `heading_pattern`, `keep_raw`, `redact` |
|
|
||||||
|
|
||||||
### Confluence-Anhaenge (eingebundene PDFs)
|
|
||||||
|
|
||||||
Bei `confluence_page`/`confluence_tree` werden **eingebundene PDF-Anhaenge**
|
|
||||||
(`view-file`/`viewpdf`-Makros) automatisch geparst und als **eigene Dokumente** im
|
|
||||||
selben Feed abgelegt (Frontmatter `kind: attachment`, `parent_url` zeigt auf die Seite).
|
|
||||||
Scope/Domain/Owner werden von der Seite geerbt. Inkrementell ueber die Attachment-Version
|
|
||||||
(unveraenderter Anhang -> kein Re-Parse). Default an, pro Quelle ueber
|
|
||||||
`options: { attachments: false }` abschaltbar.
|
|
||||||
|
|
||||||
Bilder werden nicht binaer uebernommen; der **alt-Text/Dateiname** bleibt als
|
|
||||||
`[Bild: ...]` im Markdown - gibt dem RAG Kontext ohne Volumen. Datei-Embeds ohne
|
|
||||||
Parsing-Pfad (z.B. Office-Docs) tauchen als `[Anhang: ...]`-Marker im Seitentext auf.
|
|
||||||
|
|
||||||
### Mehrere Quellen pro Tool
|
|
||||||
|
|
||||||
Ein Tool kann beliebig viele, gemischte Quellen haben – einfach weitere Eintraege
|
|
||||||
unter `sources:` anhaengen (z.B. mehrere Confluence-Quellen mit `max_depth`, eine
|
|
||||||
interne und eine externe Seite, oder zusaetzlich eine FAQ-Seite). Siehe die Tools
|
|
||||||
`pathos` (mehrere Quellen inkl. `confluence_faq` und `intern,extern`) und `nur`.
|
|
||||||
|
|
||||||
### intern vs. extern – wie wird klassifiziert?
|
|
||||||
|
|
||||||
Es gibt **keine Inhaltstrennung innerhalb einer Seite** mehr. Klassifiziert wird pro
|
|
||||||
Tool und pro Quelle:
|
|
||||||
|
|
||||||
**Tool-`scope`** (in `config/tools.yaml`):
|
|
||||||
|
|
||||||
| Tool-`scope` | Bedeutung |
|
|
||||||
|--------------|-----------|
|
|
||||||
| `intern` | alles intern (Default, restriktiv – z.B. Salesforce, NuR) |
|
|
||||||
| `extern` | alles extern |
|
|
||||||
| `allgemein` | gilt fuer intern UND extern |
|
|
||||||
| `mixed` | Tool hat Quellen mit **unterschiedlichem** Scope (z.B. eine interne und eine externe Seite) |
|
|
||||||
|
|
||||||
**Source-`scope`** (pro Quelle, gilt fuer die ganze Seite):
|
|
||||||
- `intern` | `extern` | `allgemein`
|
|
||||||
- `"intern,extern"` → die Seite wird fuer **beide** Scopes genutzt (dupliziert).
|
|
||||||
|
|
||||||
Quellen ohne eigenen `scope` erben den Tool-Scope (bei `mixed` => intern).
|
|
||||||
**Wichtig:** Hat ein Tool gemischtes Wissen, braucht es im Zweifel **zwei Seiten**
|
|
||||||
(eine interne, eine externe) als zwei Quellen – statt einer gemischten Seite.
|
|
||||||
|
|
||||||
Allgemeines, tool-uebergreifendes Wissen steht separat in **`config/general.yaml`**.
|
|
||||||
|
|
||||||
Der Tool-Katalog wird **initial** mit `scripts/bootstrap_tools.py` erzeugt und danach
|
|
||||||
**manuell** in `config/tools.yaml` gepflegt. Uebersicht aller Quellen: GitLab-Pages-Seite
|
|
||||||
**„Wissensquellen"** (`config.html`, geparst aus tools.yaml/general.yaml/approvals.yaml).
|
|
||||||
|
|
||||||
## Neues Wissen hinzufuegen (Workflow)
|
|
||||||
|
|
||||||
Jedes Wissen hat **owners** (intern Verantwortliche, Accountability) und einen
|
|
||||||
**contact** (herausgebbare Kontaktadresse, Default `einfachbahn@deutschebahn.com`).
|
|
||||||
Beide stehen im Frontmatter jedes Dokuments und auf der Pages-Seite, damit klar ist,
|
|
||||||
wer verantwortlich ist und an wen man sich wenden darf.
|
|
||||||
|
|
||||||
1. **Issue anlegen** mit der Vorlage „Neues Wissen" (`.gitlab/issue_templates/`):
|
|
||||||
Tool/Domaene, **Quellen je Zeile** (`URL | Strategie | Scope`), **Verantwortliche**
|
|
||||||
(`owners`) und optional **Kontakt** (Default `einfachbahn@deutschebahn.com`).
|
|
||||||
2. **Reviewer uebernimmt** die Angaben nach `config/tools.yaml` bzw. `config/general.yaml`
|
|
||||||
(neuer Tool-/Quellen-Eintrag inkl. `owners:`) – per GitLab Web-IDE (Variante A) oder
|
|
||||||
manuell (Variante B).
|
|
||||||
3. **Merge Request** öffnen. Die **MR-Freigabe ist das Quality Gate** (4-Augen, via
|
|
||||||
`CODEOWNERS` + Protected Branch).
|
|
||||||
4. **Vorschau** im MR: den `preview`-Job manuell starten und `PREVIEW_ONLY` auf
|
|
||||||
Tool-Id/Domaene/URL der neuen Quelle setzen. Er verarbeitet nur diese Quelle und
|
|
||||||
stellt die erzeugten Markdowns als Job-Artefakt bereit. Lokal alternativ
|
|
||||||
`python -m src.main --only "<tool|domaene|url>" --data preview`.
|
|
||||||
5. **Merge** → der ETL-Schedule verarbeitet die Quelle (stuendlich Mo–Fr, inkrementell) und
|
|
||||||
committet das Wissen; die Vektor-DB liest direkt aus `data/processed/<scope>/...`.
|
|
||||||
|
|
||||||
## Freigabe-Prozess
|
|
||||||
|
|
||||||
```
|
|
||||||
MR mergen (Mensch) -> ETL: Extract -> Transform -> Auto-Filter -> data/processed/ -> Vektor-DB
|
|
||||||
```
|
|
||||||
|
|
||||||
- **Freigabe = Merge Request (Mensch, das eigentliche Gate):** Wer eine Quelle
|
|
||||||
(Link + Strategie + Scope) in `config/tools.yaml`/`general.yaml` eintraegt, stellt einen
|
|
||||||
MR. Beim **Merge** (4-Augen via `CODEOWNERS` + Protected Branch) wird entschieden,
|
|
||||||
welche Quelle mit welchem Scope aufgenommen wird. Danach laeuft der ETL automatisch.
|
|
||||||
- **Auto-Filter im ETL (Sicherheitsnetz, kein Freigabe-Knopf):** setzt pro Dokument
|
|
||||||
`review_status` – standardmaessig **approved**:
|
|
||||||
- `approved` -> `data/processed/...` (**direkt live**, sobald gemergt + ETL gelaufen)
|
|
||||||
- `pending` -> `data/staging/pending/` (Blacklist-Treffer, Scope-Warnung, oder
|
|
||||||
Inhalt < 50 Zeichen – Grund im Frontmatter `review_notes`)
|
|
||||||
- `trusted: true` (z.B. INB) bleibt trotz Treffer approved; `redact: false` schaltet
|
|
||||||
die Redaction pro Quelle ab.
|
|
||||||
- **`config/approvals.yaml` (nachtraegliche Korrektur):** hebt einzelne
|
|
||||||
`pending`-Dokumente per URL oder `hash:<content_hash>` doch auf `approved`.
|
|
||||||
Kein Schritt, den jedes Dokument durchlaeuft. Den Hash zeigt die Pages-Seite (Detail-Fenster).
|
|
||||||
- **Vektor-DB liest direkt aus `data/processed/<scope>/<domaene>/`** (implementiert):
|
|
||||||
Nur freigegebenes Wissen liegt dort, der Scope steckt im Ordnerpfad **und** im
|
|
||||||
Frontmatter (inkl. `owners`/`contact`). Kein separater Export-/Ingestion-Schritt –
|
|
||||||
`processed/` IST der Feed. Der externe Index nimmt nur `extern/` + `allgemein/`.
|
|
||||||
- **`data/_meta.json`**: globale Statusdatei fuer nachgelagerte Systeme mit
|
|
||||||
`last_run` (letzter ETL-Lauf), `last_change` (wann sich der Bestand zuletzt
|
|
||||||
inhaltlich/metadatenseitig geaendert hat), `documents`, `by_scope` und einer
|
|
||||||
`content_signature`. Ein Index-/RAG-Consumer kann daran erkennen, ob ein erneutes
|
|
||||||
Einlesen ueberhaupt noetig ist.
|
|
||||||
- **`data/_index.json`**: dokument-genauer Katalog des gesamten Bestands.
|
|
||||||
Pro Voll-Dokument: `domain`/`tool`/`scope`, `url`, `path`, `content_hash`,
|
|
||||||
`last_updated` und - falls vorhanden - die zugehoerigen `chunks` (Anzahl + Pfad)
|
|
||||||
und `attachments` (Anhang-Dokumente: Name + Pfad). Anhang-Dokumente haben zudem
|
|
||||||
`kind: "attachment"` und `parent_url`. Aggregate: `documents_total`,
|
|
||||||
`chunks_total`, `attachments_total`, `by_domain` (mit `documents`/`chunks`/`attachments`),
|
|
||||||
`by_scope`. So sieht ein Anschliesser auf einen Blick, was pro Domaene/Tool wo liegt
|
|
||||||
und welche Dokumente zusaetzlich als Chunks oder Anhaenge vorliegen (z.B. INB =
|
|
||||||
Voll-Dokument **und** Chunks; pathOS-Seite = Voll-Dokument **und** Anhang-PDF).
|
|
||||||
Deterministisch (kein Zeitstempel) -> aendert sich nur bei echten Bestandsaenderungen.
|
|
||||||
- **`data/run_log.jsonl`**: append-only **Lauf-Historie** (1 JSON-Zeile je ETL-Lauf):
|
|
||||||
Zeitstempel, verarbeitete Dokumente, Status-Counts und **Fehler je Quelle**
|
|
||||||
(z.B. fehlgeschlagene Confluence-Abrufe). Auf die letzten 500 Laeufe gekappt. So sieht
|
|
||||||
man - auch historisch - ob ein Lauf sauber durchlief, ohne die fluechtigen CI-Job-Logs
|
|
||||||
zu durchsuchen. Die Uebersichtsseite zeigt zusaetzlich eine kompakte Health-Zeile
|
|
||||||
(„N Dok verarbeitet, M Quellen mit Fehler") in der Fusszeile.
|
|
||||||
|
|
||||||
### Wo liegt das final freigegebene Wissen?
|
|
||||||
|
|
||||||
Ausschliesslich unter **`data/processed/<scope>/<domaene>/[<tool>/]`** im Repo
|
|
||||||
(versioniert, = Single Source of Truth). `pending`-Dokumente liegen in
|
|
||||||
`data/staging/pending/` und werden zur **Transparenz ebenfalls committet** (auf der
|
|
||||||
Pages-Seite einsehbar). Der Audit-Trail steht in `data/review_report.json` und in
|
|
||||||
der Git-Historie.
|
|
||||||
|
|
||||||
## Lokal testen
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
python3 -m venv .venv
|
# CLI-Tool für Monorepo-Verwaltung
|
||||||
source .venv/bin/activate
|
cd shared/tools/monorepo-cli
|
||||||
pip install -r requirements.txt
|
pip install -e .
|
||||||
|
ctx-guard --help
|
||||||
ruff check src tests # Lint
|
|
||||||
python -m pytest -q # Offline-Tests
|
|
||||||
python -m src.main --config config/tools.yaml --data data # Live-Lauf
|
|
||||||
python -m src.chunk --data data # Chunks erzeugen (offline, optional)
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Confluence aktivieren
|
## Sicherheit
|
||||||
|
|
||||||
```bash
|
- Secrets werden via git-crypt verschlüsselt (pro Kontext eigener Schlüssel)
|
||||||
export CONFLUENCE_URL="https://arija-confluence.jaas.service.deutschebahn.com"
|
- `.secrets` ist in `.gitignore` — niemals committen
|
||||||
export CONFLUENCE_TOKEN="<PAT>" # Bearer-PAT (Server/DC)
|
- Zugriffskontrolle via `shared/config/access-config.yaml`
|
||||||
pip install atlassian-python-api
|
|
||||||
```
|
|
||||||
|
|
||||||
Ohne Credentials werden Confluence-Quellen sauber uebersprungen.
|
## Föderation
|
||||||
|
|
||||||
## Betrieb (GitLab CI)
|
Bahn-Team, dhive-Team und private Repos können als eigenständige Team-Repos synchronisiert werden (Hub-and-Spoke). Konfiguration: `shared/config/team-repos.yaml`.
|
||||||
|
|
||||||
- `quality`-Stage: `ruff` (Lint) + `gitleaks` (Secret-Scan) bei jedem Push/MR.
|
|
||||||
- `etl`-Stage: **stuendlich Mo–Fr 8–17 Uhr** (Cron `0 8-17 * * 1-5`), baut Wissen,
|
|
||||||
committet `data/` nach `main` (per `GIT_PUSH_TOKEN`); der Commit loest automatisch
|
|
||||||
einen `pages`-Refresh aus.
|
|
||||||
- `pages`-Stage: baut `public/` (Uebersicht, Hilfe, Chatbot-Anschluss, Wissensquellen)
|
|
||||||
fuer GitLab Pages auf `main`.
|
|
||||||
- `preview`-Job (MR, manuell): verarbeitet eine neue/geaenderte Quelle via
|
|
||||||
`--only` und stellt die erzeugten Markdowns als Job-Artefakt bereit (zum Pruefen
|
|
||||||
vor dem Merge). `PREVIEW_ONLY` beim Start auf Tool-Id/Domaene/URL setzen.
|
|
||||||
- **Kundeninfos inkrementell**: `incremental: true` crawlt nur NEUE Detailseiten und
|
|
||||||
ueberschreibt vorhandene nicht – Re-Runs bleiben schnell.
|
|
||||||
- **Confluence inkrementell**: `options: { incremental: true }` prueft pro Seite die
|
|
||||||
Confluence-Versionsnummer (`source_version` im Frontmatter). Unveraenderte Seiten
|
|
||||||
werden uebersprungen, nur geaenderte/neue Seiten neu verarbeitet – spart API-Last.
|
|
||||||
- **Re-Tagging bei Metadaten-Aenderung**: Aenderst du in `tools.yaml`/`general.yaml`
|
|
||||||
nur `tags`/`owners`/`contact` einer Quelle, weicht deren `meta_fingerprint`
|
|
||||||
(im Frontmatter) vom gespeicherten ab. Der naechste inkrementelle Lauf passt dann
|
|
||||||
die bestehenden Dateien dieser Quelle in-place an (nur Frontmatter, kein erneuter
|
|
||||||
Download/API-Call).
|
|
||||||
- **Scope-Wechsel**: Aenderst du den `scope` einer Quelle, wird die Seite neu
|
|
||||||
verarbeitet und an `data/processed/<neuer-scope>/...` geschrieben; die alte Ablage
|
|
||||||
wird nach erfolgreichem Schreiben **automatisch geloescht** (domaenen-intern, damit
|
|
||||||
eine URL, die legitim unter mehreren Domaenen liegt – z.B. eine pathOS-Seite auch
|
|
||||||
unter `web` – nicht versehentlich entfernt wird).
|
|
||||||
- **Domain-Umbenennung**: Die neue Ablage entsteht automatisch, die alte Domain-Ablage
|
|
||||||
bleibt aber stehen (domaenenuebergreifendes Loeschen waere wegen geteilter URLs
|
|
||||||
unsicher). Den alten `data/processed/<scope>/<alte-domain>/`-Ordner daher bei einer
|
|
||||||
Domain-Umbenennung manuell loeschen.
|
|
||||||
- CI/CD-Variablen: `GIT_PUSH_TOKEN`, `CONFLUENCE_URL`, `CONFLUENCE_TOKEN` (Masked).
|
|
||||||
- `scm-info.yaml`: Pflichtdatei fuer DB-GitLab-Compliance.
|
|
||||||
|
|
||||||
## Konfiguration
|
|
||||||
|
|
||||||
- `config/tools.yaml` – Tools, Domaenen, Quellen, Strategien, Scopes, Tags
|
|
||||||
- `config/filter_rules.json` – Blacklist-Keywords, Regex-Redaction
|
|
||||||
- `config/chunking.yaml` – gemeinsame Chunk-Defaults (pro Quelle ueber `options.chunk` ueberschreibbar)
|
|
||||||
|
|
||||||
### Chunking (optional, Default aus)
|
|
||||||
|
|
||||||
Grosse, stark gegliederte Dokumente (v.a. **INB**) werden zusaetzlich zur Voll-Datei in
|
|
||||||
**Chunks entlang der Ueberschriften** zerlegt – ein **abgeleitetes, jederzeit neu
|
|
||||||
erzeugbares** Artefakt unter `data/chunks/<scope>/<domaene>/<docslug>/`. Das Voll-Dokument
|
|
||||||
in `data/processed/` bleibt unangetastet (Parent-Document-Muster): ein Anschliesser waehlt
|
|
||||||
Voll-Dokument, Chunks oder beides. Es werden bewusst nur **deterministische, embedding-freie**
|
|
||||||
Strategien angeboten (`headings | faq | recursive`); semantisches Chunking gehoert in die
|
|
||||||
Vektor-DB des Anschliessers. Jeder Chunk bekommt eine kurze **Contextual-Retrieval**-Zeile
|
|
||||||
(`> Kontext: <Dokument> > <Abschnitt>`). Aktivierung pro Quelle via `options: { chunk: headings }`.
|
|
||||||
|
|
||||||
**Inkrementell (kein Voll-Rebuild):** Chunking laeuft am Ende jedes ETL-Laufs automatisch
|
|
||||||
mit (`src.main`) und ist Teil von `data/`. Pro Dokument wird nur dann neu gechunkt, wenn
|
|
||||||
sich der **Inhalt** (`parent_hash`) ODER die **wirksamen Optionen/Strategie**
|
|
||||||
(`chunk_fingerprint`) geaendert haben - sonst wird das Dokument uebersprungen (keine
|
|
||||||
Datei-Aenderung, kein Git-Churn). Aenderst du also bei einer Quelle z.B. `chunk_level`
|
|
||||||
oder von `headings` auf `recursive`, werden **nur die Dokumente dieser Ablage** neu
|
|
||||||
erzeugt. Wird `chunk` wieder auf `off` gestellt oder ein Voll-Dokument geloescht, werden
|
|
||||||
die zugehoerigen Chunks automatisch entfernt.
|
|
||||||
|
|
||||||
**Feinsteuerung (Default aus, pro Quelle ueberschreibbar):**
|
|
||||||
|
|
||||||
- `chunk_min_doc_chars`: Dokumente unter dieser Zeichenzahl bleiben **ganz** (kleine
|
|
||||||
FAQ-/How-to-Seiten muessen nicht zwingend gechunkt werden).
|
|
||||||
- `chunk_kind`: chunkt nur Dokumente mit passendem `kind` (`""` = alle; `attachment` =
|
|
||||||
nur Anhang-PDFs; `document` = nur Voll-Seiten).
|
|
||||||
|
|
||||||
Beispiel **pathOS** (an der Tool-Tree-Quelle in `config/tools.yaml`): grosse Anhang-PDFs
|
|
||||||
(z.B. EVU-Schnittstellen-Doku, 2,8 MB) werden gechunkt, Seiten und FAQs bleiben ganz.
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
options:
|
|
||||||
incremental: true
|
|
||||||
chunk: headings
|
|
||||||
chunk_kind: attachment # nur Anhang-PDFs chunken
|
|
||||||
chunk_min_doc_chars: 3000 # < 3k Zeichen bleiben ganz (~750 Token)
|
|
||||||
```
|
|
||||||
|
|
||||||
## Version & Changelog
|
|
||||||
|
|
||||||
Aktuelle Version steht in **`VERSION`** (SemVer), die Historie in **`CHANGELOG.md`**
|
|
||||||
(Format „Keep a Changelog"). Beides wird auf der GitLab-Pages-Seite **„Changelog"**
|
|
||||||
angezeigt (Version auch in der Navigationsleiste). Pflege-Regeln:
|
|
||||||
`.kiro/steering/changelog-versioning.md`.
|
|
||||||
|
|||||||
@@ -1,402 +0,0 @@
|
|||||||
SPEC.md
|
|
||||||
1. Executive Summary
|
|
||||||
Product / Initiative Name
|
|
||||||
|
|
||||||
Note-Graph
|
|
||||||
|
|
||||||
Interviewed Stakeholder Role
|
|
||||||
|
|
||||||
Owner, sponsor, project lead, and implementation orchestrator (“AI conductor”)
|
|
||||||
|
|
||||||
Product Type
|
|
||||||
|
|
||||||
New product
|
|
||||||
|
|
||||||
Goal in One Sentence
|
|
||||||
|
|
||||||
Create a personal knowledge infrastructure that enables reliable access to all notes, thoughts, decisions, and contextual information at any time in order to improve memory, prioritization, and project continuity.
|
|
||||||
|
|
||||||
Short Business Context
|
|
||||||
|
|
||||||
The stakeholder operates across many parallel projects, domains, meetings, and customer contexts. Information currently exists across handwritten notes, PDFs, meeting artifacts, task systems, Confluence spaces, and AI-generated summaries. The current process relies heavily on memory and chronological note-taking, causing information loss, weak follow-up tracking, and fragmented context retrieval.
|
|
||||||
|
|
||||||
Primary Target Users
|
|
||||||
Primary: Single knowledge worker / project orchestrator
|
|
||||||
Future potential: Extended collaborative usage (currently out of scope)
|
|
||||||
Core Functional Scope Summary
|
|
||||||
|
|
||||||
The system shall:
|
|
||||||
|
|
||||||
consolidate notes and knowledge artifacts from multiple formats and sources,
|
|
||||||
support flexible tagging and contextual relationships,
|
|
||||||
preserve historical context and traceability,
|
|
||||||
enable fast contextual retrieval,
|
|
||||||
connect notes with tasks, people, projects, and decisions,
|
|
||||||
support gradual AI-assisted enrichment,
|
|
||||||
maintain portability and data ownership.
|
|
||||||
In Scope
|
|
||||||
Personal knowledge management
|
|
||||||
Cross-project contextual search
|
|
||||||
Flexible labels/tags
|
|
||||||
Relationships between notes, projects, people, tasks, and decisions
|
|
||||||
Integration of handwritten notes and document-based sources
|
|
||||||
Offline-capable note capture
|
|
||||||
Mobile access
|
|
||||||
Traceability between tasks and source context
|
|
||||||
Export and publication workflows
|
|
||||||
AI-assisted enrichment and retrieval
|
|
||||||
Out of Scope
|
|
||||||
Full project management platform
|
|
||||||
CRM functionality
|
|
||||||
Team collaboration platform
|
|
||||||
Chat/messaging system
|
|
||||||
Calendar replacement
|
|
||||||
Complex automation workflows
|
|
||||||
Autonomous AI agents
|
|
||||||
Advanced graph visualization in first phase
|
|
||||||
Semantic contradiction detection in first phase
|
|
||||||
Main Open Questions
|
|
||||||
Degree of automation vs manual curation
|
|
||||||
How granular note segmentation should become in practice
|
|
||||||
How historical relationship changes should be surfaced to users
|
|
||||||
Scope of AI-generated suggestions in early versions
|
|
||||||
Exact publication/export workflows
|
|
||||||
Main Acceptance Indicators
|
|
||||||
Relevant information can be retrieved during meetings.
|
|
||||||
Meeting preparation becomes significantly easier.
|
|
||||||
Contextual information can be found reliably across projects and time.
|
|
||||||
Tasks remain traceable to their originating notes and decisions.
|
|
||||||
2. Goal
|
|
||||||
One-Sentence Goal
|
|
||||||
|
|
||||||
Provide a durable, searchable, context-aware personal knowledge system that acts as an external memory and decision-support environment.
|
|
||||||
|
|
||||||
Detailed Goal
|
|
||||||
|
|
||||||
The system shall centralize fragmented knowledge sources and allow the stakeholder to:
|
|
||||||
|
|
||||||
capture ideas quickly,
|
|
||||||
retain project continuity,
|
|
||||||
retrieve relevant context rapidly,
|
|
||||||
preserve historical relationships,
|
|
||||||
connect operational tasks with their originating discussions and decisions,
|
|
||||||
maintain structure despite frequent context switching.
|
|
||||||
Business Value
|
|
||||||
Reduced information loss
|
|
||||||
Better prioritization
|
|
||||||
Faster meeting preparation
|
|
||||||
Improved continuity across projects
|
|
||||||
Reduced cognitive load
|
|
||||||
Better follow-through on decisions and tasks
|
|
||||||
Long-term retention of institutional and personal knowledge
|
|
||||||
Intended Outcome for Users and Organization
|
|
||||||
|
|
||||||
Users should be able to trust the system as a reliable external memory and operational context assistant.
|
|
||||||
|
|
||||||
3. Stakeholder Context
|
|
||||||
Respondent Role
|
|
||||||
Owner
|
|
||||||
Sponsor
|
|
||||||
Project lead
|
|
||||||
Implementation orchestrator (“AI conductor”)
|
|
||||||
Perspective Represented
|
|
||||||
|
|
||||||
Single stakeholder representing both business and implementation direction.
|
|
||||||
|
|
||||||
Relevant Organizational or Customer Context
|
|
||||||
|
|
||||||
The stakeholder works across multiple projects, domains, customers, and meeting contexts with heterogeneous documentation ecosystems.
|
|
||||||
|
|
||||||
Whether This Spec Reflects One Interview Only
|
|
||||||
|
|
||||||
Yes. This specification reflects a single interview.
|
|
||||||
|
|
||||||
4. Target Users
|
|
||||||
User Groups
|
|
||||||
Primary User
|
|
||||||
|
|
||||||
Knowledge worker managing many parallel projects and information streams.
|
|
||||||
|
|
||||||
Potential Future Users
|
|
||||||
|
|
||||||
Additional collaborators or project participants (currently out of scope).
|
|
||||||
|
|
||||||
User Context
|
|
||||||
|
|
||||||
Users frequently:
|
|
||||||
|
|
||||||
switch contexts,
|
|
||||||
attend meetings,
|
|
||||||
create handwritten and digital notes,
|
|
||||||
consume large document collections,
|
|
||||||
derive tasks from discussions,
|
|
||||||
revisit historical decisions.
|
|
||||||
User Needs
|
|
||||||
|
|
||||||
Users need to:
|
|
||||||
|
|
||||||
retrieve information quickly,
|
|
||||||
preserve context,
|
|
||||||
avoid forgetting commitments,
|
|
||||||
connect ideas across projects,
|
|
||||||
maintain operational continuity,
|
|
||||||
understand historical decisions.
|
|
||||||
Pain Points
|
|
||||||
Media breaks between tools and formats
|
|
||||||
Difficulty finding information later
|
|
||||||
Fragmented storage locations
|
|
||||||
Reliance on memory
|
|
||||||
Missing conversion from notes to tasks
|
|
||||||
Weak cross-project traceability
|
|
||||||
Loss of contextual continuity
|
|
||||||
Important Differences Between Groups
|
|
||||||
|
|
||||||
No differentiated user groups are currently defined.
|
|
||||||
|
|
||||||
5. Current Situation / Current Process
|
|
||||||
Current Process
|
|
||||||
|
|
||||||
Information currently exists across:
|
|
||||||
|
|
||||||
handwritten meeting notes,
|
|
||||||
PDFs,
|
|
||||||
Confluence spaces,
|
|
||||||
task management tools,
|
|
||||||
AI-assisted document search tools,
|
|
||||||
digital note collections.
|
|
||||||
|
|
||||||
Meeting notes are often incrementally extended over time and searched manually using:
|
|
||||||
|
|
||||||
project names,
|
|
||||||
dates,
|
|
||||||
people,
|
|
||||||
remembered context.
|
|
||||||
Existing Workarounds
|
|
||||||
Manual search
|
|
||||||
AI-assisted document retrieval
|
|
||||||
Chronological note accumulation
|
|
||||||
Personal memory and contextual recall
|
|
||||||
Main Gaps, Bottlenecks, and Failure Points
|
|
||||||
Media discontinuity
|
|
||||||
Information fragmentation
|
|
||||||
Weak traceability
|
|
||||||
Missing task extraction
|
|
||||||
Incomplete prioritization
|
|
||||||
Context loss across projects
|
|
||||||
Inability to reliably recover ideas generated in unrelated contexts
|
|
||||||
What Should Remain Unchanged
|
|
||||||
Existing working style should remain largely compatible.
|
|
||||||
Notes may continue to originate from different tools and formats.
|
|
||||||
6. Functional Scope
|
|
||||||
Main Capabilities
|
|
||||||
|
|
||||||
The system shall:
|
|
||||||
|
|
||||||
ingest and organize knowledge artifacts,
|
|
||||||
support contextual relationships,
|
|
||||||
preserve historical information,
|
|
||||||
enable retrieval across projects and time,
|
|
||||||
support flexible tagging and labeling,
|
|
||||||
connect operational tasks to source knowledge,
|
|
||||||
support gradual AI-assisted enrichment.
|
|
||||||
Key User Interactions
|
|
||||||
|
|
||||||
Users shall be able to:
|
|
||||||
|
|
||||||
capture notes quickly,
|
|
||||||
associate notes with multiple projects,
|
|
||||||
associate notes with people and topics,
|
|
||||||
split notes into smaller contextual fragments,
|
|
||||||
search by content, project, person, or time,
|
|
||||||
review historical decisions,
|
|
||||||
export or publish derived content,
|
|
||||||
revisit unresolved items.
|
|
||||||
Major Functional Areas
|
|
||||||
Knowledge capture
|
|
||||||
Knowledge organization
|
|
||||||
Search and retrieval
|
|
||||||
Relationship management
|
|
||||||
Task traceability
|
|
||||||
Historical context management
|
|
||||||
AI-assisted enrichment
|
|
||||||
Export/publication support
|
|
||||||
Role-Based Differences in Usage
|
|
||||||
|
|
||||||
No differentiated role behavior currently defined.
|
|
||||||
|
|
||||||
Relevant Triggers, Inputs, Outputs, and Outcomes
|
|
||||||
Inputs
|
|
||||||
Handwritten notes
|
|
||||||
PDFs
|
|
||||||
Markdown files
|
|
||||||
Office documents
|
|
||||||
Meeting artifacts
|
|
||||||
AI-generated summaries
|
|
||||||
Outputs
|
|
||||||
Search results
|
|
||||||
Structured contextual views
|
|
||||||
Exported content
|
|
||||||
Task references
|
|
||||||
Meeting preparation material
|
|
||||||
Outcomes
|
|
||||||
Faster information access
|
|
||||||
Improved continuity
|
|
||||||
Better prioritization
|
|
||||||
Reduced forgotten tasks and decisions
|
|
||||||
7. Business Rules
|
|
||||||
Notes may belong to multiple projects simultaneously.
|
|
||||||
Relationships between entities may evolve over time.
|
|
||||||
Historical assignments and timestamps must remain preserved.
|
|
||||||
Original notes and source information must not be lost.
|
|
||||||
Tasks must remain traceable to originating notes or decisions.
|
|
||||||
Structured relationships and free-form associations must coexist.
|
|
||||||
Offline capture capability is required.
|
|
||||||
Mobile access is required.
|
|
||||||
Users must retain control over AI-generated enrichments or suggestions.
|
|
||||||
Relationship interpretation may change over time without overwriting historical truth.
|
|
||||||
8. Business Objects / Functional Data Objects
|
|
||||||
Note
|
|
||||||
|
|
||||||
Represents a captured thought, meeting artifact, observation, or written content.
|
|
||||||
|
|
||||||
Thought Fragment
|
|
||||||
|
|
||||||
Represents a smaller extracted or separable unit derived from a larger note.
|
|
||||||
|
|
||||||
Project
|
|
||||||
|
|
||||||
Represents a business or operational context grouping related knowledge.
|
|
||||||
|
|
||||||
Person
|
|
||||||
|
|
||||||
Represents individuals connected to notes, tasks, meetings, or decisions.
|
|
||||||
|
|
||||||
Task
|
|
||||||
|
|
||||||
Represents actionable follow-up work derived from notes or decisions.
|
|
||||||
|
|
||||||
Decision
|
|
||||||
|
|
||||||
Represents an explicitly recorded outcome or conclusion.
|
|
||||||
|
|
||||||
Source Document
|
|
||||||
|
|
||||||
Represents imported or referenced material such as PDFs, Office documents, or AI-generated summaries.
|
|
||||||
|
|
||||||
Label / Tag
|
|
||||||
|
|
||||||
Represents flexible classification and contextual grouping metadata.
|
|
||||||
|
|
||||||
Relationship
|
|
||||||
|
|
||||||
Represents contextual connections between business objects.
|
|
||||||
|
|
||||||
Time Context
|
|
||||||
|
|
||||||
Represents historical assignment and chronological relevance.
|
|
||||||
|
|
||||||
9. In Scope
|
|
||||||
Personal knowledge graph functionality
|
|
||||||
Multi-project note association
|
|
||||||
Flexible tagging and labeling
|
|
||||||
Contextual relationships
|
|
||||||
Historical traceability
|
|
||||||
Search by text, project, person, and time
|
|
||||||
Mobile access
|
|
||||||
Offline usage
|
|
||||||
Handwritten note integration
|
|
||||||
AI-assisted retrieval and enrichment
|
|
||||||
Export and publishing workflows
|
|
||||||
Task traceability to source context
|
|
||||||
Gradual enrichment through review/question workflows
|
|
||||||
10. Out of Scope
|
|
||||||
Enterprise collaboration platform
|
|
||||||
Full project management suite
|
|
||||||
CRM capabilities
|
|
||||||
Messaging/chat platform
|
|
||||||
Calendar replacement
|
|
||||||
Autonomous AI workflows
|
|
||||||
Complex process automation
|
|
||||||
Advanced graph visualizations in first release
|
|
||||||
Automatic contradiction analysis in first release
|
|
||||||
Fully automated prioritization
|
|
||||||
11. Requirements
|
|
||||||
Knowledge Capture
|
|
||||||
The system shall allow users to capture notes quickly.
|
|
||||||
The system shall support offline note capture.
|
|
||||||
The system shall support mobile access.
|
|
||||||
The system shall support handwritten note integration.
|
|
||||||
The system shall support importing external document sources.
|
|
||||||
Knowledge Organization
|
|
||||||
The system shall allow notes to belong to multiple projects simultaneously.
|
|
||||||
The system shall support labels and tags.
|
|
||||||
The system shall support both structured and free-form relationships.
|
|
||||||
The system shall allow users to split notes into smaller contextual fragments.
|
|
||||||
The system shall preserve relationships between fragmented content and original source notes.
|
|
||||||
Search and Retrieval
|
|
||||||
The system shall provide full-text search.
|
|
||||||
The system shall support contextual retrieval across projects and relationships.
|
|
||||||
The system shall support search by people.
|
|
||||||
The system shall support search by project.
|
|
||||||
The system shall support time-based navigation and retrieval.
|
|
||||||
The system shall allow users to retrieve meeting-relevant information quickly.
|
|
||||||
Task Traceability
|
|
||||||
The system shall support linking tasks to source notes or decisions.
|
|
||||||
The system shall preserve the origin context of tasks.
|
|
||||||
The system shall support identifying missing task metadata such as priorities or deadlines.
|
|
||||||
Historical Integrity
|
|
||||||
The system shall preserve original notes and timestamps.
|
|
||||||
The system shall preserve historical relationship assignments over time.
|
|
||||||
The system shall allow contextual relationships to evolve without losing historical information.
|
|
||||||
AI Assistance
|
|
||||||
The system shall support AI-assisted search and summarization.
|
|
||||||
The system shall support AI-assisted enrichment and categorization.
|
|
||||||
The system shall support gradual future expansion of AI capabilities.
|
|
||||||
The system shall support review or quiz-style interactions for knowledge enrichment.
|
|
||||||
Portability and Ownership
|
|
||||||
The system shall support export of information and derived content.
|
|
||||||
The system shall minimize dependency on proprietary structures.
|
|
||||||
The system shall preserve user data ownership.
|
|
||||||
The system shall support long-term portability and migration.
|
|
||||||
12. Open Questions / Items to Clarify
|
|
||||||
Desired level of automatic relationship generation
|
|
||||||
Exact workflow for exporting/publishing to external systems
|
|
||||||
Preferred granularity for note fragmentation
|
|
||||||
Scope of AI-generated task extraction
|
|
||||||
Desired review cadence for enrichment workflows
|
|
||||||
Long-term collaboration requirements
|
|
||||||
Definition of archival versus active information
|
|
||||||
Rules for conflict resolution between changing contextual assignments
|
|
||||||
13. Risks and Ambiguities
|
|
||||||
Excessive maintenance effort may reduce adoption.
|
|
||||||
Weak search quality could undermine trust in the system.
|
|
||||||
Too many manual steps may interrupt workflows.
|
|
||||||
Incorrect AI-generated relationships may reduce confidence.
|
|
||||||
Information overload may reduce usability.
|
|
||||||
Overly rigid structures may conflict with real-world usage patterns.
|
|
||||||
Insufficient offline support may limit operational usefulness.
|
|
||||||
Lack of clear contextual boundaries may create organizational confusion.
|
|
||||||
Future migration requirements may conflict with short-term optimizations.
|
|
||||||
14. Acceptance Perspective
|
|
||||||
|
|
||||||
The product will be considered successful if:
|
|
||||||
|
|
||||||
users can reliably retrieve relevant information during meetings,
|
|
||||||
meeting preparation becomes significantly easier,
|
|
||||||
contextual information can be recovered across projects and time,
|
|
||||||
tasks remain traceable to their origin,
|
|
||||||
users trust the system as an external memory,
|
|
||||||
the system reduces forgotten follow-ups and lost context,
|
|
||||||
users can maintain organization despite frequent context switching,
|
|
||||||
knowledge remains portable and under user control.
|
|
||||||
15. Glossary
|
|
||||||
Original Term English Explanation Notes / Context
|
|
||||||
Notiz-Graph Note graph / knowledge graph system Working title of the product
|
|
||||||
KI-Dirigent AI orchestrator Stakeholder role coordinating AI-supported workflows
|
|
||||||
Confluence Knowledge collaboration platform Existing documentation source
|
|
||||||
NotebookLM AI-supported document knowledge tool Existing AI-generated summary source
|
|
||||||
Labels Context labels Used for structured assignment
|
|
||||||
Tags Flexible metadata tags Used for contextual classification
|
|
||||||
Todo-App Task management application External task system
|
|
||||||
Medienbruch Media discontinuity Fragmentation between formats/tools
|
|
||||||
Gedächtnisstütze Memory aid Contextual reminder mechanism
|
|
||||||
Binary file not shown.
@@ -1 +0,0 @@
|
|||||||
{"device_id": "b34feb7e-2c5f-4308-a657-a20a24547171", "device_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdXRoMC11c2VyaWQiOiJhdXRoMHw2NjBkNzM2NWEyZDYzZjBmMmIyOGIwNGIiLCJkZXZpY2UtZGVzYyI6ImRlc2t0b3AtbGludXgiLCJkZXZpY2UtaWQiOiJiMzRmZWI3ZS0yYzVmLTQzMDgtYTY1Ny1hMjBhMjQ1NDcxNzEiLCJpYXQiOjE3NzkzNDc1OTEsImlzcyI6InJNIFdlYkFwcCIsImp0aSI6ImNrMHRUUkZvVXhzPSIsIm5iZiI6MTc3OTM0NzU5MSwic3ViIjoick0gRGV2aWNlIFRva2VuIn0.29K8KBvJynjJyDRpnd36LvcG8-wXwA7MBweaXE0ocmY"}
|
|
||||||
Reference in New Issue
Block a user