diff --git a/AGENTS.md b/AGENTS.md index b1715ca..900aeb0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -130,19 +130,7 @@ The repository has the following structure with specific permissions: ```text fiona-assistant/ -├── llm-wiki/ # Wiki content (see llm-wiki/AGENTS.md) -│ ├── personal/ # READ-ONLY - Never modify -│ ├── ingest/ # Input zone - parse files here -│ ├── wiki/ # READ-WRITE - synthesized content -│ │ ├── concepts/ # Processed concept notes -│ │ ├── raw/ # READ-ONLY - archived sources -│ │ │ ├── articles/ -│ │ │ ├── papers/ -│ │ │ ├── transcripts/ -│ │ │ └── assets/ -│ │ ├── index.md # Master index - update on changes -│ │ └── log.md # Action log - append all actions -│ └── utils/ # Utility scripts +├── llm-wiki/* # see `llm-wiki/wiki/schema.md` for structure │ ├── src/ # Source code │ ├── fiona/ # Main CLI package diff --git a/src/llm-wiki/AGENTS.md b/src/llm-wiki/AGENTS.md index d2428c7..282fc8f 100644 --- a/src/llm-wiki/AGENTS.md +++ b/src/llm-wiki/AGENTS.md @@ -16,13 +16,14 @@ - **`/personal/`**: Strictly **READ-ONLY**. Never create, edit, or delete notes here. - **`llm-wiki/ingest/`**: Input processing zone. Parse files dropped here into concept notes. - **`llm-wiki/wiki/`**: **READ-WRITE**. Place synthesized concept pages in `/wiki/concepts/`. - - **Raw files** (e.g., articles, papers, transcripts) must be stored in `/wiki/raw/` and **never edited**. - - Subdirectories: - - `/wiki/raw/articles/` (web articles, clippings) - - `/wiki/raw/papers/` (PDFs, arXiv papers) - - `/wiki/raw/transcripts/` (meeting notes, interviews) - - `/wiki/raw/assets/` (images, diagrams referenced by sources) - Only parsed/synthesized content belongs in `/wiki/concepts/`. +- **`llm-wiki/raw/`**: **READ-ONLY**. Raw source archive, never edited. + - **Raw files** (e.g., articles, papers, transcripts) must be stored in `/raw/` and **never edited**. + - Subdirectories: + - `/raw/articles/` (web articles, clippings) + - `/raw/papers/` (PDFs, arXiv papers) + - `/raw/transcripts/` (meeting notes, interviews) + - `/raw/assets/` (images, diagrams referenced by sources) - **`llm-wiki/utils/scripts`**: **READ-WRITE**. Place any script used and created here. diff --git a/src/llm-wiki/wiki/raw/articles/.gitkeep b/src/llm-wiki/raw/articles/.gitkeep similarity index 100% rename from src/llm-wiki/wiki/raw/articles/.gitkeep rename to src/llm-wiki/raw/articles/.gitkeep diff --git a/src/llm-wiki/wiki/raw/assets/.gitkeep b/src/llm-wiki/raw/assets/.gitkeep similarity index 100% rename from src/llm-wiki/wiki/raw/assets/.gitkeep rename to src/llm-wiki/raw/assets/.gitkeep diff --git a/src/llm-wiki/wiki/raw/papers/.gitkeep b/src/llm-wiki/raw/papers/.gitkeep similarity index 100% rename from src/llm-wiki/wiki/raw/papers/.gitkeep rename to src/llm-wiki/raw/papers/.gitkeep diff --git a/src/llm-wiki/wiki/raw/transcripts/.gitkeep b/src/llm-wiki/raw/transcripts/.gitkeep similarity index 100% rename from src/llm-wiki/wiki/raw/transcripts/.gitkeep rename to src/llm-wiki/raw/transcripts/.gitkeep diff --git a/src/llm-wiki/utils/skills/fiona/llm-wiki-mod.md b/src/llm-wiki/utils/skills/fiona/llm-wiki-mod.md index fb9d128..829e068 100644 --- a/src/llm-wiki/utils/skills/fiona/llm-wiki-mod.md +++ b/src/llm-wiki/utils/skills/fiona/llm-wiki-mod.md @@ -75,7 +75,7 @@ When the user asks to create or start a wiki: 1. Determine the wiki path (from `$WIKI_PATH` env var, or ask the user; default `~/llm-wiki`) 2. Create the directory structure above, including: - `llm-wiki/ingest/` (input processing zone) - - `llm-wiki/wiki/raw/` subdirectories (`articles/`, `papers/`, `transcripts/`, `assets/`) + - `llm-wiki/raw/` subdirectories (`articles/`, `papers/`, `transcripts/`, `assets/`) - `llm-wiki/utils/` subdirectories (`scripts/`, `skills/`, `skills_local/`, `personas/`) 3. Ask the user what domain the wiki covers — be specific 4. Write `SCHEMA.md` customized to the domain (see template below) @@ -236,10 +236,10 @@ a `_meta/topic-map.md` that groups pages by theme for faster navigation. When the user provides a source (URL, file, paste), integrate it into the wiki: ① **Capture the raw source:** - - URL → use `web_extract` to get markdown, save to `wiki/raw/articles/` - - PDF → use `web_extract` (handles PDFs), save to `wiki/raw/papers/` - - Pasted text → save to appropriate `wiki/raw/` subdirectory (e.g., `transcripts/`) - - Name the file descriptively: `wiki/raw/articles/karpathy-llm-wiki-2026.md` + - URL → use `web_extract` to get markdown, save to `raw/articles/` + - PDF → use `web_extract` (handles PDFs), save to `raw/papers/` + - Pasted text → save to appropriate `raw/` subdirectory (e.g., `transcripts/`) + - Name the file descriptively: `raw/articles/karpathy-llm-wiki-2026.md` - **Add raw frontmatter** (`source_url`, `ingested`, `sha256` of the body). On re-ingest of the same URL: recompute the sha256, compare to the stored value — skip if identical, flag drift and update if different. This is cheap enough to @@ -326,9 +326,9 @@ wiki = "" only a single source but has no confidence field set — these are candidates for either finding corroboration or demoting to `confidence: medium`. -- **Source drift:** For each file in `wiki/raw/` with a `sha256:` frontmatter, recompute +- **Source drift:** For each file in `raw/` with a `sha256:` frontmatter, recompute the hash and flag mismatches. Mismatches indicate the raw file was edited - (shouldn't happen — `wiki/raw/` is immutable) or ingested from a URL that has since + (shouldn't happen — `raw/` is immutable) or ingested from a URL that has since changed. Not a hard error, but worth reporting. ⑨ **Page size:** Flag pages over 200 lines — candidates for splitting. diff --git a/src/llm-wiki/wiki/log.md b/src/llm-wiki/wiki/log.md index 3c27760..847789c 100644 --- a/src/llm-wiki/wiki/log.md +++ b/src/llm-wiki/wiki/log.md @@ -8,3 +8,9 @@ ## [YYYY-MM-DD] create | Wiki initialized - Domain: Personal knowledge base and inventory management - Structure created with schema.md, index.md + +## [2026-09-23] update | Deduplicated raw source zone + +- Moved `wiki/raw/{articles,papers,transcripts,assets}/` to `raw/` at the root +- `llm-wiki/raw/` is the single canonical read-only source archive; `wiki/raw/` removed +- Aligned schema.md, llm-wiki/AGENTS.md, root AGENTS.md, and the fiona skill diff --git a/src/llm-wiki/wiki/schema.md b/src/llm-wiki/wiki/schema.md index 276a1ec..721eab2 100644 --- a/src/llm-wiki/wiki/schema.md +++ b/src/llm-wiki/wiki/schema.md @@ -57,7 +57,11 @@ contradictions: [other-page-slug] llm-wiki/ ├── ingest/ # Input processing zone (raw files to parse) ├── personal/ # Raw Notes from the user (**READ ONLY**) -├── raw/ # Immutable source material archive +├── raw/ # Layer 1: Immutable source material archive (NEVER EDIT) +│ ├── articles/ # Web articles, clippings +│ ├── papers/ # PDFs, arXiv papers +│ ├── transcripts/ # Meeting notes, interviews +│ └── assets/ # Images, diagrams referenced by sources ├── utils/ │ ├── personas/ # Agent personas configurations │ ├── scripts/ # Reusable utility scripts @@ -67,11 +71,6 @@ llm-wiki/ │ ├── SCHEMA.md # (This file) Conventions, structure rules, domain config │ ├── index.md # Sectioned content catalog with one-line summaries │ ├── log.md # Chronological action log (append-only, rotated yearly) -│ ├── raw/ # Layer 1: Immutable source material (NEVER EDIT) -│ │ ├── articles/ # Web articles, clippings -│ │ ├── papers/ # PDFs, arXiv papers -│ │ ├── transcripts/ # Meeting notes, interviews -│ │ └── assets/ # Images, diagrams referenced by sources │ ├── concepts/ # Layer 2: Synthesized concept notes │ ├── entities/ # Layer 2: Entity pages (people, orgs, products, models) │ ├── comparisons/ # Layer 2: Side-by-side analyses @@ -79,7 +78,8 @@ llm-wiki/ └── table_of_contents.md # Auto-generated navigation index for quick lookup ``` -- **Layer 1 — Raw Sources:** Immutable. The agent reads but never modifies these. +- **Layer 1 — Raw Sources:** `raw/` at the wiki root. The agent reads but never + modifies these. - **Layer 2 — The Wiki:** Agent-owned markdown files. Created, updated, and cross-referenced by the agent. - **Layer 3 — The Schema:** `SCHEMA.md` defines structure, conventions, and tag taxonomy. diff --git a/todo.md b/todo.md index 878ac72..da77057 100644 --- a/todo.md +++ b/todo.md @@ -35,10 +35,10 @@ block a correct install or make the docs contradict the code. - [ ] Re-verify: build a wheel, install it fresh, confirm the `fiona-assistant` console script runs and the llm-wiki template data ships -- [ ] Resolve the duplicated `raw/` zone in the llm-wiki template - - [ ] Pick one canonical location (`wiki/raw/` per the AGENTS.md - diagrams) and remove `src/llm-wiki/raw/` - - [ ] Align root `AGENTS.md`, `src/llm-wiki/AGENTS.md`, and +- [x] Resolve the duplicated `raw/` zone in the llm-wiki template + - [x] Pick one canonical location (`llm-wiki/raw/` at the wiki root, + per user decision) and remove `src/llm-wiki/wiki/raw/` + - [x] Align root `AGENTS.md`, `src/llm-wiki/AGENTS.md`, and `wiki/schema.md` on the same location - [ ] Reconcile the two competing frontmatter schemas - [ ] Decide between AGENTS.md Rule 8 (`name`/`version`/`title`) and @@ -58,6 +58,11 @@ block a correct install or make the docs contradict the code. - [ ] Remove `src/llm-wiki/utils/scripts/example.py` — `wiki init` skips `.py` files, so it can never reach user wikis - [ ] Deduplicate conftest: `src/conftest.py` vs `src/tests/conftest.py` +- [ ] `run_sanity.py` run fails, need fixing + - [ ] run sanity should show what exactly to fix + - [ ] pytests should show which test are failing, + - [ ] maskdownlint should show which files are in violation , what line + - [ ] same thing for ruff ## P1 — Wiki Lifecycle and Version Control