From 3c212179703faa0f83e5661cbbee796d57f03bac Mon Sep 17 00:00:00 2001 From: nvaldez Date: Fri, 2 Oct 2026 12:14:57 +0200 Subject: [PATCH] librarian move specifications --- SPEC.md | 7220 ++++++++++++++++++++++++++++--------------------------- todo.md | 142 ++ 2 files changed, 3827 insertions(+), 3535 deletions(-) diff --git a/SPEC.md b/SPEC.md index 9011581..34aed3b 100644 --- a/SPEC.md +++ b/SPEC.md @@ -1,3535 +1,3685 @@ ---- -name: Fiona-Assistant Specification -version: 1.5.2 -title: Project Spec Sheet -created: 2026-09-23 -updated: 2026-10-02 -type: summary ---- - -# Fiona-Assistant Specification - - - -- [1. Product Overview](#1-product-overview) - - [1.1. Document Conventions](#11-document-conventions) - - [1.2. Constraints](#12-constraints) -- [2. Package and Runtime](#2-package-and-runtime) -- [3. llm-wiki Template](#3-llm-wiki-template) - - [3.1. What Counts as a Wiki](#31-what-counts-as-a-wiki) - - [3.2. Core Files and Read-only Zones](#32-core-files-and-read-only-zones) - - [3.3. Deployed Wiki Rules](#33-deployed-wiki-rules) -- [4. Command Catalog](#4-command-catalog) - - [4.1. librarian](#41-librarian) - - [4.1.1. librarian toc](#411-librarian-toc) - - [4.2. mdoctor](#42-mdoctor) - - [4.3. fmlint](#43-fmlint) - - [4.4. python-ruff](#44-python-ruff) - - [4.5. json-lint](#45-json-lint) - - [4.6. git](#46-git) - - [4.7. wiki](#47-wiki) - - [4.7.1. wiki init](#471-wiki-init) - - [4.7.2. wiki check-update](#472-wiki-check-update) - - [4.7.3. wiki update](#473-wiki-update) - - [4.7.4. wiki sanity](#474-wiki-sanity) - - [4.7.5. wiki graph](#475-wiki-graph) - - [4.7.6. wiki ingest](#476-wiki-ingest) - - [4.8. housekeeping](#48-housekeeping) - - [4.9. Repository runners](#49-repository-runners) -- [5. Settings Schema](#5-settings-schema) - - [5.1. fiona-settings.json](#51-fiona-settingsjson) - - [5.2. settings_dev.json](#52-settings_devjson) - - [5.3. constellation themes](#53-constellation-themes) -- [6. Violation Rule Catalog](#6-violation-rule-catalog) - - [6.1. Shared Report](#61-shared-report) - - [6.2. Frontmatter Fields and FML Codes](#62-frontmatter-fields-and-fml-codes) - - [6.3. MDL Codes](#63-mdl-codes) - - [6.4. PYC Codes](#64-pyc-codes) - - [6.5. JSN Codes](#65-jsn-codes) - - [6.6. Base Linter Contract](#66-base-linter-contract) -- [7. Feature Requirements](#7-feature-requirements) - - [7.1. wiki](#71-wiki) - - [7.1.1. wiki init](#711-wiki-init) - - [7.1.2. wiki update](#712-wiki-update) - - [7.1.3. wiki sanity](#713-wiki-sanity) - - [7.1.4. wiki graph](#714-wiki-graph) - - [7.1.5. wiki ingest](#715-wiki-ingest) - - [7.2. housekeeping](#72-housekeeping) - - [7.3. librarian](#73-librarian) - - [7.3.1. Folder and In-file TOC](#731-folder-and-in-file-toc) - - [7.3.2. Non-markdown Version Tags](#732-non-markdown-version-tags) - - [7.4. mdoctor](#74-mdoctor) - - [7.5. fmlint](#75-fmlint) - - [7.6. mdlinter](#76-mdlinter) - - [7.7. pycustomlint](#77-pycustomlint) - - [7.8. python-ruff](#78-python-ruff) - - [7.9. Shellcheck](#79-shellcheck) - - [7.10. Markdown Code Blocks](#710-markdown-code-blocks) - - [7.11. json-lint](#711-json-lint) - - [7.12. Git Operations](#712-git-operations) - - [7.13. Backup Script](#713-backup-script) - - [7.14. Test and Sanity Runners](#714-test-and-sanity-runners) - - [7.15. html-js-lint](#715-html-js-lint) - - [7.16. settings setup](#716-settings-setup) - - [7.17. Logging](#717-logging) - - [7.18. Documentation](#718-documentation) - - [7.19. Tool Roles](#719-tool-roles) - - [7.20. Wiki Root Discovery](#720-wiki-root-discovery) - - [7.21. Bootstrap install modes](#721-bootstrap-install-modes) -- [Appendix A. Non-Goals](#appendix-a-non-goals) -- [Appendix B. Priority Order](#appendix-b-priority-order) -- [Appendix C. Open Questions](#appendix-c-open-questions) -- [Appendix D. Maintainer Notes](#appendix-d-maintainer-notes) - - - -This document is the build contract for Fiona-Assistant. A reader with -this file, and no source tree, must be able to produce the same -observable software. Status values are `Implemented`, `Partially -Implemented`, `Planned`, and `Won't Do`. - -Revision 0.40.0 reorganizes the sheet into contracts. Revision 0.40.1 -numbers nested requirements such as `7.14.1.1`. Revision 0.40.4 adds -the PYC001 exception in *6.4.2.*. Revision 0.41.0 installs one -`fiona` package and keeps tests and repo scripts outside it. -Revision 0.90.0 specifies `node_importance_min_point` in *7.1.4.16.* -and leaves that requirement unbuilt. Revision 0.92.0 specifies -`wiki ingest` in *7.1.5.* and leaves it unbuilt. Revision 0.93.0 adds -SHA-256 reuse across `raw/`, the complete `ingest` frontmatter map, -and the agent metadata check. Revision 0.94.0 makes `wiki update` -yield to the owner's markdown and write `-update.md` instead -(*7.1.2.11.*), and adds `fiona-assistant housekeeping` (*7.2.*). -Revision 0.96.0 compares a file without a version by SHA-256 and -shows both short hashes in the update report (*7.1.2.12.*). -Revision 0.97.0 rejects a module-level assignment that follows the -first function or class (*6.4.3.*). Revision 0.98.0 specifies that -`wiki graph` omits a file matched by `.gitignore` (*7.1.4.17.*) and -leaves that requirement unbuilt. Revision 0.99.0 builds `wiki ingest` -(*7.1.5.*). Revision 1.0.0 groups each module's functions under that -module and moves Non-Goals, Priority Order, and Open Questions -to Appendices A, B, and C. Revision 1.1.0 treats a differing -`fiona-settings.json` as a conflict and writes -`fiona-settings-update.json` (*7.1.2.3.*). Revision 1.2.0 folds -text line endings before the SHA-256 compare (*7.1.2.13.*) and -writes `corefiles.json` so a later update does not list a -core file whose version and hashes are unchanged (*7.1.2.14.*). -Revision 1.3.0 specifies directory nodes (*7.1.4.18.*) and -`wiki graph --serve` (*7.1.4.19.*) and leaves both unbuilt. -Revision 1.4.0 names the core-file record `corefiles.json` -(*7.1.2.14.*) and skips paths in `fiona.wiki.update.ignore_files` -(*7.1.2.15.*). The shipped list names `corefiles.json`. Only -`wiki update` writes that file. Revision 1.5.0 omits markdown -files matched by `fiona.wiki.graph.ignore_files` from the graph -(*7.1.4.20.*). The shipped list is `*/table_of_contents.md`, -because a table of contents links to every page. Revision 1.5.2 -gives that file frontmatter `name` and `version` (*7.3.1.1.1.*) -so `fmlint` accepts it. - -Each feature section starts with a `Status` line. Requirements use -`- *N.M.K.*`. Each `*acceptance*` sits under the requirement it tests. -`NOT_IMPLEMENTED` marks work that is specified and not built yet. -Class names, file paths, and history live in Appendix D. - ---- - -## 1. Product Overview - -Fiona-Assistant is a CLI toolkit that manages, organizes, and checks -LLM wiki content with deterministic scripts. Reserve LLM tokens for -creative and analytical work. - -### 1.1. Document Conventions - -> Status: Implemented - -- *1.1.1.* Main sections use `## N. Title`. Subsections use - `### N.M. Title`. Deeper headings use `#### N.M.K. Title`. - - *acceptance*: Every heading follows that numbering. -- *1.1.2.* A requirement is `- *N.M.K.* `. A group is numbered - the same way, and each item under it adds one segment, for example - `*7.14.1.*` then `*7.14.1.1.*`. Its acceptance is the next nested - `- *acceptance*: `, and that acceptance describes this - requirement only. - - *acceptance*: No acceptance describes a neighboring requirement. - - *acceptance*: The shared-engine bullets in section 7.14 are - numbered `7.14.1` and `7.14.1.1`, not plain dashes. -- *1.1.3.* Unbuilt requirements start with `` `NOT_IMPLEMENTED` ``. - - *acceptance*: Every planned item carries that tag. -- *1.1.4.* The block between `` and - `` is generated by `librarian toc file`. Leave that - block to the generator. - -### 1.2. Constraints - -These apply to every feature. - -- *1.2.1.* Tool reports are parseable and use the shared shapes in - sections 4, 6, and 7. - - *acceptance*: Two linters that find violations print the same - section order from section 6.1. -- *1.2.2.* A command that needs a human decision prompts on a TTY. - The same command accepts an explicit flag (`--force`, `--yes`, or - `--non-interactive`) and then runs without a prompt. - - *acceptance*: `wiki update` on a non-TTY declines unless `--force` - or `--yes` is set. -- *1.2.3.* `personal/` and `raw/` are read-only for every automated - tool. Tools may resolve links that point into those zones. The one - exception is `wiki ingest` (*7.1.5.3.*): it may add a new file under - `raw/`. It never modifies, renames, or deletes a file already there. - - *acceptance*: `wiki update` leaves every file under `personal/` - and `raw/` byte-for-byte unchanged. - - *acceptance*: A second `wiki ingest` run leaves every file that - was already under `raw/` byte-for-byte unchanged. - - *acceptance*: Ingesting bytes whose SHA-256 matches any file - already under `raw/`, from any path or URL, adds no file under - `raw/` (*7.1.5.3.3.*). -- *1.2.4.* Code changes follow test-first development. - `python scripts/run_tests.py` is green, then - `python scripts/run_sanity.py` is green, before a change is finished. - - *acceptance*: A sanity run below the coverage gate in section 5.2 - fails even when every test passes. -- *1.2.5.* Python passes Ruff. Markdown passes PyMarkdown with the - rules in section 5.1. - - *acceptance*: An unused import is reported by `python-ruff chklint` - with file, line, column, and rule code. -- *1.2.6.* Authored lengths in HTML and CSS use `em` or another - relative unit (`rem`, `%`, `vh`, `vw`). They are not written in - `px`. The wiki graph page in section 7.1.4 is that web surface. - A position read from the layout engine is written back in `rem`. - A theme length number uses the same scale: 16 is `1em`. - - *acceptance*: The graph template contains no `px` length. - ---- - -## 2. Package and Runtime - -> Status: Implemented - -Python `>= 3.11` (also 3.12 and 3.13). The default development -install from the repository root is `pip install -e ".[dev]"`. -`bootstrap --local` installs the same tree as a normal package -(`pip install .`). Section 7.21 defines those modes. The build -requires setuptools 61 or newer. - -Runtime dependencies: - -| Package | Minimum | -| --- | --- | -| pymarkdownlnt | 0.9.0 | -| ruff | 0.16.8 | -| pyyaml | 6.0.3 | -| GitPython | 3.1.0 | - -Development extras: `pytest>=7.0`, `pytest-cov>=4.0`, `pytest-mock`, -`pylint`. `html5validator` is not a dependency until section 7.15 -is implemented. `beautifulsoup4`, `markdownify`, and `pypdf` become -runtime dependencies when section 7.1.5 is implemented. - -One importable package, `fiona`, lives under `src/fiona/`. Its -subpackages are `librarian`, `mdoctor`, `customlinter`, -`python_ruff`, `gittool`, `sanity_runner`, and `wiki`. The wiki -template is package data at `src/fiona/data/llm_wiki/` and is loaded -with `importlib.resources`. It is not an importable module. Tests -live in repo-root `tests/` and are not installed. `scripts/` holds -`settings_dev.json`, shell helpers, and the launchers -`scripts/run_tests.py` and `scripts/run_sanity.py`. - -Console scripts: - -| Script | Entry | -| --- | --- | -| `fiona-assistant` | `fiona.__main__:main` | -| `librarian` | `fiona.librarian.librarian:main` | -| `mdoctor` | `fiona.mdoctor.mdoctor:main` | - -`python-ruff`, `fmlint`, `json-lint`, and `git` are reached only -through `fiona-assistant`. `html-js-lint` is not installed until -section 7.15 is implemented. - -Coverage measures `src/fiona`, omitting tests. The config lives in -`pyproject.toml`. The gate percentage is section 5.2. - -Logging: every CLI entry calls `setup_logging()` before work. Level -order is `LOG_LEVEL` environment variable, then `fiona.log_level` in -`fiona-settings.json`, then `WARNING`. Records go to stderr. When -`fiona.file_log_location` is set, a file handler writes lines -`%(asctime)s %(levelname)s %(name)s: %(message)s` at or above -`fiona.file_log_level` (fallback: `fiona.log_level`). A relative path -is resolved from the current working directory. An unopenable path -logs a warning and keeps stderr logging. Human reports stay on -stdout. Package loggers set their level explicitly because -pymarkdown resets the root logger mid-run. - -- *2.1.* The three console scripts above are installed by - `pip install -e .`. - - *acceptance*: `fiona-assistant` with no arguments prints help and - exits 0. -- *2.2.* `html-js-lint` is not an installed script until section 7.15 - exists. - - *acceptance*: A fresh install does not fail at import because of - a missing `html_js_lint` package. - ---- - -## 3. llm-wiki Template - -> Status: Implemented - -`wiki init` copies this tree from the packaged template. Text files -replace every `DATE_PLACEHOLDER` with the current UTC date -`YYYY-MM-DD`. Binary files are copied unchanged. A glob in -`fiona.llm-wiki.ignore_files` skips that file, or that directory and -its subtree. A missing settings file or a missing key copies -everything. A non-list `ignore_files` value raises `ValueError` with -the text `fiona.llm-wiki.ignore_files must be a list of strings`. - -```text -AGENTS.md -README.md -fiona-settings.json -constellation/colorful.json -constellation/dark.json -constellation/grayscale.json -constellation/night-sky.json -constellation/professional.json -table_of_contents.md -ingest/README.md -personal/README.md -raw/README.md -wiki/schema.md -wiki/index.md -wiki/log.md -utils/README.md -utils/scripts/example.py -utils/scripts/map.md -utils/skills/README.md -utils/skills/fiona/SKILL.md -utils/skills/fiona/markdown_rules.md -utils/skills/fiona/llm-wiki-mod.md -utils/skills/python-tdd/SKILL.md -utils/personas/default/SOUL.md -utils/personas/template/SOUL.md -utils/skills_local/README.md -``` - -The shipped `fiona-settings.json` is the default document in -section 5.1. `wiki/schema.md` is the page-type and tag taxonomy. -This spec's frontmatter table in section 6.2 is the field contract; -where the two disagree, this spec wins. - -### 3.1. What Counts as a Wiki - -A directory is an llm-wiki when it contains `fiona-settings.json`. -`wiki init` refuses when any of these already exist: -`AGENTS.md`, `fiona-settings.json`, `wiki/`, `ingest/`, `README.md`. - -### 3.2. Core Files and Read-only Zones - -Core files, matched by name at the wiki root: - -- `AGENTS.md` -- `fiona-settings.json` -- `README.md` -- `table_of_contents.md` - -Core directories (every file under them is core): - -- `wiki/` -- `ingest/` -- `personal/` -- `raw/` -- `utils/` -- `constellation/` - -Read-only zones, including the zone directory itself: - -- `personal/` -- `raw/` - -`wiki/` is copied by `wiki init` and then left to the LLM. -`wiki update` does not add, replace, or delete any path under -`wiki/`. The only file it writes there is an update file for a -template page that is newer than the wiki's copy (*7.1.2.11.*). - -`wiki update` adds missing core files and directories. It never -overwrites an existing markdown file: when the template carries a -newer version, it writes `-update.md` beside the file and -leaves the owner's file as it is. A differing -`fiona-settings.json` is the same kind of conflict: the owner's -file stays, and the template is written as -`fiona-settings-update.json` (*7.1.2.3.*). A confirmed update -that changes or reports anything also writes `corefiles.json` -at the wiki root (*7.1.2.14.*). That file is not part of the -template. Only `wiki update` writes it. A glob in -`fiona.wiki.update.ignore_files` is not updated (*7.1.2.15.*). -The shipped list is `corefiles.json`. It never deletes user files. It -never writes inside a read-only zone, so a missing file under -`personal/` or `raw/` is not added back. It -never writes a path matched by the repository `.gitignore` or by -the wiki's own `.gitignore`. - -After the copy, `wiki init` creates a git repository and writes -`.gitignore`. A new file starts with the fiona-assistant repository -`.gitignore` (the same patterns, including `__pycache__/`, `*.pyc`, -and `*fiona.log`). When this package is not the source checkout, -that text is the packaged `default.gitignore`. The file adds -`personal/` when that pattern is missing. -An existing `.gitignore` keeps every line it already has; missing -patterns from that set are appended. Init then stages everything -and commits `Initial llm-wiki commit`. Template paths that match -the repository `.gitignore` are not copied. `personal/` and `raw/` -are still copied. `personal/` stays untracked. If git init fails, -the copy still counts as success and stdout adds -`Note: Git repository initialization failed: `. - -- *3.2.1.* An empty directory becomes the tree above, with dates - substituted, and a git commit that does not track `personal/`. - - *acceptance*: `personal/` is absent from `git ls-files` after - `wiki init`. -- *3.2.2.* A second `wiki init` raises `FileExistsError` whose - message names each conflicting indicator, and writes nothing. - - *acceptance*: An existing `README.md` alone is enough to refuse, - and that file's bytes are unchanged. - -### 3.3. Deployed Wiki Rules - -> Status: Implemented - -Only the template tree is deployed. A wiki does not know this spec, -the root `AGENTS.md`, `scripts/`, or `tests/`. The template -`AGENTS.md` is the entry point: it says what the wiki is and loads -the Fiona skill, which loads `llm-wiki-mod.md`. The product rule -"never commit without permission" (project agent rule 2.4) covers -this repository only. - -- *3.3.1.* No markdown file in the template references a file or - rule outside the template: `SPEC`, `project agent rule`, or - `scripts/run_`. - - *acceptance*: A scan of every template `*.md` finds none of those - strings. -- *3.3.2.* The template `AGENTS.md` links the Fiona skill and - `llm-wiki-mod.md`. - - *acceptance*: Both relative links appear in that file. -- *3.3.3.* A deployed wiki commits with - `fiona-assistant git save --description TEXT` after each ingest - cycle and each housekeeping pass. `clean`, `undo`, and `restore` - run only when the user asks. - - *acceptance*: The template `AGENTS.md` and the Fiona skill both - contain `git save`, and neither says to commit only when asked. -- *3.3.4.* The template `wiki/log.md` holds one seed entry dated - `DATE_PLACEHOLDER`. - - *acceptance*: The log body has exactly one bullet. - ---- - -## 4. Command Catalog - -> Status: Implemented - -One command tree. Optional `[directory]` defaults to `.` today. -Wiki-root discovery (section 7.20) is not built. An invalid -directory prints `Error: '' is not a valid directory.` on -stderr and exits 1, unless a command below states a different -message. - -`fiona-assistant` with no command prints help and exits 0. -`fiona-assistant git` with no subcommand prints git help and exits 1. -`fiona-assistant wiki` with no subcommand prints wiki help and -exits 1. - -### 4.1. librarian - -`fiona-assistant librarian` and `librarian` with no subcommand list -the current directory. - -`list [directory]` stdout: - -```text -Found files in '': - - -``` - -`mdoctor list` uses the same listing with the prefix -`MDoctor: Found files in '':`. - -#### 4.1.1. librarian toc - -`toc [directory] [flags]` writes `table_of_contents.md` in the -target folder. Flags: `--recursive-toc`, `--non-recursive`, and -repeatable `--exclude REGEX`. Depth comes from section 5.1. - -A path that is a file or missing exits 1 on stderr and names -`librarian toc file `. A folder with no markdown prints -`no markdown files found` on stdout and writes nothing. In -`--recursive-toc` mode that message is a log line at INFO instead. - -`toc file ` writes an in-file TOC in that markdown file. -`toc file *` does that for every markdown file under the wiki root -found by section 7.3.1. `--sectionized` forces section numbering for -that run. Status lines: - -```text -wrote toc in -toc already up to date in -skipped () -generated toc for file(s), skipped -``` - -The summary line is bulk mode only. A file with no H1 is -`skipped (no H1 heading)` and is not rewritten. - -### 4.2. mdoctor - -No subcommand means `chklint .`. - -`chklint [directory]` prints the shared report (section 6.1) with -header `PyMarkdownlint Violations` and exits 0 even when violations -exist. - -`fix [directory]` runs pymarkdown auto-fix in place. Stdout: - -```text -PyMarkdownFix: file(s) fixed - - -``` - -A pymarkdown API failure prints -`Error: pymarkdown fix failed: ` on stderr and exits 1. -The command does not skip `personal/` or `raw/`. - -### 4.3. fmlint - -`fiona-assistant fmlint` with no subcommand means `chklint .`. -`chklint [directory]` prints the shared report with header -`Frontmatter Lint Violations` and exits 0 even when violations -exist. A missing directory exits 1. - -### 4.4. python-ruff - -No subcommand means `chklint .`. The report header is -`Ruff Violations`. Exit code is 0 when the ruff process itself -succeeds, including when violations are listed. `lint(path, fix=True)` -passes `--fix` to the ruff binary. There is no `fix` subcommand yet. - -### 4.5. json-lint - -No subcommand means `chklint .`. Header is `JSONLint Violations`. - -`chklint [directory]` exits 1 when the report contains violations -and exits 0 when it reports no violations. This differs from -mdoctor, fmlint, and python-ruff, which exit 0 either way. - -`chklint --fix` and `fix [directory]` rewrite indentation and -trailing commas in place, skip `personal/` and `raw/`, and print: - -```text -JSONLintFix: file(s) fixed - - -``` - -A missing directory on `fix` prints -`Error: '' does not exist.` on stderr and exits 1. - -### 4.6. git - -Operates on the git repository at the current directory. Timestamps -are UTC. - -| Command | Stdout on success | -| --- | --- | -| `save --description TEXT` | `Committed: ` | -| `snapshot --description TEXT` | `Snapshot created: commit=, tag=` | -| `restore ` | `Restored to: ` | -| `clean` | `Working tree cleaned. All changes discarded.` | -| `undo` | `Undo to: ` | -| `status` | `Status: clean` or `Status: dirty`, then `git status` | - -On failure the tool prints `Error: ` and exits 1. - -Commit message: ` `. -`save` runs `git add --all` and commits. A clean tree, including -no untracked files, raises `Working tree is clean. Nothing to commit.` - -`snapshot` commits, then creates an annotated tag -`_` with message -`Snapshot: `. - -`restore` requires a clean tree -(`Working tree is dirty. Use 'clean' to discard current edits first.`). -It accepts the full tag or the user tag suffix. The latest tag whose -name equals the argument or ends with `_` wins. It checks -that commit's tree into the index and working tree and adds a new -commit ` Restore to `. History before that -commit stays. - -`clean` requires a dirty tree -(`Working tree is already clean. Nothing to discard.`), then -`git reset --hard HEAD` and `git clean -fd`. - -`undo` requires a clean tree and at least two commits -(`Cannot undo: only one commit exists in the repository.`), then -`git reset --hard HEAD~1`. - -### 4.7. wiki - -Subcommands of `fiona-assistant wiki`. - -#### 4.7.1. wiki init - -`fiona-assistant wiki init [directory]`. Success stdout: - -```text -Successfully initialized llm-wiki directory at: -``` - -Rules are section 3. - -#### 4.7.2. wiki check-update - -`fiona-assistant wiki check-update [directory] [--json]`. -The directory must contain `fiona-settings.json`, else stderr -`Error: Directory '' is not an llm-wiki directory` and exit 1. -A missing directory is -`Error: Directory not found: ` and exit 1. -A successful check exits 0 whether or not updates exist. - -Human stdout when current: - -```text -Wiki is up to date. -``` - -When something differs, the lines that apply: - -```text -Files to update: - - ( ->