Files
fiona-assistant/SPEC.md
T
2026-10-06 11:27:26 +02:00

4008 lines
185 KiB
Markdown

---
name: Fiona-Assistant Specification
version: 1.9.1
title: Project Spec Sheet
created: 2026-09-23
updated: 2026-10-06
type: summary
---
# Fiona-Assistant Specification
<!-- toc-start: headings 2..6 -->
- [1. Product Overview](#1-product-overview)
- [1.1. Document Conventions](#11-document-conventions)
- [1.2. Constraints](#12-constraints)
- [2. Package and Runtime](#2-package-and-runtime)
- [3. llm-wiki Template](#3-llm-wiki-template)
- [3.1. What Counts as a Wiki](#31-what-counts-as-a-wiki)
- [3.2. Core Files and Read-only Zones](#32-core-files-and-read-only-zones)
- [3.3. Deployed Wiki Rules](#33-deployed-wiki-rules)
- [4. Command Catalog](#4-command-catalog)
- [4.1. librarian](#41-librarian)
- [4.1.1. librarian toc](#411-librarian-toc)
- [4.1.2. librarian move](#412-librarian-move)
- [4.2. mdoctor](#42-mdoctor)
- [4.3. fmlint](#43-fmlint)
- [4.4. python-ruff](#44-python-ruff)
- [4.5. json-lint](#45-json-lint)
- [4.6. git](#46-git)
- [4.7. wiki](#47-wiki)
- [4.7.1. wiki init](#471-wiki-init)
- [4.7.2. wiki check-update](#472-wiki-check-update)
- [4.7.3. wiki update](#473-wiki-update)
- [4.7.4. wiki sanity](#474-wiki-sanity)
- [4.7.5. wiki graph](#475-wiki-graph)
- [4.7.6. wiki ingest](#476-wiki-ingest)
- [4.8. housekeeping](#48-housekeeping)
- [4.9. Repository runners](#49-repository-runners)
- [5. Settings Schema](#5-settings-schema)
- [5.1. fiona-settings.json](#51-fiona-settingsjson)
- [5.2. settings_dev.json](#52-settings_devjson)
- [5.3. constellation themes](#53-constellation-themes)
- [6. Violation Rule Catalog](#6-violation-rule-catalog)
- [6.1. Shared Report](#61-shared-report)
- [6.2. Frontmatter Fields and FML Codes](#62-frontmatter-fields-and-fml-codes)
- [6.3. MDL Codes](#63-mdl-codes)
- [6.4. PYC Codes](#64-pyc-codes)
- [6.5. JSN Codes](#65-jsn-codes)
- [6.6. Base Linter Contract](#66-base-linter-contract)
- [7. Feature Requirements](#7-feature-requirements)
- [7.1. wiki](#71-wiki)
- [7.1.1. wiki init](#711-wiki-init)
- [7.1.2. wiki update](#712-wiki-update)
- [7.1.3. wiki sanity](#713-wiki-sanity)
- [7.1.4. wiki graph](#714-wiki-graph)
- [7.1.5. wiki ingest](#715-wiki-ingest)
- [7.2. housekeeping](#72-housekeeping)
- [7.3. librarian](#73-librarian)
- [7.3.1. Folder and In-file TOC](#731-folder-and-in-file-toc)
- [7.3.2. Non-markdown Version Tags](#732-non-markdown-version-tags)
- [7.3.3. Move with Link Rewrite](#733-move-with-link-rewrite)
- [7.4. mdoctor](#74-mdoctor)
- [7.5. fmlint](#75-fmlint)
- [7.6. mdlinter](#76-mdlinter)
- [7.7. pycustomlint](#77-pycustomlint)
- [7.8. python-ruff](#78-python-ruff)
- [7.9. Shellcheck](#79-shellcheck)
- [7.10. Markdown Code Blocks](#710-markdown-code-blocks)
- [7.11. json-lint](#711-json-lint)
- [7.12. Git Operations](#712-git-operations)
- [7.13. Backup Script](#713-backup-script)
- [7.14. Test and Sanity Runners](#714-test-and-sanity-runners)
- [7.15. html-js-lint](#715-html-js-lint)
- [7.16. settings setup](#716-settings-setup)
- [7.17. Logging](#717-logging)
- [7.18. Documentation](#718-documentation)
- [7.19. Tool Roles](#719-tool-roles)
- [7.20. Wiki Root Discovery](#720-wiki-root-discovery)
- [7.21. Bootstrap install modes](#721-bootstrap-install-modes)
- [Appendix A. Non-Goals](#appendix-a-non-goals)
- [Appendix B. Priority Order](#appendix-b-priority-order)
- [Appendix C. Open Questions](#appendix-c-open-questions)
- [Appendix D. Maintainer Notes](#appendix-d-maintainer-notes)
<!-- toc-end -->
This document is the build contract for Fiona-Assistant. A reader with
this file, and no source tree, must be able to produce the same
observable software. Status values are `Implemented`, `Partially
Implemented`, `Planned`, and `Won't Do`.
Revision 0.40.0 reorganizes the sheet into contracts. Revision 0.40.1
numbers nested requirements such as `7.14.1.1`. Revision 0.40.4 adds
the PYC001 exception in *6.4.2.*. Revision 0.41.0 installs one
`fiona` package and keeps tests and repo scripts outside it.
Revision 0.90.0 specifies `node_importance_min_point` in *7.1.4.16.*
and leaves that requirement unbuilt. Revision 0.92.0 specifies
`wiki ingest` in *7.1.5.* and leaves it unbuilt. Revision 0.93.0 adds
SHA-256 reuse across `raw/`, the complete `ingest` frontmatter map,
and the agent metadata check. Revision 0.94.0 makes `wiki update`
yield to the owner's markdown and write `<stem>-update.md` instead
(*7.1.2.11.*), and adds `fiona-assistant housekeeping` (*7.2.*).
Revision 0.96.0 compares a file without a version by SHA-256 and
shows both short hashes in the update report (*7.1.2.12.*).
Revision 0.97.0 rejects a module-level assignment that follows the
first function or class (*6.4.3.*). Revision 0.98.0 specifies that
`wiki graph` omits a file matched by `.gitignore` (*7.1.4.17.*) and
leaves that requirement unbuilt. Revision 0.99.0 builds `wiki ingest`
(*7.1.5.*). Revision 1.0.0 groups each module's functions under that
module and moves Non-Goals, Priority Order, and Open Questions
to Appendices A, B, and C. Revision 1.1.0 treats a differing
`fiona-settings.json` as a conflict and writes
`fiona-settings-update.json` (*7.1.2.3.*). Revision 1.2.0 folds
text line endings before the SHA-256 compare (*7.1.2.13.*) and
writes `corefiles.json` so a later update does not list a
core file whose version and hashes are unchanged (*7.1.2.14.*).
Revision 1.3.0 specifies directory nodes (*7.1.4.18.*) and
`wiki graph --serve` (*7.1.4.19.*) and leaves both unbuilt.
Revision 1.4.0 names the core-file record `corefiles.json`
(*7.1.2.14.*) and skips paths in `fiona.wiki.update.ignore_files`
(*7.1.2.15.*). The shipped list names `corefiles.json`. Only
`wiki update` writes that file. Revision 1.5.0 omits markdown
files matched by `fiona.wiki.graph.ignore_files` from the graph
(*7.1.4.20.*). The shipped list is `*/table_of_contents.md`,
because a table of contents links to every page. Revision 1.5.2
gives that file frontmatter `name` and `version` (*7.3.1.1.1.*)
so `fmlint` accepts it. Revision 1.6.0 specifies `librarian move`
(*7.3.3.*) and leaves it unbuilt. Revision 1.7.0 makes that
command rewrite links to the moved paths across the whole wiki
(*7.3.3.6.*). Revision 1.8.0 adds the findings of the 2026-10-02
manual test and leaves them unbuilt: one-line CLI errors with no
traceback (*1.2.7.*, *7.1.1.4.*), `--version` (*2.3.*), a quiet
default stderr (*7.17.3.*), an ingest hint for a drop-zone file that
already moved (*7.1.5.2.7.*), a dry run that does not log writes
(*7.1.5.8.1.*), a lint-clean ingest log line (*7.1.5.6.1.*), a
follow-up hint for style violations left on a page (*7.1.5.5.6.*),
single-file `mdoctor` (*7.4.4.*), compact housekeeping sanity details
(*7.2.5.1.*), a quiet graph overwrite (*7.1.4.5.9.1.*), and a usage
section in the README (*7.18.4.*). Revision 1.9.0 makes
`wiki ingest` stop with exit 4 on a URL that needs a login
(*7.1.5.2.8.*), adds `--origin` for a file saved by hand
(*7.1.5.2.9.*), and tells the agent how to get the source instead
(*7.1.5.14.*). All three are unbuilt.
Each feature section starts with a `Status` line. Requirements use
`- *N.M.K.*`. Each `*acceptance*` sits under the requirement it tests.
`NOT_IMPLEMENTED` marks work that is specified and not built yet.
Class names, file paths, and history live in Appendix D.
---
## 1. Product Overview
Fiona-Assistant is a CLI toolkit that manages, organizes, and checks
LLM wiki content with deterministic scripts. Reserve LLM tokens for
creative and analytical work.
### 1.1. Document Conventions
> Status: Implemented
- *1.1.1.* Main sections use `## N. Title`. Subsections use
`### N.M. Title`. Deeper headings use `#### N.M.K. Title`.
- *acceptance*: Every heading follows that numbering.
- *1.1.2.* A requirement is `- *N.M.K.* <text>`. A group is numbered
the same way, and each item under it adds one segment, for example
`*7.14.1.*` then `*7.14.1.1.*`. Its acceptance is the next nested
`- *acceptance*: <text>`, and that acceptance describes this
requirement only.
- *acceptance*: No acceptance describes a neighboring requirement.
- *acceptance*: The shared-engine bullets in section 7.14 are
numbered `7.14.1` and `7.14.1.1`, not plain dashes.
- *1.1.3.* Unbuilt requirements start with `` `NOT_IMPLEMENTED` ``.
- *acceptance*: Every planned item carries that tag.
- *1.1.4.* The block between `<!-- toc-start: headings 2..6 -->` and
`<!-- toc-end -->` is generated by `librarian toc file`. Leave that
block to the generator.
### 1.2. Constraints
These apply to every feature.
- *1.2.1.* Tool reports are parseable and use the shared shapes in
sections 4, 6, and 7.
- *acceptance*: Two linters that find violations print the same
section order from section 6.1.
- *1.2.2.* A command that needs a human decision prompts on a TTY.
The same command accepts an explicit flag (`--force`, `--yes`, or
`--non-interactive`) and then runs without a prompt.
- *acceptance*: `wiki update` on a non-TTY declines unless `--force`
or `--yes` is set.
- *1.2.3.* `personal/` and `raw/` are read-only for every automated
tool. Tools may resolve links that point into those zones. The one
exception is `wiki ingest` (*7.1.5.3.*): it may add a new file under
`raw/`. It never modifies, renames, or deletes a file already there.
- *acceptance*: `wiki update` leaves every file under `personal/`
and `raw/` byte-for-byte unchanged.
- *acceptance*: A second `wiki ingest` run leaves every file that
was already under `raw/` byte-for-byte unchanged.
- *acceptance*: Ingesting bytes whose SHA-256 matches any file
already under `raw/`, from any path or URL, adds no file under
`raw/` (*7.1.5.3.3.*).
- *1.2.4.* Code changes follow test-first development.
`python scripts/run_tests.py` is green, then
`python scripts/run_sanity.py` is green, before a change is finished.
- *acceptance*: A sanity run below the coverage gate in section 5.2
fails even when every test passes.
- *1.2.5.* Python passes Ruff. Markdown passes PyMarkdown with the
rules in section 5.1.
- *acceptance*: An unused import is reported by `python-ruff chklint`
with file, line, column, and rule code.
- *1.2.6.* Authored lengths in HTML and CSS use `em` or another
relative unit (`rem`, `%`, `vh`, `vw`). They are not written in
`px`. The wiki graph page in section 7.1.4 is that web surface.
A position read from the layout engine is written back in `rem`.
A theme length number uses the same scale: 16 is `1em`.
- *acceptance*: The graph template contains no `px` length.
- *1.2.7.* `NOT_IMPLEMENTED` An expected failure, such as a missing
path, a refusal, or bad input, prints one `Error: <message>` line on
stderr and exits 1. It prints no Python traceback. A traceback is
logged only at `DEBUG`.
- *acceptance*: A second `wiki init` on the same directory writes
no line containing `Traceback` on stderr (*7.1.1.4.*).
---
## 2. Package and Runtime
> Status: Partially Implemented
Python `>= 3.11` (also 3.12 and 3.13). The default development
install from the repository root is `pip install -e ".[dev]"`.
`bootstrap --local` installs the same tree as a normal package
(`pip install .`). Section 7.21 defines those modes. The build
requires setuptools 61 or newer.
Runtime dependencies:
| Package | Minimum |
| --- | --- |
| pymarkdownlnt | 0.9.0 |
| ruff | 0.16.8 |
| pyyaml | 6.0.3 |
| GitPython | 3.1.0 |
Development extras: `pytest>=7.0`, `pytest-cov>=4.0`, `pytest-mock`,
`pylint`. `html5validator` is not a dependency until section 7.15
is implemented. `beautifulsoup4`, `markdownify`, and `pypdf` become
runtime dependencies when section 7.1.5 is implemented.
One importable package, `fiona`, lives under `src/fiona/`. Its
subpackages are `librarian`, `mdoctor`, `customlinter`,
`python_ruff`, `gittool`, `sanity_runner`, and `wiki`. The wiki
template is package data at `src/fiona/data/llm_wiki/` and is loaded
with `importlib.resources`. It is not an importable module. Tests
live in repo-root `tests/` and are not installed. `scripts/` holds
`settings_dev.json`, shell helpers, and the launchers
`scripts/run_tests.py` and `scripts/run_sanity.py`.
Console scripts:
| Script | Entry |
| --- | --- |
| `fiona-assistant` | `fiona.__main__:main` |
| `librarian` | `fiona.librarian.librarian:main` |
| `mdoctor` | `fiona.mdoctor.mdoctor:main` |
`python-ruff`, `fmlint`, `json-lint`, and `git` are reached only
through `fiona-assistant`. `html-js-lint` is not installed until
section 7.15 is implemented.
Coverage measures `src/fiona`, omitting tests. The config lives in
`pyproject.toml`. The gate percentage is section 5.2.
Logging: every CLI entry calls `setup_logging()` before work. Level
order is `LOG_LEVEL` environment variable, then `fiona.log_level` in
`fiona-settings.json`, then `WARNING`. Records go to stderr. When
`fiona.file_log_location` is set, a file handler writes lines
`%(asctime)s %(levelname)s %(name)s: %(message)s` at or above
`fiona.file_log_level` (fallback: `fiona.log_level`). A relative path
is resolved from the current working directory. An unopenable path
logs a warning and keeps stderr logging. Human reports stay on
stdout. Package loggers set their level explicitly because
pymarkdown resets the root logger mid-run.
- *2.1.* The three console scripts above are installed by
`pip install -e .`.
- *acceptance*: `fiona-assistant` with no arguments prints help and
exits 0.
- *2.2.* `html-js-lint` is not an installed script until section 7.15
exists.
- *acceptance*: A fresh install does not fail at import because of
a missing `html_js_lint` package.
- *2.3.* `NOT_IMPLEMENTED` `fiona-assistant --version` prints
`fiona-assistant <package version>` on stdout and exits 0. The text
is the same string `wiki ingest` writes to `ingest.tool`
(*7.1.5.5.3.*).
- *acceptance*: The stdout of `--version` equals the `ingest.tool`
value of a page ingested by the same install.
---
## 3. llm-wiki Template
> Status: Implemented
`wiki init` copies this tree from the packaged template. Text files
replace every `DATE_PLACEHOLDER` with the current UTC date
`YYYY-MM-DD`. Binary files are copied unchanged. A glob in
`fiona.llm-wiki.ignore_files` skips that file, or that directory and
its subtree. A missing settings file or a missing key copies
everything. A non-list `ignore_files` value raises `ValueError` with
the text `fiona.llm-wiki.ignore_files must be a list of strings`.
```text
AGENTS.md
README.md
fiona-settings.json
constellation/colorful.json
constellation/dark.json
constellation/grayscale.json
constellation/night-sky.json
constellation/professional.json
table_of_contents.md
ingest/README.md
personal/README.md
raw/README.md
wiki/schema.md
wiki/index.md
wiki/log.md
utils/README.md
utils/scripts/example.py
utils/scripts/map.md
utils/skills/README.md
utils/skills/fiona/SKILL.md
utils/skills/fiona/markdown_rules.md
utils/skills/fiona/llm-wiki-mod.md
utils/skills/python-tdd/SKILL.md
utils/personas/default/SOUL.md
utils/personas/template/SOUL.md
utils/skills_local/README.md
```
The shipped `fiona-settings.json` is the default document in
section 5.1. `wiki/schema.md` is the page-type and tag taxonomy.
This spec's frontmatter table in section 6.2 is the field contract;
where the two disagree, this spec wins.
### 3.1. What Counts as a Wiki
A directory is an llm-wiki when it contains `fiona-settings.json`.
`wiki init` refuses when any of these already exist:
`AGENTS.md`, `fiona-settings.json`, `wiki/`, `ingest/`, `README.md`.
### 3.2. Core Files and Read-only Zones
Core files, matched by name at the wiki root:
- `AGENTS.md`
- `fiona-settings.json`
- `README.md`
- `table_of_contents.md`
Core directories (every file under them is core):
- `wiki/`
- `ingest/`
- `personal/`
- `raw/`
- `utils/`
- `constellation/`
Read-only zones, including the zone directory itself:
- `personal/`
- `raw/`
`wiki/` is copied by `wiki init` and then left to the LLM.
`wiki update` does not add, replace, or delete any path under
`wiki/`. The only file it writes there is an update file for a
template page that is newer than the wiki's copy (*7.1.2.11.*).
`wiki update` adds missing core files and directories. It never
overwrites an existing markdown file: when the template carries a
newer version, it writes `<stem>-update.md` beside the file and
leaves the owner's file as it is. A differing
`fiona-settings.json` is the same kind of conflict: the owner's
file stays, and the template is written as
`fiona-settings-update.json` (*7.1.2.3.*). A confirmed update
that changes or reports anything also writes `corefiles.json`
at the wiki root (*7.1.2.14.*). That file is not part of the
template. Only `wiki update` writes it. A glob in
`fiona.wiki.update.ignore_files` is not updated (*7.1.2.15.*).
The shipped list is `corefiles.json`. It never deletes user files. It
never writes inside a read-only zone, so a missing file under
`personal/` or `raw/` is not added back. It
never writes a path matched by the repository `.gitignore` or by
the wiki's own `.gitignore`.
After the copy, `wiki init` creates a git repository and writes
`.gitignore`. A new file starts with the fiona-assistant repository
`.gitignore` (the same patterns, including `__pycache__/`, `*.pyc`,
and `*fiona.log`). When this package is not the source checkout,
that text is the packaged `default.gitignore`. The file adds
`personal/` when that pattern is missing.
An existing `.gitignore` keeps every line it already has; missing
patterns from that set are appended. Init then stages everything
and commits `Initial llm-wiki commit`. Template paths that match
the repository `.gitignore` are not copied. `personal/` and `raw/`
are still copied. `personal/` stays untracked. If git init fails,
the copy still counts as success and stdout adds
`Note: Git repository initialization failed: <error>`.
- *3.2.1.* An empty directory becomes the tree above, with dates
substituted, and a git commit that does not track `personal/`.
- *acceptance*: `personal/` is absent from `git ls-files` after
`wiki init`.
- *3.2.2.* A second `wiki init` raises `FileExistsError` whose
message names each conflicting indicator, and writes nothing.
- *acceptance*: An existing `README.md` alone is enough to refuse,
and that file's bytes are unchanged.
### 3.3. Deployed Wiki Rules
> Status: Implemented
Only the template tree is deployed. A wiki does not know this spec,
the root `AGENTS.md`, `scripts/`, or `tests/`. The template
`AGENTS.md` is the entry point: it says what the wiki is and loads
the Fiona skill, which loads `llm-wiki-mod.md`. The product rule
"never commit without permission" (project agent rule 2.4) covers
this repository only.
- *3.3.1.* No markdown file in the template references a file or
rule outside the template: `SPEC`, `project agent rule`, or
`scripts/run_`.
- *acceptance*: A scan of every template `*.md` finds none of those
strings.
- *3.3.2.* The template `AGENTS.md` links the Fiona skill and
`llm-wiki-mod.md`.
- *acceptance*: Both relative links appear in that file.
- *3.3.3.* A deployed wiki commits with
`fiona-assistant git save --description TEXT` after each ingest
cycle and each housekeeping pass. `clean`, `undo`, and `restore`
run only when the user asks.
- *acceptance*: The template `AGENTS.md` and the Fiona skill both
contain `git save`, and neither says to commit only when asked.
- *3.3.4.* The template `wiki/log.md` holds one seed entry dated
`DATE_PLACEHOLDER`.
- *acceptance*: The log body has exactly one bullet.
---
## 4. Command Catalog
> Status: Implemented
One command tree. Optional `[directory]` defaults to `.` today.
Wiki-root discovery (section 7.20) is not built. An invalid
directory prints `Error: '<path>' is not a valid directory.` on
stderr and exits 1, unless a command below states a different
message.
`fiona-assistant` with no command prints help and exits 0.
`fiona-assistant git` with no subcommand prints git help and exits 1.
`fiona-assistant wiki` with no subcommand prints wiki help and
exits 1.
### 4.1. librarian
`fiona-assistant librarian` and `librarian` with no subcommand list
the current directory.
`list [directory]` stdout:
```text
Found <N> files in '<directory>':
- <path>
```
`mdoctor list` uses the same listing with the prefix
`MDoctor: Found <N> files in '<directory>':`.
#### 4.1.1. librarian toc
`toc [directory] [flags]` writes `table_of_contents.md` in the
target folder. Flags: `--recursive-toc`, `--non-recursive`, and
repeatable `--exclude REGEX`. Depth comes from section 5.1.
A path that is a file or missing exits 1 on stderr and names
`librarian toc file <path>`. A folder with no markdown prints
`no markdown files found` on stdout and writes nothing. In
`--recursive-toc` mode that message is a log line at INFO instead.
`toc file <path>` writes an in-file TOC in that markdown file.
`toc file *` does that for every markdown file under the wiki root
found by section 7.3.1. `--sectionized` forces section numbering for
that run. Status lines:
```text
wrote toc in <path>
toc already up to date in <path>
skipped <path> (<reason>)
generated toc for <N> file(s), skipped <M>
```
The summary line is bulk mode only. A file with no H1 is
`skipped <path> (no H1 heading)` and is not rewritten.
#### 4.1.2. librarian move
`NOT_IMPLEMENTED` for the whole command. Section 7.3.3 is the
contract.
`move <source> <target> [--dry-run]` moves a file or a folder and
leaves no dead link behind. Each moved markdown file gets its
relative links rewritten, so every link that resolved before the
move still resolves from the new place. Every other markdown file in
the wiki that links to a moved path gets that link rewritten to the
new path.
A `<target>` that is an existing folder receives `<source>` under
its own name, the same as `mv`. Any other `<target>` is the new
path. Missing parent folders are created.
Stdout, paths with `/` separators:
```text
moved <source> -> <target>
rewrote <N> link(s) in <path>
unresolved <N> link(s) in <path>
read-only <N> link(s) in <path>
moved <F> file(s), rewrote <L> link(s) in <R> file(s),
<U> unresolved, <K> read-only
```
`rewrote`, `unresolved`, and `read-only` lines appear only for a
file whose count is above 0, sorted by `<path>`. `<path>` is the
file's path after the move. The summary line is always last and is
one line, even though it is wrapped above. With `--dry-run`, every line starts with `[dry-run]`
and one space, and nothing is written. A refusal (*7.3.3.2.*) prints one
`Error: ...` line on stderr, writes nothing, and exits 1. Success
exits 0.
### 4.2. mdoctor
No subcommand means `chklint .`.
`chklint [directory]` prints the shared report (section 6.1) with
header `PyMarkdownlint Violations` and exits 0 even when violations
exist.
`fix [directory]` runs pymarkdown auto-fix in place. Stdout:
```text
PyMarkdownFix: <N> file(s) fixed
- <path>
```
A pymarkdown API failure prints
`Error: pymarkdown fix failed: <error>` on stderr and exits 1.
The command does not skip `personal/` or `raw/`.
`NOT_IMPLEMENTED` `chklint` and `fix` also take one markdown file in
place of `[directory]` (*7.4.4.*). Today a file prints
`Error: '<path>' is not a valid directory.` and exits 1.
### 4.3. fmlint
`fiona-assistant fmlint` with no subcommand means `chklint .`.
`chklint [directory]` prints the shared report with header
`Frontmatter Lint Violations` and exits 0 even when violations
exist. A missing directory exits 1.
### 4.4. python-ruff
No subcommand means `chklint .`. The report header is
`Ruff Violations`. Exit code is 0 when the ruff process itself
succeeds, including when violations are listed. `lint(path, fix=True)`
passes `--fix` to the ruff binary. There is no `fix` subcommand yet.
### 4.5. json-lint
No subcommand means `chklint .`. Header is `JSONLint Violations`.
`chklint [directory]` exits 1 when the report contains violations
and exits 0 when it reports no violations. This differs from
mdoctor, fmlint, and python-ruff, which exit 0 either way.
`chklint --fix` and `fix [directory]` rewrite indentation and
trailing commas in place, skip `personal/` and `raw/`, and print:
```text
JSONLintFix: <N> file(s) fixed
- <path>
```
A missing directory on `fix` prints
`Error: '<path>' does not exist.` on stderr and exits 1.
### 4.6. git
Operates on the git repository at the current directory. Timestamps
are UTC.
| Command | Stdout on success |
| --- | --- |
| `save --description TEXT` | `Committed: <full sha>` |
| `snapshot <tag_name> --description TEXT` | `Snapshot created: commit=<sha>, tag=<tag>` |
| `restore <tag_name>` | `Restored to: <sha>` |
| `clean` | `Working tree cleaned. All changes discarded.` |
| `undo` | `Undo to: <sha>` |
| `status` | `Status: clean` or `Status: dirty`, then `git status` |
On failure the tool prints `Error: <message>` and exits 1.
Commit message: `<YYYY-MM-DD HH:MM:SS> <description>`.
`save` runs `git add --all` and commits. A clean tree, including
no untracked files, raises `Working tree is clean. Nothing to commit.`
`snapshot` commits, then creates an annotated tag
`<YYYY-MM-DD_HH-MM-SS>_<tag_name>` with message
`Snapshot: <description>`.
`restore` requires a clean tree
(`Working tree is dirty. Use 'clean' to discard current edits first.`).
It accepts the full tag or the user tag suffix. The latest tag whose
name equals the argument or ends with `_<argument>` wins. It checks
that commit's tree into the index and working tree and adds a new
commit `<timestamp> Restore to <full tag>`. History before that
commit stays.
`clean` requires a dirty tree
(`Working tree is already clean. Nothing to discard.`), then
`git reset --hard HEAD` and `git clean -fd`.
`undo` requires a clean tree and at least two commits
(`Cannot undo: only one commit exists in the repository.`), then
`git reset --hard HEAD~1`.
### 4.7. wiki
Subcommands of `fiona-assistant wiki`.
#### 4.7.1. wiki init
`fiona-assistant wiki init [directory]`. Success stdout:
```text
Successfully initialized llm-wiki directory at: <resolved path>
```
Rules are section 3.
#### 4.7.2. wiki check-update
`fiona-assistant wiki check-update [directory] [--json]`.
The directory must contain `fiona-settings.json`, else stderr
`Error: Directory '<path>' is not an llm-wiki directory` and exit 1.
A missing directory is
`Error: Directory not found: <path>` and exit 1.
A successful check exits 0 whether or not updates exist.
Human stdout when current:
```text
Wiki is up to date.
```
When something differs, the lines that apply:
```text
Files to update: <N>
- <relative path> (<current> -> <template>)
Conflicts (owner copy kept): <N>
- <relative path> (<current> -> <template>) -> <update file name>
Directories to add: <N>
- <relative path>
Files to add: <N>
- <relative path>
Total: <N> file(s) need attention
Files skipped (locally modified): <N>
- <relative path>
```
The version parenthesis appears only when both versions exist.
When the file was compared by hash instead (*7.1.2.12.*), the
parenthesis is `(sha256 <current> -> <template>)`, each side the
first 8 lowercase hex digits of that SHA-256. The
`Conflicts` block appears only when a conflict exists (*7.1.2.11.*).
`Total` counts updates, conflicts, and additions.
`--json` prints one JSON object:
```json
{
"files_to_update": [
{
"path": "<path>",
"current_version": "<version or null>",
"template_version": "<version or null>",
"current_sha256": "<64 hex or null>",
"template_sha256": "<64 hex or null>",
"is_settings_file": false
}
],
"conflicts": [
{
"path": "<path>",
"update_path": "<path>",
"current_version": "<version or null>",
"template_version": "<version or null>",
"current_sha256": "<64 hex or null>",
"template_sha256": "<64 hex or null>"
}
],
"files_to_add": [],
"directories_to_add": [],
"files_skipped": [
{"path": "<path>", "reason": "locally modified"}
],
"total": 0,
"has_updates": false
}
```
#### 4.7.3. wiki update
`fiona-assistant wiki update [directory] [-f|--force] [-y|--yes]
[--dry-run] [--json]`.
Without `--force` and `--yes`, stdout shows the check-update text
and the prompt `Proceed with update? [y/N]` followed by a space.
`y` and `yes` proceed.
Any other answer, and a non-TTY, exit 1 with stderr
`Error: Update requires confirmation. Use --force or --yes to proceed without prompting.`
`--dry-run` prints `Dry run - no files will be modified:` and the
check-update text, writes nothing, and exits 0.
Success stdout:
```text
Update complete: <U> file(s) updated, <A> added, <C> conflict(s), <S> skipped.
```
When `<C>` is not zero, one more line follows:
```text
Merge each <stem>-update<suffix> into its file: run fiona-assistant housekeeping.
```
`--json` suppresses the plan and prints:
```json
{
"files_updated": [],
"files_added": [],
"directories_added": [],
"conflicts": [{"path": "<path>", "update_path": "<path>"}],
"files_skipped": [],
"total": 0
}
```
A confirmed update that had something to report writes
`corefiles.json` at the wiki root (*7.1.2.14.*).
`check-update` and `--dry-run` do not. A pristine wiki still
writes nothing (*7.1.2.5.*). A path in
`fiona.wiki.update.ignore_files` is left unchanged (*7.1.2.15.*).
Comparison and merge rules are section 7.1.2.
#### 4.7.4. wiki sanity
`fiona-assistant wiki sanity [directory] [--json] [--non-interactive]`.
Requires `fiona-settings.json`. Exit 0 when every check is clean,
exit 1 otherwise.
Checks, in this order, one thread each: `markdownlint`,
`frontmatter_lint`, `python-ruff`, `mdlinter`, `jsonlint`.
Human report:
```text
Wiki Sanity Report
TARGET: <path>
<failing check reports, omitted when clean>
Results:
<name padded>: PASS - <N> violations (MM:SS)
SANITY PASSED (MM:SS)
```
A failure ends with
`Sanity Failed - <N> checks need attention:` and one detail line per
failing check. The pass banner carries the total wall time. The
failure verdict does not. Per-check times may overlap and need not
sum to the total. Rows stay in registry order.
`--json` skips the progress display and prints:
```json
{
"target": "<path>",
"checks": {
"markdownlint": {
"result": 0,
"violation_count": 0,
"duration_seconds": 0.0,
"violations": []
}
},
"total_violations": 0,
"passed": true
}
```
`result` is 0 for pass and 1 for fail. Progress display is
section 7.14.
#### 4.7.5. wiki graph
`fiona-assistant wiki graph [directory] [-o FILE] [--format html|json]
[--include-pattern GLOB] [--exclude-pattern GLOB]
[--wikilinks|--no-wikilinks] [--json]
[--external-links|--no-external-links] [-f|--force]
[--serve [--host HOST] [--port PORT] [--no-open]]`.
Defaults: output `wiki_graph.html` in the current directory, format
`html`, include `**/*.md`, wikilinks on, external-link nodes off.
`--json` selects JSON format. Behavior is section 7.1.4.
A path in `fiona.wiki.graph.ignore_files` is not a node
(*7.1.4.20.*). The shipped pattern `*/table_of_contents.md`
also matches `table_of_contents.md` at the scan root.
`NOT_IMPLEMENTED` `--serve` runs a local live view instead of
writing once (*7.1.4.19.*). `--host`, `--port`, and `--no-open`
are accepted only with `--serve`. `NOT_IMPLEMENTED` `-f/--force`
overwrites an existing output file without the prompt of
*7.1.4.5.9.* (*7.1.4.5.9.1.*).
#### 4.7.6. wiki ingest
The contract is section 7.1.5.
```text
fiona-assistant wiki ingest <source> [--wiki DIR] [-r|--raw PATH]
[-t|--target PATH] [--origin URL] [--raw-only] [-f|--force]
[--dry-run] [--json]
fiona-assistant wiki ingest --pending [--wiki DIR] [--json]
```
`<source>` is an `http://` or `https://` URL, or a local file path.
`--wiki` is the wiki root and defaults to `.`. It must contain
`fiona-settings.json`, else stderr
`Error: Directory '<path>' is not an llm-wiki directory` and exit 1.
`<source>` and `--pending` are mutually exclusive, and one of them is
required. argparse reports a violation and exits 2.
`NOT_IMPLEMENTED` `--origin URL` names the URL a local file was saved
from (*7.1.5.2.9.*).
Human stdout on success, the lines that apply, in this order:
```text
Fetched: <url> (<media type>, <N> bytes)
Archived: <raw path>
Archived: <raw path> (already archived, sha256 <first 12 hex>)
Duplicate: <page path> already cites this source.
Wrote: <page path> (converter <name>, conversion <complete|partial>)
Warning: <text>
Next: refine <page path>, then set status: refined.
```
`Fetched:` appears only for a URL. One `Warning:` line per warning.
With no converter, or a failed conversion, the raw copy stays and
stdout ends with:
```text
No converter for <media type>. Convert <raw path> by hand.
```
`--pending` stdout:
```text
Raw pages: <N>
- <page path> (<first sources entry>)
```
`--json` prints one object and nothing else on stdout. Paths are
relative to the wiki root and use `/`:
```json
{
"source": "<argument as given>",
"origin": "<url, or the resolved file path>",
"origin_kind": "url",
"media_type": "application/pdf",
"raw_path": "raw/papers/<slug>.pdf",
"raw_reused": false,
"sha256": "<64 lowercase hex>",
"duplicate_page": null,
"page_path": "wiki/sources/<slug>.md",
"converter": "pdf",
"conversion": "complete",
"status": "raw",
"warnings": [],
"dry_run": false
}
```
`origin_kind` is `url` or `file`. With no page, `page_path` and
`converter` are `null` and `conversion` is `none`. `duplicate_page`
is the page that already cites the matching raw copy, else `null`
(*7.1.5.3.5.*). `--pending --json`
prints `{"pending": [{"page_path": ..., "sources": [...],
"origin": ...}], "total": <N>}`.
| Exit | Meaning |
| --- | --- |
| 0 | Raw copy archived. A page was written, `--raw-only` was set, or a page already cites the same bytes. |
| 1 | Error on stderr. Nothing written under the wiki. |
| 2 | argparse usage error. |
| 3 | Raw copy archived. No page: no converter, or conversion failed. |
| 4 | `NOT_IMPLEMENTED` The URL needs a login (*7.1.5.2.8.*). Error on stderr. Nothing written under the wiki. |
Errors, each on stderr with exit 1:
```text
Error: Source not found: <path>
Error: Source not found: <path> (moved to <raw path> by an earlier ingest)
Error: Source is a directory: <path>
Error: Unsupported URL scheme: <scheme>
Error: Download failed: <url>: <reason>
Error: Download exceeds <N> bytes: <url>
Error: --raw must stay inside raw/: <value>
Error: --raw cannot be used on a file already under raw/: <path>
Error: --target must stay inside wiki/: <value>
Error: --target cannot replace <path>
Error: Page exists: <path>. Use --force to replace it.
Error: --origin needs a local file: <source>
Error: --origin must be an http or https URL: <value>
```
`NOT_IMPLEMENTED` A URL that needs a login prints two lines on stderr
and exits 4 (*7.1.5.2.8.*):
```text
Error: Authentication required: <url>: <reason>
Next: save it into ingest/, then run: wiki ingest <file> --origin <url>
```
The `Next:` line names the command without the `fiona-assistant`
prefix so it fits one line. The agent fills `<file>` with the path
it saved (*7.1.5.14.*).
A failed conversion prints `Error: Conversion failed: <converter>:
<reason>` on stderr and exits 3, not 1, because the raw copy stays.
`NOT_IMPLEMENTED` The `(moved to ...)` form is *7.1.5.2.7.*.
### 4.8. housekeeping
`fiona-assistant housekeeping [directory] [--json]`. One call runs
the wiki maintenance routine and tells an LLM what to fix. The
directory must contain `fiona-settings.json`, else stderr
`Error: Directory '<path>' is not an llm-wiki directory` and exit 1.
A missing directory is `Error: Directory not found: <path>` and
exit 1. The steps and their rules are section 7.2.
Human stdout:
```text
Housekeeping Report
TARGET: <path>
[<STATUS>] <step>: <summary>
- <detail>
Next steps:
- <action>
HOUSEKEEPING PASSED
```
One `[<STATUS>]` block per step, in step order. `<STATUS>` is
`PASS`, `WARN`, or `FAIL`. Detail lines appear only when the step
has details. `Next steps:` appears only when a step has an action.
The last line is `HOUSEKEEPING PASSED`, or
`HOUSEKEEPING FAILED - <N> step(s) need attention: <names>` with
the failing step names joined by a comma and a space.
`--json` prints one object and nothing else on stdout. Paths are
relative to the wiki root and use `/`:
```json
{
"target": "<path>",
"steps": [
{
"name": "merge",
"status": "FAIL",
"summary": "1 update file(s) wait for a merge",
"details": ["AGENTS-update.md -> AGENTS.md"],
"next": ["Merge AGENTS-update.md into AGENTS.md ..."]
}
],
"passed": false
}
```
| Exit | Meaning |
| --- | --- |
| 0 | No step is `FAIL`. `WARN` steps may exist. |
| 1 | A step is `FAIL`, or the directory is not a wiki. |
---
### 4.9. Repository runners
`python scripts/run_tests.py` runs pytest with `--color=no` and a
temporary JUnit XML report (`junit_family=xunit1`). The child
process receives the repository `src` directory on `PYTHONPATH`,
joined with `os.pathsep`.
`python scripts/run_sanity.py [--non-interactive]` runs, in order:
`pytests`, `markdownlint`, `ruff`, `frontmatter_lint`,
`pycustomlint`, `jsonlint`. Same progress display, result table, and
verdict shape as wiki sanity, with label `Running Sanity` and an
extra untimed `coverage:` row under pytests. Details are section 7.14.
## 5. Settings Schema
Two files. `fiona-settings.json` ships in the template and marks a
wiki root. `scripts/settings_dev.json` is repository tooling only and is
not copied into a wiki.
### 5.1. fiona-settings.json
Absent file: pymarkdown uses its own defaults; other tools use the
defaults in the table. A typo in a pymarkdown rule id or property
name is accepted and has no effect. Confirm a new pymarkdown setting
by its effect on a scan. TOC file settings reject unknown keys and
bad types with exit 1 and the key name.
Shipped document (logical content; `wiki init` copies the file):
```json
{
"markdown": {
"pymarkdown": {
"extensions": ["front-matter"],
"respect_gitignore": true,
"md025": {"front_matter_title": "no-match"},
"md013": {"line_length": 100, "code_blocks": false, "tables": false},
"md060": {"enabled": false}
},
"frontmatter": {
"ignore_rules": [],
"ignore_file": [],
"respect_gitignore": true
},
"toc_generator": {
"max_depth_from_file": 3,
"max_depth_from_dir": null,
"file": {
"min_heading": 2,
"max_heading": 6,
"sectionize": false,
"ignore_files": [],
"*/rules.md": {"max_heading": 2}
}
}
},
"json": {
"indent": "space",
"indent_len": 2,
"trailing_commas": false,
"require_key_order": false,
"check_duplicate_keys": true,
"extensions": [".json"]
},
"python": {"ruff": {}},
"fiona": {
"log_level": "INFO",
"file_log_level": "INFO",
"file_log_location": "fiona.log",
"llm-wiki": {"ignore_files": []},
"wiki": {
"graph": {
"show_external_links": false,
"subtitle": {"enabled": true},
"show_legend": true,
"theme": "night-sky",
"show_arrows": false,
"show_source": true,
"show_glow": true,
"show_link_card": true,
"first_fit_ms": 12000,
"hover_link_tail_chars": 12,
"simplify_external_links": false,
"consolidate_external_links": false,
"consolidate_external_links_depth": 0,
"consolidate_external_links_max_char": 8,
"ignore_files": ["*/table_of_contents.md"]
},
"update": {
"ignore_files": ["corefiles.json"]
}
}
}
}
```
| Key | Type | Default when absent | Consumer |
| --- | --- | --- | --- |
| `markdown.pymarkdown.extensions` | list of strings | none enabled | mdoctor |
| `markdown.pymarkdown.respect_gitignore` | bool | false | mdoctor |
| `markdown.pymarkdown.<rule>.<prop>` | bool, int, or str | pymarkdown default | mdoctor |
| `markdown.pymarkdown.path_rule_exemptions` | list of `{path, rules}` | `[]` | wiki sanity markdownlint |
| `markdown.frontmatter.required_fields` | list of strings | `["name", "version"]` | fmlint |
| `markdown.frontmatter.type_requirements` | object of lists | `{}` | fmlint |
| `markdown.frontmatter.enabled` | bool | true | fmlint |
| `markdown.frontmatter.ignore_rules` | list of codes | `[]` | fmlint |
| `markdown.frontmatter.ignore_file` | list of globs | `[]` | fmlint |
| `markdown.frontmatter.respect_gitignore` | bool | true | fmlint |
| `markdown.toc_generator.max_depth_from_file` | int | unlimited (`0` or absent) | folder TOC |
| `markdown.toc_generator.max_depth_from_dir` | int or null | unlimited | folder TOC |
| `markdown.toc_generator.file.min_heading` | int `>= 2` | 2 | in-file TOC |
| `markdown.toc_generator.file.max_heading` | int `>= min` | 6 | in-file TOC |
| `markdown.toc_generator.file.sectionize` | bool | false | in-file TOC |
| `markdown.toc_generator.file.ignore_files` | list of globs | `[]` | in-file TOC |
| `markdown.toc_generator.file.<glob>` | object of file keys | none | per-file override |
| `json.indent` | `"space"` or `"tab"` | `"space"` | json-lint |
| `json.indent_len` | int | 2 | json-lint |
| `json.trailing_commas` | bool | false | json-lint |
| `json.require_key_order` | bool | false | json-lint |
| `json.check_duplicate_keys` | bool | true | json-lint |
| `json.extensions` | list of strings | `[".json"]` | json-lint |
| `json.max_line_length` | int or absent | no line-length check | json-lint |
| `python.ruff` | object | `{}` | reserved, unused |
| `fiona.log_level` | level name | `WARNING` | logging |
| `fiona.file_log_level` | level name | `fiona.log_level` | file log |
| `fiona.file_log_location` | string or absent | no file log | file log |
| `fiona.llm-wiki.ignore_files` | list of globs | copy all | wiki init, update |
| `fiona.wiki.update.ignore_files` | list of globs | `[]` | wiki update, housekeeping merge |
| `fiona.wiki.graph.wikilinks_enabled` | bool | true | wiki graph |
| `fiona.wiki.graph.max_files` | int | 5000 | wiki graph |
| `fiona.wiki.graph.show_external_links` | bool | false | wiki graph toggle |
| `fiona.wiki.graph.subtitle.enabled` | bool | true | wiki graph toggle |
| `fiona.wiki.graph.show_legend` | bool | true | wiki graph toggle |
| `fiona.wiki.graph.show_arrows` | bool | true | wiki graph toggle |
| `fiona.wiki.graph.show_source` | bool | true | wiki graph toggle |
| `fiona.wiki.graph.show_glow` | bool | true | wiki graph |
| `fiona.wiki.graph.show_link_card` | bool | true | wiki graph |
| `fiona.wiki.graph.first_fit_ms` | number | 12000 | wiki graph |
| `fiona.wiki.graph.hover_link_tail_chars` | int | 12 | wiki graph |
| `fiona.wiki.graph.theme` | string | `night-sky` | wiki graph |
| `fiona.wiki.graph.simplify_external_links` | bool | false | wiki graph |
| `fiona.wiki.graph.consolidate_external_links` | bool | false | wiki graph |
| `fiona.wiki.graph.consolidate_external_links_depth` | int | follow the bool | wiki graph |
| `fiona.wiki.graph.consolidate_external_links_max_char` | int | 8 | wiki graph |
| `fiona.wiki.graph.ignore_files` | list of globs | `[]` | wiki graph |
| `fiona.wiki.graph.node_importance_min_point` | int `>= 0` | 0 | wiki graph |
| `fiona.wiki.graph.directory_nodes.min_files` | int `>= 0` | 0 (off) | wiki graph |
| `fiona.wiki.graph.directory_nodes.outside_depth` | int `>= 0` | 1 | wiki graph |
| `fiona.wiki.graph.serve.host` | string | `127.0.0.1` | wiki graph `--serve` |
| `fiona.wiki.graph.serve.port` | int `0..65535` | 8765 | wiki graph `--serve` |
| `fiona.wiki.graph.serve.poll_seconds` | number `> 0` | 1 | wiki graph `--serve` |
| `fiona.wiki.graph.serve.debounce_ms` | int `>= 0` | 300 | wiki graph `--serve` |
| `fiona.wiki.ingest.timeout_seconds` | number `> 0` | 30 | wiki ingest |
| `fiona.wiki.ingest.max_download_bytes` | int `> 0` | 52428800 | wiki ingest |
| `fiona.wiki.ingest.user_agent` | string | `fiona-assistant/<package version>` | wiki ingest |
| `fiona.wiki.ingest.default_target` | string | `sources` | wiki ingest |
| `fiona.wiki.ingest.raw_categories` | object of strings | `{}` | wiki ingest |
| `fiona.wiki.ingest.login_url_patterns` | list of regex strings | see below | wiki ingest |
Bool pymarkdown properties use `set_boolean_property`, ints use
`set_integer_property`, strings use `set_string_property`, applied as
`plugins.<rule_id>.<property>`. `"enabled": false` inside a rule map
disables that rule.
`markdown-tables` stays off. pymarkdownlnt 0.9.40 raises
`BadTokenizationError` on list lines that contain pipes, which breaks
sanity on generated contents files. Until that is fixed, table rules
MD055, MD056, MD058, and MD060 do not run. MD060 is also disabled
because pymarkdown's delimiter-width check disagrees with Node
markdownlint compact tables. Project tables use compact style:
one space around cell text, including the delimiter row
(`| --- | ------- |`).
Graph colors, sizes, and shapes come from section 5.3. If
`node_colors`, `node_radius`, or `hover` is still present here, those
keys override the `colorful` theme only. Hover radius is
`min(current * size_multiplier, max)` and returns to the pre-hover
radius when the pointer leaves. `subtitle.enabled` starts the
Show Subtitles toggle. The
label is the frontmatter `name` when that field is a non-empty
string, otherwise the filename. `show_legend` starts the Show
Legend toggle. The default is on. The card sits at the upper
right of the graph. It groups shape, node size and color, and
edge color, so those three are not one mixed list.
The graph command starts from the graph section of the packaged
`fiona-settings.json`. A `fiona-settings.json` found by walking up
from the scanned directory replaces the keys it sets. A missing
file still uses that packaged section.
`show_arrows` starts the Show Arrows toggle. The default is on.
`show_source` starts the Show Sources toggle. The default is on.
It draws each frontmatter `source` value as its own node.
`show_glow` draws the glow on a hovered node, a clicked node,
and a search match. The default is on. `show_link_card` draws
the hover card beside the node. The default is on. A click
shows the frontmatter card at the upper left, including when
`show_link_card` is false. `first_fit_ms` is the first node's
fit, in milliseconds. The shipped value, and the value when
the key is absent or not a positive number, is 12000. See all,
the last node, and a theme change stay at 4200 ms.
`hover_link_tail_chars` is how many characters of a URL's last
path segment the hover card keeps. The shipped value, and the
value when the key is absent or not a positive number, is 12.
The arrowhead sits on the target end of a visible edge, so the
source file points at the target file. Turning the toggle off
removes the arrowheads. `theme` names the theme the page opens
on. The shipped value is `night-sky`. An unknown name also opens
`night-sky`. `simplify_external_links` shortens an external
subtitle to the host. The default is off.
`consolidate_external_links` draws one node per host, named
`host/*`, when depth is absent. The default of the bool is off.
`consolidate_external_links_depth` is how many levels to keep.
`0` leaves each URL alone. `1` is `host/*`. `2` is `host/a/*`.
`3` is `host/a/b/*`. A missing depth follows the bool: true is
`1`, false is `0`. A path part longer than
`consolidate_external_links_max_char` is `...`. That limit is
`8` when absent or not a positive integer. Turning depth above
`0` also shortens the subtitle to the host.
`node_importance_min_point` is how many different files must
point at a node before the HTML graph draws it. The count is
`unique_points` (*7.1.4.2.3.*). `0` turns the filter off and
every node is drawn. A missing key, or a value that is not an
integer of 0 or more, is `0`. A positive integer is *7.1.4.16.*
and is not built yet. The packaged `fiona-settings.json` omits
the key until that requirement is built. Absent means `0`.
`directory_nodes.min_files` is how many markdown pages a
directory must hold directly before the HTML graph draws it as
one directory node (*7.1.4.18.*). `0` turns consolidation off.
`directory_nodes.outside_depth` is how many hops past a
directory's own pages its view draws (*7.1.4.18.5.*).
`serve.host`, `serve.port`, `serve.poll_seconds`, and
`serve.debounce_ms` are the `--serve` defaults (*7.1.4.19.*).
A CLI flag wins over these keys. A value that is absent, of the
wrong type, or out of range uses the default in the table.
Both are not built yet. The packaged `fiona-settings.json`
omits `directory_nodes` and `serve` until they are built.
`fiona.wiki.ingest` is read by section 7.1.5. `default_target` is a
directory relative to `wiki/`. `raw_categories` maps a converter
name to a directory under `raw/`, such as `{"text": "transcripts"}`.
A value that is absent, of the wrong type, or out of range uses the
default in the table. The packaged `fiona-settings.json` omits
`fiona.wiki.ingest`. Absent keys use the defaults in the table.
`login_url_patterns` is read by *7.1.5.2.8.3.* and is not built
yet. A list replaces the default list, it does not extend it. An
entry that does not compile is skipped with one warning. The
default list:
```json
[
"/(login|logon|signin|sign-in)\\b",
"/(sso|saml2?|oauth2?|adfs|cas)(/|$)",
"login\\.microsoftonline\\.com",
"j_security_check"
]
```
A mapping key under `toc_generator.file` whose name is not one of
`min_heading`, `max_heading`, `sectionize`, `ignore_files` is an
fnmatch override against the file path relative to the wiki root.
On `wiki update`, a differing `fiona-settings.json` is not
rewritten (*7.1.2.3.*). The template is written beside it as
`fiona-settings-update.json`. The owner merges that file: start
from the template object. For each user key, if both sides are
objects, merge recursively; otherwise the user value replaces the
template value. New template keys that the user does not set are
kept. Write the result as JSON with indent 2 and a trailing
newline, then delete the update file.
- *5.1.1.* Changing `md013.line_length` changes the next mdoctor scan
with no code change.
- *acceptance*: A line between the old and new limit is clean after
the settings change and reported before it.
- *5.1.2.* `wiki update` leaves `fiona-settings.json` byte-for-byte
unchanged and writes the template as `fiona-settings-update.json`.
- *acceptance*: `markdown.pymarkdown.md013.line_length` set to 120
is still 120 in the owner file after `wiki update --yes`, and
`fiona-settings-update.json` is the template copy.
### 5.2. settings_dev.json
```json
{
"pytests": {"required-coverage": 96},
"progress": {"interval": 3, "expected-seconds": 30}
}
```
| Key | Default when absent | Meaning |
| --- | --- | --- |
| `pytests.required-coverage` | (required for sanity) | Minimum total coverage percent |
| `progress.interval` | 3 | Seconds between non-interactive dots |
| `progress.expected-seconds` | 30 | Bar scale for a running check |
- *5.2.1.* Coverage below `required-coverage` fails sanity when every
test passed.
- *acceptance*: Raising the number in this file flips a green run
to failed with no code change.
### 5.3. constellation themes
Themes for the wiki graph live in `constellation/`. Each theme is
one JSON file. The file name without `.json` is the theme name.
`colorful` is complete and listed first. The other names follow
in alphabetical order. Any other theme may omit keys. A missing
key is taken from `colorful`. A `dark` object on a theme is the
same kind of partial theme, used when the page is in dark mode.
Each file has a `version` string (semver). Bump it when that
theme changes. `version` is not a color key. A theme that omits
it does not take `colorful`'s version.
`colorful` is a warm paper field. Nodes run from warm gray
through amber to crimson. Edges use that same warm ramp, darker
so a thin line stays visible. Hover, click, and buttons are one
teal, the complement of the amber. Pages are circles and
external links are clouds. `night-sky` draws
circle pages and star external links on a dark field.
`grayscale` runs charcoal to near-black on a light field, with
a dark stroke so the pale end still stands off the canvas. Its
`dark` object is a gray night on a charcoal field, not black.
Sparse nodes stay medium gray. A mid link count is white-yellow.
Many incoming links are a light yellow-orange. `dark` is a blue-slate night. Nodes
run from slate through white-yellow to a light gold. The page and the canvas
stay blue-gray, not black. `professional` is a light boardroom
palette: pale blue nodes, a soft blue stroke, light blue
buttons, and a quiet radius.
| Key | Meaning |
| --- | --- |
| `background` | Page background |
| `graph_background` | Graph canvas background |
| `font_family` | Font for the page |
| `text_color` | Text on the page and cards |
| `node_colors.low/mid/high` | Node color from few to many incoming links |
| `node_stroke` | Outline around a node, or `none` |
| `node_radius.min/max` | Smallest and largest node radius, in em. 16 is 1em. Fractions are kept |
| `subtitle.min_radius` | Label cutoff, a fraction of `node_radius.max` from 0 to 1 |
| `subtitle.padding` | Em between the shape and the label, where 16 is 1em. Smaller sits closer |
| `subtitle.size` | Label font size in em, where 16 is 1em |
| `subtitle.on_top` | When true, labels paint above every node |
| `subtitle.outline` | When true, a stroke keeps the label readable over nodes |
| `subtitle.outline_color` | Stroke color for that outline |
| `subtitle.outline_width` | Stroke width in em, where 16 is 1em |
| `edge_colors.low/mid/high` | Edge color from one link to many |
| `edge_width` | Edge thickness in em, where 16 is 1em. `1` when absent or not positive |
| `highlight.node_color` | Node color while the pointer is over it |
| `highlight.edge_color` | Edge color while a node is highlighted |
| `highlight.size_multiplier` | Radius multiplier while highlighted |
| `clicked.node_color` | Node color after a click |
| `buttons.background/text/hover/disabled` | Button colors |
| `card.background/border` | Cards, legend, and the control bar |
| `shapes.page` | `circle`, `cloud`, `star`, or `folder` |
| `shapes.external` | Shape for an external-link node |
| `shapes.directory` | Shape for a directory node (*7.1.4.18.*). `folder` in `colorful` |
| `subtitle_color` | Label under a large node |
| `version` | Semver of this theme file. Not a color key |
`subtitle.padding` is 12 when absent. `subtitle.size` is 11 when
absent or below 1. `subtitle.on_top` and `subtitle.outline` are
false when absent. `subtitle.outline_width` is 3 when absent.
`subtitle.outline_color` uses `graph_background` when absent.
A theme that sets only some of these keeps the rest from
`colorful`.
- *5.3.1.* A theme that omits `font_family` uses the `colorful`
theme font. The HTML theme control lists `colorful`, `night-sky`,
`grayscale`, `dark`, and `professional`.
- *acceptance*: The resolved grayscale theme font is the colorful
theme font. The graph HTML contains a theme select and those names.
- *5.3.2.* Subtitle gap, size, stacking, and outline come from the
active theme's `subtitle` object.
- *acceptance*: The resolved grayscale subtitle padding, size,
`on_top`, and outline match `colorful`. The graph HTML reads
those keys from the theme.
- *5.3.3.* Each theme is one JSON file in `constellation/`. The
name is the file stem. `colorful` is first in the catalog. The
other names follow in alphabetical order. Each file has a
`version` string. Filling a theme does not copy `version` from
`colorful`, in the loader or in the page script.
- *acceptance*: The template has `constellation/colorful.json`,
`constellation/night-sky.json`, `constellation/grayscale.json`,
`constellation/dark.json`, and `constellation/professional.json`,
and no `constellation-settings.json`. Each resolved packaged
theme `version` equals the `version` in its file. A theme
object that omits `version` has no `version` after it is
filled from `colorful`.
- *5.3.4.* A `constellation/` directory at the wiki, or in a
parent, overlays the packaged themes. Each JSON file there
overlays the theme of the same name. `constellation-settings.json`
in that same directory overlays those files. The nearest
directory that has either source wins. A theme file that is
not a JSON object is skipped.
- *acceptance*: `constellation/night-sky.json` setting
`background` changes that key and keeps the packaged page
shape. A `constellation-settings.json` beside it replaces
`background` with its own value. A file of `{` does not
remove `colorful`.
---
## 6. Violation Rule Catalog
### 6.1. Shared Report
> Status: Implemented
Custom linters and the ruff wrapper print this shape. Sanity runners
omit the report for a check with zero violations; the `Results:` row
still shows the count.
```text
<Linter Name> Violations
TARGET: <path>
- DEFINITIONS:
- <code>: <rule_name>, <description>
- FILE: <path>
- <message>
- SUMMARY:
- Total files with violations: <N>
- Total violations: <N>
```
A clean standalone run that still prints a report uses
`- No violations found.` in place of the FILE section and still
prints DEFINITIONS and SUMMARY. Headers:
| Tool | Header name |
| --- | --- |
| mdoctor | PyMarkdownlint |
| fmlint | Frontmatter Lint |
| python-ruff | Ruff |
| pycustomlint | Python Custom Lint |
| mdlinter | Markdown Lint |
| json-lint | JSONLint |
Frontmatter messages omit a `line:column` prefix. Frontmatter is
always at the start of the file.
- *6.1.1.* A report with one violation contains TARGET, DEFINITIONS,
one FILE entry, and SUMMARY, in that order.
- *acceptance*: The three custom linters and ruff use this order.
### 6.2. Frontmatter Fields and FML Codes
> Status: Implemented
Required on every markdown file: `name`, `version`. Required on every
page under `wiki/`: `confidence`, `llm`, `contested`, and
`contradictions`. The Fiona skill teaches how to set those four.
fmlint enforces `name` and `version` only. All other fields are
optional.
| Field | Required | Rule |
| --- | --- | --- |
| `name` | yes | Document name. Wiki pages use lowercase and hyphens. |
| `version` | yes | Semver. Bump when the page is edited. |
| `title` | no | Human title. |
| `created` | no | `YYYY-MM-DD` |
| `updated` | no | `YYYY-MM-DD`. Bump when the page is edited. |
| `type` | no | `entity`, `concept`, `comparison`, `query`, `summary`, `toc` |
| `tags` | no | Taxonomy tags from `wiki/schema.md` |
| `sources` | no | Paths such as `raw/articles/source-name.md` |
| `confidence` | wiki pages | `trusted`, `high`, `medium`, `low`, or `zero`. The agent sets it on every page under `wiki/`. fmlint does not enforce it. |
| `llm` | wiki pages | `model` is required. `reasoning` is set when the model reports a level. The agent sets it on every page under `wiki/`. fmlint does not enforce it. |
| `contested` | wiki pages | `true` when a claim on the page is disputed, otherwise `false`. The agent sets it on every page under `wiki/`. fmlint does not enforce it. |
| `contradictions` | wiki pages | Other page slugs that disagree. `[]` when none. The agent sets it on every page under `wiki/`. fmlint does not enforce it. |
| `description` | SKILL.md only | Required on a file whose name is `SKILL.md`. |
| `status` | no | `raw` or `refined`. `wiki ingest` writes `raw`. Only the agent sets `refined` (*7.1.5.10.*). |
| `ingest` | no | Map of provenance written by `wiki ingest`: origin, media type, SHA-256, size, converter, conversion, warnings (*7.1.5.5.3.*). |
| `source_meta` | no | Map of metadata read from the source, such as author and dates. `{}` when none. |
`markdown.frontmatter.type_requirements` maps a `type` value to the
field list required for that type, in addition to the global list.
| Code | Name | Message | Fails when | Passes when |
| --- | --- | --- | --- | --- |
| FML001 | Frontmatter Parse Error | `FML001 - <error>` | YAML between `---` markers is invalid | YAML parses |
| FML002 | Missing Frontmatter | `FML002` | No frontmatter block | A block is present |
| FML003 | Missing Name Field | `FML003` | `name` absent | `name` present |
| FML004 | Missing Description | `FML004` | `SKILL.md` has no `description` | `description` present |
| FML005 | Invalid Skill Name | `FML005` | `SKILL.md` `name` fails `^[a-z0-9-]+$` | `name: python-tdd` |
| FML006 | Skill Name Mismatch | `FML006 - (expected: <folder>, got: <name>)` | `name` differs from the parent folder | folder `python-tdd` and `name: python-tdd` |
| FML007 | Missing Version Field | `FML007` | `version` absent | `version` present |
`SKILL.md` is case-sensitive. FML005 and FML006 apply only to that
filename. Scan skips `.gitignore` matches when
`respect_gitignore` is true, `ignore_file` globs, and read-only
zones.
- *6.2.1.* A markdown file with `name` and `version` and no `title`
produces no FML violation.
- *acceptance*: fmlint on that file reports no violations.
- *6.2.2.* `SKILL.md` in `python-tdd/` with `name: Python TDD` is
FML005, and `name: other-name` is FML006.
- *acceptance*: The report code matches the case above.
- *6.2.3.* Every page under `wiki/` includes `confidence`, `llm`,
`contested`, and `contradictions`. The Fiona skill requires them
and states how to choose each value. `confidence` is `trusted`
(the user stated it, or it is copied from a cited source),
`high` (more than one source agrees, or one source and no known
contradiction), `medium` (one source, or a synthesis the agent
is reasonably sure of), `low` (inference, thin evidence, or an
open gap), or `zero` (unknown, or not checked). `llm` is a list
whose `model` entry is required: the model that last edited the
page, or `none` when no model wrote it. `reasoning` is included
when that model reports a level. `contested` is `true` when any
claim on the page is disputed by a source or another page, and
`false` otherwise. `contradictions` lists the `name` of each
page that disagrees, and is `[]` when none do. fmlint enforces
`name` and `version` only.
- *acceptance*: Fiona skill section 15 marks those four fields
required on wiki pages and states the choice rules above.
### 6.3. MDL Codes
> Status: Implemented
mdlinter scans `.md` files. Fenced code, inline code, links whose
target starts with `http://`, `https://`, `mailto:`, or `#`, and any
target containing `://`, are ignored. Sources under `personal/` and
`raw/` are skipped. Targets inside those zones are still resolved.
| Code | Name | Message | Fails when | Passes when |
| --- | --- | --- | --- | --- |
| MDL001 | Dead Link | `line <n>: <target> does not exist` | A relative `](target)` does not resolve to a file | The relative file exists |
| MDL002 | Dead Wiki Link | `line <n>: [[<target>]] does not resolve to any page` | A `[[wikilink]]` matches no markdown stem | A stem matches, case-insensitive |
| MDL003 | Wrapped Table Row | `line <n>: table row continues onto the next line; keep each row on one line` | A line starts with `\|`, does not end with `\|`, and the next line is not blank and not a new row. A following line that is only `\|` counts as a continuation | The row is one line, or the next line is blank or a new row, or the split is inside a fence |
Wikilink syntax: `[[page]]`, `[[page#heading]]`, `[[page|alias]]`.
The lookup key is the text before `#` or `|`, compared to markdown
file stems in the wiki, case-insensitive. Markdown links resolve
against the linking file's directory.
- *6.3.1.* A link to a deleted file is MDL001 with file, line, and
target. A `[[missing]]` wikilink is MDL002.
- *acceptance*: Both codes appear in one wiki that has one of each.
- *6.3.2.* A table row split across two lines is MDL003 on the opening
line. The same split inside a fence is clean.
- *acceptance*: Only the unfenced split is reported.
### 6.4. PYC Codes
> Status: Implemented
pycustomlint scans `.py` files with the AST, skips `.gitignore`
matches and read-only zones, and accepts `--json` for a JSON list of
violation objects.
| Code | Name | Severity | Message | Fails when | Passes when |
| --- | --- | --- | --- | --- | --- |
| PYC001 | Inline Import | error | `PYC001 - Inline import at line <n>: <stmt>` | `import` or `from` is indented inside a function, class, `try`, or an `if` that follows a function or class | Every import is at column 0, or it sits in a module-level `if` before the first function or class |
| PYC002 | Import After Code | warning | `PYC002 - Import after code at line <n>: <stmt>` | A module-level import follows a non-import statement | Imports are grouped before other module statements |
| PYC003 | Late Declaration | error | `PYC003 - Declaration after first function or class at line <n>: <stmt>` | A module-level assignment is at or after the first function or class in the module body | Every module-level assignment is before that first function or class |
- *6.4.1.* `import os` inside a function is PYC001.
- *acceptance*: The report names the file, the line, and `import os`.
- *6.4.2.* An import inside a module-level `if`, `elif`, or `else`
that appears before the first function or class is not PYC001.
The same import after that first function or class is PYC001.
A `try` does not get this exception. An import inside a function
or class is PYC001.
- *acceptance*: `if os.name == "nt": import msvcrt` before any
function is clean. The same `if` after a function is PYC001
and the message names `import msvcrt`.
- *acceptance*: `try: import os` before any function is PYC001.
- *6.4.3.* A module-level assignment at or after the first function
or class statement in the module body is PYC003. An assignment
nested inside a function or class is not PYC003. An assignment
before that first function or class, including one inside a
module-level `if`, is not PYC003.
- *acceptance*: `SHORT_HASH_LENGTH = 8` after a function is PYC003
and the message names `SHORT_HASH_LENGTH = 8`.
- *acceptance*: The same assignment before any function or class
is clean.
- *acceptance*: `value = 1` inside a function, and `VALUE = 1`
inside a class, are clean.
- *acceptance*: `VALUE = 1` inside a module-level `if` before any
function is clean. The same `if` after a function is PYC003.
### 6.5. JSN Codes
> Status: Implemented
json-lint parses with the standard `json` module. Style checks use
section 5.1. Duplicate-key detection uses a parse hook so the last
value does not hide the earlier key.
| Code | Name | Severity | Fails when | Passes when |
| --- | --- | --- | --- | --- |
| JSN001 | JSON Syntax Error | error | The file is not JSON, including a trailing comma when that is a syntax error | `json.loads` succeeds |
| JSN002 | Indentation Error | error | Indent differs from `indent` / `indent_len` | Indent matches |
| JSN003 | Trailing Comma | error | A trailing comma is present and `trailing_commas` is false | No trailing comma, or the setting allows them |
| JSN004 | Duplicate Keys | error | An object repeats a key and `check_duplicate_keys` is true | Keys are unique |
| JSN005 | Key Order | warning | `require_key_order` is true and keys are not sorted | Keys are sorted, or the setting is false |
| JSN006 | Line Too Long | warning | `max_line_length` is set and a line exceeds it | The key is absent, or every line fits |
- *6.5.1.* A file with a missing comma reports JSN001 with line and
column. A file with duplicate keys reports JSN004.
- *acceptance*: Both cases name the file and the code.
- *6.5.2.* Four-space indent against `indent_len: 2` is JSN002.
`json-lint fix` rewrites it to two spaces.
- *acceptance*: The fixed file is clean on the next `chklint`.
### 6.6. Base Linter Contract
> Status: Implemented
- *6.6.1.* Shared behavior for fmlint, pycustomlint, mdlinter, and
json-lint.
- *6.6.1.1.* Recursive file collection by extension.
- *6.6.1.2.* `.gitignore` and read-only-zone skipping.
- *6.6.1.3.* Settings load from `fiona-settings.json`.
- *6.6.1.4.* A violation record has filepath, line, column, code,
message, rule id, and severity, plus a dict form for JSON.
- *6.6.1.5.* Report assembly follows section 6.1. The header is the
linter name.
- *6.6.1.6.* The CLI is `chklint [directory]` and defaults to `.`.
- *6.6.2.* A new rule is a new checker used by one linter. It does
not require an edit to the shared collector. Codes are unique:
`FML`, `MDL`, `PYC`, `JSN`, plus pymarkdown `MD` codes and ruff
codes.
- *acceptance*: Existing FML and PYC tests still pass after a new
MDL checker is registered.
---
## 7. Feature Requirements
### 7.1. wiki
Requirements for the `wiki` subcommands.
#### 7.1.1. wiki init
> Status: Partially Implemented
Implements sections 3 and 4.7.1.
- *7.1.1.1.* Copy is 1:1 aside from `DATE_PLACEHOLDER`,
`ignore_files`, and paths matched by the repository `.gitignore`.
- *acceptance*: A pattern in `ignore_files` omits those paths and
copies every other template file.
- *acceptance*: A template file matched by the repository
`.gitignore` is not copied.
- *7.1.1.2.* Copied markdown keeps `version` and receives
`created` / `updated` dates where the template used the placeholder.
- *acceptance*: After init, those fields match `YYYY-MM-DD` for
the UTC day of the run.
- *7.1.1.3.* `wiki init` puts or appends `.gitignore` from the
repository `.gitignore`, plus `personal/` when that pattern is
missing. Lines already in the file stay.
- *acceptance*: After init, `.gitignore` contains `__pycache__/`
and `personal/`. A line `keep-me` written before init is still
in the file.
- *7.1.1.4.* `NOT_IMPLEMENTED` The CLI turns the refusal of *3.2.2.*
into one stderr line and exit 1 (*1.2.7.*):
`Error: Directory '<path>' already contains llm-wiki files:
<names>. Run fiona-assistant wiki update to refresh the template.`
The line is one line, even though it is wrapped here. `<names>` is
the comma-separated list from the `FileExistsError` message.
- *acceptance*: A second `wiki init` exits 1, stderr is exactly that
line, and no file changes.
#### 7.1.2. wiki update
> Status: Implemented
Implements sections 4.7.2 and 4.7.3.
- *7.1.2.1.* Markdown core files are compared as follows. An existing
markdown file is never overwritten.
- *7.1.2.1.1.* Both sides have `version`, and the template version is
greater: conflict (*7.1.2.11.*). Numeric dotted versions compare
segment by segment (`1.2.0` vs `1.10.0`). Any other string
compares as text.
- *7.1.2.1.2.* The wiki version is greater, or the versions are equal
and the body differs after expanding `DATE_PLACEHOLDER` in the
template: skip the file and report `locally modified`.
- *7.1.2.1.3.* Either version is missing: compare SHA-256
(*7.1.2.12.*). Equal hashes need nothing. Different hashes are a
conflict.
- *7.1.2.2.* Other core files compare SHA-256 (*7.1.2.12.*) and update
when the hash differs.
- *7.1.2.3.* `fiona-settings.json` uses the hash comparison of
*7.1.2.12.*. Different hashes are a conflict (*7.1.2.11.*). The
owner's file stays. The template is written as
`fiona-settings-update.json`. The tool does not merge the two.
- *acceptance*: After `wiki update --yes`, the owner settings
bytes are unchanged and `fiona-settings-update.json` matches
the template.
- *7.1.2.4.* Missing template directories outside `wiki/` and outside
read-only zones are created when they hold a file that update may
add. User files that are not in the template stay. Read-only zones
are never in the update set.
- *7.1.2.5.* A pristine init reports `Wiki is up to date.` and
`wiki update` writes nothing.
- *acceptance*: A second check-update still reports up to date.
- *7.1.2.6.* An older core markdown file is listed as a conflict.
After a confirmed update its bytes are unchanged and its update
file matches the template. A locally modified file is listed
under skipped and is not overwritten.
- *acceptance*: The modified file's bytes are unchanged after
`wiki update --yes`.
- *acceptance*: An `AGENTS.md` at `1.0.0` against a template at
`2.0.0` keeps its bytes after `wiki update --yes`, and
`AGENTS-update.md` holds the expanded template text.
- *7.1.2.7.* `--dry-run` prints the plan and leaves the tree unchanged.
- *acceptance*: File hashes before and after a dry run match.
- *7.1.2.8.* `--force` and `--yes` skip the prompt, including on a
non-TTY.
- *acceptance*: The command exits 0 on a pipe when `--yes` is set
and updates exist.
- *7.1.2.9.* A path under `wiki/` is never added, overwritten, or
listed as a change, an addition, a skipped file, or a directory to
add. A template page under `wiki/` that is newer than the wiki's
copy (*7.1.2.1.1.* or *7.1.2.1.3.*) is listed as a conflict, the same
as any other markdown core file.
- *acceptance*: A `wiki/index.md` older than the template keeps
its bytes after `wiki update --yes`, and `wiki/index-update.md`
holds the expanded template text.
- *acceptance*: A `wiki/index.md` newer than the template, or
missing, is absent from the check-update report.
- *7.1.2.10.* Update skips a path matched by the repository
`.gitignore` or by the wiki's `.gitignore`. Comment lines and
blank lines are not patterns. A pattern with no slash matches any
path segment. A pattern that ends with `/` matches that directory
and everything under it.
- *acceptance*: A template file
`utils/scripts/__pycache__/example.pyc` is absent from the
check-update report.
- *acceptance*: When the wiki `.gitignore` lists `example.py`, a
drifted `utils/scripts/example.py` is left unchanged and is
absent from the report.
- *7.1.2.11.* A conflict means the template copy and the owner's
copy both exist and the tool must not overwrite the owner. That
is a newer markdown file (*7.1.2.1.*) or a differing
`fiona-settings.json` (*7.1.2.3.*). Update yields to the owner:
it keeps the file and writes the template beside it. The LLM
merges the two during housekeeping (*7.2.*).
- *7.1.2.11.1.* The update file sits in the same directory. Its name
is the file stem, then `-update`, then the suffix:
`AGENTS.md` gives `AGENTS-update.md`, `wiki/schema.md` gives
`wiki/schema-update.md`, and `fiona-settings.json` gives
`fiona-settings-update.json`. Relative links in the template still
resolve from that directory.
- *acceptance*: A conflict on `wiki/schema.md` writes
`wiki/schema-update.md` and `corefiles.json`, and no
other file.
- *7.1.2.11.2.* The update file holds the template text with
`DATE_PLACEHOLDER` expanded, as `wiki init` would write it. An
existing update file is replaced with the current template
text.
- *acceptance*: A second `wiki update --yes` leaves one update
file, with the same bytes as the first run on the same day.
- *7.1.2.11.3.* Only a confirmed update writes an update file.
`check-update` and `--dry-run` list the conflict and write
nothing.
- *acceptance*: No `*-update.md` exists after
`wiki update --dry-run`.
- *7.1.2.11.4.* A conflict clears when the owner's `version` is at
least the template's version. The merge sets that version and
deletes the update file.
- *acceptance*: After `AGENTS.md` is set to the template version
with the owner's body, check-update lists no conflict for it.
- *7.1.2.12.* A file without version control, meaning a markdown file
with no `version` on either side or any non-markdown file, is
compared by SHA-256. The wiki side and the template side both use
the text hash in *7.1.2.13.*. The template side expands
`DATE_PLACEHOLDER` before that hash. A binary file hashes its raw
bytes.
- *7.1.2.12.1.* Equal hashes: the file is up to date and is not
listed.
- *acceptance*: A freshly initialized wiki reports
`Wiki is up to date.` even when a template file holds
`DATE_PLACEHOLDER`.
- *7.1.2.12.2.* Different hashes: the report line shows both hashes
where a version pair would go, shortened to 8 hex digits:
`(sha256 1a2b3c4d -> 9f8e7d6c)`. `--json` carries the full
64-digit `current_sha256` and `template_sha256`. A file compared
by version has `null` in both.
- *acceptance*: A drifted `utils/scripts/example.py` is listed as
`utils/scripts/example.py (sha256 <8 hex> -> <8 hex>)`, and a
markdown file with no `version` that differs is a conflict with
the same parenthesis.
- *7.1.2.13.* The SHA-256 of a UTF-8 text file is the digest of that
text after every `\r\n` and every lone `\r` is folded to `\n`,
then encoded as UTF-8. Two text files that differ only in line
endings are equal. A file that is not valid UTF-8 hashes its raw
bytes, with no folding.
- *acceptance*: `fiona-settings.json` saved with CRLF against a
template saved with LF, with the same characters, is not a
conflict and is absent from the check-update report.
- *7.1.2.14.* A confirmed `wiki update` that had a file to update,
add, or conflict writes `corefiles.json` at the wiki root.
`check-update` and `--dry-run` do not write it. A pristine wiki
writes nothing (*7.1.2.5.*). The file is not part of the template
and is never itself an update, a conflict, an addition, or a
merge candidate (*7.2.3.*). Only this command writes it. It is
one JSON object:
```json
{
"files": {
"<relative posix path>": {
"version": "<owner version or null>",
"template_version": "<template version or null>",
"sha256": "<64 hex>",
"template_sha256": "<64 hex>",
"date": "YYYY-MM-DD"
}
}
}
```
Each entry is a core file the update compared, using that file's
bytes after the update. `date` is the UTC day the entry was
written. A later check does not list a core file while its owner
version, template version, owner hash, and template hash still
match that entry, including a second update on the same day. The
date does not expire the entry. A missing record, or a record that
is not this object, acknowledges nothing. When the owner file or
the template changes, the file is listed again.
- *acceptance*: After `wiki update --yes` on a drifted
`fiona-settings.json`, `corefiles.json` names
`fiona-settings.json` with `version` null, both 64-digit hashes,
and the UTC date. A second check-update does not list that file.
Editing the owner file lists it again.
- *acceptance*: `wiki update --dry-run` does not create
`corefiles.json`.
- *7.1.2.15.* `fiona.wiki.update.ignore_files` is a list of glob
patterns in the wiki's `fiona-settings.json`. A path that matches
is not an update, a conflict, an addition, or a merge candidate.
`wiki update` does not copy or replace it. A missing key is an
empty list. A value that is not a list of strings raises
`ValueError` with the text
`fiona.wiki.update.ignore_files must be a list of strings`.
The shipped list is `corefiles.json`. That path is the record in
*7.1.2.14.* and stays out of the candidate set even when the list
omits it. A confirmed update still writes the record.
- *acceptance*: A drifted `utils/scripts/example.py` whose path is
in the list is absent from check-update and unchanged after
`wiki update --yes`.
- *acceptance*: A template `corefiles.json` that differs from the
wiki copy is absent from check-update, and `wiki update --yes`
leaves the wiki copy unchanged when nothing else needs an update.
- *acceptance*: A non-list `ignore_files` makes check-update raise
`ValueError` with that text.
#### 7.1.3. wiki sanity
> Status: Implemented
Implements section 4.7.4. Markdown lint uses the wiki's own
`fiona-settings.json`. Dead links are MDL001 and MDL002 inside
mdlinter, not a separate check. New checks are added by registering
another `(name, runner)` pair in the wiki sanity list.
- *7.1.3.1.* A clean wiki exits 0. Any violation exits 1.
- *acceptance*: One dead link makes `passed` false in `--json`
and produces a non-zero exit.
- *7.1.3.2.* `--json` output contains only JSON.
- *acceptance*: The stdout of a `--json` run parses with
`json.loads` and contains no ANSI sequences.
- *7.1.3.3.* Human output uses the progress display in *7.14.2.*
with label `Running Wiki Sanity`.
- *acceptance*: A piped run prints that label and dots, and no
ANSI escapes.
- *7.1.3.4.* `markdown.pymarkdown.path_rule_exemptions` omits
matching markdownlint hits from wiki sanity only. Each entry has
`path` (glob relative to the wiki root) and `rules` (pymarkdown
rule ids). Other sanity checks still scan those paths.
- *acceptance*: A file under `tasks/` with an MD013 violation and
`{"path": "tasks/**", "rules": ["MD013"]}` in settings reports
zero markdownlint violations; without the entry it fails.
#### 7.1.4. wiki graph
> Status: Partially Implemented
*7.1.4.16.*, *7.1.4.17.*, *7.1.4.18.*, and *7.1.4.19.* are
planned. Every other requirement in this section is built.
The renderer is D3 v7, loaded from `https://d3js.org/d3.v7.min.js`,
with the visualization embedded in one HTML file. vis.js is not
used.
- *7.1.4.1.* The parser extracts links as follows.
- *7.1.4.1.1.* Markdown links `[text](path)`, including `./` and
`../`, resolve against the source file and are stored relative
to the wiki root. A path without an extension is treated as
`.md`.
- *7.1.4.1.2.* Wikilinks `[[file]]` and `[[dir/file]]` are parsed
when wikilinks are enabled.
- *7.1.4.1.3.* External URLs (`http://`, `https://`, `ftp://`, and
any `://`) are recorded. The HTML graph includes each URL as
a node. JSON lists them in metadata only with
`--external-links` or `external_links` in settings.
- *7.1.4.1.4.* A link whose target file is missing is recorded as
broken and does not abort the run.
- *7.1.4.2.* The graph is built as follows.
- *7.1.4.2.1.* A node is a markdown file. An edge is one or more
links from source to target. Weight is the link count. Each
markdown node stores that file's full frontmatter object.
- *7.1.4.2.2.* An orphan has in-degree 0 and out-degree 0.
- *7.1.4.2.3.* Each node stores `unique_points`, `total_points`,
and `link_score`. `unique_points` is how many different files
point at the node. `total_points` is the sum of those weights.
One source with weight w scores `2 * (1 - 0.5^w)`.
Ten links from one file score less than two files that each
link once.
- *7.1.4.3.* `--format json` and `--json` emit this object.
```json
{
"metadata": {
"wiki_root": "<path>",
"file_count": 1,
"link_count": 0,
"generated_at": "<UTC ISO-8601>"
},
"nodes": [
{
"id": "wiki/a.md",
"name": "a.md",
"in_degree": 0,
"out_degree": 0,
"unique_points": 0,
"total_points": 0,
"link_score": 0,
"frontmatter": {"name": "A", "version": "1.0.0"}
}
],
"edges": [
{"source": "wiki/a.md", "target": "wiki/b.md", "weight": 2}
],
"orphans": ["wiki/a.md"]
}
```
- *7.1.4.4.* When external URLs were collected, `metadata` also has
`external_links_count` (total occurrences) and `external_links`
(unique URL strings).
- *7.1.4.5.* HTML behavior.
- *7.1.4.5.1.* One self-contained file. The layout fills the
viewport and follows resize. The page script is
`templates/graph_physics.js`. Generation pastes that file
into the page. The saved HTML does not request it separately.
- *acceptance*: Generated HTML contains `function frameNodes`
and does not contain the paste marker.
- *7.1.4.5.2.* Node radius scales with `link_score` from `min` to
`max` in section 5.3. `min` and `max` are lengths in em, where
16 is 1em, and may be fractions. The drawn radius keeps that
fraction, so the sizes between min and max are not snapped to
a whole unit. Color interpolates low to mid over the
first half of `link_score` divided by the max score, then
mid to high. A score of 0 uses the low color and the min
radius. Two files that each link once outrank one file that
links ten times.
- *7.1.4.5.3.* Edge thickness is `edge_width` from the active
theme, in em, where 16 is 1em. It is 1 when that key is absent or not a
positive number. Changing the theme uses that theme's width.
Color runs from gray (weight 1) through yellow to red
(maximum weight).
- *7.1.4.5.4.* Hover: the node uses the hover color and the capped
larger radius. Connected edges use the hover edge color.
Other visible edges dim. Leaving the node restores its previous
radius and color and restores the dimmed edges. Hover does not
clear a click highlight. While both are active, edges of the
hovered node and edges of the clicked node stay highlighted.
Hover fixes the node in place the same way a click does.
Leaving the node releases it, unless that node is the clicked
one. A card appears beside the node. A line with an arrow
points from the card to the node, the same way an edge points
at its target. The card has two lines. The first line is the
node name. The second line is
`unique points:total points:in:out`. A page name stays as
the filename. An `http` or `https` link drops the scheme.
A path with more than one segment shows `host/.../last`.
When that last segment is longer than
`hover_link_tail_chars`, the card shows `host/...` plus the
last that many characters, with no slash before them. Width
follows the longer line. Height follows the two lines.
The card is a body in the layout. A link keeps it near the
node. It repels other nodes, and they repel it. The card stays
inside the graph. Leaving the node hides that card. It does
not hide a click card.
Hover uses that glow when `show_glow` is true or absent.
`show_glow` false draws no glow on hover, click, or a search
match. The hover card is shown when `show_link_card` is true
or absent. `show_link_card` false draws no hover card. It
does not change the click card.
- *7.1.4.5.5.* Single click toggles a persistent highlight and
shows that file's frontmatter card at the upper left of the
graph. This card is not the hover card. It lists every
frontmatter field and its value, then unique points, total
points, in-degree, and out-degree. A file with no frontmatter
shows an empty field list. Another node replaces the card.
Empty space hides it. Hover does not. `show_link_card` does
not hide it.
The clicked node carries the same glow as a search match
when `show_glow` is true or absent. Clearing the highlight
removes that glow. A node that is still a search match keeps
the glow when glow is on.
The clicked node is fixed where it is. No other node is fixed.
Connected nodes keep moving under the simulation, so their
edges follow. Clearing the highlight releases the node.
Dragging that node leaves it fixed at the drop point.
- *7.1.4.5.6.* Double click opens the markdown file. A double
click on a directory node opens that directory's view
instead (*7.1.4.18.4.*).
- *7.1.4.5.7.* Search highlights matching nodes in light blue at
1.5 times radius (capped at max), shows neighbors in gray, and
hides nodes that are not reachable from a match. A match also
carries a glow when `show_glow` is true or absent. Neighbors
and a cleared search do not. `show_glow` false draws no glow.
- *7.1.4.5.8.* A checkbox toggle labeled "Show Orphan Nodes" shows
or hides orphans. It starts checked. The label does not change.
- *7.1.4.5.9.* Overwrite of an existing output file asks for
confirmation on a TTY.
- *7.1.4.5.9.1.* `NOT_IMPLEMENTED` `-f/--force` skips that
prompt. Without a TTY and without `--force`, the file is still
overwritten and one `WARNING` record names it. The housekeeping
graph step (*7.2.6.*) passes `--force`, because
`wiki_graph.html` is the one file it owns (*7.2.7.*), and logs
that overwrite at `INFO`.
- *acceptance*: Housekeeping on a wiki that already has
`wiki_graph.html`, at `LOG_LEVEL=WARNING`, writes no
`Overwriting` line on stderr.
- *7.1.4.5.10.* Files above `max_files` are skipped after a warning.
The first `max_files` files are drawn.
- *7.1.4.5.11.* The run completes in under 10 seconds for 1000 files
on a current developer machine.
- *7.1.4.5.12.* A checkbox toggle labeled "Show External Links"
shows or hides external-link nodes and their edges. It starts
checked when `show_external_links` in section 5.1 is true, and
unchecked when that key is absent or false. The label does not
change. When it starts checked, those nodes and edges are drawn
on the opening fade. A click is not required. An external node
is the text cloud character. A page node stays a circle.
- *7.1.4.5.13.* A checkbox toggle labeled "Show Subtitles" shows or
hides labels under nodes. It starts checked when
`subtitle.enabled` in section 5.1 is true or absent, and
unchecked when that key is false. The label does not change.
A node shows its label when its drawn radius is at least
`subtitle.min_radius` from the active theme times that theme's
`node_radius.max`. The fraction is 0.5 when absent, and a
value outside 0 to 1 is clamped. Changing the theme uses that
theme's fraction and max. The text is the frontmatter `name`
when that field is a non-empty string, otherwise the filename.
The label sits under the shape. The gap is `subtitle.padding`
em from the active theme, where 16 is 1em, 12 when absent. A
smaller number sits closer. The font size is `subtitle.size`
on that same scale, 11 when absent or below 1. `subtitle.on_top`
true draws every label after the nodes, so another node does
not cover it. Absent or false keeps the label inside its node.
`subtitle.outline` true strokes the letters with
`subtitle.outline_width` on that same scale, 3
when absent, in `subtitle.outline_color`. Absent color uses
`graph_background`. Absent or false draws no stroke. Changing
the theme applies that theme's subtitle settings.
- *7.1.4.5.14.* A checkbox toggle labeled "Show Legend" shows or
hides a card at the upper right of the graph. It starts
checked when `show_legend` in section 5.1 is true or absent,
and unchecked when that key is false. The label does not
change. The card has three groups. Shape shows a page and an
external link in the text color. Nodes are growing circles:
few, some, and many files pointing here. Edges are lines:
linked once, a few times, and many times. Node circles use
the theme node colors. Edge lines use the theme edge colors.
- *7.1.4.5.15.* A select labeled "Theme" lists every theme from
section 5.3 and starts on `theme` from section 5.1.
That name is `night-sky` when the key is absent. Choosing a
theme updates the page background, font, buttons, cards, node
colors, node sizes, edge colors, highlight colors, the
clicked-node color, and node shapes. Choosing a theme also
runs the See all fit, so the shown nodes are centered again.
The grayscale dark-mode switch runs that fit too. The theme
applied at load does not. A missing theme key uses
`colorful`.
`grayscale` uses its `dark` object when the page is in dark mode.
- *7.1.4.5.16.* A checkbox toggle labeled "Show Arrows" draws or
hides an arrowhead on each visible edge. It starts checked
when `show_arrows` in section 5.1 is true or absent, and
unchecked when that key is false. The label does not change.
The arrowhead meets the target shape. The source file points
at the target file. The arrow uses that edge's color. A
hidden edge has no arrowhead.
- *7.1.4.5.17.* `simplify_external_links` sets an external node's
subtitle to its host. A leading `www.` is dropped. The node
stays one per URL. `consolidate_external_links` merges URLs
with the same host into one node whose id is `host/*` when
`consolidate_external_links_depth` is absent. Depth `0`
leaves each URL alone, even when the bool is true. Depth
`1` is `host/*`. Depth `2` is `host/a/*`. Depth `3` is
`host/a/b/*`. A leading `www.` is dropped from the host.
A path part longer than
`consolidate_external_links_max_char` characters is `...`.
The limit is 8 when absent. A part of that length stays.
Size and color follow `link_score` in section 7.1.4.2.3.
When depth is above 0, simplify is on even if its key is
false. The bool defaults to false. A double click opens
`https://host/` plus the real path parts before the first
`...`, with a trailing slash.
- *acceptance*: `https://www.domain.com/abc` and
`https://domain.com/acd` from two files become one node
`domain.com/*` with subtitle `domain.com` and `link_score`
2 when consolidate is true and depth is absent. With only
simplify, both URLs remain and each subtitle is
`domain.com`. Depth `2` merges
`https://domain.com/abc/one` and
`https://domain.com/abc/two` into `domain.com/abc/*` and
leaves `https://domain.com/zzz/one` as `domain.com/zzz/*`.
Depth `3` names `https://domain.com/a/b/c` as
`domain.com/a/b/*` and opens `https://domain.com/a/b/`.
`https://domain.com/abcdefghij/b/c` at depth `3` is
`domain.com/.../b/*`.
- *7.1.4.5.18.* A refresh puts every toggle, the theme select, and
the search box back to the values in section 5.1. The browser's
saved form state does not keep the previous choice.
- *acceptance*: The page sets each control again on `pageshow`.
- *7.1.4.5.19.* A checkbox toggle labeled "Show Sources" shows or
hides nodes built from the frontmatter `source` key, and the
edges from the page to those nodes. It starts checked when
`show_source` in section 5.1 is true or absent, and unchecked
when that key is false. The label does not change. A string
is one node. A list of strings is one node per entry. A blank
string is skipped. Any other value is skipped. The node is a
cloud, the same shape as an external link. Its id is that
string. Two pages with the same string share one node. A
double click opens the string when it contains `://`.
- *acceptance*: `source: https://example.com/rules.md` adds
one cloud node and one edge from that page. A list of two
URLs adds two nodes. `source: 12` adds none.
- *7.1.4.6.* Layout constants live at the top of the HTML template so
they can be tuned without hunting through the script.
| Constant | Value | Role |
| --- | --- | --- |
| `LINK_STRENGTH` | 0.4 | Spring along an edge; neighbors follow a dragged node |
| `BASE_LINK_DISTANCE` | 80 | Rest length of an edge |
| `LINK_DISTANCE_PER_DEGREE` | 0 | Distance does not grow with degree |
| `CHARGE_STRENGTH` | -140 | Repulsion that spreads the cloud |
| `CHARGE_DISTANCE_MIN` | 24 | Near distance for repulsion |
| `CHARGE_DISTANCE_MAX` | 500 | Far cutoff for repulsion |
| `NODE_ATTRACTION_STRENGTH` | 0 | Extra attraction is off |
| `CENTER_STRENGTH` | 1 | Each tick places the centroid on the viewport center |
| `GRAVITY_STRENGTH` | 0.05 | Per-node pull toward the viewport center |
| `COLLIDE_STRENGTH` | 1 | Overlap prevention |
| `BASE_COLLIDE_RADIUS` | 6 | Gap around the hitbox |
| `HITBOX_SCALE` | 1.25 | Collision circle versus the drawn radius |
| `EDGE_PADDING_MULTIPLIER` | 0 | Degree does not add collide radius |
| `CANVAS_MARGIN` | 56 | Border band that pushes a node inward |
| `CANVAS_MARGIN_STRENGTH` | 0.35 | Strength of that inward push |
| `ALPHA_DECAY` | 0.018 | How fast the simulation cools |
| `ALPHA_MIN` | 0.001 | Floor before the layout rests |
| `VELOCITY_DECAY` | 0.22 | Low damping so motion glides |
| `FLOAT_AMPLITUDE` | 2 | Idle drift, in em where 16 is 1em |
| `FLOAT_SPEED` | 0.0007 | How fast that drift cycles |
- *7.1.4.7.* Nodes start in a small disk around the viewport center
and expand under repulsion. They fade in by a breadth-first walk
from the highest in-degree node, highest connectivity first.
`d3.forceCenter` keeps the centroid on the viewport center.
`d3.forceX` and `d3.forceY` add center gravity so outliers drift
back. Both targets follow resize. The tick does not write a
canvas-edge clamp back into node coordinates. A margin force adds
inward velocity only while a node is inside `CANVAS_MARGIN` of the
border, and it skips a node that is pinned. The hitbox is
`HITBOX_SCALE` times the drawn radius, plus a fixed gap, so a
large node keeps clear of its neighbors. Each tick then pushes
any covering pair apart until the drawn circles, plus that gap,
no longer cover. An edge's rest length
is at least the two hitboxes added together. Degree does not
add padding, so a
hub is not shoved into the border. An unpinned node then drifts
at most `FLOAT_AMPLITUDE` on that em scale, and its edges
follow, so it looks like it is floating. A pinned or dragged node
does not drift. The view eases toward every shown node
over about 4200 ms, with 4em of padding. That is the same
fit as the button labeled "See all". The first node uses
`first_fit_ms` instead, so that opening zoom is slower.
It runs when the first node fades in, and again when the
last node fades in. A fit that is already moving keeps its
speed and aims at the new bounds. A second See all does not
stop the zoom.
It also runs when the theme changes.
The scale can go below 0.15 when that is what fits the
cloud, and it stays at most 6.
Hidden external links and hidden orphans are left out. A
later wheel zoom,
background drag, or node drag keeps the view the user set.
Wheel zoom and background drag pan the view. Dragging a node
reheats the simulation; releasing it lets the springs and
gravity settle.
- *7.1.4.8.* The output path must be writable. A failure is an error
message and a non-zero exit.
- *7.1.4.9.* A wiki with one relative link and one wikilink produces
two edges, and a file with neither is listed in `orphans`.
- *acceptance*: The JSON `edges` weights match the link counts.
- *7.1.4.10.* `--no-wikilinks` drops wikilink edges and keeps markdown
links.
- *acceptance*: The wikilink target is absent from `edges`.
- *7.1.4.11.* Opening the HTML in a current browser shows the graph,
hover, click highlight, search, and the orphan toggle.
- *acceptance*: A node hover changes its color to the hover color
and restores the previous radius on exit.
- *7.1.4.12.* JSON and HTML each store `frontmatter` on every markdown
node: every YAML field from that file. Missing or invalid YAML is
`{}` and does not abort the run.
- *acceptance*: A file with `name`, `version`, and `updated` has
those three keys on its node. A file with no frontmatter has
`{}`. The click card lists every frontmatter field. The
hover card has two lines: the node name, then unique
points, total points, in-degree, and out-degree.
- *7.1.4.13.* Repeated links from one file add less to size and
color than links from different files.
- *acceptance*: One source with weight 10 has `unique_points` 1,
`total_points` 10, and `link_score` below 2. Two sources with
weight 1 have `link_score` 2. HTML radius and color divide
`link_score` by `maxLinkScore`. The hover card's second line
includes the unique-point count and the total-point count.
- *7.1.4.14.* The Show Arrows toggle follows `show_arrows`.
- *acceptance*: The checkbox is checked when the key is absent
or true, and unchecked when it is false. The HTML defines
marker `edge-arrow` and sets `marker-end` from that toggle.
- *7.1.4.15.* Graph settings start from the packaged
`fiona-settings.json` graph section. A file found by walking up
from the scanned directory replaces the keys it sets. A missing
file still uses the packaged section.
- *acceptance*: A scan whose directory and parents have no
`fiona-settings.json` still applies `show_legend`,
`show_arrows`, and `simplify_external_links` from the
packaged file.
- *7.1.4.16.* `NOT_IMPLEMENTED` The HTML graph draws a node when
enough files point at it, or when a parent of that node is
clicked. The setting is `node_importance_min_point` in
section 5.1.
- *7.1.4.16.1.* `NOT_IMPLEMENTED` The count is `unique_points`
from *7.1.4.2.3.*: how many different files point at the
node. A node is below the cutoff when `unique_points` is
less than the setting. A node at the cutoff is drawn.
- *acceptance*: The setting is 5. A node with
`unique_points` 4 is absent from the opening draw. A
node with `unique_points` 5 is drawn.
- *7.1.4.16.2.* `NOT_IMPLEMENTED` `0` disables the filter. A
missing key, or a value that is not an integer of 0 or
more, is the same as `0`. Every node is drawn, still
subject to the Show Orphans, Show External Links, and
Show Sources toggles.
- *acceptance*: With the key absent or set to `0`, a node
with `unique_points` 0 is drawn when those toggles
allow it.
- *7.1.4.16.3.* `NOT_IMPLEMENTED` A parent of a node is any
node with an edge whose target is that node. While a
parent is the clicked node, each direct target is drawn
even when it is below the cutoff. The clicked node stays
drawn for as long as it stays clicked, including when it
is below the cutoff. Clearing the click hides a
below-cutoff node again. Clicking a node draws its
direct targets only, not the targets of those targets.
- *acceptance*: The setting is 5. Page A has
`unique_points` 5 and links to B. B has
`unique_points` 1 and links to C. C has
`unique_points` 1. The opening draw shows A and hides
B and C. Clicking A draws B and leaves C hidden.
Clicking B keeps B drawn and draws C. Clicking empty
space hides B and C. A node with no incoming edge
stays hidden, because it has no parent to click.
- *7.1.4.16.4.* `NOT_IMPLEMENTED` An edge is drawn only when
both of its nodes are drawn. See all, the opening fit,
and search use that same drawn set. A hidden node is
left out of the fit, as *7.1.4.7.* does for a hidden
orphan, and it is not a search match.
- *acceptance*: An edge into a below-cutoff node is
absent until a click draws that node. Searching for
its name does not draw it.
- *7.1.4.16.5.* `NOT_IMPLEMENTED` Show Orphans, Show External
Links, and Show Sources still apply. A node hidden by
one of those toggles stays hidden when it meets the
cutoff or when its parent is clicked.
- *acceptance*: With Show External Links unchecked,
clicking a page does not draw the external node it
links to.
- *7.1.4.16.6.* `NOT_IMPLEMENTED` `--format json` and `--json`
still list every node and edge. The filter changes the
HTML draw only.
- *acceptance*: JSON includes a node whose
`unique_points` is below the setting.
- *7.1.4.20.* `fiona.wiki.graph.ignore_files` is a list of glob
patterns in the wiki's `fiona-settings.json`. A markdown file
whose path relative to the scan root matches is not a graph
node. HTML and JSON both omit it. An edge that would start or
end at that file is omitted. The file does not count toward
`max_files` and is not an orphan. A missing key is an empty
list. A value that is not a list of strings raises `ValueError`
with the text `fiona.wiki.graph.ignore_files must be a list of strings`.
A pattern that starts with `*/` also matches that same path at
the scan root. The shipped list is `*/table_of_contents.md`.
A table of contents links to every page, so leaving it in the
graph hides the real links.
- *acceptance*: With that pattern, `table_of_contents.md` at the
scan root and `docs/table_of_contents.md` are absent from JSON
`nodes`, `edges`, and `orphans`. `a.md` and `b.md` stay.
An edge from the table of contents to `a.md` is absent. An
edge from `a.md` to the table of contents is absent.
- *acceptance*: `max_files` is 1. The directory holds
`table_of_contents.md` and `kept.md`. JSON `file_count` is 1
and the only node is `kept.md`.
- *acceptance*: A non-list `ignore_files` makes `wiki graph`
raise `ValueError` with that text.
- *7.1.4.17.* `NOT_IMPLEMENTED` A markdown file that `.gitignore`
matches is not a graph node. HTML and JSON both omit it. An
edge that would start or end at that file is omitted. The file
does not count toward `max_files`. Housekeeping's orphan list
(*7.2.6.*) uses this graph, so an omitted file is not an
orphan. There is no setting that turns this off.
- *acceptance*: The scanned `.gitignore` lists `secret.md`.
`secret.md` is absent from JSON `nodes`, `edges`, and
`orphans`. A page that links to it has no edge to it.
- *acceptance*: `max_files` is 1. The directory holds `kept.md`
and a gitignored `secret.md`. JSON `file_count` is 1 and the
only node is `kept.md`.
- *7.1.4.17.1.* `NOT_IMPLEMENTED` Patterns are the non-comment,
non-blank lines of the `.gitignore` in the scanned directory,
and of each `.gitignore` in a subdirectory. A subdirectory
file's patterns match only paths under that subdirectory.
Matching is *7.1.2.10.*: a pattern with no slash matches any
path segment, and a pattern that ends with `/` matches that
directory and everything under it.
- *acceptance*: The scanned `.gitignore` lists `drafts/`.
`drafts/note.md` is absent. `wiki/note.md` is present.
- *acceptance*: A line `# secret.md` does not omit
`secret.md`.
- *acceptance*: `notes/.gitignore` lists `scratch.md`.
`notes/scratch.md` is absent. `scratch.md` at the scan
root stays.
- *7.1.4.17.2.* `NOT_IMPLEMENTED` A path under `.git/` is omitted
even when no pattern names it.
- *acceptance*: With no `.gitignore`, a markdown file under
the scan root stays, and a markdown file under `.git/` is
absent.
- *7.1.4.18.* `NOT_IMPLEMENTED` The HTML graph draws a crowded
directory as one directory node. That keeps a large wiki
readable and cuts the number of shapes the page simulates. The
settings are `directory_nodes.min_files` and
`directory_nodes.outside_depth` in section 5.1.
- *7.1.4.18.1.* `NOT_IMPLEMENTED` A directory collapses when it
holds at least `min_files` markdown pages directly. Pages in
its subdirectories do not count toward that number. The scan
root never collapses. `0`, a missing key, or a value that is
not an integer of 0 or more turns collapsing off, and the
graph is drawn as it is today.
- *acceptance*: `min_files` is 3. `wiki/concepts/` holds
`a.md`, `b.md`, and `c.md`. The opening draw has one node
`wiki/concepts/` and no node for those three pages.
- *acceptance*: `min_files` is 3. `wiki/notes/` holds two
pages directly and `wiki/notes/deep/` holds three. The
opening draw shows both pages of `wiki/notes/` and one node
`wiki/notes/deep/`.
- *acceptance*: With the key absent or `0`, no directory node
is drawn.
- *7.1.4.18.2.* `NOT_IMPLEMENTED` A directory node stands for
every page under that directory, including its
subdirectories. Its id is the directory path relative to the
scan root with a trailing `/`. Its name and subtitle are the
last path segment with a trailing `/`. A collapsed directory
inside another collapsed directory is drawn only in its
parent's view (*7.1.4.18.4.*).
- *acceptance*: The node for `wiki/concepts/` has id
`wiki/concepts/` and subtitle `concepts/`.
- *7.1.4.18.3.* `NOT_IMPLEMENTED` A directory node acts like a
page node. The edges of its pages are drawn on it.
- *7.1.4.18.3.1.* `NOT_IMPLEMENTED` An edge between a page
inside and a node outside is redrawn between the directory
node and that outside node. Edges that land on the same pair
merge into one edge whose weight is the sum. An edge with
both ends inside the directory is not drawn in the view that
shows the directory node.
- *acceptance*: `wiki/index.md` links once to `a.md` and
twice to `b.md`, both inside collapsed `wiki/concepts/`.
One edge runs from `wiki/index.md` to `wiki/concepts/`
with weight 3. A link from `a.md` to `b.md` draws no edge
in that view.
- *7.1.4.18.3.2.* `NOT_IMPLEMENTED` `unique_points`,
`total_points`, `link_score`, in-degree, and out-degree are
computed from those merged edges with the rule in
*7.1.4.2.3.*. Size and color follow `link_score` like any
node.
- *acceptance*: In the example above, the directory node
has `unique_points` 1, `total_points` 3, and in-degree 1.
- *7.1.4.18.3.3.* `NOT_IMPLEMENTED` Hover, click highlight,
drag, glow, the hover card, and the toggles behave as for a
page. The click card shows `path`, `pages` (every page under
the directory), then unique points, total points, in-degree,
and out-degree. It has no frontmatter fields.
- *7.1.4.18.3.4.* `NOT_IMPLEMENTED` Search matches a directory
node when its name matches, or when the name of a page
under it matches. External links and `source` nodes of its
pages hang from the directory node.
- *acceptance*: Searching `a.md` while `wiki/concepts/` is
collapsed highlights the `wiki/concepts/` node.
- *7.1.4.18.3.5.* `NOT_IMPLEMENTED` The shape is
`shapes.directory` from the active theme (section 5.3). The
Legend's shape group adds a directory entry while a
directory node is drawn.
- *acceptance*: A theme that sets `shapes.directory` to
`star` draws the directory node as a star. A theme that
omits the key uses `folder` from `colorful`.
- *7.1.4.18.4.* `NOT_IMPLEMENTED` Double click on a directory
node opens that directory's view. The other nodes leave the
page. The view draws the pages that sit directly in the
directory, one directory node for each collapsed
subdirectory, the pages of each subdirectory that does not
collapse, and the edges between them. A collapsed
subdirectory follows *7.1.4.18.1.* inside the view. The view
runs the See all fit from *7.1.4.7.*.
- *acceptance*: Double click on `wiki/concepts/` draws
`a.md`, `b.md`, and `c.md`, the edge from `a.md` to
`b.md`, and no node that sits outside `wiki/concepts/`
except the ones in *7.1.4.18.5.*.
- *7.1.4.18.5.* `NOT_IMPLEMENTED` A node outside the directory
is drawn in its view when a page inside points at it, within
`outside_depth` hops along the direction of the edges. Depth
`1` draws the direct targets. Depth `2` also draws their
targets. Depth `0` draws no outside node. An outside node is
drawn the way the opening draw shows it: as its page, or as
the collapsed directory that holds it. Outside nodes and
their edges are drawn at half opacity. A node that only
points into the directory is not drawn.
- *acceptance*: `outside_depth` is 1. `a.md` links to
`wiki/index.md`, and `wiki/index.md` links to
`wiki/log.md`. The `wiki/concepts/` view draws
`wiki/index.md` at half opacity and does not draw
`wiki/log.md`. With depth 2 it draws both.
- *acceptance*: `outside_depth` is 1 and `a.md` links to
`wiki/other/x.md`, inside collapsed `wiki/other/`. The view
draws the node `wiki/other/` and no node `wiki/other/x.md`.
- *7.1.4.18.6.* `NOT_IMPLEMENTED` A bar at the upper center of
the graph names the open view as a path, for example
`All / wiki / concepts`. Each segment opens that view. A
button labeled "Back" opens the parent view and is disabled
on the opening view. The open view is kept in the page hash
as `#dir=<id>`, so browser back and forward move between
views and a reload reopens the same view. A hash that names
no collapsed directory opens the opening view.
- *acceptance*: From `wiki/concepts/`, Back draws the
opening view again with the `wiki/concepts/` node.
- *7.1.4.18.7.* `NOT_IMPLEMENTED` Show Orphans, Show External
Links, Show Sources, search, See all, and
`node_importance_min_point` (*7.1.4.16.*) apply to the nodes
of the open view only. A double click on an outside page in
a view opens that file. A double click on an outside
directory node opens that directory's view.
- *7.1.4.18.8.* `NOT_IMPLEMENTED` `--format json` and `--json`
list every page and edge, as today. Collapsing changes the
HTML draw only.
- *acceptance*: JSON with `min_files` 3 still lists `a.md`,
`b.md`, and `c.md` as nodes.
- *7.1.4.19.* `NOT_IMPLEMENTED` `wiki graph --serve` is a light
live view. It builds the graph, serves it over local HTTP, and
rebuilds when a scanned file changes. The open page then shows
the change without a manual refresh. It uses the standard
library only and adds no dependency.
- *7.1.4.19.1.* `NOT_IMPLEMENTED` The server binds `--host`
(default `serve.host`, `127.0.0.1`) and `--port` (default
`serve.port`, 8765). Port `0` picks a free port. On start it
prints
`Serving wiki graph at http://<host>:<port>/ (Ctrl+C to stop)`
on stdout and opens that URL in the default browser, unless
`--no-open` is set. Ctrl+C stops it, prints `Stopped`, and
exits 0. A host other than `127.0.0.1`, `localhost`, or `::1`
prints a warning on stderr that the wiki is readable from the
network.
- *acceptance*: `--serve --port 0 --no-open` prints the URL
with a port above 0, and opens no browser.
- *acceptance*: A port already in use prints an error on
stderr and exits 1.
- *7.1.4.19.2.* `NOT_IMPLEMENTED` `--serve` with `--format json`
or `--json` is a usage error with exit 2. `--host`, `--port`,
or `--no-open` without `--serve` is the same usage error.
`--serve` writes no file unless `-o` is given. Then the server
writes that file after every build.
- *7.1.4.19.3.* `NOT_IMPLEMENTED` The server answers `GET` only.
Another method is 405.
- *7.1.4.19.3.1.* `NOT_IMPLEMENTED` `/` is the current graph
HTML: the same page as *7.1.4.5.*, plus the live script in
*7.1.4.19.5.*.
- *7.1.4.19.3.2.* `NOT_IMPLEMENTED` `/graph.json` is the
current JSON from *7.1.4.3.* with `metadata.build` added.
`build` starts at 1 and adds 1 on every rebuild.
- *7.1.4.19.3.3.* `NOT_IMPLEMENTED` `/events` is a
Server-Sent Events stream. After each rebuild it sends
`event: graph` with `data: {"build": <n>}`. A comment line
goes out every 15 seconds so the connection stays open.
- *7.1.4.19.3.4.* `NOT_IMPLEMENTED` `/files/<path>` returns a
markdown file under the scan root as `text/plain;
charset=utf-8`. A path that leaves the scan root, a path
that is not a scanned file, or a missing file is 404. In a
served page, a double click on a page opens this URL in
place of `file://`.
- *acceptance*: `/files/../secret.md` and
`/files/%2e%2e/secret.md` are 404.
- *7.1.4.19.3.5.* `NOT_IMPLEMENTED` Any other path is 404.
- *7.1.4.19.4.* `NOT_IMPLEMENTED` The watcher polls every
`serve.poll_seconds` seconds. It compares the set of matched
markdown files and each file's modification time and size.
The match is the include and exclude patterns and the
`.gitignore` rule in *7.1.4.17.*. The nearest
`fiona-settings.json` and the `constellation/` files from
section 5.3 are watched too. A change waits `serve.debounce_ms`
with no newer change, then one rebuild runs. Several saves
in that window cause one rebuild.
- *acceptance*: Adding `new.md` that links to `a.md` makes
the next `/graph.json` list node `new.md` and an edge to
`a.md`, and `/events` sends one `graph` event.
- *acceptance*: Deleting a page removes its node and its
edges from the next `/graph.json`.
- *acceptance*: Saving three files within the debounce window
raises `build` by 1.
- *7.1.4.19.5.* `NOT_IMPLEMENTED` The served page listens on
`/events`. On a `graph` event whose `build` is newer than its
own, it saves its state to `sessionStorage`, then reloads.
The state is the zoom transform, each drawn node's position
by id, the clicked node, the open directory view, every
toggle, the theme, and the search text. After the reload the
page restores that state once, places known nodes where they
were, and fades in only the new nodes. The restore replaces
the reset in *7.1.4.5.18.* for that one load. A manual
refresh still resets the controls.
- *acceptance*: After a rebuild adds `new.md`, an existing
node keeps its position within 1em, the theme and toggles
are unchanged, and `new.md` fades in.
- *7.1.4.19.6.* `NOT_IMPLEMENTED` A rebuild that fails keeps the
last good graph. The error prints on stderr and the server
keeps running. No `graph` event is sent.
- *7.1.4.19.7.* `NOT_IMPLEMENTED` A file written without
`--serve` contains no live script and opens no connection.
- *acceptance*: Plain `wiki graph` output does not contain
`EventSource`.
- *7.1.4.19.8.* `NOT_IMPLEMENTED` The server makes no outbound
call. It reads only files under the scan root and the
settings and theme files of section 5.
#### 7.1.5. wiki ingest
> Status: Partially Implemented
Implements section 4.7.6. An ingest has two phases. The tool runs
the first: it turns one source into a file on disk, archives a copy
under `raw/`, converts that copy to markdown, and writes a wiki page
with `status: raw`. The agent runs the second: it refines the page
and sets `status: refined`. The Fiona skill teaches the second
phase and when to convert a source by hand. The pattern both phases
follow is section 3.1 of
[llm-wiki-mod](./src/fiona/data/llm_wiki/utils/skills/fiona/llm-wiki-mod.md).
The tool runs five stages in order. Each stage has one base class
and one job. Class names are in Appendix D.
| Stage | Input | Output |
| --- | --- | --- |
| Fetch | the `<source>` argument | a local file, its origin, and its media type |
| Archive | that local file | a file under `raw/` |
| Convert | the archived file | markdown body, title, metadata, conversion state, warnings |
| Write | the conversion | one page under `wiki/` with `status: raw` |
| Record | the run | one line appended to `wiki/log.md` |
Shipped converters, in registration order:
| Converter | Media types | Extensions | Raw category | Library |
| --- | --- | --- | --- | --- |
| `markdown` | `text/markdown` | `.md`, `.markdown` | `articles` | none |
| `text` | `text/plain` | `.txt`, `.text`, `.log` | `articles` | none |
| `html` | `text/html`, `application/xhtml+xml` | `.html`, `.htm`, `.xhtml` | `articles` | `beautifulsoup4`, `markdownify` |
| `pdf` | `application/pdf` | `.pdf` | `papers` | `pypdf` |
A source that no converter claims is archived under `raw/other/`.
Pure-Python libraries are preferred so the install stays light.
Candidates for later formats are in Appendix C.
- *7.1.5.1.* The pipeline is built from parts.
- *7.1.5.1.1.* The pipeline receives its fetchers, converter
registry, archiver, page writer, and log writer as constructor
arguments. It holds no format-specific code.
- *7.1.5.1.2.* A converter declares its name, media types,
extensions, and raw category as class attributes, and implements
one `convert` method. Adding a format is one new converter class,
one registration, and its tests.
- *acceptance*: A test registers a converter for
`application/x-test` and ingests a `.xtest` file into a page.
No pipeline, archiver, or writer module changes.
- *7.1.5.1.3.* A fetcher declares whether it accepts an argument and
returns the fetched source. The URL fetcher takes its opener as
an argument.
- *acceptance*: Tests run the URL path with a fake opener and
make no network call.
- *7.1.5.2.* Fetch.
- *7.1.5.2.1.* An argument that starts with `http://` or `https://`
is a URL. Any other `<scheme>://` prefix exits 1 with
`Error: Unsupported URL scheme: <scheme>`. Anything else is a
local path. A missing path and a directory each exit 1 with the
message from section 4.7.6.
- *acceptance*: `ftp://host/a.pdf` exits 1 and writes nothing.
- *7.1.5.2.2.* A URL is downloaded with `urllib.request` into
`ingest/<slug>.<ext>`. Redirects are followed. Proxies come from
the environment. The request sends `user_agent` and waits up to
`timeout_seconds`. No credentials are sent.
- *7.1.5.2.3.* A non-2xx response or a network error exits 1 with
`Error: Download failed: <url>: <reason>`. A response that needs
a login is *7.1.5.2.8.* instead. The reason names the
HTTP status when there is one. A body larger than
`max_download_bytes` stops the download and exits 1. In each
case the partial file is deleted and `ingest/` is unchanged.
- *acceptance*: A fake opener that answers 404 exits 1, the
message contains `404`, and `ingest/` has the same files as
before.
- *7.1.5.2.4.* The media type is the response `Content-Type`
without parameters. A `charset` parameter is kept as a decoding
hint. With no usable header, the extension decides, using the
converter table and then `mimetypes`. With no known extension,
the first 1024 bytes decide: `%PDF-` is `application/pdf`,
`<!doctype html` or `<html` in any case is `text/html`, valid
UTF-8 with no NUL byte is `text/plain`. Anything else is
`application/octet-stream`.
- *acceptance*: A local file named `notes` whose bytes start with
`%PDF-` is converted by `pdf`.
- *7.1.5.2.5.* A downloaded file takes the first extension of the
converter that claims its media type, then the URL path suffix,
then `.bin`.
- *acceptance*: `https://host/report` served as
`application/pdf` is saved as `ingest/report.pdf`.
- *7.1.5.2.6.* The slug of a URL is built from its last three
non-empty path segments, each without its extension, with a
segment named `index` dropped. An empty path uses the host. The
slug of a file is its stem. The text is lowercased, each run of
characters outside `[a-z0-9]` becomes one `-`, leading and
trailing `-` are trimmed, and the result is cut to 80
characters. An empty result is `source`.
- *acceptance*:
`https://host/job/performance_TL153/2108/console` has the slug
`performance-tl153-2108-console`.
- *7.1.5.2.7.* `NOT_IMPLEMENTED` A local path is resolved against
the current directory. `./`, `.\`, and either separator are
accepted on Windows. A missing path inside the wiki's `ingest/`
is often a file an earlier run already moved (*7.1.5.3.6.*).
When a file with the same name exists under `raw/`, the error
adds one space and `(moved to <raw path> by an earlier ingest)`.
Several matches use the first in sorted path order.
- *acceptance*: Ingesting `./ingest/a.md` and then
`.\ingest\a.md` exits 0, then exits 1 with
`Error: Source not found: .\ingest\a.md (moved to
raw/articles/a.md by an earlier ingest)` on one line.
- *7.1.5.2.8.* `NOT_IMPLEMENTED` A URL that needs a login is an
error, not a source. The tool cannot log in, so it stops and
leaves the next step to the agent (*7.1.5.14.*). A login page
must never reach `raw/` as if it were the source.
- *7.1.5.2.8.1.* The tool sends no credentials, cookies, or
tokens, and reads none from the environment or from a file
(*7.1.5.2.2.*). It never asks for a password.
- *7.1.5.2.8.2.* Three checks run in this order. The first hit
decides the reason:
| Check | Hit when | Reason text |
| --- | --- | --- |
| Status | The response is HTTP 401 or 403 | `<status> <phrase>`, e.g. `401 Unauthorized` |
| Redirect | The final URL differs from the requested URL and its host and path match one of `login_url_patterns` | `redirected to <final url>` |
| Login form | The media type is `text/html` and the body has an `<input>` whose `type` is `password`, in any case or quoting | `login page at <final url>` |
- *7.1.5.2.8.3.* `login_url_patterns` is a list of regular
expressions, searched case-insensitively in
`<host><path>` of the final URL. The query string is not
searched. The default list is in section 5.1. The redirect
check is skipped when there was no redirect, so a user can
still ingest a page whose own path holds `login`.
- *7.1.5.2.8.4.* On a hit, stderr has the `Error:
Authentication required` line and the `Next:` line from
section 4.7.6, and the command exits 4. The body is read
within `max_download_bytes` and then dropped. Nothing is
written under `ingest/`, `raw/`, or `wiki/`, and `wiki/log.md`
is unchanged. The URL in both lines is the one the user
passed, with userinfo removed.
- *7.1.5.2.8.5.* `--dry-run` runs the same checks and exits 4
the same way. `--json` prints nothing on stdout
(*7.1.5.13.*).
- *7.1.5.2.8.6.* An HTML page that passes the three checks but
has no text still converts as `partial` with the warning in
*7.1.5.4.5.5.*. Login detection does not replace that warning.
- *acceptance*: A fake opener that answers 401 exits 4. Stderr
has `Authentication required`, `401`, and the `Next:` line.
`ingest/`, `raw/`, `wiki/`, and `wiki/log.md` have the same
hashes as before.
- *acceptance*: A fake opener that redirects
`https://host/job/x/console` to
`https://host/login?from=%2Fjob%2Fx%2Fconsole` and answers 200
exits 4 with the reason `redirected to https://host/login?...`.
- *acceptance*: A fake opener that answers 200 with an HTML form
holding `<INPUT TYPE='password'>` exits 4 with the reason
`login page at <url>`.
- *acceptance*: A 200 HTML page at `https://host/docs/login`,
with no redirect and no password input, is ingested with
exit 0.
- *acceptance*: A 404 still exits 1 with `Download failed`.
- *7.1.5.2.9.* `NOT_IMPLEMENTED` `--origin URL` records where a
local file came from, so a source saved by hand after
*7.1.5.2.8.* keeps its real provenance.
- *7.1.5.2.9.1.* `--origin` is valid only when `<source>` is a
local path. With a URL source it exits 1 with
`Error: --origin needs a local file: <source>`. A value that
is not `http://` or `https://` exits 1 with
`Error: --origin must be an http or https URL: <value>`. Both
checks run before fetch.
- *7.1.5.2.9.2.* The file is fetched, archived, and moved as any
local file. The page frontmatter then has `ingest.origin` set
to the URL without userinfo, `origin_kind: url`, and
`final_url: null`. The `html` converter uses that URL to make
relative links absolute (*7.1.5.4.5.3.*). The log bullet names
that URL (*7.1.5.6.*). JSON `origin` and `origin_kind` match
the frontmatter.
- *7.1.5.2.9.3.* The slug and default raw name still come from
the file stem (*7.1.5.2.6.*), so the name the agent chose in
`ingest/` is kept.
- *acceptance*: `wiki ingest ingest/console.html --origin
https://host/job/x/console` exits 0. The page has
`origin: https://host/job/x/console`, `origin_kind: url`,
and a relative link `/a` in the file becomes
`https://host/a`. No network call is made.
- *acceptance*: `wiki ingest https://host/a --origin
https://host/b` exits 1 and makes no network call.
- *7.1.5.3.* Archive.
- *7.1.5.3.1.* The default raw path is
`raw/<category>/<slug>.<ext>`. The category is the raw category
of the converter that claims the media type, unless
`raw_categories` maps that converter name to another directory.
With no converter it is `other`.
- *acceptance*: A PDF is archived at `raw/papers/<slug>.pdf`.
- *7.1.5.3.2.* `-r/--raw PATH` is relative to `raw/`. A leading
`raw/` is accepted. A value that ends in an extension names the
file. Any other value is a directory, and the file inside it is
`<slug>.<ext>`. Missing directories are created. After resolving
`..` and symlinks the path must be inside `raw/`, else exit 1
with nothing written.
- *acceptance*: `-r jenkins/tl153` archives an HTML source at
`raw/jenkins/tl153/<slug>.html`. `-r ../wiki/x` exits 1.
- *7.1.5.3.3.* Before writing, the archive computes the SHA-256 of
the fetched file and compares it with every file under `raw/`.
Only files of the same byte size are hashed. A match anywhere
under `raw/`, at any name or directory, is reused: nothing is
written under `raw/`, `raw_path` is the matching file, and
`raw_reused` is true. When `-r` names another place, the match
still wins and adds the warning
`already archived at <raw path>; --raw ignored`. Several matches
use the first in sorted path order.
- *acceptance*: Ingesting one file twice, the second time under
another name and with `-r other/dir`, leaves one raw copy and
the second run reports `raw_reused: true` and that copy's path.
- *acceptance*: A URL whose body matches an archived file reuses
that file.
- *7.1.5.3.4.* The archive never overwrites, renames, or deletes a
file under `raw/`. When the raw path holds a file with different
bytes, the first free name of `<slug>-2.<ext>`,
`<slug>-3.<ext>`, and so on is used.
- *acceptance*: Ingesting a changed file of the same name creates
`<slug>-2.<ext>` and the first copy is unchanged.
- *7.1.5.3.5.* When the raw copy is reused and a page under `wiki/`
already has that raw path in `sources` or that hash in
`ingest.sha256`, the run is a duplicate. It prints the
`Duplicate:` line, sets `duplicate_page`, writes no page and no
log line, and exits 0. `--force` converts again and replaces
that page, at its current path unless `-t` is set.
- *acceptance*: A second ingest of the same bytes exits 0,
reports the first page as `duplicate_page`, and leaves every
file under the wiki unchanged apart from the moved source.
- *7.1.5.3.6.* A file inside the wiki's `ingest/` directory is moved
into `raw/`, so it leaves the drop zone. A downloaded file is in
`ingest/`, so it is moved too. When the raw copy is reused, the
`ingest/` file is deleted, because `raw/` already holds its
bytes. Any other file is copied and the original is left as it
was. A file already under `raw/` is not archived again, and
conversion reads it in place. `-r` on such a file exits 1.
- *acceptance*: After ingesting `ingest/a.pdf`, that path is gone
and `raw/papers/a.pdf` holds its bytes. A file ingested from
`personal/` is copied and the original is unchanged.
- *7.1.5.4.* Convert.
- *7.1.5.4.1.* The registry picks the first converter, in
registration order, whose media types contain the detected type,
then the first whose extensions contain the file extension. With
no match, no page is written, stdout ends with the
`No converter` line, and the command exits 3.
- *acceptance*: A `.xyz` file of random bytes exits 3 and is
archived under `raw/other/`.
- *7.1.5.4.2.* A converter returns the markdown body, a title or
none, a metadata map, a conversion state of `complete` or
`partial`, and a list of warnings. A converter that cannot read
a file it claims, such as a corrupt or encrypted PDF, raises a
conversion error. The command then writes no page, prints
`Error: Conversion failed: <converter>: <reason>` on stderr, and
exits 3. The raw copy stays.
- *acceptance*: An encrypted PDF exits 3, its raw copy exists,
and no page is written.
- *7.1.5.4.3.* `markdown` keeps the body byte for byte. A
frontmatter block in the source moves into `source_meta`. The
title is that frontmatter `title`, then the first H1, then the
slug.
- *7.1.5.4.4.* `text` decodes UTF-8 with any BOM stripped, then the
charset hint, then `cp1252`. A fallback decode adds a warning.
The body is one fenced `text` block whose fence is longer than
any backtick run in the text. The title is the slug.
- *7.1.5.4.5.* `html` parses with beautifulsoup4 and converts with
markdownify.
- *7.1.5.4.5.1.* The title is `<title>`, then `og:title`, then the
first `<h1>`. `source_meta` holds `lang`, the canonical link,
and the `description`, `author`, `og:*`, and
`article:published_time` meta values that are present.
- *7.1.5.4.5.2.* The content root is the first `<article>`, then
`<main>`, then `<body>`. `script`, `style`, `noscript`,
`template`, and comments are dropped.
- *7.1.5.4.5.3.* A relative `href` or `src` is made absolute
against `<base href>`, then the origin URL. A local file has no
origin URL, so its relative links are left to *7.1.5.5.4.*.
- *7.1.5.4.5.4.* Headings are ATX. Bullets use `-`. `<pre>` becomes
a fenced block. A table becomes a pipe table.
- *7.1.5.4.5.5.* A body with no text outside markup is `partial`
with the warning `no text content; the page may need
JavaScript or a login`.
- *acceptance*: For a fixture with a nav, an article, a script,
a relative link `/a`, and a table, fetched from
`https://host/p`, the body holds the article text, no script
text, the link `https://host/a`, and a pipe table.
- *7.1.5.4.6.* `pdf` reads with pypdf.
- *7.1.5.4.6.1.* `source_meta` holds the page count and the
`/Title`, `/Author`, `/Subject`, `/Keywords`, `/CreationDate`,
and `/ModDate` values that are present. Dates are ISO 8601.
The title is the metadata title, then the slug.
- *7.1.5.4.6.2.* With an outline, each outline entry becomes a
heading before the text of its destination page. The level is
the entry depth plus 2, capped at 6. Without one, each page
starts with `## Page <n>`.
- *7.1.5.4.6.3.* A page with no text adds the warning
`page <n> has no text layer` and makes the result `partial`.
When every page is empty, the one warning is
`no text layer; the PDF may be scanned`.
- *acceptance*: A two-page fixture with no outline has
`## Page 1` and `## Page 2`. A fixture with one blank page is
`partial` and names that page.
- *7.1.5.5.* Write.
- *7.1.5.5.1.* The default page is
`wiki/<default_target>/<slug>.md`. `-t/--target PATH` is
relative to `wiki/`, and a leading `wiki/` is accepted. A value
that ends in `.md` names the page. Any other value is a
directory, and the page inside it is `<slug>.md`. After
resolving `..` and symlinks the path must be inside `wiki/`, and
it must not be `wiki/index.md`, `wiki/log.md`, `wiki/schema.md`,
or a file under `wiki/logs/`. Else exit 1.
- *acceptance*: `-t concepts` writes
`wiki/concepts/<slug>.md`. `-t ../raw/x.md` and `-t index.md`
exit 1.
- *7.1.5.5.2.* An existing page exits 1 unless `--force` is set. The
check runs before fetch, so a refusal writes nothing and
downloads nothing.
- *acceptance*: A second run on the same source exits 1 and the
page bytes are unchanged. With `--force` the page is replaced.
- *7.1.5.5.3.* The page frontmatter is complete when the run ends.
Every key below is present, in this order, even when its value
is empty: `[]`, `{}`, or `null`. `sources` is the raw path
relative to the wiki root. `title` is the converter title, else
the slug. `ingest` is filled from the fetch, archive, and convert
stages, so its values are facts about this run.
```yaml
---
name: <slug>
version: 0.1.0
title: <title>
created: <YYYY-MM-DD>
updated: <YYYY-MM-DD>
type: summary
status: raw
sources: [<raw path>]
source_meta: {<key>: <value>}
ingest:
origin: <url, or resolved file path>
origin_kind: <url or file>
final_url: <url after redirects, or null>
fetched: <UTC time, YYYY-MM-DDTHH:MM:SSZ>
media_type: <media type>
size_bytes: <int>
sha256: <64 lowercase hex of the raw copy>
raw_reused: <true or false>
converter: <converter name>
conversion: <complete or partial>
warnings: [<text>]
tool: fiona-assistant <package version>
tags: []
confidence: zero
llm:
- model: none
contested: false
contradictions: []
---
```
- *acceptance*: For each fixture, the page frontmatter parses
and has every key above. `ingest.sha256` equals the SHA-256 of
the file at `sources[0]`, and `ingest.warnings` equals the
JSON `warnings`.
- *7.1.5.5.4.* The body is `# <title>`, a blank line, the line
`Raw ingest of [<raw file name>](<link to the raw file>). Not
refined yet.`, a blank line, and the converted body. When the
converted body has an H1, every heading in it moves down one
level, and H6 stays H6. A relative link or image that does not
resolve from the page is replaced by its text, and the count is
one warning.
- *acceptance*: The page has exactly one H1, and mdlinter
reports no MDL001 for it.
- *7.1.5.5.5.* After writing, the writer runs the pymarkdown fixer
on that page only. Violations still left add the warning
`<N> markdown style violations left`.
- *acceptance*: Every fixture page passes fmlint with no
violation.
- *7.1.5.5.6.* `NOT_IMPLEMENTED` When violations are left, human
stdout prints, before the `Next: refine` line,
`Next: fix the style violations listed by fiona-assistant mdoctor chklint <page path>.`
The page must pass `wiki sanity` before it is refined. The JSON
`warnings` text does not change.
- *acceptance*: An HTML fixture with a bare URL that the fixer
leaves prints that line with the page path. A clean fixture
does not.
- *7.1.5.6.* Record. A run that archives appends one
bullet to `wiki/log.md` and bumps that file's patch `version` and
`updated`. The bullet is
`- <YYYY-MM-DD> - ingest <origin> -> <raw path> -> <page path> (raw)`,
or ends `-> <raw path> (no page)` when no page was written. The tool
does not edit `wiki/index.md` or any other page.
- *acceptance*: One successful run adds one line to `log.md` and
leaves `index.md` unchanged.
- *7.1.5.6.1.* `NOT_IMPLEMENTED` The bullet passes the wiki's own
markdownlint. The origin is a code span, so a URL is not a bare
URL (MD034). Each `->` part starts a continuation line indented
two spaces, so no line is longer than the bullet's longest path:
```text
- <YYYY-MM-DD> - ingest `<origin>`
-> <raw path>
-> <page path> (raw)
```
The bullet still counts as one entry for *3.3.4.* and the first
acceptance above.
- *acceptance*: After ingesting a URL of 90 characters into a
fresh wiki, `wiki sanity` reports no markdownlint hit in
`wiki/log.md`.
- *7.1.5.7.* `--raw-only` runs fetch, archive, and
record, and skips convert and write. It exits 0.
- *acceptance*: No page is written and `conversion` is `none`.
- *7.1.5.8.* `--dry-run` fetches into a temporary
directory outside the wiki, so a URL is still downloaded. It
prints `Dry run - no files will be modified:` and then the lines
a real run would print. It writes nothing under the wiki, does not
move the source, and deletes the temporary directory. `dry_run` is
true in JSON.
- *acceptance*: File hashes under the wiki match before and after
a dry run, and an `ingest/` source is still in place.
- *7.1.5.8.1.* `NOT_IMPLEMENTED` A dry run logs no record that says
a file was archived, moved, or written. Those records read
`dry run: would archive <raw path>` and
`dry run: would write <page path>`.
- *acceptance*: A dry run at `LOG_LEVEL=INFO` has `would archive`
on stderr and no line that starts with `archived` after the
logger name.
- *7.1.5.9.* `--pending` lists every page under
`wiki/` whose frontmatter `status` is `raw`, sorted by path, and
exits 0.
- *acceptance*: A page changed to `status: refined` is no longer
listed.
- *7.1.5.10.* Refinement is the agent's phase. The
tool never writes `status: refined`. A page is refined when
`status` is `refined`, `version` is `1.0.0` or later, the quality
signals are set and `llm.model` is not `none`, the page is listed
in `wiki/index.md`, and `wiki/log.md` has a line for the
refinement.
- *acceptance*: Fiona skill section 18 lists those conditions and
the steps that meet them, and says when to convert a source by
hand.
- *7.1.5.11.* The agent checks the frontmatter the
tool wrote before it refines the body. It may correct `title`,
`type`, `tags`, `source_meta`, `ingest.conversion`, and
`ingest.warnings` when the source shows they are wrong. It does
not hand-edit `ingest.sha256`, `ingest.size_bytes`,
`ingest.fetched`, `ingest.media_type`, or `ingest.tool`. When one
of those is wrong, it re-runs the ingest with `--force` or reports
the fault to the user.
- *acceptance*: Fiona skill section 18 has a metadata check step
that names the fields it may edit and the fields it may not.
- *7.1.5.12.* Archived sources do not fail
`wiki sanity`. Every check skips `raw/`.
- *acceptance*: After ingesting a markdown file with style
violations, `wiki sanity` reports none from its raw copy.
- *7.1.5.13.* `--json` prints only the JSON object
from section 4.7.6 on stdout, on exit 0 and on exit 3. On exit 1
and exit 4, stdout is empty.
- *acceptance*: The stdout of an exit 3 run parses with
`json.loads` and has `conversion: none`.
- *7.1.5.14.* `NOT_IMPLEMENTED` On exit 4 the agent gets the source
another way, or asks the user for it. The Fiona skill teaches
this in section 18.3. The steps, in order:
- *7.1.5.14.1.* Use a tool the agent already has that can read
that host with the user's access, such as an MCP server for
Confluence, SharePoint, Jenkins, or a ticket system, or an
authenticated API script in the wiki's `utils/scripts/`. Save
what it returns into `ingest/<slug>.<ext>` in its own format:
HTML as `.html`, a PDF as `.pdf`, plain text as `.txt` or `.md`.
Then run the `Next:` command with that file and
`--origin <url>`.
- *7.1.5.14.2.* When no tool has access, stop and ask the user to
save the page into `ingest/`, for example with the browser's
Save As or Print to PDF. Name the URL, the folder, and the
command the agent will run once the file is there. Do not
retry the URL.
- *7.1.5.14.3.* Never ask the user for a password, token, or
cookie. Never put credentials in a URL, a command line, a
page, or the log.
- *7.1.5.14.4.* A tool that returns a summary instead of the
source is not the source. Archive the content the tool
returned as it is, and say in `ingest.warnings` that it came
through that tool.
- *acceptance*: Fiona skill section 6.7 lists exit 4. Section 18.3
has the three cases above (a tool with access, ask the user,
never ask for credentials) and shows the `--origin` command.
### 7.2. housekeeping
> Status: Partially Implemented
Implements section 4.8. Housekeeping is the routine an LLM runs to
maintain a wiki. It gathers what needs work; the LLM does the work.
Paths in details and actions are relative to the wiki root.
- *7.2.1.* The steps run in this order: `check-update`, `merge`,
`ingest`, `sanity`, `graph`. Every step runs, even after a step
fails.
- *acceptance*: A wiki with a pending update file and a dead link
reports `merge` and `sanity` as `FAIL`, and `graph` still runs.
- *7.2.2.* `check-update` runs the check of *7.1.2.* and writes
nothing. It is `PASS` when nothing needs an update, a conflict, or
an addition, and `WARN` otherwise. Its summary is
`<U> update(s), <C> conflict(s), <A> addition(s)`. Each item is a
detail line: `update: <path>`, `conflict: <path> -> <update path>`,
or `add: <path>`. The action is
`Ask the user, then run fiona-assistant wiki update --dry-run and wiki update --yes.`
- *acceptance*: An `AGENTS.md` older than the template gives
`WARN` and the detail `conflict: AGENTS.md -> AGENTS-update.md`,
and no update file is written.
- *7.2.3.* `merge` lists every file whose stem ends with `-update`
outside `personal/`, `raw/`, and `.git/`, except a path matched by
`fiona.wiki.update.ignore_files` and except `corefiles.json`
(*7.1.2.15.*). That includes `fiona-settings-update.json`. It is
`FAIL` when one such file exists.
Each detail is `<update path> -> <target path>`, followed by a
space and `(target missing)` when the target is absent. The target
is the same name without `-update`. The action per file is:
```text
Merge <update path> into <target path>: keep the owner's edits, take the template changes, set version to at least <template version>, then delete <update path>.
```
`<template version>` is the update file's `version`, or the text
`the template version` when it has none.
- *acceptance*: A `wiki/schema-update.md` at `1.5.0` gives the
detail `wiki/schema-update.md -> wiki/schema.md` and an action
that names `1.5.0`.
- *acceptance*: `fiona-settings-update.json` gives the detail
`fiona-settings-update.json -> fiona-settings.json`.
- *acceptance*: `corefiles.json` at the wiki root is not a detail.
A `notes-update.md` whose path is in `ignore_files` is not a
detail.
- *7.2.4.* `ingest` lists the markdown pages under `wiki/` whose
frontmatter has `status: raw`. It is `WARN` when one exists, with
one detail per page and the action
`Refine each raw page, then set status: refined.`
- *acceptance*: A page with `status: raw` is `WARN` and named in
the details. With none, the step is `PASS`.
- *7.2.5.* `sanity` runs the checks of *7.1.3.* without the progress
display. It is `FAIL` when any check has a violation. The summary
is `<N> violation(s) in <K> check(s)`. The details are each
failing check's report lines. The action is
`Fix the violations, then run fiona-assistant housekeeping again.`
- *acceptance*: One dead link makes the step `FAIL` and a detail
line names the file.
- *7.2.5.1.* `NOT_IMPLEMENTED` The details are one line per
violation, `<check>: <path>:<line>:<column> <code> <message>`,
with the path relative to the wiki root. Report headers,
DEFINITIONS, and SUMMARY lines are not details. The full report
stays with `wiki sanity`.
- *acceptance*: A wiki with three MD013 hits and one dead link
gives exactly four `sanity` detail lines.
- *7.2.6.* `graph` writes `<wiki root>/wiki_graph.html` with the
defaults of section 4.7.5 and the wiki's graph settings. The summary
is `wrote wiki_graph.html (<nodes> nodes, <edges> edges)`. A
markdown page under `wiki/` with no inbound and no outbound link is
an orphan: the step is `WARN`, one detail `orphan: <path>` per
page, and the action
`Link each orphan page from a related page or from wiki/index.md.`
Update files are not orphans. An error writing the graph is `FAIL`
with the error as the detail.
- *acceptance*: After the run `wiki_graph.html` exists at the wiki
root, and an unlinked page under `wiki/` is named as an orphan.
- *7.2.7.* Housekeeping writes only `wiki_graph.html`. It does not
run `wiki update`, merge an update file, or fix a violation.
- *acceptance*: Apart from `wiki_graph.html`, every file hash is
unchanged after a run.
- *7.2.8.* `Next steps:` lists each step's actions in step order,
without repeats. `passed` is true when no step is `FAIL`.
- *acceptance*: `--json` stdout parses with `json.loads`, and a
wiki with only `WARN` steps has `passed: true` and exit 0.
---
### 7.3. librarian
Requirements for `librarian`.
#### 7.3.1. Folder and In-file TOC
> Status: Implemented
- *7.3.1.1.* Folder TOC (`librarian toc`).
- *7.3.1.1.1.* `table_of_contents.md` has frontmatter `name`,
`version`, `title`, `created`, `updated`, and `type: toc`.
`name` is `Table of Contents` and `version` is `1.0.0`.
- *7.3.1.1.2.* Entries list subfolders, file links, and `v<version>`,
or `v0.0.0` when version is missing or unreadable.
- *7.3.1.1.3.* Headers become nested GitHub-style anchor links.
Fenced code is not scanned. Files are read as `utf-8-sig` so a
BOM does not hide the first header.
- *7.3.1.1.4.* Skip the TOC file itself, hidden folders, and
`--exclude` matches.
- *7.3.1.1.5.* `--recursive-toc` writes a TOC in every subfolder.
- *7.3.1.1.6.* `--non-recursive` lists only markdown files directly
in the folder, ignoring `max_depth_from_dir`.
- *7.3.1.1.7.* `max_depth_from_file` is the deepest header level
(3 lists h1 through h3). `max_depth_from_dir` is how many
subfolder levels to include (`1` includes the folder and its
children). Absent or `0` means unlimited. Any other non-integer
or negative value exits 1 and names the key.
- *7.3.1.2.* In-file TOC (`librarian toc file`).
- *7.3.1.2.1.* Insert directly under the H1, after frontmatter. The
H1 itself is not listed. Headings inside fences are not listed.
A `#` inside frontmatter is not the H1.
- *7.3.1.2.2.* The block is wrapped in these markers. `sectionized`
is present only when numbering was on.
```html
<!-- toc-start: headings 2..6 -->
<!-- toc-end -->
```
Example with numbering:
`<!-- toc-start: sectionized, headings 2..6 -->`.
- *7.3.1.2.3.* On regeneration, marker parameters win, then
`--sectionized`, then settings. A `sectionized` marker keeps
numbering. The recorded heading range stays, so a later settings
change does not retarget an existing TOC. A bare
`<!-- toc-start -->` has no parameters and is upgraded on
regeneration. An invalid range is ignored with a warning.
- *7.3.1.2.4.* The block is replaced in place. A current block is
reported unchanged and not rewritten. Blank lines separate the
markers from the list so MD032 passes.
- *7.3.1.2.5.* When `sectionize` is true, entries and heading lines
gain numbers such as `1.1.`. Existing numbers are stripped before
renumbering so anchors still match.
- *7.3.1.2.6.* `toc file *` root is the nearest `fiona-settings.json`,
current directory first, then subdirectories. Skip hidden folders
and `build/`, `dist/`, `__pycache__/`, `*.egg-info/`. If none is
found, use the current directory.
- *7.3.1.2.7.* `toc file *` skips read-only zones, hidden files and
folders, `table_of_contents.md`, and `ignore_files` matches.
- *7.3.1.2.8.* `toc file <explicit path>` inside `personal/` or
`raw/`, or a missing or non-markdown path, exits 1 and writes
nothing.
- *7.3.1.3.* Two markdown files in a folder produce one TOC that
lists both versions and their headers, and omits a header that
sits inside a fence.
- *acceptance*: The fence header's anchor is absent.
- *7.3.1.4.* `max_depth_from_file: 3` omits `####` and lists `###`.
Absent or `0` lists every level.
- *acceptance*: The `####` line is absent only in the capped run.
- *7.3.1.5.* `max_depth_from_dir: 1` omits files two levels down.
Absent or `0` lists them. `--non-recursive` lists only the
folder's own files.
- *acceptance*: The deep file is absent in both limited modes.
- *7.3.1.6.* Re-running `toc file` replaces the marked block and
does not append a second TOC.
- *acceptance*: The file contains one `toc-start` marker.
- *7.3.1.7.* `min_heading: 3` omits `##`. `max_heading: 3` omits
`####`.
- *acceptance*: Each setting hides only its out-of-range level.
- *7.3.1.8.* An `ignore_files` match is not modified by `toc file *`.
An explicit path in `personal/` exits 1 and is unchanged.
- *acceptance*: Both files keep their original bytes.
#### 7.3.2. Non-markdown Version Tags
> Status: Planned
- *7.3.2.1.* `NOT_IMPLEMENTED` Python and shell files gain a version
tag. The tag format is undecided (Appendix C). Do not implement
until that format is chosen.
- *acceptance*: The format is written in this spec before any
scanner ships.
#### 7.3.3. Move with Link Rewrite
> Status: Planned
Implements section 4.1.2. In this section a link is one of:
- an inline link `[text](path)`
- an image `![alt](path)`
- a reference definition `[id]: path` at the start of a line,
after up to three spaces
A relative link holds a path relative to the folder of the file
that contains it. A markdown file is a file whose suffix is `.md`,
in any case.
- *7.3.3.1.* `NOT_IMPLEMENTED` Source and target.
- *7.3.3.1.1.* `<source>` is an existing file or folder.
- *7.3.3.1.2.* When `<target>` is an existing folder, the final
path is `<target>/<source name>`. Otherwise `<target>` is the
final path. Missing parent folders of the final path are
created.
- *7.3.3.1.3.* A folder source moves every file under it,
hidden files included, and keeps their layout relative to the
folder. The emptied source folder is removed.
- *acceptance*: `move wiki/a.md wiki/concepts/` with an existing
`wiki/concepts/` creates `wiki/concepts/a.md` and removes
`wiki/a.md`.
- *acceptance*: `move wiki/a.md wiki/new/b.md` with no `wiki/new/`
creates `wiki/new/` and `wiki/new/b.md`.
- *acceptance*: `move wiki/notes wiki/archive/notes` moves
`wiki/notes/x.md` and `wiki/notes/img/p.png` to the same
relative places, and `wiki/notes/` no longer exists.
- *7.3.3.2.* `NOT_IMPLEMENTED` Refusals. Each prints its line on
stderr, writes nothing, and exits 1.
- *7.3.3.2.1.* Missing source:
`Error: '<source>' does not exist.`
- *7.3.3.2.2.* The final path exists:
`Error: '<final path>' already exists.`
- *7.3.3.2.3.* The final path is the source, or lies inside a
folder source: `Error: cannot move '<source>' into itself.`
- *7.3.3.2.4.* The source, or the final path, is inside
`personal/` or `raw/` of the wiki root (*1.2.3.*). The wiki
root is found as in *7.3.1.2.6.*:
`Error: '<path>' is in a read-only zone.`
- *acceptance*: Each refusal leaves every file under the wiki
root byte-identical and creates no folder.
- *7.3.3.3.* `NOT_IMPLEMENTED` Outbound links in each moved
markdown file are rewritten.
- *7.3.3.3.1.* The link path is percent-decoded and resolved
against the file's old folder. When it names an existing file
or folder, the new path is the relative path from the file's
new folder to that target's location after the move. A target
moved in the same run counts at its new location.
- *7.3.3.3.2.* The new path uses `/`. A `#fragment` and a
`?query` are kept as written. A link title (`"t"`, `'t'`, or
`(t)`) is kept. A path written in `<...>` stays in `<...>`. A
leading `./` is kept when the old path had one. Outside
`<...>`, a space is written as `%20`.
- *7.3.3.3.3.* A link whose new text equals the old text is not
rewritten and not counted.
- *acceptance*: `wiki/a.md` holds `[b](b.md#x)` and
`![i](img/p.png "t")`. After `move wiki/a.md wiki/concepts/a.md`
the file holds `[b](../b.md#x)` and `![i](../img/p.png "t")`,
and stdout has `rewrote 2 link(s) in wiki/concepts/a.md`.
- *acceptance*: `wiki/notes/a.md` links `b.md` (a sibling) and
`../index.md`. After `move wiki/notes wiki/archive/notes`,
`b.md` is unchanged and `../index.md` becomes
`../../index.md`.
- *acceptance*: `[d]: ../docs/My%20File.md` keeps its
`%20` after the rewrite.
- *7.3.3.4.* `NOT_IMPLEMENTED` These stay byte-identical:
- *7.3.3.4.1.* External links: any path with a URI scheme
(`http:`, `https:`, `mailto:`, or `<scheme>://`).
- *7.3.3.4.2.* Pure anchors (`#section`) and absolute paths (a
leading `/`, or a Windows drive letter).
- *7.3.3.4.3.* Wikilinks `[[page]]`. They resolve by stem, so a
move that keeps the file name does not change them. A rename is
*7.3.3.6.3.*
- *7.3.3.4.4.* Text inside fenced code and inline code, HTML
tags such as `<a href>` and `<img src>`, and autolinks
`<https://...>`.
- *7.3.3.4.5.* Frontmatter. `sources:` paths are relative to the
wiki root, not to the file.
- *7.3.3.4.6.* A relative link that did not resolve before the
move. It is counted on an `unresolved` line.
- *acceptance*: A moved file that holds one example of each item
above differs from the original only in its rewritten links,
and a dead link adds `unresolved 1 link(s) in <path>`.
- *7.3.3.5.* `NOT_IMPLEMENTED` A file that is moved or rewritten
keeps everything that is not a rewritten link: encoding, a UTF-8
BOM, line endings, the final newline, and frontmatter `version`
and `updated`. A file that is not markdown moves byte for byte.
- *acceptance*: A CRLF markdown file still has CRLF endings after
the move, and a moved PNG has the same SHA-256.
- *7.3.3.6.* `NOT_IMPLEMENTED` Inbound links. The command searches
the whole wiki for references to the moved paths and points them
at the new paths, so the move leaves no dead link.
- *7.3.3.6.1.* The search covers every markdown file under the
wiki root (*7.3.1.2.6.*) that did not move. Hidden folders
(`.git` included) are skipped.
- *7.3.3.6.2.* A relative link in such a file that resolves,
before the move, to the source, or to any file or folder under
a folder source, is rewritten to the relative path from that
file's folder to the new location. *7.3.3.3.2.* and
*7.3.3.3.3.* apply. *7.3.3.4.* applies, except its last item:
a link that does not point at a moved path is left alone and
is not counted as unresolved.
- *7.3.3.6.3.* When a moved markdown file changes its stem,
every wikilink in the wiki, in moved files too, whose target
equals the old stem (case-insensitive, as MDL002 matches) is
rewritten to the new stem. A `#section` and a `|label` are
kept. A wikilink is left alone when another markdown file in
the wiki still has the old stem after the move.
- *7.3.3.6.4.* A file inside `personal/` or `raw/` is never
written (*1.2.3.*). Each of its links that point at a moved
path is counted on a `read-only` line, so the user sees what
will be dead.
- *7.3.3.6.5.* A file with nothing to rewrite stays
byte-identical.
- *acceptance*: `wiki/index.md` holds `[a](a.md#top)`. After
`move wiki/a.md wiki/concepts/a.md` it holds
`[a](concepts/a.md#top)`, and stdout has
`rewrote 1 link(s) in wiki/index.md`.
- *acceptance*: `wiki/log.md` links `notes/x.md` and
`notes/img/p.png`. After `move wiki/notes wiki/archive/notes`
those links are `archive/notes/x.md` and
`archive/notes/img/p.png`.
- *acceptance*: `wiki/index.md` holds `[[a#intro|A page]]`.
After `move wiki/a.md wiki/b.md` it holds
`[[b#intro|A page]]`. When `wiki/other/a.md` also exists, the
wikilink is unchanged.
- *acceptance*: `raw/n.md` links `../wiki/a.md`. After the move
`raw/n.md` is byte-identical and stdout has
`read-only 1 link(s) in raw/n.md`.
- *acceptance*: After a move with no `read-only` and no
`unresolved` line, `wiki sanity` reports no MDL001 or MDL002
that the move caused.
- *7.3.3.7.* `NOT_IMPLEMENTED` Order of work: check every refusal,
compute every rewrite, then move and write. `--dry-run` stops
after the compute step and prints the same lines a real run
prints, each with the `[dry-run]` prefix. An `OSError` during
the write step prints `Error: <message>` on stderr and exits 1.
- *acceptance*: `--dry-run` leaves the tree byte-identical, and
its stdout equals the stdout of the real run with each line
prefixed.
- *7.3.3.8.* `NOT_IMPLEMENTED` The command runs no git operation.
The wiki records the move with `git save` (*3.3.3.*).
- *acceptance*: In a git wiki the commit count is the same before
and after `move`.
### 7.4. mdoctor
> Status: Partially Implemented
Implements section 4.2 and the pymarkdown keys in section 5.1.
Settings load at construction. The default settings path is the
packaged template `fiona-settings.json`. Callers, including
`run_sanity.py`, may pass another path. A missing file means plain
pymarkdown defaults.
- *7.4.1.* `mdoctor fix` repairs an auto-fixable violation such as
MD047 and lists the path.
- *acceptance*: The file ends with a newline and stdout contains
`PyMarkdownFix: 1 file(s) fixed`.
- *7.4.2.* A tree with nothing to fix prints
`PyMarkdownFix: 0 file(s) fixed`.
- *acceptance*: That line is the only stdout line.
- *7.4.3.* A scan does not ask PyMarkdown to run git. When
`respect_gitignore` is true and the path is inside the working
directory, directory names from `.gitignore` and `.git` are
exclude patterns. A path outside the working directory is still
skipped by the directory-name check after the scan.
`respect_gitignore` false still scans those directories, except
`.git` when the path is inside the working directory.
- *acceptance*: `scan_path` is called with `respect_gitignore`
false. Inside the working directory the exclude list contains
`.git/` and each plain directory name from `.gitignore`.
- *7.4.4.* `NOT_IMPLEMENTED` `chklint` and `fix` accept one markdown
file as well as a directory, the same as `fmlint chklint`. A path
that does not exist prints `Error: '<path>' does not exist.` on
stderr and exits 1.
- *acceptance*: `mdoctor fix wiki/sources/a.md` lists only that
file, and every other file hash is unchanged.
### 7.5. fmlint
> Status: Implemented
Implements sections 4.3 and 6.2.
- *7.5.1.* `required_fields` in settings replaces the default pair
on the next run.
- *acceptance*: Adding `title` to the list reports a file that
previously passed.
- *7.5.2.* `enabled: false` reports no frontmatter violations.
- *acceptance*: A file with no frontmatter is clean for fmlint.
- *7.5.3.* A code in `ignore_rules` is omitted. A glob in
`ignore_file` skips those paths.
- *acceptance*: `ignore_rules: ["FML002"]` hides a missing-block
finding and still reports a missing `name`.
### 7.6. mdlinter
> Status: Implemented
Implements section 6.3. Checks run in registry order. A new check
is a method plus an entry in that registry. `lint_file` resolves
wikilinks against the file's parent tree. `lint_directory` scans
the wiki.
- *7.6.1.* The report header is `Markdown Lint Violations` and the
section order matches section 6.1.
- *acceptance*: DEFINITIONS lists `MDL001: Dead Link` when a dead
markdown link exists.
### 7.7. pycustomlint
> Status: Implemented
Implements section 6.4. The sanity runner in section 7.14 includes
it as `pycustomlint`. There is no `fiona-assistant pycustomlint`
command; the repository runner calls the library.
- *7.7.1.* `--json` prints a JSON list of violation objects.
- *acceptance*: `json.loads` succeeds and each object has `code`.
### 7.8. python-ruff
> Status: Implemented
Implements section 4.4. The sanity check name is `ruff` in
`run_sanity.py` and `python-ruff` in wiki sanity.
- *7.8.1.* An unused import appears in DEFINITIONS, FILE, and
SUMMARY.
- *acceptance*: The FILE line contains the ruff rule code.
- *7.8.2.* `NOT_IMPLEMENTED` A `python-ruff fix` subcommand mirrors
`mdoctor fix` and calls the existing `fix=True` path.
- *acceptance*: An auto-fixable ruff finding is gone after the
command and the path is listed.
### 7.9. Shellcheck
> Status: Planned
- *7.9.1.* `NOT_IMPLEMENTED` Wrap the `shellcheck` binary and print
the shared report.
- *acceptance*: A known shellcheck finding shows file, line, and
rule code.
- *7.9.2.* `NOT_IMPLEMENTED` A missing binary prints an install hint
and exits non-zero, with no traceback.
- *acceptance*: The message names `shellcheck`.
- *7.9.3.* `NOT_IMPLEMENTED` Wiki sanity runs this check.
- *acceptance*: The `Results:` table includes the new check name.
### 7.10. Markdown Code Blocks
> Status: Planned
- *7.10.1.* `NOT_IMPLEMENTED` Extract fenced blocks. Lint Python
blocks with Ruff and shell blocks with shellcheck. Skip other
languages.
- *acceptance*: An unused import inside a Python fence is
reported. Prose outside fences is not.
- *7.10.2.* `NOT_IMPLEMENTED` The finding names the markdown path,
the fence index, the line inside the fence, and the markdown
line number.
- *acceptance*: The reported markdown line is the fence line that
contains the import.
- *7.10.3.* `NOT_IMPLEMENTED` Wiki sanity runs this check.
- *acceptance*: The check appears in the `Results:` table.
### 7.11. json-lint
> Status: Implemented
Implements sections 4.5 and 6.5. Both sanity runners include it.
- *7.11.1.* A JSON violation fails `run_sanity.py` and wiki sanity.
- *acceptance*: The `Results:` row for `jsonlint` is FAIL and the
process exit code is non-zero.
- *7.11.2.* An empty tree of JSON files prints no violations.
- *acceptance*: The report says no violations were found.
### 7.12. Git Operations
> Status: Implemented
Implements section 4.6. `wiki init` writes the repository
`.gitignore` patterns and `personal/` (section 7.1.1). Those paths
are never staged by these commands except where `git add --all`
respects that ignore file.
- *7.12.1.* `save --description note` on a dirty tree creates one
commit whose message starts with the UTC timestamp and ends with
`note`.
- *acceptance*: `git log -1 --format=%s` matches
`<YYYY-MM-DD HH:MM:SS> note`.
- *7.12.2.* `snapshot demo --description note` then
`restore demo` returns file bytes to the tagged tree and adds a
new commit. The pre-restore history is still reachable.
- *acceptance*: `git rev-parse HEAD~1` after restore is the
snapshot commit or a descendant, and the tagged tree matches
the worktree.
- *7.12.3.* `undo` and `restore` on a dirty tree exit 1 and mention
`clean`.
- *acceptance*: The HEAD sha is unchanged.
- *7.12.4.* `clean` leaves `git status` empty of changes.
- *acceptance*: A following `status` prints `Status: clean`.
### 7.13. Backup Script
> Status: Planned
- *7.13.1.* `NOT_IMPLEMENTED` A bundled command writes a restorable
archive of the wiki.
- *acceptance*: Restoring that archive reproduces the wiki tree
from before the archive.
- *7.13.2.* `NOT_IMPLEMENTED` The archive is stored outside the wiki
tree.
- *acceptance*: The archive path is not under the wiki root.
- *7.13.3.* `NOT_IMPLEMENTED` The filename is
`YYYYMMDD_hhmm_<description>.zstd`.
- *acceptance*: The file is valid zstd and the name matches.
- *7.13.4.* `NOT_IMPLEMENTED` The command asks for a directory and
uses `~/Downloads` when the user accepts the default.
- *acceptance*: Pressing Enter writes the archive under
`~/Downloads`.
### 7.14. Test and Sanity Runners
> Status: Implemented
- *7.14.1.* Shared engine, used by `run_sanity.py` and wiki sanity.
- *7.14.1.1.* One thread per check. Results stay in registry order.
- *7.14.1.2.* Summary line `PASS/FAIL - <detail> (MM:SS)`. Minutes
may exceed 59 and are zero-padded. A missing duration omits the
suffix.
- *7.14.1.3.* `SANITY PASSED (MM:SS)` lists check names in registry
order. `Sanity Failed - N checks need attention:` has no total
time.
- *7.14.1.4.* A runner may add an extra result row, such as
coverage, and per-check failure details.
- *7.14.1.5.* `progress(name, result)` runs when a check finishes.
`heartbeat(remaining_names)` runs about every 0.5 seconds. Start
and finish are logged at INFO.
- *7.14.2.* Progress display.
- *7.14.2.1.* On a TTY, one bar per check, redrawn in place with
cursor-up and line clear. Example:
`pytests [###------------] 22% 6.5s`.
- *7.14.2.2.* The bar fills as elapsed / `expected-seconds`, capped
at 100%.
- *7.14.2.3.* A finished check shows a full bar and `(MM:SS)`:
`markdownlint [###############] 100% done (00:07)`.
- *7.14.2.4.* The final block stays on screen above the reports.
- *7.14.2.5.* `--non-interactive`, also when stdout is not a TTY,
prints `Running Sanity` or `Running Wiki Sanity` once, then one
dot per `interval` seconds. A run shorter than the interval
prints no dots. A newline ends the line before the report. No
ANSI.
- *7.14.2.6.* `--json` on wiki sanity skips the display.
- *7.14.2.7.* Bar scale and dot interval come from section 5.2.
- *7.14.3.* `run_tests.py` and the pytests check.
- *7.14.3.1.* The command includes `--color=no` and JUnit XML
`junit_family=xunit1`.
- *7.14.3.2.* Displayed messages prefer regex-parsed pytest output.
JUnit fills missing fields and replaces empty messages. Match by
basename plus test name so path spellings do not duplicate a
failure.
- *7.14.3.3.* A missing short summary logs a WARNING that names
`PYTEST_ADDOPTS`, `--no-summary`, and plugin output as likely
causes.
- *7.14.3.4.* `LOG_LEVEL=DEBUG` logs the pytest command, return
code, parsed failures, and raw output when the parse disagrees.
- *7.14.3.5.* The report nests failures as `Class : test_name`
under the file, then the assert text, or the exception text when
there is no assert line.
- *7.14.3.6.* The `Sanity Failed` verdict lists failing test names
only. Assert text stays in the report body.
- *7.14.3.7.* The `coverage:` line is
`PASS/FAIL - <percent>% (required <percent>%)` and is untimed. A
shortfall is its own verdict entry.
- *7.14.4.* `run_sanity.py` check order is `pytests`, `markdownlint`,
`ruff`, `frontmatter_lint`, `pycustomlint`, `jsonlint`.
- *7.14.5.* A failing test is listed under its file with the assert
text when pytest is run with `--no-summary` or `PY_COLORS=1`.
- *acceptance*: The name and assert still appear.
- *7.14.6.* A green run prints every `Results:` line with `(MM:SS)`
and ends with `SANITY PASSED (MM:SS)`.
- *acceptance*: The banner time is the wall time of the whole run.
- *7.14.7.* A piped run contains no ANSI and prints one dot per
interval after the label, unless the run is shorter than the
interval.
- *acceptance*: The captured stdout has no ESC byte.
### 7.15. html-js-lint
> Status: Planned
The `html-js-lint` console script is not installed until this
section is built. The intended tool validates HTML with
`html5validator` (Java 8+) and JavaScript with `eslint` on PATH.
- *7.15.1.* `NOT_IMPLEMENTED` HTML files (default `.html` and
`.htm`) are checked. Findings include path, line, and column.
`.gitignore` and read-only zones are skipped.
- *acceptance*: An unclosed tag is reported. A valid file is not.
A tree with no HTML prints no HTML findings.
- *7.15.2.* `NOT_IMPLEMENTED` JavaScript files (default `.js` and
`.javascript`) are checked with eslint, honoring `.eslintrc`.
- *acceptance*: A syntax error names path, line, and message.
A clean file produces no finding.
- *7.15.3.* `NOT_IMPLEMENTED` Inline `<script>` is linted only when
`html.lint_inline_js` is true (default false). Findings use the
HTML path and line.
- *acceptance*: With the flag off, inline script is ignored.
- *7.15.4.* `NOT_IMPLEMENTED` CLI is
`fiona-assistant html-js-lint`, default `chklint .`, shared
report, exit 0 even with violations, and `--fix` where the
underlying tool can fix.
- *acceptance*: `--fix` reports how many files changed.
- *7.15.5.* `NOT_IMPLEMENTED` Settings, under a dedicated section,
with defaults when the section is missing:
| Key | Default |
| --- | --- |
| `html.extensions` | `["html", "htm"]` |
| `html.ignore_patterns` | `[]` |
| `html.lint_inline_js` | false |
| `javascript.extensions` | `["js", "javascript"]` |
| `javascript.eslint_config` | `.eslintrc` |
| `javascript.ignore_patterns` | `[]` |
- *acceptance*: A custom extension list is the set that is scanned.
- *7.15.6.* `NOT_IMPLEMENTED` Both `run_sanity.py` and wiki sanity
run this check.
- *acceptance*: A violation fails that sanity run.
### 7.16. settings setup
> Status: Planned
`fiona-assistant init` remains the non-interactive wiki bootstrap
(section 7.1.1): it copies the template and does not ask questions.
`fiona-assistant setup` is a separate command for reviewing settings
on an existing wiki. It does not create the wiki tree.
- *7.16.1.* `NOT_IMPLEMENTED` `setup` walks every configurable key
in `fiona-settings.json`, shows the current value, and accepts
Enter as "keep current".
- *acceptance*: Enter through every prompt leaves the file
semantically unchanged.
- *7.16.2.* `NOT_IMPLEMENTED` Booleans can be toggled. Strings and
integers accept a typed value. Values are checked before write.
- *acceptance*: A non-integer for `indent_len` is rejected and the
file is not written.
- *7.16.3.* `NOT_IMPLEMENTED` `--yes` or `--default` applies defaults
with no prompts.
- *acceptance*: The command exits 0 on a pipe when the flag is set.
- *7.16.4.* `NOT_IMPLEMENTED` A missing settings file is created
from the template defaults. A corrupt file is reported and not
silently replaced.
- *acceptance*: Exit 0 after creating a missing file. Exit non-zero
on corrupt JSON, with the parse error on stderr.
- *7.16.5.* `NOT_IMPLEMENTED` `setup` can be run again later to
change an existing wiki.
- *acceptance*: A second run can change `log_level` and leave
every other key as it was.
### 7.17. Logging
> Status: Partially Implemented
The contract is section 2. Reports stay on stdout. Diagnostics go
through the logging module to stderr and, when configured, to the
file sink.
- *7.17.1.* Each CLI entry calls the shared setup before doing work.
- *acceptance*: `LOG_LEVEL=DEBUG` emits debug lines on stderr and
the human report stays on stdout.
- *7.17.2.* Dropping `file_log_location` leaves no file handler.
- *acceptance*: No log file is created for that run.
- *7.17.3.* `NOT_IMPLEMENTED` The shipped `fiona-settings.json` sets
`fiona.log_level` to `WARNING`, the same as the default in
section 5.1. `fiona.file_log_level` stays `INFO`, so `fiona.log`
keeps the detail. PowerShell shows every native stderr line as a
`NativeCommandError`, so an `INFO` record looks like a failure
even when the exit code is 0. When this is built, the shipped
document in section 5.1 changes to match.
- *acceptance*: In a fresh wiki, `wiki sanity` on the clean
template and `wiki ingest` of a local markdown file write nothing
on stderr, and `fiona.log` has their `INFO` lines.
### 7.18. Documentation
> Status: Partially Implemented
`README.md` links to this spec. `src/README.md`,
`src/fiona/librarian/README.md`, and `src/fiona/mdoctor/README.md` exist and
are behind the current CLIs.
- *7.18.1.* `NOT_IMPLEMENTED` `librarian/README.md` documents `list`
and `toc`. `mdoctor/README.md` documents `list`, `chklint`, and
`fix`.
- *acceptance*: A new reader can install and run each command
from those files alone.
- *7.18.2.* `src/README.md` describes `src/fiona/` (including
`wiki/`, `python_ruff/`, and `data/llm_wiki/`), repo-root `tests/`,
and `scripts/` (`run_tests.py`, `run_sanity.py`, `settings_dev.json`).
- *acceptance*: Those names appear in the structure section.
- *7.18.3.* Documentation markdown passes `mdoctor chklint`.
- *acceptance*: `README.md` contains a markdown link to `SPEC.md`
and the lint report is clean.
- *7.18.4.* `NOT_IMPLEMENTED` The README Quick Start has a short
"After ingest" and "On Windows" part. It says that a file in
`ingest/` moves into `raw/` and a second run on it fails, that
`LOG_LEVEL=ERROR` hides diagnostics until *7.17.3.* ships, the loop
`housekeeping`, then fix style violations, then refine and set
`status: refined`, that `--version` (*2.3.*) or `pip show
fiona-assistant` gives the installed version, and that commands
run from the wiki root until section 7.20 is built. The Fiona skill
carries the same loop.
- *acceptance*: Each of those five points appears in `README.md`,
and the after-ingest loop appears in the Fiona skill.
### 7.19. Tool Roles
> Status: Planned
- *7.19.1.* `NOT_IMPLEMENTED` Each `src/` tool has one role:
ingestion (`ingest/` to `raw/`), processing (`raw/` to
`wiki/concepts/`), or maintenance (`index.md` / `log.md` and
the linters).
- *acceptance*: A table in this spec or the README assigns every
tool exactly one role.
- *7.19.2.* `NOT_IMPLEMENTED` Ingestion is `wiki ingest`
(section 7.1.5). Missing processing steps become scripts under the
template `utils/` directory, such as `process.py`.
- *acceptance*: Each new script is named in the role table.
### 7.20. Wiki Root Discovery
> Status: Planned
Today every optional directory defaults to `.`. `toc file *` already
searches for `fiona-settings.json` as specified in section 7.3.1.
Graph settings already walk parent directories. The dispatcher does
not.
- *7.20.1.* `NOT_IMPLEMENTED` When the directory argument is omitted,
commands use the wiki root from Appendix C's discovery rule.
- *acceptance*: `fiona-assistant mdoctor` from a subdirectory lints
that root, not the subdirectory.
- *7.20.2.* `NOT_IMPLEMENTED` An explicit directory always wins.
- *acceptance*: `mdoctor chklint sub/dir` lints `sub/dir` even when
a settings file exists higher up.
- *7.20.3.* `NOT_IMPLEMENTED` With no settings file found, the
default stays `.`.
- *acceptance*: Behavior matches today's commands.
- *7.20.4.* `NOT_IMPLEMENTED` The file is loaded once per process
and shared. mdoctor already reads it for rules.
- *acceptance*: Two tools in one invocation see the same object.
### 7.21. Bootstrap install modes
> Status: Implemented
`bootstrap` (bash, sourced) and `bootstrap.ps1` (PowerShell) prepare
a Python environment and install this repository. With no install
flag, both create or reuse a virtual environment and run
`python -m pip install -e ".[dev]"`. `scripts/bootstrap_install.py`
is the pip-argument source both scripts call.
- *7.21.1.* `--local` installs this repository as a normal package.
The pip arguments are `install` and `.`. They do not include `-e`
and they do not include the `dev` extra.
- *acceptance*: `pip_install_args(local=True, no_env=False)` is
`["install", "."]`.
- *7.21.2.* `--no-env` does not create or activate a virtual
environment. The install adds `--user`, so the console scripts
land in the user scripts directory and run without activating an
environment.
- *acceptance*: `pip_install_args(local=False, no_env=True)`
contains `--user`. PowerShell skips virtual environment creation
when `$NoEnv` is set. Bash skips it when `NO_ENV` is `true`.
- *7.21.3.* `--local` and `--no-env` combine. The install is
`install`, `--user`, and `.`, with no `-e`.
- *acceptance*: `pip_install_args(local=True, no_env=True)` is
`["install", "--user", "."]`.
- *7.21.4.* Neither flag keeps the default: `-e` and `.[dev]`, and
no `--user`.
- *acceptance*: `pip_install_args(local=False, no_env=False)` is
`["install", "-e", ".[dev]"]`.
- *7.21.5.* Bash adds `--break-system-packages` on every install.
PowerShell does not.
- *acceptance*: `break_system_packages=True` inserts that flag
after `--user` when both are set, and after `install` when
`--user` is absent.
- *7.21.6.* A virtual-environment install on PowerShell must be
dot-sourced so the environment stays active. `--no-env` may be
executed with `-File`, because there is no environment to keep
active. Bash is sourced for both modes.
- *acceptance*: PowerShell reports a dot-source error when
`--local` is used without `--no-env` and the script is not
dot-sourced. The message does not call `--local` an unknown
option.
## Appendix A. Non-Goals
- *A.1.* No LLM or network API inside Fiona-Assistant. The tools are
deterministic.
- *A.2.* No automated edit of `personal/` or `raw/` content.
`wiki ingest` adds new files under `raw/` and edits none
(*1.2.3.*).
- *A.3.* No cloud sync and no remote backup. Version control is local.
The planned archive in *7.13.* is a local file.
- *A.4.* Deterministic tools make no cloud API calls. Each command
uses only local files and finishes with no network access.
- *acceptance*: With the network unavailable, `wiki sanity` and
`wiki graph` still complete on a local wiki.
- *acceptance*: Opening the generated graph HTML is an exception:
the browser loads D3 from `d3js.org` (*7.1.4.*). The generator
itself does not call that host.
- *acceptance*: `wiki ingest <url>` is the other exception. It
downloads only the URL the user passed (*7.1.5.2.*). With a local
path it makes no network call.
---
## Appendix B. Priority Order
| Priority | Feature | Section | Status |
| --- | --- | --- | --- |
| P1 | wiki update / check-update | 7.1.2 | Implemented |
| P1 | Git save, snapshot, restore, clean, undo, status | 7.12 | Implemented |
| P1 | wiki sanity, including dead links | 7.1.3 | Implemented |
| P1 | housekeeping | 7.2 | Partially Implemented |
| P1 | wiki init | 7.1.1 | Partially Implemented |
| P1 | CLI errors without traceback | 1.2.7 | Planned |
| P1 | wiki graph | 7.1.4 | Partially Implemented |
| P2 | fmlint | 7.5 | Implemented |
| P2 | json-lint | 7.11 | Implemented |
| P2 | mdoctor fix | 7.4 | Partially Implemented |
| P2 | Logging | 7.17 | Partially Implemented |
| P2 | Settings schema | 5 | Implemented |
| P2 | Sanity reporting, coverage, progress | 7.14 | Implemented |
| P2 | Shellcheck | 7.9 | Planned |
| P2 | Markdown code-block lint | 7.10 | Planned |
| P2 | html-js-lint | 7.15 | Planned |
| P2 | python-ruff fix subcommand | 7.8.2 | Planned |
| P2 | Wiki root discovery | 7.20 | Planned |
| P2 | settings setup | 7.16 | Planned |
| P2 | Bootstrap install modes | 7.21 | Implemented |
| P2 | wiki ingest | 7.1.5 | Partially Implemented |
| P2 | Ingest login detection and `--origin` | 7.1.5.2.8, 7.1.5.2.9, 7.1.5.14 | Planned |
| P3 | `--version` | 2.3 | Planned |
| P3 | fmlint enforcement of quality signals | 6.2 | Planned |
| P3 | Backup archive | 7.13 | Planned |
| P3 | Documentation refresh | 7.18 | Partially Implemented |
| P2 | Graph skips gitignored files | 7.1.4.17 | Planned |
| P3 | Node importance filter | 7.1.4.16 | Planned |
| P2 | Graph directory nodes | 7.1.4.18 | Planned |
| P2 | Graph live view (`--serve`) | 7.1.4.19 | Planned |
| P2 | librarian move with link rewrite | 7.3.3 | Planned |
| P4 | Non-markdown version tags | 7.3.2 | Planned |
| P4 | Tool-role table | 7.19 | Planned |
---
## Appendix C. Open Questions
- Python and shell version-tag format (section 7.3.2) is undecided.
Do not build a scanner until this section names the format.
- Wiki root discovery when several `fiona-settings.json` files exist
under the start directory: the rule is not chosen. Candidates are
"the nearest directory wins" and "error when two files sit at the
same depth". Parent walk is already what graph settings do, and is
a third candidate for the default directory. Section 7.20 stays
planned until one rule is written here.
- json-lint `chklint` exits 1 when violations exist. mdoctor, fmlint,
and python-ruff exit 0. A future change may align them. Until this
line changes, implement the split in section 4.
- `wiki ingest` takes one file or URL. A directory argument, and a
bare `wiki ingest` that drains `ingest/`, are not specified.
- More converters for section 7.1.5, each one class under *7.1.5.1.2.*:
Office files (`mammoth` for docx, `python-pptx`, `openpyxl`), or
`markitdown` as one optional extra that covers them all but pulls
in `magika` and `onnxruntime`. OCR for scanned PDFs is not chosen.
PDF tables need `pdfplumber` or similar; pypdf reads text only.
- Images and other assets referenced by an ingested source are not
copied into `raw/assets/`.
- Third-party converters could register through a package entry
point group such as `fiona.ingest_converters`. Not decided.
- The SHA-256 check in *7.1.5.3.3.* scans `raw/` on every run. A
cached hash index is not specified until a large `raw/` makes the
scan slow.
- A directory view (*7.1.4.18.5.*) draws outside nodes that its
pages point at. Drawing outside nodes that point into the
directory, as a second setting, is not decided.
- `--serve` (*7.1.4.19.5.*) reloads the page on every rebuild.
Patching the drawn graph in place, with no reload, is not
specified. It needs the D3 joins keyed by node id.
- `librarian move` (*7.3.3.4.4.*) does not rewrite links in HTML
tags (`<a href>`, `<img src>`). Whether it should is not
decided.
- `git undo` and `git clean` discard work with no prompt. The
2026-10-02 manual test lost a `git save` commit this way. A TTY
prompt with `--yes` (*1.2.2.*) would make agents pass `--yes` on
every call. Not decided.
- `wiki ingest` sends no credentials (*7.1.5.2.8.1.*). A cookie
file, a bearer token from an environment variable, or a
`--header` option would let it read some login-only hosts
itself. Each one puts secrets near the wiki. Not decided.
- On exit 4, `--json` prints nothing (*7.1.5.13.*). A JSON error
object such as `{"error": "auth_required", "url": ...,
"reason": ...}` would save the agent parsing stderr. Not decided.
---
## Appendix D. Maintainer Notes
These notes are not requirements. They point at the current tree.
| Area | Module |
| --- | --- |
| CLI dispatch | `src/fiona/__main__.py` |
| Logging | `src/fiona/logging_config.py` |
| template path | `src/fiona/paths.py` |
| librarian list | `src/fiona/librarian/librarian.py` |
| TOC | `src/fiona/librarian/toc_generator.py` |
| librarian move (planned) | `src/fiona/librarian/mover.py`, `src/fiona/librarian/link_rewriter.py` |
| mdoctor | `src/fiona/mdoctor/mdoctor.py` |
| fmlint, mdlinter, pycustomlint, json-lint | `src/fiona/customlinter/` |
| ruff wrapper | `src/fiona/python_ruff/python_ruff.py` |
| wiki init | `src/fiona/wiki/wiki.py` |
| wiki gitignore | `src/fiona/wiki/ignore.py` |
| wiki update | `src/fiona/wiki/update_manager.py` |
| wiki sanity | `src/fiona/wiki/sanity_manager.py` |
| housekeeping | `src/fiona/wiki/housekeeping.py` |
| wiki graph | `src/fiona/wiki/graph_generator.py`, `src/fiona/wiki/templates/graph_template.html`, `src/fiona/wiki/templates/graph_physics.js` |
| wiki graph `--serve` (planned) | `src/fiona/wiki/graph_server.py`, `src/fiona/wiki/templates/graph_live.js` |
| git | `src/fiona/gittool/git_tool.py` |
| sanity engine | `src/fiona/sanity_runner/sanity_runner.py` |
| runners | `src/fiona/run_tests.py`, `src/fiona/run_sanity.py` |
| launchers | `scripts/run_tests.py`, `scripts/run_sanity.py` |
| bootstrap | `bootstrap`, `bootstrap.ps1`, `scripts/bootstrap_install.py` |
Layout for section 7.1.5, under `src/fiona/wiki/ingest/`:
| Module | Classes | Role |
| --- | --- | --- |
| `models.py` | `FetchedSource`, `ConversionResult`, `IngestResult` | Dataclasses passed between stages. `IngestResult.to_dict()` is the JSON in section 4.7.6. |
| `fetchers.py` | `SourceFetcher` (ABC), `LocalFileFetcher`, `UrlFetcher` | `accepts(argument)` and `fetch(argument, dest_dir)`. `UrlFetcher` takes the opener. |
| `media.py` | `detect_media_type`, `make_slug` | *7.1.5.2.4.* and *7.1.5.2.6.* |
| `converters/base.py` | `SourceConverter` (ABC), `ConverterRegistry`, `ConversionError` | Class attributes `name`, `media_types`, `extensions`, `raw_category`; `convert(source)`. `ConverterRegistry.find(media_type, extension)`. |
| `converters/markdown.py`, `text.py`, `html.py`, `pdf.py` | `MarkdownConverter`, `TextConverter`, `HtmlConverter`, `PdfConverter` | One format each. |
| `archiver.py` | `RawArchiver` | *7.1.5.3.*; `find_by_sha256(digest, size)` scans `raw/` for a reusable copy. |
| `page_writer.py` | `RawPageWriter` | *7.1.5.5.* |
| `log_writer.py` | `IngestLog` | *7.1.5.6.* |
| `pipeline.py` | `IngestPipeline`, `default_pipeline()` | Runs the stages; `list_pending(wiki_root)` is *7.1.5.9.* |
Known defects to avoid copying blindly:
- `json-lint fix` walks the file list twice. The observable contract
is one count and one path line per fixed file.
- The previous spec's graph physics (`mass = 1 + in_degree * 0.5`,
center `0.1`, `alphaMin` `0.1`, `velocityDecay` `0.6`, and
"D3 or vis.js") does not match the template. Section 7.1.4 is the
contract.
- The previous spec's external-link JSON used `{url, count}` objects.
The implementation emits unique URL strings plus a total count.
- pymarkdown rule typos fail silently (section 5.1).
- `mdoctor fix` does not skip read-only zones (section 4.2).