Files
fiona-assistant/SPEC.md
T

138 KiB

name, version, title, created, updated, type
name version title created updated type
Fiona-Assistant Specification 1.1.0 Project Spec Sheet 2026-09-23 2026-10-01 summary

Fiona-Assistant Specification

This document is the build contract for Fiona-Assistant. A reader with this file, and no source tree, must be able to produce the same observable software. Status values are Implemented, Partially Implemented, Planned, and Won't Do.

Revision 0.40.0 reorganizes the sheet into contracts. Revision 0.40.1 numbers nested requirements such as 7.14.1.1. Revision 0.40.4 adds the PYC001 exception in 6.4.2.. Revision 0.41.0 installs one fiona package and keeps tests and repo scripts outside it. Revision 0.90.0 specifies node_importance_min_point in 7.1.4.16. and leaves that requirement unbuilt. Revision 0.92.0 specifies wiki ingest in 7.1.5. and leaves it unbuilt. Revision 0.93.0 adds SHA-256 reuse across raw/, the complete ingest frontmatter map, and the agent metadata check. Revision 0.94.0 makes wiki update yield to the owner's markdown and write <stem>-update.md instead (7.1.2.11.), and adds fiona-assistant housekeeping (7.2.). Revision 0.96.0 compares a file without a version by SHA-256 and shows both short hashes in the update report (7.1.2.12.). Revision 0.97.0 rejects a module-level assignment that follows the first function or class (6.4.3.). Revision 0.98.0 specifies that wiki graph omits a file matched by .gitignore (7.1.4.17.) and leaves that requirement unbuilt. Revision 0.99.0 builds wiki ingest (7.1.5.). Revision 1.0.0 groups each module's functions under that module and moves Non-Goals, Priority Order, and Open Questions to Appendices A, B, and C. Revision 1.1.0 treats a differing fiona-settings.json as a conflict and writes fiona-settings-update.json (7.1.2.3.).

Each feature section starts with a Status line. Requirements use - *N.M.K.*. Each *acceptance* sits under the requirement it tests. NOT_IMPLEMENTED marks work that is specified and not built yet. Class names, file paths, and history live in Appendix D.


1. Product Overview

Fiona-Assistant is a CLI toolkit that manages, organizes, and checks LLM wiki content with deterministic scripts. Reserve LLM tokens for creative and analytical work.

1.1. Document Conventions

Status: Implemented

  • 1.1.1. Main sections use ## N. Title. Subsections use ### N.M. Title. Deeper headings use #### N.M.K. Title.
    • acceptance: Every heading follows that numbering.
  • 1.1.2. A requirement is - *N.M.K.* <text>. A group is numbered the same way, and each item under it adds one segment, for example *7.14.1.* then *7.14.1.1.*. Its acceptance is the next nested - *acceptance*: <text>, and that acceptance describes this requirement only.
    • acceptance: No acceptance describes a neighboring requirement.
    • acceptance: The shared-engine bullets in section 7.14 are numbered 7.14.1 and 7.14.1.1, not plain dashes.
  • 1.1.3. Unbuilt requirements start with `NOT_IMPLEMENTED`.
    • acceptance: Every planned item carries that tag.
  • 1.1.4. The block between <!-- toc-start: headings 2..6 --> and <!-- toc-end --> is generated by librarian toc file. Leave that block to the generator.

1.2. Constraints

These apply to every feature.

  • 1.2.1. Tool reports are parseable and use the shared shapes in sections 4, 6, and 7.
    • acceptance: Two linters that find violations print the same section order from section 6.1.
  • 1.2.2. A command that needs a human decision prompts on a TTY. The same command accepts an explicit flag (--force, --yes, or --non-interactive) and then runs without a prompt.
    • acceptance: wiki update on a non-TTY declines unless --force or --yes is set.
  • 1.2.3. personal/ and raw/ are read-only for every automated tool. Tools may resolve links that point into those zones. The one exception is wiki ingest (7.1.5.3.): it may add a new file under raw/. It never modifies, renames, or deletes a file already there.
    • acceptance: wiki update leaves every file under personal/ and raw/ byte-for-byte unchanged.
    • acceptance: A second wiki ingest run leaves every file that was already under raw/ byte-for-byte unchanged.
    • acceptance: Ingesting bytes whose SHA-256 matches any file already under raw/, from any path or URL, adds no file under raw/ (7.1.5.3.3.).
  • 1.2.4. Code changes follow test-first development. python scripts/run_tests.py is green, then python scripts/run_sanity.py is green, before a change is finished.
    • acceptance: A sanity run below the coverage gate in section 5.2 fails even when every test passes.
  • 1.2.5. Python passes Ruff. Markdown passes PyMarkdown with the rules in section 5.1.
    • acceptance: An unused import is reported by python-ruff chklint with file, line, column, and rule code.
  • 1.2.6. Authored lengths in HTML and CSS use em or another relative unit (rem, %, vh, vw). They are not written in px. The wiki graph page in section 7.1.4 is that web surface. A position read from the layout engine is written back in rem. A theme length number uses the same scale: 16 is 1em.
    • acceptance: The graph template contains no px length.

2. Package and Runtime

Status: Implemented

Python >= 3.11 (also 3.12 and 3.13). The default development install from the repository root is pip install -e ".[dev]". bootstrap --local installs the same tree as a normal package (pip install .). Section 7.21 defines those modes. The build requires setuptools 61 or newer.

Runtime dependencies:

Package Minimum
pymarkdownlnt 0.9.0
ruff 0.16.8
pyyaml 6.0.3
GitPython 3.1.0

Development extras: pytest>=7.0, pytest-cov>=4.0, pytest-mock, pylint. html5validator is not a dependency until section 7.15 is implemented. beautifulsoup4, markdownify, and pypdf become runtime dependencies when section 7.1.5 is implemented.

One importable package, fiona, lives under src/fiona/. Its subpackages are librarian, mdoctor, customlinter, python_ruff, gittool, sanity_runner, and wiki. The wiki template is package data at src/fiona/data/llm_wiki/ and is loaded with importlib.resources. It is not an importable module. Tests live in repo-root tests/ and are not installed. scripts/ holds settings_dev.json, shell helpers, and the launchers scripts/run_tests.py and scripts/run_sanity.py.

Console scripts:

Script Entry
fiona-assistant fiona.__main__:main
librarian fiona.librarian.librarian:main
mdoctor fiona.mdoctor.mdoctor:main

python-ruff, fmlint, json-lint, and git are reached only through fiona-assistant. html-js-lint is not installed until section 7.15 is implemented.

Coverage measures src/fiona, omitting tests. The config lives in pyproject.toml. The gate percentage is section 5.2.

Logging: every CLI entry calls setup_logging() before work. Level order is LOG_LEVEL environment variable, then fiona.log_level in fiona-settings.json, then WARNING. Records go to stderr. When fiona.file_log_location is set, a file handler writes lines %(asctime)s %(levelname)s %(name)s: %(message)s at or above fiona.file_log_level (fallback: fiona.log_level). A relative path is resolved from the current working directory. An unopenable path logs a warning and keeps stderr logging. Human reports stay on stdout. Package loggers set their level explicitly because pymarkdown resets the root logger mid-run.

  • 2.1. The three console scripts above are installed by pip install -e ..
    • acceptance: fiona-assistant with no arguments prints help and exits 0.
  • 2.2. html-js-lint is not an installed script until section 7.15 exists.
    • acceptance: A fresh install does not fail at import because of a missing html_js_lint package.

3. llm-wiki Template

Status: Implemented

wiki init copies this tree from the packaged template. Text files replace every DATE_PLACEHOLDER with the current UTC date YYYY-MM-DD. Binary files are copied unchanged. A glob in fiona.llm-wiki.ignore_files skips that file, or that directory and its subtree. A missing settings file or a missing key copies everything. A non-list ignore_files value raises ValueError with the text fiona.llm-wiki.ignore_files must be a list of strings.

AGENTS.md
README.md
fiona-settings.json
table_of_contents.md
ingest/README.md
personal/README.md
raw/README.md
wiki/schema.md
wiki/index.md
wiki/log.md
utils/README.md
utils/scripts/example.py
utils/scripts/map.md
utils/skills/README.md
utils/skills/fiona/SKILL.md
utils/skills/fiona/markdown_rules.md
utils/skills/fiona/llm-wiki-mod.md
utils/skills/python-tdd/SKILL.md
utils/personas/default/SOUL.md
utils/personas/template/SOUL.md
utils/skills_local/README.md

The shipped fiona-settings.json is the default document in section 5.1. wiki/schema.md is the page-type and tag taxonomy. This spec's frontmatter table in section 6.2 is the field contract; where the two disagree, this spec wins.

3.1. What Counts as a Wiki

A directory is an llm-wiki when it contains fiona-settings.json. wiki init refuses when any of these already exist: AGENTS.md, fiona-settings.json, wiki/, ingest/, README.md.

3.2. Core Files and Read-only Zones

Core files, matched by name at the wiki root:

  • AGENTS.md
  • fiona-settings.json
  • README.md
  • table_of_contents.md

Core directories (every file under them is core):

  • wiki/
  • ingest/
  • personal/
  • raw/
  • utils/

Read-only zones, including the zone directory itself:

  • personal/
  • raw/

wiki/ is copied by wiki init and then left to the LLM. wiki update does not add, replace, or delete any path under wiki/. The only file it writes there is an update file for a template page that is newer than the wiki's copy (7.1.2.11.).

wiki update adds missing core files and directories. It never overwrites an existing markdown file: when the template carries a newer version, it writes <stem>-update.md beside the file and leaves the owner's file as it is. A differing fiona-settings.json is the same kind of conflict: the owner's file stays, and the template is written as fiona-settings-update.json (7.1.2.3.). It never deletes user files. It never writes inside a read-only zone, so a missing file under personal/ or raw/ is not added back. It never writes a path matched by the repository .gitignore or by the wiki's own .gitignore.

After the copy, wiki init creates a git repository and writes .gitignore. A new file starts with the fiona-assistant repository .gitignore (the same patterns, including __pycache__/, *.pyc, and *fiona.log). When this package is not the source checkout, that text is the packaged default.gitignore. The file adds personal/ when that pattern is missing. An existing .gitignore keeps every line it already has; missing patterns from that set are appended. Init then stages everything and commits Initial llm-wiki commit. Template paths that match the repository .gitignore are not copied. personal/ and raw/ are still copied. personal/ stays untracked. If git init fails, the copy still counts as success and stdout adds Note: Git repository initialization failed: <error>.

  • 3.2.1. An empty directory becomes the tree above, with dates substituted, and a git commit that does not track personal/.
    • acceptance: personal/ is absent from git ls-files after wiki init.
  • 3.2.2. A second wiki init raises FileExistsError whose message names each conflicting indicator, and writes nothing.
    • acceptance: An existing README.md alone is enough to refuse, and that file's bytes are unchanged.

3.3. Deployed Wiki Rules

Status: Implemented

Only the template tree is deployed. A wiki does not know this spec, the root AGENTS.md, scripts/, or tests/. The template AGENTS.md is the entry point: it says what the wiki is and loads the Fiona skill, which loads llm-wiki-mod.md. The product rule "never commit without permission" (project agent rule 2.4) covers this repository only.

  • 3.3.1. No markdown file in the template references a file or rule outside the template: SPEC, project agent rule, or scripts/run_.
    • acceptance: A scan of every template *.md finds none of those strings.
  • 3.3.2. The template AGENTS.md links the Fiona skill and llm-wiki-mod.md.
    • acceptance: Both relative links appear in that file.
  • 3.3.3. A deployed wiki commits with fiona-assistant git save --description TEXT after each ingest cycle and each housekeeping pass. clean, undo, and restore run only when the user asks.
    • acceptance: The template AGENTS.md and the Fiona skill both contain git save, and neither says to commit only when asked.
  • 3.3.4. The template wiki/log.md holds one seed entry dated DATE_PLACEHOLDER.
    • acceptance: The log body has exactly one bullet.

4. Command Catalog

Status: Implemented

One command tree. Optional [directory] defaults to . today. Wiki-root discovery (section 7.20) is not built. An invalid directory prints Error: '<path>' is not a valid directory. on stderr and exits 1, unless a command below states a different message.

fiona-assistant with no command prints help and exits 0. fiona-assistant git with no subcommand prints git help and exits 1. fiona-assistant wiki with no subcommand prints wiki help and exits 1.

4.1. librarian

fiona-assistant librarian and librarian with no subcommand list the current directory.

list [directory] stdout:

Found <N> files in '<directory>':
  - <path>

mdoctor list uses the same listing with the prefix MDoctor: Found <N> files in '<directory>':.

4.1.1. librarian toc

toc [directory] [flags] writes table_of_contents.md in the target folder. Flags: --recursive-toc, --non-recursive, and repeatable --exclude REGEX. Depth comes from section 5.1.

A path that is a file or missing exits 1 on stderr and names librarian toc file <path>. A folder with no markdown prints no markdown files found on stdout and writes nothing. In --recursive-toc mode that message is a log line at INFO instead.

toc file <path> writes an in-file TOC in that markdown file. toc file * does that for every markdown file under the wiki root found by section 7.3.1. --sectionized forces section numbering for that run. Status lines:

wrote toc in <path>
toc already up to date in <path>
skipped <path> (<reason>)
generated toc for <N> file(s), skipped <M>

The summary line is bulk mode only. A file with no H1 is skipped <path> (no H1 heading) and is not rewritten.

4.2. mdoctor

No subcommand means chklint ..

chklint [directory] prints the shared report (section 6.1) with header PyMarkdownlint Violations and exits 0 even when violations exist.

fix [directory] runs pymarkdown auto-fix in place. Stdout:

PyMarkdownFix: <N> file(s) fixed
  - <path>

A pymarkdown API failure prints Error: pymarkdown fix failed: <error> on stderr and exits 1. The command does not skip personal/ or raw/.

4.3. fmlint

fiona-assistant fmlint with no subcommand means chklint .. chklint [directory] prints the shared report with header Frontmatter Lint Violations and exits 0 even when violations exist. A missing directory exits 1.

4.4. python-ruff

No subcommand means chklint .. The report header is Ruff Violations. Exit code is 0 when the ruff process itself succeeds, including when violations are listed. lint(path, fix=True) passes --fix to the ruff binary. There is no fix subcommand yet.

4.5. json-lint

No subcommand means chklint .. Header is JSONLint Violations.

chklint [directory] exits 1 when the report contains violations and exits 0 when it reports no violations. This differs from mdoctor, fmlint, and python-ruff, which exit 0 either way.

chklint --fix and fix [directory] rewrite indentation and trailing commas in place, skip personal/ and raw/, and print:

JSONLintFix: <N> file(s) fixed
  - <path>

A missing directory on fix prints Error: '<path>' does not exist. on stderr and exits 1.

4.6. git

Operates on the git repository at the current directory. Timestamps are UTC.

Command Stdout on success
save --description TEXT Committed: <full sha>
snapshot <tag_name> --description TEXT Snapshot created: commit=<sha>, tag=<tag>
restore <tag_name> Restored to: <sha>
clean Working tree cleaned. All changes discarded.
undo Undo to: <sha>
status Status: clean or Status: dirty, then git status

On failure the tool prints Error: <message> and exits 1.

Commit message: <YYYY-MM-DD HH:MM:SS> <description>. save runs git add --all and commits. A clean tree, including no untracked files, raises Working tree is clean. Nothing to commit.

snapshot commits, then creates an annotated tag <YYYY-MM-DD_HH-MM-SS>_<tag_name> with message Snapshot: <description>.

restore requires a clean tree (Working tree is dirty. Use 'clean' to discard current edits first.). It accepts the full tag or the user tag suffix. The latest tag whose name equals the argument or ends with _<argument> wins. It checks that commit's tree into the index and working tree and adds a new commit <timestamp> Restore to <full tag>. History before that commit stays.

clean requires a dirty tree (Working tree is already clean. Nothing to discard.), then git reset --hard HEAD and git clean -fd.

undo requires a clean tree and at least two commits (Cannot undo: only one commit exists in the repository.), then git reset --hard HEAD~1.

4.7. wiki

Subcommands of fiona-assistant wiki.

4.7.1. wiki init

fiona-assistant wiki init [directory]. Success stdout:

Successfully initialized llm-wiki directory at: <resolved path>

Rules are section 3.

4.7.2. wiki check-update

fiona-assistant wiki check-update [directory] [--json]. The directory must contain fiona-settings.json, else stderr Error: Directory '<path>' is not an llm-wiki directory and exit 1. A missing directory is Error: Directory not found: <path> and exit 1. A successful check exits 0 whether or not updates exist.

Human stdout when current:

Wiki is up to date.

When something differs, the lines that apply:

Files to update: <N>
  - <relative path> (<current> -> <template>)
Conflicts (owner copy kept): <N>
  - <relative path> (<current> -> <template>) -> <update file name>
Directories to add: <N>
  - <relative path>
Files to add: <N>
  - <relative path>

Total: <N> file(s) need attention
Files skipped (locally modified): <N>
  - <relative path>

The version parenthesis appears only when both versions exist. When the file was compared by hash instead (7.1.2.12.), the parenthesis is (sha256 <current> -> <template>), each side the first 8 lowercase hex digits of that SHA-256. The Conflicts block appears only when a conflict exists (7.1.2.11.). Total counts updates, conflicts, and additions. --json prints one JSON object:

{
  "files_to_update": [
    {
      "path": "<path>",
      "current_version": "<version or null>",
      "template_version": "<version or null>",
      "current_sha256": "<64 hex or null>",
      "template_sha256": "<64 hex or null>",
      "is_settings_file": false
    }
  ],
  "conflicts": [
    {
      "path": "<path>",
      "update_path": "<path>",
      "current_version": "<version or null>",
      "template_version": "<version or null>",
      "current_sha256": "<64 hex or null>",
      "template_sha256": "<64 hex or null>"
    }
  ],
  "files_to_add": [],
  "directories_to_add": [],
  "files_skipped": [
    {"path": "<path>", "reason": "locally modified"}
  ],
  "total": 0,
  "has_updates": false
}

4.7.3. wiki update

fiona-assistant wiki update [directory] [-f|--force] [-y|--yes] [--dry-run] [--json].

Without --force and --yes, stdout shows the check-update text and the prompt Proceed with update? [y/N] followed by a space. y and yes proceed. Any other answer, and a non-TTY, exit 1 with stderr Error: Update requires confirmation. Use --force or --yes to proceed without prompting.

--dry-run prints Dry run - no files will be modified: and the check-update text, writes nothing, and exits 0.

Success stdout:

Update complete: <U> file(s) updated, <A> added, <C> conflict(s), <S> skipped.

When <C> is not zero, one more line follows:

Merge each <stem>-update<suffix> into its file: run fiona-assistant housekeeping.

--json suppresses the plan and prints:

{
  "files_updated": [],
  "files_added": [],
  "directories_added": [],
  "conflicts": [{"path": "<path>", "update_path": "<path>"}],
  "files_skipped": [],
  "total": 0
}

Comparison and merge rules are section 7.1.2.

4.7.4. wiki sanity

fiona-assistant wiki sanity [directory] [--json] [--non-interactive]. Requires fiona-settings.json. Exit 0 when every check is clean, exit 1 otherwise.

Checks, in this order, one thread each: markdownlint, frontmatter_lint, python-ruff, mdlinter, jsonlint.

Human report:

Wiki Sanity Report
TARGET: <path>

<failing check reports, omitted when clean>

Results:
<name padded>: PASS - <N> violations (MM:SS)

SANITY PASSED (MM:SS)

A failure ends with Sanity Failed - <N> checks need attention: and one detail line per failing check. The pass banner carries the total wall time. The failure verdict does not. Per-check times may overlap and need not sum to the total. Rows stay in registry order.

--json skips the progress display and prints:

{
  "target": "<path>",
  "checks": {
    "markdownlint": {
      "result": 0,
      "violation_count": 0,
      "duration_seconds": 0.0,
      "violations": []
    }
  },
  "total_violations": 0,
  "passed": true
}

result is 0 for pass and 1 for fail. Progress display is section 7.14.

4.7.5. wiki graph

fiona-assistant wiki graph [directory] [-o FILE] [--format html|json] [--include-pattern GLOB] [--exclude-pattern GLOB] [--wikilinks|--no-wikilinks] [--json] [--external-links|--no-external-links].

Defaults: output wiki_graph.html in the current directory, format html, include **/*.md, wikilinks on, external-link nodes off. --json selects JSON format. Behavior is section 7.1.4.

4.7.6. wiki ingest

The contract is section 7.1.5.

fiona-assistant wiki ingest <source> [--wiki DIR] [-r|--raw PATH]
    [-t|--target PATH] [--raw-only] [-f|--force] [--dry-run] [--json]
fiona-assistant wiki ingest --pending [--wiki DIR] [--json]

<source> is an http:// or https:// URL, or a local file path. --wiki is the wiki root and defaults to .. It must contain fiona-settings.json, else stderr Error: Directory '<path>' is not an llm-wiki directory and exit 1. <source> and --pending are mutually exclusive, and one of them is required. argparse reports a violation and exits 2.

Human stdout on success, the lines that apply, in this order:

Fetched: <url> (<media type>, <N> bytes)
Archived: <raw path>
Archived: <raw path> (already archived, sha256 <first 12 hex>)
Duplicate: <page path> already cites this source.
Wrote: <page path> (converter <name>, conversion <complete|partial>)
Warning: <text>
Next: refine <page path>, then set status: refined.

Fetched: appears only for a URL. One Warning: line per warning. With no converter, or a failed conversion, the raw copy stays and stdout ends with:

No converter for <media type>. Convert <raw path> by hand.

--pending stdout:

Raw pages: <N>
  - <page path> (<first sources entry>)

--json prints one object and nothing else on stdout. Paths are relative to the wiki root and use /:

{
  "source": "<argument as given>",
  "origin": "<url, or the resolved file path>",
  "origin_kind": "url",
  "media_type": "application/pdf",
  "raw_path": "raw/papers/<slug>.pdf",
  "raw_reused": false,
  "sha256": "<64 lowercase hex>",
  "duplicate_page": null,
  "page_path": "wiki/sources/<slug>.md",
  "converter": "pdf",
  "conversion": "complete",
  "status": "raw",
  "warnings": [],
  "dry_run": false
}

origin_kind is url or file. With no page, page_path and converter are null and conversion is none. duplicate_page is the page that already cites the matching raw copy, else null (7.1.5.3.5.). --pending --json prints {"pending": [{"page_path": ..., "sources": [...], "origin": ...}], "total": <N>}.

Exit Meaning
0 Raw copy archived. A page was written, --raw-only was set, or a page already cites the same bytes.
1 Error on stderr. Nothing written under the wiki.
2 argparse usage error.
3 Raw copy archived. No page: no converter, or conversion failed.

Errors, each on stderr with exit 1:

Error: Source not found: <path>
Error: Source is a directory: <path>
Error: Unsupported URL scheme: <scheme>
Error: Download failed: <url>: <reason>
Error: Download exceeds <N> bytes: <url>
Error: --raw must stay inside raw/: <value>
Error: --raw cannot be used on a file already under raw/: <path>
Error: --target must stay inside wiki/: <value>
Error: --target cannot replace <path>
Error: Page exists: <path>. Use --force to replace it.

A failed conversion prints Error: Conversion failed: <converter>: <reason> on stderr and exits 3, not 1, because the raw copy stays.

4.8. housekeeping

fiona-assistant housekeeping [directory] [--json]. One call runs the wiki maintenance routine and tells an LLM what to fix. The directory must contain fiona-settings.json, else stderr Error: Directory '<path>' is not an llm-wiki directory and exit 1. A missing directory is Error: Directory not found: <path> and exit 1. The steps and their rules are section 7.2.

Human stdout:

Housekeeping Report
TARGET: <path>

[<STATUS>] <step>: <summary>
  - <detail>

Next steps:
  - <action>

HOUSEKEEPING PASSED

One [<STATUS>] block per step, in step order. <STATUS> is PASS, WARN, or FAIL. Detail lines appear only when the step has details. Next steps: appears only when a step has an action. The last line is HOUSEKEEPING PASSED, or HOUSEKEEPING FAILED - <N> step(s) need attention: <names> with the failing step names joined by a comma and a space.

--json prints one object and nothing else on stdout. Paths are relative to the wiki root and use /:

{
  "target": "<path>",
  "steps": [
    {
      "name": "merge",
      "status": "FAIL",
      "summary": "1 update file(s) wait for a merge",
      "details": ["AGENTS-update.md -> AGENTS.md"],
      "next": ["Merge AGENTS-update.md into AGENTS.md ..."]
    }
  ],
  "passed": false
}
Exit Meaning
0 No step is FAIL. WARN steps may exist.
1 A step is FAIL, or the directory is not a wiki.

4.9. Repository runners

python scripts/run_tests.py runs pytest with --color=no and a temporary JUnit XML report (junit_family=xunit1). The child process receives the repository src directory on PYTHONPATH, joined with os.pathsep.

python scripts/run_sanity.py [--non-interactive] runs, in order: pytests, markdownlint, ruff, frontmatter_lint, pycustomlint, jsonlint. Same progress display, result table, and verdict shape as wiki sanity, with label Running Sanity and an extra untimed coverage: row under pytests. Details are section 7.14.

5. Settings Schema

Two files. fiona-settings.json ships in the template and marks a wiki root. scripts/settings_dev.json is repository tooling only and is not copied into a wiki.

5.1. fiona-settings.json

Absent file: pymarkdown uses its own defaults; other tools use the defaults in the table. A typo in a pymarkdown rule id or property name is accepted and has no effect. Confirm a new pymarkdown setting by its effect on a scan. TOC file settings reject unknown keys and bad types with exit 1 and the key name.

Shipped document (logical content; wiki init copies the file):

{
  "markdown": {
    "pymarkdown": {
      "extensions": ["front-matter"],
      "respect_gitignore": true,
      "md025": {"front_matter_title": "no-match"},
      "md013": {"line_length": 100, "code_blocks": false, "tables": false},
      "md060": {"enabled": false}
    },
    "frontmatter": {
      "ignore_rules": [],
      "ignore_file": [],
      "respect_gitignore": true
    },
    "toc_generator": {
      "max_depth_from_file": 3,
      "max_depth_from_dir": null,
      "file": {
        "min_heading": 2,
        "max_heading": 6,
        "sectionize": false,
        "ignore_files": [],
        "*/rules.md": {"max_heading": 2}
      }
    }
  },
  "json": {
    "indent": "space",
    "indent_len": 2,
    "trailing_commas": false,
    "require_key_order": false,
    "check_duplicate_keys": true,
    "extensions": [".json"]
  },
  "python": {"ruff": {}},
  "fiona": {
    "log_level": "INFO",
    "file_log_level": "INFO",
    "file_log_location": "fiona.log",
    "llm-wiki": {"ignore_files": []},
    "wiki": {
      "graph": {
        "show_external_links": false,
        "subtitle": {"enabled": true},
        "show_legend": true,
        "theme": "night-sky",
        "show_arrows": false,
        "show_source": true,
        "show_glow": true,
        "show_link_card": true,
        "first_fit_ms": 12000,
        "hover_link_tail_chars": 12,
        "simplify_external_links": false,
        "consolidate_external_links": false,
        "consolidate_external_links_depth": 0,
        "consolidate_external_links_max_char": 8
      }
    }
  }
}
Key Type Default when absent Consumer
markdown.pymarkdown.extensions list of strings none enabled mdoctor
markdown.pymarkdown.respect_gitignore bool false mdoctor
markdown.pymarkdown.<rule>.<prop> bool, int, or str pymarkdown default mdoctor
markdown.frontmatter.required_fields list of strings ["name", "version"] fmlint
markdown.frontmatter.type_requirements object of lists {} fmlint
markdown.frontmatter.enabled bool true fmlint
markdown.frontmatter.ignore_rules list of codes [] fmlint
markdown.frontmatter.ignore_file list of globs [] fmlint
markdown.frontmatter.respect_gitignore bool true fmlint
markdown.toc_generator.max_depth_from_file int unlimited (0 or absent) folder TOC
markdown.toc_generator.max_depth_from_dir int or null unlimited folder TOC
markdown.toc_generator.file.min_heading int >= 2 2 in-file TOC
markdown.toc_generator.file.max_heading int >= min 6 in-file TOC
markdown.toc_generator.file.sectionize bool false in-file TOC
markdown.toc_generator.file.ignore_files list of globs [] in-file TOC
markdown.toc_generator.file.<glob> object of file keys none per-file override
json.indent "space" or "tab" "space" json-lint
json.indent_len int 2 json-lint
json.trailing_commas bool false json-lint
json.require_key_order bool false json-lint
json.check_duplicate_keys bool true json-lint
json.extensions list of strings [".json"] json-lint
json.max_line_length int or absent no line-length check json-lint
python.ruff object {} reserved, unused
fiona.log_level level name WARNING logging
fiona.file_log_level level name fiona.log_level file log
fiona.file_log_location string or absent no file log file log
fiona.llm-wiki.ignore_files list of globs copy all wiki init, update
fiona.wiki.graph.wikilinks_enabled bool true wiki graph
fiona.wiki.graph.max_files int 5000 wiki graph
fiona.wiki.graph.show_external_links bool false wiki graph toggle
fiona.wiki.graph.subtitle.enabled bool true wiki graph toggle
fiona.wiki.graph.show_legend bool true wiki graph toggle
fiona.wiki.graph.show_arrows bool true wiki graph toggle
fiona.wiki.graph.show_source bool true wiki graph toggle
fiona.wiki.graph.show_glow bool true wiki graph
fiona.wiki.graph.show_link_card bool true wiki graph
fiona.wiki.graph.first_fit_ms number 12000 wiki graph
fiona.wiki.graph.hover_link_tail_chars int 12 wiki graph
fiona.wiki.graph.theme string night-sky wiki graph
fiona.wiki.graph.simplify_external_links bool false wiki graph
fiona.wiki.graph.consolidate_external_links bool false wiki graph
fiona.wiki.graph.consolidate_external_links_depth int follow the bool wiki graph
fiona.wiki.graph.consolidate_external_links_max_char int 8 wiki graph
fiona.wiki.graph.node_importance_min_point int >= 0 0 wiki graph
fiona.wiki.ingest.timeout_seconds number > 0 30 wiki ingest
fiona.wiki.ingest.max_download_bytes int > 0 52428800 wiki ingest
fiona.wiki.ingest.user_agent string fiona-assistant/<package version> wiki ingest
fiona.wiki.ingest.default_target string sources wiki ingest
fiona.wiki.ingest.raw_categories object of strings {} wiki ingest

Bool pymarkdown properties use set_boolean_property, ints use set_integer_property, strings use set_string_property, applied as plugins.<rule_id>.<property>. "enabled": false inside a rule map disables that rule.

markdown-tables stays off. pymarkdownlnt 0.9.40 raises BadTokenizationError on list lines that contain pipes, which breaks sanity on generated contents files. Until that is fixed, table rules MD055, MD056, MD058, and MD060 do not run. MD060 is also disabled because pymarkdown's delimiter-width check disagrees with Node markdownlint compact tables. Project tables use compact style: one space around cell text, including the delimiter row (| --- | ------- |).

Graph colors, sizes, and shapes come from section 5.3. If node_colors, node_radius, or hover is still present here, those keys override the colorful theme only. Hover radius is min(current * size_multiplier, max) and returns to the pre-hover radius when the pointer leaves. subtitle.enabled starts the Show Subtitles toggle. The label is the frontmatter name when that field is a non-empty string, otherwise the filename. show_legend starts the Show Legend toggle. The default is on. The card sits at the upper right of the graph. It groups shape, node size and color, and edge color, so those three are not one mixed list. The graph command starts from the graph section of the packaged fiona-settings.json. A fiona-settings.json found by walking up from the scanned directory replaces the keys it sets. A missing file still uses that packaged section. show_arrows starts the Show Arrows toggle. The default is on. show_source starts the Show Sources toggle. The default is on. It draws each frontmatter source value as its own node. show_glow draws the glow on a hovered node, a clicked node, and a search match. The default is on. show_link_card draws the hover card beside the node. The default is on. A click shows the frontmatter card at the upper left, including when show_link_card is false. first_fit_ms is the first node's fit, in milliseconds. The shipped value, and the value when the key is absent or not a positive number, is 12000. See all, the last node, and a theme change stay at 4200 ms. hover_link_tail_chars is how many characters of a URL's last path segment the hover card keeps. The shipped value, and the value when the key is absent or not a positive number, is 12. The arrowhead sits on the target end of a visible edge, so the source file points at the target file. Turning the toggle off removes the arrowheads. theme names the theme the page opens on. The shipped value is night-sky. An unknown name also opens night-sky. simplify_external_links shortens an external subtitle to the host. The default is off. consolidate_external_links draws one node per host, named host/*, when depth is absent. The default of the bool is off. consolidate_external_links_depth is how many levels to keep. 0 leaves each URL alone. 1 is host/*. 2 is host/a/*. 3 is host/a/b/*. A missing depth follows the bool: true is 1, false is 0. A path part longer than consolidate_external_links_max_char is .... That limit is 8 when absent or not a positive integer. Turning depth above 0 also shortens the subtitle to the host. node_importance_min_point is how many different files must point at a node before the HTML graph draws it. The count is unique_points (7.1.4.2.3.). 0 turns the filter off and every node is drawn. A missing key, or a value that is not an integer of 0 or more, is 0. A positive integer is 7.1.4.16. and is not built yet. The packaged fiona-settings.json omits the key until that requirement is built. Absent means 0.

fiona.wiki.ingest is read by section 7.1.5. default_target is a directory relative to wiki/. raw_categories maps a converter name to a directory under raw/, such as {"text": "transcripts"}. A value that is absent, of the wrong type, or out of range uses the default in the table. The packaged fiona-settings.json omits fiona.wiki.ingest. Absent keys use the defaults in the table.

A mapping key under toc_generator.file whose name is not one of min_heading, max_heading, sectionize, ignore_files is an fnmatch override against the file path relative to the wiki root.

On wiki update, a differing fiona-settings.json is not rewritten (7.1.2.3.). The template is written beside it as fiona-settings-update.json. The owner merges that file: start from the template object. For each user key, if both sides are objects, merge recursively; otherwise the user value replaces the template value. New template keys that the user does not set are kept. Write the result as JSON with indent 2 and a trailing newline, then delete the update file.

  • 5.1.1. Changing md013.line_length changes the next mdoctor scan with no code change.
    • acceptance: A line between the old and new limit is clean after the settings change and reported before it.
  • 5.1.2. wiki update leaves fiona-settings.json byte-for-byte unchanged and writes the template as fiona-settings-update.json.
    • acceptance: markdown.pymarkdown.md013.line_length set to 120 is still 120 in the owner file after wiki update --yes, and fiona-settings-update.json is the template copy.

5.2. settings_dev.json

{
  "pytests": {"required-coverage": 96},
  "progress": {"interval": 3, "expected-seconds": 30}
}
Key Default when absent Meaning
pytests.required-coverage (required for sanity) Minimum total coverage percent
progress.interval 3 Seconds between non-interactive dots
progress.expected-seconds 30 Bar scale for a running check
  • 5.2.1. Coverage below required-coverage fails sanity when every test passed.
    • acceptance: Raising the number in this file flips a green run to failed with no code change.

5.3. constellation-settings.json

Themes for the wiki graph. The file has a themes object. colorful is complete. Any other theme may omit keys. A missing key is taken from colorful. A dark object on a theme is the same kind of partial theme, used when the page is in dark mode.

colorful is a warm paper field. Nodes run from warm gray through amber to crimson. Edges use that same warm ramp, darker so a thin line stays visible. Hover, click, and buttons are one teal, the complement of the amber. Pages are circles and external links are clouds. night-sky draws circle pages and star external links on a dark field. grayscale runs charcoal to near-black on a light field, with a dark stroke so the pale end still stands off the canvas. Its dark object is a gray night on a charcoal field, not black. Sparse nodes stay medium gray. A mid link count is white-yellow. Many incoming links are a light yellow-orange. dark is a blue-slate night. Nodes run from slate through white-yellow to a light gold. The page and the canvas stay blue-gray, not black. professional is a light boardroom palette: pale blue nodes, a soft blue stroke, light blue buttons, and a quiet radius.

Key Meaning
background Page background
graph_background Graph canvas background
font_family Font for the page
text_color Text on the page and cards
node_colors.low/mid/high Node color from few to many incoming links
node_stroke Outline around a node, or none
node_radius.min/max Smallest and largest node radius, in em. 16 is 1em. Fractions are kept
subtitle.min_radius Label cutoff, a fraction of node_radius.max from 0 to 1
subtitle.padding Em between the shape and the label, where 16 is 1em. Smaller sits closer
subtitle.size Label font size in em, where 16 is 1em
subtitle.on_top When true, labels paint above every node
subtitle.outline When true, a stroke keeps the label readable over nodes
subtitle.outline_color Stroke color for that outline
subtitle.outline_width Stroke width in em, where 16 is 1em
edge_colors.low/mid/high Edge color from one link to many
edge_width Edge thickness in em, where 16 is 1em. 1 when absent or not positive
highlight.node_color Node color while the pointer is over it
highlight.edge_color Edge color while a node is highlighted
highlight.size_multiplier Radius multiplier while highlighted
clicked.node_color Node color after a click
buttons.background/text/hover/disabled Button colors
card.background/border Cards, legend, and the control bar
shapes.page circle, cloud, or star
shapes.external Shape for an external-link node
subtitle_color Label under a large node

subtitle.padding is 12 when absent. subtitle.size is 11 when absent or below 1. subtitle.on_top and subtitle.outline are false when absent. subtitle.outline_width is 3 when absent. subtitle.outline_color uses graph_background when absent. A theme that sets only some of these keeps the rest from colorful.

  • 5.3.1. A theme that omits font_family uses the colorful theme font. The HTML theme control lists colorful, night-sky, grayscale, dark, and professional.
    • acceptance: The resolved grayscale theme font is the colorful theme font. The graph HTML contains a theme select and those names.
  • 5.3.2. Subtitle gap, size, stacking, and outline come from the active theme's subtitle object.
    • acceptance: The resolved grayscale subtitle padding, size, on_top, and outline match colorful. The graph HTML reads those keys from the theme.

6. Violation Rule Catalog

6.1. Shared Report

Status: Implemented

Custom linters and the ruff wrapper print this shape. Sanity runners omit the report for a check with zero violations; the Results: row still shows the count.

<Linter Name> Violations
TARGET: <path>

- DEFINITIONS:
  - <code>: <rule_name>, <description>

- FILE: <path>
  - <message>

- SUMMARY:
  - Total files with violations: <N>
  - Total violations: <N>

A clean standalone run that still prints a report uses - No violations found. in place of the FILE section and still prints DEFINITIONS and SUMMARY. Headers:

Tool Header name
mdoctor PyMarkdownlint
fmlint Frontmatter Lint
python-ruff Ruff
pycustomlint Python Custom Lint
mdlinter Markdown Lint
json-lint JSONLint

Frontmatter messages omit a line:column prefix. Frontmatter is always at the start of the file.

  • 6.1.1. A report with one violation contains TARGET, DEFINITIONS, one FILE entry, and SUMMARY, in that order.
    • acceptance: The three custom linters and ruff use this order.

6.2. Frontmatter Fields and FML Codes

Status: Implemented

Required on every markdown file: name, version. Required on every page under wiki/: confidence, llm, contested, and contradictions. The Fiona skill teaches how to set those four. fmlint enforces name and version only. All other fields are optional.

Field Required Rule
name yes Document name. Wiki pages use lowercase and hyphens.
version yes Semver. Bump when the page is edited.
title no Human title.
created no YYYY-MM-DD
updated no YYYY-MM-DD. Bump when the page is edited.
type no entity, concept, comparison, query, summary, toc
tags no Taxonomy tags from wiki/schema.md
sources no Paths such as raw/articles/source-name.md
confidence wiki pages trusted, high, medium, low, or zero. The agent sets it on every page under wiki/. fmlint does not enforce it.
llm wiki pages model is required. reasoning is set when the model reports a level. The agent sets it on every page under wiki/. fmlint does not enforce it.
contested wiki pages true when a claim on the page is disputed, otherwise false. The agent sets it on every page under wiki/. fmlint does not enforce it.
contradictions wiki pages Other page slugs that disagree. [] when none. The agent sets it on every page under wiki/. fmlint does not enforce it.
description SKILL.md only Required on a file whose name is SKILL.md.
status no raw or refined. wiki ingest writes raw. Only the agent sets refined (7.1.5.10.).
ingest no Map of provenance written by wiki ingest: origin, media type, SHA-256, size, converter, conversion, warnings (7.1.5.5.3.).
source_meta no Map of metadata read from the source, such as author and dates. {} when none.

markdown.frontmatter.type_requirements maps a type value to the field list required for that type, in addition to the global list.

Code Name Message Fails when Passes when
FML001 Frontmatter Parse Error FML001 - <error> YAML between --- markers is invalid YAML parses
FML002 Missing Frontmatter FML002 No frontmatter block A block is present
FML003 Missing Name Field FML003 name absent name present
FML004 Missing Description FML004 SKILL.md has no description description present
FML005 Invalid Skill Name FML005 SKILL.md name fails ^[a-z0-9-]+$ name: python-tdd
FML006 Skill Name Mismatch FML006 - (expected: <folder>, got: <name>) name differs from the parent folder folder python-tdd and name: python-tdd
FML007 Missing Version Field FML007 version absent version present

SKILL.md is case-sensitive. FML005 and FML006 apply only to that filename. Scan skips .gitignore matches when respect_gitignore is true, ignore_file globs, and read-only zones.

  • 6.2.1. A markdown file with name and version and no title produces no FML violation.
    • acceptance: fmlint on that file reports no violations.
  • 6.2.2. SKILL.md in python-tdd/ with name: Python TDD is FML005, and name: other-name is FML006.
    • acceptance: The report code matches the case above.
  • 6.2.3. Every page under wiki/ includes confidence, llm, contested, and contradictions. The Fiona skill requires them and states how to choose each value. confidence is trusted (the user stated it, or it is copied from a cited source), high (more than one source agrees, or one source and no known contradiction), medium (one source, or a synthesis the agent is reasonably sure of), low (inference, thin evidence, or an open gap), or zero (unknown, or not checked). llm is a list whose model entry is required: the model that last edited the page, or none when no model wrote it. reasoning is included when that model reports a level. contested is true when any claim on the page is disputed by a source or another page, and false otherwise. contradictions lists the name of each page that disagrees, and is [] when none do. fmlint enforces name and version only.
    • acceptance: Fiona skill section 13 marks those four fields required on wiki pages and states the choice rules above.

6.3. MDL Codes

Status: Implemented

mdlinter scans .md files. Fenced code, inline code, links whose target starts with http://, https://, mailto:, or #, and any target containing ://, are ignored. Sources under personal/ and raw/ are skipped. Targets inside those zones are still resolved.

Code Name Message Fails when Passes when
MDL001 Dead Link line <n>: <target> does not exist A relative ](target) does not resolve to a file The relative file exists
MDL002 Dead Wiki Link line <n>: [[<target>]] does not resolve to any page A [[wikilink]] matches no markdown stem A stem matches, case-insensitive
MDL003 Wrapped Table Row line <n>: table row continues onto the next line; keep each row on one line A line starts with |, does not end with |, and the next line is not blank and not a new row. A following line that is only | counts as a continuation The row is one line, or the next line is blank or a new row, or the split is inside a fence

Wikilink syntax: [[page]], [[page#heading]], [[page|alias]]. The lookup key is the text before # or |, compared to markdown file stems in the wiki, case-insensitive. Markdown links resolve against the linking file's directory.

  • 6.3.1. A link to a deleted file is MDL001 with file, line, and target. A [[missing]] wikilink is MDL002.
    • acceptance: Both codes appear in one wiki that has one of each.
  • 6.3.2. A table row split across two lines is MDL003 on the opening line. The same split inside a fence is clean.
    • acceptance: Only the unfenced split is reported.

6.4. PYC Codes

Status: Implemented

pycustomlint scans .py files with the AST, skips .gitignore matches and read-only zones, and accepts --json for a JSON list of violation objects.

Code Name Severity Message Fails when Passes when
PYC001 Inline Import error PYC001 - Inline import at line <n>: <stmt> import or from is indented inside a function, class, try, or an if that follows a function or class Every import is at column 0, or it sits in a module-level if before the first function or class
PYC002 Import After Code warning PYC002 - Import after code at line <n>: <stmt> A module-level import follows a non-import statement Imports are grouped before other module statements
PYC003 Late Declaration error PYC003 - Declaration after first function or class at line <n>: <stmt> A module-level assignment is at or after the first function or class in the module body Every module-level assignment is before that first function or class
  • 6.4.1. import os inside a function is PYC001.
    • acceptance: The report names the file, the line, and import os.
  • 6.4.2. An import inside a module-level if, elif, or else that appears before the first function or class is not PYC001. The same import after that first function or class is PYC001. A try does not get this exception. An import inside a function or class is PYC001.
    • acceptance: if os.name == "nt": import msvcrt before any function is clean. The same if after a function is PYC001 and the message names import msvcrt.
    • acceptance: try: import os before any function is PYC001.
  • 6.4.3. A module-level assignment at or after the first function or class statement in the module body is PYC003. An assignment nested inside a function or class is not PYC003. An assignment before that first function or class, including one inside a module-level if, is not PYC003.
    • acceptance: SHORT_HASH_LENGTH = 8 after a function is PYC003 and the message names SHORT_HASH_LENGTH = 8.
    • acceptance: The same assignment before any function or class is clean.
    • acceptance: value = 1 inside a function, and VALUE = 1 inside a class, are clean.
    • acceptance: VALUE = 1 inside a module-level if before any function is clean. The same if after a function is PYC003.

6.5. JSN Codes

Status: Implemented

json-lint parses with the standard json module. Style checks use section 5.1. Duplicate-key detection uses a parse hook so the last value does not hide the earlier key.

Code Name Severity Fails when Passes when
JSN001 JSON Syntax Error error The file is not JSON, including a trailing comma when that is a syntax error json.loads succeeds
JSN002 Indentation Error error Indent differs from indent / indent_len Indent matches
JSN003 Trailing Comma error A trailing comma is present and trailing_commas is false No trailing comma, or the setting allows them
JSN004 Duplicate Keys error An object repeats a key and check_duplicate_keys is true Keys are unique
JSN005 Key Order warning require_key_order is true and keys are not sorted Keys are sorted, or the setting is false
JSN006 Line Too Long warning max_line_length is set and a line exceeds it The key is absent, or every line fits
  • 6.5.1. A file with a missing comma reports JSN001 with line and column. A file with duplicate keys reports JSN004.
    • acceptance: Both cases name the file and the code.
  • 6.5.2. Four-space indent against indent_len: 2 is JSN002. json-lint fix rewrites it to two spaces.
    • acceptance: The fixed file is clean on the next chklint.

6.6. Base Linter Contract

Status: Implemented

  • 6.6.1. Shared behavior for fmlint, pycustomlint, mdlinter, and json-lint.
    • 6.6.1.1. Recursive file collection by extension.
    • 6.6.1.2. .gitignore and read-only-zone skipping.
    • 6.6.1.3. Settings load from fiona-settings.json.
    • 6.6.1.4. A violation record has filepath, line, column, code, message, rule id, and severity, plus a dict form for JSON.
    • 6.6.1.5. Report assembly follows section 6.1. The header is the linter name.
    • 6.6.1.6. The CLI is chklint [directory] and defaults to ..
  • 6.6.2. A new rule is a new checker used by one linter. It does not require an edit to the shared collector. Codes are unique: FML, MDL, PYC, JSN, plus pymarkdown MD codes and ruff codes.
    • acceptance: Existing FML and PYC tests still pass after a new MDL checker is registered.

7. Feature Requirements

7.1. wiki

Requirements for the wiki subcommands.

7.1.1. wiki init

Status: Implemented

Implements sections 3 and 4.7.1.

  • 7.1.1.1. Copy is 1:1 aside from DATE_PLACEHOLDER, ignore_files, and paths matched by the repository .gitignore.
    • acceptance: A pattern in ignore_files omits those paths and copies every other template file.
    • acceptance: A template file matched by the repository .gitignore is not copied.
  • 7.1.1.2. Copied markdown keeps version and receives created / updated dates where the template used the placeholder.
    • acceptance: After init, those fields match YYYY-MM-DD for the UTC day of the run.
  • 7.1.1.3. wiki init puts or appends .gitignore from the repository .gitignore, plus personal/ when that pattern is missing. Lines already in the file stay.
    • acceptance: After init, .gitignore contains __pycache__/ and personal/. A line keep-me written before init is still in the file.

7.1.2. wiki update

Status: Implemented

Implements sections 4.7.2 and 4.7.3.

  • 7.1.2.1. Markdown core files are compared as follows. An existing markdown file is never overwritten.
    • 7.1.2.1.1. Both sides have version, and the template version is greater: conflict (7.1.2.11.). Numeric dotted versions compare segment by segment (1.2.0 vs 1.10.0). Any other string compares as text.
    • 7.1.2.1.2. The wiki version is greater, or the versions are equal and the body differs after expanding DATE_PLACEHOLDER in the template: skip the file and report locally modified.
    • 7.1.2.1.3. Either version is missing: compare SHA-256 (7.1.2.12.). Equal hashes need nothing. Different hashes are a conflict.
  • 7.1.2.2. Other core files compare SHA-256 (7.1.2.12.) and update when the hash differs.
  • 7.1.2.3. fiona-settings.json uses the hash comparison of 7.1.2.12.. Different hashes are a conflict (7.1.2.11.). The owner's file stays. The template is written as fiona-settings-update.json. The tool does not merge the two.
    • acceptance: After wiki update --yes, the owner settings bytes are unchanged and fiona-settings-update.json matches the template.
  • 7.1.2.4. Missing template directories outside wiki/ and outside read-only zones are created when they hold a file that update may add. User files that are not in the template stay. Read-only zones are never in the update set.
  • 7.1.2.5. A pristine init reports Wiki is up to date. and wiki update writes nothing.
    • acceptance: A second check-update still reports up to date.
  • 7.1.2.6. An older core markdown file is listed as a conflict. After a confirmed update its bytes are unchanged and its update file matches the template. A locally modified file is listed under skipped and is not overwritten.
    • acceptance: The modified file's bytes are unchanged after wiki update --yes.
    • acceptance: An AGENTS.md at 1.0.0 against a template at 2.0.0 keeps its bytes after wiki update --yes, and AGENTS-update.md holds the expanded template text.
  • 7.1.2.7. --dry-run prints the plan and leaves the tree unchanged.
    • acceptance: File hashes before and after a dry run match.
  • 7.1.2.8. --force and --yes skip the prompt, including on a non-TTY.
    • acceptance: The command exits 0 on a pipe when --yes is set and updates exist.
  • 7.1.2.9. A path under wiki/ is never added, overwritten, or listed as a change, an addition, a skipped file, or a directory to add. A template page under wiki/ that is newer than the wiki's copy (7.1.2.1.1. or 7.1.2.1.3.) is listed as a conflict, the same as any other markdown core file.
    • acceptance: A wiki/index.md older than the template keeps its bytes after wiki update --yes, and wiki/index-update.md holds the expanded template text.
    • acceptance: A wiki/index.md newer than the template, or missing, is absent from the check-update report.
  • 7.1.2.10. Update skips a path matched by the repository .gitignore or by the wiki's .gitignore. Comment lines and blank lines are not patterns. A pattern with no slash matches any path segment. A pattern that ends with / matches that directory and everything under it.
    • acceptance: A template file utils/scripts/__pycache__/example.pyc is absent from the check-update report.
    • acceptance: When the wiki .gitignore lists example.py, a drifted utils/scripts/example.py is left unchanged and is absent from the report.
  • 7.1.2.11. A conflict means the template copy and the owner's copy both exist and the tool must not overwrite the owner. That is a newer markdown file (7.1.2.1.) or a differing fiona-settings.json (7.1.2.3.). Update yields to the owner: it keeps the file and writes the template beside it. The LLM merges the two during housekeeping (7.2.).
    • 7.1.2.11.1. The update file sits in the same directory. Its name is the file stem, then -update, then the suffix: AGENTS.md gives AGENTS-update.md, wiki/schema.md gives wiki/schema-update.md, and fiona-settings.json gives fiona-settings-update.json. Relative links in the template still resolve from that directory.
      • acceptance: A conflict on wiki/schema.md writes wiki/schema-update.md and no other file.
    • 7.1.2.11.2. The update file holds the template text with DATE_PLACEHOLDER expanded, as wiki init would write it. An existing update file is replaced with the current template text.
      • acceptance: A second wiki update --yes leaves one update file, with the same bytes as the first run on the same day.
    • 7.1.2.11.3. Only a confirmed update writes an update file. check-update and --dry-run list the conflict and write nothing.
      • acceptance: No *-update.md exists after wiki update --dry-run.
    • 7.1.2.11.4. A conflict clears when the owner's version is at least the template's version. The merge sets that version and deletes the update file.
      • acceptance: After AGENTS.md is set to the template version with the owner's body, check-update lists no conflict for it.
  • 7.1.2.12. A file without version control, meaning a markdown file with no version on either side or any non-markdown file, is compared by SHA-256. The wiki side hashes the file's bytes. The template side hashes the text wiki init would write: UTF-8 with DATE_PLACEHOLDER expanded and line endings kept, or the raw bytes for a binary file.
    • 7.1.2.12.1. Equal hashes: the file is up to date and is not listed.
      • acceptance: A freshly initialized wiki reports Wiki is up to date. even when a template file holds DATE_PLACEHOLDER.
    • 7.1.2.12.2. Different hashes: the report line shows both hashes where a version pair would go, shortened to 8 hex digits: (sha256 1a2b3c4d -> 9f8e7d6c). --json carries the full 64-digit current_sha256 and template_sha256. A file compared by version has null in both.
      • acceptance: A drifted utils/scripts/example.py is listed as utils/scripts/example.py (sha256 <8 hex> -> <8 hex>), and a markdown file with no version that differs is a conflict with the same parenthesis.

7.1.3. wiki sanity

Status: Implemented

Implements section 4.7.4. Markdown lint uses the wiki's own fiona-settings.json. Dead links are MDL001 and MDL002 inside mdlinter, not a separate check. New checks are added by registering another (name, runner) pair in the wiki sanity list.

  • 7.1.3.1. A clean wiki exits 0. Any violation exits 1.
    • acceptance: One dead link makes passed false in --json and produces a non-zero exit.
  • 7.1.3.2. --json output contains only JSON.
    • acceptance: The stdout of a --json run parses with json.loads and contains no ANSI sequences.
  • 7.1.3.3. Human output uses the progress display in 7.14.2. with label Running Wiki Sanity.
    • acceptance: A piped run prints that label and dots, and no ANSI escapes.

7.1.4. wiki graph

Status: Partially Implemented

7.1.4.16. and 7.1.4.17. are planned. Every other requirement in this section is built.

The renderer is D3 v7, loaded from https://d3js.org/d3.v7.min.js, with the visualization embedded in one HTML file. vis.js is not used.

  • 7.1.4.1. The parser extracts links as follows.
    • 7.1.4.1.1. Markdown links [text](path), including ./ and ../, resolve against the source file and are stored relative to the wiki root. A path without an extension is treated as .md.
    • 7.1.4.1.2. Wikilinks [[file]] and [[dir/file]] are parsed when wikilinks are enabled.
    • 7.1.4.1.3. External URLs (http://, https://, ftp://, and any ://) are recorded. The HTML graph includes each URL as a node. JSON lists them in metadata only with --external-links or external_links in settings.
    • 7.1.4.1.4. A link whose target file is missing is recorded as broken and does not abort the run.
  • 7.1.4.2. The graph is built as follows.
    • 7.1.4.2.1. A node is a markdown file. An edge is one or more links from source to target. Weight is the link count. Each markdown node stores that file's full frontmatter object.
    • 7.1.4.2.2. An orphan has in-degree 0 and out-degree 0.
    • 7.1.4.2.3. Each node stores unique_points, total_points, and link_score. unique_points is how many different files point at the node. total_points is the sum of those weights. One source with weight w scores 2 * (1 - 0.5^w). Ten links from one file score less than two files that each link once.
  • 7.1.4.3. --format json and --json emit this object.
{
  "metadata": {
    "wiki_root": "<path>",
    "file_count": 1,
    "link_count": 0,
    "generated_at": "<UTC ISO-8601>"
  },
  "nodes": [
    {
      "id": "wiki/a.md",
      "name": "a.md",
      "in_degree": 0,
      "out_degree": 0,
      "unique_points": 0,
      "total_points": 0,
      "link_score": 0,
      "frontmatter": {"name": "A", "version": "1.0.0"}
    }
  ],
  "edges": [
    {"source": "wiki/a.md", "target": "wiki/b.md", "weight": 2}
  ],
  "orphans": ["wiki/a.md"]
}
  • 7.1.4.4. When external URLs were collected, metadata also has external_links_count (total occurrences) and external_links (unique URL strings).
  • 7.1.4.5. HTML behavior.
    • 7.1.4.5.1. One self-contained file. The layout fills the viewport and follows resize. The page script is templates/graph_physics.js. Generation pastes that file into the page. The saved HTML does not request it separately.
      • acceptance: Generated HTML contains function frameNodes and does not contain the paste marker.
    • 7.1.4.5.2. Node radius scales with link_score from min to max in section 5.3. min and max are lengths in em, where 16 is 1em, and may be fractions. The drawn radius keeps that fraction, so the sizes between min and max are not snapped to a whole unit. Color interpolates low to mid over the first half of link_score divided by the max score, then mid to high. A score of 0 uses the low color and the min radius. Two files that each link once outrank one file that links ten times.
    • 7.1.4.5.3. Edge thickness is edge_width from the active theme, in em, where 16 is 1em. It is 1 when that key is absent or not a positive number. Changing the theme uses that theme's width. Color runs from gray (weight 1) through yellow to red (maximum weight).
    • 7.1.4.5.4. Hover: the node uses the hover color and the capped larger radius. Connected edges use the hover edge color. Other visible edges dim. Leaving the node restores its previous radius and color and restores the dimmed edges. Hover does not clear a click highlight. While both are active, edges of the hovered node and edges of the clicked node stay highlighted. Hover fixes the node in place the same way a click does. Leaving the node releases it, unless that node is the clicked one. A card appears beside the node. A line with an arrow points from the card to the node, the same way an edge points at its target. The card has two lines. The first line is the node name. The second line is unique points:total points:in:out. A page name stays as the filename. An http or https link drops the scheme. A path with more than one segment shows host/.../last. When that last segment is longer than hover_link_tail_chars, the card shows host/... plus the last that many characters, with no slash before them. Width follows the longer line. Height follows the two lines. The card is a body in the layout. A link keeps it near the node. It repels other nodes, and they repel it. The card stays inside the graph. Leaving the node hides that card. It does not hide a click card. Hover uses that glow when show_glow is true or absent. show_glow false draws no glow on hover, click, or a search match. The hover card is shown when show_link_card is true or absent. show_link_card false draws no hover card. It does not change the click card.
    • 7.1.4.5.5. Single click toggles a persistent highlight and shows that file's frontmatter card at the upper left of the graph. This card is not the hover card. It lists every frontmatter field and its value, then unique points, total points, in-degree, and out-degree. A file with no frontmatter shows an empty field list. Another node replaces the card. Empty space hides it. Hover does not. show_link_card does not hide it. The clicked node carries the same glow as a search match when show_glow is true or absent. Clearing the highlight removes that glow. A node that is still a search match keeps the glow when glow is on. The clicked node is fixed where it is. No other node is fixed. Connected nodes keep moving under the simulation, so their edges follow. Clearing the highlight releases the node. Dragging that node leaves it fixed at the drop point.
    • 7.1.4.5.6. Double click opens the markdown file.
    • 7.1.4.5.7. Search highlights matching nodes in light blue at 1.5 times radius (capped at max), shows neighbors in gray, and hides nodes that are not reachable from a match. A match also carries a glow when show_glow is true or absent. Neighbors and a cleared search do not. show_glow false draws no glow.
    • 7.1.4.5.8. A checkbox toggle labeled "Show Orphan Nodes" shows or hides orphans. It starts checked. The label does not change.
    • 7.1.4.5.9. Overwrite of an existing output file asks for confirmation on a TTY.
    • 7.1.4.5.10. Files above max_files are skipped after a warning. The first max_files files are drawn.
    • 7.1.4.5.11. The run completes in under 10 seconds for 1000 files on a current developer machine.
    • 7.1.4.5.12. A checkbox toggle labeled "Show External Links" shows or hides external-link nodes and their edges. It starts checked when show_external_links in section 5.1 is true, and unchecked when that key is absent or false. The label does not change. When it starts checked, those nodes and edges are drawn on the opening fade. A click is not required. An external node is the text cloud character. A page node stays a circle.
    • 7.1.4.5.13. A checkbox toggle labeled "Show Subtitles" shows or hides labels under nodes. It starts checked when subtitle.enabled in section 5.1 is true or absent, and unchecked when that key is false. The label does not change. A node shows its label when its drawn radius is at least subtitle.min_radius from the active theme times that theme's node_radius.max. The fraction is 0.5 when absent, and a value outside 0 to 1 is clamped. Changing the theme uses that theme's fraction and max. The text is the frontmatter name when that field is a non-empty string, otherwise the filename. The label sits under the shape. The gap is subtitle.padding em from the active theme, where 16 is 1em, 12 when absent. A smaller number sits closer. The font size is subtitle.size on that same scale, 11 when absent or below 1. subtitle.on_top true draws every label after the nodes, so another node does not cover it. Absent or false keeps the label inside its node. subtitle.outline true strokes the letters with subtitle.outline_width on that same scale, 3 when absent, in subtitle.outline_color. Absent color uses graph_background. Absent or false draws no stroke. Changing the theme applies that theme's subtitle settings.
    • 7.1.4.5.14. A checkbox toggle labeled "Show Legend" shows or hides a card at the upper right of the graph. It starts checked when show_legend in section 5.1 is true or absent, and unchecked when that key is false. The label does not change. The card has three groups. Shape shows a page and an external link in the text color. Nodes are growing circles: few, some, and many files pointing here. Edges are lines: linked once, a few times, and many times. Node circles use the theme node colors. Edge lines use the theme edge colors.
    • 7.1.4.5.15. A select labeled "Theme" lists every theme from section 5.3 and starts on theme from section 5.1. That name is night-sky when the key is absent. Choosing a theme updates the page background, font, buttons, cards, node colors, node sizes, edge colors, highlight colors, the clicked-node color, and node shapes. Choosing a theme also runs the See all fit, so the shown nodes are centered again. The grayscale dark-mode switch runs that fit too. The theme applied at load does not. A missing theme key uses colorful. grayscale uses its dark object when the page is in dark mode.
    • 7.1.4.5.16. A checkbox toggle labeled "Show Arrows" draws or hides an arrowhead on each visible edge. It starts checked when show_arrows in section 5.1 is true or absent, and unchecked when that key is false. The label does not change. The arrowhead meets the target shape. The source file points at the target file. The arrow uses that edge's color. A hidden edge has no arrowhead.
    • 7.1.4.5.17. simplify_external_links sets an external node's subtitle to its host. A leading www. is dropped. The node stays one per URL. consolidate_external_links merges URLs with the same host into one node whose id is host/* when consolidate_external_links_depth is absent. Depth 0 leaves each URL alone, even when the bool is true. Depth 1 is host/*. Depth 2 is host/a/*. Depth 3 is host/a/b/*. A leading www. is dropped from the host. A path part longer than consolidate_external_links_max_char characters is .... The limit is 8 when absent. A part of that length stays. Size and color follow link_score in section 7.1.4.2.3. When depth is above 0, simplify is on even if its key is false. The bool defaults to false. A double click opens https://host/ plus the real path parts before the first ..., with a trailing slash.
      • acceptance: https://www.domain.com/abc and https://domain.com/acd from two files become one node domain.com/* with subtitle domain.com and link_score 2 when consolidate is true and depth is absent. With only simplify, both URLs remain and each subtitle is domain.com. Depth 2 merges https://domain.com/abc/one and https://domain.com/abc/two into domain.com/abc/* and leaves https://domain.com/zzz/one as domain.com/zzz/*. Depth 3 names https://domain.com/a/b/c as domain.com/a/b/* and opens https://domain.com/a/b/. https://domain.com/abcdefghij/b/c at depth 3 is domain.com/.../b/*.
    • 7.1.4.5.18. A refresh puts every toggle, the theme select, and the search box back to the values in section 5.1. The browser's saved form state does not keep the previous choice.
      • acceptance: The page sets each control again on pageshow.
    • 7.1.4.5.19. A checkbox toggle labeled "Show Sources" shows or hides nodes built from the frontmatter source key, and the edges from the page to those nodes. It starts checked when show_source in section 5.1 is true or absent, and unchecked when that key is false. The label does not change. A string is one node. A list of strings is one node per entry. A blank string is skipped. Any other value is skipped. The node is a cloud, the same shape as an external link. Its id is that string. Two pages with the same string share one node. A double click opens the string when it contains ://.
      • acceptance: source: https://example.com/rules.md adds one cloud node and one edge from that page. A list of two URLs adds two nodes. source: 12 adds none.
  • 7.1.4.6. Layout constants live at the top of the HTML template so they can be tuned without hunting through the script.
Constant Value Role
LINK_STRENGTH 0.4 Spring along an edge; neighbors follow a dragged node
BASE_LINK_DISTANCE 80 Rest length of an edge
LINK_DISTANCE_PER_DEGREE 0 Distance does not grow with degree
CHARGE_STRENGTH -140 Repulsion that spreads the cloud
CHARGE_DISTANCE_MIN 24 Near distance for repulsion
CHARGE_DISTANCE_MAX 500 Far cutoff for repulsion
NODE_ATTRACTION_STRENGTH 0 Extra attraction is off
CENTER_STRENGTH 1 Each tick places the centroid on the viewport center
GRAVITY_STRENGTH 0.05 Per-node pull toward the viewport center
COLLIDE_STRENGTH 1 Overlap prevention
BASE_COLLIDE_RADIUS 6 Gap around the hitbox
HITBOX_SCALE 1.25 Collision circle versus the drawn radius
EDGE_PADDING_MULTIPLIER 0 Degree does not add collide radius
CANVAS_MARGIN 56 Border band that pushes a node inward
CANVAS_MARGIN_STRENGTH 0.35 Strength of that inward push
ALPHA_DECAY 0.018 How fast the simulation cools
ALPHA_MIN 0.001 Floor before the layout rests
VELOCITY_DECAY 0.22 Low damping so motion glides
FLOAT_AMPLITUDE 2 Idle drift, in em where 16 is 1em
FLOAT_SPEED 0.0007 How fast that drift cycles
  • 7.1.4.7. Nodes start in a small disk around the viewport center and expand under repulsion. They fade in by a breadth-first walk from the highest in-degree node, highest connectivity first. d3.forceCenter keeps the centroid on the viewport center. d3.forceX and d3.forceY add center gravity so outliers drift back. Both targets follow resize. The tick does not write a canvas-edge clamp back into node coordinates. A margin force adds inward velocity only while a node is inside CANVAS_MARGIN of the border, and it skips a node that is pinned. The hitbox is HITBOX_SCALE times the drawn radius, plus a fixed gap, so a large node keeps clear of its neighbors. Each tick then pushes any covering pair apart until the drawn circles, plus that gap, no longer cover. An edge's rest length is at least the two hitboxes added together. Degree does not add padding, so a hub is not shoved into the border. An unpinned node then drifts at most FLOAT_AMPLITUDE on that em scale, and its edges follow, so it looks like it is floating. A pinned or dragged node does not drift. The view eases toward every shown node over about 4200 ms, with 4em of padding. That is the same fit as the button labeled "See all". The first node uses first_fit_ms instead, so that opening zoom is slower. It runs when the first node fades in, and again when the last node fades in. A fit that is already moving keeps its speed and aims at the new bounds. A second See all does not stop the zoom. It also runs when the theme changes. The scale can go below 0.15 when that is what fits the cloud, and it stays at most 6. Hidden external links and hidden orphans are left out. A later wheel zoom, background drag, or node drag keeps the view the user set. Wheel zoom and background drag pan the view. Dragging a node reheats the simulation; releasing it lets the springs and gravity settle.
  • 7.1.4.8. The output path must be writable. A failure is an error message and a non-zero exit.
  • 7.1.4.9. A wiki with one relative link and one wikilink produces two edges, and a file with neither is listed in orphans.
    • acceptance: The JSON edges weights match the link counts.
  • 7.1.4.10. --no-wikilinks drops wikilink edges and keeps markdown links.
    • acceptance: The wikilink target is absent from edges.
  • 7.1.4.11. Opening the HTML in a current browser shows the graph, hover, click highlight, search, and the orphan toggle.
    • acceptance: A node hover changes its color to the hover color and restores the previous radius on exit.
  • 7.1.4.12. JSON and HTML each store frontmatter on every markdown node: every YAML field from that file. Missing or invalid YAML is {} and does not abort the run.
    • acceptance: A file with name, version, and updated has those three keys on its node. A file with no frontmatter has {}. The click card lists every frontmatter field. The hover card has two lines: the node name, then unique points, total points, in-degree, and out-degree.
  • 7.1.4.13. Repeated links from one file add less to size and color than links from different files.
    • acceptance: One source with weight 10 has unique_points 1, total_points 10, and link_score below 2. Two sources with weight 1 have link_score 2. HTML radius and color divide link_score by maxLinkScore. The hover card's second line includes the unique-point count and the total-point count.
  • 7.1.4.14. The Show Arrows toggle follows show_arrows.
    • acceptance: The checkbox is checked when the key is absent or true, and unchecked when it is false. The HTML defines marker edge-arrow and sets marker-end from that toggle.
  • 7.1.4.15. Graph settings start from the packaged fiona-settings.json graph section. A file found by walking up from the scanned directory replaces the keys it sets. A missing file still uses the packaged section.
    • acceptance: A scan whose directory and parents have no fiona-settings.json still applies show_legend, show_arrows, and simplify_external_links from the packaged file.
  • 7.1.4.16. NOT_IMPLEMENTED The HTML graph draws a node when enough files point at it, or when a parent of that node is clicked. The setting is node_importance_min_point in section 5.1.
    • 7.1.4.16.1. NOT_IMPLEMENTED The count is unique_points from 7.1.4.2.3.: how many different files point at the node. A node is below the cutoff when unique_points is less than the setting. A node at the cutoff is drawn.
      • acceptance: The setting is 5. A node with unique_points 4 is absent from the opening draw. A node with unique_points 5 is drawn.
    • 7.1.4.16.2. NOT_IMPLEMENTED 0 disables the filter. A missing key, or a value that is not an integer of 0 or more, is the same as 0. Every node is drawn, still subject to the Show Orphans, Show External Links, and Show Sources toggles.
      • acceptance: With the key absent or set to 0, a node with unique_points 0 is drawn when those toggles allow it.
    • 7.1.4.16.3. NOT_IMPLEMENTED A parent of a node is any node with an edge whose target is that node. While a parent is the clicked node, each direct target is drawn even when it is below the cutoff. The clicked node stays drawn for as long as it stays clicked, including when it is below the cutoff. Clearing the click hides a below-cutoff node again. Clicking a node draws its direct targets only, not the targets of those targets.
      • acceptance: The setting is 5. Page A has unique_points 5 and links to B. B has unique_points 1 and links to C. C has unique_points 1. The opening draw shows A and hides B and C. Clicking A draws B and leaves C hidden. Clicking B keeps B drawn and draws C. Clicking empty space hides B and C. A node with no incoming edge stays hidden, because it has no parent to click.
    • 7.1.4.16.4. NOT_IMPLEMENTED An edge is drawn only when both of its nodes are drawn. See all, the opening fit, and search use that same drawn set. A hidden node is left out of the fit, as 7.1.4.7. does for a hidden orphan, and it is not a search match.
      • acceptance: An edge into a below-cutoff node is absent until a click draws that node. Searching for its name does not draw it.
    • 7.1.4.16.5. NOT_IMPLEMENTED Show Orphans, Show External Links, and Show Sources still apply. A node hidden by one of those toggles stays hidden when it meets the cutoff or when its parent is clicked.
      • acceptance: With Show External Links unchecked, clicking a page does not draw the external node it links to.
    • 7.1.4.16.6. NOT_IMPLEMENTED --format json and --json still list every node and edge. The filter changes the HTML draw only.
      • acceptance: JSON includes a node whose unique_points is below the setting.
  • 7.1.4.17. NOT_IMPLEMENTED A markdown file that .gitignore matches is not a graph node. HTML and JSON both omit it. An edge that would start or end at that file is omitted. The file does not count toward max_files. Housekeeping's orphan list (7.2.6.) uses this graph, so an omitted file is not an orphan. There is no setting that turns this off.
    • acceptance: The scanned .gitignore lists secret.md. secret.md is absent from JSON nodes, edges, and orphans. A page that links to it has no edge to it.
    • acceptance: max_files is 1. The directory holds kept.md and a gitignored secret.md. JSON file_count is 1 and the only node is kept.md.
    • 7.1.4.17.1. NOT_IMPLEMENTED Patterns are the non-comment, non-blank lines of the .gitignore in the scanned directory, and of each .gitignore in a subdirectory. A subdirectory file's patterns match only paths under that subdirectory. Matching is 7.1.2.10.: a pattern with no slash matches any path segment, and a pattern that ends with / matches that directory and everything under it.
      • acceptance: The scanned .gitignore lists drafts/. drafts/note.md is absent. wiki/note.md is present.
      • acceptance: A line # secret.md does not omit secret.md.
      • acceptance: notes/.gitignore lists scratch.md. notes/scratch.md is absent. scratch.md at the scan root stays.
    • 7.1.4.17.2. NOT_IMPLEMENTED A path under .git/ is omitted even when no pattern names it.
      • acceptance: With no .gitignore, a markdown file under the scan root stays, and a markdown file under .git/ is absent.

7.1.5. wiki ingest

Status: Implemented

Implements section 4.7.6. An ingest has two phases. The tool runs the first: it turns one source into a file on disk, archives a copy under raw/, converts that copy to markdown, and writes a wiki page with status: raw. The agent runs the second: it refines the page and sets status: refined. The Fiona skill teaches the second phase and when to convert a source by hand. The pattern both phases follow is section 3.1 of llm-wiki-mod.

The tool runs five stages in order. Each stage has one base class and one job. Class names are in Appendix D.

Stage Input Output
Fetch the <source> argument a local file, its origin, and its media type
Archive that local file a file under raw/
Convert the archived file markdown body, title, metadata, conversion state, warnings
Write the conversion one page under wiki/ with status: raw
Record the run one line appended to wiki/log.md

Shipped converters, in registration order:

Converter Media types Extensions Raw category Library
markdown text/markdown .md, .markdown articles none
text text/plain .txt, .text, .log articles none
html text/html, application/xhtml+xml .html, .htm, .xhtml articles beautifulsoup4, markdownify
pdf application/pdf .pdf papers pypdf

A source that no converter claims is archived under raw/other/. Pure-Python libraries are preferred so the install stays light. Candidates for later formats are in Appendix C.

  • 7.1.5.1. The pipeline is built from parts.
    • 7.1.5.1.1. The pipeline receives its fetchers, converter registry, archiver, page writer, and log writer as constructor arguments. It holds no format-specific code.
    • 7.1.5.1.2. A converter declares its name, media types, extensions, and raw category as class attributes, and implements one convert method. Adding a format is one new converter class, one registration, and its tests.
      • acceptance: A test registers a converter for application/x-test and ingests a .xtest file into a page. No pipeline, archiver, or writer module changes.
    • 7.1.5.1.3. A fetcher declares whether it accepts an argument and returns the fetched source. The URL fetcher takes its opener as an argument.
      • acceptance: Tests run the URL path with a fake opener and make no network call.
  • 7.1.5.2. Fetch.
    • 7.1.5.2.1. An argument that starts with http:// or https:// is a URL. Any other <scheme>:// prefix exits 1 with Error: Unsupported URL scheme: <scheme>. Anything else is a local path. A missing path and a directory each exit 1 with the message from section 4.7.6.
      • acceptance: ftp://host/a.pdf exits 1 and writes nothing.
    • 7.1.5.2.2. A URL is downloaded with urllib.request into ingest/<slug>.<ext>. Redirects are followed. Proxies come from the environment. The request sends user_agent and waits up to timeout_seconds. No credentials are sent.
    • 7.1.5.2.3. A non-2xx response or a network error exits 1 with Error: Download failed: <url>: <reason>. The reason names the HTTP status when there is one. A body larger than max_download_bytes stops the download and exits 1. In each case the partial file is deleted and ingest/ is unchanged.
      • acceptance: A fake opener that answers 404 exits 1, the message contains 404, and ingest/ has the same files as before.
    • 7.1.5.2.4. The media type is the response Content-Type without parameters. A charset parameter is kept as a decoding hint. With no usable header, the extension decides, using the converter table and then mimetypes. With no known extension, the first 1024 bytes decide: %PDF- is application/pdf, <!doctype html or <html in any case is text/html, valid UTF-8 with no NUL byte is text/plain. Anything else is application/octet-stream.
      • acceptance: A local file named notes whose bytes start with %PDF- is converted by pdf.
    • 7.1.5.2.5. A downloaded file takes the first extension of the converter that claims its media type, then the URL path suffix, then .bin.
      • acceptance: https://host/report served as application/pdf is saved as ingest/report.pdf.
    • 7.1.5.2.6. The slug of a URL is built from its last three non-empty path segments, each without its extension, with a segment named index dropped. An empty path uses the host. The slug of a file is its stem. The text is lowercased, each run of characters outside [a-z0-9] becomes one -, leading and trailing - are trimmed, and the result is cut to 80 characters. An empty result is source.
      • acceptance: https://host/job/performance_TL153/2108/console has the slug performance-tl153-2108-console.
  • 7.1.5.3. Archive.
    • 7.1.5.3.1. The default raw path is raw/<category>/<slug>.<ext>. The category is the raw category of the converter that claims the media type, unless raw_categories maps that converter name to another directory. With no converter it is other.
      • acceptance: A PDF is archived at raw/papers/<slug>.pdf.
    • 7.1.5.3.2. -r/--raw PATH is relative to raw/. A leading raw/ is accepted. A value that ends in an extension names the file. Any other value is a directory, and the file inside it is <slug>.<ext>. Missing directories are created. After resolving .. and symlinks the path must be inside raw/, else exit 1 with nothing written.
      • acceptance: -r jenkins/tl153 archives an HTML source at raw/jenkins/tl153/<slug>.html. -r ../wiki/x exits 1.
    • 7.1.5.3.3. Before writing, the archive computes the SHA-256 of the fetched file and compares it with every file under raw/. Only files of the same byte size are hashed. A match anywhere under raw/, at any name or directory, is reused: nothing is written under raw/, raw_path is the matching file, and raw_reused is true. When -r names another place, the match still wins and adds the warning already archived at <raw path>; --raw ignored. Several matches use the first in sorted path order.
      • acceptance: Ingesting one file twice, the second time under another name and with -r other/dir, leaves one raw copy and the second run reports raw_reused: true and that copy's path.
      • acceptance: A URL whose body matches an archived file reuses that file.
    • 7.1.5.3.4. The archive never overwrites, renames, or deletes a file under raw/. When the raw path holds a file with different bytes, the first free name of <slug>-2.<ext>, <slug>-3.<ext>, and so on is used.
      • acceptance: Ingesting a changed file of the same name creates <slug>-2.<ext> and the first copy is unchanged.
    • 7.1.5.3.5. When the raw copy is reused and a page under wiki/ already has that raw path in sources or that hash in ingest.sha256, the run is a duplicate. It prints the Duplicate: line, sets duplicate_page, writes no page and no log line, and exits 0. --force converts again and replaces that page, at its current path unless -t is set.
      • acceptance: A second ingest of the same bytes exits 0, reports the first page as duplicate_page, and leaves every file under the wiki unchanged apart from the moved source.
    • 7.1.5.3.6. A file inside the wiki's ingest/ directory is moved into raw/, so it leaves the drop zone. A downloaded file is in ingest/, so it is moved too. When the raw copy is reused, the ingest/ file is deleted, because raw/ already holds its bytes. Any other file is copied and the original is left as it was. A file already under raw/ is not archived again, and conversion reads it in place. -r on such a file exits 1.
      • acceptance: After ingesting ingest/a.pdf, that path is gone and raw/papers/a.pdf holds its bytes. A file ingested from personal/ is copied and the original is unchanged.
  • 7.1.5.4. Convert.
    • 7.1.5.4.1. The registry picks the first converter, in registration order, whose media types contain the detected type, then the first whose extensions contain the file extension. With no match, no page is written, stdout ends with the No converter line, and the command exits 3.
      • acceptance: A .xyz file of random bytes exits 3 and is archived under raw/other/.
    • 7.1.5.4.2. A converter returns the markdown body, a title or none, a metadata map, a conversion state of complete or partial, and a list of warnings. A converter that cannot read a file it claims, such as a corrupt or encrypted PDF, raises a conversion error. The command then writes no page, prints Error: Conversion failed: <converter>: <reason> on stderr, and exits 3. The raw copy stays.
      • acceptance: An encrypted PDF exits 3, its raw copy exists, and no page is written.
    • 7.1.5.4.3. markdown keeps the body byte for byte. A frontmatter block in the source moves into source_meta. The title is that frontmatter title, then the first H1, then the slug.
    • 7.1.5.4.4. text decodes UTF-8 with any BOM stripped, then the charset hint, then cp1252. A fallback decode adds a warning. The body is one fenced text block whose fence is longer than any backtick run in the text. The title is the slug.
    • 7.1.5.4.5. html parses with beautifulsoup4 and converts with markdownify.
      • 7.1.5.4.5.1. The title is <title>, then og:title, then the first <h1>. source_meta holds lang, the canonical link, and the description, author, og:*, and article:published_time meta values that are present.
      • 7.1.5.4.5.2. The content root is the first <article>, then <main>, then <body>. script, style, noscript, template, and comments are dropped.
      • 7.1.5.4.5.3. A relative href or src is made absolute against <base href>, then the origin URL. A local file has no origin URL, so its relative links are left to 7.1.5.5.4..
      • 7.1.5.4.5.4. Headings are ATX. Bullets use -. <pre> becomes a fenced block. A table becomes a pipe table.
      • 7.1.5.4.5.5. A body with no text outside markup is partial with the warning no text content; the page may need JavaScript or a login.
      • acceptance: For a fixture with a nav, an article, a script, a relative link /a, and a table, fetched from https://host/p, the body holds the article text, no script text, the link https://host/a, and a pipe table.
    • 7.1.5.4.6. pdf reads with pypdf.
      • 7.1.5.4.6.1. source_meta holds the page count and the /Title, /Author, /Subject, /Keywords, /CreationDate, and /ModDate values that are present. Dates are ISO 8601. The title is the metadata title, then the slug.
      • 7.1.5.4.6.2. With an outline, each outline entry becomes a heading before the text of its destination page. The level is the entry depth plus 2, capped at 6. Without one, each page starts with ## Page <n>.
      • 7.1.5.4.6.3. A page with no text adds the warning page <n> has no text layer and makes the result partial. When every page is empty, the one warning is no text layer; the PDF may be scanned.
      • acceptance: A two-page fixture with no outline has ## Page 1 and ## Page 2. A fixture with one blank page is partial and names that page.
  • 7.1.5.5. Write.
    • 7.1.5.5.1. The default page is wiki/<default_target>/<slug>.md. -t/--target PATH is relative to wiki/, and a leading wiki/ is accepted. A value that ends in .md names the page. Any other value is a directory, and the page inside it is <slug>.md. After resolving .. and symlinks the path must be inside wiki/, and it must not be wiki/index.md, wiki/log.md, wiki/schema.md, or a file under wiki/logs/. Else exit 1.

      • acceptance: -t concepts writes wiki/concepts/<slug>.md. -t ../raw/x.md and -t index.md exit 1.
    • 7.1.5.5.2. An existing page exits 1 unless --force is set. The check runs before fetch, so a refusal writes nothing and downloads nothing.

      • acceptance: A second run on the same source exits 1 and the page bytes are unchanged. With --force the page is replaced.
    • 7.1.5.5.3. The page frontmatter is complete when the run ends. Every key below is present, in this order, even when its value is empty: [], {}, or null. sources is the raw path relative to the wiki root. title is the converter title, else the slug. ingest is filled from the fetch, archive, and convert stages, so its values are facts about this run.

      ---
      name: <slug>
      version: 0.1.0
      title: <title>
      created: <YYYY-MM-DD>
      updated: <YYYY-MM-DD>
      type: summary
      status: raw
      sources: [<raw path>]
      source_meta: {<key>: <value>}
      ingest:
        origin: <url, or resolved file path>
        origin_kind: <url or file>
        final_url: <url after redirects, or null>
        fetched: <UTC time, YYYY-MM-DDTHH:MM:SSZ>
        media_type: <media type>
        size_bytes: <int>
        sha256: <64 lowercase hex of the raw copy>
        raw_reused: <true or false>
        converter: <converter name>
        conversion: <complete or partial>
        warnings: [<text>]
        tool: fiona-assistant <package version>
      tags: []
      confidence: zero
      llm:
        - model: none
      contested: false
      contradictions: []
      ---
      
      • acceptance: For each fixture, the page frontmatter parses and has every key above. ingest.sha256 equals the SHA-256 of the file at sources[0], and ingest.warnings equals the JSON warnings.
    • 7.1.5.5.4. The body is # <title>, a blank line, the line Raw ingest of [<raw file name>](<link to the raw file>). Not refined yet., a blank line, and the converted body. When the converted body has an H1, every heading in it moves down one level, and H6 stays H6. A relative link or image that does not resolve from the page is replaced by its text, and the count is one warning.

      • acceptance: The page has exactly one H1, and mdlinter reports no MDL001 for it.
    • 7.1.5.5.5. After writing, the writer runs the pymarkdown fixer on that page only. Violations still left add the warning <N> markdown style violations left.

      • acceptance: Every fixture page passes fmlint with no violation.
  • 7.1.5.6. Record. A run that archives appends one bullet to wiki/log.md and bumps that file's patch version and updated. The bullet is - <YYYY-MM-DD> - ingest <origin> -> <raw path> -> <page path> (raw), or ends -> <raw path> (no page) when no page was written. The tool does not edit wiki/index.md or any other page.
    • acceptance: One successful run adds one line to log.md and leaves index.md unchanged.
  • 7.1.5.7. --raw-only runs fetch, archive, and record, and skips convert and write. It exits 0.
    • acceptance: No page is written and conversion is none.
  • 7.1.5.8. --dry-run fetches into a temporary directory outside the wiki, so a URL is still downloaded. It prints Dry run - no files will be modified: and then the lines a real run would print. It writes nothing under the wiki, does not move the source, and deletes the temporary directory. dry_run is true in JSON.
    • acceptance: File hashes under the wiki match before and after a dry run, and an ingest/ source is still in place.
  • 7.1.5.9. --pending lists every page under wiki/ whose frontmatter status is raw, sorted by path, and exits 0.
    • acceptance: A page changed to status: refined is no longer listed.
  • 7.1.5.10. Refinement is the agent's phase. The tool never writes status: refined. A page is refined when status is refined, version is 1.0.0 or later, the quality signals are set and llm.model is not none, the page is listed in wiki/index.md, and wiki/log.md has a line for the refinement.
    • acceptance: Fiona skill section 16 lists those conditions and the steps that meet them, and says when to convert a source by hand.
  • 7.1.5.11. The agent checks the frontmatter the tool wrote before it refines the body. It may correct title, type, tags, source_meta, ingest.conversion, and ingest.warnings when the source shows they are wrong. It does not hand-edit ingest.sha256, ingest.size_bytes, ingest.fetched, ingest.media_type, or ingest.tool. When one of those is wrong, it re-runs the ingest with --force or reports the fault to the user.
    • acceptance: Fiona skill section 16 has a metadata check step that names the fields it may edit and the fields it may not.
  • 7.1.5.12. Archived sources do not fail wiki sanity. Every check skips raw/.
    • acceptance: After ingesting a markdown file with style violations, wiki sanity reports none from its raw copy.
  • 7.1.5.13. --json prints only the JSON object from section 4.7.6 on stdout, on exit 0 and on exit 3. On exit 1, stdout is empty.
    • acceptance: The stdout of an exit 3 run parses with json.loads and has conversion: none.

7.2. housekeeping

Status: Implemented

Implements section 4.8. Housekeeping is the routine an LLM runs to maintain a wiki. It gathers what needs work; the LLM does the work. Paths in details and actions are relative to the wiki root.

  • 7.2.1. The steps run in this order: check-update, merge, ingest, sanity, graph. Every step runs, even after a step fails.

    • acceptance: A wiki with a pending update file and a dead link reports merge and sanity as FAIL, and graph still runs.
  • 7.2.2. check-update runs the check of 7.1.2. and writes nothing. It is PASS when nothing needs an update, a conflict, or an addition, and WARN otherwise. Its summary is <U> update(s), <C> conflict(s), <A> addition(s). Each item is a detail line: update: <path>, conflict: <path> -> <update path>, or add: <path>. The action is Ask the user, then run fiona-assistant wiki update --dry-run and wiki update --yes.

    • acceptance: An AGENTS.md older than the template gives WARN and the detail conflict: AGENTS.md -> AGENTS-update.md, and no update file is written.
  • 7.2.3. merge lists every file whose stem ends with -update outside personal/, raw/, and .git/. That includes fiona-settings-update.json. It is FAIL when one exists. Each detail is <update path> -> <target path>, followed by a space and (target missing) when the target is absent. The target is the same name without -update. The action per file is:

    Merge <update path> into <target path>: keep the owner's edits, take the template changes, set version to at least <template version>, then delete <update path>.
    

    <template version> is the update file's version, or the text the template version when it has none.

    • acceptance: A wiki/schema-update.md at 1.5.0 gives the detail wiki/schema-update.md -> wiki/schema.md and an action that names 1.5.0.
    • acceptance: fiona-settings-update.json gives the detail fiona-settings-update.json -> fiona-settings.json.
  • 7.2.4. ingest lists the markdown pages under wiki/ whose frontmatter has status: raw. It is WARN when one exists, with one detail per page and the action Refine each raw page, then set status: refined.

    • acceptance: A page with status: raw is WARN and named in the details. With none, the step is PASS.
  • 7.2.5. sanity runs the checks of 7.1.3. without the progress display. It is FAIL when any check has a violation. The summary is <N> violation(s) in <K> check(s). The details are each failing check's report lines. The action is Fix the violations, then run fiona-assistant housekeeping again.

    • acceptance: One dead link makes the step FAIL and a detail line names the file.
  • 7.2.6. graph writes <wiki root>/wiki_graph.html with the defaults of section 4.7.5 and the wiki's graph settings. The summary is wrote wiki_graph.html (<nodes> nodes, <edges> edges). A markdown page under wiki/ with no inbound and no outbound link is an orphan: the step is WARN, one detail orphan: <path> per page, and the action Link each orphan page from a related page or from wiki/index.md. Update files are not orphans. An error writing the graph is FAIL with the error as the detail.

    • acceptance: After the run wiki_graph.html exists at the wiki root, and an unlinked page under wiki/ is named as an orphan.
  • 7.2.7. Housekeeping writes only wiki_graph.html. It does not run wiki update, merge an update file, or fix a violation.

    • acceptance: Apart from wiki_graph.html, every file hash is unchanged after a run.
  • 7.2.8. Next steps: lists each step's actions in step order, without repeats. passed is true when no step is FAIL.

    • acceptance: --json stdout parses with json.loads, and a wiki with only WARN steps has passed: true and exit 0.

7.3. librarian

Requirements for librarian.

7.3.1. Folder and In-file TOC

Status: Implemented

  • 7.3.1.1. Folder TOC (librarian toc).
    • 7.3.1.1.1. table_of_contents.md has frontmatter title, created, updated, and type: toc.
    • 7.3.1.1.2. Entries list subfolders, file links, and v<version>, or v0.0.0 when version is missing or unreadable.
    • 7.3.1.1.3. Headers become nested GitHub-style anchor links. Fenced code is not scanned. Files are read as utf-8-sig so a BOM does not hide the first header.
    • 7.3.1.1.4. Skip the TOC file itself, hidden folders, and --exclude matches.
    • 7.3.1.1.5. --recursive-toc writes a TOC in every subfolder.
    • 7.3.1.1.6. --non-recursive lists only markdown files directly in the folder, ignoring max_depth_from_dir.
    • 7.3.1.1.7. max_depth_from_file is the deepest header level (3 lists h1 through h3). max_depth_from_dir is how many subfolder levels to include (1 includes the folder and its children). Absent or 0 means unlimited. Any other non-integer or negative value exits 1 and names the key.
  • 7.3.1.2. In-file TOC (librarian toc file).
    • 7.3.1.2.1. Insert directly under the H1, after frontmatter. The H1 itself is not listed. Headings inside fences are not listed. A # inside frontmatter is not the H1.

    • 7.3.1.2.2. The block is wrapped in these markers. sectionized is present only when numbering was on.

      <!-- toc-start: headings 2..6 -->
      <!-- toc-end -->
      

      Example with numbering: <!-- toc-start: sectionized, headings 2..6 -->.

    • 7.3.1.2.3. On regeneration, marker parameters win, then --sectionized, then settings. A sectionized marker keeps numbering. The recorded heading range stays, so a later settings change does not retarget an existing TOC. A bare <!-- toc-start --> has no parameters and is upgraded on regeneration. An invalid range is ignored with a warning.

    • 7.3.1.2.4. The block is replaced in place. A current block is reported unchanged and not rewritten. Blank lines separate the markers from the list so MD032 passes.

    • 7.3.1.2.5. When sectionize is true, entries and heading lines gain numbers such as 1.1.. Existing numbers are stripped before renumbering so anchors still match.

    • 7.3.1.2.6. toc file * root is the nearest fiona-settings.json, current directory first, then subdirectories. Skip hidden folders and build/, dist/, __pycache__/, *.egg-info/. If none is found, use the current directory.

    • 7.3.1.2.7. toc file * skips read-only zones, hidden files and folders, table_of_contents.md, and ignore_files matches.

    • 7.3.1.2.8. toc file <explicit path> inside personal/ or raw/, or a missing or non-markdown path, exits 1 and writes nothing.

  • 7.3.1.3. Two markdown files in a folder produce one TOC that lists both versions and their headers, and omits a header that sits inside a fence.
    • acceptance: The fence header's anchor is absent.
  • 7.3.1.4. max_depth_from_file: 3 omits #### and lists ###. Absent or 0 lists every level.
    • acceptance: The #### line is absent only in the capped run.
  • 7.3.1.5. max_depth_from_dir: 1 omits files two levels down. Absent or 0 lists them. --non-recursive lists only the folder's own files.
    • acceptance: The deep file is absent in both limited modes.
  • 7.3.1.6. Re-running toc file replaces the marked block and does not append a second TOC.
    • acceptance: The file contains one toc-start marker.
  • 7.3.1.7. min_heading: 3 omits ##. max_heading: 3 omits ####.
    • acceptance: Each setting hides only its out-of-range level.
  • 7.3.1.8. An ignore_files match is not modified by toc file *. An explicit path in personal/ exits 1 and is unchanged.
    • acceptance: Both files keep their original bytes.

7.3.2. Non-markdown Version Tags

Status: Planned

  • 7.3.2.1. NOT_IMPLEMENTED Python and shell files gain a version tag. The tag format is undecided (Appendix C). Do not implement until that format is chosen.
    • acceptance: The format is written in this spec before any scanner ships.

7.4. mdoctor

Status: Implemented

Implements section 4.2 and the pymarkdown keys in section 5.1. Settings load at construction. The default settings path is the packaged template fiona-settings.json. Callers, including run_sanity.py, may pass another path. A missing file means plain pymarkdown defaults.

  • 7.4.1. mdoctor fix repairs an auto-fixable violation such as MD047 and lists the path.
    • acceptance: The file ends with a newline and stdout contains PyMarkdownFix: 1 file(s) fixed.
  • 7.4.2. A tree with nothing to fix prints PyMarkdownFix: 0 file(s) fixed.
    • acceptance: That line is the only stdout line.
  • 7.4.3. A scan does not ask PyMarkdown to run git. When respect_gitignore is true and the path is inside the working directory, directory names from .gitignore and .git are exclude patterns. A path outside the working directory is still skipped by the directory-name check after the scan. respect_gitignore false still scans those directories, except .git when the path is inside the working directory.
    • acceptance: scan_path is called with respect_gitignore false. Inside the working directory the exclude list contains .git/ and each plain directory name from .gitignore.

7.5. fmlint

Status: Implemented

Implements sections 4.3 and 6.2.

  • 7.5.1. required_fields in settings replaces the default pair on the next run.
    • acceptance: Adding title to the list reports a file that previously passed.
  • 7.5.2. enabled: false reports no frontmatter violations.
    • acceptance: A file with no frontmatter is clean for fmlint.
  • 7.5.3. A code in ignore_rules is omitted. A glob in ignore_file skips those paths.
    • acceptance: ignore_rules: ["FML002"] hides a missing-block finding and still reports a missing name.

7.6. mdlinter

Status: Implemented

Implements section 6.3. Checks run in registry order. A new check is a method plus an entry in that registry. lint_file resolves wikilinks against the file's parent tree. lint_directory scans the wiki.

  • 7.6.1. The report header is Markdown Lint Violations and the section order matches section 6.1.
    • acceptance: DEFINITIONS lists MDL001: Dead Link when a dead markdown link exists.

7.7. pycustomlint

Status: Implemented

Implements section 6.4. The sanity runner in section 7.14 includes it as pycustomlint. There is no fiona-assistant pycustomlint command; the repository runner calls the library.

  • 7.7.1. --json prints a JSON list of violation objects.
    • acceptance: json.loads succeeds and each object has code.

7.8. python-ruff

Status: Implemented

Implements section 4.4. The sanity check name is ruff in run_sanity.py and python-ruff in wiki sanity.

  • 7.8.1. An unused import appears in DEFINITIONS, FILE, and SUMMARY.
    • acceptance: The FILE line contains the ruff rule code.
  • 7.8.2. NOT_IMPLEMENTED A python-ruff fix subcommand mirrors mdoctor fix and calls the existing fix=True path.
    • acceptance: An auto-fixable ruff finding is gone after the command and the path is listed.

7.9. Shellcheck

Status: Planned

  • 7.9.1. NOT_IMPLEMENTED Wrap the shellcheck binary and print the shared report.
    • acceptance: A known shellcheck finding shows file, line, and rule code.
  • 7.9.2. NOT_IMPLEMENTED A missing binary prints an install hint and exits non-zero, with no traceback.
    • acceptance: The message names shellcheck.
  • 7.9.3. NOT_IMPLEMENTED Wiki sanity runs this check.
    • acceptance: The Results: table includes the new check name.

7.10. Markdown Code Blocks

Status: Planned

  • 7.10.1. NOT_IMPLEMENTED Extract fenced blocks. Lint Python blocks with Ruff and shell blocks with shellcheck. Skip other languages.
    • acceptance: An unused import inside a Python fence is reported. Prose outside fences is not.
  • 7.10.2. NOT_IMPLEMENTED The finding names the markdown path, the fence index, the line inside the fence, and the markdown line number.
    • acceptance: The reported markdown line is the fence line that contains the import.
  • 7.10.3. NOT_IMPLEMENTED Wiki sanity runs this check.
    • acceptance: The check appears in the Results: table.

7.11. json-lint

Status: Implemented

Implements sections 4.5 and 6.5. Both sanity runners include it.

  • 7.11.1. A JSON violation fails run_sanity.py and wiki sanity.
    • acceptance: The Results: row for jsonlint is FAIL and the process exit code is non-zero.
  • 7.11.2. An empty tree of JSON files prints no violations.
    • acceptance: The report says no violations were found.

7.12. Git Operations

Status: Implemented

Implements section 4.6. wiki init writes the repository .gitignore patterns and personal/ (section 7.1.1). Those paths are never staged by these commands except where git add --all respects that ignore file.

  • 7.12.1. save --description note on a dirty tree creates one commit whose message starts with the UTC timestamp and ends with note.
    • acceptance: git log -1 --format=%s matches <YYYY-MM-DD HH:MM:SS> note.
  • 7.12.2. snapshot demo --description note then restore demo returns file bytes to the tagged tree and adds a new commit. The pre-restore history is still reachable.
    • acceptance: git rev-parse HEAD~1 after restore is the snapshot commit or a descendant, and the tagged tree matches the worktree.
  • 7.12.3. undo and restore on a dirty tree exit 1 and mention clean.
    • acceptance: The HEAD sha is unchanged.
  • 7.12.4. clean leaves git status empty of changes.
    • acceptance: A following status prints Status: clean.

7.13. Backup Script

Status: Planned

  • 7.13.1. NOT_IMPLEMENTED A bundled command writes a restorable archive of the wiki.
    • acceptance: Restoring that archive reproduces the wiki tree from before the archive.
  • 7.13.2. NOT_IMPLEMENTED The archive is stored outside the wiki tree.
    • acceptance: The archive path is not under the wiki root.
  • 7.13.3. NOT_IMPLEMENTED The filename is YYYYMMDD_hhmm_<description>.zstd.
    • acceptance: The file is valid zstd and the name matches.
  • 7.13.4. NOT_IMPLEMENTED The command asks for a directory and uses ~/Downloads when the user accepts the default.
    • acceptance: Pressing Enter writes the archive under ~/Downloads.

7.14. Test and Sanity Runners

Status: Implemented

  • 7.14.1. Shared engine, used by run_sanity.py and wiki sanity.
    • 7.14.1.1. One thread per check. Results stay in registry order.
    • 7.14.1.2. Summary line PASS/FAIL - <detail> (MM:SS). Minutes may exceed 59 and are zero-padded. A missing duration omits the suffix.
    • 7.14.1.3. SANITY PASSED (MM:SS) lists check names in registry order. Sanity Failed - N checks need attention: has no total time.
    • 7.14.1.4. A runner may add an extra result row, such as coverage, and per-check failure details.
    • 7.14.1.5. progress(name, result) runs when a check finishes. heartbeat(remaining_names) runs about every 0.5 seconds. Start and finish are logged at INFO.
  • 7.14.2. Progress display.
    • 7.14.2.1. On a TTY, one bar per check, redrawn in place with cursor-up and line clear. Example: pytests [###------------] 22% 6.5s.
    • 7.14.2.2. The bar fills as elapsed / expected-seconds, capped at 100%.
    • 7.14.2.3. A finished check shows a full bar and (MM:SS): markdownlint [###############] 100% done (00:07).
    • 7.14.2.4. The final block stays on screen above the reports.
    • 7.14.2.5. --non-interactive, also when stdout is not a TTY, prints Running Sanity or Running Wiki Sanity once, then one dot per interval seconds. A run shorter than the interval prints no dots. A newline ends the line before the report. No ANSI.
    • 7.14.2.6. --json on wiki sanity skips the display.
    • 7.14.2.7. Bar scale and dot interval come from section 5.2.
  • 7.14.3. run_tests.py and the pytests check.
    • 7.14.3.1. The command includes --color=no and JUnit XML junit_family=xunit1.
    • 7.14.3.2. Displayed messages prefer regex-parsed pytest output. JUnit fills missing fields and replaces empty messages. Match by basename plus test name so path spellings do not duplicate a failure.
    • 7.14.3.3. A missing short summary logs a WARNING that names PYTEST_ADDOPTS, --no-summary, and plugin output as likely causes.
    • 7.14.3.4. LOG_LEVEL=DEBUG logs the pytest command, return code, parsed failures, and raw output when the parse disagrees.
    • 7.14.3.5. The report nests failures as Class : test_name under the file, then the assert text, or the exception text when there is no assert line.
    • 7.14.3.6. The Sanity Failed verdict lists failing test names only. Assert text stays in the report body.
    • 7.14.3.7. The coverage: line is PASS/FAIL - <percent>% (required <percent>%) and is untimed. A shortfall is its own verdict entry.
  • 7.14.4. run_sanity.py check order is pytests, markdownlint, ruff, frontmatter_lint, pycustomlint, jsonlint.
  • 7.14.5. A failing test is listed under its file with the assert text when pytest is run with --no-summary or PY_COLORS=1.
    • acceptance: The name and assert still appear.
  • 7.14.6. A green run prints every Results: line with (MM:SS) and ends with SANITY PASSED (MM:SS).
    • acceptance: The banner time is the wall time of the whole run.
  • 7.14.7. A piped run contains no ANSI and prints one dot per interval after the label, unless the run is shorter than the interval.
    • acceptance: The captured stdout has no ESC byte.

7.15. html-js-lint

Status: Planned

The html-js-lint console script is not installed until this section is built. The intended tool validates HTML with html5validator (Java 8+) and JavaScript with eslint on PATH.

  • 7.15.1. NOT_IMPLEMENTED HTML files (default .html and .htm) are checked. Findings include path, line, and column. .gitignore and read-only zones are skipped.

    • acceptance: An unclosed tag is reported. A valid file is not. A tree with no HTML prints no HTML findings.
  • 7.15.2. NOT_IMPLEMENTED JavaScript files (default .js and .javascript) are checked with eslint, honoring .eslintrc.

    • acceptance: A syntax error names path, line, and message. A clean file produces no finding.
  • 7.15.3. NOT_IMPLEMENTED Inline <script> is linted only when html.lint_inline_js is true (default false). Findings use the HTML path and line.

    • acceptance: With the flag off, inline script is ignored.
  • 7.15.4. NOT_IMPLEMENTED CLI is fiona-assistant html-js-lint, default chklint ., shared report, exit 0 even with violations, and --fix where the underlying tool can fix.

    • acceptance: --fix reports how many files changed.
  • 7.15.5. NOT_IMPLEMENTED Settings, under a dedicated section, with defaults when the section is missing:

    Key Default
    html.extensions ["html", "htm"]
    html.ignore_patterns []
    html.lint_inline_js false
    javascript.extensions ["js", "javascript"]
    javascript.eslint_config .eslintrc
    javascript.ignore_patterns []
    • acceptance: A custom extension list is the set that is scanned.
  • 7.15.6. NOT_IMPLEMENTED Both run_sanity.py and wiki sanity run this check.

    • acceptance: A violation fails that sanity run.

7.16. settings setup

Status: Planned

fiona-assistant init remains the non-interactive wiki bootstrap (section 7.1.1): it copies the template and does not ask questions. fiona-assistant setup is a separate command for reviewing settings on an existing wiki. It does not create the wiki tree.

  • 7.16.1. NOT_IMPLEMENTED setup walks every configurable key in fiona-settings.json, shows the current value, and accepts Enter as "keep current".
    • acceptance: Enter through every prompt leaves the file semantically unchanged.
  • 7.16.2. NOT_IMPLEMENTED Booleans can be toggled. Strings and integers accept a typed value. Values are checked before write.
    • acceptance: A non-integer for indent_len is rejected and the file is not written.
  • 7.16.3. NOT_IMPLEMENTED --yes or --default applies defaults with no prompts.
    • acceptance: The command exits 0 on a pipe when the flag is set.
  • 7.16.4. NOT_IMPLEMENTED A missing settings file is created from the template defaults. A corrupt file is reported and not silently replaced.
    • acceptance: Exit 0 after creating a missing file. Exit non-zero on corrupt JSON, with the parse error on stderr.
  • 7.16.5. NOT_IMPLEMENTED setup can be run again later to change an existing wiki.
    • acceptance: A second run can change log_level and leave every other key as it was.

7.17. Logging

Status: Implemented

The contract is section 2. Reports stay on stdout. Diagnostics go through the logging module to stderr and, when configured, to the file sink.

  • 7.17.1. Each CLI entry calls the shared setup before doing work.
    • acceptance: LOG_LEVEL=DEBUG emits debug lines on stderr and the human report stays on stdout.
  • 7.17.2. Dropping file_log_location leaves no file handler.
    • acceptance: No log file is created for that run.

7.18. Documentation

Status: Partially Implemented

README.md links to this spec. src/README.md, src/fiona/librarian/README.md, and src/fiona/mdoctor/README.md exist and are behind the current CLIs.

  • 7.18.1. NOT_IMPLEMENTED librarian/README.md documents list and toc. mdoctor/README.md documents list, chklint, and fix.
    • acceptance: A new reader can install and run each command from those files alone.
  • 7.18.2. src/README.md describes src/fiona/ (including wiki/, python_ruff/, and data/llm_wiki/), repo-root tests/, and scripts/ (run_tests.py, run_sanity.py, settings_dev.json).
    • acceptance: Those names appear in the structure section.
  • 7.18.3. Documentation markdown passes mdoctor chklint.
    • acceptance: README.md contains a markdown link to SPEC.md and the lint report is clean.

7.19. Tool Roles

Status: Planned

  • 7.19.1. NOT_IMPLEMENTED Each src/ tool has one role: ingestion (ingest/ to raw/), processing (raw/ to wiki/concepts/), or maintenance (index.md / log.md and the linters).
    • acceptance: A table in this spec or the README assigns every tool exactly one role.
  • 7.19.2. NOT_IMPLEMENTED Ingestion is wiki ingest (section 7.1.5). Missing processing steps become scripts under the template utils/ directory, such as process.py.
    • acceptance: Each new script is named in the role table.

7.20. Wiki Root Discovery

Status: Planned

Today every optional directory defaults to .. toc file * already searches for fiona-settings.json as specified in section 7.3.1. Graph settings already walk parent directories. The dispatcher does not.

  • 7.20.1. NOT_IMPLEMENTED When the directory argument is omitted, commands use the wiki root from Appendix C's discovery rule.
    • acceptance: fiona-assistant mdoctor from a subdirectory lints that root, not the subdirectory.
  • 7.20.2. NOT_IMPLEMENTED An explicit directory always wins.
    • acceptance: mdoctor chklint sub/dir lints sub/dir even when a settings file exists higher up.
  • 7.20.3. NOT_IMPLEMENTED With no settings file found, the default stays ..
    • acceptance: Behavior matches today's commands.
  • 7.20.4. NOT_IMPLEMENTED The file is loaded once per process and shared. mdoctor already reads it for rules.
    • acceptance: Two tools in one invocation see the same object.

7.21. Bootstrap install modes

Status: Implemented

bootstrap (bash, sourced) and bootstrap.ps1 (PowerShell) prepare a Python environment and install this repository. With no install flag, both create or reuse a virtual environment and run python -m pip install -e ".[dev]". scripts/bootstrap_install.py is the pip-argument source both scripts call.

  • 7.21.1. --local installs this repository as a normal package. The pip arguments are install and .. They do not include -e and they do not include the dev extra.
    • acceptance: pip_install_args(local=True, no_env=False) is ["install", "."].
  • 7.21.2. --no-env does not create or activate a virtual environment. The install adds --user, so the console scripts land in the user scripts directory and run without activating an environment.
    • acceptance: pip_install_args(local=False, no_env=True) contains --user. PowerShell skips virtual environment creation when $NoEnv is set. Bash skips it when NO_ENV is true.
  • 7.21.3. --local and --no-env combine. The install is install, --user, and ., with no -e.
    • acceptance: pip_install_args(local=True, no_env=True) is ["install", "--user", "."].
  • 7.21.4. Neither flag keeps the default: -e and .[dev], and no --user.
    • acceptance: pip_install_args(local=False, no_env=False) is ["install", "-e", ".[dev]"].
  • 7.21.5. Bash adds --break-system-packages on every install. PowerShell does not.
    • acceptance: break_system_packages=True inserts that flag after --user when both are set, and after install when --user is absent.
  • 7.21.6. A virtual-environment install on PowerShell must be dot-sourced so the environment stays active. --no-env may be executed with -File, because there is no environment to keep active. Bash is sourced for both modes.
    • acceptance: PowerShell reports a dot-source error when --local is used without --no-env and the script is not dot-sourced. The message does not call --local an unknown option.

Appendix A. Non-Goals

  • A.1. No LLM or network API inside Fiona-Assistant. The tools are deterministic.
  • A.2. No automated edit of personal/ or raw/ content. wiki ingest adds new files under raw/ and edits none (1.2.3.).
  • A.3. No cloud sync and no remote backup. Version control is local. The planned archive in 7.13. is a local file.
  • A.4. Deterministic tools make no cloud API calls. Each command uses only local files and finishes with no network access.
    • acceptance: With the network unavailable, wiki sanity and wiki graph still complete on a local wiki.
    • acceptance: Opening the generated graph HTML is an exception: the browser loads D3 from d3js.org (7.1.4.). The generator itself does not call that host.
    • acceptance: wiki ingest <url> is the other exception. It downloads only the URL the user passed (7.1.5.2.). With a local path it makes no network call.

Appendix B. Priority Order

Priority Feature Section Status
P1 wiki update / check-update 7.1.2 Implemented
P1 Git save, snapshot, restore, clean, undo, status 7.12 Implemented
P1 wiki sanity, including dead links 7.1.3 Implemented
P1 housekeeping 7.2 Implemented
P1 wiki graph 7.1.4 Partially Implemented
P2 fmlint 7.5 Implemented
P2 json-lint 7.11 Implemented
P2 mdoctor fix 7.4 Implemented
P2 Settings schema 5 Implemented
P2 Sanity reporting, coverage, progress 7.14 Implemented
P2 Shellcheck 7.9 Planned
P2 Markdown code-block lint 7.10 Planned
P2 html-js-lint 7.15 Planned
P2 python-ruff fix subcommand 7.8.2 Planned
P2 Wiki root discovery 7.20 Planned
P2 settings setup 7.16 Planned
P2 Bootstrap install modes 7.21 Implemented
P2 wiki ingest 7.1.5 Implemented
P3 fmlint enforcement of quality signals 6.2 Planned
P3 Backup archive 7.13 Planned
P3 Documentation refresh 7.18 Partially Implemented
P2 Graph skips gitignored files 7.1.4.17 Planned
P3 Node importance filter 7.1.4.16 Planned
P4 Non-markdown version tags 7.3.2 Planned
P4 Tool-role table 7.19 Planned

Appendix C. Open Questions

  • Python and shell version-tag format (section 7.3.2) is undecided. Do not build a scanner until this section names the format.
  • Wiki root discovery when several fiona-settings.json files exist under the start directory: the rule is not chosen. Candidates are "the nearest directory wins" and "error when two files sit at the same depth". Parent walk is already what graph settings do, and is a third candidate for the default directory. Section 7.20 stays planned until one rule is written here.
  • json-lint chklint exits 1 when violations exist. mdoctor, fmlint, and python-ruff exit 0. A future change may align them. Until this line changes, implement the split in section 4.
  • wiki ingest takes one file or URL. A directory argument, and a bare wiki ingest that drains ingest/, are not specified.
  • More converters for section 7.1.5, each one class under 7.1.5.1.2.: Office files (mammoth for docx, python-pptx, openpyxl), or markitdown as one optional extra that covers them all but pulls in magika and onnxruntime. OCR for scanned PDFs is not chosen. PDF tables need pdfplumber or similar; pypdf reads text only.
  • Images and other assets referenced by an ingested source are not copied into raw/assets/.
  • Third-party converters could register through a package entry point group such as fiona.ingest_converters. Not decided.
  • The SHA-256 check in 7.1.5.3.3. scans raw/ on every run. A cached hash index is not specified until a large raw/ makes the scan slow.

Appendix D. Maintainer Notes

These notes are not requirements. They point at the current tree.

Area Module
CLI dispatch src/fiona/__main__.py
Logging src/fiona/logging_config.py
template path src/fiona/paths.py
librarian list src/fiona/librarian/librarian.py
TOC src/fiona/librarian/toc_generator.py
mdoctor src/fiona/mdoctor/mdoctor.py
fmlint, mdlinter, pycustomlint, json-lint src/fiona/customlinter/
ruff wrapper src/fiona/python_ruff/python_ruff.py
wiki init src/fiona/wiki/wiki.py
wiki gitignore src/fiona/wiki/ignore.py
wiki update src/fiona/wiki/update_manager.py
wiki sanity src/fiona/wiki/sanity_manager.py
housekeeping src/fiona/wiki/housekeeping.py
wiki graph src/fiona/wiki/graph_generator.py, src/fiona/wiki/templates/graph_template.html, src/fiona/wiki/templates/graph_physics.js
git src/fiona/gittool/git_tool.py
sanity engine src/fiona/sanity_runner/sanity_runner.py
runners src/fiona/run_tests.py, src/fiona/run_sanity.py
launchers scripts/run_tests.py, scripts/run_sanity.py
bootstrap bootstrap, bootstrap.ps1, scripts/bootstrap_install.py

Layout for section 7.1.5, under src/fiona/wiki/ingest/:

Module Classes Role
models.py FetchedSource, ConversionResult, IngestResult Dataclasses passed between stages. IngestResult.to_dict() is the JSON in section 4.7.6.
fetchers.py SourceFetcher (ABC), LocalFileFetcher, UrlFetcher accepts(argument) and fetch(argument, dest_dir). UrlFetcher takes the opener.
media.py detect_media_type, make_slug 7.1.5.2.4. and 7.1.5.2.6.
converters/base.py SourceConverter (ABC), ConverterRegistry, ConversionError Class attributes name, media_types, extensions, raw_category; convert(source). ConverterRegistry.find(media_type, extension).
converters/markdown.py, text.py, html.py, pdf.py MarkdownConverter, TextConverter, HtmlConverter, PdfConverter One format each.
archiver.py RawArchiver 7.1.5.3.; find_by_sha256(digest, size) scans raw/ for a reusable copy.
page_writer.py RawPageWriter 7.1.5.5.
log_writer.py IngestLog 7.1.5.6.
pipeline.py IngestPipeline, default_pipeline() Runs the stages; list_pending(wiki_root) is 7.1.5.9.

Known defects to avoid copying blindly:

  • json-lint fix walks the file list twice. The observable contract is one count and one path line per fixed file.
  • The previous spec's graph physics (mass = 1 + in_degree * 0.5, center 0.1, alphaMin 0.1, velocityDecay 0.6, and "D3 or vis.js") does not match the template. Section 7.1.4 is the contract.
  • The previous spec's external-link JSON used {url, count} objects. The implementation emits unique URL strings plus a total count.
  • pymarkdown rule typos fail silently (section 5.1).
  • mdoctor fix does not skip read-only zones (section 4.2).