Files
nvaldez de28441943 feat: sync Nokia fork product through SPEC 1.18 (v1.16.0)
MEMORY.md and skill memory template, ingest auth and html_sweep, TOC slug fix, run_tests CI output, and matching tests. Excludes GitLab CI and publishing docs.
2026-10-19 09:12:04 +02:00

96 KiB

Fiona Project TODO List

Ordered by priority. Each item references its section in SPEC.md (the source of truth for requirements and acceptance criteria).

Done

  • Include the llm-wiki template as part of the PyPI package
  • MD frontmatter parser for version control of core files (SPEC 6.2)
    • version tag support
  • fiona-assistant wiki init — copy packaged template into a target directory (basic flow; hardening tracked under P1)
  • Windows PowerShell bootstrap (bootstrap.ps1): repo-local venv under .venvs\<name>, same flags as the bash bootstrap

P0 — Packaging and Structure Fixes

From the 2026-09-23 structure audit, plus the 2026-09-29 layout audit below. These are repo-hygiene items that block a correct install or make the docs contradict the code.

  • Fix PyPI packaging so a non-editable install works (wheel build confirmed python_ruff is missing entirely)
    • Add python_ruff to the packages.find include list in pyproject.toml
    • Add src/python_ruff/__init__.py
    • Add src/fiona/__init__.py
    • Re-verify: build a wheel, install it fresh, confirm the fiona-assistant console script runs and the llm-wiki template data ships
  • Resolve the duplicated raw/ zone in the llm-wiki template
    • Pick one canonical location (llm-wiki/raw/ at the wiki root, per user decision) and remove src/llm-wiki/wiki/raw/
    • Align root AGENTS.md, src/llm-wiki/AGENTS.md, and wiki/schema.md on the same location
  • Reconcile the two competing frontmatter schemas
    • Decide between project agent rule 10.1 (name/version/title) and wiki/schema.md (title/sources/tags); one schema wins: name/version required, title optional (user decision, 2026-09-25; documented in SPEC 6.2)
    • Add missing type values (e.g. toc) to the allowed types list (project agent rule 10.1, wiki/schema.md, SPEC 6.2)
    • Stamp frontmatter on the template md files that lack it; only personal/README.md and raw/README.md were missing frontmatter (all other template files already carry name/version; title is optional, so wiki/index.md, wiki/log.md, and the other READMEs already conform)
  • Update the root AGENTS.md structure diagram to match reality: llm-wiki lives at src/llm-wiki/ (packaged template), and src/ also contains wiki/, run_tests.py, run_sanity.py, settings_dev.json, utils.sh, install_packages.sh
  • Raise test coverage from 87% to the required 90% (settings_dev.json) so run_sanity.py can pass
  • Document the src/llm-wiki/__init__.py setuptools hack (hyphenated, non-importable data-only "package") in pyproject.toml comments
  • Remove src/llm-wiki/utils/scripts/example.py — won't do: wiki init no longer skips .py files (it now makes a 1:1 copy of the template, so example.py ships to user wikis; user decision, 2026-09-27; SPEC 7.1.1, version 0.30.0)
  • Deduplicate conftest: src/conftest.py vs src/tests/conftest.py
    • Consolidated into src/tests/conftest.py (sys.path bootstrap moved there, outer file removed, 2026-09-27); verified python src/run_tests.py (619/619), python -m pytest without PYTHONPATH, and python src/run_sanity.py all pass — the 6055ac6 revert was a misdiagnosis: the then-failing sanity was the terminal-width issue below, not the conftest change
  • run_sanity.py run fails, need fixing
    • run sanity should show what exactly to fix
      • pytests should show which test are failing,
      • markdownlint should show which files are in violation , what line
      • same thing for ruff
  • Remove the phantom html_js_lint package and html-js-lint console script from pyproject.toml until SPEC 7.15 is implemented. The directory src/html_js_lint/ does not exist. Detail is item 2 in the 2026-09-29 audit below

2026-09-29 structure audit

pyproject.toml declares a PEP 621 project and the code lives under src/, but the tree is nine sibling import packages plus loose scripts. A src layout is meant to install one import package. Target shape when this cluster is done:

src/fiona/
├── __main__.py
├── librarian/
├── mdoctor/
├── customlinter/
├── python_ruff/
├── gittool/
├── sanity_runner/
├── wiki/
└── data/llm_wiki/     # templates, no fake package
tests/                 # repo root, not installed

Keep a single required entry point: fiona-assistant = fiona.__main__:main. Optional thin scripts can stay as extra entry points that call into those submodules. Do this as one change set: import paths, tests, entry points, and docs move together. Half-moving a package leaves both layouts on sys.path.

  • Collapse the nine top-level packages into fiona subpackages. Imports, pyproject.toml, tests, and docs moved with the directories.
    • Current install names, from packages.find include and from src/fiona_assistant.egg-info/top_level.txt: customlinter, fiona, gittool, librarian, llm-wiki, mdoctor, python_ruff, sanity_runner, wiki
    • fiona/__main__.py imports them as top-level modules (from librarian.librarian import main, from wiki.wiki import wiki_init, and the same for customlinter, gittool, mdoctor, python_ruff)
    • Same-name module pattern to preserve while moving: librarian/librarian.py, mdoctor/mdoctor.py, python_ruff/python_ruff.py, sanity_runner/sanity_runner.py. gittool is the odd one: the module is git_tool.py
    • wiki is a generic top-level name and will collide with any other installed project that also ships a wiki package
    • After the move, packages.find include should list fiona only. An allow-list is how python_ruff was shipped late and how html_js_lint stayed declared after the directory vanished
  • Remove the phantom html_js_lint declaration (same checkbox as the item above this section)
    • [project.scripts] maps html-js-lint to html_js_lint.html_js_lint:main
    • packages.find include lists html_js_lint
    • src/fiona_assistant.egg-info/entry_points.txt still has that script from the last editable install
    • src/html_js_lint/ does not exist. Invoking html-js-lint fails at import. SPEC 7.15 is the future feature; the entry point should not ship until that directory exists
    • html5validator is in runtime dependencies and is not imported anywhere under src/. Drop it with the phantom script, or keep it only when SPEC 7.15 lands
  • Delete src/__init__.py
    • The file's only content is the comment # Fiona tools package
    • An __init__.py in src/ makes src itself a package whenever the repo root is on sys.path. The src layout exists so packages are importable only after install
    • That is why bytecode for run_sanity.py and run_tests.py lands in src/__pycache__/ (run_sanity.cpython-313.pyc, run_tests.cpython-313.pyc, __init__.cpython-313.pyc)
  • Stop tests from editing sys.path
    • src/tests/conftest.py does sys.path.insert(0, os.path.dirname(os.path.dirname(...))), which puts src/ on the path, then imports librarian, mdoctor, python_ruff, and wiki as top-level names
    • src/tests/__init__.py exists, so tests is a package sitting inside the source root. Standard layout is a repo-root tests/ directory that is not installed
    • [tool.pytest.ini_options] sets testpaths = ["src/tests"] and has no pythonpath. Coverage source = ["src"] treats the mixed script-and-package directory as the measured tree
    • Replacement: pip install -e . in the dev env, or pythonpath = ["src"] once there is a single fiona package. The path edit hides "package not installed" failures
    • The 2026-09-27 conftest consolidation (outer src/conftest.py removed) stays valid. This item replaces the remaining sys.path bootstrap, it does not undo that merge
  • Decide where the loose src/ files live
    • src/run_sanity.py and src/run_tests.py are modules, not packages. They are absent from packages.find, so a wheel does not install them. They are how this repo runs its own sanity and tests (python src/run_sanity.py)
    • src/install_packages.sh, src/utils.sh, and src/settings_dev.json sit in the same directory. Dev config and shell helpers are not package source
    • Either give run_sanity / run_tests real entry points under fiona, or move them to a repo-root scripts/ (or tools/) directory that is not on the package path
  • Replace the hyphenated llm-wiki package with package data
    • import llm-wiki is a SyntaxError. src/llm-wiki/__init__.py exists only so setuptools detects the directory. The pyproject.toml comment already documents that hack
    • [tool.setuptools.package-data] maps "llm-wiki" = ["*", "**/*"]
    • src/llm-wiki/__init__.py sets __version__ = "0.1.0" while [project] version is 0.36.0. Two versions for one distribution
    • Move the template tree to something importable, for example src/fiona/data/llm_wiki/, and load files with importlib.resources. Update wiki init / wiki update path helpers (get_llm_wiki_source_path) in the same change
    • Root README.md links ./llm-wiki/AGENTS.md. The file is src/llm-wiki/AGENTS.md
  • Get setuptools metadata out of src/
    • src/fiona_assistant.egg-info/ holds PKG-INFO, entry_points.txt, requires.txt, SOURCES.txt, top_level.txt. That is install metadata. It belongs at the repo root (or nowhere, if it is only a local editable-install artifact)
    • .gitignore already ignores *.egg-info/, so this directory should not be committed, but where = ["src"] can still see it during package discovery
    • The egg name fiona_assistant is the setuptools normalization of project name fiona-assistant. Leave the normalization alone; only the directory location is wrong
  • Keep a single coverage config, and point it at the code that exists
    • Both .coveragerc and [tool.coverage] in pyproject.toml exist. Coverage reads .coveragerc first, so the pyproject block is unused
    • .coveragerc still omits src/mdoctor/fmlint.py and src/python_ruff/pycustomlint.py. Those modules live under src/customlinter/ (fmlint.py, pycustomlint.py)
    • Both files share the same exclude_lines list. After deleting .coveragerc, confirm pytest --cov still honors the pyproject source, omit, and exclude_lines
    • coverage.xml at the repo root is a generated report. It is not matched by .gitignore (*.log and .coverage are; coverage.xml is not)
  • Raise the setuptools build pin so [project] is valid
    • [build-system] requires is setuptools>=42
    • The [project] table (PEP 621) needs setuptools 61 or newer. An older setuptools that satisfies >=42 can fail the build
    • Set requires = ["setuptools>=61"] (or a current floor) and rebuild the wheel
  • Sort runtime dependencies from repo-only tools
    • ruff is imported by src/python_ruff/python_ruff.py, so it stays in dependencies
    • pylint is in dependencies with no version pin, and nothing under src/ imports it. Move it to the dev extra or drop it
    • There is no [tool.ruff] table and no ruff.toml. If this repo is linted with Ruff, the rule set belongs in pyproject.toml
    • requires-python is >=3.8. Local bytecode in src/__pycache__/ is CPython 3.13. Classifiers only say Programming Language :: Python :: 3. The bootstrap examples use 3.11. Record the versions this repo actually runs, and confirm syntax such as str | None is either covered by from __future__ import annotations or by a higher requires-python
  • Make the structure docs match the tree (also listed under P3)
    • src/README.md "Project Structure" lists only fiona/, librarian/, mdoctor/, tests/, and README.md. Missing from that diagram: customlinter/, python_ruff/, gittool/, sanity_runner/, wiki/, llm-wiki/, run_sanity.py, run_tests.py, settings_dev.json, utils.sh, install_packages.sh
    • MDoctor's description in that file is still [Description to be added], and Contributing is still [Contribution guidelines to be added]
    • Refresh this diagram again after the package collapse, so the docs describe src/fiona/ rather than today's sibling packages

Code defects — 2026-09-29 audit

Source review after the package directories moved under src/fiona/. src/ on disk is fiona/ plus README.md. Tests live in tests/. The template lives in src/fiona/data/llm_wiki/. The package-move items below are done. The command defects after them are still open.

Finish the package move

  • Point every import at fiona.<package>
    • src/fiona/__main__.py still imports top-level librarian, mdoctor, customlinter, python_ruff, gittool, and wiki. Those packages are src/fiona/<name>/.
    • tests/test_fiona_main.py still does from librarian import toc_generator.
    • TOOL_LOGGER_NAMES in src/fiona/logging_config.py still lists librarian, mdoctor, python_ruff, and wiki. Loggers are now fiona.* via getLogger(__name__). The parent name fiona already covers them; drop the stale names.
  • Point template and settings paths at fiona/data/llm_wiki
    • get_llm_wiki_source_path in src/fiona/wiki/wiki.py and src/fiona/wiki/update_manager.py looks for a sibling llm-wiki (src/fiona/llm-wiki). The template is src/fiona/data/llm_wiki/. wiki init and wiki update miss it. Load it with importlib.resources (same end state as the P0 llm-wiki package-data item).
    • The default settings path in logging_config.py, customlinter/base.py, mdoctor/mdoctor.py, and librarian/toc_generator.py is parent.parent / "llm-wiki" / "fiona-settings.json", which is src/llm-wiki/.... The file is src/fiona/data/llm_wiki/fiona-settings.json.
    • src/fiona/run_sanity.py still lints ./src/llm-wiki for frontmatter. Point it at src/fiona/data/llm_wiki.
  • Match packaging and the test runner to the new tree
    • pyproject.toml packages.find include and [project.scripts] still name librarian, mdoctor, python_ruff, customlinter, sanity_runner, llm-wiki, wiki, gittool, and html_js_lint. librarian and mdoctor console scripts import modules that are no longer top-level.
    • include = ["fiona"] does not match subpackages. Use fiona* so fiona.wiki and the rest are installed.
    • [tool.pytest.ini_options] testpaths is still src/tests. The suite is tests/.
    • tests/conftest.py walks two directories up and inserts the repo root. That math used to reach src/ when the file lived at src/tests/conftest.py. Insert src/ so import fiona works without an editable install.
    • src/__init__.py is already gone. Leave it gone.
    • run_tests.py and run_sanity.py already sit in src/fiona/. Give them entry points or document python -m fiona.run_tests / python -m fiona.run_sanity in place of python src/run_tests.py (SPEC 4.9).

Fixes in commands the spec already calls Implemented

  • json-lint fix walks the file list twice (SPEC appendix, SPEC 4.5). The second fix_file pass finds the files already clean, so the path lines never print. Walk once: one count and one path line per fixed file.
  • json-lint chklint --fix is parsed and ignored (src/fiona/customlinter/jsonlint.py). SPEC 4.5 says chklint --fix rewrites the same way fix does.
  • json-lint fix treats any path part named personal or raw as a read-only zone, including a folder outside the wiki. Skip with the wiki-relative check in BaseLinter (src/fiona/customlinter/base.py), the same rule as chklint.
  • git failures print Error: on stdout and catch Exception (src/fiona/gittool/git_tool.py). SPEC 4.6 and section 2: the message goes to stderr. Catch the errors GitTool raises. This is the concrete case of the P2 logging item.
  • wiki graph prints Error: Directory not found on stdout (src/fiona/wiki/graph_generator.py). The other graph errors use stderr. SPEC section 4: an invalid directory is stderr and exit 1.
  • wiki graph replaces --wikilinks / --no-wikilinks and --external-links with fiona.wiki.graph settings after the CLI has set them (load_graph_settings in wiki_graph_main). An explicit flag wins. Settings fill the defaults only (SPEC 4.7.5).
  • wiki graph drops files above max_files with a log line only. Print the kept count and the limit on stderr.
  • mdoctor fix rewrites personal/ and raw/ (SPEC 4.2, appendix). Constraint 1.2.3 says those zones are read-only for every automated tool. Skip them in fix_file_programmatically, then change SPEC 4.2 to say the command skips those zones.
  • graph_generator.py logs with f-strings. Use lazy % formatting, the same style as logging_config.py.
  • fiona-assistant python-ruff has an empty help string and a #TODO in src/fiona/__main__.py. Use a one-line help like the other linters.
  • fiona.__main__ rewrites sys.argv and calls each tool main(), which parses the arguments a second time. Call the library functions directly so each command is parsed once.

Manual test findings — 2026-10-02

Black-box test from the README and SPEC only. Report: tests/wikitest/wiki/queries/fiona-assistant-manual-test-2026-10-02.md. Each item below was re-run on 2026-10-02 before it was written down. SPEC 1.8.0 holds the requirements. Order is the build order.

  • Second wiki init prints a Python traceback (SPEC 1.2.7, 7.1.1.4). Reproduced: the FileExistsError reaches the top level. Catch it in the wiki init CLI handler, print the one Error: line on stderr, exit 1. Then sweep the other handlers in src/fiona/__main__.py for expected errors that escape
    • Test: second init, stderr has no Traceback, exit 1, file hashes unchanged
  • Shipped fiona.log_level is INFO, the code default is WARNING (SPEC 7.17.3). Every INFO record shows as a red NativeCommandError in PowerShell even on exit 0. Ship WARNING, keep file_log_level INFO, and update the shipped document in SPEC 5.1
    • Test: fresh wiki, clean wiki sanity and a local wiki ingest write nothing on stderr; fiona.log has INFO
    • Existing wikis get this through wiki update as a fiona-settings-update.json conflict. Check the housekeeping merge action is clear enough for that one key
  • Ingest log bullet fails the wiki's own lint (SPEC 7.1.5.6.1). Reproduced in the test wiki: wiki/log.md has MD013 (112 and 156 chars) and MD034 (bare origin URL). Put the origin in a code span and each -> <path> on a two-space continuation line in log_writer.py
    • Test: 90-char URL ingest, then wiki sanity has no markdownlint hit in wiki/log.md; still one bullet per run
  • mdoctor chklint and mdoctor fix reject a file path (SPEC 4.2, 7.4.4). Reproduced: both print is not a valid directory. Accept a single .md file the way fmlint chklint does
    • Test: fix on one file lists only that file; other hashes unchanged; missing path gives does not exist
  • Post-ingest hint for leftover style violations (SPEC 7.1.5.5.6). Depends on the mdoctor file item above. Print Next: fix the style violations listed by fiona-assistant mdoctor chklint <page>. before Next: refine
  • "Relative path fails on Windows" is a misdiagnosis (SPEC 7.1.5.2.7). ./ingest/x.md and .\ingest\x.md both work from the wiki root. The tester's file was already moved to raw/articles/tester-note.md by the earlier absolute-path run (SPEC 7.1.5.3.6), so the second run failed. Make the error say so: add (moved to <raw path> by an earlier ingest) when the missing path is under ingest/ and raw/ has that name
    • Test: both separators ingest; second run gets the hint
  • --dry-run logs archived <raw path> although nothing is written (SPEC 7.1.5.8.1). Reproduced. Log dry run: would archive / dry run: would write instead
  • Housekeeping sanity details repeat the whole PyMarkdown report (SPEC 7.2.5.1). One line per violation: <check>: <path>:<line>:<column> <code> <message>
  • Housekeeping logs WARNING Overwriting existing file: wiki_graph.html on every run (SPEC 7.1.4.5.9.1). Add wiki graph -f/--force; housekeeping passes it and logs at INFO
  • fiona-assistant --version (SPEC 2.3). Reproduced: argparse rejects it. Print the same string as ingest.tool
  • README "After ingest" and "On Windows" notes (SPEC 7.18.4), and the same after-ingest loop in the Fiona skill. Bump both versions
  • Wiki root discovery from wiki/ (SPEC 7.20) is already tracked under P2. The tester calls it daily friction for agents. It is blocked on the Appendix C rule; decide that rule, then consider moving SPEC 7.20 to P1
  • librarian move is already tracked under P1. The tester hit invalid choice; no new action, since SPEC 4.1.2 marks it NOT_IMPLEMENTED and the README does not list it
  • Decide whether git undo and git clean prompt on a TTY (SPEC Appendix C). The tester lost a git save commit with git undo. Listed under P4 as well

Ingest of a URL that needs a login

Users paste links to Jenkins, Confluence, SharePoint, and ticket pages that need a login. Today a 401 or 403 exits 1 with the generic Download failed, and an SSO redirect that ends on a 200 login page is archived under raw/ and converted as if it were the source. SPEC 1.9.0 holds the requirements (7.1.5.2.8, 7.1.5.2.9, 7.1.5.14). Spec written 2026-10-05; built 2026-10-07 (SPEC 1.13.0). Order is the build order.

  • Detect a login in the URL fetcher (SPEC 7.1.5.2.8). In src/fiona/wiki/ingest/fetchers.py, raise a new AuthRequiredError(IngestError) that carries the URL and the reason, before anything is written to ingest/
    • HTTP 401 and 403 from HTTPError: reason <code> <phrase>. Other non-2xx stay Download failed (SPEC 7.1.5.2.3)
    • Redirect check: final URL differs from the requested URL and <host><path> matches login_url_patterns (case-insensitive, query string ignored)
    • Login form check: text/html body with an <input> whose type is password, any case or quoting. Run it on the bytes already read, so the size limit still applies first
    • Tests with a fake opener, no network: 401, 403, redirect to /login?from=..., 200 page with <INPUT TYPE='password'>, a /docs/login page with no redirect that still ingests, and a 404 that still exits 1. Each auth case leaves ingest/, raw/, wiki/, and wiki/log.md unchanged
  • login_url_patterns setting (SPEC 5.1). Add it to src/fiona/wiki/ingest/settings.py with the default list from SPEC 5.1. A user list replaces the default. A regex that does not compile is skipped with one warning
    • Tests: default list, user override, one bad regex
  • Exit 4 and the two stderr lines (SPEC 4.7.6). Catch AuthRequiredError in the ingest CLI handler, print the Error: Authentication required: <url>: <reason> line and the Next: line, exit 4. URL without userinfo. --json stdout stays empty (SPEC 7.1.5.13); --dry-run exits 4 the same way
    • Tests: exit code, both stderr lines, empty stdout with --json, dry run, no Traceback (SPEC 1.2.7)
  • --origin URL (SPEC 7.1.5.2.9). Valid only with a local file, http/https only, both checks before fetch. Sets ingest.origin, origin_kind: url, final_url: null, feeds the html converter's base URL and the log bullet. Slug stays the file stem
    • Tests: saved HTML with a relative link /a becomes https://host/a; URL source plus --origin exits 1; bad scheme exits 1; no network call in any case
  • Fiona skill (SPEC 7.1.5.14). In src/fiona/data/llm_wiki/utils/skills/fiona/SKILL.md: add exit 4 to the section 4.6 exit table and --origin to its options; rewrite the authentication bullet in section 16.2 into the three steps (use a tool with access, else ask the user to save it into ingest/, never ask for credentials) with the --origin command. Bump the skill version
  • README ingest notes (SPEC 7.18.4): one line on exit 4 and --origin
  • Verify the spec (project agent rule 2.1 step 4): drop the NOT_IMPLEMENTED markers on 4.7.6, 7.1.5.2.8, 7.1.5.2.9, 7.1.5.14, and the login_url_patterns prose in 5.1; set the Appendix B row to Implemented; bump SPEC
  • [-] Open questions left in SPEC Appendix C, not part of this work: credential support (cookie file, token, --header) and a JSON error object on exit 4

HTML block sweep for ingest

wiki ingest of a landing page kept only one testimonial card, because the html converter takes the first <article> as the whole page (repro: tests/wikitest/raw/articles/agent-harness.html, page wiki/sources/agent-harness.md). SPEC 1.10.0 replaces that with a block sweep that scores every block as keep, review, or drop (SPEC 7.1.5.4.5.6, settings in 5.1). Spec written 2026-10-07; built 2026-10-07 (SPEC 1.13.0). Order is the build order. Bootstrap first (. .\bootstrap.ps1) and load the python-tdd skill.

  • Settings (SPEC 5.1). In src/fiona/wiki/ingest/settings.py add html_min_words: int = 4 and html_max_link_density: float = 0.5 to IngestSettings and default_settings(). In load_ingest_settings() read the nested html map: min_words is an int >= 1, not a bool; max_link_density is an int or float in 0..1, not a bool. Anything else keeps the default
    • Tests: defaults, valid override, 0, true, "4", 1.5 each fall back
  • Hand the settings to the converter. HtmlConverter.__init__ takes min_words and max_link_density with the same defaults, so HtmlConverter() still works in existing tests. default_registry(settings: IngestSettings | None = None) registers HtmlConverter(...) as an instance (ConverterRegistry.register already accepts instances). In default_pipeline() load the settings once, before the registry, and pass the same object to both
    • Test: a wiki with min_words: 2 makes a three-word <div> keep through the full pipeline
  • New module src/fiona/wiki/ingest/converters/html_sweep.py with no format code outside it. Keep html.py as the converter shell: decode, parse, drop, absolutize, title, meta, then call the sweep
    • Block dataclass: nodes (the element, or the run of nodes of a loose-text block), kind (heading, p, list, pre, table, quote, figure, loose), text, link_chars, words, verdict, reason
    • Shadow roots (SPEC 7.1.5.4.5.6.1): before the drop loop in html.py, unwrap() every template[shadowrootmode], then drop the remaining template tags. Check that html.parser keeps template children as normal children; add a test that proves it
    • Layout tables: a table whose td/th holds a table, div, or any block tag is not a block; walk its cells as containers
    • Walk (SPEC 7.1.5.4.5.6.1.1, 7.1.5.4.5.6.1.2): recursive over <body>. A block tag (not a layout table) is emitted whole and not entered. A container's children are read in order: text nodes and inline tags join the current run; a block or container child flushes the run as one loose block, then is walked. An inline tag that holds a block or container descendant (a card as <a><div>...</div></a>) is treated as a container. A tag with a hyphen in its name is a container
    • Text (SPEC 7.1.5.4.5.6.2): replace each br with a space, then get_text() with no separator, then " ".join(text.split()). No separator keeps <b>Fa</b>st as one word. link_chars is the folded text length of the a tags inside the block. words is len(text.split())
    • Ancestors: one helper _inside(block, predicate) that walks parent up to <body>. For a loose run use the first node's parent. Hidden test: hidden attribute, aria-hidden="true", or a style matching display\s*:\s*none|visibility\s*:\s*hidden (case-insensitive)
    • Classify in document order (SPEC 7.1.5.4.5.6.3 to .5) with one seen set of case-folded texts of earlier keep and review blocks. Order of checks: no letter or digit (any(ch.isalnum() ...)) -> drop; short and seen -> drop; inside nav/button/select/hidden -> drop; chrome; repeat; links (link_chars / len(text) > max_link_density); short (only p and loose); else keep
    • Render (SPEC 7.1.5.4.5.6.6): each block through markdownify with the current options (ATX, -). A loose run is the joined str() of its nodes. Strip each result; skip one that renders empty. Wrap review blocks in <!-- fiona:review <reason> --> and <!-- /fiona:review -->, each on its own line. Join blocks with one blank line
    • Report (SPEC 7.1.5.4.5.6.7): append html sweep: <k> keep, <r> review, <d> drop to the warnings. No keep and at least one review -> partial with the second warning. The empty-body rule of 7.1.5.4.5.5 still runs first
  • Wire it into html.py. Run _absolutize on <body> before the sweep, so link rewriting covers every block. Pass body instead of the old root to _html_title. Remove the article/main root line
  • Check the page passes the wiki's own lint: the raw page goes through mdoctor fix after writing. Confirm the markers stay on their own lines, MD033 allows HTML comments, and no blank line inside a marker pair is collapsed. Add one test that runs wiki sanity on an ingested fixture with review blocks
  • Fixtures under tests/fixtures/html/ (or inline strings, matching tests/test_ingest_edges.py), one test per SPEC acceptance of 7.1.5.4.5.6:
    • landing.html: no <main>, section with <h2> and a twelve-word <div>, then two <article> cards. All present, in order (the agent-harness case)
    • boilerplate.html: nav list, footer paragraph, <span>•</span>, Share twice, twelve-word paragraph twice, all-link list, two-word <div>. Check the verdicts and reasons chrome, repeat, links, short
    • encyclopedia.html: side nav, infobox, paragraphs with <sup><a> citations, reference list, navbox table
    • qa.html: question, two answers with the same pre, vote counts 42, 7, 3, related aside
    • forum.html: custom-element posts, one in <template shadowrootmode="open">, nested replies, one reply with style="display: none"
    • layout-table.html: table cells that hold div posts give one block per post and no pipe table
    • Determinism: convert the same fixture twice, compare bodies byte for byte, and check the html sweep: counts
  • Re-check the old tests. test_html_keeps_article_link_and_table (tests/test_ingest.py) must still pass: nav gone, article text, absolute /a, pipe table. Review test_html_edges (tests/test_ingest_edges.py) for asserts that relied on the first-<article> root; a change there is a behavior change, say so in the commit
  • Manual check: in tests/wikitest, run fiona-assistant wiki ingest <agent-harness URL> --force (or wiki ingest raw/articles/agent-harness.html --force) and confirm the sections "Test and evaluate your agents systematically" through "Agent Testing Resources" and both customer cards are in the page
  • Fiona skill (SPEC 7.1.5.4.5.6.9). In src/fiona/data/llm_wiki/utils/skills/fiona/SKILL.md section 18.5 add the step: for each fiona:review block, keep it by deleting the two marker lines or delete the block with them; a refined page has no marker. Mention the reasons chrome, repeat, links, short in one line each. Bump the skill version
  • Verify the spec (project agent rule 2.1 step 4): drop the NOT_IMPLEMENTED on 7.1.5.4.5.6, remove the "until the block sweep replaces that rule" sentence and the first-<article> rule from 7.1.5.4.5.2, drop "not built yet" from the html.* settings prose in 5.1, set the Appendix B row to Implemented, bump SPEC. Run python scripts/run_tests.py and python scripts/run_sanity.py --non-interactive
  • [-] Open, not part of this work (now in SPEC Appendix C, with image-only blocks): per-host URL rewrites before fetch (for example Reddit to old.reddit.com), and pages that build their text with JavaScript
  • after sweep,run the automatic md fix from pymarkdown

A plain keyword search over the wiki that prints <path>:<line>: <text> and shortens long lines around the match. SPEC 1.11.0 holds the requirements (SPEC 4.7.7, 7.1.6, settings in 5.1). Spec written 2026-10-07; nothing below is built. Order is the build order. Bootstrap first and load the python-tdd skill.

  • Settings (SPEC 5.1). New SearchSettings dataclass with max_line_chars: int = 160 and max_results: int = 200, and load_search_settings(wiki_root) that reads fiona.wiki.search. max_line_chars is an int >= 20, max_results an int >= 0, neither a bool; anything else keeps the default. Factor the JSON walk to fiona.wiki.<name> out of load_ingest_settings() into one shared helper instead of copying it
    • Tests: defaults, overrides, 19, -1, true, a broken fiona-settings.json
  • New module src/fiona/wiki/search.py, pure functions plus two dataclasses, no printing inside the search itself:
    • SearchMatch(path: str, line: int, text: str, columns: list[int]) and SearchResult(keywords, matches, total, files, truncated)
    • iter_files(wiki_root, includes) -> list[pathlib.Path] (SPEC 7.1.6.1.1, 7.1.6.1.3): default ["wiki/**/*.md"]; each --include replaces the default; union of wiki_root.glob(pattern), files only, deduplicated, sorted by the POSIX relative path. Python 3.11 ** matches zero directories, so wiki/**/*.md includes wiki/index.md; add a test for it
    • read_lines(path) (SPEC 7.1.6.1.2): read_bytes(), strip a UTF-8 BOM, decode("utf-8", errors="replace"), splitlines(). Line numbers start at 1. Frontmatter lines count
    • match_columns(line, keywords, case_sensitive, all_words) (SPEC 7.1.6.2): literal str.find loop per keyword, on casefold() copies unless case-sensitive; return the sorted 1-based columns of every occurrence, or [] when the line does not match (any keyword by default, every keyword with all_words). Never compile a regex from user input
    • shorten(text, start, end, max_chars) -> str (SPEC 7.1.6.4), where text is the stripped line and start/end the span of the first match in it:
      • len(text) <= max_chars -> return text
      • room = max_chars - 8 (worst case: both 4-character markers, "... " and " ..."). If end - start >= room, the window is text[start:start + room]
      • Else split room - (end - start) evenly: left = s - half, right = e + (rest - half); clamp to 0/len(text) and give the unused side's room to the other side
      • Snap: when left > 0 and the cut falls inside a word, move left to just after the next space if that is still <= start; when right < len(text) and inside a word, move right back to the last space if that is still >= end. Strip the window
      • Prefix "... " only when left > 0, suffix " ..." only when right < len(text). Assert len(result) <= max_chars
    • search_wiki(wiki_root, keywords, settings, *, includes, case_sensitive, all_words) -> SearchResult: for each match line, columns are against the original line (SPEC 4.7.7), text is shorten() of the stripped line around the first column. total counts every matching line; only the first max_results (all when 0) go into matches; truncated is total > len(matches); files counts files with at least one match
    • render_text(result) -> str and render_json(result) -> str (SPEC 4.7.7): text lines <path>:<line>: <text>, then ... <N> more matches. Narrow the keywords or raise max_results. when truncated, then Matches: <N> lines in <F> files. JSON keys in the order of SPEC 4.7.7
  • CLI in src/fiona/__main__.py: a search parser under the wiki subparsers (prog="fiona-assistant wiki search"), keywords with nargs="+", --wiki default ., --all-words, --case-sensitive, --include with action="append", --json. Dispatch next to ingest
    • Not a wiki (no fiona-settings.json): reuse the ingest message Error: Directory '<path>' is not an llm-wiki directory, exit 1. Empty or whitespace keyword: Error: Empty keyword, exit 1. No traceback (SPEC 1.2.7)
    • A matched line can hold any Unicode. The CLI has no stdout encoding handling today, so a cp1252 PowerShell console can raise UnicodeEncodeError. Call sys.stdout.reconfigure(errors="replace") in the search handler and test it with a non-ASCII line
  • Tests in tests/test_wiki_search.py, one per SPEC acceptance of 7.1.6, on a tmp_path wiki:
    • Default scope vs --include "**/*.md" and path order (7.1.6.1)
    • Agent Harness works: harness, --case-sensitive, harness eval, --all-words, a.b vs axb (7.1.6.2)
    • Two matches on one line print once; summary text (7.1.6.3)
    • max_line_chars 40: match at column 100 of a 200-character line, match in the first 10 characters, a 40-character line unchanged, a 60-character match longer than the window (7.1.6.4)
    • max_results 2 with five matches: the ... 3 more line and Matches: 5 ...; JSON truncated: true, total: 5 (7.1.6.5)
    • No writes: hashes and st_mtime_ns of every file unchanged (7.1.6.6)
    • CLI: exit 0 with zero matches, exit 1 for a non-wiki and an empty keyword, exit 2 with no keyword, --json parses with json.loads
  • Fiona skill: add Where is a term mentioned? -> wiki search <keyword> --json to the question table in section 5 ("Which command"), and the command with its flags and exit codes to section 6 ("wiki"). Tell the agent to search before reading whole pages. Bump the skill version
  • README usage section (SPEC 7.18.4): one example line
  • Verify the spec (project agent rule 2.1 step 4): drop the NOT_IMPLEMENTED on 4.7.7 and 7.1.6.1 to 7.1.6.6, set 7.1.6 Status: Implemented, drop "not built yet" from the fiona.wiki.search prose in 5.1, set the Appendix B row to Implemented, bump SPEC. Run python scripts/run_tests.py and python scripts/run_sanity.py --non-interactive
  • Open, not part of this work: should --include reject a pattern that leaves the wiki root (../**), and should wiki search honor .gitignore like wiki graph (SPEC 7.1.4.17)

Wiki audit — 2026-10-08

Audit of tests/wikitest against Karpathy's LLM Wiki pattern. There are three layers: immutable raw/, an LLM-compiled wiki/, and a schema. One ingest writes a summary, then updates the entity and concept pages, index.md, and log.md. A single source "might touch 10-15 wiki pages". A query reads index.md first. A lint pass looks for contradictions, orphans, and concepts that have no page yet.

State on 2026-10-08: the log has 5 ingest entries. Under wiki/ there are 3 source pages and 1 query page. 0 pages are refined, and there are 0 concept, entity, or comparison pages. Git has 1 commit (the init). wiki sanity reports 72 violations. The tool's phase works: it fetches, archives, converts, and logs. The agent's phase (refine, cross-link, index, save) has never run once. So the wiki is a converted archive, not a compiled knowledge base. Nothing builds on earlier work yet. Order below is by impact on that.

Way of working

  • No ingest cycle ever finished (SPEC 7.1.5.10, skill 18.5). All 3 source pages are status: raw, confidence: zero, tags: [], the oldest since 2026-10-02. Two sources cover the same topic (LangSmith and Wikipedia on agent harnesses), but no concepts/agent-harness.md joins them. Each ingest touched 1 page plus log.md. Housekeeping only says WARN, and new ingests keep stacking on the pending pile
    • Ingest stdout and --json end with the pending count and the refine command, e.g. Next: refine wiki/sources/x.md (3 pages pending)
    • Skill: do not start a new ingest while --pending is not empty, unless the user asked for a batch. Finish the cycle on the page you just wrote first
    • Decide (P4): housekeeping ingest step turns FAIL when a raw page is older than a setting (fiona.wiki.housekeeping.max_raw_age_days)
  • Nothing was saved to git after init (AGENTS.md section 4, skill 2.3). 5 ingests over 3 days are all untracked. The save trigger is "end of ingest cycle", and no cycle ever ends, so it never fires. Add a housekeeping git step: WARN when wiki/ or raw/ has uncommitted changes, with the action fiona-assistant git save --description ...
  • wiki/index.md is a dead catalog. It lists sources/example-com.md (deleted) and tester-note.md, and is missing agent-harness.md and wiki-agent-harness.md. The tool never edits the index (SPEC 7.1.5.6), and the refine step that should edit it was skipped. So the query path Karpathy relies on (read the index, then drill in) finds nothing
    • Housekeeping index step: every page under wiki/ other than index.md, log.md, and schema.md is linked from index.md. WARN with one detail per missing page
    • Decide (P4): should ingest add the raw page under an ## Pending (raw) heading in index.md, so a query can find it before refine? That changes SPEC 7.1.5.6
  • Schema never set up. wiki/schema.md Domain still reads Replace this line with the wiki's subject. The taxonomy is only wiki and schema, so refine has no tags to pick from. Karpathy has the schema grow with the wiki. Housekeeping WARN while the placeholder line is present. Skill 18.5.7: when a page needs a tag that is not there, add it to the taxonomy
  • Meta content in the user's wiki. The only page an agent wrote is the fiona-assistant test report in wiki/queries/. It is about the tool, not the wiki's domain. Its README/SPEC citations are placeholders (https://github.com/). Skill rule: feedback about the tool goes to the fiona-assistant repo. queries/ holds answers to the user's questions and cites wiki pages
  • Template drift goes unnoticed. The wiki runs Fiona skill 1.15.0, but the package ships 2.1.0 (also schema 1.6.0 to 1.6.1, llm-wiki-mod 1.4.0 to 1.4.1, per wiki check-update). The 1.2.0 converter writes fiona:review markers. The skill steps that tell the agent what to do with them (18.5.2.1) are only in 2.x, so the agent in this wiki has no instruction for that output. wiki ingest should warn when the wiki's skill version is older than the one the tool ships: Fiona skill 1.15.0 is older than 2.1.0; run wiki update

Ingest defects found in the audit

  • Re-ingesting a URL makes a second raw copy and orphans the first (SPEC 7.1.5.3.3, 7.1.5.3.5). On 10-07 and 10-08 the same Wikipedia URL was fetched twice. Both bodies were 102788 bytes but had different bytes (sha256 6c78... and 8c07...), so the byte check did not match them, and wiki-agent-harness-2.html was written. --force then rewrote the page so it cites only -2.html, and reset created from 2026-10-07 to 2026-10-08 and version to 0.1.0. Now raw/articles/wiki-agent-harness.html is cited by no page
    • Before fetching, match the normalized URL against ingest.origin and ingest.final_url of existing pages. On a match, print Duplicate origin: <page>, fetch nothing, and exit 0. --force (or a new --refresh) fetches again
    • On --force, keep created, bump version instead of resetting it, and add the new raw path to sources instead of replacing the list. Mark the log bullet (replaced)
    • Lint: a file under raw/ that no page cites in sources (an orphan raw copy). WARN in housekeeping
  • Tracking parameters are stored everywhere. The LangSmith origin carries utm_*, gclid, gbraid, hsa_*, and gad_* (about 700 characters). They end up in ingest.origin, final_url, the log bullet (one 717-character line), and every absolutized link in the body. Strip known tracking parameters before fetch, with the list in settings (fiona.wiki.ingest.strip_query_params), and record the clean URL
  • The landing-page manual check is marked done above (HTML block sweep, "Manual check"), but tests/wikitest/wiki/sources/agent-harness.md still holds only the one testimonial card. It has tool: fiona-assistant 1.2.0 and no html sweep: warning. Re-run it with --force and confirm, or reopen that item
  • conversion: complete on a 56 KB page that came out as one quote. Add a yield check: when the page body has a small fraction of the words of the source's visible text, mark it partial with the warning low yield: <n> of <m> words. That sends the agent to convert by hand (skill 18.3.1.2)
  • Wikipedia citation markers trip our own linter. Converted [[1]](https://...#cite_note-...) gives 19 MDL002 dead-wikilink errors and one MD011 in wiki-agent-harness.md. Have the converter escape a bracketed number ([\[1\]](...)), or have mdlinter skip [[...]] that is directly followed by (. Add the encyclopedia.html fixture to the wiki sanity test
  • Deleted sources leave references that nothing checks. example-com.md and its raw file are gone (lost to git undo on 10-02). index.md and the query page body still link to them (MDL001 catches that). The query page's frontmatter sources: [raw/articles/example-com.html] and the log bullets are not checked. Lint sources: entries that do not exist
  • Local paths in provenance. A local-file ingest records C:\Users\nvaldez\repositories\tests\wikitest\ingest\tester-note.md in ingest.origin and in the log. That leaks the user name and breaks on another machine. When the source is inside the wiki root, record the path relative to it
  • Stray wiki/fiona.log (0 bytes). file_log_location: "fiona.log" resolves against the working directory, so a run from wiki/ makes a second log file. Resolve it against the wiki root

Lint and housekeeping

  • Sanity is always red, so it stops being a signal. Of the 72 violations, almost all are in files the tool wrote: log bullets (tracked above), raw pages (MD013 on 1000-character paragraphs), and the citation false positives. An agent that sees FAIL every run learns to ignore it. Report violations in status: raw pages as one ingest detail per page ("fixed by refine"), apart from violations in refined and core pages. This depends on the mdoctor fix after-sweep item above
  • table_of_contents.md is the template copy from 2026-09-24 and nothing refreshes it. It lists AGENTS.md headings that no longer exist (Pre-requisites, Directory Permissions, Linking Rules), a libarian folder that does not exist, and nothing under wiki/sources/ or wiki/queries/. Either wiki init generates it with librarian toc, and housekeeping warns when it is stale, or the template stops shipping it
  • Housekeeping checks only mechanics. Karpathy's lint also asks the LLM about meaning: contradictions, superseded claims, concepts that 2+ sources mention but that have no page, and missing cross-links. Add a skill section "Semantic lint" that the agent runs after housekeeping. Housekeeping can feed it a list of bold terms and headings that appear in 2+ source pages but have no page under concepts/ or entities/

P1 — Wiki Lifecycle and Version Control

  • wiki init hardening (SPEC 7.1.1)
    • Warn or refuse when the target is already an llm-wiki directory
    • Do not clobber user content on re-run (idempotent)
  • fiona-assistant wiki update (SPEC 7.1.2)
    • Create src/wiki/update_manager.py with stub functions
    • Add CLI commands: wiki check-update and wiki update
    • Add comprehensive test suite (48 tests)
    • Update core md files / scripts to current template versions (compare version frontmatter)
    • Update directory structure when the template changed
    • Skip locally-modified files and report them
  • fiona-assistant wiki sanity (SPEC 7.1.3)
    • Run all linters: mdlint, python-ruff, shellcheck (once P2 lands)
    • Dead-link check (relative and wiki-internal links)
  • Git version control for llm-wiki (SPEC 7.12)
    • GitPython library wrapper
    • Basic operations:
      • save — commit current edits with yyyy-mm-dd HH:MM:SS <description>
      • snapshot — save plus a git tag yyyy-mm-dd_HH-MM-SS_<tag_name>
      • restore — create new commit restoring to <tag_name> state
      • clean — discard current edits (staging area); git status must be clean afterwards
      • undo — go back one commit; error if the working tree is dirty and suggest clean first
  • fiona-assistant librarian move <source> <target> (SPEC 4.1.2, SPEC 7.3.3). Moves a file or folder, rewrites the relative links inside each moved markdown file, and rewrites every link elsewhere in the wiki that points at a moved path, so no dead link is left (SPEC 7.3.3.6). Lifecycle is project agent rule 2.1. The spec text is written. Start at step 2
    • Step 0, shared link scanner with no behavior change. mdlinter.py (MARKDOWN_LINK, INLINE_CODE, fence toggle, _is_ignorable_target), graph_generator.py (MARKDOWN_LINK_PATTERN), and ingest/page_writer.py (_FENCE, _LINK, _IMAGE) each parse links their own way. Do not merge them in this task. Write the new scanner so it can replace them later, and leave a P2 item for that merge
    • New module src/fiona/librarian/link_rewriter.py, pure functions, no disk writes
      • LinkSpan dataclass: start, end (offsets in the line), path, suffix (#fragment or ?query), title, angled (bool), kind (inline, image, reference)
      • iter_links(line) -> Iterator[LinkSpan]. Inline links, images, and reference definitions (^ {0,3}\[id\]:\s+path). Parse <path with spaces> and the three title forms ("t", 't', (t)). Mask inline code before matching (replace with spaces so offsets stay valid)
      • is_rewritable(path) -> bool: false for empty, #..., a URI scheme (re.match(r"^[A-Za-z][A-Za-z0-9+.-]*:", p); that also covers C: drive letters, which are absolute), and a leading / or \. SPEC 7.3.3.4.1 and 7.3.3.4.2
      • new_link_path(old_path, old_dir, new_dir, moves) -> str | None. urllib.parse.unquote, resolve against old_dir, return None when it does not exist (counted unresolved). Map the resolved target through moves (dict[Path, Path], old absolute to new absolute, with a folder entry covering its children). Then os.path.relpath(target, new_dir) with /
      • format_path(new, old_path, angled): keep ./ when the old path had it; outside <...> write a space as %20 and keep any %xx the old path used (quote only characters that the old path quoted, plus spaces). SPEC 7.3.3.3.2
      • rewrite_text(text, old_file, new_file, moves, *, inbound=False) -> RewriteResult(text, rewritten, unresolved, hits). Split with splitlines(keepends=True) so CRLF stays. Skip the frontmatter block (--- first line through the next ---) and fenced blocks (``` or ~~~, matched to the opening fence length). Rebuild each line from the spans right to left so offsets stay valid. A span whose new text equals the old text is not counted (SPEC 7.3.3.3.3)
      • Inbound mode (inbound=True, file did not move, so old_file == new_file): rewrite only a link whose resolved target is a key in moves or lies under a moved folder. Every other link is left alone and not counted as unresolved (SPEC 7.3.3.6.2). hits counts the links that point at a moved path; the caller uses it for the read-only count without writing
      • rewrite_wikilinks(text, renames) -> (text, count). renames maps old stem (lowercase) to new stem. Reuse the WIKILINK grammar from mdlinter.py, but capture the #section and |label parts so they are written back unchanged. Same fence, inline-code, and frontmatter skipping as rewrite_text (SPEC 7.3.3.6.3)
    • New module src/fiona/librarian/mover.py
      • MovePlan dataclass: source, final, files (list of (old, new) absolute paths), rewrites (dict[path_after_move, RewriteResult] for every markdown file that changes, moved or not), read_only (dict[path, int] from SPEC 7.3.3.6.4), is_dir
      • MoveError(Exception) carrying the exact stderr text from SPEC 7.3.3.2
      • resolve_final(source, target) -> Path: an existing folder target means target / source.name (SPEC 7.3.3.1.2)
      • check(source, final, wiki_root): raise MoveError in this order: source missing, final exists, final equals source or final.is_relative_to(source) for a folder, then read-only zone for source or final. Get the root with find_wiki_root(source.parent) from toc_generator.py (SPEC 7.3.1.2.6). Test the first part of the path relative to that root against READONLY_DIRS. Do not copy _file_toc_single, which matches any path part named personal or raw, the same defect listed for json-lint fix in the 2026-09-29 audit
      • plan(source, target, wiki_root) -> MovePlan: call check, list files (rglob("*") for a folder, hidden included; SPEC 7.3.3.1.3), build moves, read each markdown file as bytes, detect a UTF-8 BOM, decode, call rewrite_text. Disk is read only
      • Inbound scan inside plan (SPEC 7.3.3.6): walk every *.md under wiki_root, skip hidden folders (.git[[[^|]|(.)]] included) and the moved files. Call rewrite_text(..., inbound=True) on each. A file whose first part relative to the root is in READONLY_DIRS goes to plan.read_only with its hits count and is never put in rewrites (SPEC 7.3.3.6.4). A file with no change is not added (SPEC 7.3.3.6.5). Read each file once; a large wiki is one pass over its markdown
      • Wikilink renames: build renames from moved markdown files whose stem changed. Drop an old stem when any other markdown file under the root, after the move, still has that stem (case-insensitive). Run rewrite_wikilinks on every scanned file, moved files included, and add its count to the same rewrote total for that file
      • apply(plan): create final.parent, move with shutil.move(source, final) (one call moves a folder), then write each changed markdown file at its path after the move with write_bytes (re-add the BOM, no newline translation). Wrap OSError into MoveError(f"Error: {exc}") (SPEC 7.3.3.7)
      • report_lines(plan, dry_run) -> list[str]: the line shapes from SPEC 4.1.2, paths relative to the current directory with /, [dry-run] prefix when asked. One moved line for the source, then the rewrote, unresolved, and read-only lines sorted by path, then the summary line with file count, link count, rewritten file count, unresolved, and read-only
      • move_main(source, target, dry_run) -> int: plan, print, apply unless dry run. MoveError goes to stderr, return 1. No git call (SPEC 7.3.3.8)
    • CLI in src/fiona/__main__.py: add a move subparser under librarian_subparsers with source, target, and --dry-run, help Move a file or folder and fix its relative links. Dispatch straight to move_main(...) and sys.exit its return value. Do not rewrite sys.argv (open item in the 2026-09-29 audit). Also add move to the standalone librarian script in librarian.py, or record in SPEC 2 that it is reached only through fiona-assistant
    • Tests, tests/test_link_rewriter.py (pure, no disk except tmp_path targets for existence)
      • One test per link form: inline, image, reference definition, <angled path>, each title form, #fragment, ?query, ./ prefix kept, %20 kept, space quoted
      • Unchanged cases from SPEC 7.3.3.4: https://, mailto:, #anchor, /abs, C:/abs, [[wikilink]], fenced (backtick and tilde), inline code, <a href>, <https://autolink>, frontmatter sources:
      • A dead link is left as is and counted unresolved
      • Two links on one line are both rewritten at the right offsets
      • CRLF input gives CRLF output
      • Inbound mode rewrites a link to a moved file and to a file under a moved folder, and leaves a dead link to an unrelated path untouched and uncounted
      • rewrite_wikilinks keeps #section and |label, matches the stem case-insensitively, and skips fenced and inline code
    • Tests, tests/test_librarian_move.py, one per acceptance in SPEC 7.3.3, on a tmp_path wiki with fiona-settings.json
      • Move into an existing folder; move to a new path that creates parents; folder move keeps layout and removes the empty source (SPEC 7.3.3.1)
      • Each refusal in SPEC 7.3.3.2: exact stderr line, exit 1, tree hash before equals tree hash after (helper that SHA-256s every file plus the folder list)
      • The wiki/a.md -> wiki/concepts/a.md example and its rewrote 2 link(s) line (SPEC 7.3.3.3)
      • Folder move: sibling link unchanged, ../index.md becomes ../../index.md
      • Link from one moved file to another moved file (folder move, and a link that leaves the folder and comes back)
      • BOM, CRLF, and frontmatter version unchanged; a moved PNG keeps its SHA-256 (SPEC 7.3.3.5)
      • Inbound acceptances in SPEC 7.3.3.6: wiki/index.md [a](a.md#top) becomes [a](concepts/a.md#top) with its rewrote 1 link(s) line; folder move rewrites notes/x.md and notes/img/p.png in wiki/log.md; [[a#intro|A page]] becomes [[b#intro|A page]] on a rename and stays when wiki/other/a.md exists; raw/n.md is byte-identical with a read-only 1 link(s) line; a file with no reference is byte-identical
      • End-to-end: after a move with no read-only and no unresolved line, find_dead_links(wiki_root) from mdlinter.py returns nothing new compared with before the move
      • --dry-run stdout equals the real run's stdout with the prefix, and the tree is unchanged (SPEC 7.3.3.7)
      • OSError from shutil.move (patch it) gives stderr and 1
      • Git wiki: commit count unchanged (SPEC 7.3.3.8)
      • CLI test through fiona.__main__.main with sys.argv patched, as tests/test_fiona_main.py does for toc
    • Verify: remove each NOT_IMPLEMENTED in SPEC 4.1.2 and 7.3.3, set the 7.3.3 status to Implemented, update Appendix B, drop "(planned)" from the Appendix D row, add move to src/fiona/librarian/README.md (SPEC 7.18). Run python scripts/run_tests.py and python scripts/run_sanity.py --non-interactive, bump the SPEC version
    • Manual check on a copy of a real wiki: move a page into a subfolder, then run fiona-assistant wiki sanity and confirm the move added no MDL001 or MDL002, in the moved page or in the pages that link to it

P2 — Quality Tooling

  • Restore the run_sanity progress heartbeat ([WAITING] lines while checks run) on top of the shared SanityRunner engine (SPEC 7.14.1); dropped when the engine was generalized
    • SanityRunner.run_all takes optional progress and heartbeat callbacks; check lifecycle logged at INFO
    • run_sanity passes stdout callbacks (DONE markers and WAITING lines, assigned via a carriage-return constant); wiki sanity output unchanged
  • Frontmatter linter (fmlint) (SPEC 7.5)
    • Implement src/customlinter/fmlint.py
    • Write TDD tests for fmlint
    • Add CLI integration: fiona-assistant fmlint chklint [directory]
    • Integrate into run_sanity.py and wiki sanity
    • Add settings support via fiona-settings.json
    • SKILL.md name, folder, and description checks (FML004-FML006)
  • Generalize compress and format violations for all linters (SPEC 6.6, 6.1)
    • BaseLinter provides compress_violations() and format_compressed_violations(); report header comes from the linter_name class attribute
    • fmlint delegates format_violations and lint_directory compression to the shared base methods (output unchanged)
    • Migrate mdoctor (compress_violations/format_violations, same compressed shape as fmlint) onto the shared base format
    • Migrate python_ruff (different compressed shape: dict[rel_path, list[dict]]) onto the shared base format
    • Keep all linter reports diffable: same sections, same order (update SPEC 7.4/7.8 and tests together)
  • JSON linter (json-lint) (SPEC 7.11)
    • Implement src/customlinter/jsonlint.py
    • Write TDD tests for json-lint
    • JSON syntax validation using the standard json module
    • JSON style rules (indentation, trailing commas, key ordering)
    • CLI: fiona-assistant json-lint chklint and fix
    • Settings via the json section of fiona-settings.json
    • Integrated into run_sanity.py and wiki sanity
  • Shellcheck library (SPEC 7.9)
    • add shellcheck to toml
    • Wrap shellcheck like python_ruff (AI-friendly violation report)
    • Clear install-hint error when shellcheck is missing
  • Lint fenced code blocks inside MD files (SPEC 7.10)
    • Python blocks via Ruff, shell blocks via shellcheck
  • Logging library instead of print for diagnostics (SPEC 7.17)
    • Shared fiona.logging_config.setup_logging() on CLI entry points
    • Error and warning diagnostics go to stderr through logging. CLI reports stay stdout prints. git_tool, graph_generator, and several CLIs still print errors
  • Enable every Ruff rule (select = ["ALL"] in pyproject.toml)
    • The table under [tool.ruff] currently leaves Ruff on its built-in default. That skips style, docstrings, line length, pyupgrade, and other families
    • Turn on ALL, then clear the findings on src/ or add an explicit ignore only where a rule fights the spec (CLI print on stdout is one)
    • Keep test_lint_detects_unused_import green. It must find unused-import F401 even when another rule is reported first
  • Clear existing lint violations (SPEC 1 constraint: Ruff and PyMarkdown must pass)
    • Ruff: 0 violations — removed unused imports and unnecessary pass statements, sorted imports, made dates tz-aware (UTC), added explicit check=False to subprocess.run, added from __future__ import annotations to wiki.py
    • Markdownlint: 0 violations — fixed blanks around headings/lists, line length, fence languages; mdoctor now enables the pymarkdown front-matter extension (files with frontmatter no longer raise phantom MD003/MD022/MD041) and ignores the frontmatter title for MD025
  • Settings manager, fiona-assistant setup (SPEC 7.16)
  • When running sanity, make sure that markdownlint tests also include outer dir of the src/ because there are markdown files there
  • in run_sanity.py, show the failing test function and assert ex.
  • add pyspellchecker in spec for spelling checks in markdown
  • add in spec, we add a new function in toc_generator,
    • generate toc for files.
      • All headings except H1 # will be included
      • the toc will be generated below the H1,
      • make a logic to detect of a toc exists then regenerate to update.
      • make a logic to add section numbers, (also add section numbers to the actual headings to preserve link is working)
      • in fiona-settings.json, add specific section to modify parameters such as, max heading to include e.g min:2, max:6, sectionize: true/false, ignore_files:[], etc
      • fiona-assistant librarian toc file <path_to_file> will generate the toc for specific file
      • fiona-assistant librarian toc file * will generate the toc for all files recursively in the current directory/llm-wiki dir if detected
  - test_md_frontmatter.py       43/44 passed, 1 failed
     - TestParseFrontmatter : test_parse_valid_frontmatter
     - assert 'Test Document' == 'Test Documen'
  - test_mdoctor.py              33/33 passed
  • prevent the pytest short summary missing from output
WARNING run_tests: pytest short summary missing from output; failure details come from the junit report (check PYTEST_ADDOPTS, --no-summary, or plugin output)
  • Fix terminal-width-dependent failure of the failure-report integration tests (TestFailedTestReportIntegration in src/tests/test_run_sanity.py)

    • Symptom: test_real_error_shows_function_and_message (and its assert-based sibling) fail on python src/run_sanity.py in an interactive terminal with the report literally showing RuntimeErro..., but pass when CI is set in the environment
    • Root cause: pytest trims the short-summary line ERROR <nodeid> - <message> to the terminal width (_format_trimmed in _pytest/terminal.py; width from shutil.get_terminal_size, i.e. COLUMNS or the tty) unless CI/BUILD_NUMBER is set or -vv is used. The temp-test nodeids (~117 chars, ../../../../tmp/pytest-of-user/...) push the line past a ~140-column terminal, so the message is cut to RuntimeErro.... run_tests._parse_failed_tests_output captures that trimmed text as a real (non-generic) message and _merge_failure_details keeps it over the junit report's full RuntimeError: boom, so the truncated text reaches the report and the exact-match assertion fails
    • Reproduce: env -u CI COLUMNS=140 python -m pytest -q fails; CI=true python -m pytest -q passes. Bisect context: last commit where sanity passes in a plain terminal is 43cb7f3 ("clean up pytests"); e750a3d/cb7c3d8 introduced the env-sensitive integration tests (057ac8b additionally has a genuinely broken test, fixed later)
    • Recommended fix (TDD): in run_tests.py, treat short-summary messages that end in ... as unreliable — same as the generic fallback — so _merge_failure_details lets the junit entry (which always carries the full message) replace them. Write the failing test first: unit-test _merge_failure_details with a ...-trimmed regex entry vs a junit entry, assert junit wins
    • Alternative one-liner: set CI=1 in the sanitized subprocess environment in _sanitize_pytest_env (pytest skips summary trimming when CI is set); keep junit parsing as the safety net
  • Make scripts/run_sanity.py fast on Windows (2026-09-29 investigation). A clean run takes about 2 to 3.5 minutes on Windows and at most 20 seconds on Linux

    • Symptom, measured with LOG_LEVEL=INFO and --non-interactive. The machine had other load during the run, so the absolute times are high. The ratios hold
      • pytests took 03:26 and markdownlint took 00:26. ruff, frontmatter_lint, pycustomlint, and jsonlint each took 00:00
      • SanityRunner.run_all runs the checks in one thread each. The total is the slowest check, which is pytests
      • python -m pytest tests alone, without coverage, took 165 s for 962 tests. The sanity run adds --cov, which makes it slower
    • Root cause: each git process is slow to start on Windows, and the run starts hundreds of them
      • Measured cost per subprocess.run on this Windows box: git --version about 198 ms, python -c pass about 150 ms, cmd /c exit about 51 ms. On Linux a git call takes about 1 to 3 ms
      • C:\Program Files\Git\cmd\git.exe is a wrapper that starts mingw64\bin\git.exe, so each call creates two processes. Endpoint security that scans each new process likely adds to the cost
    • Source 1: pymarkdown respect_gitignore starts git twice for every directory it walks
      • fiona-settings.json sets respect_gitignore: true. MDoctor.scan_file_programmatically passes it to PyMarkdownApi.scan_path(..., recurse_if_directory=True)
      • In application_file_scanner (a pymarkdownlnt dependency), each directory in the os.walk loop calls GitProcessor.get_check_ignores. That runs git rev-parse --show-toplevel and then git check-ignore (application_file_scanner.py, about line 850; git_processor.py)
      • One more batched git check-ignore runs over the final file list (__remove_any_gitignored_paths)
      • Repo-level run_markdownlint scans .. That walks the repo root, including .git, .ruff_cache, and the other directories that are not ignored. It took 18 to 26 s alone, which is longer than the whole Linux run
      • MDoctor._ignored_by_gitignore reads .gitignore without git, but it only filters the violations afterwards. It does not stop pymarkdown from walking and calling git
    • Source 2: the test suite runs real sanity scans and real git
      • Slowest single item: the session fixtures in tests/conftest.py. pristine_wiki calls wiki_init (1.7 s). pristine_check_results calls run_wiki_checks. The two together take 24 s of setup on test_sanity_manager.py::TestSanityChecksRegistry::test_registry_drives_results
      • The initialized wiki is a git repo with 96 files and 75 directories. Timed per check, run_wiki_checks spends 29.2 s in markdownlint: about 150 git calls at about 190 ms each. frontmatter_lint takes 0.1 s, python-ruff 0.3 s, mdlinter 0.1 s, and jsonlint 0.0 s
      • Other tests repeat full wiki sanity scans at 1.3 to 2.6 s each (test_sanity_manager.py, test_fiona_main.py test_main_wiki_sanity*, test_mdoctor.py::TestMDoctorGitignore::test_scan_skips_gitignored_files)
      • test_gittool.py: the git_repo fixture uses GitPython, which starts real git processes. Setup takes 1.3 to 2 s per test, and calls take up to 3.4 s (test_restore_by_full_tag, test_restore_by_partial_tag, test_undo, test_clean)
      • wiki_init tests take 2 to 3.3 s each (test_wiki.py, test_fiona_main.py::test_main_wiki_init). Each copies the template and runs a git init and commit
      • TestFailedTestReportIntegration in test_run_sanity.py starts a nested pytest subprocess, about 2.2 s per test
      • The other ~900 tests average about 60 ms each. Temp directory and file creation is slow on Windows, more so with antivirus scanning
    • Recommended fixes, in order of impact (TDD, SPEC first when behavior changes)
      • Stop pymarkdown from calling git. Pass respect_gitignore=False to scan_path and fix_path. Build exclude_patterns from the .gitignore directory names that MDoctor._gitignore_dir_names already parses, plus .git. Or collect the file list with the librarian and scan each file. This removes nearly all git calls from the markdownlint check and from the wiki-sanity tests. Done in MDoctor._pymarkdown_path_options. Patterns are omitted when the scan root is outside the working directory, because the matcher rejects those paths. The post-scan directory check still drops ignored files. SPEC 7.4.3
      • Make the pristine and minimal wiki fixtures cheaper. Use a wiki that is not a git repo, or turn off respect_gitignore in the fixture's settings. The scan change above removes the git calls from pristine_check_results, which was the 29 s cost. The fixtures themselves are unchanged. wiki_init is still about 1.7 s once per session
      • Share one session-scoped git repo across the read-only test_gittool.py tests. Keep per-test repos only for tests that change state. Fixture is read_only_repo
      • Run pytest in parallel with pytest-xdist (-n auto) in TestRunner.run_pytest. Confirm coverage still combines correctly (pytest-cov supports xdist). Tried on 2026-09-30: coverage still merged at 96.0%, and pytests went from 01:56 to 01:40. Worker progress lines ([gwN] [100%] FAILED path) do not match the short-summary parser, so TestFailedTestReportIntegration reported Tests: 0/1 passed and dropped the assert text. The flag was removed. Parallel workers are not a win on this box
      • Optional, outside the repo: exclude the repo directory and git.exe from antivirus scanning, if IT policy allows it
    • Acceptance: record Windows and Linux timings for python scripts/run_sanity.py --non-interactive before and after the change. Windows should finish well under 1 minute. Linux was not re-measured (this machine is Windows). The todo baseline is at most 20 s on Linux. Windows before, this machine: about 02:50 in the session that wrote the git fix, and the 2026-09-29 note above (pytests 03:26, markdownlint 00:26). Windows after the git fix, shared read-only repo, and os.pathsep (2026-09-30, LOG_LEVEL=INFO): pytests 01:54, markdownlint 00:10, SANITY PASSED 01:54 (968 tests, coverage 96.0%). An earlier pass the same day was 01:56. That is under the old 2 to 3.5 minutes and still above 1 minute. The remaining time is pytest itself: temp files, the tests that still start git, and nested pytest
    • Side finding, not a speed issue: run_tests.py joins PYTHONPATH with ":" (f"{src_dir}:{current_pythonpath}"). On Windows the separator is ;. Use os.pathsep. Done. SPEC 4.9
  • in toc generator settings, per-file glob overrides in the file section (SPEC 7.3.1.2)

  • instead of current directory as default, find a fiona-settings.json in the current directory / subdirectory, that will be the llm-wiki root and default directory for commands (SPEC 7.20)

  • Enable the pymarkdown markdown-tables extension in fiona-settings.json so table rules (MD055/MD056/MD058) run in run_sanity.py (SPEC 5.1 known limitation)

    • Blocked: pymarkdownlnt 0.9.40 crashes with BadTokenizationError on list lines containing pipes (e.g. generated table_of_contents.md entries); upgrade pymarkdownlnt once fixed and re-run sanity

P3 — Metadata, Backup, and Documentation

  • Backup script for llm-wiki so users can recover if something goes wrong (SPEC 7.13)
  • Frontmatter tags on MD files (SPEC 6.2)
    • version
    • date tag
    • Quality signal: confidence
    • contested flag (true if claims are disputed)
  • Project documentation (SPEC 7.18)
    • Root README.md links SPEC.md
    • Root README and AGENTS.md document bootstrap and bootstrap.ps1
    • Refresh librarian/README.md and mdoctor/README.md for the current subcommands (list, toc, chklint, fix)
    • Refresh src/README.md structure (wiki/, python_ruff/, run_tests.py, run_sanity.py, settings_dev.json). Full gap list is the last item of the P0 2026-09-29 structure audit, including the root README link to ./llm-wiki/AGENTS.md
  • Process hygiene: use semantic commit prefixes (feat:, fix:, docs:, ...) per project agent rule 8.1
  • fiona-assistant wiki graph (SPEC 7.1.4): link parser, graph, HTML template, and CLI in src/wiki/graph_generator.py
  • Draw only important graph nodes until a parent is clicked (SPEC 7.1.4.16)
    • Setting fiona.wiki.graph.node_importance_min_point
    • 0, or a missing key, draws every node
    • A positive integer hides a node whose unique_points is below that integer until one of its parents is clicked
  • Omit gitignored markdown from the wiki graph (SPEC 7.1.4.17)
    • Failing tests in tests/test_graph_generator.py for the acceptances in SPEC 7.1.4.17, 7.1.4.17.1, and 7.1.4.17.2: secret.md absent from JSON nodes, edges, and orphans; a link to it creates no edge; max_files 1 still keeps kept.md; drafts/ omits drafts/note.md and keeps wiki/note.md; a # secret.md comment does not omit; notes/.gitignore omits notes/scratch.md only; with no .gitignore, .git/*.md is absent and a root page stays
    • In wiki_graph_main (src/fiona/wiki/graph_generator.py), drop a match before the max_files cut. Read patterns with gitignore_patterns_from_text and test them with path_is_ignored (src/fiona/wiki/ignore.py), the same rules as SPEC 7.1.2.10. The scan-root .gitignore applies to the whole tree. A subdirectory .gitignore applies only under that subdirectory. Also drop any path under .git/
    • Do not parse an omitted file. Drop an edge whose target resolves to an omitted file, so it never becomes a node
    • HTML uses that same graph. No new settings key: the filter always runs
    • Housekeeping's orphan list follows automatically, because graph_step builds the graph through wiki_graph_main (SPEC 7.2.6)
  • Graph live view, wiki graph --serve (SPEC 7.1.4.19). Build this before directory nodes. It is smaller, needs no D3 refactor, and gives a live page to watch while the directory work lands. Lifecycle is project agent rule 2.1. The spec text is written. Start at step 2
    • Step 0, refactor with no behavior change. Split wiki_graph_main in src/fiona/wiki/graph_generator.py into collect_files(directory, include, exclude, max_files), build_graph(directory, files, settings) -> WikiGraph, and render(graph, settings, format_type, live=False) -> str. wiki_graph_main calls those three and writes the file. The server calls the same three without writing. The existing tests in tests/test_graph_generator.py stay green
    • Settings helpers beside consolidate_depth: serve_settings(settings) -> dict returns host, port, poll_seconds, debounce_ms with the SPEC 5.1 defaults for absent, mistyped, or out-of-range values. One test per key and one per bad value
    • CLI in src/fiona/__main__.py: add --serve, --host, --port (type=int), --no-open. Reject --serve with --format json / --json, and --host/--port/--no-open without --serve, through parser.error (exit 2). A flag wins over fiona.wiki.graph.serve.*. Dispatch --serve to wiki_graph_serve(...) in the new module
    • New module src/fiona/wiki/graph_server.py, stdlib only (http.server, threading, webbrowser, urllib.parse, pathlib, json, time)
      • GraphState: holds html, json, build, and the scanned relative paths, behind a threading.Condition. publish(html, json_text, files) raises build by 1 and calls notify_all(). wait_for_build(after, timeout) returns the newer build or None
      • FileWatcher(directory, include, exclude, clock=time.monotonic): snapshot() maps each matched markdown file, the nearest fiona-settings.json, and every constellation/*.json to (st_mtime_ns, st_size). poll() returns true once the snapshot changed and no newer change came within debounce_ms. Inject the clock so tests do not sleep
      • GraphRequestHandler(BaseHTTPRequestHandler): route /, /graph.json, /events, /files/<path>, else 404. do_POST, do_PUT, do_PATCH, do_DELETE send 405. Silence log_message into logging at DEBUG
      • /events: send Content-Type: text/event-stream and Cache-Control: no-cache, then loop on state.wait_for_build(last, 15). A build writes event: graph and data: {"build": n}. A timeout writes : keepalive. A BrokenPipeError or ConnectionResetError ends that handler quietly
      • /files/<path>: urllib.parse.unquote, join on the scan root, resolve(), then require is_relative_to(root) and membership in the scanned set. Anything else is 404. Node ids are absolute paths today (the SPEC 7.1.4.3 example shows relative ones), so the page must send the path relative to the scan root. Add a rel field to each page node in render, or strip the root in the live script
      • wiki_graph_serve(...): first build, then ThreadingHTTPServer((host, port), handler) with daemon_threads = True. OSError on bind prints the error on stderr and returns 1. Print the URL line from SPEC 7.1.4.19.1 with the real bound port (server.server_address[1]). Warn on stderr for a host that is not loopback. Open the browser unless --no-open
      • Watcher thread: every poll_seconds, poll(), then rebuild with the step-0 functions and state.publish. On any rebuild error log it on stderr, keep the last good state, and send no event (SPEC 7.1.4.19.6). Catch the specific errors the build raises (OSError, UnicodeDecodeError, ValueError), not Exception
      • -o FILE given with --serve: write that file after each successful build. Without -o, write nothing
      • KeyboardInterrupt: server.shutdown(), stop the watcher with a threading.Event, print Stopped, return 0
    • Live script src/fiona/wiki/templates/graph_live.js
      • graph_template.html gets a {{graph_live_js}} marker. render(..., live=False) replaces it with an empty string, so a written file has no EventSource (SPEC 7.1.4.19.7)
      • In graph_physics.js, expose window.fionaGraph = {getState(), restoreState(state)}. State is the zoom transform, {id: [x, y]} for drawn nodes, the clicked id, the open view hash, each toggle, the theme, and the search text
      • restoreState sets known node x/y before the simulation starts, skips the opening fade for those nodes, applies the transform with no fit, and lets only new nodes fade in. Store a one-shot sessionStorage['fiona-live-state']. The pageshow reset (SPEC 7.1.4.5.18) is skipped only when that key exists, and the key is removed after use
      • new EventSource('/events'). On graph with build > window.FIONA_BUILD, save state, then location.reload(). The server writes window.FIONA_BUILD = <n> into the served page
      • Double click on a page uses '/files/' + rel when window.FIONA_BUILD is defined, else file:// as today
    • Tests, tests/test_graph_server.py, one per acceptance in SPEC 7.1.4.19. Start the server on port 0 in a thread and read it with urllib.request
      • URL line printed with a port above 0; --no-open does not call webbrowser.open (patch it)
      • Port in use: bind a socket first, expect stderr and 1
      • Usage errors exit 2 for each bad flag combination
      • POST / is 405; /nope is 404
      • /files/a.md returns the text; /files/../secret.md and /files/%2e%2e/secret.md are 404
      • Add new.md, drive FileWatcher with a fake clock, and assert /graph.json gains the node and edge and build rises by 1. Deleting a page drops it
      • Three saves inside the debounce window raise build by 1
      • Read /events with a short socket timeout and assert one event: graph line after a rebuild
      • A failing rebuild (patch build_graph to raise OSError) keeps the old HTML and sends no event
      • Plain wiki graph output has no EventSource
    • Verify: flip each NOT_IMPLEMENTED in SPEC 7.1.4.19 and the --serve line in SPEC 4.7.5, add serve to the packaged fiona-settings.json, update the SPEC 5.1 prose and Appendix B status, run python scripts/run_tests.py and python scripts/run_sanity.py --non-interactive, bump the SPEC version
    • Manual check: fiona-assistant wiki graph <wiki> --serve, add a page linking an existing page, confirm it fades in and the view and theme stay put
  • Directory nodes in the HTML graph (SPEC 7.1.4.18). Spec is written. Start at step 2
    • Step 0, refactor graph_physics.js with no behavior change. This is the large part
      • Key every node and edge by id. d3.forceLink(...).id(d => d.id) in place of d.index, and drop nodeData[d.source.index] lookups in the distance callback
      • Move the one-shot .data(...).enter() blocks for edges, nodes, labels, glow, and arrows into one renderView(nodes, edges) that uses .join() keyed by id, resets visibleNodes / visibleEdges, recomputes maxInDegree and maxLinkScore, restarts the simulation, and runs the See all fit
      • Toggles, search, legend, and theme change work from the current view's arrays, not the global nodeData
      • Existing HTML string tests in tests/test_graph_generator.py stay green. Open a real wiki in the browser and check hover, click, drag, search, and every toggle by hand
    • Settings helper directory_settings(settings) -> (min_files, outside_depth) with the SPEC 5.1 fallbacks. Tests for absent, 0, negative, string, and float
    • Python view builder, so the logic is tested with pytest and the page only draws. New pure functions in graph_generator.py (or graph_views.py if it grows)
      • collapsed_directories(page_rel_paths, min_files) -> set[str]: count pages per parent directory (direct only), keep those at or above min_files, never the scan root ""
      • display_id(page, view_dir, collapsed) -> str: walk the page's ancestors below view_dir; the first collapsed one is the shown id, else the page id. Outside pages use view_dir = "" (the opening view), per SPEC 7.1.4.18.5
      • build_view(view_dir, nodes, edges, collapsed, depth) -> dict: map each edge end through display_id, drop self loops and edges with both ends inside a collapsed node, sum weights per pair, then add outside targets by a breadth-first walk along edge direction up to depth hops. Mark those nodes and edges outside: true
      • Directory node metrics from the merged edges with repeated_link_score (SPEC 7.1.4.18.3.2). Page nodes keep their global metrics from SPEC 7.1.4.2.3. Write that down in the SPEC if a test shows it matters
      • Directory node fields: id (wiki/concepts/), name (concepts/), kind: "directory", path, pages (every page under it), members (page names, for search)
      • render embeds views = {"": {...}, "wiki/concepts/": {...}} for the root and each collapsed directory, plus the existing page data. JSON output is unchanged (SPEC 7.1.4.18.8). With min_files 0, views holds only "" and the draw equals today's
    • Page script
      • Opening draw is renderView(views[hashDir() || ""])
      • Double click on kind === "directory" sets location.hash = '#dir=' + encodeURIComponent(id). A hashchange listener calls renderView. An unknown hash opens ""
      • Breadcrumb bar at the upper center (All / wiki / concepts, each segment a link to that view) and a "Back" button, disabled on "". Lengths in em (SPEC 1.2.6)
      • Outside nodes and edges at opacity: 0.5
      • Search: a directory node matches its own name or any members entry
      • Click card for a directory: path, pages, points, in-degree, out-degree; no frontmatter rows
      • Legend: a directory shape entry while a directory node is drawn
    • Folder shape: add a folder path generator next to circle, cloud, and star. Add shapes.directory: "folder" to constellation/colorful.json and pick a shape for night-sky, grayscale, dark, and professional. Bump each theme's version
    • Tests, one per acceptance in SPEC 7.1.4.18
      • collapsed_directories: three pages in wiki/concepts/ collapse at 3; two direct pages in wiki/notes/ plus three in wiki/notes/deep/ collapse only deep/
      • build_view(""): one wiki/concepts/ node with subtitle concepts/; edge from wiki/index.md with weight 3; no a.md-b.md edge; unique_points 1, total_points 3, in-degree 1
      • build_view("wiki/concepts/"): draws a.md, b.md, c.md and their edge; depth 1 adds wiki/index.md as outside and not wiki/log.md; depth 2 adds both; depth 0 adds none; a target in collapsed wiki/other/ shows as wiki/other/
      • HTML: views JSON present, shapes.directory read from the theme, hashchange handler present, "Back" label present
      • JSON with min_files 3 still lists every page
    • Verify: flip each NOT_IMPLEMENTED in SPEC 7.1.4.18, update the 7.1.4 status line, SPEC 5.1 prose, and Appendix B. Add directory_nodes to the packaged fiona-settings.json. Measure SPEC 7.1.4.5.11 (1000 files under 10 s) with min_files on. Run tests and sanity, bump the SPEC version
  • in spec, refine the sectioning by compiling the modules, and their functions as subsections
    • eg 4.1. librarian then 4.1.1 librarian toc, to make it cleaner

P4 — Design Decisions Needed

  • Should git undo and git clean prompt on a TTY, with --yes to skip it? Agents would then pass --yes every time (SPEC Appendix C, 2026-10-02 manual test)
  • Librarian: version tracking for non-MD files (SPEC 7.3.2)
    • Python files — format TBD
    • Bash scripts — format TBD
    • Decide the tagging format before implementing
  • Define LLM wiki tool roles (SPEC 7.19)
    • Document purpose of src/ tools
    • Classify tools by role:
      • Ingestion: moving files from ingest/ to raw/
      • Processing: converting raw/ to wiki/concepts/
      • Maintenance: updating index.md / log.md

Open Questions (SPEC)

  • Tagging format for python files and bash scripts (SPEC 7.3.2)
  • Git wrapper location: implemented at src/gittool/git_tool.py (SPEC 7.12)
  • Wiki root discovery (SPEC 7.20): the nearest fiona-settings.json wins; error when two at the same depth are equally near. Ancestor search is still open.