MEMORY.md and skill memory template, ingest auth and html_sweep, TOC slug fix, run_tests CI output, and matching tests. Excludes GitLab CI and publishing docs.
96 KiB
Fiona Project TODO List
- Done
- P0 — Packaging and Structure Fixes
- Code defects — 2026-09-29 audit
- Manual test findings — 2026-10-02
- Ingest of a URL that needs a login
- HTML block sweep for ingest
- wiki search
- Wiki audit — 2026-10-08
- P1 — Wiki Lifecycle and Version Control
- P2 — Quality Tooling
- P3 — Metadata, Backup, and Documentation
- P4 — Design Decisions Needed
- Open Questions (SPEC)
Ordered by priority. Each item references its section in SPEC.md (the source of truth for requirements and acceptance criteria).
Done
- Include the llm-wiki template as part of the PyPI package
- MD frontmatter parser for version control of core files (SPEC 6.2)
versiontag support
fiona-assistant wiki init— copy packaged template into a target directory (basic flow; hardening tracked under P1)- Windows PowerShell bootstrap (
bootstrap.ps1): repo-local venv under.venvs\<name>, same flags as the bashbootstrap
P0 — Packaging and Structure Fixes
From the 2026-09-23 structure audit, plus the 2026-09-29 layout audit below. These are repo-hygiene items that block a correct install or make the docs contradict the code.
- Fix PyPI packaging so a non-editable install works (wheel build
confirmed
python_ruffis missing entirely)- Add
python_ruffto thepackages.findinclude list inpyproject.toml - Add
src/python_ruff/__init__.py - Add
src/fiona/__init__.py - Re-verify: build a wheel, install it fresh, confirm the
fiona-assistantconsole script runs and the llm-wiki template data ships
- Add
- Resolve the duplicated
raw/zone in the llm-wiki template- Pick one canonical location (
llm-wiki/raw/at the wiki root, per user decision) and removesrc/llm-wiki/wiki/raw/ - Align root
AGENTS.md,src/llm-wiki/AGENTS.md, andwiki/schema.mdon the same location
- Pick one canonical location (
- Reconcile the two competing frontmatter schemas
- Decide between project agent rule 10.1 (
name/version/title) andwiki/schema.md(title/sources/tags); one schema wins:name/versionrequired,titleoptional (user decision, 2026-09-25; documented in SPEC 6.2) - Add missing
typevalues (e.g.toc) to the allowed types list (project agent rule 10.1,wiki/schema.md, SPEC 6.2) - Stamp frontmatter on the template md files that lack it; only
personal/README.mdandraw/README.mdwere missing frontmatter (all other template files already carryname/version;titleis optional, sowiki/index.md,wiki/log.md, and the other READMEs already conform)
- Decide between project agent rule 10.1 (
- Update the root AGENTS.md structure diagram to match reality:
llm-wiki lives at
src/llm-wiki/(packaged template), andsrc/also containswiki/,run_tests.py,run_sanity.py,settings_dev.json,utils.sh,install_packages.sh - Raise test coverage from 87% to the required 90%
(
settings_dev.json) sorun_sanity.pycan pass - Document the
src/llm-wiki/__init__.pysetuptools hack (hyphenated, non-importable data-only "package") inpyproject.tomlcomments - Remove
src/llm-wiki/utils/scripts/example.py— won't do:wiki initno longer skips.pyfiles (it now makes a 1:1 copy of the template, soexample.pyships to user wikis; user decision, 2026-09-27; SPEC 7.1.1, version 0.30.0) - Deduplicate conftest:
src/conftest.pyvssrc/tests/conftest.py- Consolidated into
src/tests/conftest.py(sys.path bootstrap moved there, outer file removed, 2026-09-27); verifiedpython src/run_tests.py(619/619),python -m pytestwithout PYTHONPATH, andpython src/run_sanity.pyall pass — the6055ac6revert was a misdiagnosis: the then-failing sanity was the terminal-width issue below, not the conftest change
- Consolidated into
run_sanity.pyrun fails, need fixing- run sanity should show what exactly to fix
- pytests should show which test are failing,
- markdownlint should show which files are in violation , what line
- same thing for ruff
- run sanity should show what exactly to fix
- Remove the phantom
html_js_lintpackage andhtml-js-lintconsole script frompyproject.tomluntil SPEC 7.15 is implemented. The directorysrc/html_js_lint/does not exist. Detail is item 2 in the 2026-09-29 audit below
2026-09-29 structure audit
pyproject.toml declares a PEP 621 project and the code lives under
src/, but the tree is nine sibling import packages plus loose scripts.
A src layout is meant to install one import package. Target shape when
this cluster is done:
src/fiona/
├── __main__.py
├── librarian/
├── mdoctor/
├── customlinter/
├── python_ruff/
├── gittool/
├── sanity_runner/
├── wiki/
└── data/llm_wiki/ # templates, no fake package
tests/ # repo root, not installed
Keep a single required entry point:
fiona-assistant = fiona.__main__:main. Optional thin scripts can stay
as extra entry points that call into those submodules. Do this as one
change set: import paths, tests, entry points, and docs move together.
Half-moving a package leaves both layouts on sys.path.
- Collapse the nine top-level packages into
fionasubpackages. Imports,pyproject.toml, tests, and docs moved with the directories.- Current install names, from
packages.findincludeand fromsrc/fiona_assistant.egg-info/top_level.txt:customlinter,fiona,gittool,librarian,llm-wiki,mdoctor,python_ruff,sanity_runner,wiki fiona/__main__.pyimports them as top-level modules (from librarian.librarian import main,from wiki.wiki import wiki_init, and the same forcustomlinter,gittool,mdoctor,python_ruff)- Same-name module pattern to preserve while moving:
librarian/librarian.py,mdoctor/mdoctor.py,python_ruff/python_ruff.py,sanity_runner/sanity_runner.py.gittoolis the odd one: the module isgit_tool.py wikiis a generic top-level name and will collide with any other installed project that also ships awikipackage- After the move,
packages.findincludeshould listfionaonly. An allow-list is howpython_ruffwas shipped late and howhtml_js_lintstayed declared after the directory vanished
- Current install names, from
- Remove the phantom
html_js_lintdeclaration (same checkbox as the item above this section)[project.scripts]mapshtml-js-linttohtml_js_lint.html_js_lint:mainpackages.findincludelistshtml_js_lintsrc/fiona_assistant.egg-info/entry_points.txtstill has that script from the last editable installsrc/html_js_lint/does not exist. Invokinghtml-js-lintfails at import. SPEC 7.15 is the future feature; the entry point should not ship until that directory existshtml5validatoris in runtimedependenciesand is not imported anywhere undersrc/. Drop it with the phantom script, or keep it only when SPEC 7.15 lands
- Delete
src/__init__.py- The file's only content is the comment
# Fiona tools package - An
__init__.pyinsrc/makessrcitself a package whenever the repo root is onsys.path. The src layout exists so packages are importable only after install - That is why bytecode for
run_sanity.pyandrun_tests.pylands insrc/__pycache__/(run_sanity.cpython-313.pyc,run_tests.cpython-313.pyc,__init__.cpython-313.pyc)
- The file's only content is the comment
- Stop tests from editing
sys.pathsrc/tests/conftest.pydoessys.path.insert(0, os.path.dirname(os.path.dirname(...))), which putssrc/on the path, then importslibrarian,mdoctor,python_ruff, andwikias top-level namessrc/tests/__init__.pyexists, sotestsis a package sitting inside the source root. Standard layout is a repo-roottests/directory that is not installed[tool.pytest.ini_options]setstestpaths = ["src/tests"]and has nopythonpath. Coveragesource = ["src"]treats the mixed script-and-package directory as the measured tree- Replacement:
pip install -e .in the dev env, orpythonpath = ["src"]once there is a singlefionapackage. The path edit hides "package not installed" failures - The 2026-09-27 conftest consolidation (outer
src/conftest.pyremoved) stays valid. This item replaces the remainingsys.pathbootstrap, it does not undo that merge
- Decide where the loose
src/files livesrc/run_sanity.pyandsrc/run_tests.pyare modules, not packages. They are absent frompackages.find, so a wheel does not install them. They are how this repo runs its own sanity and tests (python src/run_sanity.py)src/install_packages.sh,src/utils.sh, andsrc/settings_dev.jsonsit in the same directory. Dev config and shell helpers are not package source- Either give
run_sanity/run_testsreal entry points underfiona, or move them to a repo-rootscripts/(ortools/) directory that is not on the package path
- Replace the hyphenated
llm-wikipackage with package dataimport llm-wikiis aSyntaxError.src/llm-wiki/__init__.pyexists only so setuptools detects the directory. Thepyproject.tomlcomment already documents that hack[tool.setuptools.package-data]maps"llm-wiki" = ["*", "**/*"]src/llm-wiki/__init__.pysets__version__ = "0.1.0"while[project]versionis0.36.0. Two versions for one distribution- Move the template tree to something importable, for example
src/fiona/data/llm_wiki/, and load files withimportlib.resources. Updatewiki init/wiki updatepath helpers (get_llm_wiki_source_path) in the same change - Root
README.mdlinks./llm-wiki/AGENTS.md. The file issrc/llm-wiki/AGENTS.md
- Get setuptools metadata out of
src/src/fiona_assistant.egg-info/holdsPKG-INFO,entry_points.txt,requires.txt,SOURCES.txt,top_level.txt. That is install metadata. It belongs at the repo root (or nowhere, if it is only a local editable-install artifact).gitignorealready ignores*.egg-info/, so this directory should not be committed, butwhere = ["src"]can still see it during package discovery- The egg name
fiona_assistantis the setuptools normalization of project namefiona-assistant. Leave the normalization alone; only the directory location is wrong
- Keep a single coverage config, and point it at the code that exists
- Both
.coveragercand[tool.coverage]inpyproject.tomlexist. Coverage reads.coveragercfirst, so the pyproject block is unused .coveragercstill omitssrc/mdoctor/fmlint.pyandsrc/python_ruff/pycustomlint.py. Those modules live undersrc/customlinter/(fmlint.py,pycustomlint.py)- Both files share the same
exclude_lineslist. After deleting.coveragerc, confirmpytest --covstill honors the pyprojectsource,omit, andexclude_lines coverage.xmlat the repo root is a generated report. It is not matched by.gitignore(*.logand.coverageare;coverage.xmlis not)
- Both
- Raise the setuptools build pin so
[project]is valid[build-system]requiresissetuptools>=42- The
[project]table (PEP 621) needs setuptools 61 or newer. An older setuptools that satisfies>=42can fail the build - Set
requires = ["setuptools>=61"](or a current floor) and rebuild the wheel
- Sort runtime dependencies from repo-only tools
ruffis imported bysrc/python_ruff/python_ruff.py, so it stays independenciespylintis independencieswith no version pin, and nothing undersrc/imports it. Move it to thedevextra or drop it- There is no
[tool.ruff]table and noruff.toml. If this repo is linted with Ruff, the rule set belongs inpyproject.toml requires-pythonis>=3.8. Local bytecode insrc/__pycache__/is CPython 3.13. Classifiers only sayProgramming Language :: Python :: 3. The bootstrap examples use 3.11. Record the versions this repo actually runs, and confirm syntax such asstr | Noneis either covered byfrom __future__ import annotationsor by a higherrequires-python
- Make the structure docs match the tree (also listed under P3)
src/README.md"Project Structure" lists onlyfiona/,librarian/,mdoctor/,tests/, andREADME.md. Missing from that diagram:customlinter/,python_ruff/,gittool/,sanity_runner/,wiki/,llm-wiki/,run_sanity.py,run_tests.py,settings_dev.json,utils.sh,install_packages.sh- MDoctor's description in that file is still
[Description to be added], and Contributing is still[Contribution guidelines to be added] - Refresh this diagram again after the package collapse, so the
docs describe
src/fiona/rather than today's sibling packages
Code defects — 2026-09-29 audit
Source review after the package directories moved under src/fiona/.
src/ on disk is fiona/ plus README.md. Tests live in tests/.
The template lives in src/fiona/data/llm_wiki/. The package-move
items below are done. The command defects after them are still open.
Finish the package move
- Point every import at
fiona.<package>src/fiona/__main__.pystill imports top-levellibrarian,mdoctor,customlinter,python_ruff,gittool, andwiki. Those packages aresrc/fiona/<name>/.tests/test_fiona_main.pystill doesfrom librarian import toc_generator.TOOL_LOGGER_NAMESinsrc/fiona/logging_config.pystill listslibrarian,mdoctor,python_ruff, andwiki. Loggers are nowfiona.*viagetLogger(__name__). The parent namefionaalready covers them; drop the stale names.
- Point template and settings paths at
fiona/data/llm_wikiget_llm_wiki_source_pathinsrc/fiona/wiki/wiki.pyandsrc/fiona/wiki/update_manager.pylooks for a siblingllm-wiki(src/fiona/llm-wiki). The template issrc/fiona/data/llm_wiki/.wiki initandwiki updatemiss it. Load it withimportlib.resources(same end state as the P0 llm-wiki package-data item).- The default settings path in
logging_config.py,customlinter/base.py,mdoctor/mdoctor.py, andlibrarian/toc_generator.pyisparent.parent / "llm-wiki" / "fiona-settings.json", which issrc/llm-wiki/.... The file issrc/fiona/data/llm_wiki/fiona-settings.json. src/fiona/run_sanity.pystill lints./src/llm-wikifor frontmatter. Point it atsrc/fiona/data/llm_wiki.
- Match packaging and the test runner to the new tree
pyproject.tomlpackages.findincludeand[project.scripts]still namelibrarian,mdoctor,python_ruff,customlinter,sanity_runner,llm-wiki,wiki,gittool, andhtml_js_lint.librarianandmdoctorconsole scripts import modules that are no longer top-level.include = ["fiona"]does not match subpackages. Usefiona*sofiona.wikiand the rest are installed.[tool.pytest.ini_options]testpathsis stillsrc/tests. The suite istests/.tests/conftest.pywalks two directories up and inserts the repo root. That math used to reachsrc/when the file lived atsrc/tests/conftest.py. Insertsrc/soimport fionaworks without an editable install.src/__init__.pyis already gone. Leave it gone.run_tests.pyandrun_sanity.pyalready sit insrc/fiona/. Give them entry points or documentpython -m fiona.run_tests/python -m fiona.run_sanityin place ofpython src/run_tests.py(SPEC 4.9).
Fixes in commands the spec already calls Implemented
json-lint fixwalks the file list twice (SPEC appendix, SPEC 4.5). The secondfix_filepass finds the files already clean, so the path lines never print. Walk once: one count and one path line per fixed file.json-lint chklint --fixis parsed and ignored (src/fiona/customlinter/jsonlint.py). SPEC 4.5 sayschklint --fixrewrites the same wayfixdoes.json-lint fixtreats any path part namedpersonalorrawas a read-only zone, including a folder outside the wiki. Skip with the wiki-relative check inBaseLinter(src/fiona/customlinter/base.py), the same rule aschklint.gitfailures printError:on stdout and catchException(src/fiona/gittool/git_tool.py). SPEC 4.6 and section 2: the message goes to stderr. Catch the errorsGitToolraises. This is the concrete case of the P2 logging item.wiki graphprintsError: Directory not foundon stdout (src/fiona/wiki/graph_generator.py). The other graph errors use stderr. SPEC section 4: an invalid directory is stderr and exit 1.wiki graphreplaces--wikilinks/--no-wikilinksand--external-linkswithfiona.wiki.graphsettings after the CLI has set them (load_graph_settingsinwiki_graph_main). An explicit flag wins. Settings fill the defaults only (SPEC 4.7.5).wiki graphdrops files abovemax_fileswith a log line only. Print the kept count and the limit on stderr.mdoctor fixrewritespersonal/andraw/(SPEC 4.2, appendix). Constraint 1.2.3 says those zones are read-only for every automated tool. Skip them infix_file_programmatically, then change SPEC 4.2 to say the command skips those zones.graph_generator.pylogs with f-strings. Use lazy%formatting, the same style aslogging_config.py.fiona-assistant python-ruffhas an empty help string and a#TODOinsrc/fiona/__main__.py. Use a one-line help like the other linters.fiona.__main__rewritessys.argvand calls each toolmain(), which parses the arguments a second time. Call the library functions directly so each command is parsed once.
Manual test findings — 2026-10-02
Black-box test from the README and SPEC only. Report:
tests/wikitest/wiki/queries/fiona-assistant-manual-test-2026-10-02.md.
Each item below was re-run on 2026-10-02 before it was written down.
SPEC 1.8.0 holds the requirements. Order is the build order.
- Second
wiki initprints a Python traceback (SPEC 1.2.7, 7.1.1.4). Reproduced: theFileExistsErrorreaches the top level. Catch it in thewiki initCLI handler, print the oneError:line on stderr, exit 1. Then sweep the other handlers insrc/fiona/__main__.pyfor expected errors that escape- Test: second init, stderr has no
Traceback, exit 1, file hashes unchanged
- Test: second init, stderr has no
- Shipped
fiona.log_levelisINFO, the code default isWARNING(SPEC 7.17.3). Every INFO record shows as a redNativeCommandErrorin PowerShell even on exit 0. ShipWARNING, keepfile_log_levelINFO, and update the shipped document in SPEC 5.1- Test: fresh wiki, clean
wiki sanityand a localwiki ingestwrite nothing on stderr;fiona.loghas INFO - Existing wikis get this through
wiki updateas afiona-settings-update.jsonconflict. Check the housekeeping merge action is clear enough for that one key
- Test: fresh wiki, clean
- Ingest log bullet fails the wiki's own lint (SPEC 7.1.5.6.1).
Reproduced in the test wiki:
wiki/log.mdhas MD013 (112 and 156 chars) and MD034 (bare origin URL). Put the origin in a code span and each-> <path>on a two-space continuation line inlog_writer.py- Test: 90-char URL ingest, then
wiki sanityhas no markdownlint hit inwiki/log.md; still one bullet per run
- Test: 90-char URL ingest, then
mdoctor chklintandmdoctor fixreject a file path (SPEC 4.2, 7.4.4). Reproduced: both printis not a valid directory. Accept a single.mdfile the wayfmlint chklintdoes- Test:
fixon one file lists only that file; other hashes unchanged; missing path givesdoes not exist
- Test:
- Post-ingest hint for leftover style violations
(SPEC 7.1.5.5.6). Depends on the
mdoctorfile item above. PrintNext: fix the style violations listed by fiona-assistant mdoctor chklint <page>.beforeNext: refine - "Relative path fails on Windows" is a misdiagnosis
(SPEC 7.1.5.2.7).
./ingest/x.mdand.\ingest\x.mdboth work from the wiki root. The tester's file was already moved toraw/articles/tester-note.mdby the earlier absolute-path run (SPEC 7.1.5.3.6), so the second run failed. Make the error say so: add(moved to <raw path> by an earlier ingest)when the missing path is underingest/andraw/has that name- Test: both separators ingest; second run gets the hint
--dry-runlogsarchived <raw path>although nothing is written (SPEC 7.1.5.8.1). Reproduced. Logdry run: would archive/dry run: would writeinstead- Housekeeping sanity details repeat the whole PyMarkdown report
(SPEC 7.2.5.1). One line per violation:
<check>: <path>:<line>:<column> <code> <message> - Housekeeping logs
WARNING Overwriting existing file: wiki_graph.htmlon every run (SPEC 7.1.4.5.9.1). Addwiki graph -f/--force; housekeeping passes it and logs at INFO fiona-assistant --version(SPEC 2.3). Reproduced: argparse rejects it. Print the same string asingest.tool- README "After ingest" and "On Windows" notes (SPEC 7.18.4), and the same after-ingest loop in the Fiona skill. Bump both versions
- Wiki root discovery from
wiki/(SPEC 7.20) is already tracked under P2. The tester calls it daily friction for agents. It is blocked on the Appendix C rule; decide that rule, then consider moving SPEC 7.20 to P1 librarian moveis already tracked under P1. The tester hitinvalid choice; no new action, since SPEC 4.1.2 marks itNOT_IMPLEMENTEDand the README does not list it- Decide whether
git undoandgit cleanprompt on a TTY (SPEC Appendix C). The tester lost agit savecommit withgit undo. Listed under P4 as well
Ingest of a URL that needs a login
Users paste links to Jenkins, Confluence, SharePoint, and ticket
pages that need a login. Today a 401 or 403 exits 1 with the generic
Download failed, and an SSO redirect that ends on a 200 login page
is archived under raw/ and converted as if it were the source.
SPEC 1.9.0 holds the requirements (7.1.5.2.8, 7.1.5.2.9, 7.1.5.14).
Spec written 2026-10-05; built 2026-10-07 (SPEC 1.13.0). Order is
the build order.
- Detect a login in the URL fetcher (SPEC 7.1.5.2.8). In
src/fiona/wiki/ingest/fetchers.py, raise a newAuthRequiredError(IngestError)that carries the URL and the reason, before anything is written toingest/- HTTP 401 and 403 from
HTTPError: reason<code> <phrase>. Other non-2xx stayDownload failed(SPEC 7.1.5.2.3) - Redirect check: final URL differs from the requested URL
and
<host><path>matcheslogin_url_patterns(case-insensitive, query string ignored) - Login form check:
text/htmlbody with an<input>whosetypeispassword, any case or quoting. Run it on the bytes already read, so the size limit still applies first - Tests with a fake opener, no network: 401, 403, redirect to
/login?from=..., 200 page with<INPUT TYPE='password'>, a/docs/loginpage with no redirect that still ingests, and a 404 that still exits 1. Each auth case leavesingest/,raw/,wiki/, andwiki/log.mdunchanged
- HTTP 401 and 403 from
login_url_patternssetting (SPEC 5.1). Add it tosrc/fiona/wiki/ingest/settings.pywith the default list from SPEC 5.1. A user list replaces the default. A regex that does not compile is skipped with one warning- Tests: default list, user override, one bad regex
- Exit 4 and the two stderr lines (SPEC 4.7.6). Catch
AuthRequiredErrorin the ingest CLI handler, print theError: Authentication required: <url>: <reason>line and theNext:line, exit 4. URL without userinfo.--jsonstdout stays empty (SPEC 7.1.5.13);--dry-runexits 4 the same way- Tests: exit code, both stderr lines, empty stdout with
--json, dry run, noTraceback(SPEC 1.2.7)
- Tests: exit code, both stderr lines, empty stdout with
--origin URL(SPEC 7.1.5.2.9). Valid only with a local file,http/httpsonly, both checks before fetch. Setsingest.origin,origin_kind: url,final_url: null, feeds thehtmlconverter's base URL and the log bullet. Slug stays the file stem- Tests: saved HTML with a relative link
/abecomeshttps://host/a; URL source plus--originexits 1; bad scheme exits 1; no network call in any case
- Tests: saved HTML with a relative link
- Fiona skill (SPEC 7.1.5.14). In
src/fiona/data/llm_wiki/utils/skills/fiona/SKILL.md: add exit 4 to the section 4.6 exit table and--originto its options; rewrite the authentication bullet in section 16.2 into the three steps (use a tool with access, else ask the user to save it intoingest/, never ask for credentials) with the--origincommand. Bump the skill version - README ingest notes (SPEC 7.18.4): one line on exit 4 and
--origin - Verify the spec (project agent rule 2.1 step 4): drop the
NOT_IMPLEMENTEDmarkers on 4.7.6, 7.1.5.2.8, 7.1.5.2.9, 7.1.5.14, and thelogin_url_patternsprose in 5.1; set the Appendix B row toImplemented; bump SPEC - [-] Open questions left in SPEC Appendix C, not part of this
work: credential support (cookie file, token,
--header) and a JSON error object on exit 4
HTML block sweep for ingest
wiki ingest of a landing page kept only one testimonial card,
because the html converter takes the first <article> as the whole
page (repro: tests/wikitest/raw/articles/agent-harness.html, page
wiki/sources/agent-harness.md). SPEC 1.10.0 replaces that with a
block sweep that scores every block as keep, review, or drop
(SPEC 7.1.5.4.5.6, settings in 5.1). Spec written 2026-10-07;
built 2026-10-07 (SPEC 1.13.0). Order is the build order. Bootstrap first
(. .\bootstrap.ps1) and load the python-tdd skill.
- Settings (SPEC 5.1). In
src/fiona/wiki/ingest/settings.pyaddhtml_min_words: int = 4andhtml_max_link_density: float = 0.5toIngestSettingsanddefault_settings(). Inload_ingest_settings()read the nestedhtmlmap:min_wordsis anint >= 1, not abool;max_link_densityis anintorfloatin0..1, not abool. Anything else keeps the default- Tests: defaults, valid override,
0,true,"4",1.5each fall back
- Tests: defaults, valid override,
- Hand the settings to the converter.
HtmlConverter.__init__takesmin_wordsandmax_link_densitywith the same defaults, soHtmlConverter()still works in existing tests.default_registry(settings: IngestSettings | None = None)registersHtmlConverter(...)as an instance (ConverterRegistry.registeralready accepts instances). Indefault_pipeline()load the settings once, before the registry, and pass the same object to both- Test: a wiki with
min_words: 2makes a three-word<div>keepthrough the full pipeline
- Test: a wiki with
- New module
src/fiona/wiki/ingest/converters/html_sweep.pywith no format code outside it. Keephtml.pyas the converter shell: decode, parse, drop, absolutize, title, meta, then call the sweepBlockdataclass:nodes(the element, or the run of nodes of a loose-text block),kind(heading,p,list,pre,table,quote,figure,loose),text,link_chars,words,verdict,reason- Shadow roots (SPEC 7.1.5.4.5.6.1): before the drop loop in
html.py,unwrap()everytemplate[shadowrootmode], then drop the remainingtemplatetags. Check thathtml.parserkeeps template children as normal children; add a test that proves it - Layout tables: a
tablewhosetd/thholds atable,div, or any block tag is not a block; walk its cells as containers - Walk (SPEC 7.1.5.4.5.6.1.1, 7.1.5.4.5.6.1.2): recursive over
<body>. A block tag (not a layout table) is emitted whole and not entered. A container's children are read in order: text nodes and inline tags join the current run; a block or container child flushes the run as onelooseblock, then is walked. An inline tag that holds a block or container descendant (a card as<a><div>...</div></a>) is treated as a container. A tag with a hyphen in its name is a container - Text (SPEC 7.1.5.4.5.6.2): replace each
brwith a space, thenget_text()with no separator, then" ".join(text.split()). No separator keeps<b>Fa</b>stas one word.link_charsis the folded text length of theatags inside the block.wordsislen(text.split()) - Ancestors: one helper
_inside(block, predicate)that walksparentup to<body>. For a loose run use the first node's parent. Hidden test:hiddenattribute,aria-hidden="true", or astylematchingdisplay\s*:\s*none|visibility\s*:\s*hidden(case-insensitive) - Classify in document order (SPEC 7.1.5.4.5.6.3 to .5) with
one
seenset of case-folded texts of earlierkeepandreviewblocks. Order of checks: no letter or digit (any(ch.isalnum() ...)) -> drop; short and seen -> drop; insidenav/button/select/hidden -> drop;chrome;repeat;links(link_chars / len(text) > max_link_density);short(onlypandloose); elsekeep - Render (SPEC 7.1.5.4.5.6.6): each block through
markdownifywith the current options (ATX,-). A loose run is the joinedstr()of its nodes. Strip each result; skip one that renders empty. Wrapreviewblocks in<!-- fiona:review <reason> -->and<!-- /fiona:review -->, each on its own line. Join blocks with one blank line - Report (SPEC 7.1.5.4.5.6.7): append
html sweep: <k> keep, <r> review, <d> dropto the warnings. Nokeepand at least onereview->partialwith the second warning. The empty-body rule of 7.1.5.4.5.5 still runs first
- Wire it into
html.py. Run_absolutizeon<body>before the sweep, so link rewriting covers every block. Passbodyinstead of the old root to_html_title. Remove thearticle/mainroot line - Check the page passes the wiki's own lint: the raw page goes
through
mdoctor fixafter writing. Confirm the markers stay on their own lines, MD033 allows HTML comments, and no blank line inside a marker pair is collapsed. Add one test that runswiki sanityon an ingested fixture with review blocks - Fixtures under
tests/fixtures/html/(or inline strings, matchingtests/test_ingest_edges.py), one test per SPEC acceptance of 7.1.5.4.5.6:landing.html: no<main>, section with<h2>and a twelve-word<div>, then two<article>cards. All present, in order (the agent-harness case)boilerplate.html: nav list, footer paragraph,<span>•</span>,Sharetwice, twelve-word paragraph twice, all-link list, two-word<div>. Check the verdicts and reasonschrome,repeat,links,shortencyclopedia.html: sidenav, infobox, paragraphs with<sup><a>citations, reference list, navbox tableqa.html: question, two answers with the samepre, vote counts42,7,3, relatedasideforum.html: custom-element posts, one in<template shadowrootmode="open">, nested replies, one reply withstyle="display: none"layout-table.html: table cells that holddivposts give one block per post and no pipe table- Determinism: convert the same fixture twice, compare bodies
byte for byte, and check the
html sweep:counts
- Re-check the old tests.
test_html_keeps_article_link_and_table(tests/test_ingest.py) must still pass: nav gone, article text, absolute/a, pipe table. Reviewtest_html_edges(tests/test_ingest_edges.py) for asserts that relied on the first-<article>root; a change there is a behavior change, say so in the commit - Manual check: in
tests/wikitest, runfiona-assistant wiki ingest <agent-harness URL> --force(orwiki ingest raw/articles/agent-harness.html --force) and confirm the sections "Test and evaluate your agents systematically" through "Agent Testing Resources" and both customer cards are in the page - Fiona skill (SPEC 7.1.5.4.5.6.9). In
src/fiona/data/llm_wiki/utils/skills/fiona/SKILL.mdsection 18.5 add the step: for eachfiona:reviewblock, keep it by deleting the two marker lines or delete the block with them; a refined page has no marker. Mention the reasonschrome,repeat,links,shortin one line each. Bump the skill version - Verify the spec (project agent rule 2.1 step 4): drop the
NOT_IMPLEMENTEDon 7.1.5.4.5.6, remove the "until the block sweep replaces that rule" sentence and the first-<article>rule from 7.1.5.4.5.2, drop "not built yet" from thehtml.*settings prose in 5.1, set the Appendix B row toImplemented, bump SPEC. Runpython scripts/run_tests.pyandpython scripts/run_sanity.py --non-interactive - [-] Open, not part of this work (now in SPEC Appendix C, with
image-only blocks): per-host URL rewrites before
fetch (for example Reddit to
old.reddit.com), and pages that build their text with JavaScript - after sweep,run the automatic md fix from pymarkdown
wiki search
A plain keyword search over the wiki that prints
<path>:<line>: <text> and shortens long lines around the match.
SPEC 1.11.0 holds the requirements (SPEC 4.7.7, 7.1.6, settings in
5.1). Spec written 2026-10-07; nothing below is built. Order is the
build order. Bootstrap first and load the python-tdd skill.
- Settings (SPEC 5.1). New
SearchSettingsdataclass withmax_line_chars: int = 160andmax_results: int = 200, andload_search_settings(wiki_root)that readsfiona.wiki.search.max_line_charsis anint >= 20,max_resultsanint >= 0, neither abool; anything else keeps the default. Factor the JSON walk tofiona.wiki.<name>out ofload_ingest_settings()into one shared helper instead of copying it- Tests: defaults, overrides,
19,-1,true, a brokenfiona-settings.json
- Tests: defaults, overrides,
- New module
src/fiona/wiki/search.py, pure functions plus two dataclasses, no printing inside the search itself:SearchMatch(path: str, line: int, text: str, columns: list[int])andSearchResult(keywords, matches, total, files, truncated)iter_files(wiki_root, includes) -> list[pathlib.Path](SPEC 7.1.6.1.1, 7.1.6.1.3): default["wiki/**/*.md"]; each--includereplaces the default; union ofwiki_root.glob(pattern), files only, deduplicated, sorted by the POSIX relative path. Python 3.11**matches zero directories, sowiki/**/*.mdincludeswiki/index.md; add a test for itread_lines(path)(SPEC 7.1.6.1.2):read_bytes(), strip a UTF-8 BOM,decode("utf-8", errors="replace"),splitlines(). Line numbers start at 1. Frontmatter lines countmatch_columns(line, keywords, case_sensitive, all_words)(SPEC 7.1.6.2): literalstr.findloop per keyword, oncasefold()copies unless case-sensitive; return the sorted 1-based columns of every occurrence, or[]when the line does not match (any keyword by default, every keyword withall_words). Never compile a regex from user inputshorten(text, start, end, max_chars) -> str(SPEC 7.1.6.4), wheretextis the stripped line andstart/endthe span of the first match in it:len(text) <= max_chars-> returntextroom = max_chars - 8(worst case: both 4-character markers,"... "and" ..."). Ifend - start >= room, the window istext[start:start + room]- Else split
room - (end - start)evenly:left = s - half,right = e + (rest - half); clamp to0/len(text)and give the unused side's room to the other side - Snap: when
left > 0and the cut falls inside a word, moveleftto just after the next space if that is still<= start; whenright < len(text)and inside a word, moverightback to the last space if that is still>= end. Strip the window - Prefix
"... "only whenleft > 0, suffix" ..."only whenright < len(text). Assertlen(result) <= max_chars
search_wiki(wiki_root, keywords, settings, *, includes, case_sensitive, all_words) -> SearchResult: for each match line,columnsare against the original line (SPEC 4.7.7),textisshorten()of the stripped line around the first column.totalcounts every matching line; only the firstmax_results(all when0) go intomatches;truncatedistotal > len(matches);filescounts files with at least one matchrender_text(result) -> strandrender_json(result) -> str(SPEC 4.7.7): text lines<path>:<line>: <text>, then... <N> more matches. Narrow the keywords or raise max_results.when truncated, thenMatches: <N> lines in <F> files. JSON keys in the order of SPEC 4.7.7
- CLI in
src/fiona/__main__.py: asearchparser under thewikisubparsers (prog="fiona-assistant wiki search"),keywordswithnargs="+",--wikidefault.,--all-words,--case-sensitive,--includewithaction="append",--json. Dispatch next toingest- Not a wiki (no
fiona-settings.json): reuse the ingest messageError: Directory '<path>' is not an llm-wiki directory, exit 1. Empty or whitespace keyword:Error: Empty keyword, exit 1. No traceback (SPEC 1.2.7) - A matched line can hold any Unicode. The CLI has no stdout
encoding handling today, so a cp1252 PowerShell console can
raise
UnicodeEncodeError. Callsys.stdout.reconfigure(errors="replace")in the search handler and test it with a non-ASCII line
- Not a wiki (no
- Tests in
tests/test_wiki_search.py, one per SPEC acceptance of 7.1.6, on atmp_pathwiki:- Default scope vs
--include "**/*.md"and path order (7.1.6.1) Agent Harness works:harness,--case-sensitive,harness eval,--all-words,a.bvsaxb(7.1.6.2)- Two matches on one line print once; summary text (7.1.6.3)
max_line_chars40: match at column 100 of a 200-character line, match in the first 10 characters, a 40-character line unchanged, a 60-character match longer than the window (7.1.6.4)max_results2 with five matches: the... 3 moreline andMatches: 5 ...; JSONtruncated: true,total: 5(7.1.6.5)- No writes: hashes and
st_mtime_nsof every file unchanged (7.1.6.6) - CLI: exit 0 with zero matches, exit 1 for a non-wiki and an
empty keyword, exit 2 with no keyword,
--jsonparses withjson.loads
- Default scope vs
- Fiona skill: add
Where is a term mentioned?->wiki search <keyword> --jsonto the question table in section 5 ("Which command"), and the command with its flags and exit codes to section 6 ("wiki"). Tell the agent to search before reading whole pages. Bump the skill version - README usage section (SPEC 7.18.4): one example line
- Verify the spec (project agent rule 2.1 step 4): drop the
NOT_IMPLEMENTEDon 4.7.7 and 7.1.6.1 to 7.1.6.6, set 7.1.6Status: Implemented, drop "not built yet" from thefiona.wiki.searchprose in 5.1, set the Appendix B row toImplemented, bump SPEC. Runpython scripts/run_tests.pyandpython scripts/run_sanity.py --non-interactive - Open, not part of this work: should
--includereject a pattern that leaves the wiki root (../**), and shouldwiki searchhonor.gitignorelikewiki graph(SPEC 7.1.4.17)
Wiki audit — 2026-10-08
Audit of tests/wikitest against Karpathy's
LLM Wiki
pattern. There are three layers: immutable raw/, an
LLM-compiled wiki/, and a schema. One ingest writes a summary,
then updates the entity and concept pages, index.md, and log.md.
A single source "might touch 10-15 wiki pages". A query reads
index.md first. A lint pass looks for contradictions, orphans,
and concepts that have no page yet.
State on 2026-10-08: the log has 5 ingest entries. Under wiki/
there are 3 source pages and 1 query page. 0 pages are refined,
and there are 0 concept, entity, or comparison pages. Git has 1
commit (the init). wiki sanity reports 72 violations. The
tool's phase works: it fetches, archives, converts, and logs. The
agent's phase (refine, cross-link, index, save) has never run once.
So the wiki is a converted archive, not a compiled knowledge base.
Nothing builds on earlier work yet. Order below is by impact on
that.
Way of working
- No ingest cycle ever finished (SPEC 7.1.5.10, skill 18.5).
All 3 source pages are
status: raw,confidence: zero,tags: [], the oldest since 2026-10-02. Two sources cover the same topic (LangSmith and Wikipedia on agent harnesses), but noconcepts/agent-harness.mdjoins them. Each ingest touched 1 page pluslog.md. Housekeeping only saysWARN, and new ingests keep stacking on the pending pile- Ingest stdout and
--jsonend with the pending count and the refine command, e.g.Next: refine wiki/sources/x.md (3 pages pending) - Skill: do not start a new ingest while
--pendingis not empty, unless the user asked for a batch. Finish the cycle on the page you just wrote first - Decide (P4): housekeeping
ingeststep turnsFAILwhen a raw page is older than a setting (fiona.wiki.housekeeping.max_raw_age_days)
- Ingest stdout and
- Nothing was saved to git after init (AGENTS.md section 4,
skill 2.3). 5 ingests over 3 days are all untracked. The save
trigger is "end of ingest cycle", and no cycle ever ends, so
it never fires. Add a housekeeping
gitstep:WARNwhenwiki/orraw/has uncommitted changes, with the actionfiona-assistant git save --description ... wiki/index.mdis a dead catalog. It listssources/example-com.md(deleted) andtester-note.md, and is missingagent-harness.mdandwiki-agent-harness.md. The tool never edits the index (SPEC 7.1.5.6), and the refine step that should edit it was skipped. So the query path Karpathy relies on (read the index, then drill in) finds nothing- Housekeeping
indexstep: every page underwiki/other thanindex.md,log.md, andschema.mdis linked fromindex.md.WARNwith one detail per missing page - Decide (P4): should ingest add the raw page under an
## Pending (raw)heading inindex.md, so a query can find it before refine? That changes SPEC 7.1.5.6
- Housekeeping
- Schema never set up.
wiki/schema.mdDomain still readsReplace this line with the wiki's subject. The taxonomy is onlywikiandschema, so refine has no tags to pick from. Karpathy has the schema grow with the wiki. HousekeepingWARNwhile the placeholder line is present. Skill 18.5.7: when a page needs a tag that is not there, add it to the taxonomy - Meta content in the user's wiki. The only page an agent wrote
is the fiona-assistant test report in
wiki/queries/. It is about the tool, not the wiki's domain. Its README/SPEC citations are placeholders (https://github.com/). Skill rule: feedback about the tool goes to the fiona-assistant repo.queries/holds answers to the user's questions and cites wiki pages - Template drift goes unnoticed. The wiki runs Fiona skill
1.15.0, but the package ships 2.1.0 (also schema 1.6.0 to
1.6.1, llm-wiki-mod 1.4.0 to 1.4.1, per
wiki check-update). The 1.2.0 converter writesfiona:reviewmarkers. The skill steps that tell the agent what to do with them (18.5.2.1) are only in 2.x, so the agent in this wiki has no instruction for that output.wiki ingestshould warn when the wiki's skill version is older than the one the tool ships:Fiona skill 1.15.0 is older than 2.1.0; run wiki update
Ingest defects found in the audit
- Re-ingesting a URL makes a second raw copy and orphans the
first (SPEC 7.1.5.3.3, 7.1.5.3.5). On 10-07 and 10-08 the
same Wikipedia URL was fetched twice. Both bodies were
102788 bytes but had different bytes (sha256
6c78...and8c07...), so the byte check did not match them, andwiki-agent-harness-2.htmlwas written.--forcethen rewrote the page so it cites only-2.html, and resetcreatedfrom 2026-10-07 to 2026-10-08 andversionto 0.1.0. Nowraw/articles/wiki-agent-harness.htmlis cited by no page- Before fetching, match the normalized URL against
ingest.originandingest.final_urlof existing pages. On a match, printDuplicate origin: <page>, fetch nothing, and exit 0.--force(or a new--refresh) fetches again - On
--force, keepcreated, bumpversioninstead of resetting it, and add the new raw path tosourcesinstead of replacing the list. Mark the log bullet(replaced) - Lint: a file under
raw/that no page cites insources(an orphan raw copy).WARNin housekeeping
- Before fetching, match the normalized URL against
- Tracking parameters are stored everywhere. The LangSmith
origin carries
utm_*,gclid,gbraid,hsa_*, andgad_*(about 700 characters). They end up iningest.origin,final_url, the log bullet (one 717-character line), and every absolutized link in the body. Strip known tracking parameters before fetch, with the list in settings (fiona.wiki.ingest.strip_query_params), and record the clean URL - The landing-page manual check is marked done above (HTML
block sweep, "Manual check"), but
tests/wikitest/wiki/sources/agent-harness.mdstill holds only the one testimonial card. It hastool: fiona-assistant 1.2.0and nohtml sweep:warning. Re-run it with--forceand confirm, or reopen that item conversion: completeon a 56 KB page that came out as one quote. Add a yield check: when the page body has a small fraction of the words of the source's visible text, mark itpartialwith the warninglow yield: <n> of <m> words. That sends the agent to convert by hand (skill 18.3.1.2)- Wikipedia citation markers trip our own linter. Converted
[[1]](https://...#cite_note-...)gives 19MDL002dead-wikilink errors and oneMD011inwiki-agent-harness.md. Have the converter escape a bracketed number ([\[1\]](...)), or havemdlinterskip[[...]]that is directly followed by(. Add theencyclopedia.htmlfixture to thewiki sanitytest - Deleted sources leave references that nothing checks.
example-com.mdand its raw file are gone (lost togit undoon 10-02).index.mdand the query page body still link to them (MDL001 catches that). The query page's frontmattersources: [raw/articles/example-com.html]and the log bullets are not checked. Lintsources:entries that do not exist - Local paths in provenance. A local-file ingest records
C:\Users\nvaldez\repositories\tests\wikitest\ingest\tester-note.mdiningest.originand in the log. That leaks the user name and breaks on another machine. When the source is inside the wiki root, record the path relative to it - Stray
wiki/fiona.log(0 bytes).file_log_location: "fiona.log"resolves against the working directory, so a run fromwiki/makes a second log file. Resolve it against the wiki root
Lint and housekeeping
- Sanity is always red, so it stops being a signal. Of the 72
violations, almost all are in files the tool wrote: log
bullets (tracked above), raw pages (MD013 on 1000-character
paragraphs), and the citation false positives. An agent that
sees
FAILevery run learns to ignore it. Report violations instatus: rawpages as oneingestdetail per page ("fixed by refine"), apart from violations in refined and core pages. This depends on themdoctor fixafter-sweep item above table_of_contents.mdis the template copy from 2026-09-24 and nothing refreshes it. It listsAGENTS.mdheadings that no longer exist (Pre-requisites, Directory Permissions, Linking Rules), alibarianfolder that does not exist, and nothing underwiki/sources/orwiki/queries/. Eitherwiki initgenerates it withlibrarian toc, and housekeeping warns when it is stale, or the template stops shipping it- Housekeeping checks only mechanics. Karpathy's lint also asks
the LLM about meaning: contradictions, superseded claims,
concepts that 2+ sources mention but that have no page, and
missing cross-links. Add a skill section "Semantic lint" that
the agent runs after housekeeping. Housekeeping can feed it
a list of bold terms and headings that appear in 2+ source
pages but have no page under
concepts/orentities/
P1 — Wiki Lifecycle and Version Control
wiki inithardening (SPEC 7.1.1)- Warn or refuse when the target is already an llm-wiki directory
- Do not clobber user content on re-run (idempotent)
fiona-assistant wiki update(SPEC 7.1.2)- Create
src/wiki/update_manager.pywith stub functions - Add CLI commands:
wiki check-updateandwiki update - Add comprehensive test suite (48 tests)
- Update core md files / scripts to current template versions
(compare
versionfrontmatter) - Update directory structure when the template changed
- Skip locally-modified files and report them
- Create
fiona-assistant wiki sanity(SPEC 7.1.3)- Run all linters: mdlint, python-ruff, shellcheck (once P2 lands)
- Dead-link check (relative and wiki-internal links)
- Git version control for llm-wiki (SPEC 7.12)
- GitPython library wrapper
- Basic operations:
save— commit current edits withyyyy-mm-dd HH:MM:SS <description>snapshot—saveplus a git tagyyyy-mm-dd_HH-MM-SS_<tag_name>restore— create new commit restoring to<tag_name>stateclean— discard current edits (staging area);git statusmust be clean afterwardsundo— go back one commit; error if the working tree is dirty and suggestcleanfirst
fiona-assistant librarian move <source> <target>(SPEC 4.1.2, SPEC 7.3.3). Moves a file or folder, rewrites the relative links inside each moved markdown file, and rewrites every link elsewhere in the wiki that points at a moved path, so no dead link is left (SPEC 7.3.3.6). Lifecycle is project agent rule 2.1. The spec text is written. Start at step 2- Step 0, shared link scanner with no behavior change.
mdlinter.py(MARKDOWN_LINK,INLINE_CODE, fence toggle,_is_ignorable_target),graph_generator.py(MARKDOWN_LINK_PATTERN), andingest/page_writer.py(_FENCE,_LINK,_IMAGE) each parse links their own way. Do not merge them in this task. Write the new scanner so it can replace them later, and leave a P2 item for that merge - New module
src/fiona/librarian/link_rewriter.py, pure functions, no disk writesLinkSpandataclass:start,end(offsets in the line),path,suffix(#fragmentor?query),title,angled(bool),kind(inline,image,reference)iter_links(line) -> Iterator[LinkSpan]. Inline links, images, and reference definitions (^ {0,3}\[id\]:\s+path). Parse<path with spaces>and the three title forms ("t",'t',(t)). Mask inline code before matching (replace with spaces so offsets stay valid)is_rewritable(path) -> bool: false for empty,#..., a URI scheme (re.match(r"^[A-Za-z][A-Za-z0-9+.-]*:", p); that also coversC:drive letters, which are absolute), and a leading/or\. SPEC 7.3.3.4.1 and 7.3.3.4.2new_link_path(old_path, old_dir, new_dir, moves) -> str | None.urllib.parse.unquote, resolve againstold_dir, returnNonewhen it does not exist (counted unresolved). Map the resolved target throughmoves(dict[Path, Path], old absolute to new absolute, with a folder entry covering its children). Thenos.path.relpath(target, new_dir)with/format_path(new, old_path, angled): keep./when the old path had it; outside<...>write a space as%20and keep any%xxthe old path used (quote only characters that the old path quoted, plus spaces). SPEC 7.3.3.3.2rewrite_text(text, old_file, new_file, moves, *, inbound=False) -> RewriteResult(text, rewritten, unresolved, hits). Split withsplitlines(keepends=True)so CRLF stays. Skip the frontmatter block (---first line through the next---) and fenced blocks (```or~~~, matched to the opening fence length). Rebuild each line from the spans right to left so offsets stay valid. A span whose new text equals the old text is not counted (SPEC 7.3.3.3.3)- Inbound mode (
inbound=True, file did not move, soold_file == new_file): rewrite only a link whose resolved target is a key inmovesor lies under a moved folder. Every other link is left alone and not counted as unresolved (SPEC 7.3.3.6.2).hitscounts the links that point at a moved path; the caller uses it for theread-onlycount without writing rewrite_wikilinks(text, renames) -> (text, count).renamesmaps old stem (lowercase) to new stem. Reuse theWIKILINKgrammar frommdlinter.py, but capture the#sectionand|labelparts so they are written back unchanged. Same fence, inline-code, and frontmatter skipping asrewrite_text(SPEC 7.3.3.6.3)
- New module
src/fiona/librarian/mover.pyMovePlandataclass:source,final,files(list of(old, new)absolute paths),rewrites(dict[path_after_move, RewriteResult]for every markdown file that changes, moved or not),read_only(dict[path, int]from SPEC 7.3.3.6.4),is_dirMoveError(Exception)carrying the exact stderr text from SPEC 7.3.3.2resolve_final(source, target) -> Path: an existing folder target meanstarget / source.name(SPEC 7.3.3.1.2)check(source, final, wiki_root): raiseMoveErrorin this order: source missing, final exists, final equals source orfinal.is_relative_to(source)for a folder, then read-only zone for source or final. Get the root withfind_wiki_root(source.parent)fromtoc_generator.py(SPEC 7.3.1.2.6). Test the first part of the path relative to that root againstREADONLY_DIRS. Do not copy_file_toc_single, which matches any path part namedpersonalorraw, the same defect listed forjson-lint fixin the 2026-09-29 auditplan(source, target, wiki_root) -> MovePlan: callcheck, list files (rglob("*")for a folder, hidden included; SPEC 7.3.3.1.3), buildmoves, read each markdown file as bytes, detect a UTF-8 BOM, decode, callrewrite_text. Disk is read only- Inbound scan inside
plan(SPEC 7.3.3.6): walk every*.mdunderwiki_root, skip hidden folders (.git[[[^|]|(.)]] included) and the moved files. Callrewrite_text(..., inbound=True)on each. A file whose first part relative to the root is inREADONLY_DIRSgoes toplan.read_onlywith itshitscount and is never put inrewrites(SPEC 7.3.3.6.4). A file with no change is not added (SPEC 7.3.3.6.5). Read each file once; a large wiki is one pass over its markdown - Wikilink renames: build
renamesfrom moved markdown files whose stem changed. Drop an old stem when any other markdown file under the root, after the move, still has that stem (case-insensitive). Runrewrite_wikilinkson every scanned file, moved files included, and add its count to the samerewrotetotal for that file apply(plan): createfinal.parent, move withshutil.move(source, final)(one call moves a folder), then write each changed markdown file at its path after the move withwrite_bytes(re-add the BOM, no newline translation). WrapOSErrorintoMoveError(f"Error: {exc}")(SPEC 7.3.3.7)report_lines(plan, dry_run) -> list[str]: the line shapes from SPEC 4.1.2, paths relative to the current directory with/,[dry-run]prefix when asked. Onemovedline for the source, then therewrote,unresolved, andread-onlylines sorted by path, then the summary line with file count, link count, rewritten file count, unresolved, and read-onlymove_main(source, target, dry_run) -> int: plan, print, apply unless dry run.MoveErrorgoes to stderr, return 1. No git call (SPEC 7.3.3.8)
- CLI in
src/fiona/__main__.py: add amovesubparser underlibrarian_subparserswithsource,target, and--dry-run, helpMove a file or folder and fix its relative links. Dispatch straight tomove_main(...)andsys.exitits return value. Do not rewritesys.argv(open item in the 2026-09-29 audit). Also addmoveto the standalonelibrarianscript inlibrarian.py, or record in SPEC 2 that it is reached only throughfiona-assistant - Tests,
tests/test_link_rewriter.py(pure, no disk excepttmp_pathtargets for existence)- One test per link form: inline, image, reference
definition,
<angled path>, each title form,#fragment,?query,./prefix kept,%20kept, space quoted - Unchanged cases from SPEC 7.3.3.4:
https://,mailto:,#anchor,/abs,C:/abs,[[wikilink]], fenced (backtick and tilde), inline code,<a href>,<https://autolink>, frontmattersources: - A dead link is left as is and counted unresolved
- Two links on one line are both rewritten at the right offsets
- CRLF input gives CRLF output
- Inbound mode rewrites a link to a moved file and to a file under a moved folder, and leaves a dead link to an unrelated path untouched and uncounted
rewrite_wikilinkskeeps#sectionand|label, matches the stem case-insensitively, and skips fenced and inline code
- One test per link form: inline, image, reference
definition,
- Tests,
tests/test_librarian_move.py, one per acceptance in SPEC 7.3.3, on atmp_pathwiki withfiona-settings.json- Move into an existing folder; move to a new path that creates parents; folder move keeps layout and removes the empty source (SPEC 7.3.3.1)
- Each refusal in SPEC 7.3.3.2: exact stderr line, exit 1, tree hash before equals tree hash after (helper that SHA-256s every file plus the folder list)
- The
wiki/a.md->wiki/concepts/a.mdexample and itsrewrote 2 link(s)line (SPEC 7.3.3.3) - Folder move: sibling link unchanged,
../index.mdbecomes../../index.md - Link from one moved file to another moved file (folder move, and a link that leaves the folder and comes back)
- BOM, CRLF, and frontmatter
versionunchanged; a moved PNG keeps its SHA-256 (SPEC 7.3.3.5) - Inbound acceptances in SPEC 7.3.3.6:
wiki/index.md[a](a.md#top)becomes[a](concepts/a.md#top)with itsrewrote 1 link(s)line; folder move rewritesnotes/x.mdandnotes/img/p.pnginwiki/log.md;[[a#intro|A page]]becomes[[b#intro|A page]]on a rename and stays whenwiki/other/a.mdexists;raw/n.mdis byte-identical with aread-only 1 link(s)line; a file with no reference is byte-identical - End-to-end: after a move with no
read-onlyand nounresolvedline,find_dead_links(wiki_root)frommdlinter.pyreturns nothing new compared with before the move --dry-runstdout equals the real run's stdout with the prefix, and the tree is unchanged (SPEC 7.3.3.7)OSErrorfromshutil.move(patch it) gives stderr and 1- Git wiki: commit count unchanged (SPEC 7.3.3.8)
- CLI test through
fiona.__main__.mainwithsys.argvpatched, astests/test_fiona_main.pydoes fortoc
- Verify: remove each
NOT_IMPLEMENTEDin SPEC 4.1.2 and 7.3.3, set the 7.3.3 status toImplemented, update Appendix B, drop "(planned)" from the Appendix D row, addmovetosrc/fiona/librarian/README.md(SPEC 7.18). Runpython scripts/run_tests.pyandpython scripts/run_sanity.py --non-interactive, bump the SPEC version - Manual check on a copy of a real wiki: move a page into a
subfolder, then run
fiona-assistant wiki sanityand confirm the move added no MDL001 or MDL002, in the moved page or in the pages that link to it
- Step 0, shared link scanner with no behavior change.
P2 — Quality Tooling
- Restore the run_sanity progress heartbeat (
[WAITING]lines while checks run) on top of the shared SanityRunner engine (SPEC 7.14.1); dropped when the engine was generalizedSanityRunner.run_alltakes optionalprogressandheartbeatcallbacks; check lifecycle logged at INFO- run_sanity passes stdout callbacks (DONE markers and WAITING lines, assigned via a carriage-return constant); wiki sanity output unchanged
- Frontmatter linter (fmlint) (SPEC 7.5)
- Implement
src/customlinter/fmlint.py - Write TDD tests for fmlint
- Add CLI integration:
fiona-assistant fmlint chklint [directory] - Integrate into
run_sanity.pyandwiki sanity - Add settings support via
fiona-settings.json - SKILL.md name, folder, and description checks (FML004-FML006)
- Implement
- Generalize compress and format violations for all linters
(SPEC 6.6, 6.1)
BaseLinterprovidescompress_violations()andformat_compressed_violations(); report header comes from thelinter_nameclass attributefmlintdelegatesformat_violationsandlint_directorycompression to the shared base methods (output unchanged)- Migrate
mdoctor(compress_violations/format_violations, same compressed shape as fmlint) onto the shared base format - Migrate
python_ruff(different compressed shape:dict[rel_path, list[dict]]) onto the shared base format - Keep all linter reports diffable: same sections, same order (update SPEC 7.4/7.8 and tests together)
- JSON linter (json-lint) (SPEC 7.11)
- Implement
src/customlinter/jsonlint.py - Write TDD tests for json-lint
- JSON syntax validation using the standard
jsonmodule - JSON style rules (indentation, trailing commas, key ordering)
- CLI:
fiona-assistant json-lint chklintandfix - Settings via the
jsonsection offiona-settings.json - Integrated into
run_sanity.pyandwiki sanity
- Implement
- Shellcheck library (SPEC 7.9)
- add shellcheck to toml
- Wrap shellcheck like
python_ruff(AI-friendly violation report) - Clear install-hint error when shellcheck is missing
- Lint fenced code blocks inside MD files (SPEC 7.10)
- Python blocks via Ruff, shell blocks via shellcheck
- Logging library instead of
printfor diagnostics (SPEC 7.17)- Shared
fiona.logging_config.setup_logging()on CLI entry points - Error and warning diagnostics go to stderr through logging.
CLI reports stay stdout prints.
git_tool,graph_generator, and several CLIs stillprinterrors
- Shared
- Enable every Ruff rule (
select = ["ALL"]inpyproject.toml)- The table under
[tool.ruff]currently leaves Ruff on its built-in default. That skips style, docstrings, line length, pyupgrade, and other families - Turn on
ALL, then clear the findings onsrc/or add an explicitignoreonly where a rule fights the spec (CLIprinton stdout is one) - Keep
test_lint_detects_unused_importgreen. It must find unused-importF401even when another rule is reported first
- The table under
- Clear existing lint violations (SPEC 1 constraint: Ruff and
PyMarkdown must pass)
- Ruff: 0 violations — removed unused imports and unnecessary
passstatements, sorted imports, made dates tz-aware (UTC), added explicitcheck=Falseto subprocess.run, addedfrom __future__ import annotationsto wiki.py - Markdownlint: 0 violations — fixed blanks around headings/lists,
line length, fence languages; mdoctor now enables the pymarkdown
front-matterextension (files with frontmatter no longer raise phantom MD003/MD022/MD041) and ignores the frontmattertitlefor MD025
- Ruff: 0 violations — removed unused imports and unnecessary
- Settings manager,
fiona-assistant setup(SPEC 7.16) - When running sanity, make sure that markdownlint tests also include outer dir of the src/ because there are markdown files there
- in
run_sanity.py, show the failing test function and assert ex. - add pyspellchecker in spec for spelling checks in markdown
- add in spec, we add a new function in toc_generator,
- generate toc for files.
- All headings except H1
#will be included - the toc will be generated below the H1,
- make a logic to detect of a toc exists then regenerate to update.
- make a logic to add section numbers, (also add section numbers to the actual headings to preserve link is working)
- in fiona-settings.json, add specific section to modify parameters such as, max heading to include e.g min:2, max:6, sectionize: true/false, ignore_files:[], etc
fiona-assistant librarian toc file <path_to_file>will generate the toc for specific filefiona-assistant librarian toc file *will generate the toc for all files recursively in the current directory/llm-wiki dir if detected
- All headings except H1
- generate toc for files.
- test_md_frontmatter.py 43/44 passed, 1 failed
- TestParseFrontmatter : test_parse_valid_frontmatter
- assert 'Test Document' == 'Test Documen'
- test_mdoctor.py 33/33 passed
- prevent the pytest short summary missing from output
WARNING run_tests: pytest short summary missing from output; failure details come from the junit report (check PYTEST_ADDOPTS, --no-summary, or plugin output)
-
Fix terminal-width-dependent failure of the failure-report integration tests (
TestFailedTestReportIntegrationinsrc/tests/test_run_sanity.py)- Symptom:
test_real_error_shows_function_and_message(and its assert-based sibling) fail onpython src/run_sanity.pyin an interactive terminal with the report literally showingRuntimeErro..., but pass whenCIis set in the environment - Root cause: pytest trims the short-summary line
ERROR <nodeid> - <message>to the terminal width (_format_trimmedin_pytest/terminal.py; width fromshutil.get_terminal_size, i.e.COLUMNSor the tty) unlessCI/BUILD_NUMBERis set or-vvis used. The temp-test nodeids (~117 chars,../../../../tmp/pytest-of-user/...) push the line past a ~140-column terminal, so the message is cut toRuntimeErro....run_tests._parse_failed_tests_outputcaptures that trimmed text as a real (non-generic) message and_merge_failure_detailskeeps it over the junit report's fullRuntimeError: boom, so the truncated text reaches the report and the exact-match assertion fails - Reproduce:
env -u CI COLUMNS=140 python -m pytest -qfails;CI=true python -m pytest -qpasses. Bisect context: last commit where sanity passes in a plain terminal is43cb7f3("clean up pytests");e750a3d/cb7c3d8introduced the env-sensitive integration tests (057ac8badditionally has a genuinely broken test, fixed later) - Recommended fix (TDD): in
run_tests.py, treat short-summary messages that end in...as unreliable — same as the generic fallback — so_merge_failure_detailslets the junit entry (which always carries the full message) replace them. Write the failing test first: unit-test_merge_failure_detailswith a...-trimmed regex entry vs a junit entry, assert junit wins - Alternative one-liner: set
CI=1in the sanitized subprocess environment in_sanitize_pytest_env(pytest skips summary trimming whenCIis set); keep junit parsing as the safety net
- Symptom:
-
Make
scripts/run_sanity.pyfast on Windows (2026-09-29 investigation). A clean run takes about 2 to 3.5 minutes on Windows and at most 20 seconds on Linux- Symptom, measured with
LOG_LEVEL=INFOand--non-interactive. The machine had other load during the run, so the absolute times are high. The ratios holdpyteststook 03:26 andmarkdownlinttook 00:26.ruff,frontmatter_lint,pycustomlint, andjsonlinteach took 00:00SanityRunner.run_allruns the checks in one thread each. The total is the slowest check, which ispytestspython -m pytest testsalone, without coverage, took 165 s for 962 tests. The sanity run adds--cov, which makes it slower
- Root cause: each
gitprocess is slow to start on Windows, and the run starts hundreds of them- Measured cost per
subprocess.runon this Windows box:git --versionabout 198 ms,python -c passabout 150 ms,cmd /c exitabout 51 ms. On Linux a git call takes about 1 to 3 ms C:\Program Files\Git\cmd\git.exeis a wrapper that startsmingw64\bin\git.exe, so each call creates two processes. Endpoint security that scans each new process likely adds to the cost
- Measured cost per
- Source 1: pymarkdown
respect_gitignorestarts git twice for every directory it walksfiona-settings.jsonsetsrespect_gitignore: true.MDoctor.scan_file_programmaticallypasses it toPyMarkdownApi.scan_path(..., recurse_if_directory=True)- In
application_file_scanner(a pymarkdownlnt dependency), each directory in theos.walkloop callsGitProcessor.get_check_ignores. That runsgit rev-parse --show-topleveland thengit check-ignore(application_file_scanner.py, about line 850;git_processor.py) - One more batched
git check-ignoreruns over the final file list (__remove_any_gitignored_paths) - Repo-level
run_markdownlintscans.. That walks the repo root, including.git,.ruff_cache, and the other directories that are not ignored. It took 18 to 26 s alone, which is longer than the whole Linux run MDoctor._ignored_by_gitignorereads.gitignorewithout git, but it only filters the violations afterwards. It does not stop pymarkdown from walking and calling git
- Source 2: the test suite runs real sanity scans and real git
- Slowest single item: the session fixtures in
tests/conftest.py.pristine_wikicallswiki_init(1.7 s).pristine_check_resultscallsrun_wiki_checks. The two together take 24 s of setup ontest_sanity_manager.py::TestSanityChecksRegistry::test_registry_drives_results - The initialized wiki is a git repo with 96 files and 75
directories. Timed per check,
run_wiki_checksspends 29.2 s inmarkdownlint: about 150 git calls at about 190 ms each.frontmatter_linttakes 0.1 s,python-ruff0.3 s,mdlinter0.1 s, andjsonlint0.0 s - Other tests repeat full wiki sanity scans at 1.3 to 2.6 s
each (
test_sanity_manager.py,test_fiona_main.pytest_main_wiki_sanity*,test_mdoctor.py::TestMDoctorGitignore::test_scan_skips_gitignored_files) test_gittool.py: thegit_repofixture uses GitPython, which starts realgitprocesses. Setup takes 1.3 to 2 s per test, and calls take up to 3.4 s (test_restore_by_full_tag,test_restore_by_partial_tag,test_undo,test_clean)wiki_inittests take 2 to 3.3 s each (test_wiki.py,test_fiona_main.py::test_main_wiki_init). Each copies the template and runs a git init and commitTestFailedTestReportIntegrationintest_run_sanity.pystarts a nested pytest subprocess, about 2.2 s per test- The other ~900 tests average about 60 ms each. Temp directory and file creation is slow on Windows, more so with antivirus scanning
- Slowest single item: the session fixtures in
- Recommended fixes, in order of impact (TDD, SPEC first when
behavior changes)
- Stop pymarkdown from calling git. Pass
respect_gitignore=Falsetoscan_pathandfix_path. Buildexclude_patternsfrom the.gitignoredirectory names thatMDoctor._gitignore_dir_namesalready parses, plus.git. Or collect the file list with the librarian and scan each file. This removes nearly all git calls from themarkdownlintcheck and from the wiki-sanity tests. Done inMDoctor._pymarkdown_path_options. Patterns are omitted when the scan root is outside the working directory, because the matcher rejects those paths. The post-scan directory check still drops ignored files. SPEC 7.4.3 - Make the pristine and minimal wiki fixtures cheaper. Use a
wiki that is not a git repo, or turn off
respect_gitignorein the fixture's settings. The scan change above removes the git calls frompristine_check_results, which was the 29 s cost. The fixtures themselves are unchanged.wiki_initis still about 1.7 s once per session - Share one session-scoped git repo across the read-only
test_gittool.pytests. Keep per-test repos only for tests that change state. Fixture isread_only_repo - Run pytest in parallel with
pytest-xdist(-n auto) inTestRunner.run_pytest. Confirm coverage still combines correctly (pytest-covsupports xdist). Tried on 2026-09-30: coverage still merged at 96.0%, and pytests went from 01:56 to 01:40. Worker progress lines ([gwN] [100%] FAILED path) do not match the short-summary parser, soTestFailedTestReportIntegrationreportedTests: 0/1 passedand dropped the assert text. The flag was removed. Parallel workers are not a win on this box - Optional, outside the repo: exclude the repo directory and
git.exefrom antivirus scanning, if IT policy allows it
- Stop pymarkdown from calling git. Pass
- Acceptance: record Windows and Linux timings for
python scripts/run_sanity.py --non-interactivebefore and after the change. Windows should finish well under 1 minute. Linux was not re-measured (this machine is Windows). The todo baseline is at most 20 s on Linux. Windows before, this machine: about 02:50 in the session that wrote the git fix, and the 2026-09-29 note above (pytests 03:26, markdownlint 00:26). Windows after the git fix, shared read-only repo, andos.pathsep(2026-09-30,LOG_LEVEL=INFO): pytests 01:54, markdownlint 00:10, SANITY PASSED 01:54 (968 tests, coverage 96.0%). An earlier pass the same day was 01:56. That is under the old 2 to 3.5 minutes and still above 1 minute. The remaining time is pytest itself: temp files, the tests that still start git, and nested pytest - Side finding, not a speed issue:
run_tests.pyjoinsPYTHONPATHwith":"(f"{src_dir}:{current_pythonpath}"). On Windows the separator is;. Useos.pathsep. Done. SPEC 4.9
- Symptom, measured with
-
in toc generator settings, per-file glob overrides in the
filesection (SPEC 7.3.1.2) -
instead of current directory as default, find a fiona-settings.json in the current directory / subdirectory, that will be the llm-wiki root and default directory for commands (SPEC 7.20)
-
Enable the pymarkdown
markdown-tablesextension infiona-settings.jsonso table rules (MD055/MD056/MD058) run inrun_sanity.py(SPEC 5.1 known limitation)- Blocked: pymarkdownlnt 0.9.40 crashes with BadTokenizationError
on list lines containing pipes (e.g. generated
table_of_contents.mdentries); upgrade pymarkdownlnt once fixed and re-run sanity
- Blocked: pymarkdownlnt 0.9.40 crashes with BadTokenizationError
on list lines containing pipes (e.g. generated
P3 — Metadata, Backup, and Documentation
- Backup script for llm-wiki so users can recover if something goes wrong (SPEC 7.13)
- Frontmatter tags on MD files (SPEC 6.2)
versiondatetag- Quality signal:
confidence contestedflag (true if claims are disputed)
- Project documentation (SPEC 7.18)
- Root
README.mdlinksSPEC.md - Root README and
AGENTS.mddocumentbootstrapandbootstrap.ps1 - Refresh
librarian/README.mdandmdoctor/README.mdfor the current subcommands (list,toc,chklint,fix) - Refresh
src/README.mdstructure (wiki/,python_ruff/,run_tests.py,run_sanity.py,settings_dev.json). Full gap list is the last item of the P0 2026-09-29 structure audit, including the root README link to./llm-wiki/AGENTS.md
- Root
- Process hygiene: use semantic commit prefixes
(
feat:,fix:,docs:, ...) per project agent rule 8.1 fiona-assistant wiki graph(SPEC 7.1.4): link parser, graph, HTML template, and CLI insrc/wiki/graph_generator.py- Draw only important graph nodes until a parent is clicked
(SPEC 7.1.4.16)
- Setting
fiona.wiki.graph.node_importance_min_point 0, or a missing key, draws every node- A positive integer hides a node whose
unique_pointsis below that integer until one of its parents is clicked
- Setting
- Omit gitignored markdown from the wiki graph (SPEC 7.1.4.17)
- Failing tests in
tests/test_graph_generator.pyfor the acceptances in SPEC 7.1.4.17, 7.1.4.17.1, and 7.1.4.17.2:secret.mdabsent from JSON nodes, edges, and orphans; a link to it creates no edge;max_files1 still keepskept.md;drafts/omitsdrafts/note.mdand keepswiki/note.md; a# secret.mdcomment does not omit;notes/.gitignoreomitsnotes/scratch.mdonly; with no.gitignore,.git/*.mdis absent and a root page stays - In
wiki_graph_main(src/fiona/wiki/graph_generator.py), drop a match before themax_filescut. Read patterns withgitignore_patterns_from_textand test them withpath_is_ignored(src/fiona/wiki/ignore.py), the same rules as SPEC 7.1.2.10. The scan-root.gitignoreapplies to the whole tree. A subdirectory.gitignoreapplies only under that subdirectory. Also drop any path under.git/ - Do not parse an omitted file. Drop an edge whose target resolves to an omitted file, so it never becomes a node
- HTML uses that same graph. No new settings key: the filter always runs
- Housekeeping's orphan list follows automatically, because
graph_stepbuilds the graph throughwiki_graph_main(SPEC 7.2.6)
- Failing tests in
- Graph live view,
wiki graph --serve(SPEC 7.1.4.19). Build this before directory nodes. It is smaller, needs no D3 refactor, and gives a live page to watch while the directory work lands. Lifecycle is project agent rule 2.1. The spec text is written. Start at step 2- Step 0, refactor with no behavior change. Split
wiki_graph_maininsrc/fiona/wiki/graph_generator.pyintocollect_files(directory, include, exclude, max_files),build_graph(directory, files, settings) -> WikiGraph, andrender(graph, settings, format_type, live=False) -> str.wiki_graph_maincalls those three and writes the file. The server calls the same three without writing. The existing tests intests/test_graph_generator.pystay green - Settings helpers beside
consolidate_depth:serve_settings(settings) -> dictreturnshost,port,poll_seconds,debounce_mswith the SPEC 5.1 defaults for absent, mistyped, or out-of-range values. One test per key and one per bad value - CLI in
src/fiona/__main__.py: add--serve,--host,--port(type=int),--no-open. Reject--servewith--format json/--json, and--host/--port/--no-openwithout--serve, throughparser.error(exit 2). A flag wins overfiona.wiki.graph.serve.*. Dispatch--servetowiki_graph_serve(...)in the new module - New module
src/fiona/wiki/graph_server.py, stdlib only (http.server,threading,webbrowser,urllib.parse,pathlib,json,time)GraphState: holdshtml,json,build, and the scanned relative paths, behind athreading.Condition.publish(html, json_text, files)raisesbuildby 1 and callsnotify_all().wait_for_build(after, timeout)returns the newer build orNoneFileWatcher(directory, include, exclude, clock=time.monotonic):snapshot()maps each matched markdown file, the nearestfiona-settings.json, and everyconstellation/*.jsonto(st_mtime_ns, st_size).poll()returns true once the snapshot changed and no newer change came withindebounce_ms. Inject the clock so tests do not sleepGraphRequestHandler(BaseHTTPRequestHandler): route/,/graph.json,/events,/files/<path>, else 404.do_POST,do_PUT,do_PATCH,do_DELETEsend 405. Silencelog_messageintologgingat DEBUG/events: sendContent-Type: text/event-streamandCache-Control: no-cache, then loop onstate.wait_for_build(last, 15). A build writesevent: graphanddata: {"build": n}. A timeout writes: keepalive. ABrokenPipeErrororConnectionResetErrorends that handler quietly/files/<path>:urllib.parse.unquote, join on the scan root,resolve(), then requireis_relative_to(root)and membership in the scanned set. Anything else is 404. Node ids are absolute paths today (the SPEC 7.1.4.3 example shows relative ones), so the page must send the path relative to the scan root. Add arelfield to each page node inrender, or strip the root in the live scriptwiki_graph_serve(...): first build, thenThreadingHTTPServer((host, port), handler)withdaemon_threads = True.OSErroron bind prints the error on stderr and returns 1. Print the URL line from SPEC 7.1.4.19.1 with the real bound port (server.server_address[1]). Warn on stderr for a host that is not loopback. Open the browser unless--no-open- Watcher thread: every
poll_seconds,poll(), then rebuild with the step-0 functions andstate.publish. On any rebuild error log it on stderr, keep the last good state, and send no event (SPEC 7.1.4.19.6). Catch the specific errors the build raises (OSError,UnicodeDecodeError,ValueError), notException -o FILEgiven with--serve: write that file after each successful build. Without-o, write nothingKeyboardInterrupt:server.shutdown(), stop the watcher with athreading.Event, printStopped, return 0
- Live script
src/fiona/wiki/templates/graph_live.jsgraph_template.htmlgets a{{graph_live_js}}marker.render(..., live=False)replaces it with an empty string, so a written file has noEventSource(SPEC 7.1.4.19.7)- In
graph_physics.js, exposewindow.fionaGraph = {getState(), restoreState(state)}. State is the zoom transform,{id: [x, y]}for drawn nodes, the clicked id, the open view hash, each toggle, the theme, and the search text restoreStatesets known nodex/ybefore the simulation starts, skips the opening fade for those nodes, applies the transform with no fit, and lets only new nodes fade in. Store a one-shotsessionStorage['fiona-live-state']. Thepageshowreset (SPEC 7.1.4.5.18) is skipped only when that key exists, and the key is removed after usenew EventSource('/events'). Ongraphwithbuild > window.FIONA_BUILD, save state, thenlocation.reload(). The server writeswindow.FIONA_BUILD = <n>into the served page- Double click on a page uses
'/files/' + relwhenwindow.FIONA_BUILDis defined, elsefile://as today
- Tests,
tests/test_graph_server.py, one per acceptance in SPEC 7.1.4.19. Start the server on port 0 in a thread and read it withurllib.request- URL line printed with a port above 0;
--no-opendoes not callwebbrowser.open(patch it) - Port in use: bind a socket first, expect stderr and 1
- Usage errors exit 2 for each bad flag combination
POST /is 405;/nopeis 404/files/a.mdreturns the text;/files/../secret.mdand/files/%2e%2e/secret.mdare 404- Add
new.md, driveFileWatcherwith a fake clock, and assert/graph.jsongains the node and edge andbuildrises by 1. Deleting a page drops it - Three saves inside the debounce window raise
buildby 1 - Read
/eventswith a short socket timeout and assert oneevent: graphline after a rebuild - A failing rebuild (patch
build_graphto raiseOSError) keeps the old HTML and sends no event - Plain
wiki graphoutput has noEventSource
- URL line printed with a port above 0;
- Verify: flip each
NOT_IMPLEMENTEDin SPEC 7.1.4.19 and the--serveline in SPEC 4.7.5, addserveto the packagedfiona-settings.json, update the SPEC 5.1 prose and Appendix B status, runpython scripts/run_tests.pyandpython scripts/run_sanity.py --non-interactive, bump the SPEC version - Manual check:
fiona-assistant wiki graph <wiki> --serve, add a page linking an existing page, confirm it fades in and the view and theme stay put
- Step 0, refactor with no behavior change. Split
- Directory nodes in the HTML graph (SPEC 7.1.4.18). Spec is
written. Start at step 2
- Step 0, refactor
graph_physics.jswith no behavior change. This is the large part- Key every node and edge by id.
d3.forceLink(...).id(d => d.id)in place ofd.index, and dropnodeData[d.source.index]lookups in the distance callback - Move the one-shot
.data(...).enter()blocks for edges, nodes, labels, glow, and arrows into onerenderView(nodes, edges)that uses.join()keyed by id, resetsvisibleNodes/visibleEdges, recomputesmaxInDegreeandmaxLinkScore, restarts the simulation, and runs the See all fit - Toggles, search, legend, and theme change work from the
current view's arrays, not the global
nodeData - Existing HTML string tests in
tests/test_graph_generator.pystay green. Open a real wiki in the browser and check hover, click, drag, search, and every toggle by hand
- Key every node and edge by id.
- Settings helper
directory_settings(settings) -> (min_files, outside_depth)with the SPEC 5.1 fallbacks. Tests for absent,0, negative, string, and float - Python view builder, so the logic is tested with pytest and
the page only draws. New pure functions in
graph_generator.py(orgraph_views.pyif it grows)collapsed_directories(page_rel_paths, min_files) -> set[str]: count pages per parent directory (direct only), keep those at or abovemin_files, never the scan root""display_id(page, view_dir, collapsed) -> str: walk the page's ancestors belowview_dir; the first collapsed one is the shown id, else the page id. Outside pages useview_dir = ""(the opening view), per SPEC 7.1.4.18.5build_view(view_dir, nodes, edges, collapsed, depth) -> dict: map each edge end throughdisplay_id, drop self loops and edges with both ends inside a collapsed node, sum weights per pair, then add outside targets by a breadth-first walk along edge direction up todepthhops. Mark those nodes and edgesoutside: true- Directory node metrics from the merged edges with
repeated_link_score(SPEC 7.1.4.18.3.2). Page nodes keep their global metrics from SPEC 7.1.4.2.3. Write that down in the SPEC if a test shows it matters - Directory node fields:
id(wiki/concepts/),name(concepts/),kind: "directory",path,pages(every page under it),members(page names, for search) renderembedsviews = {"": {...}, "wiki/concepts/": {...}}for the root and each collapsed directory, plus the existing page data. JSON output is unchanged (SPEC 7.1.4.18.8). Withmin_files0,viewsholds only""and the draw equals today's
- Page script
- Opening draw is
renderView(views[hashDir() || ""]) - Double click on
kind === "directory"setslocation.hash = '#dir=' + encodeURIComponent(id). Ahashchangelistener callsrenderView. An unknown hash opens"" - Breadcrumb bar at the upper center (
All / wiki / concepts, each segment a link to that view) and a "Back" button, disabled on"". Lengths in em (SPEC 1.2.6) - Outside nodes and edges at
opacity: 0.5 - Search: a directory node matches its own name or any
membersentry - Click card for a directory:
path,pages, points, in-degree, out-degree; no frontmatter rows - Legend: a directory shape entry while a directory node is drawn
- Opening draw is
- Folder shape: add a
folderpath generator next tocircle,cloud, andstar. Addshapes.directory: "folder"toconstellation/colorful.jsonand pick a shape fornight-sky,grayscale,dark, andprofessional. Bump each theme'sversion - Tests, one per acceptance in SPEC 7.1.4.18
collapsed_directories: three pages inwiki/concepts/collapse at 3; two direct pages inwiki/notes/plus three inwiki/notes/deep/collapse onlydeep/build_view(""): onewiki/concepts/node with subtitleconcepts/; edge fromwiki/index.mdwith weight 3; noa.md-b.mdedge;unique_points1,total_points3, in-degree 1build_view("wiki/concepts/"): drawsa.md,b.md,c.mdand their edge; depth 1 addswiki/index.mdas outside and notwiki/log.md; depth 2 adds both; depth 0 adds none; a target in collapsedwiki/other/shows aswiki/other/- HTML:
viewsJSON present,shapes.directoryread from the theme,hashchangehandler present, "Back" label present - JSON with
min_files3 still lists every page
- Verify: flip each
NOT_IMPLEMENTEDin SPEC 7.1.4.18, update the 7.1.4 status line, SPEC 5.1 prose, and Appendix B. Adddirectory_nodesto the packagedfiona-settings.json. Measure SPEC 7.1.4.5.11 (1000 files under 10 s) withmin_fileson. Run tests and sanity, bump the SPEC version
- Step 0, refactor
- in spec, refine the sectioning by compiling the modules,
and their functions as subsections
- eg 4.1. librarian then 4.1.1 librarian toc, to make it cleaner
P4 — Design Decisions Needed
- Should
git undoandgit cleanprompt on a TTY, with--yesto skip it? Agents would then pass--yesevery time (SPEC Appendix C, 2026-10-02 manual test) - Librarian: version tracking for non-MD files (SPEC 7.3.2)
- Python files — format TBD
- Bash scripts — format TBD
- Decide the tagging format before implementing
- Define LLM wiki tool roles (SPEC 7.19)
- Document purpose of
src/tools - Classify tools by role:
- Ingestion: moving files from
ingest/toraw/ - Processing: converting
raw/towiki/concepts/ - Maintenance: updating
index.md/log.md
- Ingestion: moving files from
- Document purpose of
Open Questions (SPEC)
- Tagging format for python files and bash scripts (SPEC 7.3.2)
- Git wrapper location: implemented at
src/gittool/git_tool.py(SPEC 7.12) - Wiki root discovery (SPEC 7.20): the nearest
fiona-settings.jsonwins; error when two at the same depth are equally near. Ancestor search is still open.