52 KiB
Fiona Project TODO List
- Done
- P0 — Packaging and Structure Fixes
- Code defects — 2026-09-29 audit
- P1 — Wiki Lifecycle and Version Control
- P2 — Quality Tooling
- P3 — Metadata, Backup, and Documentation
- P4 — Design Decisions Needed
- Open Questions (SPEC)
Ordered by priority. Each item references its section in SPEC.md (the source of truth for requirements and acceptance criteria).
Done
- Include the llm-wiki template as part of the PyPI package
- MD frontmatter parser for version control of core files (SPEC 6.2)
versiontag support
fiona-assistant wiki init— copy packaged template into a target directory (basic flow; hardening tracked under P1)- Windows PowerShell bootstrap (
bootstrap.ps1): repo-local venv under.venvs\<name>, same flags as the bashbootstrap
P0 — Packaging and Structure Fixes
From the 2026-09-23 structure audit, plus the 2026-09-29 layout audit below. These are repo-hygiene items that block a correct install or make the docs contradict the code.
- Fix PyPI packaging so a non-editable install works (wheel build
confirmed
python_ruffis missing entirely)- Add
python_ruffto thepackages.findinclude list inpyproject.toml - Add
src/python_ruff/__init__.py - Add
src/fiona/__init__.py - Re-verify: build a wheel, install it fresh, confirm the
fiona-assistantconsole script runs and the llm-wiki template data ships
- Add
- Resolve the duplicated
raw/zone in the llm-wiki template- Pick one canonical location (
llm-wiki/raw/at the wiki root, per user decision) and removesrc/llm-wiki/wiki/raw/ - Align root
AGENTS.md,src/llm-wiki/AGENTS.md, andwiki/schema.mdon the same location
- Pick one canonical location (
- Reconcile the two competing frontmatter schemas
- Decide between project agent rule 10.1 (
name/version/title) andwiki/schema.md(title/sources/tags); one schema wins:name/versionrequired,titleoptional (user decision, 2026-09-25; documented in SPEC 6.2) - Add missing
typevalues (e.g.toc) to the allowed types list (project agent rule 10.1,wiki/schema.md, SPEC 6.2) - Stamp frontmatter on the template md files that lack it; only
personal/README.mdandraw/README.mdwere missing frontmatter (all other template files already carryname/version;titleis optional, sowiki/index.md,wiki/log.md, and the other READMEs already conform)
- Decide between project agent rule 10.1 (
- Update the root AGENTS.md structure diagram to match reality:
llm-wiki lives at
src/llm-wiki/(packaged template), andsrc/also containswiki/,run_tests.py,run_sanity.py,settings_dev.json,utils.sh,install_packages.sh - Raise test coverage from 87% to the required 90%
(
settings_dev.json) sorun_sanity.pycan pass - Document the
src/llm-wiki/__init__.pysetuptools hack (hyphenated, non-importable data-only "package") inpyproject.tomlcomments - Remove
src/llm-wiki/utils/scripts/example.py— won't do:wiki initno longer skips.pyfiles (it now makes a 1:1 copy of the template, soexample.pyships to user wikis; user decision, 2026-09-27; SPEC 7.1.1, version 0.30.0) - Deduplicate conftest:
src/conftest.pyvssrc/tests/conftest.py- Consolidated into
src/tests/conftest.py(sys.path bootstrap moved there, outer file removed, 2026-09-27); verifiedpython src/run_tests.py(619/619),python -m pytestwithout PYTHONPATH, andpython src/run_sanity.pyall pass — the6055ac6revert was a misdiagnosis: the then-failing sanity was the terminal-width issue below, not the conftest change
- Consolidated into
run_sanity.pyrun fails, need fixing- run sanity should show what exactly to fix
- pytests should show which test are failing,
- markdownlint should show which files are in violation , what line
- same thing for ruff
- run sanity should show what exactly to fix
- Remove the phantom
html_js_lintpackage andhtml-js-lintconsole script frompyproject.tomluntil SPEC 7.15 is implemented. The directorysrc/html_js_lint/does not exist. Detail is item 2 in the 2026-09-29 audit below
2026-09-29 structure audit
pyproject.toml declares a PEP 621 project and the code lives under
src/, but the tree is nine sibling import packages plus loose scripts.
A src layout is meant to install one import package. Target shape when
this cluster is done:
src/fiona/
├── __main__.py
├── librarian/
├── mdoctor/
├── customlinter/
├── python_ruff/
├── gittool/
├── sanity_runner/
├── wiki/
└── data/llm_wiki/ # templates, no fake package
tests/ # repo root, not installed
Keep a single required entry point:
fiona-assistant = fiona.__main__:main. Optional thin scripts can stay
as extra entry points that call into those submodules. Do this as one
change set: import paths, tests, entry points, and docs move together.
Half-moving a package leaves both layouts on sys.path.
- Collapse the nine top-level packages into
fionasubpackages. Imports,pyproject.toml, tests, and docs moved with the directories.- Current install names, from
packages.findincludeand fromsrc/fiona_assistant.egg-info/top_level.txt:customlinter,fiona,gittool,librarian,llm-wiki,mdoctor,python_ruff,sanity_runner,wiki fiona/__main__.pyimports them as top-level modules (from librarian.librarian import main,from wiki.wiki import wiki_init, and the same forcustomlinter,gittool,mdoctor,python_ruff)- Same-name module pattern to preserve while moving:
librarian/librarian.py,mdoctor/mdoctor.py,python_ruff/python_ruff.py,sanity_runner/sanity_runner.py.gittoolis the odd one: the module isgit_tool.py wikiis a generic top-level name and will collide with any other installed project that also ships awikipackage- After the move,
packages.findincludeshould listfionaonly. An allow-list is howpython_ruffwas shipped late and howhtml_js_lintstayed declared after the directory vanished
- Current install names, from
- Remove the phantom
html_js_lintdeclaration (same checkbox as the item above this section)[project.scripts]mapshtml-js-linttohtml_js_lint.html_js_lint:mainpackages.findincludelistshtml_js_lintsrc/fiona_assistant.egg-info/entry_points.txtstill has that script from the last editable installsrc/html_js_lint/does not exist. Invokinghtml-js-lintfails at import. SPEC 7.15 is the future feature; the entry point should not ship until that directory existshtml5validatoris in runtimedependenciesand is not imported anywhere undersrc/. Drop it with the phantom script, or keep it only when SPEC 7.15 lands
- Delete
src/__init__.py- The file's only content is the comment
# Fiona tools package - An
__init__.pyinsrc/makessrcitself a package whenever the repo root is onsys.path. The src layout exists so packages are importable only after install - That is why bytecode for
run_sanity.pyandrun_tests.pylands insrc/__pycache__/(run_sanity.cpython-313.pyc,run_tests.cpython-313.pyc,__init__.cpython-313.pyc)
- The file's only content is the comment
- Stop tests from editing
sys.pathsrc/tests/conftest.pydoessys.path.insert(0, os.path.dirname(os.path.dirname(...))), which putssrc/on the path, then importslibrarian,mdoctor,python_ruff, andwikias top-level namessrc/tests/__init__.pyexists, sotestsis a package sitting inside the source root. Standard layout is a repo-roottests/directory that is not installed[tool.pytest.ini_options]setstestpaths = ["src/tests"]and has nopythonpath. Coveragesource = ["src"]treats the mixed script-and-package directory as the measured tree- Replacement:
pip install -e .in the dev env, orpythonpath = ["src"]once there is a singlefionapackage. The path edit hides "package not installed" failures - The 2026-09-27 conftest consolidation (outer
src/conftest.pyremoved) stays valid. This item replaces the remainingsys.pathbootstrap, it does not undo that merge
- Decide where the loose
src/files livesrc/run_sanity.pyandsrc/run_tests.pyare modules, not packages. They are absent frompackages.find, so a wheel does not install them. They are how this repo runs its own sanity and tests (python src/run_sanity.py)src/install_packages.sh,src/utils.sh, andsrc/settings_dev.jsonsit in the same directory. Dev config and shell helpers are not package source- Either give
run_sanity/run_testsreal entry points underfiona, or move them to a repo-rootscripts/(ortools/) directory that is not on the package path
- Replace the hyphenated
llm-wikipackage with package dataimport llm-wikiis aSyntaxError.src/llm-wiki/__init__.pyexists only so setuptools detects the directory. Thepyproject.tomlcomment already documents that hack[tool.setuptools.package-data]maps"llm-wiki" = ["*", "**/*"]src/llm-wiki/__init__.pysets__version__ = "0.1.0"while[project]versionis0.36.0. Two versions for one distribution- Move the template tree to something importable, for example
src/fiona/data/llm_wiki/, and load files withimportlib.resources. Updatewiki init/wiki updatepath helpers (get_llm_wiki_source_path) in the same change - Root
README.mdlinks./llm-wiki/AGENTS.md. The file issrc/llm-wiki/AGENTS.md
- Get setuptools metadata out of
src/src/fiona_assistant.egg-info/holdsPKG-INFO,entry_points.txt,requires.txt,SOURCES.txt,top_level.txt. That is install metadata. It belongs at the repo root (or nowhere, if it is only a local editable-install artifact).gitignorealready ignores*.egg-info/, so this directory should not be committed, butwhere = ["src"]can still see it during package discovery- The egg name
fiona_assistantis the setuptools normalization of project namefiona-assistant. Leave the normalization alone; only the directory location is wrong
- Keep a single coverage config, and point it at the code that exists
- Both
.coveragercand[tool.coverage]inpyproject.tomlexist. Coverage reads.coveragercfirst, so the pyproject block is unused .coveragercstill omitssrc/mdoctor/fmlint.pyandsrc/python_ruff/pycustomlint.py. Those modules live undersrc/customlinter/(fmlint.py,pycustomlint.py)- Both files share the same
exclude_lineslist. After deleting.coveragerc, confirmpytest --covstill honors the pyprojectsource,omit, andexclude_lines coverage.xmlat the repo root is a generated report. It is not matched by.gitignore(*.logand.coverageare;coverage.xmlis not)
- Both
- Raise the setuptools build pin so
[project]is valid[build-system]requiresissetuptools>=42- The
[project]table (PEP 621) needs setuptools 61 or newer. An older setuptools that satisfies>=42can fail the build - Set
requires = ["setuptools>=61"](or a current floor) and rebuild the wheel
- Sort runtime dependencies from repo-only tools
ruffis imported bysrc/python_ruff/python_ruff.py, so it stays independenciespylintis independencieswith no version pin, and nothing undersrc/imports it. Move it to thedevextra or drop it- There is no
[tool.ruff]table and noruff.toml. If this repo is linted with Ruff, the rule set belongs inpyproject.toml requires-pythonis>=3.8. Local bytecode insrc/__pycache__/is CPython 3.13. Classifiers only sayProgramming Language :: Python :: 3. The bootstrap examples use 3.11. Record the versions this repo actually runs, and confirm syntax such asstr | Noneis either covered byfrom __future__ import annotationsor by a higherrequires-python
- Make the structure docs match the tree (also listed under P3)
src/README.md"Project Structure" lists onlyfiona/,librarian/,mdoctor/,tests/, andREADME.md. Missing from that diagram:customlinter/,python_ruff/,gittool/,sanity_runner/,wiki/,llm-wiki/,run_sanity.py,run_tests.py,settings_dev.json,utils.sh,install_packages.sh- MDoctor's description in that file is still
[Description to be added], and Contributing is still[Contribution guidelines to be added] - Refresh this diagram again after the package collapse, so the
docs describe
src/fiona/rather than today's sibling packages
Code defects — 2026-09-29 audit
Source review after the package directories moved under src/fiona/.
src/ on disk is fiona/ plus README.md. Tests live in tests/.
The template lives in src/fiona/data/llm_wiki/. The package-move
items below are done. The command defects after them are still open.
Finish the package move
- Point every import at
fiona.<package>src/fiona/__main__.pystill imports top-levellibrarian,mdoctor,customlinter,python_ruff,gittool, andwiki. Those packages aresrc/fiona/<name>/.tests/test_fiona_main.pystill doesfrom librarian import toc_generator.TOOL_LOGGER_NAMESinsrc/fiona/logging_config.pystill listslibrarian,mdoctor,python_ruff, andwiki. Loggers are nowfiona.*viagetLogger(__name__). The parent namefionaalready covers them; drop the stale names.
- Point template and settings paths at
fiona/data/llm_wikiget_llm_wiki_source_pathinsrc/fiona/wiki/wiki.pyandsrc/fiona/wiki/update_manager.pylooks for a siblingllm-wiki(src/fiona/llm-wiki). The template issrc/fiona/data/llm_wiki/.wiki initandwiki updatemiss it. Load it withimportlib.resources(same end state as the P0 llm-wiki package-data item).- The default settings path in
logging_config.py,customlinter/base.py,mdoctor/mdoctor.py, andlibrarian/toc_generator.pyisparent.parent / "llm-wiki" / "fiona-settings.json", which issrc/llm-wiki/.... The file issrc/fiona/data/llm_wiki/fiona-settings.json. src/fiona/run_sanity.pystill lints./src/llm-wikifor frontmatter. Point it atsrc/fiona/data/llm_wiki.
- Match packaging and the test runner to the new tree
pyproject.tomlpackages.findincludeand[project.scripts]still namelibrarian,mdoctor,python_ruff,customlinter,sanity_runner,llm-wiki,wiki,gittool, andhtml_js_lint.librarianandmdoctorconsole scripts import modules that are no longer top-level.include = ["fiona"]does not match subpackages. Usefiona*sofiona.wikiand the rest are installed.[tool.pytest.ini_options]testpathsis stillsrc/tests. The suite istests/.tests/conftest.pywalks two directories up and inserts the repo root. That math used to reachsrc/when the file lived atsrc/tests/conftest.py. Insertsrc/soimport fionaworks without an editable install.src/__init__.pyis already gone. Leave it gone.run_tests.pyandrun_sanity.pyalready sit insrc/fiona/. Give them entry points or documentpython -m fiona.run_tests/python -m fiona.run_sanityin place ofpython src/run_tests.py(SPEC 4.9).
Fixes in commands the spec already calls Implemented
json-lint fixwalks the file list twice (SPEC appendix, SPEC 4.5). The secondfix_filepass finds the files already clean, so the path lines never print. Walk once: one count and one path line per fixed file.json-lint chklint --fixis parsed and ignored (src/fiona/customlinter/jsonlint.py). SPEC 4.5 sayschklint --fixrewrites the same wayfixdoes.json-lint fixtreats any path part namedpersonalorrawas a read-only zone, including a folder outside the wiki. Skip with the wiki-relative check inBaseLinter(src/fiona/customlinter/base.py), the same rule aschklint.gitfailures printError:on stdout and catchException(src/fiona/gittool/git_tool.py). SPEC 4.6 and section 2: the message goes to stderr. Catch the errorsGitToolraises. This is the concrete case of the P2 logging item.wiki graphprintsError: Directory not foundon stdout (src/fiona/wiki/graph_generator.py). The other graph errors use stderr. SPEC section 4: an invalid directory is stderr and exit 1.wiki graphreplaces--wikilinks/--no-wikilinksand--external-linkswithfiona.wiki.graphsettings after the CLI has set them (load_graph_settingsinwiki_graph_main). An explicit flag wins. Settings fill the defaults only (SPEC 4.7.5).wiki graphdrops files abovemax_fileswith a log line only. Print the kept count and the limit on stderr.mdoctor fixrewritespersonal/andraw/(SPEC 4.2, appendix). Constraint 1.2.3 says those zones are read-only for every automated tool. Skip them infix_file_programmatically, then change SPEC 4.2 to say the command skips those zones.graph_generator.pylogs with f-strings. Use lazy%formatting, the same style aslogging_config.py.fiona-assistant python-ruffhas an empty help string and a#TODOinsrc/fiona/__main__.py. Use a one-line help like the other linters.fiona.__main__rewritessys.argvand calls each toolmain(), which parses the arguments a second time. Call the library functions directly so each command is parsed once.
P1 — Wiki Lifecycle and Version Control
wiki inithardening (SPEC 7.1.1)- Warn or refuse when the target is already an llm-wiki directory
- Do not clobber user content on re-run (idempotent)
fiona-assistant wiki update(SPEC 7.1.2)- Create
src/wiki/update_manager.pywith stub functions - Add CLI commands:
wiki check-updateandwiki update - Add comprehensive test suite (48 tests)
- Update core md files / scripts to current template versions
(compare
versionfrontmatter) - Update directory structure when the template changed
- Skip locally-modified files and report them
- Create
fiona-assistant wiki sanity(SPEC 7.1.3)- Run all linters: mdlint, python-ruff, shellcheck (once P2 lands)
- Dead-link check (relative and wiki-internal links)
- Git version control for llm-wiki (SPEC 7.12)
- GitPython library wrapper
- Basic operations:
save— commit current edits withyyyy-mm-dd HH:MM:SS <description>snapshot—saveplus a git tagyyyy-mm-dd_HH-MM-SS_<tag_name>restore— create new commit restoring to<tag_name>stateclean— discard current edits (staging area);git statusmust be clean afterwardsundo— go back one commit; error if the working tree is dirty and suggestcleanfirst
P2 — Quality Tooling
- Restore the run_sanity progress heartbeat (
[WAITING]lines while checks run) on top of the shared SanityRunner engine (SPEC 7.14.1); dropped when the engine was generalizedSanityRunner.run_alltakes optionalprogressandheartbeatcallbacks; check lifecycle logged at INFO- run_sanity passes stdout callbacks (DONE markers and WAITING lines, assigned via a carriage-return constant); wiki sanity output unchanged
- Frontmatter linter (fmlint) (SPEC 7.5)
- Implement
src/customlinter/fmlint.py - Write TDD tests for fmlint
- Add CLI integration:
fiona-assistant fmlint chklint [directory] - Integrate into
run_sanity.pyandwiki sanity - Add settings support via
fiona-settings.json - SKILL.md name, folder, and description checks (FML004-FML006)
- Implement
- Generalize compress and format violations for all linters
(SPEC 6.6, 6.1)
BaseLinterprovidescompress_violations()andformat_compressed_violations(); report header comes from thelinter_nameclass attributefmlintdelegatesformat_violationsandlint_directorycompression to the shared base methods (output unchanged)- Migrate
mdoctor(compress_violations/format_violations, same compressed shape as fmlint) onto the shared base format - Migrate
python_ruff(different compressed shape:dict[rel_path, list[dict]]) onto the shared base format - Keep all linter reports diffable: same sections, same order (update SPEC 7.4/7.8 and tests together)
- JSON linter (json-lint) (SPEC 7.11)
- Implement
src/customlinter/jsonlint.py - Write TDD tests for json-lint
- JSON syntax validation using the standard
jsonmodule - JSON style rules (indentation, trailing commas, key ordering)
- CLI:
fiona-assistant json-lint chklintandfix - Settings via the
jsonsection offiona-settings.json - Integrated into
run_sanity.pyandwiki sanity
- Implement
- Shellcheck library (SPEC 7.9)
- add shellcheck to toml
- Wrap shellcheck like
python_ruff(AI-friendly violation report) - Clear install-hint error when shellcheck is missing
- Lint fenced code blocks inside MD files (SPEC 7.10)
- Python blocks via Ruff, shell blocks via shellcheck
- Logging library instead of
printfor diagnostics (SPEC 7.17)- Shared
fiona.logging_config.setup_logging()on CLI entry points - Error and warning diagnostics go to stderr through logging.
CLI reports stay stdout prints.
git_tool,graph_generator, and several CLIs stillprinterrors
- Shared
- Enable every Ruff rule (
select = ["ALL"]inpyproject.toml)- The table under
[tool.ruff]currently leaves Ruff on its built-in default. That skips style, docstrings, line length, pyupgrade, and other families - Turn on
ALL, then clear the findings onsrc/or add an explicitignoreonly where a rule fights the spec (CLIprinton stdout is one) - Keep
test_lint_detects_unused_importgreen. It must find unused-importF401even when another rule is reported first
- The table under
- Clear existing lint violations (SPEC 1 constraint: Ruff and
PyMarkdown must pass)
- Ruff: 0 violations — removed unused imports and unnecessary
passstatements, sorted imports, made dates tz-aware (UTC), added explicitcheck=Falseto subprocess.run, addedfrom __future__ import annotationsto wiki.py - Markdownlint: 0 violations — fixed blanks around headings/lists,
line length, fence languages; mdoctor now enables the pymarkdown
front-matterextension (files with frontmatter no longer raise phantom MD003/MD022/MD041) and ignores the frontmattertitlefor MD025
- Ruff: 0 violations — removed unused imports and unnecessary
- Settings manager,
fiona-assistant setup(SPEC 7.16) - When running sanity, make sure that markdownlint tests also include outer dir of the src/ because there are markdown files there
- in
run_sanity.py, show the failing test function and assert ex. - add pyspellchecker in spec for spelling checks in markdown
- add in spec, we add a new function in toc_generator,
- generate toc for files.
- All headings except H1
#will be included - the toc will be generated below the H1,
- make a logic to detect of a toc exists then regenerate to update.
- make a logic to add section numbers, (also add section numbers to the actual headings to preserve link is working)
- in fiona-settings.json, add specific section to modify parameters such as, max heading to include e.g min:2, max:6, sectionize: true/false, ignore_files:[], etc
fiona-assistant librarian toc file <path_to_file>will generate the toc for specific filefiona-assistant librarian toc file *will generate the toc for all files recursively in the current directory/llm-wiki dir if detected
- All headings except H1
- generate toc for files.
- test_md_frontmatter.py 43/44 passed, 1 failed
- TestParseFrontmatter : test_parse_valid_frontmatter
- assert 'Test Document' == 'Test Documen'
- test_mdoctor.py 33/33 passed
- prevent the pytest short summary missing from output
WARNING run_tests: pytest short summary missing from output; failure details come from the junit report (check PYTEST_ADDOPTS, --no-summary, or plugin output)
-
Fix terminal-width-dependent failure of the failure-report integration tests (
TestFailedTestReportIntegrationinsrc/tests/test_run_sanity.py)- Symptom:
test_real_error_shows_function_and_message(and its assert-based sibling) fail onpython src/run_sanity.pyin an interactive terminal with the report literally showingRuntimeErro..., but pass whenCIis set in the environment - Root cause: pytest trims the short-summary line
ERROR <nodeid> - <message>to the terminal width (_format_trimmedin_pytest/terminal.py; width fromshutil.get_terminal_size, i.e.COLUMNSor the tty) unlessCI/BUILD_NUMBERis set or-vvis used. The temp-test nodeids (~117 chars,../../../../tmp/pytest-of-user/...) push the line past a ~140-column terminal, so the message is cut toRuntimeErro....run_tests._parse_failed_tests_outputcaptures that trimmed text as a real (non-generic) message and_merge_failure_detailskeeps it over the junit report's fullRuntimeError: boom, so the truncated text reaches the report and the exact-match assertion fails - Reproduce:
env -u CI COLUMNS=140 python -m pytest -qfails;CI=true python -m pytest -qpasses. Bisect context: last commit where sanity passes in a plain terminal is43cb7f3("clean up pytests");e750a3d/cb7c3d8introduced the env-sensitive integration tests (057ac8badditionally has a genuinely broken test, fixed later) - Recommended fix (TDD): in
run_tests.py, treat short-summary messages that end in...as unreliable — same as the generic fallback — so_merge_failure_detailslets the junit entry (which always carries the full message) replace them. Write the failing test first: unit-test_merge_failure_detailswith a...-trimmed regex entry vs a junit entry, assert junit wins - Alternative one-liner: set
CI=1in the sanitized subprocess environment in_sanitize_pytest_env(pytest skips summary trimming whenCIis set); keep junit parsing as the safety net
- Symptom:
-
Make
scripts/run_sanity.pyfast on Windows (2026-09-29 investigation). A clean run takes about 2 to 3.5 minutes on Windows and at most 20 seconds on Linux- Symptom, measured with
LOG_LEVEL=INFOand--non-interactive. The machine had other load during the run, so the absolute times are high. The ratios holdpyteststook 03:26 andmarkdownlinttook 00:26.ruff,frontmatter_lint,pycustomlint, andjsonlinteach took 00:00SanityRunner.run_allruns the checks in one thread each. The total is the slowest check, which ispytestspython -m pytest testsalone, without coverage, took 165 s for 962 tests. The sanity run adds--cov, which makes it slower
- Root cause: each
gitprocess is slow to start on Windows, and the run starts hundreds of them- Measured cost per
subprocess.runon this Windows box:git --versionabout 198 ms,python -c passabout 150 ms,cmd /c exitabout 51 ms. On Linux a git call takes about 1 to 3 ms C:\Program Files\Git\cmd\git.exeis a wrapper that startsmingw64\bin\git.exe, so each call creates two processes. Endpoint security that scans each new process likely adds to the cost
- Measured cost per
- Source 1: pymarkdown
respect_gitignorestarts git twice for every directory it walksfiona-settings.jsonsetsrespect_gitignore: true.MDoctor.scan_file_programmaticallypasses it toPyMarkdownApi.scan_path(..., recurse_if_directory=True)- In
application_file_scanner(a pymarkdownlnt dependency), each directory in theos.walkloop callsGitProcessor.get_check_ignores. That runsgit rev-parse --show-topleveland thengit check-ignore(application_file_scanner.py, about line 850;git_processor.py) - One more batched
git check-ignoreruns over the final file list (__remove_any_gitignored_paths) - Repo-level
run_markdownlintscans.. That walks the repo root, including.git,.ruff_cache, and the other directories that are not ignored. It took 18 to 26 s alone, which is longer than the whole Linux run MDoctor._ignored_by_gitignorereads.gitignorewithout git, but it only filters the violations afterwards. It does not stop pymarkdown from walking and calling git
- Source 2: the test suite runs real sanity scans and real git
- Slowest single item: the session fixtures in
tests/conftest.py.pristine_wikicallswiki_init(1.7 s).pristine_check_resultscallsrun_wiki_checks. The two together take 24 s of setup ontest_sanity_manager.py::TestSanityChecksRegistry::test_registry_drives_results - The initialized wiki is a git repo with 96 files and 75
directories. Timed per check,
run_wiki_checksspends 29.2 s inmarkdownlint: about 150 git calls at about 190 ms each.frontmatter_linttakes 0.1 s,python-ruff0.3 s,mdlinter0.1 s, andjsonlint0.0 s - Other tests repeat full wiki sanity scans at 1.3 to 2.6 s
each (
test_sanity_manager.py,test_fiona_main.pytest_main_wiki_sanity*,test_mdoctor.py::TestMDoctorGitignore::test_scan_skips_gitignored_files) test_gittool.py: thegit_repofixture uses GitPython, which starts realgitprocesses. Setup takes 1.3 to 2 s per test, and calls take up to 3.4 s (test_restore_by_full_tag,test_restore_by_partial_tag,test_undo,test_clean)wiki_inittests take 2 to 3.3 s each (test_wiki.py,test_fiona_main.py::test_main_wiki_init). Each copies the template and runs a git init and commitTestFailedTestReportIntegrationintest_run_sanity.pystarts a nested pytest subprocess, about 2.2 s per test- The other ~900 tests average about 60 ms each. Temp directory and file creation is slow on Windows, more so with antivirus scanning
- Slowest single item: the session fixtures in
- Recommended fixes, in order of impact (TDD, SPEC first when
behavior changes)
- Stop pymarkdown from calling git. Pass
respect_gitignore=Falsetoscan_pathandfix_path. Buildexclude_patternsfrom the.gitignoredirectory names thatMDoctor._gitignore_dir_namesalready parses, plus.git. Or collect the file list with the librarian and scan each file. This removes nearly all git calls from themarkdownlintcheck and from the wiki-sanity tests. Done inMDoctor._pymarkdown_path_options. Patterns are omitted when the scan root is outside the working directory, because the matcher rejects those paths. The post-scan directory check still drops ignored files. SPEC 7.4.3 - Make the pristine and minimal wiki fixtures cheaper. Use a
wiki that is not a git repo, or turn off
respect_gitignorein the fixture's settings. The scan change above removes the git calls frompristine_check_results, which was the 29 s cost. The fixtures themselves are unchanged.wiki_initis still about 1.7 s once per session - Share one session-scoped git repo across the read-only
test_gittool.pytests. Keep per-test repos only for tests that change state. Fixture isread_only_repo - Run pytest in parallel with
pytest-xdist(-n auto) inTestRunner.run_pytest. Confirm coverage still combines correctly (pytest-covsupports xdist). Tried on 2026-09-30: coverage still merged at 96.0%, and pytests went from 01:56 to 01:40. Worker progress lines ([gwN] [100%] FAILED path) do not match the short-summary parser, soTestFailedTestReportIntegrationreportedTests: 0/1 passedand dropped the assert text. The flag was removed. Parallel workers are not a win on this box - Optional, outside the repo: exclude the repo directory and
git.exefrom antivirus scanning, if IT policy allows it
- Stop pymarkdown from calling git. Pass
- Acceptance: record Windows and Linux timings for
python scripts/run_sanity.py --non-interactivebefore and after the change. Windows should finish well under 1 minute. Linux was not re-measured (this machine is Windows). The todo baseline is at most 20 s on Linux. Windows before, this machine: about 02:50 in the session that wrote the git fix, and the 2026-09-29 note above (pytests 03:26, markdownlint 00:26). Windows after the git fix, shared read-only repo, andos.pathsep(2026-09-30,LOG_LEVEL=INFO): pytests 01:54, markdownlint 00:10, SANITY PASSED 01:54 (968 tests, coverage 96.0%). An earlier pass the same day was 01:56. That is under the old 2 to 3.5 minutes and still above 1 minute. The remaining time is pytest itself: temp files, the tests that still start git, and nested pytest - Side finding, not a speed issue:
run_tests.pyjoinsPYTHONPATHwith":"(f"{src_dir}:{current_pythonpath}"). On Windows the separator is;. Useos.pathsep. Done. SPEC 4.9
- Symptom, measured with
-
in toc generator settings, per-file glob overrides in the
filesection (SPEC 7.3.1.2) -
instead of current directory as default, find a fiona-settings.json in the current directory / subdirectory, that will be the llm-wiki root and default directory for commands (SPEC 7.20)
-
Enable the pymarkdown
markdown-tablesextension infiona-settings.jsonso table rules (MD055/MD056/MD058) run inrun_sanity.py(SPEC 5.1 known limitation)- Blocked: pymarkdownlnt 0.9.40 crashes with BadTokenizationError
on list lines containing pipes (e.g. generated
table_of_contents.mdentries); upgrade pymarkdownlnt once fixed and re-run sanity
- Blocked: pymarkdownlnt 0.9.40 crashes with BadTokenizationError
on list lines containing pipes (e.g. generated
P3 — Metadata, Backup, and Documentation
- Backup script for llm-wiki so users can recover if something goes wrong (SPEC 7.13)
- Frontmatter tags on MD files (SPEC 6.2)
versiondatetag- Quality signal:
confidence contestedflag (true if claims are disputed)
- Project documentation (SPEC 7.18)
- Root
README.mdlinksSPEC.md - Root README and
AGENTS.mddocumentbootstrapandbootstrap.ps1 - Refresh
librarian/README.mdandmdoctor/README.mdfor the current subcommands (list,toc,chklint,fix) - Refresh
src/README.mdstructure (wiki/,python_ruff/,run_tests.py,run_sanity.py,settings_dev.json). Full gap list is the last item of the P0 2026-09-29 structure audit, including the root README link to./llm-wiki/AGENTS.md
- Root
- Process hygiene: use semantic commit prefixes
(
feat:,fix:,docs:, ...) per project agent rule 8.1 fiona-assistant wiki graph(SPEC 7.1.4): link parser, graph, HTML template, and CLI insrc/wiki/graph_generator.py- Draw only important graph nodes until a parent is clicked
(SPEC 7.1.4.16)
- Setting
fiona.wiki.graph.node_importance_min_point 0, or a missing key, draws every node- A positive integer hides a node whose
unique_pointsis below that integer until one of its parents is clicked
- Setting
- Omit gitignored markdown from the wiki graph (SPEC 7.1.4.17)
- Failing tests in
tests/test_graph_generator.pyfor the acceptances in SPEC 7.1.4.17, 7.1.4.17.1, and 7.1.4.17.2:secret.mdabsent from JSON nodes, edges, and orphans; a link to it creates no edge;max_files1 still keepskept.md;drafts/omitsdrafts/note.mdand keepswiki/note.md; a# secret.mdcomment does not omit;notes/.gitignoreomitsnotes/scratch.mdonly; with no.gitignore,.git/*.mdis absent and a root page stays - In
wiki_graph_main(src/fiona/wiki/graph_generator.py), drop a match before themax_filescut. Read patterns withgitignore_patterns_from_textand test them withpath_is_ignored(src/fiona/wiki/ignore.py), the same rules as SPEC 7.1.2.10. The scan-root.gitignoreapplies to the whole tree. A subdirectory.gitignoreapplies only under that subdirectory. Also drop any path under.git/ - Do not parse an omitted file. Drop an edge whose target resolves to an omitted file, so it never becomes a node
- HTML uses that same graph. No new settings key: the filter always runs
- Housekeeping's orphan list follows automatically, because
graph_stepbuilds the graph throughwiki_graph_main(SPEC 7.2.6)
- Failing tests in
- Graph live view,
wiki graph --serve(SPEC 7.1.4.19). Build this before directory nodes. It is smaller, needs no D3 refactor, and gives a live page to watch while the directory work lands. Lifecycle is project agent rule 2.1. The spec text is written. Start at step 2- Step 0, refactor with no behavior change. Split
wiki_graph_maininsrc/fiona/wiki/graph_generator.pyintocollect_files(directory, include, exclude, max_files),build_graph(directory, files, settings) -> WikiGraph, andrender(graph, settings, format_type, live=False) -> str.wiki_graph_maincalls those three and writes the file. The server calls the same three without writing. The existing tests intests/test_graph_generator.pystay green - Settings helpers beside
consolidate_depth:serve_settings(settings) -> dictreturnshost,port,poll_seconds,debounce_mswith the SPEC 5.1 defaults for absent, mistyped, or out-of-range values. One test per key and one per bad value - CLI in
src/fiona/__main__.py: add--serve,--host,--port(type=int),--no-open. Reject--servewith--format json/--json, and--host/--port/--no-openwithout--serve, throughparser.error(exit 2). A flag wins overfiona.wiki.graph.serve.*. Dispatch--servetowiki_graph_serve(...)in the new module - New module
src/fiona/wiki/graph_server.py, stdlib only (http.server,threading,webbrowser,urllib.parse,pathlib,json,time)GraphState: holdshtml,json,build, and the scanned relative paths, behind athreading.Condition.publish(html, json_text, files)raisesbuildby 1 and callsnotify_all().wait_for_build(after, timeout)returns the newer build orNoneFileWatcher(directory, include, exclude, clock=time.monotonic):snapshot()maps each matched markdown file, the nearestfiona-settings.json, and everyconstellation/*.jsonto(st_mtime_ns, st_size).poll()returns true once the snapshot changed and no newer change came withindebounce_ms. Inject the clock so tests do not sleepGraphRequestHandler(BaseHTTPRequestHandler): route/,/graph.json,/events,/files/<path>, else 404.do_POST,do_PUT,do_PATCH,do_DELETEsend 405. Silencelog_messageintologgingat DEBUG/events: sendContent-Type: text/event-streamandCache-Control: no-cache, then loop onstate.wait_for_build(last, 15). A build writesevent: graphanddata: {"build": n}. A timeout writes: keepalive. ABrokenPipeErrororConnectionResetErrorends that handler quietly/files/<path>:urllib.parse.unquote, join on the scan root,resolve(), then requireis_relative_to(root)and membership in the scanned set. Anything else is 404. Node ids are absolute paths today (the SPEC 7.1.4.3 example shows relative ones), so the page must send the path relative to the scan root. Add arelfield to each page node inrender, or strip the root in the live scriptwiki_graph_serve(...): first build, thenThreadingHTTPServer((host, port), handler)withdaemon_threads = True.OSErroron bind prints the error on stderr and returns 1. Print the URL line from SPEC 7.1.4.19.1 with the real bound port (server.server_address[1]). Warn on stderr for a host that is not loopback. Open the browser unless--no-open- Watcher thread: every
poll_seconds,poll(), then rebuild with the step-0 functions andstate.publish. On any rebuild error log it on stderr, keep the last good state, and send no event (SPEC 7.1.4.19.6). Catch the specific errors the build raises (OSError,UnicodeDecodeError,ValueError), notException -o FILEgiven with--serve: write that file after each successful build. Without-o, write nothingKeyboardInterrupt:server.shutdown(), stop the watcher with athreading.Event, printStopped, return 0
- Live script
src/fiona/wiki/templates/graph_live.jsgraph_template.htmlgets a{{graph_live_js}}marker.render(..., live=False)replaces it with an empty string, so a written file has noEventSource(SPEC 7.1.4.19.7)- In
graph_physics.js, exposewindow.fionaGraph = {getState(), restoreState(state)}. State is the zoom transform,{id: [x, y]}for drawn nodes, the clicked id, the open view hash, each toggle, the theme, and the search text restoreStatesets known nodex/ybefore the simulation starts, skips the opening fade for those nodes, applies the transform with no fit, and lets only new nodes fade in. Store a one-shotsessionStorage['fiona-live-state']. Thepageshowreset (SPEC 7.1.4.5.18) is skipped only when that key exists, and the key is removed after usenew EventSource('/events'). Ongraphwithbuild > window.FIONA_BUILD, save state, thenlocation.reload(). The server writeswindow.FIONA_BUILD = <n>into the served page- Double click on a page uses
'/files/' + relwhenwindow.FIONA_BUILDis defined, elsefile://as today
- Tests,
tests/test_graph_server.py, one per acceptance in SPEC 7.1.4.19. Start the server on port 0 in a thread and read it withurllib.request- URL line printed with a port above 0;
--no-opendoes not callwebbrowser.open(patch it) - Port in use: bind a socket first, expect stderr and 1
- Usage errors exit 2 for each bad flag combination
POST /is 405;/nopeis 404/files/a.mdreturns the text;/files/../secret.mdand/files/%2e%2e/secret.mdare 404- Add
new.md, driveFileWatcherwith a fake clock, and assert/graph.jsongains the node and edge andbuildrises by 1. Deleting a page drops it - Three saves inside the debounce window raise
buildby 1 - Read
/eventswith a short socket timeout and assert oneevent: graphline after a rebuild - A failing rebuild (patch
build_graphto raiseOSError) keeps the old HTML and sends no event - Plain
wiki graphoutput has noEventSource
- URL line printed with a port above 0;
- Verify: flip each
NOT_IMPLEMENTEDin SPEC 7.1.4.19 and the--serveline in SPEC 4.7.5, addserveto the packagedfiona-settings.json, update the SPEC 5.1 prose and Appendix B status, runpython scripts/run_tests.pyandpython scripts/run_sanity.py --non-interactive, bump the SPEC version - Manual check:
fiona-assistant wiki graph <wiki> --serve, add a page linking an existing page, confirm it fades in and the view and theme stay put
- Step 0, refactor with no behavior change. Split
- Directory nodes in the HTML graph (SPEC 7.1.4.18). Spec is
written. Start at step 2
- Step 0, refactor
graph_physics.jswith no behavior change. This is the large part- Key every node and edge by id.
d3.forceLink(...).id(d => d.id)in place ofd.index, and dropnodeData[d.source.index]lookups in the distance callback - Move the one-shot
.data(...).enter()blocks for edges, nodes, labels, glow, and arrows into onerenderView(nodes, edges)that uses.join()keyed by id, resetsvisibleNodes/visibleEdges, recomputesmaxInDegreeandmaxLinkScore, restarts the simulation, and runs the See all fit - Toggles, search, legend, and theme change work from the
current view's arrays, not the global
nodeData - Existing HTML string tests in
tests/test_graph_generator.pystay green. Open a real wiki in the browser and check hover, click, drag, search, and every toggle by hand
- Key every node and edge by id.
- Settings helper
directory_settings(settings) -> (min_files, outside_depth)with the SPEC 5.1 fallbacks. Tests for absent,0, negative, string, and float - Python view builder, so the logic is tested with pytest and
the page only draws. New pure functions in
graph_generator.py(orgraph_views.pyif it grows)collapsed_directories(page_rel_paths, min_files) -> set[str]: count pages per parent directory (direct only), keep those at or abovemin_files, never the scan root""display_id(page, view_dir, collapsed) -> str: walk the page's ancestors belowview_dir; the first collapsed one is the shown id, else the page id. Outside pages useview_dir = ""(the opening view), per SPEC 7.1.4.18.5build_view(view_dir, nodes, edges, collapsed, depth) -> dict: map each edge end throughdisplay_id, drop self loops and edges with both ends inside a collapsed node, sum weights per pair, then add outside targets by a breadth-first walk along edge direction up todepthhops. Mark those nodes and edgesoutside: true- Directory node metrics from the merged edges with
repeated_link_score(SPEC 7.1.4.18.3.2). Page nodes keep their global metrics from SPEC 7.1.4.2.3. Write that down in the SPEC if a test shows it matters - Directory node fields:
id(wiki/concepts/),name(concepts/),kind: "directory",path,pages(every page under it),members(page names, for search) renderembedsviews = {"": {...}, "wiki/concepts/": {...}}for the root and each collapsed directory, plus the existing page data. JSON output is unchanged (SPEC 7.1.4.18.8). Withmin_files0,viewsholds only""and the draw equals today's
- Page script
- Opening draw is
renderView(views[hashDir() || ""]) - Double click on
kind === "directory"setslocation.hash = '#dir=' + encodeURIComponent(id). Ahashchangelistener callsrenderView. An unknown hash opens"" - Breadcrumb bar at the upper center (
All / wiki / concepts, each segment a link to that view) and a "Back" button, disabled on"". Lengths in em (SPEC 1.2.6) - Outside nodes and edges at
opacity: 0.5 - Search: a directory node matches its own name or any
membersentry - Click card for a directory:
path,pages, points, in-degree, out-degree; no frontmatter rows - Legend: a directory shape entry while a directory node is drawn
- Opening draw is
- Folder shape: add a
folderpath generator next tocircle,cloud, andstar. Addshapes.directory: "folder"toconstellation/colorful.jsonand pick a shape fornight-sky,grayscale,dark, andprofessional. Bump each theme'sversion - Tests, one per acceptance in SPEC 7.1.4.18
collapsed_directories: three pages inwiki/concepts/collapse at 3; two direct pages inwiki/notes/plus three inwiki/notes/deep/collapse onlydeep/build_view(""): onewiki/concepts/node with subtitleconcepts/; edge fromwiki/index.mdwith weight 3; noa.md-b.mdedge;unique_points1,total_points3, in-degree 1build_view("wiki/concepts/"): drawsa.md,b.md,c.mdand their edge; depth 1 addswiki/index.mdas outside and notwiki/log.md; depth 2 adds both; depth 0 adds none; a target in collapsedwiki/other/shows aswiki/other/- HTML:
viewsJSON present,shapes.directoryread from the theme,hashchangehandler present, "Back" label present - JSON with
min_files3 still lists every page
- Verify: flip each
NOT_IMPLEMENTEDin SPEC 7.1.4.18, update the 7.1.4 status line, SPEC 5.1 prose, and Appendix B. Adddirectory_nodesto the packagedfiona-settings.json. Measure SPEC 7.1.4.5.11 (1000 files under 10 s) withmin_fileson. Run tests and sanity, bump the SPEC version
- Step 0, refactor
- in spec, refine the sectioning by compiling the modules,
and their functions as subsections
- eg 4.1. librarian then 4.1.1 librarian toc, to make it cleaner
P4 — Design Decisions Needed
- Librarian: version tracking for non-MD files (SPEC 7.3.2)
- Python files — format TBD
- Bash scripts — format TBD
- Decide the tagging format before implementing
- Define LLM wiki tool roles (SPEC 7.19)
- Document purpose of
src/tools - Classify tools by role:
- Ingestion: moving files from
ingest/toraw/ - Processing: converting
raw/towiki/concepts/ - Maintenance: updating
index.md/log.md
- Ingestion: moving files from
- Document purpose of
Open Questions (SPEC)
- Tagging format for python files and bash scripts (SPEC 7.3.2)
- Git wrapper location: implemented at
src/gittool/git_tool.py(SPEC 7.12) - Wiki root discovery (SPEC 7.20): the nearest
fiona-settings.jsonwins; error when two at the same depth are equally near. Ancestor search is still open.