38 KiB
Fiona Project TODO List
- Done
- P0 — Packaging and Structure Fixes
- Code defects — 2026-09-29 audit
- P1 — Wiki Lifecycle and Version Control
- P2 — Quality Tooling
- P3 — Metadata, Backup, and Documentation
- P4 — Design Decisions Needed
- Open Questions (SPEC)
Ordered by priority. Each item references its section in SPEC.md (the source of truth for requirements and acceptance criteria).
Done
- Include the llm-wiki template as part of the PyPI package
- MD frontmatter parser for version control of core files (SPEC 6.2)
versiontag support
fiona-assistant wiki init— copy packaged template into a target directory (basic flow; hardening tracked under P1)- Windows PowerShell bootstrap (
bootstrap.ps1): repo-local venv under.venvs\<name>, same flags as the bashbootstrap
P0 — Packaging and Structure Fixes
From the 2026-09-23 structure audit, plus the 2026-09-29 layout audit below. These are repo-hygiene items that block a correct install or make the docs contradict the code.
- Fix PyPI packaging so a non-editable install works (wheel build
confirmed
python_ruffis missing entirely)- Add
python_ruffto thepackages.findinclude list inpyproject.toml - Add
src/python_ruff/__init__.py - Add
src/fiona/__init__.py - Re-verify: build a wheel, install it fresh, confirm the
fiona-assistantconsole script runs and the llm-wiki template data ships
- Add
- Resolve the duplicated
raw/zone in the llm-wiki template- Pick one canonical location (
llm-wiki/raw/at the wiki root, per user decision) and removesrc/llm-wiki/wiki/raw/ - Align root
AGENTS.md,src/llm-wiki/AGENTS.md, andwiki/schema.mdon the same location
- Pick one canonical location (
- Reconcile the two competing frontmatter schemas
- Decide between project agent rule 10.1 (
name/version/title) andwiki/schema.md(title/sources/tags); one schema wins:name/versionrequired,titleoptional (user decision, 2026-09-25; documented in SPEC 6.2) - Add missing
typevalues (e.g.toc) to the allowed types list (project agent rule 10.1,wiki/schema.md, SPEC 6.2) - Stamp frontmatter on the template md files that lack it; only
personal/README.mdandraw/README.mdwere missing frontmatter (all other template files already carryname/version;titleis optional, sowiki/index.md,wiki/log.md, and the other READMEs already conform)
- Decide between project agent rule 10.1 (
- Update the root AGENTS.md structure diagram to match reality:
llm-wiki lives at
src/llm-wiki/(packaged template), andsrc/also containswiki/,run_tests.py,run_sanity.py,settings_dev.json,utils.sh,install_packages.sh - Raise test coverage from 87% to the required 90%
(
settings_dev.json) sorun_sanity.pycan pass - Document the
src/llm-wiki/__init__.pysetuptools hack (hyphenated, non-importable data-only "package") inpyproject.tomlcomments - Remove
src/llm-wiki/utils/scripts/example.py— won't do:wiki initno longer skips.pyfiles (it now makes a 1:1 copy of the template, soexample.pyships to user wikis; user decision, 2026-09-27; SPEC 7.1, version 0.30.0) - Deduplicate conftest:
src/conftest.pyvssrc/tests/conftest.py- Consolidated into
src/tests/conftest.py(sys.path bootstrap moved there, outer file removed, 2026-09-27); verifiedpython src/run_tests.py(619/619),python -m pytestwithout PYTHONPATH, andpython src/run_sanity.pyall pass — the6055ac6revert was a misdiagnosis: the then-failing sanity was the terminal-width issue below, not the conftest change
- Consolidated into
run_sanity.pyrun fails, need fixing- run sanity should show what exactly to fix
- pytests should show which test are failing,
- markdownlint should show which files are in violation , what line
- same thing for ruff
- run sanity should show what exactly to fix
- Remove the phantom
html_js_lintpackage andhtml-js-lintconsole script frompyproject.tomluntil SPEC 7.17 is implemented. The directorysrc/html_js_lint/does not exist. Detail is item 2 in the 2026-09-29 audit below
2026-09-29 structure audit
pyproject.toml declares a PEP 621 project and the code lives under
src/, but the tree is nine sibling import packages plus loose scripts.
A src layout is meant to install one import package. Target shape when
this cluster is done:
src/fiona/
├── __main__.py
├── librarian/
├── mdoctor/
├── customlinter/
├── python_ruff/
├── gittool/
├── sanity_runner/
├── wiki/
└── data/llm_wiki/ # templates, no fake package
tests/ # repo root, not installed
Keep a single required entry point:
fiona-assistant = fiona.__main__:main. Optional thin scripts can stay
as extra entry points that call into those submodules. Do this as one
change set: import paths, tests, entry points, and docs move together.
Half-moving a package leaves both layouts on sys.path.
- Collapse the nine top-level packages into
fionasubpackages. Imports,pyproject.toml, tests, and docs moved with the directories.- Current install names, from
packages.findincludeand fromsrc/fiona_assistant.egg-info/top_level.txt:customlinter,fiona,gittool,librarian,llm-wiki,mdoctor,python_ruff,sanity_runner,wiki fiona/__main__.pyimports them as top-level modules (from librarian.librarian import main,from wiki.wiki import wiki_init, and the same forcustomlinter,gittool,mdoctor,python_ruff)- Same-name module pattern to preserve while moving:
librarian/librarian.py,mdoctor/mdoctor.py,python_ruff/python_ruff.py,sanity_runner/sanity_runner.py.gittoolis the odd one: the module isgit_tool.py wikiis a generic top-level name and will collide with any other installed project that also ships awikipackage- After the move,
packages.findincludeshould listfionaonly. An allow-list is howpython_ruffwas shipped late and howhtml_js_lintstayed declared after the directory vanished
- Current install names, from
- Remove the phantom
html_js_lintdeclaration (same checkbox as the item above this section)[project.scripts]mapshtml-js-linttohtml_js_lint.html_js_lint:mainpackages.findincludelistshtml_js_lintsrc/fiona_assistant.egg-info/entry_points.txtstill has that script from the last editable installsrc/html_js_lint/does not exist. Invokinghtml-js-lintfails at import. SPEC 7.17 is the future feature; the entry point should not ship until that directory existshtml5validatoris in runtimedependenciesand is not imported anywhere undersrc/. Drop it with the phantom script, or keep it only when SPEC 7.17 lands
- Delete
src/__init__.py- The file's only content is the comment
# Fiona tools package - An
__init__.pyinsrc/makessrcitself a package whenever the repo root is onsys.path. The src layout exists so packages are importable only after install - That is why bytecode for
run_sanity.pyandrun_tests.pylands insrc/__pycache__/(run_sanity.cpython-313.pyc,run_tests.cpython-313.pyc,__init__.cpython-313.pyc)
- The file's only content is the comment
- Stop tests from editing
sys.pathsrc/tests/conftest.pydoessys.path.insert(0, os.path.dirname(os.path.dirname(...))), which putssrc/on the path, then importslibrarian,mdoctor,python_ruff, andwikias top-level namessrc/tests/__init__.pyexists, sotestsis a package sitting inside the source root. Standard layout is a repo-roottests/directory that is not installed[tool.pytest.ini_options]setstestpaths = ["src/tests"]and has nopythonpath. Coveragesource = ["src"]treats the mixed script-and-package directory as the measured tree- Replacement:
pip install -e .in the dev env, orpythonpath = ["src"]once there is a singlefionapackage. The path edit hides "package not installed" failures - The 2026-09-27 conftest consolidation (outer
src/conftest.pyremoved) stays valid. This item replaces the remainingsys.pathbootstrap, it does not undo that merge
- Decide where the loose
src/files livesrc/run_sanity.pyandsrc/run_tests.pyare modules, not packages. They are absent frompackages.find, so a wheel does not install them. They are how this repo runs its own sanity and tests (python src/run_sanity.py)src/install_packages.sh,src/utils.sh, andsrc/settings_dev.jsonsit in the same directory. Dev config and shell helpers are not package source- Either give
run_sanity/run_testsreal entry points underfiona, or move them to a repo-rootscripts/(ortools/) directory that is not on the package path
- Replace the hyphenated
llm-wikipackage with package dataimport llm-wikiis aSyntaxError.src/llm-wiki/__init__.pyexists only so setuptools detects the directory. Thepyproject.tomlcomment already documents that hack[tool.setuptools.package-data]maps"llm-wiki" = ["*", "**/*"]src/llm-wiki/__init__.pysets__version__ = "0.1.0"while[project]versionis0.36.0. Two versions for one distribution- Move the template tree to something importable, for example
src/fiona/data/llm_wiki/, and load files withimportlib.resources. Updatewiki init/wiki updatepath helpers (get_llm_wiki_source_path) in the same change - Root
README.mdlinks./llm-wiki/AGENTS.md. The file issrc/llm-wiki/AGENTS.md
- Get setuptools metadata out of
src/src/fiona_assistant.egg-info/holdsPKG-INFO,entry_points.txt,requires.txt,SOURCES.txt,top_level.txt. That is install metadata. It belongs at the repo root (or nowhere, if it is only a local editable-install artifact).gitignorealready ignores*.egg-info/, so this directory should not be committed, butwhere = ["src"]can still see it during package discovery- The egg name
fiona_assistantis the setuptools normalization of project namefiona-assistant. Leave the normalization alone; only the directory location is wrong
- Keep a single coverage config, and point it at the code that exists
- Both
.coveragercand[tool.coverage]inpyproject.tomlexist. Coverage reads.coveragercfirst, so the pyproject block is unused .coveragercstill omitssrc/mdoctor/fmlint.pyandsrc/python_ruff/pycustomlint.py. Those modules live undersrc/customlinter/(fmlint.py,pycustomlint.py)- Both files share the same
exclude_lineslist. After deleting.coveragerc, confirmpytest --covstill honors the pyprojectsource,omit, andexclude_lines coverage.xmlat the repo root is a generated report. It is not matched by.gitignore(*.logand.coverageare;coverage.xmlis not)
- Both
- Raise the setuptools build pin so
[project]is valid[build-system]requiresissetuptools>=42- The
[project]table (PEP 621) needs setuptools 61 or newer. An older setuptools that satisfies>=42can fail the build - Set
requires = ["setuptools>=61"](or a current floor) and rebuild the wheel
- Sort runtime dependencies from repo-only tools
ruffis imported bysrc/python_ruff/python_ruff.py, so it stays independenciespylintis independencieswith no version pin, and nothing undersrc/imports it. Move it to thedevextra or drop it- There is no
[tool.ruff]table and noruff.toml. If this repo is linted with Ruff, the rule set belongs inpyproject.toml requires-pythonis>=3.8. Local bytecode insrc/__pycache__/is CPython 3.13. Classifiers only sayProgramming Language :: Python :: 3. The bootstrap examples use 3.11. Record the versions this repo actually runs, and confirm syntax such asstr | Noneis either covered byfrom __future__ import annotationsor by a higherrequires-python
- Make the structure docs match the tree (also listed under P3)
src/README.md"Project Structure" lists onlyfiona/,librarian/,mdoctor/,tests/, andREADME.md. Missing from that diagram:customlinter/,python_ruff/,gittool/,sanity_runner/,wiki/,llm-wiki/,run_sanity.py,run_tests.py,settings_dev.json,utils.sh,install_packages.sh- MDoctor's description in that file is still
[Description to be added], and Contributing is still[Contribution guidelines to be added] - Refresh this diagram again after the package collapse, so the
docs describe
src/fiona/rather than today's sibling packages
Code defects — 2026-09-29 audit
Source review after the package directories moved under src/fiona/.
src/ on disk is fiona/ plus README.md. Tests live in tests/.
The template lives in src/fiona/data/llm_wiki/. The package-move
items below are done. The command defects after them are still open.
Finish the package move
- Point every import at
fiona.<package>src/fiona/__main__.pystill imports top-levellibrarian,mdoctor,customlinter,python_ruff,gittool, andwiki. Those packages aresrc/fiona/<name>/.tests/test_fiona_main.pystill doesfrom librarian import toc_generator.TOOL_LOGGER_NAMESinsrc/fiona/logging_config.pystill listslibrarian,mdoctor,python_ruff, andwiki. Loggers are nowfiona.*viagetLogger(__name__). The parent namefionaalready covers them; drop the stale names.
- Point template and settings paths at
fiona/data/llm_wikiget_llm_wiki_source_pathinsrc/fiona/wiki/wiki.pyandsrc/fiona/wiki/update_manager.pylooks for a siblingllm-wiki(src/fiona/llm-wiki). The template issrc/fiona/data/llm_wiki/.wiki initandwiki updatemiss it. Load it withimportlib.resources(same end state as the P0 llm-wiki package-data item).- The default settings path in
logging_config.py,customlinter/base.py,mdoctor/mdoctor.py, andlibrarian/toc_generator.pyisparent.parent / "llm-wiki" / "fiona-settings.json", which issrc/llm-wiki/.... The file issrc/fiona/data/llm_wiki/fiona-settings.json. src/fiona/run_sanity.pystill lints./src/llm-wikifor frontmatter. Point it atsrc/fiona/data/llm_wiki.
- Match packaging and the test runner to the new tree
pyproject.tomlpackages.findincludeand[project.scripts]still namelibrarian,mdoctor,python_ruff,customlinter,sanity_runner,llm-wiki,wiki,gittool, andhtml_js_lint.librarianandmdoctorconsole scripts import modules that are no longer top-level.include = ["fiona"]does not match subpackages. Usefiona*sofiona.wikiand the rest are installed.[tool.pytest.ini_options]testpathsis stillsrc/tests. The suite istests/.tests/conftest.pywalks two directories up and inserts the repo root. That math used to reachsrc/when the file lived atsrc/tests/conftest.py. Insertsrc/soimport fionaworks without an editable install.src/__init__.pyis already gone. Leave it gone.run_tests.pyandrun_sanity.pyalready sit insrc/fiona/. Give them entry points or documentpython -m fiona.run_tests/python -m fiona.run_sanityin place ofpython src/run_tests.py(SPEC 4.13).
Fixes in commands the spec already calls Implemented
json-lint fixwalks the file list twice (SPEC appendix, SPEC 4.6). The secondfix_filepass finds the files already clean, so the path lines never print. Walk once: one count and one path line per fixed file.json-lint chklint --fixis parsed and ignored (src/fiona/customlinter/jsonlint.py). SPEC 4.6 sayschklint --fixrewrites the same wayfixdoes.json-lint fixtreats any path part namedpersonalorrawas a read-only zone, including a folder outside the wiki. Skip with the wiki-relative check inBaseLinter(src/fiona/customlinter/base.py), the same rule aschklint.gitfailures printError:on stdout and catchException(src/fiona/gittool/git_tool.py). SPEC 4.7 and section 2: the message goes to stderr. Catch the errorsGitToolraises. This is the concrete case of the P2 logging item.wiki graphprintsError: Directory not foundon stdout (src/fiona/wiki/graph_generator.py). The other graph errors use stderr. SPEC section 4: an invalid directory is stderr and exit 1.wiki graphreplaces--wikilinks/--no-wikilinksand--external-linkswithfiona.wiki.graphsettings after the CLI has set them (load_graph_settingsinwiki_graph_main). An explicit flag wins. Settings fill the defaults only (SPEC 4.12).wiki graphdrops files abovemax_fileswith a log line only. Print the kept count and the limit on stderr.mdoctor fixrewritespersonal/andraw/(SPEC 4.3, appendix). Constraint 1.2.3 says those zones are read-only for every automated tool. Skip them infix_file_programmatically, then change SPEC 4.3 to say the command skips those zones.graph_generator.pylogs with f-strings. Use lazy%formatting, the same style aslogging_config.py.fiona-assistant python-ruffhas an empty help string and a#TODOinsrc/fiona/__main__.py. Use a one-line help like the other linters.fiona.__main__rewritessys.argvand calls each toolmain(), which parses the arguments a second time. Call the library functions directly so each command is parsed once.
P1 — Wiki Lifecycle and Version Control
wiki inithardening (SPEC 7.1)- Warn or refuse when the target is already an llm-wiki directory
- Do not clobber user content on re-run (idempotent)
fiona-assistant wiki update(SPEC 7.2)- Create
src/wiki/update_manager.pywith stub functions - Add CLI commands:
wiki check-updateandwiki update - Add comprehensive test suite (48 tests)
- Update core md files / scripts to current template versions
(compare
versionfrontmatter) - Update directory structure when the template changed
- Skip locally-modified files and report them
- Create
fiona-assistant wiki sanity(SPEC 7.3)- Run all linters: mdlint, python-ruff, shellcheck (once P2 lands)
- Dead-link check (relative and wiki-internal links)
- Git version control for llm-wiki (SPEC 7.5)
- GitPython library wrapper
- Basic operations:
save— commit current edits withyyyy-mm-dd HH:MM:SS <description>snapshot—saveplus a git tagyyyy-mm-dd_HH-MM-SS_<tag_name>restore— create new commit restoring to<tag_name>stateclean— discard current edits (staging area);git statusmust be clean afterwardsundo— go back one commit; error if the working tree is dirty and suggestcleanfirst
P2 — Quality Tooling
- Restore the run_sanity progress heartbeat (
[WAITING]lines while checks run) on top of the shared SanityRunner engine (SPEC 7.16.1); dropped when the engine was generalizedSanityRunner.run_alltakes optionalprogressandheartbeatcallbacks; check lifecycle logged at INFO- run_sanity passes stdout callbacks (DONE markers and WAITING lines, assigned via a carriage-return constant); wiki sanity output unchanged
- Frontmatter linter (fmlint) (SPEC 7.8)
- Implement
src/customlinter/fmlint.py - Write TDD tests for fmlint
- Add CLI integration:
fiona-assistant fmlint chklint [directory] - Integrate into
run_sanity.pyandwiki sanity - Add settings support via
fiona-settings.json - SKILL.md name, folder, and description checks (FML004-FML006)
- Implement
- Generalize compress and format violations for all linters
(SPEC 6.6, 6.1)
BaseLinterprovidescompress_violations()andformat_compressed_violations(); report header comes from thelinter_nameclass attributefmlintdelegatesformat_violationsandlint_directorycompression to the shared base methods (output unchanged)- Migrate
mdoctor(compress_violations/format_violations, same compressed shape as fmlint) onto the shared base format - Migrate
python_ruff(different compressed shape:dict[rel_path, list[dict]]) onto the shared base format - Keep all linter reports diffable: same sections, same order (update SPEC 7.7/7.11 and tests together)
- JSON linter (json-lint) (SPEC 7.18)
- Implement
src/customlinter/jsonlint.py - Write TDD tests for json-lint
- JSON syntax validation using the standard
jsonmodule - JSON style rules (indentation, trailing commas, key ordering)
- CLI:
fiona-assistant json-lint chklintandfix - Settings via the
jsonsection offiona-settings.json - Integrated into
run_sanity.pyandwiki sanity
- Implement
- Shellcheck library (SPEC 7.12)
- add shellcheck to toml
- Wrap shellcheck like
python_ruff(AI-friendly violation report) - Clear install-hint error when shellcheck is missing
- Lint fenced code blocks inside MD files (SPEC 7.13)
- Python blocks via Ruff, shell blocks via shellcheck
- Logging library instead of
printfor diagnostics (SPEC 7.20)- Shared
fiona.logging_config.setup_logging()on CLI entry points - Error and warning diagnostics go to stderr through logging.
CLI reports stay stdout prints.
git_tool,graph_generator, and several CLIs stillprinterrors
- Shared
- Enable every Ruff rule (
select = ["ALL"]inpyproject.toml)- The table under
[tool.ruff]currently leaves Ruff on its built-in default. That skips style, docstrings, line length, pyupgrade, and other families - Turn on
ALL, then clear the findings onsrc/or add an explicitignoreonly where a rule fights the spec (CLIprinton stdout is one) - Keep
test_lint_detects_unused_importgreen. It must find unused-importF401even when another rule is reported first
- The table under
- Clear existing lint violations (SPEC 1 constraint: Ruff and
PyMarkdown must pass)
- Ruff: 0 violations — removed unused imports and unnecessary
passstatements, sorted imports, made dates tz-aware (UTC), added explicitcheck=Falseto subprocess.run, addedfrom __future__ import annotationsto wiki.py - Markdownlint: 0 violations — fixed blanks around headings/lists,
line length, fence languages; mdoctor now enables the pymarkdown
front-matterextension (files with frontmatter no longer raise phantom MD003/MD022/MD041) and ignores the frontmattertitlefor MD025
- Ruff: 0 violations — removed unused imports and unnecessary
- Settings manager,
fiona-assistant setup(SPEC 7.19) - When running sanity, make sure that markdownlint tests also include outer dir of the src/ because there are markdown files there
- in
run_sanity.py, show the failing test function and assert ex. - add pyspellchecker in spec for spelling checks in markdown
- add in spec, we add a new function in toc_generator,
- generate toc for files.
- All headings except H1
#will be included - the toc will be generated below the H1,
- make a logic to detect of a toc exists then regenerate to update.
- make a logic to add section numbers, (also add section numbers to the actual headings to preserve link is working)
- in fiona-settings.json, add specific section to modify parameters such as, max heading to include e.g min:2, max:6, sectionize: true/false, ignore_files:[], etc
fiona-assistant librarian toc file <path_to_file>will generate the toc for specific filefiona-assistant librarian toc file *will generate the toc for all files recursively in the current directory/llm-wiki dir if detected
- All headings except H1
- generate toc for files.
- test_md_frontmatter.py 43/44 passed, 1 failed
- TestParseFrontmatter : test_parse_valid_frontmatter
- assert 'Test Document' == 'Test Documen'
- test_mdoctor.py 33/33 passed
- prevent the pytest short summary missing from output
WARNING run_tests: pytest short summary missing from output; failure details come from the junit report (check PYTEST_ADDOPTS, --no-summary, or plugin output)
-
Fix terminal-width-dependent failure of the failure-report integration tests (
TestFailedTestReportIntegrationinsrc/tests/test_run_sanity.py)- Symptom:
test_real_error_shows_function_and_message(and its assert-based sibling) fail onpython src/run_sanity.pyin an interactive terminal with the report literally showingRuntimeErro..., but pass whenCIis set in the environment - Root cause: pytest trims the short-summary line
ERROR <nodeid> - <message>to the terminal width (_format_trimmedin_pytest/terminal.py; width fromshutil.get_terminal_size, i.e.COLUMNSor the tty) unlessCI/BUILD_NUMBERis set or-vvis used. The temp-test nodeids (~117 chars,../../../../tmp/pytest-of-user/...) push the line past a ~140-column terminal, so the message is cut toRuntimeErro....run_tests._parse_failed_tests_outputcaptures that trimmed text as a real (non-generic) message and_merge_failure_detailskeeps it over the junit report's fullRuntimeError: boom, so the truncated text reaches the report and the exact-match assertion fails - Reproduce:
env -u CI COLUMNS=140 python -m pytest -qfails;CI=true python -m pytest -qpasses. Bisect context: last commit where sanity passes in a plain terminal is43cb7f3("clean up pytests");e750a3d/cb7c3d8introduced the env-sensitive integration tests (057ac8badditionally has a genuinely broken test, fixed later) - Recommended fix (TDD): in
run_tests.py, treat short-summary messages that end in...as unreliable — same as the generic fallback — so_merge_failure_detailslets the junit entry (which always carries the full message) replace them. Write the failing test first: unit-test_merge_failure_detailswith a...-trimmed regex entry vs a junit entry, assert junit wins - Alternative one-liner: set
CI=1in the sanitized subprocess environment in_sanitize_pytest_env(pytest skips summary trimming whenCIis set); keep junit parsing as the safety net
- Symptom:
-
Make
scripts/run_sanity.pyfast on Windows (2026-09-29 investigation). A clean run takes about 2 to 3.5 minutes on Windows and at most 20 seconds on Linux- Symptom, measured with
LOG_LEVEL=INFOand--non-interactive. The machine had other load during the run, so the absolute times are high. The ratios holdpyteststook 03:26 andmarkdownlinttook 00:26.ruff,frontmatter_lint,pycustomlint, andjsonlinteach took 00:00SanityRunner.run_allruns the checks in one thread each. The total is the slowest check, which ispytestspython -m pytest testsalone, without coverage, took 165 s for 962 tests. The sanity run adds--cov, which makes it slower
- Root cause: each
gitprocess is slow to start on Windows, and the run starts hundreds of them- Measured cost per
subprocess.runon this Windows box:git --versionabout 198 ms,python -c passabout 150 ms,cmd /c exitabout 51 ms. On Linux a git call takes about 1 to 3 ms C:\Program Files\Git\cmd\git.exeis a wrapper that startsmingw64\bin\git.exe, so each call creates two processes. Endpoint security that scans each new process likely adds to the cost
- Measured cost per
- Source 1: pymarkdown
respect_gitignorestarts git twice for every directory it walksfiona-settings.jsonsetsrespect_gitignore: true.MDoctor.scan_file_programmaticallypasses it toPyMarkdownApi.scan_path(..., recurse_if_directory=True)- In
application_file_scanner(a pymarkdownlnt dependency), each directory in theos.walkloop callsGitProcessor.get_check_ignores. That runsgit rev-parse --show-topleveland thengit check-ignore(application_file_scanner.py, about line 850;git_processor.py) - One more batched
git check-ignoreruns over the final file list (__remove_any_gitignored_paths) - Repo-level
run_markdownlintscans.. That walks the repo root, including.git,.ruff_cache, and the other directories that are not ignored. It took 18 to 26 s alone, which is longer than the whole Linux run MDoctor._ignored_by_gitignorereads.gitignorewithout git, but it only filters the violations afterwards. It does not stop pymarkdown from walking and calling git
- Source 2: the test suite runs real sanity scans and real git
- Slowest single item: the session fixtures in
tests/conftest.py.pristine_wikicallswiki_init(1.7 s).pristine_check_resultscallsrun_wiki_checks. The two together take 24 s of setup ontest_sanity_manager.py::TestSanityChecksRegistry::test_registry_drives_results - The initialized wiki is a git repo with 96 files and 75
directories. Timed per check,
run_wiki_checksspends 29.2 s inmarkdownlint: about 150 git calls at about 190 ms each.frontmatter_linttakes 0.1 s,python-ruff0.3 s,mdlinter0.1 s, andjsonlint0.0 s - Other tests repeat full wiki sanity scans at 1.3 to 2.6 s
each (
test_sanity_manager.py,test_fiona_main.pytest_main_wiki_sanity*,test_mdoctor.py::TestMDoctorGitignore::test_scan_skips_gitignored_files) test_gittool.py: thegit_repofixture uses GitPython, which starts realgitprocesses. Setup takes 1.3 to 2 s per test, and calls take up to 3.4 s (test_restore_by_full_tag,test_restore_by_partial_tag,test_undo,test_clean)wiki_inittests take 2 to 3.3 s each (test_wiki.py,test_fiona_main.py::test_main_wiki_init). Each copies the template and runs a git init and commitTestFailedTestReportIntegrationintest_run_sanity.pystarts a nested pytest subprocess, about 2.2 s per test- The other ~900 tests average about 60 ms each. Temp directory and file creation is slow on Windows, more so with antivirus scanning
- Slowest single item: the session fixtures in
- Recommended fixes, in order of impact (TDD, SPEC first when
behavior changes)
- Stop pymarkdown from calling git. Pass
respect_gitignore=Falsetoscan_pathandfix_path. Buildexclude_patternsfrom the.gitignoredirectory names thatMDoctor._gitignore_dir_namesalready parses, plus.git. Or collect the file list with the librarian and scan each file. This removes nearly all git calls from themarkdownlintcheck and from the wiki-sanity tests. Done inMDoctor._pymarkdown_path_options. Patterns are omitted when the scan root is outside the working directory, because the matcher rejects those paths. The post-scan directory check still drops ignored files. SPEC 7.7.3 - Make the pristine and minimal wiki fixtures cheaper. Use a
wiki that is not a git repo, or turn off
respect_gitignorein the fixture's settings. The scan change above removes the git calls frompristine_check_results, which was the 29 s cost. The fixtures themselves are unchanged.wiki_initis still about 1.7 s once per session - Share one session-scoped git repo across the read-only
test_gittool.pytests. Keep per-test repos only for tests that change state. Fixture isread_only_repo - Run pytest in parallel with
pytest-xdist(-n auto) inTestRunner.run_pytest. Confirm coverage still combines correctly (pytest-covsupports xdist). Tried on 2026-09-30: coverage still merged at 96.0%, and pytests went from 01:56 to 01:40. Worker progress lines ([gwN] [100%] FAILED path) do not match the short-summary parser, soTestFailedTestReportIntegrationreportedTests: 0/1 passedand dropped the assert text. The flag was removed. Parallel workers are not a win on this box - Optional, outside the repo: exclude the repo directory and
git.exefrom antivirus scanning, if IT policy allows it
- Stop pymarkdown from calling git. Pass
- Acceptance: record Windows and Linux timings for
python scripts/run_sanity.py --non-interactivebefore and after the change. Windows should finish well under 1 minute. Linux was not re-measured (this machine is Windows). The todo baseline is at most 20 s on Linux. Windows before, this machine: about 02:50 in the session that wrote the git fix, and the 2026-09-29 note above (pytests 03:26, markdownlint 00:26). Windows after the git fix, shared read-only repo, andos.pathsep(2026-09-30,LOG_LEVEL=INFO): pytests 01:54, markdownlint 00:10, SANITY PASSED 01:54 (968 tests, coverage 96.0%). An earlier pass the same day was 01:56. That is under the old 2 to 3.5 minutes and still above 1 minute. The remaining time is pytest itself: temp files, the tests that still start git, and nested pytest - Side finding, not a speed issue:
run_tests.pyjoinsPYTHONPATHwith":"(f"{src_dir}:{current_pythonpath}"). On Windows the separator is;. Useos.pathsep. Done. SPEC 4.13
- Symptom, measured with
-
in toc generator settings, per-file glob overrides in the
filesection (SPEC 7.15.2) -
instead of current directory as default, find a fiona-settings.json in the current directory / subdirectory, that will be the llm-wiki root and default directory for commands (SPEC 7.23)
-
Enable the pymarkdown
markdown-tablesextension infiona-settings.jsonso table rules (MD055/MD056/MD058) run inrun_sanity.py(SPEC 5.1 known limitation)- Blocked: pymarkdownlnt 0.9.40 crashes with BadTokenizationError
on list lines containing pipes (e.g. generated
table_of_contents.mdentries); upgrade pymarkdownlnt once fixed and re-run sanity
- Blocked: pymarkdownlnt 0.9.40 crashes with BadTokenizationError
on list lines containing pipes (e.g. generated
P3 — Metadata, Backup, and Documentation
- Backup script for llm-wiki so users can recover if something goes wrong (SPEC 7.6)
- Frontmatter tags on MD files (SPEC 6.2)
versiondatetag- Quality signal:
confidence contestedflag (true if claims are disputed)
- Project documentation (SPEC 7.21)
- Root
README.mdlinksSPEC.md - Root README and
AGENTS.mddocumentbootstrapandbootstrap.ps1 - Refresh
librarian/README.mdandmdoctor/README.mdfor the current subcommands (list,toc,chklint,fix) - Refresh
src/README.mdstructure (wiki/,python_ruff/,run_tests.py,run_sanity.py,settings_dev.json). Full gap list is the last item of the P0 2026-09-29 structure audit, including the root README link to./llm-wiki/AGENTS.md
- Root
- Process hygiene: use semantic commit prefixes
(
feat:,fix:,docs:, ...) per project agent rule 8.1 fiona-assistant wiki graph(SPEC 7.4): link parser, graph, HTML template, and CLI insrc/wiki/graph_generator.py- Draw only important graph nodes until a parent is clicked
(SPEC 7.4.16)
- Setting
fiona.wiki.graph.node_importance_min_point 0, or a missing key, draws every node- A positive integer hides a node whose
unique_pointsis below that integer until one of its parents is clicked
- Setting
P4 — Design Decisions Needed
- Librarian: version tracking for non-MD files (SPEC 7.14)
- Python files — format TBD
- Bash scripts — format TBD
- Decide the tagging format before implementing
- Define LLM wiki tool roles (SPEC 7.22)
- Document purpose of
src/tools - Classify tools by role:
- Ingestion: moving files from
ingest/toraw/ - Processing: converting
raw/towiki/concepts/ - Maintenance: updating
index.md/log.md
- Ingestion: moving files from
- Add scripts to
utils/for these tasks (e.g.ingest.py,process.py)
- Document purpose of
Open Questions (SPEC)
- Tagging format for python files and bash scripts (SPEC 7.14)
- Git wrapper location: implemented at
src/gittool/git_tool.py(SPEC 7.5) - Wiki root discovery (SPEC 7.23): the nearest
fiona-settings.jsonwins; error when two at the same depth are equally near. Ancestor search is still open.