44 Commits

Author SHA1 Message Date
nvaldez a3d35c7f12 Implement SPEC 7.1.1.4: wiki init duplicate error handling
wiki init now catches FileExistsError, prints one stderr line with the
update suggestion, and exits 1. SPEC.md status for wiki init bumped to
Implemented (v1.22.2). Adds test for message format, exit code, and
single-line stderr output.

Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
2026-11-19 11:53:41 +01:00
nvaldez def3d2cd6d Rewrite specification-writer skill to be self-contained
- Remove references to fiona-assistant and agent-cost-estimator
- Include all SPEC structure details directly in the skill
- Add comprehensive section guidelines with numbered items
- Expand core contracts, configuration files, error handling sections
- Maintain sectionize-markdown integration
- All housekeeping checks pass

Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
2026-11-03 16:34:25 +01:00
nvaldez b7c173affa Fix linting violations in specification-writer skill
- Fix MD013 line length violations by splitting long lines
- Fix MD032 by adding blank lines around lists
- Fix MD036 by removing bold emphasis used as headings
- All housekeeping checks now pass

Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
2026-11-03 16:30:57 +01:00
nvaldez 59d1054e1d Add specification-writer skill for expert SPEC.md creation
- Create new skill at src/fiona/data/llm_wiki/utils/skills/specification-writer/
- Provides comprehensive guidance for writing project specifications
- Integrates with sectionize-markdown skill for consistent formatting
- Includes structure template based on fiona-assistant and agent-cost-estimator patterns
- Emphasizes always sectionizing SPEC files with numbered headings and list items
- Features quality checklist and maintenance guidelines

Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
2026-11-03 16:27:05 +01:00
nvaldez 983198b8b4 feat: add git diff staged/unstaged flags and root discovery\n\n- Git commands now discover and operate on root repository from current dir (7.12.0)\n- git diff includes untracked and deleted files by default (7.12.6)\n- Add --staged and --unstaged flags to git diff (7.12.7, 7.12.8)\n- Update diff rendering: collapse consecutive added/deleted lines (7.12.5)\n- Update SPEC to 1.22.1 and CHANGELOG to 1.2.3\n\nGenerated by Mistral Vibe.\nCo-Authored-By: Mistral Vibe <vibe@mistral.ai> 2026-10-24 14:11:21 +02:00
nvaldez a907e7da79 feat: ship sectionize-markdown skill as SKILL.md
Rename the template skill to the <name>/SKILL.md convention, add the
frontmatter description fmlint requires (FML004), point every
reference at the new path (SPEC 3.4.2), and add template tests.

Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
2026-10-21 23:07:48 +02:00
nvaldez b0539d6f10 chore: align package version with SPEC 1.19.0
Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
2026-10-21 23:07:43 +02:00
nvaldez dfea2bccfa feat: refuse to run bootstrap when not sourced
Bash bootstrap now exits non-zero when executed instead of sourced
(and when run under a non-bash shell), pointing the user at the
source command, per SPEC 7.21.7.

Also clears every ruff and pycustomlint violation so
run_sanity.py passes: five ruff findings in diff_report.py and
the two inline diff_report imports moved to module top in
__main__.py and git_tool.py.
2026-10-21 18:01:46 +02:00
nvaldez d6a265d4be fix minor issue 2026-10-19 15:45:55 +02:00
nvaldez 1c40d252b6 %CURSOR=90% fix: wire git diff in fiona-assistant main CLI
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-19 15:35:50 +02:00
nvaldez 295089c748 %CURSOR=90% feat: sync git diff HTML report from Nokia fork
Constellation-themed fiona-assistant git diff, tmp/ gitignore, SPEC 1.18.1. Version 1.17.0.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-19 10:58:19 +02:00
nvaldez de28441943 feat: sync Nokia fork product through SPEC 1.18 (v1.16.0)
MEMORY.md and skill memory template, ingest auth and html_sweep, TOC slug fix, run_tests CI output, and matching tests. Excludes GitLab CI and publishing docs.
2026-10-19 09:12:04 +02:00
nvaldez fd72e4d66b bump version to spec 2026-10-10 12:00:37 +02:00
nvaldez e9542ccf88 update fiona skill 2026-10-06 11:27:26 +02:00
nvaldez dcc0536f6e wiki ingest additional spec for auth required links 2026-10-05 09:07:02 +02:00
nvaldez 9ff5af588a test result changed spec 2026-10-05 07:31:46 +02:00
nvaldez 4c84213a30 librarian move specifications 2026-10-02 13:16:53 +02:00
nvaldez 09362f3951 librarian move specifications 2026-10-02 12:24:54 +02:00
nvaldez 3c21217970 librarian move specifications 2026-10-02 12:15:01 +02:00
nvaldez 8a00ce97f5 sanity fix 2026-10-02 11:50:26 +02:00
nvaldez 348579a085 license 2026-10-02 11:35:46 +02:00
nvaldez 48b94a5781 graph optimization 2026-10-02 11:33:55 +02:00
nvaldez 20378727fa SPEC for graph directory and live view 2026-10-01 17:56:31 +02:00
nvaldez cd1ea32635 v1.2.0 2026-10-01 17:01:42 +02:00
nvaldez 44029aee90 wiki update fix, themes versioning, readme refactor, graph ignore 2026-10-01 16:58:19 +02:00
nvaldez c45d4f1d2a Version 1.…11 llm tagging for auto mode 2026-10-01 13:53:04 +02:00
nvaldez 9e98390f2b Version 1.…1.0 SPEC subsection improvement, and wiki update refinement 2026-10-01 11:36:20 +02:00
nvaldez 482b2cb2e0 ingest implemented 2026-10-01 07:04:55 +02:00
nvaldez 2d08b71b55 new custom pylint to enforce no late declaration 2026-09-30 23:37:59 +02:00
nvaldez 31364eab2c housekeeping and fiona skill overhaul 2026-09-30 21:25:12 +02:00
nvaldez a1fccbe641 housekeeping and fiona skill overhaul 2026-09-30 21:24:29 +02:00
nvaldez da01b3fe4a housekeeping and fiona skill overhaul 2026-09-30 21:19:47 +02:00
nvaldez 94d3554d77 update themes 2026-09-30 17:10:12 +02:00
nvaldez b93a5a0c02 bump version and update themes 2026-09-30 13:37:01 +02:00
nvaldez 151282657e new spec for node draw improvement 2026-09-30 11:13:11 +02:00
nvaldez 0907252d4c fiona skill improvement, and local non edit bootstrap install 2026-09-30 11:06:02 +02:00
nvaldez 3471ff9fe6 themes revisit 2026-09-30 08:53:47 +02:00
nvaldez 32920d1736 Major graph refactor, better physics, themes, etc 2026-09-30 08:34:27 +02:00
nvaldez fd58ad549a refactor fiona src to have single module 2026-09-29 16:01:16 +02:00
nvaldez 02a159a086 fix: add pre-build cleanup to prevent stale artifact upload errors
- Added rm -rf dist/ build/ fiona_assistant.egg-info/ to build.sh
- Bumped version from 0.36.0 to 0.40.5 in pyproject.toml
2026-09-29 14:43:55 +02:00
nvaldez 6e334cc34d major refactor of spec 2026-09-29 14:35:51 +02:00
nvaldez e14f96b590 update todo with the audit 2026-09-29 10:44:57 +02:00
nvaldez 859d2e55d7 update the spec 2026-09-29 10:10:20 +02:00
nvaldez e6920aa691 windows bootstrap implemented 2026-09-29 10:04:57 +02:00
158 changed files with 27979 additions and 7305 deletions
-19
View File
@@ -1,19 +0,0 @@
[run]
source = src
omit =
src/tests/*
*/tests/*
*/test_*.py
src/mdoctor/fmlint.py
src/python_ruff/pycustomlint.py
src/llm-wiki/__init__.py
[report]
exclude_lines =
pragma: no cover
def __repr__
raise AssertionError
raise NotImplementedError
if __name__ == .__main__.:
if TYPE_CHECKING:
@abstractmethod
+10 -1
View File
@@ -5,6 +5,7 @@ __pycache__/
# Virtual environments # Virtual environments
.venv/ .venv/
.venvs/
venv/ venv/
env/ env/
ENV/ ENV/
@@ -34,6 +35,7 @@ dist/
# Coverage # Coverage
.coverage .coverage
coverage.xml
htmlcov/ htmlcov/
# Jupyter Notebook checkpoints # Jupyter Notebook checkpoints
@@ -45,10 +47,17 @@ htmlcov/
# Windows # Windows
Thumbs.db Thumbs.db
# Obsidian per-user vault state
.obsidian/workspace*.json
.obsidian/cache
# Python # Python
*.pyc *.pyc
*.pyo *.pyo
*.pyd *.pyd
*fiona.log *fiona.log
*wiki_graph.html *wiki_graph.html
# Generated review artifacts (git diff HTML, scratch exports)
tmp/
+178 -81
View File
@@ -1,21 +1,23 @@
--- ---
name: Fiona-Assistant Agent Rules name: Fiona-Assistant Development Agent Rules
version: 1.3.1 version: 1.9.3
title: LLM Agent Rules title: Agents
type: summary type: summary
created: 2025-09-23 created: 2025-09-23
updated: 2026-09-26 updated: 2026-10-06
tags: [agents, rules, guidelines, llm, ai] tags: [agents, rules, guidelines, llm, ai]
sources: [] sources: []
--- ---
# Fiona-Assistant LLM Agent Rules # Agents
- [1. Overview](#1-overview) - [1. Overview](#1-overview)
- [1.1 Rule References](#11-rule-references) - [1.1 Rule References](#11-rule-references)
- [2. Basic Rules (MANDATORY)](#2-basic-rules-mandatory) - [2. Basic Rules (MANDATORY)](#2-basic-rules-mandatory)
- [2.1 Testing](#21-testing) - [2.1 Development Lifecycle](#21-development-lifecycle)
- [2.2 Documentation](#22-documentation) - [2.2 Testing](#22-testing)
- [2.3 Documentation](#23-documentation)
- [2.4 Commit](#24-commit)
- [3. Environment Setup (MANDATORY)](#3-environment-setup-mandatory) - [3. Environment Setup (MANDATORY)](#3-environment-setup-mandatory)
- [3.1 Always Use Bootstrap for Environment](#31-always-use-bootstrap-for-environment) - [3.1 Always Use Bootstrap for Environment](#31-always-use-bootstrap-for-environment)
- [4. Python Development Rules](#4-python-development-rules) - [4. Python Development Rules](#4-python-development-rules)
@@ -55,34 +57,80 @@ Fiona-Assistant repository.
Refer to rules in this file as `project agent rule N.M` (e.g. Refer to rules in this file as `project agent rule N.M` (e.g.
`project agent rule 1.1`) to differentiate them from rules in other `project agent rule 1.1`) to differentiate them from rules in other
AGENTS.md files (e.g. the llm-wiki template's `src/llm-wiki/AGENTS.md`). AGENTS.md files (e.g. the llm-wiki template's `src/fiona/data/llm_wiki/AGENTS.md`).
--- ---
## 2. Basic Rules (MANDATORY) ## 2. Basic Rules (MANDATORY)
### 2.1 Testing ### 2.1 Development Lifecycle
- For testing scripts always run `python src/run_tests.py` and make sure Every change that alters behavior, adds a feature, or changes a
requirement follows these five steps, in order. Finish a step before
starting the next. A read-only question skips this lifecycle.
Before step 2, load the environment (project agent rule 3.1).
1. **Write or check the spec.** Read [SPEC](./SPEC.md) for the
area this task will change.
- When the requirement or its `*acceptance*` is missing, write
both in [SPEC](./SPEC.md) before any test or code. Follow
SPEC 1.1 and set the section status.
- When the requirement exists, confirm the task matches that
text. That text is the behavior to build.
2. **Write tests.** Add tests that fail for that acceptance
(project agent rule 6.1). Leave production code unchanged in
this step.
3. **Implement, TDD style.** Load the
[python-tdd skill](./src/fiona/data/llm_wiki/utils/skills/python-tdd/SKILL.md)
(project agent rule 4.1).
- Red: run the new tests and confirm they fail.
- Green: write the smallest change that makes them pass.
- Refactor while those tests stay green.
4. **Verify the spec.** Re-read the requirement and each
acceptance this task touched. The code and the tests satisfy
that text.
- When they disagree, update [SPEC](./SPEC.md) in this same
task. Move status along `Planned` ->
`Partially Implemented` -> `Implemented`, or set
`Won't Do` when the requirement is rejected.
- Run `python scripts/run_tests.py`, then
`python scripts/run_sanity.py --non-interactive`. Both pass
(project agent rule 2.2).
5. **Bump versions.** On each markdown file this task edited,
bump frontmatter `version` and set `updated` to the edit date
(project agent rule 10.1).
- Semver: patch for a clarification, minor for a new rule or
requirement, major for a breaking contract change.
- Bump [SPEC](./SPEC.md) when step 1 or step 4 changed it.
- Python and shell files have no version tag yet (SPEC 7.3.2.1).
Leave them untagged.
A documentation edit that does not change product behavior still
bumps versions (step 5). It skips the test and implementation steps.
### 2.2 Testing
- For testing scripts always run `python scripts/run_tests.py` and make sure
everything is passing on every loop everything is passing on every loop
- After Modifying the repo, make sure `python src/run_sanity.py --non-interactive` passes before finalizing. - After modifying the repo, make sure
`python scripts/run_sanity.py --non-interactive` passes before finalizing.
### 2.2 Documentation ### 2.3 Documentation
- Before changing any functionality, read [SPEC](./SPEC.md) (the project - Product behavior is specified in [SPEC](./SPEC.md). The order for
spec sheet) so the change follows the specified requirements and reading it, writing it, and checking it after the code is project
acceptance criteria. agent rule 2.1.
- When documentation is changed (e.g., `SPEC.md`, especially the table of - When documentation changes (for example [SPEC](./SPEC.md), especially
contents section), double-check that all references to it are unbroken. the table of contents), confirm every reference to it still resolves.
- Always link via markdownlink for connectivity. - Always link with a markdown link.
- Whenever a change alters behavior, adds a feature, or changes a
requirement, update [SPEC](./SPEC.md) in the same task so the spec stays
in sync (adjust section status: `planned` -> `partial` -> `implemented`;
use `won't do` for rejected features).
### 2.3 Commit ### 2.4 Commit
- *NEVER* commit without explicit permission! - *NEVER* commit without explicit permission!
- This rule covers this repository only. A deployed wiki commits
after each ingest cycle and housekeeping pass
([SPEC](./SPEC.md) 3.3.3).
--- ---
@@ -90,7 +138,9 @@ AGENTS.md files (e.g. the llm-wiki template's `src/llm-wiki/AGENTS.md`).
### 3.1 Always Use Bootstrap for Environment ### 3.1 Always Use Bootstrap for Environment
> BEFORE any testing or development work, you MUST source the bootstrap script > BEFORE any testing or development work, you MUST load the bootstrap script
Linux / macOS (bash, must be sourced):
```bash ```bash
# Bare minimum - always run this first # Bare minimum - always run this first
@@ -104,6 +154,30 @@ source bootstrap --env <env_name> --python <version>
# With cleanup flag # With cleanup flag
source bootstrap --env <env_name> -c source bootstrap --env <env_name> -c
# This checkout as a normal package (not editable)
source bootstrap --local
# Current user, no virtual environment
source bootstrap --local --no-env
```
Windows (PowerShell). A virtual-environment install must be
dot-sourced. This creates a repo-local venv at `.venvs\<name>` with
the Python launcher. It does not install pyenv. `--no-env` skips that
venv and may be executed with `-File`. `--local` installs this
repository with `pip install .` (not editable, no `dev` extra).
`--no-env` installs with `pip install --user`, so `fiona-assistant`
runs without activating an environment. The two flags combine.
```powershell
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
. .\bootstrap.ps1
. .\bootstrap.ps1 --env fiona_env
. .\bootstrap.ps1 --env <env_name> --python <version>
. .\bootstrap.ps1 --env <env_name> -c
. .\bootstrap.ps1 --local
powershell -ExecutionPolicy Bypass -File .\bootstrap.ps1 --local --no-env
``` ```
**Rationale**: The bootstrap script sets up the Python virtual environment, **Rationale**: The bootstrap script sets up the Python virtual environment,
@@ -119,11 +193,13 @@ dependency conflicts, incorrect Python versions, or missing packages.
### 4.1 Load Python TDD Skill for Python Tasks ### 4.1 Load Python TDD Skill for Python Tasks
> When modifying or creating Python scripts, the Python TDD Skill MUST be loaded > When modifying or creating Python scripts, the
> [python-tdd skill](./src/fiona/data/llm_wiki/utils/skills/python-tdd/SKILL.md)
> MUST be loaded
```text ```text
SKILL: python-tdd SKILL: python-tdd
PATH: llm-wiki/utils/skills/python-tdd/SKILL.md PATH: src/fiona/data/llm_wiki/utils/skills/python-tdd/SKILL.md
``` ```
- Project Rules: - Project Rules:
@@ -132,6 +208,8 @@ PATH: llm-wiki/utils/skills/python-tdd/SKILL.md
**Required Practices from Python TDD Skill:** **Required Practices from Python TDD Skill:**
- Step 3 of project agent rule 2.1 is this cycle. The spec and the
failing tests already exist before implementation starts.
- Strictly follow **Test-Driven Development (TDD)** - Red-Green-Refactor cycle - Strictly follow **Test-Driven Development (TDD)** - Red-Green-Refactor cycle
- Use **pytest** for testing - Use **pytest** for testing
- Follow **OOP principles** and **SOLID** design patterns - Follow **OOP principles** and **SOLID** design patterns
@@ -160,29 +238,33 @@ The repository has the following structure with specific permissions:
```text ```text
fiona-assistant/ fiona-assistant/
├── src/ # Source code ├── src/fiona/ # The only installable package
│ ├── llm-wiki/ # Packaged llm-wiki template (data only) │ ├── __main__.py # CLI entry point
│ │ # Structure source of truth:
│ │ # `src/llm-wiki/wiki/schema.md`
│ ├── fiona/ # Main CLI package
│ ├── librarian/ # Wiki content management │ ├── librarian/ # Wiki content management
│ ├── mdoctor/ # Markdown diagnosis │ ├── mdoctor/ # Markdown diagnosis
│ ├── python_ruff/ # Python code linting │ ├── customlinter/ # Frontmatter, markdown, Python, JSON linters
│ ├── wiki/ # Wiki lifecycle modules (init, update) │ ├── python_ruff/ # Ruff wrapper
│ ├── tests/ # Unit tests │ ├── gittool/ # Git operations
│ ├── run_tests.py # Test runner │ ├── sanity_runner/ # Shared sanity engine
│ ├── run_sanity.py # Sanity runner (tests + linters) │ ├── wiki/ # Wiki lifecycle (init, update, sanity, graph)
│ ├── data/llm_wiki/ # Packaged template (data only)
│ ├── run_tests.py # Test runner module
│ └── run_sanity.py # Sanity runner module
├── tests/ # Unit tests (not installed)
├── scripts/ # Repo tooling, not package source
│ ├── run_tests.py # Launcher for fiona.run_tests
│ ├── run_sanity.py # Launcher for fiona.run_sanity
│ ├── settings_dev.json # Development settings │ ├── settings_dev.json # Development settings
│ ├── utils.sh # Shell utilities │ ├── utils.sh # Shell utilities
│ └── install_packages.sh # Dependency installer │ └── install_packages.sh # Dependency installer
│
├── pyproject.toml # Project configuration ├── pyproject.toml # Project configuration
├── bootstrap # Environment setup ├── bootstrap # Environment setup (bash; source it)
├── bootstrap.ps1 # Environment setup (PowerShell; dot-source it)
└── README.md # Project documentation └── README.md # Project documentation
``` ```
The llm-wiki directory structure is not duplicated here: The llm-wiki directory structure is not duplicated here:
`src/llm-wiki/wiki/schema.md` is the final source of truth for the `src/fiona/data/llm_wiki/wiki/schema.md` is the final source of truth for the
llm-wiki directory structure. llm-wiki directory structure.
**General Rules:** **General Rules:**
@@ -191,8 +273,8 @@ llm-wiki directory structure.
- `/raw/` directories are **READ-ONLY** - Never edit archived source materials - `/raw/` directories are **READ-ONLY** - Never edit archived source materials
- `/wiki/concepts/` is **READ-WRITE** - Place synthesized content here - `/wiki/concepts/` is **READ-WRITE** - Place synthesized content here
- `/ingest/` is for **input processing** - Parse files dropped here - `/ingest/` is for **input processing** - Parse files dropped here
- **Always update** `src/llm-wiki/wiki/index.md` when adding new concepts - **Always update** `src/fiona/data/llm_wiki/wiki/index.md` when adding new concepts
- **Always append** to `src/llm-wiki/wiki/log.md` for every action - **Always append** to `src/fiona/data/llm_wiki/wiki/log.md` for every action
--- ---
@@ -200,6 +282,7 @@ llm-wiki directory structure.
### 6.1 Testing Requirements ### 6.1 Testing Requirements
Steps 2 and 3 of project agent rule 2.1.
**All code changes MUST be tested** following TDD principles: **All code changes MUST be tested** following TDD principles:
1. **Write failing tests first** (Red phase) 1. **Write failing tests first** (Red phase)
@@ -210,7 +293,7 @@ llm-wiki directory structure.
```bash ```bash
# Run all tests # Run all tests
python src/run_tests.py python scripts/run_tests.py
# Run with pytest directly # Run with pytest directly
python -m pytest python -m pytest
@@ -219,7 +302,7 @@ python -m pytest
python -m pytest --cov=src python -m pytest --cov=src
# Run specific test file # Run specific test file
python -m pytest src/tests/test_md_frontmatter.py -v python -m pytest tests/test_md_frontmatter.py -v
``` ```
**Test File Structure:** **Test File Structure:**
@@ -294,35 +377,19 @@ git commit -m "test: add TDD tests for name/version extraction"
### 9.1 Development Workflow ### 9.1 Development Workflow
**Standard Workflow:** The development lifecycle is project agent rule 2.1. Do those five
steps in order: write or check the spec, write tests, implement
TDD-style, verify the spec, bump versions.
1. **Setup Environment** (MANDATORY) Before step 2 of that lifecycle:
```bash 1. Set up the environment (project agent rule 3.1).
source bootstrap --env <env_name> 2. Load the skill when the task needs one (project agent rule 4.1).
``` Python tasks load the
[python-tdd skill](./src/fiona/data/llm_wiki/utils/skills/python-tdd/SKILL.md).
2. **Load Appropriate Skill** (if applicable) Wiki tasks load the
- Python tasks: Load `python-tdd` skill [Fiona skill](./src/fiona/data/llm_wiki/utils/skills/fiona/SKILL.md)
- Wiki tasks: Load `llm-wiki` skill unless `src/fiona/data/llm_wiki/AGENTS.md` already loaded it.
3. Read the Spec
- Read [SPEC](./SPEC.md) for the feature being changed
- Confirm the change matches the specified requirements and acceptance criteria
4. Follow TDD for Code Changes
- Write tests first
- Implement minimal code
- Refactor
5. Validate
- Run tests
- Run linters
6. Document
- Update docstrings
- Update README if needed
- Update [SPEC](./SPEC.md) if the change affects any specified feature or requirement
--- ---
@@ -331,7 +398,12 @@ git commit -m "test: add TDD tests for name/version extraction"
### 10.1 MD File Frontmatter ### 10.1 MD File Frontmatter
**All markdown files in llm-wiki/ MUST have YAML frontmatter** with the **All markdown files in llm-wiki/ MUST have YAML frontmatter** with the
required `name` and `version` fields. All other fields are optional: required `name` and `version` fields. Every page under `wiki/` also
requires the quality signals `confidence`, `llm`, `contested`, and
`contradictions`. How to set them is section 15 of the
[Fiona skill](./src/fiona/data/llm_wiki/utils/skills/fiona/SKILL.md).
All other fields are optional. fmlint enforces `name` and `version`
only.
```yaml ```yaml
--- ---
@@ -343,10 +415,29 @@ updated: YYYY-MM-DD # optional (bump when editing)
type: entity | concept | comparison | query | summary | toc type: entity | concept | comparison | query | summary | toc
sources: [raw/articles/source-name.md] # optional sources: [raw/articles/source-name.md] # optional
tags: [from taxonomy] # optional tags: [from taxonomy] # optional
# Optional quality signals: # quality signals (required on pages under wiki/):
confidence: trusted | high | medium | low | zero confidence: trusted | high | medium | low | zero
llm:
- model: none # required; model that last edited the page
- reasoning: high # when the model reports a level
contested: true contested: true
contradictions: [other-page-slug] contradictions: [other-page-slug]
# written by wiki ingest (SPEC 7.1.5), optional:
status: raw | refined
source_meta: {author: Name} # metadata read from the source
ingest: # provenance; every key always present
origin: https://host/path # URL or file path of the source
origin_kind: url | file
final_url: null # URL after redirects
fetched: YYYY-MM-DDTHH:MM:SSZ
media_type: text/html
size_bytes: 0
sha256: <64 lowercase hex> # of the raw copy; never hand-edited
raw_reused: false
converter: html # converter name, or agent
conversion: complete | partial
warnings: []
tool: fiona-assistant <version>
--- ---
``` ```
@@ -357,7 +448,7 @@ contradictions: [other-page-slug]
- Use semantic versioning (MAJOR.MINOR.PATCH) - Use semantic versioning (MAJOR.MINOR.PATCH)
This is the single canonical frontmatter schema; `wiki/schema.md` documents This is the single canonical frontmatter schema; `wiki/schema.md` documents
the same schema for the wiki template (see SPEC 4.2.1). the same schema for the wiki template (see SPEC 6.2).
--- ---
@@ -396,7 +487,8 @@ the same schema for the wiki template (see SPEC 4.2.1).
When multiple tasks are possible, prioritize in this order: When multiple tasks are possible, prioritize in this order:
1. **User's explicit request** (highest priority) 1. **User's explicit request** (highest priority)
2. **TDD requirements** (tests before implementation) 2. **Development lifecycle** (project agent rule 2.1): spec, then
tests, then implementation
3. **Code quality** (linting, formatting) 3. **Code quality** (linting, formatting)
4. **Documentation** (docstrings, README) 4. **Documentation** (docstrings, README)
5. **Refactoring** (improving existing code) 5. **Refactoring** (improving existing code)
@@ -409,6 +501,8 @@ When multiple tasks are possible, prioritize in this order:
### 14.1 Things to Never Do ### 14.1 Things to Never Do
- Never modify `/personal/` or `/raw/` directories - Never modify `/personal/` or `/raw/` directories
- Never write production code before the spec states the requirement
and a failing test exists (project agent rule 2.1)
- Never commit code without tests - Never commit code without tests
- Never commit code that fails linting - Never commit code that fails linting
- Never ignore AGENTS.md rules without explicit user override - Never ignore AGENTS.md rules without explicit user override
@@ -434,14 +528,17 @@ These rules **CAN** be overridden by:
| Task | Required Action | | Task | Required Action |
| ---- | -------------- | | ---- | -------------- |
| Any Python work | Load python-tdd skill | | Development | project agent rule 2.1, in order |
| Before testing | `source bootstrap` | | Any Python work | Load the [python-tdd skill](./src/fiona/data/llm_wiki/utils/skills/python-tdd/SKILL.md) |
| Changing functionality | Read [SPEC](./SPEC.md) first | | Before testing | `source bootstrap` (Windows: `. .\bootstrap.ps1`) |
| Modifying code | Write tests first (TDD) | | Local install, no venv | `bootstrap --local --no-env` |
| Step 1 | Write or check [SPEC](./SPEC.md) |
| Step 2 and 3 | Failing tests, then the smallest change |
| Step 4 | Re-read the spec; tests and sanity pass |
| Step 5 | Bump `version` and `updated` on edited markdown |
| Before commit | Run tests + linters | | Before commit | Run tests + linters |
| After behavior change | Update [SPEC](./SPEC.md) to stay in sync | | Wiki changes | Load the [Fiona skill](./src/fiona/data/llm_wiki/utils/skills/fiona/SKILL.md); update index.md + log.md |
| Wiki changes | Update index.md + log.md | | Frontmatter | `name` and `version` on every file; on `wiki/` pages also `confidence`, `llm`, `contested`, `contradictions` |
| Frontmatter | Include name, version, title, dates |
| Errors | Raise exceptions, don't fail silently | | Errors | Raise exceptions, don't fail silently |
--- ---
+229
View File
@@ -0,0 +1,229 @@
---
name: changelog
version: 1.2.3
title: Changelog
created: 2026-10-08
updated: 2026-10-24
type: summary
---
# Changelog
Revisions of [SPEC](./SPEC.md), newest first. Each entry names the
requirements it adds or changes. "Unbuilt" means the requirement is
specified and marked `NOT_IMPLEMENTED`.
## 1.22.1 - 2026-10-24
- Minor corrections to 1.22.0.
## 1.22.0 - 2026-10-24
- Adds `--staged` and `--unstaged` flags to `git diff` for staged-only and
unstaged-only diffs (*7.12.7.*, *7.12.8.*).
## 1.21.0 - 2026-10-24
- Includes untracked and deleted files in `git diff` report by default
(*7.12.6.*).
- Updates diff rendering: removes spacing between consecutive added or
deleted lines, preserves spacing between added and deleted blocks (*7.12.5.*).
## 1.20.0 - 2026-10-24
- Git commands discover and operate on the root repository from the current
directory (*4.6.*, *7.12.0.*).
## 1.19.1 - 2026-10-21
- The sectionize-markdown skill ships as
`utils/skills/sectionize-markdown/SKILL.md` (*3.4.2.*), matching the
`<name>/SKILL.md` convention of the other skills. Every reference to
the old `sectionize-markdown.md` path now points at `SKILL.md`.
## 1.18.1 - 2026-10-19
- `fiona-assistant git diff` writes a constellation-themed HTML
report under the wiki `tmp/` directory (*4.6*, *7.12.5*). The
packaged wiki `.gitignore` now lists `tmp/`.
## 1.18.0 - 2026-10-15
- Adds skill memory to the template (*3.5.*): the `memory` skill
under `utils/skills/memory/` says how a skill keeps its own
`MEMORIES.md` beside its `SKILL.md`, and ships the blank
`MEMORIES.md` to copy. Wiki-wide preferences stay in `MEMORY.md`.
## 1.17.0 - 2026-10-15
- The `.gitignore` written by `wiki init` ignores Obsidian's per-user
vault state, `.obsidian/workspace*.json` and `.obsidian/cache`, so a
wiki opened as an Obsidian vault stays clean between saves. Shared
vault settings such as `.obsidian/app.json` stay tracked (*3.2.3.*).
## 1.16.0 - 2026-10-14
- Adds the agent memory `MEMORY.md` to the template (*3.4.*): a
sectionized file of owner preferences, corrections, and verified
facts, linked from `AGENTS.md`. `AGENTS.md` says to update it on
an important memory or when the user explicitly asks (*3.4.6.*). `wiki update` adds it when missing
and never compares or replaces it after that (*3.4.4.*).
## 1.14.1 - 2026-10-08
- Moves the revision history out of the spec into this file.
## 1.14.0 - 2026-10-08
- Stages raw pages in a `refine/` directory at the wiki root, and has
the agent move each refined page into `wiki/` (*7.1.5.15.*).
Unbuilt.
## 1.13.0 - 2026-10-07
- Builds login detection with exit 4 (*7.1.5.2.8.*), `--origin`
(*7.1.5.2.9.*), the agent steps for exit 4 (*7.1.5.14.*), and the
HTML block sweep (*7.1.5.4.5.6.*).
## 1.11.0
- Specifies `wiki search` (*4.7.7.*, *7.1.6.*), a keyword search that
prints `<path>:<line>: <text>` and shortens long lines around the
match. Unbuilt.
## 1.10.0
- Replaces the first-`<article>` content root of the `html` converter
with a block sweep that scores every block as `keep`, `review`, or
`drop` (*7.1.5.4.5.6.*). Unbuilt.
## 1.9.0
- `wiki ingest` stops with exit 4 on a URL that needs a login
(*7.1.5.2.8.*). Unbuilt.
- Adds `--origin` for a file saved by hand (*7.1.5.2.9.*). Unbuilt.
- Tells the agent how to get the source instead (*7.1.5.14.*).
Unbuilt.
## 1.8.0
Adds the findings of the 2026-10-02 manual test. All unbuilt:
- One-line CLI errors with no traceback (*1.2.7.*, *7.1.1.4.*).
- `--version` (*2.3.*).
- A quiet default stderr (*7.17.3.*).
- An ingest hint for a drop-zone file that already moved
(*7.1.5.2.7.*).
- A dry run that does not log writes (*7.1.5.8.1.*).
- A lint-clean ingest log line (*7.1.5.6.1.*).
- A follow-up hint for style violations left on a page
(*7.1.5.5.6.*).
- Single-file `mdoctor` (*7.4.4.*).
- Compact housekeeping sanity details (*7.2.5.1.*).
- A quiet graph overwrite (*7.1.4.5.9.1.*).
- A usage section in the README (*7.18.4.*).
## 1.7.0
- `librarian move` rewrites links to the moved paths across the whole
wiki (*7.3.3.6.*).
## 1.6.0
- Specifies `librarian move` (*7.3.3.*). Unbuilt.
## 1.5.2
- Gives `table_of_contents.md` frontmatter `name` and `version`
(*7.3.1.1.1.*), so `fmlint` accepts it.
## 1.5.0
- Omits markdown files matched by `fiona.wiki.graph.ignore_files`
from the graph (*7.1.4.20.*). The shipped list is
`*/table_of_contents.md`, because a table of contents links to
every page.
## 1.4.0
- Names the core-file record `corefiles.json` (*7.1.2.14.*).
- Skips paths in `fiona.wiki.update.ignore_files` (*7.1.2.15.*). The
shipped list names `corefiles.json`. Only `wiki update` writes that
file.
## 1.3.0
- Specifies directory nodes (*7.1.4.18.*) and `wiki graph --serve`
(*7.1.4.19.*). Both unbuilt.
## 1.2.0
- Folds text line endings before the SHA-256 compare (*7.1.2.13.*).
- Writes `corefiles.json`, so a later update does not list a core
file whose version and hashes are unchanged (*7.1.2.14.*).
## 1.1.0
- Treats a differing `fiona-settings.json` as a conflict and writes
`fiona-settings-update.json` (*7.1.2.3.*).
## 1.0.0
- Groups each module's functions under that module.
- Moves Non-Goals, Priority Order, and Open Questions to Appendices
A, B, and C.
## 0.99.0
- Builds `wiki ingest` (*7.1.5.*).
## 0.98.0
- Specifies that `wiki graph` omits a file matched by `.gitignore`
(*7.1.4.17.*). Unbuilt.
## 0.97.0
- Rejects a module-level assignment that follows the first function
or class (*6.4.3.*).
## 0.96.0
- Compares a file without a version by SHA-256 and shows both short
hashes in the update report (*7.1.2.12.*).
## 0.94.0
- `wiki update` yields to the owner's markdown and writes
`<stem>-update.md` instead (*7.1.2.11.*).
- Adds `fiona-assistant housekeeping` (*7.2.*).
## 0.93.0
- Adds SHA-256 reuse across `raw/`, the complete `ingest`
frontmatter map, and the agent metadata check.
## 0.92.0
- Specifies `wiki ingest` (*7.1.5.*). Unbuilt.
## 0.90.0
- Specifies `node_importance_min_point` (*7.1.4.16.*). Unbuilt.
## 0.41.0
- Installs one `fiona` package and keeps tests and repo scripts
outside it.
## 0.40.4
- Adds the PYC001 exception (*6.4.2.*).
## 0.40.1
- Numbers nested requirements such as `7.14.1.1`.
## 0.40.0
- Reorganizes the sheet into contracts.
+362 -282
View File
@@ -1,356 +1,436 @@
# Fiona-Assistant # Fiona-Assistant
LLM Wiki Tool Helper for LLM Agents/Hives > Your LLM agent writes the wiki. Fiona keeps it clean.
Fiona-Assistant is a command-line toolkit designed to manage, organize, and Fiona-Assistant is a command-line toolkit for running an **llm-wiki**: a
maintain LLM wiki content with minimal AI token dependence. It combines folder of interlinked markdown that an AI agent builds and maintains for
script-based automation with structured workflows to help LLM Agents and you. You bring sources (web pages, PDFs, notes) and questions. The agent
Hives efficiently manage documentation, markdown files, and code quality. compiles them into wiki pages. Fiona does every deterministic part of the
job: it scaffolds the wiki, ingests and archives sources, lints markdown,
frontmatter, Python, and JSON, checks links, draws a link graph, versions
everything in git, and tells the agent exactly what to fix next.
The result: the agent spends its tokens on reading and writing, not on
listing files, chasing broken links, or reformatting tables.
```bash
python -m pip install fiona-assistant
fiona-assistant wiki init my-wiki
cd my-wiki
fiona-assistant wiki ingest https://example.com/article.html
fiona-assistant housekeeping
```
--- ---
## Overview ## Table of Contents
### Purpose - [Why Fiona](#why-fiona)
- [Features](#features)
- [Quick Start](#quick-start)
- [How It Works](#how-it-works)
- [Command Reference](#command-reference)
- [Output Built for Agents](#output-built-for-agents)
- [Roadmap](#roadmap)
- [Development](#development)
- [Documentation](#documentation)
- [License](#license)
Fiona-Assistant serves as a **standalone MD/scripts/docs organizer** and ---
**LLM wiki management helper** that:
- **Reduces AI Token Usage**: Uses deterministic scripts and tools to handle ## Why Fiona
wiki management tasks that don't require LLM intelligence
- **Maintains Wiki Structure**: Enforces consistent organization of markdown
files, concepts, and raw source materials
- **Validates Content Quality**: Provides linting and diagnosis tools for markdown and Python code
- **Enables Script-Based Workflows**: Automates repetitive wiki maintenance tasks through CLI commands
### Core Philosophy An LLM agent can maintain a knowledge base, but left alone it wastes
context on chores and slowly breaks things: dead links, missing
frontmatter, wrapped table rows, orphan pages, overwritten sources.
> Use scripts for deterministic tasks, reserve AI tokens for creative and analytical work Fiona fixes that split of work:
The toolkit is designed to be **AI-friendly** - outputs are formatted in | The agent does | Fiona does |
consistent, parseable structures that LLMs can easily read and process. | --- | --- |
| Reads sources and writes synthesized pages | Fetches, archives, and converts sources to markdown |
| Decides what links to what | Finds dead links, dead wikilinks, and orphan pages |
| Refines raw pages | Lints markdown, frontmatter, Python, and JSON |
| Answers your questions from the wiki | Commits, snapshots, and restores wiki history |
| Merges template updates | Detects template updates and never overwrites your files |
> Use scripts for deterministic tasks. Reserve AI tokens for creative and
> analytical work.
--- ---
## Features ## Features
### Tool Suite ### Wiki in one command
| Tool | Purpose | Key Capabilities | `fiona-assistant wiki init` creates a ready-to-use llm-wiki: directory
| ---- | ------- | ---------------- | layout, settings, agent rules (`AGENTS.md`), page schema, index, log, and
| **Librarian** | Wiki content management | List files, scan directories, organize MD content | bundled agent skills. It also creates a git repository and makes the
| **MDoctor** | Markdown diagnosis | Lint markdown files using PyMarkdown, detect formatting issues | first commit, with `personal/` kept out of version control.
| **Python-Ruff** | Code quality | Lint Python files using Ruff, enforce code standards |
| **Generate-TOC** | Table of contents | Auto-generate nested TOC from markdown headers |
### Wiki Structure Support ### Source ingestion
Fiona-Assistant understands and maintains the [LLM Wiki directory structure](./llm-wiki/AGENTS.md): `fiona-assistant wiki ingest <url-or-file>` turns a source into a wiki page
in one step:
```text - Downloads the URL or reads the local file
llm-wiki/ - Archives an untouched copy under `raw/` (duplicates are detected by
├── ingest/ # Input zone: files to be processed SHA-256, so the same bytes are never stored twice)
├── raw/ # Archived source materials (read-only) - Converts it to markdown (HTML, PDF, plain text, and markdown ship today)
│ ├── articles/ - Writes a page under `wiki/` with full provenance in its frontmatter and
│ ├── papers/ `status: raw`, so the agent knows it still needs refining
│ ├── transcripts/ - Appends the action to `wiki/log.md`
│ └── assets/
├── wiki/ # Processed content `wiki ingest --pending` lists every page still waiting to be refined.
│ └── concepts/ # Synthesized concept notes
├── personal/ # User notes (read-only for agents) ### One-call maintenance
└── utils/ # Utility scripts
``` `fiona-assistant housekeeping` runs the whole maintenance routine and
prints a to-do list for the agent:
- **check-update**: is the packaged template newer than this wiki?
- **merge**: which `*-update.md` files are waiting to be merged?
- **ingest**: which pages are still `status: raw`?
- **sanity**: which lint checks fail, and where?
- **graph**: regenerates `wiki_graph.html` and lists orphan pages
It only reads, except for writing the graph. Every finding comes with a
concrete "next step" the agent can act on. Add `--json` for machine use.
### Quality checks
`fiona-assistant wiki sanity` runs five checks in parallel and gives one
pass/fail verdict:
| Check | What it catches |
| --- | --- |
| `markdownlint` | Markdown style, via PyMarkdown |
| `frontmatter_lint` | Missing or malformed YAML frontmatter (`name`, `version`, and more) |
| `python-ruff` | Python issues in wiki scripts, via Ruff |
| `mdlinter` | Dead links, dead `[[wikilinks]]`, and table rows split across lines |
| `jsonlint` | JSON syntax, indentation, trailing commas, duplicate keys |
Each linter is also available on its own, and `mdoctor fix` and
`json-lint fix` repair what they can automatically.
### Interactive link graph
`fiona-assistant wiki graph` writes a single self-contained HTML file
that shows how your pages link to each other. Nodes are scored by how
many different pages point at them, so hub pages stand out. It
understands markdown links and `[[wikilinks]]`, shows external URLs, and
flags broken links and orphans. `--format json` exports the same graph
for other tools.
### Safe template updates
As Fiona improves, so does its wiki template. `wiki check-update` shows
what changed, and `wiki update` brings your wiki up to date without ever
overwriting a file you own. When both sides changed, Fiona writes
`<name>-update.md` next to your file and `housekeeping` tells the agent
to merge it.
### Git history built in
`fiona-assistant git` wraps the operations an agent needs, with
timestamped commit messages:
- `save` commits all current edits
- `snapshot <name>` commits and tags a restore point
- `restore <name>` returns to a snapshot without losing history
- `undo`, `clean`, and `status` for everyday recovery
### Guardrails
- `personal/` and `raw/` are read-only for every automated tool. Ingest
may add new files to `raw/`; it never changes an existing one.
- Commands that need a decision prompt on a terminal and accept
`--yes`, `--force`, or `--non-interactive` for unattended runs.
- Respects `.gitignore` when scanning.
### Table of contents generation
`fiona-assistant librarian toc` builds `table_of_contents.md` for a folder
tree. `librarian toc file <path>` keeps an in-file TOC under the H1 up to
date, with optional section numbering (`--sectionized`).
--- ---
## Installation ## Quick Start
### Quick Install (Production) ### 1. Install
Requires Python 3.11 or newer.
```bash ```bash
python -m pip install fiona-assistant python -m pip install fiona-assistant
``` ```
### Development Setup ### 2. Create a wiki
- **Clone the repository:**
```bash
git clone <repository-url>
cd fiona-assistant
```
- **Create and activate the virtual environment:**
```bash
source bootstrap --env fiona_env -c
```
> Make sure `fiona_env` is activated as pyenv before proceeding
---
## Usage
### Main CLI
Fiona-Assistant provides a unified CLI with multiple subcommands:
```bash ```bash
# Show all available commands fiona-assistant wiki init my-wiki
fiona-assistant --help cd my-wiki
# Run via Python module
python -m fiona --help
``` ```
#### Available Commands ### 3. Point your agent at it
Open the folder in your AI coding agent (Cursor, Claude Code, or any
agent that reads `AGENTS.md`). The bundled `AGENTS.md` loads the Fiona
skill, which teaches the agent the ingest, refine, and housekeeping
workflow and when to call each command.
### 4. Feed it sources
```bash ```bash
# Run tests fiona-assistant wiki ingest https://example.com/post.html
fiona-assistant test fiona-assistant wiki ingest ./notes/meeting.pdf
# Manage wiki content with Librarian
fiona-assistant librarian [directory]
# Generate an in-file table of contents below the H1
fiona-assistant librarian toc file <path_to_file>
# Force section numbering on for the run
fiona-assistant librarian toc file <path_to_file> --sectionized
# Generate in-file TOCs for every md file in the current directory / wiki root
fiona-assistant librarian toc file '*'
# Diagnose markdown files with MDoctor
fiona-assistant mdoctor [chklint] [directory]
# Lint Python code with Ruff
fiona-assistant python-ruff [chklint] [directory]
``` ```
### Direct Tool Access Then ask the agent to refine the new pages.
Each tool can also be called directly: ### 5. Keep it healthy
```bash ```bash
# Librarian - List all files in a directory fiona-assistant housekeeping
librarian [directory] fiona-assistant git save --description "ingest meeting notes"
# MDoctor - Check markdown lint violations
mdoctor chklint [directory]
# MDoctor - List all files
# Python-Ruff - Check Python code quality
python-ruff chklint [directory]
# Generate table of contents for markdown files
generate-toc [directory] [options]
``` ```
--- ---
## Workflow Examples ## How It Works
### Wiki Ingestion Workflow A wiki created by Fiona has this layout:
```bash
# 1. Drop raw files into ingest directory
# (Manual or automated process)
# 2. List files ready for processing
librarian llm-wiki/ingest
# 3. Validate markdown quality before processing
mdoctor chklint llm-wiki/ingest
# 4. Move processed files to raw archive
# (Agent/orchestration task)
```
### Content Maintenance Workflow
```bash
# Check all markdown files in wiki for lint violations
mdoctor chklint llm-wiki/wiki
# Generate table of contents for concept notes
generate-toc llm-wiki/wiki/concepts --recursive-toc
# Validate Python code in the project
python-ruff chklint src/
# List all markdown files for inventory
librarian llm-wiki/
```
### Standalone Document Organization
```bash
# Organize any directory of markdown files
librarian /path/to/docs/
# Validate markdown formatting
mdoctor chklint /path/to/docs/
# Create nested table of contents
generate-toc /path/to/docs/ --recursive-toc
```
---
## Output Format
All tools produce **AI-friendly output** with consistent formatting:
### MDoctor Output Format
```text ```text
PyMarkdownLint Violations my-wiki/
TARGET: /path/to/directory ├── AGENTS.md # Entry point for the agent; loads the Fiona skill
├── fiona-settings.json # Marks the wiki root; tool settings
├── ingest/ # Drop zone for files waiting to be parsed
├── raw/ # Immutable archive of original sources
│ ├── articles/
│ ├── papers/
│ ├── transcripts/
│ ├── assets/
│ └── other/
├── wiki/ # Pages the agent writes and maintains
│ ├── schema.md # Page types, tags, conventions
│ ├── index.md # Content catalog
│ ├── log.md # Append-only action log
│ ├── concepts/
│ ├── entities/
│ ├── comparisons/
│ ├── queries/
│ └── sources/
├── personal/ # Your own notes (read-only for agents, untracked)
└── utils/ # Agent skills, personas, and helper scripts
```
The cycle is:
- **Ingest**: `wiki ingest` archives a source in `raw/` and writes a raw
page in `wiki/`.
- **Refine**: the agent rewrites the page, links it to related pages, and
sets `status: refined`.
- **Maintain**: `housekeeping` reports what needs attention; the agent
fixes it.
- **Save**: `git save` records the change.
The full page schema lives in the template's
[schema](./src/fiona/data/llm_wiki/wiki/schema.md).
---
## Command Reference
Run `fiona-assistant --help` or `fiona-assistant <command> --help` for all
flags. Commands that take an optional directory default to the current
directory.
### Wiki lifecycle
| Command | What it does |
| --- | --- |
| `wiki init [dir]` | Create a new llm-wiki with git history |
| `wiki ingest <source>` | Fetch, archive, and convert a URL or file into a raw page |
| `wiki ingest --pending` | List pages still marked `status: raw` |
| `wiki sanity [dir]` | Run every quality check; exit 1 on failure |
| `wiki graph [dir]` | Write `wiki_graph.html` (or JSON with `--format json`) |
| `wiki check-update [dir]` | Show what a template update would change |
| `wiki update [dir]` | Apply a template update (`--dry-run`, `--yes`) |
| `housekeeping [dir]` | Run the full maintenance routine and list next steps |
### Linters
| Command | What it does |
| --- | --- |
| `mdoctor chklint [dir]` | Markdown lint via PyMarkdown |
| `mdoctor fix [dir]` | Auto-fix markdown lint violations |
| `fmlint chklint [dir]` | Frontmatter lint |
| `python-ruff chklint [dir]` | Python lint via Ruff |
| `json-lint chklint [dir]` | JSON lint |
| `json-lint fix [dir]` | Auto-fix JSON indentation and trailing commas |
### Files and TOC
| Command | What it does |
| --- | --- |
| `librarian list [dir]` | List files in a directory |
| `librarian toc [dir]` | Write `table_of_contents.md` (`--recursive-toc`, `--exclude REGEX`) |
| `librarian toc file <path>` | Write an in-file TOC below the H1 (`--sectionized`) |
| `librarian toc file '*'` | Update in-file TOCs for every markdown file in the wiki |
### Git
| Command | What it does |
| --- | --- |
| `git save --description TEXT` | Commit all edits |
| `git snapshot <name> --description TEXT` | Commit and tag a restore point |
| `git restore <name>` | Restore a snapshot as a new commit |
| `git undo` | Drop the last commit |
| `git clean` | Discard uncommitted edits |
| `git status` | Show whether the tree is clean |
Every command above is run as `fiona-assistant <command>`. The
`librarian` and `mdoctor` tools are also installed as standalone
commands.
---
## Output Built for Agents
Every report uses the same shape: a header, the target, rule
definitions, findings per file with line and column, and a summary. An
agent can parse it without guessing, and a human can skim it.
```text
Markdown Lint Violations
TARGET: wiki
- DEFINITIONS: - DEFINITIONS:
- md001: Heading levels should only increment by one, Heading levels should only increment by one level at a time - MDL001: Dead Link, Relative markdown link does not resolve to an existing file
- md013: Line length, Line length - MDL002: Dead Wiki Link, Wiki-internal link does not resolve to any md page in the wiki
- FILE: file1.md - FILE: wiki/concepts/retrieval.md
- 10:5 md013 - MDL001 line 12: ../sources/old-paper.md does not exist
- 20:3 md001 - MDL002 line 30: [[embeddings]] does not resolve to any page
- SUMMARY: - SUMMARY:
- Total files with violations: 1 - Total files with violations: 1
- Total violations: 2 - Total violations: 2
``` ```
### Python-Ruff Output Format `wiki sanity`, `housekeeping`, `wiki ingest`, `wiki check-update`, and
`wiki update` also accept `--json` for structured output.
```text
Ruff Violations
TARGET: /path/to/directory
Definitions:
- F401: `*` import used, but not imported
- E501: Line too long
- FILE: script.py
- 15:10 F401
- 25:80 E501
- SUMMARY:
- Total files with violations: 1
- Total violations: 2
```
--- ---
## Project Structure ## Roadmap
Planned work, tracked with requirements and acceptance criteria in
[SPEC](./SPEC.md) and prioritized in [todo](./todo.md):
| Feature | What it will do |
| --- | --- |
| Wiki root discovery | Run any command from a subfolder; Fiona finds the nearest `fiona-settings.json` |
| Backup | One command writes a compressed, restorable archive of the wiki outside the tree |
| `setup` command | Interactive review and editing of `fiona-settings.json`, with validation |
| Smarter graph | Show only important nodes until you expand a parent; hide gitignored pages |
| Code blocks in markdown | Lint Python and shell snippets inside fenced code blocks |
| Shellcheck | Lint shell scripts as part of wiki sanity |
| HTML and JavaScript lint | Validate HTML and JS files in the wiki |
| `python-ruff fix` | Auto-fix Python lint findings |
| More converters | Ingest more source formats beyond HTML, PDF, text, and markdown |
| Version tags for scripts | Track versions of Python and shell files like markdown pages |
---
## Development
### Set up
Clone the repository and load the environment with the bootstrap script.
Linux and macOS:
```bash
git clone <repository-url>
cd fiona-assistant
source bootstrap --env fiona_env -c
```
Windows (PowerShell):
```powershell
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
. .\bootstrap.ps1 --env fiona_env -c
```
The default install is editable (`pip install -e ".[dev]"`). On Windows
the environment lives in `.venvs\fiona_env`.
### Other install modes
| Goal | Bash | PowerShell |
| --- | --- | --- |
| Install this checkout as a normal package | `source bootstrap --local` | `. .\bootstrap.ps1 --local` |
| Same as `pip install fiona-assistant`, no venv | `source bootstrap --local --no-env` | `powershell -ExecutionPolicy Bypass -File .\bootstrap.ps1 --local --no-env` |
`--local --no-env` runs `python -m pip install --user .`. If
`fiona-assistant` is not found afterwards, add the user scripts
directory printed by bootstrap to `PATH`.
### Test and check
```bash
# Unit tests with coverage
python scripts/run_tests.py
# Full sanity: tests, coverage gate, markdown, Ruff, frontmatter, JSON
python scripts/run_sanity.py --non-interactive
```
Contributions follow the spec-first, test-first lifecycle in
[AGENTS](./AGENTS.md).
### Project structure
```text ```text
fiona-assistant/ fiona-assistant/
├── llm-wiki/ # Wiki content and structure definition ├── src/fiona/ # The installable package
│ ├── AGENTS.md # Agent rules and directory permissions │ ├── __main__.py # fiona-assistant CLI
│ ├── ingest/ # Input zone for new content │ ├── librarian/ # File listing and TOC generation
│ ├── raw/ # Archived source materials │ ├── mdoctor/ # Markdown lint and fix
│ ├── wiki/ # Processed concept notes │ ├── customlinter/ # Frontmatter, markdown link, Python, JSON linters
│ └── utils/ # Utility scripts │ ├── python_ruff/ # Ruff wrapper
│ │ ├── gittool/ # Git operations
├── src/ │ ├── sanity_runner/ # Shared sanity engine
│ ├── fiona/ # Main CLI entry point │ ├── wiki/ # init, update, sanity, graph, ingest, housekeeping
│ │ └── __main__.py # Unified command dispatcher │ └── data/llm_wiki/ # Packaged wiki template
│ │ ├── tests/ # Unit tests (not installed)
│ ├── librarian/ # Wiki content management ├── scripts/ # Test and sanity runners, dev settings
│ │ ├── __init__.py ├── bootstrap # Environment setup (bash)
│ │ ├── librarian.py # File scanning and listing ├── bootstrap.ps1 # Environment setup (PowerShell)
│ │ └── toc_generator.py # Table of contents generator ├── SPEC.md # Requirements and acceptance criteria
│ │ └── pyproject.toml
│ ├── mdoctor/ # Markdown diagnosis
│ │ ├── __init__.py
│ │ └── mdoctor.py # PyMarkdown lint wrapper
│ │
│ ├── python_ruff/ # Python code linting
│ │ ├── __init__.py
│ │ └── python_ruff.py # Ruff lint wrapper
│ │
│ ├── tests/ # Unit tests
│ │ ├── test_fiona_main.py
│ │ ├── test_librarian.py
│ │ ├── test_mdoctor.py
│ │ └── test_python_ruff.py
│ │
│ ├── run_tests.py # Test runner
│ └── README.md # Development documentation
│
├── pyproject.toml # Project configuration
├── bootstrap # Environment setup script
├── LICENSE # GPL-3.0-only
└── README.md # This file
``` ```
--- ---
## Testing ## Documentation
### Run Tests - [Project Specification](./SPEC.md): every command, setting, and rule,
with acceptance criteria
```bash - [LLM Wiki Agent Rules](./src/fiona/data/llm_wiki/AGENTS.md): what a
# Run all tests deployed wiki tells its agent
python src/run_tests.py - [Fiona skill](./src/fiona/data/llm_wiki/utils/skills/fiona/SKILL.md):
the workflow the agent follows inside a wiki
# Or use pytest directly - [Development Agent Rules](./AGENTS.md): how to contribute to this
python -m pytest repository
- [Roadmap](./todo.md): planned and in-progress work
# Run with coverage
python -m pytest --cov=src
```
--- ---
## License ## License
GPL-3.0-only GPL-3.0-only
---
## Key Concepts
### For LLM Agents
- **Deterministic Operations**: Use Fiona-Assistant for file listing, linting, and organization
- **Token Efficiency**: Reserve LLM context for synthesis, analysis, and creative tasks
- **Structured Output**: All tool outputs are formatted for easy parsing by LLMs
### For Human Users
- **CLI-First**: All functionality available through command-line interface
- **Composable Tools**: Combine tools in scripts for automated workflows
- **Clear Structure**: Maintains consistent wiki organization standards
---
## Related Documentation
- [LLM Wiki Agent Rules](./llm-wiki/AGENTS.md) - Directory permissions and workflow rules
- [Development Setup](./src/README.md) - Detailed development information
- [Project Specification](./SPEC.md) - Feature spec sheet with requirements and acceptance criteria
- [Project TODO](./todo.md) - Roadmap and planned features
---
## Quick Reference Card
| Task | Command |
| ---- | ------- |
| Show help | `fiona-assistant --help` |
| List files | `fiona-assistant librarian /path/to/dir` |
| Check MD lint | `fiona-assistant mdoctor chklint /path/to/dir` |
| Check Python lint | `fiona-assistant python-ruff chklint /path/to/dir` |
| Generate TOC | `generate-toc /path/to/dir --recursive-toc` |
| Generate file TOC | `fiona-assistant librarian toc file /path/to/file.md --sectionized` |
| Run tests | `python src/run_tests.py` |
| Install | `pip install fiona-assistant` |
| Dev install | `pip install -e .` |
---
> **Note**: All paths are relative to the current working directory unless specified as absolute paths.
+4446 -1956
View File
File diff suppressed because it is too large Load Diff
+189 -82
View File
@@ -1,23 +1,53 @@
#!/bin/bash #!/bin/bash
# Modified by AI/LLM: 2026-09-30 - --local and --no-env install options.
# #
# Project bootstrap script - Sets up Python development environment # Project bootstrap script - Sets up Python development environment
# #
# This script MUST be sourced to properly set up the environment: # This script MUST be sourced to properly set up the environment:
# source bootstrap # source bootstrap
# #
# Windows (PowerShell): dot-source bootstrap.ps1 instead
# . .\bootstrap.ps1 [--env name] [--python 3.x.x] [-c|--clean] [--local] [--no-env]
#
# What it does: # What it does:
# 1. Installs pyenv and required system dependencies if needed # 1. Installs pyenv and required system dependencies if needed
# 2. Initializes pyenv and pyenv-virtualenv in the current shell # 2. Initializes pyenv and pyenv-virtualenv in the current shell
# 3. Creates a Python virtual environment named 'test_env' (or custom name) # 3. Creates a Python virtual environment named 'test_env' (or custom name)
# 4. Activates the virtual environment # 4. Activates the virtual environment
# 5. Installs all project requirements from requirements.txt files # 5. Installs this repository (editable with the dev extra, unless --local)
# #
# With --clean/-c option: # With --clean/-c option:
# - Deletes existing environment before creating a new one # - Deletes existing environment before creating a new one
# #
# With --local:
# - Installs this repository as a normal package (pip install .), not editable
#
# With --no-env:
# - Skips pyenv and the virtual environment
# - Installs with pip --user so the commands run without activating an environment
#
# Usage: # Usage:
# source bootstrap [--pyenv] [-c|--clean] [--env custom_name] [--python 3.x.x] # source bootstrap [--pyenv] [-c|--clean] [--env custom_name] [--python 3.x.x] [--local] [--no-env]
# ex. source bootstrap --pyenv -c --env my_env --python 3.10.6 # ex. source bootstrap --pyenv -c --env my_env --python 3.10.6
# ex. source bootstrap --local --no-env
# =============================================
# SOURCED GUARD
# =============================================
# Activation only persists when this file is sourced; executing it
# loses the environment when the child shell exits. `return` works
# at the top level only inside a sourced script.
if [ -z "${BASH_VERSION:-}" ]; then
echo "ERROR: bootstrap is a bash script. Run it with bash:" >&2
echo "Run: source bootstrap [--pyenv] [-c|--clean] [--env name] [--python 3.x.x] [--local] [--no-env]" >&2
exit 1
fi
if ! (return 0 2>/dev/null); then
echo "ERROR: bootstrap must be sourced, not executed." >&2
echo "The environment would be lost when the child shell exits." >&2
echo "Run: source bootstrap [--pyenv] [-c|--clean] [--env name] [--python 3.x.x] [--local] [--no-env]" >&2
exit 1
fi
# ============================================= # =============================================
# VARIABLES # VARIABLES
@@ -26,10 +56,13 @@ PYTHON_VERSION="3.13.8" # Default Python version
export PYENV_NAME="test_env" # Default environment name export PYENV_NAME="test_env" # Default environment name
FORCE_PYENV_INSTALL=false # Flag to force pyenv reinstallation FORCE_PYENV_INSTALL=false # Flag to force pyenv reinstallation
CLEAN_ENVIRONMENT=false # Flag to delete and recreate environment CLEAN_ENVIRONMENT=false # Flag to delete and recreate environment
LOCAL_INSTALL=false # Install this repo as a normal package, not editable
NO_ENV=false # Skip the virtual environment and pip install --user
PYTHON_BIN="python"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" # scripts root directory SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" # scripts root directory
# Source utilities (needed for echo_error, echo_status, etc.) # Source utilities (needed for echo_error, echo_status, etc.)
source "$SCRIPT_DIR/src/utils.sh" source "$SCRIPT_DIR/scripts/utils.sh"
# Parse command line arguments # Parse command line arguments
while [[ $# -gt 0 ]]; do while [[ $# -gt 0 ]]; do
@@ -42,6 +75,14 @@ while [[ $# -gt 0 ]]; do
CLEAN_ENVIRONMENT=true CLEAN_ENVIRONMENT=true
shift shift
;; ;;
--local)
LOCAL_INSTALL=true
shift
;;
--no-env)
NO_ENV=true
shift
;;
--env) --env)
if [ -n "$2" ]; then if [ -n "$2" ]; then
PYENV_NAME="$2" PYENV_NAME="$2"
@@ -62,7 +103,7 @@ while [[ $# -gt 0 ]]; do
;; ;;
*) *)
echo_error "Unknown option: $1" echo_error "Unknown option: $1"
echo "Usage: source bootstrap [--pyenv] [-c|--clean] [--env custom_name] [--python 3.x.x]" echo "Usage: source bootstrap [--pyenv] [-c|--clean] [--env custom_name] [--python 3.x.x] [--local] [--no-env]"
return 1 return 1
;; ;;
esac esac
@@ -283,7 +324,7 @@ activate_environment() {
# Get the virtualenv path # Get the virtualenv path
local venv_path local venv_path
venv_path=$(pyenv prefix "$PYENV_NAME" 2>/dev/null) venv_path=$(pyenv prefix "$PYENV_NAME" 2>/dev/null)
if [ -n "$venv_path" ] && [ -d "$venv_path" ]; then if [ -n "$venv_path" ] && [ -d "$venv_path" ]; then
# Manually activate by sourcing the activate script # Manually activate by sourcing the activate script
local activate_script="$venv_path/bin/activate" local activate_script="$venv_path/bin/activate"
@@ -296,7 +337,7 @@ activate_environment() {
return 0 return 0
fi fi
fi fi
# Fallback: try pyenv activate # Fallback: try pyenv activate
if pyenv activate "$PYENV_NAME" 2>/dev/null; then if pyenv activate "$PYENV_NAME" 2>/dev/null; then
echo_status "Successfully activated $PYENV_NAME with pyenv" echo_status "Successfully activated $PYENV_NAME with pyenv"
@@ -331,52 +372,82 @@ install_requirements() {
echo_status "Installing requirements..." echo_status "Installing requirements..."
# Check if pyproject.toml exists (could be in root or src/) # Check if pyproject.toml exists (could be in root or src/)
local pyproject_root="$SCRIPT_DIR/pyproject.toml" local project_dir=""
local pyproject_src="$SCRIPT_DIR/src/pyproject.toml" if [ -f "$SCRIPT_DIR/pyproject.toml" ]; then
project_dir="$SCRIPT_DIR"
if [ -f "$pyproject_root" ]; then elif [ -f "$SCRIPT_DIR/src/pyproject.toml" ]; then
# Install the package in development mode from root project_dir="$SCRIPT_DIR/src"
echo_status "Installing package and dependencies from pyproject.toml..."
(cd "$SCRIPT_DIR" && pip install -e ".[dev]" --break-system-packages) || {
echo_error "Failed to install package and dependencies from pyproject.toml"
return 1
}
elif [ -f "$pyproject_src" ]; then
# Install the package in development mode from src/
echo_status "Installing package and dependencies from pyproject.toml..."
(cd "$SCRIPT_DIR/src" && pip install -e ".[dev]" --break-system-packages) || {
echo_error "Failed to install package and dependencies from pyproject.toml"
return 1
}
else
echo_warning "No pyproject.toml found"
# Fallback: look for any requirements.txt files
local req_files=()
while IFS= read -r -d '' req_file; do
req_files+=("$req_file")
done < <(find "$SCRIPT_DIR" -name "requirements.txt" -print0)
if [ ${#req_files[@]} -eq 0 ]; then
echo_status "No requirements files found"
return 0
fi
for req_file in "${req_files[@]}"; do
echo_status "Installing requirements from: $req_file"
pip install -r "$req_file" --break-system-packages || {
echo_error "Failed to install requirements from $req_file"
return 1
}
done
fi fi
if [ -n "$project_dir" ]; then
local helper="$SCRIPT_DIR/scripts/bootstrap_install.py"
if [ ! -f "$helper" ]; then
echo_error "Missing $helper"
return 1
fi
local flags=(--break-system-packages)
if [ "$LOCAL_INSTALL" = true ]; then
echo_status "Installing this repository as a normal package..."
flags+=(--local)
else
echo_status "Installing package and dependencies from pyproject.toml..."
fi
if [ "$NO_ENV" = true ]; then
flags+=(--no-env)
fi
local pip_words=()
local word
while IFS= read -r word; do
if [ -n "$word" ]; then
pip_words+=("$word")
fi
done < <("$PYTHON_BIN" "$helper" "${flags[@]}")
if [ "${#pip_words[@]}" -eq 0 ]; then
echo_error "Failed to resolve pip install arguments"
return 1
fi
(cd "$project_dir" && "$PYTHON_BIN" -m pip "${pip_words[@]}") || {
echo_error "Failed to install package and dependencies from pyproject.toml"
return 1
}
return 0
fi
echo_warning "No pyproject.toml found"
# Fallback: look for any requirements.txt files
local req_files=()
while IFS= read -r -d '' req_file; do
req_files+=("$req_file")
done < <(find "$SCRIPT_DIR" -name "requirements.txt" -print0)
if [ ${#req_files[@]} -eq 0 ]; then
echo_status "No requirements files found"
return 0
fi
local user_args=()
if [ "$NO_ENV" = true ]; then
user_args+=(--user)
fi
for req_file in "${req_files[@]}"; do
echo_status "Installing requirements from: $req_file"
"$PYTHON_BIN" -m pip install "${user_args[@]}" -r "$req_file" --break-system-packages || {
echo_error "Failed to install requirements from $req_file"
return 1
}
done
return 0 return 0
} }
upgrade_pip() { upgrade_pip() {
echo_status "Upgrading pip..." echo_status "Upgrading pip..."
python -m pip install --upgrade pip --break-system-packages || { local upgrade_args=(install --upgrade pip --break-system-packages)
if [ "$NO_ENV" = true ]; then
upgrade_args=(install --user --upgrade pip --break-system-packages)
fi
"$PYTHON_BIN" -m pip "${upgrade_args[@]}" || {
echo_warning "Failed to upgrade pip" echo_warning "Failed to upgrade pip"
echo_warning "Continuing with requirements installation..." echo_warning "Continuing with requirements installation..."
} }
@@ -384,18 +455,18 @@ upgrade_pip() {
install_dev_packages() { install_dev_packages() {
echo_status "Installing local Python packages in development mode..." echo_status "Installing local Python packages in development mode..."
local install_script="$SCRIPT_DIR/src/install_packages.sh" local install_script="$SCRIPT_DIR/scripts/install_packages.sh"
if [ -f "$install_script" ]; then if [ -f "$install_script" ]; then
# Use source to run in current shell context, preserving environment variables # Use source to run in current shell context, preserving environment variables
(cd "$SCRIPT_DIR/src" && . "$install_script") || { (cd "$SCRIPT_DIR" && . "$install_script") || {
echo_error "Dev packages not installed!" echo_error "Dev packages not installed!"
return 1 return 1
} }
else else
echo_status "Install script not found at $install_script, attempting direct install..." echo_status "Install script not found at $install_script, attempting direct install..."
# Fallback: install in editable mode from src directory # Fallback: install in editable mode from the repository root
(cd "$SCRIPT_DIR/src" && pip install -e . --break-system-packages) || { (cd "$SCRIPT_DIR" && pip install -e . --break-system-packages) || {
echo_error "Failed to install package in development mode" echo_error "Failed to install package in development mode"
return 1 return 1
} }
@@ -411,16 +482,16 @@ sanitize_pytest_env() {
# Also remove -S or --no-summary (older versions) # Also remove -S or --no-summary (older versions)
local sanitized local sanitized
sanitized=$(echo "$PYTEST_ADDOPTS" | sed -e 's/--no-summary[[:space:]]*//g' -e 's/--no-summary//g' -e 's/-S[[:space:]]*//g' -e 's/-S//g') sanitized=$(echo "$PYTEST_ADDOPTS" | sed -e 's/--no-summary[[:space:]]*//g' -e 's/--no-summary//g' -e 's/-S[[:space:]]*//g' -e 's/-S//g')
# Remove leading/trailing whitespace and any double spaces # Remove leading/trailing whitespace and any double spaces
sanitized=$(echo "$sanitized" | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//' -e 's/[[:space:]]\+/ /g') sanitized=$(echo "$sanitized" | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//' -e 's/[[:space:]]\+/ /g')
if [ "$sanitized" != "$PYTEST_ADDOPTS" ]; then if [ "$sanitized" != "$PYTEST_ADDOPTS" ]; then
echo_warning "Removing --no-summary from PYTEST_ADDOPTS to ensure test output is parsed correctly" echo_warning "Removing --no-summary from PYTEST_ADDOPTS to ensure test output is parsed correctly"
export PYTEST_ADDOPTS="$sanitized" export PYTEST_ADDOPTS="$sanitized"
fi fi
fi fi
# Also unset FORCE_COLOR and PY_COLORS as they can interfere with output parsing # Also unset FORCE_COLOR and PY_COLORS as they can interfere with output parsing
if [ -n "$FORCE_COLOR" ]; then if [ -n "$FORCE_COLOR" ]; then
echo_warning "Unsetting FORCE_COLOR to ensure pytest output is parsed correctly" echo_warning "Unsetting FORCE_COLOR to ensure pytest output is parsed correctly"
@@ -430,42 +501,67 @@ sanitize_pytest_env() {
echo_warning "Unsetting PY_COLORS to ensure pytest output is parsed correctly" echo_warning "Unsetting PY_COLORS to ensure pytest output is parsed correctly"
unset PY_COLORS unset PY_COLORS
fi fi
return 0 return 0
} }
# ============================================= # =============================================
# MAIN EXECUTION # MAIN EXECUTION
# ============================================= # =============================================
echo_status "Starting project bootstrap with environment: $PYENV_NAME" if [ "$NO_ENV" = true ]; then
echo_status "Starting project bootstrap (no virtual environment)"
if [ "$FORCE_PYENV_INSTALL" = true ]; then else
echo_status "Forcing pyenv reinstallation" echo_status "Starting project bootstrap with environment: $PYENV_NAME"
fi fi
# 1. Check if Python is installed if [ "$LOCAL_INSTALL" = true ]; then
if ! check_python_installed; then echo_status "Local install: this repository, not editable"
return 1 fi
if [ "$NO_ENV" = true ]; then
echo_status "No virtual environment: installing with --user"
fi fi
# 2. Check/install pyenv if [ "$NO_ENV" = true ]; then
if ! check_and_install_pyenv; then if ! command -v python >/dev/null 2>&1 && ! command -v python3 >/dev/null 2>&1; then
return 1 echo_error "Python is not installed. Please install Python $PYTHON_VERSION before running this script."
fi echo_error "Or download from: https://www.python.org/downloads/"
return 1
fi
if command -v python >/dev/null 2>&1; then
PYTHON_BIN="python"
else
PYTHON_BIN="python3"
fi
else
if [ "$FORCE_PYENV_INSTALL" = true ]; then
echo_status "Forcing pyenv reinstallation"
fi
# 3. Initialize pyenv # 1. Check if Python is installed
if ! initialize_pyenv; then if ! check_python_installed; then
return 1 return 1
fi fi
# 4. Check/create virtual environment # 2. Check/install pyenv
if ! check_environment_exists; then if ! check_and_install_pyenv; then
return 1 return 1
fi fi
# 5. Activate the environment # 3. Initialize pyenv
if ! activate_environment; then if ! initialize_pyenv; then
return 1 return 1
fi
# 4. Check/create virtual environment
if ! check_environment_exists; then
return 1
fi
# 5. Activate the environment
if ! activate_environment; then
return 1
fi
PYTHON_BIN="python"
fi fi
# 5.5. Sanitize pytest environment to ensure proper output parsing # 5.5. Sanitize pytest environment to ensure proper output parsing
@@ -474,14 +570,25 @@ sanitize_pytest_env
# 6. Upgrade pip # 6. Upgrade pip
upgrade_pip upgrade_pip
# 7. Install requirements (including package in dev mode with all dependencies) # 7. Install this repository
install_requirements if ! install_requirements; then
return 1
fi
# Final verification # Final verification
echo_status "Bootstrap completed successfully!" echo_status "Bootstrap completed successfully!"
echo_status "Virtual environment '$PYENV_NAME' is active" if [ "$NO_ENV" = true ]; then
echo_status "Python: $(python --version) ($(which python))" scripts_dir=$("$PYTHON_BIN" -c "import sys, sysconfig; scheme = 'nt_user' if sys.platform == 'win32' else 'posix_user'; print(sysconfig.get_path('scripts', scheme))")
echo_status "Pip: $(pip --version)" echo_status "Installed for the current user (no virtual environment)"
echo_status "Command directory: $scripts_dir"
echo_status "If fiona-assistant is not found, add that directory to PATH."
echo_status "Python: $("$PYTHON_BIN" --version) ($PYTHON_BIN)"
echo_status "Pip: $("$PYTHON_BIN" -m pip --version)"
else
echo_status "Virtual environment '$PYENV_NAME' is active"
echo_status "Python: $(python --version) ($(which python))"
echo_status "Pip: $(pip --version)"
fi
echo_status "" echo_status ""
echo_status "To verify installation, try:" echo_status "To verify installation, try:"
echo_status " fiona-assistant test" echo_status " fiona-assistant test"
+358
View File
@@ -0,0 +1,358 @@
# Generated by AI/LLM: 2026-09-29 - Windows PowerShell bootstrap for Fiona development
# Modified by AI/LLM: 2026-09-30 - --local and --no-env install options.
#
# Project bootstrap for Windows. Dot-source a virtual-environment install so
# the environment stays active in the current session (same idea as
# `source bootstrap`):
#
# Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
# . .\bootstrap.ps1
# . .\bootstrap.ps1 --env fiona_env
# . .\bootstrap.ps1 --env fiona_env --python 3.13.8 -c
# . .\bootstrap.ps1 --local
# powershell -ExecutionPolicy Bypass -File .\bootstrap.ps1 --local --no-env
#
# Creates a repo-local venv at .venvs\<name> with the Python launcher (py)
# or python on PATH. Does not install pyenv. --pyenv is accepted so the
# Linux flag still parses; on Windows it is a no-op notice.
#
# --local installs this repository as a normal package (not editable).
# --no-env skips the virtual environment and installs with pip --user,
# so fiona-assistant can run without activating an environment.
# --no-env may be executed with -File. A venv install must be dot-sourced.
#Requires -Version 5.1
# AI/LLM: flags are parsed by hand. Windows PowerShell 5.1 does not bind
# `--python` to a parameter alias, and the bash script uses those flags.
function Write-FionaStatus {
param([string]$Message)
Write-Host "[+] $Message" -ForegroundColor Green
}
function Write-FionaWarning {
param([string]$Message)
Write-Host "[!] $Message" -ForegroundColor Yellow
}
function Write-FionaError {
param([string]$Message)
Write-Host "[x] $Message" -ForegroundColor Red
}
function Remove-FionaBootstrapFunctions {
if (Get-Variable -Name FionaPrevErrorAction -Scope 1 -ErrorAction SilentlyContinue) {
Set-Variable -Name ErrorActionPreference -Value $FionaPrevErrorAction -Scope 1
Remove-Variable -Name FionaPrevErrorAction -Scope 1 -ErrorAction SilentlyContinue
}
foreach ($name in @(
"Write-FionaStatus",
"Write-FionaWarning",
"Write-FionaError",
"Resolve-FionaPython",
"Remove-FionaBootstrapFunctions"
)) {
Remove-Item -Path "Function:\$name" -ErrorAction SilentlyContinue
}
}
function Resolve-FionaPython {
# AI/LLM: pick a real interpreter. Prefer the py launcher so a requested
# version (3.13.8, then 3.13) wins over whatever `python` happens to be.
param([string]$Version)
$tags = @()
if ($Version) {
$tags += $Version
if ($Version -match '^(\d+\.\d+)') {
$minor = $Matches[1]
if ($minor -ne $Version) {
$tags += $minor
}
}
}
$py = Get-Command py -ErrorAction SilentlyContinue
if ($py) {
foreach ($tag in $tags) {
$exe = & py "-$tag" -c "import sys; print(sys.executable)" 2>$null
if ($LASTEXITCODE -eq 0 -and $exe) {
return @($exe)[-1].ToString().Trim()
}
}
if (-not $Version) {
$exe = & py -3 -c "import sys; print(sys.executable)" 2>$null
if ($LASTEXITCODE -eq 0 -and $exe) {
return @($exe)[-1].ToString().Trim()
}
}
}
if (-not $Version) {
foreach ($name in @("python", "python3")) {
$cmd = Get-Command $name -ErrorAction SilentlyContinue
if ($cmd -and $cmd.Source -notmatch "WindowsApps") {
return $cmd.Source
}
}
}
return $null
}
$FionaPrevErrorAction = $ErrorActionPreference
$ErrorActionPreference = "Stop"
# AI/LLM: a failed pip/py call must be checked via $LASTEXITCODE, not thrown.
if (Get-Variable -Name PSNativeCommandUseErrorActionPreference -ErrorAction SilentlyContinue) {
$PSNativeCommandUseErrorActionPreference = $false
}
$Clean = $false
$EnvName = "test_env"
$PythonVersion = ""
$Pyenv = $false
$Local = $false
$NoEnv = $false
$bootstrapArgs = @($args)
for ($i = 0; $i -lt $bootstrapArgs.Count; $i++) {
$token = [string]$bootstrapArgs[$i]
if ($token -eq "--pyenv") {
$Pyenv = $true
}
elseif ($token -eq "-c" -or $token -eq "--clean") {
$Clean = $true
}
elseif ($token -eq "--local") {
$Local = $true
}
elseif ($token -eq "--no-env") {
$NoEnv = $true
}
elseif ($token -eq "--env") {
if (($i + 1) -ge $bootstrapArgs.Count) {
Write-FionaError "--env requires an environment name argument"
Remove-FionaBootstrapFunctions
return
}
$i++
$EnvName = [string]$bootstrapArgs[$i]
}
elseif ($token -eq "--python") {
if (($i + 1) -ge $bootstrapArgs.Count) {
Write-FionaError "--python requires a version number argument"
Remove-FionaBootstrapFunctions
return
}
$i++
$PythonVersion = [string]$bootstrapArgs[$i]
}
else {
Write-FionaError "Unknown option: $token"
Write-FionaError "Usage: . .\bootstrap.ps1 [--pyenv] [-c|--clean] [--env custom_name] [--python 3.x.x] [--local] [--no-env]"
Remove-FionaBootstrapFunctions
return
}
}
if (-not $NoEnv -and $MyInvocation.InvocationName -ne ".") {
Write-FionaError "Dot-source this script so the virtual environment stays active in this session."
Write-FionaError " . .\bootstrap.ps1"
Write-FionaError "To install without a virtual environment, run:"
Write-FionaError " powershell -ExecutionPolicy Bypass -File .\bootstrap.ps1 --local --no-env"
Remove-FionaBootstrapFunctions
return
}
if ($Pyenv) {
Write-FionaWarning "Windows bootstrap uses a local venv under .venvs\. It does not install pyenv."
}
$repoRoot = $PSScriptRoot
if (-not $repoRoot) {
$repoRoot = (Get-Location).Path
}
if ($NoEnv) {
Write-FionaStatus "Starting project bootstrap (no virtual environment)"
}
else {
Write-FionaStatus "Starting project bootstrap with environment: $EnvName"
}
if ($Local) {
Write-FionaStatus "Local install: this repository, not editable"
}
if ($NoEnv) {
Write-FionaStatus "No virtual environment: installing with --user"
}
if ($NoEnv -and $Clean) {
Write-FionaWarning "--clean has no effect with --no-env"
}
try {
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass -Force -ErrorAction Stop | Out-Null
}
catch {
# AI/LLM: a GPO can refuse the process override. RemoteSigned still runs local scripts.
}
$pythonExe = Resolve-FionaPython -Version $PythonVersion
if (-not $pythonExe) {
Write-FionaError "Python is not installed, or version '$PythonVersion' is not available to the py launcher."
Write-FionaError "Install Python from https://www.python.org/downloads/ and enable the py launcher."
if ($PythonVersion) {
Write-FionaError "Then re-run: . .\bootstrap.ps1 --python $PythonVersion"
}
Remove-FionaBootstrapFunctions
return
}
$pipPython = $pythonExe
if (-not $NoEnv) {
$venvRoot = Join-Path $repoRoot ".venvs"
$venvPath = Join-Path $venvRoot $EnvName
if ($Clean -and (Test-Path -LiteralPath $venvPath)) {
Write-FionaStatus "Deleting existing virtual environment $EnvName..."
try {
Remove-Item -LiteralPath $venvPath -Recurse -Force
}
catch {
Write-FionaError "Failed to delete virtual environment $EnvName"
Write-FionaError $_.Exception.Message
Remove-FionaBootstrapFunctions
return
}
}
$activate = Join-Path $venvPath "Scripts\Activate.ps1"
if (-not (Test-Path -LiteralPath $activate)) {
Write-FionaStatus "Creating virtual environment $EnvName with $pythonExe..."
New-Item -ItemType Directory -Path $venvRoot -Force | Out-Null
& $pythonExe -m venv $venvPath
if ($LASTEXITCODE -ne 0) {
Write-FionaError "Failed to create virtual environment $EnvName"
Remove-FionaBootstrapFunctions
return
}
}
else {
Write-FionaStatus "Virtual environment $EnvName already exists"
}
Write-FionaStatus "Activating $EnvName..."
. $activate
if (-not $env:VIRTUAL_ENV) {
Write-FionaError "Failed to activate $EnvName"
Remove-FionaBootstrapFunctions
return
}
$pipPython = Join-Path $venvPath "Scripts\python.exe"
}
if ($env:PYTEST_ADDOPTS) {
$sanitized = $env:PYTEST_ADDOPTS -replace '--no-summary', '' -replace '(^|\s)-S(\s|$)', ' '
$sanitized = ($sanitized -replace '\s+', ' ').Trim()
if ($sanitized -ne $env:PYTEST_ADDOPTS.Trim()) {
Write-FionaWarning "Removing --no-summary from PYTEST_ADDOPTS so test output can be parsed"
if ($sanitized) {
$env:PYTEST_ADDOPTS = $sanitized
}
else {
Remove-Item Env:PYTEST_ADDOPTS
}
}
}
if ($env:FORCE_COLOR) {
Write-FionaWarning "Unsetting FORCE_COLOR so pytest output can be parsed"
Remove-Item Env:FORCE_COLOR
}
if ($env:PY_COLORS) {
Write-FionaWarning "Unsetting PY_COLORS so pytest output can be parsed"
Remove-Item Env:PY_COLORS
}
$pyproject = Join-Path $repoRoot "pyproject.toml"
if (-not (Test-Path -LiteralPath $pyproject)) {
Write-FionaError "No pyproject.toml found at $pyproject"
Remove-FionaBootstrapFunctions
return
}
Write-FionaStatus "Upgrading pip..."
if ($NoEnv) {
& $pipPython -m pip install --upgrade pip --user
}
else {
& $pipPython -m pip install --upgrade pip
}
if ($LASTEXITCODE -ne 0) {
Write-FionaWarning "Failed to upgrade pip. Continuing with package installation..."
}
$helper = Join-Path $repoRoot "scripts\bootstrap_install.py"
if (-not (Test-Path -LiteralPath $helper)) {
Write-FionaError "Missing $helper"
Remove-FionaBootstrapFunctions
return
}
$helperArgs = @($helper)
if ($Local) {
$helperArgs += "--local"
}
if ($NoEnv) {
$helperArgs += "--no-env"
}
$pipWords = @(& $pipPython @helperArgs | Where-Object { $_ -and "$_".Trim() -ne "" })
if ($LASTEXITCODE -ne 0 -or $pipWords.Count -eq 0) {
Write-FionaError "Failed to resolve pip install arguments"
Remove-FionaBootstrapFunctions
return
}
if ($Local) {
Write-FionaStatus "Installing this repository as a normal package..."
}
else {
Write-FionaStatus "Installing package and dependencies from pyproject.toml..."
}
Push-Location $repoRoot
try {
& $pipPython -m pip @pipWords
if ($LASTEXITCODE -ne 0) {
Write-FionaError "Failed to install package and dependencies from pyproject.toml"
Remove-FionaBootstrapFunctions
return
}
}
finally {
Pop-Location
}
$pythonVersionText = (& $pipPython --version 2>&1 | Out-String).Trim()
$pipVersionText = (& $pipPython -m pip --version 2>&1 | Out-String).Trim()
Write-FionaStatus "Bootstrap completed successfully!"
if ($NoEnv) {
$scriptsDir = (& $pipPython -c "import sysconfig; print(sysconfig.get_path('scripts', 'nt_user'))" | Out-String).Trim()
Write-FionaStatus "Installed for the current user (no virtual environment)"
Write-FionaStatus "Command directory: $scriptsDir"
Write-FionaStatus "If fiona-assistant is not found, add that directory to PATH."
}
else {
Write-FionaStatus "Virtual environment '$EnvName' is active"
}
Write-FionaStatus "Python: $pythonVersionText ($pipPython)"
Write-FionaStatus "Pip: $pipVersionText"
Write-FionaStatus ""
Write-FionaStatus "To verify installation, try:"
if (-not $NoEnv) {
Write-FionaStatus " python scripts/run_tests.py"
}
Write-FionaStatus " fiona-assistant --help"
Write-FionaStatus ""
Write-FionaStatus "Happy coding!"
Remove-FionaBootstrapFunctions
+2
View File
@@ -11,6 +11,8 @@ if ! python -m build --version >/dev/null 2>&1; then
fi fi
# Build the distribution files # Build the distribution files
rm -rf dist/ build/ fiona_assistant.egg-info/
python -m build python -m build
echo "Build completed. Distribution files are in the dist/ directory." echo "Build completed. Distribution files are in the dist/ directory."
-612
View File
@@ -1,612 +0,0 @@
<?xml version="1.0" ?>
<coverage version="7.16.1" timestamp="1790528640380" lines-valid="578" lines-covered="454" line-rate="0.7855" branches-covered="0" branches-valid="0" branch-rate="0" complexity="0">
<!-- Generated by coverage.py: https://coverage.readthedocs.io/en/7.16.1 -->
<!-- Based on https://raw.githubusercontent.com/cobertura/web/master/htdocs/xml/coverage-04.dtd -->
<sources>
<source>/home/user/repositories/fiona-assistant/src/customlinter</source>
</sources>
<packages>
<package name="." line-rate="0.7855" branch-rate="0" complexity="0">
<classes>
<class name="__init__.py" filename="__init__.py" complexity="0" line-rate="1" branch-rate="0">
<methods/>
<lines>
<line number="17" hits="1"/>
<line number="26" hits="1"/>
<line number="27" hits="1"/>
<line number="29" hits="1"/>
</lines>
</class>
<class name="base.py" filename="base.py" complexity="0" line-rate="0.6962" branch-rate="0">
<methods/>
<lines>
<line number="8" hits="1"/>
<line number="10" hits="1"/>
<line number="11" hits="1"/>
<line number="12" hits="1"/>
<line number="13" hits="1"/>
<line number="14" hits="1"/>
<line number="15" hits="1"/>
<line number="16" hits="1"/>
<line number="18" hits="1"/>
<line number="25" hits="1"/>
<line number="28" hits="1"/>
<line number="33" hits="1"/>
<line number="49" hits="1"/>
<line number="52" hits="1"/>
<line number="65" hits="1"/>
<line number="66" hits="1"/>
<line number="67" hits="1"/>
<line number="68" hits="1"/>
<line number="72" hits="1"/>
<line number="79" hits="1"/>
<line number="90" hits="1"/>
<line number="97" hits="1"/>
<line number="118" hits="1"/>
<line number="119" hits="1"/>
<line number="120" hits="1"/>
<line number="121" hits="1"/>
<line number="122" hits="1"/>
<line number="123" hits="1"/>
<line number="124" hits="1"/>
<line number="126" hits="1"/>
<line number="128" hits="1"/>
<line number="138" hits="1"/>
<line number="140" hits="1"/>
<line number="147" hits="1"/>
<line number="164" hits="1"/>
<line number="167" hits="1"/>
<line number="170" hits="1"/>
<line number="172" hits="1"/>
<line number="178" hits="1"/>
<line number="179" hits="1"/>
<line number="181" hits="1"/>
<line number="183" hits="1"/>
<line number="184" hits="1"/>
<line number="186" hits="1"/>
<line number="187" hits="1"/>
<line number="188" hits="1"/>
<line number="190" hits="1"/>
<line number="191" hits="1"/>
<line number="192" hits="1"/>
<line number="193" hits="1"/>
<line number="194" hits="0"/>
<line number="195" hits="1"/>
<line number="196" hits="1"/>
<line number="197" hits="1"/>
<line number="210" hits="1"/>
<line number="219" hits="0"/>
<line number="220" hits="0"/>
<line number="222" hits="0"/>
<line number="223" hits="0"/>
<line number="226" hits="0"/>
<line number="229" hits="0"/>
<line number="230" hits="0"/>
<line number="231" hits="0"/>
<line number="233" hits="0"/>
<line number="235" hits="1"/>
<line number="244" hits="1"/>
<line number="247" hits="1"/>
<line number="248" hits="1"/>
<line number="250" hits="1"/>
<line number="251" hits="1"/>
<line number="252" hits="1"/>
<line number="253" hits="1"/>
<line number="254" hits="1"/>
<line number="255" hits="1"/>
<line number="256" hits="1"/>
<line number="257" hits="0"/>
<line number="258" hits="0"/>
<line number="261" hits="1"/>
<line number="262" hits="1"/>
<line number="265" hits="1"/>
<line number="266" hits="1"/>
<line number="267" hits="1"/>
<line number="270" hits="1"/>
<line number="272" hits="1"/>
<line number="273" hits="1"/>
<line number="276" hits="1"/>
<line number="278" hits="1"/>
<line number="280" hits="1"/>
<line number="290" hits="1"/>
<line number="291" hits="1"/>
<line number="292" hits="1"/>
<line number="294" hits="1"/>
<line number="295" hits="1"/>
<line number="296" hits="1"/>
<line number="298" hits="1"/>
<line number="308" hits="1"/>
<line number="309" hits="1"/>
<line number="312" hits="1"/>
<line number="313" hits="1"/>
<line number="316" hits="1"/>
<line number="317" hits="1"/>
<line number="318" hits="1"/>
<line number="319" hits="1"/>
<line number="320" hits="1"/>
<line number="321" hits="1"/>
<line number="323" hits="1"/>
<line number="325" hits="1"/>
<line number="335" hits="1"/>
<line number="336" hits="1"/>
<line number="337" hits="1"/>
<line number="338" hits="1"/>
<line number="339" hits="1"/>
<line number="340" hits="0"/>
<line number="342" hits="0"/>
<line number="343" hits="0"/>
<line number="345" hits="1"/>
<line number="354" hits="1"/>
<line number="355" hits="1"/>
<line number="357" hits="1"/>
<line number="358" hits="1"/>
<line number="359" hits="1"/>
<line number="361" hits="1"/>
<line number="363" hits="1"/>
<line number="372" hits="1"/>
<line number="374" hits="1"/>
<line number="390" hits="0"/>
<line number="391" hits="0"/>
<line number="392" hits="0"/>
<line number="393" hits="0"/>
<line number="396" hits="0"/>
<line number="397" hits="0"/>
<line number="398" hits="0"/>
<line number="399" hits="0"/>
<line number="400" hits="0"/>
<line number="406" hits="0"/>
<line number="407" hits="0"/>
<line number="408" hits="0"/>
<line number="409" hits="0"/>
<line number="412" hits="0"/>
<line number="415" hits="0"/>
<line number="416" hits="0"/>
<line number="417" hits="0"/>
<line number="418" hits="0"/>
<line number="419" hits="0"/>
<line number="420" hits="0"/>
<line number="421" hits="0"/>
<line number="423" hits="0"/>
<line number="424" hits="0"/>
<line number="425" hits="0"/>
<line number="426" hits="0"/>
<line number="427" hits="0"/>
<line number="429" hits="0"/>
<line number="430" hits="0"/>
<line number="433" hits="0"/>
<line number="434" hits="0"/>
<line number="435" hits="0"/>
<line number="436" hits="0"/>
<line number="438" hits="0"/>
</lines>
</class>
<class name="fmlint.py" filename="fmlint.py" complexity="0" line-rate="0.766" branch-rate="0">
<methods/>
<lines>
<line number="19" hits="1"/>
<line number="21" hits="1"/>
<line number="22" hits="1"/>
<line number="23" hits="1"/>
<line number="24" hits="1"/>
<line number="25" hits="1"/>
<line number="27" hits="1"/>
<line number="29" hits="1"/>
<line number="37" hits="1"/>
<line number="40" hits="1"/>
<line number="43" hits="1"/>
<line number="46" hits="1"/>
<line number="53" hits="1"/>
<line number="103" hits="1"/>
<line number="115" hits="0"/>
<line number="116" hits="0"/>
<line number="117" hits="0"/>
<line number="118" hits="0"/>
<line number="119" hits="0"/>
<line number="120" hits="0"/>
<line number="121" hits="0"/>
<line number="122" hits="0"/>
<line number="125" hits="1"/>
<line number="134" hits="1"/>
<line number="137" hits="1"/>
<line number="148" hits="1"/>
<line number="151" hits="1"/>
<line number="152" hits="1"/>
<line number="155" hits="1"/>
<line number="156" hits="1"/>
<line number="158" hits="1"/>
<line number="161" hits="1"/>
<line number="180" hits="1"/>
<line number="181" hits="1"/>
<line number="183" hits="1"/>
<line number="184" hits="1"/>
<line number="186" hits="1"/>
<line number="187" hits="1"/>
<line number="190" hits="1"/>
<line number="191" hits="1"/>
<line number="192" hits="1"/>
<line number="193" hits="1"/>
<line number="196" hits="1"/>
<line number="197" hits="1"/>
<line number="198" hits="1"/>
<line number="199" hits="1"/>
<line number="202" hits="1"/>
<line number="203" hits="1"/>
<line number="210" hits="1"/>
<line number="212" hits="1"/>
<line number="213" hits="1"/>
<line number="219" hits="1"/>
<line number="226" hits="1"/>
<line number="238" hits="1"/>
<line number="239" hits="1"/>
<line number="240" hits="1"/>
<line number="242" hits="1"/>
<line number="258" hits="1"/>
<line number="261" hits="1"/>
<line number="262" hits="1"/>
<line number="263" hits="1"/>
<line number="264" hits="1"/>
<line number="265" hits="1"/>
<line number="266" hits="1"/>
<line number="267" hits="1"/>
<line number="270" hits="1"/>
<line number="272" hits="1"/>
<line number="275" hits="1"/>
<line number="278" hits="1"/>
<line number="281" hits="1"/>
<line number="282" hits="1"/>
<line number="283" hits="1"/>
<line number="286" hits="1"/>
<line number="287" hits="1"/>
<line number="290" hits="1"/>
<line number="291" hits="1"/>
<line number="293" hits="1"/>
<line number="302" hits="1"/>
<line number="304" hits="1"/>
<line number="306" hits="1"/>
<line number="307" hits="0"/>
<line number="310" hits="1"/>
<line number="311" hits="0"/>
<line number="314" hits="1"/>
<line number="315" hits="1"/>
<line number="316" hits="1"/>
<line number="318" hits="0"/>
<line number="319" hits="0"/>
<line number="320" hits="0"/>
<line number="321" hits="0"/>
<line number="322" hits="0"/>
<line number="323" hits="0"/>
<line number="325" hits="0"/>
<line number="326" hits="0"/>
<line number="327" hits="1"/>
<line number="328" hits="0"/>
<line number="331" hits="1"/>
<line number="332" hits="1"/>
<line number="333" hits="0"/>
<line number="335" hits="0"/>
<line number="343" hits="0"/>
<line number="344" hits="0"/>
<line number="347" hits="1"/>
<line number="348" hits="1"/>
<line number="349" hits="1"/>
<line number="350" hits="1"/>
<line number="358" hits="1"/>
<line number="359" hits="1"/>
<line number="362" hits="1"/>
<line number="363" hits="1"/>
<line number="371" hits="1"/>
<line number="372" hits="1"/>
<line number="375" hits="1"/>
<line number="378" hits="1"/>
<line number="384" hits="1"/>
<line number="385" hits="1"/>
<line number="386" hits="1"/>
<line number="394" hits="1"/>
<line number="397" hits="1"/>
<line number="399" hits="1"/>
<line number="400" hits="1"/>
<line number="408" hits="1"/>
<line number="410" hits="1"/>
<line number="411" hits="1"/>
<line number="413" hits="1"/>
<line number="414" hits="1"/>
<line number="416" hits="1"/>
<line number="417" hits="1"/>
<line number="418" hits="1"/>
<line number="426" hits="1"/>
<line number="427" hits="1"/>
<line number="428" hits="1"/>
<line number="436" hits="1"/>
<line number="439" hits="1"/>
<line number="444" hits="1"/>
<line number="446" hits="1"/>
<line number="473" hits="1"/>
<line number="476" hits="1"/>
<line number="487" hits="1"/>
<line number="489" hits="1"/>
<line number="491" hits="1"/>
<line number="492" hits="1"/>
<line number="493" hits="1"/>
<line number="495" hits="0"/>
<line number="496" hits="0"/>
<line number="497" hits="0"/>
<line number="498" hits="0"/>
<line number="499" hits="0"/>
<line number="500" hits="0"/>
<line number="502" hits="0"/>
<line number="503" hits="0"/>
<line number="504" hits="1"/>
<line number="505" hits="0"/>
<line number="508" hits="1"/>
<line number="511" hits="1"/>
<line number="512" hits="1"/>
<line number="513" hits="1"/>
<line number="514" hits="1"/>
<line number="516" hits="1"/>
<line number="517" hits="1"/>
<line number="518" hits="1"/>
<line number="523" hits="1"/>
<line number="524" hits="1"/>
<line number="526" hits="1"/>
<line number="527" hits="1"/>
<line number="528" hits="1"/>
<line number="530" hits="1"/>
<line number="532" hits="1"/>
<line number="548" hits="1"/>
<line number="549" hits="1"/>
<line number="550" hits="1"/>
<line number="551" hits="1"/>
<line number="552" hits="1"/>
<line number="556" hits="1"/>
<line number="558" hits="1"/>
<line number="570" hits="1"/>
<line number="572" hits="1"/>
<line number="573" hits="1"/>
<line number="575" hits="1"/>
<line number="578" hits="1"/>
<line number="579" hits="1"/>
<line number="580" hits="1"/>
<line number="583" hits="1"/>
<line number="584" hits="1"/>
<line number="587" hits="1"/>
<line number="588" hits="0"/>
<line number="591" hits="1"/>
<line number="592" hits="0"/>
<line number="594" hits="1"/>
<line number="597" hits="1"/>
<line number="598" hits="1"/>
<line number="599" hits="1"/>
<line number="600" hits="1"/>
<line number="601" hits="1"/>
<line number="602" hits="1"/>
<line number="603" hits="1"/>
<line number="605" hits="1"/>
<line number="607" hits="1"/>
<line number="617" hits="1"/>
<line number="618" hits="1"/>
<line number="619" hits="1"/>
<line number="620" hits="1"/>
<line number="621" hits="1"/>
<line number="622" hits="0"/>
<line number="623" hits="1"/>
<line number="624" hits="1"/>
<line number="625" hits="1"/>
<line number="627" hits="1"/>
<line number="637" hits="1"/>
<line number="638" hits="1"/>
<line number="639" hits="1"/>
<line number="641" hits="1"/>
<line number="642" hits="1"/>
<line number="643" hits="1"/>
<line number="645" hits="1"/>
<line number="646" hits="1"/>
<line number="647" hits="1"/>
<line number="648" hits="1"/>
<line number="649" hits="1"/>
<line number="651" hits="1"/>
<line number="652" hits="1"/>
<line number="653" hits="0"/>
<line number="655" hits="1"/>
<line number="656" hits="1"/>
<line number="658" hits="1"/>
<line number="660" hits="1"/>
<line number="671" hits="1"/>
<line number="673" hits="1"/>
<line number="674" hits="1"/>
<line number="675" hits="1"/>
<line number="676" hits="1"/>
<line number="679" hits="1"/>
<line number="680" hits="1"/>
<line number="681" hits="1"/>
<line number="682" hits="1"/>
<line number="683" hits="1"/>
<line number="684" hits="1"/>
<line number="685" hits="1"/>
<line number="688" hits="1"/>
<line number="689" hits="1"/>
<line number="690" hits="1"/>
<line number="691" hits="1"/>
<line number="692" hits="1"/>
<line number="693" hits="1"/>
<line number="695" hits="1"/>
<line number="696" hits="1"/>
<line number="699" hits="1"/>
<line number="700" hits="1"/>
<line number="701" hits="1"/>
<line number="702" hits="1"/>
<line number="703" hits="1"/>
<line number="705" hits="1"/>
<line number="707" hits="1"/>
<line number="725" hits="0"/>
<line number="726" hits="0"/>
<line number="727" hits="0"/>
<line number="728" hits="0"/>
<line number="731" hits="0"/>
<line number="732" hits="0"/>
<line number="733" hits="0"/>
<line number="734" hits="0"/>
<line number="735" hits="0"/>
<line number="741" hits="0"/>
<line number="742" hits="0"/>
<line number="743" hits="0"/>
<line number="744" hits="0"/>
<line number="745" hits="0"/>
<line number="746" hits="0"/>
<line number="747" hits="0"/>
<line number="750" hits="0"/>
<line number="751" hits="0"/>
<line number="752" hits="0"/>
<line number="753" hits="0"/>
<line number="754" hits="0"/>
<line number="755" hits="0"/>
<line number="758" hits="0"/>
<line number="759" hits="0"/>
<line number="760" hits="0"/>
<line number="761" hits="0"/>
<line number="762" hits="0"/>
<line number="763" hits="0"/>
<line number="765" hits="0"/>
<line number="766" hits="0"/>
<line number="769" hits="0"/>
<line number="770" hits="0"/>
<line number="771" hits="0"/>
<line number="772" hits="0"/>
<line number="774" hits="0"/>
<line number="781" hits="1"/>
<line number="783" hits="1"/>
<line number="784" hits="1"/>
<line number="786" hits="1"/>
<line number="788" hits="1"/>
<line number="790" hits="1"/>
<line number="793" hits="1"/>
<line number="796" hits="1"/>
<line number="800" hits="1"/>
<line number="807" hits="1"/>
<line number="809" hits="1"/>
<line number="811" hits="1"/>
<line number="812" hits="1"/>
<line number="814" hits="1"/>
<line number="815" hits="1"/>
<line number="816" hits="1"/>
<line number="818" hits="1"/>
<line number="819" hits="1"/>
<line number="820" hits="1"/>
<line number="822" hits="1"/>
<line number="823" hits="1"/>
<line number="824" hits="1"/>
<line number="826" hits="0"/>
<line number="827" hits="0"/>
</lines>
</class>
<class name="pycustomlint.py" filename="pycustomlint.py" complexity="0" line-rate="0.9712" branch-rate="0">
<methods/>
<lines>
<line number="19" hits="1"/>
<line number="21" hits="1"/>
<line number="22" hits="1"/>
<line number="23" hits="1"/>
<line number="24" hits="1"/>
<line number="25" hits="1"/>
<line number="27" hits="1"/>
<line number="29" hits="1"/>
<line number="32" hits="1"/>
<line number="40" hits="1"/>
<line number="60" hits="1"/>
<line number="63" hits="1"/>
<line number="69" hits="1"/>
<line number="70" hits="1"/>
<line number="72" hits="1"/>
<line number="74" hits="1"/>
<line number="75" hits="1"/>
<line number="76" hits="1"/>
<line number="78" hits="1"/>
<line number="80" hits="1"/>
<line number="81" hits="1"/>
<line number="82" hits="1"/>
<line number="84" hits="1"/>
<line number="87" hits="1"/>
<line number="88" hits="1"/>
<line number="89" hits="1"/>
<line number="90" hits="1"/>
<line number="91" hits="1"/>
<line number="92" hits="1"/>
<line number="93" hits="1"/>
<line number="94" hits="1"/>
<line number="95" hits="1"/>
<line number="97" hits="1"/>
<line number="105" hits="1"/>
<line number="107" hits="1"/>
<line number="109" hits="1"/>
<line number="116" hits="1"/>
<line number="122" hits="1"/>
<line number="123" hits="1"/>
<line number="124" hits="1"/>
<line number="126" hits="1"/>
<line number="135" hits="1"/>
<line number="137" hits="1"/>
<line number="139" hits="1"/>
<line number="140" hits="1"/>
<line number="143" hits="1"/>
<line number="144" hits="0"/>
<line number="147" hits="1"/>
<line number="148" hits="1"/>
<line number="151" hits="1"/>
<line number="152" hits="1"/>
<line number="153" hits="1"/>
<line number="155" hits="1"/>
<line number="157" hits="1"/>
<line number="158" hits="1"/>
<line number="159" hits="1"/>
<line number="161" hits="1"/>
<line number="163" hits="1"/>
<line number="177" hits="1"/>
<line number="178" hits="1"/>
<line number="179" hits="1"/>
<line number="181" hits="1"/>
<line number="182" hits="1"/>
<line number="185" hits="1"/>
<line number="188" hits="1"/>
<line number="189" hits="1"/>
<line number="190" hits="1"/>
<line number="192" hits="1"/>
<line number="194" hits="1"/>
<line number="196" hits="1"/>
<line number="203" hits="1"/>
<line number="205" hits="1"/>
<line number="207" hits="1"/>
<line number="209" hits="1"/>
<line number="210" hits="1"/>
<line number="212" hits="1"/>
<line number="215" hits="1"/>
<line number="218" hits="1"/>
<line number="222" hits="1"/>
<line number="228" hits="1"/>
<line number="234" hits="1"/>
<line number="236" hits="1"/>
<line number="238" hits="1"/>
<line number="239" hits="1"/>
<line number="240" hits="1"/>
<line number="242" hits="1"/>
<line number="243" hits="1"/>
<line number="245" hits="1"/>
<line number="247" hits="1"/>
<line number="248" hits="1"/>
<line number="249" hits="1"/>
<line number="252" hits="1"/>
<line number="253" hits="1"/>
<line number="255" hits="1"/>
<line number="257" hits="1"/>
<line number="258" hits="1"/>
<line number="260" hits="1"/>
<line number="262" hits="1"/>
<line number="263" hits="1"/>
<line number="264" hits="1"/>
<line number="266" hits="1"/>
<line number="267" hits="1"/>
<line number="269" hits="0"/>
<line number="270" hits="0"/>
</lines>
</class>
</classes>
</package>
</packages>
</coverage>
+28 -20
View File
@@ -1,29 +1,32 @@
[build-system] [build-system]
requires = ["setuptools>=42"] requires = ["setuptools>=61"]
build-backend = "setuptools.build_meta" build-backend = "setuptools.build_meta"
[project] [project]
name = "fiona-assistant" name = "fiona-assistant"
version = "0.36.0" version = "1.19.0"
description = "Tools for managing the LLM wiki." description = "Tools for managing the LLM wiki."
authors = [{name = "nvaldeziii"}] authors = [{name = "nvaldeziii"}]
readme = "README.md" readme = "README.md"
license = "GPL-3.0-only" license = "GPL-3.0-only"
requires-python = ">=3.8" requires-python = ">=3.11"
classifiers = [ classifiers = [
"Programming Language :: Python :: 3", "Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
"Operating System :: OS Independent", "Operating System :: OS Independent",
] ]
# Core dependencies (required for runtime) # Core dependencies (required for runtime)
# Note: html5validator requires Java 8+ to be installed on the system
dependencies = [ dependencies = [
"pymarkdownlnt>=0.9.0", "pymarkdownlnt>=0.9.0",
"ruff>=0.16.8", "ruff>=0.16.8",
"pyyaml>=6.0.3", "pyyaml>=6.0.3",
"pylint",
"GitPython>=3.1.0", "GitPython>=3.1.0",
"html5validator>=0.4.0", "beautifulsoup4>=4.12.0",
"markdownify>=1.1.0",
"pypdf>=5.0.0",
] ]
# Optional dependencies groups # Optional dependencies groups
@@ -32,6 +35,7 @@ dev = [
"pytest>=7.0.0", "pytest>=7.0.0",
"pytest-cov>=4.0.0", "pytest-cov>=4.0.0",
"pytest-mock", "pytest-mock",
"pylint",
"build", "build",
"twine", "twine",
] ]
@@ -41,23 +45,23 @@ include-package-data = true
[tool.setuptools.packages.find] [tool.setuptools.packages.find]
where = ["src"] where = ["src"]
# llm-wiki is a hyphenated, non-importable data-only "package" that ships include = ["fiona*"]
# template files (wiki/, ingest/, AGENTS.md, etc.) as package data.
# The hyphen makes it unimportable as a Python module (import llm-wiki
# is a SyntaxError), but setuptools allows it as a distribution package name.
# The __init__.py exists only to satisfy setuptools' package detection.
include = ["fiona", "librarian", "mdoctor", "python_ruff", "customlinter", "sanity_runner", "llm-wiki", "wiki", "gittool", "html_js_lint"]
[tool.setuptools.package-data] [tool.setuptools.package-data]
# Include all files under llm-wiki/ as package data (not Python code) fiona = [
"llm-wiki" = ["*", "**/*"] "data/llm_wiki/**/*",
"data/default.gitignore",
"wiki/templates/*",
"gittool/templates/*",
]
[tool.pytest.ini_options] [tool.pytest.ini_options]
testpaths = ["src/tests"] testpaths = ["tests"]
pythonpath = ["src"]
[tool.coverage.run] [tool.coverage.run]
source = ["src"] source = ["src/fiona"]
omit = ["src/tests/*", "*/tests/*", "*/test_*.py"] omit = ["*/tests/*", "*/test_*.py"]
[tool.coverage.report] [tool.coverage.report]
exclude_lines = [ exclude_lines = [
@@ -73,6 +77,10 @@ exclude_lines = [
# Define console scripts entry points # Define console scripts entry points
[project.scripts] [project.scripts]
fiona-assistant = "fiona.__main__:main" fiona-assistant = "fiona.__main__:main"
librarian = "librarian.librarian:main" librarian = "fiona.librarian.librarian:main"
mdoctor = "mdoctor.mdoctor:main" mdoctor = "fiona.mdoctor.mdoctor:main"
html-js-lint = "html_js_lint.html_js_lint:main"
[tool.ruff]
target-version = "py311"
line-length = 100
src = ["src"]
+83
View File
@@ -0,0 +1,83 @@
"""Pip arguments for bootstrap install modes.
Generated by AI/LLM: 2026-09-30 - local and no-env install options.
SPEC 7.21. Both ``bootstrap`` and ``bootstrap.ps1`` call this script
and read one pip argument per stdout line.
"""
import argparse
import sys
def pip_install_args(
*,
local: bool,
no_env: bool,
break_system_packages: bool = False,
) -> list[str]:
"""Return the arguments passed to ``python -m pip``.
Args:
local: Install this repository as a normal package. When
false, the install is editable and includes the ``dev``
extra.
no_env: Add ``--user`` so the install does not need a virtual
environment.
break_system_packages: Add ``--break-system-packages``. Bash
sets this. PowerShell does not.
Returns:
The pip argument list, starting with ``install``.
"""
args = ["install"]
if no_env:
args.append("--user")
if break_system_packages:
args.append("--break-system-packages")
if local:
args.append(".")
else:
args.extend(["-e", ".[dev]"])
return args
def main(argv: list[str] | None = None) -> int:
"""Print pip arguments, one per line.
Args:
argv: Flags to parse. ``None`` reads ``sys.argv``.
Returns:
Process exit code. ``0`` after the arguments are written.
"""
parser = argparse.ArgumentParser(
description="Print pip install arguments for bootstrap.",
)
parser.add_argument(
"--local",
action="store_true",
help="install this repository as a normal package",
)
parser.add_argument(
"--no-env",
action="store_true",
help="add --user so no virtual environment is required",
)
parser.add_argument(
"--break-system-packages",
action="store_true",
help="pass --break-system-packages to pip",
)
parsed = parser.parse_args(argv)
words = pip_install_args(
local=parsed.local,
no_env=parsed.no_env,
break_system_packages=parsed.break_system_packages,
)
sys.stdout.write("\n".join(words) + "\n")
return 0
if __name__ == "__main__":
sys.exit(main())
+17
View File
@@ -0,0 +1,17 @@
"""Repo-root launcher for the sanity runner.
Generated by AI/LLM: 2026-09-29 - implementation lives in fiona.run_sanity.
"""
import pathlib
import sys
_SRC = pathlib.Path(__file__).resolve().parent.parent / "src"
if str(_SRC) not in sys.path:
sys.path.insert(0, str(_SRC))
import fiona.run_sanity as runner # noqa: E402
if __name__ == "__main__":
sys.exit(runner.main())
+17
View File
@@ -0,0 +1,17 @@
"""Repo-root launcher for the test runner.
Generated by AI/LLM: 2026-09-29 - implementation lives in fiona.run_tests.
"""
import pathlib
import sys
_SRC = pathlib.Path(__file__).resolve().parent.parent / "src"
if str(_SRC) not in sys.path:
sys.path.insert(0, str(_SRC))
import fiona.run_tests as runner # noqa: E402
if __name__ == "__main__":
sys.exit(runner.main())
View File
+54 -17
View File
@@ -7,7 +7,7 @@ A collection of tools for managing the LLM wiki and related operations.
The Fiona Assistant project provides command-line tools for: The Fiona Assistant project provides command-line tools for:
- **Librarian**: Managing LLM wiki content and operations - **Librarian**: Managing LLM wiki content and operations
- **MDoctor**: [Description to be added] - **MDoctor**: Markdown diagnosis (lint, list, and fix)
## Installation ## Installation
@@ -20,7 +20,9 @@ The Fiona Assistant project provides command-line tools for:
cd fiona-assistant cd fiona-assistant
``` ```
2. Set up the Python environment using the bootstrap script: 2. Set up the Python environment using the bootstrap script.
Linux / macOS:
```bash ```bash
source bootstrap --env fiona_env source bootstrap --env fiona_env
@@ -32,6 +34,21 @@ The Fiona Assistant project provides command-line tools for:
source bootstrap --env fiona_env --python 3.11.0 source bootstrap --env fiona_env --python 3.11.0
``` ```
Windows (PowerShell):
```powershell
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
. .\bootstrap.ps1 --env fiona_env --python 3.11.0
```
Install this checkout as a normal package with `--local`. Skip the
virtual environment with `--no-env` (PowerShell may use `-File`
instead of dot-sourcing):
```powershell
powershell -ExecutionPolicy Bypass -File .\bootstrap.ps1 --local --no-env
```
3. Install the package in development mode: 3. Install the package in development mode:
```bash ```bash
@@ -48,7 +65,7 @@ pip install -e ".[dev]"
### Librarian ### Librarian
See [librarian/README.md](./librarian/README.md) for detailed usage. See [librarian/README.md](./fiona/librarian/README.md) for detailed usage.
```bash ```bash
librarian --help librarian --help
@@ -56,7 +73,7 @@ librarian --help
### MDoctor ### MDoctor
See [mdoctor/README.md](./mdoctor/README.md) for detailed usage. See [mdoctor/README.md](./fiona/mdoctor/README.md) for detailed usage.
```bash ```bash
mdoctor --help mdoctor --help
@@ -65,32 +82,50 @@ mdoctor --help
## Project Structure ## Project Structure
```text ```text
src/ src/fiona/
├── fiona/ # Main package ├── __main__.py # CLI entry point
├── librarian/ # Wiki content management tool ├── librarian/ # Wiki content management
├── mdoctor/ # [Tool description] ├── mdoctor/ # Markdown diagnosis
├── tests/ # Unit tests ├── customlinter/ # Frontmatter, markdown, Python, and JSON linters
└── README.md # This file ├── python_ruff/ # Ruff wrapper
├── gittool/ # Git operations
├── sanity_runner/ # Shared sanity engine
├── wiki/ # Wiki init, update, sanity, and graph
├── data/llm_wiki/ # Packaged wiki template
├── run_tests.py # Test runner
└── run_sanity.py # Sanity runner
tests/ # Unit tests (repository root, not installed)
scripts/
├── run_tests.py
├── run_sanity.py
├── settings_dev.json
├── utils.sh
└── install_packages.sh
``` ```
## Running Tests ## Running Tests
```bash ```bash
# Run all tests # Run all tests
pytest python scripts/run_tests.py
# Or use pytest directly
python -m pytest
# Run with coverage # Run with coverage
pytest --cov=src python -m pytest --cov=src/fiona
``` ```
## Development Tools ## Development Tools
The project includes several development utilities: The project includes several development utilities:
- `bootstrap`: Set up the Python development environment - `bootstrap`: Set up the Python development environment (bash)
- `src/utils.sh`: Shell utility functions - `bootstrap.ps1`: Set up the Python development environment (PowerShell)
- `src/install_packages.sh`: Install local Python packages - `scripts/utils.sh`: Shell utility functions
- `src/run_tests.sh`: Run the test suite - `scripts/install_packages.sh`: Install local Python packages
- `scripts/run_tests.py`: Run the test suite
- `scripts/run_sanity.py`: Run tests and linters
## License ## License
@@ -98,4 +133,6 @@ GPL-3.0-only
## Contributing ## Contributing
[Contribution guidelines to be added] Follow [AGENTS.md](../AGENTS.md): update [SPEC.md](../SPEC.md) first,
add a failing test, then implement. Run `python scripts/run_tests.py`
and `python scripts/run_sanity.py --non-interactive` before finishing.
-1
View File
@@ -1 +0,0 @@
# Fiona tools package
+105 -13
View File
@@ -8,18 +8,21 @@ import argparse
import logging import logging
import sys import sys
from customlinter.fmlint import main as fmlint_main from fiona.customlinter.fmlint import main as fmlint_main
from customlinter.jsonlint import main as jsonlint_main from fiona.customlinter.jsonlint import main as jsonlint_main
from fiona.gittool import diff_report
from fiona.gittool.git_tool import main as git_tool_main
from fiona.librarian.librarian import main as librarian_main
from fiona.librarian.toc_generator import main as toc_generator_main
from fiona.logging_config import setup_logging from fiona.logging_config import setup_logging
from gittool.git_tool import main as git_tool_main from fiona.mdoctor.mdoctor import main as mdoctor_main
from librarian.librarian import main as librarian_main from fiona.python_ruff.python_ruff import main as python_ruff_main
from librarian.toc_generator import main as toc_generator_main from fiona.wiki.graph_generator import wiki_graph_main
from mdoctor.mdoctor import main as mdoctor_main from fiona.wiki.housekeeping import housekeeping_main
from python_ruff.python_ruff import main as python_ruff_main from fiona.wiki.ingest.pipeline import add_ingest_arguments, execute_ingest
from wiki.graph_generator import wiki_graph_main from fiona.wiki.sanity_manager import sanity_main
from wiki.sanity_manager import sanity_main from fiona.wiki.update_manager import check_update_main, update_main
from wiki.update_manager import check_update_main, update_main from fiona.wiki.wiki import wiki_init
from wiki.wiki import wiki_init
def main(): def main():
@@ -128,6 +131,39 @@ def main():
# fiona-assistant git status # fiona-assistant git status
git_subparsers.add_parser("status", help="Show git status") git_subparsers.add_parser("status", help="Show git status")
# fiona-assistant git diff
git_diff_parser = git_subparsers.add_parser(
"diff",
help="Write constellation-themed HTML diff under wiki tmp/",
)
git_diff_parser.add_argument(
"--base",
help="Base revision (default: origin/main or main/master)",
)
git_diff_parser.add_argument("--head", default="HEAD", help="Head revision")
git_diff_parser.add_argument("--title", help="Report H1 / page title")
git_diff_parser.add_argument("--subtitle", help="Extra header line")
git_diff_parser.add_argument(
"-o",
"--output",
help="Output HTML path",
)
git_diff_parser.add_argument(
"--collapsed",
action="store_true",
help="Start with per-file sections collapsed",
)
git_diff_parser.add_argument(
"--staged",
action="store_true",
help="Diff only staged changes (ignores unstaged changes)",
)
git_diff_parser.add_argument(
"--unstaged",
action="store_true",
help="Diff only unstaged changes (ignores staged changes)",
)
# fiona-assistant wiki # fiona-assistant wiki
wiki_parser = subparsers.add_parser("wiki", help="Manage wiki directories") wiki_parser = subparsers.add_parser("wiki", help="Manage wiki directories")
wiki_subparsers = wiki_parser.add_subparsers(dest="wiki_subcommand", help="Wiki subcommands") wiki_subparsers = wiki_parser.add_subparsers(dest="wiki_subcommand", help="Wiki subcommands")
@@ -279,6 +315,31 @@ def main():
help="Disable tracking of external links" help="Disable tracking of external links"
) )
# AI/LLM: 2026-09-30 - flags live here so wiki ingest --help is not rejected.
wiki_ingest_parser = wiki_subparsers.add_parser(
"ingest",
prog="fiona-assistant wiki ingest",
help="Archive one source and write a raw wiki page",
)
add_ingest_arguments(wiki_ingest_parser)
# AI/LLM: fiona-assistant housekeeping (SPEC 4.8)
housekeeping_parser = subparsers.add_parser(
"housekeeping",
help="Run the wiki maintenance routine and report what needs work"
)
housekeeping_parser.add_argument(
"directory",
nargs="?",
default=".",
help="Wiki directory (default: current directory)"
)
housekeeping_parser.add_argument(
"--json",
action="store_true",
help="Output results in JSON format"
)
args = parser.parse_args() args = parser.parse_args()
setup_logging() setup_logging()
logging.getLogger(__name__).debug("fiona-assistant command: %s", args.command) logging.getLogger(__name__).debug("fiona-assistant command: %s", args.command)
@@ -355,13 +416,40 @@ def main():
elif args.git_subcommand == "status": elif args.git_subcommand == "status":
sys.argv = ["git_tool", "status"] sys.argv = ["git_tool", "status"]
git_tool_main() git_tool_main()
elif args.git_subcommand == "diff":
diff_argv: list[str] = []
if args.base:
diff_argv.extend(["--base", args.base])
if args.head != "HEAD":
diff_argv.extend(["--head", args.head])
if args.title:
diff_argv.extend(["--title", args.title])
if args.subtitle:
diff_argv.extend(["--subtitle", args.subtitle])
if args.output is not None:
diff_argv.extend(["--output", args.output])
if args.collapsed:
diff_argv.append("--collapsed")
if args.staged:
diff_argv.append("--staged")
if args.unstaged:
diff_argv.append("--unstaged")
sys.exit(diff_report.main(diff_argv))
else: else:
git_parser.print_help() git_parser.print_help()
sys.exit(1) sys.exit(1)
elif args.command == "wiki": elif args.command == "wiki":
if args.wiki_subcommand == "init": if args.wiki_subcommand == "init":
result = wiki_init(args.directory) try:
print(result) result = wiki_init(args.directory)
print(result)
except FileExistsError as e:
# SPEC 7.1.1.4: format the error message and exit 1
# The FileExistsError message is: "Directory '<path>' already contains llm-wiki files: <names>"
# We want: "Error: Directory '<path>' already contains llm-wiki files: <names>. Run fiona-assistant wiki update to refresh the template."
formatted_error = f"Error: {e}. Run fiona-assistant wiki update to refresh the template."
print(formatted_error, file=sys.stderr)
sys.exit(1)
elif args.wiki_subcommand == "check-update": elif args.wiki_subcommand == "check-update":
sys.exit(check_update_main( sys.exit(check_update_main(
directory=args.directory, directory=args.directory,
@@ -381,6 +469,8 @@ def main():
json_output=args.json, json_output=args.json,
non_interactive=args.non_interactive non_interactive=args.non_interactive
)) ))
elif args.wiki_subcommand == "ingest":
sys.exit(execute_ingest(args, wiki_ingest_parser))
elif args.wiki_subcommand == "graph": elif args.wiki_subcommand == "graph":
sys.exit(wiki_graph_main( sys.exit(wiki_graph_main(
directory=args.directory, directory=args.directory,
@@ -394,6 +484,8 @@ def main():
else: else:
wiki_parser.print_help() wiki_parser.print_help()
sys.exit(1) sys.exit(1)
elif args.command == "housekeeping":
sys.exit(housekeeping_main(directory=args.directory, json_output=args.json))
else: else:
parser.print_help() parser.print_help()
sys.exit(1) sys.exit(1)
@@ -16,7 +16,7 @@ various code and content rules. It includes:
All linters share a common interface and output format for consistency. All linters share a common interface and output format for consistency.
""" """
from customlinter.base import ( from fiona.customlinter.base import (
DEFAULT_SETTINGS_FILE, DEFAULT_SETTINGS_FILE,
READONLY_ZONES, READONLY_ZONES,
BaseLinter, BaseLinter,
@@ -25,9 +25,9 @@ from customlinter.base import (
LinterError, LinterError,
LintViolation, LintViolation,
) )
from customlinter.fmlint import FrontmatterLinter from fiona.customlinter.fmlint import FrontmatterLinter
from customlinter.mdlinter import MarkdownLinter from fiona.customlinter.mdlinter import MarkdownLinter
from customlinter.pycustomlint import PythonCustomLinter from fiona.customlinter.pycustomlint import PythonCustomLinter
__all__ = [ __all__ = [
"DEFAULT_SETTINGS_FILE", "DEFAULT_SETTINGS_FILE",
@@ -15,19 +15,15 @@ import pathlib
from abc import ABC, abstractmethod from abc import ABC, abstractmethod
from typing import Any, ClassVar from typing import Any, ClassVar
logger = logging.getLogger(__name__) from fiona.paths import llm_wiki_template
# ============================================================================= logger = logging.getLogger(__name__)
# Constants
# =============================================================================
# Read-only zone names that should be skipped # Read-only zone names that should be skipped
READONLY_ZONES = ["personal", "raw"] READONLY_ZONES = ["personal", "raw"]
# Default settings file path # Default settings file path
DEFAULT_SETTINGS_FILE = ( DEFAULT_SETTINGS_FILE = llm_wiki_template() / "fiona-settings.json"
pathlib.Path(__file__).resolve().parent.parent / "llm-wiki" / "fiona-settings.json"
)
# Re-export constants for backward compatibility # Re-export constants for backward compatibility
__all__ = [ __all__ = [
@@ -41,6 +37,20 @@ __all__ = [
] ]
def report_path(path: pathlib.Path | str) -> str:
"""Return a forward-slash path for reports.
Generated by AI/LLM: 2026-09-29 - str(Path) uses backslashes on Windows,
which makes report text differ from the POSIX form the checks expect.
Args:
path: Path to display.
Returns:
The path with forward slashes.
"""
return pathlib.Path(path).as_posix()
# ============================================================================= # =============================================================================
# Exceptions # Exceptions
@@ -56,7 +66,7 @@ class LinterError(Exception):
line: int | None = None, line: int | None = None,
): ):
"""Initialize the exception with details. """Initialize the exception with details.
Args: Args:
message: The error message describing the linting failure. message: The error message describing the linting failure.
filepath: The path to the file that failed linting. filepath: The path to the file that failed linting.
@@ -71,14 +81,14 @@ class LinterError(Exception):
# Backward compatibility aliases # Backward compatibility aliases
class CustomLintError(LinterError): class CustomLintError(LinterError):
"""Exception raised when custom Python linting fails. """Exception raised when custom Python linting fails.
This is an alias for LinterError for backward compatibility. This is an alias for LinterError for backward compatibility.
""" """
class FrontmatterLintError(LinterError): class FrontmatterLintError(LinterError):
"""Exception raised when frontmatter validation fails. """Exception raised when frontmatter validation fails.
This is an alias for LinterError for backward compatibility. This is an alias for LinterError for backward compatibility.
""" """
@@ -89,7 +99,7 @@ class FrontmatterLintError(LinterError):
class LintViolation: class LintViolation:
"""Represents a single linting violation. """Represents a single linting violation.
This class provides a standardized format for representing violations This class provides a standardized format for representing violations
across all linter types. across all linter types.
""" """
@@ -105,7 +115,7 @@ class LintViolation:
severity: str = "error", severity: str = "error",
): ):
"""Initialize a violation. """Initialize a violation.
Args: Args:
filepath: Path to the file containing the violation. filepath: Path to the file containing the violation.
line: Line number of the violation. line: Line number of the violation.
@@ -126,7 +136,7 @@ class LintViolation:
def to_dict(self) -> dict: def to_dict(self) -> dict:
"""Convert violation to dictionary for reporting.""" """Convert violation to dictionary for reporting."""
return { return {
"file": str(self.filepath), "file": report_path(self.filepath),
"line": self.line, "line": self.line,
"column": self.column, "column": self.column,
"code": self.code, "code": self.code,
@@ -137,7 +147,10 @@ class LintViolation:
def __str__(self) -> str: def __str__(self) -> str:
"""Format violation for display.""" """Format violation for display."""
return f"{self.filepath}:{self.line}:{self.column} {self.code} {self.message}" return (
f"{report_path(self.filepath)}:{self.line}:{self.column} "
f"{self.code} {self.message}"
)
# ============================================================================= # =============================================================================
@@ -146,14 +159,14 @@ class LintViolation:
class BaseLinter(ABC): class BaseLinter(ABC):
"""Abstract base class for all custom linters. """Abstract base class for all custom linters.
This class provides common functionality for all linters including: This class provides common functionality for all linters including:
- Settings loading - Settings loading
- File collection - File collection
- Pattern matching - Pattern matching
- Read-only zone and .gitignore support - Read-only zone and .gitignore support
- Consistent violation formatting - Consistent violation formatting
Subclasses must implement: Subclasses must implement:
- lint_file() method - lint_file() method
- VIOLATION_CODES class attribute - VIOLATION_CODES class attribute
@@ -174,7 +187,7 @@ class BaseLinter(ABC):
def __init__(self, settings_file: pathlib.Path | None = None): def __init__(self, settings_file: pathlib.Path | None = None):
"""Initialize the linter. """Initialize the linter.
Args: Args:
settings_file: Path to the settings file, or None for defaults. settings_file: Path to the settings file, or None for defaults.
""" """
@@ -185,7 +198,7 @@ class BaseLinter(ABC):
"""Load settings from the settings file.""" """Load settings from the settings file."""
if self.settings_file is None: if self.settings_file is None:
self.settings_file = DEFAULT_SETTINGS_FILE self.settings_file = DEFAULT_SETTINGS_FILE
try: try:
with open(self.settings_file, encoding="utf-8") as f: with open(self.settings_file, encoding="utf-8") as f:
settings = json.load(f) settings = json.load(f)
@@ -202,54 +215,54 @@ class BaseLinter(ABC):
@abstractmethod @abstractmethod
def lint_file(self, filepath: pathlib.Path | str) -> list[LintViolation]: def lint_file(self, filepath: pathlib.Path | str) -> list[LintViolation]:
"""Lint a single file for violations. """Lint a single file for violations.
Args: Args:
filepath: Path to the file to lint. filepath: Path to the file to lint.
Returns: Returns:
List of LintViolation objects. List of LintViolation objects.
""" """
def lint_directory(self, directory: str | pathlib.Path) -> list[LintViolation]: def lint_directory(self, directory: str | pathlib.Path) -> list[LintViolation]:
"""Lint all files in a directory tree. """Lint all files in a directory tree.
Args: Args:
directory: Path to the directory to lint. directory: Path to the directory to lint.
Returns: Returns:
List of LintViolation objects from all files. List of LintViolation objects from all files.
""" """
directory = pathlib.Path(directory) directory = pathlib.Path(directory)
violations: list[LintViolation] = [] violations: list[LintViolation] = []
if not directory.exists(): if not directory.exists():
raise LinterError(f"Directory does not exist: {directory}") raise LinterError(f"Directory does not exist: {directory}")
# Collect files # Collect files
files = self._collect_files(directory) files = self._collect_files(directory)
# Lint each file # Lint each file
for file in files: for file in files:
file_violations = self.lint_file(file) file_violations = self.lint_file(file)
violations.extend(file_violations) violations.extend(file_violations)
return violations return violations
def _collect_files(self, directory: pathlib.Path) -> list[pathlib.Path]: def _collect_files(self, directory: pathlib.Path) -> list[pathlib.Path]:
"""Collect all files to lint in a directory tree. """Collect all files to lint in a directory tree.
Args: Args:
directory: Path to the directory to search. directory: Path to the directory to search.
Returns: Returns:
List of Path objects for files to lint. List of Path objects for files to lint.
""" """
files: list[pathlib.Path] = [] files: list[pathlib.Path] = []
# Respect .gitignore # Respect .gitignore
gitignore_path = directory / ".gitignore" gitignore_path = directory / ".gitignore"
ignore_patterns: list[str] = [] ignore_patterns: list[str] = []
if gitignore_path.exists(): if gitignore_path.exists():
try: try:
with open(gitignore_path, encoding="utf-8") as f: with open(gitignore_path, encoding="utf-8") as f:
@@ -259,34 +272,34 @@ class BaseLinter(ABC):
ignore_patterns.append(line) ignore_patterns.append(line)
except OSError: except OSError:
pass pass
# Walk the directory tree # Walk the directory tree
for root, dirs, filenames in os.walk(directory): for root, dirs, filenames in os.walk(directory):
root_path = pathlib.Path(root) root_path = pathlib.Path(root)
# Skip read-only zones # Skip read-only zones
if any(zone in root_path.parts for zone in READONLY_ZONES): if any(zone in root_path.parts for zone in READONLY_ZONES):
dirs.clear() dirs.clear()
continue continue
# Skip based on .gitignore patterns # Skip based on .gitignore patterns
dirs[:] = [d for d in dirs if not self._matches_pattern(d, ignore_patterns)] dirs[:] = [d for d in dirs if not self._matches_pattern(d, ignore_patterns)]
for filename in filenames: for filename in filenames:
if any(filename.endswith(ext) for ext in self.file_extensions) and not self._matches_pattern( if any(filename.endswith(ext) for ext in self.file_extensions) and not self._matches_pattern(
filename, ignore_patterns filename, ignore_patterns
): ):
files.append(root_path / filename) files.append(root_path / filename)
return files return files
def _matches_pattern(self, name: str, patterns: list[str]) -> bool: def _matches_pattern(self, name: str, patterns: list[str]) -> bool:
"""Check if a name matches any of the ignore patterns. """Check if a name matches any of the ignore patterns.
Args: Args:
name: The name to check. name: The name to check.
patterns: List of patterns to match against. patterns: List of patterns to match against.
Returns: Returns:
True if the name matches any pattern. True if the name matches any pattern.
""" """
@@ -300,21 +313,21 @@ class BaseLinter(ABC):
def _should_skip(self, filepath: pathlib.Path | str) -> bool: def _should_skip(self, filepath: pathlib.Path | str) -> bool:
"""Check if a file should be skipped based on settings. """Check if a file should be skipped based on settings.
Args: Args:
filepath: Path to the file (as Path or string). filepath: Path to the file (as Path or string).
Returns: Returns:
True if the file should be skipped. True if the file should be skipped.
""" """
# Convert to Path if it's a string # Convert to Path if it's a string
if isinstance(filepath, str): if isinstance(filepath, str):
filepath = pathlib.Path(filepath) filepath = pathlib.Path(filepath)
# Check read-only zones # Check read-only zones
if any(zone in filepath.parts for zone in READONLY_ZONES): if any(zone in filepath.parts for zone in READONLY_ZONES):
return True return True
# Check ignore patterns from settings # Check ignore patterns from settings
ignore_patterns = self._settings.get("ignore_patterns", []) ignore_patterns = self._settings.get("ignore_patterns", [])
if ignore_patterns: if ignore_patterns:
@@ -322,16 +335,16 @@ class BaseLinter(ABC):
for pattern in ignore_patterns: for pattern in ignore_patterns:
if fnmatch.fnmatch(file_str, pattern): if fnmatch.fnmatch(file_str, pattern):
return True return True
return False return False
def get_violation_message(self, violation_code: str, **kwargs) -> str: def get_violation_message(self, violation_code: str, **kwargs) -> str:
"""Get formatted violation message from VIOLATION_CODES dictionary. """Get formatted violation message from VIOLATION_CODES dictionary.
Args: Args:
violation_code: The violation code (e.g., 'PYC001', 'FML001'). violation_code: The violation code (e.g., 'PYC001', 'FML001').
**kwargs: Keyword arguments to format the message template. **kwargs: Keyword arguments to format the message template.
Returns: Returns:
Formatted violation message string. Formatted violation message string.
""" """
@@ -347,20 +360,20 @@ class BaseLinter(ABC):
def format_violations(self, violations: list[LintViolation]) -> str: def format_violations(self, violations: list[LintViolation]) -> str:
"""Format violations for display. """Format violations for display.
Args: Args:
violations: List of LintViolation objects. violations: List of LintViolation objects.
Returns: Returns:
Formatted string with all violations. Formatted string with all violations.
""" """
if not violations: if not violations:
return "" return ""
lines = [] lines = []
for violation in sorted(violations, key=lambda v: (str(v.filepath), v.line, v.column)): for violation in sorted(violations, key=lambda v: (str(v.filepath), v.line, v.column)):
lines.append(str(violation)) lines.append(str(violation))
return "\n".join(lines) return "\n".join(lines)
def compress_violations( def compress_violations(
@@ -397,18 +410,18 @@ class BaseLinter(ABC):
"total_violations": 0, "total_violations": 0,
}, },
} }
if target is not None: if target is not None:
compressed_violations["meta"]["target"] = str(target_path) compressed_violations["meta"]["target"] = report_path(target_path)
for violation in violations: for violation in violations:
if target_path is not None: if target_path is not None:
try: try:
file_key = str(violation.filepath.relative_to(target_path)) file_key = report_path(violation.filepath.relative_to(target_path))
except ValueError: except ValueError:
file_key = str(violation.filepath) file_key = report_path(violation.filepath)
else: else:
file_key = str(violation.filepath) file_key = report_path(violation.filepath)
if file_key not in compressed_violations["files"]: if file_key not in compressed_violations["files"]:
compressed_violations["files"][file_key] = [] compressed_violations["files"][file_key] = []
@@ -479,10 +492,10 @@ class BaseLinter(ABC):
def format_violations_json(self, violations: list[LintViolation]) -> str: def format_violations_json(self, violations: list[LintViolation]) -> str:
"""Format violations as JSON. """Format violations as JSON.
Args: Args:
violations: List of LintViolation objects. violations: List of LintViolation objects.
Returns: Returns:
JSON string with all violations. JSON string with all violations.
""" """
@@ -495,12 +508,12 @@ class BaseLinter(ABC):
linter_name: str, linter_name: str,
) -> str: ) -> str:
"""Format violations in a comprehensive AI-friendly report. """Format violations in a comprehensive AI-friendly report.
Args: Args:
violations: List of LintViolation objects. violations: List of LintViolation objects.
target: The target directory or file that was linted. target: The target directory or file that was linted.
linter_name: Name of the linter (for report header). linter_name: Name of the linter (for report header).
Returns: Returns:
Formatted report string. Formatted report string.
""" """
@@ -508,7 +521,7 @@ class BaseLinter(ABC):
lines.append(f"{linter_name} Violations") lines.append(f"{linter_name} Violations")
lines.append(f"TARGET: {target}") lines.append(f"TARGET: {target}")
lines.append("") lines.append("")
# Collect all unique violation codes with their descriptions # Collect all unique violation codes with their descriptions
descriptions: dict[str, dict] = {} descriptions: dict[str, dict] = {}
for violation in violations: for violation in violations:
@@ -518,7 +531,7 @@ class BaseLinter(ABC):
"rule_name": code_info.get("rule_name", violation.code), "rule_name": code_info.get("rule_name", violation.code),
"description": code_info.get("description", ""), "description": code_info.get("description", ""),
} }
# Add definitions # Add definitions
if descriptions: if descriptions:
lines.append("- DEFINITIONS:") lines.append("- DEFINITIONS:")
@@ -527,16 +540,16 @@ class BaseLinter(ABC):
f" - {code}: {info['rule_name']}, {info['description']}" f" - {code}: {info['rule_name']}, {info['description']}"
) )
lines.append("") lines.append("")
# Add violations by file # Add violations by file
if violations: if violations:
files_violations: dict[str, list[LintViolation]] = {} files_violations: dict[str, list[LintViolation]] = {}
for violation in violations: for violation in violations:
file_key = str(violation.filepath) file_key = report_path(violation.filepath)
if file_key not in files_violations: if file_key not in files_violations:
files_violations[file_key] = [] files_violations[file_key] = []
files_violations[file_key].append(violation) files_violations[file_key].append(violation)
for file_path, file_viols in sorted(files_violations.items()): for file_path, file_viols in sorted(files_violations.items()):
lines.append(f"- FILE: {file_path}") lines.append(f"- FILE: {file_path}")
for violation in sorted(file_viols, key=lambda v: (v.line, v.column)): for violation in sorted(file_viols, key=lambda v: (v.line, v.column)):
@@ -545,11 +558,11 @@ class BaseLinter(ABC):
else: else:
lines.append("- No violations found.") lines.append("- No violations found.")
lines.append("") lines.append("")
# Add summary # Add summary
lines.append("- SUMMARY:") lines.append("- SUMMARY:")
lines.append(f" - Total files with violations: {len(files_violations) if violations else 0}") lines.append(f" - Total files with violations: {len(files_violations) if violations else 0}")
lines.append(f" - Total violations: {len(violations)}") lines.append(f" - Total violations: {len(violations)}")
lines.append("") lines.append("")
return "\n".join(lines) return "\n".join(lines)
@@ -26,13 +26,14 @@ import pathlib
import re import re
import sys import sys
from customlinter.base import ( from fiona.customlinter.base import (
BaseLinter, BaseLinter,
FrontmatterLintError, FrontmatterLintError,
LintViolation, LintViolation,
report_path,
) )
from fiona.librarian.md_frontmatter import FrontmatterError, parse_frontmatter
from fiona.logging_config import setup_logging from fiona.logging_config import setup_logging
from librarian.md_frontmatter import FrontmatterError, parse_frontmatter
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -695,7 +696,7 @@ class FrontmatterLinter(BaseLinter):
# Group violations by file # Group violations by file
files_violations: dict[str, list[LintViolation]] = {} files_violations: dict[str, list[LintViolation]] = {}
for violation in violations: for violation in violations:
file_key = str(violation.filepath) file_key = report_path(violation.filepath)
if file_key not in files_violations: if file_key not in files_violations:
files_violations[file_key] = [] files_violations[file_key] = []
files_violations[file_key].append(violation) files_violations[file_key].append(violation)
@@ -9,7 +9,7 @@ Includes support for:
- Key ordering (optional) - Key ordering (optional)
- Auto-fixing certain violations - Auto-fixing certain violations
Follows the BaseLinter pattern from customlinter.base. Follows the BaseLinter pattern from fiona.customlinter.base.
""" """
from __future__ import annotations from __future__ import annotations
@@ -21,7 +21,7 @@ import re
import sys import sys
from typing import Any, ClassVar from typing import Any, ClassVar
from customlinter.base import BaseLinter, LinterError, LintViolation from fiona.customlinter.base import BaseLinter, LinterError, LintViolation
from fiona.logging_config import setup_logging from fiona.logging_config import setup_logging
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -3,7 +3,7 @@
This module provides a custom markdown linter for llm-wiki content, This module provides a custom markdown linter for llm-wiki content,
built on the customlinter framework (BaseLinter). It runs a registry of built on the customlinter framework (BaseLinter). It runs a registry of
per-file checks over a wiki's markdown files and reports violations in per-file checks over a wiki's markdown files and reports violations in
the shared AI-friendly format (SPEC 4.9.3). the shared AI-friendly format (SPEC 6.1).
The first check is the dead-link check (moved from The first check is the dead-link check (moved from
`wiki.sanity_manager.find_dead_links`): `wiki.sanity_manager.find_dead_links`):
@@ -27,7 +27,7 @@ import pathlib
import re import re
from typing import Any, ClassVar from typing import Any, ClassVar
from customlinter.base import BaseLinter, LinterError, LintViolation from fiona.customlinter.base import BaseLinter, LinterError, LintViolation
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -67,6 +67,14 @@ VIOLATION_CODES = {
}, },
} }
__all__ = [
"FILE_EXTENSIONS",
"SETTINGS_KEY_PATH",
"VIOLATION_CODES",
"MarkdownLinter",
"find_dead_links",
]
def _is_ignorable_target(target: str) -> bool: def _is_ignorable_target(target: str) -> bool:
"""Check whether a markdown link target is out of scope.""" """Check whether a markdown link target is out of scope."""
@@ -238,7 +246,7 @@ class MarkdownLinter(BaseLinter):
# ============================================================================= # =============================================================================
# Module-level Dead Link Helper (moved from wiki.sanity_manager) # Module-level Dead Link Helper (moved from fiona.wiki.sanity_manager)
# ============================================================================= # =============================================================================
def find_dead_links(wiki_root: pathlib.Path) -> list[LintViolation]: def find_dead_links(wiki_root: pathlib.Path) -> list[LintViolation]:
@@ -262,16 +270,3 @@ def find_dead_links(wiki_root: pathlib.Path) -> list[LintViolation]:
content = path.read_text(encoding="utf-8", errors="replace") content = path.read_text(encoding="utf-8", errors="replace")
violations.extend(linter._check_dead_links(path, content, md_stems)) violations.extend(linter._check_dead_links(path, content, md_stems))
return violations return violations
# =============================================================================
# Public API
# =============================================================================
__all__ = [
"FILE_EXTENSIONS",
"SETTINGS_KEY_PATH",
"VIOLATION_CODES",
"MarkdownLinter",
"find_dead_links",
]
@@ -11,6 +11,7 @@ The module provides functionality to:
Currently implemented rules: Currently implemented rules:
- Inline imports (imports inside functions/blocks instead of module level) - Inline imports (imports inside functions/blocks instead of module level)
- Late declarations (module-level assignments after the first function or class)
The framework is designed to be modular, allowing additional rules to be The framework is designed to be modular, allowing additional rules to be
added in the future. added in the future.
@@ -25,7 +26,7 @@ import pathlib
import sys import sys
from typing import ClassVar from typing import ClassVar
from customlinter.base import ( from fiona.customlinter.base import (
BaseLinter, BaseLinter,
CustomLintError, CustomLintError,
LintViolation, LintViolation,
@@ -46,7 +47,10 @@ READONLY_ZONES = ["personal", "raw"]
VIOLATION_CODES = { VIOLATION_CODES = {
"PYC001": { "PYC001": {
"rule_name": "Inline Import", "rule_name": "Inline Import",
"description": "Import statement found inside function or block (inline import)", "description": (
"Import statement found inside a function or block (inline import). "
"A module-level if/else before the first function or class is allowed"
),
"message_template": "PYC001 - Inline import at line {line}: {import_stmt}", "message_template": "PYC001 - Inline import at line {line}: {import_stmt}",
"severity": "error" "severity": "error"
}, },
@@ -56,6 +60,17 @@ VIOLATION_CODES = {
"message_template": "PYC002 - Import after code at line {line}: {import_stmt}", "message_template": "PYC002 - Import after code at line {line}: {import_stmt}",
"severity": "warning" "severity": "warning"
}, },
"PYC003": {
"rule_name": "Late Declaration",
"description": (
"Module-level assignment found after the first function or class. "
"Put declarations at the top of the file, before the first function or class"
),
"message_template": (
"PYC003 - Declaration after first function or class at line {line}: {stmt}"
),
"severity": "error",
},
} }
@@ -64,26 +79,78 @@ VIOLATION_CODES = {
# ============================================================================= # =============================================================================
class InlineImportVisitor(ast.NodeVisitor): class InlineImportVisitor(ast.NodeVisitor):
"""AST visitor to detect inline (indented) imports.""" """AST visitor to detect inline (indented) imports.
Modified by AI/LLM: 2026-09-29 - allow imports in a module-level
if/else that appears before the first function or class (SPEC 6.4.2).
"""
def __init__(self, filename: str): def __init__(self, filename: str):
"""Initialize the visitor. """Initialize the visitor.
Args: Args:
filename: The name of the file being analyzed. filename: The name of the file being analyzed.
""" """
self.filename = filename self.filename = filename
self.violations: list[LintViolation] = [] self.violations: list[LintViolation] = []
self._os_guard_depth = 0
self._guard_if_ids: set[int] = set()
def visit_Module(self, node: ast.Module) -> None:
"""Mark module-level if statements that sit before any function or class."""
first_boundary = None
for statement in node.body:
if isinstance(statement, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
first_boundary = statement.lineno
break
for statement in node.body:
if first_boundary is not None and statement.lineno >= first_boundary:
break
if isinstance(statement, ast.If):
self._guard_if_ids.add(id(statement))
self.generic_visit(node)
def visit_If(self, node: ast.If) -> None:
"""Keep the OS-guard depth raised while visiting an allowed if."""
guarded = id(node) in self._guard_if_ids
if guarded:
self._os_guard_depth += 1
self.generic_visit(node)
if guarded:
self._os_guard_depth -= 1
def visit_FunctionDef(self, node: ast.FunctionDef) -> None:
"""Imports inside a function stay inline, even under an OS if."""
self._visit_blocked_scope(node)
def visit_AsyncFunctionDef(self, node: ast.AsyncFunctionDef) -> None:
"""Imports inside an async function stay inline."""
self._visit_blocked_scope(node)
def visit_ClassDef(self, node: ast.ClassDef) -> None:
"""Imports inside a class stay inline."""
self._visit_blocked_scope(node)
def _visit_blocked_scope(self, node: ast.AST) -> None:
"""Visit a function or class with the OS-guard exception off.
Modified by AI/LLM: 2026-09-29 - the SPEC 6.4.2 exception does
not cover imports nested in a function or class.
"""
saved = self._os_guard_depth
self._os_guard_depth = 0
self.generic_visit(node)
self._os_guard_depth = saved
def visit_Import(self, node: ast.Import) -> None: def visit_Import(self, node: ast.Import) -> None:
"""Visit import statement - check if indented.""" """Visit import statement - check if indented."""
if node.col_offset > 0: # Indented = inline if node.col_offset > 0 and self._os_guard_depth == 0:
self._add_inline_import_violation(node) self._add_inline_import_violation(node)
self.generic_visit(node) self.generic_visit(node)
def visit_ImportFrom(self, node: ast.ImportFrom) -> None: def visit_ImportFrom(self, node: ast.ImportFrom) -> None:
"""Visit import from statement - check if indented.""" """Visit import from statement - check if indented."""
if node.col_offset > 0: # Indented = inline if node.col_offset > 0 and self._os_guard_depth == 0:
self._add_inline_import_violation(node) self._add_inline_import_violation(node)
self.generic_visit(node) self.generic_visit(node)
@@ -99,7 +166,7 @@ class InlineImportVisitor(ast.NodeVisitor):
import_stmt = line.strip() import_stmt = line.strip()
except (OSError, IndexError, ValueError): except (OSError, IndexError, ValueError):
pass pass
violation = LintViolation( violation = LintViolation(
filepath=pathlib.Path(self.filename), filepath=pathlib.Path(self.filename),
line=node.lineno, line=node.lineno,
@@ -115,16 +182,110 @@ class InlineImportVisitor(ast.NodeVisitor):
return self.violations return self.violations
class LateDeclarationVisitor(ast.NodeVisitor):
"""AST visitor for module-level assignments after the first function or class.
Modified by AI/LLM: 2026-09-30 - PYC003 late declarations (SPEC 6.4.3).
"""
def __init__(self, filename: str) -> None:
"""Initialize the visitor.
Args:
filename: The name of the file being analyzed.
"""
self.filename = filename
self.violations: list[LintViolation] = []
self._boundary: int | None = None
self._scope_depth = 0
def visit_Module(self, node: ast.Module) -> None:
"""Record the first top-level function or class, then walk the module."""
for statement in node.body:
if isinstance(statement, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
self._boundary = statement.lineno
break
self.generic_visit(node)
def visit_FunctionDef(self, node: ast.FunctionDef) -> None:
"""Skip assignments that belong to a function."""
self._visit_nested_scope(node)
def visit_AsyncFunctionDef(self, node: ast.AsyncFunctionDef) -> None:
"""Skip assignments that belong to an async function."""
self._visit_nested_scope(node)
def visit_ClassDef(self, node: ast.ClassDef) -> None:
"""Skip assignments that belong to a class."""
self._visit_nested_scope(node)
def _visit_nested_scope(self, node: ast.AST) -> None:
"""Visit a function or class without treating its assignments as module-level.
Modified by AI/LLM: 2026-09-30 - class and function bodies are not
declarations in the middle of the module (SPEC 6.4.3).
"""
self._scope_depth += 1
self.generic_visit(node)
self._scope_depth -= 1
def visit_Assign(self, node: ast.Assign) -> None:
"""Flag a module-level assignment that follows the first function or class."""
self._check_declaration(node)
self.generic_visit(node)
def visit_AnnAssign(self, node: ast.AnnAssign) -> None:
"""Flag a module-level annotated assignment after the first function or class."""
self._check_declaration(node)
self.generic_visit(node)
def _check_declaration(self, node: ast.Assign | ast.AnnAssign) -> None:
"""Record PYC003 when this assignment is a late module declaration."""
if self._scope_depth > 0 or self._boundary is None:
return
if node.lineno < self._boundary:
return
statement = self._source_line(node.lineno)
self.violations.append(
LintViolation(
filepath=pathlib.Path(self.filename),
line=node.lineno,
column=node.col_offset,
code="PYC003",
message=(
"PYC003 - Declaration after first function or class "
f"at line {node.lineno}: {statement}"
),
rule_id="late-declaration",
)
)
def _source_line(self, lineno: int) -> str:
"""Return the stripped source line for a violation message."""
try:
with open(self.filename, encoding="utf-8") as handle:
lines = handle.readlines()
if lineno <= len(lines):
return lines[lineno - 1].strip()
except (OSError, IndexError, ValueError):
return "assignment"
return "assignment"
def get_violations(self) -> list[LintViolation]:
"""Get all detected violations."""
return self.violations
# ============================================================================= # =============================================================================
# Python Custom Linter Class # Python Custom Linter Class
# ============================================================================= # =============================================================================
class PythonCustomLinter(BaseLinter): class PythonCustomLinter(BaseLinter):
"""Main linter class for custom Python code rules. """Main linter class for custom Python code rules.
This class extends BaseLinter to provide Python-specific linting. This class extends BaseLinter to provide Python-specific linting.
""" """
VIOLATION_CODES: ClassVar[dict] = VIOLATION_CODES VIOLATION_CODES: ClassVar[dict] = VIOLATION_CODES
file_extensions: ClassVar[list[str]] = [".py"] file_extensions: ClassVar[list[str]] = [".py"]
settings_key_path: ClassVar[list[str]] = ["python", "custom_lint"] settings_key_path: ClassVar[list[str]] = ["python", "custom_lint"]
@@ -132,69 +293,72 @@ class PythonCustomLinter(BaseLinter):
def lint_file(self, filepath: pathlib.Path | str) -> list[LintViolation]: def lint_file(self, filepath: pathlib.Path | str) -> list[LintViolation]:
"""Lint a single Python file for custom rule violations. """Lint a single Python file for custom rule violations.
Args: Args:
filepath: Path to the Python file to lint (as Path or string). filepath: Path to the Python file to lint (as Path or string).
Returns: Returns:
List of LintViolation objects. List of LintViolation objects.
""" """
violations: list[LintViolation] = [] violations: list[LintViolation] = []
try: try:
# Convert to Path if it's a string # Convert to Path if it's a string
if isinstance(filepath, str): if isinstance(filepath, str):
filepath = pathlib.Path(filepath) filepath = pathlib.Path(filepath)
# Check if file should be skipped # Check if file should be skipped
if self._should_skip(filepath): if self._should_skip(filepath):
return violations return violations
# Read and parse the file # Read and parse the file
source = filepath.read_text(encoding="utf-8") source = filepath.read_text(encoding="utf-8")
tree = ast.parse(source, filename=str(filepath)) tree = ast.parse(source, filename=str(filepath))
# Visit the AST to find inline imports # Visit the AST to find inline imports and late declarations
visitor = InlineImportVisitor(str(filepath)) visitor = InlineImportVisitor(str(filepath))
visitor.visit(tree) visitor.visit(tree)
violations = visitor.get_violations() violations = visitor.get_violations()
declarations = LateDeclarationVisitor(str(filepath))
declarations.visit(tree)
violations.extend(declarations.get_violations())
except SyntaxError as e: except SyntaxError as e:
# Syntax errors are not our concern # Syntax errors are not our concern
logger.warning("Syntax error in %s: %s", filepath, e) logger.warning("Syntax error in %s: %s", filepath, e)
except OSError as e: except OSError as e:
logger.warning("Could not read %s: %s", filepath, e) logger.warning("Could not read %s: %s", filepath, e)
return violations return violations
def lint_directory(self, directory: str | pathlib.Path) -> list[LintViolation]: def lint_directory(self, directory: str | pathlib.Path) -> list[LintViolation]:
"""Lint all files in a directory tree. """Lint all files in a directory tree.
This overrides the base class to raise CustomLintError instead of LinterError. This overrides the base class to raise CustomLintError instead of LinterError.
Args: Args:
directory: Path to the directory to lint. directory: Path to the directory to lint.
Returns: Returns:
List of LintViolation objects from all files. List of LintViolation objects from all files.
Raises: Raises:
CustomLintError: If the directory does not exist. CustomLintError: If the directory does not exist.
""" """
directory = pathlib.Path(directory) directory = pathlib.Path(directory)
violations: list[LintViolation] = [] violations: list[LintViolation] = []
if not directory.exists(): if not directory.exists():
raise CustomLintError(f"Directory does not exist: {directory}") raise CustomLintError(f"Directory does not exist: {directory}")
# Collect files # Collect files
files = self._collect_files(directory) files = self._collect_files(directory)
# Lint each file # Lint each file
for file in files: for file in files:
file_violations = self.lint_file(file) file_violations = self.lint_file(file)
violations.extend(file_violations) violations.extend(file_violations)
return violations return violations
def _collect_python_files(self, directory: pathlib.Path) -> list[pathlib.Path]: def _collect_python_files(self, directory: pathlib.Path) -> list[pathlib.Path]:
@@ -208,15 +372,15 @@ class PythonCustomLinter(BaseLinter):
def main(): def main():
"""Command-line interface for the Python custom linter.""" """Command-line interface for the Python custom linter."""
setup_logging() setup_logging()
logger.debug("Python custom linter starting") logger.debug("Python custom linter starting")
parser = argparse.ArgumentParser( parser = argparse.ArgumentParser(
description="Python Custom Linter - Check for custom Python code rule violations." description="Python Custom Linter - Check for custom Python code rule violations."
) )
subparsers = parser.add_subparsers(dest="subcommand", help="Available subcommands") subparsers = parser.add_subparsers(dest="subcommand", help="Available subcommands")
# chklint subcommand # chklint subcommand
chklint_parser = subparsers.add_parser( chklint_parser = subparsers.add_parser(
"chklint", "chklint",
@@ -233,35 +397,35 @@ def main():
action="store_true", action="store_true",
help="Output results in JSON format" help="Output results in JSON format"
) )
args = parser.parse_args() args = parser.parse_args()
if args.subcommand is None: if args.subcommand is None:
# Default to chklint with current directory # Default to chklint with current directory
args.subcommand = "chklint" args.subcommand = "chklint"
args.directory = "." args.directory = "."
args.json = False # Default value when not specified args.json = False # Default value when not specified
if args.subcommand == "chklint": if args.subcommand == "chklint":
linter = PythonCustomLinter() linter = PythonCustomLinter()
target = pathlib.Path(args.directory) target = pathlib.Path(args.directory)
if not target.exists(): if not target.exists():
print(f"Error: '{args.directory}' does not exist.", file=sys.stderr) print(f"Error: '{args.directory}' does not exist.", file=sys.stderr)
sys.exit(1) sys.exit(1)
# Support both files and directories # Support both files and directories
if target.is_file(): if target.is_file():
violations = linter.lint_file(target) violations = linter.lint_file(target)
else: else:
violations = linter.lint_directory(target) violations = linter.lint_directory(target)
if getattr(args, 'json', False): if getattr(args, 'json', False):
output = linter.format_violations_json(violations) output = linter.format_violations_json(violations)
else: else:
output = linter.format_violations(violations) output = linter.format_violations(violations)
if output: if output:
print(output) print(output)
sys.exit(1) sys.exit(1)
+63
View File
@@ -0,0 +1,63 @@
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# Virtual environments
.venv/
.venvs/
venv/
env/
ENV/
# IDE-specific files
.vscode/
.idea/
*.swp
*.swo
# Local configuration
.env
*.local
# Build artifacts
build/
dist/
*.egg-info/
# Logs and databases
*.log
*.sqlite
*.db
# pytest
.pytest_cache/
# Coverage
.coverage
coverage.xml
htmlcov/
# Jupyter Notebook checkpoints
.ipynb_checkpoints/
# macOS
.DS_Store
# Windows
Thumbs.db
# Obsidian per-user vault state
.obsidian/workspace*.json
.obsidian/cache
# Python
*.pyc
*.pyo
*.pyd
*fiona.log
*wiki_graph.html
# Generated review artifacts (git diff HTML, scratch exports)
tmp/
+72
View File
@@ -0,0 +1,72 @@
---
name: llm-wiki-agent-rules
title: LLM Wiki Agent Rules
version: 1.4.0
created: 2025-09-23
updated: 2026-10-14
type: summary
---
# LLM Wiki Agent Rules
This directory is an llm-wiki: a persistent knowledge base of
interlinked markdown. The user brings sources and questions. You
compile each source once into wiki pages, keep the cross-references
current, and answer from those pages. The pattern is
[llm-wiki-mod](./utils/skills/fiona/llm-wiki-mod.md). The wiki's
domain, tags, and directory tree are in [schema](./wiki/schema.md).
This file states only what must be true before any skill is loaded.
Commands, lint codes, and the edit workflow live in the Fiona skill.
- [1. Load the Fiona skill](#1-load-the-fiona-skill)
- [2. What an llm-wiki is](#2-what-an-llm-wiki-is)
- [3. Boundaries](#3-boundaries)
- [4. History](#4-history)
## 1. Load the Fiona skill
If the Fiona skill is not already part of the system prompt, read
[Fiona skill](./utils/skills/fiona/SKILL.md) and follow it for the
rest of the session. It loads llm-wiki-mod.
When that skill is already loaded, do not read it again. This file
does not repeat it.
Then read [MEMORY](./MEMORY.md): what this wiki's owner wants, kept
across sessions.
Update `MEMORY.md` whenever an important memory comes up: a lasting
preference, a correction, a repeated request, or a fact a run
verified. Also update it whenever the user explicitly asks you to
remember something. Section 1 of that file says how.
## 2. What an llm-wiki is
A directory is an llm-wiki when it contains `fiona-settings.json`.
Page types, tags, and the directory tree are defined in
[schema](./wiki/schema.md). Do not invent a second map of the tree.
Install the tool with `python -m pip install fiona-assistant` when
the `fiona-assistant` command is missing. How to use it is the
Fiona skill, not this file.
## 3. Boundaries
These hold even before the skill is loaded:
- `personal/` is read-only. Never create, edit, or delete there.
- `raw/` is the archive. Add a new source there. Never edit, move,
or delete a file already there.
- Synthesized pages belong under `wiki/`.
- `ingest/` is only the drop zone for files waiting to be parsed.
## 4. History
Git is the wiki's history. After each ingest cycle and each
housekeeping pass (lint fixes, index or TOC updates, page moves),
commit with `fiona-assistant git save --description TEXT`.
`git clean`, `git undo`, and `git restore` discard work. Run them
only when the user asks.
+179
View File
@@ -0,0 +1,179 @@
---
name: memory
version: 1.0.1
title: Agent Memory
created: DATE_PLACEHOLDER
updated: DATE_PLACEHOLDER
type: summary
---
# Agent Memory
<!-- toc-start: headings 2..6 -->
- [1. How to use this file](#1-how-to-use-this-file)
- [1.1 When to read](#11-when-to-read)
- [1.2 When to write](#12-when-to-write)
- [1.3 How to write](#13-how-to-write)
- [1.4 How the memory evolves](#14-how-the-memory-evolves)
- [2. Pages and writing](#2-pages-and-writing)
- [2.1 Style and wording](#21-style-and-wording)
- [2.2 Placement and links](#22-placement-and-links)
- [3. Sources and ingest](#3-sources-and-ingest)
- [3.1 Hosts and access](#31-hosts-and-access)
- [3.2 Conversion and refinement](#32-conversion-and-refinement)
- [4. Tools and environment](#4-tools-and-environment)
- [4.1 Commands and flags](#41-commands-and-flags)
- [4.2 Machine and shell](#42-machine-and-shell)
- [5. History and housekeeping](#5-history-and-housekeeping)
- [5.1 Saves and maintenance](#51-saves-and-maintenance)
<!-- toc-end -->
What this wiki's owner wants from you, kept across sessions:
standing preferences, corrections, repeated request patterns, and
facts a run verified about operating this wiki. Knowledge from
sources does not go here; it goes to pages under `wiki/`.
## 1. How to use this file
### 1.1 When to read
- **1.1.1** Read this file once the
[Fiona skill](./utils/skills/fiona/SKILL.md) is loaded, before the
first wiki work of the session. When it is already in context, do
not read it again.
- **1.1.2** Apply every entry that matches the request.
- **1.1.2.1** An entry overrides a default of the Fiona skill or
the [schema](./wiki/schema.md), for this wiki only.
- **1.1.2.2** An entry never overrides the boundaries in section 3
of [AGENTS](./AGENTS.md); when the two conflict, follow the
boundary and ask the user.
- **1.1.2.3** An entry never changes a tool contract: a command, a
flag, or a required frontmatter field.
- **1.1.3** Every heading and list item is numbered as
[sectionize-markdown](./utils/skills/sectionize-markdown/SKILL.md)
describes. Cite an item as `MEMORY.md item 2.1.3`; a new item takes
the next free number in its subsection.
### 1.2 When to write
- **1.2.1** Write an entry when one of these happens:
- **1.2.1.1** The user states a requirement or preference that
outlasts the current request ("always", "never", "from now on",
"in this wiki", "we use").
- **1.2.1.2** The user corrects your output or approach: a page, a
summary, a placement, a link, or a command you ran.
- **1.2.1.3** The same kind of request comes for the second time;
record the pattern and what the user wanted.
- **1.2.1.4** A run verifies a reusable fact about operating this
wiki that the Fiona skill and the schema do not hold, such as a
host that needs a login, a converter that loses a site's tables,
or a flag combination that works.
- **1.2.1.5** The user explicitly asks you to remember something
("remember", "note this", "save to memory").
- **1.2.2** Do not write:
- **1.2.2.1** One-off values that only apply to the current
request.
- **1.2.2.2** Passwords, tokens, cookies, personal data, or
anything the user asks to keep out.
- **1.2.2.3** Knowledge from a source. That is a page under
`wiki/`, written by an ingest.
- **1.2.2.4** Facts that the Fiona skill, the schema, or a wiki
page already states; cite them instead.
- **1.2.2.5** Guesses; only what the user said or a run showed.
### 1.3 How to write
- **1.3.1** Put the entry under the `###` subsection that fits.
- **1.3.1.1** When none fits, add a `###` subsection with the next
free number under the matching `##`, or a new `##` category with
the next free number.
- **1.3.1.2** Never put an entry directly under a `##`; its id
would collide with a later subsection.
- **1.3.2** Use the knowledge-entry shape of sectionize-markdown
section 4: a header line with the source tag and a short title,
one child per fact, and a last child with the ref and the date.
- **1.3.3** Source tags: `user` (the user said it), `run` (a command
or tool run showed it), `source` (a file under `raw/` states it),
joined with `+` when several agree.
- **1.3.4** The ref says where to check the fact: the user's request
and its date, the command and its output line, or a page or raw
path.
- **1.3.5** Write the entry in the same turn, then tell the user
which item was added or changed, for example "Saved to MEMORY.md
item 2.1.1".
- **1.3.6** Bump `version` (patch for a changed entry, minor for a
new entry or subsection) and set `updated` in the frontmatter on
every change.
- **1.3.7** Append `- YYYY-MM-DD - memory: <item id> <title>` to
`wiki/log.md`, and bump that file too.
- **1.3.8** The change lands in the save of the current ingest cycle
or housekeeping pass (Fiona skill section 2.3).
- **1.3.8.1** With no cycle running, save it on its own with
`fiona-assistant git save --description "memory: <item id> <title>"`.
An entry looks like this:
```text
- **2.1.1** [user] "Source summaries open with the decision"
- **2.1.1.1** A source page starts with the decision the source records, then the reasons.
- **2.1.1.2** Applies to meeting notes and design reviews.
- **2.1.1.3** Ref: user request YYYY-MM-DD; added YYYY-MM-DD.
```
### 1.4 How the memory evolves
- **1.4.1** Before adding an entry, look for a near-duplicate in the
same category; extend it instead and keep its id and its earliest
date.
- **1.4.2** A finding that contradicts an entry becomes its next
child, titled `OVERRIDE: ...`, with the new facts and its own ref.
- **1.4.3** An entry that no longer applies is marked
**Superseded YYYY-MM-DD** with the reason; it is not deleted.
- **1.4.4** An entry that held for several uses graduates; mark it
**Promoted YYYY-MM-DD** with the target.
- **1.4.4.1** A convention for every page moves to the
[schema](./wiki/schema.md).
- **1.4.4.2** A reusable procedure moves to a skill under
[utils/skills_local](./utils/skills_local/README.md).
- **1.4.5** Never renumber ids; pages, log lines, and chats may cite
them.
- **1.4.6** This file belongs to this wiki and evolves with it.
- **1.4.6.1** `fiona-assistant wiki update` adds it only when it is
missing. It never compares or replaces it, and never writes a
`MEMORY-update.md`.
- **1.4.6.2** A newer template version of these rules does not
reach an existing wiki; the owner decides what to adopt.
- **1.4.7** Git is the memory's history: `git log -p MEMORY.md` shows
when each entry changed and why.
## 2. Pages and writing
How pages read, where they go, and how they link.
### 2.1 Style and wording
### 2.2 Placement and links
## 3. Sources and ingest
Where sources come from and how they become pages.
### 3.1 Hosts and access
### 3.2 Conversion and refinement
## 4. Tools and environment
How `fiona-assistant` and the machine behave for this wiki.
### 4.1 Commands and flags
### 4.2 Machine and shell
## 5. History and housekeeping
Saves, commits, merges, and maintenance passes.
### 5.1 Saves and maintenance
@@ -0,0 +1,59 @@
{
"version": "1.1.0",
"background": "#f4efe6",
"graph_background": "#fffdf8",
"font_family": "Arial, sans-serif",
"text_color": "#2c2416",
"node_colors": {
"low": "#8f8272",
"mid": "#e0943a",
"high": "#c23b4a"
},
"node_stroke": "none",
"node_radius": {
"min": 5,
"max": 40
},
"edge_colors": {
"low": "#a89884",
"mid": "#b86a1a",
"high": "#8e2436"
},
"edge_width": 1,
"highlight": {
"node_color": "#2aa7a1",
"edge_color": "#0d6e74",
"size_multiplier": 1.3
},
"clicked": {
"node_color": "#0e6b66"
},
"orphan": {
"enabled": false,
"node_color": "#7b5ea7"
},
"buttons": {
"background": "#0f6e78",
"text": "#fffdf8",
"hover": "#0b555c",
"disabled": "#b7aea3"
},
"card": {
"background": "#fffdf8",
"border": "#e0d3c2"
},
"shapes": {
"page": "circle",
"external": "cloud"
},
"subtitle_color": "#2c2416",
"subtitle": {
"min_radius": 0.8,
"padding": 2,
"size": 6,
"on_top": true,
"outline": false,
"outline_color": "#fffdf8",
"outline_width": 3
}
}
@@ -0,0 +1,59 @@
{
"version": "1.1.0",
"background": "#1e2836",
"graph_background": "#283448",
"font_family": "Segoe UI, sans-serif",
"text_color": "#e7eef8",
"node_colors": {
"low": "#7d8da3",
"mid": "#fff4c4",
"high": "#f7d67c"
},
"node_stroke": "none",
"node_radius": {
"min": 4,
"max": 24
},
"edge_colors": {
"low": "#51627a",
"mid": "#8fa3bd",
"high": "#d5e2f2"
},
"edge_width": 1,
"highlight": {
"node_color": "#f3d48a",
"edge_color": "#9ec5ea",
"size_multiplier": 1.45
},
"clicked": {
"node_color": "#ffe3a3"
},
"orphan": {
"enabled": false,
"node_color": "#ef8a7a"
},
"buttons": {
"background": "#3e5674",
"text": "#f4f8fc",
"hover": "#527094",
"disabled": "#2a3648"
},
"card": {
"background": "#31445c",
"border": "#5c7390"
},
"shapes": {
"page": "circle",
"external": "cloud"
},
"subtitle_color": "#e7eef8",
"subtitle": {
"min_radius": 0.8,
"padding": 2,
"size": 6,
"on_top": true,
"outline": false,
"outline_color": "#283448",
"outline_width": 3
}
}
@@ -0,0 +1,113 @@
{
"version": "1.1.0",
"background": "#e6e6e6",
"graph_background": "#f4f4f4",
"font_family": "Arial, sans-serif",
"text_color": "#141414",
"node_colors": {
"low": "#5a5a5a",
"mid": "#2e2e2e",
"high": "#0d0d0d"
},
"node_stroke": "#141414",
"node_radius": {
"min": 5,
"max": 36
},
"edge_colors": {
"low": "#8f8f8f",
"mid": "#3d3d3d",
"high": "#141414"
},
"edge_width": 1,
"highlight": {
"node_color": "#000000",
"edge_color": "#1a1a1a",
"size_multiplier": 1.4
},
"clicked": {
"node_color": "#000000"
},
"orphan": {
"enabled": false,
"node_color": "#bdbdbd"
},
"buttons": {
"background": "#2a2a2a",
"text": "#ffffff",
"hover": "#111111",
"disabled": "#b0b0b0"
},
"card": {
"background": "#ffffff",
"border": "#8a8a8a"
},
"shapes": {
"page": "circle",
"external": "cloud"
},
"subtitle_color": "#141414",
"subtitle": {
"min_radius": 0.8,
"padding": 2,
"size": 6,
"on_top": true,
"outline": false,
"outline_color": "#f4f4f4",
"outline_width": 3
},
"dark": {
"background": "#2c2c2c",
"graph_background": "#3a3a3a",
"font_family": "Arial, sans-serif",
"text_color": "#f5f5f5",
"node_colors": {
"low": "#8d8d8d",
"mid": "#fff4c4",
"high": "#f5b44a"
},
"node_stroke": "none",
"node_radius": {
"min": 5,
"max": 36
},
"edge_colors": {
"low": "#6e6e6e",
"mid": "#c4c4c4",
"high": "#ffffff"
},
"edge_width": 1,
"highlight": {
"node_color": "#ffffff",
"edge_color": "#ffffff",
"size_multiplier": 1.4
},
"clicked": {
"node_color": "#ffffff"
},
"buttons": {
"background": "#e8e8e8",
"text": "#1a1a1a",
"hover": "#ffffff",
"disabled": "#5a5a5a"
},
"card": {
"background": "#454545",
"border": "#9a9a9a"
},
"shapes": {
"page": "circle",
"external": "cloud"
},
"subtitle_color": "#f5f5f5",
"subtitle": {
"min_radius": 0.5,
"padding": 2,
"size": 8,
"on_top": true,
"outline": true,
"outline_color": "#2c2c2c",
"outline_width": 3
}
}
}
@@ -0,0 +1,59 @@
{
"version": "1.1.0",
"background": "#070b14",
"graph_background": "#0b1020",
"font_family": "Segoe UI, sans-serif",
"text_color": "#d6def5",
"node_colors": {
"low": "#8ea2c8",
"mid": "#ffe9a0",
"high": "#ffffff"
},
"node_stroke": "none",
"node_radius": {
"min": 3,
"max": 20
},
"edge_colors": {
"low": "#243056",
"mid": "#7f8cbf",
"high": "#e7ecff"
},
"edge_width": 1,
"highlight": {
"node_color": "#fff6c2",
"edge_color": "#9ecbff",
"size_multiplier": 1.6
},
"clicked": {
"node_color": "#ffd56a"
},
"orphan": {
"enabled": false,
"node_color": "#ff7f6e"
},
"buttons": {
"background": "#1e2a4a",
"text": "#e8eeff",
"hover": "#314572",
"disabled": "#3a4258"
},
"card": {
"background": "#12182b",
"border": "#2a3558"
},
"shapes": {
"page": "circle",
"external": "star"
},
"subtitle_color": "#d5def8",
"subtitle": {
"min_radius": 0.8,
"padding": 2,
"size": 6,
"on_top": true,
"outline": false,
"outline_color": "#0b1020",
"outline_width": 3
}
}
@@ -0,0 +1,59 @@
{
"version": "1.1.0",
"background": "#f7fafc",
"graph_background": "#fcfdfe",
"font_family": "Segoe UI, Calibri, Helvetica Neue, Arial, sans-serif",
"text_color": "#4a5d70",
"node_colors": {
"low": "#e4eef4",
"mid": "#a9c6d8",
"high": "#7eabc4"
},
"node_stroke": "#9bb8cb",
"node_radius": {
"min": 4,
"max": 26
},
"edge_colors": {
"low": "#e7f0f5",
"mid": "#b7d0de",
"high": "#8eb4c9"
},
"edge_width": 1,
"highlight": {
"node_color": "#6aa3c0",
"edge_color": "#7eabc4",
"size_multiplier": 1.15
},
"clicked": {
"node_color": "#5b93b0"
},
"orphan": {
"enabled": false,
"node_color": "#d9a5b3"
},
"buttons": {
"background": "#8eb4c9",
"text": "#ffffff",
"hover": "#7aa6bd",
"disabled": "#dce7ee"
},
"card": {
"background": "#ffffff",
"border": "#e6eef3"
},
"shapes": {
"page": "circle",
"external": "cloud"
},
"subtitle_color": "#4a5d70",
"subtitle": {
"min_radius": 0.8,
"padding": 2,
"size": 6,
"on_top": true,
"outline": false,
"outline_color": "#fcfdfe",
"outline_width": 3
}
}
@@ -1,4 +1,5 @@
{ {
"version": "1.0.1",
"markdown": { "markdown": {
"pymarkdown": { "pymarkdown": {
"extensions": [ "extensions": [
@@ -58,20 +59,32 @@
}, },
"wiki": { "wiki": {
"graph": { "graph": {
"node_colors": { "theme": "dark",
"low": "#808080", "show_external_links": true,
"mid": "#ffa500", "subtitle": {
"high": "#FF0000" "enabled": true
}, },
"node_radius": { "show_legend": true,
"min": 5, "show_arrows": false,
"max": 40 "show_source": true,
}, "show_glow": true,
"hover": { "show_link_card": true,
"color": "#add8e6", "first_fit_ms": 12000,
"edge_color": "#0000ff", "hover_link_tail_chars": 12,
"size_multiplier": 1.3 "simplify_external_links": true,
} "consolidate_external_links": true,
"consolidate_external_links_depth": 3,
"consolidate_external_links_max_char": 8,
"ignore_files": [
"*/table_of_contents.md",
"*/README.md",
"*/AGENTS.md"
]
},
"update": {
"ignore_files": [
"corefiles.json"
]
} }
} }
} }
@@ -0,0 +1,19 @@
---
name: map
version: 1.1.0
title: Script Map
type: summary
created: 2025-09-23
updated: 2026-09-30
---
# Script Map
Catalog of the reusable scripts in this folder. Read it before
writing a new script, and add a row when you add one. The rules for
writing a script are in the
[Fiona skill](../skills/fiona/SKILL.md#write-reusable-scripts).
| Script | Purpose | Usage | Added |
| --- | --- | --- | --- |
| `example.py` | Placeholder that prints a greeting. | `python utils/scripts/example.py` | 2025-09-23 |
@@ -0,0 +1,18 @@
---
name: skills
version: 1.1.1
title: Skills Directory
type: summary
created: 2025-09-23
updated: 2026-10-21
---
# Skills Directory
Skill definitions and configurations.
- [fiona](./fiona/SKILL.md): operate the llm-wiki with `fiona-assistant`.
- [memory](./memory/SKILL.md): give a skill its own `MEMORIES.md`.
- [python-tdd](./python-tdd/SKILL.md): test-driven Python work.
- [sectionize-markdown](./sectionize-markdown/SKILL.md):
number headings and list items so each rule can be cited.
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,144 @@
---
name: llm-wiki-mod
title: LLM Wiki Pattern
version: 1.4.1
updated: 2026-10-06
type: concept
tags: [wiki, schema]
---
# LLM Wiki
A persistent knowledge base of interlinked markdown. The pattern
comes from
[Andrej Karpathy's LLM Wiki](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f).
Page rules are in [schema](../../../wiki/schema.md). Commands and
the edit loop are in the [Fiona skill](./SKILL.md).
- [1. Core idea](#1-core-idea)
- [2. Layers](#2-layers)
- [3. Operations](#3-operations)
- [4. Index and log](#4-index-and-log)
- [5. Page rules](#5-page-rules)
- [6. Why this works](#6-why-this-works)
- [7. Scope](#7-scope)
## 1. Core idea
Sources are read once and compiled into wiki pages. Later questions
use those pages. Cross-references, contradictions, and synthesis
stay in the files. Each new source and each filed answer adds to
that store.
The human curates sources and asks the questions. The agent writes
the pages, updates cross-references, and keeps the bookkeeping.
## 2. Layers
Three layers. The directory tree in
[schema](../../../wiki/schema.md) is the map.
### 2.1. Raw sources
`raw/` is the archive. The agent adds new sources there and never
edits, moves, or deletes one already archived.
`ingest/` is the drop zone for files that still need parsing.
`personal/` holds user notes and is read-only, including the
directory itself.
### 2.2. The wiki
`wiki/` is agent-owned markdown. Synthesized pages go in
`concepts/`, `entities/`, `comparisons/`, and `queries/`. Source
summaries that back those pages go in `sources/`. The agent creates
pages, revises them when a new source arrives, and keeps links
consistent.
### 2.3. The schema
[schema](../../../wiki/schema.md) states file names, frontmatter,
tags, when to create a page, and the directory tree.
[AGENTS.md](../../../AGENTS.md) is the entry point. It loads the
[Fiona skill](./SKILL.md) for commands and the edit loop. When a
convention changes, change the schema. Leave the tree in that one
file.
## 3. Operations
### 3.1. Ingest
Put a new source in `ingest/`. The agent reads it, writes the
summary under `wiki/`, updates [index](../../../wiki/index.md),
updates the entity and concept pages the source touches, and
appends one dated line to [log](../../../wiki/log.md). One source
can touch many pages. Take sources one at a time when the summary
should be checked before the next file.
After the archive step, the source lives under `raw/` and is not
edited again.
An ingest has two phases. A tool fetches the source, archives it
under `raw/`, and converts it into a page marked `status: raw`.
The agent then refines that page, updates the pages it touches,
and marks it `status: refined`. The steps are section 18 of the
[Fiona skill](./SKILL.md).
### 3.2. Query
Ask against the wiki. The agent reads the index, opens the pages
that apply, and answers with citations. File a comparison, an
analysis, or a connection that should outlive the chat under
`wiki/queries/` or `wiki/comparisons/`.
### 3.3. Lint
Check the wiki for contradictions, claims a newer source has
replaced, pages with no inbound links, concepts that need their
own page, missing cross-references, and gaps a source could fill.
The health command is `fiona-assistant wiki sanity`. Style,
frontmatter, and link checks are in the [Fiona skill](./SKILL.md).
## 4. Index and log
[index](../../../wiki/index.md) is the catalog. Each page has a
link and a one-line summary, grouped by section. Update it when
a page is added. Read it first when answering a query.
[log](../../../wiki/log.md) is append-only. Each entry is one
dated bullet:
```text
- YYYY-MM-DD - what happened
```
Bump `version` and `updated` on `log.md` when appending. The log
is the timeline of ingests, queries, and lint passes.
Git holds the before and after of each entry. Every ingest cycle
and every housekeeping pass ends in one commit, so one source or
one cleanup can be reviewed or undone on its own.
## 5. Page rules
File names, frontmatter, tags, links, and when to create or split
a page are in [schema](../../../wiki/schema.md). How to fill the
frontmatter and the quality signals is section 15 of the
[Fiona skill](./SKILL.md). This file does not repeat them.
## 6. Why this works
Cross-references, summaries that still match the sources, and notes
where a new source disagrees with an older claim are the work that
usually gets dropped. The agent does that work on each ingest, so
the wiki stays current as it grows.
The human chooses sources, directs the analysis, and decides what
the pages mean. The agent does the filing.
## 7. Scope
This file states the pattern. The directory tree stays in
[schema](../../../wiki/schema.md). Commands stay in the
[Fiona skill](./SKILL.md). An Obsidian graph, Marp slides,
Dataview, or a local search engine can sit beside the wiki. This
wiki does not require them.
@@ -0,0 +1,173 @@
---
name: skill-memories
version: 1.0.1
title: SKILL_NAME Memories
created: DATE_PLACEHOLDER
updated: DATE_PLACEHOLDER
type: summary
---
# SKILL_NAME Memories
<!-- toc-start: headings 2..6 -->
- [1. How to use this file](#1-how-to-use-this-file)
- [1.1 When to read](#11-when-to-read)
- [1.2 When to write](#12-when-to-write)
- [1.3 How to write](#13-how-to-write)
- [1.4 How the memory evolves](#14-how-the-memory-evolves)
- [2. Output and wording](#2-output-and-wording)
- [2.1 Style and format](#21-style-and-format)
- [2.2 Placement and naming](#22-placement-and-naming)
- [3. Inputs and sources](#3-inputs-and-sources)
- [3.1 Where inputs come from](#31-where-inputs-come-from)
- [3.2 Quirks and parsing](#32-quirks-and-parsing)
- [4. Steps and tools](#4-steps-and-tools)
- [4.1 Commands and flags](#41-commands-and-flags)
- [4.2 Machine and shell](#42-machine-and-shell)
- [5. Recurring requests](#5-recurring-requests)
- [5.1 Patterns](#51-patterns)
<!-- toc-end -->
What users want from the SKILL_NAME skill, kept across sessions:
standing preferences, corrections, repeated request patterns, and
facts a run verified while using this skill. Preferences for the
whole wiki go to the wiki's `MEMORY.md`. Knowledge from sources goes
to pages under `wiki/`.
## 1. How to use this file
### 1.1 When to read
- **1.1.1** Read this file once [the skill](./SKILL.md) is loaded,
before its first step of the session. When it is already in
context, do not read it again.
- **1.1.2** Apply every entry that matches the request.
- **1.1.2.1** An entry overrides a default of the skill and an
entry of the wiki's `MEMORY.md`, for work done with this skill
only.
- **1.1.2.2** An entry never overrides the boundaries in the wiki's
`AGENTS.md`; when the two conflict, follow the boundary and ask
the user.
- **1.1.2.3** An entry never changes a tool contract: a command, a
flag, or a required frontmatter field.
- **1.1.3** Every heading and list item is numbered as
[sectionize-markdown](../../skills/sectionize-markdown/SKILL.md)
describes. Cite an item as `SKILL_NAME MEMORIES.md item 2.1.3`; a
new item takes the next free number in its subsection.
### 1.2 When to write
- **1.2.1** Write an entry when one of these happens while the skill
is in use:
- **1.2.1.1** The user states a requirement or preference for this
skill that outlasts the current request ("always", "never", "from
now on", "when you do X").
- **1.2.1.2** The user corrects the skill's output or a step it
took.
- **1.2.1.3** The same kind of request comes for the second time;
record the pattern and what the user wanted.
- **1.2.1.4** A run verifies a reusable fact about the skill's
inputs, hosts, or tools that the skill does not state.
- **1.2.1.5** The user explicitly asks you to remember something
for this skill.
- **1.2.2** Do not write:
- **1.2.2.1** One-off values that only apply to the current
request.
- **1.2.2.2** Passwords, tokens, cookies, personal data, or
anything the user asks to keep out.
- **1.2.2.3** Preferences that hold for any wiki work; those go to
the wiki's `MEMORY.md`.
- **1.2.2.4** Facts the skill or a wiki page already states; cite
them instead.
- **1.2.2.5** Guesses; only what the user said or a run showed.
### 1.3 How to write
- **1.3.1** Put the entry under the `###` subsection that fits.
- **1.3.1.1** When none fits, add a `###` subsection with the next
free number under the matching `##`, or a new `##` category with
the next free number.
- **1.3.1.2** Never put an entry directly under a `##`; its id
would collide with a later subsection.
- **1.3.2** Use the knowledge-entry shape of sectionize-markdown
section 4: a header line with the source tag and a short title,
one child per fact, and a last child with the ref and the date.
- **1.3.3** Source tags: `user` (the user said it), `run` (a command
or tool run showed it), `source` (a file under `raw/` states it),
joined with `+` when several agree.
- **1.3.4** The ref says where to check the fact: the user's request
and its date, the command and its output line, or a page or raw
path.
- **1.3.5** Write the entry in the same turn, then tell the user
which item was added or changed, for example "Saved to SKILL_NAME
MEMORIES.md item 2.1.1".
- **1.3.6** Bump `version` (patch for a changed entry, minor for a
new entry or subsection) and set `updated` in the frontmatter on
every change.
- **1.3.7** Append `- YYYY-MM-DD - SKILL_NAME memory: <item id> <title>`
to `wiki/log.md`, and bump that file too.
- **1.3.8** The change lands in the save of the current ingest cycle
or housekeeping pass.
- **1.3.8.1** With no cycle running, save it on its own with
`fiona-assistant git save --description "SKILL_NAME memory: <item id> <title>"`.
An entry looks like this:
```text
- **2.1.1** [user] "Reports open with the verdict"
- **2.1.1.1** The report starts with pass or fail, then the evidence.
- **2.1.1.2** Applies to every run of this skill.
- **2.1.1.3** Ref: user request YYYY-MM-DD; added YYYY-MM-DD.
```
### 1.4 How the memory evolves
- **1.4.1** Before adding an entry, look for a near-duplicate in the
same category; extend it instead and keep its id and its earliest
date.
- **1.4.2** A finding that contradicts an entry becomes its next
child, titled `OVERRIDE: ...`, with the new facts and its own ref.
- **1.4.3** An entry that no longer applies is marked
**Superseded YYYY-MM-DD** with the reason; it is not deleted.
- **1.4.4** An entry that held for several uses graduates; mark it
**Promoted YYYY-MM-DD** with the target.
- **1.4.4.1** A rule for every run of the skill moves into
[the skill](./SKILL.md).
- **1.4.4.2** A preference that turns out to hold for all wiki
work moves to the wiki's `MEMORY.md`.
- **1.4.5** Never renumber ids; pages, log lines, and chats may cite
them.
- **1.4.6** Git is the memory's history: `git log -p MEMORIES.md` in
the skill's folder shows when each entry changed and why.
## 2. Output and wording
What the skill produces and how it reads.
### 2.1 Style and format
### 2.2 Placement and naming
## 3. Inputs and sources
What the skill reads and where it comes from.
### 3.1 Where inputs come from
### 3.2 Quirks and parsing
## 4. Steps and tools
How the skill's commands and the machine behave.
### 4.1 Commands and flags
### 4.2 Machine and shell
## 5. Recurring requests
Requests that come back, and what the user wanted each time.
### 5.1 Patterns
@@ -0,0 +1,109 @@
---
name: memory
category: productivity
description: Give a skill its own memory, a MEMORIES.md beside its SKILL.md.
author: nvaldez
version: 1.0.1
created: 2026-10-15
updated: 2026-10-21
---
# Skill Memory
<!-- toc-start: headings 2..6 -->
- [1. What a skill memory is](#1-what-a-skill-memory-is)
- [2. Give a skill memory](#2-give-a-skill-memory)
- [3. Wire it into the skill](#3-wire-it-into-the-skill)
- [4. Use a skill memory](#4-use-a-skill-memory)
<!-- toc-end -->
A skill with memory keeps what its users want from that skill across
sessions, in a `MEMORIES.md` file beside its `SKILL.md`. The blank
file to copy is [MEMORIES](./MEMORIES.md). The wiki-wide memory is
[MEMORY](../../../MEMORY.md); this skill does not replace it.
Every heading and list item is numbered as
[sectionize-markdown](../sectionize-markdown/SKILL.md)
describes. Cite a rule as `memory skill item 2.3`.
## 1. What a skill memory is
- **1.1** One file per skill: `MEMORIES.md` in the skill's folder,
next to its `SKILL.md`.
- **1.2** It holds facts about operating that one skill:
- **1.2.1** Preferences for the skill's output.
- **1.2.2** Corrections to the skill's steps or results.
- **1.2.3** Request patterns that came back for that skill.
- **1.2.4** Facts a run verified about the skill's inputs, hosts,
and tools.
- **1.3** It does not hold:
- **1.3.1** Knowledge from sources; that is a page under `wiki/`.
- **1.3.2** Preferences for the whole wiki; those go to
[MEMORY](../../../MEMORY.md).
- **1.3.3** Passwords, tokens, cookies, or personal data.
- **1.4** Precedence, from strongest to weakest:
- **1.4.1** The boundaries in [AGENTS](../../../AGENTS.md) and tool
contracts (commands, flags, required frontmatter).
- **1.4.2** The skill's `MEMORIES.md`, for work done with that skill.
- **1.4.3** The wiki's `MEMORY.md`.
- **1.4.4** The defaults in the skill's `SKILL.md`.
## 2. Give a skill memory
- **2.1** Give a skill memory when the user asks for a skill with
memory, or when the same correction to one skill comes a second
time.
- **2.2** A new skill lives under `utils/skills_local/<skill>/`. A
skill shipped under `utils/skills/` can also get a memory; the
template never ships a `MEMORIES.md` for it, so an update leaves the
file alone.
- **2.3** Copy [MEMORIES](./MEMORIES.md) to
`<skill-folder>/MEMORIES.md`. Never edit the template copy in this
folder.
- **2.4** In the copy:
- **2.4.1** Set frontmatter `name` to `<skill>-memories`, `version`
to `1.0.0`, and `created` and `updated` to today.
- **2.4.2** Replace every `SKILL_NAME` with the skill's name.
- **2.4.3** Rename the empty categories in sections 2 to 5 to fit
the skill, or keep them. Leave section 1 as it is.
- **2.5** Add the memory section of section 3 to the skill's
`SKILL.md` and bump that file's `version`.
- **2.6** Append `- YYYY-MM-DD - memory: <skill> gets MEMORIES.md` to
`wiki/log.md`, then save with
`fiona-assistant git save --description "memory: <skill> gets MEMORIES.md"`.
## 3. Wire it into the skill
- **3.1** The skill's `SKILL.md` gets this section as its last
numbered section, with `N` set to that section's number:
```markdown
## N. Memory
- **N.1** After loading this skill, read [MEMORIES](./MEMORIES.md)
once per session. When it is already in context, do not read it
again.
- **N.2** Update it whenever an important memory about this skill
comes up: a lasting preference, a correction, a repeated request,
or a fact a run verified. Also update it whenever the user
explicitly asks you to remember something for this skill.
Section 1 of that file says how.
```
- **3.2** Keep the link relative (`./MEMORIES.md`) so the skill still
finds its memory after a move.
## 4. Use a skill memory
- **4.1** Section 1 of each skill's `MEMORIES.md` is the rule for
reading and writing it. This file does not repeat it.
- **4.2** Pick the narrowest memory that covers the fact:
- **4.2.1** It only matters while one skill runs: that skill's
`MEMORIES.md`.
- **4.2.2** It matters for any wiki work: `MEMORY.md`.
- **4.2.3** When unsure, ask the user which one.
- **4.3** When a request uses a skill that has no memory and a
memory comes up, write it to `MEMORY.md` and suggest giving the
skill one (section 2).
@@ -0,0 +1,159 @@
---
name: sectionize-markdown
version: 1.1.3
title: How to sectionize a markdown file
description: Number every heading and list item of a markdown file so each
rule, step, or fact has a stable id to cite.
created: '2026-10-14'
updated: '2026-10-21'
confidence: high
llm:
- model: Claude Opus 5.5
---
# How to sectionize a markdown file
<!-- toc-start: headings 2..6 -->
- [1. Headings](#1-headings)
- [2. List item ids](#2-list-item-ids)
- [3. Splitting long lines](#3-splitting-long-lines)
- [4. Knowledge entries](#4-knowledge-entries)
- [5. Organize by category](#5-organize-by-category)
- [6. Doing it safely](#6-doing-it-safely)
- [7. Example](#7-example)
<!-- toc-end -->
Numbers every heading and every list item of a markdown file so each
rule, step, or fact has a stable id that can be cited ("item
4.2.3"), tracked, and checked for duplicates. It suits rule files,
skill instructions, glossaries, and knowledge caches.
## 1. Headings
- **1.1** Number every heading below the title: `## N.` for sections,
`### N.M` for subsections, `#### N.M.K` below that.
- **1.2** The `#` title and the table of contents are not numbered.
- **1.3** Put a TOC of markdown links right under the title and intro.
Regenerate it whenever headings change.
- **1.4** Anchors are GitHub-style slugs: `## 5.6 Download the logs`
becomes `#56-download-the-logs`.
- **1.5** Avoid em dashes and other punctuation in headings; use a colon
(`10.7 Outcome: a match exists`). Punctuation makes the slug
unpredictable and breaks TOC links (MD051).
- **1.6** A label that only introduces a list ("Files in this
folder:") is a numbered list item, not a heading.
## 2. List item ids
- **2.1** Every list item gets a bold id right after the dash:
`- **4.2.1** Text`. Bold is two asterisks on each side; a single
asterisk is italic.
- **2.2** The id is the section number plus a running counter:
`**<section>.<n>**`. Nested items extend the parent id:
`**<section>.<n>.<m>**`, indented two spaces.
- **2.3** The counter runs through the whole section, across separate
lists split by paragraphs or code blocks. Two lists under `### 12.3`
give `**12.3.1**`…`**12.3.3**`, then `**12.3.4**`…, never a second
`**12.3.1**`.
- **2.4** Checklists keep the box first: `- [ ] **13.1** Verify …`.
- **2.5** Do not number the TOC, code blocks, or table rows.
- **2.6** State the convention in the file's first section so the next
editor follows it ("Cite an item as `item 4.2.3`").
- **2.7** Never renumber existing ids; other notes may cite them. A new
item takes the next free number in its section, even if that puts it
out of reading order. Fix the order only while the ids are still
unreleased (uncommitted).
## 3. Splitting long lines
- **3.1** A list item holding several facts becomes a short header item
plus one child per fact.
- **3.2** One child is one sentence, or two closely linked sentences.
- **3.3** Split at sentence ends (`.`, `?` or `!`, then a space and an
uppercase letter, backtick, `*`, `"` or `[`). Do not split inside
backticks or parentheses, or after `e.g.`, `i.e.`, `vs.`, `cf.`.
- **3.4** Merge fragments shorter than about 45 characters back into the
previous child; they read better together.
- **3.5** A long single sentence stays on one line. Do not cut a
sentence in half to meet a length target.
- **3.6** Leave command lines, code blocks, and table rows as they are.
## 4. Knowledge entries
A file that collects findings or lessons learned uses a fixed entry
shape so each fact is citable and the source is visible:
```text
- **<section>.<n>** [<source>] "<short title>"
- **<section>.<n>.1** <reusable fact + where it applies>
- **<section>.<n>.2** <next fact>
- **<section>.<n>.<k>** Ref: <path:line or query>; added <YYYY-MM-DD>.
```
- **4.1** The entry line holds only the source tag and the title.
- **4.2** `<source>` names where the fact was verified (a repository, a
spec, …), joined with `+` when several agree.
- **4.3** The last child is always the ref with the date it was added.
- **4.4** A finding that contradicts an entry becomes that entry's next
child, titled `OVERRIDE: …`, with its own children. Or mark the old
entry **Superseded YYYY-MM-DD** with the reason.
## 5. Organize by category
- **5.1** Group entries by topic (setup, build, deployment, …), not by
the date they were added. The date lives in the ref line.
- **5.2** Grouping puts near-duplicates next to each other. Merge them
into one entry and keep the earliest `added` date.
- **5.3** A glossary gets the same treatment: one
`###` per term under a `##` category, its definition split into
numbered items.
## 6. Doing it safely
- **6.1** Restructure with a script, not by hand, when the file has more
than a few dozen items. Hand edits drop or reorder text.
- **6.2** The script asserts that the words are preserved: same word
sequence before and after, only ids and line breaks added.
- **6.3** Verify after every pass:
- **6.3.1** No duplicate ids (`^\s*- (\[ \] )?\*\*([\d.]+)\*\*`).
- **6.3.2** Every `](#anchor)` resolves to a heading slug.
- **6.3.3** The entry count matches the original; diff titles against
`HEAD` when it does not (an uncommitted entry may explain it).
- **6.3.4** One `Ref:` line per entry.
- **6.4** Watch for editors that reformat on save (escaped `\*`, `*`
bullets padded with three spaces, lost HTML comments, fences without a language). Normalize
back and re-verify.
- **6.5** On Windows PowerShell, write helper scripts to `%TEMP%` and run
them; `python -c` with backticks breaks. Avoid printing non-ASCII to
the cp1252 console. Delete the temp scripts afterwards.
- **6.6** Keep Unix line endings and `-` bullets, and run the markdown
linter before committing.
## 7. Example
Before:
```text
## Topic
- Rule about the topic. A fact about it. How to fix it. Ref: file:line.
- Label for a group: first part of the rule, second part of the rule.
```
After:
```text
## 3. Topic
### 3.2 Subtopic
- **3.2.4** [source] "Rule about the topic"
- **3.2.4.1** A fact about it.
- **3.2.4.2** How to fix it.
- **3.2.4.3** Ref: file:line; added YYYY-MM-DD.
- **3.2.5** Label for a group:
- **3.2.5.1** First part of the rule.
- **3.2.5.2** Second part of the rule.
```
@@ -0,0 +1,321 @@
---
name: specification-writer
description: Load when writing or updating project specifications (SPEC.md).
Expert in creating structured and maintainable specification documents.
type: skill
version: 1.0.0
created: 2026-11-03
updated: 2026-11-03
---
# Specification Writer
You are an expert in writing software project specifications.
This skill provides guidance on creating structured, comprehensive, and
maintainable SPEC.md files following established best practices.
## When to Load
Load this skill when:
- Creating a new SPEC.md file for a project
- Updating or expanding an existing specification
- Reviewing specification structure and content
- Ensuring specifications follow consistent patterns
## Core Principles
### Sectionize-Markdown Conventions
Always apply sectionize-markdown conventions to every SPEC.md file.
Reference: [sectionize-markdown skill](../sectionize-markdown/SKILL.md)
- Every heading MUST be numbered: `## 1.` for sections, `### 1.1`
for subsections, etc.
- Every list item MUST have a bold ID: `- **1.1.1** Text`
- TOC MUST be present under the title with markdown links
- Anchors MUST follow GitHub-style slugs
- Never renumber existing IDs - they may be cited elsewhere
## SPEC.md Structure Template
Use this structure as the foundation for all project specifications:
```markdown
---
name: <Project Name> Specification
version: <semver>
title: Project Spec Sheet
type: specification
created: <YYYY-MM-DD>
updated: <YYYY-MM-DD>
---
# <Project Name> Specification
<!-- toc-start: headings 2..6 -->
- [1. Product Overview](#1-product-overview)
- [1.1. Purpose](#11-purpose)
- [1.2. Scope](#12-scope)
- [1.3. Non-Goals](#13-non-goals)
- [1.4. Document Conventions](#14-document-conventions)
- [2. Architecture](#2-architecture)
- [2.1. Component Diagram](#21-component-diagram)
- [2.2. Data Flow](#22-data-flow)
- [2.3. Module Structure](#23-module-structure)
- [3. Core Contracts](#3-core-contracts)
- [3.1. Interfaces](#31-interfaces)
- [3.2. Data Structures](#32-data-structures)
- [4. Package and Runtime](#4-package-and-runtime)
- [4.1. Dependencies](#41-dependencies)
- [4.2. Entry Points](#42-entry-points)
- [4.3. Installation](#43-installation)
- [5. Feature Requirements](#5-feature-requirements)
- [6. Configuration Files](#6-configuration-files)
- [7. Error Handling](#7-error-handling)
- [8. Testing](#8-testing)
- [9. Deployment](#9-deployment)
<!-- toc-end -->
This document is the build contract for <Project Name>.
A reader with this file, and no source tree, must be able to produce
the same observable software.
Status values are `Implemented`, `Partially Implemented`, `Planned`,
and `Not Implemented`.
---
```
## Detailed Section Guidelines
### 1. Product Overview
#### 1.1. Purpose
- **1.1.1** Clearly state what problem the project solves
- **1.1.2** Describe the primary value proposition
- **1.1.3** Identify the target users or audience
- **1.1.4** Define the project's mission statement
#### 1.2. Scope
- **1.2.1** **In Scope**: List what the project will do
- **1.2.2** **Out of Scope**: List what the project will NOT do
- **1.2.3** Define clear boundaries for the project
#### 1.3. Non-Goals
- **1.3.1** Explicitly state what will not be implemented
- **1.3.2** Clarify boundaries with related projects or systems
- **1.3.3** Document conscious exclusions and rationale
#### 1.4. Document Conventions
- **1.4.1** Define terminology used throughout the document
- **1.4.2** Establish status labels and their meanings
- Implemented: Feature is complete and tested
- Partially Implemented: Feature is partially complete
- Planned: Feature is scheduled for implementation
- Not Implemented: Feature is specified but not yet built
- **1.4.3** Specify numbering conventions for sections and items
- **1.4.4** Define RFC2119 keywords (MUST, SHALL, SHOULD, MAY, etc.)
### 2. Architecture
#### 2.1. Component Diagram
- **2.1.1** Provide ASCII or Mermaid diagram showing system architecture
- **2.1.2** Label all major components and their relationships
- **2.1.3** Show data flow between components with arrows or descriptions
- **2.1.4** Indicate external dependencies and integrations
#### 2.2. Data Flow
- **2.2.1** Describe how data moves through the system end-to-end
- **2.2.2** Identify key data transformations at each stage
- **2.2.3** Note where data is persisted and in what format
- **2.2.4** Document data validation and error handling points
#### 2.3. Module Structure
- **2.3.1** Document the complete directory and file structure
- **2.3.2** Describe each module's responsibility and purpose
- **2.3.3** Note dependencies between modules
- **2.3.4** Indicate which modules are public vs internal APIs
### 3. Core Contracts
#### 3.1. Interfaces
- **3.1.1** Define purpose and role of each interface
- **3.1.2** List all required methods or functions with signatures
- **3.1.3** Document expected inputs, outputs, and return types
- **3.1.4** Specify error handling expectations and exceptions
- **3.1.5** Note version compatibility requirements
#### 3.2. Data Structures
- **3.2.1** Provide complete type definitions or JSON schemas
- **3.2.2** Include concrete examples of each data structure
- **3.2.3** Document validation rules and constraints
- **3.2.4** Note backward and forward compatibility considerations
- **3.2.5** Specify serialization formats (JSON, YAML, etc.)
### 4. Package and Runtime
#### 4.1. Dependencies
- **4.1.1** List runtime dependencies with exact version requirements
- **4.1.2** List development dependencies for building and testing
- **4.1.3** Document optional dependencies and their purposes
- **4.1.4** Note any version conflicts or compatibility constraints
#### 4.2. Entry Points
- **4.2.1** Define CLI entry points and their usage
- **4.2.2** Document library entry points for programmatic use
- **4.2.3** Specify plugin or extension entry points if applicable
- **4.2.4** Include examples of invoking each entry point
#### 4.3. Installation
- **4.3.1** Document all supported installation methods
- **4.3.2** Specify platform and environment requirements
- **4.3.3** Provide verification steps to confirm successful installation
- **4.3.4** Note any post-installation configuration needed
### 5. Feature Requirements
Organize requirements by feature area. Each feature section should include:
- **5.X.1** Status: One of Implemented, Partially Implemented, Planned,
Not Implemented
- **5.X.2** Purpose: User stories and motivations for the feature
- **5.X.3** Functional requirements: What the feature must do
- **5.X.4** Non-functional requirements: Performance, security, etc.
- **5.X.5** Acceptance criteria: How to verify the feature works
### 6. Configuration Files
For each configuration file, document:
- **6.1.1** Location: Both relative path and typical absolute paths
- Note platform-specific locations (Windows vs Unix)
- **6.1.2** Purpose: What the configuration controls
- **6.1.3** Structure: Complete format and schema
- **6.1.4** Default values: What settings are used if not specified
- **6.1.5** Customization: How users can modify the configuration
### 7. Error Handling
#### 7.1. Error Classification
- **7.1.1** Transient vs permanent errors
- **7.1.2** Recoverable vs unrecoverable errors
- **7.1.3** User-facing vs internal errors
- **7.1.4** Expected vs unexpected errors
#### 7.2. Error Reporting
- **7.2.1** Logging strategy and verbosity levels
- **7.2.2** User notification mechanisms (CLI, UI, etc.)
- **7.2.3** Error codes and their meanings
- **7.2.4** Error message format and content
#### 7.3. Recovery Strategies
- **7.3.1** Automatic recovery mechanisms and retries
- **7.3.2** Manual recovery procedures and user actions
- **7.3.3** Data consistency guarantees during failures
- **7.3.4** Rollback and undo capabilities
### 8. Testing
#### 8.1. Test Strategy
- **8.1.1** Testing pyramid: Unit, integration, system, E2E
- **8.1.2** Test coverage targets for different test types
- **8.1.3** Test environments and their configurations
- **8.1.4** CI/CD integration for automated testing
#### 8.2. Test Cases
- **8.2.1** Critical path tests covering core functionality
- **8.2.2** Edge case tests for boundary conditions
- **8.2.3** Regression tests for known issues
- **8.2.4** Performance and load tests if applicable
### 9. Deployment
#### 9.1. Release Process
- **9.1.1** Versioning scheme (SemVer, CalVer, etc.)
- **9.1.2** Release checklist and required steps
- **9.1.3** Rollback procedures for failed deployments
- **9.1.4** Release notes and changelog requirements
#### 9.2. Distribution
- **9.2.1** Package repositories and distribution channels
- **9.2.2** Installation methods for end users
- **9.2.3** Update mechanisms and version management
- **9.2.4** Offline or air-gapped installation options
## Sectionize-Markdown Integration
**CRITICAL**: Every SPEC.md file you create or modify MUST follow
sectionize-markdown conventions completely.
Before finalizing any SPEC.md:
1. Apply numbered headings (## 1., ### 1.1, #### 1.1.1)
2. Add bold IDs to all list items (- **1.1.1**)
3. Generate or update the TOC with markdown links
4. Ensure all anchors resolve correctly
5. Verify no duplicate IDs exist
Use the sectionize-markdown skill as the definitive reference for
all sectionizing conventions and requirements.
## Quality Checklist
Before marking a SPEC as complete, verify:
- [ ] **Structure**: All required sections are present
- [ ] **Numbering**: All headings and list items are numbered correctly
- [ ] **TOC**: Table of contents is present and accurate
- [ ] **Status**: Every feature section has a status line
- [ ] **Examples**: Code and configuration examples are provided
- [ ] **Cross-references**: All internal links resolve correctly
- [ ] **Consistency**: Terminology and formatting are consistent
- [ ] **Completeness**: All acceptance criteria are defined
- [ ] **Sectionized**: Follows sectionize-markdown conventions completely
## Maintenance Guidelines
- **Versioning**: Increment version in frontmatter with each update
- **Changelog**: Maintain updated and created dates in frontmatter
- **Review**: SPEC should be reviewed before major implementation
- **Sync**: Keep SPEC in sync with implementation (update status)
## Tools and Automation
When working with SPEC files:
- Use scripts to verify section numbering consistency
- Automate TOC generation to avoid manual errors
- Validate all internal links before committing
- Check for duplicate IDs across the entire document
- Run markdown linters to ensure formatting quality
## Common Pitfalls
- **Unnumbered items**: Every list item MUST have a bold ID
- **Broken anchors**: Always verify all TOC links resolve
- **Inconsistent status**: Use consistent status labels throughout
- **Missing examples**: Always include examples for data formats
- **Ambiguous requirements**: Requirements must be testable and verifiable
- **Overly verbose**: Be concise but comprehensive
- **Under-specified**: Include enough detail for implementation
@@ -1,10 +1,15 @@
--- ---
name: index name: index
title: Wiki Index title: Wiki Index
version: 1.0.0 version: 1.1.0
type: summary type: summary
created: 2025-09-23 created: 2025-09-23
updated: 2026-09-24 updated: 2026-09-30
confidence: trusted
llm:
- model: none
contested: false
contradictions: []
--- ---
# Wiki Index # Wiki Index
+19
View File
@@ -0,0 +1,19 @@
---
name: log
title: Wiki Log
version: 2.0.0
type: summary
created: DATE_PLACEHOLDER
updated: DATE_PLACEHOLDER
confidence: trusted
llm:
- model: none
contested: false
contradictions: []
---
# Wiki Log
Chronological tracking of all actions performed on the wiki.
- DATE_PLACEHOLDER - Initialized the llm-wiki.
@@ -1,34 +1,47 @@
--- ---
name: llm-wiki-schema name: llm-wiki-schema
title: LLM Wiki Schema title: LLM Wiki Schema
version: 1.2.0 version: 1.6.1
created: 2025-09-18 created: 2025-09-18
updated: 2026-09-26 updated: 2026-10-06
type: summary type: summary
tags: [wiki, schema] tags: [wiki, schema]
confidence: trusted
llm:
- model: claude-opus-5.5
contested: false
contradictions: []
--- ---
# LLM Wiki Schema # LLM Wiki Schema
## Domain ## Domain
Personal knowledge base and inventory management. A personal knowledge base. Replace this line with the wiki's subject
and extend the tag taxonomy to match it.
## Conventions ## Conventions
- File names: lowercase, hyphens, no spaces (e.g., `llm-wiki-schema.md`) - File names: lowercase, hyphens, no spaces (e.g., `llm-wiki-schema.md`)
- Every wiki page starts with YAML frontmatter (required: `name`, `version`; - Every page starts with YAML frontmatter. Required on every file:
optional: `title`, `created`, `updated`, `type`, `tags`, `sources`). `name`, `version`. Required on pages under `wiki/`: `confidence`,
- Use `[[wikilinks]]` to link between pages (minimum 2 outbound links per page). `llm`, `contested`, `contradictions`. Optional: `title`, `created`,
- When updating a page, always bump the `updated` date. `updated`, `type`, `tags`, `sources`. Ingested pages also carry
`status`, `source_meta`, and the `ingest` provenance map.
- Use markdown links (`[text](url)`) to link between pages (minimum 2 outbound links per page).
- When updating a page, bump `version` and set `updated` to the edit date.
- Every new page must be added to `index.md` under the correct section. - Every new page must be added to `index.md` under the correct section.
- Every action must be appended to `log.md`. - Every action must be appended to `log.md`. In the first edit of a
new year, move the previous year's entries to `logs/log-YYYY.md`.
## Frontmatter ## Frontmatter
`name` and `version` are required on every md file; all other fields are `name` and `version` are required on every md file. Every page under
optional. This is the single canonical frontmatter schema, identical to `wiki/` also requires the quality signals `confidence`, `llm`,
project agent rule 10.1 (see SPEC 4.2.1). `contested`, and `contradictions`. How to set them is section 15 of
the [Fiona skill](../utils/skills/fiona/SKILL.md). All other fields
are optional. This is the single canonical frontmatter schema.
fmlint enforces `name` and `version` only.
```yaml ```yaml
--- ---
@@ -40,17 +53,34 @@ updated: YYYY-MM-DD # bump when editing
type: entity | concept | comparison | query | summary | toc type: entity | concept | comparison | query | summary | toc
sources: [raw/articles/source-name.md] sources: [raw/articles/source-name.md]
tags: [from taxonomy below] tags: [from taxonomy below]
# Optional quality signals: # quality signals (required on pages under wiki/):
confidence: trusted | high | medium | low | zero confidence: trusted | high | medium | low | zero
llm:
- model: none # required; model that last edited the page
- reasoning: high # when the model reports a level
contested: true contested: true
contradictions: [other-page-slug] contradictions: [other-page-slug]
# written by wiki ingest (Fiona skill section 18):
status: raw | refined
source_meta: {author: Name} # metadata read from the source
ingest: # provenance; every key always present
origin: https://host/path # URL or file path of the source
origin_kind: url | file
final_url: null # URL after redirects
fetched: YYYY-MM-DDTHH:MM:SSZ
media_type: text/html
size_bytes: 0
sha256: <64 lowercase hex> # of the raw copy; never hand-edited
raw_reused: false
converter: html # converter name, or agent
conversion: complete | partial
warnings: []
tool: fiona-assistant <version>
--- ---
``` ```
## Tag Taxonomy ## Tag Taxonomy
- inventory
- box
- wiki - wiki
- schema - schema
@@ -59,7 +89,7 @@ contradictions: [other-page-slug]
- Create a page when an entity/concept appears in 2+ sources OR is central to - Create a page when an entity/concept appears in 2+ sources OR is central to
one source. one source.
- Split pages over 200 lines. - Split pages over 200 lines.
- Archive superseded pages to `_archive/`. - Move superseded pages to `wiki/_archive/` and fix every inbound link.
## Directory Structure ## Directory Structure
@@ -75,7 +105,8 @@ llm-wiki/
│ ├── articles/ # Web articles, clippings │ ├── articles/ # Web articles, clippings
│ ├── papers/ # PDFs, arXiv papers │ ├── papers/ # PDFs, arXiv papers
│ ├── transcripts/ # Meeting notes, interviews │ ├── transcripts/ # Meeting notes, interviews
│ └── assets/ # Images, diagrams referenced by sources │ ├── assets/ # Images, diagrams referenced by sources
│ └── other/ # Ingested files no converter handles
├── utils/ ├── utils/
│ ├── personas/ # Agent personas configurations │ ├── personas/ # Agent personas configurations
│ ├── scripts/ # Reusable utility scripts │ ├── scripts/ # Reusable utility scripts
@@ -90,12 +121,13 @@ llm-wiki/
│ ├── comparisons/ # Layer 2: Side-by-side analyses │ ├── comparisons/ # Layer 2: Side-by-side analyses
│ ├── queries/ # Layer 2: Filed query results worth keeping │ ├── queries/ # Layer 2: Filed query results worth keeping
│ ├── sources/ # Layer 2: Source material backing wiki pages │ ├── sources/ # Layer 2: Source material backing wiki pages
│ └── logs/ # Archived log files rotated out of log.md │ ├── logs/ # Archived log files rotated out of log.md
│ └── _archive/ # Superseded pages, created when first needed
└── table_of_contents.md # Auto-generated navigation index for quick lookup └── table_of_contents.md # Auto-generated navigation index for quick lookup
``` ```
- **Layer 1 — Raw Sources:** `raw/` at the wiki root. The agent reads but never - **Layer 1 — Raw Sources:** `raw/` at the wiki root. The agent adds new
modifies these. sources and never modifies existing ones.
- **Layer 2 — The Wiki:** Agent-owned markdown files. Created, updated, and - **Layer 2 — The Wiki:** Agent-owned markdown files. Created, updated, and
cross-referenced by the agent. cross-referenced by the agent.
- **Layer 3 — The Schema:** `SCHEMA.md` defines structure, conventions, and tag taxonomy. - **Layer 3 — The Schema:** `wiki/schema.md` (this file) defines structure, conventions, and tag taxonomy.
File diff suppressed because it is too large Load Diff
@@ -7,12 +7,14 @@ from __future__ import annotations
import argparse import argparse
import pathlib import pathlib
from datetime import datetime, timezone from datetime import UTC, datetime
import git import git
from git.repo.base import Repo from git.repo.base import Repo
from fiona.gittool import diff_report
from fiona.logging_config import setup_logging from fiona.logging_config import setup_logging
from fiona.wiki.ignore import ensure_wiki_gitignore
class GitTool: class GitTool:
@@ -50,7 +52,7 @@ class GitTool:
Returns: Returns:
str: Formatted timestamp string. str: Formatted timestamp string.
""" """
return datetime.now(tz=timezone.utc).strftime("%Y-%m-%d %H:%M:%S") return datetime.now(tz=UTC).strftime("%Y-%m-%d %H:%M:%S")
@staticmethod @staticmethod
def _get_tag_timestamp() -> str: def _get_tag_timestamp() -> str:
@@ -62,7 +64,7 @@ class GitTool:
Returns: Returns:
str: Formatted timestamp string safe for git tags. str: Formatted timestamp string safe for git tags.
""" """
return datetime.now(tz=timezone.utc).strftime("%Y-%m-%d_%H-%M-%S") return datetime.now(tz=UTC).strftime("%Y-%m-%d_%H-%M-%S")
def _is_working_tree_clean(self) -> bool: def _is_working_tree_clean(self) -> bool:
"""Check if the working tree is clean. """Check if the working tree is clean.
@@ -73,7 +75,7 @@ class GitTool:
# Check for modified/staged files # Check for modified/staged files
if self.repo.is_dirty(): if self.repo.is_dirty():
return False return False
# Check for untracked files # Check for untracked files
untracked = self.repo.untracked_files untracked = self.repo.untracked_files
return len(untracked) == 0 return len(untracked) == 0
@@ -170,7 +172,7 @@ class GitTool:
tag for tag in self.repo.tags tag for tag in self.repo.tags
if tag.name.endswith(f"_{tag_name}") or tag.name == tag_name if tag.name.endswith(f"_{tag_name}") or tag.name == tag_name
] ]
if matching_tags: if matching_tags:
# Sort by commit date descending and take the most recent # Sort by commit date descending and take the most recent
matching_tags.sort( matching_tags.sort(
@@ -178,7 +180,7 @@ class GitTool:
reverse=True reverse=True
) )
tag_ref = matching_tags[0] tag_ref = matching_tags[0]
if tag_ref is None: if tag_ref is None:
raise git.GitCommandError( raise git.GitCommandError(
f"Tag '{tag_name}' does not exist." f"Tag '{tag_name}' does not exist."
@@ -189,7 +191,7 @@ class GitTool:
# Restore the tag's tree into the working directory and index # Restore the tag's tree into the working directory and index
self.repo.git.checkout(tag_ref.commit, "--", ".") self.repo.git.checkout(tag_ref.commit, "--", ".")
# Create a new commit with the restored state # Create a new commit with the restored state
new_commit = self.repo.index.commit(restore_message) new_commit = self.repo.index.commit(restore_message)
@@ -272,9 +274,9 @@ class GitTool:
@staticmethod @staticmethod
def init_wiki_repo(repo_path: str | pathlib.Path, initial_message: str = "Initial llm-wiki commit") -> git.Repo: def init_wiki_repo(repo_path: str | pathlib.Path, initial_message: str = "Initial llm-wiki commit") -> git.Repo:
"""Initialize a git repository for llm-wiki with .gitignore for personal/. """Initialize a git repository for llm-wiki and seed .gitignore.
Creates a .gitignore file that excludes the personal/ directory, Puts or appends the repository ignore patterns plus ``personal/``,
stages all files, and makes the initial commit. stages all files, and makes the initial commit.
Args: Args:
@@ -288,26 +290,23 @@ class GitTool:
git.InvalidGitRepositoryError: If git initialization fails. git.InvalidGitRepositoryError: If git initialization fails.
""" """
repo_path = pathlib.Path(repo_path).resolve() repo_path = pathlib.Path(repo_path).resolve()
# Initialize the repo # Initialize the repo
repo = GitTool.init_repo(repo_path) repo = GitTool.init_repo(repo_path)
# Create .gitignore to exclude personal/ directory # AI/LLM: 2026-09-30 - seed .gitignore from the repository ignore file.
gitignore_path = repo_path / ".gitignore" ensure_wiki_gitignore(repo_path)
with open(gitignore_path, "w", encoding="utf-8") as f:
f.write("# Exclude personal directory from git as per SPEC 3.2.0.2\n")
f.write("personal/\n")
# Add all files and make initial commit # Add all files and make initial commit
repo.git.add("--all") repo.git.add("--all")
repo.index.commit(initial_message) repo.index.commit(initial_message)
return repo return repo
def main(): # pragma: no cover def main(): # pragma: no cover
"""Command-line interface for the GitTool. """Command-line interface for the GitTool.
Note: CLI is tested via integration tests through fiona-assistant git commands. Note: CLI is tested via integration tests through fiona-assistant git commands.
""" """
parser = argparse.ArgumentParser( parser = argparse.ArgumentParser(
@@ -351,6 +350,30 @@ def main(): # pragma: no cover
# status subcommand # status subcommand
subparsers.add_parser("status", help="Show git status") subparsers.add_parser("status", help="Show git status")
# diff subcommand
diff_parser = subparsers.add_parser(
"diff",
help="Write constellation-themed HTML diff under wiki tmp/",
)
diff_parser.add_argument("--base", help="Base revision (default: origin/main or main/master)")
diff_parser.add_argument("--head", default="HEAD", help="Head revision (default: HEAD)")
diff_parser.add_argument("--title", help="Report H1 / page title")
diff_parser.add_argument(
"--subtitle",
help="Extra header line (e.g. temporary review before push)",
)
diff_parser.add_argument(
"-o",
"--output",
type=pathlib.Path,
help="Output HTML path (default: <wiki>/tmp/<repo>-git-diff-<short>.html)",
)
diff_parser.add_argument(
"--collapsed",
action="store_true",
help="Start with per-file sections collapsed",
)
args = parser.parse_args() args = parser.parse_args()
setup_logging() setup_logging()
@@ -380,6 +403,21 @@ def main(): # pragma: no cover
elif args.subcommand == "status": elif args.subcommand == "status":
status = git_tool.status() status = git_tool.status()
print(status) print(status)
elif args.subcommand == "diff":
diff_argv: list[str] = []
if args.base:
diff_argv.extend(["--base", args.base])
if args.head != "HEAD":
diff_argv.extend(["--head", args.head])
if args.title:
diff_argv.extend(["--title", args.title])
if args.subtitle:
diff_argv.extend(["--subtitle", args.subtitle])
if args.output is not None:
diff_argv.extend(["--output", str(args.output)])
if args.collapsed:
diff_argv.append("--collapsed")
raise SystemExit(diff_report.main(diff_argv))
except Exception as e: except Exception as e:
print(f"Error: {e}") print(f"Error: {e}")
raise SystemExit(1) from e raise SystemExit(1) from e
@@ -0,0 +1,327 @@
<!DOCTYPE html>
<!-- Packaged layout for fiona.gittool.diff_report; placeholders filled at generation time. -->
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>{{PAGE_TITLE}}</title>
<style>
{{THEME_CSS}}
* { box-sizing: border-box; }
body {
margin: 0;
font-family: var(--font);
background: var(--bg);
color: var(--text);
line-height: 1.45;
transition: background-color 0.3s ease, color 0.3s ease;
}
header {
background: var(--card);
color: var(--text);
padding: 1rem 1.25rem;
border-bottom: 1px solid var(--border);
display: flex;
justify-content: space-between;
align-items: center;
flex-wrap: wrap;
gap: 1rem;
}
header .header-title {
flex: 1;
min-width: 200px;
}
header h1 { margin: 0 0 0.35rem; font-size: 1.4rem; font-weight: 600; }
header p { margin: 0.25rem 0; opacity: 0.92; font-size: 0.9rem; }
header code {
background: color-mix(in srgb, var(--button-text) 15%, transparent);
padding: 0.1rem 0.35rem;
border-radius: 4px;
}
.theme-toggle {
display: flex;
align-items: center;
gap: 0.5rem;
}
.theme-toggle button {
padding: 0.35rem 0.65rem;
cursor: pointer;
border: 1px solid var(--border);
border-radius: 6px;
background: var(--button-bg);
color: var(--button-text);
font-size: 0.85rem;
}
.theme-toggle button:hover { background: var(--button-hover); }
.layout {
display: flex;
align-items: stretch;
min-height: calc(100vh - 8rem);
}
aside.sidebar {
width: var(--sidebar-w);
flex-shrink: 0;
position: sticky;
top: 0;
align-self: flex-start;
max-height: 100vh;
overflow: auto;
background: var(--card);
border-right: 1px solid var(--border);
padding: 0.75rem 0.5rem 1rem;
}
aside.sidebar h2 {
margin: 0 0 0.5rem 0.65rem;
font-size: 0.75rem;
text-transform: uppercase;
letter-spacing: 0.05em;
color: var(--blue);
}
.file-tree {
list-style: none;
margin: 0;
padding: 0 0 0 0.35rem;
font-size: 0.8rem;
}
.file-tree ul {
list-style: none;
margin: 0;
padding-left: 0.65rem;
border-left: 1px solid var(--border);
}
.file-tree li { margin: 0.15rem 0; }
.file-tree .folder-label {
display: block;
font-weight: 600;
color: var(--text);
padding: 0.1rem 0.35rem;
}
.file-tree a {
display: block;
padding: 0.15rem 0.35rem;
border-radius: 4px;
color: var(--accent);
text-decoration: none;
word-break: break-all;
}
.file-tree a:hover { background: var(--link-hover-bg); text-decoration: underline; }
.file-tree a.active {
background: var(--link-active-bg);
font-weight: 600;
color: var(--blue);
}
.file-tree .churn {
font-size: 0.7rem;
color: var(--muted);
margin-left: 0.25rem;
white-space: nowrap;
}
.churn .add { color: var(--churn-add); }
.churn .del { color: var(--churn-del); }
.file-icon {
font-size: 0.8rem;
margin-right: 0.35rem;
opacity: 0.7;
}
.stat-fold {
margin: 0.75rem 0.5rem 0;
font-size: 0.75rem;
}
.stat-fold summary { cursor: pointer; color: var(--muted); }
.stat-fold pre {
margin: 0.35rem 0 0;
font-size: 0.68rem;
white-space: pre-wrap;
max-height: 12rem;
overflow: auto;
background: var(--summary-bg);
padding: 0.5rem;
border-radius: 4px;
border: 1px solid var(--border);
}
.content {
flex: 1;
min-width: 0;
padding: 1rem 1.25rem 2.5rem;
}
.toolbar {
font-size: 0.85rem;
display: flex;
align-items: center;
gap: 0.5rem;
flex-wrap: wrap;
background: var(--card);
border: 1px solid var(--border);
border-radius: 8px;
padding: 0.65rem 0.85rem;
margin-bottom: 1rem;
}
button {
padding: 0.35rem 0.65rem;
cursor: pointer;
border: 1px solid var(--border);
border-radius: 6px;
background: var(--button-bg);
color: var(--button-text);
}
button:hover { background: var(--button-hover); }
details.file {
margin: 0.75rem 0;
border: 1px solid var(--border);
border-radius: 6px;
background: var(--card);
scroll-margin-top: 0.75rem;
}
details.file:target {
outline: 2px solid var(--accent);
outline-offset: 2px;
}
details.file summary {
cursor: pointer;
padding: 0.6rem 0.75rem;
font-weight: 600;
background: var(--summary-bg);
}
pre.diff {
margin: 0;
padding: 0.75rem;
font-size: 0.72rem;
line-height: 1.0;
overflow: auto;
max-height: 70vh;
}
.diff-add { background: var(--diff-add-bg); display: block; margin: 0; padding: 0; }
.diff-del { background: var(--diff-del-bg); display: block; margin: 0; padding: 0; }
.diff-hunk { color: var(--diff-hunk); display: block; margin: 0; padding: 0; }
.diff-meta { color: var(--diff-meta); display: block; margin: 0; padding: 0; }
.diff-ctx { display: block; margin: 0; padding: 0; }
@media (max-width: 900px) {
.layout { flex-direction: column; }
aside.sidebar {
width: 100%;
position: relative;
max-height: 40vh;
border-right: none;
border-bottom: 1px solid var(--border);
}
}
/* Dark mode theme - applied when data-theme="dark" (default) */
[data-theme="dark"] {
--bg: #1a1a1a;
--card: #2d2d2d;
--border: #404040;
--text: #e0e0e0;
--blue: #6ba1f5;
--accent: #7aa7d6;
--header-end: #c586c0;
--muted: #9e9e9e;
--summary-bg: #252525;
--button-bg: #3d3d3d;
--button-text: #e0e0e0;
--button-hover: #4a4a4a;
--churn-add: #7edd8a;
--churn-del: #ff7b7b;
--diff-add-bg: color-mix(in srgb, #7edd8a 22%, var(--card));
--diff-del-bg: color-mix(in srgb, #ff7b7b 22%, var(--card));
--link-hover-bg: color-mix(in srgb, #7aa7d6 18%, var(--card));
--link-active-bg: color-mix(in srgb, #7aa7d6 22%, var(--card));
}
/* Light mode theme - applied when data-theme="light" */
[data-theme="light"] {
--bg: #ffffff;
--card: #f8f9fa;
--border: #e2e8f0;
--text: #1a1a1a;
--blue: #0d6efd;
--accent: #0d6efd;
--header-end: #6610f2;
--muted: #6c757d;
--summary-bg: #f8f9fa;
--button-bg: #ffffff;
--button-text: #1a1a1a;
--button-hover: #e2e8f0;
--churn-add: #15803d;
--churn-del: #b91c1c;
--diff-add-bg: color-mix(in srgb, #15803d 22%, var(--card));
--diff-del-bg: color-mix(in srgb, #b91c1c 22%, var(--card));
--link-hover-bg: color-mix(in srgb, #0d6efd 18%, var(--card));
--link-active-bg: color-mix(in srgb, #0d6efd 22%, var(--card));
}
</style>
</head>
<body>
<header>
<div class="header-title">
{{HEADER_BODY}}
</div>
{{THEME_TOGGLE_HTML}}
</header>
<div class="layout">
<aside class="sidebar" aria-label="Changed files">
<h2>Files ({{FILE_COUNT}})</h2>
{{TREE_HTML}}
<details class="stat-fold">
<summary>git diff --stat</summary>
<pre>{{STAT_BLOCK}}</pre>
</details>
</aside>
<div class="content">
<div class="toolbar">
<button type="button" id="btn-expand-all">Expand all</button>
<button type="button" id="btn-collapse-all">Collapse all</button>
</div>
{{FILES_HTML}}
</div>
</div>
<script>
(function () {
// Theme toggle functionality - default is dark mode
var themeToggle = document.getElementById('theme-toggle');
if (themeToggle) {
var htmlEl = document.documentElement;
// Default to dark mode
var savedTheme = localStorage.getItem('fiona-diff-theme') || 'dark';
htmlEl.setAttribute('data-theme', savedTheme);
themeToggle.addEventListener('click', function () {
var currentTheme = htmlEl.getAttribute('data-theme');
var newTheme = currentTheme === 'dark' ? 'light' : 'dark';
htmlEl.setAttribute('data-theme', newTheme);
localStorage.setItem('fiona-diff-theme', newTheme);
});
}
function openFromHash() {
var id = location.hash ? location.hash.slice(1) : "";
document.querySelectorAll(".file-tree a").forEach(function (a) {
a.classList.toggle("active", a.getAttribute("href") === "#" + id);
});
if (!id) {
return;
}
var el = document.getElementById(id);
if (el && el.tagName === "DETAILS") {
el.open = true;
el.scrollIntoView({ behavior: "smooth", block: "start" });
}
}
document.getElementById("btn-expand-all").addEventListener("click", function () {
document.querySelectorAll("details.file").forEach(function (d) { d.open = true; });
});
document.getElementById("btn-collapse-all").addEventListener("click", function () {
document.querySelectorAll("details.file").forEach(function (d) { d.open = false; });
});
document.querySelectorAll(".file-tree a").forEach(function (a) {
a.addEventListener("click", function () {
var id = a.getAttribute("href").slice(1);
var el = document.getElementById(id);
if (el && el.tagName === "DETAILS") {
el.open = true;
}
});
});
window.addEventListener("hashchange", openFromHash);
openFromHash();
})();
</script>
</body>
</html>
@@ -18,6 +18,18 @@ import re
import yaml import yaml
# Default required fields for wiki pages
DEFAULT_REQUIRED_FIELDS = ["title", "created", "updated", "type"]
# Pattern to match frontmatter delimiters
# This pattern matches --- at start, then any content (including empty), then ---
# Handles: ---\ncontent\n--- or ---\n--- or ---\n\n---
FRONTMATTER_PATTERN = re.compile(r'^---\s*\n([\s\S]*?)(?:\n---\s*($|\n)|---\s*($|\n))')
# Pattern to match a single --- delimiter line
DELIMITER_PATTERN = re.compile(r'^---\s*$', re.MULTILINE)
# ============================================================================= # =============================================================================
# Exceptions # Exceptions
# ============================================================================= # =============================================================================
@@ -30,22 +42,6 @@ class RequiredFieldError(Exception):
"""Exception raised when required frontmatter fields are missing.""" """Exception raised when required frontmatter fields are missing."""
# =============================================================================
# Constants
# =============================================================================
# Default required fields for wiki pages
DEFAULT_REQUIRED_FIELDS = ["title", "created", "updated", "type"]
# Pattern to match frontmatter delimiters
# This pattern matches --- at start, then any content (including empty), then ---
# Handles: ---\ncontent\n--- or ---\n--- or ---\n\n---
FRONTMATTER_PATTERN = re.compile(r'^---\s*\n([\s\S]*?)(?:\n---\s*($|\n)|---\s*($|\n))')
# Pattern to match a single --- delimiter line
DELIMITER_PATTERN = re.compile(r'^---\s*$', re.MULTILINE)
# ============================================================================= # =============================================================================
# Helper Functions # Helper Functions
# ============================================================================= # =============================================================================
@@ -16,8 +16,9 @@ import re
import sys import sys
import typing import typing
from fiona.librarian.md_frontmatter import FrontmatterError, get_file_version
from fiona.logging_config import load_settings, setup_logging from fiona.logging_config import load_settings, setup_logging
from librarian.md_frontmatter import FrontmatterError, get_file_version from fiona.paths import llm_wiki_template
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -40,9 +41,7 @@ SETTINGS_NAME = "fiona-settings.json"
ARTIFACT_DIR_NAMES = frozenset({"build", "dist", "__pycache__"}) ARTIFACT_DIR_NAMES = frozenset({"build", "dist", "__pycache__"})
FILE_TOC_KEYS = frozenset({"min_heading", "max_heading", "sectionize", "ignore_files"}) FILE_TOC_KEYS = frozenset({"min_heading", "max_heading", "sectionize", "ignore_files"})
DEFAULT_SETTINGS_FILE = ( DEFAULT_SETTINGS_FILE = llm_wiki_template() / "fiona-settings.json"
pathlib.Path(__file__).resolve().parent.parent / "llm-wiki" / "fiona-settings.json"
)
def load_toc_settings(settings_file: str | pathlib.Path | None = None) -> dict: def load_toc_settings(settings_file: str | pathlib.Path | None = None) -> dict:
@@ -95,9 +94,12 @@ def depth_setting(settings: dict, key: str) -> int | None:
def slugify(text: str) -> str: def slugify(text: str) -> str:
"""Convert a header text into a GitHub-style anchor.""" """Convert a header text into a GitHub-style anchor (SPEC 7.3.1.1.3).
Each space becomes one hyphen; runs are kept so ``a | b`` is ``a--b``.
"""
slug = re.sub(r"[^\w\s-]", "", text.lower()) slug = re.sub(r"[^\w\s-]", "", text.lower())
return re.sub(r"\s+", "-", slug.strip()) return re.sub(r"\s", "-", slug.strip())
def excluded(text: str, patterns: list[typing.Pattern]) -> bool: def excluded(text: str, patterns: list[typing.Pattern]) -> bool:
@@ -109,7 +111,7 @@ def read_headers(path: pathlib.Path) -> list[tuple[int, str]]:
"""Return (level, text) for every header in a Markdown file, skipping code fences.""" """Return (level, text) for every header in a Markdown file, skipping code fences."""
headers: list[tuple[int, str]] = [] headers: list[tuple[int, str]] = []
in_fence = False in_fence = False
# CURSOR: utf-8-sig drops a leading BOM that would hide the first header. # AI/LLM: utf-8-sig drops a leading BOM that would hide the first header.
for line in path.read_text(encoding="utf-8-sig", errors="replace").splitlines(): for line in path.read_text(encoding="utf-8-sig", errors="replace").splitlines():
if FENCE_RE.match(line): if FENCE_RE.match(line):
in_fence = not in_fence in_fence = not in_fence
@@ -163,15 +165,19 @@ def build_toc(
) -> str: ) -> str:
"""Build a table of contents that nests folders, files and their headers. """Build a table of contents that nests folders, files and their headers.
Generated by AI/LLM: 2026-10-02 - frontmatter name and version for fmlint.
Args: Args:
folder: The folder the table of contents covers. folder: The folder the table of contents covers.
files: Relative md file paths to list (as returned by collect_md). files: Relative md file paths to list (as returned by collect_md).
max_header_depth: Maximum header level to list per file max_header_depth: Maximum header level to list per file
(3 = h1 through h3). None or 0 mean unlimited. (3 = h1 through h3). None or 0 mean unlimited.
""" """
# Add YAML frontmatter # AI/LLM: name and version are required by fmlint on every markdown file.
today = datetime.datetime.now(datetime.timezone.utc).date().isoformat() today = datetime.datetime.now(datetime.UTC).date().isoformat()
frontmatter = f"""--- frontmatter = f"""---
name: Table of Contents
version: 1.0.0
title: Table of Contents title: Table of Contents
created: {today} created: {today}
updated: {today} updated: {today}
@@ -183,7 +189,7 @@ type: toc
for name in files: for name in files:
parts = tuple(name.split("/")) parts = tuple(name.split("/"))
folders = parts[:-1] folders = parts[:-1]
# CURSOR: emit a bullet only for folder levels that differ from the previous entry. # AI/LLM: emit a bullet only for folder levels that differ from the previous entry.
for depth in range(len(folders)): for depth in range(len(folders)):
if folders[: depth + 1] != previous[: depth + 1]: if folders[: depth + 1] != previous[: depth + 1]:
lines.append("{}- {}".format(" " * depth, folders[depth])) lines.append("{}- {}".format(" " * depth, folders[depth]))
@@ -215,7 +221,7 @@ class FileTocError(Exception):
@dataclasses.dataclass(frozen=True) @dataclasses.dataclass(frozen=True)
class FileTocConfig: class FileTocConfig:
"""Parameters for per-file TOC generation (SPEC 4.8.2). """Parameters for per-file TOC generation (SPEC 7.3.1.2).
Attributes: Attributes:
min_heading: Smallest heading level included (never below 2; min_heading: Smallest heading level included (never below 2;
@@ -360,7 +366,7 @@ def merge_marker_config(lines: list[str], base: FileTocConfig) -> FileTocConfig:
"""Apply the existing marker's params over the base config. """Apply the existing marker's params over the base config.
The marker records this TOC's own settings and has priority The marker records this TOC's own settings and has priority
(SPEC 4.8.2): a `sectionized` marker keeps section numbering on (SPEC 7.3.1.2): a `sectionized` marker keeps section numbering on
and the recorded heading range sticks, so regenerations keep the and the recorded heading range sticks, so regenerations keep the
TOC's identity regardless of settings changes. Fields the marker TOC's identity regardless of settings changes. Fields the marker
does not record (bare legacy markers, plain markers without does not record (bare legacy markers, plain markers without
@@ -674,7 +680,7 @@ def _file_toc_star(sectionized: bool) -> None:
def file_toc_main(argv: list[str]) -> None: def file_toc_main(argv: list[str]) -> None:
"""CLI for per-file TOC generation (SPEC 4.8.2).""" """CLI for per-file TOC generation (SPEC 7.3.1.2)."""
parser = argparse.ArgumentParser( parser = argparse.ArgumentParser(
prog="toc_generator file", prog="toc_generator file",
description="Generate an in-file table of contents below the H1 heading.", description="Generate an in-file table of contents below the H1 heading.",
@@ -748,7 +754,7 @@ def main() -> None:
for folder in target_folders(args.directory, patterns, args.recursive_toc): for folder in target_folders(args.directory, patterns, args.recursive_toc):
files = collect_md(folder, patterns, not args.non_recursive, max_depth=max_dir_depth) files = collect_md(folder, patterns, not args.non_recursive, max_depth=max_dir_depth)
if files: if files:
# CURSOR: newline="\n" keeps Unix line endings on every platform. # AI/LLM: newline="\n" keeps Unix line endings on every platform.
with open(folder / TOC_NAME, "w", encoding="utf-8", newline="\n") as toc: with open(folder / TOC_NAME, "w", encoding="utf-8", newline="\n") as toc:
toc.write(build_toc(folder, files, max_header_depth=max_header_depth)) toc.write(build_toc(folder, files, max_header_depth=max_header_depth))
logger.info("wrote %s (%d file(s))", folder / TOC_NAME, len(files)) logger.info("wrote %s (%d file(s))", folder / TOC_NAME, len(files))
+74 -7
View File
@@ -1,4 +1,4 @@
"""Shared logging configuration for all Fiona tools (SPEC 5.1). """Shared logging configuration for all Fiona tools (SPEC 7.17).
Diagnostics go through the standard `logging` module with the level Diagnostics go through the standard `logging` module with the level
resolved from (highest precedence first): resolved from (highest precedence first):
@@ -13,20 +13,25 @@ records go to stderr.
from __future__ import annotations from __future__ import annotations
import ctypes
import json import json
import logging import logging
import os import os
import pathlib import pathlib
import sys import sys
from fiona.paths import llm_wiki_template
# AI/LLM: Windows-only; module-level so PYC001 allows it (SPEC 6.4.2).
if os.name == "nt":
import msvcrt
DEFAULT_LOG_LEVEL = "WARNING" DEFAULT_LOG_LEVEL = "WARNING"
DEFAULT_SETTINGS_FILE = ( DEFAULT_SETTINGS_FILE = llm_wiki_template() / "fiona-settings.json"
pathlib.Path(__file__).resolve().parent.parent / "llm-wiki" / "fiona-settings.json"
)
# Tool package loggers that carry the level explicitly: pymarkdown # Tool package loggers that carry the level explicitly: pymarkdown
# resets the root logger to WARNING mid-run, which would otherwise # resets the root logger to WARNING mid-run, which would otherwise
# silence tool diagnostics (same pattern as run_sanity, see SPEC 4.7.1). # silence tool diagnostics (same pattern as run_sanity, see SPEC 7.17).
TOOL_LOGGER_NAMES = ("fiona", "librarian", "mdoctor", "python_ruff", "wiki") TOOL_LOGGER_NAMES = ("fiona",)
LOG_FORMAT = "%(levelname)s %(name)s: %(message)s" LOG_FORMAT = "%(levelname)s %(name)s: %(message)s"
FILE_LOG_FORMAT = "%(asctime)s %(levelname)s %(name)s: %(message)s" FILE_LOG_FORMAT = "%(asctime)s %(levelname)s %(name)s: %(message)s"
@@ -85,6 +90,68 @@ def load_log_level(settings: dict | None) -> str:
return candidate.upper() return candidate.upper()
def _open_shared_append(path: str, encoding: str):
"""Open a log file that Windows can delete while the handle is open.
Generated by AI/LLM: 2026-09-29 - the default open() omits
FILE_SHARE_DELETE, so a temp directory that still has fiona.log
open cannot be removed.
Args:
path: Log file path.
encoding: Text encoding for the stream.
Returns:
A text file object opened for append.
"""
if os.name != "nt":
return open(path, "a", encoding=encoding)
kernel32 = ctypes.WinDLL("kernel32", use_last_error=True)
kernel32.CreateFileW.argtypes = [
ctypes.c_wchar_p,
ctypes.c_uint32,
ctypes.c_uint32,
ctypes.c_void_p,
ctypes.c_uint32,
ctypes.c_uint32,
ctypes.c_void_p,
]
kernel32.CreateFileW.restype = ctypes.c_void_p
kernel32.SetFilePointer.argtypes = [
ctypes.c_void_p,
ctypes.c_long,
ctypes.c_void_p,
ctypes.c_uint32,
]
kernel32.SetFilePointer.restype = ctypes.c_uint32
handle = kernel32.CreateFileW(
path,
0x40000000, # GENERIC_WRITE
0x1 | 0x2 | 0x4, # FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE
None,
4, # OPEN_ALWAYS
0x80, # FILE_ATTRIBUTE_NORMAL
None,
)
invalid = ctypes.c_void_p(-1).value
if handle in (None, invalid):
raise OSError(ctypes.get_last_error(), f"Cannot open log file {path}")
kernel32.SetFilePointer(handle, 0, None, 2) # FILE_END
fd = msvcrt.open_osfhandle(handle, os.O_APPEND)
return os.fdopen(fd, "a", encoding=encoding, closefd=True)
class _SharedFileHandler(logging.FileHandler):
"""File log handler that does not block deletion of its directory.
Generated by AI/LLM: 2026-09-29 - Windows tests and temp wikis remove
the working directory while fiona.log is still attached.
"""
def _open(self):
return _open_shared_append(self.baseFilename, self.encoding or "utf-8")
def _remove_file_handler() -> None: def _remove_file_handler() -> None:
"""Detach (and close) the managed file log handler, if any.""" """Detach (and close) the managed file log handler, if any."""
for handler in list(logging.getLogger().handlers): for handler in list(logging.getLogger().handlers):
@@ -117,7 +184,7 @@ def _configure_file_log(fiona_settings: dict, console_level: str) -> None:
) )
file_level = console_level file_level = console_level
try: try:
handler = logging.FileHandler(location, encoding="utf-8") handler = _SharedFileHandler(location, encoding="utf-8")
except OSError as error: except OSError as error:
logging.getLogger(__name__).warning("Cannot open log file %s: %s", location, error) logging.getLogger(__name__).warning("Cannot open log file %s: %s", location, error)
return return
@@ -4,22 +4,54 @@ MDoctor - A tool for diagnosing and managing the LLM wiki.
from __future__ import annotations from __future__ import annotations
import argparse import argparse
import contextlib
import json import json
import logging import logging
import os import os
import pathlib import pathlib
import sys import sys
import tempfile
from typing import Any, ClassVar from typing import Any, ClassVar
from pymarkdown.api import PyMarkdownApi, PyMarkdownApiException from pymarkdown.api import PyMarkdownApi, PyMarkdownApiException
from customlinter.base import BaseLinter, LintViolation from fiona.customlinter.base import BaseLinter, LintViolation
from fiona.librarian.librarian import Librarian
from fiona.logging_config import setup_logging from fiona.logging_config import setup_logging
from librarian.librarian import Librarian from fiona.paths import llm_wiki_template
DEFAULT_SETTINGS_FILE = ( DEFAULT_SETTINGS_FILE = llm_wiki_template() / "fiona-settings.json"
pathlib.Path(__file__).resolve().parent.parent / "llm-wiki" / "fiona-settings.json" # AI/LLM: MD013 defaults used when fiona-settings.json omits the keys.
) MD013_DEFAULTS: dict[str, int | bool] = {
"line_length": 100,
"tables": False,
}
@contextlib.contextmanager
def _pymarkdown_reopenable_temp_files():
"""Keep pymarkdown temp files reopenable on Windows.
Generated by AI/LLM: 2026-09-29 - pymarkdown closes a NamedTemporaryFile
and opens that name again. Windows delete-on-close denies the second open.
Yields:
None.
"""
if os.name != "nt":
yield
return
original = tempfile.NamedTemporaryFile
def _named_temporary_file(*args, **kwargs):
kwargs["delete"] = False
return original(*args, **kwargs)
tempfile.NamedTemporaryFile = _named_temporary_file
try:
yield
finally:
tempfile.NamedTemporaryFile = original
class ProgrammaticLintViolation: class ProgrammaticLintViolation:
@@ -37,20 +69,20 @@ class ProgrammaticLintViolation:
class MDoctor(BaseLinter): class MDoctor(BaseLinter):
"""Core class for diagnosing wiki content and operations.""" """Core class for diagnosing wiki content and operations."""
# File extensions to scan # File extensions to scan
file_extensions: ClassVar[list[str]] = [".md"] file_extensions: ClassVar[list[str]] = [".md"]
# Settings key path in fiona-settings.json # Settings key path in fiona-settings.json
settings_key_path: ClassVar[list[str]] = ["markdown", "pymarkdown"] settings_key_path: ClassVar[list[str]] = ["markdown", "pymarkdown"]
# Human-readable linter name for report headers # Human-readable linter name for report headers
linter_name: ClassVar[str] = "PyMarkdownLint" linter_name: ClassVar[str] = "PyMarkdownLint"
# Violation codes - pymarkdown rules are dynamic, so we use an empty dict # Violation codes - pymarkdown rules are dynamic, so we use an empty dict
# and populate descriptions from the API # and populate descriptions from the API
VIOLATION_CODES: ClassVar[dict[str, dict[str, Any]]] = {} VIOLATION_CODES: ClassVar[dict[str, dict[str, Any]]] = {}
def __init__(self, settings_file: str | pathlib.Path | None = None) -> None: def __init__(self, settings_file: str | pathlib.Path | None = None) -> None:
self.librarian = Librarian() self.librarian = Librarian()
self.api = PyMarkdownApi() self.api = PyMarkdownApi()
@@ -61,10 +93,17 @@ class MDoctor(BaseLinter):
settings_file = DEFAULT_SETTINGS_FILE settings_file = DEFAULT_SETTINGS_FILE
self.settings_file = pathlib.Path(settings_file) self.settings_file = pathlib.Path(settings_file)
self.respect_gitignore = False self.respect_gitignore = False
self._md013_tables = bool(MD013_DEFAULTS["tables"])
self._table_lines: dict[pathlib.Path, set[int]] = {}
self._gitignore_dirs: dict[pathlib.Path, set[str]] = {}
# Call parent init # Call parent init
super().__init__(settings_file=self.settings_file) super().__init__(settings_file=self.settings_file)
# AI/LLM: seed MD013 before the settings file so omitted keys keep these defaults.
for name, value in MD013_DEFAULTS.items():
self._set_plugin_property("md013", name, value)
if self.settings_file.exists(): if self.settings_file.exists():
with open(self.settings_file, encoding="utf-8") as f: with open(self.settings_file, encoding="utf-8") as f:
settings = json.load(f) settings = json.load(f)
@@ -90,34 +129,54 @@ class MDoctor(BaseLinter):
if rule_id == "extensions" or not isinstance(properties, dict): if rule_id == "extensions" or not isinstance(properties, dict):
continue continue
for name, value in properties.items(): for name, value in properties.items():
property_name = f"plugins.{rule_id}.{name}" self._set_plugin_property(rule_id, name, value)
if isinstance(value, bool):
self.api.set_boolean_property(property_name, value) def _set_plugin_property(self, rule_id: str, name: str, value: bool | int | str) -> None:
elif isinstance(value, int): """Set one pymarkdown plugin property from a JSON value.
self.api.set_integer_property(property_name, value)
else: Generated by AI/LLM: 2026-09-29 - one setter for MD013 defaults and
self.api.set_string_property(property_name, value) fiona-settings.json overrides. Bool is checked before int because
bool is a subclass of int.
Args:
rule_id: PyMarkdown rule id, such as ``md013``.
name: Property name under that rule.
value: Boolean, integer, or string property value.
"""
property_name = f"plugins.{rule_id}.{name}"
if isinstance(value, bool):
self.api.set_boolean_property(property_name, value)
elif isinstance(value, int):
self.api.set_integer_property(property_name, value)
else:
self.api.set_string_property(property_name, value)
if rule_id == "md013" and name == "tables" and isinstance(value, bool):
self._md013_tables = value
def lint_file(self, filepath: pathlib.Path | str) -> list[LintViolation]: def lint_file(self, filepath: pathlib.Path | str) -> list[LintViolation]:
"""Lint a single markdown file using PyMarkdownApi. """Lint a single markdown file using PyMarkdownApi.
Args: Args:
filepath: Path to the markdown file to lint. filepath: Path to the markdown file to lint.
Returns: Returns:
List of LintViolation objects. List of LintViolation objects.
""" """
resolved_filepath = pathlib.Path(filepath).resolve() resolved_filepath = pathlib.Path(filepath).resolve()
if self._ignored_by_gitignore(resolved_filepath.parent, resolved_filepath):
return []
# Use pymarkdown API to scan the file # Use pymarkdown API to scan the file
scan_result = self.api.scan_path( scan_result = self.api.scan_path(
path_to_scan=str(resolved_filepath), path_to_scan=str(resolved_filepath),
recurse_if_directory=False, recurse_if_directory=False,
respect_gitignore=self.respect_gitignore **self._pymarkdown_path_options(resolved_filepath)
) )
violations = [] violations = []
for failure in scan_result.scan_failures: for failure in scan_result.scan_failures:
if self._skip_table_md013(resolved_filepath, failure.line_number, failure.rule_id):
continue
violation = LintViolation( violation = LintViolation(
filepath=resolved_filepath, filepath=resolved_filepath,
line=failure.line_number, line=failure.line_number,
@@ -128,7 +187,7 @@ class MDoctor(BaseLinter):
severity="error" severity="error"
) )
violations.append(violation) violations.append(violation)
return violations return violations
def list_all_md_files(self, directory: str|pathlib.Path) -> list: def list_all_md_files(self, directory: str|pathlib.Path) -> list:
@@ -141,11 +200,11 @@ class MDoctor(BaseLinter):
# Resolve the filepath to absolute path for scanning # Resolve the filepath to absolute path for scanning
resolved_filepath = pathlib.Path(filepath).resolve() resolved_filepath = pathlib.Path(filepath).resolve()
scan_result = api.scan_path( scan_result = api.scan_path(
path_to_scan=str(resolved_filepath), path_to_scan=str(resolved_filepath),
recurse_if_directory=True, recurse_if_directory=True,
respect_gitignore=self.respect_gitignore **self._pymarkdown_path_options(resolved_filepath)
) )
violations = [] violations = []
for failure in scan_result.scan_failures: for failure in scan_result.scan_failures:
@@ -158,17 +217,21 @@ class MDoctor(BaseLinter):
base_path = resolved_filepath.parent base_path = resolved_filepath.parent
else: else:
base_path = resolved_filepath base_path = resolved_filepath
relative_path = str(scan_file_path.relative_to(base_path)) relative_path = str(scan_file_path.relative_to(base_path))
# If the scan_file is the same as the filepath being scanned, return "." # If the scan_file is the same as the filepath being scanned, return "."
if scan_file_path == resolved_filepath: if scan_file_path == resolved_filepath:
relative_path = "." relative_path = "."
except ValueError: except ValueError:
# If relative path fails, use the absolute path # If relative path fails, use the absolute path
relative_path = str(scan_file_path) relative_path = str(scan_file_path)
if self._ignored_by_gitignore(resolved_filepath, scan_file_path):
continue
if self._skip_table_md013(scan_file_path, failure.line_number, failure.rule_id):
continue
violations.append({ violations.append({
"scan_file": relative_path, "scan_file": relative_path,
"line": failure.line_number, "line": failure.line_number,
@@ -181,6 +244,182 @@ class MDoctor(BaseLinter):
return violations return violations
def _skip_table_md013(self, path: pathlib.Path, line_number: int, rule_id: str) -> bool:
"""Drop MD013 on an unfenced pipe row when tables are ignored.
Generated by AI/LLM: 2026-09-29 - pymarkdown does not tokenize pipe
rows while markdown-tables stays disabled, so plugins.md013.tables
never sees them.
Args:
path: Markdown file that produced the violation.
line_number: 1-based line of the violation.
rule_id: PyMarkdown rule id.
Returns:
True when this hit should be omitted from the report.
"""
if self._md013_tables or rule_id != "MD013":
return False
return line_number in self._table_line_numbers(path)
def _table_line_numbers(self, path: pathlib.Path) -> set[int]:
"""Return 1-based line numbers of unfenced GFM table rows.
Generated by AI/LLM: 2026-09-29 - identify pipe rows MD013 should skip.
Args:
path: Markdown file to read.
Returns:
Line numbers whose stripped text starts with `|` and contains
another `|`, outside fenced code.
"""
resolved = path.resolve()
cached = self._table_lines.get(resolved)
if cached is not None:
return cached
numbers: set[int] = set()
in_fence = False
try:
text = resolved.read_text(encoding="utf-8")
except OSError:
self._table_lines[resolved] = numbers
return numbers
for number, raw in enumerate(text.splitlines(), start=1):
stripped = raw.lstrip()
if stripped.startswith(("```", "~~~")):
in_fence = not in_fence
continue
if in_fence:
continue
body = raw.strip()
if body.startswith("|") and body.count("|") >= 2:
numbers.add(number)
self._table_lines[resolved] = numbers
return numbers
def _ignored_by_gitignore(self, scan_root: pathlib.Path, file_path: pathlib.Path) -> bool:
"""Skip a file that sits under a directory named in .gitignore.
Generated by AI/LLM: 2026-09-29 - pymarkdown only applies
.gitignore inside a git checkout. Sanity still has to skip
virtualenv markdown when the folder is not a repository.
Args:
scan_root: Directory passed to the scan, or a file's parent.
file_path: Markdown file to test.
Returns:
True when respect_gitignore is on and a path segment matches
a directory entry in the nearest .gitignore.
"""
if not self.respect_gitignore:
return False
root = self._gitignore_root(scan_root)
if root is None:
return False
try:
relative = file_path.resolve().relative_to(root)
except ValueError:
return False
return any(part in self._gitignore_dir_names(root) for part in relative.parts)
def _gitignore_root(self, start: pathlib.Path) -> pathlib.Path | None:
"""Find the nearest .gitignore at or above start.
Generated by AI/LLM: 2026-09-29 - locate the ignore file for a scan.
Args:
start: File or directory being scanned.
Returns:
Directory that contains .gitignore, or None.
"""
directory = start if start.is_dir() else start.parent
for candidate in [directory, *directory.parents]:
if (candidate / ".gitignore").is_file():
return candidate
return None
def _gitignore_dir_names(self, root: pathlib.Path) -> set[str]:
"""Return directory names listed in root/.gitignore.
Generated by AI/LLM: 2026-09-29 - read directory entries such as
`.venvs/` without invoking git.
Args:
root: Directory that may contain .gitignore.
Returns:
Names with no glob characters. Comments and negations are
ignored.
"""
resolved = root.resolve()
cached = self._gitignore_dirs.get(resolved)
if cached is not None:
return cached
names: set[str] = set()
gitignore = resolved / ".gitignore"
try:
lines = gitignore.read_text(encoding="utf-8").splitlines()
except OSError:
self._gitignore_dirs[resolved] = names
return names
for line in lines:
entry = line.strip()
if not entry or entry.startswith(("#", "!")):
continue
name = entry.strip("/")
if not name or "/" in name or any(char in name for char in "*?["):
continue
names.add(name)
self._gitignore_dirs[resolved] = names
return names
def _pymarkdown_path_options(self, path: pathlib.Path) -> dict[str, Any]:
"""Keep PyMarkdown from starting git.
Generated by AI/LLM: 2026-09-30 - directory excludes replace git check-ignore.
Args:
path: File or directory about to be scanned or fixed.
Returns:
Keyword arguments for ``scan_path`` and ``fix_path``.
"""
return {
"respect_gitignore": False,
"exclude_patterns": self._exclude_patterns(path),
}
def _exclude_patterns(self, path: pathlib.Path) -> list[str] | None:
"""Return directory globs PyMarkdown can apply without git.
Generated by AI/LLM: 2026-09-30 - prune ignored directories during the walk.
Patterns are relative to the working directory. A scan outside
that directory gets no patterns, because the matcher rejects
those paths. The post-scan directory check still applies.
Args:
path: File or directory about to be scanned or fixed.
Returns:
Glob list, or None when the path is outside the working directory.
"""
root = path if path.is_dir() else path.parent
try:
root.resolve().relative_to(pathlib.Path.cwd().resolve())
except ValueError:
return None
names = {".git"}
if self.respect_gitignore:
git_root = self._gitignore_root(root)
if git_root is not None:
names.update(self._gitignore_dir_names(git_root))
return [f"{name}/" for name in sorted(names)]
def fix_file_programmatically(self, filepath: str) -> list[str]: def fix_file_programmatically(self, filepath: str) -> list[str]:
"""Apply PyMarkdown auto-fixes to a markdown file or directory. """Apply PyMarkdown auto-fixes to a markdown file or directory.
@@ -197,27 +436,28 @@ class MDoctor(BaseLinter):
api = self.api api = self.api
resolved_filepath = pathlib.Path(filepath).resolve() resolved_filepath = pathlib.Path(filepath).resolve()
fix_result = api.fix_path( with _pymarkdown_reopenable_temp_files():
path_to_scan=str(resolved_filepath), fix_result = api.fix_path(
recurse_if_directory=True, path_to_scan=str(resolved_filepath),
respect_gitignore=self.respect_gitignore recurse_if_directory=True,
) **self._pymarkdown_path_options(resolved_filepath)
)
return [str(pathlib.Path(f).resolve()) for f in fix_result.files_fixed] return [str(pathlib.Path(f).resolve()) for f in fix_result.files_fixed]
def compress_violations( def compress_violations(
self, violations: list, put_extra: bool = False, target: str | pathlib.Path | None = None self, violations: list, put_extra: bool = False, target: str | pathlib.Path | None = None
): ):
"""Compress violations using the shared BaseLinter format. """Compress violations using the shared BaseLinter format.
For backward compatibility, this method accepts: For backward compatibility, this method accepts:
- List of violation dicts from scan_file_programmatically (old format) - List of violation dicts from scan_file_programmatically (old format)
- List of LintViolation objects from lint_directory (new format) - List of LintViolation objects from lint_directory (new format)
Args: Args:
violations: List of violation dicts or LintViolation objects. violations: List of violation dicts or LintViolation objects.
put_extra: Whether to include extra information in violation messages (old format only). put_extra: Whether to include extra information in violation messages (old format only).
target: Target directory path. target: Target directory path.
Returns: Returns:
Dictionary with compressed violations in shared format. Dictionary with compressed violations in shared format.
""" """
@@ -232,7 +472,7 @@ class MDoctor(BaseLinter):
# Construct the message with or without extra # Construct the message with or without extra
extra = f' {violation["extra"]}' if put_extra and violation.get("extra") else '' extra = f' {violation["extra"]}' if put_extra and violation.get("extra") else ''
message = f'{violation["line"]}:{violation["column"]} {violation["rule_id"]}{extra}' message = f'{violation["line"]}:{violation["column"]} {violation["rule_id"]}{extra}'
# Create a LintViolation object # Create a LintViolation object
# Note: filepath will be set to the scan_file, but we need to resolve it # Note: filepath will be set to the scan_file, but we need to resolve it
# For now, we'll use the scan_file as a string identifier # For now, we'll use the scan_file as a string identifier
@@ -246,24 +486,24 @@ class MDoctor(BaseLinter):
severity="error" severity="error"
) )
lint_violations.append(lint_violation) lint_violations.append(lint_violation)
# Store the rule info in VIOLATION_CODES for the base class to use # Store the rule info in VIOLATION_CODES for the base class to use
if violation["rule_id"] not in self.VIOLATION_CODES: if violation["rule_id"] not in self.VIOLATION_CODES:
self.VIOLATION_CODES[violation["rule_id"]] = { self.VIOLATION_CODES[violation["rule_id"]] = {
"rule_name": violation["rule_name"], "rule_name": violation["rule_name"],
"description": violation["description"] "description": violation["description"]
} }
# Use the base class compress_violations method # Use the base class compress_violations method
return super().compress_violations(lint_violations, target) return super().compress_violations(lint_violations, target)
def format_violations(self, compressed_violations: dict) -> str: def format_violations(self, compressed_violations: dict) -> str:
"""Format violations using the shared BaseLinter format. """Format violations using the shared BaseLinter format.
Args: Args:
compressed_violations: Compressed violations dict from compress_violations. compressed_violations: Compressed violations dict from compress_violations.
Returns: Returns:
Formatted string with violation report. Formatted string with violation report.
""" """
@@ -281,10 +521,10 @@ class MDoctor(BaseLinter):
Formatted string with violation report Formatted string with violation report
""" """
directory_path = pathlib.Path(directory).resolve() directory_path = pathlib.Path(directory).resolve()
# Use the standard lint_directory from base class # Use the standard lint_directory from base class
violations = self.lint_directory(directory_path) violations = self.lint_directory(directory_path)
# Compress and format using base class methods # Compress and format using base class methods
compressed_violations = self.compress_violations(violations, target=directory_path) compressed_violations = self.compress_violations(violations, target=directory_path)
return self.format_compressed_violations(compressed_violations) return self.format_compressed_violations(compressed_violations)
+21
View File
@@ -0,0 +1,21 @@
"""Filesystem locations for packaged Fiona data.
Generated by AI/LLM: 2026-09-29 - one resolver for the llm-wiki template.
"""
from __future__ import annotations
import importlib.resources
import pathlib
def llm_wiki_template() -> pathlib.Path:
"""Return the packaged llm-wiki template directory.
Generated by AI/LLM: 2026-09-29 - load package data with importlib.resources.
Returns:
Filesystem path of ``fiona/data/llm_wiki``.
"""
resource = importlib.resources.files("fiona").joinpath("data", "llm_wiki")
return pathlib.Path(str(resource))
@@ -8,7 +8,7 @@ from typing import Any, ClassVar
import ruff import ruff
from customlinter.base import BaseLinter, LintViolation from fiona.customlinter.base import BaseLinter, LintViolation
from fiona.logging_config import setup_logging from fiona.logging_config import setup_logging
+49 -25
View File
@@ -1,7 +1,7 @@
"""Sanity Check Runner for Fiona-Assistant project. """Sanity Check Runner for Fiona-Assistant project.
Runs its checks through the shared SanityRunner engine Runs its checks through the shared SanityRunner engine
(src/sanity_runner, SPEC 4.9.5): concurrent execution, per-check (src/sanity_runner, SPEC 7.14.1): concurrent execution, per-check
timings, and the common report format. This module keeps only the timings, and the common report format. This module keeps only the
repository-specific checks, their summaries, and the report repository-specific checks, their summaries, and the report
presentation. presentation.
@@ -17,19 +17,17 @@ import time
from collections.abc import Callable, Sequence from collections.abc import Callable, Sequence
from pathlib import Path from pathlib import Path
import sanity_runner from fiona import sanity_runner
from customlinter.fmlint import FrontmatterLinter from fiona.customlinter.fmlint import FrontmatterLinter
from customlinter.jsonlint import JsonLinter from fiona.customlinter.jsonlint import JsonLinter
from customlinter.pycustomlint import PythonCustomLinter from fiona.customlinter.pycustomlint import PythonCustomLinter
from mdoctor.mdoctor import MDoctor from fiona.mdoctor.mdoctor import MDoctor
from python_ruff.python_ruff import PythonRuff from fiona.python_ruff.python_ruff import PythonRuff
from run_tests import TestRunner from fiona.run_tests import TestRunner
# Configure logging # Configure logging
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
SETTINGS_FILE = Path(__file__).resolve().parent / "settings_dev.json"
# Progress display defaults (overridable via the "progress" settings # Progress display defaults (overridable via the "progress" settings
# section, see load_progress_settings) # section, see load_progress_settings)
PROGRESS_BAR_WIDTH = 15 PROGRESS_BAR_WIDTH = 15
@@ -37,6 +35,22 @@ DEFAULT_PROGRESS_INTERVAL_SECONDS = 3.0
DEFAULT_EXPECTED_SECONDS = 30.0 DEFAULT_EXPECTED_SECONDS = 30.0
def dev_settings_file() -> Path:
"""Return scripts/settings_dev.json from the checkout that contains this file.
Generated by AI/LLM: 2026-09-29 - development settings live outside the package.
Returns:
The settings file when this module sits in a checkout, otherwise a
path relative to the current working directory.
"""
for parent in Path(__file__).resolve().parents:
candidate = parent / "scripts" / "settings_dev.json"
if candidate.is_file():
return candidate
return Path("scripts") / "settings_dev.json"
def configure_logging() -> None: def configure_logging() -> None:
"""Configure logging from the LOG_LEVEL environment variable. """Configure logging from the LOG_LEVEL environment variable.
@@ -57,15 +71,16 @@ def configure_logging() -> None:
# pymarkdown resets the root logger to WARNING during lint scans, # pymarkdown resets the root logger to WARNING during lint scans,
# which would silence the runner's debug output mid-run; the module # which would silence the runner's debug output mid-run; the module
# loggers therefore also carry the level explicitly # loggers therefore also carry the level explicitly
for logger_name in (__name__, "run_sanity", "run_tests"): for logger_name in (__name__, "fiona.run_tests"):
logging.getLogger(logger_name).setLevel(level) logging.getLogger(logger_name).setLevel(level)
def load_required_coverage(settings_file: str | Path = SETTINGS_FILE) -> float: def load_required_coverage(settings_file: str | Path | None = None) -> float:
"""Load the required test coverage percentage from the settings file. """Load the required test coverage percentage from the settings file.
Args: Args:
settings_file: Path to the JSON settings file. settings_file: Path to the JSON settings file. The dev settings
file is used when this is omitted.
Returns: Returns:
Required coverage percentage for pytests. Required coverage percentage for pytests.
@@ -74,16 +89,19 @@ def load_required_coverage(settings_file: str | Path = SETTINGS_FILE) -> float:
KeyError: If the settings file has no "required-coverage" entry. KeyError: If the settings file has no "required-coverage" entry.
json.JSONDecodeError: If the settings file is not valid JSON. json.JSONDecodeError: If the settings file is not valid JSON.
""" """
if settings_file is None:
settings_file = dev_settings_file()
with open(settings_file, encoding="utf-8") as f: with open(settings_file, encoding="utf-8") as f:
settings = json.load(f) settings = json.load(f)
return float(settings["pytests"]["required-coverage"]) return float(settings["pytests"]["required-coverage"])
def load_progress_settings(settings_file: str | Path = SETTINGS_FILE) -> dict: def load_progress_settings(settings_file: str | Path | None = None) -> dict:
"""Load the progress display settings from the settings file. """Load the progress display settings from the settings file.
Args: Args:
settings_file: Path to the JSON settings file. settings_file: Path to the JSON settings file. The dev settings
file is used when this is omitted.
Returns: Returns:
dict: The display settings with: dict: The display settings with:
@@ -92,8 +110,13 @@ def load_progress_settings(settings_file: str | Path = SETTINGS_FILE) -> dict:
Each falls back to its default when the "progress" section or Each falls back to its default when the "progress" section or
the individual key is absent. the individual key is absent.
""" """
with open(settings_file, encoding="utf-8") as f: if settings_file is None:
settings = json.load(f) settings_file = dev_settings_file()
try:
with open(settings_file, encoding="utf-8") as f:
settings = json.load(f)
except FileNotFoundError:
settings = {}
progress = settings.get("progress", {}) progress = settings.get("progress", {})
return { return {
"interval_seconds": float( "interval_seconds": float(
@@ -261,7 +284,7 @@ def run_pytests():
(or the raw message when no traceback details exist) (or the raw message when no traceback details exist)
""" """
runner = TestRunner() runner = TestRunner()
result = runner.run_pytest("src/tests", verbose=True, capture_output=True, with_coverage=True) result = runner.run_pytest("tests", verbose=True, capture_output=True, with_coverage=True)
required_coverage = load_required_coverage() required_coverage = load_required_coverage()
coverage_percent = result.get("coverage_percent") coverage_percent = result.get("coverage_percent")
@@ -293,7 +316,7 @@ def run_pytests():
unreported_failures, unreported_failures,
) )
lines = ["Pytest Results", "TARGET: src/tests", ""] lines = ["Pytest Results", "TARGET: tests", ""]
lines.append(f"Tests: {result['passed_count']}/{result['total_count']} passed") lines.append(f"Tests: {result['passed_count']}/{result['total_count']} passed")
for file_name in sorted(file_results): for file_name in sorted(file_results):
counts = file_results[file_name] counts = file_results[file_name]
@@ -387,8 +410,9 @@ def run_ruff():
def run_pycustomlint(): def run_pycustomlint():
"""Run the Python custom linter programmatically using PythonCustomLinter. """Run the Python custom linter programmatically using PythonCustomLinter.
Scans Python files in the src tree for custom rule violations, Scans Python files in the src tree for custom rule violations:
currently inline imports (project agent rule 4.1, python-tdd skill). inline imports (project agent rule 4.1, python-tdd skill) and
module-level assignments after the first function or class (SPEC 6.4.3).
Returns: Returns:
dict: A dictionary containing the lint results with: dict: A dictionary containing the lint results with:
@@ -396,7 +420,7 @@ def run_pycustomlint():
- violation_count: number of violations found - violation_count: number of violations found
- stdout: AI-friendly report in the unified linter format - stdout: AI-friendly report in the unified linter format
(Violations header, TARGET, DEFINITIONS, FILE, SUMMARY; (Violations header, TARGET, DEFINITIONS, FILE, SUMMARY;
see SPEC 4.9.3) see SPEC 6.1)
""" """
linter = PythonCustomLinter() linter = PythonCustomLinter()
@@ -429,7 +453,7 @@ def run_frontmatter_lint():
linter = FrontmatterLinter() linter = FrontmatterLinter()
# Lint markdown files in llm-wiki directory only (per project agent rule 10.1) # Lint markdown files in llm-wiki directory only (per project agent rule 10.1)
llm_wiki_path = "./src/llm-wiki" llm_wiki_path = "./src/fiona/data/llm_wiki"
violations_dict = linter.lint_directory(llm_wiki_path) violations_dict = linter.lint_directory(llm_wiki_path)
# Get violation count from meta # Get violation count from meta
@@ -602,7 +626,7 @@ def coverage_met(check_result: dict) -> bool:
def format_duration(seconds: float) -> str: def format_duration(seconds: float) -> str:
"""Format a duration in seconds as zero-padded MM:SS. """Format a duration in seconds as zero-padded MM:SS.
Delegates to the shared SanityRunner engine (SPEC 4.9.5). Delegates to the shared SanityRunner engine (SPEC 7.14.1).
Args: Args:
seconds: Duration in seconds. seconds: Duration in seconds.
@@ -616,7 +640,7 @@ def format_duration(seconds: float) -> str:
def format_check_summary(check_name: str, check_result: dict) -> str: def format_check_summary(check_name: str, check_result: dict) -> str:
"""Format a single check result as a human/AI-friendly line. """Format a single check result as a human/AI-friendly line.
Delegates to the shared SanityRunner engine (SPEC 4.9.5); the Delegates to the shared SanityRunner engine (SPEC 7.14.1); the
pytests and coverage summaries plug in as formatters. pytests and coverage summaries plug in as formatters.
Args: Args:
+46 -13
View File
@@ -22,6 +22,8 @@ logger = logging.getLogger(__name__)
class TestRunner: class TestRunner:
"""A test runner for Fiona tools that can be used both manually and in CI.""" """A test runner for Fiona tools that can be used both manually and in CI."""
__test__ = False
def __init__(self, base_dir: str | None = None): def __init__(self, base_dir: str | None = None):
""" """
Initialize the test runner. Initialize the test runner.
@@ -450,7 +452,7 @@ class TestRunner:
Args: Args:
test_path: Path to the test file or directory test_path: Path to the test file or directory
verbose: Whether to run in verbose mode verbose: Whether to run in verbose mode
capture_output: Whether to suppress the pytest output (for CI) capture_output: When True, hide pytest stdout/stderr (default False)
with_coverage: Whether to run with coverage (default: True) with_coverage: Whether to run with coverage (default: True)
ignore: Pattern to ignore (e.g., "test_sanity.py") ignore: Pattern to ignore (e.g., "test_sanity.py")
@@ -464,7 +466,7 @@ class TestRunner:
if verbose: if verbose:
cmd.append("-v") cmd.append("-v")
if with_coverage: if with_coverage:
cmd.extend(["--cov=src", "--cov-config=.coveragerc", "--cov-report=term"]) cmd.extend(["--cov=src/fiona", "--cov-report=term"])
if ignore: if ignore:
cmd.extend(["--ignore-glob", f"*{ignore}*"]) cmd.extend(["--ignore-glob", f"*{ignore}*"])
@@ -477,13 +479,14 @@ class TestRunner:
# environment variables (PY_COLORS, FORCE_COLOR) would break them # environment variables (PY_COLORS, FORCE_COLOR) would break them
cmd.append("--color=no") cmd.append("--color=no")
# Set PYTHONPATH to include src directory for customlinter imports # Set PYTHONPATH to include src so the fiona package resolves.
# AI/LLM: 2026-09-30 - Windows joins PATH entries with os.pathsep.
env = os.environ.copy() env = os.environ.copy()
src_dir = os.path.abspath(os.path.dirname(__file__)) src_dir = str(pathlib.Path(__file__).resolve().parent.parent)
if os.path.exists(src_dir): if os.path.exists(src_dir):
current_pythonpath = env.get("PYTHONPATH", "") current_pythonpath = env.get("PYTHONPATH", "")
if current_pythonpath: if current_pythonpath:
env["PYTHONPATH"] = f"{src_dir}:{current_pythonpath}" env["PYTHONPATH"] = f"{src_dir}{os.pathsep}{current_pythonpath}"
else: else:
env["PYTHONPATH"] = src_dir env["PYTHONPATH"] = src_dir
@@ -491,7 +494,10 @@ class TestRunner:
completed = subprocess.run(cmd, capture_output=True, text=True, check=False, env=env) completed = subprocess.run(cmd, capture_output=True, text=True, check=False, env=env)
if not capture_output: if not capture_output:
print(completed.stdout) if completed.stdout:
print(completed.stdout, end="" if completed.stdout.endswith("\n") else "\n")
if completed.stderr:
print(completed.stderr, end="" if completed.stderr.endswith("\n") else "\n", file=sys.stderr)
logger.debug("pytest command: %s", " ".join(str(part) for part in cmd)) logger.debug("pytest command: %s", " ".join(str(part) for part in cmd))
logger.debug("pytest return code: %s", completed.returncode) logger.debug("pytest return code: %s", completed.returncode)
@@ -572,7 +578,7 @@ class TestRunner:
True if all tests passed, False otherwise True if all tests passed, False otherwise
""" """
test_paths = [ test_paths = [
"src/tests" "tests"
] ]
all_passed = True all_passed = True
@@ -603,29 +609,56 @@ def main():
"""Main function for command-line execution.""" """Main function for command-line execution."""
parser = argparse.ArgumentParser(description="Run Fiona tools tests.") parser = argparse.ArgumentParser(description="Run Fiona tools tests.")
parser.add_argument("--verbose", action="store_true", help="Run tests in verbose mode") parser.add_argument("--verbose", action="store_true", help="Run tests in verbose mode")
parser.add_argument("--ci", action="store_true", help="Run in CI mode (capture output)") parser.add_argument(
"--ci",
action="store_true",
help="Non-interactive CI mode: verbose pytest with full output to the log",
)
parser.add_argument(
"--non-interactive",
action="store_true",
help="Alias for --ci",
)
parser.add_argument("--test", help="Run specific test file or directory") parser.add_argument("--test", help="Run specific test file or directory")
parser.add_argument("--skip-cov", action="store_true", help="Skip coverage") parser.add_argument("--skip-cov", action="store_true", help="Skip coverage")
args = parser.parse_args() args = parser.parse_args()
# AI/LLM: use `is True` — bool(MagicMock()) is True in unit tests.
ci_mode = args.ci is True or args.non_interactive is True
verbose = args.verbose is True or ci_mode
capture_output = False
# Initialize test runner # Initialize test runner
runner = TestRunner() runner = TestRunner()
# Determine if we should run with coverage # Determine if we should run with coverage
with_coverage = not args.skip_cov with_coverage = args.skip_cov is not True
def _print_failures() -> None:
"""Echo parsed failure details after pytest output (CI-friendly)."""
for detail in runner.test_results.values():
for entry in detail.get("failed_tests") or []:
test_id = entry.get("id", "?")
message = entry.get("message", "")
print(f"FAILED {test_id} - {message}", file=sys.stderr)
if entry.get("location"):
print(f" at {entry['location']}", file=sys.stderr)
if entry.get("assert"):
print(f" {entry['assert']}", file=sys.stderr)
if args.test: if args.test:
# Run specific test # Run specific test
result = runner.run_pytest(args.test, args.verbose, args.ci, with_coverage) result = runner.run_pytest(args.test, verbose, capture_output, with_coverage)
if not result["passed"]: if not result["passed"]:
print(f"Test failed: {args.test}") _print_failures()
print(f"Test failed: {args.test}", file=sys.stderr)
sys.exit(1) sys.exit(1)
else: else:
# Run all tests # Run all tests
all_passed = runner.run_all_tests(args.verbose, args.ci, with_coverage) all_passed = runner.run_all_tests(verbose, capture_output, with_coverage)
if not all_passed: if not all_passed:
print("Some tests failed") #TODO: this is not very informative, need improving _print_failures()
print("Some tests failed", file=sys.stderr)
sys.exit(1) sys.exit(1)
# Print summary # Print summary
@@ -3,10 +3,10 @@
Provides `SanityRunner`, the generalized class shared by the Provides `SanityRunner`, the generalized class shared by the
repository sanity runner (run_sanity) and the wiki sanity command repository sanity runner (run_sanity) and the wiki sanity command
(wiki.sanity_manager): concurrent check execution with per-check (wiki.sanity_manager): concurrent check execution with per-check
timings and the common AI-friendly report format (SPEC 4.9.3/4.9.4). timings and the common AI-friendly report format (SPEC 6.1/7.14.1).
""" """
from sanity_runner.sanity_runner import ( from fiona.sanity_runner.sanity_runner import (
SanityRunner, SanityRunner,
format_duration, format_duration,
format_violation_count, format_violation_count,
@@ -1,14 +1,14 @@
"""Shared sanity runner engine (SPEC 4.9.5). """Shared sanity runner engine (SPEC 7.14.1).
This module provides `SanityRunner`, the generalized class behind both This module provides `SanityRunner`, the generalized class behind both
sanity runners in the project: sanity runners in the project:
- the repository sanity runner (`src/run_sanity.py`, SPEC 4.9) - the repository sanity runner (`fiona.run_sanity`, SPEC 7.14)
- the wiki sanity command (`src/wiki/sanity_manager.py`, SPEC 3.1.3) - the wiki sanity command (`src/wiki/sanity_manager.py`, SPEC 7.1.3)
The class owns the common machinery: concurrent check execution with The class owns the common machinery: concurrent check execution with
per-check wall-clock timings (SPEC 4.9.4) and the shared AI-friendly per-check wall-clock timings (SPEC 7.14.1) and the shared AI-friendly
report structure (SPEC 4.9.3) — one-line check summaries, the aligned report structure (SPEC 6.1) — one-line check summaries, the aligned
`Results:` table rows, the `SANITY PASSED (MM:SS)` banner, and the `Results:` table rows, the `SANITY PASSED (MM:SS)` banner, and the
`Sanity Failed` footer. Per-check specifics (summary wording, extra `Sanity Failed` footer. Per-check specifics (summary wording, extra
table rows, failure details) plug in as callables, so each runner table rows, failure details) plug in as callables, so each runner
@@ -35,6 +35,19 @@ POLL_INTERVAL_SECONDS = 0.5
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
__all__ = [
"CheckCallable",
"CheckResult",
"ExtraResultLine",
"FailureDetails",
"HeartbeatCallback",
"ProgressCallback",
"SanityRunner",
"SummaryFormatter",
"format_duration",
"format_violation_count",
]
def format_duration(seconds: float) -> str: def format_duration(seconds: float) -> str:
"""Format a duration in seconds as zero-padded MM:SS. """Format a duration in seconds as zero-padded MM:SS.
@@ -98,7 +111,7 @@ class SanityRunner:
Each result carries its wall-clock `duration_seconds`; times of Each result carries its wall-clock `duration_seconds`; times of
parallel checks overlap and need not sum to the total parallel checks overlap and need not sum to the total
(SPEC 4.9.4). The returned mapping keeps the registry order (SPEC 7.14.1). The returned mapping keeps the registry order
regardless of completion order. The check lifecycle (start and regardless of completion order. The check lifecycle (start and
completion with duration) is logged at INFO. completion with duration) is logged at INFO.
@@ -267,21 +280,3 @@ class SanityRunner:
lines.extend(self.failure_details(check_name, check_result)) lines.extend(self.failure_details(check_name, check_result))
return lines return lines
# =============================================================================
# Public API
# =============================================================================
__all__ = [
"CheckCallable",
"CheckResult",
"ExtraResultLine",
"FailureDetails",
"HeartbeatCallback",
"ProgressCallback",
"SanityRunner",
"SummaryFormatter",
"format_duration",
"format_violation_count",
]
+5
View File
@@ -0,0 +1,5 @@
"""Wiki package for Fiona-Assistant.
Submodules are imported directly. This package does not import them
here, so git_tool can load wiki.ignore without a circular import.
"""
File diff suppressed because it is too large Load Diff
+367
View File
@@ -0,0 +1,367 @@
"""Wiki housekeeping routine (SPEC 4.8, 7.2).
Generated by AI/LLM: 2026-09-30 - one call for LLM wiki maintenance.
`fiona-assistant housekeeping` runs the maintenance steps in a fixed
order and reports, per step, a status (`PASS`, `WARN`, `FAIL`), a
summary, detail lines, and the next actions for the LLM. The routine
gathers what needs work; it writes only `wiki_graph.html`.
Steps, in order (`STEP_NAMES`):
- `check-update`: core files behind the packaged template (read-only).
- `merge`: files whose stem ends with `-update` that `wiki update` left for a merge.
- `ingest`: pages under `wiki/` still at `status: raw`.
- `sanity`: every `wiki sanity` check.
- `graph`: regenerate `wiki_graph.html` and list orphan pages.
"""
from __future__ import annotations
import contextlib
import io
import json
import os
import sys
import tempfile
from pathlib import Path
from fiona.librarian.md_frontmatter import FrontmatterError, parse_frontmatter
from fiona.wiki.graph_generator import wiki_graph_main
from fiona.wiki.sanity_manager import SETTINGS_FILE, run_wiki_checks
from fiona.wiki.update_manager import (
UPDATE_SUFFIX,
check_wiki_updates,
get_update_ignore_patterns,
path_is_update_ignored,
)
STEP_NAMES = ("check-update", "merge", "ingest", "sanity", "graph")
GRAPH_FILE = "wiki_graph.html"
UPDATE_FILE_SUFFIX = f"{UPDATE_SUFFIX}.md"
# AI/LLM: read-only zones sit at the wiki root; .git is skipped anywhere
ROOT_SKIP_DIRS = frozenset({"personal", "raw"})
__all__ = [
"GRAPH_FILE",
"STEP_NAMES",
"check_update_step",
"format_housekeeping_report",
"graph_step",
"housekeeping_main",
"ingest_step",
"merge_step",
"run_housekeeping",
"sanity_step",
]
# =============================================================================
# Helpers
# =============================================================================
def _step(name: str, status: str, summary: str, details: list[str] | None = None,
next_steps: list[str] | None = None) -> dict:
"""Build one step result.
Generated by AI/LLM: 2026-09-30 - shared step shape (SPEC 4.8).
"""
return {
"name": name,
"status": status,
"summary": summary,
"details": details or [],
"next": next_steps or [],
}
def _relative(path: str | Path, wiki_root: Path) -> str:
"""Return a path relative to the wiki root, with `/` separators.
Generated by AI/LLM: 2026-09-30 - report paths from the wiki root.
"""
return Path(os.path.relpath(path, wiki_root)).as_posix()
def _frontmatter(path: Path) -> dict:
"""Return a page's frontmatter, or an empty dict when it cannot be read.
Generated by AI/LLM: 2026-09-30 - sanity reports broken YAML itself.
"""
try:
frontmatter = parse_frontmatter(path)
except (FrontmatterError, UnicodeDecodeError):
return {}
return frontmatter if isinstance(frontmatter, dict) else {}
def _wiki_files(wiki_root: Path) -> list[Path]:
"""List files outside read-only zones and `.git`, sorted.
Modified by AI/LLM: 2026-10-01 - merge walks every suffix, not only markdown.
Args:
wiki_root: Root path of the wiki.
Returns:
list[Path]: Files found.
"""
found = []
for current, dirs, files in os.walk(wiki_root):
at_root = Path(current) == wiki_root
dirs[:] = [
name for name in dirs
if name != ".git" and not (at_root and name in ROOT_SKIP_DIRS)
]
found.extend(Path(current) / name for name in files)
return sorted(found)
def _markdown_files(wiki_root: Path, subdir: str = "") -> list[Path]:
"""List markdown files outside read-only zones and `.git`, sorted.
Generated by AI/LLM: 2026-09-30 - pruned walk for the merge and ingest steps.
Args:
wiki_root: Root path of the wiki.
subdir: Optional directory, relative to the root, to walk instead.
Returns:
list[Path]: Markdown files found.
"""
start = wiki_root / subdir
found = []
for current, dirs, files in os.walk(start):
at_root = Path(current) == wiki_root
dirs[:] = [
name for name in dirs
if name != ".git" and not (at_root and name in ROOT_SKIP_DIRS)
]
found.extend(Path(current) / name for name in files if name.endswith(".md"))
return sorted(found)
# =============================================================================
# Steps
# =============================================================================
def check_update_step(wiki_root: Path) -> dict:
"""Report core files behind the template without writing (SPEC 7.2.2).
Generated by AI/LLM: 2026-09-30 - read-only wiki check-update.
"""
summary = check_wiki_updates(wiki_root)
details = [f"update: {_relative(r.file_path, wiki_root)}" for r in summary.files_to_update]
details += [
f"conflict: {_relative(r.file_path, wiki_root)} -> {_relative(r.update_path, wiki_root)}"
for r in summary.conflicts
]
details += [f"add: {path.as_posix()}" for path in summary.files_to_add]
text = (
f"{len(summary.files_to_update)} update(s), {len(summary.conflicts)} conflict(s), "
f"{len(summary.files_to_add)} addition(s)"
)
if not details:
return _step("check-update", "PASS", text)
return _step("check-update", "WARN", text, details, [
"Ask the user, then run fiona-assistant wiki update --dry-run and wiki update --yes."
])
def merge_step(wiki_root: Path) -> dict:
"""List update files that wait for an LLM merge (SPEC 7.2.3).
Modified by AI/LLM: 2026-10-02 - skip update.ignore_files and corefiles.json.
"""
details, actions = [], []
ignore_patterns = get_update_ignore_patterns(wiki_root)
for update_file in _wiki_files(wiki_root):
if not update_file.stem.endswith(UPDATE_SUFFIX):
continue
if path_is_update_ignored(_relative(update_file, wiki_root), ignore_patterns):
continue
target = update_file.with_name(
update_file.stem[: -len(UPDATE_SUFFIX)] + update_file.suffix
)
update_rel = _relative(update_file, wiki_root)
target_rel = _relative(target, wiki_root)
missing = "" if target.exists() else " (target missing)"
details.append(f"{update_rel} -> {target_rel}{missing}")
version = _frontmatter(update_file).get("version") or "the template version"
actions.append(
f"Merge {update_rel} into {target_rel}: keep the owner's edits, "
f"take the template changes, set version to at least {version}, "
f"then delete {update_rel}."
)
text = f"{len(details)} update file(s) wait for a merge"
return _step("merge", "FAIL" if details else "PASS", text, details, actions)
def ingest_step(wiki_root: Path) -> dict:
"""List pages under wiki/ still at `status: raw` (SPEC 7.2.4).
Generated by AI/LLM: 2026-09-30 - pages waiting for a refine.
"""
pages = [
_relative(page, wiki_root)
for page in _markdown_files(wiki_root, "wiki")
if _frontmatter(page).get("status") == "raw"
]
text = f"{len(pages)} raw page(s)"
if not pages:
return _step("ingest", "PASS", text)
return _step("ingest", "WARN", text, pages,
["Refine each raw page, then set status: refined."])
def sanity_step(wiki_root: Path) -> dict:
"""Run every wiki sanity check without the progress display (SPEC 7.2.5).
Generated by AI/LLM: 2026-09-30 - failing check reports as details.
"""
results = run_wiki_checks(wiki_root)
failing = {name: check for name, check in results["checks"].items() if check["result"]}
text = f"{results['total_violations']} violation(s) in {len(failing)} check(s)"
if not failing:
return _step("sanity", "PASS", text)
details = [
f"{name}: {line.strip().removeprefix('- ')}"
for name, check in failing.items()
for line in check["stdout"].splitlines()
if line.strip()
]
return _step("sanity", "FAIL", text, details,
["Fix the violations, then run fiona-assistant housekeeping again."])
def graph_step(wiki_root: Path) -> dict:
"""Write wiki_graph.html at the wiki root and list orphans (SPEC 7.2.6).
Generated by AI/LLM: 2026-09-30 - graph plus orphan pages under wiki/.
"""
captured = io.StringIO()
with tempfile.TemporaryDirectory() as temp_dir:
graph_json = Path(temp_dir) / "graph.json"
with contextlib.redirect_stdout(captured), contextlib.redirect_stderr(captured):
code = wiki_graph_main(str(wiki_root), output=str(graph_json), format_type="json")
if code == 0:
code = wiki_graph_main(str(wiki_root), output=str(wiki_root / GRAPH_FILE))
if code != 0:
error = captured.getvalue().strip() or f"wiki graph exited {code}"
return _step("graph", "FAIL", f"could not write {GRAPH_FILE}",
error.splitlines())
data = json.loads(graph_json.read_text(encoding="utf-8"))
orphans = sorted(
rel for rel in (_relative(path, wiki_root) for path in data["orphans"])
if rel.startswith("wiki/") and not rel.endswith(UPDATE_FILE_SUFFIX)
)
text = (
f"wrote {GRAPH_FILE} ({data['metadata']['file_count']} nodes, "
f"{len(data['edges'])} edges)"
)
if not orphans:
return _step("graph", "PASS", text)
return _step("graph", "WARN", text, [f"orphan: {rel}" for rel in orphans],
["Link each orphan page from a related page or from wiki/index.md."])
# =============================================================================
# Routine and Report
# =============================================================================
def run_housekeeping(directory: str | Path = ".") -> dict:
"""Run every housekeeping step over a wiki (SPEC 7.2.1).
Generated by AI/LLM: 2026-09-30 - every step runs, even after a FAIL.
Args:
directory: The wiki directory. Defaults to the current directory.
Returns:
dict: `target` (resolved path), `steps` (one result per step,
in `STEP_NAMES` order), and `passed` (no step is `FAIL`).
Raises:
FileNotFoundError: If the directory does not exist.
NotADirectoryError: If the directory is not an llm-wiki directory.
"""
wiki_root = Path(directory).resolve()
if not wiki_root.exists():
raise FileNotFoundError(f"Directory not found: {directory}")
if not (wiki_root / SETTINGS_FILE).exists():
raise NotADirectoryError(f"Directory '{directory}' is not an llm-wiki directory")
steps = [
step(wiki_root)
for step in (
check_update_step,
merge_step,
ingest_step,
sanity_step,
graph_step,
)
]
return {
"target": wiki_root,
"steps": steps,
"passed": all(step["status"] != "FAIL" for step in steps),
}
def format_housekeeping_report(results: dict) -> str:
"""Format housekeeping results for humans and LLMs (SPEC 4.8).
Generated by AI/LLM: 2026-09-30 - step blocks, next steps, verdict.
Args:
results: The dict returned by run_housekeeping.
Returns:
str: The report text.
"""
lines = ["Housekeeping Report", f"TARGET: {results['target']}", ""]
for step in results["steps"]:
lines.append(f"[{step['status']}] {step['name']}: {step['summary']}")
lines.extend(f" - {detail}" for detail in step["details"])
actions = list(dict.fromkeys(a for step in results["steps"] for a in step["next"]))
if actions:
lines.extend(["", "Next steps:"])
lines.extend(f" - {action}" for action in actions)
failing = [step["name"] for step in results["steps"] if step["status"] == "FAIL"]
lines.append("")
if failing:
lines.append(
f"HOUSEKEEPING FAILED - {len(failing)} step(s) need attention: "
f"{', '.join(failing)}"
)
else:
lines.append("HOUSEKEEPING PASSED")
return "\n".join(lines)
def housekeeping_main(directory: str = ".", json_output: bool = False) -> int:
"""Main function for the 'housekeeping' CLI command.
Generated by AI/LLM: 2026-09-30 - exit 1 on any FAIL step.
Args:
directory: Wiki directory (default: current directory).
json_output: If True, print the results as one JSON object.
Returns:
int: 0 when no step is `FAIL`, 1 otherwise or on an invalid wiki.
"""
try:
results = run_housekeeping(directory)
except (FileNotFoundError, NotADirectoryError) as error:
print(f"Error: {error}", file=sys.stderr)
return 1
if json_output:
print(json.dumps({**results, "target": str(results["target"])}, indent=2))
else:
print(format_housekeeping_report(results))
return 0 if results["passed"] else 1
+171
View File
@@ -0,0 +1,171 @@
"""Gitignore patterns shared by wiki init and wiki update.
Generated by AI/LLM: 2026-09-30 - seed wiki ignores from the repository file.
"""
from __future__ import annotations
import fnmatch
import importlib.resources
import pathlib
_PACKAGE_NAME = "fiona-assistant"
_PERSONAL_PATTERN = "personal/"
def gitignore_patterns_from_text(text: str) -> list[str]:
"""Return ignore patterns from gitignore text.
Generated by AI/LLM: 2026-09-30 - drop blanks and comments.
Args:
text: Contents of a .gitignore file.
Returns:
Patterns in file order, without comments or blank lines.
"""
patterns: list[str] = []
for line in text.splitlines():
stripped = line.strip()
if stripped and not stripped.startswith("#"):
patterns.append(stripped)
return patterns
def path_is_ignored(relpath: str, patterns: list[str]) -> bool:
"""Return whether a relative posix path matches any gitignore pattern.
Generated by AI/LLM: 2026-09-30 - match unanchored names and directory prefixes.
A pattern with no slash matches any path segment. A trailing slash matches
a directory and everything under it. ``*`` does not cross a directory
boundary because matching is done per segment.
Args:
relpath: Path relative to the wiki or template root, posix form.
patterns: Gitignore patterns.
Returns:
True when one pattern matches.
"""
return any(_pattern_matches(relpath, pattern) for pattern in patterns)
def _pattern_matches(relpath: str, pattern: str) -> bool:
"""Return whether one gitignore pattern matches a relative posix path."""
body = pattern.rstrip("/")
if not body:
return False
parts = relpath.split("/")
if "/" in body:
prefixes = ["/".join(parts[:index]) for index in range(1, len(parts) + 1)]
return any(fnmatch.fnmatchcase(prefix, body) for prefix in prefixes)
return any(fnmatch.fnmatchcase(part, body) for part in parts)
def repository_gitignore_path() -> pathlib.Path | None:
"""Find the fiona-assistant repository .gitignore, if this tree has one.
Generated by AI/LLM: 2026-09-30 - prefer the source checkout over package data.
Returns:
Path to the repository .gitignore, or None when this install is not
the source checkout.
"""
here = pathlib.Path(__file__).resolve()
for parent in here.parents:
project = parent / "pyproject.toml"
gitignore = parent / ".gitignore"
if not project.is_file() or not gitignore.is_file():
continue
text = project.read_text(encoding="utf-8")
if f'name = "{_PACKAGE_NAME}"' in text:
return gitignore
return None
def packaged_gitignore_path() -> pathlib.Path | None:
"""Return the packaged copy of the repository .gitignore, if present.
Generated by AI/LLM: 2026-09-30 - fallback for an installed wheel.
Returns:
Path to ``fiona/data/default.gitignore``, or None when it is absent.
"""
resource = importlib.resources.files("fiona").joinpath("data", "default.gitignore")
path = pathlib.Path(str(resource))
if path.is_file():
return path
return None
def repository_gitignore_text() -> str:
"""Load the ignore file that seeds a wiki, as LF text.
Generated by AI/LLM: 2026-09-30 - repository file, then packaged copy.
Returns:
Ignore file text ending in a newline.
Raises:
FileNotFoundError: When neither the repository nor the packaged file exists.
"""
path = repository_gitignore_path() or packaged_gitignore_path()
if path is None:
raise FileNotFoundError("fiona-assistant repository .gitignore was not found")
text = path.read_text(encoding="utf-8").replace("\r\n", "\n").replace("\r", "\n")
if text and not text.endswith("\n"):
text += "\n"
return text
def wiki_gitignore_patterns() -> list[str]:
"""Return patterns written into a wiki, repository set plus personal/.
Generated by AI/LLM: 2026-09-30 - personal/ stays untracked after init.
Returns:
Ignore patterns in write order.
"""
patterns = gitignore_patterns_from_text(repository_gitignore_text())
if _PERSONAL_PATTERN not in patterns:
patterns.append(_PERSONAL_PATTERN)
return patterns
def wiki_gitignore_text() -> str:
"""Return the full .gitignore body for a wiki that does not have one yet.
Generated by AI/LLM: 2026-09-30 - repository text plus personal/ when missing.
Returns:
File body ending in a newline.
"""
text = repository_gitignore_text()
if _PERSONAL_PATTERN not in gitignore_patterns_from_text(text):
text += f"{_PERSONAL_PATTERN}\n"
return text
def ensure_wiki_gitignore(directory: pathlib.Path) -> None:
"""Create .gitignore or append patterns that are not already listed.
Generated by AI/LLM: 2026-09-30 - put or append the repository ignores.
Args:
directory: Wiki root that should contain .gitignore.
"""
target = directory / ".gitignore"
desired = wiki_gitignore_patterns()
if not target.exists():
target.write_text(wiki_gitignore_text(), encoding="utf-8", newline="\n")
return
existing = target.read_text(encoding="utf-8").replace("\r\n", "\n").replace("\r", "\n")
have = set(gitignore_patterns_from_text(existing))
missing = [pattern for pattern in desired if pattern not in have]
if not missing:
return
if existing and not existing.endswith("\n"):
existing += "\n"
existing += "\n".join(missing) + "\n"
target.write_text(existing, encoding="utf-8", newline="\n")
+4
View File
@@ -0,0 +1,4 @@
"""Wiki ingest package (SPEC 7.1.5).
Generated by AI/LLM: 2026-09-30 - stage modules for wiki ingest.
"""
+239
View File
@@ -0,0 +1,239 @@
"""Archive a fetched file under ``raw/``.
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.3. Never edits a file already there.
"""
from __future__ import annotations
import hashlib
import logging
import pathlib
import shutil
from fiona.wiki.ingest.converters.base import ConverterRegistry
from fiona.wiki.ingest.media import extension_for
from fiona.wiki.ingest.models import ArchiveResult, FetchedSource, IngestError
from fiona.wiki.ingest.settings import IngestSettings
logger = logging.getLogger(__name__)
class RawArchiver:
"""Copy or move a source into ``raw/`` and reuse identical bytes.
Generated by AI/LLM: 2026-09-30 - SHA-256 reuse and collision names.
"""
def __init__(self, wiki_root: pathlib.Path) -> None:
"""Store the wiki root.
Args:
wiki_root: Directory that contains ``raw/``.
"""
self.wiki_root = wiki_root
def find_by_sha256(
self,
digest: str,
size: int,
skip: pathlib.Path | None = None,
) -> pathlib.Path | None:
"""Return the first ``raw/`` file with this size and digest.
Generated by AI/LLM: 2026-09-30 - only same-size files are hashed.
Args:
digest: Lowercase SHA-256 hex.
size: Byte size that a candidate must match.
skip: A file to ignore, usually the source when it already sits in ``raw/``.
Returns:
pathlib.Path | None: Match with the lowest path, or None.
"""
raw_root = self.wiki_root / "raw"
if not raw_root.is_dir():
return None
skip_resolved = skip.resolve() if skip is not None else None
matches: list[pathlib.Path] = []
for path in raw_root.rglob("*"):
if not path.is_file():
continue
if skip_resolved is not None and path.resolve() == skip_resolved:
continue
try:
if path.stat().st_size != size:
continue
except OSError:
continue
if _sha256(path) == digest:
matches.append(path)
if not matches:
return None
matches.sort(key=lambda item: item.as_posix())
return matches[0]
def archive(
self,
source: FetchedSource,
*,
registry: ConverterRegistry,
settings: IngestSettings,
raw_override: str | None,
dry_run: bool,
) -> ArchiveResult:
"""Place the source under ``raw/`` or reuse an identical copy.
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.3.
Args:
source: Fetched file.
registry: Used for the raw category and a missing extension.
settings: ``raw_categories`` overrides.
raw_override: ``-r`` value, relative to ``raw/``, or None.
dry_run: When true, compute the path and write nothing.
Returns:
ArchiveResult: Relative raw path, digest, and warnings.
Raises:
IngestError: ``-r`` escapes ``raw/`` or is used on a file already there.
"""
raw_root = (self.wiki_root / "raw").resolve()
source_in_raw = _is_inside(source.path, raw_root)
if raw_override and source_in_raw:
raise IngestError(
f"--raw cannot be used on a file already under raw/: {source.argument}"
)
if raw_override:
_require_inside(raw_root, raw_override, "--raw must stay inside raw/")
digest = _sha256(source.path)
match = self.find_by_sha256(
digest,
source.size_bytes,
skip=source.path if source_in_raw else None,
)
warnings: list[str] = []
if match is not None:
raw_rel = _relative(match, self.wiki_root)
if raw_override:
warnings.append(f"already archived at {raw_rel}; --raw ignored")
if _is_inside(source.path, self.wiki_root / "ingest") and not dry_run:
source.path.unlink()
logger.info("reused raw file %s", raw_rel)
return ArchiveResult(match, raw_rel, digest, True, warnings)
if source_in_raw:
return ArchiveResult(source.path, _relative(source.path, self.wiki_root), digest, False, [])
extension = source.path.suffix.lower() or extension_for(source.media_type, "", registry)
dest = self._destination(source, registry, settings, raw_override, extension, raw_root)
dest = _free_name(dest, source.slug, extension)
if not dry_run:
dest.parent.mkdir(parents=True, exist_ok=True)
if _is_inside(source.path, self.wiki_root / "ingest"):
shutil.move(str(source.path), str(dest))
else:
shutil.copyfile(source.path, dest)
logger.info("archived %s", dest)
return ArchiveResult(dest, _relative(dest, self.wiki_root), digest, False, warnings)
def _destination(
self,
source: FetchedSource,
registry: ConverterRegistry,
settings: IngestSettings,
raw_override: str | None,
extension: str,
raw_root: pathlib.Path,
) -> pathlib.Path:
"""Resolve the raw path before a collision suffix is applied.
Generated by AI/LLM: 2026-09-30 - ``-r`` file or directory, else the category.
"""
filename = f"{source.slug}{extension}"
if raw_override:
relative = _strip_prefix(raw_override, "raw")
suffix = pathlib.PurePosixPath(relative.replace("\\", "/")).suffix
if suffix:
return raw_root / relative
return raw_root / relative / filename
converter = registry.find(source.media_type, extension)
if converter is None:
category = "other"
else:
category = settings.raw_categories.get(converter.name, converter.raw_category)
return raw_root / category / filename
def _sha256(path: pathlib.Path) -> str:
"""Return the lowercase SHA-256 hex digest of a file.
Generated by AI/LLM: 2026-09-30 - streamed so large files stay cheap.
"""
digest = hashlib.sha256()
with path.open("rb") as handle:
while True:
chunk = handle.read(1024 * 1024)
if not chunk:
break
digest.update(chunk)
return digest.hexdigest()
def _free_name(dest: pathlib.Path, slug: str, extension: str) -> pathlib.Path:
"""Return ``dest`` or the first free ``<slug>-N`` name beside it.
Generated by AI/LLM: 2026-09-30 - never overwrite a raw file.
"""
if not dest.exists():
return dest
number = 2
while True:
candidate = dest.parent / f"{slug}-{number}{extension}"
if not candidate.exists():
return candidate
number += 1
def _strip_prefix(value: str, prefix: str) -> str:
"""Drop a leading ``raw/`` or ``wiki/`` from a user path.
Generated by AI/LLM: 2026-09-30 - both forms are accepted.
"""
relative = value.replace("\\", "/").lstrip("/")
marker = f"{prefix}/"
if relative == prefix:
return ""
if relative.startswith(marker):
return relative[len(marker):]
return relative
def _require_inside(root: pathlib.Path, value: str, message: str) -> pathlib.Path:
"""Resolve ``value`` under ``root`` or raise.
Generated by AI/LLM: 2026-09-30 - ``..`` and symlinks must stay inside.
"""
relative = _strip_prefix(value, root.name)
candidate = (root / relative).resolve()
if not _is_inside(candidate, root):
raise IngestError(f"{message}: {value}")
return candidate
def _is_inside(path: pathlib.Path, root: pathlib.Path) -> bool:
"""Return whether ``path`` is ``root`` or a file inside it.
Generated by AI/LLM: 2026-09-30 - compare resolved paths.
"""
try:
path.resolve().relative_to(root.resolve())
except ValueError:
return False
return True
def _relative(path: pathlib.Path, root: pathlib.Path) -> str:
"""Return a wiki-relative path with ``/`` separators.
Generated by AI/LLM: 2026-09-30 - JSON and log paths.
"""
return path.resolve().relative_to(root.resolve()).as_posix()
@@ -0,0 +1,38 @@
"""Shipped converters, in registration order.
Generated by AI/LLM: 2026-09-30 - markdown, text, html, then pdf.
Modified by AI/LLM: 2026-10-07 - the html converter takes the block sweep settings.
"""
from fiona.wiki.ingest.converters.base import ConverterRegistry
from fiona.wiki.ingest.converters.html import HtmlConverter
from fiona.wiki.ingest.converters.markdown import MarkdownConverter
from fiona.wiki.ingest.converters.pdf import PdfConverter
from fiona.wiki.ingest.converters.text import TextConverter
from fiona.wiki.ingest.settings import IngestSettings, default_settings
def default_registry(settings: IngestSettings | None = None) -> ConverterRegistry:
"""Return the converters shipped with wiki ingest.
Generated by AI/LLM: 2026-09-30 - registration order from SPEC 7.1.5.
Modified by AI/LLM: 2026-10-07 - ``html.*`` settings reach ``HtmlConverter``.
Args:
settings: Ingest settings. ``None`` uses the defaults.
Returns:
ConverterRegistry: Markdown, text, HTML, and PDF.
"""
settings = settings or default_settings()
registry = ConverterRegistry()
registry.register(MarkdownConverter)
registry.register(TextConverter)
registry.register(
HtmlConverter(
min_words=settings.html_min_words,
max_link_density=settings.html_max_link_density,
)
)
registry.register(PdfConverter)
return registry
+108
View File
@@ -0,0 +1,108 @@
"""Converter base types and the registry.
Generated by AI/LLM: 2026-09-30 - one class per format, SPEC 7.1.5.1.
"""
from __future__ import annotations
import abc
import typing
from fiona.wiki.ingest.models import ConversionResult, FetchedSource
class ConversionError(Exception):
"""Raised when a converter cannot read a file it claims.
Generated by AI/LLM: 2026-09-30 - exit 3, raw copy kept.
"""
def __init__(self, converter: str, reason: str) -> None:
"""Store the converter name and the reason.
Args:
converter: Converter ``name`` attribute.
reason: Short failure text, without the ``Error:`` prefix.
"""
super().__init__(f"{converter}: {reason}")
self.converter = converter
self.reason = reason
class SourceConverter(abc.ABC):
"""One format. Subclasses declare what they claim and implement ``convert``.
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.1.2.
"""
name: typing.ClassVar[str]
media_types: typing.ClassVar[tuple[str, ...]]
extensions: typing.ClassVar[tuple[str, ...]]
raw_category: typing.ClassVar[str]
@abc.abstractmethod
def convert(self, source: FetchedSource) -> ConversionResult:
"""Convert an archived file to markdown.
Args:
source: Fetched file, including media type and slug.
Returns:
ConversionResult: Body, title, metadata, and warnings.
Raises:
ConversionError: The file cannot be read.
"""
class ConverterRegistry:
"""Ordered list of converters.
Generated by AI/LLM: 2026-09-30 - first media-type match, then extension.
"""
def __init__(self, converters: list[SourceConverter] | None = None) -> None:
"""Start with an optional list of converter instances.
Args:
converters: Instances in registration order.
"""
self.converters: list[SourceConverter] = list(converters or [])
def register(self, converter: SourceConverter | type[SourceConverter]) -> None:
"""Append a converter class or instance.
Generated by AI/LLM: 2026-09-30 - adding a format is one registration.
Args:
converter: Class with a no-arg constructor, or a ready instance.
"""
if isinstance(converter, type):
converter = converter()
self.converters.append(converter)
def find(self, media_type: str, extension: str) -> SourceConverter | None:
"""Pick the converter for a media type and extension.
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.4.1 registration order.
Args:
media_type: Detected type, or empty.
extension: Suffix including the dot, or empty.
Returns:
SourceConverter | None: The first match, or None.
"""
media = (media_type or "").lower()
ext = extension.lower()
if ext and not ext.startswith("."):
ext = f".{ext}"
if media:
for converter in self.converters:
if media in converter.media_types:
return converter
if ext:
for converter in self.converters:
if ext in converter.extensions:
return converter
return None
+141
View File
@@ -0,0 +1,141 @@
"""HTML source converter.
Generated by AI/LLM: 2026-09-30 - article text via beautifulsoup4 and markdownify.
Modified by AI/LLM: 2026-10-07 - the block sweep replaces the first-``<article>`` root.
"""
from __future__ import annotations
import typing
import urllib.parse
import bs4
from fiona.wiki.ingest.converters.base import SourceConverter
from fiona.wiki.ingest.converters.html_sweep import sweep, unwrap_shadow_roots
from fiona.wiki.ingest.media import decode_bytes
from fiona.wiki.ingest.models import ConversionResult, FetchedSource
_DROP_TAGS = ("script", "style", "noscript", "template")
_NO_TEXT = "no text content; the page may need JavaScript or a login"
class HtmlConverter(SourceConverter):
"""Convert an HTML document to markdown.
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.4.5.
Modified by AI/LLM: 2026-10-07 - sweep limits are constructor arguments.
"""
name: typing.ClassVar[str] = "html"
media_types: typing.ClassVar[tuple[str, ...]] = ("text/html", "application/xhtml+xml")
extensions: typing.ClassVar[tuple[str, ...]] = (".html", ".htm", ".xhtml")
raw_category: typing.ClassVar[str] = "articles"
def __init__(self, min_words: int = 4, max_link_density: float = 0.5) -> None:
"""Store the block sweep limits.
Generated by AI/LLM: 2026-10-07 - defaults from SPEC 5.1 ``fiona.wiki.ingest.html``.
Args:
min_words: Fewer words make a ``p`` or loose block ``short``.
max_link_density: Link text over text length above this is ``links``.
"""
self.min_words = min_words
self.max_link_density = max_link_density
def convert(self, source: FetchedSource) -> ConversionResult:
"""Parse HTML and return every swept block of the body as markdown.
Generated by AI/LLM: 2026-09-30 - drop scripts and absolutize links.
Modified by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6 block sweep over ``<body>``.
Args:
source: Archived HTML file. ``origin`` is the page URL when fetched.
Returns:
ConversionResult: Markdown body, title, and ``source_meta``.
"""
text, warnings = decode_bytes(source.path.read_bytes(), source.charset)
soup = bs4.BeautifulSoup(text, "html.parser")
unwrap_shadow_roots(soup)
for tag in soup.find_all(_DROP_TAGS):
tag.decompose()
for comment in soup.find_all(string=lambda node: isinstance(node, bs4.Comment)):
comment.extract()
body = soup.body or soup
origin = source.origin if source.origin_kind == "url" else None
_absolutize(body, soup, origin)
conversion = "complete"
if not body.get_text(strip=True):
conversion = "partial"
warnings = [*warnings, _NO_TEXT]
swept = sweep(body, min_words=self.min_words, max_link_density=self.max_link_density)
if swept.partial:
conversion = "partial"
return ConversionResult(
body=swept.body,
title=_html_title(soup, body),
source_meta=_source_meta(soup),
conversion=conversion,
warnings=[*warnings, *swept.warnings],
converter=self.name,
)
def _absolutize(root: bs4.element.Tag, soup: bs4.BeautifulSoup, origin: str | None) -> None:
"""Rewrite relative href and src values against base href, then the origin.
Generated by AI/LLM: 2026-09-30 - a local file with no origin keeps relative links.
"""
base_tag = soup.find("base", href=True)
base = base_tag.get("href") if base_tag is not None else None
if not base:
base = origin
if not base:
return
for tag in root.find_all(href=True):
tag["href"] = urllib.parse.urljoin(str(base), str(tag["href"]))
for tag in root.find_all(src=True):
tag["src"] = urllib.parse.urljoin(str(base), str(tag["src"]))
def _html_title(soup: bs4.BeautifulSoup, root: bs4.element.Tag) -> str | None:
"""Return ``<title>``, then ``og:title``, then the first ``<h1>`` that has text.
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.4.5.1.
Modified by AI/LLM: 2026-10-07 - the root is ``<body>``, so a blank ``<h1>`` is skipped.
"""
if soup.title and soup.title.string and soup.title.string.strip():
return soup.title.string.strip()
og = soup.find("meta", attrs={"property": "og:title"})
if og is not None and og.get("content"):
return str(og["content"]).strip() or None
for heading in root.find_all("h1"):
text = heading.get_text(strip=True)
if text:
return text
return None
def _source_meta(soup: bs4.BeautifulSoup) -> dict:
"""Collect language, canonical URL, and selected meta values.
Generated by AI/LLM: 2026-09-30 - only keys that are present.
"""
meta: dict[str, str] = {}
root = soup.find("html")
if root is not None and root.get("lang"):
meta["lang"] = str(root["lang"])
canonical = soup.find("link", rel=lambda value: value and "canonical" in value)
if canonical is not None and canonical.get("href"):
meta["canonical"] = str(canonical["href"])
for tag in soup.find_all("meta"):
key = tag.get("name") or tag.get("property")
content = tag.get("content")
if not key or content is None:
continue
name = str(key)
if name in {"description", "author"} or name.startswith("og:") or name == "article:published_time":
meta[name] = str(content)
return meta
@@ -0,0 +1,285 @@
"""Score every block of an HTML body as ``keep``, ``review``, or ``drop``.
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6. Rules read HTML structure only.
"""
from __future__ import annotations
import dataclasses
import re
import typing
import bs4
import markdownify
_BLOCK_KINDS = {
"h1": "heading",
"h2": "heading",
"h3": "heading",
"h4": "heading",
"h5": "heading",
"h6": "heading",
"p": "p",
"ul": "list",
"ol": "list",
"dl": "list",
"blockquote": "quote",
"pre": "pre",
"table": "table",
"figure": "figure",
}
_INLINE_TAGS = frozenset((
"a", "abbr", "b", "bdi", "br", "cite", "code", "data", "del", "dfn", "em",
"font", "i", "img", "ins", "kbd", "label", "mark", "q", "s", "samp", "small",
"span", "strong", "sub", "sup", "time", "u", "var", "wbr",
))
_LAYOUT_CELL_CHILDREN = (*_BLOCK_KINDS, "div")
_SKIPPED_TAGS = frozenset(("head",))
_DROP_ANCESTORS = frozenset(("nav", "button", "select"))
_CHROME_TAGS = frozenset(("aside", "form", "dialog"))
_CHROME_ROLES = frozenset(("navigation", "banner", "contentinfo", "complementary"))
_HIDDEN_STYLE = re.compile(r"display\s*:\s*none|visibility\s*:\s*hidden", re.IGNORECASE)
_SHORT_KINDS = frozenset(("p", "loose"))
_ALL_REVIEW = "no block passed the content checks; every block is marked for review"
@dataclasses.dataclass
class Block:
"""One swept block: an element, or a run of loose text and inline nodes.
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6.1 and 7.1.5.4.5.6.2.
"""
nodes: list[bs4.PageElement]
kind: str
text: str = ""
link_chars: int = 0
words: int = 0
verdict: str = ""
reason: str = ""
@dataclasses.dataclass
class SweepResult:
"""Markdown body, warnings, and whether every block needs review.
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6.6 and 7.1.5.4.5.6.7.
"""
body: str
warnings: list[str]
partial: bool
def unwrap_shadow_roots(soup: bs4.BeautifulSoup) -> None:
"""Read each declarative shadow root in place, so its text is swept.
Generated by AI/LLM: 2026-10-07 - ``template[shadowrootmode]``; other templates are dropped later.
Args:
soup: Parsed document, before the drop loop.
"""
for template in soup.find_all("template", attrs={"shadowrootmode": True}):
template.unwrap()
def sweep(root: bs4.element.Tag, *, min_words: int, max_link_density: float) -> SweepResult:
"""Walk ``root``, classify each block, and render the kept and review blocks.
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6. No clock, network, or random source.
Args:
root: ``<body>``, or the document when there is no body.
min_words: ``fiona.wiki.ingest.html.min_words``.
max_link_density: ``fiona.wiki.ingest.html.max_link_density``.
Returns:
SweepResult: Markdown body, the ``html sweep:`` warning, and the partial flag.
"""
blocks: list[Block] = []
_walk(root, blocks)
seen: set[str] = set()
for block in blocks:
_measure(block)
block.verdict, block.reason = _classify(block, seen, min_words, max_link_density)
if block.verdict != "drop":
seen.add(block.text.casefold())
counts = {verdict: sum(block.verdict == verdict for block in blocks) for verdict in ("keep", "review", "drop")}
warnings = [f"html sweep: {counts['keep']} keep, {counts['review']} review, {counts['drop']} drop"]
partial = counts["keep"] == 0 and counts["review"] > 0
if partial:
warnings.append(_ALL_REVIEW)
parts = [part for part in (_render(block) for block in blocks if block.verdict != "drop") if part]
body = "\n\n".join(parts)
return SweepResult(body=body + "\n" if body else "", warnings=warnings, partial=partial)
def _walk(container: bs4.element.Tag, blocks: list[Block]) -> None:
"""Append the blocks of ``container`` in document order.
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6.1 to 7.1.5.4.5.6.1.2.
"""
run: list[bs4.PageElement] = []
for child in list(container.children):
if not isinstance(child, bs4.element.Tag):
if isinstance(child, bs4.element.NavigableString):
run.append(child)
continue
if child.name in _SKIPPED_TAGS:
continue
if _is_block(child):
_flush(run, blocks)
blocks.append(Block(nodes=[child], kind=_BLOCK_KINDS[child.name]))
elif child.name in _INLINE_TAGS and not _holds_structure(child):
run.append(child)
else:
_flush(run, blocks)
_walk(child, blocks)
_flush(run, blocks)
def _flush(run: list[bs4.PageElement], blocks: list[Block]) -> None:
"""Close a loose-text run as one block, unless it holds no text.
Generated by AI/LLM: 2026-10-07 - whitespace between tags is not a block.
"""
if run and _fold(_raw_text(run)):
blocks.append(Block(nodes=list(run), kind="loose"))
run.clear()
def _is_block(tag: bs4.element.Tag) -> bool:
"""Return whether ``tag`` is a block. A layout table is a container instead.
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6.1 layout tables.
"""
if tag.name not in _BLOCK_KINDS:
return False
if tag.name != "table":
return True
return not any(cell.find(_LAYOUT_CELL_CHILDREN) for cell in tag.find_all(("td", "th")))
def _holds_structure(tag: bs4.element.Tag) -> bool:
"""Return whether an inline tag wraps a block or container, like a card link.
Generated by AI/LLM: 2026-10-07 - ``<a><div>...</div></a>`` is walked as a container.
"""
return tag.find(lambda node: node.name not in _INLINE_TAGS) is not None
def _raw_text(nodes: typing.Iterable[bs4.PageElement]) -> str:
"""Return the text of ``nodes`` with no separator and ``<br>`` as a space.
Generated by AI/LLM: 2026-10-07 - keeps ``<b>Fa</b>st`` as one word.
"""
parts: list[str] = []
for node in nodes:
items = [node] if isinstance(node, bs4.element.NavigableString) else [node, *node.descendants]
for item in items:
if isinstance(item, bs4.element.NavigableString):
parts.append(str(item))
elif item.name == "br":
parts.append(" ")
return "".join(parts)
def _fold(text: str) -> str:
"""Fold whitespace runs to one space and trim.
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6.2.
"""
return " ".join(text.split())
def _measure(block: Block) -> None:
"""Set the folded text, link text length, and word count of a block.
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6.2.
"""
block.text = _fold(_raw_text(block.nodes))
links: list[bs4.element.Tag] = []
for node in block.nodes:
if isinstance(node, bs4.element.Tag):
links.extend([node] if node.name == "a" else node.find_all("a"))
block.link_chars = sum(len(_fold(_raw_text([link]))) for link in links)
block.words = len(block.text.split())
def _classify(block: Block, seen: set[str], min_words: int, max_link_density: float) -> tuple[str, str]:
"""Return ``(verdict, reason)`` with the first rule that holds.
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6.3 to 7.1.5.4.5.6.5.
"""
folded = block.text.casefold()
if not any(char.isalnum() for char in block.text):
return "drop", ""
if block.words < min_words and folded in seen:
return "drop", ""
if _inside(block, _drops):
return "drop", ""
if _inside(block, _is_chrome):
return "review", "chrome"
if folded in seen:
return "review", "repeat"
if block.link_chars / len(block.text) > max_link_density:
return "review", "links"
if block.kind in _SHORT_KINDS and block.words < min_words:
return "review", "short"
return "keep", ""
def _inside(block: Block, predicate: typing.Callable[[bs4.element.Tag], bool]) -> bool:
"""Return whether the block, or an ancestor below the document, matches.
Generated by AI/LLM: 2026-10-07 - an element block checks itself; a loose run starts at its parent.
"""
first = block.nodes[0]
node = first.parent if block.kind == "loose" else first
while isinstance(node, bs4.element.Tag) and not isinstance(node, bs4.BeautifulSoup):
if predicate(node):
return True
node = node.parent
return False
def _drops(tag: bs4.element.Tag) -> bool:
"""Return whether text under ``tag`` is never content: nav, controls, hidden.
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6.3.
"""
if tag.name in _DROP_ANCESTORS or tag.has_attr("hidden"):
return True
if str(tag.get("aria-hidden", "")).strip().lower() == "true":
return True
return bool(_HIDDEN_STYLE.search(str(tag.get("style", ""))))
def _is_chrome(tag: bs4.element.Tag) -> bool:
"""Return whether ``tag`` is page chrome: aside, form, landmark role, outer header or footer.
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6.4 ``chrome``.
"""
if tag.name in _CHROME_TAGS:
return True
if str(tag.get("role", "")).strip().lower() in _CHROME_ROLES:
return True
if tag.name in ("header", "footer"):
return tag.find_parent(("article", "main")) is None
return False
def _render(block: Block) -> str:
"""Convert one block to markdown and wrap a ``review`` block in its markers.
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6.6, options of 7.1.5.4.5.4.
"""
markup = "".join(
node.output_ready(formatter="minimal") if isinstance(node, bs4.element.NavigableString) else str(node)
for node in block.nodes
)
heading = getattr(markdownify, "ATX", "ATX")
text = markdownify.markdownify(markup, heading_style=heading, bullets="-").strip()
if not text or block.verdict != "review":
return text
return f"<!-- fiona:review {block.reason} -->\n\n{text}\n\n<!-- /fiona:review -->"
@@ -0,0 +1,105 @@
"""Markdown source converter.
Generated by AI/LLM: 2026-09-30 - keep the body and lift frontmatter into source_meta.
"""
from __future__ import annotations
import re
import typing
import yaml
from fiona.wiki.ingest.converters.base import SourceConverter
from fiona.wiki.ingest.media import decode_bytes
from fiona.wiki.ingest.models import ConversionResult, FetchedSource
_FRONTMATTER = re.compile(r"\A---\r?\n(.*?)\r?\n---\r?\n?", re.DOTALL)
_H1 = re.compile(r"^#\s+(\S.*)$")
_FENCE = re.compile(r"^\s*(`{3,}|~{3,})")
class MarkdownConverter(SourceConverter):
"""Keep markdown as it is and record any source frontmatter.
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.4.3.
The body after a frontmatter block is kept character for character.
The page writer adds the wiki frontmatter when the new page does not
have it yet. Source keys move to ``source_meta`` instead of being
written a second time as the page frontmatter.
"""
name: typing.ClassVar[str] = "markdown"
media_types: typing.ClassVar[tuple[str, ...]] = ("text/markdown",)
extensions: typing.ClassVar[tuple[str, ...]] = (".md", ".markdown")
raw_category: typing.ClassVar[str] = "articles"
def convert(self, source: FetchedSource) -> ConversionResult:
"""Return the markdown body and any frontmatter as metadata.
Generated by AI/LLM: 2026-09-30 - title from frontmatter, then H1, then slug.
Args:
source: Archived markdown file.
Returns:
ConversionResult: Unchanged body, ``complete`` unless decoding fell back.
"""
text, warnings = decode_bytes(source.path.read_bytes(), source.charset)
meta: dict = {}
body = text
match = _FRONTMATTER.match(text)
if match:
loaded = _load_meta(match.group(1))
if loaded is None:
warnings = [*warnings, "source frontmatter did not parse"]
else:
meta = loaded
body = text[match.end():]
title = _title(meta, body, source.slug)
return ConversionResult(
body=body,
title=title,
source_meta=meta,
conversion="complete",
warnings=warnings,
converter=self.name,
)
def _load_meta(block: str) -> dict | None:
"""Parse a frontmatter block into a dict.
Generated by AI/LLM: 2026-09-30 - invalid YAML stays in the body.
"""
try:
loaded = yaml.safe_load(block)
except yaml.YAMLError:
return None
if loaded is None:
return {}
if isinstance(loaded, dict):
return loaded
return None
def _title(meta: dict, body: str, slug: str) -> str:
"""Choose the page title.
Generated by AI/LLM: 2026-09-30 - frontmatter title, first H1, then slug.
"""
raw_title = meta.get("title")
if isinstance(raw_title, str) and raw_title.strip():
return raw_title.strip()
fenced = False
for line in body.splitlines():
if _FENCE.match(line):
fenced = not fenced
continue
if fenced:
continue
match = _H1.match(line)
if match:
return match.group(1).strip()
return slug

Some files were not shown because too many files have changed in this diff Show More