From 44029aee901d68e6d645fcc2753fd48508228d76 Mon Sep 17 00:00:00 2001 From: nvaldez Date: Thu, 1 Oct 2026 16:58:19 +0200 Subject: [PATCH] wiki update fix, themes versioning, readme refactor, graph ignore --- README.md | 674 +- SPEC.md | 6354 +++++++++-------- .../data/llm_wiki/constellation-settings.json | 329 - .../data/llm_wiki/constellation/colorful.json | 55 + .../data/llm_wiki/constellation/dark.json | 55 + .../llm_wiki/constellation/grayscale.json | 109 + .../llm_wiki/constellation/night-sky.json | 55 + .../llm_wiki/constellation/professional.json | 55 + .../data/llm_wiki/utils/skills/fiona/SKILL.md | 22 +- src/fiona/wiki/graph_generator.py | 122 +- src/fiona/wiki/templates/graph_physics.js | 5 + src/fiona/wiki/update_manager.py | 192 +- tests/test_graph_generator.py | 119 +- tests/test_update_manager.py | 112 +- tests/test_wiki.py | 8 +- 15 files changed, 4453 insertions(+), 3813 deletions(-) delete mode 100644 src/fiona/data/llm_wiki/constellation-settings.json create mode 100644 src/fiona/data/llm_wiki/constellation/colorful.json create mode 100644 src/fiona/data/llm_wiki/constellation/dark.json create mode 100644 src/fiona/data/llm_wiki/constellation/grayscale.json create mode 100644 src/fiona/data/llm_wiki/constellation/night-sky.json create mode 100644 src/fiona/data/llm_wiki/constellation/professional.json diff --git a/README.md b/README.md index 3a86961..8733e18 100644 --- a/README.md +++ b/README.md @@ -1,390 +1,436 @@ # Fiona-Assistant -LLM Wiki Tool Helper for LLM Agents/Hives +> Your LLM agent writes the wiki. Fiona keeps it clean. -Fiona-Assistant is a command-line toolkit designed to manage, organize, and -maintain LLM wiki content with minimal AI token dependence. It combines -script-based automation with structured workflows to help LLM Agents and -Hives efficiently manage documentation, markdown files, and code quality. +Fiona-Assistant is a command-line toolkit for running an **llm-wiki**: a +folder of interlinked markdown that an AI agent builds and maintains for +you. You bring sources (web pages, PDFs, notes) and questions. The agent +compiles them into wiki pages. Fiona does every deterministic part of the +job: it scaffolds the wiki, ingests and archives sources, lints markdown, +frontmatter, Python, and JSON, checks links, draws a link graph, versions +everything in git, and tells the agent exactly what to fix next. + +The result: the agent spends its tokens on reading and writing, not on +listing files, chasing broken links, or reformatting tables. + +```bash +python -m pip install fiona-assistant +fiona-assistant wiki init my-wiki +cd my-wiki +fiona-assistant wiki ingest https://example.com/article.html +fiona-assistant housekeeping +``` --- -## Overview +## Table of Contents -### Purpose +- [Why Fiona](#why-fiona) +- [Features](#features) +- [Quick Start](#quick-start) +- [How It Works](#how-it-works) +- [Command Reference](#command-reference) +- [Output Built for Agents](#output-built-for-agents) +- [Roadmap](#roadmap) +- [Development](#development) +- [Documentation](#documentation) +- [License](#license) -Fiona-Assistant serves as a **standalone MD/scripts/docs organizer** and -**LLM wiki management helper** that: +--- -- **Reduces AI Token Usage**: Uses deterministic scripts and tools to handle - wiki management tasks that don't require LLM intelligence -- **Maintains Wiki Structure**: Enforces consistent organization of markdown - files, concepts, and raw source materials -- **Validates Content Quality**: Provides linting and diagnosis tools for markdown and Python code -- **Enables Script-Based Workflows**: Automates repetitive wiki maintenance tasks through CLI commands +## Why Fiona -### Core Philosophy +An LLM agent can maintain a knowledge base, but left alone it wastes +context on chores and slowly breaks things: dead links, missing +frontmatter, wrapped table rows, orphan pages, overwritten sources. -> Use scripts for deterministic tasks, reserve AI tokens for creative and analytical work +Fiona fixes that split of work: -The toolkit is designed to be **AI-friendly** - outputs are formatted in -consistent, parseable structures that LLMs can easily read and process. +| The agent does | Fiona does | +| --- | --- | +| Reads sources and writes synthesized pages | Fetches, archives, and converts sources to markdown | +| Decides what links to what | Finds dead links, dead wikilinks, and orphan pages | +| Refines raw pages | Lints markdown, frontmatter, Python, and JSON | +| Answers your questions from the wiki | Commits, snapshots, and restores wiki history | +| Merges template updates | Detects template updates and never overwrites your files | + +> Use scripts for deterministic tasks. Reserve AI tokens for creative and +> analytical work. --- ## Features -### Tool Suite +### Wiki in one command -| Tool | Purpose | Key Capabilities | -| ---- | ------- | ---------------- | -| **Librarian** | Wiki content management | List files, scan directories, organize MD content | -| **MDoctor** | Markdown diagnosis | Lint markdown files using PyMarkdown, detect formatting issues | -| **Python-Ruff** | Code quality | Lint Python files using Ruff, enforce code standards | -| **Generate-TOC** | Table of contents | Auto-generate nested TOC from markdown headers | +`fiona-assistant wiki init` creates a ready-to-use llm-wiki: directory +layout, settings, agent rules (`AGENTS.md`), page schema, index, log, and +bundled agent skills. It also creates a git repository and makes the +first commit, with `personal/` kept out of version control. -### Wiki Structure Support +### Source ingestion -Fiona-Assistant understands and maintains the [LLM Wiki directory structure](./src/fiona/data/llm_wiki/AGENTS.md): +`fiona-assistant wiki ingest ` turns a source into a wiki page +in one step: -```text -llm-wiki/ -├── ingest/ # Input zone: files to be processed -├── raw/ # Archived source materials (read-only) -│ ├── articles/ -│ ├── papers/ -│ ├── transcripts/ -│ └── assets/ -├── wiki/ # Processed content -│ └── concepts/ # Synthesized concept notes -├── personal/ # User notes (read-only for agents) -└── utils/ # Utility scripts -``` +- Downloads the URL or reads the local file +- Archives an untouched copy under `raw/` (duplicates are detected by + SHA-256, so the same bytes are never stored twice) +- Converts it to markdown (HTML, PDF, plain text, and markdown ship today) +- Writes a page under `wiki/` with full provenance in its frontmatter and + `status: raw`, so the agent knows it still needs refining +- Appends the action to `wiki/log.md` + +`wiki ingest --pending` lists every page still waiting to be refined. + +### One-call maintenance + +`fiona-assistant housekeeping` runs the whole maintenance routine and +prints a to-do list for the agent: + +- **check-update**: is the packaged template newer than this wiki? +- **merge**: which `*-update.md` files are waiting to be merged? +- **ingest**: which pages are still `status: raw`? +- **sanity**: which lint checks fail, and where? +- **graph**: regenerates `wiki_graph.html` and lists orphan pages + +It only reads, except for writing the graph. Every finding comes with a +concrete "next step" the agent can act on. Add `--json` for machine use. + +### Quality checks + +`fiona-assistant wiki sanity` runs five checks in parallel and gives one +pass/fail verdict: + +| Check | What it catches | +| --- | --- | +| `markdownlint` | Markdown style, via PyMarkdown | +| `frontmatter_lint` | Missing or malformed YAML frontmatter (`name`, `version`, and more) | +| `python-ruff` | Python issues in wiki scripts, via Ruff | +| `mdlinter` | Dead links, dead `[[wikilinks]]`, and table rows split across lines | +| `jsonlint` | JSON syntax, indentation, trailing commas, duplicate keys | + +Each linter is also available on its own, and `mdoctor fix` and +`json-lint fix` repair what they can automatically. + +### Interactive link graph + +`fiona-assistant wiki graph` writes a single self-contained HTML file +that shows how your pages link to each other. Nodes are scored by how +many different pages point at them, so hub pages stand out. It +understands markdown links and `[[wikilinks]]`, shows external URLs, and +flags broken links and orphans. `--format json` exports the same graph +for other tools. + +### Safe template updates + +As Fiona improves, so does its wiki template. `wiki check-update` shows +what changed, and `wiki update` brings your wiki up to date without ever +overwriting a file you own. When both sides changed, Fiona writes +`-update.md` next to your file and `housekeeping` tells the agent +to merge it. + +### Git history built in + +`fiona-assistant git` wraps the operations an agent needs, with +timestamped commit messages: + +- `save` commits all current edits +- `snapshot ` commits and tags a restore point +- `restore ` returns to a snapshot without losing history +- `undo`, `clean`, and `status` for everyday recovery + +### Guardrails + +- `personal/` and `raw/` are read-only for every automated tool. Ingest + may add new files to `raw/`; it never changes an existing one. +- Commands that need a decision prompt on a terminal and accept + `--yes`, `--force`, or `--non-interactive` for unattended runs. +- Respects `.gitignore` when scanning. + +### Table of contents generation + +`fiona-assistant librarian toc` builds `table_of_contents.md` for a folder +tree. `librarian toc file ` keeps an in-file TOC under the H1 up to +date, with optional section numbering (`--sectionized`). --- -## Installation +## Quick Start -### Quick Install (Production) +### 1. Install + +Requires Python 3.11 or newer. ```bash python -m pip install fiona-assistant ``` -### Development Setup - -- **Clone the repository:** - - ```bash - git clone - cd fiona-assistant - ``` - -- **Create and activate the virtual environment:** - - Linux / macOS: - - ```bash - source bootstrap --env fiona_env -c - ``` - - Windows (PowerShell): - - ```powershell - Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass - . .\bootstrap.ps1 --env fiona_env -c - ``` - - > Make sure `fiona_env` is activated before proceeding. On Windows the - > environment lives in `.venvs\fiona_env`. - -### Local install (not editable) - -Install this checkout as a normal package. The default bootstrap install -is editable (`pip install -e ".[dev]"`). +### 2. Create a wiki ```bash -source bootstrap --local +fiona-assistant wiki init my-wiki +cd my-wiki ``` -```powershell -. .\bootstrap.ps1 --local -``` +### 3. Point your agent at it -### Same as `pip install fiona-assistant` +Open the folder in your AI coding agent (Cursor, Claude Code, or any +agent that reads `AGENTS.md`). The bundled `AGENTS.md` loads the Fiona +skill, which teaches the agent the ingest, refine, and housekeeping +workflow and when to call each command. -`--local --no-env` installs this repository the same way -`python -m pip install fiona-assistant` does. Pip builds a wheel from -this directory and installs it into your user site-packages. The -`fiona-assistant` command then runs with no virtual environment. The -flag name is `--no-env`. +### 4. Feed it sources ```bash -source bootstrap --local --no-env +fiona-assistant wiki ingest https://example.com/post.html +fiona-assistant wiki ingest ./notes/meeting.pdf ``` -```powershell -powershell -ExecutionPolicy Bypass -File .\bootstrap.ps1 --local --no-env -``` +Then ask the agent to refine the new pages. -That command is `python -m pip install --user .` from the repository -root. The package files are the ones in this checkout. If the command -is not found, add the user scripts directory printed by bootstrap to -`PATH`. - ---- - -## Usage - -### Main CLI - -Fiona-Assistant provides a unified CLI with multiple subcommands: +### 5. Keep it healthy ```bash -# Show all available commands -fiona-assistant --help - -# Run via Python module -python -m fiona --help -``` - -#### Available Commands - -```bash -# Run tests -fiona-assistant test - -# Manage wiki content with Librarian -fiona-assistant librarian [directory] - -# Generate an in-file table of contents below the H1 -fiona-assistant librarian toc file - -# Force section numbering on for the run -fiona-assistant librarian toc file --sectionized - -# Generate in-file TOCs for every md file in the current directory / wiki root -fiona-assistant librarian toc file '*' - -# Diagnose markdown files with MDoctor -fiona-assistant mdoctor [chklint] [directory] - -# Lint Python code with Ruff -fiona-assistant python-ruff [chklint] [directory] -``` - -### Direct Tool Access - -Each tool can also be called directly: - -```bash -# Librarian - List all files in a directory -librarian [directory] - -# MDoctor - Check markdown lint violations -mdoctor chklint [directory] - -# MDoctor - List all files - -# Python-Ruff - Check Python code quality -python-ruff chklint [directory] - -# Generate table of contents for markdown files -generate-toc [directory] [options] +fiona-assistant housekeeping +fiona-assistant git save --description "ingest meeting notes" ``` --- -## Workflow Examples +## How It Works -### Wiki Ingestion Workflow - -```bash -# 1. Drop raw files into ingest directory -# (Manual or automated process) - -# 2. List files ready for processing -librarian llm-wiki/ingest - -# 3. Validate markdown quality before processing -mdoctor chklint llm-wiki/ingest - -# 4. Move processed files to raw archive -# (Agent/orchestration task) -``` - -### Content Maintenance Workflow - -```bash -# Check all markdown files in wiki for lint violations -mdoctor chklint llm-wiki/wiki - -# Generate table of contents for concept notes -generate-toc llm-wiki/wiki/concepts --recursive-toc - -# Validate Python code in the project -python-ruff chklint src/ - -# List all markdown files for inventory -librarian llm-wiki/ -``` - -### Standalone Document Organization - -```bash -# Organize any directory of markdown files -librarian /path/to/docs/ - -# Validate markdown formatting -mdoctor chklint /path/to/docs/ - -# Create nested table of contents -generate-toc /path/to/docs/ --recursive-toc -``` - ---- - -## Output Format - -All tools produce **AI-friendly output** with consistent formatting: - -### MDoctor Output Format +A wiki created by Fiona has this layout: ```text -PyMarkdownLint Violations -TARGET: /path/to/directory +my-wiki/ +├── AGENTS.md # Entry point for the agent; loads the Fiona skill +├── fiona-settings.json # Marks the wiki root; tool settings +├── ingest/ # Drop zone for files waiting to be parsed +├── raw/ # Immutable archive of original sources +│ ├── articles/ +│ ├── papers/ +│ ├── transcripts/ +│ ├── assets/ +│ └── other/ +├── wiki/ # Pages the agent writes and maintains +│ ├── schema.md # Page types, tags, conventions +│ ├── index.md # Content catalog +│ ├── log.md # Append-only action log +│ ├── concepts/ +│ ├── entities/ +│ ├── comparisons/ +│ ├── queries/ +│ └── sources/ +├── personal/ # Your own notes (read-only for agents, untracked) +└── utils/ # Agent skills, personas, and helper scripts +``` + +The cycle is: + +- **Ingest**: `wiki ingest` archives a source in `raw/` and writes a raw + page in `wiki/`. +- **Refine**: the agent rewrites the page, links it to related pages, and + sets `status: refined`. +- **Maintain**: `housekeeping` reports what needs attention; the agent + fixes it. +- **Save**: `git save` records the change. + +The full page schema lives in the template's +[schema](./src/fiona/data/llm_wiki/wiki/schema.md). + +--- + +## Command Reference + +Run `fiona-assistant --help` or `fiona-assistant --help` for all +flags. Commands that take an optional directory default to the current +directory. + +### Wiki lifecycle + +| Command | What it does | +| --- | --- | +| `wiki init [dir]` | Create a new llm-wiki with git history | +| `wiki ingest ` | Fetch, archive, and convert a URL or file into a raw page | +| `wiki ingest --pending` | List pages still marked `status: raw` | +| `wiki sanity [dir]` | Run every quality check; exit 1 on failure | +| `wiki graph [dir]` | Write `wiki_graph.html` (or JSON with `--format json`) | +| `wiki check-update [dir]` | Show what a template update would change | +| `wiki update [dir]` | Apply a template update (`--dry-run`, `--yes`) | +| `housekeeping [dir]` | Run the full maintenance routine and list next steps | + +### Linters + +| Command | What it does | +| --- | --- | +| `mdoctor chklint [dir]` | Markdown lint via PyMarkdown | +| `mdoctor fix [dir]` | Auto-fix markdown lint violations | +| `fmlint chklint [dir]` | Frontmatter lint | +| `python-ruff chklint [dir]` | Python lint via Ruff | +| `json-lint chklint [dir]` | JSON lint | +| `json-lint fix [dir]` | Auto-fix JSON indentation and trailing commas | + +### Files and TOC + +| Command | What it does | +| --- | --- | +| `librarian list [dir]` | List files in a directory | +| `librarian toc [dir]` | Write `table_of_contents.md` (`--recursive-toc`, `--exclude REGEX`) | +| `librarian toc file ` | Write an in-file TOC below the H1 (`--sectionized`) | +| `librarian toc file '*'` | Update in-file TOCs for every markdown file in the wiki | + +### Git + +| Command | What it does | +| --- | --- | +| `git save --description TEXT` | Commit all edits | +| `git snapshot --description TEXT` | Commit and tag a restore point | +| `git restore ` | Restore a snapshot as a new commit | +| `git undo` | Drop the last commit | +| `git clean` | Discard uncommitted edits | +| `git status` | Show whether the tree is clean | + +Every command above is run as `fiona-assistant `. The +`librarian` and `mdoctor` tools are also installed as standalone +commands. + +--- + +## Output Built for Agents + +Every report uses the same shape: a header, the target, rule +definitions, findings per file with line and column, and a summary. An +agent can parse it without guessing, and a human can skim it. + +```text +Markdown Lint Violations +TARGET: wiki - DEFINITIONS: - - md001: Heading levels should only increment by one, Heading levels should only increment by one level at a time - - md013: Line length, Line length + - MDL001: Dead Link, Relative markdown link does not resolve to an existing file + - MDL002: Dead Wiki Link, Wiki-internal link does not resolve to any md page in the wiki -- FILE: file1.md - - 10:5 md013 - - 20:3 md001 +- FILE: wiki/concepts/retrieval.md + - MDL001 line 12: ../sources/old-paper.md does not exist + - MDL002 line 30: [[embeddings]] does not resolve to any page - SUMMARY: - Total files with violations: 1 - Total violations: 2 ``` -### Python-Ruff Output Format - -```text -Ruff Violations -TARGET: /path/to/directory - -Definitions: - - F401: `*` import used, but not imported - - E501: Line too long - -- FILE: script.py - - 15:10 F401 - - 25:80 E501 - -- SUMMARY: - - Total files with violations: 1 - - Total violations: 2 -``` +`wiki sanity`, `housekeeping`, `wiki ingest`, `wiki check-update`, and +`wiki update` also accept `--json` for structured output. --- -## Project Structure +## Roadmap + +Planned work, tracked with requirements and acceptance criteria in +[SPEC](./SPEC.md) and prioritized in [todo](./todo.md): + +| Feature | What it will do | +| --- | --- | +| Wiki root discovery | Run any command from a subfolder; Fiona finds the nearest `fiona-settings.json` | +| Backup | One command writes a compressed, restorable archive of the wiki outside the tree | +| `setup` command | Interactive review and editing of `fiona-settings.json`, with validation | +| Smarter graph | Show only important nodes until you expand a parent; hide gitignored pages | +| Code blocks in markdown | Lint Python and shell snippets inside fenced code blocks | +| Shellcheck | Lint shell scripts as part of wiki sanity | +| HTML and JavaScript lint | Validate HTML and JS files in the wiki | +| `python-ruff fix` | Auto-fix Python lint findings | +| More converters | Ingest more source formats beyond HTML, PDF, text, and markdown | +| Version tags for scripts | Track versions of Python and shell files like markdown pages | + +--- + +## Development + +### Set up + +Clone the repository and load the environment with the bootstrap script. + +Linux and macOS: + +```bash +git clone +cd fiona-assistant +source bootstrap --env fiona_env -c +``` + +Windows (PowerShell): + +```powershell +Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass +. .\bootstrap.ps1 --env fiona_env -c +``` + +The default install is editable (`pip install -e ".[dev]"`). On Windows +the environment lives in `.venvs\fiona_env`. + +### Other install modes + +| Goal | Bash | PowerShell | +| --- | --- | --- | +| Install this checkout as a normal package | `source bootstrap --local` | `. .\bootstrap.ps1 --local` | +| Same as `pip install fiona-assistant`, no venv | `source bootstrap --local --no-env` | `powershell -ExecutionPolicy Bypass -File .\bootstrap.ps1 --local --no-env` | + +`--local --no-env` runs `python -m pip install --user .`. If +`fiona-assistant` is not found afterwards, add the user scripts +directory printed by bootstrap to `PATH`. + +### Test and check + +```bash +# Unit tests with coverage +python scripts/run_tests.py + +# Full sanity: tests, coverage gate, markdown, Ruff, frontmatter, JSON +python scripts/run_sanity.py --non-interactive +``` + +Contributions follow the spec-first, test-first lifecycle in +[AGENTS](./AGENTS.md). + +### Project structure ```text fiona-assistant/ -├── src/fiona/ # Installable package -│ ├── __main__.py # Unified command dispatcher -│ ├── librarian/ # Wiki content management -│ ├── mdoctor/ # Markdown diagnosis -│ ├── python_ruff/ # Python code linting -│ ├── customlinter/ # Custom linters -│ ├── gittool/ # Git operations -│ ├── sanity_runner/ # Shared sanity engine -│ ├── wiki/ # Wiki lifecycle -│ └── data/llm_wiki/ # Packaged wiki template -│ ├── AGENTS.md # Agent rules and directory permissions -│ ├── ingest/ # Input zone for new content -│ ├── raw/ # Archived source materials -│ ├── wiki/ # Processed concept notes -│ └── utils/ # Utility scripts -├── tests/ # Unit tests (not installed) -├── scripts/ # Repo tooling (not installed) -│ ├── run_tests.py -│ ├── run_sanity.py -│ └── settings_dev.json -├── pyproject.toml # Project configuration -├── bootstrap # Environment setup script (bash) -├── bootstrap.ps1 # Environment setup script (PowerShell) -├── LICENSE # GPL-3.0-only -└── README.md # This file +├── src/fiona/ # The installable package +│ ├── __main__.py # fiona-assistant CLI +│ ├── librarian/ # File listing and TOC generation +│ ├── mdoctor/ # Markdown lint and fix +│ ├── customlinter/ # Frontmatter, markdown link, Python, JSON linters +│ ├── python_ruff/ # Ruff wrapper +│ ├── gittool/ # Git operations +│ ├── sanity_runner/ # Shared sanity engine +│ ├── wiki/ # init, update, sanity, graph, ingest, housekeeping +│ └── data/llm_wiki/ # Packaged wiki template +├── tests/ # Unit tests (not installed) +├── scripts/ # Test and sanity runners, dev settings +├── bootstrap # Environment setup (bash) +├── bootstrap.ps1 # Environment setup (PowerShell) +├── SPEC.md # Requirements and acceptance criteria +└── pyproject.toml ``` --- -## Testing +## Documentation -### Run Tests - -```bash -# Run all tests -python scripts/run_tests.py - -# Or use pytest directly -python -m pytest - -# Run with coverage -python -m pytest --cov=src/fiona -``` +- [Project Specification](./SPEC.md): every command, setting, and rule, + with acceptance criteria +- [LLM Wiki Agent Rules](./src/fiona/data/llm_wiki/AGENTS.md): what a + deployed wiki tells its agent +- [Fiona skill](./src/fiona/data/llm_wiki/utils/skills/fiona/SKILL.md): + the workflow the agent follows inside a wiki +- [Development Agent Rules](./AGENTS.md): how to contribute to this + repository +- [Roadmap](./todo.md): planned and in-progress work --- ## License GPL-3.0-only - ---- - -## Key Concepts - -### For LLM Agents - -- **Deterministic Operations**: Use Fiona-Assistant for file listing, linting, and organization -- **Token Efficiency**: Reserve LLM context for synthesis, analysis, and creative tasks -- **Structured Output**: All tool outputs are formatted for easy parsing by LLMs - -### For Human Users - -- **CLI-First**: All functionality available through command-line interface -- **Composable Tools**: Combine tools in scripts for automated workflows -- **Clear Structure**: Maintains consistent wiki organization standards - ---- - -## Related Documentation - -- [LLM Wiki Agent Rules](./src/fiona/data/llm_wiki/AGENTS.md) - Directory permissions and workflow rules -- [Development Setup](./src/README.md) - Detailed development information -- [Project Specification](./SPEC.md) - Feature spec sheet with requirements and acceptance criteria -- [Project TODO](./todo.md) - Roadmap and planned features - ---- - -## Quick Reference Card - -| Task | Command | -| ---- | ------- | -| Show help | `fiona-assistant --help` | -| List files | `fiona-assistant librarian /path/to/dir` | -| Check MD lint | `fiona-assistant mdoctor chklint /path/to/dir` | -| Check Python lint | `fiona-assistant python-ruff chklint /path/to/dir` | -| Generate TOC | `generate-toc /path/to/dir --recursive-toc` | -| Generate file TOC | `fiona-assistant librarian toc file /path/to/file.md --sectionized` | -| Run tests | `python scripts/run_tests.py` | -| Install | `pip install fiona-assistant` | -| Dev install | `pip install -e .` | -| Local install | `bootstrap --local` | -| Install like `pip install fiona-assistant` | `bootstrap --local --no-env` | - ---- - -> **Note**: All paths are relative to the current working directory unless specified as absolute paths. diff --git a/SPEC.md b/SPEC.md index 1715ce3..c05782b 100644 --- a/SPEC.md +++ b/SPEC.md @@ -1,3131 +1,3223 @@ ---- -name: Fiona-Assistant Specification -version: 1.1.0 -title: Project Spec Sheet -created: 2026-09-23 -updated: 2026-10-01 -type: summary ---- - -# Fiona-Assistant Specification - - - -- [1. Product Overview](#1-product-overview) - - [1.1. Document Conventions](#11-document-conventions) - - [1.2. Constraints](#12-constraints) -- [2. Package and Runtime](#2-package-and-runtime) -- [3. llm-wiki Template](#3-llm-wiki-template) - - [3.1. What Counts as a Wiki](#31-what-counts-as-a-wiki) - - [3.2. Core Files and Read-only Zones](#32-core-files-and-read-only-zones) - - [3.3. Deployed Wiki Rules](#33-deployed-wiki-rules) -- [4. Command Catalog](#4-command-catalog) - - [4.1. librarian](#41-librarian) - - [4.1.1. librarian toc](#411-librarian-toc) - - [4.2. mdoctor](#42-mdoctor) - - [4.3. fmlint](#43-fmlint) - - [4.4. python-ruff](#44-python-ruff) - - [4.5. json-lint](#45-json-lint) - - [4.6. git](#46-git) - - [4.7. wiki](#47-wiki) - - [4.7.1. wiki init](#471-wiki-init) - - [4.7.2. wiki check-update](#472-wiki-check-update) - - [4.7.3. wiki update](#473-wiki-update) - - [4.7.4. wiki sanity](#474-wiki-sanity) - - [4.7.5. wiki graph](#475-wiki-graph) - - [4.7.6. wiki ingest](#476-wiki-ingest) - - [4.8. housekeeping](#48-housekeeping) - - [4.9. Repository runners](#49-repository-runners) -- [5. Settings Schema](#5-settings-schema) - - [5.1. fiona-settings.json](#51-fiona-settingsjson) - - [5.2. settings_dev.json](#52-settings_devjson) - - [5.3. constellation-settings.json](#53-constellation-settingsjson) -- [6. Violation Rule Catalog](#6-violation-rule-catalog) - - [6.1. Shared Report](#61-shared-report) - - [6.2. Frontmatter Fields and FML Codes](#62-frontmatter-fields-and-fml-codes) - - [6.3. MDL Codes](#63-mdl-codes) - - [6.4. PYC Codes](#64-pyc-codes) - - [6.5. JSN Codes](#65-jsn-codes) - - [6.6. Base Linter Contract](#66-base-linter-contract) -- [7. Feature Requirements](#7-feature-requirements) - - [7.1. wiki](#71-wiki) - - [7.1.1. wiki init](#711-wiki-init) - - [7.1.2. wiki update](#712-wiki-update) - - [7.1.3. wiki sanity](#713-wiki-sanity) - - [7.1.4. wiki graph](#714-wiki-graph) - - [7.1.5. wiki ingest](#715-wiki-ingest) - - [7.2. housekeeping](#72-housekeeping) - - [7.3. librarian](#73-librarian) - - [7.3.1. Folder and In-file TOC](#731-folder-and-in-file-toc) - - [7.3.2. Non-markdown Version Tags](#732-non-markdown-version-tags) - - [7.4. mdoctor](#74-mdoctor) - - [7.5. fmlint](#75-fmlint) - - [7.6. mdlinter](#76-mdlinter) - - [7.7. pycustomlint](#77-pycustomlint) - - [7.8. python-ruff](#78-python-ruff) - - [7.9. Shellcheck](#79-shellcheck) - - [7.10. Markdown Code Blocks](#710-markdown-code-blocks) - - [7.11. json-lint](#711-json-lint) - - [7.12. Git Operations](#712-git-operations) - - [7.13. Backup Script](#713-backup-script) - - [7.14. Test and Sanity Runners](#714-test-and-sanity-runners) - - [7.15. html-js-lint](#715-html-js-lint) - - [7.16. settings setup](#716-settings-setup) - - [7.17. Logging](#717-logging) - - [7.18. Documentation](#718-documentation) - - [7.19. Tool Roles](#719-tool-roles) - - [7.20. Wiki Root Discovery](#720-wiki-root-discovery) - - [7.21. Bootstrap install modes](#721-bootstrap-install-modes) -- [Appendix A. Non-Goals](#appendix-a-non-goals) -- [Appendix B. Priority Order](#appendix-b-priority-order) -- [Appendix C. Open Questions](#appendix-c-open-questions) -- [Appendix D. Maintainer Notes](#appendix-d-maintainer-notes) - - - -This document is the build contract for Fiona-Assistant. A reader with -this file, and no source tree, must be able to produce the same -observable software. Status values are `Implemented`, `Partially -Implemented`, `Planned`, and `Won't Do`. - -Revision 0.40.0 reorganizes the sheet into contracts. Revision 0.40.1 -numbers nested requirements such as `7.14.1.1`. Revision 0.40.4 adds -the PYC001 exception in *6.4.2.*. Revision 0.41.0 installs one -`fiona` package and keeps tests and repo scripts outside it. -Revision 0.90.0 specifies `node_importance_min_point` in *7.1.4.16.* -and leaves that requirement unbuilt. Revision 0.92.0 specifies -`wiki ingest` in *7.1.5.* and leaves it unbuilt. Revision 0.93.0 adds -SHA-256 reuse across `raw/`, the complete `ingest` frontmatter map, -and the agent metadata check. Revision 0.94.0 makes `wiki update` -yield to the owner's markdown and write `-update.md` instead -(*7.1.2.11.*), and adds `fiona-assistant housekeeping` (*7.2.*). -Revision 0.96.0 compares a file without a version by SHA-256 and -shows both short hashes in the update report (*7.1.2.12.*). -Revision 0.97.0 rejects a module-level assignment that follows the -first function or class (*6.4.3.*). Revision 0.98.0 specifies that -`wiki graph` omits a file matched by `.gitignore` (*7.1.4.17.*) and -leaves that requirement unbuilt. Revision 0.99.0 builds `wiki ingest` -(*7.1.5.*). Revision 1.0.0 groups each module's functions under that -module and moves Non-Goals, Priority Order, and Open Questions -to Appendices A, B, and C. Revision 1.1.0 treats a differing -`fiona-settings.json` as a conflict and writes -`fiona-settings-update.json` (*7.1.2.3.*). - -Each feature section starts with a `Status` line. Requirements use -`- *N.M.K.*`. Each `*acceptance*` sits under the requirement it tests. -`NOT_IMPLEMENTED` marks work that is specified and not built yet. -Class names, file paths, and history live in Appendix D. - ---- - -## 1. Product Overview - -Fiona-Assistant is a CLI toolkit that manages, organizes, and checks -LLM wiki content with deterministic scripts. Reserve LLM tokens for -creative and analytical work. - -### 1.1. Document Conventions - -> Status: Implemented - -- *1.1.1.* Main sections use `## N. Title`. Subsections use - `### N.M. Title`. Deeper headings use `#### N.M.K. Title`. - - *acceptance*: Every heading follows that numbering. -- *1.1.2.* A requirement is `- *N.M.K.* `. A group is numbered - the same way, and each item under it adds one segment, for example - `*7.14.1.*` then `*7.14.1.1.*`. Its acceptance is the next nested - `- *acceptance*: `, and that acceptance describes this - requirement only. - - *acceptance*: No acceptance describes a neighboring requirement. - - *acceptance*: The shared-engine bullets in section 7.14 are - numbered `7.14.1` and `7.14.1.1`, not plain dashes. -- *1.1.3.* Unbuilt requirements start with `` `NOT_IMPLEMENTED` ``. - - *acceptance*: Every planned item carries that tag. -- *1.1.4.* The block between `` and - `` is generated by `librarian toc file`. Leave that - block to the generator. - -### 1.2. Constraints - -These apply to every feature. - -- *1.2.1.* Tool reports are parseable and use the shared shapes in - sections 4, 6, and 7. - - *acceptance*: Two linters that find violations print the same - section order from section 6.1. -- *1.2.2.* A command that needs a human decision prompts on a TTY. - The same command accepts an explicit flag (`--force`, `--yes`, or - `--non-interactive`) and then runs without a prompt. - - *acceptance*: `wiki update` on a non-TTY declines unless `--force` - or `--yes` is set. -- *1.2.3.* `personal/` and `raw/` are read-only for every automated - tool. Tools may resolve links that point into those zones. The one - exception is `wiki ingest` (*7.1.5.3.*): it may add a new file under - `raw/`. It never modifies, renames, or deletes a file already there. - - *acceptance*: `wiki update` leaves every file under `personal/` - and `raw/` byte-for-byte unchanged. - - *acceptance*: A second `wiki ingest` run leaves every file that - was already under `raw/` byte-for-byte unchanged. - - *acceptance*: Ingesting bytes whose SHA-256 matches any file - already under `raw/`, from any path or URL, adds no file under - `raw/` (*7.1.5.3.3.*). -- *1.2.4.* Code changes follow test-first development. - `python scripts/run_tests.py` is green, then - `python scripts/run_sanity.py` is green, before a change is finished. - - *acceptance*: A sanity run below the coverage gate in section 5.2 - fails even when every test passes. -- *1.2.5.* Python passes Ruff. Markdown passes PyMarkdown with the - rules in section 5.1. - - *acceptance*: An unused import is reported by `python-ruff chklint` - with file, line, column, and rule code. -- *1.2.6.* Authored lengths in HTML and CSS use `em` or another - relative unit (`rem`, `%`, `vh`, `vw`). They are not written in - `px`. The wiki graph page in section 7.1.4 is that web surface. - A position read from the layout engine is written back in `rem`. - A theme length number uses the same scale: 16 is `1em`. - - *acceptance*: The graph template contains no `px` length. - ---- - -## 2. Package and Runtime - -> Status: Implemented - -Python `>= 3.11` (also 3.12 and 3.13). The default development -install from the repository root is `pip install -e ".[dev]"`. -`bootstrap --local` installs the same tree as a normal package -(`pip install .`). Section 7.21 defines those modes. The build -requires setuptools 61 or newer. - -Runtime dependencies: - -| Package | Minimum | -| --- | --- | -| pymarkdownlnt | 0.9.0 | -| ruff | 0.16.8 | -| pyyaml | 6.0.3 | -| GitPython | 3.1.0 | - -Development extras: `pytest>=7.0`, `pytest-cov>=4.0`, `pytest-mock`, -`pylint`. `html5validator` is not a dependency until section 7.15 -is implemented. `beautifulsoup4`, `markdownify`, and `pypdf` become -runtime dependencies when section 7.1.5 is implemented. - -One importable package, `fiona`, lives under `src/fiona/`. Its -subpackages are `librarian`, `mdoctor`, `customlinter`, -`python_ruff`, `gittool`, `sanity_runner`, and `wiki`. The wiki -template is package data at `src/fiona/data/llm_wiki/` and is loaded -with `importlib.resources`. It is not an importable module. Tests -live in repo-root `tests/` and are not installed. `scripts/` holds -`settings_dev.json`, shell helpers, and the launchers -`scripts/run_tests.py` and `scripts/run_sanity.py`. - -Console scripts: - -| Script | Entry | -| --- | --- | -| `fiona-assistant` | `fiona.__main__:main` | -| `librarian` | `fiona.librarian.librarian:main` | -| `mdoctor` | `fiona.mdoctor.mdoctor:main` | - -`python-ruff`, `fmlint`, `json-lint`, and `git` are reached only -through `fiona-assistant`. `html-js-lint` is not installed until -section 7.15 is implemented. - -Coverage measures `src/fiona`, omitting tests. The config lives in -`pyproject.toml`. The gate percentage is section 5.2. - -Logging: every CLI entry calls `setup_logging()` before work. Level -order is `LOG_LEVEL` environment variable, then `fiona.log_level` in -`fiona-settings.json`, then `WARNING`. Records go to stderr. When -`fiona.file_log_location` is set, a file handler writes lines -`%(asctime)s %(levelname)s %(name)s: %(message)s` at or above -`fiona.file_log_level` (fallback: `fiona.log_level`). A relative path -is resolved from the current working directory. An unopenable path -logs a warning and keeps stderr logging. Human reports stay on -stdout. Package loggers set their level explicitly because -pymarkdown resets the root logger mid-run. - -- *2.1.* The three console scripts above are installed by - `pip install -e .`. - - *acceptance*: `fiona-assistant` with no arguments prints help and - exits 0. -- *2.2.* `html-js-lint` is not an installed script until section 7.15 - exists. - - *acceptance*: A fresh install does not fail at import because of - a missing `html_js_lint` package. - ---- - -## 3. llm-wiki Template - -> Status: Implemented - -`wiki init` copies this tree from the packaged template. Text files -replace every `DATE_PLACEHOLDER` with the current UTC date -`YYYY-MM-DD`. Binary files are copied unchanged. A glob in -`fiona.llm-wiki.ignore_files` skips that file, or that directory and -its subtree. A missing settings file or a missing key copies -everything. A non-list `ignore_files` value raises `ValueError` with -the text `fiona.llm-wiki.ignore_files must be a list of strings`. - -```text -AGENTS.md -README.md -fiona-settings.json -table_of_contents.md -ingest/README.md -personal/README.md -raw/README.md -wiki/schema.md -wiki/index.md -wiki/log.md -utils/README.md -utils/scripts/example.py -utils/scripts/map.md -utils/skills/README.md -utils/skills/fiona/SKILL.md -utils/skills/fiona/markdown_rules.md -utils/skills/fiona/llm-wiki-mod.md -utils/skills/python-tdd/SKILL.md -utils/personas/default/SOUL.md -utils/personas/template/SOUL.md -utils/skills_local/README.md -``` - -The shipped `fiona-settings.json` is the default document in -section 5.1. `wiki/schema.md` is the page-type and tag taxonomy. -This spec's frontmatter table in section 6.2 is the field contract; -where the two disagree, this spec wins. - -### 3.1. What Counts as a Wiki - -A directory is an llm-wiki when it contains `fiona-settings.json`. -`wiki init` refuses when any of these already exist: -`AGENTS.md`, `fiona-settings.json`, `wiki/`, `ingest/`, `README.md`. - -### 3.2. Core Files and Read-only Zones - -Core files, matched by name at the wiki root: - -- `AGENTS.md` -- `fiona-settings.json` -- `README.md` -- `table_of_contents.md` - -Core directories (every file under them is core): - -- `wiki/` -- `ingest/` -- `personal/` -- `raw/` -- `utils/` - -Read-only zones, including the zone directory itself: - -- `personal/` -- `raw/` - -`wiki/` is copied by `wiki init` and then left to the LLM. -`wiki update` does not add, replace, or delete any path under -`wiki/`. The only file it writes there is an update file for a -template page that is newer than the wiki's copy (*7.1.2.11.*). - -`wiki update` adds missing core files and directories. It never -overwrites an existing markdown file: when the template carries a -newer version, it writes `-update.md` beside the file and -leaves the owner's file as it is. A differing -`fiona-settings.json` is the same kind of conflict: the owner's -file stays, and the template is written as -`fiona-settings-update.json` (*7.1.2.3.*). It never deletes user files. It -never writes inside a read-only zone, so a missing file under -`personal/` or `raw/` is not added back. It -never writes a path matched by the repository `.gitignore` or by -the wiki's own `.gitignore`. - -After the copy, `wiki init` creates a git repository and writes -`.gitignore`. A new file starts with the fiona-assistant repository -`.gitignore` (the same patterns, including `__pycache__/`, `*.pyc`, -and `*fiona.log`). When this package is not the source checkout, -that text is the packaged `default.gitignore`. The file adds -`personal/` when that pattern is missing. -An existing `.gitignore` keeps every line it already has; missing -patterns from that set are appended. Init then stages everything -and commits `Initial llm-wiki commit`. Template paths that match -the repository `.gitignore` are not copied. `personal/` and `raw/` -are still copied. `personal/` stays untracked. If git init fails, -the copy still counts as success and stdout adds -`Note: Git repository initialization failed: `. - -- *3.2.1.* An empty directory becomes the tree above, with dates - substituted, and a git commit that does not track `personal/`. - - *acceptance*: `personal/` is absent from `git ls-files` after - `wiki init`. -- *3.2.2.* A second `wiki init` raises `FileExistsError` whose - message names each conflicting indicator, and writes nothing. - - *acceptance*: An existing `README.md` alone is enough to refuse, - and that file's bytes are unchanged. - -### 3.3. Deployed Wiki Rules - -> Status: Implemented - -Only the template tree is deployed. A wiki does not know this spec, -the root `AGENTS.md`, `scripts/`, or `tests/`. The template -`AGENTS.md` is the entry point: it says what the wiki is and loads -the Fiona skill, which loads `llm-wiki-mod.md`. The product rule -"never commit without permission" (project agent rule 2.4) covers -this repository only. - -- *3.3.1.* No markdown file in the template references a file or - rule outside the template: `SPEC`, `project agent rule`, or - `scripts/run_`. - - *acceptance*: A scan of every template `*.md` finds none of those - strings. -- *3.3.2.* The template `AGENTS.md` links the Fiona skill and - `llm-wiki-mod.md`. - - *acceptance*: Both relative links appear in that file. -- *3.3.3.* A deployed wiki commits with - `fiona-assistant git save --description TEXT` after each ingest - cycle and each housekeeping pass. `clean`, `undo`, and `restore` - run only when the user asks. - - *acceptance*: The template `AGENTS.md` and the Fiona skill both - contain `git save`, and neither says to commit only when asked. -- *3.3.4.* The template `wiki/log.md` holds one seed entry dated - `DATE_PLACEHOLDER`. - - *acceptance*: The log body has exactly one bullet. - ---- - -## 4. Command Catalog - -> Status: Implemented - -One command tree. Optional `[directory]` defaults to `.` today. -Wiki-root discovery (section 7.20) is not built. An invalid -directory prints `Error: '' is not a valid directory.` on -stderr and exits 1, unless a command below states a different -message. - -`fiona-assistant` with no command prints help and exits 0. -`fiona-assistant git` with no subcommand prints git help and exits 1. -`fiona-assistant wiki` with no subcommand prints wiki help and -exits 1. - -### 4.1. librarian - -`fiona-assistant librarian` and `librarian` with no subcommand list -the current directory. - -`list [directory]` stdout: - -```text -Found files in '': - - -``` - -`mdoctor list` uses the same listing with the prefix -`MDoctor: Found files in '':`. - -#### 4.1.1. librarian toc - -`toc [directory] [flags]` writes `table_of_contents.md` in the -target folder. Flags: `--recursive-toc`, `--non-recursive`, and -repeatable `--exclude REGEX`. Depth comes from section 5.1. - -A path that is a file or missing exits 1 on stderr and names -`librarian toc file `. A folder with no markdown prints -`no markdown files found` on stdout and writes nothing. In -`--recursive-toc` mode that message is a log line at INFO instead. - -`toc file ` writes an in-file TOC in that markdown file. -`toc file *` does that for every markdown file under the wiki root -found by section 7.3.1. `--sectionized` forces section numbering for -that run. Status lines: - -```text -wrote toc in -toc already up to date in -skipped () -generated toc for file(s), skipped -``` - -The summary line is bulk mode only. A file with no H1 is -`skipped (no H1 heading)` and is not rewritten. - -### 4.2. mdoctor - -No subcommand means `chklint .`. - -`chklint [directory]` prints the shared report (section 6.1) with -header `PyMarkdownlint Violations` and exits 0 even when violations -exist. - -`fix [directory]` runs pymarkdown auto-fix in place. Stdout: - -```text -PyMarkdownFix: file(s) fixed - - -``` - -A pymarkdown API failure prints -`Error: pymarkdown fix failed: ` on stderr and exits 1. -The command does not skip `personal/` or `raw/`. - -### 4.3. fmlint - -`fiona-assistant fmlint` with no subcommand means `chklint .`. -`chklint [directory]` prints the shared report with header -`Frontmatter Lint Violations` and exits 0 even when violations -exist. A missing directory exits 1. - -### 4.4. python-ruff - -No subcommand means `chklint .`. The report header is -`Ruff Violations`. Exit code is 0 when the ruff process itself -succeeds, including when violations are listed. `lint(path, fix=True)` -passes `--fix` to the ruff binary. There is no `fix` subcommand yet. - -### 4.5. json-lint - -No subcommand means `chklint .`. Header is `JSONLint Violations`. - -`chklint [directory]` exits 1 when the report contains violations -and exits 0 when it reports no violations. This differs from -mdoctor, fmlint, and python-ruff, which exit 0 either way. - -`chklint --fix` and `fix [directory]` rewrite indentation and -trailing commas in place, skip `personal/` and `raw/`, and print: - -```text -JSONLintFix: file(s) fixed - - -``` - -A missing directory on `fix` prints -`Error: '' does not exist.` on stderr and exits 1. - -### 4.6. git - -Operates on the git repository at the current directory. Timestamps -are UTC. - -| Command | Stdout on success | -| --- | --- | -| `save --description TEXT` | `Committed: ` | -| `snapshot --description TEXT` | `Snapshot created: commit=, tag=` | -| `restore ` | `Restored to: ` | -| `clean` | `Working tree cleaned. All changes discarded.` | -| `undo` | `Undo to: ` | -| `status` | `Status: clean` or `Status: dirty`, then `git status` | - -On failure the tool prints `Error: ` and exits 1. - -Commit message: ` `. -`save` runs `git add --all` and commits. A clean tree, including -no untracked files, raises `Working tree is clean. Nothing to commit.` - -`snapshot` commits, then creates an annotated tag -`_` with message -`Snapshot: `. - -`restore` requires a clean tree -(`Working tree is dirty. Use 'clean' to discard current edits first.`). -It accepts the full tag or the user tag suffix. The latest tag whose -name equals the argument or ends with `_` wins. It checks -that commit's tree into the index and working tree and adds a new -commit ` Restore to `. History before that -commit stays. - -`clean` requires a dirty tree -(`Working tree is already clean. Nothing to discard.`), then -`git reset --hard HEAD` and `git clean -fd`. - -`undo` requires a clean tree and at least two commits -(`Cannot undo: only one commit exists in the repository.`), then -`git reset --hard HEAD~1`. - -### 4.7. wiki - -Subcommands of `fiona-assistant wiki`. - -#### 4.7.1. wiki init - -`fiona-assistant wiki init [directory]`. Success stdout: - -```text -Successfully initialized llm-wiki directory at: -``` - -Rules are section 3. - -#### 4.7.2. wiki check-update - -`fiona-assistant wiki check-update [directory] [--json]`. -The directory must contain `fiona-settings.json`, else stderr -`Error: Directory '' is not an llm-wiki directory` and exit 1. -A missing directory is -`Error: Directory not found: ` and exit 1. -A successful check exits 0 whether or not updates exist. - -Human stdout when current: - -```text -Wiki is up to date. -``` - -When something differs, the lines that apply: - -```text -Files to update: - - ( ->