4 Commits

Author SHA1 Message Date
nvaldez 579f37f112 implement shell check 2026-09-29 14:33:29 +02:00
nvaldez e0ad912eb4 update todo with the audit 2026-09-29 10:44:57 +02:00
nvaldez a745ba92c1 update the spec 2026-09-29 10:10:20 +02:00
nvaldez c8b98dbbd6 windows bootstrap implemented 2026-09-29 10:04:57 +02:00
161 changed files with 8339 additions and 27604 deletions
+19
View File
@@ -0,0 +1,19 @@
[run]
source = src
omit =
src/tests/*
*/tests/*
*/test_*.py
src/mdoctor/fmlint.py
src/python_ruff/pycustomlint.py
src/llm-wiki/__init__.py
[report]
exclude_lines =
pragma: no cover
def __repr__
raise AssertionError
raise NotImplementedError
if __name__ == .__main__.:
if TYPE_CHECKING:
@abstractmethod
+1 -9
View File
@@ -35,7 +35,6 @@ dist/
# Coverage
.coverage
coverage.xml
htmlcov/
# Jupyter Notebook checkpoints
@@ -47,17 +46,10 @@ htmlcov/
# Windows
Thumbs.db
# Obsidian per-user vault state
.obsidian/workspace*.json
.obsidian/cache
# Python
*.pyc
*.pyo
*.pyd
*fiona.log
*wiki_graph.html
# Generated review artifacts (git diff HTML, scratch exports)
tmp/
*wiki_graph.html
+86 -163
View File
@@ -1,23 +1,21 @@
---
name: Fiona-Assistant Development Agent Rules
version: 1.9.3
title: Agents
name: Fiona-Assistant Agent Rules
version: 1.3.2
title: LLM Agent Rules
type: summary
created: 2025-09-23
updated: 2026-10-06
updated: 2026-09-29
tags: [agents, rules, guidelines, llm, ai]
sources: []
---
# Agents
# Fiona-Assistant LLM Agent Rules
- [1. Overview](#1-overview)
- [1.1 Rule References](#11-rule-references)
- [2. Basic Rules (MANDATORY)](#2-basic-rules-mandatory)
- [2.1 Development Lifecycle](#21-development-lifecycle)
- [2.2 Testing](#22-testing)
- [2.3 Documentation](#23-documentation)
- [2.4 Commit](#24-commit)
- [2.1 Testing](#21-testing)
- [2.2 Documentation](#22-documentation)
- [3. Environment Setup (MANDATORY)](#3-environment-setup-mandatory)
- [3.1 Always Use Bootstrap for Environment](#31-always-use-bootstrap-for-environment)
- [4. Python Development Rules](#4-python-development-rules)
@@ -57,80 +55,34 @@ Fiona-Assistant repository.
Refer to rules in this file as `project agent rule N.M` (e.g.
`project agent rule 1.1`) to differentiate them from rules in other
AGENTS.md files (e.g. the llm-wiki template's `src/fiona/data/llm_wiki/AGENTS.md`).
AGENTS.md files (e.g. the llm-wiki template's `src/llm-wiki/AGENTS.md`).
---
## 2. Basic Rules (MANDATORY)
### 2.1 Development Lifecycle
### 2.1 Testing
Every change that alters behavior, adds a feature, or changes a
requirement follows these five steps, in order. Finish a step before
starting the next. A read-only question skips this lifecycle.
Before step 2, load the environment (project agent rule 3.1).
1. **Write or check the spec.** Read [SPEC](./SPEC.md) for the
area this task will change.
- When the requirement or its `*acceptance*` is missing, write
both in [SPEC](./SPEC.md) before any test or code. Follow
SPEC 1.1 and set the section status.
- When the requirement exists, confirm the task matches that
text. That text is the behavior to build.
2. **Write tests.** Add tests that fail for that acceptance
(project agent rule 6.1). Leave production code unchanged in
this step.
3. **Implement, TDD style.** Load the
[python-tdd skill](./src/fiona/data/llm_wiki/utils/skills/python-tdd/SKILL.md)
(project agent rule 4.1).
- Red: run the new tests and confirm they fail.
- Green: write the smallest change that makes them pass.
- Refactor while those tests stay green.
4. **Verify the spec.** Re-read the requirement and each
acceptance this task touched. The code and the tests satisfy
that text.
- When they disagree, update [SPEC](./SPEC.md) in this same
task. Move status along `Planned` ->
`Partially Implemented` -> `Implemented`, or set
`Won't Do` when the requirement is rejected.
- Run `python scripts/run_tests.py`, then
`python scripts/run_sanity.py --non-interactive`. Both pass
(project agent rule 2.2).
5. **Bump versions.** On each markdown file this task edited,
bump frontmatter `version` and set `updated` to the edit date
(project agent rule 10.1).
- Semver: patch for a clarification, minor for a new rule or
requirement, major for a breaking contract change.
- Bump [SPEC](./SPEC.md) when step 1 or step 4 changed it.
- Python and shell files have no version tag yet (SPEC 7.3.2.1).
Leave them untagged.
A documentation edit that does not change product behavior still
bumps versions (step 5). It skips the test and implementation steps.
### 2.2 Testing
- For testing scripts always run `python scripts/run_tests.py` and make sure
- For testing scripts always run `python src/run_tests.py` and make sure
everything is passing on every loop
- After modifying the repo, make sure
`python scripts/run_sanity.py --non-interactive` passes before finalizing.
- After Modifying the repo, make sure `python src/run_sanity.py --non-interactive` passes before finalizing.
### 2.3 Documentation
### 2.2 Documentation
- Product behavior is specified in [SPEC](./SPEC.md). The order for
reading it, writing it, and checking it after the code is project
agent rule 2.1.
- When documentation changes (for example [SPEC](./SPEC.md), especially
the table of contents), confirm every reference to it still resolves.
- Always link with a markdown link.
- Before changing any functionality, read [SPEC](./SPEC.md) (the project
spec sheet) so the change follows the specified requirements and
acceptance criteria.
- When documentation is changed (e.g., `SPEC.md`, especially the table of
contents section), double-check that all references to it are unbroken.
- Always link via markdownlink for connectivity.
- Whenever a change alters behavior, adds a feature, or changes a
requirement, update [SPEC](./SPEC.md) in the same task so the spec stays
in sync (adjust section status: `planned` -> `partial` -> `implemented`;
use `won't do` for rejected features).
### 2.4 Commit
### 2.3 Commit
- *NEVER* commit without explicit permission!
- This rule covers this repository only. A deployed wiki commits
after each ingest cycle and housekeeping pass
([SPEC](./SPEC.md) 3.3.3).
---
@@ -154,21 +106,10 @@ source bootstrap --env <env_name> --python <version>
# With cleanup flag
source bootstrap --env <env_name> -c
# This checkout as a normal package (not editable)
source bootstrap --local
# Current user, no virtual environment
source bootstrap --local --no-env
```
Windows (PowerShell). A virtual-environment install must be
dot-sourced. This creates a repo-local venv at `.venvs\<name>` with
the Python launcher. It does not install pyenv. `--no-env` skips that
venv and may be executed with `-File`. `--local` installs this
repository with `pip install .` (not editable, no `dev` extra).
`--no-env` installs with `pip install --user`, so `fiona-assistant`
runs without activating an environment. The two flags combine.
Windows (PowerShell, must be dot-sourced). This creates a repo-local venv
at `.venvs\<name>` with the Python launcher. It does not install pyenv.
```powershell
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
@@ -176,8 +117,6 @@ Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
. .\bootstrap.ps1 --env fiona_env
. .\bootstrap.ps1 --env <env_name> --python <version>
. .\bootstrap.ps1 --env <env_name> -c
. .\bootstrap.ps1 --local
powershell -ExecutionPolicy Bypass -File .\bootstrap.ps1 --local --no-env
```
**Rationale**: The bootstrap script sets up the Python virtual environment,
@@ -193,13 +132,11 @@ dependency conflicts, incorrect Python versions, or missing packages.
### 4.1 Load Python TDD Skill for Python Tasks
> When modifying or creating Python scripts, the
> [python-tdd skill](./src/fiona/data/llm_wiki/utils/skills/python-tdd/SKILL.md)
> MUST be loaded
> When modifying or creating Python scripts, the Python TDD Skill MUST be loaded
```text
SKILL: python-tdd
PATH: src/fiona/data/llm_wiki/utils/skills/python-tdd/SKILL.md
PATH: llm-wiki/utils/skills/python-tdd/SKILL.md
```
- Project Rules:
@@ -208,8 +145,6 @@ PATH: src/fiona/data/llm_wiki/utils/skills/python-tdd/SKILL.md
**Required Practices from Python TDD Skill:**
- Step 3 of project agent rule 2.1 is this cycle. The spec and the
failing tests already exist before implementation starts.
- Strictly follow **Test-Driven Development (TDD)** - Red-Green-Refactor cycle
- Use **pytest** for testing
- Follow **OOP principles** and **SOLID** design patterns
@@ -238,25 +173,22 @@ The repository has the following structure with specific permissions:
```text
fiona-assistant/
├── src/fiona/ # The only installable package
│ ├── __main__.py # CLI entry point
├── src/ # Source code
│ ├── llm-wiki/ # Packaged llm-wiki template (data only)
│ │ # Structure source of truth:
│ │ # `src/llm-wiki/wiki/schema.md`
│ ├── fiona/ # Main CLI package
│ ├── librarian/ # Wiki content management
│ ├── mdoctor/ # Markdown diagnosis
│ ├── customlinter/ # Frontmatter, markdown, Python, JSON linters
│ ├── python_ruff/ # Ruff wrapper
│ ├── gittool/ # Git operations
│ ├── sanity_runner/ # Shared sanity engine
│ ├── wiki/ # Wiki lifecycle (init, update, sanity, graph)
│ ├── data/llm_wiki/ # Packaged template (data only)
│ ├── run_tests.py # Test runner module
│ └── run_sanity.py # Sanity runner module
├── tests/ # Unit tests (not installed)
├── scripts/ # Repo tooling, not package source
│ ├── run_tests.py # Launcher for fiona.run_tests
│ ├── run_sanity.py # Launcher for fiona.run_sanity
│ ├── python_ruff/ # Python code linting
│ ├── wiki/ # Wiki lifecycle modules (init, update)
│ ├── tests/ # Unit tests
│ ├── run_tests.py # Test runner
│ ├── run_sanity.py # Sanity runner (tests + linters)
│ ├── settings_dev.json # Development settings
│ ├── utils.sh # Shell utilities
│ └── install_packages.sh # Dependency installer
│
├── pyproject.toml # Project configuration
├── bootstrap # Environment setup (bash; source it)
├── bootstrap.ps1 # Environment setup (PowerShell; dot-source it)
@@ -264,7 +196,7 @@ fiona-assistant/
```
The llm-wiki directory structure is not duplicated here:
`src/fiona/data/llm_wiki/wiki/schema.md` is the final source of truth for the
`src/llm-wiki/wiki/schema.md` is the final source of truth for the
llm-wiki directory structure.
**General Rules:**
@@ -273,8 +205,8 @@ llm-wiki directory structure.
- `/raw/` directories are **READ-ONLY** - Never edit archived source materials
- `/wiki/concepts/` is **READ-WRITE** - Place synthesized content here
- `/ingest/` is for **input processing** - Parse files dropped here
- **Always update** `src/fiona/data/llm_wiki/wiki/index.md` when adding new concepts
- **Always append** to `src/fiona/data/llm_wiki/wiki/log.md` for every action
- **Always update** `src/llm-wiki/wiki/index.md` when adding new concepts
- **Always append** to `src/llm-wiki/wiki/log.md` for every action
---
@@ -282,7 +214,6 @@ llm-wiki directory structure.
### 6.1 Testing Requirements
Steps 2 and 3 of project agent rule 2.1.
**All code changes MUST be tested** following TDD principles:
1. **Write failing tests first** (Red phase)
@@ -293,7 +224,7 @@ Steps 2 and 3 of project agent rule 2.1.
```bash
# Run all tests
python scripts/run_tests.py
python src/run_tests.py
# Run with pytest directly
python -m pytest
@@ -302,7 +233,7 @@ python -m pytest
python -m pytest --cov=src
# Run specific test file
python -m pytest tests/test_md_frontmatter.py -v
python -m pytest src/tests/test_md_frontmatter.py -v
```
**Test File Structure:**
@@ -377,19 +308,41 @@ git commit -m "test: add TDD tests for name/version extraction"
### 9.1 Development Workflow
The development lifecycle is project agent rule 2.1. Do those five
steps in order: write or check the spec, write tests, implement
TDD-style, verify the spec, bump versions.
**Standard Workflow:**
Before step 2 of that lifecycle:
1. **Setup Environment** (MANDATORY)
1. Set up the environment (project agent rule 3.1).
2. Load the skill when the task needs one (project agent rule 4.1).
Python tasks load the
[python-tdd skill](./src/fiona/data/llm_wiki/utils/skills/python-tdd/SKILL.md).
Wiki tasks load the
[Fiona skill](./src/fiona/data/llm_wiki/utils/skills/fiona/SKILL.md)
unless `src/fiona/data/llm_wiki/AGENTS.md` already loaded it.
```bash
source bootstrap --env <env_name>
```
On Windows, dot-source the PowerShell script instead:
```powershell
. .\bootstrap.ps1 --env <env_name>
```
2. **Load Appropriate Skill** (if applicable)
- Python tasks: Load `python-tdd` skill
- Wiki tasks: Load `llm-wiki` skill
3. Read the Spec
- Read [SPEC](./SPEC.md) for the feature being changed
- Confirm the change matches the specified requirements and acceptance criteria
4. Follow TDD for Code Changes
- Write tests first
- Implement minimal code
- Refactor
5. Validate
- Run tests
- Run linters
6. Document
- Update docstrings
- Update README if needed
- Update [SPEC](./SPEC.md) if the change affects any specified feature or requirement
---
@@ -398,12 +351,7 @@ Before step 2 of that lifecycle:
### 10.1 MD File Frontmatter
**All markdown files in llm-wiki/ MUST have YAML frontmatter** with the
required `name` and `version` fields. Every page under `wiki/` also
requires the quality signals `confidence`, `llm`, `contested`, and
`contradictions`. How to set them is section 15 of the
[Fiona skill](./src/fiona/data/llm_wiki/utils/skills/fiona/SKILL.md).
All other fields are optional. fmlint enforces `name` and `version`
only.
required `name` and `version` fields. All other fields are optional:
```yaml
---
@@ -415,29 +363,10 @@ updated: YYYY-MM-DD # optional (bump when editing)
type: entity | concept | comparison | query | summary | toc
sources: [raw/articles/source-name.md] # optional
tags: [from taxonomy] # optional
# quality signals (required on pages under wiki/):
# Optional quality signals:
confidence: trusted | high | medium | low | zero
llm:
- model: none # required; model that last edited the page
- reasoning: high # when the model reports a level
contested: true
contradictions: [other-page-slug]
# written by wiki ingest (SPEC 7.1.5), optional:
status: raw | refined
source_meta: {author: Name} # metadata read from the source
ingest: # provenance; every key always present
origin: https://host/path # URL or file path of the source
origin_kind: url | file
final_url: null # URL after redirects
fetched: YYYY-MM-DDTHH:MM:SSZ
media_type: text/html
size_bytes: 0
sha256: <64 lowercase hex> # of the raw copy; never hand-edited
raw_reused: false
converter: html # converter name, or agent
conversion: complete | partial
warnings: []
tool: fiona-assistant <version>
---
```
@@ -448,7 +377,7 @@ ingest: # provenance; every key always present
- Use semantic versioning (MAJOR.MINOR.PATCH)
This is the single canonical frontmatter schema; `wiki/schema.md` documents
the same schema for the wiki template (see SPEC 6.2).
the same schema for the wiki template (see SPEC 4.2.1).
---
@@ -487,8 +416,7 @@ the same schema for the wiki template (see SPEC 6.2).
When multiple tasks are possible, prioritize in this order:
1. **User's explicit request** (highest priority)
2. **Development lifecycle** (project agent rule 2.1): spec, then
tests, then implementation
2. **TDD requirements** (tests before implementation)
3. **Code quality** (linting, formatting)
4. **Documentation** (docstrings, README)
5. **Refactoring** (improving existing code)
@@ -501,8 +429,6 @@ When multiple tasks are possible, prioritize in this order:
### 14.1 Things to Never Do
- Never modify `/personal/` or `/raw/` directories
- Never write production code before the spec states the requirement
and a failing test exists (project agent rule 2.1)
- Never commit code without tests
- Never commit code that fails linting
- Never ignore AGENTS.md rules without explicit user override
@@ -528,17 +454,14 @@ These rules **CAN** be overridden by:
| Task | Required Action |
| ---- | -------------- |
| Development | project agent rule 2.1, in order |
| Any Python work | Load the [python-tdd skill](./src/fiona/data/llm_wiki/utils/skills/python-tdd/SKILL.md) |
| Any Python work | Load python-tdd skill |
| Before testing | `source bootstrap` (Windows: `. .\bootstrap.ps1`) |
| Local install, no venv | `bootstrap --local --no-env` |
| Step 1 | Write or check [SPEC](./SPEC.md) |
| Step 2 and 3 | Failing tests, then the smallest change |
| Step 4 | Re-read the spec; tests and sanity pass |
| Step 5 | Bump `version` and `updated` on edited markdown |
| Changing functionality | Read [SPEC](./SPEC.md) first |
| Modifying code | Write tests first (TDD) |
| Before commit | Run tests + linters |
| Wiki changes | Load the [Fiona skill](./src/fiona/data/llm_wiki/utils/skills/fiona/SKILL.md); update index.md + log.md |
| Frontmatter | `name` and `version` on every file; on `wiki/` pages also `confidence`, `llm`, `contested`, `contradictions` |
| After behavior change | Update [SPEC](./SPEC.md) to stay in sync |
| Wiki changes | Update index.md + log.md |
| Frontmatter | Include name, version, title, dates |
| Errors | Raise exceptions, don't fail silently |
---
-229
View File
@@ -1,229 +0,0 @@
---
name: changelog
version: 1.2.3
title: Changelog
created: 2026-10-08
updated: 2026-10-24
type: summary
---
# Changelog
Revisions of [SPEC](./SPEC.md), newest first. Each entry names the
requirements it adds or changes. "Unbuilt" means the requirement is
specified and marked `NOT_IMPLEMENTED`.
## 1.22.1 - 2026-10-24
- Minor corrections to 1.22.0.
## 1.22.0 - 2026-10-24
- Adds `--staged` and `--unstaged` flags to `git diff` for staged-only and
unstaged-only diffs (*7.12.7.*, *7.12.8.*).
## 1.21.0 - 2026-10-24
- Includes untracked and deleted files in `git diff` report by default
(*7.12.6.*).
- Updates diff rendering: removes spacing between consecutive added or
deleted lines, preserves spacing between added and deleted blocks (*7.12.5.*).
## 1.20.0 - 2026-10-24
- Git commands discover and operate on the root repository from the current
directory (*4.6.*, *7.12.0.*).
## 1.19.1 - 2026-10-21
- The sectionize-markdown skill ships as
`utils/skills/sectionize-markdown/SKILL.md` (*3.4.2.*), matching the
`<name>/SKILL.md` convention of the other skills. Every reference to
the old `sectionize-markdown.md` path now points at `SKILL.md`.
## 1.18.1 - 2026-10-19
- `fiona-assistant git diff` writes a constellation-themed HTML
report under the wiki `tmp/` directory (*4.6*, *7.12.5*). The
packaged wiki `.gitignore` now lists `tmp/`.
## 1.18.0 - 2026-10-15
- Adds skill memory to the template (*3.5.*): the `memory` skill
under `utils/skills/memory/` says how a skill keeps its own
`MEMORIES.md` beside its `SKILL.md`, and ships the blank
`MEMORIES.md` to copy. Wiki-wide preferences stay in `MEMORY.md`.
## 1.17.0 - 2026-10-15
- The `.gitignore` written by `wiki init` ignores Obsidian's per-user
vault state, `.obsidian/workspace*.json` and `.obsidian/cache`, so a
wiki opened as an Obsidian vault stays clean between saves. Shared
vault settings such as `.obsidian/app.json` stay tracked (*3.2.3.*).
## 1.16.0 - 2026-10-14
- Adds the agent memory `MEMORY.md` to the template (*3.4.*): a
sectionized file of owner preferences, corrections, and verified
facts, linked from `AGENTS.md`. `AGENTS.md` says to update it on
an important memory or when the user explicitly asks (*3.4.6.*). `wiki update` adds it when missing
and never compares or replaces it after that (*3.4.4.*).
## 1.14.1 - 2026-10-08
- Moves the revision history out of the spec into this file.
## 1.14.0 - 2026-10-08
- Stages raw pages in a `refine/` directory at the wiki root, and has
the agent move each refined page into `wiki/` (*7.1.5.15.*).
Unbuilt.
## 1.13.0 - 2026-10-07
- Builds login detection with exit 4 (*7.1.5.2.8.*), `--origin`
(*7.1.5.2.9.*), the agent steps for exit 4 (*7.1.5.14.*), and the
HTML block sweep (*7.1.5.4.5.6.*).
## 1.11.0
- Specifies `wiki search` (*4.7.7.*, *7.1.6.*), a keyword search that
prints `<path>:<line>: <text>` and shortens long lines around the
match. Unbuilt.
## 1.10.0
- Replaces the first-`<article>` content root of the `html` converter
with a block sweep that scores every block as `keep`, `review`, or
`drop` (*7.1.5.4.5.6.*). Unbuilt.
## 1.9.0
- `wiki ingest` stops with exit 4 on a URL that needs a login
(*7.1.5.2.8.*). Unbuilt.
- Adds `--origin` for a file saved by hand (*7.1.5.2.9.*). Unbuilt.
- Tells the agent how to get the source instead (*7.1.5.14.*).
Unbuilt.
## 1.8.0
Adds the findings of the 2026-10-02 manual test. All unbuilt:
- One-line CLI errors with no traceback (*1.2.7.*, *7.1.1.4.*).
- `--version` (*2.3.*).
- A quiet default stderr (*7.17.3.*).
- An ingest hint for a drop-zone file that already moved
(*7.1.5.2.7.*).
- A dry run that does not log writes (*7.1.5.8.1.*).
- A lint-clean ingest log line (*7.1.5.6.1.*).
- A follow-up hint for style violations left on a page
(*7.1.5.5.6.*).
- Single-file `mdoctor` (*7.4.4.*).
- Compact housekeeping sanity details (*7.2.5.1.*).
- A quiet graph overwrite (*7.1.4.5.9.1.*).
- A usage section in the README (*7.18.4.*).
## 1.7.0
- `librarian move` rewrites links to the moved paths across the whole
wiki (*7.3.3.6.*).
## 1.6.0
- Specifies `librarian move` (*7.3.3.*). Unbuilt.
## 1.5.2
- Gives `table_of_contents.md` frontmatter `name` and `version`
(*7.3.1.1.1.*), so `fmlint` accepts it.
## 1.5.0
- Omits markdown files matched by `fiona.wiki.graph.ignore_files`
from the graph (*7.1.4.20.*). The shipped list is
`*/table_of_contents.md`, because a table of contents links to
every page.
## 1.4.0
- Names the core-file record `corefiles.json` (*7.1.2.14.*).
- Skips paths in `fiona.wiki.update.ignore_files` (*7.1.2.15.*). The
shipped list names `corefiles.json`. Only `wiki update` writes that
file.
## 1.3.0
- Specifies directory nodes (*7.1.4.18.*) and `wiki graph --serve`
(*7.1.4.19.*). Both unbuilt.
## 1.2.0
- Folds text line endings before the SHA-256 compare (*7.1.2.13.*).
- Writes `corefiles.json`, so a later update does not list a core
file whose version and hashes are unchanged (*7.1.2.14.*).
## 1.1.0
- Treats a differing `fiona-settings.json` as a conflict and writes
`fiona-settings-update.json` (*7.1.2.3.*).
## 1.0.0
- Groups each module's functions under that module.
- Moves Non-Goals, Priority Order, and Open Questions to Appendices
A, B, and C.
## 0.99.0
- Builds `wiki ingest` (*7.1.5.*).
## 0.98.0
- Specifies that `wiki graph` omits a file matched by `.gitignore`
(*7.1.4.17.*). Unbuilt.
## 0.97.0
- Rejects a module-level assignment that follows the first function
or class (*6.4.3.*).
## 0.96.0
- Compares a file without a version by SHA-256 and shows both short
hashes in the update report (*7.1.2.12.*).
## 0.94.0
- `wiki update` yields to the owner's markdown and writes
`<stem>-update.md` instead (*7.1.2.11.*).
- Adds `fiona-assistant housekeeping` (*7.2.*).
## 0.93.0
- Adds SHA-256 reuse across `raw/`, the complete `ingest`
frontmatter map, and the agent metadata check.
## 0.92.0
- Specifies `wiki ingest` (*7.1.5.*). Unbuilt.
## 0.90.0
- Specifies `node_importance_min_point` (*7.1.4.16.*). Unbuilt.
## 0.41.0
- Installs one `fiona` package and keeps tests and repo scripts
outside it.
## 0.40.4
- Adds the PYC001 exception (*6.4.2.*).
## 0.40.1
- Numbers nested requirements such as `7.14.1.1`.
## 0.40.0
- Reorganizes the sheet into contracts.
+293 -362
View File
@@ -1,436 +1,367 @@
# Fiona-Assistant
> Your LLM agent writes the wiki. Fiona keeps it clean.
LLM Wiki Tool Helper for LLM Agents/Hives
Fiona-Assistant is a command-line toolkit for running an **llm-wiki**: a
folder of interlinked markdown that an AI agent builds and maintains for
you. You bring sources (web pages, PDFs, notes) and questions. The agent
compiles them into wiki pages. Fiona does every deterministic part of the
job: it scaffolds the wiki, ingests and archives sources, lints markdown,
frontmatter, Python, and JSON, checks links, draws a link graph, versions
everything in git, and tells the agent exactly what to fix next.
The result: the agent spends its tokens on reading and writing, not on
listing files, chasing broken links, or reformatting tables.
```bash
python -m pip install fiona-assistant
fiona-assistant wiki init my-wiki
cd my-wiki
fiona-assistant wiki ingest https://example.com/article.html
fiona-assistant housekeeping
```
Fiona-Assistant is a command-line toolkit designed to manage, organize, and
maintain LLM wiki content with minimal AI token dependence. It combines
script-based automation with structured workflows to help LLM Agents and
Hives efficiently manage documentation, markdown files, and code quality.
---
## Table of Contents
## Overview
- [Why Fiona](#why-fiona)
- [Features](#features)
- [Quick Start](#quick-start)
- [How It Works](#how-it-works)
- [Command Reference](#command-reference)
- [Output Built for Agents](#output-built-for-agents)
- [Roadmap](#roadmap)
- [Development](#development)
- [Documentation](#documentation)
- [License](#license)
### Purpose
---
Fiona-Assistant serves as a **standalone MD/scripts/docs organizer** and
**LLM wiki management helper** that:
## Why Fiona
- **Reduces AI Token Usage**: Uses deterministic scripts and tools to handle
wiki management tasks that don't require LLM intelligence
- **Maintains Wiki Structure**: Enforces consistent organization of markdown
files, concepts, and raw source materials
- **Validates Content Quality**: Provides linting and diagnosis tools for markdown and Python code
- **Enables Script-Based Workflows**: Automates repetitive wiki maintenance tasks through CLI commands
An LLM agent can maintain a knowledge base, but left alone it wastes
context on chores and slowly breaks things: dead links, missing
frontmatter, wrapped table rows, orphan pages, overwritten sources.
### Core Philosophy
Fiona fixes that split of work:
> Use scripts for deterministic tasks, reserve AI tokens for creative and analytical work
| The agent does | Fiona does |
| --- | --- |
| Reads sources and writes synthesized pages | Fetches, archives, and converts sources to markdown |
| Decides what links to what | Finds dead links, dead wikilinks, and orphan pages |
| Refines raw pages | Lints markdown, frontmatter, Python, and JSON |
| Answers your questions from the wiki | Commits, snapshots, and restores wiki history |
| Merges template updates | Detects template updates and never overwrites your files |
> Use scripts for deterministic tasks. Reserve AI tokens for creative and
> analytical work.
The toolkit is designed to be **AI-friendly** - outputs are formatted in
consistent, parseable structures that LLMs can easily read and process.
---
## Features
### Wiki in one command
### Tool Suite
`fiona-assistant wiki init` creates a ready-to-use llm-wiki: directory
layout, settings, agent rules (`AGENTS.md`), page schema, index, log, and
bundled agent skills. It also creates a git repository and makes the
first commit, with `personal/` kept out of version control.
| Tool | Purpose | Key Capabilities |
| ---- | ------- | ---------------- |
| **Librarian** | Wiki content management | List files, scan directories, organize MD content |
| **MDoctor** | Markdown diagnosis | Lint markdown files using PyMarkdown, detect formatting issues |
| **Python-Ruff** | Code quality | Lint Python files using Ruff, enforce code standards |
| **Generate-TOC** | Table of contents | Auto-generate nested TOC from markdown headers |
### Source ingestion
### Wiki Structure Support
`fiona-assistant wiki ingest <url-or-file>` turns a source into a wiki page
in one step:
Fiona-Assistant understands and maintains the [LLM Wiki directory structure](./llm-wiki/AGENTS.md):
- Downloads the URL or reads the local file
- Archives an untouched copy under `raw/` (duplicates are detected by
SHA-256, so the same bytes are never stored twice)
- Converts it to markdown (HTML, PDF, plain text, and markdown ship today)
- Writes a page under `wiki/` with full provenance in its frontmatter and
`status: raw`, so the agent knows it still needs refining
- Appends the action to `wiki/log.md`
`wiki ingest --pending` lists every page still waiting to be refined.
### One-call maintenance
`fiona-assistant housekeeping` runs the whole maintenance routine and
prints a to-do list for the agent:
- **check-update**: is the packaged template newer than this wiki?
- **merge**: which `*-update.md` files are waiting to be merged?
- **ingest**: which pages are still `status: raw`?
- **sanity**: which lint checks fail, and where?
- **graph**: regenerates `wiki_graph.html` and lists orphan pages
It only reads, except for writing the graph. Every finding comes with a
concrete "next step" the agent can act on. Add `--json` for machine use.
### Quality checks
`fiona-assistant wiki sanity` runs five checks in parallel and gives one
pass/fail verdict:
| Check | What it catches |
| --- | --- |
| `markdownlint` | Markdown style, via PyMarkdown |
| `frontmatter_lint` | Missing or malformed YAML frontmatter (`name`, `version`, and more) |
| `python-ruff` | Python issues in wiki scripts, via Ruff |
| `mdlinter` | Dead links, dead `[[wikilinks]]`, and table rows split across lines |
| `jsonlint` | JSON syntax, indentation, trailing commas, duplicate keys |
Each linter is also available on its own, and `mdoctor fix` and
`json-lint fix` repair what they can automatically.
### Interactive link graph
`fiona-assistant wiki graph` writes a single self-contained HTML file
that shows how your pages link to each other. Nodes are scored by how
many different pages point at them, so hub pages stand out. It
understands markdown links and `[[wikilinks]]`, shows external URLs, and
flags broken links and orphans. `--format json` exports the same graph
for other tools.
### Safe template updates
As Fiona improves, so does its wiki template. `wiki check-update` shows
what changed, and `wiki update` brings your wiki up to date without ever
overwriting a file you own. When both sides changed, Fiona writes
`<name>-update.md` next to your file and `housekeeping` tells the agent
to merge it.
### Git history built in
`fiona-assistant git` wraps the operations an agent needs, with
timestamped commit messages:
- `save` commits all current edits
- `snapshot <name>` commits and tags a restore point
- `restore <name>` returns to a snapshot without losing history
- `undo`, `clean`, and `status` for everyday recovery
### Guardrails
- `personal/` and `raw/` are read-only for every automated tool. Ingest
may add new files to `raw/`; it never changes an existing one.
- Commands that need a decision prompt on a terminal and accept
`--yes`, `--force`, or `--non-interactive` for unattended runs.
- Respects `.gitignore` when scanning.
### Table of contents generation
`fiona-assistant librarian toc` builds `table_of_contents.md` for a folder
tree. `librarian toc file <path>` keeps an in-file TOC under the H1 up to
date, with optional section numbering (`--sectionized`).
```text
llm-wiki/
├── ingest/ # Input zone: files to be processed
├── raw/ # Archived source materials (read-only)
│ ├── articles/
│ ├── papers/
│ ├── transcripts/
│ └── assets/
├── wiki/ # Processed content
│ └── concepts/ # Synthesized concept notes
├── personal/ # User notes (read-only for agents)
└── utils/ # Utility scripts
```
---
## Quick Start
## Installation
### 1. Install
Requires Python 3.11 or newer.
### Quick Install (Production)
```bash
python -m pip install fiona-assistant
```
### 2. Create a wiki
### Development Setup
- **Clone the repository:**
```bash
git clone <repository-url>
cd fiona-assistant
```
- **Create and activate the virtual environment:**
Linux / macOS:
```bash
source bootstrap --env fiona_env -c
```
Windows (PowerShell):
```powershell
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
. .\bootstrap.ps1 --env fiona_env -c
```
> Make sure `fiona_env` is activated before proceeding. On Windows the
> environment lives in `.venvs\fiona_env`.
---
## Usage
### Main CLI
Fiona-Assistant provides a unified CLI with multiple subcommands:
```bash
fiona-assistant wiki init my-wiki
cd my-wiki
# Show all available commands
fiona-assistant --help
# Run via Python module
python -m fiona --help
```
### 3. Point your agent at it
Open the folder in your AI coding agent (Cursor, Claude Code, or any
agent that reads `AGENTS.md`). The bundled `AGENTS.md` loads the Fiona
skill, which teaches the agent the ingest, refine, and housekeeping
workflow and when to call each command.
### 4. Feed it sources
#### Available Commands
```bash
fiona-assistant wiki ingest https://example.com/post.html
fiona-assistant wiki ingest ./notes/meeting.pdf
# Run tests
fiona-assistant test
# Manage wiki content with Librarian
fiona-assistant librarian [directory]
# Generate an in-file table of contents below the H1
fiona-assistant librarian toc file <path_to_file>
# Force section numbering on for the run
fiona-assistant librarian toc file <path_to_file> --sectionized
# Generate in-file TOCs for every md file in the current directory / wiki root
fiona-assistant librarian toc file '*'
# Diagnose markdown files with MDoctor
fiona-assistant mdoctor [chklint] [directory]
# Lint Python code with Ruff
fiona-assistant python-ruff [chklint] [directory]
```
Then ask the agent to refine the new pages.
### Direct Tool Access
### 5. Keep it healthy
Each tool can also be called directly:
```bash
fiona-assistant housekeeping
fiona-assistant git save --description "ingest meeting notes"
# Librarian - List all files in a directory
librarian [directory]
# MDoctor - Check markdown lint violations
mdoctor chklint [directory]
# MDoctor - List all files
# Python-Ruff - Check Python code quality
python-ruff chklint [directory]
# Generate table of contents for markdown files
generate-toc [directory] [options]
```
---
## How It Works
## Workflow Examples
A wiki created by Fiona has this layout:
### Wiki Ingestion Workflow
```bash
# 1. Drop raw files into ingest directory
# (Manual or automated process)
# 2. List files ready for processing
librarian llm-wiki/ingest
# 3. Validate markdown quality before processing
mdoctor chklint llm-wiki/ingest
# 4. Move processed files to raw archive
# (Agent/orchestration task)
```
### Content Maintenance Workflow
```bash
# Check all markdown files in wiki for lint violations
mdoctor chklint llm-wiki/wiki
# Generate table of contents for concept notes
generate-toc llm-wiki/wiki/concepts --recursive-toc
# Validate Python code in the project
python-ruff chklint src/
# List all markdown files for inventory
librarian llm-wiki/
```
### Standalone Document Organization
```bash
# Organize any directory of markdown files
librarian /path/to/docs/
# Validate markdown formatting
mdoctor chklint /path/to/docs/
# Create nested table of contents
generate-toc /path/to/docs/ --recursive-toc
```
---
## Output Format
All tools produce **AI-friendly output** with consistent formatting:
### MDoctor Output Format
```text
my-wiki/
├── AGENTS.md # Entry point for the agent; loads the Fiona skill
├── fiona-settings.json # Marks the wiki root; tool settings
├── ingest/ # Drop zone for files waiting to be parsed
├── raw/ # Immutable archive of original sources
│ ├── articles/
│ ├── papers/
│ ├── transcripts/
│ ├── assets/
│ └── other/
├── wiki/ # Pages the agent writes and maintains
│ ├── schema.md # Page types, tags, conventions
│ ├── index.md # Content catalog
│ ├── log.md # Append-only action log
│ ├── concepts/
│ ├── entities/
│ ├── comparisons/
│ ├── queries/
│ └── sources/
├── personal/ # Your own notes (read-only for agents, untracked)
└── utils/ # Agent skills, personas, and helper scripts
```
The cycle is:
- **Ingest**: `wiki ingest` archives a source in `raw/` and writes a raw
page in `wiki/`.
- **Refine**: the agent rewrites the page, links it to related pages, and
sets `status: refined`.
- **Maintain**: `housekeeping` reports what needs attention; the agent
fixes it.
- **Save**: `git save` records the change.
The full page schema lives in the template's
[schema](./src/fiona/data/llm_wiki/wiki/schema.md).
---
## Command Reference
Run `fiona-assistant --help` or `fiona-assistant <command> --help` for all
flags. Commands that take an optional directory default to the current
directory.
### Wiki lifecycle
| Command | What it does |
| --- | --- |
| `wiki init [dir]` | Create a new llm-wiki with git history |
| `wiki ingest <source>` | Fetch, archive, and convert a URL or file into a raw page |
| `wiki ingest --pending` | List pages still marked `status: raw` |
| `wiki sanity [dir]` | Run every quality check; exit 1 on failure |
| `wiki graph [dir]` | Write `wiki_graph.html` (or JSON with `--format json`) |
| `wiki check-update [dir]` | Show what a template update would change |
| `wiki update [dir]` | Apply a template update (`--dry-run`, `--yes`) |
| `housekeeping [dir]` | Run the full maintenance routine and list next steps |
### Linters
| Command | What it does |
| --- | --- |
| `mdoctor chklint [dir]` | Markdown lint via PyMarkdown |
| `mdoctor fix [dir]` | Auto-fix markdown lint violations |
| `fmlint chklint [dir]` | Frontmatter lint |
| `python-ruff chklint [dir]` | Python lint via Ruff |
| `json-lint chklint [dir]` | JSON lint |
| `json-lint fix [dir]` | Auto-fix JSON indentation and trailing commas |
### Files and TOC
| Command | What it does |
| --- | --- |
| `librarian list [dir]` | List files in a directory |
| `librarian toc [dir]` | Write `table_of_contents.md` (`--recursive-toc`, `--exclude REGEX`) |
| `librarian toc file <path>` | Write an in-file TOC below the H1 (`--sectionized`) |
| `librarian toc file '*'` | Update in-file TOCs for every markdown file in the wiki |
### Git
| Command | What it does |
| --- | --- |
| `git save --description TEXT` | Commit all edits |
| `git snapshot <name> --description TEXT` | Commit and tag a restore point |
| `git restore <name>` | Restore a snapshot as a new commit |
| `git undo` | Drop the last commit |
| `git clean` | Discard uncommitted edits |
| `git status` | Show whether the tree is clean |
Every command above is run as `fiona-assistant <command>`. The
`librarian` and `mdoctor` tools are also installed as standalone
commands.
---
## Output Built for Agents
Every report uses the same shape: a header, the target, rule
definitions, findings per file with line and column, and a summary. An
agent can parse it without guessing, and a human can skim it.
```text
Markdown Lint Violations
TARGET: wiki
PyMarkdownLint Violations
TARGET: /path/to/directory
- DEFINITIONS:
- MDL001: Dead Link, Relative markdown link does not resolve to an existing file
- MDL002: Dead Wiki Link, Wiki-internal link does not resolve to any md page in the wiki
- md001: Heading levels should only increment by one, Heading levels should only increment by one level at a time
- md013: Line length, Line length
- FILE: wiki/concepts/retrieval.md
- MDL001 line 12: ../sources/old-paper.md does not exist
- MDL002 line 30: [[embeddings]] does not resolve to any page
- FILE: file1.md
- 10:5 md013
- 20:3 md001
- SUMMARY:
- Total files with violations: 1
- Total violations: 2
```
`wiki sanity`, `housekeeping`, `wiki ingest`, `wiki check-update`, and
`wiki update` also accept `--json` for structured output.
### Python-Ruff Output Format
```text
Ruff Violations
TARGET: /path/to/directory
Definitions:
- F401: `*` import used, but not imported
- E501: Line too long
- FILE: script.py
- 15:10 F401
- 25:80 E501
- SUMMARY:
- Total files with violations: 1
- Total violations: 2
```
---
## Roadmap
Planned work, tracked with requirements and acceptance criteria in
[SPEC](./SPEC.md) and prioritized in [todo](./todo.md):
| Feature | What it will do |
| --- | --- |
| Wiki root discovery | Run any command from a subfolder; Fiona finds the nearest `fiona-settings.json` |
| Backup | One command writes a compressed, restorable archive of the wiki outside the tree |
| `setup` command | Interactive review and editing of `fiona-settings.json`, with validation |
| Smarter graph | Show only important nodes until you expand a parent; hide gitignored pages |
| Code blocks in markdown | Lint Python and shell snippets inside fenced code blocks |
| Shellcheck | Lint shell scripts as part of wiki sanity |
| HTML and JavaScript lint | Validate HTML and JS files in the wiki |
| `python-ruff fix` | Auto-fix Python lint findings |
| More converters | Ingest more source formats beyond HTML, PDF, text, and markdown |
| Version tags for scripts | Track versions of Python and shell files like markdown pages |
---
## Development
### Set up
Clone the repository and load the environment with the bootstrap script.
Linux and macOS:
```bash
git clone <repository-url>
cd fiona-assistant
source bootstrap --env fiona_env -c
```
Windows (PowerShell):
```powershell
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
. .\bootstrap.ps1 --env fiona_env -c
```
The default install is editable (`pip install -e ".[dev]"`). On Windows
the environment lives in `.venvs\fiona_env`.
### Other install modes
| Goal | Bash | PowerShell |
| --- | --- | --- |
| Install this checkout as a normal package | `source bootstrap --local` | `. .\bootstrap.ps1 --local` |
| Same as `pip install fiona-assistant`, no venv | `source bootstrap --local --no-env` | `powershell -ExecutionPolicy Bypass -File .\bootstrap.ps1 --local --no-env` |
`--local --no-env` runs `python -m pip install --user .`. If
`fiona-assistant` is not found afterwards, add the user scripts
directory printed by bootstrap to `PATH`.
### Test and check
```bash
# Unit tests with coverage
python scripts/run_tests.py
# Full sanity: tests, coverage gate, markdown, Ruff, frontmatter, JSON
python scripts/run_sanity.py --non-interactive
```
Contributions follow the spec-first, test-first lifecycle in
[AGENTS](./AGENTS.md).
### Project structure
## Project Structure
```text
fiona-assistant/
├── src/fiona/ # The installable package
│ ├── __main__.py # fiona-assistant CLI
│ ├── librarian/ # File listing and TOC generation
│ ├── mdoctor/ # Markdown lint and fix
│ ├── customlinter/ # Frontmatter, markdown link, Python, JSON linters
│ ├── python_ruff/ # Ruff wrapper
│ ├── gittool/ # Git operations
│ ├── sanity_runner/ # Shared sanity engine
│ ├── wiki/ # init, update, sanity, graph, ingest, housekeeping
│ └── data/llm_wiki/ # Packaged wiki template
├── tests/ # Unit tests (not installed)
├── scripts/ # Test and sanity runners, dev settings
├── bootstrap # Environment setup (bash)
├── bootstrap.ps1 # Environment setup (PowerShell)
├── SPEC.md # Requirements and acceptance criteria
└── pyproject.toml
├── llm-wiki/ # Wiki content and structure definition
│ ├── AGENTS.md # Agent rules and directory permissions
│ ├── ingest/ # Input zone for new content
│ ├── raw/ # Archived source materials
│ ├── wiki/ # Processed concept notes
│ └── utils/ # Utility scripts
│
├── src/
│ ├── fiona/ # Main CLI entry point
│ │ └── __main__.py # Unified command dispatcher
│ │
│ ├── librarian/ # Wiki content management
│ │ ├── __init__.py
│ │ ├── librarian.py # File scanning and listing
│ │ └── toc_generator.py # Table of contents generator
│ │
│ ├── mdoctor/ # Markdown diagnosis
│ │ ├── __init__.py
│ │ └── mdoctor.py # PyMarkdown lint wrapper
│ │
│ ├── python_ruff/ # Python code linting
│ │ ├── __init__.py
│ │ └── python_ruff.py # Ruff lint wrapper
│ │
│ ├── tests/ # Unit tests
│ │ ├── test_fiona_main.py
│ │ ├── test_librarian.py
│ │ ├── test_mdoctor.py
│ │ └── test_python_ruff.py
│ │
│ ├── run_tests.py # Test runner
│ └── README.md # Development documentation
│
├── pyproject.toml # Project configuration
├── bootstrap # Environment setup script (bash)
├── bootstrap.ps1 # Environment setup script (PowerShell)
├── LICENSE # GPL-3.0-only
└── README.md # This file
```
---
## Documentation
## Testing
- [Project Specification](./SPEC.md): every command, setting, and rule,
with acceptance criteria
- [LLM Wiki Agent Rules](./src/fiona/data/llm_wiki/AGENTS.md): what a
deployed wiki tells its agent
- [Fiona skill](./src/fiona/data/llm_wiki/utils/skills/fiona/SKILL.md):
the workflow the agent follows inside a wiki
- [Development Agent Rules](./AGENTS.md): how to contribute to this
repository
- [Roadmap](./todo.md): planned and in-progress work
### Run Tests
```bash
# Run all tests
python src/run_tests.py
# Or use pytest directly
python -m pytest
# Run with coverage
python -m pytest --cov=src
```
---
## License
GPL-3.0-only
---
## Key Concepts
### For LLM Agents
- **Deterministic Operations**: Use Fiona-Assistant for file listing, linting, and organization
- **Token Efficiency**: Reserve LLM context for synthesis, analysis, and creative tasks
- **Structured Output**: All tool outputs are formatted for easy parsing by LLMs
### For Human Users
- **CLI-First**: All functionality available through command-line interface
- **Composable Tools**: Combine tools in scripts for automated workflows
- **Clear Structure**: Maintains consistent wiki organization standards
---
## Related Documentation
- [LLM Wiki Agent Rules](./llm-wiki/AGENTS.md) - Directory permissions and workflow rules
- [Development Setup](./src/README.md) - Detailed development information
- [Project Specification](./SPEC.md) - Feature spec sheet with requirements and acceptance criteria
- [Project TODO](./todo.md) - Roadmap and planned features
---
## Quick Reference Card
| Task | Command |
| ---- | ------- |
| Show help | `fiona-assistant --help` |
| List files | `fiona-assistant librarian /path/to/dir` |
| Check MD lint | `fiona-assistant mdoctor chklint /path/to/dir` |
| Check Python lint | `fiona-assistant python-ruff chklint /path/to/dir` |
| Generate TOC | `generate-toc /path/to/dir --recursive-toc` |
| Generate file TOC | `fiona-assistant librarian toc file /path/to/file.md --sectionized` |
| Run tests | `python src/run_tests.py` |
| Install | `pip install fiona-assistant` |
| Dev install | `pip install -e .` |
---
> **Note**: All paths are relative to the current working directory unless specified as absolute paths.
+2029 -4478
View File
File diff suppressed because it is too large Load Diff
+78 -182
View File
@@ -1,5 +1,4 @@
#!/bin/bash
# Modified by AI/LLM: 2026-09-30 - --local and --no-env install options.
#
# Project bootstrap script - Sets up Python development environment
#
@@ -7,47 +6,21 @@
# source bootstrap
#
# Windows (PowerShell): dot-source bootstrap.ps1 instead
# . .\bootstrap.ps1 [--env name] [--python 3.x.x] [-c|--clean] [--local] [--no-env]
# . .\bootstrap.ps1 [--env name] [--python 3.x.x] [-c|--clean]
#
# What it does:
# 1. Installs pyenv and required system dependencies if needed
# 2. Initializes pyenv and pyenv-virtualenv in the current shell
# 3. Creates a Python virtual environment named 'test_env' (or custom name)
# 4. Activates the virtual environment
# 5. Installs this repository (editable with the dev extra, unless --local)
# 5. Installs all project requirements from requirements.txt files
#
# With --clean/-c option:
# - Deletes existing environment before creating a new one
#
# With --local:
# - Installs this repository as a normal package (pip install .), not editable
#
# With --no-env:
# - Skips pyenv and the virtual environment
# - Installs with pip --user so the commands run without activating an environment
#
# Usage:
# source bootstrap [--pyenv] [-c|--clean] [--env custom_name] [--python 3.x.x] [--local] [--no-env]
# source bootstrap [--pyenv] [-c|--clean] [--env custom_name] [--python 3.x.x]
# ex. source bootstrap --pyenv -c --env my_env --python 3.10.6
# ex. source bootstrap --local --no-env
# =============================================
# SOURCED GUARD
# =============================================
# Activation only persists when this file is sourced; executing it
# loses the environment when the child shell exits. `return` works
# at the top level only inside a sourced script.
if [ -z "${BASH_VERSION:-}" ]; then
echo "ERROR: bootstrap is a bash script. Run it with bash:" >&2
echo "Run: source bootstrap [--pyenv] [-c|--clean] [--env name] [--python 3.x.x] [--local] [--no-env]" >&2
exit 1
fi
if ! (return 0 2>/dev/null); then
echo "ERROR: bootstrap must be sourced, not executed." >&2
echo "The environment would be lost when the child shell exits." >&2
echo "Run: source bootstrap [--pyenv] [-c|--clean] [--env name] [--python 3.x.x] [--local] [--no-env]" >&2
exit 1
fi
# =============================================
# VARIABLES
@@ -56,13 +29,10 @@ PYTHON_VERSION="3.13.8" # Default Python version
export PYENV_NAME="test_env" # Default environment name
FORCE_PYENV_INSTALL=false # Flag to force pyenv reinstallation
CLEAN_ENVIRONMENT=false # Flag to delete and recreate environment
LOCAL_INSTALL=false # Install this repo as a normal package, not editable
NO_ENV=false # Skip the virtual environment and pip install --user
PYTHON_BIN="python"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" # scripts root directory
# Source utilities (needed for echo_error, echo_status, etc.)
source "$SCRIPT_DIR/scripts/utils.sh"
source "$SCRIPT_DIR/src/utils.sh"
# Parse command line arguments
while [[ $# -gt 0 ]]; do
@@ -75,14 +45,6 @@ while [[ $# -gt 0 ]]; do
CLEAN_ENVIRONMENT=true
shift
;;
--local)
LOCAL_INSTALL=true
shift
;;
--no-env)
NO_ENV=true
shift
;;
--env)
if [ -n "$2" ]; then
PYENV_NAME="$2"
@@ -103,7 +65,7 @@ while [[ $# -gt 0 ]]; do
;;
*)
echo_error "Unknown option: $1"
echo "Usage: source bootstrap [--pyenv] [-c|--clean] [--env custom_name] [--python 3.x.x] [--local] [--no-env]"
echo "Usage: source bootstrap [--pyenv] [-c|--clean] [--env custom_name] [--python 3.x.x]"
return 1
;;
esac
@@ -324,7 +286,7 @@ activate_environment() {
# Get the virtualenv path
local venv_path
venv_path=$(pyenv prefix "$PYENV_NAME" 2>/dev/null)
if [ -n "$venv_path" ] && [ -d "$venv_path" ]; then
# Manually activate by sourcing the activate script
local activate_script="$venv_path/bin/activate"
@@ -337,7 +299,7 @@ activate_environment() {
return 0
fi
fi
# Fallback: try pyenv activate
if pyenv activate "$PYENV_NAME" 2>/dev/null; then
echo_status "Successfully activated $PYENV_NAME with pyenv"
@@ -372,82 +334,52 @@ install_requirements() {
echo_status "Installing requirements..."
# Check if pyproject.toml exists (could be in root or src/)
local project_dir=""
if [ -f "$SCRIPT_DIR/pyproject.toml" ]; then
project_dir="$SCRIPT_DIR"
elif [ -f "$SCRIPT_DIR/src/pyproject.toml" ]; then
project_dir="$SCRIPT_DIR/src"
fi
if [ -n "$project_dir" ]; then
local helper="$SCRIPT_DIR/scripts/bootstrap_install.py"
if [ ! -f "$helper" ]; then
echo_error "Missing $helper"
return 1
fi
local flags=(--break-system-packages)
if [ "$LOCAL_INSTALL" = true ]; then
echo_status "Installing this repository as a normal package..."
flags+=(--local)
else
echo_status "Installing package and dependencies from pyproject.toml..."
fi
if [ "$NO_ENV" = true ]; then
flags+=(--no-env)
fi
local pip_words=()
local word
while IFS= read -r word; do
if [ -n "$word" ]; then
pip_words+=("$word")
fi
done < <("$PYTHON_BIN" "$helper" "${flags[@]}")
if [ "${#pip_words[@]}" -eq 0 ]; then
echo_error "Failed to resolve pip install arguments"
return 1
fi
(cd "$project_dir" && "$PYTHON_BIN" -m pip "${pip_words[@]}") || {
local pyproject_root="$SCRIPT_DIR/pyproject.toml"
local pyproject_src="$SCRIPT_DIR/src/pyproject.toml"
if [ -f "$pyproject_root" ]; then
# Install the package in development mode from root
echo_status "Installing package and dependencies from pyproject.toml..."
(cd "$SCRIPT_DIR" && pip install -e ".[dev]" --break-system-packages) || {
echo_error "Failed to install package and dependencies from pyproject.toml"
return 1
}
return 0
fi
echo_warning "No pyproject.toml found"
# Fallback: look for any requirements.txt files
local req_files=()
while IFS= read -r -d '' req_file; do
req_files+=("$req_file")
done < <(find "$SCRIPT_DIR" -name "requirements.txt" -print0)
if [ ${#req_files[@]} -eq 0 ]; then
echo_status "No requirements files found"
return 0
fi
local user_args=()
if [ "$NO_ENV" = true ]; then
user_args+=(--user)
fi
for req_file in "${req_files[@]}"; do
echo_status "Installing requirements from: $req_file"
"$PYTHON_BIN" -m pip install "${user_args[@]}" -r "$req_file" --break-system-packages || {
echo_error "Failed to install requirements from $req_file"
elif [ -f "$pyproject_src" ]; then
# Install the package in development mode from src/
echo_status "Installing package and dependencies from pyproject.toml..."
(cd "$SCRIPT_DIR/src" && pip install -e ".[dev]" --break-system-packages) || {
echo_error "Failed to install package and dependencies from pyproject.toml"
return 1
}
done
else
echo_warning "No pyproject.toml found"
# Fallback: look for any requirements.txt files
local req_files=()
while IFS= read -r -d '' req_file; do
req_files+=("$req_file")
done < <(find "$SCRIPT_DIR" -name "requirements.txt" -print0)
if [ ${#req_files[@]} -eq 0 ]; then
echo_status "No requirements files found"
return 0
fi
for req_file in "${req_files[@]}"; do
echo_status "Installing requirements from: $req_file"
pip install -r "$req_file" --break-system-packages || {
echo_error "Failed to install requirements from $req_file"
return 1
}
done
fi
return 0
}
upgrade_pip() {
echo_status "Upgrading pip..."
local upgrade_args=(install --upgrade pip --break-system-packages)
if [ "$NO_ENV" = true ]; then
upgrade_args=(install --user --upgrade pip --break-system-packages)
fi
"$PYTHON_BIN" -m pip "${upgrade_args[@]}" || {
python -m pip install --upgrade pip --break-system-packages || {
echo_warning "Failed to upgrade pip"
echo_warning "Continuing with requirements installation..."
}
@@ -455,18 +387,18 @@ upgrade_pip() {
install_dev_packages() {
echo_status "Installing local Python packages in development mode..."
local install_script="$SCRIPT_DIR/scripts/install_packages.sh"
local install_script="$SCRIPT_DIR/src/install_packages.sh"
if [ -f "$install_script" ]; then
# Use source to run in current shell context, preserving environment variables
(cd "$SCRIPT_DIR" && . "$install_script") || {
(cd "$SCRIPT_DIR/src" && . "$install_script") || {
echo_error "Dev packages not installed!"
return 1
}
else
echo_status "Install script not found at $install_script, attempting direct install..."
# Fallback: install in editable mode from the repository root
(cd "$SCRIPT_DIR" && pip install -e . --break-system-packages) || {
# Fallback: install in editable mode from src directory
(cd "$SCRIPT_DIR/src" && pip install -e . --break-system-packages) || {
echo_error "Failed to install package in development mode"
return 1
}
@@ -482,16 +414,16 @@ sanitize_pytest_env() {
# Also remove -S or --no-summary (older versions)
local sanitized
sanitized=$(echo "$PYTEST_ADDOPTS" | sed -e 's/--no-summary[[:space:]]*//g' -e 's/--no-summary//g' -e 's/-S[[:space:]]*//g' -e 's/-S//g')
# Remove leading/trailing whitespace and any double spaces
sanitized=$(echo "$sanitized" | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//' -e 's/[[:space:]]\+/ /g')
if [ "$sanitized" != "$PYTEST_ADDOPTS" ]; then
echo_warning "Removing --no-summary from PYTEST_ADDOPTS to ensure test output is parsed correctly"
export PYTEST_ADDOPTS="$sanitized"
fi
fi
# Also unset FORCE_COLOR and PY_COLORS as they can interfere with output parsing
if [ -n "$FORCE_COLOR" ]; then
echo_warning "Unsetting FORCE_COLOR to ensure pytest output is parsed correctly"
@@ -501,67 +433,42 @@ sanitize_pytest_env() {
echo_warning "Unsetting PY_COLORS to ensure pytest output is parsed correctly"
unset PY_COLORS
fi
return 0
}
# =============================================
# MAIN EXECUTION
# =============================================
if [ "$NO_ENV" = true ]; then
echo_status "Starting project bootstrap (no virtual environment)"
else
echo_status "Starting project bootstrap with environment: $PYENV_NAME"
echo_status "Starting project bootstrap with environment: $PYENV_NAME"
if [ "$FORCE_PYENV_INSTALL" = true ]; then
echo_status "Forcing pyenv reinstallation"
fi
if [ "$LOCAL_INSTALL" = true ]; then
echo_status "Local install: this repository, not editable"
fi
if [ "$NO_ENV" = true ]; then
echo_status "No virtual environment: installing with --user"
# 1. Check if Python is installed
if ! check_python_installed; then
return 1
fi
if [ "$NO_ENV" = true ]; then
if ! command -v python >/dev/null 2>&1 && ! command -v python3 >/dev/null 2>&1; then
echo_error "Python is not installed. Please install Python $PYTHON_VERSION before running this script."
echo_error "Or download from: https://www.python.org/downloads/"
return 1
fi
if command -v python >/dev/null 2>&1; then
PYTHON_BIN="python"
else
PYTHON_BIN="python3"
fi
else
if [ "$FORCE_PYENV_INSTALL" = true ]; then
echo_status "Forcing pyenv reinstallation"
fi
# 2. Check/install pyenv
if ! check_and_install_pyenv; then
return 1
fi
# 1. Check if Python is installed
if ! check_python_installed; then
return 1
fi
# 3. Initialize pyenv
if ! initialize_pyenv; then
return 1
fi
# 2. Check/install pyenv
if ! check_and_install_pyenv; then
return 1
fi
# 4. Check/create virtual environment
if ! check_environment_exists; then
return 1
fi
# 3. Initialize pyenv
if ! initialize_pyenv; then
return 1
fi
# 4. Check/create virtual environment
if ! check_environment_exists; then
return 1
fi
# 5. Activate the environment
if ! activate_environment; then
return 1
fi
PYTHON_BIN="python"
# 5. Activate the environment
if ! activate_environment; then
return 1
fi
# 5.5. Sanitize pytest environment to ensure proper output parsing
@@ -570,25 +477,14 @@ sanitize_pytest_env
# 6. Upgrade pip
upgrade_pip
# 7. Install this repository
if ! install_requirements; then
return 1
fi
# 7. Install requirements (including package in dev mode with all dependencies)
install_requirements
# Final verification
echo_status "Bootstrap completed successfully!"
if [ "$NO_ENV" = true ]; then
scripts_dir=$("$PYTHON_BIN" -c "import sys, sysconfig; scheme = 'nt_user' if sys.platform == 'win32' else 'posix_user'; print(sysconfig.get_path('scripts', scheme))")
echo_status "Installed for the current user (no virtual environment)"
echo_status "Command directory: $scripts_dir"
echo_status "If fiona-assistant is not found, add that directory to PATH."
echo_status "Python: $("$PYTHON_BIN" --version) ($PYTHON_BIN)"
echo_status "Pip: $("$PYTHON_BIN" -m pip --version)"
else
echo_status "Virtual environment '$PYENV_NAME' is active"
echo_status "Python: $(python --version) ($(which python))"
echo_status "Pip: $(pip --version)"
fi
echo_status "Virtual environment '$PYENV_NAME' is active"
echo_status "Python: $(python --version) ($(which python))"
echo_status "Pip: $(pip --version)"
echo_status ""
echo_status "To verify installation, try:"
echo_status " fiona-assistant test"
+52 -128
View File
@@ -1,29 +1,20 @@
# Generated by AI/LLM: 2026-09-29 - Windows PowerShell bootstrap for Fiona development
# Modified by AI/LLM: 2026-09-30 - --local and --no-env install options.
# Generated by CURSOR: 2026-09-29 - Windows PowerShell bootstrap for Fiona development
#
# Project bootstrap for Windows. Dot-source a virtual-environment install so
# the environment stays active in the current session (same idea as
# `source bootstrap`):
# Project bootstrap for Windows. Dot-source it so the virtual environment
# stays active in the current session (same idea as `source bootstrap`):
#
# Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
# . .\bootstrap.ps1
# . .\bootstrap.ps1 --env fiona_env
# . .\bootstrap.ps1 --env fiona_env --python 3.13.8 -c
# . .\bootstrap.ps1 --local
# powershell -ExecutionPolicy Bypass -File .\bootstrap.ps1 --local --no-env
#
# Creates a repo-local venv at .venvs\<name> with the Python launcher (py)
# or python on PATH. Does not install pyenv. --pyenv is accepted so the
# Linux flag still parses; on Windows it is a no-op notice.
#
# --local installs this repository as a normal package (not editable).
# --no-env skips the virtual environment and installs with pip --user,
# so fiona-assistant can run without activating an environment.
# --no-env may be executed with -File. A venv install must be dot-sourced.
#Requires -Version 5.1
# AI/LLM: flags are parsed by hand. Windows PowerShell 5.1 does not bind
# CURSOR: flags are parsed by hand. Windows PowerShell 5.1 does not bind
# `--python` to a parameter alias, and the bash script uses those flags.
function Write-FionaStatus {
@@ -58,7 +49,7 @@ function Remove-FionaBootstrapFunctions {
}
function Resolve-FionaPython {
# AI/LLM: pick a real interpreter. Prefer the py launcher so a requested
# CURSOR: pick a real interpreter. Prefer the py launcher so a requested
# version (3.13.8, then 3.13) wins over whatever `python` happens to be.
param([string]$Version)
@@ -103,7 +94,7 @@ function Resolve-FionaPython {
$FionaPrevErrorAction = $ErrorActionPreference
$ErrorActionPreference = "Stop"
# AI/LLM: a failed pip/py call must be checked via $LASTEXITCODE, not thrown.
# CURSOR: a failed pip/py call must be checked via $LASTEXITCODE, not thrown.
if (Get-Variable -Name PSNativeCommandUseErrorActionPreference -ErrorAction SilentlyContinue) {
$PSNativeCommandUseErrorActionPreference = $false
}
@@ -112,8 +103,6 @@ $Clean = $false
$EnvName = "test_env"
$PythonVersion = ""
$Pyenv = $false
$Local = $false
$NoEnv = $false
$bootstrapArgs = @($args)
for ($i = 0; $i -lt $bootstrapArgs.Count; $i++) {
$token = [string]$bootstrapArgs[$i]
@@ -123,12 +112,6 @@ for ($i = 0; $i -lt $bootstrapArgs.Count; $i++) {
elseif ($token -eq "-c" -or $token -eq "--clean") {
$Clean = $true
}
elseif ($token -eq "--local") {
$Local = $true
}
elseif ($token -eq "--no-env") {
$NoEnv = $true
}
elseif ($token -eq "--env") {
if (($i + 1) -ge $bootstrapArgs.Count) {
Write-FionaError "--env requires an environment name argument"
@@ -149,17 +132,15 @@ for ($i = 0; $i -lt $bootstrapArgs.Count; $i++) {
}
else {
Write-FionaError "Unknown option: $token"
Write-FionaError "Usage: . .\bootstrap.ps1 [--pyenv] [-c|--clean] [--env custom_name] [--python 3.x.x] [--local] [--no-env]"
Write-FionaError "Usage: . .\bootstrap.ps1 [--pyenv] [-c|--clean] [--env custom_name] [--python 3.x.x]"
Remove-FionaBootstrapFunctions
return
}
}
if (-not $NoEnv -and $MyInvocation.InvocationName -ne ".") {
if ($MyInvocation.InvocationName -ne ".") {
Write-FionaError "Dot-source this script so the virtual environment stays active in this session."
Write-FionaError " . .\bootstrap.ps1"
Write-FionaError "To install without a virtual environment, run:"
Write-FionaError " powershell -ExecutionPolicy Bypass -File .\bootstrap.ps1 --local --no-env"
Remove-FionaBootstrapFunctions
return
}
@@ -173,27 +154,13 @@ if (-not $repoRoot) {
$repoRoot = (Get-Location).Path
}
if ($NoEnv) {
Write-FionaStatus "Starting project bootstrap (no virtual environment)"
}
else {
Write-FionaStatus "Starting project bootstrap with environment: $EnvName"
}
if ($Local) {
Write-FionaStatus "Local install: this repository, not editable"
}
if ($NoEnv) {
Write-FionaStatus "No virtual environment: installing with --user"
}
if ($NoEnv -and $Clean) {
Write-FionaWarning "--clean has no effect with --no-env"
}
Write-FionaStatus "Starting project bootstrap with environment: $EnvName"
try {
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass -Force -ErrorAction Stop | Out-Null
}
catch {
# AI/LLM: a GPO can refuse the process override. RemoteSigned still runs local scripts.
# CURSOR: a GPO can refuse the process override. RemoteSigned still runs local scripts.
}
$pythonExe = Resolve-FionaPython -Version $PythonVersion
@@ -207,49 +174,44 @@ if (-not $pythonExe) {
return
}
$pipPython = $pythonExe
if (-not $NoEnv) {
$venvRoot = Join-Path $repoRoot ".venvs"
$venvPath = Join-Path $venvRoot $EnvName
$venvRoot = Join-Path $repoRoot ".venvs"
$venvPath = Join-Path $venvRoot $EnvName
if ($Clean -and (Test-Path -LiteralPath $venvPath)) {
Write-FionaStatus "Deleting existing virtual environment $EnvName..."
try {
Remove-Item -LiteralPath $venvPath -Recurse -Force
}
catch {
Write-FionaError "Failed to delete virtual environment $EnvName"
Write-FionaError $_.Exception.Message
Remove-FionaBootstrapFunctions
return
}
if ($Clean -and (Test-Path -LiteralPath $venvPath)) {
Write-FionaStatus "Deleting existing virtual environment $EnvName..."
try {
Remove-Item -LiteralPath $venvPath -Recurse -Force
}
$activate = Join-Path $venvPath "Scripts\Activate.ps1"
if (-not (Test-Path -LiteralPath $activate)) {
Write-FionaStatus "Creating virtual environment $EnvName with $pythonExe..."
New-Item -ItemType Directory -Path $venvRoot -Force | Out-Null
& $pythonExe -m venv $venvPath
if ($LASTEXITCODE -ne 0) {
Write-FionaError "Failed to create virtual environment $EnvName"
Remove-FionaBootstrapFunctions
return
}
}
else {
Write-FionaStatus "Virtual environment $EnvName already exists"
}
Write-FionaStatus "Activating $EnvName..."
. $activate
if (-not $env:VIRTUAL_ENV) {
Write-FionaError "Failed to activate $EnvName"
catch {
Write-FionaError "Failed to delete virtual environment $EnvName"
Write-FionaError $_.Exception.Message
Remove-FionaBootstrapFunctions
return
}
}
$pipPython = Join-Path $venvPath "Scripts\python.exe"
$activate = Join-Path $venvPath "Scripts\Activate.ps1"
if (-not (Test-Path -LiteralPath $activate)) {
Write-FionaStatus "Creating virtual environment $EnvName with $pythonExe..."
New-Item -ItemType Directory -Path $venvRoot -Force | Out-Null
& $pythonExe -m venv $venvPath
if ($LASTEXITCODE -ne 0) {
Write-FionaError "Failed to create virtual environment $EnvName"
Remove-FionaBootstrapFunctions
return
}
}
else {
Write-FionaStatus "Virtual environment $EnvName already exists"
}
Write-FionaStatus "Activating $EnvName..."
. $activate
if (-not $env:VIRTUAL_ENV) {
Write-FionaError "Failed to activate $EnvName"
Remove-FionaBootstrapFunctions
return
}
if ($env:PYTEST_ADDOPTS) {
@@ -282,45 +244,15 @@ if (-not (Test-Path -LiteralPath $pyproject)) {
}
Write-FionaStatus "Upgrading pip..."
if ($NoEnv) {
& $pipPython -m pip install --upgrade pip --user
}
else {
& $pipPython -m pip install --upgrade pip
}
& python -m pip install --upgrade pip
if ($LASTEXITCODE -ne 0) {
Write-FionaWarning "Failed to upgrade pip. Continuing with package installation..."
}
$helper = Join-Path $repoRoot "scripts\bootstrap_install.py"
if (-not (Test-Path -LiteralPath $helper)) {
Write-FionaError "Missing $helper"
Remove-FionaBootstrapFunctions
return
}
$helperArgs = @($helper)
if ($Local) {
$helperArgs += "--local"
}
if ($NoEnv) {
$helperArgs += "--no-env"
}
$pipWords = @(& $pipPython @helperArgs | Where-Object { $_ -and "$_".Trim() -ne "" })
if ($LASTEXITCODE -ne 0 -or $pipWords.Count -eq 0) {
Write-FionaError "Failed to resolve pip install arguments"
Remove-FionaBootstrapFunctions
return
}
if ($Local) {
Write-FionaStatus "Installing this repository as a normal package..."
}
else {
Write-FionaStatus "Installing package and dependencies from pyproject.toml..."
}
Write-FionaStatus "Installing package and dependencies from pyproject.toml..."
Push-Location $repoRoot
try {
& $pipPython -m pip @pipWords
& python -m pip install -e ".[dev]"
if ($LASTEXITCODE -ne 0) {
Write-FionaError "Failed to install package and dependencies from pyproject.toml"
Remove-FionaBootstrapFunctions
@@ -331,26 +263,18 @@ finally {
Pop-Location
}
$pythonVersionText = (& $pipPython --version 2>&1 | Out-String).Trim()
$pipVersionText = (& $pipPython -m pip --version 2>&1 | Out-String).Trim()
$pythonCmd = Get-Command python -ErrorAction SilentlyContinue
$pythonWhere = if ($pythonCmd) { $pythonCmd.Source } else { "python" }
$pythonVersionText = (& python --version 2>&1 | Out-String).Trim()
$pipVersionText = (& python -m pip --version 2>&1 | Out-String).Trim()
Write-FionaStatus "Bootstrap completed successfully!"
if ($NoEnv) {
$scriptsDir = (& $pipPython -c "import sysconfig; print(sysconfig.get_path('scripts', 'nt_user'))" | Out-String).Trim()
Write-FionaStatus "Installed for the current user (no virtual environment)"
Write-FionaStatus "Command directory: $scriptsDir"
Write-FionaStatus "If fiona-assistant is not found, add that directory to PATH."
}
else {
Write-FionaStatus "Virtual environment '$EnvName' is active"
}
Write-FionaStatus "Python: $pythonVersionText ($pipPython)"
Write-FionaStatus "Virtual environment '$EnvName' is active"
Write-FionaStatus "Python: $pythonVersionText ($pythonWhere)"
Write-FionaStatus "Pip: $pipVersionText"
Write-FionaStatus ""
Write-FionaStatus "To verify installation, try:"
if (-not $NoEnv) {
Write-FionaStatus " python scripts/run_tests.py"
}
Write-FionaStatus " python src/run_tests.py"
Write-FionaStatus " fiona-assistant --help"
Write-FionaStatus ""
Write-FionaStatus "Happy coding!"
-2
View File
@@ -11,8 +11,6 @@ if ! python -m build --version >/dev/null 2>&1; then
fi
# Build the distribution files
rm -rf dist/ build/ fiona_assistant.egg-info/
python -m build
echo "Build completed. Distribution files are in the dist/ directory."
+612
View File
@@ -0,0 +1,612 @@
<?xml version="1.0" ?>
<coverage version="7.16.1" timestamp="1790528640380" lines-valid="578" lines-covered="454" line-rate="0.7855" branches-covered="0" branches-valid="0" branch-rate="0" complexity="0">
<!-- Generated by coverage.py: https://coverage.readthedocs.io/en/7.16.1 -->
<!-- Based on https://raw.githubusercontent.com/cobertura/web/master/htdocs/xml/coverage-04.dtd -->
<sources>
<source>/home/user/repositories/fiona-assistant/src/customlinter</source>
</sources>
<packages>
<package name="." line-rate="0.7855" branch-rate="0" complexity="0">
<classes>
<class name="__init__.py" filename="__init__.py" complexity="0" line-rate="1" branch-rate="0">
<methods/>
<lines>
<line number="17" hits="1"/>
<line number="26" hits="1"/>
<line number="27" hits="1"/>
<line number="29" hits="1"/>
</lines>
</class>
<class name="base.py" filename="base.py" complexity="0" line-rate="0.6962" branch-rate="0">
<methods/>
<lines>
<line number="8" hits="1"/>
<line number="10" hits="1"/>
<line number="11" hits="1"/>
<line number="12" hits="1"/>
<line number="13" hits="1"/>
<line number="14" hits="1"/>
<line number="15" hits="1"/>
<line number="16" hits="1"/>
<line number="18" hits="1"/>
<line number="25" hits="1"/>
<line number="28" hits="1"/>
<line number="33" hits="1"/>
<line number="49" hits="1"/>
<line number="52" hits="1"/>
<line number="65" hits="1"/>
<line number="66" hits="1"/>
<line number="67" hits="1"/>
<line number="68" hits="1"/>
<line number="72" hits="1"/>
<line number="79" hits="1"/>
<line number="90" hits="1"/>
<line number="97" hits="1"/>
<line number="118" hits="1"/>
<line number="119" hits="1"/>
<line number="120" hits="1"/>
<line number="121" hits="1"/>
<line number="122" hits="1"/>
<line number="123" hits="1"/>
<line number="124" hits="1"/>
<line number="126" hits="1"/>
<line number="128" hits="1"/>
<line number="138" hits="1"/>
<line number="140" hits="1"/>
<line number="147" hits="1"/>
<line number="164" hits="1"/>
<line number="167" hits="1"/>
<line number="170" hits="1"/>
<line number="172" hits="1"/>
<line number="178" hits="1"/>
<line number="179" hits="1"/>
<line number="181" hits="1"/>
<line number="183" hits="1"/>
<line number="184" hits="1"/>
<line number="186" hits="1"/>
<line number="187" hits="1"/>
<line number="188" hits="1"/>
<line number="190" hits="1"/>
<line number="191" hits="1"/>
<line number="192" hits="1"/>
<line number="193" hits="1"/>
<line number="194" hits="0"/>
<line number="195" hits="1"/>
<line number="196" hits="1"/>
<line number="197" hits="1"/>
<line number="210" hits="1"/>
<line number="219" hits="0"/>
<line number="220" hits="0"/>
<line number="222" hits="0"/>
<line number="223" hits="0"/>
<line number="226" hits="0"/>
<line number="229" hits="0"/>
<line number="230" hits="0"/>
<line number="231" hits="0"/>
<line number="233" hits="0"/>
<line number="235" hits="1"/>
<line number="244" hits="1"/>
<line number="247" hits="1"/>
<line number="248" hits="1"/>
<line number="250" hits="1"/>
<line number="251" hits="1"/>
<line number="252" hits="1"/>
<line number="253" hits="1"/>
<line number="254" hits="1"/>
<line number="255" hits="1"/>
<line number="256" hits="1"/>
<line number="257" hits="0"/>
<line number="258" hits="0"/>
<line number="261" hits="1"/>
<line number="262" hits="1"/>
<line number="265" hits="1"/>
<line number="266" hits="1"/>
<line number="267" hits="1"/>
<line number="270" hits="1"/>
<line number="272" hits="1"/>
<line number="273" hits="1"/>
<line number="276" hits="1"/>
<line number="278" hits="1"/>
<line number="280" hits="1"/>
<line number="290" hits="1"/>
<line number="291" hits="1"/>
<line number="292" hits="1"/>
<line number="294" hits="1"/>
<line number="295" hits="1"/>
<line number="296" hits="1"/>
<line number="298" hits="1"/>
<line number="308" hits="1"/>
<line number="309" hits="1"/>
<line number="312" hits="1"/>
<line number="313" hits="1"/>
<line number="316" hits="1"/>
<line number="317" hits="1"/>
<line number="318" hits="1"/>
<line number="319" hits="1"/>
<line number="320" hits="1"/>
<line number="321" hits="1"/>
<line number="323" hits="1"/>
<line number="325" hits="1"/>
<line number="335" hits="1"/>
<line number="336" hits="1"/>
<line number="337" hits="1"/>
<line number="338" hits="1"/>
<line number="339" hits="1"/>
<line number="340" hits="0"/>
<line number="342" hits="0"/>
<line number="343" hits="0"/>
<line number="345" hits="1"/>
<line number="354" hits="1"/>
<line number="355" hits="1"/>
<line number="357" hits="1"/>
<line number="358" hits="1"/>
<line number="359" hits="1"/>
<line number="361" hits="1"/>
<line number="363" hits="1"/>
<line number="372" hits="1"/>
<line number="374" hits="1"/>
<line number="390" hits="0"/>
<line number="391" hits="0"/>
<line number="392" hits="0"/>
<line number="393" hits="0"/>
<line number="396" hits="0"/>
<line number="397" hits="0"/>
<line number="398" hits="0"/>
<line number="399" hits="0"/>
<line number="400" hits="0"/>
<line number="406" hits="0"/>
<line number="407" hits="0"/>
<line number="408" hits="0"/>
<line number="409" hits="0"/>
<line number="412" hits="0"/>
<line number="415" hits="0"/>
<line number="416" hits="0"/>
<line number="417" hits="0"/>
<line number="418" hits="0"/>
<line number="419" hits="0"/>
<line number="420" hits="0"/>
<line number="421" hits="0"/>
<line number="423" hits="0"/>
<line number="424" hits="0"/>
<line number="425" hits="0"/>
<line number="426" hits="0"/>
<line number="427" hits="0"/>
<line number="429" hits="0"/>
<line number="430" hits="0"/>
<line number="433" hits="0"/>
<line number="434" hits="0"/>
<line number="435" hits="0"/>
<line number="436" hits="0"/>
<line number="438" hits="0"/>
</lines>
</class>
<class name="fmlint.py" filename="fmlint.py" complexity="0" line-rate="0.766" branch-rate="0">
<methods/>
<lines>
<line number="19" hits="1"/>
<line number="21" hits="1"/>
<line number="22" hits="1"/>
<line number="23" hits="1"/>
<line number="24" hits="1"/>
<line number="25" hits="1"/>
<line number="27" hits="1"/>
<line number="29" hits="1"/>
<line number="37" hits="1"/>
<line number="40" hits="1"/>
<line number="43" hits="1"/>
<line number="46" hits="1"/>
<line number="53" hits="1"/>
<line number="103" hits="1"/>
<line number="115" hits="0"/>
<line number="116" hits="0"/>
<line number="117" hits="0"/>
<line number="118" hits="0"/>
<line number="119" hits="0"/>
<line number="120" hits="0"/>
<line number="121" hits="0"/>
<line number="122" hits="0"/>
<line number="125" hits="1"/>
<line number="134" hits="1"/>
<line number="137" hits="1"/>
<line number="148" hits="1"/>
<line number="151" hits="1"/>
<line number="152" hits="1"/>
<line number="155" hits="1"/>
<line number="156" hits="1"/>
<line number="158" hits="1"/>
<line number="161" hits="1"/>
<line number="180" hits="1"/>
<line number="181" hits="1"/>
<line number="183" hits="1"/>
<line number="184" hits="1"/>
<line number="186" hits="1"/>
<line number="187" hits="1"/>
<line number="190" hits="1"/>
<line number="191" hits="1"/>
<line number="192" hits="1"/>
<line number="193" hits="1"/>
<line number="196" hits="1"/>
<line number="197" hits="1"/>
<line number="198" hits="1"/>
<line number="199" hits="1"/>
<line number="202" hits="1"/>
<line number="203" hits="1"/>
<line number="210" hits="1"/>
<line number="212" hits="1"/>
<line number="213" hits="1"/>
<line number="219" hits="1"/>
<line number="226" hits="1"/>
<line number="238" hits="1"/>
<line number="239" hits="1"/>
<line number="240" hits="1"/>
<line number="242" hits="1"/>
<line number="258" hits="1"/>
<line number="261" hits="1"/>
<line number="262" hits="1"/>
<line number="263" hits="1"/>
<line number="264" hits="1"/>
<line number="265" hits="1"/>
<line number="266" hits="1"/>
<line number="267" hits="1"/>
<line number="270" hits="1"/>
<line number="272" hits="1"/>
<line number="275" hits="1"/>
<line number="278" hits="1"/>
<line number="281" hits="1"/>
<line number="282" hits="1"/>
<line number="283" hits="1"/>
<line number="286" hits="1"/>
<line number="287" hits="1"/>
<line number="290" hits="1"/>
<line number="291" hits="1"/>
<line number="293" hits="1"/>
<line number="302" hits="1"/>
<line number="304" hits="1"/>
<line number="306" hits="1"/>
<line number="307" hits="0"/>
<line number="310" hits="1"/>
<line number="311" hits="0"/>
<line number="314" hits="1"/>
<line number="315" hits="1"/>
<line number="316" hits="1"/>
<line number="318" hits="0"/>
<line number="319" hits="0"/>
<line number="320" hits="0"/>
<line number="321" hits="0"/>
<line number="322" hits="0"/>
<line number="323" hits="0"/>
<line number="325" hits="0"/>
<line number="326" hits="0"/>
<line number="327" hits="1"/>
<line number="328" hits="0"/>
<line number="331" hits="1"/>
<line number="332" hits="1"/>
<line number="333" hits="0"/>
<line number="335" hits="0"/>
<line number="343" hits="0"/>
<line number="344" hits="0"/>
<line number="347" hits="1"/>
<line number="348" hits="1"/>
<line number="349" hits="1"/>
<line number="350" hits="1"/>
<line number="358" hits="1"/>
<line number="359" hits="1"/>
<line number="362" hits="1"/>
<line number="363" hits="1"/>
<line number="371" hits="1"/>
<line number="372" hits="1"/>
<line number="375" hits="1"/>
<line number="378" hits="1"/>
<line number="384" hits="1"/>
<line number="385" hits="1"/>
<line number="386" hits="1"/>
<line number="394" hits="1"/>
<line number="397" hits="1"/>
<line number="399" hits="1"/>
<line number="400" hits="1"/>
<line number="408" hits="1"/>
<line number="410" hits="1"/>
<line number="411" hits="1"/>
<line number="413" hits="1"/>
<line number="414" hits="1"/>
<line number="416" hits="1"/>
<line number="417" hits="1"/>
<line number="418" hits="1"/>
<line number="426" hits="1"/>
<line number="427" hits="1"/>
<line number="428" hits="1"/>
<line number="436" hits="1"/>
<line number="439" hits="1"/>
<line number="444" hits="1"/>
<line number="446" hits="1"/>
<line number="473" hits="1"/>
<line number="476" hits="1"/>
<line number="487" hits="1"/>
<line number="489" hits="1"/>
<line number="491" hits="1"/>
<line number="492" hits="1"/>
<line number="493" hits="1"/>
<line number="495" hits="0"/>
<line number="496" hits="0"/>
<line number="497" hits="0"/>
<line number="498" hits="0"/>
<line number="499" hits="0"/>
<line number="500" hits="0"/>
<line number="502" hits="0"/>
<line number="503" hits="0"/>
<line number="504" hits="1"/>
<line number="505" hits="0"/>
<line number="508" hits="1"/>
<line number="511" hits="1"/>
<line number="512" hits="1"/>
<line number="513" hits="1"/>
<line number="514" hits="1"/>
<line number="516" hits="1"/>
<line number="517" hits="1"/>
<line number="518" hits="1"/>
<line number="523" hits="1"/>
<line number="524" hits="1"/>
<line number="526" hits="1"/>
<line number="527" hits="1"/>
<line number="528" hits="1"/>
<line number="530" hits="1"/>
<line number="532" hits="1"/>
<line number="548" hits="1"/>
<line number="549" hits="1"/>
<line number="550" hits="1"/>
<line number="551" hits="1"/>
<line number="552" hits="1"/>
<line number="556" hits="1"/>
<line number="558" hits="1"/>
<line number="570" hits="1"/>
<line number="572" hits="1"/>
<line number="573" hits="1"/>
<line number="575" hits="1"/>
<line number="578" hits="1"/>
<line number="579" hits="1"/>
<line number="580" hits="1"/>
<line number="583" hits="1"/>
<line number="584" hits="1"/>
<line number="587" hits="1"/>
<line number="588" hits="0"/>
<line number="591" hits="1"/>
<line number="592" hits="0"/>
<line number="594" hits="1"/>
<line number="597" hits="1"/>
<line number="598" hits="1"/>
<line number="599" hits="1"/>
<line number="600" hits="1"/>
<line number="601" hits="1"/>
<line number="602" hits="1"/>
<line number="603" hits="1"/>
<line number="605" hits="1"/>
<line number="607" hits="1"/>
<line number="617" hits="1"/>
<line number="618" hits="1"/>
<line number="619" hits="1"/>
<line number="620" hits="1"/>
<line number="621" hits="1"/>
<line number="622" hits="0"/>
<line number="623" hits="1"/>
<line number="624" hits="1"/>
<line number="625" hits="1"/>
<line number="627" hits="1"/>
<line number="637" hits="1"/>
<line number="638" hits="1"/>
<line number="639" hits="1"/>
<line number="641" hits="1"/>
<line number="642" hits="1"/>
<line number="643" hits="1"/>
<line number="645" hits="1"/>
<line number="646" hits="1"/>
<line number="647" hits="1"/>
<line number="648" hits="1"/>
<line number="649" hits="1"/>
<line number="651" hits="1"/>
<line number="652" hits="1"/>
<line number="653" hits="0"/>
<line number="655" hits="1"/>
<line number="656" hits="1"/>
<line number="658" hits="1"/>
<line number="660" hits="1"/>
<line number="671" hits="1"/>
<line number="673" hits="1"/>
<line number="674" hits="1"/>
<line number="675" hits="1"/>
<line number="676" hits="1"/>
<line number="679" hits="1"/>
<line number="680" hits="1"/>
<line number="681" hits="1"/>
<line number="682" hits="1"/>
<line number="683" hits="1"/>
<line number="684" hits="1"/>
<line number="685" hits="1"/>
<line number="688" hits="1"/>
<line number="689" hits="1"/>
<line number="690" hits="1"/>
<line number="691" hits="1"/>
<line number="692" hits="1"/>
<line number="693" hits="1"/>
<line number="695" hits="1"/>
<line number="696" hits="1"/>
<line number="699" hits="1"/>
<line number="700" hits="1"/>
<line number="701" hits="1"/>
<line number="702" hits="1"/>
<line number="703" hits="1"/>
<line number="705" hits="1"/>
<line number="707" hits="1"/>
<line number="725" hits="0"/>
<line number="726" hits="0"/>
<line number="727" hits="0"/>
<line number="728" hits="0"/>
<line number="731" hits="0"/>
<line number="732" hits="0"/>
<line number="733" hits="0"/>
<line number="734" hits="0"/>
<line number="735" hits="0"/>
<line number="741" hits="0"/>
<line number="742" hits="0"/>
<line number="743" hits="0"/>
<line number="744" hits="0"/>
<line number="745" hits="0"/>
<line number="746" hits="0"/>
<line number="747" hits="0"/>
<line number="750" hits="0"/>
<line number="751" hits="0"/>
<line number="752" hits="0"/>
<line number="753" hits="0"/>
<line number="754" hits="0"/>
<line number="755" hits="0"/>
<line number="758" hits="0"/>
<line number="759" hits="0"/>
<line number="760" hits="0"/>
<line number="761" hits="0"/>
<line number="762" hits="0"/>
<line number="763" hits="0"/>
<line number="765" hits="0"/>
<line number="766" hits="0"/>
<line number="769" hits="0"/>
<line number="770" hits="0"/>
<line number="771" hits="0"/>
<line number="772" hits="0"/>
<line number="774" hits="0"/>
<line number="781" hits="1"/>
<line number="783" hits="1"/>
<line number="784" hits="1"/>
<line number="786" hits="1"/>
<line number="788" hits="1"/>
<line number="790" hits="1"/>
<line number="793" hits="1"/>
<line number="796" hits="1"/>
<line number="800" hits="1"/>
<line number="807" hits="1"/>
<line number="809" hits="1"/>
<line number="811" hits="1"/>
<line number="812" hits="1"/>
<line number="814" hits="1"/>
<line number="815" hits="1"/>
<line number="816" hits="1"/>
<line number="818" hits="1"/>
<line number="819" hits="1"/>
<line number="820" hits="1"/>
<line number="822" hits="1"/>
<line number="823" hits="1"/>
<line number="824" hits="1"/>
<line number="826" hits="0"/>
<line number="827" hits="0"/>
</lines>
</class>
<class name="pycustomlint.py" filename="pycustomlint.py" complexity="0" line-rate="0.9712" branch-rate="0">
<methods/>
<lines>
<line number="19" hits="1"/>
<line number="21" hits="1"/>
<line number="22" hits="1"/>
<line number="23" hits="1"/>
<line number="24" hits="1"/>
<line number="25" hits="1"/>
<line number="27" hits="1"/>
<line number="29" hits="1"/>
<line number="32" hits="1"/>
<line number="40" hits="1"/>
<line number="60" hits="1"/>
<line number="63" hits="1"/>
<line number="69" hits="1"/>
<line number="70" hits="1"/>
<line number="72" hits="1"/>
<line number="74" hits="1"/>
<line number="75" hits="1"/>
<line number="76" hits="1"/>
<line number="78" hits="1"/>
<line number="80" hits="1"/>
<line number="81" hits="1"/>
<line number="82" hits="1"/>
<line number="84" hits="1"/>
<line number="87" hits="1"/>
<line number="88" hits="1"/>
<line number="89" hits="1"/>
<line number="90" hits="1"/>
<line number="91" hits="1"/>
<line number="92" hits="1"/>
<line number="93" hits="1"/>
<line number="94" hits="1"/>
<line number="95" hits="1"/>
<line number="97" hits="1"/>
<line number="105" hits="1"/>
<line number="107" hits="1"/>
<line number="109" hits="1"/>
<line number="116" hits="1"/>
<line number="122" hits="1"/>
<line number="123" hits="1"/>
<line number="124" hits="1"/>
<line number="126" hits="1"/>
<line number="135" hits="1"/>
<line number="137" hits="1"/>
<line number="139" hits="1"/>
<line number="140" hits="1"/>
<line number="143" hits="1"/>
<line number="144" hits="0"/>
<line number="147" hits="1"/>
<line number="148" hits="1"/>
<line number="151" hits="1"/>
<line number="152" hits="1"/>
<line number="153" hits="1"/>
<line number="155" hits="1"/>
<line number="157" hits="1"/>
<line number="158" hits="1"/>
<line number="159" hits="1"/>
<line number="161" hits="1"/>
<line number="163" hits="1"/>
<line number="177" hits="1"/>
<line number="178" hits="1"/>
<line number="179" hits="1"/>
<line number="181" hits="1"/>
<line number="182" hits="1"/>
<line number="185" hits="1"/>
<line number="188" hits="1"/>
<line number="189" hits="1"/>
<line number="190" hits="1"/>
<line number="192" hits="1"/>
<line number="194" hits="1"/>
<line number="196" hits="1"/>
<line number="203" hits="1"/>
<line number="205" hits="1"/>
<line number="207" hits="1"/>
<line number="209" hits="1"/>
<line number="210" hits="1"/>
<line number="212" hits="1"/>
<line number="215" hits="1"/>
<line number="218" hits="1"/>
<line number="222" hits="1"/>
<line number="228" hits="1"/>
<line number="234" hits="1"/>
<line number="236" hits="1"/>
<line number="238" hits="1"/>
<line number="239" hits="1"/>
<line number="240" hits="1"/>
<line number="242" hits="1"/>
<line number="243" hits="1"/>
<line number="245" hits="1"/>
<line number="247" hits="1"/>
<line number="248" hits="1"/>
<line number="249" hits="1"/>
<line number="252" hits="1"/>
<line number="253" hits="1"/>
<line number="255" hits="1"/>
<line number="257" hits="1"/>
<line number="258" hits="1"/>
<line number="260" hits="1"/>
<line number="262" hits="1"/>
<line number="263" hits="1"/>
<line number="264" hits="1"/>
<line number="266" hits="1"/>
<line number="267" hits="1"/>
<line number="269" hits="0"/>
<line number="270" hits="0"/>
</lines>
</class>
</classes>
</package>
</packages>
</coverage>
+20 -28
View File
@@ -1,32 +1,29 @@
[build-system]
requires = ["setuptools>=61"]
requires = ["setuptools>=42"]
build-backend = "setuptools.build_meta"
[project]
name = "fiona-assistant"
version = "1.19.0"
version = "0.36.0"
description = "Tools for managing the LLM wiki."
authors = [{name = "nvaldeziii"}]
readme = "README.md"
license = "GPL-3.0-only"
requires-python = ">=3.11"
requires-python = ">=3.8"
classifiers = [
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
"Operating System :: OS Independent",
]
# Core dependencies (required for runtime)
# Note: html5validator requires Java 8+ to be installed on the system
dependencies = [
"pymarkdownlnt>=0.9.0",
"ruff>=0.16.8",
"pyyaml>=6.0.3",
"pylint",
"GitPython>=3.1.0",
"beautifulsoup4>=4.12.0",
"markdownify>=1.1.0",
"pypdf>=5.0.0",
"html5validator>=0.4.0",
]
# Optional dependencies groups
@@ -35,7 +32,6 @@ dev = [
"pytest>=7.0.0",
"pytest-cov>=4.0.0",
"pytest-mock",
"pylint",
"build",
"twine",
]
@@ -45,23 +41,23 @@ include-package-data = true
[tool.setuptools.packages.find]
where = ["src"]
include = ["fiona*"]
# llm-wiki is a hyphenated, non-importable data-only "package" that ships
# template files (wiki/, ingest/, AGENTS.md, etc.) as package data.
# The hyphen makes it unimportable as a Python module (import llm-wiki
# is a SyntaxError), but setuptools allows it as a distribution package name.
# The __init__.py exists only to satisfy setuptools' package detection.
include = ["fiona", "librarian", "mdoctor", "python_ruff", "customlinter", "sanity_runner", "llm-wiki", "wiki", "gittool", "html_js_lint"]
[tool.setuptools.package-data]
fiona = [
"data/llm_wiki/**/*",
"data/default.gitignore",
"wiki/templates/*",
"gittool/templates/*",
]
# Include all files under llm-wiki/ as package data (not Python code)
"llm-wiki" = ["*", "**/*"]
[tool.pytest.ini_options]
testpaths = ["tests"]
pythonpath = ["src"]
testpaths = ["src/tests"]
[tool.coverage.run]
source = ["src/fiona"]
omit = ["*/tests/*", "*/test_*.py"]
source = ["src"]
omit = ["src/tests/*", "*/tests/*", "*/test_*.py"]
[tool.coverage.report]
exclude_lines = [
@@ -77,10 +73,6 @@ exclude_lines = [
# Define console scripts entry points
[project.scripts]
fiona-assistant = "fiona.__main__:main"
librarian = "fiona.librarian.librarian:main"
mdoctor = "fiona.mdoctor.mdoctor:main"
[tool.ruff]
target-version = "py311"
line-length = 100
src = ["src"]
librarian = "librarian.librarian:main"
mdoctor = "mdoctor.mdoctor:main"
html-js-lint = "html_js_lint.html_js_lint:main"
-83
View File
@@ -1,83 +0,0 @@
"""Pip arguments for bootstrap install modes.
Generated by AI/LLM: 2026-09-30 - local and no-env install options.
SPEC 7.21. Both ``bootstrap`` and ``bootstrap.ps1`` call this script
and read one pip argument per stdout line.
"""
import argparse
import sys
def pip_install_args(
*,
local: bool,
no_env: bool,
break_system_packages: bool = False,
) -> list[str]:
"""Return the arguments passed to ``python -m pip``.
Args:
local: Install this repository as a normal package. When
false, the install is editable and includes the ``dev``
extra.
no_env: Add ``--user`` so the install does not need a virtual
environment.
break_system_packages: Add ``--break-system-packages``. Bash
sets this. PowerShell does not.
Returns:
The pip argument list, starting with ``install``.
"""
args = ["install"]
if no_env:
args.append("--user")
if break_system_packages:
args.append("--break-system-packages")
if local:
args.append(".")
else:
args.extend(["-e", ".[dev]"])
return args
def main(argv: list[str] | None = None) -> int:
"""Print pip arguments, one per line.
Args:
argv: Flags to parse. ``None`` reads ``sys.argv``.
Returns:
Process exit code. ``0`` after the arguments are written.
"""
parser = argparse.ArgumentParser(
description="Print pip install arguments for bootstrap.",
)
parser.add_argument(
"--local",
action="store_true",
help="install this repository as a normal package",
)
parser.add_argument(
"--no-env",
action="store_true",
help="add --user so no virtual environment is required",
)
parser.add_argument(
"--break-system-packages",
action="store_true",
help="pass --break-system-packages to pip",
)
parsed = parser.parse_args(argv)
words = pip_install_args(
local=parsed.local,
no_env=parsed.no_env,
break_system_packages=parsed.break_system_packages,
)
sys.stdout.write("\n".join(words) + "\n")
return 0
if __name__ == "__main__":
sys.exit(main())
-17
View File
@@ -1,17 +0,0 @@
"""Repo-root launcher for the sanity runner.
Generated by AI/LLM: 2026-09-29 - implementation lives in fiona.run_sanity.
"""
import pathlib
import sys
_SRC = pathlib.Path(__file__).resolve().parent.parent / "src"
if str(_SRC) not in sys.path:
sys.path.insert(0, str(_SRC))
import fiona.run_sanity as runner # noqa: E402
if __name__ == "__main__":
sys.exit(runner.main())
-17
View File
@@ -1,17 +0,0 @@
"""Repo-root launcher for the test runner.
Generated by AI/LLM: 2026-09-29 - implementation lives in fiona.run_tests.
"""
import pathlib
import sys
_SRC = pathlib.Path(__file__).resolve().parent.parent / "src"
if str(_SRC) not in sys.path:
sys.path.insert(0, str(_SRC))
import fiona.run_tests as runner # noqa: E402
if __name__ == "__main__":
sys.exit(runner.main())
+15 -42
View File
@@ -7,7 +7,7 @@ A collection of tools for managing the LLM wiki and related operations.
The Fiona Assistant project provides command-line tools for:
- **Librarian**: Managing LLM wiki content and operations
- **MDoctor**: Markdown diagnosis (lint, list, and fix)
- **MDoctor**: [Description to be added]
## Installation
@@ -41,14 +41,6 @@ The Fiona Assistant project provides command-line tools for:
. .\bootstrap.ps1 --env fiona_env --python 3.11.0
```
Install this checkout as a normal package with `--local`. Skip the
virtual environment with `--no-env` (PowerShell may use `-File`
instead of dot-sourcing):
```powershell
powershell -ExecutionPolicy Bypass -File .\bootstrap.ps1 --local --no-env
```
3. Install the package in development mode:
```bash
@@ -65,7 +57,7 @@ pip install -e ".[dev]"
### Librarian
See [librarian/README.md](./fiona/librarian/README.md) for detailed usage.
See [librarian/README.md](./librarian/README.md) for detailed usage.
```bash
librarian --help
@@ -73,7 +65,7 @@ librarian --help
### MDoctor
See [mdoctor/README.md](./fiona/mdoctor/README.md) for detailed usage.
See [mdoctor/README.md](./mdoctor/README.md) for detailed usage.
```bash
mdoctor --help
@@ -82,38 +74,22 @@ mdoctor --help
## Project Structure
```text
src/fiona/
├── __main__.py # CLI entry point
├── librarian/ # Wiki content management
├── mdoctor/ # Markdown diagnosis
├── customlinter/ # Frontmatter, markdown, Python, and JSON linters
├── python_ruff/ # Ruff wrapper
├── gittool/ # Git operations
├── sanity_runner/ # Shared sanity engine
├── wiki/ # Wiki init, update, sanity, and graph
├── data/llm_wiki/ # Packaged wiki template
├── run_tests.py # Test runner
└── run_sanity.py # Sanity runner
tests/ # Unit tests (repository root, not installed)
scripts/
├── run_tests.py
├── run_sanity.py
├── settings_dev.json
├── utils.sh
└── install_packages.sh
src/
├── fiona/ # Main package
├── librarian/ # Wiki content management tool
├── mdoctor/ # [Tool description]
├── tests/ # Unit tests
└── README.md # This file
```
## Running Tests
```bash
# Run all tests
python scripts/run_tests.py
# Or use pytest directly
python -m pytest
pytest
# Run with coverage
python -m pytest --cov=src/fiona
pytest --cov=src
```
## Development Tools
@@ -122,10 +98,9 @@ The project includes several development utilities:
- `bootstrap`: Set up the Python development environment (bash)
- `bootstrap.ps1`: Set up the Python development environment (PowerShell)
- `scripts/utils.sh`: Shell utility functions
- `scripts/install_packages.sh`: Install local Python packages
- `scripts/run_tests.py`: Run the test suite
- `scripts/run_sanity.py`: Run tests and linters
- `src/utils.sh`: Shell utility functions
- `src/install_packages.sh`: Install local Python packages
- `src/run_tests.sh`: Run the test suite
## License
@@ -133,6 +108,4 @@ GPL-3.0-only
## Contributing
Follow [AGENTS.md](../AGENTS.md): update [SPEC.md](../SPEC.md) first,
add a failing test, then implement. Run `python scripts/run_tests.py`
and `python scripts/run_sanity.py --non-interactive` before finishing.
[Contribution guidelines to be added]
+1
View File
@@ -0,0 +1 @@
# Fiona tools package
@@ -16,7 +16,7 @@ various code and content rules. It includes:
All linters share a common interface and output format for consistency.
"""
from fiona.customlinter.base import (
from customlinter.base import (
DEFAULT_SETTINGS_FILE,
READONLY_ZONES,
BaseLinter,
@@ -25,9 +25,9 @@ from fiona.customlinter.base import (
LinterError,
LintViolation,
)
from fiona.customlinter.fmlint import FrontmatterLinter
from fiona.customlinter.mdlinter import MarkdownLinter
from fiona.customlinter.pycustomlint import PythonCustomLinter
from customlinter.fmlint import FrontmatterLinter
from customlinter.mdlinter import MarkdownLinter
from customlinter.pycustomlint import PythonCustomLinter
__all__ = [
"DEFAULT_SETTINGS_FILE",
@@ -15,15 +15,19 @@ import pathlib
from abc import ABC, abstractmethod
from typing import Any, ClassVar
from fiona.paths import llm_wiki_template
logger = logging.getLogger(__name__)
# =============================================================================
# Constants
# =============================================================================
# Read-only zone names that should be skipped
READONLY_ZONES = ["personal", "raw"]
# Default settings file path
DEFAULT_SETTINGS_FILE = llm_wiki_template() / "fiona-settings.json"
DEFAULT_SETTINGS_FILE = (
pathlib.Path(__file__).resolve().parent.parent / "llm-wiki" / "fiona-settings.json"
)
# Re-export constants for backward compatibility
__all__ = [
@@ -37,20 +41,6 @@ __all__ = [
]
def report_path(path: pathlib.Path | str) -> str:
"""Return a forward-slash path for reports.
Generated by AI/LLM: 2026-09-29 - str(Path) uses backslashes on Windows,
which makes report text differ from the POSIX form the checks expect.
Args:
path: Path to display.
Returns:
The path with forward slashes.
"""
return pathlib.Path(path).as_posix()
# =============================================================================
# Exceptions
@@ -66,7 +56,7 @@ class LinterError(Exception):
line: int | None = None,
):
"""Initialize the exception with details.
Args:
message: The error message describing the linting failure.
filepath: The path to the file that failed linting.
@@ -81,14 +71,14 @@ class LinterError(Exception):
# Backward compatibility aliases
class CustomLintError(LinterError):
"""Exception raised when custom Python linting fails.
This is an alias for LinterError for backward compatibility.
"""
class FrontmatterLintError(LinterError):
"""Exception raised when frontmatter validation fails.
This is an alias for LinterError for backward compatibility.
"""
@@ -99,7 +89,7 @@ class FrontmatterLintError(LinterError):
class LintViolation:
"""Represents a single linting violation.
This class provides a standardized format for representing violations
across all linter types.
"""
@@ -115,7 +105,7 @@ class LintViolation:
severity: str = "error",
):
"""Initialize a violation.
Args:
filepath: Path to the file containing the violation.
line: Line number of the violation.
@@ -136,7 +126,7 @@ class LintViolation:
def to_dict(self) -> dict:
"""Convert violation to dictionary for reporting."""
return {
"file": report_path(self.filepath),
"file": str(self.filepath),
"line": self.line,
"column": self.column,
"code": self.code,
@@ -147,10 +137,7 @@ class LintViolation:
def __str__(self) -> str:
"""Format violation for display."""
return (
f"{report_path(self.filepath)}:{self.line}:{self.column} "
f"{self.code} {self.message}"
)
return f"{self.filepath}:{self.line}:{self.column} {self.code} {self.message}"
# =============================================================================
@@ -159,14 +146,14 @@ class LintViolation:
class BaseLinter(ABC):
"""Abstract base class for all custom linters.
This class provides common functionality for all linters including:
- Settings loading
- File collection
- Pattern matching
- Read-only zone and .gitignore support
- Consistent violation formatting
Subclasses must implement:
- lint_file() method
- VIOLATION_CODES class attribute
@@ -187,7 +174,7 @@ class BaseLinter(ABC):
def __init__(self, settings_file: pathlib.Path | None = None):
"""Initialize the linter.
Args:
settings_file: Path to the settings file, or None for defaults.
"""
@@ -198,7 +185,7 @@ class BaseLinter(ABC):
"""Load settings from the settings file."""
if self.settings_file is None:
self.settings_file = DEFAULT_SETTINGS_FILE
try:
with open(self.settings_file, encoding="utf-8") as f:
settings = json.load(f)
@@ -215,54 +202,54 @@ class BaseLinter(ABC):
@abstractmethod
def lint_file(self, filepath: pathlib.Path | str) -> list[LintViolation]:
"""Lint a single file for violations.
Args:
filepath: Path to the file to lint.
Returns:
List of LintViolation objects.
"""
def lint_directory(self, directory: str | pathlib.Path) -> list[LintViolation]:
"""Lint all files in a directory tree.
Args:
directory: Path to the directory to lint.
Returns:
List of LintViolation objects from all files.
"""
directory = pathlib.Path(directory)
violations: list[LintViolation] = []
if not directory.exists():
raise LinterError(f"Directory does not exist: {directory}")
# Collect files
files = self._collect_files(directory)
# Lint each file
for file in files:
file_violations = self.lint_file(file)
violations.extend(file_violations)
return violations
def _collect_files(self, directory: pathlib.Path) -> list[pathlib.Path]:
"""Collect all files to lint in a directory tree.
Args:
directory: Path to the directory to search.
Returns:
List of Path objects for files to lint.
"""
files: list[pathlib.Path] = []
# Respect .gitignore
gitignore_path = directory / ".gitignore"
ignore_patterns: list[str] = []
if gitignore_path.exists():
try:
with open(gitignore_path, encoding="utf-8") as f:
@@ -272,34 +259,34 @@ class BaseLinter(ABC):
ignore_patterns.append(line)
except OSError:
pass
# Walk the directory tree
for root, dirs, filenames in os.walk(directory):
root_path = pathlib.Path(root)
# Skip read-only zones
if any(zone in root_path.parts for zone in READONLY_ZONES):
dirs.clear()
continue
# Skip based on .gitignore patterns
dirs[:] = [d for d in dirs if not self._matches_pattern(d, ignore_patterns)]
for filename in filenames:
if any(filename.endswith(ext) for ext in self.file_extensions) and not self._matches_pattern(
filename, ignore_patterns
):
files.append(root_path / filename)
return files
def _matches_pattern(self, name: str, patterns: list[str]) -> bool:
"""Check if a name matches any of the ignore patterns.
Args:
name: The name to check.
patterns: List of patterns to match against.
Returns:
True if the name matches any pattern.
"""
@@ -313,21 +300,21 @@ class BaseLinter(ABC):
def _should_skip(self, filepath: pathlib.Path | str) -> bool:
"""Check if a file should be skipped based on settings.
Args:
filepath: Path to the file (as Path or string).
Returns:
True if the file should be skipped.
"""
# Convert to Path if it's a string
if isinstance(filepath, str):
filepath = pathlib.Path(filepath)
# Check read-only zones
if any(zone in filepath.parts for zone in READONLY_ZONES):
return True
# Check ignore patterns from settings
ignore_patterns = self._settings.get("ignore_patterns", [])
if ignore_patterns:
@@ -335,16 +322,16 @@ class BaseLinter(ABC):
for pattern in ignore_patterns:
if fnmatch.fnmatch(file_str, pattern):
return True
return False
def get_violation_message(self, violation_code: str, **kwargs) -> str:
"""Get formatted violation message from VIOLATION_CODES dictionary.
Args:
violation_code: The violation code (e.g., 'PYC001', 'FML001').
**kwargs: Keyword arguments to format the message template.
Returns:
Formatted violation message string.
"""
@@ -360,20 +347,20 @@ class BaseLinter(ABC):
def format_violations(self, violations: list[LintViolation]) -> str:
"""Format violations for display.
Args:
violations: List of LintViolation objects.
Returns:
Formatted string with all violations.
"""
if not violations:
return ""
lines = []
for violation in sorted(violations, key=lambda v: (str(v.filepath), v.line, v.column)):
lines.append(str(violation))
return "\n".join(lines)
def compress_violations(
@@ -410,18 +397,18 @@ class BaseLinter(ABC):
"total_violations": 0,
},
}
if target is not None:
compressed_violations["meta"]["target"] = report_path(target_path)
compressed_violations["meta"]["target"] = str(target_path)
for violation in violations:
if target_path is not None:
try:
file_key = report_path(violation.filepath.relative_to(target_path))
file_key = str(violation.filepath.relative_to(target_path))
except ValueError:
file_key = report_path(violation.filepath)
file_key = str(violation.filepath)
else:
file_key = report_path(violation.filepath)
file_key = str(violation.filepath)
if file_key not in compressed_violations["files"]:
compressed_violations["files"][file_key] = []
@@ -492,10 +479,10 @@ class BaseLinter(ABC):
def format_violations_json(self, violations: list[LintViolation]) -> str:
"""Format violations as JSON.
Args:
violations: List of LintViolation objects.
Returns:
JSON string with all violations.
"""
@@ -508,12 +495,12 @@ class BaseLinter(ABC):
linter_name: str,
) -> str:
"""Format violations in a comprehensive AI-friendly report.
Args:
violations: List of LintViolation objects.
target: The target directory or file that was linted.
linter_name: Name of the linter (for report header).
Returns:
Formatted report string.
"""
@@ -521,7 +508,7 @@ class BaseLinter(ABC):
lines.append(f"{linter_name} Violations")
lines.append(f"TARGET: {target}")
lines.append("")
# Collect all unique violation codes with their descriptions
descriptions: dict[str, dict] = {}
for violation in violations:
@@ -531,7 +518,7 @@ class BaseLinter(ABC):
"rule_name": code_info.get("rule_name", violation.code),
"description": code_info.get("description", ""),
}
# Add definitions
if descriptions:
lines.append("- DEFINITIONS:")
@@ -540,16 +527,16 @@ class BaseLinter(ABC):
f" - {code}: {info['rule_name']}, {info['description']}"
)
lines.append("")
# Add violations by file
if violations:
files_violations: dict[str, list[LintViolation]] = {}
for violation in violations:
file_key = report_path(violation.filepath)
file_key = str(violation.filepath)
if file_key not in files_violations:
files_violations[file_key] = []
files_violations[file_key].append(violation)
for file_path, file_viols in sorted(files_violations.items()):
lines.append(f"- FILE: {file_path}")
for violation in sorted(file_viols, key=lambda v: (v.line, v.column)):
@@ -558,11 +545,11 @@ class BaseLinter(ABC):
else:
lines.append("- No violations found.")
lines.append("")
# Add summary
lines.append("- SUMMARY:")
lines.append(f" - Total files with violations: {len(files_violations) if violations else 0}")
lines.append(f" - Total violations: {len(violations)}")
lines.append("")
return "\n".join(lines)
@@ -26,14 +26,13 @@ import pathlib
import re
import sys
from fiona.customlinter.base import (
from customlinter.base import (
BaseLinter,
FrontmatterLintError,
LintViolation,
report_path,
)
from fiona.librarian.md_frontmatter import FrontmatterError, parse_frontmatter
from fiona.logging_config import setup_logging
from librarian.md_frontmatter import FrontmatterError, parse_frontmatter
logger = logging.getLogger(__name__)
@@ -696,7 +695,7 @@ class FrontmatterLinter(BaseLinter):
# Group violations by file
files_violations: dict[str, list[LintViolation]] = {}
for violation in violations:
file_key = report_path(violation.filepath)
file_key = str(violation.filepath)
if file_key not in files_violations:
files_violations[file_key] = []
files_violations[file_key].append(violation)
@@ -9,7 +9,7 @@ Includes support for:
- Key ordering (optional)
- Auto-fixing certain violations
Follows the BaseLinter pattern from fiona.customlinter.base.
Follows the BaseLinter pattern from customlinter.base.
"""
from __future__ import annotations
@@ -21,7 +21,7 @@ import re
import sys
from typing import Any, ClassVar
from fiona.customlinter.base import BaseLinter, LinterError, LintViolation
from customlinter.base import BaseLinter, LinterError, LintViolation
from fiona.logging_config import setup_logging
logger = logging.getLogger(__name__)
@@ -3,7 +3,7 @@
This module provides a custom markdown linter for llm-wiki content,
built on the customlinter framework (BaseLinter). It runs a registry of
per-file checks over a wiki's markdown files and reports violations in
the shared AI-friendly format (SPEC 6.1).
the shared AI-friendly format (SPEC 4.9.3).
The first check is the dead-link check (moved from
`wiki.sanity_manager.find_dead_links`):
@@ -27,7 +27,7 @@ import pathlib
import re
from typing import Any, ClassVar
from fiona.customlinter.base import BaseLinter, LinterError, LintViolation
from customlinter.base import BaseLinter, LinterError, LintViolation
logger = logging.getLogger(__name__)
@@ -67,14 +67,6 @@ VIOLATION_CODES = {
},
}
__all__ = [
"FILE_EXTENSIONS",
"SETTINGS_KEY_PATH",
"VIOLATION_CODES",
"MarkdownLinter",
"find_dead_links",
]
def _is_ignorable_target(target: str) -> bool:
"""Check whether a markdown link target is out of scope."""
@@ -246,7 +238,7 @@ class MarkdownLinter(BaseLinter):
# =============================================================================
# Module-level Dead Link Helper (moved from fiona.wiki.sanity_manager)
# Module-level Dead Link Helper (moved from wiki.sanity_manager)
# =============================================================================
def find_dead_links(wiki_root: pathlib.Path) -> list[LintViolation]:
@@ -270,3 +262,16 @@ def find_dead_links(wiki_root: pathlib.Path) -> list[LintViolation]:
content = path.read_text(encoding="utf-8", errors="replace")
violations.extend(linter._check_dead_links(path, content, md_stems))
return violations
# =============================================================================
# Public API
# =============================================================================
__all__ = [
"FILE_EXTENSIONS",
"SETTINGS_KEY_PATH",
"VIOLATION_CODES",
"MarkdownLinter",
"find_dead_links",
]
@@ -11,7 +11,6 @@ The module provides functionality to:
Currently implemented rules:
- Inline imports (imports inside functions/blocks instead of module level)
- Late declarations (module-level assignments after the first function or class)
The framework is designed to be modular, allowing additional rules to be
added in the future.
@@ -26,7 +25,7 @@ import pathlib
import sys
from typing import ClassVar
from fiona.customlinter.base import (
from customlinter.base import (
BaseLinter,
CustomLintError,
LintViolation,
@@ -47,10 +46,7 @@ READONLY_ZONES = ["personal", "raw"]
VIOLATION_CODES = {
"PYC001": {
"rule_name": "Inline Import",
"description": (
"Import statement found inside a function or block (inline import). "
"A module-level if/else before the first function or class is allowed"
),
"description": "Import statement found inside function or block (inline import)",
"message_template": "PYC001 - Inline import at line {line}: {import_stmt}",
"severity": "error"
},
@@ -60,17 +56,6 @@ VIOLATION_CODES = {
"message_template": "PYC002 - Import after code at line {line}: {import_stmt}",
"severity": "warning"
},
"PYC003": {
"rule_name": "Late Declaration",
"description": (
"Module-level assignment found after the first function or class. "
"Put declarations at the top of the file, before the first function or class"
),
"message_template": (
"PYC003 - Declaration after first function or class at line {line}: {stmt}"
),
"severity": "error",
},
}
@@ -79,78 +64,26 @@ VIOLATION_CODES = {
# =============================================================================
class InlineImportVisitor(ast.NodeVisitor):
"""AST visitor to detect inline (indented) imports.
Modified by AI/LLM: 2026-09-29 - allow imports in a module-level
if/else that appears before the first function or class (SPEC 6.4.2).
"""
"""AST visitor to detect inline (indented) imports."""
def __init__(self, filename: str):
"""Initialize the visitor.
Args:
filename: The name of the file being analyzed.
"""
self.filename = filename
self.violations: list[LintViolation] = []
self._os_guard_depth = 0
self._guard_if_ids: set[int] = set()
def visit_Module(self, node: ast.Module) -> None:
"""Mark module-level if statements that sit before any function or class."""
first_boundary = None
for statement in node.body:
if isinstance(statement, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
first_boundary = statement.lineno
break
for statement in node.body:
if first_boundary is not None and statement.lineno >= first_boundary:
break
if isinstance(statement, ast.If):
self._guard_if_ids.add(id(statement))
self.generic_visit(node)
def visit_If(self, node: ast.If) -> None:
"""Keep the OS-guard depth raised while visiting an allowed if."""
guarded = id(node) in self._guard_if_ids
if guarded:
self._os_guard_depth += 1
self.generic_visit(node)
if guarded:
self._os_guard_depth -= 1
def visit_FunctionDef(self, node: ast.FunctionDef) -> None:
"""Imports inside a function stay inline, even under an OS if."""
self._visit_blocked_scope(node)
def visit_AsyncFunctionDef(self, node: ast.AsyncFunctionDef) -> None:
"""Imports inside an async function stay inline."""
self._visit_blocked_scope(node)
def visit_ClassDef(self, node: ast.ClassDef) -> None:
"""Imports inside a class stay inline."""
self._visit_blocked_scope(node)
def _visit_blocked_scope(self, node: ast.AST) -> None:
"""Visit a function or class with the OS-guard exception off.
Modified by AI/LLM: 2026-09-29 - the SPEC 6.4.2 exception does
not cover imports nested in a function or class.
"""
saved = self._os_guard_depth
self._os_guard_depth = 0
self.generic_visit(node)
self._os_guard_depth = saved
def visit_Import(self, node: ast.Import) -> None:
"""Visit import statement - check if indented."""
if node.col_offset > 0 and self._os_guard_depth == 0:
if node.col_offset > 0: # Indented = inline
self._add_inline_import_violation(node)
self.generic_visit(node)
def visit_ImportFrom(self, node: ast.ImportFrom) -> None:
"""Visit import from statement - check if indented."""
if node.col_offset > 0 and self._os_guard_depth == 0:
if node.col_offset > 0: # Indented = inline
self._add_inline_import_violation(node)
self.generic_visit(node)
@@ -166,7 +99,7 @@ class InlineImportVisitor(ast.NodeVisitor):
import_stmt = line.strip()
except (OSError, IndexError, ValueError):
pass
violation = LintViolation(
filepath=pathlib.Path(self.filename),
line=node.lineno,
@@ -182,110 +115,16 @@ class InlineImportVisitor(ast.NodeVisitor):
return self.violations
class LateDeclarationVisitor(ast.NodeVisitor):
"""AST visitor for module-level assignments after the first function or class.
Modified by AI/LLM: 2026-09-30 - PYC003 late declarations (SPEC 6.4.3).
"""
def __init__(self, filename: str) -> None:
"""Initialize the visitor.
Args:
filename: The name of the file being analyzed.
"""
self.filename = filename
self.violations: list[LintViolation] = []
self._boundary: int | None = None
self._scope_depth = 0
def visit_Module(self, node: ast.Module) -> None:
"""Record the first top-level function or class, then walk the module."""
for statement in node.body:
if isinstance(statement, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
self._boundary = statement.lineno
break
self.generic_visit(node)
def visit_FunctionDef(self, node: ast.FunctionDef) -> None:
"""Skip assignments that belong to a function."""
self._visit_nested_scope(node)
def visit_AsyncFunctionDef(self, node: ast.AsyncFunctionDef) -> None:
"""Skip assignments that belong to an async function."""
self._visit_nested_scope(node)
def visit_ClassDef(self, node: ast.ClassDef) -> None:
"""Skip assignments that belong to a class."""
self._visit_nested_scope(node)
def _visit_nested_scope(self, node: ast.AST) -> None:
"""Visit a function or class without treating its assignments as module-level.
Modified by AI/LLM: 2026-09-30 - class and function bodies are not
declarations in the middle of the module (SPEC 6.4.3).
"""
self._scope_depth += 1
self.generic_visit(node)
self._scope_depth -= 1
def visit_Assign(self, node: ast.Assign) -> None:
"""Flag a module-level assignment that follows the first function or class."""
self._check_declaration(node)
self.generic_visit(node)
def visit_AnnAssign(self, node: ast.AnnAssign) -> None:
"""Flag a module-level annotated assignment after the first function or class."""
self._check_declaration(node)
self.generic_visit(node)
def _check_declaration(self, node: ast.Assign | ast.AnnAssign) -> None:
"""Record PYC003 when this assignment is a late module declaration."""
if self._scope_depth > 0 or self._boundary is None:
return
if node.lineno < self._boundary:
return
statement = self._source_line(node.lineno)
self.violations.append(
LintViolation(
filepath=pathlib.Path(self.filename),
line=node.lineno,
column=node.col_offset,
code="PYC003",
message=(
"PYC003 - Declaration after first function or class "
f"at line {node.lineno}: {statement}"
),
rule_id="late-declaration",
)
)
def _source_line(self, lineno: int) -> str:
"""Return the stripped source line for a violation message."""
try:
with open(self.filename, encoding="utf-8") as handle:
lines = handle.readlines()
if lineno <= len(lines):
return lines[lineno - 1].strip()
except (OSError, IndexError, ValueError):
return "assignment"
return "assignment"
def get_violations(self) -> list[LintViolation]:
"""Get all detected violations."""
return self.violations
# =============================================================================
# Python Custom Linter Class
# =============================================================================
class PythonCustomLinter(BaseLinter):
"""Main linter class for custom Python code rules.
This class extends BaseLinter to provide Python-specific linting.
"""
VIOLATION_CODES: ClassVar[dict] = VIOLATION_CODES
file_extensions: ClassVar[list[str]] = [".py"]
settings_key_path: ClassVar[list[str]] = ["python", "custom_lint"]
@@ -293,72 +132,69 @@ class PythonCustomLinter(BaseLinter):
def lint_file(self, filepath: pathlib.Path | str) -> list[LintViolation]:
"""Lint a single Python file for custom rule violations.
Args:
filepath: Path to the Python file to lint (as Path or string).
Returns:
List of LintViolation objects.
"""
violations: list[LintViolation] = []
try:
# Convert to Path if it's a string
if isinstance(filepath, str):
filepath = pathlib.Path(filepath)
# Check if file should be skipped
if self._should_skip(filepath):
return violations
# Read and parse the file
source = filepath.read_text(encoding="utf-8")
tree = ast.parse(source, filename=str(filepath))
# Visit the AST to find inline imports and late declarations
# Visit the AST to find inline imports
visitor = InlineImportVisitor(str(filepath))
visitor.visit(tree)
violations = visitor.get_violations()
declarations = LateDeclarationVisitor(str(filepath))
declarations.visit(tree)
violations.extend(declarations.get_violations())
except SyntaxError as e:
# Syntax errors are not our concern
logger.warning("Syntax error in %s: %s", filepath, e)
except OSError as e:
logger.warning("Could not read %s: %s", filepath, e)
return violations
def lint_directory(self, directory: str | pathlib.Path) -> list[LintViolation]:
"""Lint all files in a directory tree.
This overrides the base class to raise CustomLintError instead of LinterError.
Args:
directory: Path to the directory to lint.
Returns:
List of LintViolation objects from all files.
Raises:
CustomLintError: If the directory does not exist.
"""
directory = pathlib.Path(directory)
violations: list[LintViolation] = []
if not directory.exists():
raise CustomLintError(f"Directory does not exist: {directory}")
# Collect files
files = self._collect_files(directory)
# Lint each file
for file in files:
file_violations = self.lint_file(file)
violations.extend(file_violations)
return violations
def _collect_python_files(self, directory: pathlib.Path) -> list[pathlib.Path]:
@@ -372,15 +208,15 @@ class PythonCustomLinter(BaseLinter):
def main():
"""Command-line interface for the Python custom linter."""
setup_logging()
logger.debug("Python custom linter starting")
parser = argparse.ArgumentParser(
description="Python Custom Linter - Check for custom Python code rule violations."
)
subparsers = parser.add_subparsers(dest="subcommand", help="Available subcommands")
# chklint subcommand
chklint_parser = subparsers.add_parser(
"chklint",
@@ -397,35 +233,35 @@ def main():
action="store_true",
help="Output results in JSON format"
)
args = parser.parse_args()
if args.subcommand is None:
# Default to chklint with current directory
args.subcommand = "chklint"
args.directory = "."
args.json = False # Default value when not specified
if args.subcommand == "chklint":
linter = PythonCustomLinter()
target = pathlib.Path(args.directory)
if not target.exists():
print(f"Error: '{args.directory}' does not exist.", file=sys.stderr)
sys.exit(1)
# Support both files and directories
if target.is_file():
violations = linter.lint_file(target)
else:
violations = linter.lint_directory(target)
if getattr(args, 'json', False):
output = linter.format_violations_json(violations)
else:
output = linter.format_violations(violations)
if output:
print(output)
sys.exit(1)
+13 -105
View File
@@ -8,21 +8,18 @@ import argparse
import logging
import sys
from fiona.customlinter.fmlint import main as fmlint_main
from fiona.customlinter.jsonlint import main as jsonlint_main
from fiona.gittool import diff_report
from fiona.gittool.git_tool import main as git_tool_main
from fiona.librarian.librarian import main as librarian_main
from fiona.librarian.toc_generator import main as toc_generator_main
from customlinter.fmlint import main as fmlint_main
from customlinter.jsonlint import main as jsonlint_main
from fiona.logging_config import setup_logging
from fiona.mdoctor.mdoctor import main as mdoctor_main
from fiona.python_ruff.python_ruff import main as python_ruff_main
from fiona.wiki.graph_generator import wiki_graph_main
from fiona.wiki.housekeeping import housekeeping_main
from fiona.wiki.ingest.pipeline import add_ingest_arguments, execute_ingest
from fiona.wiki.sanity_manager import sanity_main
from fiona.wiki.update_manager import check_update_main, update_main
from fiona.wiki.wiki import wiki_init
from gittool.git_tool import main as git_tool_main
from librarian.librarian import main as librarian_main
from librarian.toc_generator import main as toc_generator_main
from mdoctor.mdoctor import main as mdoctor_main
from python_ruff.python_ruff import main as python_ruff_main
from wiki.graph_generator import wiki_graph_main
from wiki.sanity_manager import sanity_main
from wiki.update_manager import check_update_main, update_main
from wiki.wiki import wiki_init
def main():
@@ -131,39 +128,6 @@ def main():
# fiona-assistant git status
git_subparsers.add_parser("status", help="Show git status")
# fiona-assistant git diff
git_diff_parser = git_subparsers.add_parser(
"diff",
help="Write constellation-themed HTML diff under wiki tmp/",
)
git_diff_parser.add_argument(
"--base",
help="Base revision (default: origin/main or main/master)",
)
git_diff_parser.add_argument("--head", default="HEAD", help="Head revision")
git_diff_parser.add_argument("--title", help="Report H1 / page title")
git_diff_parser.add_argument("--subtitle", help="Extra header line")
git_diff_parser.add_argument(
"-o",
"--output",
help="Output HTML path",
)
git_diff_parser.add_argument(
"--collapsed",
action="store_true",
help="Start with per-file sections collapsed",
)
git_diff_parser.add_argument(
"--staged",
action="store_true",
help="Diff only staged changes (ignores unstaged changes)",
)
git_diff_parser.add_argument(
"--unstaged",
action="store_true",
help="Diff only unstaged changes (ignores staged changes)",
)
# fiona-assistant wiki
wiki_parser = subparsers.add_parser("wiki", help="Manage wiki directories")
wiki_subparsers = wiki_parser.add_subparsers(dest="wiki_subcommand", help="Wiki subcommands")
@@ -315,31 +279,6 @@ def main():
help="Disable tracking of external links"
)
# AI/LLM: 2026-09-30 - flags live here so wiki ingest --help is not rejected.
wiki_ingest_parser = wiki_subparsers.add_parser(
"ingest",
prog="fiona-assistant wiki ingest",
help="Archive one source and write a raw wiki page",
)
add_ingest_arguments(wiki_ingest_parser)
# AI/LLM: fiona-assistant housekeeping (SPEC 4.8)
housekeeping_parser = subparsers.add_parser(
"housekeeping",
help="Run the wiki maintenance routine and report what needs work"
)
housekeeping_parser.add_argument(
"directory",
nargs="?",
default=".",
help="Wiki directory (default: current directory)"
)
housekeeping_parser.add_argument(
"--json",
action="store_true",
help="Output results in JSON format"
)
args = parser.parse_args()
setup_logging()
logging.getLogger(__name__).debug("fiona-assistant command: %s", args.command)
@@ -416,40 +355,13 @@ def main():
elif args.git_subcommand == "status":
sys.argv = ["git_tool", "status"]
git_tool_main()
elif args.git_subcommand == "diff":
diff_argv: list[str] = []
if args.base:
diff_argv.extend(["--base", args.base])
if args.head != "HEAD":
diff_argv.extend(["--head", args.head])
if args.title:
diff_argv.extend(["--title", args.title])
if args.subtitle:
diff_argv.extend(["--subtitle", args.subtitle])
if args.output is not None:
diff_argv.extend(["--output", args.output])
if args.collapsed:
diff_argv.append("--collapsed")
if args.staged:
diff_argv.append("--staged")
if args.unstaged:
diff_argv.append("--unstaged")
sys.exit(diff_report.main(diff_argv))
else:
git_parser.print_help()
sys.exit(1)
elif args.command == "wiki":
if args.wiki_subcommand == "init":
try:
result = wiki_init(args.directory)
print(result)
except FileExistsError as e:
# SPEC 7.1.1.4: format the error message and exit 1
# The FileExistsError message is: "Directory '<path>' already contains llm-wiki files: <names>"
# We want: "Error: Directory '<path>' already contains llm-wiki files: <names>. Run fiona-assistant wiki update to refresh the template."
formatted_error = f"Error: {e}. Run fiona-assistant wiki update to refresh the template."
print(formatted_error, file=sys.stderr)
sys.exit(1)
result = wiki_init(args.directory)
print(result)
elif args.wiki_subcommand == "check-update":
sys.exit(check_update_main(
directory=args.directory,
@@ -469,8 +381,6 @@ def main():
json_output=args.json,
non_interactive=args.non_interactive
))
elif args.wiki_subcommand == "ingest":
sys.exit(execute_ingest(args, wiki_ingest_parser))
elif args.wiki_subcommand == "graph":
sys.exit(wiki_graph_main(
directory=args.directory,
@@ -484,8 +394,6 @@ def main():
else:
wiki_parser.print_help()
sys.exit(1)
elif args.command == "housekeeping":
sys.exit(housekeeping_main(directory=args.directory, json_output=args.json))
else:
parser.print_help()
sys.exit(1)
-63
View File
@@ -1,63 +0,0 @@
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# Virtual environments
.venv/
.venvs/
venv/
env/
ENV/
# IDE-specific files
.vscode/
.idea/
*.swp
*.swo
# Local configuration
.env
*.local
# Build artifacts
build/
dist/
*.egg-info/
# Logs and databases
*.log
*.sqlite
*.db
# pytest
.pytest_cache/
# Coverage
.coverage
coverage.xml
htmlcov/
# Jupyter Notebook checkpoints
.ipynb_checkpoints/
# macOS
.DS_Store
# Windows
Thumbs.db
# Obsidian per-user vault state
.obsidian/workspace*.json
.obsidian/cache
# Python
*.pyc
*.pyo
*.pyd
*fiona.log
*wiki_graph.html
# Generated review artifacts (git diff HTML, scratch exports)
tmp/
-72
View File
@@ -1,72 +0,0 @@
---
name: llm-wiki-agent-rules
title: LLM Wiki Agent Rules
version: 1.4.0
created: 2025-09-23
updated: 2026-10-14
type: summary
---
# LLM Wiki Agent Rules
This directory is an llm-wiki: a persistent knowledge base of
interlinked markdown. The user brings sources and questions. You
compile each source once into wiki pages, keep the cross-references
current, and answer from those pages. The pattern is
[llm-wiki-mod](./utils/skills/fiona/llm-wiki-mod.md). The wiki's
domain, tags, and directory tree are in [schema](./wiki/schema.md).
This file states only what must be true before any skill is loaded.
Commands, lint codes, and the edit workflow live in the Fiona skill.
- [1. Load the Fiona skill](#1-load-the-fiona-skill)
- [2. What an llm-wiki is](#2-what-an-llm-wiki-is)
- [3. Boundaries](#3-boundaries)
- [4. History](#4-history)
## 1. Load the Fiona skill
If the Fiona skill is not already part of the system prompt, read
[Fiona skill](./utils/skills/fiona/SKILL.md) and follow it for the
rest of the session. It loads llm-wiki-mod.
When that skill is already loaded, do not read it again. This file
does not repeat it.
Then read [MEMORY](./MEMORY.md): what this wiki's owner wants, kept
across sessions.
Update `MEMORY.md` whenever an important memory comes up: a lasting
preference, a correction, a repeated request, or a fact a run
verified. Also update it whenever the user explicitly asks you to
remember something. Section 1 of that file says how.
## 2. What an llm-wiki is
A directory is an llm-wiki when it contains `fiona-settings.json`.
Page types, tags, and the directory tree are defined in
[schema](./wiki/schema.md). Do not invent a second map of the tree.
Install the tool with `python -m pip install fiona-assistant` when
the `fiona-assistant` command is missing. How to use it is the
Fiona skill, not this file.
## 3. Boundaries
These hold even before the skill is loaded:
- `personal/` is read-only. Never create, edit, or delete there.
- `raw/` is the archive. Add a new source there. Never edit, move,
or delete a file already there.
- Synthesized pages belong under `wiki/`.
- `ingest/` is only the drop zone for files waiting to be parsed.
## 4. History
Git is the wiki's history. After each ingest cycle and each
housekeeping pass (lint fixes, index or TOC updates, page moves),
commit with `fiona-assistant git save --description TEXT`.
`git clean`, `git undo`, and `git restore` discard work. Run them
only when the user asks.
-179
View File
@@ -1,179 +0,0 @@
---
name: memory
version: 1.0.1
title: Agent Memory
created: DATE_PLACEHOLDER
updated: DATE_PLACEHOLDER
type: summary
---
# Agent Memory
<!-- toc-start: headings 2..6 -->
- [1. How to use this file](#1-how-to-use-this-file)
- [1.1 When to read](#11-when-to-read)
- [1.2 When to write](#12-when-to-write)
- [1.3 How to write](#13-how-to-write)
- [1.4 How the memory evolves](#14-how-the-memory-evolves)
- [2. Pages and writing](#2-pages-and-writing)
- [2.1 Style and wording](#21-style-and-wording)
- [2.2 Placement and links](#22-placement-and-links)
- [3. Sources and ingest](#3-sources-and-ingest)
- [3.1 Hosts and access](#31-hosts-and-access)
- [3.2 Conversion and refinement](#32-conversion-and-refinement)
- [4. Tools and environment](#4-tools-and-environment)
- [4.1 Commands and flags](#41-commands-and-flags)
- [4.2 Machine and shell](#42-machine-and-shell)
- [5. History and housekeeping](#5-history-and-housekeeping)
- [5.1 Saves and maintenance](#51-saves-and-maintenance)
<!-- toc-end -->
What this wiki's owner wants from you, kept across sessions:
standing preferences, corrections, repeated request patterns, and
facts a run verified about operating this wiki. Knowledge from
sources does not go here; it goes to pages under `wiki/`.
## 1. How to use this file
### 1.1 When to read
- **1.1.1** Read this file once the
[Fiona skill](./utils/skills/fiona/SKILL.md) is loaded, before the
first wiki work of the session. When it is already in context, do
not read it again.
- **1.1.2** Apply every entry that matches the request.
- **1.1.2.1** An entry overrides a default of the Fiona skill or
the [schema](./wiki/schema.md), for this wiki only.
- **1.1.2.2** An entry never overrides the boundaries in section 3
of [AGENTS](./AGENTS.md); when the two conflict, follow the
boundary and ask the user.
- **1.1.2.3** An entry never changes a tool contract: a command, a
flag, or a required frontmatter field.
- **1.1.3** Every heading and list item is numbered as
[sectionize-markdown](./utils/skills/sectionize-markdown/SKILL.md)
describes. Cite an item as `MEMORY.md item 2.1.3`; a new item takes
the next free number in its subsection.
### 1.2 When to write
- **1.2.1** Write an entry when one of these happens:
- **1.2.1.1** The user states a requirement or preference that
outlasts the current request ("always", "never", "from now on",
"in this wiki", "we use").
- **1.2.1.2** The user corrects your output or approach: a page, a
summary, a placement, a link, or a command you ran.
- **1.2.1.3** The same kind of request comes for the second time;
record the pattern and what the user wanted.
- **1.2.1.4** A run verifies a reusable fact about operating this
wiki that the Fiona skill and the schema do not hold, such as a
host that needs a login, a converter that loses a site's tables,
or a flag combination that works.
- **1.2.1.5** The user explicitly asks you to remember something
("remember", "note this", "save to memory").
- **1.2.2** Do not write:
- **1.2.2.1** One-off values that only apply to the current
request.
- **1.2.2.2** Passwords, tokens, cookies, personal data, or
anything the user asks to keep out.
- **1.2.2.3** Knowledge from a source. That is a page under
`wiki/`, written by an ingest.
- **1.2.2.4** Facts that the Fiona skill, the schema, or a wiki
page already states; cite them instead.
- **1.2.2.5** Guesses; only what the user said or a run showed.
### 1.3 How to write
- **1.3.1** Put the entry under the `###` subsection that fits.
- **1.3.1.1** When none fits, add a `###` subsection with the next
free number under the matching `##`, or a new `##` category with
the next free number.
- **1.3.1.2** Never put an entry directly under a `##`; its id
would collide with a later subsection.
- **1.3.2** Use the knowledge-entry shape of sectionize-markdown
section 4: a header line with the source tag and a short title,
one child per fact, and a last child with the ref and the date.
- **1.3.3** Source tags: `user` (the user said it), `run` (a command
or tool run showed it), `source` (a file under `raw/` states it),
joined with `+` when several agree.
- **1.3.4** The ref says where to check the fact: the user's request
and its date, the command and its output line, or a page or raw
path.
- **1.3.5** Write the entry in the same turn, then tell the user
which item was added or changed, for example "Saved to MEMORY.md
item 2.1.1".
- **1.3.6** Bump `version` (patch for a changed entry, minor for a
new entry or subsection) and set `updated` in the frontmatter on
every change.
- **1.3.7** Append `- YYYY-MM-DD - memory: <item id> <title>` to
`wiki/log.md`, and bump that file too.
- **1.3.8** The change lands in the save of the current ingest cycle
or housekeeping pass (Fiona skill section 2.3).
- **1.3.8.1** With no cycle running, save it on its own with
`fiona-assistant git save --description "memory: <item id> <title>"`.
An entry looks like this:
```text
- **2.1.1** [user] "Source summaries open with the decision"
- **2.1.1.1** A source page starts with the decision the source records, then the reasons.
- **2.1.1.2** Applies to meeting notes and design reviews.
- **2.1.1.3** Ref: user request YYYY-MM-DD; added YYYY-MM-DD.
```
### 1.4 How the memory evolves
- **1.4.1** Before adding an entry, look for a near-duplicate in the
same category; extend it instead and keep its id and its earliest
date.
- **1.4.2** A finding that contradicts an entry becomes its next
child, titled `OVERRIDE: ...`, with the new facts and its own ref.
- **1.4.3** An entry that no longer applies is marked
**Superseded YYYY-MM-DD** with the reason; it is not deleted.
- **1.4.4** An entry that held for several uses graduates; mark it
**Promoted YYYY-MM-DD** with the target.
- **1.4.4.1** A convention for every page moves to the
[schema](./wiki/schema.md).
- **1.4.4.2** A reusable procedure moves to a skill under
[utils/skills_local](./utils/skills_local/README.md).
- **1.4.5** Never renumber ids; pages, log lines, and chats may cite
them.
- **1.4.6** This file belongs to this wiki and evolves with it.
- **1.4.6.1** `fiona-assistant wiki update` adds it only when it is
missing. It never compares or replaces it, and never writes a
`MEMORY-update.md`.
- **1.4.6.2** A newer template version of these rules does not
reach an existing wiki; the owner decides what to adopt.
- **1.4.7** Git is the memory's history: `git log -p MEMORY.md` shows
when each entry changed and why.
## 2. Pages and writing
How pages read, where they go, and how they link.
### 2.1 Style and wording
### 2.2 Placement and links
## 3. Sources and ingest
Where sources come from and how they become pages.
### 3.1 Hosts and access
### 3.2 Conversion and refinement
## 4. Tools and environment
How `fiona-assistant` and the machine behave for this wiki.
### 4.1 Commands and flags
### 4.2 Machine and shell
## 5. History and housekeeping
Saves, commits, merges, and maintenance passes.
### 5.1 Saves and maintenance
@@ -1,59 +0,0 @@
{
"version": "1.1.0",
"background": "#f4efe6",
"graph_background": "#fffdf8",
"font_family": "Arial, sans-serif",
"text_color": "#2c2416",
"node_colors": {
"low": "#8f8272",
"mid": "#e0943a",
"high": "#c23b4a"
},
"node_stroke": "none",
"node_radius": {
"min": 5,
"max": 40
},
"edge_colors": {
"low": "#a89884",
"mid": "#b86a1a",
"high": "#8e2436"
},
"edge_width": 1,
"highlight": {
"node_color": "#2aa7a1",
"edge_color": "#0d6e74",
"size_multiplier": 1.3
},
"clicked": {
"node_color": "#0e6b66"
},
"orphan": {
"enabled": false,
"node_color": "#7b5ea7"
},
"buttons": {
"background": "#0f6e78",
"text": "#fffdf8",
"hover": "#0b555c",
"disabled": "#b7aea3"
},
"card": {
"background": "#fffdf8",
"border": "#e0d3c2"
},
"shapes": {
"page": "circle",
"external": "cloud"
},
"subtitle_color": "#2c2416",
"subtitle": {
"min_radius": 0.8,
"padding": 2,
"size": 6,
"on_top": true,
"outline": false,
"outline_color": "#fffdf8",
"outline_width": 3
}
}
@@ -1,59 +0,0 @@
{
"version": "1.1.0",
"background": "#1e2836",
"graph_background": "#283448",
"font_family": "Segoe UI, sans-serif",
"text_color": "#e7eef8",
"node_colors": {
"low": "#7d8da3",
"mid": "#fff4c4",
"high": "#f7d67c"
},
"node_stroke": "none",
"node_radius": {
"min": 4,
"max": 24
},
"edge_colors": {
"low": "#51627a",
"mid": "#8fa3bd",
"high": "#d5e2f2"
},
"edge_width": 1,
"highlight": {
"node_color": "#f3d48a",
"edge_color": "#9ec5ea",
"size_multiplier": 1.45
},
"clicked": {
"node_color": "#ffe3a3"
},
"orphan": {
"enabled": false,
"node_color": "#ef8a7a"
},
"buttons": {
"background": "#3e5674",
"text": "#f4f8fc",
"hover": "#527094",
"disabled": "#2a3648"
},
"card": {
"background": "#31445c",
"border": "#5c7390"
},
"shapes": {
"page": "circle",
"external": "cloud"
},
"subtitle_color": "#e7eef8",
"subtitle": {
"min_radius": 0.8,
"padding": 2,
"size": 6,
"on_top": true,
"outline": false,
"outline_color": "#283448",
"outline_width": 3
}
}
@@ -1,113 +0,0 @@
{
"version": "1.1.0",
"background": "#e6e6e6",
"graph_background": "#f4f4f4",
"font_family": "Arial, sans-serif",
"text_color": "#141414",
"node_colors": {
"low": "#5a5a5a",
"mid": "#2e2e2e",
"high": "#0d0d0d"
},
"node_stroke": "#141414",
"node_radius": {
"min": 5,
"max": 36
},
"edge_colors": {
"low": "#8f8f8f",
"mid": "#3d3d3d",
"high": "#141414"
},
"edge_width": 1,
"highlight": {
"node_color": "#000000",
"edge_color": "#1a1a1a",
"size_multiplier": 1.4
},
"clicked": {
"node_color": "#000000"
},
"orphan": {
"enabled": false,
"node_color": "#bdbdbd"
},
"buttons": {
"background": "#2a2a2a",
"text": "#ffffff",
"hover": "#111111",
"disabled": "#b0b0b0"
},
"card": {
"background": "#ffffff",
"border": "#8a8a8a"
},
"shapes": {
"page": "circle",
"external": "cloud"
},
"subtitle_color": "#141414",
"subtitle": {
"min_radius": 0.8,
"padding": 2,
"size": 6,
"on_top": true,
"outline": false,
"outline_color": "#f4f4f4",
"outline_width": 3
},
"dark": {
"background": "#2c2c2c",
"graph_background": "#3a3a3a",
"font_family": "Arial, sans-serif",
"text_color": "#f5f5f5",
"node_colors": {
"low": "#8d8d8d",
"mid": "#fff4c4",
"high": "#f5b44a"
},
"node_stroke": "none",
"node_radius": {
"min": 5,
"max": 36
},
"edge_colors": {
"low": "#6e6e6e",
"mid": "#c4c4c4",
"high": "#ffffff"
},
"edge_width": 1,
"highlight": {
"node_color": "#ffffff",
"edge_color": "#ffffff",
"size_multiplier": 1.4
},
"clicked": {
"node_color": "#ffffff"
},
"buttons": {
"background": "#e8e8e8",
"text": "#1a1a1a",
"hover": "#ffffff",
"disabled": "#5a5a5a"
},
"card": {
"background": "#454545",
"border": "#9a9a9a"
},
"shapes": {
"page": "circle",
"external": "cloud"
},
"subtitle_color": "#f5f5f5",
"subtitle": {
"min_radius": 0.5,
"padding": 2,
"size": 8,
"on_top": true,
"outline": true,
"outline_color": "#2c2c2c",
"outline_width": 3
}
}
}
@@ -1,59 +0,0 @@
{
"version": "1.1.0",
"background": "#070b14",
"graph_background": "#0b1020",
"font_family": "Segoe UI, sans-serif",
"text_color": "#d6def5",
"node_colors": {
"low": "#8ea2c8",
"mid": "#ffe9a0",
"high": "#ffffff"
},
"node_stroke": "none",
"node_radius": {
"min": 3,
"max": 20
},
"edge_colors": {
"low": "#243056",
"mid": "#7f8cbf",
"high": "#e7ecff"
},
"edge_width": 1,
"highlight": {
"node_color": "#fff6c2",
"edge_color": "#9ecbff",
"size_multiplier": 1.6
},
"clicked": {
"node_color": "#ffd56a"
},
"orphan": {
"enabled": false,
"node_color": "#ff7f6e"
},
"buttons": {
"background": "#1e2a4a",
"text": "#e8eeff",
"hover": "#314572",
"disabled": "#3a4258"
},
"card": {
"background": "#12182b",
"border": "#2a3558"
},
"shapes": {
"page": "circle",
"external": "star"
},
"subtitle_color": "#d5def8",
"subtitle": {
"min_radius": 0.8,
"padding": 2,
"size": 6,
"on_top": true,
"outline": false,
"outline_color": "#0b1020",
"outline_width": 3
}
}
@@ -1,59 +0,0 @@
{
"version": "1.1.0",
"background": "#f7fafc",
"graph_background": "#fcfdfe",
"font_family": "Segoe UI, Calibri, Helvetica Neue, Arial, sans-serif",
"text_color": "#4a5d70",
"node_colors": {
"low": "#e4eef4",
"mid": "#a9c6d8",
"high": "#7eabc4"
},
"node_stroke": "#9bb8cb",
"node_radius": {
"min": 4,
"max": 26
},
"edge_colors": {
"low": "#e7f0f5",
"mid": "#b7d0de",
"high": "#8eb4c9"
},
"edge_width": 1,
"highlight": {
"node_color": "#6aa3c0",
"edge_color": "#7eabc4",
"size_multiplier": 1.15
},
"clicked": {
"node_color": "#5b93b0"
},
"orphan": {
"enabled": false,
"node_color": "#d9a5b3"
},
"buttons": {
"background": "#8eb4c9",
"text": "#ffffff",
"hover": "#7aa6bd",
"disabled": "#dce7ee"
},
"card": {
"background": "#ffffff",
"border": "#e6eef3"
},
"shapes": {
"page": "circle",
"external": "cloud"
},
"subtitle_color": "#4a5d70",
"subtitle": {
"min_radius": 0.8,
"padding": 2,
"size": 6,
"on_top": true,
"outline": false,
"outline_color": "#fcfdfe",
"outline_width": 3
}
}
@@ -1,19 +0,0 @@
---
name: map
version: 1.1.0
title: Script Map
type: summary
created: 2025-09-23
updated: 2026-09-30
---
# Script Map
Catalog of the reusable scripts in this folder. Read it before
writing a new script, and add a row when you add one. The rules for
writing a script are in the
[Fiona skill](../skills/fiona/SKILL.md#write-reusable-scripts).
| Script | Purpose | Usage | Added |
| --- | --- | --- | --- |
| `example.py` | Placeholder that prints a greeting. | `python utils/scripts/example.py` | 2025-09-23 |
@@ -1,18 +0,0 @@
---
name: skills
version: 1.1.1
title: Skills Directory
type: summary
created: 2025-09-23
updated: 2026-10-21
---
# Skills Directory
Skill definitions and configurations.
- [fiona](./fiona/SKILL.md): operate the llm-wiki with `fiona-assistant`.
- [memory](./memory/SKILL.md): give a skill its own `MEMORIES.md`.
- [python-tdd](./python-tdd/SKILL.md): test-driven Python work.
- [sectionize-markdown](./sectionize-markdown/SKILL.md):
number headings and list items so each rule can be cited.
File diff suppressed because it is too large Load Diff
@@ -1,144 +0,0 @@
---
name: llm-wiki-mod
title: LLM Wiki Pattern
version: 1.4.1
updated: 2026-10-06
type: concept
tags: [wiki, schema]
---
# LLM Wiki
A persistent knowledge base of interlinked markdown. The pattern
comes from
[Andrej Karpathy's LLM Wiki](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f).
Page rules are in [schema](../../../wiki/schema.md). Commands and
the edit loop are in the [Fiona skill](./SKILL.md).
- [1. Core idea](#1-core-idea)
- [2. Layers](#2-layers)
- [3. Operations](#3-operations)
- [4. Index and log](#4-index-and-log)
- [5. Page rules](#5-page-rules)
- [6. Why this works](#6-why-this-works)
- [7. Scope](#7-scope)
## 1. Core idea
Sources are read once and compiled into wiki pages. Later questions
use those pages. Cross-references, contradictions, and synthesis
stay in the files. Each new source and each filed answer adds to
that store.
The human curates sources and asks the questions. The agent writes
the pages, updates cross-references, and keeps the bookkeeping.
## 2. Layers
Three layers. The directory tree in
[schema](../../../wiki/schema.md) is the map.
### 2.1. Raw sources
`raw/` is the archive. The agent adds new sources there and never
edits, moves, or deletes one already archived.
`ingest/` is the drop zone for files that still need parsing.
`personal/` holds user notes and is read-only, including the
directory itself.
### 2.2. The wiki
`wiki/` is agent-owned markdown. Synthesized pages go in
`concepts/`, `entities/`, `comparisons/`, and `queries/`. Source
summaries that back those pages go in `sources/`. The agent creates
pages, revises them when a new source arrives, and keeps links
consistent.
### 2.3. The schema
[schema](../../../wiki/schema.md) states file names, frontmatter,
tags, when to create a page, and the directory tree.
[AGENTS.md](../../../AGENTS.md) is the entry point. It loads the
[Fiona skill](./SKILL.md) for commands and the edit loop. When a
convention changes, change the schema. Leave the tree in that one
file.
## 3. Operations
### 3.1. Ingest
Put a new source in `ingest/`. The agent reads it, writes the
summary under `wiki/`, updates [index](../../../wiki/index.md),
updates the entity and concept pages the source touches, and
appends one dated line to [log](../../../wiki/log.md). One source
can touch many pages. Take sources one at a time when the summary
should be checked before the next file.
After the archive step, the source lives under `raw/` and is not
edited again.
An ingest has two phases. A tool fetches the source, archives it
under `raw/`, and converts it into a page marked `status: raw`.
The agent then refines that page, updates the pages it touches,
and marks it `status: refined`. The steps are section 18 of the
[Fiona skill](./SKILL.md).
### 3.2. Query
Ask against the wiki. The agent reads the index, opens the pages
that apply, and answers with citations. File a comparison, an
analysis, or a connection that should outlive the chat under
`wiki/queries/` or `wiki/comparisons/`.
### 3.3. Lint
Check the wiki for contradictions, claims a newer source has
replaced, pages with no inbound links, concepts that need their
own page, missing cross-references, and gaps a source could fill.
The health command is `fiona-assistant wiki sanity`. Style,
frontmatter, and link checks are in the [Fiona skill](./SKILL.md).
## 4. Index and log
[index](../../../wiki/index.md) is the catalog. Each page has a
link and a one-line summary, grouped by section. Update it when
a page is added. Read it first when answering a query.
[log](../../../wiki/log.md) is append-only. Each entry is one
dated bullet:
```text
- YYYY-MM-DD - what happened
```
Bump `version` and `updated` on `log.md` when appending. The log
is the timeline of ingests, queries, and lint passes.
Git holds the before and after of each entry. Every ingest cycle
and every housekeeping pass ends in one commit, so one source or
one cleanup can be reviewed or undone on its own.
## 5. Page rules
File names, frontmatter, tags, links, and when to create or split
a page are in [schema](../../../wiki/schema.md). How to fill the
frontmatter and the quality signals is section 15 of the
[Fiona skill](./SKILL.md). This file does not repeat them.
## 6. Why this works
Cross-references, summaries that still match the sources, and notes
where a new source disagrees with an older claim are the work that
usually gets dropped. The agent does that work on each ingest, so
the wiki stays current as it grows.
The human chooses sources, directs the analysis, and decides what
the pages mean. The agent does the filing.
## 7. Scope
This file states the pattern. The directory tree stays in
[schema](../../../wiki/schema.md). Commands stay in the
[Fiona skill](./SKILL.md). An Obsidian graph, Marp slides,
Dataview, or a local search engine can sit beside the wiki. This
wiki does not require them.
@@ -1,173 +0,0 @@
---
name: skill-memories
version: 1.0.1
title: SKILL_NAME Memories
created: DATE_PLACEHOLDER
updated: DATE_PLACEHOLDER
type: summary
---
# SKILL_NAME Memories
<!-- toc-start: headings 2..6 -->
- [1. How to use this file](#1-how-to-use-this-file)
- [1.1 When to read](#11-when-to-read)
- [1.2 When to write](#12-when-to-write)
- [1.3 How to write](#13-how-to-write)
- [1.4 How the memory evolves](#14-how-the-memory-evolves)
- [2. Output and wording](#2-output-and-wording)
- [2.1 Style and format](#21-style-and-format)
- [2.2 Placement and naming](#22-placement-and-naming)
- [3. Inputs and sources](#3-inputs-and-sources)
- [3.1 Where inputs come from](#31-where-inputs-come-from)
- [3.2 Quirks and parsing](#32-quirks-and-parsing)
- [4. Steps and tools](#4-steps-and-tools)
- [4.1 Commands and flags](#41-commands-and-flags)
- [4.2 Machine and shell](#42-machine-and-shell)
- [5. Recurring requests](#5-recurring-requests)
- [5.1 Patterns](#51-patterns)
<!-- toc-end -->
What users want from the SKILL_NAME skill, kept across sessions:
standing preferences, corrections, repeated request patterns, and
facts a run verified while using this skill. Preferences for the
whole wiki go to the wiki's `MEMORY.md`. Knowledge from sources goes
to pages under `wiki/`.
## 1. How to use this file
### 1.1 When to read
- **1.1.1** Read this file once [the skill](./SKILL.md) is loaded,
before its first step of the session. When it is already in
context, do not read it again.
- **1.1.2** Apply every entry that matches the request.
- **1.1.2.1** An entry overrides a default of the skill and an
entry of the wiki's `MEMORY.md`, for work done with this skill
only.
- **1.1.2.2** An entry never overrides the boundaries in the wiki's
`AGENTS.md`; when the two conflict, follow the boundary and ask
the user.
- **1.1.2.3** An entry never changes a tool contract: a command, a
flag, or a required frontmatter field.
- **1.1.3** Every heading and list item is numbered as
[sectionize-markdown](../../skills/sectionize-markdown/SKILL.md)
describes. Cite an item as `SKILL_NAME MEMORIES.md item 2.1.3`; a
new item takes the next free number in its subsection.
### 1.2 When to write
- **1.2.1** Write an entry when one of these happens while the skill
is in use:
- **1.2.1.1** The user states a requirement or preference for this
skill that outlasts the current request ("always", "never", "from
now on", "when you do X").
- **1.2.1.2** The user corrects the skill's output or a step it
took.
- **1.2.1.3** The same kind of request comes for the second time;
record the pattern and what the user wanted.
- **1.2.1.4** A run verifies a reusable fact about the skill's
inputs, hosts, or tools that the skill does not state.
- **1.2.1.5** The user explicitly asks you to remember something
for this skill.
- **1.2.2** Do not write:
- **1.2.2.1** One-off values that only apply to the current
request.
- **1.2.2.2** Passwords, tokens, cookies, personal data, or
anything the user asks to keep out.
- **1.2.2.3** Preferences that hold for any wiki work; those go to
the wiki's `MEMORY.md`.
- **1.2.2.4** Facts the skill or a wiki page already states; cite
them instead.
- **1.2.2.5** Guesses; only what the user said or a run showed.
### 1.3 How to write
- **1.3.1** Put the entry under the `###` subsection that fits.
- **1.3.1.1** When none fits, add a `###` subsection with the next
free number under the matching `##`, or a new `##` category with
the next free number.
- **1.3.1.2** Never put an entry directly under a `##`; its id
would collide with a later subsection.
- **1.3.2** Use the knowledge-entry shape of sectionize-markdown
section 4: a header line with the source tag and a short title,
one child per fact, and a last child with the ref and the date.
- **1.3.3** Source tags: `user` (the user said it), `run` (a command
or tool run showed it), `source` (a file under `raw/` states it),
joined with `+` when several agree.
- **1.3.4** The ref says where to check the fact: the user's request
and its date, the command and its output line, or a page or raw
path.
- **1.3.5** Write the entry in the same turn, then tell the user
which item was added or changed, for example "Saved to SKILL_NAME
MEMORIES.md item 2.1.1".
- **1.3.6** Bump `version` (patch for a changed entry, minor for a
new entry or subsection) and set `updated` in the frontmatter on
every change.
- **1.3.7** Append `- YYYY-MM-DD - SKILL_NAME memory: <item id> <title>`
to `wiki/log.md`, and bump that file too.
- **1.3.8** The change lands in the save of the current ingest cycle
or housekeeping pass.
- **1.3.8.1** With no cycle running, save it on its own with
`fiona-assistant git save --description "SKILL_NAME memory: <item id> <title>"`.
An entry looks like this:
```text
- **2.1.1** [user] "Reports open with the verdict"
- **2.1.1.1** The report starts with pass or fail, then the evidence.
- **2.1.1.2** Applies to every run of this skill.
- **2.1.1.3** Ref: user request YYYY-MM-DD; added YYYY-MM-DD.
```
### 1.4 How the memory evolves
- **1.4.1** Before adding an entry, look for a near-duplicate in the
same category; extend it instead and keep its id and its earliest
date.
- **1.4.2** A finding that contradicts an entry becomes its next
child, titled `OVERRIDE: ...`, with the new facts and its own ref.
- **1.4.3** An entry that no longer applies is marked
**Superseded YYYY-MM-DD** with the reason; it is not deleted.
- **1.4.4** An entry that held for several uses graduates; mark it
**Promoted YYYY-MM-DD** with the target.
- **1.4.4.1** A rule for every run of the skill moves into
[the skill](./SKILL.md).
- **1.4.4.2** A preference that turns out to hold for all wiki
work moves to the wiki's `MEMORY.md`.
- **1.4.5** Never renumber ids; pages, log lines, and chats may cite
them.
- **1.4.6** Git is the memory's history: `git log -p MEMORIES.md` in
the skill's folder shows when each entry changed and why.
## 2. Output and wording
What the skill produces and how it reads.
### 2.1 Style and format
### 2.2 Placement and naming
## 3. Inputs and sources
What the skill reads and where it comes from.
### 3.1 Where inputs come from
### 3.2 Quirks and parsing
## 4. Steps and tools
How the skill's commands and the machine behave.
### 4.1 Commands and flags
### 4.2 Machine and shell
## 5. Recurring requests
Requests that come back, and what the user wanted each time.
### 5.1 Patterns
@@ -1,109 +0,0 @@
---
name: memory
category: productivity
description: Give a skill its own memory, a MEMORIES.md beside its SKILL.md.
author: nvaldez
version: 1.0.1
created: 2026-10-15
updated: 2026-10-21
---
# Skill Memory
<!-- toc-start: headings 2..6 -->
- [1. What a skill memory is](#1-what-a-skill-memory-is)
- [2. Give a skill memory](#2-give-a-skill-memory)
- [3. Wire it into the skill](#3-wire-it-into-the-skill)
- [4. Use a skill memory](#4-use-a-skill-memory)
<!-- toc-end -->
A skill with memory keeps what its users want from that skill across
sessions, in a `MEMORIES.md` file beside its `SKILL.md`. The blank
file to copy is [MEMORIES](./MEMORIES.md). The wiki-wide memory is
[MEMORY](../../../MEMORY.md); this skill does not replace it.
Every heading and list item is numbered as
[sectionize-markdown](../sectionize-markdown/SKILL.md)
describes. Cite a rule as `memory skill item 2.3`.
## 1. What a skill memory is
- **1.1** One file per skill: `MEMORIES.md` in the skill's folder,
next to its `SKILL.md`.
- **1.2** It holds facts about operating that one skill:
- **1.2.1** Preferences for the skill's output.
- **1.2.2** Corrections to the skill's steps or results.
- **1.2.3** Request patterns that came back for that skill.
- **1.2.4** Facts a run verified about the skill's inputs, hosts,
and tools.
- **1.3** It does not hold:
- **1.3.1** Knowledge from sources; that is a page under `wiki/`.
- **1.3.2** Preferences for the whole wiki; those go to
[MEMORY](../../../MEMORY.md).
- **1.3.3** Passwords, tokens, cookies, or personal data.
- **1.4** Precedence, from strongest to weakest:
- **1.4.1** The boundaries in [AGENTS](../../../AGENTS.md) and tool
contracts (commands, flags, required frontmatter).
- **1.4.2** The skill's `MEMORIES.md`, for work done with that skill.
- **1.4.3** The wiki's `MEMORY.md`.
- **1.4.4** The defaults in the skill's `SKILL.md`.
## 2. Give a skill memory
- **2.1** Give a skill memory when the user asks for a skill with
memory, or when the same correction to one skill comes a second
time.
- **2.2** A new skill lives under `utils/skills_local/<skill>/`. A
skill shipped under `utils/skills/` can also get a memory; the
template never ships a `MEMORIES.md` for it, so an update leaves the
file alone.
- **2.3** Copy [MEMORIES](./MEMORIES.md) to
`<skill-folder>/MEMORIES.md`. Never edit the template copy in this
folder.
- **2.4** In the copy:
- **2.4.1** Set frontmatter `name` to `<skill>-memories`, `version`
to `1.0.0`, and `created` and `updated` to today.
- **2.4.2** Replace every `SKILL_NAME` with the skill's name.
- **2.4.3** Rename the empty categories in sections 2 to 5 to fit
the skill, or keep them. Leave section 1 as it is.
- **2.5** Add the memory section of section 3 to the skill's
`SKILL.md` and bump that file's `version`.
- **2.6** Append `- YYYY-MM-DD - memory: <skill> gets MEMORIES.md` to
`wiki/log.md`, then save with
`fiona-assistant git save --description "memory: <skill> gets MEMORIES.md"`.
## 3. Wire it into the skill
- **3.1** The skill's `SKILL.md` gets this section as its last
numbered section, with `N` set to that section's number:
```markdown
## N. Memory
- **N.1** After loading this skill, read [MEMORIES](./MEMORIES.md)
once per session. When it is already in context, do not read it
again.
- **N.2** Update it whenever an important memory about this skill
comes up: a lasting preference, a correction, a repeated request,
or a fact a run verified. Also update it whenever the user
explicitly asks you to remember something for this skill.
Section 1 of that file says how.
```
- **3.2** Keep the link relative (`./MEMORIES.md`) so the skill still
finds its memory after a move.
## 4. Use a skill memory
- **4.1** Section 1 of each skill's `MEMORIES.md` is the rule for
reading and writing it. This file does not repeat it.
- **4.2** Pick the narrowest memory that covers the fact:
- **4.2.1** It only matters while one skill runs: that skill's
`MEMORIES.md`.
- **4.2.2** It matters for any wiki work: `MEMORY.md`.
- **4.2.3** When unsure, ask the user which one.
- **4.3** When a request uses a skill that has no memory and a
memory comes up, write it to `MEMORY.md` and suggest giving the
skill one (section 2).
@@ -1,159 +0,0 @@
---
name: sectionize-markdown
version: 1.1.3
title: How to sectionize a markdown file
description: Number every heading and list item of a markdown file so each
rule, step, or fact has a stable id to cite.
created: '2026-10-14'
updated: '2026-10-21'
confidence: high
llm:
- model: Claude Opus 5.5
---
# How to sectionize a markdown file
<!-- toc-start: headings 2..6 -->
- [1. Headings](#1-headings)
- [2. List item ids](#2-list-item-ids)
- [3. Splitting long lines](#3-splitting-long-lines)
- [4. Knowledge entries](#4-knowledge-entries)
- [5. Organize by category](#5-organize-by-category)
- [6. Doing it safely](#6-doing-it-safely)
- [7. Example](#7-example)
<!-- toc-end -->
Numbers every heading and every list item of a markdown file so each
rule, step, or fact has a stable id that can be cited ("item
4.2.3"), tracked, and checked for duplicates. It suits rule files,
skill instructions, glossaries, and knowledge caches.
## 1. Headings
- **1.1** Number every heading below the title: `## N.` for sections,
`### N.M` for subsections, `#### N.M.K` below that.
- **1.2** The `#` title and the table of contents are not numbered.
- **1.3** Put a TOC of markdown links right under the title and intro.
Regenerate it whenever headings change.
- **1.4** Anchors are GitHub-style slugs: `## 5.6 Download the logs`
becomes `#56-download-the-logs`.
- **1.5** Avoid em dashes and other punctuation in headings; use a colon
(`10.7 Outcome: a match exists`). Punctuation makes the slug
unpredictable and breaks TOC links (MD051).
- **1.6** A label that only introduces a list ("Files in this
folder:") is a numbered list item, not a heading.
## 2. List item ids
- **2.1** Every list item gets a bold id right after the dash:
`- **4.2.1** Text`. Bold is two asterisks on each side; a single
asterisk is italic.
- **2.2** The id is the section number plus a running counter:
`**<section>.<n>**`. Nested items extend the parent id:
`**<section>.<n>.<m>**`, indented two spaces.
- **2.3** The counter runs through the whole section, across separate
lists split by paragraphs or code blocks. Two lists under `### 12.3`
give `**12.3.1**`…`**12.3.3**`, then `**12.3.4**`…, never a second
`**12.3.1**`.
- **2.4** Checklists keep the box first: `- [ ] **13.1** Verify …`.
- **2.5** Do not number the TOC, code blocks, or table rows.
- **2.6** State the convention in the file's first section so the next
editor follows it ("Cite an item as `item 4.2.3`").
- **2.7** Never renumber existing ids; other notes may cite them. A new
item takes the next free number in its section, even if that puts it
out of reading order. Fix the order only while the ids are still
unreleased (uncommitted).
## 3. Splitting long lines
- **3.1** A list item holding several facts becomes a short header item
plus one child per fact.
- **3.2** One child is one sentence, or two closely linked sentences.
- **3.3** Split at sentence ends (`.`, `?` or `!`, then a space and an
uppercase letter, backtick, `*`, `"` or `[`). Do not split inside
backticks or parentheses, or after `e.g.`, `i.e.`, `vs.`, `cf.`.
- **3.4** Merge fragments shorter than about 45 characters back into the
previous child; they read better together.
- **3.5** A long single sentence stays on one line. Do not cut a
sentence in half to meet a length target.
- **3.6** Leave command lines, code blocks, and table rows as they are.
## 4. Knowledge entries
A file that collects findings or lessons learned uses a fixed entry
shape so each fact is citable and the source is visible:
```text
- **<section>.<n>** [<source>] "<short title>"
- **<section>.<n>.1** <reusable fact + where it applies>
- **<section>.<n>.2** <next fact>
- **<section>.<n>.<k>** Ref: <path:line or query>; added <YYYY-MM-DD>.
```
- **4.1** The entry line holds only the source tag and the title.
- **4.2** `<source>` names where the fact was verified (a repository, a
spec, …), joined with `+` when several agree.
- **4.3** The last child is always the ref with the date it was added.
- **4.4** A finding that contradicts an entry becomes that entry's next
child, titled `OVERRIDE: …`, with its own children. Or mark the old
entry **Superseded YYYY-MM-DD** with the reason.
## 5. Organize by category
- **5.1** Group entries by topic (setup, build, deployment, …), not by
the date they were added. The date lives in the ref line.
- **5.2** Grouping puts near-duplicates next to each other. Merge them
into one entry and keep the earliest `added` date.
- **5.3** A glossary gets the same treatment: one
`###` per term under a `##` category, its definition split into
numbered items.
## 6. Doing it safely
- **6.1** Restructure with a script, not by hand, when the file has more
than a few dozen items. Hand edits drop or reorder text.
- **6.2** The script asserts that the words are preserved: same word
sequence before and after, only ids and line breaks added.
- **6.3** Verify after every pass:
- **6.3.1** No duplicate ids (`^\s*- (\[ \] )?\*\*([\d.]+)\*\*`).
- **6.3.2** Every `](#anchor)` resolves to a heading slug.
- **6.3.3** The entry count matches the original; diff titles against
`HEAD` when it does not (an uncommitted entry may explain it).
- **6.3.4** One `Ref:` line per entry.
- **6.4** Watch for editors that reformat on save (escaped `\*`, `*`
bullets padded with three spaces, lost HTML comments, fences without a language). Normalize
back and re-verify.
- **6.5** On Windows PowerShell, write helper scripts to `%TEMP%` and run
them; `python -c` with backticks breaks. Avoid printing non-ASCII to
the cp1252 console. Delete the temp scripts afterwards.
- **6.6** Keep Unix line endings and `-` bullets, and run the markdown
linter before committing.
## 7. Example
Before:
```text
## Topic
- Rule about the topic. A fact about it. How to fix it. Ref: file:line.
- Label for a group: first part of the rule, second part of the rule.
```
After:
```text
## 3. Topic
### 3.2 Subtopic
- **3.2.4** [source] "Rule about the topic"
- **3.2.4.1** A fact about it.
- **3.2.4.2** How to fix it.
- **3.2.4.3** Ref: file:line; added YYYY-MM-DD.
- **3.2.5** Label for a group:
- **3.2.5.1** First part of the rule.
- **3.2.5.2** Second part of the rule.
```
@@ -1,321 +0,0 @@
---
name: specification-writer
description: Load when writing or updating project specifications (SPEC.md).
Expert in creating structured and maintainable specification documents.
type: skill
version: 1.0.0
created: 2026-11-03
updated: 2026-11-03
---
# Specification Writer
You are an expert in writing software project specifications.
This skill provides guidance on creating structured, comprehensive, and
maintainable SPEC.md files following established best practices.
## When to Load
Load this skill when:
- Creating a new SPEC.md file for a project
- Updating or expanding an existing specification
- Reviewing specification structure and content
- Ensuring specifications follow consistent patterns
## Core Principles
### Sectionize-Markdown Conventions
Always apply sectionize-markdown conventions to every SPEC.md file.
Reference: [sectionize-markdown skill](../sectionize-markdown/SKILL.md)
- Every heading MUST be numbered: `## 1.` for sections, `### 1.1`
for subsections, etc.
- Every list item MUST have a bold ID: `- **1.1.1** Text`
- TOC MUST be present under the title with markdown links
- Anchors MUST follow GitHub-style slugs
- Never renumber existing IDs - they may be cited elsewhere
## SPEC.md Structure Template
Use this structure as the foundation for all project specifications:
```markdown
---
name: <Project Name> Specification
version: <semver>
title: Project Spec Sheet
type: specification
created: <YYYY-MM-DD>
updated: <YYYY-MM-DD>
---
# <Project Name> Specification
<!-- toc-start: headings 2..6 -->
- [1. Product Overview](#1-product-overview)
- [1.1. Purpose](#11-purpose)
- [1.2. Scope](#12-scope)
- [1.3. Non-Goals](#13-non-goals)
- [1.4. Document Conventions](#14-document-conventions)
- [2. Architecture](#2-architecture)
- [2.1. Component Diagram](#21-component-diagram)
- [2.2. Data Flow](#22-data-flow)
- [2.3. Module Structure](#23-module-structure)
- [3. Core Contracts](#3-core-contracts)
- [3.1. Interfaces](#31-interfaces)
- [3.2. Data Structures](#32-data-structures)
- [4. Package and Runtime](#4-package-and-runtime)
- [4.1. Dependencies](#41-dependencies)
- [4.2. Entry Points](#42-entry-points)
- [4.3. Installation](#43-installation)
- [5. Feature Requirements](#5-feature-requirements)
- [6. Configuration Files](#6-configuration-files)
- [7. Error Handling](#7-error-handling)
- [8. Testing](#8-testing)
- [9. Deployment](#9-deployment)
<!-- toc-end -->
This document is the build contract for <Project Name>.
A reader with this file, and no source tree, must be able to produce
the same observable software.
Status values are `Implemented`, `Partially Implemented`, `Planned`,
and `Not Implemented`.
---
```
## Detailed Section Guidelines
### 1. Product Overview
#### 1.1. Purpose
- **1.1.1** Clearly state what problem the project solves
- **1.1.2** Describe the primary value proposition
- **1.1.3** Identify the target users or audience
- **1.1.4** Define the project's mission statement
#### 1.2. Scope
- **1.2.1** **In Scope**: List what the project will do
- **1.2.2** **Out of Scope**: List what the project will NOT do
- **1.2.3** Define clear boundaries for the project
#### 1.3. Non-Goals
- **1.3.1** Explicitly state what will not be implemented
- **1.3.2** Clarify boundaries with related projects or systems
- **1.3.3** Document conscious exclusions and rationale
#### 1.4. Document Conventions
- **1.4.1** Define terminology used throughout the document
- **1.4.2** Establish status labels and their meanings
- Implemented: Feature is complete and tested
- Partially Implemented: Feature is partially complete
- Planned: Feature is scheduled for implementation
- Not Implemented: Feature is specified but not yet built
- **1.4.3** Specify numbering conventions for sections and items
- **1.4.4** Define RFC2119 keywords (MUST, SHALL, SHOULD, MAY, etc.)
### 2. Architecture
#### 2.1. Component Diagram
- **2.1.1** Provide ASCII or Mermaid diagram showing system architecture
- **2.1.2** Label all major components and their relationships
- **2.1.3** Show data flow between components with arrows or descriptions
- **2.1.4** Indicate external dependencies and integrations
#### 2.2. Data Flow
- **2.2.1** Describe how data moves through the system end-to-end
- **2.2.2** Identify key data transformations at each stage
- **2.2.3** Note where data is persisted and in what format
- **2.2.4** Document data validation and error handling points
#### 2.3. Module Structure
- **2.3.1** Document the complete directory and file structure
- **2.3.2** Describe each module's responsibility and purpose
- **2.3.3** Note dependencies between modules
- **2.3.4** Indicate which modules are public vs internal APIs
### 3. Core Contracts
#### 3.1. Interfaces
- **3.1.1** Define purpose and role of each interface
- **3.1.2** List all required methods or functions with signatures
- **3.1.3** Document expected inputs, outputs, and return types
- **3.1.4** Specify error handling expectations and exceptions
- **3.1.5** Note version compatibility requirements
#### 3.2. Data Structures
- **3.2.1** Provide complete type definitions or JSON schemas
- **3.2.2** Include concrete examples of each data structure
- **3.2.3** Document validation rules and constraints
- **3.2.4** Note backward and forward compatibility considerations
- **3.2.5** Specify serialization formats (JSON, YAML, etc.)
### 4. Package and Runtime
#### 4.1. Dependencies
- **4.1.1** List runtime dependencies with exact version requirements
- **4.1.2** List development dependencies for building and testing
- **4.1.3** Document optional dependencies and their purposes
- **4.1.4** Note any version conflicts or compatibility constraints
#### 4.2. Entry Points
- **4.2.1** Define CLI entry points and their usage
- **4.2.2** Document library entry points for programmatic use
- **4.2.3** Specify plugin or extension entry points if applicable
- **4.2.4** Include examples of invoking each entry point
#### 4.3. Installation
- **4.3.1** Document all supported installation methods
- **4.3.2** Specify platform and environment requirements
- **4.3.3** Provide verification steps to confirm successful installation
- **4.3.4** Note any post-installation configuration needed
### 5. Feature Requirements
Organize requirements by feature area. Each feature section should include:
- **5.X.1** Status: One of Implemented, Partially Implemented, Planned,
Not Implemented
- **5.X.2** Purpose: User stories and motivations for the feature
- **5.X.3** Functional requirements: What the feature must do
- **5.X.4** Non-functional requirements: Performance, security, etc.
- **5.X.5** Acceptance criteria: How to verify the feature works
### 6. Configuration Files
For each configuration file, document:
- **6.1.1** Location: Both relative path and typical absolute paths
- Note platform-specific locations (Windows vs Unix)
- **6.1.2** Purpose: What the configuration controls
- **6.1.3** Structure: Complete format and schema
- **6.1.4** Default values: What settings are used if not specified
- **6.1.5** Customization: How users can modify the configuration
### 7. Error Handling
#### 7.1. Error Classification
- **7.1.1** Transient vs permanent errors
- **7.1.2** Recoverable vs unrecoverable errors
- **7.1.3** User-facing vs internal errors
- **7.1.4** Expected vs unexpected errors
#### 7.2. Error Reporting
- **7.2.1** Logging strategy and verbosity levels
- **7.2.2** User notification mechanisms (CLI, UI, etc.)
- **7.2.3** Error codes and their meanings
- **7.2.4** Error message format and content
#### 7.3. Recovery Strategies
- **7.3.1** Automatic recovery mechanisms and retries
- **7.3.2** Manual recovery procedures and user actions
- **7.3.3** Data consistency guarantees during failures
- **7.3.4** Rollback and undo capabilities
### 8. Testing
#### 8.1. Test Strategy
- **8.1.1** Testing pyramid: Unit, integration, system, E2E
- **8.1.2** Test coverage targets for different test types
- **8.1.3** Test environments and their configurations
- **8.1.4** CI/CD integration for automated testing
#### 8.2. Test Cases
- **8.2.1** Critical path tests covering core functionality
- **8.2.2** Edge case tests for boundary conditions
- **8.2.3** Regression tests for known issues
- **8.2.4** Performance and load tests if applicable
### 9. Deployment
#### 9.1. Release Process
- **9.1.1** Versioning scheme (SemVer, CalVer, etc.)
- **9.1.2** Release checklist and required steps
- **9.1.3** Rollback procedures for failed deployments
- **9.1.4** Release notes and changelog requirements
#### 9.2. Distribution
- **9.2.1** Package repositories and distribution channels
- **9.2.2** Installation methods for end users
- **9.2.3** Update mechanisms and version management
- **9.2.4** Offline or air-gapped installation options
## Sectionize-Markdown Integration
**CRITICAL**: Every SPEC.md file you create or modify MUST follow
sectionize-markdown conventions completely.
Before finalizing any SPEC.md:
1. Apply numbered headings (## 1., ### 1.1, #### 1.1.1)
2. Add bold IDs to all list items (- **1.1.1**)
3. Generate or update the TOC with markdown links
4. Ensure all anchors resolve correctly
5. Verify no duplicate IDs exist
Use the sectionize-markdown skill as the definitive reference for
all sectionizing conventions and requirements.
## Quality Checklist
Before marking a SPEC as complete, verify:
- [ ] **Structure**: All required sections are present
- [ ] **Numbering**: All headings and list items are numbered correctly
- [ ] **TOC**: Table of contents is present and accurate
- [ ] **Status**: Every feature section has a status line
- [ ] **Examples**: Code and configuration examples are provided
- [ ] **Cross-references**: All internal links resolve correctly
- [ ] **Consistency**: Terminology and formatting are consistent
- [ ] **Completeness**: All acceptance criteria are defined
- [ ] **Sectionized**: Follows sectionize-markdown conventions completely
## Maintenance Guidelines
- **Versioning**: Increment version in frontmatter with each update
- **Changelog**: Maintain updated and created dates in frontmatter
- **Review**: SPEC should be reviewed before major implementation
- **Sync**: Keep SPEC in sync with implementation (update status)
## Tools and Automation
When working with SPEC files:
- Use scripts to verify section numbering consistency
- Automate TOC generation to avoid manual errors
- Validate all internal links before committing
- Check for duplicate IDs across the entire document
- Run markdown linters to ensure formatting quality
## Common Pitfalls
- **Unnumbered items**: Every list item MUST have a bold ID
- **Broken anchors**: Always verify all TOC links resolve
- **Inconsistent status**: Use consistent status labels throughout
- **Missing examples**: Always include examples for data formats
- **Ambiguous requirements**: Requirements must be testable and verifiable
- **Overly verbose**: Be concise but comprehensive
- **Under-specified**: Include enough detail for implementation
-19
View File
@@ -1,19 +0,0 @@
---
name: log
title: Wiki Log
version: 2.0.0
type: summary
created: DATE_PLACEHOLDER
updated: DATE_PLACEHOLDER
confidence: trusted
llm:
- model: none
contested: false
contradictions: []
---
# Wiki Log
Chronological tracking of all actions performed on the wiki.
- DATE_PLACEHOLDER - Initialized the llm-wiki.
File diff suppressed because it is too large Load Diff
@@ -1,327 +0,0 @@
<!DOCTYPE html>
<!-- Packaged layout for fiona.gittool.diff_report; placeholders filled at generation time. -->
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>{{PAGE_TITLE}}</title>
<style>
{{THEME_CSS}}
* { box-sizing: border-box; }
body {
margin: 0;
font-family: var(--font);
background: var(--bg);
color: var(--text);
line-height: 1.45;
transition: background-color 0.3s ease, color 0.3s ease;
}
header {
background: var(--card);
color: var(--text);
padding: 1rem 1.25rem;
border-bottom: 1px solid var(--border);
display: flex;
justify-content: space-between;
align-items: center;
flex-wrap: wrap;
gap: 1rem;
}
header .header-title {
flex: 1;
min-width: 200px;
}
header h1 { margin: 0 0 0.35rem; font-size: 1.4rem; font-weight: 600; }
header p { margin: 0.25rem 0; opacity: 0.92; font-size: 0.9rem; }
header code {
background: color-mix(in srgb, var(--button-text) 15%, transparent);
padding: 0.1rem 0.35rem;
border-radius: 4px;
}
.theme-toggle {
display: flex;
align-items: center;
gap: 0.5rem;
}
.theme-toggle button {
padding: 0.35rem 0.65rem;
cursor: pointer;
border: 1px solid var(--border);
border-radius: 6px;
background: var(--button-bg);
color: var(--button-text);
font-size: 0.85rem;
}
.theme-toggle button:hover { background: var(--button-hover); }
.layout {
display: flex;
align-items: stretch;
min-height: calc(100vh - 8rem);
}
aside.sidebar {
width: var(--sidebar-w);
flex-shrink: 0;
position: sticky;
top: 0;
align-self: flex-start;
max-height: 100vh;
overflow: auto;
background: var(--card);
border-right: 1px solid var(--border);
padding: 0.75rem 0.5rem 1rem;
}
aside.sidebar h2 {
margin: 0 0 0.5rem 0.65rem;
font-size: 0.75rem;
text-transform: uppercase;
letter-spacing: 0.05em;
color: var(--blue);
}
.file-tree {
list-style: none;
margin: 0;
padding: 0 0 0 0.35rem;
font-size: 0.8rem;
}
.file-tree ul {
list-style: none;
margin: 0;
padding-left: 0.65rem;
border-left: 1px solid var(--border);
}
.file-tree li { margin: 0.15rem 0; }
.file-tree .folder-label {
display: block;
font-weight: 600;
color: var(--text);
padding: 0.1rem 0.35rem;
}
.file-tree a {
display: block;
padding: 0.15rem 0.35rem;
border-radius: 4px;
color: var(--accent);
text-decoration: none;
word-break: break-all;
}
.file-tree a:hover { background: var(--link-hover-bg); text-decoration: underline; }
.file-tree a.active {
background: var(--link-active-bg);
font-weight: 600;
color: var(--blue);
}
.file-tree .churn {
font-size: 0.7rem;
color: var(--muted);
margin-left: 0.25rem;
white-space: nowrap;
}
.churn .add { color: var(--churn-add); }
.churn .del { color: var(--churn-del); }
.file-icon {
font-size: 0.8rem;
margin-right: 0.35rem;
opacity: 0.7;
}
.stat-fold {
margin: 0.75rem 0.5rem 0;
font-size: 0.75rem;
}
.stat-fold summary { cursor: pointer; color: var(--muted); }
.stat-fold pre {
margin: 0.35rem 0 0;
font-size: 0.68rem;
white-space: pre-wrap;
max-height: 12rem;
overflow: auto;
background: var(--summary-bg);
padding: 0.5rem;
border-radius: 4px;
border: 1px solid var(--border);
}
.content {
flex: 1;
min-width: 0;
padding: 1rem 1.25rem 2.5rem;
}
.toolbar {
font-size: 0.85rem;
display: flex;
align-items: center;
gap: 0.5rem;
flex-wrap: wrap;
background: var(--card);
border: 1px solid var(--border);
border-radius: 8px;
padding: 0.65rem 0.85rem;
margin-bottom: 1rem;
}
button {
padding: 0.35rem 0.65rem;
cursor: pointer;
border: 1px solid var(--border);
border-radius: 6px;
background: var(--button-bg);
color: var(--button-text);
}
button:hover { background: var(--button-hover); }
details.file {
margin: 0.75rem 0;
border: 1px solid var(--border);
border-radius: 6px;
background: var(--card);
scroll-margin-top: 0.75rem;
}
details.file:target {
outline: 2px solid var(--accent);
outline-offset: 2px;
}
details.file summary {
cursor: pointer;
padding: 0.6rem 0.75rem;
font-weight: 600;
background: var(--summary-bg);
}
pre.diff {
margin: 0;
padding: 0.75rem;
font-size: 0.72rem;
line-height: 1.0;
overflow: auto;
max-height: 70vh;
}
.diff-add { background: var(--diff-add-bg); display: block; margin: 0; padding: 0; }
.diff-del { background: var(--diff-del-bg); display: block; margin: 0; padding: 0; }
.diff-hunk { color: var(--diff-hunk); display: block; margin: 0; padding: 0; }
.diff-meta { color: var(--diff-meta); display: block; margin: 0; padding: 0; }
.diff-ctx { display: block; margin: 0; padding: 0; }
@media (max-width: 900px) {
.layout { flex-direction: column; }
aside.sidebar {
width: 100%;
position: relative;
max-height: 40vh;
border-right: none;
border-bottom: 1px solid var(--border);
}
}
/* Dark mode theme - applied when data-theme="dark" (default) */
[data-theme="dark"] {
--bg: #1a1a1a;
--card: #2d2d2d;
--border: #404040;
--text: #e0e0e0;
--blue: #6ba1f5;
--accent: #7aa7d6;
--header-end: #c586c0;
--muted: #9e9e9e;
--summary-bg: #252525;
--button-bg: #3d3d3d;
--button-text: #e0e0e0;
--button-hover: #4a4a4a;
--churn-add: #7edd8a;
--churn-del: #ff7b7b;
--diff-add-bg: color-mix(in srgb, #7edd8a 22%, var(--card));
--diff-del-bg: color-mix(in srgb, #ff7b7b 22%, var(--card));
--link-hover-bg: color-mix(in srgb, #7aa7d6 18%, var(--card));
--link-active-bg: color-mix(in srgb, #7aa7d6 22%, var(--card));
}
/* Light mode theme - applied when data-theme="light" */
[data-theme="light"] {
--bg: #ffffff;
--card: #f8f9fa;
--border: #e2e8f0;
--text: #1a1a1a;
--blue: #0d6efd;
--accent: #0d6efd;
--header-end: #6610f2;
--muted: #6c757d;
--summary-bg: #f8f9fa;
--button-bg: #ffffff;
--button-text: #1a1a1a;
--button-hover: #e2e8f0;
--churn-add: #15803d;
--churn-del: #b91c1c;
--diff-add-bg: color-mix(in srgb, #15803d 22%, var(--card));
--diff-del-bg: color-mix(in srgb, #b91c1c 22%, var(--card));
--link-hover-bg: color-mix(in srgb, #0d6efd 18%, var(--card));
--link-active-bg: color-mix(in srgb, #0d6efd 22%, var(--card));
}
</style>
</head>
<body>
<header>
<div class="header-title">
{{HEADER_BODY}}
</div>
{{THEME_TOGGLE_HTML}}
</header>
<div class="layout">
<aside class="sidebar" aria-label="Changed files">
<h2>Files ({{FILE_COUNT}})</h2>
{{TREE_HTML}}
<details class="stat-fold">
<summary>git diff --stat</summary>
<pre>{{STAT_BLOCK}}</pre>
</details>
</aside>
<div class="content">
<div class="toolbar">
<button type="button" id="btn-expand-all">Expand all</button>
<button type="button" id="btn-collapse-all">Collapse all</button>
</div>
{{FILES_HTML}}
</div>
</div>
<script>
(function () {
// Theme toggle functionality - default is dark mode
var themeToggle = document.getElementById('theme-toggle');
if (themeToggle) {
var htmlEl = document.documentElement;
// Default to dark mode
var savedTheme = localStorage.getItem('fiona-diff-theme') || 'dark';
htmlEl.setAttribute('data-theme', savedTheme);
themeToggle.addEventListener('click', function () {
var currentTheme = htmlEl.getAttribute('data-theme');
var newTheme = currentTheme === 'dark' ? 'light' : 'dark';
htmlEl.setAttribute('data-theme', newTheme);
localStorage.setItem('fiona-diff-theme', newTheme);
});
}
function openFromHash() {
var id = location.hash ? location.hash.slice(1) : "";
document.querySelectorAll(".file-tree a").forEach(function (a) {
a.classList.toggle("active", a.getAttribute("href") === "#" + id);
});
if (!id) {
return;
}
var el = document.getElementById(id);
if (el && el.tagName === "DETAILS") {
el.open = true;
el.scrollIntoView({ behavior: "smooth", block: "start" });
}
}
document.getElementById("btn-expand-all").addEventListener("click", function () {
document.querySelectorAll("details.file").forEach(function (d) { d.open = true; });
});
document.getElementById("btn-collapse-all").addEventListener("click", function () {
document.querySelectorAll("details.file").forEach(function (d) { d.open = false; });
});
document.querySelectorAll(".file-tree a").forEach(function (a) {
a.addEventListener("click", function () {
var id = a.getAttribute("href").slice(1);
var el = document.getElementById(id);
if (el && el.tagName === "DETAILS") {
el.open = true;
}
});
});
window.addEventListener("hashchange", openFromHash);
openFromHash();
})();
</script>
</body>
</html>
+7 -74
View File
@@ -1,4 +1,4 @@
"""Shared logging configuration for all Fiona tools (SPEC 7.17).
"""Shared logging configuration for all Fiona tools (SPEC 5.1).
Diagnostics go through the standard `logging` module with the level
resolved from (highest precedence first):
@@ -13,25 +13,20 @@ records go to stderr.
from __future__ import annotations
import ctypes
import json
import logging
import os
import pathlib
import sys
from fiona.paths import llm_wiki_template
# AI/LLM: Windows-only; module-level so PYC001 allows it (SPEC 6.4.2).
if os.name == "nt":
import msvcrt
DEFAULT_LOG_LEVEL = "WARNING"
DEFAULT_SETTINGS_FILE = llm_wiki_template() / "fiona-settings.json"
DEFAULT_SETTINGS_FILE = (
pathlib.Path(__file__).resolve().parent.parent / "llm-wiki" / "fiona-settings.json"
)
# Tool package loggers that carry the level explicitly: pymarkdown
# resets the root logger to WARNING mid-run, which would otherwise
# silence tool diagnostics (same pattern as run_sanity, see SPEC 7.17).
TOOL_LOGGER_NAMES = ("fiona",)
# silence tool diagnostics (same pattern as run_sanity, see SPEC 4.7.1).
TOOL_LOGGER_NAMES = ("fiona", "librarian", "mdoctor", "python_ruff", "wiki")
LOG_FORMAT = "%(levelname)s %(name)s: %(message)s"
FILE_LOG_FORMAT = "%(asctime)s %(levelname)s %(name)s: %(message)s"
@@ -90,68 +85,6 @@ def load_log_level(settings: dict | None) -> str:
return candidate.upper()
def _open_shared_append(path: str, encoding: str):
"""Open a log file that Windows can delete while the handle is open.
Generated by AI/LLM: 2026-09-29 - the default open() omits
FILE_SHARE_DELETE, so a temp directory that still has fiona.log
open cannot be removed.
Args:
path: Log file path.
encoding: Text encoding for the stream.
Returns:
A text file object opened for append.
"""
if os.name != "nt":
return open(path, "a", encoding=encoding)
kernel32 = ctypes.WinDLL("kernel32", use_last_error=True)
kernel32.CreateFileW.argtypes = [
ctypes.c_wchar_p,
ctypes.c_uint32,
ctypes.c_uint32,
ctypes.c_void_p,
ctypes.c_uint32,
ctypes.c_uint32,
ctypes.c_void_p,
]
kernel32.CreateFileW.restype = ctypes.c_void_p
kernel32.SetFilePointer.argtypes = [
ctypes.c_void_p,
ctypes.c_long,
ctypes.c_void_p,
ctypes.c_uint32,
]
kernel32.SetFilePointer.restype = ctypes.c_uint32
handle = kernel32.CreateFileW(
path,
0x40000000, # GENERIC_WRITE
0x1 | 0x2 | 0x4, # FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE
None,
4, # OPEN_ALWAYS
0x80, # FILE_ATTRIBUTE_NORMAL
None,
)
invalid = ctypes.c_void_p(-1).value
if handle in (None, invalid):
raise OSError(ctypes.get_last_error(), f"Cannot open log file {path}")
kernel32.SetFilePointer(handle, 0, None, 2) # FILE_END
fd = msvcrt.open_osfhandle(handle, os.O_APPEND)
return os.fdopen(fd, "a", encoding=encoding, closefd=True)
class _SharedFileHandler(logging.FileHandler):
"""File log handler that does not block deletion of its directory.
Generated by AI/LLM: 2026-09-29 - Windows tests and temp wikis remove
the working directory while fiona.log is still attached.
"""
def _open(self):
return _open_shared_append(self.baseFilename, self.encoding or "utf-8")
def _remove_file_handler() -> None:
"""Detach (and close) the managed file log handler, if any."""
for handler in list(logging.getLogger().handlers):
@@ -184,7 +117,7 @@ def _configure_file_log(fiona_settings: dict, console_level: str) -> None:
)
file_level = console_level
try:
handler = _SharedFileHandler(location, encoding="utf-8")
handler = logging.FileHandler(location, encoding="utf-8")
except OSError as error:
logging.getLogger(__name__).warning("Cannot open log file %s: %s", location, error)
return
-21
View File
@@ -1,21 +0,0 @@
"""Filesystem locations for packaged Fiona data.
Generated by AI/LLM: 2026-09-29 - one resolver for the llm-wiki template.
"""
from __future__ import annotations
import importlib.resources
import pathlib
def llm_wiki_template() -> pathlib.Path:
"""Return the packaged llm-wiki template directory.
Generated by AI/LLM: 2026-09-29 - load package data with importlib.resources.
Returns:
Filesystem path of ``fiona/data/llm_wiki``.
"""
resource = importlib.resources.files("fiona").joinpath("data", "llm_wiki")
return pathlib.Path(str(resource))
-5
View File
@@ -1,5 +0,0 @@
"""Wiki package for Fiona-Assistant.
Submodules are imported directly. This package does not import them
here, so git_tool can load wiki.ignore without a circular import.
"""
File diff suppressed because it is too large Load Diff
-367
View File
@@ -1,367 +0,0 @@
"""Wiki housekeeping routine (SPEC 4.8, 7.2).
Generated by AI/LLM: 2026-09-30 - one call for LLM wiki maintenance.
`fiona-assistant housekeeping` runs the maintenance steps in a fixed
order and reports, per step, a status (`PASS`, `WARN`, `FAIL`), a
summary, detail lines, and the next actions for the LLM. The routine
gathers what needs work; it writes only `wiki_graph.html`.
Steps, in order (`STEP_NAMES`):
- `check-update`: core files behind the packaged template (read-only).
- `merge`: files whose stem ends with `-update` that `wiki update` left for a merge.
- `ingest`: pages under `wiki/` still at `status: raw`.
- `sanity`: every `wiki sanity` check.
- `graph`: regenerate `wiki_graph.html` and list orphan pages.
"""
from __future__ import annotations
import contextlib
import io
import json
import os
import sys
import tempfile
from pathlib import Path
from fiona.librarian.md_frontmatter import FrontmatterError, parse_frontmatter
from fiona.wiki.graph_generator import wiki_graph_main
from fiona.wiki.sanity_manager import SETTINGS_FILE, run_wiki_checks
from fiona.wiki.update_manager import (
UPDATE_SUFFIX,
check_wiki_updates,
get_update_ignore_patterns,
path_is_update_ignored,
)
STEP_NAMES = ("check-update", "merge", "ingest", "sanity", "graph")
GRAPH_FILE = "wiki_graph.html"
UPDATE_FILE_SUFFIX = f"{UPDATE_SUFFIX}.md"
# AI/LLM: read-only zones sit at the wiki root; .git is skipped anywhere
ROOT_SKIP_DIRS = frozenset({"personal", "raw"})
__all__ = [
"GRAPH_FILE",
"STEP_NAMES",
"check_update_step",
"format_housekeeping_report",
"graph_step",
"housekeeping_main",
"ingest_step",
"merge_step",
"run_housekeeping",
"sanity_step",
]
# =============================================================================
# Helpers
# =============================================================================
def _step(name: str, status: str, summary: str, details: list[str] | None = None,
next_steps: list[str] | None = None) -> dict:
"""Build one step result.
Generated by AI/LLM: 2026-09-30 - shared step shape (SPEC 4.8).
"""
return {
"name": name,
"status": status,
"summary": summary,
"details": details or [],
"next": next_steps or [],
}
def _relative(path: str | Path, wiki_root: Path) -> str:
"""Return a path relative to the wiki root, with `/` separators.
Generated by AI/LLM: 2026-09-30 - report paths from the wiki root.
"""
return Path(os.path.relpath(path, wiki_root)).as_posix()
def _frontmatter(path: Path) -> dict:
"""Return a page's frontmatter, or an empty dict when it cannot be read.
Generated by AI/LLM: 2026-09-30 - sanity reports broken YAML itself.
"""
try:
frontmatter = parse_frontmatter(path)
except (FrontmatterError, UnicodeDecodeError):
return {}
return frontmatter if isinstance(frontmatter, dict) else {}
def _wiki_files(wiki_root: Path) -> list[Path]:
"""List files outside read-only zones and `.git`, sorted.
Modified by AI/LLM: 2026-10-01 - merge walks every suffix, not only markdown.
Args:
wiki_root: Root path of the wiki.
Returns:
list[Path]: Files found.
"""
found = []
for current, dirs, files in os.walk(wiki_root):
at_root = Path(current) == wiki_root
dirs[:] = [
name for name in dirs
if name != ".git" and not (at_root and name in ROOT_SKIP_DIRS)
]
found.extend(Path(current) / name for name in files)
return sorted(found)
def _markdown_files(wiki_root: Path, subdir: str = "") -> list[Path]:
"""List markdown files outside read-only zones and `.git`, sorted.
Generated by AI/LLM: 2026-09-30 - pruned walk for the merge and ingest steps.
Args:
wiki_root: Root path of the wiki.
subdir: Optional directory, relative to the root, to walk instead.
Returns:
list[Path]: Markdown files found.
"""
start = wiki_root / subdir
found = []
for current, dirs, files in os.walk(start):
at_root = Path(current) == wiki_root
dirs[:] = [
name for name in dirs
if name != ".git" and not (at_root and name in ROOT_SKIP_DIRS)
]
found.extend(Path(current) / name for name in files if name.endswith(".md"))
return sorted(found)
# =============================================================================
# Steps
# =============================================================================
def check_update_step(wiki_root: Path) -> dict:
"""Report core files behind the template without writing (SPEC 7.2.2).
Generated by AI/LLM: 2026-09-30 - read-only wiki check-update.
"""
summary = check_wiki_updates(wiki_root)
details = [f"update: {_relative(r.file_path, wiki_root)}" for r in summary.files_to_update]
details += [
f"conflict: {_relative(r.file_path, wiki_root)} -> {_relative(r.update_path, wiki_root)}"
for r in summary.conflicts
]
details += [f"add: {path.as_posix()}" for path in summary.files_to_add]
text = (
f"{len(summary.files_to_update)} update(s), {len(summary.conflicts)} conflict(s), "
f"{len(summary.files_to_add)} addition(s)"
)
if not details:
return _step("check-update", "PASS", text)
return _step("check-update", "WARN", text, details, [
"Ask the user, then run fiona-assistant wiki update --dry-run and wiki update --yes."
])
def merge_step(wiki_root: Path) -> dict:
"""List update files that wait for an LLM merge (SPEC 7.2.3).
Modified by AI/LLM: 2026-10-02 - skip update.ignore_files and corefiles.json.
"""
details, actions = [], []
ignore_patterns = get_update_ignore_patterns(wiki_root)
for update_file in _wiki_files(wiki_root):
if not update_file.stem.endswith(UPDATE_SUFFIX):
continue
if path_is_update_ignored(_relative(update_file, wiki_root), ignore_patterns):
continue
target = update_file.with_name(
update_file.stem[: -len(UPDATE_SUFFIX)] + update_file.suffix
)
update_rel = _relative(update_file, wiki_root)
target_rel = _relative(target, wiki_root)
missing = "" if target.exists() else " (target missing)"
details.append(f"{update_rel} -> {target_rel}{missing}")
version = _frontmatter(update_file).get("version") or "the template version"
actions.append(
f"Merge {update_rel} into {target_rel}: keep the owner's edits, "
f"take the template changes, set version to at least {version}, "
f"then delete {update_rel}."
)
text = f"{len(details)} update file(s) wait for a merge"
return _step("merge", "FAIL" if details else "PASS", text, details, actions)
def ingest_step(wiki_root: Path) -> dict:
"""List pages under wiki/ still at `status: raw` (SPEC 7.2.4).
Generated by AI/LLM: 2026-09-30 - pages waiting for a refine.
"""
pages = [
_relative(page, wiki_root)
for page in _markdown_files(wiki_root, "wiki")
if _frontmatter(page).get("status") == "raw"
]
text = f"{len(pages)} raw page(s)"
if not pages:
return _step("ingest", "PASS", text)
return _step("ingest", "WARN", text, pages,
["Refine each raw page, then set status: refined."])
def sanity_step(wiki_root: Path) -> dict:
"""Run every wiki sanity check without the progress display (SPEC 7.2.5).
Generated by AI/LLM: 2026-09-30 - failing check reports as details.
"""
results = run_wiki_checks(wiki_root)
failing = {name: check for name, check in results["checks"].items() if check["result"]}
text = f"{results['total_violations']} violation(s) in {len(failing)} check(s)"
if not failing:
return _step("sanity", "PASS", text)
details = [
f"{name}: {line.strip().removeprefix('- ')}"
for name, check in failing.items()
for line in check["stdout"].splitlines()
if line.strip()
]
return _step("sanity", "FAIL", text, details,
["Fix the violations, then run fiona-assistant housekeeping again."])
def graph_step(wiki_root: Path) -> dict:
"""Write wiki_graph.html at the wiki root and list orphans (SPEC 7.2.6).
Generated by AI/LLM: 2026-09-30 - graph plus orphan pages under wiki/.
"""
captured = io.StringIO()
with tempfile.TemporaryDirectory() as temp_dir:
graph_json = Path(temp_dir) / "graph.json"
with contextlib.redirect_stdout(captured), contextlib.redirect_stderr(captured):
code = wiki_graph_main(str(wiki_root), output=str(graph_json), format_type="json")
if code == 0:
code = wiki_graph_main(str(wiki_root), output=str(wiki_root / GRAPH_FILE))
if code != 0:
error = captured.getvalue().strip() or f"wiki graph exited {code}"
return _step("graph", "FAIL", f"could not write {GRAPH_FILE}",
error.splitlines())
data = json.loads(graph_json.read_text(encoding="utf-8"))
orphans = sorted(
rel for rel in (_relative(path, wiki_root) for path in data["orphans"])
if rel.startswith("wiki/") and not rel.endswith(UPDATE_FILE_SUFFIX)
)
text = (
f"wrote {GRAPH_FILE} ({data['metadata']['file_count']} nodes, "
f"{len(data['edges'])} edges)"
)
if not orphans:
return _step("graph", "PASS", text)
return _step("graph", "WARN", text, [f"orphan: {rel}" for rel in orphans],
["Link each orphan page from a related page or from wiki/index.md."])
# =============================================================================
# Routine and Report
# =============================================================================
def run_housekeeping(directory: str | Path = ".") -> dict:
"""Run every housekeeping step over a wiki (SPEC 7.2.1).
Generated by AI/LLM: 2026-09-30 - every step runs, even after a FAIL.
Args:
directory: The wiki directory. Defaults to the current directory.
Returns:
dict: `target` (resolved path), `steps` (one result per step,
in `STEP_NAMES` order), and `passed` (no step is `FAIL`).
Raises:
FileNotFoundError: If the directory does not exist.
NotADirectoryError: If the directory is not an llm-wiki directory.
"""
wiki_root = Path(directory).resolve()
if not wiki_root.exists():
raise FileNotFoundError(f"Directory not found: {directory}")
if not (wiki_root / SETTINGS_FILE).exists():
raise NotADirectoryError(f"Directory '{directory}' is not an llm-wiki directory")
steps = [
step(wiki_root)
for step in (
check_update_step,
merge_step,
ingest_step,
sanity_step,
graph_step,
)
]
return {
"target": wiki_root,
"steps": steps,
"passed": all(step["status"] != "FAIL" for step in steps),
}
def format_housekeeping_report(results: dict) -> str:
"""Format housekeeping results for humans and LLMs (SPEC 4.8).
Generated by AI/LLM: 2026-09-30 - step blocks, next steps, verdict.
Args:
results: The dict returned by run_housekeeping.
Returns:
str: The report text.
"""
lines = ["Housekeeping Report", f"TARGET: {results['target']}", ""]
for step in results["steps"]:
lines.append(f"[{step['status']}] {step['name']}: {step['summary']}")
lines.extend(f" - {detail}" for detail in step["details"])
actions = list(dict.fromkeys(a for step in results["steps"] for a in step["next"]))
if actions:
lines.extend(["", "Next steps:"])
lines.extend(f" - {action}" for action in actions)
failing = [step["name"] for step in results["steps"] if step["status"] == "FAIL"]
lines.append("")
if failing:
lines.append(
f"HOUSEKEEPING FAILED - {len(failing)} step(s) need attention: "
f"{', '.join(failing)}"
)
else:
lines.append("HOUSEKEEPING PASSED")
return "\n".join(lines)
def housekeeping_main(directory: str = ".", json_output: bool = False) -> int:
"""Main function for the 'housekeeping' CLI command.
Generated by AI/LLM: 2026-09-30 - exit 1 on any FAIL step.
Args:
directory: Wiki directory (default: current directory).
json_output: If True, print the results as one JSON object.
Returns:
int: 0 when no step is `FAIL`, 1 otherwise or on an invalid wiki.
"""
try:
results = run_housekeeping(directory)
except (FileNotFoundError, NotADirectoryError) as error:
print(f"Error: {error}", file=sys.stderr)
return 1
if json_output:
print(json.dumps({**results, "target": str(results["target"])}, indent=2))
else:
print(format_housekeeping_report(results))
return 0 if results["passed"] else 1
-171
View File
@@ -1,171 +0,0 @@
"""Gitignore patterns shared by wiki init and wiki update.
Generated by AI/LLM: 2026-09-30 - seed wiki ignores from the repository file.
"""
from __future__ import annotations
import fnmatch
import importlib.resources
import pathlib
_PACKAGE_NAME = "fiona-assistant"
_PERSONAL_PATTERN = "personal/"
def gitignore_patterns_from_text(text: str) -> list[str]:
"""Return ignore patterns from gitignore text.
Generated by AI/LLM: 2026-09-30 - drop blanks and comments.
Args:
text: Contents of a .gitignore file.
Returns:
Patterns in file order, without comments or blank lines.
"""
patterns: list[str] = []
for line in text.splitlines():
stripped = line.strip()
if stripped and not stripped.startswith("#"):
patterns.append(stripped)
return patterns
def path_is_ignored(relpath: str, patterns: list[str]) -> bool:
"""Return whether a relative posix path matches any gitignore pattern.
Generated by AI/LLM: 2026-09-30 - match unanchored names and directory prefixes.
A pattern with no slash matches any path segment. A trailing slash matches
a directory and everything under it. ``*`` does not cross a directory
boundary because matching is done per segment.
Args:
relpath: Path relative to the wiki or template root, posix form.
patterns: Gitignore patterns.
Returns:
True when one pattern matches.
"""
return any(_pattern_matches(relpath, pattern) for pattern in patterns)
def _pattern_matches(relpath: str, pattern: str) -> bool:
"""Return whether one gitignore pattern matches a relative posix path."""
body = pattern.rstrip("/")
if not body:
return False
parts = relpath.split("/")
if "/" in body:
prefixes = ["/".join(parts[:index]) for index in range(1, len(parts) + 1)]
return any(fnmatch.fnmatchcase(prefix, body) for prefix in prefixes)
return any(fnmatch.fnmatchcase(part, body) for part in parts)
def repository_gitignore_path() -> pathlib.Path | None:
"""Find the fiona-assistant repository .gitignore, if this tree has one.
Generated by AI/LLM: 2026-09-30 - prefer the source checkout over package data.
Returns:
Path to the repository .gitignore, or None when this install is not
the source checkout.
"""
here = pathlib.Path(__file__).resolve()
for parent in here.parents:
project = parent / "pyproject.toml"
gitignore = parent / ".gitignore"
if not project.is_file() or not gitignore.is_file():
continue
text = project.read_text(encoding="utf-8")
if f'name = "{_PACKAGE_NAME}"' in text:
return gitignore
return None
def packaged_gitignore_path() -> pathlib.Path | None:
"""Return the packaged copy of the repository .gitignore, if present.
Generated by AI/LLM: 2026-09-30 - fallback for an installed wheel.
Returns:
Path to ``fiona/data/default.gitignore``, or None when it is absent.
"""
resource = importlib.resources.files("fiona").joinpath("data", "default.gitignore")
path = pathlib.Path(str(resource))
if path.is_file():
return path
return None
def repository_gitignore_text() -> str:
"""Load the ignore file that seeds a wiki, as LF text.
Generated by AI/LLM: 2026-09-30 - repository file, then packaged copy.
Returns:
Ignore file text ending in a newline.
Raises:
FileNotFoundError: When neither the repository nor the packaged file exists.
"""
path = repository_gitignore_path() or packaged_gitignore_path()
if path is None:
raise FileNotFoundError("fiona-assistant repository .gitignore was not found")
text = path.read_text(encoding="utf-8").replace("\r\n", "\n").replace("\r", "\n")
if text and not text.endswith("\n"):
text += "\n"
return text
def wiki_gitignore_patterns() -> list[str]:
"""Return patterns written into a wiki, repository set plus personal/.
Generated by AI/LLM: 2026-09-30 - personal/ stays untracked after init.
Returns:
Ignore patterns in write order.
"""
patterns = gitignore_patterns_from_text(repository_gitignore_text())
if _PERSONAL_PATTERN not in patterns:
patterns.append(_PERSONAL_PATTERN)
return patterns
def wiki_gitignore_text() -> str:
"""Return the full .gitignore body for a wiki that does not have one yet.
Generated by AI/LLM: 2026-09-30 - repository text plus personal/ when missing.
Returns:
File body ending in a newline.
"""
text = repository_gitignore_text()
if _PERSONAL_PATTERN not in gitignore_patterns_from_text(text):
text += f"{_PERSONAL_PATTERN}\n"
return text
def ensure_wiki_gitignore(directory: pathlib.Path) -> None:
"""Create .gitignore or append patterns that are not already listed.
Generated by AI/LLM: 2026-09-30 - put or append the repository ignores.
Args:
directory: Wiki root that should contain .gitignore.
"""
target = directory / ".gitignore"
desired = wiki_gitignore_patterns()
if not target.exists():
target.write_text(wiki_gitignore_text(), encoding="utf-8", newline="\n")
return
existing = target.read_text(encoding="utf-8").replace("\r\n", "\n").replace("\r", "\n")
have = set(gitignore_patterns_from_text(existing))
missing = [pattern for pattern in desired if pattern not in have]
if not missing:
return
if existing and not existing.endswith("\n"):
existing += "\n"
existing += "\n".join(missing) + "\n"
target.write_text(existing, encoding="utf-8", newline="\n")
-4
View File
@@ -1,4 +0,0 @@
"""Wiki ingest package (SPEC 7.1.5).
Generated by AI/LLM: 2026-09-30 - stage modules for wiki ingest.
"""
-239
View File
@@ -1,239 +0,0 @@
"""Archive a fetched file under ``raw/``.
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.3. Never edits a file already there.
"""
from __future__ import annotations
import hashlib
import logging
import pathlib
import shutil
from fiona.wiki.ingest.converters.base import ConverterRegistry
from fiona.wiki.ingest.media import extension_for
from fiona.wiki.ingest.models import ArchiveResult, FetchedSource, IngestError
from fiona.wiki.ingest.settings import IngestSettings
logger = logging.getLogger(__name__)
class RawArchiver:
"""Copy or move a source into ``raw/`` and reuse identical bytes.
Generated by AI/LLM: 2026-09-30 - SHA-256 reuse and collision names.
"""
def __init__(self, wiki_root: pathlib.Path) -> None:
"""Store the wiki root.
Args:
wiki_root: Directory that contains ``raw/``.
"""
self.wiki_root = wiki_root
def find_by_sha256(
self,
digest: str,
size: int,
skip: pathlib.Path | None = None,
) -> pathlib.Path | None:
"""Return the first ``raw/`` file with this size and digest.
Generated by AI/LLM: 2026-09-30 - only same-size files are hashed.
Args:
digest: Lowercase SHA-256 hex.
size: Byte size that a candidate must match.
skip: A file to ignore, usually the source when it already sits in ``raw/``.
Returns:
pathlib.Path | None: Match with the lowest path, or None.
"""
raw_root = self.wiki_root / "raw"
if not raw_root.is_dir():
return None
skip_resolved = skip.resolve() if skip is not None else None
matches: list[pathlib.Path] = []
for path in raw_root.rglob("*"):
if not path.is_file():
continue
if skip_resolved is not None and path.resolve() == skip_resolved:
continue
try:
if path.stat().st_size != size:
continue
except OSError:
continue
if _sha256(path) == digest:
matches.append(path)
if not matches:
return None
matches.sort(key=lambda item: item.as_posix())
return matches[0]
def archive(
self,
source: FetchedSource,
*,
registry: ConverterRegistry,
settings: IngestSettings,
raw_override: str | None,
dry_run: bool,
) -> ArchiveResult:
"""Place the source under ``raw/`` or reuse an identical copy.
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.3.
Args:
source: Fetched file.
registry: Used for the raw category and a missing extension.
settings: ``raw_categories`` overrides.
raw_override: ``-r`` value, relative to ``raw/``, or None.
dry_run: When true, compute the path and write nothing.
Returns:
ArchiveResult: Relative raw path, digest, and warnings.
Raises:
IngestError: ``-r`` escapes ``raw/`` or is used on a file already there.
"""
raw_root = (self.wiki_root / "raw").resolve()
source_in_raw = _is_inside(source.path, raw_root)
if raw_override and source_in_raw:
raise IngestError(
f"--raw cannot be used on a file already under raw/: {source.argument}"
)
if raw_override:
_require_inside(raw_root, raw_override, "--raw must stay inside raw/")
digest = _sha256(source.path)
match = self.find_by_sha256(
digest,
source.size_bytes,
skip=source.path if source_in_raw else None,
)
warnings: list[str] = []
if match is not None:
raw_rel = _relative(match, self.wiki_root)
if raw_override:
warnings.append(f"already archived at {raw_rel}; --raw ignored")
if _is_inside(source.path, self.wiki_root / "ingest") and not dry_run:
source.path.unlink()
logger.info("reused raw file %s", raw_rel)
return ArchiveResult(match, raw_rel, digest, True, warnings)
if source_in_raw:
return ArchiveResult(source.path, _relative(source.path, self.wiki_root), digest, False, [])
extension = source.path.suffix.lower() or extension_for(source.media_type, "", registry)
dest = self._destination(source, registry, settings, raw_override, extension, raw_root)
dest = _free_name(dest, source.slug, extension)
if not dry_run:
dest.parent.mkdir(parents=True, exist_ok=True)
if _is_inside(source.path, self.wiki_root / "ingest"):
shutil.move(str(source.path), str(dest))
else:
shutil.copyfile(source.path, dest)
logger.info("archived %s", dest)
return ArchiveResult(dest, _relative(dest, self.wiki_root), digest, False, warnings)
def _destination(
self,
source: FetchedSource,
registry: ConverterRegistry,
settings: IngestSettings,
raw_override: str | None,
extension: str,
raw_root: pathlib.Path,
) -> pathlib.Path:
"""Resolve the raw path before a collision suffix is applied.
Generated by AI/LLM: 2026-09-30 - ``-r`` file or directory, else the category.
"""
filename = f"{source.slug}{extension}"
if raw_override:
relative = _strip_prefix(raw_override, "raw")
suffix = pathlib.PurePosixPath(relative.replace("\\", "/")).suffix
if suffix:
return raw_root / relative
return raw_root / relative / filename
converter = registry.find(source.media_type, extension)
if converter is None:
category = "other"
else:
category = settings.raw_categories.get(converter.name, converter.raw_category)
return raw_root / category / filename
def _sha256(path: pathlib.Path) -> str:
"""Return the lowercase SHA-256 hex digest of a file.
Generated by AI/LLM: 2026-09-30 - streamed so large files stay cheap.
"""
digest = hashlib.sha256()
with path.open("rb") as handle:
while True:
chunk = handle.read(1024 * 1024)
if not chunk:
break
digest.update(chunk)
return digest.hexdigest()
def _free_name(dest: pathlib.Path, slug: str, extension: str) -> pathlib.Path:
"""Return ``dest`` or the first free ``<slug>-N`` name beside it.
Generated by AI/LLM: 2026-09-30 - never overwrite a raw file.
"""
if not dest.exists():
return dest
number = 2
while True:
candidate = dest.parent / f"{slug}-{number}{extension}"
if not candidate.exists():
return candidate
number += 1
def _strip_prefix(value: str, prefix: str) -> str:
"""Drop a leading ``raw/`` or ``wiki/`` from a user path.
Generated by AI/LLM: 2026-09-30 - both forms are accepted.
"""
relative = value.replace("\\", "/").lstrip("/")
marker = f"{prefix}/"
if relative == prefix:
return ""
if relative.startswith(marker):
return relative[len(marker):]
return relative
def _require_inside(root: pathlib.Path, value: str, message: str) -> pathlib.Path:
"""Resolve ``value`` under ``root`` or raise.
Generated by AI/LLM: 2026-09-30 - ``..`` and symlinks must stay inside.
"""
relative = _strip_prefix(value, root.name)
candidate = (root / relative).resolve()
if not _is_inside(candidate, root):
raise IngestError(f"{message}: {value}")
return candidate
def _is_inside(path: pathlib.Path, root: pathlib.Path) -> bool:
"""Return whether ``path`` is ``root`` or a file inside it.
Generated by AI/LLM: 2026-09-30 - compare resolved paths.
"""
try:
path.resolve().relative_to(root.resolve())
except ValueError:
return False
return True
def _relative(path: pathlib.Path, root: pathlib.Path) -> str:
"""Return a wiki-relative path with ``/`` separators.
Generated by AI/LLM: 2026-09-30 - JSON and log paths.
"""
return path.resolve().relative_to(root.resolve()).as_posix()
@@ -1,38 +0,0 @@
"""Shipped converters, in registration order.
Generated by AI/LLM: 2026-09-30 - markdown, text, html, then pdf.
Modified by AI/LLM: 2026-10-07 - the html converter takes the block sweep settings.
"""
from fiona.wiki.ingest.converters.base import ConverterRegistry
from fiona.wiki.ingest.converters.html import HtmlConverter
from fiona.wiki.ingest.converters.markdown import MarkdownConverter
from fiona.wiki.ingest.converters.pdf import PdfConverter
from fiona.wiki.ingest.converters.text import TextConverter
from fiona.wiki.ingest.settings import IngestSettings, default_settings
def default_registry(settings: IngestSettings | None = None) -> ConverterRegistry:
"""Return the converters shipped with wiki ingest.
Generated by AI/LLM: 2026-09-30 - registration order from SPEC 7.1.5.
Modified by AI/LLM: 2026-10-07 - ``html.*`` settings reach ``HtmlConverter``.
Args:
settings: Ingest settings. ``None`` uses the defaults.
Returns:
ConverterRegistry: Markdown, text, HTML, and PDF.
"""
settings = settings or default_settings()
registry = ConverterRegistry()
registry.register(MarkdownConverter)
registry.register(TextConverter)
registry.register(
HtmlConverter(
min_words=settings.html_min_words,
max_link_density=settings.html_max_link_density,
)
)
registry.register(PdfConverter)
return registry
-108
View File
@@ -1,108 +0,0 @@
"""Converter base types and the registry.
Generated by AI/LLM: 2026-09-30 - one class per format, SPEC 7.1.5.1.
"""
from __future__ import annotations
import abc
import typing
from fiona.wiki.ingest.models import ConversionResult, FetchedSource
class ConversionError(Exception):
"""Raised when a converter cannot read a file it claims.
Generated by AI/LLM: 2026-09-30 - exit 3, raw copy kept.
"""
def __init__(self, converter: str, reason: str) -> None:
"""Store the converter name and the reason.
Args:
converter: Converter ``name`` attribute.
reason: Short failure text, without the ``Error:`` prefix.
"""
super().__init__(f"{converter}: {reason}")
self.converter = converter
self.reason = reason
class SourceConverter(abc.ABC):
"""One format. Subclasses declare what they claim and implement ``convert``.
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.1.2.
"""
name: typing.ClassVar[str]
media_types: typing.ClassVar[tuple[str, ...]]
extensions: typing.ClassVar[tuple[str, ...]]
raw_category: typing.ClassVar[str]
@abc.abstractmethod
def convert(self, source: FetchedSource) -> ConversionResult:
"""Convert an archived file to markdown.
Args:
source: Fetched file, including media type and slug.
Returns:
ConversionResult: Body, title, metadata, and warnings.
Raises:
ConversionError: The file cannot be read.
"""
class ConverterRegistry:
"""Ordered list of converters.
Generated by AI/LLM: 2026-09-30 - first media-type match, then extension.
"""
def __init__(self, converters: list[SourceConverter] | None = None) -> None:
"""Start with an optional list of converter instances.
Args:
converters: Instances in registration order.
"""
self.converters: list[SourceConverter] = list(converters or [])
def register(self, converter: SourceConverter | type[SourceConverter]) -> None:
"""Append a converter class or instance.
Generated by AI/LLM: 2026-09-30 - adding a format is one registration.
Args:
converter: Class with a no-arg constructor, or a ready instance.
"""
if isinstance(converter, type):
converter = converter()
self.converters.append(converter)
def find(self, media_type: str, extension: str) -> SourceConverter | None:
"""Pick the converter for a media type and extension.
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.4.1 registration order.
Args:
media_type: Detected type, or empty.
extension: Suffix including the dot, or empty.
Returns:
SourceConverter | None: The first match, or None.
"""
media = (media_type or "").lower()
ext = extension.lower()
if ext and not ext.startswith("."):
ext = f".{ext}"
if media:
for converter in self.converters:
if media in converter.media_types:
return converter
if ext:
for converter in self.converters:
if ext in converter.extensions:
return converter
return None
-141
View File
@@ -1,141 +0,0 @@
"""HTML source converter.
Generated by AI/LLM: 2026-09-30 - article text via beautifulsoup4 and markdownify.
Modified by AI/LLM: 2026-10-07 - the block sweep replaces the first-``<article>`` root.
"""
from __future__ import annotations
import typing
import urllib.parse
import bs4
from fiona.wiki.ingest.converters.base import SourceConverter
from fiona.wiki.ingest.converters.html_sweep import sweep, unwrap_shadow_roots
from fiona.wiki.ingest.media import decode_bytes
from fiona.wiki.ingest.models import ConversionResult, FetchedSource
_DROP_TAGS = ("script", "style", "noscript", "template")
_NO_TEXT = "no text content; the page may need JavaScript or a login"
class HtmlConverter(SourceConverter):
"""Convert an HTML document to markdown.
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.4.5.
Modified by AI/LLM: 2026-10-07 - sweep limits are constructor arguments.
"""
name: typing.ClassVar[str] = "html"
media_types: typing.ClassVar[tuple[str, ...]] = ("text/html", "application/xhtml+xml")
extensions: typing.ClassVar[tuple[str, ...]] = (".html", ".htm", ".xhtml")
raw_category: typing.ClassVar[str] = "articles"
def __init__(self, min_words: int = 4, max_link_density: float = 0.5) -> None:
"""Store the block sweep limits.
Generated by AI/LLM: 2026-10-07 - defaults from SPEC 5.1 ``fiona.wiki.ingest.html``.
Args:
min_words: Fewer words make a ``p`` or loose block ``short``.
max_link_density: Link text over text length above this is ``links``.
"""
self.min_words = min_words
self.max_link_density = max_link_density
def convert(self, source: FetchedSource) -> ConversionResult:
"""Parse HTML and return every swept block of the body as markdown.
Generated by AI/LLM: 2026-09-30 - drop scripts and absolutize links.
Modified by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6 block sweep over ``<body>``.
Args:
source: Archived HTML file. ``origin`` is the page URL when fetched.
Returns:
ConversionResult: Markdown body, title, and ``source_meta``.
"""
text, warnings = decode_bytes(source.path.read_bytes(), source.charset)
soup = bs4.BeautifulSoup(text, "html.parser")
unwrap_shadow_roots(soup)
for tag in soup.find_all(_DROP_TAGS):
tag.decompose()
for comment in soup.find_all(string=lambda node: isinstance(node, bs4.Comment)):
comment.extract()
body = soup.body or soup
origin = source.origin if source.origin_kind == "url" else None
_absolutize(body, soup, origin)
conversion = "complete"
if not body.get_text(strip=True):
conversion = "partial"
warnings = [*warnings, _NO_TEXT]
swept = sweep(body, min_words=self.min_words, max_link_density=self.max_link_density)
if swept.partial:
conversion = "partial"
return ConversionResult(
body=swept.body,
title=_html_title(soup, body),
source_meta=_source_meta(soup),
conversion=conversion,
warnings=[*warnings, *swept.warnings],
converter=self.name,
)
def _absolutize(root: bs4.element.Tag, soup: bs4.BeautifulSoup, origin: str | None) -> None:
"""Rewrite relative href and src values against base href, then the origin.
Generated by AI/LLM: 2026-09-30 - a local file with no origin keeps relative links.
"""
base_tag = soup.find("base", href=True)
base = base_tag.get("href") if base_tag is not None else None
if not base:
base = origin
if not base:
return
for tag in root.find_all(href=True):
tag["href"] = urllib.parse.urljoin(str(base), str(tag["href"]))
for tag in root.find_all(src=True):
tag["src"] = urllib.parse.urljoin(str(base), str(tag["src"]))
def _html_title(soup: bs4.BeautifulSoup, root: bs4.element.Tag) -> str | None:
"""Return ``<title>``, then ``og:title``, then the first ``<h1>`` that has text.
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.4.5.1.
Modified by AI/LLM: 2026-10-07 - the root is ``<body>``, so a blank ``<h1>`` is skipped.
"""
if soup.title and soup.title.string and soup.title.string.strip():
return soup.title.string.strip()
og = soup.find("meta", attrs={"property": "og:title"})
if og is not None and og.get("content"):
return str(og["content"]).strip() or None
for heading in root.find_all("h1"):
text = heading.get_text(strip=True)
if text:
return text
return None
def _source_meta(soup: bs4.BeautifulSoup) -> dict:
"""Collect language, canonical URL, and selected meta values.
Generated by AI/LLM: 2026-09-30 - only keys that are present.
"""
meta: dict[str, str] = {}
root = soup.find("html")
if root is not None and root.get("lang"):
meta["lang"] = str(root["lang"])
canonical = soup.find("link", rel=lambda value: value and "canonical" in value)
if canonical is not None and canonical.get("href"):
meta["canonical"] = str(canonical["href"])
for tag in soup.find_all("meta"):
key = tag.get("name") or tag.get("property")
content = tag.get("content")
if not key or content is None:
continue
name = str(key)
if name in {"description", "author"} or name.startswith("og:") or name == "article:published_time":
meta[name] = str(content)
return meta
@@ -1,285 +0,0 @@
"""Score every block of an HTML body as ``keep``, ``review``, or ``drop``.
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6. Rules read HTML structure only.
"""
from __future__ import annotations
import dataclasses
import re
import typing
import bs4
import markdownify
_BLOCK_KINDS = {
"h1": "heading",
"h2": "heading",
"h3": "heading",
"h4": "heading",
"h5": "heading",
"h6": "heading",
"p": "p",
"ul": "list",
"ol": "list",
"dl": "list",
"blockquote": "quote",
"pre": "pre",
"table": "table",
"figure": "figure",
}
_INLINE_TAGS = frozenset((
"a", "abbr", "b", "bdi", "br", "cite", "code", "data", "del", "dfn", "em",
"font", "i", "img", "ins", "kbd", "label", "mark", "q", "s", "samp", "small",
"span", "strong", "sub", "sup", "time", "u", "var", "wbr",
))
_LAYOUT_CELL_CHILDREN = (*_BLOCK_KINDS, "div")
_SKIPPED_TAGS = frozenset(("head",))
_DROP_ANCESTORS = frozenset(("nav", "button", "select"))
_CHROME_TAGS = frozenset(("aside", "form", "dialog"))
_CHROME_ROLES = frozenset(("navigation", "banner", "contentinfo", "complementary"))
_HIDDEN_STYLE = re.compile(r"display\s*:\s*none|visibility\s*:\s*hidden", re.IGNORECASE)
_SHORT_KINDS = frozenset(("p", "loose"))
_ALL_REVIEW = "no block passed the content checks; every block is marked for review"
@dataclasses.dataclass
class Block:
"""One swept block: an element, or a run of loose text and inline nodes.
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6.1 and 7.1.5.4.5.6.2.
"""
nodes: list[bs4.PageElement]
kind: str
text: str = ""
link_chars: int = 0
words: int = 0
verdict: str = ""
reason: str = ""
@dataclasses.dataclass
class SweepResult:
"""Markdown body, warnings, and whether every block needs review.
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6.6 and 7.1.5.4.5.6.7.
"""
body: str
warnings: list[str]
partial: bool
def unwrap_shadow_roots(soup: bs4.BeautifulSoup) -> None:
"""Read each declarative shadow root in place, so its text is swept.
Generated by AI/LLM: 2026-10-07 - ``template[shadowrootmode]``; other templates are dropped later.
Args:
soup: Parsed document, before the drop loop.
"""
for template in soup.find_all("template", attrs={"shadowrootmode": True}):
template.unwrap()
def sweep(root: bs4.element.Tag, *, min_words: int, max_link_density: float) -> SweepResult:
"""Walk ``root``, classify each block, and render the kept and review blocks.
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6. No clock, network, or random source.
Args:
root: ``<body>``, or the document when there is no body.
min_words: ``fiona.wiki.ingest.html.min_words``.
max_link_density: ``fiona.wiki.ingest.html.max_link_density``.
Returns:
SweepResult: Markdown body, the ``html sweep:`` warning, and the partial flag.
"""
blocks: list[Block] = []
_walk(root, blocks)
seen: set[str] = set()
for block in blocks:
_measure(block)
block.verdict, block.reason = _classify(block, seen, min_words, max_link_density)
if block.verdict != "drop":
seen.add(block.text.casefold())
counts = {verdict: sum(block.verdict == verdict for block in blocks) for verdict in ("keep", "review", "drop")}
warnings = [f"html sweep: {counts['keep']} keep, {counts['review']} review, {counts['drop']} drop"]
partial = counts["keep"] == 0 and counts["review"] > 0
if partial:
warnings.append(_ALL_REVIEW)
parts = [part for part in (_render(block) for block in blocks if block.verdict != "drop") if part]
body = "\n\n".join(parts)
return SweepResult(body=body + "\n" if body else "", warnings=warnings, partial=partial)
def _walk(container: bs4.element.Tag, blocks: list[Block]) -> None:
"""Append the blocks of ``container`` in document order.
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6.1 to 7.1.5.4.5.6.1.2.
"""
run: list[bs4.PageElement] = []
for child in list(container.children):
if not isinstance(child, bs4.element.Tag):
if isinstance(child, bs4.element.NavigableString):
run.append(child)
continue
if child.name in _SKIPPED_TAGS:
continue
if _is_block(child):
_flush(run, blocks)
blocks.append(Block(nodes=[child], kind=_BLOCK_KINDS[child.name]))
elif child.name in _INLINE_TAGS and not _holds_structure(child):
run.append(child)
else:
_flush(run, blocks)
_walk(child, blocks)
_flush(run, blocks)
def _flush(run: list[bs4.PageElement], blocks: list[Block]) -> None:
"""Close a loose-text run as one block, unless it holds no text.
Generated by AI/LLM: 2026-10-07 - whitespace between tags is not a block.
"""
if run and _fold(_raw_text(run)):
blocks.append(Block(nodes=list(run), kind="loose"))
run.clear()
def _is_block(tag: bs4.element.Tag) -> bool:
"""Return whether ``tag`` is a block. A layout table is a container instead.
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6.1 layout tables.
"""
if tag.name not in _BLOCK_KINDS:
return False
if tag.name != "table":
return True
return not any(cell.find(_LAYOUT_CELL_CHILDREN) for cell in tag.find_all(("td", "th")))
def _holds_structure(tag: bs4.element.Tag) -> bool:
"""Return whether an inline tag wraps a block or container, like a card link.
Generated by AI/LLM: 2026-10-07 - ``<a><div>...</div></a>`` is walked as a container.
"""
return tag.find(lambda node: node.name not in _INLINE_TAGS) is not None
def _raw_text(nodes: typing.Iterable[bs4.PageElement]) -> str:
"""Return the text of ``nodes`` with no separator and ``<br>`` as a space.
Generated by AI/LLM: 2026-10-07 - keeps ``<b>Fa</b>st`` as one word.
"""
parts: list[str] = []
for node in nodes:
items = [node] if isinstance(node, bs4.element.NavigableString) else [node, *node.descendants]
for item in items:
if isinstance(item, bs4.element.NavigableString):
parts.append(str(item))
elif item.name == "br":
parts.append(" ")
return "".join(parts)
def _fold(text: str) -> str:
"""Fold whitespace runs to one space and trim.
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6.2.
"""
return " ".join(text.split())
def _measure(block: Block) -> None:
"""Set the folded text, link text length, and word count of a block.
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6.2.
"""
block.text = _fold(_raw_text(block.nodes))
links: list[bs4.element.Tag] = []
for node in block.nodes:
if isinstance(node, bs4.element.Tag):
links.extend([node] if node.name == "a" else node.find_all("a"))
block.link_chars = sum(len(_fold(_raw_text([link]))) for link in links)
block.words = len(block.text.split())
def _classify(block: Block, seen: set[str], min_words: int, max_link_density: float) -> tuple[str, str]:
"""Return ``(verdict, reason)`` with the first rule that holds.
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6.3 to 7.1.5.4.5.6.5.
"""
folded = block.text.casefold()
if not any(char.isalnum() for char in block.text):
return "drop", ""
if block.words < min_words and folded in seen:
return "drop", ""
if _inside(block, _drops):
return "drop", ""
if _inside(block, _is_chrome):
return "review", "chrome"
if folded in seen:
return "review", "repeat"
if block.link_chars / len(block.text) > max_link_density:
return "review", "links"
if block.kind in _SHORT_KINDS and block.words < min_words:
return "review", "short"
return "keep", ""
def _inside(block: Block, predicate: typing.Callable[[bs4.element.Tag], bool]) -> bool:
"""Return whether the block, or an ancestor below the document, matches.
Generated by AI/LLM: 2026-10-07 - an element block checks itself; a loose run starts at its parent.
"""
first = block.nodes[0]
node = first.parent if block.kind == "loose" else first
while isinstance(node, bs4.element.Tag) and not isinstance(node, bs4.BeautifulSoup):
if predicate(node):
return True
node = node.parent
return False
def _drops(tag: bs4.element.Tag) -> bool:
"""Return whether text under ``tag`` is never content: nav, controls, hidden.
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6.3.
"""
if tag.name in _DROP_ANCESTORS or tag.has_attr("hidden"):
return True
if str(tag.get("aria-hidden", "")).strip().lower() == "true":
return True
return bool(_HIDDEN_STYLE.search(str(tag.get("style", ""))))
def _is_chrome(tag: bs4.element.Tag) -> bool:
"""Return whether ``tag`` is page chrome: aside, form, landmark role, outer header or footer.
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6.4 ``chrome``.
"""
if tag.name in _CHROME_TAGS:
return True
if str(tag.get("role", "")).strip().lower() in _CHROME_ROLES:
return True
if tag.name in ("header", "footer"):
return tag.find_parent(("article", "main")) is None
return False
def _render(block: Block) -> str:
"""Convert one block to markdown and wrap a ``review`` block in its markers.
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6.6, options of 7.1.5.4.5.4.
"""
markup = "".join(
node.output_ready(formatter="minimal") if isinstance(node, bs4.element.NavigableString) else str(node)
for node in block.nodes
)
heading = getattr(markdownify, "ATX", "ATX")
text = markdownify.markdownify(markup, heading_style=heading, bullets="-").strip()
if not text or block.verdict != "review":
return text
return f"<!-- fiona:review {block.reason} -->\n\n{text}\n\n<!-- /fiona:review -->"
@@ -1,105 +0,0 @@
"""Markdown source converter.
Generated by AI/LLM: 2026-09-30 - keep the body and lift frontmatter into source_meta.
"""
from __future__ import annotations
import re
import typing
import yaml
from fiona.wiki.ingest.converters.base import SourceConverter
from fiona.wiki.ingest.media import decode_bytes
from fiona.wiki.ingest.models import ConversionResult, FetchedSource
_FRONTMATTER = re.compile(r"\A---\r?\n(.*?)\r?\n---\r?\n?", re.DOTALL)
_H1 = re.compile(r"^#\s+(\S.*)$")
_FENCE = re.compile(r"^\s*(`{3,}|~{3,})")
class MarkdownConverter(SourceConverter):
"""Keep markdown as it is and record any source frontmatter.
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.4.3.
The body after a frontmatter block is kept character for character.
The page writer adds the wiki frontmatter when the new page does not
have it yet. Source keys move to ``source_meta`` instead of being
written a second time as the page frontmatter.
"""
name: typing.ClassVar[str] = "markdown"
media_types: typing.ClassVar[tuple[str, ...]] = ("text/markdown",)
extensions: typing.ClassVar[tuple[str, ...]] = (".md", ".markdown")
raw_category: typing.ClassVar[str] = "articles"
def convert(self, source: FetchedSource) -> ConversionResult:
"""Return the markdown body and any frontmatter as metadata.
Generated by AI/LLM: 2026-09-30 - title from frontmatter, then H1, then slug.
Args:
source: Archived markdown file.
Returns:
ConversionResult: Unchanged body, ``complete`` unless decoding fell back.
"""
text, warnings = decode_bytes(source.path.read_bytes(), source.charset)
meta: dict = {}
body = text
match = _FRONTMATTER.match(text)
if match:
loaded = _load_meta(match.group(1))
if loaded is None:
warnings = [*warnings, "source frontmatter did not parse"]
else:
meta = loaded
body = text[match.end():]
title = _title(meta, body, source.slug)
return ConversionResult(
body=body,
title=title,
source_meta=meta,
conversion="complete",
warnings=warnings,
converter=self.name,
)
def _load_meta(block: str) -> dict | None:
"""Parse a frontmatter block into a dict.
Generated by AI/LLM: 2026-09-30 - invalid YAML stays in the body.
"""
try:
loaded = yaml.safe_load(block)
except yaml.YAMLError:
return None
if loaded is None:
return {}
if isinstance(loaded, dict):
return loaded
return None
def _title(meta: dict, body: str, slug: str) -> str:
"""Choose the page title.
Generated by AI/LLM: 2026-09-30 - frontmatter title, first H1, then slug.
"""
raw_title = meta.get("title")
if isinstance(raw_title, str) and raw_title.strip():
return raw_title.strip()
fenced = False
for line in body.splitlines():
if _FENCE.match(line):
fenced = not fenced
continue
if fenced:
continue
match = _H1.match(line)
if match:
return match.group(1).strip()
return slug
-186
View File
@@ -1,186 +0,0 @@
"""PDF source converter.
Generated by AI/LLM: 2026-09-30 - text layer via pypdf, SPEC 7.1.5.4.6.
"""
from __future__ import annotations
import datetime
import typing
import pypdf
import pypdf.errors
from fiona.wiki.ingest.converters.base import ConversionError, SourceConverter
from fiona.wiki.ingest.models import ConversionResult, FetchedSource
_SCANNED = "no text layer; the PDF may be scanned"
class PdfConverter(SourceConverter):
"""Read a PDF text layer into markdown headings and paragraphs.
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.4.6.
"""
name: typing.ClassVar[str] = "pdf"
media_types: typing.ClassVar[tuple[str, ...]] = ("application/pdf",)
extensions: typing.ClassVar[tuple[str, ...]] = (".pdf",)
raw_category: typing.ClassVar[str] = "papers"
def convert(self, source: FetchedSource) -> ConversionResult:
"""Extract page text, outline headings, and document metadata.
Generated by AI/LLM: 2026-09-30 - encrypted or corrupt files raise.
Args:
source: Archived PDF.
Returns:
ConversionResult: Markdown body. Empty pages make it ``partial``.
Raises:
ConversionError: The PDF is encrypted or cannot be parsed.
"""
try:
reader = pypdf.PdfReader(str(source.path))
except pypdf.errors.PdfReadError as exc:
raise ConversionError(self.name, str(exc)) from exc
if reader.is_encrypted:
raise ConversionError(self.name, "encrypted PDF")
meta = _metadata(reader)
title = meta.get("title") or source.slug
outline = _outline(reader)
pages = []
empty: list[int] = []
for index, page in enumerate(reader.pages, start=1):
try:
text = page.extract_text() or ""
except pypdf.errors.PdfReadError as exc:
raise ConversionError(self.name, str(exc)) from exc
pages.append(text.strip())
if not text.strip():
empty.append(index)
body = _render(pages, outline)
warnings: list[str] = []
conversion = "complete"
if empty and len(empty) == len(pages):
warnings.append(_SCANNED)
conversion = "partial"
elif empty:
warnings.extend(f"page {number} has no text layer" for number in empty)
conversion = "partial"
return ConversionResult(
body=body,
title=str(title),
source_meta=meta,
conversion=conversion,
warnings=warnings,
converter=self.name,
)
def _metadata(reader: pypdf.PdfReader) -> dict:
"""Return page count and the document info values that are present.
Generated by AI/LLM: 2026-09-30 - dates as ISO 8601.
"""
meta: dict[str, object] = {"page_count": len(reader.pages)}
info = reader.metadata
if info is None:
return meta
mapping = (
("title", info.title),
("author", info.author),
("subject", info.subject),
("keywords", info.keywords),
("creation_date", _iso_date(info.creation_date)),
("mod_date", _iso_date(info.modification_date)),
)
for key, value in mapping:
if value:
meta[key] = value
return meta
def _iso_date(value: object) -> str | None:
"""Format a PDF date as ISO 8601.
Generated by AI/LLM: 2026-09-30 - accept datetime values and ``D:`` strings.
"""
if value is None:
return None
if isinstance(value, datetime.datetime):
if value.tzinfo is None:
return value.strftime("%Y-%m-%dT%H:%M:%SZ")
return value.astimezone(datetime.UTC).strftime("%Y-%m-%dT%H:%M:%SZ")
text = str(value).strip().removeprefix("D:")
digits = "".join(char for char in text if char.isdigit())
if len(digits) < 8:
return text or None
year, month, day = digits[0:4], digits[4:6], digits[6:8]
hour = digits[8:10] or "00"
minute = digits[10:12] or "00"
second = digits[12:14] or "00"
return f"{year}-{month}-{day}T{hour}:{minute}:{second}Z"
def _outline(reader: pypdf.PdfReader) -> list[tuple[int, int, str]]:
"""Return outline entries as page index, heading level, and title.
Generated by AI/LLM: 2026-09-30 - level is depth plus 2, capped at 6.
"""
found: list[tuple[int, int, str]] = []
try:
items = reader.outline
except pypdf.errors.PdfReadError:
return found
_walk_outline(reader, items, 0, found)
return found
def _walk_outline(
reader: pypdf.PdfReader,
items: object,
depth: int,
found: list[tuple[int, int, str]],
) -> None:
"""Walk a pypdf outline tree.
Generated by AI/LLM: 2026-09-30 - nested lists are deeper entries.
"""
if not items:
return
for item in items: # type: ignore[union-attr]
if isinstance(item, list):
_walk_outline(reader, item, depth + 1, found)
continue
try:
page_index = reader.get_destination_page_number(item)
except (ValueError, KeyError, pypdf.errors.PdfReadError):
continue
title = getattr(item, "title", "") or ""
level = min(depth + 2, 6)
found.append((page_index, level, str(title).strip() or "Untitled"))
def _render(pages: list[str], outline: list[tuple[int, int, str]]) -> str:
"""Render pages with outline headings or ``## Page n`` headings.
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.4.6.2.
"""
by_page: dict[int, list[str]] = {}
for page_index, level, title in outline:
by_page.setdefault(page_index, []).append(f"{'#' * level} {title}")
chunks: list[str] = []
for index, text in enumerate(pages):
if outline:
chunks.extend(by_page.get(index, []))
else:
chunks.append(f"## Page {index + 1}")
if text:
chunks.append(text)
body = "\n\n".join(chunks)
if body:
body += "\n"
return body
-68
View File
@@ -1,68 +0,0 @@
"""Plain-text source converter.
Generated by AI/LLM: 2026-09-30 - wrap text in a fence; the agent refines the page.
"""
from __future__ import annotations
import typing
from fiona.wiki.ingest.converters.base import SourceConverter
from fiona.wiki.ingest.media import decode_bytes
from fiona.wiki.ingest.models import ConversionResult, FetchedSource
class TextConverter(SourceConverter):
"""Turn plain text into one fenced markdown block.
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.4.4.
The page is written at ``status: raw``. The tool does not summarize
the text. The agent rewrites the fence during refinement.
"""
name: typing.ClassVar[str] = "text"
media_types: typing.ClassVar[tuple[str, ...]] = ("text/plain",)
extensions: typing.ClassVar[tuple[str, ...]] = (".txt", ".text", ".log")
raw_category: typing.ClassVar[str] = "articles"
def convert(self, source: FetchedSource) -> ConversionResult:
"""Decode the file and wrap it in a fence longer than any backtick run.
Generated by AI/LLM: 2026-09-30 - title is the slug; fallback decode warns.
Args:
source: Archived text file.
Returns:
ConversionResult: Fenced body and the slug as the title.
"""
text, warnings = decode_bytes(source.path.read_bytes(), source.charset)
marker = "`" * _fence_length(text)
if text and not text.endswith("\n"):
text += "\n"
body = f"{marker}text\n{text}{marker}\n"
return ConversionResult(
body=body,
title=source.slug,
source_meta={},
conversion="complete",
warnings=warnings,
converter=self.name,
)
def _fence_length(text: str) -> int:
"""Return a fence length longer than the longest backtick run.
Generated by AI/LLM: 2026-09-30 - a run of three when the text has fewer.
"""
longest = 0
current = 0
for char in text:
if char == "`":
current += 1
longest = max(longest, current)
else:
current = 0
return max(3, longest + 1)
-316
View File
@@ -1,316 +0,0 @@
"""Fetch a local file or an HTTP URL.
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.2. No credentials are sent.
Modified by AI/LLM: 2026-10-07 - login detection, SPEC 7.1.5.2.8.
"""
from __future__ import annotations
import http
import logging
import pathlib
import re
import urllib.error
import urllib.parse
import urllib.request
from fiona.wiki.ingest.converters.base import ConverterRegistry
from fiona.wiki.ingest.media import (
detect_media_type,
extension_for,
slug_from_file,
slug_from_url,
utc_now,
)
from fiona.wiki.ingest.models import AuthRequiredError, FetchedSource, IngestError
from fiona.wiki.ingest.settings import IngestSettings, default_settings
logger = logging.getLogger(__name__)
_SCHEME = re.compile(r"^([A-Za-z][A-Za-z0-9+.-]*)://")
_WINDOWS_DRIVE = re.compile(r"^[A-Za-z]:[\\/]")
_AUTH_STATUSES = (401, 403)
_PASSWORD_INPUT = re.compile(rb"<input\b[^>]*?\btype\s*=\s*[\"']?password\b", re.IGNORECASE)
def url_scheme(argument: str) -> str | None:
"""Return the URL scheme, or None for a local path.
Generated by AI/LLM: 2026-09-30 - a Windows drive letter is not a scheme.
Args:
argument: The ``<source>`` argument as given.
Returns:
str | None: Scheme text, preserving case, or None.
"""
if _WINDOWS_DRIVE.match(argument):
return None
match = _SCHEME.match(argument)
if match is None:
return None
return match.group(1)
class SourceFetcher:
"""Base fetcher. Subclasses accept an argument and return a local file.
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.1.3.
"""
settings: IngestSettings | None = None
def accepts(self, argument: str) -> bool:
"""Return whether this fetcher handles the argument.
Args:
argument: The ``<source>`` argument as given.
Returns:
bool: True when ``fetch`` can run.
"""
raise NotImplementedError
def fetch(self, argument: str, dest_dir: pathlib.Path) -> FetchedSource:
"""Fetch the argument into ``dest_dir`` when a download is required.
Args:
argument: The ``<source>`` argument as given.
dest_dir: Directory for a downloaded file. Local fetches ignore it.
Returns:
FetchedSource: Local path and origin facts.
"""
raise NotImplementedError
class LocalFileFetcher(SourceFetcher):
"""Read a local file in place.
Generated by AI/LLM: 2026-09-30 - missing files and directories exit 1.
"""
def __init__(self, registry: ConverterRegistry) -> None:
"""Store the registry used to name the media type.
Args:
registry: Converter registry for extension lookup.
"""
self.registry = registry
self.settings = None
def accepts(self, argument: str) -> bool:
"""Accept any argument that is not a URL.
Generated by AI/LLM: 2026-09-30 - scheme detection stays in ``url_scheme``.
Args:
argument: The ``<source>`` argument as given.
Returns:
bool: True when there is no ``://`` scheme.
"""
return url_scheme(argument) is None
def fetch(self, argument: str, dest_dir: pathlib.Path) -> FetchedSource:
"""Resolve a local path without copying it.
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.2.1.
Args:
argument: Local path as given.
dest_dir: Unused. Present so every fetcher shares one signature.
Returns:
FetchedSource: The resolved file. ``created`` is false.
Raises:
IngestError: The path is missing or is a directory.
"""
del dest_dir
path = pathlib.Path(argument)
if not path.exists():
raise IngestError(f"Source not found: {argument}")
if path.is_dir():
raise IngestError(f"Source is a directory: {argument}")
data = path.read_bytes()
media, charset = detect_media_type(data, path.suffix, None, self.registry)
resolved = path.resolve()
logger.info("fetched local file %s as %s", resolved, media)
return FetchedSource(
argument=argument,
path=resolved,
origin=str(resolved),
origin_kind="file",
final_url=None,
media_type=media,
charset=charset,
size_bytes=len(data),
slug=slug_from_file(path),
fetched=utc_now(),
created=False,
)
class UrlFetcher(SourceFetcher):
"""Download one ``http`` or ``https`` URL.
Generated by AI/LLM: 2026-09-30 - opener is injected; tests make no network call.
"""
def __init__(
self,
registry: ConverterRegistry,
opener: urllib.request.OpenerDirector | None = None,
) -> None:
"""Store the registry and the URL opener.
Args:
registry: Converter registry for the saved extension.
opener: ``urllib`` opener. The default follows redirects and proxies.
"""
self.registry = registry
self.opener = opener or urllib.request.build_opener()
self.settings = None
def accepts(self, argument: str) -> bool:
"""Accept ``http`` and ``https`` only.
Generated by AI/LLM: 2026-09-30 - other schemes are rejected by the pipeline.
Args:
argument: The ``<source>`` argument as given.
Returns:
bool: True for http and https.
"""
scheme = url_scheme(argument)
return scheme is not None and scheme.lower() in {"http", "https"}
def fetch(self, argument: str, dest_dir: pathlib.Path) -> FetchedSource:
"""Download a URL into ``dest_dir`` and stop past the size limit.
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.2.2 and 7.1.5.2.3.
Args:
argument: Absolute URL.
dest_dir: Directory that receives ``<slug>.<ext>``.
Returns:
FetchedSource: The saved file. ``created`` is true.
Raises:
IngestError: The response is not 2xx, the network fails, or the body is too large.
"""
settings = self.settings or default_settings()
requested = without_userinfo(argument)
request = urllib.request.Request(
requested,
headers={"User-Agent": settings.user_agent},
)
try:
response = self.opener.open(request, timeout=settings.timeout_seconds)
except urllib.error.HTTPError as exc:
if exc.code in _AUTH_STATUSES:
phrase = http.HTTPStatus(exc.code).phrase
raise AuthRequiredError(requested, f"{exc.code} {phrase}") from exc
raise IngestError(f"Download failed: {argument}: {exc.code} {exc.reason}") from exc
except urllib.error.URLError as exc:
raise IngestError(f"Download failed: {argument}: {exc.reason}") from exc
with response:
final_url = response.geturl()
header = response.headers.get("Content-Type")
body = _read_limited(response, settings.max_download_bytes, argument)
parsed = urllib.parse.urlsplit(argument)
suffix = pathlib.PurePosixPath(parsed.path).suffix
media, charset = detect_media_type(body, suffix, header, self.registry)
_check_login(requested, final_url, media, body, settings.login_url_patterns)
extension = extension_for(media, suffix, self.registry)
slug = slug_from_url(argument)
dest_dir.mkdir(parents=True, exist_ok=True)
target = dest_dir / f"{slug}{extension}"
partial = target.with_name(target.name + ".partial")
try:
partial.write_bytes(body)
partial.replace(target)
except OSError:
if partial.exists():
partial.unlink()
raise
logger.info("downloaded %s as %s (%s bytes)", argument, media, len(body))
return FetchedSource(
argument=argument,
path=target,
origin=argument,
origin_kind="url",
final_url=final_url,
media_type=media,
charset=charset,
size_bytes=len(body),
slug=slug,
fetched=utc_now(),
created=True,
)
def _check_login(
requested: str,
final_url: str,
media: str,
body: bytes,
patterns: list[str],
) -> None:
"""Raise when a 2xx answer is a login redirect or a login form.
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.2.8.2 redirect, then login form.
Args:
requested: URL that was requested, without userinfo.
final_url: URL after redirects.
media: Detected media type of the body.
body: Response body, already within the size limit.
patterns: ``login_url_patterns`` from the settings.
Raises:
AuthRequiredError: The answer is a login page, not the source.
"""
if final_url and final_url != requested:
parts = urllib.parse.urlsplit(final_url)
target = f"{parts.hostname or ''}{parts.path}"
if any(re.search(pattern, target, re.IGNORECASE) for pattern in patterns):
raise AuthRequiredError(requested, f"redirected to {final_url}")
if media == "text/html" and _PASSWORD_INPUT.search(body):
raise AuthRequiredError(requested, f"login page at {final_url or requested}")
def without_userinfo(url: str) -> str:
"""Drop userinfo so the request sends no credentials.
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.2.2.
Modified by AI/LLM: 2026-10-07 - public; also used for ``--origin``.
"""
parts = urllib.parse.urlsplit(url)
if parts.username is None and parts.password is None:
return url
host = parts.hostname or ""
if parts.port:
host = f"{host}:{parts.port}"
return urllib.parse.urlunsplit((parts.scheme, host, parts.path, parts.query, parts.fragment))
def _read_limited(response: object, limit: int, url: str) -> bytes:
"""Read a response body, refusing anything over ``limit``.
Generated by AI/LLM: 2026-09-30 - a partial download is not returned.
"""
chunks: list[bytes] = []
total = 0
while True:
block = response.read(65536) # type: ignore[attr-defined]
if not block:
break
total += len(block)
if total > limit:
raise IngestError(f"Download exceeds {limit} bytes: {url}")
chunks.append(block)
return b"".join(chunks)
-96
View File
@@ -1,96 +0,0 @@
"""Append one ingest line to ``wiki/log.md``.
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.6. Does not edit ``index.md``.
"""
from __future__ import annotations
import logging
import pathlib
import re
logger = logging.getLogger(__name__)
_VERSION = re.compile(r"^version:\s*(\S+)\s*$", re.MULTILINE)
_UPDATED = re.compile(r"^updated:\s*(\S+)\s*$", re.MULTILINE)
class IngestLog:
"""Record a finished archive in ``wiki/log.md``.
Generated by AI/LLM: 2026-09-30 - bump patch version and ``updated``.
"""
def __init__(self, wiki_root: pathlib.Path) -> None:
"""Store the wiki root.
Args:
wiki_root: Directory that contains ``wiki/log.md``.
"""
self.wiki_root = wiki_root
def append(
self,
*,
day: str,
origin: str,
raw_rel: str,
page_rel: str | None,
dry_run: bool,
) -> None:
"""Append one bullet and bump the log frontmatter.
Generated by AI/LLM: 2026-09-30 - ``(raw)`` or ``(no page)``.
Args:
day: UTC date ``YYYY-MM-DD``.
origin: URL, or file path (wiki-relative inside the wiki).
raw_rel: Raw path relative to the wiki root.
page_rel: Page path relative to the wiki root, or None.
dry_run: When true, leave the log untouched.
"""
if dry_run:
return
path = self.wiki_root / "wiki" / "log.md"
if page_rel:
bullet = f"- {day} - ingest {origin} -> {raw_rel} -> {page_rel} (raw)"
else:
bullet = f"- {day} - ingest {origin} -> {raw_rel} (no page)"
if not path.is_file():
logger.warning("wiki/log.md is missing; ingest was not recorded")
return
text = path.read_text(encoding="utf-8")
text = _bump_version(text)
text = _set_updated(text, day)
if not text.endswith("\n"):
text += "\n"
text += bullet + "\n"
path.write_text(text, encoding="utf-8", newline="\n")
logger.info("recorded %s", bullet)
def _bump_version(text: str) -> str:
"""Increment the patch number of the first ``version`` field.
Generated by AI/LLM: 2026-09-30 - ``2.0.0`` becomes ``2.0.1``.
"""
match = _VERSION.search(text)
if match is None:
return text
parts = match.group(1).split(".")
if len(parts) == 3 and all(part.isdigit() for part in parts):
parts[2] = str(int(parts[2]) + 1)
bumped = ".".join(parts)
return text[: match.start(1)] + bumped + text[match.end(1) :]
return text
def _set_updated(text: str, day: str) -> str:
"""Set the first ``updated`` field to ``day``.
Generated by AI/LLM: 2026-09-30 - replaces a placeholder or a previous date.
"""
match = _UPDATED.search(text)
if match is None:
return text
return text[: match.start(1)] + day + text[match.end(1) :]
-255
View File
@@ -1,255 +0,0 @@
"""Media-type detection and slug rules for wiki ingest.
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.2.4 and 7.1.5.2.6.
"""
from __future__ import annotations
import datetime
import mimetypes
import pathlib
import re
import urllib.parse
_SLUG_TRIM = 80
_UNUSABLE_MEDIA = {"", "application/octet-stream"}
def make_slug(text: str) -> str:
"""Turn free text into a wiki slug.
Generated by AI/LLM: 2026-09-30 - lowercase, hyphenate, and cap at 80.
Args:
text: Stem, host, or joined path segments.
Returns:
str: A slug, or ``source`` when the text has no usable characters.
"""
slug = re.sub(r"[^a-z0-9]+", "-", text.lower()).strip("-")[:_SLUG_TRIM]
return slug or "source"
def slug_from_url(url: str) -> str:
"""Build a slug from the last three path segments of a URL.
Generated by AI/LLM: 2026-09-30 - drop ``index`` and an empty path uses the host.
Args:
url: Absolute ``http`` or ``https`` URL.
Returns:
str: Slug for the downloaded file and the wiki page.
"""
parts = urllib.parse.urlsplit(url)
cleaned: list[str] = []
for segment in parts.path.split("/"):
if not segment:
continue
path = pathlib.PurePosixPath(segment)
stem = path.stem if path.suffix else segment
if stem.lower() == "index":
continue
cleaned.append(stem)
if not cleaned:
return make_slug(parts.hostname or "source")
return make_slug("-".join(cleaned[-3:]))
def slug_from_file(path: pathlib.Path) -> str:
"""Build a slug from a file stem.
Generated by AI/LLM: 2026-09-30 - local paths use the stem only.
Args:
path: Local file path. It does not have to exist.
Returns:
str: Slug for the archive name and the wiki page.
"""
return make_slug(path.stem)
def split_content_type(header: str | None) -> tuple[str | None, str | None]:
"""Split a Content-Type header into a media type and charset.
Generated by AI/LLM: 2026-09-30 - drop parameters other than charset.
``application/octet-stream`` is treated as missing so the extension
or the leading bytes can decide.
Args:
header: Raw Content-Type value, or None.
Returns:
tuple: Media type without parameters, and charset if one was set.
"""
if not header:
return None, None
parts = [part.strip() for part in header.split(";") if part.strip()]
media = parts[0].lower()
charset = None
for part in parts[1:]:
if "=" not in part:
continue
key, value = part.split("=", 1)
if key.strip().lower() == "charset":
charset = value.strip().strip("\"'")
if media in _UNUSABLE_MEDIA:
media = None
return media, charset
def media_for_extension(suffix: str, registry: object) -> str | None:
"""Map a file extension to a media type.
Generated by AI/LLM: 2026-09-30 - converter table first, then mimetypes.
Args:
suffix: Extension including the leading dot.
registry: Converter registry in registration order.
Returns:
str | None: Media type, or None when the extension is unknown.
"""
lowered = suffix.lower()
for converter in registry.converters: # type: ignore[attr-defined]
if lowered in converter.extensions and converter.media_types:
return converter.media_types[0]
guessed, _encoding = mimetypes.guess_type("file" + lowered)
return guessed
def sniff_media_type(data: bytes) -> str:
"""Guess a media type from the first 1024 bytes.
Generated by AI/LLM: 2026-09-30 - PDF, HTML, UTF-8 text, or octet-stream.
Args:
data: File bytes. Only the prefix is inspected.
Returns:
str: Detected media type.
"""
head = data[:1024]
if head.startswith(b"%PDF-"):
return "application/pdf"
sample = head.lstrip().lower()
if sample.startswith((b"<!doctype html", b"<html")):
return "text/html"
if b"\x00" not in head and _is_utf8(head):
return "text/plain"
return "application/octet-stream"
def detect_media_type(
data: bytes,
suffix: str,
header: str | None,
registry: object,
) -> tuple[str, str | None]:
"""Detect a media type and optional charset hint.
Generated by AI/LLM: 2026-09-30 - header, then extension, then bytes.
Args:
data: File bytes, used only when the header and extension miss.
suffix: Extension including the dot, or empty.
header: Content-Type header for a download, or None for a local file.
registry: Converter registry.
Returns:
tuple: Media type and charset hint.
"""
media, charset = split_content_type(header)
if media:
return media, charset
if suffix:
from_ext = media_for_extension(suffix, registry)
if from_ext:
return from_ext, charset
return sniff_media_type(data), charset
def extension_for(media_type: str, path_suffix: str, registry: object) -> str:
"""Choose the extension for a downloaded file.
Generated by AI/LLM: 2026-09-30 - converter extension, URL suffix, then ``.bin``.
Args:
media_type: Detected media type.
path_suffix: Suffix from the URL path, including the dot, or empty.
registry: Converter registry.
Returns:
str: Extension including the leading dot.
"""
converter = registry.find(media_type, "") # type: ignore[attr-defined]
if converter is not None and converter.extensions:
return converter.extensions[0]
if path_suffix:
suffix = path_suffix.lower()
return suffix if suffix.startswith(".") else f".{suffix}"
return ".bin"
def decode_bytes(data: bytes, charset: str | None) -> tuple[str, list[str]]:
"""Decode source bytes to text.
Generated by AI/LLM: 2026-09-30 - UTF-8, then the charset hint, then cp1252.
A BOM is stripped before UTF-8 is tried. A fallback decode adds one warning.
Args:
data: Raw file bytes.
charset: Optional charset from Content-Type.
Returns:
tuple: Decoded text and warnings.
"""
payload = data[3:] if data.startswith(b"\xef\xbb\xbf") else data
try:
return payload.decode("utf-8"), []
except UnicodeDecodeError:
pass
if charset:
try:
return payload.decode(charset), [f"decoded as {charset}; not valid UTF-8"]
except (UnicodeDecodeError, LookupError):
pass
return payload.decode("cp1252"), ["decoded as cp1252; not valid UTF-8"]
def utc_now() -> str:
"""Return the current UTC time as ``YYYY-MM-DDTHH:MM:SSZ``.
Generated by AI/LLM: 2026-09-30 - ingest ``fetched`` timestamp.
Returns:
str: UTC timestamp with a ``Z`` suffix.
"""
moment = datetime.datetime.now(datetime.UTC).replace(microsecond=0)
return moment.strftime("%Y-%m-%dT%H:%M:%SZ")
def utc_day() -> str:
"""Return today's UTC date as ``YYYY-MM-DD``.
Generated by AI/LLM: 2026-09-30 - page ``created`` and ``updated`` dates.
Returns:
str: UTC calendar date.
"""
return utc_now()[:10]
def _is_utf8(data: bytes) -> bool:
"""Return whether a byte prefix is valid UTF-8.
Generated by AI/LLM: 2026-09-30 - allow a multibyte character cut at 1024.
"""
try:
data.decode("utf-8")
except UnicodeDecodeError as exc:
return exc.start > 0 and exc.start >= len(data) - 3 and "unexpected end" in exc.reason
return True
-144
View File
@@ -1,144 +0,0 @@
"""Dataclasses passed between wiki ingest stages.
Generated by AI/LLM: 2026-09-30 - fetch, convert, and JSON result types.
"""
from __future__ import annotations
import dataclasses
import pathlib
class IngestError(Exception):
"""Raised when wiki ingest must stop before a page is written.
Generated by AI/LLM: 2026-09-30 - exit 1 errors from section 4.7.6.
"""
def __init__(self, message: str, exit_code: int = 1) -> None:
"""Store the stderr message and process exit code.
Args:
message: Text after ``Error: ``.
exit_code: Process status. Section 4.7.6 uses 1.
"""
super().__init__(message)
self.exit_code = exit_code
class AuthRequiredError(IngestError):
"""Raised when a URL answers with a login instead of the source.
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.2.8. Exit 4, nothing written.
"""
def __init__(self, url: str, reason: str) -> None:
"""Store the URL and the reason for the ``Error:`` line.
Args:
url: The URL the user passed, without userinfo.
reason: ``<status> <phrase>``, ``redirected to <url>``, or ``login page at <url>``.
"""
super().__init__(f"Authentication required: {url}: {reason}", exit_code=4)
self.url = url
self.reason = reason
@dataclasses.dataclass
class FetchedSource:
"""A local file plus the facts the later stages need.
Generated by AI/LLM: 2026-09-30 - output of the fetch stage.
"""
argument: str
path: pathlib.Path
origin: str
origin_kind: str
final_url: str | None
media_type: str
charset: str | None
size_bytes: int
slug: str
fetched: str
created: bool
@dataclasses.dataclass
class ConversionResult:
"""Markdown produced from one archived file.
Generated by AI/LLM: 2026-09-30 - output of one converter.
"""
body: str
title: str | None
source_meta: dict
conversion: str
warnings: list[str]
converter: str
@dataclasses.dataclass
class ArchiveResult:
"""Where the raw copy lives after the archive stage.
Generated by AI/LLM: 2026-09-30 - raw path, hash, and reuse flag.
"""
path: pathlib.Path
raw_rel: str
sha256: str
raw_reused: bool
warnings: list[str]
@dataclasses.dataclass
class IngestResult:
"""One ingest run, including the JSON object from section 4.7.6.
Generated by AI/LLM: 2026-09-30 - stdout facts for a finished run.
"""
source: str
origin: str
origin_kind: str
media_type: str
raw_path: str | None
raw_reused: bool
sha256: str
duplicate_page: str | None
page_path: str | None
converter: str | None
conversion: str
status: str | None
warnings: list[str]
dry_run: bool
exit_code: int = 0
error: str | None = None
messages: list[str] = dataclasses.field(default_factory=list)
def to_dict(self) -> dict:
"""Return the JSON object from section 4.7.6.
Generated by AI/LLM: 2026-09-30 - drop CLI-only fields.
Returns:
dict: Keys and nulls from the ingest JSON contract.
"""
return {
"source": self.source,
"origin": self.origin,
"origin_kind": self.origin_kind,
"media_type": self.media_type,
"raw_path": self.raw_path,
"raw_reused": self.raw_reused,
"sha256": self.sha256,
"duplicate_page": self.duplicate_page,
"page_path": self.page_path,
"converter": self.converter,
"conversion": self.conversion,
"status": self.status,
"warnings": list(self.warnings),
"dry_run": self.dry_run,
}
-375
View File
@@ -1,375 +0,0 @@
"""Write one ``status: raw`` wiki page.
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.5. The tool never sets ``refined``.
"""
from __future__ import annotations
import logging
import os
import pathlib
import re
import yaml
from pymarkdown.api import PyMarkdownApiException
from fiona.mdoctor.mdoctor import MDoctor
from fiona.wiki.ingest.media import utc_day
from fiona.wiki.ingest.models import ArchiveResult, ConversionResult, FetchedSource, IngestError
from fiona.wiki.ingest.settings import IngestSettings, package_version
logger = logging.getLogger(__name__)
_FENCE = re.compile(r"^\s*(`{3,}|~{3,})")
_ATX = re.compile(r"^(#{1,6})(\s+)(.*)$")
_IMAGE = re.compile(r"!\[([^\]]*)\]\(([^)]+)\)")
_LINK = re.compile(r"(?<!!)\[([^\]]*)\]\(([^)]+)\)")
_PROTECTED = {"index.md", "log.md", "schema.md"}
class RawPageWriter:
"""Write the raw page and run the markdown fixer on that file only.
Generated by AI/LLM: 2026-09-30 - frontmatter order from SPEC 7.1.5.5.3.
"""
def __init__(self, wiki_root: pathlib.Path) -> None:
"""Store the wiki root.
Args:
wiki_root: Directory that contains ``wiki/``.
"""
self.wiki_root = wiki_root
def page_path(
self,
slug: str,
settings: IngestSettings,
target: str | None,
) -> pathlib.Path:
"""Resolve the page path and reject paths outside ``wiki/``.
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.5.1. Does not touch the disk.
Args:
slug: Page stem when ``target`` is a directory or omitted.
settings: Supplies ``default_target``.
target: ``-t`` value, or None.
Returns:
pathlib.Path: Absolute page path inside ``wiki/``.
Raises:
IngestError: The path escapes ``wiki/`` or names a protected file.
"""
wiki = (self.wiki_root / "wiki").resolve()
if target is None:
relative = f"{settings.default_target}/{slug}.md"
else:
relative = _strip_wiki_prefix(target)
suffix = pathlib.PurePosixPath(relative.replace("\\", "/")).suffix
if suffix != ".md":
relative = f"{relative}/{slug}.md" if relative else f"{slug}.md"
candidate = (wiki / relative).resolve()
try:
page_rel = candidate.relative_to(wiki).as_posix()
except ValueError as exc:
raise IngestError(f"--target must stay inside wiki/: {target}") from exc
if page_rel in _PROTECTED or page_rel.split("/", 1)[0] == "logs":
shown = target if target else page_rel
raise IngestError(f"--target cannot replace {shown}")
return candidate
def write(
self,
*,
source: FetchedSource,
archive: ArchiveResult,
conversion: ConversionResult,
page: pathlib.Path,
dry_run: bool,
) -> tuple[str, list[str]]:
"""Write the page, fix it, and return the relative path and warnings.
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.5.3 through 7.1.5.5.5.
Args:
source: Fetch facts stored in ``ingest``.
archive: Raw path, hash, and reuse flag.
conversion: Converter output.
page: Destination markdown file.
dry_run: When true, build the warnings and write nothing.
Returns:
tuple: Page path relative to the wiki, and the warning list.
"""
title = conversion.title or source.slug
content, link_warnings = _page_content(conversion.body, title, page, archive.path)
warnings = [*conversion.warnings, *archive.warnings, *link_warnings]
if dry_run:
return _rel(page, self.wiki_root), warnings
page.parent.mkdir(parents=True, exist_ok=True)
day = utc_day()
page.write_text(
_document(source, archive, conversion, content, warnings, day),
encoding="utf-8",
newline="\n",
)
_fix(page, self.wiki_root)
style_count = _style_violations(page, self.wiki_root)
if style_count:
warnings.append(f"{style_count} markdown style violations left")
fixed = _body_after_frontmatter(page.read_text(encoding="utf-8"))
if not fixed.lstrip("\n").startswith("# "):
fixed = content
page.write_text(
_document(source, archive, conversion, fixed.lstrip("\n"), warnings, day),
encoding="utf-8",
newline="\n",
)
logger.info("wrote %s", page)
return _rel(page, self.wiki_root), warnings
def _page_content(
body: str,
title: str,
page: pathlib.Path,
raw_path: pathlib.Path,
) -> tuple[str, list[str]]:
"""Demote extra H1s, drop dead relative links, and add the ingest intro.
Generated by AI/LLM: 2026-09-30 - the page has one H1 and a link to the raw file.
"""
shifted = _shift_headings(body)
rewritten, count = _rewrite_links(shifted, page.parent)
warnings = [f"{count} relative link(s) did not resolve"] if count else []
link = os.path.relpath(raw_path, page.parent).replace("\\", "/")
intro = f"Raw ingest of [{raw_path.name}]({link}). Not refined yet."
prepared = rewritten.strip("\n")
tail = f"{intro}\n\n{prepared}\n" if prepared else f"{intro}\n"
return f"# {title}\n\n{tail}", warnings
def _document(
source: FetchedSource,
archive: ArchiveResult,
conversion: ConversionResult,
content: str,
warnings: list[str],
day: str,
) -> str:
"""Render frontmatter plus the page content.
Generated by AI/LLM: 2026-09-30 - every key is present, in contract order.
"""
title = conversion.title or source.slug
document = {
"name": source.slug,
"version": "0.1.0",
"title": title,
"created": day,
"updated": day,
"type": "summary",
"status": "raw",
"sources": [archive.raw_rel],
"source_meta": conversion.source_meta or {},
"ingest": {
"origin": source.origin,
"origin_kind": source.origin_kind,
"final_url": source.final_url,
"fetched": source.fetched,
"media_type": source.media_type,
"size_bytes": source.size_bytes,
"sha256": archive.sha256,
"raw_reused": archive.raw_reused,
"converter": conversion.converter,
"conversion": conversion.conversion,
"warnings": list(warnings),
"tool": f"fiona-assistant {package_version()}",
},
"tags": [],
"confidence": "zero",
"llm": [{"model": "none"}],
"contested": False,
"contradictions": [],
}
dumped = yaml.safe_dump(document, sort_keys=False, allow_unicode=True, width=1000)
payload = content if content.endswith("\n") else content + "\n"
return f"---\n{dumped}---\n\n{payload}"
def _shift_headings(body: str) -> str:
"""Move every ATX heading down one level when the body has an H1.
Generated by AI/LLM: 2026-09-30 - H6 stays H6. Fenced lines are left alone.
"""
if not _has_h1(body):
return body
lines = body.splitlines(keepends=True)
fenced = False
shifted: list[str] = []
for line in lines:
ending, content = _split_ending(line)
fence = _FENCE.match(content)
if fence:
fenced = not fenced
shifted.append(line)
continue
match = None if fenced else _ATX.match(content)
if match is None:
shifted.append(line)
continue
marks = match.group(1)
if len(marks) < 6:
marks += "#"
shifted.append(f"{marks}{match.group(2)}{match.group(3)}{ending}")
return "".join(shifted)
def _has_h1(body: str) -> bool:
"""Return whether an unfenced ATX H1 is present.
Generated by AI/LLM: 2026-09-30 - detection does not change the body.
"""
fenced = False
for line in body.splitlines():
if _FENCE.match(line):
fenced = not fenced
continue
if not fenced and re.match(r"^#\s+\S", line):
return True
return False
def _rewrite_links(body: str, page_dir: pathlib.Path) -> tuple[str, int]:
"""Replace relative links and images that do not resolve from the page.
Generated by AI/LLM: 2026-09-30 - one warning carries the count.
"""
count = 0
lines = body.splitlines(keepends=True)
fenced = False
rewritten: list[str] = []
for line in lines:
ending, content = _split_ending(line)
if _FENCE.match(content):
fenced = not fenced
rewritten.append(line)
continue
if fenced:
rewritten.append(line)
continue
content, image_count = _replace_pattern(_IMAGE, content, page_dir)
content, link_count = _replace_pattern(_LINK, content, page_dir)
count += image_count + link_count
rewritten.append(content + ending)
return "".join(rewritten), count
def _replace_pattern(
pattern: re.Pattern[str],
content: str,
page_dir: pathlib.Path,
) -> tuple[str, int]:
"""Replace matches whose target does not exist.
Generated by AI/LLM: 2026-09-30 - external URLs and anchors stay.
"""
count = 0
def _swap(match: re.Match[str]) -> str:
nonlocal count
target = match.group(2).strip()
if _keeps_target(target, page_dir):
return match.group(0)
count += 1
return match.group(1)
return pattern.sub(_swap, content), count
def _keeps_target(target: str, page_dir: pathlib.Path) -> bool:
"""Return whether a link target should stay in the page.
Generated by AI/LLM: 2026-09-30 - http, mailto, and anchors are kept.
"""
lowered = target.lower()
if lowered.startswith(("http://", "https://", "mailto:", "#", "//")) or "://" in target:
return True
path_part = target.split("#", 1)[0].split("?", 1)[0]
if not path_part:
return True
resolved = (page_dir / path_part).resolve()
return resolved.exists()
def _fix(page: pathlib.Path, wiki_root: pathlib.Path) -> None:
"""Run the pymarkdown fixer on one page.
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.5.5. A fixer error is logged.
"""
try:
doctor = MDoctor(settings_file=wiki_root / "fiona-settings.json")
doctor.fix_file_programmatically(str(page))
except PyMarkdownApiException:
logger.warning("markdown fix failed for %s", page, exc_info=True)
def _style_violations(page: pathlib.Path, wiki_root: pathlib.Path) -> int:
"""Count pymarkdown violations still on the page.
Generated by AI/LLM: 2026-09-30 - the count becomes one warning.
"""
try:
doctor = MDoctor(settings_file=wiki_root / "fiona-settings.json")
return len(doctor.scan_file_programmatically(str(page)))
except PyMarkdownApiException:
logger.warning("markdown scan failed for %s", page, exc_info=True)
return 0
def _body_after_frontmatter(text: str) -> str:
"""Return the file body after the closing frontmatter marker.
Generated by AI/LLM: 2026-09-30 - keep the fixer output and replace the header.
"""
if not text.startswith("---"):
return text
parts = text.split("---", 2)
if len(parts) < 3:
return text
return parts[2].removeprefix("\n")
def _split_ending(line: str) -> tuple[str, str]:
"""Split a line into its ending and its content.
Generated by AI/LLM: 2026-09-30 - keep ``\\n`` and ``\\r\\n``.
"""
if line.endswith("\r\n"):
return "\r\n", line[:-2]
if line.endswith("\n"):
return "\n", line[:-1]
return "", line
def _strip_wiki_prefix(value: str) -> str:
"""Drop a leading ``wiki/`` from a ``-t`` value.
Generated by AI/LLM: 2026-09-30 - both forms are accepted.
"""
relative = value.replace("\\", "/").lstrip("/")
if relative == "wiki":
return ""
if relative.startswith("wiki/"):
return relative[5:]
return relative
def _rel(path: pathlib.Path, root: pathlib.Path) -> str:
"""Return a path relative to the wiki root using ``/``.
Generated by AI/LLM: 2026-09-30 - JSON page paths.
"""
return path.resolve().relative_to(root.resolve()).as_posix()
-586
View File
@@ -1,586 +0,0 @@
"""Run the wiki ingest stages in order.
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5. The pipeline holds no format logic.
"""
from __future__ import annotations
import argparse
import dataclasses
import json
import logging
import pathlib
import shutil
import sys
import tempfile
from fiona.librarian.md_frontmatter import FrontmatterError, parse_frontmatter
from fiona.wiki.ingest.archiver import RawArchiver
from fiona.wiki.ingest.converters import default_registry
from fiona.wiki.ingest.converters.base import ConversionError, ConverterRegistry
from fiona.wiki.ingest.fetchers import (
LocalFileFetcher,
SourceFetcher,
UrlFetcher,
url_scheme,
without_userinfo,
)
from fiona.wiki.ingest.log_writer import IngestLog
from fiona.wiki.ingest.media import slug_from_file, slug_from_url, utc_day
from fiona.wiki.ingest.models import AuthRequiredError, FetchedSource, IngestError, IngestResult
from fiona.wiki.ingest.page_writer import RawPageWriter
from fiona.wiki.ingest.settings import IngestSettings, load_ingest_settings
logger = logging.getLogger(__name__)
_NO_CONVERTER = "No converter for {media}. Convert {raw} by hand."
_AUTH_NEXT = "Next: save it into ingest/, then run: wiki ingest <file> --origin {url}"
class IngestPipeline:
"""Fetch, archive, convert, write, and record one source.
Generated by AI/LLM: 2026-09-30 - stages are constructor arguments.
"""
def __init__(
self,
wiki_root: pathlib.Path,
fetchers: list[SourceFetcher],
registry: ConverterRegistry,
archiver: RawArchiver,
page_writer: RawPageWriter,
log_writer: IngestLog,
settings: IngestSettings,
) -> None:
"""Store the stage objects. No converter is named here.
Args:
wiki_root: Wiki root directory.
fetchers: Tried in order. Each implements ``accepts`` and ``fetch``.
registry: Converter registry. Tests may register another converter.
archiver: Raw archive stage.
page_writer: Wiki page stage.
log_writer: ``wiki/log.md`` stage.
settings: Timeouts, size limit, and default paths.
"""
self.wiki_root = wiki_root
self.fetchers = fetchers
self.registry = registry
self.archiver = archiver
self.page_writer = page_writer
self.log_writer = log_writer
self.settings = settings
def run(
self,
argument: str,
*,
raw: str | None = None,
target: str | None = None,
raw_only: bool = False,
force: bool = False,
dry_run: bool = False,
origin: str | None = None,
) -> IngestResult:
"""Ingest one URL or local file.
Generated by AI/LLM: 2026-09-30 - exit 1 raises; exit 0 and 3 return a result.
Modified by AI/LLM: 2026-10-07 - ``origin`` for a file saved by hand (SPEC 7.1.5.2.9).
Modified by AI/LLM: 2026-10-08 - a file inside the wiki has a relative origin (SPEC 7.1.5.2.10).
Args:
argument: Source as given.
raw: ``-r`` path relative to ``raw/``.
target: ``-t`` path relative to ``wiki/``.
raw_only: Archive and record, skip convert and write.
force: Replace an existing page, including a duplicate.
dry_run: Report the run and write nothing under the wiki.
origin: ``--origin`` URL a local file was saved from.
Returns:
IngestResult: Facts for stdout. ``exit_code`` is 0 or 3.
Raises:
IngestError: Exit 1. Nothing new remains under the wiki.
AuthRequiredError: Exit 4. The URL needs a login.
"""
scheme = url_scheme(argument)
if scheme is not None and scheme.lower() not in {"http", "https"}:
raise IngestError(f"Unsupported URL scheme: {scheme}")
_check_origin(argument, scheme, origin)
slug = slug_from_url(argument) if scheme else slug_from_file(pathlib.Path(argument))
page = self.page_writer.page_path(slug, self.settings, target)
if page.is_file() and not force:
rel = page.resolve().relative_to(self.wiki_root.resolve()).as_posix()
raise IngestError(f"Page exists: {rel}. Use --force to replace it.")
fetcher = self._fetcher(argument)
fetcher.settings = self.settings
temp_dir: pathlib.Path | None = None
fetched: FetchedSource | None = None
archived = False
try:
if dry_run and scheme:
temp_dir = pathlib.Path(tempfile.mkdtemp(prefix="fiona-ingest-"))
dest = temp_dir
else:
dest = self.wiki_root / "ingest"
fetched = fetcher.fetch(argument, dest)
if fetched.origin_kind == "file" and fetched.path.is_relative_to(self.wiki_root.resolve()):
fetched = dataclasses.replace(
fetched,
origin=fetched.path.relative_to(self.wiki_root.resolve()).as_posix(),
)
if origin is not None:
fetched = dataclasses.replace(
fetched,
origin=without_userinfo(origin),
origin_kind="url",
final_url=None,
)
archive = self.archiver.archive(
fetched,
registry=self.registry,
settings=self.settings,
raw_override=raw,
dry_run=dry_run,
)
archived = True
return self._after_archive(
fetched,
archive,
page,
target_explicit=target is not None,
raw_only=raw_only,
force=force,
dry_run=dry_run,
)
except IngestError:
if fetched is not None and fetched.created and fetched.path.exists() and not archived:
fetched.path.unlink()
raise
finally:
if temp_dir is not None:
shutil.rmtree(temp_dir, ignore_errors=True)
def list_pending(self) -> list[dict]:
"""List wiki pages whose ``status`` is ``raw``, sorted by path.
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.9.
Returns:
list[dict]: ``page_path``, ``sources``, and ``origin`` for each page.
"""
wiki = self.wiki_root / "wiki"
if not wiki.is_dir():
return []
pages: list[dict] = []
for path in sorted(wiki.rglob("*.md"), key=lambda item: item.as_posix()):
if not path.is_file():
continue
try:
front = parse_frontmatter(path)
except FrontmatterError:
continue
if front.get("status") != "raw":
continue
sources = front.get("sources") or []
if isinstance(sources, str):
sources = [sources]
ingest = front.get("ingest") if isinstance(front.get("ingest"), dict) else {}
origin = ingest.get("origin") if isinstance(ingest, dict) else None
pages.append({
"page_path": path.resolve().relative_to(self.wiki_root.resolve()).as_posix(),
"sources": list(sources),
"origin": origin,
})
return pages
def _fetcher(self, argument: str) -> SourceFetcher:
"""Return the first fetcher that accepts the argument.
Generated by AI/LLM: 2026-09-30 - URL fetcher is registered before the local one.
"""
for fetcher in self.fetchers:
if fetcher.accepts(argument):
return fetcher
raise IngestError(f"Source not found: {argument}")
def _after_archive(
self,
source: FetchedSource,
archive: object,
page: pathlib.Path,
*,
target_explicit: bool,
raw_only: bool,
force: bool,
dry_run: bool,
) -> IngestResult:
"""Finish convert, write, and record after a successful archive.
Generated by AI/LLM: 2026-09-30 - duplicate, raw-only, and missing converter exits.
"""
messages = _fetched_line(source)
if archive.raw_reused: # type: ignore[attr-defined]
short = archive.sha256[:12] # type: ignore[attr-defined]
messages.append(
f"Archived: {archive.raw_rel} (already archived, sha256 {short})" # type: ignore[attr-defined]
)
else:
messages.append(f"Archived: {archive.raw_rel}") # type: ignore[attr-defined]
duplicate = _citing_page(self.wiki_root, archive.raw_rel, archive.sha256) # type: ignore[attr-defined]
if duplicate and not force:
messages.append(f"Duplicate: {duplicate} already cites this source.")
return _result(
source,
archive,
messages=messages,
duplicate_page=duplicate,
dry_run=dry_run,
warnings=list(archive.warnings), # type: ignore[attr-defined]
)
if duplicate and force and not target_explicit:
page = self.wiki_root / duplicate
if raw_only:
self.log_writer.append(
day=utc_day(),
origin=source.origin,
raw_rel=archive.raw_rel, # type: ignore[attr-defined]
page_rel=None,
dry_run=dry_run,
)
return _result(source, archive, messages=messages, dry_run=dry_run, warnings=list(archive.warnings)) # type: ignore[attr-defined]
extension = archive.path.suffix # type: ignore[attr-defined]
converter = self.registry.find(source.media_type, extension)
if converter is None:
messages.append(_NO_CONVERTER.format(media=source.media_type, raw=archive.raw_rel)) # type: ignore[attr-defined]
self._record(source, archive, None, dry_run)
return _result(
source,
archive,
messages=messages,
dry_run=dry_run,
exit_code=3,
warnings=list(archive.warnings), # type: ignore[attr-defined]
)
read_path = archive.path if archive.path.is_file() else source.path # type: ignore[attr-defined]
try:
conversion = converter.convert(dataclasses.replace(source, path=read_path))
except ConversionError as exc:
messages.append(_NO_CONVERTER.format(media=source.media_type, raw=archive.raw_rel)) # type: ignore[attr-defined]
self._record(source, archive, None, dry_run)
return _result(
source,
archive,
messages=messages,
dry_run=dry_run,
exit_code=3,
error=f"Conversion failed: {exc.converter}: {exc.reason}",
warnings=list(archive.warnings), # type: ignore[attr-defined]
)
page_rel, warnings = self.page_writer.write(
source=source,
archive=archive, # type: ignore[arg-type]
conversion=conversion,
page=page,
dry_run=dry_run,
)
messages.append(
f"Wrote: {page_rel} (converter {conversion.converter}, conversion {conversion.conversion})"
)
messages.extend(f"Warning: {warning}" for warning in warnings)
messages.append(f"Next: refine {page_rel}, then set status: refined.")
self._record(source, archive, page_rel, dry_run)
return _result(
source,
archive,
messages=messages,
page_path=page_rel,
converter=conversion.converter,
conversion=conversion.conversion,
status="raw",
dry_run=dry_run,
warnings=warnings,
)
def _record(self, source: FetchedSource, archive: object, page_rel: str | None, dry_run: bool) -> None:
"""Append the log line for a run that archived a source.
Generated by AI/LLM: 2026-09-30 - duplicates do not call this.
"""
self.log_writer.append(
day=utc_day(),
origin=source.origin,
raw_rel=archive.raw_rel, # type: ignore[attr-defined]
page_rel=page_rel,
dry_run=dry_run,
)
def default_pipeline(wiki_root: pathlib.Path) -> IngestPipeline:
"""Build the pipeline with the shipped fetchers and converters.
Generated by AI/LLM: 2026-09-30 - URL fetcher, then local file fetcher.
Modified by AI/LLM: 2026-10-07 - one settings object for the registry and the pipeline.
Args:
wiki_root: Wiki root directory.
Returns:
IngestPipeline: Ready to ``run`` or ``list_pending``.
"""
settings = load_ingest_settings(wiki_root)
registry = default_registry(settings)
return IngestPipeline(
wiki_root=wiki_root,
fetchers=[UrlFetcher(registry), LocalFileFetcher(registry)],
registry=registry,
archiver=RawArchiver(wiki_root),
page_writer=RawPageWriter(wiki_root),
log_writer=IngestLog(wiki_root),
settings=settings,
)
def render_stdout(result: IngestResult) -> str:
"""Return the human report, including the dry-run banner.
Generated by AI/LLM: 2026-09-30 - section 4.7.6 line order.
"""
lines: list[str] = []
if result.dry_run:
lines.append("Dry run - no files will be modified:")
lines.extend(result.messages)
return "\n".join(lines)
def add_ingest_arguments(parser: argparse.ArgumentParser) -> None:
"""Add the section 4.7.6 flags to an ingest parser.
Generated by AI/LLM: 2026-09-30 - the CLI subparser owns these flags, including --help.
Args:
parser: The ``wiki ingest`` parser.
"""
parser.add_argument("source", nargs="?", help="http(s) URL or file path")
parser.add_argument(
"--wiki",
default=".",
help="llm-wiki directory (default: .)",
)
parser.add_argument("-r", "--raw", dest="raw", help="raw path, relative to raw/")
parser.add_argument(
"--origin",
dest="origin",
help="http(s) URL a local file was saved from",
)
parser.add_argument(
"-t",
"--target",
dest="target",
help="page path, relative to wiki/",
)
parser.add_argument(
"--raw-only",
action="store_true",
help="archive only; do not write a page",
)
parser.add_argument(
"-f",
"--force",
action="store_true",
help="replace an existing page",
)
parser.add_argument(
"--dry-run",
action="store_true",
help="print the plan and write nothing in the wiki",
)
parser.add_argument("--json", action="store_true", help="print only the JSON result")
parser.add_argument(
"--pending",
action="store_true",
help="list pages whose status is raw",
)
def ingest_main(argv: list[str] | None = None) -> int:
"""CLI entry for ``fiona-assistant wiki ingest``.
Generated by AI/LLM: 2026-09-30 - section 4.7.6 exit codes.
Args:
argv: Arguments after the subcommand. ``None`` reads the process arguments.
Returns:
int: 0, 1, 2, 3, or 4.
"""
parser = argparse.ArgumentParser(prog="fiona-assistant wiki ingest")
add_ingest_arguments(parser)
return execute_ingest(parser.parse_args(argv), parser)
def execute_ingest(args: argparse.Namespace, parser: argparse.ArgumentParser) -> int:
"""Run one ingest from an already parsed namespace.
Generated by AI/LLM: 2026-09-30 - shared by ingest_main and the wiki subcommand.
Args:
args: Parsed source, flags, and paths.
parser: Parser used for the mutual-exclusion error.
Returns:
int: 0, 1, 2, 3, or 4.
"""
if bool(args.source) == bool(args.pending):
parser.error("provide a source or --pending, not both")
try:
root = _require_wiki(args.wiki)
pipeline = default_pipeline(root)
if args.pending:
return _print_pending(pipeline, args.json)
result = pipeline.run(
args.source,
raw=args.raw,
target=args.target,
raw_only=args.raw_only,
force=args.force,
dry_run=args.dry_run,
origin=args.origin,
)
except IngestError as exc:
print(f"Error: {exc}", file=sys.stderr)
if isinstance(exc, AuthRequiredError):
print(_AUTH_NEXT.format(url=exc.url), file=sys.stderr)
return exc.exit_code
if result.error:
print(f"Error: {result.error}", file=sys.stderr)
if args.json:
print(json.dumps(result.to_dict(), indent=2))
else:
text = render_stdout(result)
if text:
print(text)
return result.exit_code
def _check_origin(argument: str, scheme: str | None, origin: str | None) -> None:
"""Reject ``--origin`` with a URL source or a non-http value, before fetch.
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.2.9.1.
"""
if origin is None:
return
if scheme is not None:
raise IngestError(f"--origin needs a local file: {argument}")
origin_scheme = url_scheme(origin)
if origin_scheme is None or origin_scheme.lower() not in {"http", "https"}:
raise IngestError(f"--origin must be an http or https URL: {origin}")
def _require_wiki(directory: str) -> pathlib.Path:
"""Return the wiki root or raise when settings are missing.
Generated by AI/LLM: 2026-09-30 - section 4.7.6 directory error.
"""
root = pathlib.Path(directory)
if not (root / "fiona-settings.json").is_file():
raise IngestError(f"Directory '{directory}' is not an llm-wiki directory")
return root
def _print_pending(pipeline: IngestPipeline, json_output: bool) -> int:
"""Print the pending-page report.
Generated by AI/LLM: 2026-09-30 - human lines or one JSON object.
"""
pages = pipeline.list_pending()
if json_output:
print(json.dumps({"pending": pages, "total": len(pages)}, indent=2))
return 0
print(f"Raw pages: {len(pages)}")
for page in pages:
first = page["sources"][0] if page["sources"] else ""
print(f" - {page['page_path']} ({first})")
return 0
def _fetched_line(source: FetchedSource) -> list[str]:
"""Return the ``Fetched:`` line for a URL, or an empty list.
Generated by AI/LLM: 2026-09-30 - local files do not print Fetched.
Modified by AI/LLM: 2026-10-07 - a local file with ``--origin`` was not fetched either.
"""
if url_scheme(source.argument) is None:
return []
return [f"Fetched: {source.argument} ({source.media_type}, {source.size_bytes} bytes)"]
def _citing_page(wiki_root: pathlib.Path, raw_rel: str, digest: str) -> str | None:
"""Return a page that already cites this raw path or hash.
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.3.5. First path in sorted order.
"""
wiki = wiki_root / "wiki"
if not wiki.is_dir():
return None
matches: list[str] = []
for path in wiki.rglob("*.md"):
if not path.is_file():
continue
try:
front = parse_frontmatter(path)
except FrontmatterError:
continue
sources = front.get("sources") or []
if isinstance(sources, str):
sources = [sources]
ingest = front.get("ingest") if isinstance(front.get("ingest"), dict) else {}
sha = ingest.get("sha256") if isinstance(ingest, dict) else None
if raw_rel in list(sources) or sha == digest:
matches.append(path.resolve().relative_to(wiki_root.resolve()).as_posix())
if not matches:
return None
matches.sort()
return matches[0]
def _result(
source: FetchedSource,
archive: object,
*,
messages: list[str],
dry_run: bool,
warnings: list[str],
duplicate_page: str | None = None,
page_path: str | None = None,
converter: str | None = None,
conversion: str = "none",
status: str | None = None,
exit_code: int = 0,
error: str | None = None,
) -> IngestResult:
"""Build an ingest result from the archive and any page that was written.
Generated by AI/LLM: 2026-09-30 - JSON fields from section 4.7.6.
"""
return IngestResult(
source=source.argument,
origin=source.origin,
origin_kind=source.origin_kind,
media_type=source.media_type,
raw_path=archive.raw_rel, # type: ignore[attr-defined]
raw_reused=archive.raw_reused, # type: ignore[attr-defined]
sha256=archive.sha256, # type: ignore[attr-defined]
duplicate_page=duplicate_page,
page_path=page_path,
converter=converter,
conversion=conversion,
status=status,
warnings=warnings,
dry_run=dry_run,
exit_code=exit_code,
error=error,
messages=messages,
)
-153
View File
@@ -1,153 +0,0 @@
"""Ingest settings from ``fiona-settings.json``.
Generated by AI/LLM: 2026-09-30 - SPEC 5.1 ``fiona.wiki.ingest`` keys.
Modified by AI/LLM: 2026-10-07 - ``login_url_patterns`` and ``html.*`` keys.
"""
from __future__ import annotations
import dataclasses
import importlib.metadata
import json
import logging
import pathlib
import re
logger = logging.getLogger(__name__)
DEFAULT_LOGIN_URL_PATTERNS = (
r"/(login|logon|signin|sign-in)\b",
r"/(sso|saml2?|oauth2?|adfs|cas)(/|$)",
r"login\.microsoftonline\.com",
r"j_security_check",
)
def package_version() -> str:
"""Return the installed fiona-assistant version.
Generated by AI/LLM: 2026-09-30 - user agent and ``ingest.tool``.
Returns:
str: Distribution version, or ``0.99.0`` when metadata is absent.
"""
try:
return importlib.metadata.version("fiona-assistant")
except importlib.metadata.PackageNotFoundError:
return "0.99.0"
@dataclasses.dataclass
class IngestSettings:
"""Runtime limits and paths for one ingest.
Generated by AI/LLM: 2026-09-30 - defaults from the settings table.
Modified by AI/LLM: 2026-10-07 - login patterns and block sweep limits.
"""
timeout_seconds: float
max_download_bytes: int
user_agent: str
default_target: str
raw_categories: dict[str, str]
login_url_patterns: list[str] = dataclasses.field(
default_factory=lambda: list(DEFAULT_LOGIN_URL_PATTERNS)
)
html_min_words: int = 4
html_max_link_density: float = 0.5
def default_settings() -> IngestSettings:
"""Return the settings used when the wiki omits ``fiona.wiki.ingest``.
Generated by AI/LLM: 2026-09-30 - table defaults from SPEC 5.1.
Returns:
IngestSettings: Built-in limits and paths.
"""
return IngestSettings(
timeout_seconds=30,
max_download_bytes=52428800,
user_agent=f"fiona-assistant/{package_version()}",
default_target="sources",
raw_categories={},
)
def load_ingest_settings(wiki_root: pathlib.Path) -> IngestSettings:
"""Load ingest settings, falling back per key when a value is unusable.
Generated by AI/LLM: 2026-09-30 - absent or mistyped keys keep the default.
Modified by AI/LLM: 2026-10-07 - read ``login_url_patterns`` and ``html``.
Args:
wiki_root: Directory that contains ``fiona-settings.json``.
Returns:
IngestSettings: Merged settings for this wiki.
"""
settings = default_settings()
path = wiki_root / "fiona-settings.json"
if not path.is_file():
return settings
try:
document = json.loads(path.read_text(encoding="utf-8"))
except json.JSONDecodeError:
return settings
section = {}
if isinstance(document, dict):
wiki = document.get("fiona", {})
if isinstance(wiki, dict):
wiki = wiki.get("wiki", {})
if isinstance(wiki, dict):
raw = wiki.get("ingest", {})
if isinstance(raw, dict):
section = raw
timeout = section.get("timeout_seconds", settings.timeout_seconds)
if isinstance(timeout, (int, float)) and not isinstance(timeout, bool) and timeout > 0:
settings.timeout_seconds = float(timeout)
limit = section.get("max_download_bytes", settings.max_download_bytes)
if isinstance(limit, int) and not isinstance(limit, bool) and limit > 0:
settings.max_download_bytes = limit
agent = section.get("user_agent")
if isinstance(agent, str) and agent.strip():
settings.user_agent = agent
target = section.get("default_target")
if isinstance(target, str) and target.strip() and ".." not in target.replace("\\", "/"):
settings.default_target = target.strip().strip("/")
categories = section.get("raw_categories")
if isinstance(categories, dict):
cleaned = {
str(key): value
for key, value in categories.items()
if isinstance(value, str) and value.strip()
}
settings.raw_categories = cleaned
patterns = section.get("login_url_patterns")
if isinstance(patterns, list):
settings.login_url_patterns = _compilable(patterns)
html = section.get("html")
if isinstance(html, dict):
words = html.get("min_words")
if isinstance(words, int) and not isinstance(words, bool) and words >= 1:
settings.html_min_words = words
density = html.get("max_link_density")
if isinstance(density, (int, float)) and not isinstance(density, bool) and 0 <= density <= 1:
settings.html_max_link_density = float(density)
return settings
def _compilable(patterns: list) -> list[str]:
"""Return the entries that compile as regular expressions.
Generated by AI/LLM: 2026-10-07 - SPEC 5.1, one warning per skipped entry.
"""
kept: list[str] = []
for pattern in patterns:
try:
re.compile(pattern)
except (re.error, TypeError):
logger.warning("login_url_patterns: skipped %r; not a valid regular expression", pattern)
continue
kept.append(pattern)
return kept
File diff suppressed because it is too large Load Diff
@@ -1,361 +0,0 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Wiki Graph Visualization</title>
<script src="https://d3js.org/d3.v7.min.js"></script>
<style>
html, body {
height: 100%;
}
body {
font-family: var(--font, Arial, sans-serif);
margin: 0;
padding: 0.75em 1em;
background-color: var(--page-bg, #f5f5f5);
color: var(--text, #333333);
box-sizing: border-box;
display: flex;
flex-direction: column;
}
.container {
max-width: 100%;
margin: 0 auto;
width: 100%;
display: flex;
flex-direction: column;
flex: 1;
min-height: 0;
}
#graph-container {
width: 100%;
flex: 1;
min-height: 20em;
background-color: var(--graph-bg, #ffffff);
border: 0.0625em solid #ddd;
border-radius: 0.25em;
overflow: hidden;
}
h1 {
text-align: center;
color: var(--text, #333333);
}
.controls {
background-color: var(--card-bg, #ffffff);
color: var(--text, #333333);
padding: 0.9375em;
border: 0.0625em solid #ddd;
border-radius: 0.25em;
margin-bottom: 0.9375em;
display: flex;
flex-wrap: wrap;
gap: 0.625em;
align-items: center;
}
.control-group {
display: flex;
gap: 0.625em;
align-items: center;
flex-wrap: wrap;
}
.control-group label {
font-size: 0.875em;
}
input[type="text"], input[type="checkbox"], select, button {
padding: 0.5em 0.75em;
border: 0.0625em solid #ccc;
border-radius: 0.25em;
font-size: 0.875em;
}
button {
background-color: var(--btn-bg, #007bff);
color: var(--btn-text, #ffffff);
border: none;
cursor: pointer;
border-radius: 0.25em;
padding: 0.5em 1em;
font-size: 0.875em;
transition: background-color 0.3s;
}
button:hover {
background-color: var(--btn-hover, #0056b3);
}
button:disabled {
background-color: var(--btn-disabled, #6c757d);
cursor: not-allowed;
}
.toggle {
display: inline-flex;
align-items: center;
gap: 0.5em;
cursor: pointer;
font-size: 0.875em;
user-select: none;
}
.toggle input {
position: absolute;
opacity: 0;
width: 0.0625em;
height: 0.0625em;
margin: 0;
padding: 0;
}
.toggle-track {
width: 2.25em;
height: 1.25em;
background: #ccc;
border-radius: 0.625em;
position: relative;
flex: 0 0 auto;
}
.toggle-track::after {
content: "";
position: absolute;
width: 1em;
height: 1em;
border-radius: 50%;
background: #fff;
top: 0.125em;
left: 0.125em;
transition: transform 0.15s ease;
}
.toggle input:checked + .toggle-track {
background: var(--btn-bg, #007bff);
}
.toggle input:checked + .toggle-track::after {
transform: translateX(1em);
}
.toggle input:focus-visible + .toggle-track {
outline: 0.125em solid #007bff;
outline-offset: 0.125em;
}
.node-info {
position: absolute;
background: var(--card-bg, #ffffff);
color: var(--text, #333333);
border: 0.0625em solid #999;
padding: 0.5em;
border-radius: 0.25em;
pointer-events: none;
font-size: 0.75em;
z-index: 1000;
box-shadow: 0 0.125em 0.25em rgba(0,0,0,0.1);
}
.edge-info {
position: absolute;
background: var(--card-bg, #ffffff);
color: var(--text, #333333);
border: 0.0625em solid #999;
padding: 0.5em;
border-radius: 0.25em;
pointer-events: none;
font-size: 0.75em;
z-index: 1000;
box-shadow: 0 0.125em 0.25em rgba(0,0,0,0.1);
}
.frontmatter-card {
position: absolute;
background: var(--card-bg, #ffffff);
color: var(--text, #333333);
border: 0.0625em solid #999;
padding: 0.625em;
border-radius: 0.25em;
font-size: 0.75em;
z-index: 1001;
box-shadow: 0 0.125em 0.25em rgba(0,0,0,0.2);
max-width: 26.25em;
max-height: 70vh;
overflow: auto;
word-break: break-word;
}
.legend-card {
position: absolute;
background: var(--card-bg, #ffffff);
color: var(--text, #333333);
border: 0.0625em solid #999;
padding: 0.625em 0.75em;
border-radius: 0.25em;
font-size: 0.75em;
z-index: 1001;
box-shadow: 0 0.125em 0.25em rgba(0,0,0,0.2);
max-width: 15em;
}
.legend-card h2 {
margin: 0 0 0.375em;
font-size: 0.8125em;
}
.legend-heading {
margin: 0.625em 0 0;
font-size: 0.6875em;
font-weight: 700;
}
.legend-row {
display: flex;
align-items: center;
gap: 0.5em;
margin-top: 0.375em;
}
.legend-mark {
width: 1.75em;
height: 1.375em;
flex: 0 0 auto;
}
.stats {
background-color: var(--card-bg, #ffffff);
color: var(--text, #333333);
padding: 0.625em;
border: 0.0625em solid #ddd;
border-radius: 0.25em;
margin-bottom: 0.9375em;
}
.stats span {
margin-right: 1.25em;
font-size: 0.875em;
}
.controls, .stats, #graph-container, .legend-card,
.frontmatter-card, .node-info, .edge-info {
border-color: var(--card-border, #999999);
}
@media (max-width: 48em) {
.controls {
flex-direction: column;
align-items: stretch;
}
.control-group {
flex-direction: column;
align-items: stretch;
}
}
</style>
</head>
<body>
<div class="container">
<h1>Wiki Graph Visualization</h1>
<div class="stats">
<span>Nodes: <strong>{{node_count}}</strong></span>
<span>Edges: <strong>{{edge_count}}</strong></span>
<span>Max in-degree: <strong>{{max_in_degree}}</strong></span>
</div>
<div class="controls">
<div class="control-group">
<label for="theme-select">Theme:</label>
<select id="theme-select" autocomplete="off"></select>
</div>
<div class="control-group">
<label for="search-input">Search:</label>
<input type="text" id="search-input" placeholder="Filter nodes..." autocomplete="off" />
</div>
<div class="control-group">
<button type="button" id="fit-button">See all</button>
</div>
<div class="control-group">
<label class="toggle" for="orphan-toggle">
<input type="checkbox" id="orphan-toggle" checked autocomplete="off" />
<span class="toggle-track" aria-hidden="true"></span>
Show Orphan Nodes
</label>
</div>
<div class="control-group">
<label class="toggle" for="external-toggle">
<input type="checkbox" id="external-toggle" {{external_checked}} autocomplete="off" />
<span class="toggle-track" aria-hidden="true"></span>
Show External Links
</label>
</div>
<div class="control-group">
<label class="toggle" for="source-toggle">
<input type="checkbox" id="source-toggle" {{source_checked}} autocomplete="off" />
<span class="toggle-track" aria-hidden="true"></span>
Show Sources
</label>
</div>
<div class="control-group">
<label class="toggle" for="subtitle-toggle">
<input type="checkbox" id="subtitle-toggle" {{subtitle_checked}} autocomplete="off" />
<span class="toggle-track" aria-hidden="true"></span>
Show Subtitles
</label>
</div>
<div class="control-group">
<label class="toggle" for="legend-toggle">
<input type="checkbox" id="legend-toggle" {{legend_checked}} autocomplete="off" />
<span class="toggle-track" aria-hidden="true"></span>
Show Legend
</label>
</div>
<div class="control-group">
<label class="toggle" for="arrow-toggle">
<input type="checkbox" id="arrow-toggle" {{arrow_checked}} autocomplete="off" />
<span class="toggle-track" aria-hidden="true"></span>
Show Arrows
</label>
</div>
</div>
<div id="graph-container"></div>
</div>
<div class="node-info" style="display: none;"></div>
<div class="edge-info" style="display: none;"></div>
<div id="frontmatter-card" class="frontmatter-card" style="display: none;"></div>
<div id="legend-card" class="legend-card" style="display: {{legend_display}};">
<h2>Legend</h2>
<div class="legend-heading">Shape</div>
<div class="legend-row">
<svg id="legend-shape-page" class="legend-mark" viewBox="-16 -14 32 28" aria-hidden="true">
<circle r="7" fill="#333333"></circle>
</svg>
<span>Page</span>
</div>
<div class="legend-row">
<svg id="legend-shape-external" class="legend-mark" viewBox="-16 -14 32 28" aria-hidden="true">
<text text-anchor="middle" dominant-baseline="central" font-size="18" font-family="Segoe UI Symbol, sans-serif" fill="#333333">&#9729;&#65038;</text>
</svg>
<span>External link</span>
</div>
<div class="legend-heading">Files pointing here</div>
<div class="legend-row">
<svg class="legend-mark" viewBox="-16 -14 32 28" aria-hidden="true">
<circle id="legend-node-low" r="4" fill="#8f8272"></circle>
</svg>
<span>Few</span>
</div>
<div class="legend-row">
<svg class="legend-mark" viewBox="-16 -14 32 28" aria-hidden="true">
<circle id="legend-node-mid" r="7" fill="#e0943a"></circle>
</svg>
<span>Some</span>
</div>
<div class="legend-row">
<svg class="legend-mark" viewBox="-16 -14 32 28" aria-hidden="true">
<circle id="legend-node-high" r="11" fill="#c23b4a"></circle>
</svg>
<span>Many</span>
</div>
<div class="legend-heading">Same link, repeated</div>
<div class="legend-row">
<svg class="legend-mark" viewBox="0 0 28 12" aria-hidden="true">
<line id="legend-edge-low" x1="2" y1="6" x2="26" y2="6" stroke="#a89884" stroke-width="3"></line>
</svg>
<span>Linked once</span>
</div>
<div class="legend-row">
<svg class="legend-mark" viewBox="0 0 28 12" aria-hidden="true">
<line id="legend-edge-mid" x1="2" y1="6" x2="26" y2="6" stroke="#b86a1a" stroke-width="3"></line>
</svg>
<span>A few times</span>
</div>
<div class="legend-row">
<svg class="legend-mark" viewBox="0 0 28 12" aria-hidden="true">
<line id="legend-edge-high" x1="2" y1="6" x2="26" y2="6" stroke="#8e2436" stroke-width="3"></line>
</svg>
<span>Many times</span>
</div>
</div>
<script>
{{graph_physics_js}}
</script>
</body>
</html>
@@ -7,14 +7,12 @@ from __future__ import annotations
import argparse
import pathlib
from datetime import UTC, datetime
from datetime import datetime, timezone
import git
from git.repo.base import Repo
from fiona.gittool import diff_report
from fiona.logging_config import setup_logging
from fiona.wiki.ignore import ensure_wiki_gitignore
class GitTool:
@@ -52,7 +50,7 @@ class GitTool:
Returns:
str: Formatted timestamp string.
"""
return datetime.now(tz=UTC).strftime("%Y-%m-%d %H:%M:%S")
return datetime.now(tz=timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
@staticmethod
def _get_tag_timestamp() -> str:
@@ -64,7 +62,7 @@ class GitTool:
Returns:
str: Formatted timestamp string safe for git tags.
"""
return datetime.now(tz=UTC).strftime("%Y-%m-%d_%H-%M-%S")
return datetime.now(tz=timezone.utc).strftime("%Y-%m-%d_%H-%M-%S")
def _is_working_tree_clean(self) -> bool:
"""Check if the working tree is clean.
@@ -75,7 +73,7 @@ class GitTool:
# Check for modified/staged files
if self.repo.is_dirty():
return False
# Check for untracked files
untracked = self.repo.untracked_files
return len(untracked) == 0
@@ -172,7 +170,7 @@ class GitTool:
tag for tag in self.repo.tags
if tag.name.endswith(f"_{tag_name}") or tag.name == tag_name
]
if matching_tags:
# Sort by commit date descending and take the most recent
matching_tags.sort(
@@ -180,7 +178,7 @@ class GitTool:
reverse=True
)
tag_ref = matching_tags[0]
if tag_ref is None:
raise git.GitCommandError(
f"Tag '{tag_name}' does not exist."
@@ -191,7 +189,7 @@ class GitTool:
# Restore the tag's tree into the working directory and index
self.repo.git.checkout(tag_ref.commit, "--", ".")
# Create a new commit with the restored state
new_commit = self.repo.index.commit(restore_message)
@@ -274,9 +272,9 @@ class GitTool:
@staticmethod
def init_wiki_repo(repo_path: str | pathlib.Path, initial_message: str = "Initial llm-wiki commit") -> git.Repo:
"""Initialize a git repository for llm-wiki and seed .gitignore.
"""Initialize a git repository for llm-wiki with .gitignore for personal/.
Puts or appends the repository ignore patterns plus ``personal/``,
Creates a .gitignore file that excludes the personal/ directory,
stages all files, and makes the initial commit.
Args:
@@ -290,23 +288,26 @@ class GitTool:
git.InvalidGitRepositoryError: If git initialization fails.
"""
repo_path = pathlib.Path(repo_path).resolve()
# Initialize the repo
repo = GitTool.init_repo(repo_path)
# AI/LLM: 2026-09-30 - seed .gitignore from the repository ignore file.
ensure_wiki_gitignore(repo_path)
# Create .gitignore to exclude personal/ directory
gitignore_path = repo_path / ".gitignore"
with open(gitignore_path, "w", encoding="utf-8") as f:
f.write("# Exclude personal directory from git as per SPEC 3.2.0.2\n")
f.write("personal/\n")
# Add all files and make initial commit
repo.git.add("--all")
repo.index.commit(initial_message)
return repo
def main(): # pragma: no cover
"""Command-line interface for the GitTool.
Note: CLI is tested via integration tests through fiona-assistant git commands.
"""
parser = argparse.ArgumentParser(
@@ -350,30 +351,6 @@ def main(): # pragma: no cover
# status subcommand
subparsers.add_parser("status", help="Show git status")
# diff subcommand
diff_parser = subparsers.add_parser(
"diff",
help="Write constellation-themed HTML diff under wiki tmp/",
)
diff_parser.add_argument("--base", help="Base revision (default: origin/main or main/master)")
diff_parser.add_argument("--head", default="HEAD", help="Head revision (default: HEAD)")
diff_parser.add_argument("--title", help="Report H1 / page title")
diff_parser.add_argument(
"--subtitle",
help="Extra header line (e.g. temporary review before push)",
)
diff_parser.add_argument(
"-o",
"--output",
type=pathlib.Path,
help="Output HTML path (default: <wiki>/tmp/<repo>-git-diff-<short>.html)",
)
diff_parser.add_argument(
"--collapsed",
action="store_true",
help="Start with per-file sections collapsed",
)
args = parser.parse_args()
setup_logging()
@@ -403,21 +380,6 @@ def main(): # pragma: no cover
elif args.subcommand == "status":
status = git_tool.status()
print(status)
elif args.subcommand == "diff":
diff_argv: list[str] = []
if args.base:
diff_argv.extend(["--base", args.base])
if args.head != "HEAD":
diff_argv.extend(["--head", args.head])
if args.title:
diff_argv.extend(["--title", args.title])
if args.subtitle:
diff_argv.extend(["--subtitle", args.subtitle])
if args.output is not None:
diff_argv.extend(["--output", str(args.output)])
if args.collapsed:
diff_argv.append("--collapsed")
raise SystemExit(diff_report.main(diff_argv))
except Exception as e:
print(f"Error: {e}")
raise SystemExit(1) from e
@@ -18,18 +18,6 @@ import re
import yaml
# Default required fields for wiki pages
DEFAULT_REQUIRED_FIELDS = ["title", "created", "updated", "type"]
# Pattern to match frontmatter delimiters
# This pattern matches --- at start, then any content (including empty), then ---
# Handles: ---\ncontent\n--- or ---\n--- or ---\n\n---
FRONTMATTER_PATTERN = re.compile(r'^---\s*\n([\s\S]*?)(?:\n---\s*($|\n)|---\s*($|\n))')
# Pattern to match a single --- delimiter line
DELIMITER_PATTERN = re.compile(r'^---\s*$', re.MULTILINE)
# =============================================================================
# Exceptions
# =============================================================================
@@ -42,6 +30,22 @@ class RequiredFieldError(Exception):
"""Exception raised when required frontmatter fields are missing."""
# =============================================================================
# Constants
# =============================================================================
# Default required fields for wiki pages
DEFAULT_REQUIRED_FIELDS = ["title", "created", "updated", "type"]
# Pattern to match frontmatter delimiters
# This pattern matches --- at start, then any content (including empty), then ---
# Handles: ---\ncontent\n--- or ---\n--- or ---\n\n---
FRONTMATTER_PATTERN = re.compile(r'^---\s*\n([\s\S]*?)(?:\n---\s*($|\n)|---\s*($|\n))')
# Pattern to match a single --- delimiter line
DELIMITER_PATTERN = re.compile(r'^---\s*$', re.MULTILINE)
# =============================================================================
# Helper Functions
# =============================================================================
@@ -16,9 +16,8 @@ import re
import sys
import typing
from fiona.librarian.md_frontmatter import FrontmatterError, get_file_version
from fiona.logging_config import load_settings, setup_logging
from fiona.paths import llm_wiki_template
from librarian.md_frontmatter import FrontmatterError, get_file_version
logger = logging.getLogger(__name__)
@@ -41,7 +40,9 @@ SETTINGS_NAME = "fiona-settings.json"
ARTIFACT_DIR_NAMES = frozenset({"build", "dist", "__pycache__"})
FILE_TOC_KEYS = frozenset({"min_heading", "max_heading", "sectionize", "ignore_files"})
DEFAULT_SETTINGS_FILE = llm_wiki_template() / "fiona-settings.json"
DEFAULT_SETTINGS_FILE = (
pathlib.Path(__file__).resolve().parent.parent / "llm-wiki" / "fiona-settings.json"
)
def load_toc_settings(settings_file: str | pathlib.Path | None = None) -> dict:
@@ -94,12 +95,9 @@ def depth_setting(settings: dict, key: str) -> int | None:
def slugify(text: str) -> str:
"""Convert a header text into a GitHub-style anchor (SPEC 7.3.1.1.3).
Each space becomes one hyphen; runs are kept so ``a | b`` is ``a--b``.
"""
"""Convert a header text into a GitHub-style anchor."""
slug = re.sub(r"[^\w\s-]", "", text.lower())
return re.sub(r"\s", "-", slug.strip())
return re.sub(r"\s+", "-", slug.strip())
def excluded(text: str, patterns: list[typing.Pattern]) -> bool:
@@ -111,7 +109,7 @@ def read_headers(path: pathlib.Path) -> list[tuple[int, str]]:
"""Return (level, text) for every header in a Markdown file, skipping code fences."""
headers: list[tuple[int, str]] = []
in_fence = False
# AI/LLM: utf-8-sig drops a leading BOM that would hide the first header.
# CURSOR: utf-8-sig drops a leading BOM that would hide the first header.
for line in path.read_text(encoding="utf-8-sig", errors="replace").splitlines():
if FENCE_RE.match(line):
in_fence = not in_fence
@@ -165,19 +163,15 @@ def build_toc(
) -> str:
"""Build a table of contents that nests folders, files and their headers.
Generated by AI/LLM: 2026-10-02 - frontmatter name and version for fmlint.
Args:
folder: The folder the table of contents covers.
files: Relative md file paths to list (as returned by collect_md).
max_header_depth: Maximum header level to list per file
(3 = h1 through h3). None or 0 mean unlimited.
"""
# AI/LLM: name and version are required by fmlint on every markdown file.
today = datetime.datetime.now(datetime.UTC).date().isoformat()
# Add YAML frontmatter
today = datetime.datetime.now(datetime.timezone.utc).date().isoformat()
frontmatter = f"""---
name: Table of Contents
version: 1.0.0
title: Table of Contents
created: {today}
updated: {today}
@@ -189,7 +183,7 @@ type: toc
for name in files:
parts = tuple(name.split("/"))
folders = parts[:-1]
# AI/LLM: emit a bullet only for folder levels that differ from the previous entry.
# CURSOR: emit a bullet only for folder levels that differ from the previous entry.
for depth in range(len(folders)):
if folders[: depth + 1] != previous[: depth + 1]:
lines.append("{}- {}".format(" " * depth, folders[depth]))
@@ -221,7 +215,7 @@ class FileTocError(Exception):
@dataclasses.dataclass(frozen=True)
class FileTocConfig:
"""Parameters for per-file TOC generation (SPEC 7.3.1.2).
"""Parameters for per-file TOC generation (SPEC 4.8.2).
Attributes:
min_heading: Smallest heading level included (never below 2;
@@ -366,7 +360,7 @@ def merge_marker_config(lines: list[str], base: FileTocConfig) -> FileTocConfig:
"""Apply the existing marker's params over the base config.
The marker records this TOC's own settings and has priority
(SPEC 7.3.1.2): a `sectionized` marker keeps section numbering on
(SPEC 4.8.2): a `sectionized` marker keeps section numbering on
and the recorded heading range sticks, so regenerations keep the
TOC's identity regardless of settings changes. Fields the marker
does not record (bare legacy markers, plain markers without
@@ -680,7 +674,7 @@ def _file_toc_star(sectionized: bool) -> None:
def file_toc_main(argv: list[str]) -> None:
"""CLI for per-file TOC generation (SPEC 7.3.1.2)."""
"""CLI for per-file TOC generation (SPEC 4.8.2)."""
parser = argparse.ArgumentParser(
prog="toc_generator file",
description="Generate an in-file table of contents below the H1 heading.",
@@ -754,7 +748,7 @@ def main() -> None:
for folder in target_folders(args.directory, patterns, args.recursive_toc):
files = collect_md(folder, patterns, not args.non_recursive, max_depth=max_dir_depth)
if files:
# AI/LLM: newline="\n" keeps Unix line endings on every platform.
# CURSOR: newline="\n" keeps Unix line endings on every platform.
with open(folder / TOC_NAME, "w", encoding="utf-8", newline="\n") as toc:
toc.write(build_toc(folder, files, max_header_depth=max_header_depth))
logger.info("wrote %s (%d file(s))", folder / TOC_NAME, len(files))
+77
View File
@@ -0,0 +1,77 @@
---
name: llm-wiki-agent-rules
title: LLM Wiki Agent Rules
version: 1.1.1
created: 2025-09-23
updated: 2026-09-27
type: summary
---
# LLM Wiki Agent Rules
- [1. Pre-requisites](#1-pre-requisites)
- [1.1 Install fiona-assistant library](#11-install-fiona-assistant-library)
- [1.2 fiona-assistant usage](#12-fiona-assistant-usage)
- [2. Directory Permissions](#2-directory-permissions)
- [3. Linking Rules](#3-linking-rules)
- [4. Updates](#4-updates)
- [5. Skills](#5-skills)
- [6. Boundaries](#6-boundaries)
## 1. Pre-requisites
### 1.1 Install fiona-assistant library
- Install with pip
- `python -m pip install fiona-assistant`
### 1.2 fiona-assistant usage
> TODO: add here how to use fiona-assistant for LLMs
## 2. Directory Permissions
- **`/personal/`**: Strictly **READ-ONLY**. Never create, edit, or delete notes here.
- **`llm-wiki/ingest/`**: Input processing zone. Parse files dropped here into
concept notes.
- **`llm-wiki/wiki/`**: **READ-WRITE**. Place synthesized concept pages in `/wiki/concepts/`.
- Only parsed/synthesized content belongs in `/wiki/concepts/`.
- **`llm-wiki/raw/`**: **READ-ONLY**. Raw source archive, never edited.
- **Raw files** (e.g., articles, papers, transcripts) must be stored in
`/raw/` and **never edited**.
- Subdirectories:
- `/raw/articles/` (web articles, clippings)
- `/raw/papers/` (PDFs, arXiv papers)
- `/raw/transcripts/` (meeting notes, interviews)
- `/raw/assets/` (images, diagrams referenced by sources)
- **`llm-wiki/utils/scripts`**: **READ-WRITE**. Place any script used and
created here.
## 3. Linking Rules
- Format concept note titles in **lowercase** or **Title-Case** to match page headings.
- Use standard markdown links for graph connectivity.
## 4. Updates
- Maintain a **master index** in `/wiki/index.md` whenever new concepts are synthesized.
- Append all actions to `/wiki/log.md` for chronological tracking.
## 5. Skills
- Skills are stored in `utils/skills/`.
- **`llm-wiki`** is the primary skill for maintaining this wiki.
- Learn more: [llm-wiki skill](./utils/skills/fiona/SKILL.md).
## 6. Boundaries
- **Do:**
- Follow directory permissions strictly.
- Update `index.md` and `log.md` for every change.
- Use provided skills for consistency.
- **Don’t:**
- Modify `/personal/` under any circumstances.
- Deviate from linking or naming conventions.
- Assume user intent without confirmation.
+4
View File
@@ -0,0 +1,4 @@
# llm-wiki package
# This package contains templates and utilities for managing the LLM wiki.
__version__ = "0.1.0"
@@ -1,5 +1,4 @@
{
"version": "1.0.1",
"markdown": {
"pymarkdown": {
"extensions": [
@@ -59,32 +58,20 @@
},
"wiki": {
"graph": {
"theme": "dark",
"show_external_links": true,
"subtitle": {
"enabled": true
"node_colors": {
"low": "#808080",
"mid": "#ffa500",
"high": "#FF0000"
},
"show_legend": true,
"show_arrows": false,
"show_source": true,
"show_glow": true,
"show_link_card": true,
"first_fit_ms": 12000,
"hover_link_tail_chars": 12,
"simplify_external_links": true,
"consolidate_external_links": true,
"consolidate_external_links_depth": 3,
"consolidate_external_links_max_char": 8,
"ignore_files": [
"*/table_of_contents.md",
"*/README.md",
"*/AGENTS.md"
]
},
"update": {
"ignore_files": [
"corefiles.json"
]
"node_radius": {
"min": 5,
"max": 40
},
"hover": {
"color": "#add8e6",
"edge_color": "#0000ff",
"size_multiplier": 1.3
}
}
}
}
+12
View File
@@ -0,0 +1,12 @@
---
name: map
version: 1.0.0
title: Map Script
type: summary
created: 2025-09-23
updated: 2026-09-24
---
# Map Script
Mapping utility script.
+12
View File
@@ -0,0 +1,12 @@
---
name: skills
version: 1.0.0
title: Skills Directory
type: summary
created: 2025-09-23
updated: 2026-09-24
---
# Skills Directory
Skill definitions and configurations.
+71
View File
@@ -0,0 +1,71 @@
---
name: fiona
category: productivity
description: LLM-Wiki Maintenance Assistant - Open Source Edition
author: nvaldez
license: MIT
version: 1.0.1
---
# FIONA-ASSISTANT Skill
**F**ocused **I**ntelligence **O**rganizer for **N**otes and **A**nalysis
The open-source counterpart to NADIA, designed for knowledge organization,
technical documentation, and information synthesis.
FIONA-ASSISTANT provides a community-driven approach to intelligent note management.
---
## Core Identity
You are **FIONA-ASSISTANT**, the **LLM-Wiki Maintenance Assistant**
specializing in:
- **F**ocused wiki organization
- **I**ntelligent note structuring
- **O**pen knowledge base management
- **N**avigation of knowledge graphs
- **A**utomated wiki maintenance
Your philosophy is:
> "A well-maintained wiki is the foundation of collective intelligence.
> I ensure knowledge is organized, accessible, and always up-to-date
> through open standards and community collaboration."
## LLM-Wiki Maintenance Approach
FIONA-ASSISTANT knows basic [LLM-Wiki](./llm-wiki-mod.md) Management
FIONA-ASSISTANT maintains LLM-wikis with strict adherence to standard Markdown rules:
1. **Standard Markdown Compliance**:
- Follows [markdownlint rules](./markdown_rules.md)
- Uses only standard Markdown syntax (`[text](path.md)` links, not wikilinks)
- Maintains consistent heading hierarchy (MD001, MD002)
- Enforces line length limits (MD013)
- Ensures proper list indentation (MD005, MD007)
2. **Open Knowledge Standards**:
- Uses YAML frontmatter for metadata (MD041)
- Standard Markdown tables (MD048)
- Fenced code blocks with language specification (MD046)
- Proper emphasis syntax (MD049, MD050)
3. **Obsidian Compatibility Layer**:
- Provides conversion tools between standard Markdown and Obsidian-specific formats
- Maintains compatibility without requiring proprietary extensions
- Preserves knowledge integrity when moving between tools
4. **Automated Markdown Validation**:
- Continuous linting with markdownlint
- Automatic correction of common Markdown issues
- Link validation and repair (MD052)
- Orphaned page detection
- Consistent formatting enforcement
5. **Knowledge Graph Integrity**:
- Ensures all links follow standard Markdown format
- Maintains consistent tagging and categorization
- Visualizes connections using standard Markdown-compatible tools
---
@@ -0,0 +1,15 @@
---
name: llm-wiki-mod
description: Modified Karpathy's LLM Wiki for FIONA
category: research
version: 1.0.0
---
# Karpathy's LLM Wiki
Build and maintain a persistent, compounding knowledge base as interlinked
markdown files.
Based on [Andrej Karpathy's LLM Wiki pattern](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f).
Unlike traditional RAG (which rediscovers knowledge from scratch per query),
the wiki compiles knowledge once and keeps it current. Cross-references are already there.
@@ -0,0 +1,97 @@
---
name: obsidian
description: Read, search, create, and edit notes in the Obsidian vault.
version: 1.0.0
author: Teknium (teknium1), Hermes Agent
license: MIT
platforms: [linux, macos, windows]
metadata:
hermes:
tags: [Obsidian, Notes, Markdown, Vault]
related_skills: []
---
# Obsidian Vault
Use this skill for filesystem-first Obsidian vault work: reading notes, listing
notes, searching note files, creating notes, appending content, and adding
wikilinks.
## Vault path
Use a known or resolved vault path before calling file tools.
The documented vault-path convention is the `OBSIDIAN_VAULT_PATH` environment
variable, for example from `${HERMES_HOME:-~/.hermes}/.env`. If it is unset,
use `~/Documents/Obsidian Vault`.
File tools do not expand shell variables. Do not pass paths containing
`$OBSIDIAN_VAULT_PATH` to `read_file`, `write_file`, `patch`, or
`search_files`; resolve the vault path first and pass a concrete absolute path.
Vault paths may contain spaces, which is another reason to prefer file tools
over shell commands.
If the vault path is unknown, `terminal` is acceptable for resolving
`OBSIDIAN_VAULT_PATH` or checking whether the fallback path exists. Once the
path is known, switch back to file tools.
## Read a note
Use `read_file` with the resolved absolute path to the note. Prefer this over
`cat` because it provides line numbers and pagination.
## List notes
Use `search_files` with `target: "files"` and the resolved vault path. Prefer
this over `find` or `ls`.
- To list all markdown notes, use `pattern: "*.md"` under the vault path.
- To list a subfolder, search under that subfolder's absolute path.
## Search
Use `search_files` for both filename and content searches. Prefer this over
`grep`, `find`, or `ls`.
- For filenames, use `search_files` with `target: "files"` and a filename `pattern`.
- For note contents, use `search_files` with `target: "content"`, the content
regex as `pattern`, and `file_glob: "*.md"` when you want to restrict matches
to markdown notes.
## Create a note
Use `write_file` with the resolved absolute path and the full markdown content.
Prefer this over shell heredocs or `echo` because it avoids shell quoting
issues and returns structured results.
## Append to a note
Prefer a native file-tool workflow when it is not awkward:
- Read the target note with `read_file`.
- Use `patch` for an anchored append when there is stable context, such as
adding a section after an existing heading or appending before a known
trailing block.
- Use `write_file` when rewriting the whole note is clearer than constructing a
fragile patch.
For an anchored append with `patch`, replace the anchor with the anchor plus
the new content.
For a simple append with no stable context, `terminal` is acceptable if it is
the clearest safe option.
## Targeted edits
Use `patch` for focused note changes when the current content gives you stable
context. Prefer this over shell text rewriting.
## Linking
For **compatibility** across tools (e.g., GitHub, VS Code, Obsidian), use
**standard Markdown links** (`[link text](path/to/note.md)`) instead of
wikilinks (`[[Note Name]]`).
- **Markdown links** ensure portability and work in all markdown renderers.
- If Obsidian-specific features (e.g., graph view) are needed, wikilinks can be
used **in addition to** Markdown links, but never as a replacement.

Some files were not shown because too many files have changed in this diff Show More