Compare commits
44 Commits
shellcheck
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| a3d35c7f12 | |||
| def3d2cd6d | |||
| b7c173affa | |||
| 59d1054e1d | |||
| 983198b8b4 | |||
| a907e7da79 | |||
| b0539d6f10 | |||
| dfea2bccfa | |||
| d6a265d4be | |||
| 1c40d252b6 | |||
| 295089c748 | |||
| de28441943 | |||
| fd72e4d66b | |||
| e9542ccf88 | |||
| dcc0536f6e | |||
| 9ff5af588a | |||
| 4c84213a30 | |||
| 09362f3951 | |||
| 3c21217970 | |||
| 8a00ce97f5 | |||
| 348579a085 | |||
| 48b94a5781 | |||
| 20378727fa | |||
| cd1ea32635 | |||
| 44029aee90 | |||
| c45d4f1d2a | |||
| 9e98390f2b | |||
| 482b2cb2e0 | |||
| 2d08b71b55 | |||
| 31364eab2c | |||
| a1fccbe641 | |||
| da01b3fe4a | |||
| 94d3554d77 | |||
| b93a5a0c02 | |||
| 151282657e | |||
| 0907252d4c | |||
| 3471ff9fe6 | |||
| 32920d1736 | |||
| fd58ad549a | |||
| 02a159a086 | |||
| 6e334cc34d | |||
| e14f96b590 | |||
| 859d2e55d7 | |||
| e6920aa691 |
-19
@@ -1,19 +0,0 @@
|
||||
[run]
|
||||
source = src
|
||||
omit =
|
||||
src/tests/*
|
||||
*/tests/*
|
||||
*/test_*.py
|
||||
src/mdoctor/fmlint.py
|
||||
src/python_ruff/pycustomlint.py
|
||||
src/llm-wiki/__init__.py
|
||||
|
||||
[report]
|
||||
exclude_lines =
|
||||
pragma: no cover
|
||||
def __repr__
|
||||
raise AssertionError
|
||||
raise NotImplementedError
|
||||
if __name__ == .__main__.:
|
||||
if TYPE_CHECKING:
|
||||
@abstractmethod
|
||||
+10
-1
@@ -5,6 +5,7 @@ __pycache__/
|
||||
|
||||
# Virtual environments
|
||||
.venv/
|
||||
.venvs/
|
||||
venv/
|
||||
env/
|
||||
ENV/
|
||||
@@ -34,6 +35,7 @@ dist/
|
||||
|
||||
# Coverage
|
||||
.coverage
|
||||
coverage.xml
|
||||
htmlcov/
|
||||
|
||||
# Jupyter Notebook checkpoints
|
||||
@@ -45,10 +47,17 @@ htmlcov/
|
||||
# Windows
|
||||
Thumbs.db
|
||||
|
||||
# Obsidian per-user vault state
|
||||
.obsidian/workspace*.json
|
||||
.obsidian/cache
|
||||
|
||||
# Python
|
||||
*.pyc
|
||||
*.pyo
|
||||
*.pyd
|
||||
|
||||
*fiona.log
|
||||
*wiki_graph.html
|
||||
*wiki_graph.html
|
||||
|
||||
# Generated review artifacts (git diff HTML, scratch exports)
|
||||
tmp/
|
||||
@@ -1,21 +1,23 @@
|
||||
---
|
||||
name: Fiona-Assistant Agent Rules
|
||||
version: 1.3.1
|
||||
title: LLM Agent Rules
|
||||
name: Fiona-Assistant Development Agent Rules
|
||||
version: 1.9.3
|
||||
title: Agents
|
||||
type: summary
|
||||
created: 2025-09-23
|
||||
updated: 2026-09-26
|
||||
updated: 2026-10-06
|
||||
tags: [agents, rules, guidelines, llm, ai]
|
||||
sources: []
|
||||
---
|
||||
|
||||
# Fiona-Assistant LLM Agent Rules
|
||||
# Agents
|
||||
|
||||
- [1. Overview](#1-overview)
|
||||
- [1.1 Rule References](#11-rule-references)
|
||||
- [2. Basic Rules (MANDATORY)](#2-basic-rules-mandatory)
|
||||
- [2.1 Testing](#21-testing)
|
||||
- [2.2 Documentation](#22-documentation)
|
||||
- [2.1 Development Lifecycle](#21-development-lifecycle)
|
||||
- [2.2 Testing](#22-testing)
|
||||
- [2.3 Documentation](#23-documentation)
|
||||
- [2.4 Commit](#24-commit)
|
||||
- [3. Environment Setup (MANDATORY)](#3-environment-setup-mandatory)
|
||||
- [3.1 Always Use Bootstrap for Environment](#31-always-use-bootstrap-for-environment)
|
||||
- [4. Python Development Rules](#4-python-development-rules)
|
||||
@@ -55,34 +57,80 @@ Fiona-Assistant repository.
|
||||
|
||||
Refer to rules in this file as `project agent rule N.M` (e.g.
|
||||
`project agent rule 1.1`) to differentiate them from rules in other
|
||||
AGENTS.md files (e.g. the llm-wiki template's `src/llm-wiki/AGENTS.md`).
|
||||
AGENTS.md files (e.g. the llm-wiki template's `src/fiona/data/llm_wiki/AGENTS.md`).
|
||||
|
||||
---
|
||||
|
||||
## 2. Basic Rules (MANDATORY)
|
||||
|
||||
### 2.1 Testing
|
||||
### 2.1 Development Lifecycle
|
||||
|
||||
- For testing scripts always run `python src/run_tests.py` and make sure
|
||||
Every change that alters behavior, adds a feature, or changes a
|
||||
requirement follows these five steps, in order. Finish a step before
|
||||
starting the next. A read-only question skips this lifecycle.
|
||||
|
||||
Before step 2, load the environment (project agent rule 3.1).
|
||||
|
||||
1. **Write or check the spec.** Read [SPEC](./SPEC.md) for the
|
||||
area this task will change.
|
||||
- When the requirement or its `*acceptance*` is missing, write
|
||||
both in [SPEC](./SPEC.md) before any test or code. Follow
|
||||
SPEC 1.1 and set the section status.
|
||||
- When the requirement exists, confirm the task matches that
|
||||
text. That text is the behavior to build.
|
||||
2. **Write tests.** Add tests that fail for that acceptance
|
||||
(project agent rule 6.1). Leave production code unchanged in
|
||||
this step.
|
||||
3. **Implement, TDD style.** Load the
|
||||
[python-tdd skill](./src/fiona/data/llm_wiki/utils/skills/python-tdd/SKILL.md)
|
||||
(project agent rule 4.1).
|
||||
- Red: run the new tests and confirm they fail.
|
||||
- Green: write the smallest change that makes them pass.
|
||||
- Refactor while those tests stay green.
|
||||
4. **Verify the spec.** Re-read the requirement and each
|
||||
acceptance this task touched. The code and the tests satisfy
|
||||
that text.
|
||||
- When they disagree, update [SPEC](./SPEC.md) in this same
|
||||
task. Move status along `Planned` ->
|
||||
`Partially Implemented` -> `Implemented`, or set
|
||||
`Won't Do` when the requirement is rejected.
|
||||
- Run `python scripts/run_tests.py`, then
|
||||
`python scripts/run_sanity.py --non-interactive`. Both pass
|
||||
(project agent rule 2.2).
|
||||
5. **Bump versions.** On each markdown file this task edited,
|
||||
bump frontmatter `version` and set `updated` to the edit date
|
||||
(project agent rule 10.1).
|
||||
- Semver: patch for a clarification, minor for a new rule or
|
||||
requirement, major for a breaking contract change.
|
||||
- Bump [SPEC](./SPEC.md) when step 1 or step 4 changed it.
|
||||
- Python and shell files have no version tag yet (SPEC 7.3.2.1).
|
||||
Leave them untagged.
|
||||
|
||||
A documentation edit that does not change product behavior still
|
||||
bumps versions (step 5). It skips the test and implementation steps.
|
||||
|
||||
### 2.2 Testing
|
||||
|
||||
- For testing scripts always run `python scripts/run_tests.py` and make sure
|
||||
everything is passing on every loop
|
||||
- After Modifying the repo, make sure `python src/run_sanity.py --non-interactive` passes before finalizing.
|
||||
- After modifying the repo, make sure
|
||||
`python scripts/run_sanity.py --non-interactive` passes before finalizing.
|
||||
|
||||
### 2.2 Documentation
|
||||
### 2.3 Documentation
|
||||
|
||||
- Before changing any functionality, read [SPEC](./SPEC.md) (the project
|
||||
spec sheet) so the change follows the specified requirements and
|
||||
acceptance criteria.
|
||||
- When documentation is changed (e.g., `SPEC.md`, especially the table of
|
||||
contents section), double-check that all references to it are unbroken.
|
||||
- Always link via markdownlink for connectivity.
|
||||
- Whenever a change alters behavior, adds a feature, or changes a
|
||||
requirement, update [SPEC](./SPEC.md) in the same task so the spec stays
|
||||
in sync (adjust section status: `planned` -> `partial` -> `implemented`;
|
||||
use `won't do` for rejected features).
|
||||
- Product behavior is specified in [SPEC](./SPEC.md). The order for
|
||||
reading it, writing it, and checking it after the code is project
|
||||
agent rule 2.1.
|
||||
- When documentation changes (for example [SPEC](./SPEC.md), especially
|
||||
the table of contents), confirm every reference to it still resolves.
|
||||
- Always link with a markdown link.
|
||||
|
||||
### 2.3 Commit
|
||||
### 2.4 Commit
|
||||
|
||||
- *NEVER* commit without explicit permission!
|
||||
- This rule covers this repository only. A deployed wiki commits
|
||||
after each ingest cycle and housekeeping pass
|
||||
([SPEC](./SPEC.md) 3.3.3).
|
||||
|
||||
---
|
||||
|
||||
@@ -90,7 +138,9 @@ AGENTS.md files (e.g. the llm-wiki template's `src/llm-wiki/AGENTS.md`).
|
||||
|
||||
### 3.1 Always Use Bootstrap for Environment
|
||||
|
||||
> BEFORE any testing or development work, you MUST source the bootstrap script
|
||||
> BEFORE any testing or development work, you MUST load the bootstrap script
|
||||
|
||||
Linux / macOS (bash, must be sourced):
|
||||
|
||||
```bash
|
||||
# Bare minimum - always run this first
|
||||
@@ -104,6 +154,30 @@ source bootstrap --env <env_name> --python <version>
|
||||
|
||||
# With cleanup flag
|
||||
source bootstrap --env <env_name> -c
|
||||
|
||||
# This checkout as a normal package (not editable)
|
||||
source bootstrap --local
|
||||
|
||||
# Current user, no virtual environment
|
||||
source bootstrap --local --no-env
|
||||
```
|
||||
|
||||
Windows (PowerShell). A virtual-environment install must be
|
||||
dot-sourced. This creates a repo-local venv at `.venvs\<name>` with
|
||||
the Python launcher. It does not install pyenv. `--no-env` skips that
|
||||
venv and may be executed with `-File`. `--local` installs this
|
||||
repository with `pip install .` (not editable, no `dev` extra).
|
||||
`--no-env` installs with `pip install --user`, so `fiona-assistant`
|
||||
runs without activating an environment. The two flags combine.
|
||||
|
||||
```powershell
|
||||
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
|
||||
. .\bootstrap.ps1
|
||||
. .\bootstrap.ps1 --env fiona_env
|
||||
. .\bootstrap.ps1 --env <env_name> --python <version>
|
||||
. .\bootstrap.ps1 --env <env_name> -c
|
||||
. .\bootstrap.ps1 --local
|
||||
powershell -ExecutionPolicy Bypass -File .\bootstrap.ps1 --local --no-env
|
||||
```
|
||||
|
||||
**Rationale**: The bootstrap script sets up the Python virtual environment,
|
||||
@@ -119,11 +193,13 @@ dependency conflicts, incorrect Python versions, or missing packages.
|
||||
|
||||
### 4.1 Load Python TDD Skill for Python Tasks
|
||||
|
||||
> When modifying or creating Python scripts, the Python TDD Skill MUST be loaded
|
||||
> When modifying or creating Python scripts, the
|
||||
> [python-tdd skill](./src/fiona/data/llm_wiki/utils/skills/python-tdd/SKILL.md)
|
||||
> MUST be loaded
|
||||
|
||||
```text
|
||||
SKILL: python-tdd
|
||||
PATH: llm-wiki/utils/skills/python-tdd/SKILL.md
|
||||
PATH: src/fiona/data/llm_wiki/utils/skills/python-tdd/SKILL.md
|
||||
```
|
||||
|
||||
- Project Rules:
|
||||
@@ -132,6 +208,8 @@ PATH: llm-wiki/utils/skills/python-tdd/SKILL.md
|
||||
|
||||
**Required Practices from Python TDD Skill:**
|
||||
|
||||
- Step 3 of project agent rule 2.1 is this cycle. The spec and the
|
||||
failing tests already exist before implementation starts.
|
||||
- Strictly follow **Test-Driven Development (TDD)** - Red-Green-Refactor cycle
|
||||
- Use **pytest** for testing
|
||||
- Follow **OOP principles** and **SOLID** design patterns
|
||||
@@ -160,29 +238,33 @@ The repository has the following structure with specific permissions:
|
||||
|
||||
```text
|
||||
fiona-assistant/
|
||||
├── src/ # Source code
|
||||
│ ├── llm-wiki/ # Packaged llm-wiki template (data only)
|
||||
│ │ # Structure source of truth:
|
||||
│ │ # `src/llm-wiki/wiki/schema.md`
|
||||
│ ├── fiona/ # Main CLI package
|
||||
├── src/fiona/ # The only installable package
|
||||
│ ├── __main__.py # CLI entry point
|
||||
│ ├── librarian/ # Wiki content management
|
||||
│ ├── mdoctor/ # Markdown diagnosis
|
||||
│ ├── python_ruff/ # Python code linting
|
||||
│ ├── wiki/ # Wiki lifecycle modules (init, update)
|
||||
│ ├── tests/ # Unit tests
|
||||
│ ├── run_tests.py # Test runner
|
||||
│ ├── run_sanity.py # Sanity runner (tests + linters)
|
||||
│ ├── customlinter/ # Frontmatter, markdown, Python, JSON linters
|
||||
│ ├── python_ruff/ # Ruff wrapper
|
||||
│ ├── gittool/ # Git operations
|
||||
│ ├── sanity_runner/ # Shared sanity engine
|
||||
│ ├── wiki/ # Wiki lifecycle (init, update, sanity, graph)
|
||||
│ ├── data/llm_wiki/ # Packaged template (data only)
|
||||
│ ├── run_tests.py # Test runner module
|
||||
│ └── run_sanity.py # Sanity runner module
|
||||
├── tests/ # Unit tests (not installed)
|
||||
├── scripts/ # Repo tooling, not package source
|
||||
│ ├── run_tests.py # Launcher for fiona.run_tests
|
||||
│ ├── run_sanity.py # Launcher for fiona.run_sanity
|
||||
│ ├── settings_dev.json # Development settings
|
||||
│ ├── utils.sh # Shell utilities
|
||||
│ └── install_packages.sh # Dependency installer
|
||||
│
|
||||
├── pyproject.toml # Project configuration
|
||||
├── bootstrap # Environment setup
|
||||
├── bootstrap # Environment setup (bash; source it)
|
||||
├── bootstrap.ps1 # Environment setup (PowerShell; dot-source it)
|
||||
└── README.md # Project documentation
|
||||
```
|
||||
|
||||
The llm-wiki directory structure is not duplicated here:
|
||||
`src/llm-wiki/wiki/schema.md` is the final source of truth for the
|
||||
`src/fiona/data/llm_wiki/wiki/schema.md` is the final source of truth for the
|
||||
llm-wiki directory structure.
|
||||
|
||||
**General Rules:**
|
||||
@@ -191,8 +273,8 @@ llm-wiki directory structure.
|
||||
- `/raw/` directories are **READ-ONLY** - Never edit archived source materials
|
||||
- `/wiki/concepts/` is **READ-WRITE** - Place synthesized content here
|
||||
- `/ingest/` is for **input processing** - Parse files dropped here
|
||||
- **Always update** `src/llm-wiki/wiki/index.md` when adding new concepts
|
||||
- **Always append** to `src/llm-wiki/wiki/log.md` for every action
|
||||
- **Always update** `src/fiona/data/llm_wiki/wiki/index.md` when adding new concepts
|
||||
- **Always append** to `src/fiona/data/llm_wiki/wiki/log.md` for every action
|
||||
|
||||
---
|
||||
|
||||
@@ -200,6 +282,7 @@ llm-wiki directory structure.
|
||||
|
||||
### 6.1 Testing Requirements
|
||||
|
||||
Steps 2 and 3 of project agent rule 2.1.
|
||||
**All code changes MUST be tested** following TDD principles:
|
||||
|
||||
1. **Write failing tests first** (Red phase)
|
||||
@@ -210,7 +293,7 @@ llm-wiki directory structure.
|
||||
|
||||
```bash
|
||||
# Run all tests
|
||||
python src/run_tests.py
|
||||
python scripts/run_tests.py
|
||||
|
||||
# Run with pytest directly
|
||||
python -m pytest
|
||||
@@ -219,7 +302,7 @@ python -m pytest
|
||||
python -m pytest --cov=src
|
||||
|
||||
# Run specific test file
|
||||
python -m pytest src/tests/test_md_frontmatter.py -v
|
||||
python -m pytest tests/test_md_frontmatter.py -v
|
||||
```
|
||||
|
||||
**Test File Structure:**
|
||||
@@ -294,35 +377,19 @@ git commit -m "test: add TDD tests for name/version extraction"
|
||||
|
||||
### 9.1 Development Workflow
|
||||
|
||||
**Standard Workflow:**
|
||||
The development lifecycle is project agent rule 2.1. Do those five
|
||||
steps in order: write or check the spec, write tests, implement
|
||||
TDD-style, verify the spec, bump versions.
|
||||
|
||||
1. **Setup Environment** (MANDATORY)
|
||||
Before step 2 of that lifecycle:
|
||||
|
||||
```bash
|
||||
source bootstrap --env <env_name>
|
||||
```
|
||||
|
||||
2. **Load Appropriate Skill** (if applicable)
|
||||
- Python tasks: Load `python-tdd` skill
|
||||
- Wiki tasks: Load `llm-wiki` skill
|
||||
|
||||
3. Read the Spec
|
||||
- Read [SPEC](./SPEC.md) for the feature being changed
|
||||
- Confirm the change matches the specified requirements and acceptance criteria
|
||||
|
||||
4. Follow TDD for Code Changes
|
||||
- Write tests first
|
||||
- Implement minimal code
|
||||
- Refactor
|
||||
|
||||
5. Validate
|
||||
- Run tests
|
||||
- Run linters
|
||||
|
||||
6. Document
|
||||
- Update docstrings
|
||||
- Update README if needed
|
||||
- Update [SPEC](./SPEC.md) if the change affects any specified feature or requirement
|
||||
1. Set up the environment (project agent rule 3.1).
|
||||
2. Load the skill when the task needs one (project agent rule 4.1).
|
||||
Python tasks load the
|
||||
[python-tdd skill](./src/fiona/data/llm_wiki/utils/skills/python-tdd/SKILL.md).
|
||||
Wiki tasks load the
|
||||
[Fiona skill](./src/fiona/data/llm_wiki/utils/skills/fiona/SKILL.md)
|
||||
unless `src/fiona/data/llm_wiki/AGENTS.md` already loaded it.
|
||||
|
||||
---
|
||||
|
||||
@@ -331,7 +398,12 @@ git commit -m "test: add TDD tests for name/version extraction"
|
||||
### 10.1 MD File Frontmatter
|
||||
|
||||
**All markdown files in llm-wiki/ MUST have YAML frontmatter** with the
|
||||
required `name` and `version` fields. All other fields are optional:
|
||||
required `name` and `version` fields. Every page under `wiki/` also
|
||||
requires the quality signals `confidence`, `llm`, `contested`, and
|
||||
`contradictions`. How to set them is section 15 of the
|
||||
[Fiona skill](./src/fiona/data/llm_wiki/utils/skills/fiona/SKILL.md).
|
||||
All other fields are optional. fmlint enforces `name` and `version`
|
||||
only.
|
||||
|
||||
```yaml
|
||||
---
|
||||
@@ -343,10 +415,29 @@ updated: YYYY-MM-DD # optional (bump when editing)
|
||||
type: entity | concept | comparison | query | summary | toc
|
||||
sources: [raw/articles/source-name.md] # optional
|
||||
tags: [from taxonomy] # optional
|
||||
# Optional quality signals:
|
||||
# quality signals (required on pages under wiki/):
|
||||
confidence: trusted | high | medium | low | zero
|
||||
llm:
|
||||
- model: none # required; model that last edited the page
|
||||
- reasoning: high # when the model reports a level
|
||||
contested: true
|
||||
contradictions: [other-page-slug]
|
||||
# written by wiki ingest (SPEC 7.1.5), optional:
|
||||
status: raw | refined
|
||||
source_meta: {author: Name} # metadata read from the source
|
||||
ingest: # provenance; every key always present
|
||||
origin: https://host/path # URL or file path of the source
|
||||
origin_kind: url | file
|
||||
final_url: null # URL after redirects
|
||||
fetched: YYYY-MM-DDTHH:MM:SSZ
|
||||
media_type: text/html
|
||||
size_bytes: 0
|
||||
sha256: <64 lowercase hex> # of the raw copy; never hand-edited
|
||||
raw_reused: false
|
||||
converter: html # converter name, or agent
|
||||
conversion: complete | partial
|
||||
warnings: []
|
||||
tool: fiona-assistant <version>
|
||||
---
|
||||
```
|
||||
|
||||
@@ -357,7 +448,7 @@ contradictions: [other-page-slug]
|
||||
- Use semantic versioning (MAJOR.MINOR.PATCH)
|
||||
|
||||
This is the single canonical frontmatter schema; `wiki/schema.md` documents
|
||||
the same schema for the wiki template (see SPEC 4.2.1).
|
||||
the same schema for the wiki template (see SPEC 6.2).
|
||||
|
||||
---
|
||||
|
||||
@@ -396,7 +487,8 @@ the same schema for the wiki template (see SPEC 4.2.1).
|
||||
When multiple tasks are possible, prioritize in this order:
|
||||
|
||||
1. **User's explicit request** (highest priority)
|
||||
2. **TDD requirements** (tests before implementation)
|
||||
2. **Development lifecycle** (project agent rule 2.1): spec, then
|
||||
tests, then implementation
|
||||
3. **Code quality** (linting, formatting)
|
||||
4. **Documentation** (docstrings, README)
|
||||
5. **Refactoring** (improving existing code)
|
||||
@@ -409,6 +501,8 @@ When multiple tasks are possible, prioritize in this order:
|
||||
### 14.1 Things to Never Do
|
||||
|
||||
- Never modify `/personal/` or `/raw/` directories
|
||||
- Never write production code before the spec states the requirement
|
||||
and a failing test exists (project agent rule 2.1)
|
||||
- Never commit code without tests
|
||||
- Never commit code that fails linting
|
||||
- Never ignore AGENTS.md rules without explicit user override
|
||||
@@ -434,14 +528,17 @@ These rules **CAN** be overridden by:
|
||||
|
||||
| Task | Required Action |
|
||||
| ---- | -------------- |
|
||||
| Any Python work | Load python-tdd skill |
|
||||
| Before testing | `source bootstrap` |
|
||||
| Changing functionality | Read [SPEC](./SPEC.md) first |
|
||||
| Modifying code | Write tests first (TDD) |
|
||||
| Development | project agent rule 2.1, in order |
|
||||
| Any Python work | Load the [python-tdd skill](./src/fiona/data/llm_wiki/utils/skills/python-tdd/SKILL.md) |
|
||||
| Before testing | `source bootstrap` (Windows: `. .\bootstrap.ps1`) |
|
||||
| Local install, no venv | `bootstrap --local --no-env` |
|
||||
| Step 1 | Write or check [SPEC](./SPEC.md) |
|
||||
| Step 2 and 3 | Failing tests, then the smallest change |
|
||||
| Step 4 | Re-read the spec; tests and sanity pass |
|
||||
| Step 5 | Bump `version` and `updated` on edited markdown |
|
||||
| Before commit | Run tests + linters |
|
||||
| After behavior change | Update [SPEC](./SPEC.md) to stay in sync |
|
||||
| Wiki changes | Update index.md + log.md |
|
||||
| Frontmatter | Include name, version, title, dates |
|
||||
| Wiki changes | Load the [Fiona skill](./src/fiona/data/llm_wiki/utils/skills/fiona/SKILL.md); update index.md + log.md |
|
||||
| Frontmatter | `name` and `version` on every file; on `wiki/` pages also `confidence`, `llm`, `contested`, `contradictions` |
|
||||
| Errors | Raise exceptions, don't fail silently |
|
||||
|
||||
---
|
||||
|
||||
+229
@@ -0,0 +1,229 @@
|
||||
---
|
||||
name: changelog
|
||||
version: 1.2.3
|
||||
title: Changelog
|
||||
created: 2026-10-08
|
||||
updated: 2026-10-24
|
||||
type: summary
|
||||
---
|
||||
|
||||
# Changelog
|
||||
|
||||
Revisions of [SPEC](./SPEC.md), newest first. Each entry names the
|
||||
requirements it adds or changes. "Unbuilt" means the requirement is
|
||||
specified and marked `NOT_IMPLEMENTED`.
|
||||
|
||||
## 1.22.1 - 2026-10-24
|
||||
|
||||
- Minor corrections to 1.22.0.
|
||||
|
||||
## 1.22.0 - 2026-10-24
|
||||
|
||||
- Adds `--staged` and `--unstaged` flags to `git diff` for staged-only and
|
||||
unstaged-only diffs (*7.12.7.*, *7.12.8.*).
|
||||
|
||||
## 1.21.0 - 2026-10-24
|
||||
|
||||
- Includes untracked and deleted files in `git diff` report by default
|
||||
(*7.12.6.*).
|
||||
- Updates diff rendering: removes spacing between consecutive added or
|
||||
deleted lines, preserves spacing between added and deleted blocks (*7.12.5.*).
|
||||
|
||||
## 1.20.0 - 2026-10-24
|
||||
|
||||
- Git commands discover and operate on the root repository from the current
|
||||
directory (*4.6.*, *7.12.0.*).
|
||||
|
||||
## 1.19.1 - 2026-10-21
|
||||
|
||||
- The sectionize-markdown skill ships as
|
||||
`utils/skills/sectionize-markdown/SKILL.md` (*3.4.2.*), matching the
|
||||
`<name>/SKILL.md` convention of the other skills. Every reference to
|
||||
the old `sectionize-markdown.md` path now points at `SKILL.md`.
|
||||
|
||||
## 1.18.1 - 2026-10-19
|
||||
|
||||
- `fiona-assistant git diff` writes a constellation-themed HTML
|
||||
report under the wiki `tmp/` directory (*4.6*, *7.12.5*). The
|
||||
packaged wiki `.gitignore` now lists `tmp/`.
|
||||
|
||||
## 1.18.0 - 2026-10-15
|
||||
|
||||
- Adds skill memory to the template (*3.5.*): the `memory` skill
|
||||
under `utils/skills/memory/` says how a skill keeps its own
|
||||
`MEMORIES.md` beside its `SKILL.md`, and ships the blank
|
||||
`MEMORIES.md` to copy. Wiki-wide preferences stay in `MEMORY.md`.
|
||||
|
||||
## 1.17.0 - 2026-10-15
|
||||
|
||||
- The `.gitignore` written by `wiki init` ignores Obsidian's per-user
|
||||
vault state, `.obsidian/workspace*.json` and `.obsidian/cache`, so a
|
||||
wiki opened as an Obsidian vault stays clean between saves. Shared
|
||||
vault settings such as `.obsidian/app.json` stay tracked (*3.2.3.*).
|
||||
|
||||
## 1.16.0 - 2026-10-14
|
||||
|
||||
- Adds the agent memory `MEMORY.md` to the template (*3.4.*): a
|
||||
sectionized file of owner preferences, corrections, and verified
|
||||
facts, linked from `AGENTS.md`. `AGENTS.md` says to update it on
|
||||
an important memory or when the user explicitly asks (*3.4.6.*). `wiki update` adds it when missing
|
||||
and never compares or replaces it after that (*3.4.4.*).
|
||||
|
||||
## 1.14.1 - 2026-10-08
|
||||
|
||||
- Moves the revision history out of the spec into this file.
|
||||
|
||||
## 1.14.0 - 2026-10-08
|
||||
|
||||
- Stages raw pages in a `refine/` directory at the wiki root, and has
|
||||
the agent move each refined page into `wiki/` (*7.1.5.15.*).
|
||||
Unbuilt.
|
||||
|
||||
## 1.13.0 - 2026-10-07
|
||||
|
||||
- Builds login detection with exit 4 (*7.1.5.2.8.*), `--origin`
|
||||
(*7.1.5.2.9.*), the agent steps for exit 4 (*7.1.5.14.*), and the
|
||||
HTML block sweep (*7.1.5.4.5.6.*).
|
||||
|
||||
## 1.11.0
|
||||
|
||||
- Specifies `wiki search` (*4.7.7.*, *7.1.6.*), a keyword search that
|
||||
prints `<path>:<line>: <text>` and shortens long lines around the
|
||||
match. Unbuilt.
|
||||
|
||||
## 1.10.0
|
||||
|
||||
- Replaces the first-`<article>` content root of the `html` converter
|
||||
with a block sweep that scores every block as `keep`, `review`, or
|
||||
`drop` (*7.1.5.4.5.6.*). Unbuilt.
|
||||
|
||||
## 1.9.0
|
||||
|
||||
- `wiki ingest` stops with exit 4 on a URL that needs a login
|
||||
(*7.1.5.2.8.*). Unbuilt.
|
||||
- Adds `--origin` for a file saved by hand (*7.1.5.2.9.*). Unbuilt.
|
||||
- Tells the agent how to get the source instead (*7.1.5.14.*).
|
||||
Unbuilt.
|
||||
|
||||
## 1.8.0
|
||||
|
||||
Adds the findings of the 2026-10-02 manual test. All unbuilt:
|
||||
|
||||
- One-line CLI errors with no traceback (*1.2.7.*, *7.1.1.4.*).
|
||||
- `--version` (*2.3.*).
|
||||
- A quiet default stderr (*7.17.3.*).
|
||||
- An ingest hint for a drop-zone file that already moved
|
||||
(*7.1.5.2.7.*).
|
||||
- A dry run that does not log writes (*7.1.5.8.1.*).
|
||||
- A lint-clean ingest log line (*7.1.5.6.1.*).
|
||||
- A follow-up hint for style violations left on a page
|
||||
(*7.1.5.5.6.*).
|
||||
- Single-file `mdoctor` (*7.4.4.*).
|
||||
- Compact housekeeping sanity details (*7.2.5.1.*).
|
||||
- A quiet graph overwrite (*7.1.4.5.9.1.*).
|
||||
- A usage section in the README (*7.18.4.*).
|
||||
|
||||
## 1.7.0
|
||||
|
||||
- `librarian move` rewrites links to the moved paths across the whole
|
||||
wiki (*7.3.3.6.*).
|
||||
|
||||
## 1.6.0
|
||||
|
||||
- Specifies `librarian move` (*7.3.3.*). Unbuilt.
|
||||
|
||||
## 1.5.2
|
||||
|
||||
- Gives `table_of_contents.md` frontmatter `name` and `version`
|
||||
(*7.3.1.1.1.*), so `fmlint` accepts it.
|
||||
|
||||
## 1.5.0
|
||||
|
||||
- Omits markdown files matched by `fiona.wiki.graph.ignore_files`
|
||||
from the graph (*7.1.4.20.*). The shipped list is
|
||||
`*/table_of_contents.md`, because a table of contents links to
|
||||
every page.
|
||||
|
||||
## 1.4.0
|
||||
|
||||
- Names the core-file record `corefiles.json` (*7.1.2.14.*).
|
||||
- Skips paths in `fiona.wiki.update.ignore_files` (*7.1.2.15.*). The
|
||||
shipped list names `corefiles.json`. Only `wiki update` writes that
|
||||
file.
|
||||
|
||||
## 1.3.0
|
||||
|
||||
- Specifies directory nodes (*7.1.4.18.*) and `wiki graph --serve`
|
||||
(*7.1.4.19.*). Both unbuilt.
|
||||
|
||||
## 1.2.0
|
||||
|
||||
- Folds text line endings before the SHA-256 compare (*7.1.2.13.*).
|
||||
- Writes `corefiles.json`, so a later update does not list a core
|
||||
file whose version and hashes are unchanged (*7.1.2.14.*).
|
||||
|
||||
## 1.1.0
|
||||
|
||||
- Treats a differing `fiona-settings.json` as a conflict and writes
|
||||
`fiona-settings-update.json` (*7.1.2.3.*).
|
||||
|
||||
## 1.0.0
|
||||
|
||||
- Groups each module's functions under that module.
|
||||
- Moves Non-Goals, Priority Order, and Open Questions to Appendices
|
||||
A, B, and C.
|
||||
|
||||
## 0.99.0
|
||||
|
||||
- Builds `wiki ingest` (*7.1.5.*).
|
||||
|
||||
## 0.98.0
|
||||
|
||||
- Specifies that `wiki graph` omits a file matched by `.gitignore`
|
||||
(*7.1.4.17.*). Unbuilt.
|
||||
|
||||
## 0.97.0
|
||||
|
||||
- Rejects a module-level assignment that follows the first function
|
||||
or class (*6.4.3.*).
|
||||
|
||||
## 0.96.0
|
||||
|
||||
- Compares a file without a version by SHA-256 and shows both short
|
||||
hashes in the update report (*7.1.2.12.*).
|
||||
|
||||
## 0.94.0
|
||||
|
||||
- `wiki update` yields to the owner's markdown and writes
|
||||
`<stem>-update.md` instead (*7.1.2.11.*).
|
||||
- Adds `fiona-assistant housekeeping` (*7.2.*).
|
||||
|
||||
## 0.93.0
|
||||
|
||||
- Adds SHA-256 reuse across `raw/`, the complete `ingest`
|
||||
frontmatter map, and the agent metadata check.
|
||||
|
||||
## 0.92.0
|
||||
|
||||
- Specifies `wiki ingest` (*7.1.5.*). Unbuilt.
|
||||
|
||||
## 0.90.0
|
||||
|
||||
- Specifies `node_importance_min_point` (*7.1.4.16.*). Unbuilt.
|
||||
|
||||
## 0.41.0
|
||||
|
||||
- Installs one `fiona` package and keeps tests and repo scripts
|
||||
outside it.
|
||||
|
||||
## 0.40.4
|
||||
|
||||
- Adds the PYC001 exception (*6.4.2.*).
|
||||
|
||||
## 0.40.1
|
||||
|
||||
- Numbers nested requirements such as `7.14.1.1`.
|
||||
|
||||
## 0.40.0
|
||||
|
||||
- Reorganizes the sheet into contracts.
|
||||
@@ -1,356 +1,436 @@
|
||||
# Fiona-Assistant
|
||||
|
||||
LLM Wiki Tool Helper for LLM Agents/Hives
|
||||
> Your LLM agent writes the wiki. Fiona keeps it clean.
|
||||
|
||||
Fiona-Assistant is a command-line toolkit designed to manage, organize, and
|
||||
maintain LLM wiki content with minimal AI token dependence. It combines
|
||||
script-based automation with structured workflows to help LLM Agents and
|
||||
Hives efficiently manage documentation, markdown files, and code quality.
|
||||
Fiona-Assistant is a command-line toolkit for running an **llm-wiki**: a
|
||||
folder of interlinked markdown that an AI agent builds and maintains for
|
||||
you. You bring sources (web pages, PDFs, notes) and questions. The agent
|
||||
compiles them into wiki pages. Fiona does every deterministic part of the
|
||||
job: it scaffolds the wiki, ingests and archives sources, lints markdown,
|
||||
frontmatter, Python, and JSON, checks links, draws a link graph, versions
|
||||
everything in git, and tells the agent exactly what to fix next.
|
||||
|
||||
The result: the agent spends its tokens on reading and writing, not on
|
||||
listing files, chasing broken links, or reformatting tables.
|
||||
|
||||
```bash
|
||||
python -m pip install fiona-assistant
|
||||
fiona-assistant wiki init my-wiki
|
||||
cd my-wiki
|
||||
fiona-assistant wiki ingest https://example.com/article.html
|
||||
fiona-assistant housekeeping
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
## Table of Contents
|
||||
|
||||
### Purpose
|
||||
- [Why Fiona](#why-fiona)
|
||||
- [Features](#features)
|
||||
- [Quick Start](#quick-start)
|
||||
- [How It Works](#how-it-works)
|
||||
- [Command Reference](#command-reference)
|
||||
- [Output Built for Agents](#output-built-for-agents)
|
||||
- [Roadmap](#roadmap)
|
||||
- [Development](#development)
|
||||
- [Documentation](#documentation)
|
||||
- [License](#license)
|
||||
|
||||
Fiona-Assistant serves as a **standalone MD/scripts/docs organizer** and
|
||||
**LLM wiki management helper** that:
|
||||
---
|
||||
|
||||
- **Reduces AI Token Usage**: Uses deterministic scripts and tools to handle
|
||||
wiki management tasks that don't require LLM intelligence
|
||||
- **Maintains Wiki Structure**: Enforces consistent organization of markdown
|
||||
files, concepts, and raw source materials
|
||||
- **Validates Content Quality**: Provides linting and diagnosis tools for markdown and Python code
|
||||
- **Enables Script-Based Workflows**: Automates repetitive wiki maintenance tasks through CLI commands
|
||||
## Why Fiona
|
||||
|
||||
### Core Philosophy
|
||||
An LLM agent can maintain a knowledge base, but left alone it wastes
|
||||
context on chores and slowly breaks things: dead links, missing
|
||||
frontmatter, wrapped table rows, orphan pages, overwritten sources.
|
||||
|
||||
> Use scripts for deterministic tasks, reserve AI tokens for creative and analytical work
|
||||
Fiona fixes that split of work:
|
||||
|
||||
The toolkit is designed to be **AI-friendly** - outputs are formatted in
|
||||
consistent, parseable structures that LLMs can easily read and process.
|
||||
| The agent does | Fiona does |
|
||||
| --- | --- |
|
||||
| Reads sources and writes synthesized pages | Fetches, archives, and converts sources to markdown |
|
||||
| Decides what links to what | Finds dead links, dead wikilinks, and orphan pages |
|
||||
| Refines raw pages | Lints markdown, frontmatter, Python, and JSON |
|
||||
| Answers your questions from the wiki | Commits, snapshots, and restores wiki history |
|
||||
| Merges template updates | Detects template updates and never overwrites your files |
|
||||
|
||||
> Use scripts for deterministic tasks. Reserve AI tokens for creative and
|
||||
> analytical work.
|
||||
|
||||
---
|
||||
|
||||
## Features
|
||||
|
||||
### Tool Suite
|
||||
### Wiki in one command
|
||||
|
||||
| Tool | Purpose | Key Capabilities |
|
||||
| ---- | ------- | ---------------- |
|
||||
| **Librarian** | Wiki content management | List files, scan directories, organize MD content |
|
||||
| **MDoctor** | Markdown diagnosis | Lint markdown files using PyMarkdown, detect formatting issues |
|
||||
| **Python-Ruff** | Code quality | Lint Python files using Ruff, enforce code standards |
|
||||
| **Generate-TOC** | Table of contents | Auto-generate nested TOC from markdown headers |
|
||||
`fiona-assistant wiki init` creates a ready-to-use llm-wiki: directory
|
||||
layout, settings, agent rules (`AGENTS.md`), page schema, index, log, and
|
||||
bundled agent skills. It also creates a git repository and makes the
|
||||
first commit, with `personal/` kept out of version control.
|
||||
|
||||
### Wiki Structure Support
|
||||
### Source ingestion
|
||||
|
||||
Fiona-Assistant understands and maintains the [LLM Wiki directory structure](./llm-wiki/AGENTS.md):
|
||||
`fiona-assistant wiki ingest <url-or-file>` turns a source into a wiki page
|
||||
in one step:
|
||||
|
||||
```text
|
||||
llm-wiki/
|
||||
├── ingest/ # Input zone: files to be processed
|
||||
├── raw/ # Archived source materials (read-only)
|
||||
│ ├── articles/
|
||||
│ ├── papers/
|
||||
│ ├── transcripts/
|
||||
│ └── assets/
|
||||
├── wiki/ # Processed content
|
||||
│ └── concepts/ # Synthesized concept notes
|
||||
├── personal/ # User notes (read-only for agents)
|
||||
└── utils/ # Utility scripts
|
||||
```
|
||||
- Downloads the URL or reads the local file
|
||||
- Archives an untouched copy under `raw/` (duplicates are detected by
|
||||
SHA-256, so the same bytes are never stored twice)
|
||||
- Converts it to markdown (HTML, PDF, plain text, and markdown ship today)
|
||||
- Writes a page under `wiki/` with full provenance in its frontmatter and
|
||||
`status: raw`, so the agent knows it still needs refining
|
||||
- Appends the action to `wiki/log.md`
|
||||
|
||||
`wiki ingest --pending` lists every page still waiting to be refined.
|
||||
|
||||
### One-call maintenance
|
||||
|
||||
`fiona-assistant housekeeping` runs the whole maintenance routine and
|
||||
prints a to-do list for the agent:
|
||||
|
||||
- **check-update**: is the packaged template newer than this wiki?
|
||||
- **merge**: which `*-update.md` files are waiting to be merged?
|
||||
- **ingest**: which pages are still `status: raw`?
|
||||
- **sanity**: which lint checks fail, and where?
|
||||
- **graph**: regenerates `wiki_graph.html` and lists orphan pages
|
||||
|
||||
It only reads, except for writing the graph. Every finding comes with a
|
||||
concrete "next step" the agent can act on. Add `--json` for machine use.
|
||||
|
||||
### Quality checks
|
||||
|
||||
`fiona-assistant wiki sanity` runs five checks in parallel and gives one
|
||||
pass/fail verdict:
|
||||
|
||||
| Check | What it catches |
|
||||
| --- | --- |
|
||||
| `markdownlint` | Markdown style, via PyMarkdown |
|
||||
| `frontmatter_lint` | Missing or malformed YAML frontmatter (`name`, `version`, and more) |
|
||||
| `python-ruff` | Python issues in wiki scripts, via Ruff |
|
||||
| `mdlinter` | Dead links, dead `[[wikilinks]]`, and table rows split across lines |
|
||||
| `jsonlint` | JSON syntax, indentation, trailing commas, duplicate keys |
|
||||
|
||||
Each linter is also available on its own, and `mdoctor fix` and
|
||||
`json-lint fix` repair what they can automatically.
|
||||
|
||||
### Interactive link graph
|
||||
|
||||
`fiona-assistant wiki graph` writes a single self-contained HTML file
|
||||
that shows how your pages link to each other. Nodes are scored by how
|
||||
many different pages point at them, so hub pages stand out. It
|
||||
understands markdown links and `[[wikilinks]]`, shows external URLs, and
|
||||
flags broken links and orphans. `--format json` exports the same graph
|
||||
for other tools.
|
||||
|
||||
### Safe template updates
|
||||
|
||||
As Fiona improves, so does its wiki template. `wiki check-update` shows
|
||||
what changed, and `wiki update` brings your wiki up to date without ever
|
||||
overwriting a file you own. When both sides changed, Fiona writes
|
||||
`<name>-update.md` next to your file and `housekeeping` tells the agent
|
||||
to merge it.
|
||||
|
||||
### Git history built in
|
||||
|
||||
`fiona-assistant git` wraps the operations an agent needs, with
|
||||
timestamped commit messages:
|
||||
|
||||
- `save` commits all current edits
|
||||
- `snapshot <name>` commits and tags a restore point
|
||||
- `restore <name>` returns to a snapshot without losing history
|
||||
- `undo`, `clean`, and `status` for everyday recovery
|
||||
|
||||
### Guardrails
|
||||
|
||||
- `personal/` and `raw/` are read-only for every automated tool. Ingest
|
||||
may add new files to `raw/`; it never changes an existing one.
|
||||
- Commands that need a decision prompt on a terminal and accept
|
||||
`--yes`, `--force`, or `--non-interactive` for unattended runs.
|
||||
- Respects `.gitignore` when scanning.
|
||||
|
||||
### Table of contents generation
|
||||
|
||||
`fiona-assistant librarian toc` builds `table_of_contents.md` for a folder
|
||||
tree. `librarian toc file <path>` keeps an in-file TOC under the H1 up to
|
||||
date, with optional section numbering (`--sectionized`).
|
||||
|
||||
---
|
||||
|
||||
## Installation
|
||||
## Quick Start
|
||||
|
||||
### Quick Install (Production)
|
||||
### 1. Install
|
||||
|
||||
Requires Python 3.11 or newer.
|
||||
|
||||
```bash
|
||||
python -m pip install fiona-assistant
|
||||
```
|
||||
|
||||
### Development Setup
|
||||
|
||||
- **Clone the repository:**
|
||||
|
||||
```bash
|
||||
git clone <repository-url>
|
||||
cd fiona-assistant
|
||||
```
|
||||
|
||||
- **Create and activate the virtual environment:**
|
||||
|
||||
```bash
|
||||
source bootstrap --env fiona_env -c
|
||||
```
|
||||
|
||||
> Make sure `fiona_env` is activated as pyenv before proceeding
|
||||
|
||||
---
|
||||
|
||||
## Usage
|
||||
|
||||
### Main CLI
|
||||
|
||||
Fiona-Assistant provides a unified CLI with multiple subcommands:
|
||||
### 2. Create a wiki
|
||||
|
||||
```bash
|
||||
# Show all available commands
|
||||
fiona-assistant --help
|
||||
|
||||
# Run via Python module
|
||||
python -m fiona --help
|
||||
fiona-assistant wiki init my-wiki
|
||||
cd my-wiki
|
||||
```
|
||||
|
||||
#### Available Commands
|
||||
### 3. Point your agent at it
|
||||
|
||||
Open the folder in your AI coding agent (Cursor, Claude Code, or any
|
||||
agent that reads `AGENTS.md`). The bundled `AGENTS.md` loads the Fiona
|
||||
skill, which teaches the agent the ingest, refine, and housekeeping
|
||||
workflow and when to call each command.
|
||||
|
||||
### 4. Feed it sources
|
||||
|
||||
```bash
|
||||
# Run tests
|
||||
fiona-assistant test
|
||||
|
||||
# Manage wiki content with Librarian
|
||||
fiona-assistant librarian [directory]
|
||||
|
||||
# Generate an in-file table of contents below the H1
|
||||
fiona-assistant librarian toc file <path_to_file>
|
||||
|
||||
# Force section numbering on for the run
|
||||
fiona-assistant librarian toc file <path_to_file> --sectionized
|
||||
|
||||
# Generate in-file TOCs for every md file in the current directory / wiki root
|
||||
fiona-assistant librarian toc file '*'
|
||||
|
||||
# Diagnose markdown files with MDoctor
|
||||
fiona-assistant mdoctor [chklint] [directory]
|
||||
|
||||
# Lint Python code with Ruff
|
||||
fiona-assistant python-ruff [chklint] [directory]
|
||||
fiona-assistant wiki ingest https://example.com/post.html
|
||||
fiona-assistant wiki ingest ./notes/meeting.pdf
|
||||
```
|
||||
|
||||
### Direct Tool Access
|
||||
Then ask the agent to refine the new pages.
|
||||
|
||||
Each tool can also be called directly:
|
||||
### 5. Keep it healthy
|
||||
|
||||
```bash
|
||||
# Librarian - List all files in a directory
|
||||
librarian [directory]
|
||||
|
||||
# MDoctor - Check markdown lint violations
|
||||
mdoctor chklint [directory]
|
||||
|
||||
# MDoctor - List all files
|
||||
|
||||
# Python-Ruff - Check Python code quality
|
||||
python-ruff chklint [directory]
|
||||
|
||||
# Generate table of contents for markdown files
|
||||
generate-toc [directory] [options]
|
||||
fiona-assistant housekeeping
|
||||
fiona-assistant git save --description "ingest meeting notes"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Workflow Examples
|
||||
## How It Works
|
||||
|
||||
### Wiki Ingestion Workflow
|
||||
|
||||
```bash
|
||||
# 1. Drop raw files into ingest directory
|
||||
# (Manual or automated process)
|
||||
|
||||
# 2. List files ready for processing
|
||||
librarian llm-wiki/ingest
|
||||
|
||||
# 3. Validate markdown quality before processing
|
||||
mdoctor chklint llm-wiki/ingest
|
||||
|
||||
# 4. Move processed files to raw archive
|
||||
# (Agent/orchestration task)
|
||||
```
|
||||
|
||||
### Content Maintenance Workflow
|
||||
|
||||
```bash
|
||||
# Check all markdown files in wiki for lint violations
|
||||
mdoctor chklint llm-wiki/wiki
|
||||
|
||||
# Generate table of contents for concept notes
|
||||
generate-toc llm-wiki/wiki/concepts --recursive-toc
|
||||
|
||||
# Validate Python code in the project
|
||||
python-ruff chklint src/
|
||||
|
||||
# List all markdown files for inventory
|
||||
librarian llm-wiki/
|
||||
```
|
||||
|
||||
### Standalone Document Organization
|
||||
|
||||
```bash
|
||||
# Organize any directory of markdown files
|
||||
librarian /path/to/docs/
|
||||
|
||||
# Validate markdown formatting
|
||||
mdoctor chklint /path/to/docs/
|
||||
|
||||
# Create nested table of contents
|
||||
generate-toc /path/to/docs/ --recursive-toc
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Output Format
|
||||
|
||||
All tools produce **AI-friendly output** with consistent formatting:
|
||||
|
||||
### MDoctor Output Format
|
||||
A wiki created by Fiona has this layout:
|
||||
|
||||
```text
|
||||
PyMarkdownLint Violations
|
||||
TARGET: /path/to/directory
|
||||
my-wiki/
|
||||
├── AGENTS.md # Entry point for the agent; loads the Fiona skill
|
||||
├── fiona-settings.json # Marks the wiki root; tool settings
|
||||
├── ingest/ # Drop zone for files waiting to be parsed
|
||||
├── raw/ # Immutable archive of original sources
|
||||
│ ├── articles/
|
||||
│ ├── papers/
|
||||
│ ├── transcripts/
|
||||
│ ├── assets/
|
||||
│ └── other/
|
||||
├── wiki/ # Pages the agent writes and maintains
|
||||
│ ├── schema.md # Page types, tags, conventions
|
||||
│ ├── index.md # Content catalog
|
||||
│ ├── log.md # Append-only action log
|
||||
│ ├── concepts/
|
||||
│ ├── entities/
|
||||
│ ├── comparisons/
|
||||
│ ├── queries/
|
||||
│ └── sources/
|
||||
├── personal/ # Your own notes (read-only for agents, untracked)
|
||||
└── utils/ # Agent skills, personas, and helper scripts
|
||||
```
|
||||
|
||||
The cycle is:
|
||||
|
||||
- **Ingest**: `wiki ingest` archives a source in `raw/` and writes a raw
|
||||
page in `wiki/`.
|
||||
- **Refine**: the agent rewrites the page, links it to related pages, and
|
||||
sets `status: refined`.
|
||||
- **Maintain**: `housekeeping` reports what needs attention; the agent
|
||||
fixes it.
|
||||
- **Save**: `git save` records the change.
|
||||
|
||||
The full page schema lives in the template's
|
||||
[schema](./src/fiona/data/llm_wiki/wiki/schema.md).
|
||||
|
||||
---
|
||||
|
||||
## Command Reference
|
||||
|
||||
Run `fiona-assistant --help` or `fiona-assistant <command> --help` for all
|
||||
flags. Commands that take an optional directory default to the current
|
||||
directory.
|
||||
|
||||
### Wiki lifecycle
|
||||
|
||||
| Command | What it does |
|
||||
| --- | --- |
|
||||
| `wiki init [dir]` | Create a new llm-wiki with git history |
|
||||
| `wiki ingest <source>` | Fetch, archive, and convert a URL or file into a raw page |
|
||||
| `wiki ingest --pending` | List pages still marked `status: raw` |
|
||||
| `wiki sanity [dir]` | Run every quality check; exit 1 on failure |
|
||||
| `wiki graph [dir]` | Write `wiki_graph.html` (or JSON with `--format json`) |
|
||||
| `wiki check-update [dir]` | Show what a template update would change |
|
||||
| `wiki update [dir]` | Apply a template update (`--dry-run`, `--yes`) |
|
||||
| `housekeeping [dir]` | Run the full maintenance routine and list next steps |
|
||||
|
||||
### Linters
|
||||
|
||||
| Command | What it does |
|
||||
| --- | --- |
|
||||
| `mdoctor chklint [dir]` | Markdown lint via PyMarkdown |
|
||||
| `mdoctor fix [dir]` | Auto-fix markdown lint violations |
|
||||
| `fmlint chklint [dir]` | Frontmatter lint |
|
||||
| `python-ruff chklint [dir]` | Python lint via Ruff |
|
||||
| `json-lint chklint [dir]` | JSON lint |
|
||||
| `json-lint fix [dir]` | Auto-fix JSON indentation and trailing commas |
|
||||
|
||||
### Files and TOC
|
||||
|
||||
| Command | What it does |
|
||||
| --- | --- |
|
||||
| `librarian list [dir]` | List files in a directory |
|
||||
| `librarian toc [dir]` | Write `table_of_contents.md` (`--recursive-toc`, `--exclude REGEX`) |
|
||||
| `librarian toc file <path>` | Write an in-file TOC below the H1 (`--sectionized`) |
|
||||
| `librarian toc file '*'` | Update in-file TOCs for every markdown file in the wiki |
|
||||
|
||||
### Git
|
||||
|
||||
| Command | What it does |
|
||||
| --- | --- |
|
||||
| `git save --description TEXT` | Commit all edits |
|
||||
| `git snapshot <name> --description TEXT` | Commit and tag a restore point |
|
||||
| `git restore <name>` | Restore a snapshot as a new commit |
|
||||
| `git undo` | Drop the last commit |
|
||||
| `git clean` | Discard uncommitted edits |
|
||||
| `git status` | Show whether the tree is clean |
|
||||
|
||||
Every command above is run as `fiona-assistant <command>`. The
|
||||
`librarian` and `mdoctor` tools are also installed as standalone
|
||||
commands.
|
||||
|
||||
---
|
||||
|
||||
## Output Built for Agents
|
||||
|
||||
Every report uses the same shape: a header, the target, rule
|
||||
definitions, findings per file with line and column, and a summary. An
|
||||
agent can parse it without guessing, and a human can skim it.
|
||||
|
||||
```text
|
||||
Markdown Lint Violations
|
||||
TARGET: wiki
|
||||
|
||||
- DEFINITIONS:
|
||||
- md001: Heading levels should only increment by one, Heading levels should only increment by one level at a time
|
||||
- md013: Line length, Line length
|
||||
- MDL001: Dead Link, Relative markdown link does not resolve to an existing file
|
||||
- MDL002: Dead Wiki Link, Wiki-internal link does not resolve to any md page in the wiki
|
||||
|
||||
- FILE: file1.md
|
||||
- 10:5 md013
|
||||
- 20:3 md001
|
||||
- FILE: wiki/concepts/retrieval.md
|
||||
- MDL001 line 12: ../sources/old-paper.md does not exist
|
||||
- MDL002 line 30: [[embeddings]] does not resolve to any page
|
||||
|
||||
- SUMMARY:
|
||||
- Total files with violations: 1
|
||||
- Total violations: 2
|
||||
```
|
||||
|
||||
### Python-Ruff Output Format
|
||||
|
||||
```text
|
||||
Ruff Violations
|
||||
TARGET: /path/to/directory
|
||||
|
||||
Definitions:
|
||||
- F401: `*` import used, but not imported
|
||||
- E501: Line too long
|
||||
|
||||
- FILE: script.py
|
||||
- 15:10 F401
|
||||
- 25:80 E501
|
||||
|
||||
- SUMMARY:
|
||||
- Total files with violations: 1
|
||||
- Total violations: 2
|
||||
```
|
||||
`wiki sanity`, `housekeeping`, `wiki ingest`, `wiki check-update`, and
|
||||
`wiki update` also accept `--json` for structured output.
|
||||
|
||||
---
|
||||
|
||||
## Project Structure
|
||||
## Roadmap
|
||||
|
||||
Planned work, tracked with requirements and acceptance criteria in
|
||||
[SPEC](./SPEC.md) and prioritized in [todo](./todo.md):
|
||||
|
||||
| Feature | What it will do |
|
||||
| --- | --- |
|
||||
| Wiki root discovery | Run any command from a subfolder; Fiona finds the nearest `fiona-settings.json` |
|
||||
| Backup | One command writes a compressed, restorable archive of the wiki outside the tree |
|
||||
| `setup` command | Interactive review and editing of `fiona-settings.json`, with validation |
|
||||
| Smarter graph | Show only important nodes until you expand a parent; hide gitignored pages |
|
||||
| Code blocks in markdown | Lint Python and shell snippets inside fenced code blocks |
|
||||
| Shellcheck | Lint shell scripts as part of wiki sanity |
|
||||
| HTML and JavaScript lint | Validate HTML and JS files in the wiki |
|
||||
| `python-ruff fix` | Auto-fix Python lint findings |
|
||||
| More converters | Ingest more source formats beyond HTML, PDF, text, and markdown |
|
||||
| Version tags for scripts | Track versions of Python and shell files like markdown pages |
|
||||
|
||||
---
|
||||
|
||||
## Development
|
||||
|
||||
### Set up
|
||||
|
||||
Clone the repository and load the environment with the bootstrap script.
|
||||
|
||||
Linux and macOS:
|
||||
|
||||
```bash
|
||||
git clone <repository-url>
|
||||
cd fiona-assistant
|
||||
source bootstrap --env fiona_env -c
|
||||
```
|
||||
|
||||
Windows (PowerShell):
|
||||
|
||||
```powershell
|
||||
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
|
||||
. .\bootstrap.ps1 --env fiona_env -c
|
||||
```
|
||||
|
||||
The default install is editable (`pip install -e ".[dev]"`). On Windows
|
||||
the environment lives in `.venvs\fiona_env`.
|
||||
|
||||
### Other install modes
|
||||
|
||||
| Goal | Bash | PowerShell |
|
||||
| --- | --- | --- |
|
||||
| Install this checkout as a normal package | `source bootstrap --local` | `. .\bootstrap.ps1 --local` |
|
||||
| Same as `pip install fiona-assistant`, no venv | `source bootstrap --local --no-env` | `powershell -ExecutionPolicy Bypass -File .\bootstrap.ps1 --local --no-env` |
|
||||
|
||||
`--local --no-env` runs `python -m pip install --user .`. If
|
||||
`fiona-assistant` is not found afterwards, add the user scripts
|
||||
directory printed by bootstrap to `PATH`.
|
||||
|
||||
### Test and check
|
||||
|
||||
```bash
|
||||
# Unit tests with coverage
|
||||
python scripts/run_tests.py
|
||||
|
||||
# Full sanity: tests, coverage gate, markdown, Ruff, frontmatter, JSON
|
||||
python scripts/run_sanity.py --non-interactive
|
||||
```
|
||||
|
||||
Contributions follow the spec-first, test-first lifecycle in
|
||||
[AGENTS](./AGENTS.md).
|
||||
|
||||
### Project structure
|
||||
|
||||
```text
|
||||
fiona-assistant/
|
||||
├── llm-wiki/ # Wiki content and structure definition
|
||||
│ ├── AGENTS.md # Agent rules and directory permissions
|
||||
│ ├── ingest/ # Input zone for new content
|
||||
│ ├── raw/ # Archived source materials
|
||||
│ ├── wiki/ # Processed concept notes
|
||||
│ └── utils/ # Utility scripts
|
||||
│
|
||||
├── src/
|
||||
│ ├── fiona/ # Main CLI entry point
|
||||
│ │ └── __main__.py # Unified command dispatcher
|
||||
│ │
|
||||
│ ├── librarian/ # Wiki content management
|
||||
│ │ ├── __init__.py
|
||||
│ │ ├── librarian.py # File scanning and listing
|
||||
│ │ └── toc_generator.py # Table of contents generator
|
||||
│ │
|
||||
│ ├── mdoctor/ # Markdown diagnosis
|
||||
│ │ ├── __init__.py
|
||||
│ │ └── mdoctor.py # PyMarkdown lint wrapper
|
||||
│ │
|
||||
│ ├── python_ruff/ # Python code linting
|
||||
│ │ ├── __init__.py
|
||||
│ │ └── python_ruff.py # Ruff lint wrapper
|
||||
│ │
|
||||
│ ├── tests/ # Unit tests
|
||||
│ │ ├── test_fiona_main.py
|
||||
│ │ ├── test_librarian.py
|
||||
│ │ ├── test_mdoctor.py
|
||||
│ │ └── test_python_ruff.py
|
||||
│ │
|
||||
│ ├── run_tests.py # Test runner
|
||||
│ └── README.md # Development documentation
|
||||
│
|
||||
├── pyproject.toml # Project configuration
|
||||
├── bootstrap # Environment setup script
|
||||
├── LICENSE # GPL-3.0-only
|
||||
└── README.md # This file
|
||||
├── src/fiona/ # The installable package
|
||||
│ ├── __main__.py # fiona-assistant CLI
|
||||
│ ├── librarian/ # File listing and TOC generation
|
||||
│ ├── mdoctor/ # Markdown lint and fix
|
||||
│ ├── customlinter/ # Frontmatter, markdown link, Python, JSON linters
|
||||
│ ├── python_ruff/ # Ruff wrapper
|
||||
│ ├── gittool/ # Git operations
|
||||
│ ├── sanity_runner/ # Shared sanity engine
|
||||
│ ├── wiki/ # init, update, sanity, graph, ingest, housekeeping
|
||||
│ └── data/llm_wiki/ # Packaged wiki template
|
||||
├── tests/ # Unit tests (not installed)
|
||||
├── scripts/ # Test and sanity runners, dev settings
|
||||
├── bootstrap # Environment setup (bash)
|
||||
├── bootstrap.ps1 # Environment setup (PowerShell)
|
||||
├── SPEC.md # Requirements and acceptance criteria
|
||||
└── pyproject.toml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
## Documentation
|
||||
|
||||
### Run Tests
|
||||
|
||||
```bash
|
||||
# Run all tests
|
||||
python src/run_tests.py
|
||||
|
||||
# Or use pytest directly
|
||||
python -m pytest
|
||||
|
||||
# Run with coverage
|
||||
python -m pytest --cov=src
|
||||
```
|
||||
- [Project Specification](./SPEC.md): every command, setting, and rule,
|
||||
with acceptance criteria
|
||||
- [LLM Wiki Agent Rules](./src/fiona/data/llm_wiki/AGENTS.md): what a
|
||||
deployed wiki tells its agent
|
||||
- [Fiona skill](./src/fiona/data/llm_wiki/utils/skills/fiona/SKILL.md):
|
||||
the workflow the agent follows inside a wiki
|
||||
- [Development Agent Rules](./AGENTS.md): how to contribute to this
|
||||
repository
|
||||
- [Roadmap](./todo.md): planned and in-progress work
|
||||
|
||||
---
|
||||
|
||||
## License
|
||||
|
||||
GPL-3.0-only
|
||||
|
||||
---
|
||||
|
||||
## Key Concepts
|
||||
|
||||
### For LLM Agents
|
||||
|
||||
- **Deterministic Operations**: Use Fiona-Assistant for file listing, linting, and organization
|
||||
- **Token Efficiency**: Reserve LLM context for synthesis, analysis, and creative tasks
|
||||
- **Structured Output**: All tool outputs are formatted for easy parsing by LLMs
|
||||
|
||||
### For Human Users
|
||||
|
||||
- **CLI-First**: All functionality available through command-line interface
|
||||
- **Composable Tools**: Combine tools in scripts for automated workflows
|
||||
- **Clear Structure**: Maintains consistent wiki organization standards
|
||||
|
||||
---
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [LLM Wiki Agent Rules](./llm-wiki/AGENTS.md) - Directory permissions and workflow rules
|
||||
- [Development Setup](./src/README.md) - Detailed development information
|
||||
- [Project Specification](./SPEC.md) - Feature spec sheet with requirements and acceptance criteria
|
||||
- [Project TODO](./todo.md) - Roadmap and planned features
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference Card
|
||||
|
||||
| Task | Command |
|
||||
| ---- | ------- |
|
||||
| Show help | `fiona-assistant --help` |
|
||||
| List files | `fiona-assistant librarian /path/to/dir` |
|
||||
| Check MD lint | `fiona-assistant mdoctor chklint /path/to/dir` |
|
||||
| Check Python lint | `fiona-assistant python-ruff chklint /path/to/dir` |
|
||||
| Generate TOC | `generate-toc /path/to/dir --recursive-toc` |
|
||||
| Generate file TOC | `fiona-assistant librarian toc file /path/to/file.md --sectionized` |
|
||||
| Run tests | `python src/run_tests.py` |
|
||||
| Install | `pip install fiona-assistant` |
|
||||
| Dev install | `pip install -e .` |
|
||||
|
||||
---
|
||||
|
||||
> **Note**: All paths are relative to the current working directory unless specified as absolute paths.
|
||||
|
||||
@@ -1,23 +1,53 @@
|
||||
#!/bin/bash
|
||||
# Modified by AI/LLM: 2026-09-30 - --local and --no-env install options.
|
||||
#
|
||||
# Project bootstrap script - Sets up Python development environment
|
||||
#
|
||||
# This script MUST be sourced to properly set up the environment:
|
||||
# source bootstrap
|
||||
#
|
||||
# Windows (PowerShell): dot-source bootstrap.ps1 instead
|
||||
# . .\bootstrap.ps1 [--env name] [--python 3.x.x] [-c|--clean] [--local] [--no-env]
|
||||
#
|
||||
# What it does:
|
||||
# 1. Installs pyenv and required system dependencies if needed
|
||||
# 2. Initializes pyenv and pyenv-virtualenv in the current shell
|
||||
# 3. Creates a Python virtual environment named 'test_env' (or custom name)
|
||||
# 4. Activates the virtual environment
|
||||
# 5. Installs all project requirements from requirements.txt files
|
||||
# 5. Installs this repository (editable with the dev extra, unless --local)
|
||||
#
|
||||
# With --clean/-c option:
|
||||
# - Deletes existing environment before creating a new one
|
||||
#
|
||||
# With --local:
|
||||
# - Installs this repository as a normal package (pip install .), not editable
|
||||
#
|
||||
# With --no-env:
|
||||
# - Skips pyenv and the virtual environment
|
||||
# - Installs with pip --user so the commands run without activating an environment
|
||||
#
|
||||
# Usage:
|
||||
# source bootstrap [--pyenv] [-c|--clean] [--env custom_name] [--python 3.x.x]
|
||||
# source bootstrap [--pyenv] [-c|--clean] [--env custom_name] [--python 3.x.x] [--local] [--no-env]
|
||||
# ex. source bootstrap --pyenv -c --env my_env --python 3.10.6
|
||||
# ex. source bootstrap --local --no-env
|
||||
|
||||
# =============================================
|
||||
# SOURCED GUARD
|
||||
# =============================================
|
||||
# Activation only persists when this file is sourced; executing it
|
||||
# loses the environment when the child shell exits. `return` works
|
||||
# at the top level only inside a sourced script.
|
||||
if [ -z "${BASH_VERSION:-}" ]; then
|
||||
echo "ERROR: bootstrap is a bash script. Run it with bash:" >&2
|
||||
echo "Run: source bootstrap [--pyenv] [-c|--clean] [--env name] [--python 3.x.x] [--local] [--no-env]" >&2
|
||||
exit 1
|
||||
fi
|
||||
if ! (return 0 2>/dev/null); then
|
||||
echo "ERROR: bootstrap must be sourced, not executed." >&2
|
||||
echo "The environment would be lost when the child shell exits." >&2
|
||||
echo "Run: source bootstrap [--pyenv] [-c|--clean] [--env name] [--python 3.x.x] [--local] [--no-env]" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# =============================================
|
||||
# VARIABLES
|
||||
@@ -26,10 +56,13 @@ PYTHON_VERSION="3.13.8" # Default Python version
|
||||
export PYENV_NAME="test_env" # Default environment name
|
||||
FORCE_PYENV_INSTALL=false # Flag to force pyenv reinstallation
|
||||
CLEAN_ENVIRONMENT=false # Flag to delete and recreate environment
|
||||
LOCAL_INSTALL=false # Install this repo as a normal package, not editable
|
||||
NO_ENV=false # Skip the virtual environment and pip install --user
|
||||
PYTHON_BIN="python"
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" # scripts root directory
|
||||
|
||||
# Source utilities (needed for echo_error, echo_status, etc.)
|
||||
source "$SCRIPT_DIR/src/utils.sh"
|
||||
source "$SCRIPT_DIR/scripts/utils.sh"
|
||||
|
||||
# Parse command line arguments
|
||||
while [[ $# -gt 0 ]]; do
|
||||
@@ -42,6 +75,14 @@ while [[ $# -gt 0 ]]; do
|
||||
CLEAN_ENVIRONMENT=true
|
||||
shift
|
||||
;;
|
||||
--local)
|
||||
LOCAL_INSTALL=true
|
||||
shift
|
||||
;;
|
||||
--no-env)
|
||||
NO_ENV=true
|
||||
shift
|
||||
;;
|
||||
--env)
|
||||
if [ -n "$2" ]; then
|
||||
PYENV_NAME="$2"
|
||||
@@ -62,7 +103,7 @@ while [[ $# -gt 0 ]]; do
|
||||
;;
|
||||
*)
|
||||
echo_error "Unknown option: $1"
|
||||
echo "Usage: source bootstrap [--pyenv] [-c|--clean] [--env custom_name] [--python 3.x.x]"
|
||||
echo "Usage: source bootstrap [--pyenv] [-c|--clean] [--env custom_name] [--python 3.x.x] [--local] [--no-env]"
|
||||
return 1
|
||||
;;
|
||||
esac
|
||||
@@ -283,7 +324,7 @@ activate_environment() {
|
||||
# Get the virtualenv path
|
||||
local venv_path
|
||||
venv_path=$(pyenv prefix "$PYENV_NAME" 2>/dev/null)
|
||||
|
||||
|
||||
if [ -n "$venv_path" ] && [ -d "$venv_path" ]; then
|
||||
# Manually activate by sourcing the activate script
|
||||
local activate_script="$venv_path/bin/activate"
|
||||
@@ -296,7 +337,7 @@ activate_environment() {
|
||||
return 0
|
||||
fi
|
||||
fi
|
||||
|
||||
|
||||
# Fallback: try pyenv activate
|
||||
if pyenv activate "$PYENV_NAME" 2>/dev/null; then
|
||||
echo_status "Successfully activated $PYENV_NAME with pyenv"
|
||||
@@ -331,52 +372,82 @@ install_requirements() {
|
||||
echo_status "Installing requirements..."
|
||||
|
||||
# Check if pyproject.toml exists (could be in root or src/)
|
||||
local pyproject_root="$SCRIPT_DIR/pyproject.toml"
|
||||
local pyproject_src="$SCRIPT_DIR/src/pyproject.toml"
|
||||
|
||||
if [ -f "$pyproject_root" ]; then
|
||||
# Install the package in development mode from root
|
||||
echo_status "Installing package and dependencies from pyproject.toml..."
|
||||
(cd "$SCRIPT_DIR" && pip install -e ".[dev]" --break-system-packages) || {
|
||||
echo_error "Failed to install package and dependencies from pyproject.toml"
|
||||
return 1
|
||||
}
|
||||
elif [ -f "$pyproject_src" ]; then
|
||||
# Install the package in development mode from src/
|
||||
echo_status "Installing package and dependencies from pyproject.toml..."
|
||||
(cd "$SCRIPT_DIR/src" && pip install -e ".[dev]" --break-system-packages) || {
|
||||
echo_error "Failed to install package and dependencies from pyproject.toml"
|
||||
return 1
|
||||
}
|
||||
else
|
||||
echo_warning "No pyproject.toml found"
|
||||
|
||||
# Fallback: look for any requirements.txt files
|
||||
local req_files=()
|
||||
while IFS= read -r -d '' req_file; do
|
||||
req_files+=("$req_file")
|
||||
done < <(find "$SCRIPT_DIR" -name "requirements.txt" -print0)
|
||||
|
||||
if [ ${#req_files[@]} -eq 0 ]; then
|
||||
echo_status "No requirements files found"
|
||||
return 0
|
||||
fi
|
||||
|
||||
for req_file in "${req_files[@]}"; do
|
||||
echo_status "Installing requirements from: $req_file"
|
||||
pip install -r "$req_file" --break-system-packages || {
|
||||
echo_error "Failed to install requirements from $req_file"
|
||||
return 1
|
||||
}
|
||||
done
|
||||
local project_dir=""
|
||||
if [ -f "$SCRIPT_DIR/pyproject.toml" ]; then
|
||||
project_dir="$SCRIPT_DIR"
|
||||
elif [ -f "$SCRIPT_DIR/src/pyproject.toml" ]; then
|
||||
project_dir="$SCRIPT_DIR/src"
|
||||
fi
|
||||
|
||||
|
||||
if [ -n "$project_dir" ]; then
|
||||
local helper="$SCRIPT_DIR/scripts/bootstrap_install.py"
|
||||
if [ ! -f "$helper" ]; then
|
||||
echo_error "Missing $helper"
|
||||
return 1
|
||||
fi
|
||||
local flags=(--break-system-packages)
|
||||
if [ "$LOCAL_INSTALL" = true ]; then
|
||||
echo_status "Installing this repository as a normal package..."
|
||||
flags+=(--local)
|
||||
else
|
||||
echo_status "Installing package and dependencies from pyproject.toml..."
|
||||
fi
|
||||
if [ "$NO_ENV" = true ]; then
|
||||
flags+=(--no-env)
|
||||
fi
|
||||
local pip_words=()
|
||||
local word
|
||||
while IFS= read -r word; do
|
||||
if [ -n "$word" ]; then
|
||||
pip_words+=("$word")
|
||||
fi
|
||||
done < <("$PYTHON_BIN" "$helper" "${flags[@]}")
|
||||
if [ "${#pip_words[@]}" -eq 0 ]; then
|
||||
echo_error "Failed to resolve pip install arguments"
|
||||
return 1
|
||||
fi
|
||||
(cd "$project_dir" && "$PYTHON_BIN" -m pip "${pip_words[@]}") || {
|
||||
echo_error "Failed to install package and dependencies from pyproject.toml"
|
||||
return 1
|
||||
}
|
||||
return 0
|
||||
fi
|
||||
|
||||
echo_warning "No pyproject.toml found"
|
||||
|
||||
# Fallback: look for any requirements.txt files
|
||||
local req_files=()
|
||||
while IFS= read -r -d '' req_file; do
|
||||
req_files+=("$req_file")
|
||||
done < <(find "$SCRIPT_DIR" -name "requirements.txt" -print0)
|
||||
|
||||
if [ ${#req_files[@]} -eq 0 ]; then
|
||||
echo_status "No requirements files found"
|
||||
return 0
|
||||
fi
|
||||
|
||||
local user_args=()
|
||||
if [ "$NO_ENV" = true ]; then
|
||||
user_args+=(--user)
|
||||
fi
|
||||
for req_file in "${req_files[@]}"; do
|
||||
echo_status "Installing requirements from: $req_file"
|
||||
"$PYTHON_BIN" -m pip install "${user_args[@]}" -r "$req_file" --break-system-packages || {
|
||||
echo_error "Failed to install requirements from $req_file"
|
||||
return 1
|
||||
}
|
||||
done
|
||||
|
||||
return 0
|
||||
}
|
||||
|
||||
upgrade_pip() {
|
||||
echo_status "Upgrading pip..."
|
||||
python -m pip install --upgrade pip --break-system-packages || {
|
||||
local upgrade_args=(install --upgrade pip --break-system-packages)
|
||||
if [ "$NO_ENV" = true ]; then
|
||||
upgrade_args=(install --user --upgrade pip --break-system-packages)
|
||||
fi
|
||||
"$PYTHON_BIN" -m pip "${upgrade_args[@]}" || {
|
||||
echo_warning "Failed to upgrade pip"
|
||||
echo_warning "Continuing with requirements installation..."
|
||||
}
|
||||
@@ -384,18 +455,18 @@ upgrade_pip() {
|
||||
|
||||
install_dev_packages() {
|
||||
echo_status "Installing local Python packages in development mode..."
|
||||
local install_script="$SCRIPT_DIR/src/install_packages.sh"
|
||||
|
||||
local install_script="$SCRIPT_DIR/scripts/install_packages.sh"
|
||||
|
||||
if [ -f "$install_script" ]; then
|
||||
# Use source to run in current shell context, preserving environment variables
|
||||
(cd "$SCRIPT_DIR/src" && . "$install_script") || {
|
||||
(cd "$SCRIPT_DIR" && . "$install_script") || {
|
||||
echo_error "Dev packages not installed!"
|
||||
return 1
|
||||
}
|
||||
else
|
||||
echo_status "Install script not found at $install_script, attempting direct install..."
|
||||
# Fallback: install in editable mode from src directory
|
||||
(cd "$SCRIPT_DIR/src" && pip install -e . --break-system-packages) || {
|
||||
# Fallback: install in editable mode from the repository root
|
||||
(cd "$SCRIPT_DIR" && pip install -e . --break-system-packages) || {
|
||||
echo_error "Failed to install package in development mode"
|
||||
return 1
|
||||
}
|
||||
@@ -411,16 +482,16 @@ sanitize_pytest_env() {
|
||||
# Also remove -S or --no-summary (older versions)
|
||||
local sanitized
|
||||
sanitized=$(echo "$PYTEST_ADDOPTS" | sed -e 's/--no-summary[[:space:]]*//g' -e 's/--no-summary//g' -e 's/-S[[:space:]]*//g' -e 's/-S//g')
|
||||
|
||||
|
||||
# Remove leading/trailing whitespace and any double spaces
|
||||
sanitized=$(echo "$sanitized" | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//' -e 's/[[:space:]]\+/ /g')
|
||||
|
||||
|
||||
if [ "$sanitized" != "$PYTEST_ADDOPTS" ]; then
|
||||
echo_warning "Removing --no-summary from PYTEST_ADDOPTS to ensure test output is parsed correctly"
|
||||
export PYTEST_ADDOPTS="$sanitized"
|
||||
fi
|
||||
fi
|
||||
|
||||
|
||||
# Also unset FORCE_COLOR and PY_COLORS as they can interfere with output parsing
|
||||
if [ -n "$FORCE_COLOR" ]; then
|
||||
echo_warning "Unsetting FORCE_COLOR to ensure pytest output is parsed correctly"
|
||||
@@ -430,42 +501,67 @@ sanitize_pytest_env() {
|
||||
echo_warning "Unsetting PY_COLORS to ensure pytest output is parsed correctly"
|
||||
unset PY_COLORS
|
||||
fi
|
||||
|
||||
|
||||
return 0
|
||||
}
|
||||
|
||||
# =============================================
|
||||
# MAIN EXECUTION
|
||||
# =============================================
|
||||
echo_status "Starting project bootstrap with environment: $PYENV_NAME"
|
||||
|
||||
if [ "$FORCE_PYENV_INSTALL" = true ]; then
|
||||
echo_status "Forcing pyenv reinstallation"
|
||||
if [ "$NO_ENV" = true ]; then
|
||||
echo_status "Starting project bootstrap (no virtual environment)"
|
||||
else
|
||||
echo_status "Starting project bootstrap with environment: $PYENV_NAME"
|
||||
fi
|
||||
|
||||
# 1. Check if Python is installed
|
||||
if ! check_python_installed; then
|
||||
return 1
|
||||
if [ "$LOCAL_INSTALL" = true ]; then
|
||||
echo_status "Local install: this repository, not editable"
|
||||
fi
|
||||
if [ "$NO_ENV" = true ]; then
|
||||
echo_status "No virtual environment: installing with --user"
|
||||
fi
|
||||
|
||||
# 2. Check/install pyenv
|
||||
if ! check_and_install_pyenv; then
|
||||
return 1
|
||||
fi
|
||||
if [ "$NO_ENV" = true ]; then
|
||||
if ! command -v python >/dev/null 2>&1 && ! command -v python3 >/dev/null 2>&1; then
|
||||
echo_error "Python is not installed. Please install Python $PYTHON_VERSION before running this script."
|
||||
echo_error "Or download from: https://www.python.org/downloads/"
|
||||
return 1
|
||||
fi
|
||||
if command -v python >/dev/null 2>&1; then
|
||||
PYTHON_BIN="python"
|
||||
else
|
||||
PYTHON_BIN="python3"
|
||||
fi
|
||||
else
|
||||
if [ "$FORCE_PYENV_INSTALL" = true ]; then
|
||||
echo_status "Forcing pyenv reinstallation"
|
||||
fi
|
||||
|
||||
# 3. Initialize pyenv
|
||||
if ! initialize_pyenv; then
|
||||
return 1
|
||||
fi
|
||||
# 1. Check if Python is installed
|
||||
if ! check_python_installed; then
|
||||
return 1
|
||||
fi
|
||||
|
||||
# 4. Check/create virtual environment
|
||||
if ! check_environment_exists; then
|
||||
return 1
|
||||
fi
|
||||
# 2. Check/install pyenv
|
||||
if ! check_and_install_pyenv; then
|
||||
return 1
|
||||
fi
|
||||
|
||||
# 5. Activate the environment
|
||||
if ! activate_environment; then
|
||||
return 1
|
||||
# 3. Initialize pyenv
|
||||
if ! initialize_pyenv; then
|
||||
return 1
|
||||
fi
|
||||
|
||||
# 4. Check/create virtual environment
|
||||
if ! check_environment_exists; then
|
||||
return 1
|
||||
fi
|
||||
|
||||
# 5. Activate the environment
|
||||
if ! activate_environment; then
|
||||
return 1
|
||||
fi
|
||||
PYTHON_BIN="python"
|
||||
fi
|
||||
|
||||
# 5.5. Sanitize pytest environment to ensure proper output parsing
|
||||
@@ -474,14 +570,25 @@ sanitize_pytest_env
|
||||
# 6. Upgrade pip
|
||||
upgrade_pip
|
||||
|
||||
# 7. Install requirements (including package in dev mode with all dependencies)
|
||||
install_requirements
|
||||
# 7. Install this repository
|
||||
if ! install_requirements; then
|
||||
return 1
|
||||
fi
|
||||
|
||||
# Final verification
|
||||
echo_status "Bootstrap completed successfully!"
|
||||
echo_status "Virtual environment '$PYENV_NAME' is active"
|
||||
echo_status "Python: $(python --version) ($(which python))"
|
||||
echo_status "Pip: $(pip --version)"
|
||||
if [ "$NO_ENV" = true ]; then
|
||||
scripts_dir=$("$PYTHON_BIN" -c "import sys, sysconfig; scheme = 'nt_user' if sys.platform == 'win32' else 'posix_user'; print(sysconfig.get_path('scripts', scheme))")
|
||||
echo_status "Installed for the current user (no virtual environment)"
|
||||
echo_status "Command directory: $scripts_dir"
|
||||
echo_status "If fiona-assistant is not found, add that directory to PATH."
|
||||
echo_status "Python: $("$PYTHON_BIN" --version) ($PYTHON_BIN)"
|
||||
echo_status "Pip: $("$PYTHON_BIN" -m pip --version)"
|
||||
else
|
||||
echo_status "Virtual environment '$PYENV_NAME' is active"
|
||||
echo_status "Python: $(python --version) ($(which python))"
|
||||
echo_status "Pip: $(pip --version)"
|
||||
fi
|
||||
echo_status ""
|
||||
echo_status "To verify installation, try:"
|
||||
echo_status " fiona-assistant test"
|
||||
|
||||
+358
@@ -0,0 +1,358 @@
|
||||
# Generated by AI/LLM: 2026-09-29 - Windows PowerShell bootstrap for Fiona development
|
||||
# Modified by AI/LLM: 2026-09-30 - --local and --no-env install options.
|
||||
#
|
||||
# Project bootstrap for Windows. Dot-source a virtual-environment install so
|
||||
# the environment stays active in the current session (same idea as
|
||||
# `source bootstrap`):
|
||||
#
|
||||
# Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
|
||||
# . .\bootstrap.ps1
|
||||
# . .\bootstrap.ps1 --env fiona_env
|
||||
# . .\bootstrap.ps1 --env fiona_env --python 3.13.8 -c
|
||||
# . .\bootstrap.ps1 --local
|
||||
# powershell -ExecutionPolicy Bypass -File .\bootstrap.ps1 --local --no-env
|
||||
#
|
||||
# Creates a repo-local venv at .venvs\<name> with the Python launcher (py)
|
||||
# or python on PATH. Does not install pyenv. --pyenv is accepted so the
|
||||
# Linux flag still parses; on Windows it is a no-op notice.
|
||||
#
|
||||
# --local installs this repository as a normal package (not editable).
|
||||
# --no-env skips the virtual environment and installs with pip --user,
|
||||
# so fiona-assistant can run without activating an environment.
|
||||
# --no-env may be executed with -File. A venv install must be dot-sourced.
|
||||
|
||||
#Requires -Version 5.1
|
||||
|
||||
# AI/LLM: flags are parsed by hand. Windows PowerShell 5.1 does not bind
|
||||
# `--python` to a parameter alias, and the bash script uses those flags.
|
||||
|
||||
function Write-FionaStatus {
|
||||
param([string]$Message)
|
||||
Write-Host "[+] $Message" -ForegroundColor Green
|
||||
}
|
||||
|
||||
function Write-FionaWarning {
|
||||
param([string]$Message)
|
||||
Write-Host "[!] $Message" -ForegroundColor Yellow
|
||||
}
|
||||
|
||||
function Write-FionaError {
|
||||
param([string]$Message)
|
||||
Write-Host "[x] $Message" -ForegroundColor Red
|
||||
}
|
||||
|
||||
function Remove-FionaBootstrapFunctions {
|
||||
if (Get-Variable -Name FionaPrevErrorAction -Scope 1 -ErrorAction SilentlyContinue) {
|
||||
Set-Variable -Name ErrorActionPreference -Value $FionaPrevErrorAction -Scope 1
|
||||
Remove-Variable -Name FionaPrevErrorAction -Scope 1 -ErrorAction SilentlyContinue
|
||||
}
|
||||
foreach ($name in @(
|
||||
"Write-FionaStatus",
|
||||
"Write-FionaWarning",
|
||||
"Write-FionaError",
|
||||
"Resolve-FionaPython",
|
||||
"Remove-FionaBootstrapFunctions"
|
||||
)) {
|
||||
Remove-Item -Path "Function:\$name" -ErrorAction SilentlyContinue
|
||||
}
|
||||
}
|
||||
|
||||
function Resolve-FionaPython {
|
||||
# AI/LLM: pick a real interpreter. Prefer the py launcher so a requested
|
||||
# version (3.13.8, then 3.13) wins over whatever `python` happens to be.
|
||||
param([string]$Version)
|
||||
|
||||
$tags = @()
|
||||
if ($Version) {
|
||||
$tags += $Version
|
||||
if ($Version -match '^(\d+\.\d+)') {
|
||||
$minor = $Matches[1]
|
||||
if ($minor -ne $Version) {
|
||||
$tags += $minor
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
$py = Get-Command py -ErrorAction SilentlyContinue
|
||||
if ($py) {
|
||||
foreach ($tag in $tags) {
|
||||
$exe = & py "-$tag" -c "import sys; print(sys.executable)" 2>$null
|
||||
if ($LASTEXITCODE -eq 0 -and $exe) {
|
||||
return @($exe)[-1].ToString().Trim()
|
||||
}
|
||||
}
|
||||
if (-not $Version) {
|
||||
$exe = & py -3 -c "import sys; print(sys.executable)" 2>$null
|
||||
if ($LASTEXITCODE -eq 0 -and $exe) {
|
||||
return @($exe)[-1].ToString().Trim()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (-not $Version) {
|
||||
foreach ($name in @("python", "python3")) {
|
||||
$cmd = Get-Command $name -ErrorAction SilentlyContinue
|
||||
if ($cmd -and $cmd.Source -notmatch "WindowsApps") {
|
||||
return $cmd.Source
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return $null
|
||||
}
|
||||
|
||||
$FionaPrevErrorAction = $ErrorActionPreference
|
||||
$ErrorActionPreference = "Stop"
|
||||
# AI/LLM: a failed pip/py call must be checked via $LASTEXITCODE, not thrown.
|
||||
if (Get-Variable -Name PSNativeCommandUseErrorActionPreference -ErrorAction SilentlyContinue) {
|
||||
$PSNativeCommandUseErrorActionPreference = $false
|
||||
}
|
||||
|
||||
$Clean = $false
|
||||
$EnvName = "test_env"
|
||||
$PythonVersion = ""
|
||||
$Pyenv = $false
|
||||
$Local = $false
|
||||
$NoEnv = $false
|
||||
$bootstrapArgs = @($args)
|
||||
for ($i = 0; $i -lt $bootstrapArgs.Count; $i++) {
|
||||
$token = [string]$bootstrapArgs[$i]
|
||||
if ($token -eq "--pyenv") {
|
||||
$Pyenv = $true
|
||||
}
|
||||
elseif ($token -eq "-c" -or $token -eq "--clean") {
|
||||
$Clean = $true
|
||||
}
|
||||
elseif ($token -eq "--local") {
|
||||
$Local = $true
|
||||
}
|
||||
elseif ($token -eq "--no-env") {
|
||||
$NoEnv = $true
|
||||
}
|
||||
elseif ($token -eq "--env") {
|
||||
if (($i + 1) -ge $bootstrapArgs.Count) {
|
||||
Write-FionaError "--env requires an environment name argument"
|
||||
Remove-FionaBootstrapFunctions
|
||||
return
|
||||
}
|
||||
$i++
|
||||
$EnvName = [string]$bootstrapArgs[$i]
|
||||
}
|
||||
elseif ($token -eq "--python") {
|
||||
if (($i + 1) -ge $bootstrapArgs.Count) {
|
||||
Write-FionaError "--python requires a version number argument"
|
||||
Remove-FionaBootstrapFunctions
|
||||
return
|
||||
}
|
||||
$i++
|
||||
$PythonVersion = [string]$bootstrapArgs[$i]
|
||||
}
|
||||
else {
|
||||
Write-FionaError "Unknown option: $token"
|
||||
Write-FionaError "Usage: . .\bootstrap.ps1 [--pyenv] [-c|--clean] [--env custom_name] [--python 3.x.x] [--local] [--no-env]"
|
||||
Remove-FionaBootstrapFunctions
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
if (-not $NoEnv -and $MyInvocation.InvocationName -ne ".") {
|
||||
Write-FionaError "Dot-source this script so the virtual environment stays active in this session."
|
||||
Write-FionaError " . .\bootstrap.ps1"
|
||||
Write-FionaError "To install without a virtual environment, run:"
|
||||
Write-FionaError " powershell -ExecutionPolicy Bypass -File .\bootstrap.ps1 --local --no-env"
|
||||
Remove-FionaBootstrapFunctions
|
||||
return
|
||||
}
|
||||
|
||||
if ($Pyenv) {
|
||||
Write-FionaWarning "Windows bootstrap uses a local venv under .venvs\. It does not install pyenv."
|
||||
}
|
||||
|
||||
$repoRoot = $PSScriptRoot
|
||||
if (-not $repoRoot) {
|
||||
$repoRoot = (Get-Location).Path
|
||||
}
|
||||
|
||||
if ($NoEnv) {
|
||||
Write-FionaStatus "Starting project bootstrap (no virtual environment)"
|
||||
}
|
||||
else {
|
||||
Write-FionaStatus "Starting project bootstrap with environment: $EnvName"
|
||||
}
|
||||
if ($Local) {
|
||||
Write-FionaStatus "Local install: this repository, not editable"
|
||||
}
|
||||
if ($NoEnv) {
|
||||
Write-FionaStatus "No virtual environment: installing with --user"
|
||||
}
|
||||
if ($NoEnv -and $Clean) {
|
||||
Write-FionaWarning "--clean has no effect with --no-env"
|
||||
}
|
||||
|
||||
try {
|
||||
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass -Force -ErrorAction Stop | Out-Null
|
||||
}
|
||||
catch {
|
||||
# AI/LLM: a GPO can refuse the process override. RemoteSigned still runs local scripts.
|
||||
}
|
||||
|
||||
$pythonExe = Resolve-FionaPython -Version $PythonVersion
|
||||
if (-not $pythonExe) {
|
||||
Write-FionaError "Python is not installed, or version '$PythonVersion' is not available to the py launcher."
|
||||
Write-FionaError "Install Python from https://www.python.org/downloads/ and enable the py launcher."
|
||||
if ($PythonVersion) {
|
||||
Write-FionaError "Then re-run: . .\bootstrap.ps1 --python $PythonVersion"
|
||||
}
|
||||
Remove-FionaBootstrapFunctions
|
||||
return
|
||||
}
|
||||
|
||||
$pipPython = $pythonExe
|
||||
if (-not $NoEnv) {
|
||||
$venvRoot = Join-Path $repoRoot ".venvs"
|
||||
$venvPath = Join-Path $venvRoot $EnvName
|
||||
|
||||
if ($Clean -and (Test-Path -LiteralPath $venvPath)) {
|
||||
Write-FionaStatus "Deleting existing virtual environment $EnvName..."
|
||||
try {
|
||||
Remove-Item -LiteralPath $venvPath -Recurse -Force
|
||||
}
|
||||
catch {
|
||||
Write-FionaError "Failed to delete virtual environment $EnvName"
|
||||
Write-FionaError $_.Exception.Message
|
||||
Remove-FionaBootstrapFunctions
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
$activate = Join-Path $venvPath "Scripts\Activate.ps1"
|
||||
if (-not (Test-Path -LiteralPath $activate)) {
|
||||
Write-FionaStatus "Creating virtual environment $EnvName with $pythonExe..."
|
||||
New-Item -ItemType Directory -Path $venvRoot -Force | Out-Null
|
||||
& $pythonExe -m venv $venvPath
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
Write-FionaError "Failed to create virtual environment $EnvName"
|
||||
Remove-FionaBootstrapFunctions
|
||||
return
|
||||
}
|
||||
}
|
||||
else {
|
||||
Write-FionaStatus "Virtual environment $EnvName already exists"
|
||||
}
|
||||
|
||||
Write-FionaStatus "Activating $EnvName..."
|
||||
. $activate
|
||||
|
||||
if (-not $env:VIRTUAL_ENV) {
|
||||
Write-FionaError "Failed to activate $EnvName"
|
||||
Remove-FionaBootstrapFunctions
|
||||
return
|
||||
}
|
||||
|
||||
$pipPython = Join-Path $venvPath "Scripts\python.exe"
|
||||
}
|
||||
|
||||
if ($env:PYTEST_ADDOPTS) {
|
||||
$sanitized = $env:PYTEST_ADDOPTS -replace '--no-summary', '' -replace '(^|\s)-S(\s|$)', ' '
|
||||
$sanitized = ($sanitized -replace '\s+', ' ').Trim()
|
||||
if ($sanitized -ne $env:PYTEST_ADDOPTS.Trim()) {
|
||||
Write-FionaWarning "Removing --no-summary from PYTEST_ADDOPTS so test output can be parsed"
|
||||
if ($sanitized) {
|
||||
$env:PYTEST_ADDOPTS = $sanitized
|
||||
}
|
||||
else {
|
||||
Remove-Item Env:PYTEST_ADDOPTS
|
||||
}
|
||||
}
|
||||
}
|
||||
if ($env:FORCE_COLOR) {
|
||||
Write-FionaWarning "Unsetting FORCE_COLOR so pytest output can be parsed"
|
||||
Remove-Item Env:FORCE_COLOR
|
||||
}
|
||||
if ($env:PY_COLORS) {
|
||||
Write-FionaWarning "Unsetting PY_COLORS so pytest output can be parsed"
|
||||
Remove-Item Env:PY_COLORS
|
||||
}
|
||||
|
||||
$pyproject = Join-Path $repoRoot "pyproject.toml"
|
||||
if (-not (Test-Path -LiteralPath $pyproject)) {
|
||||
Write-FionaError "No pyproject.toml found at $pyproject"
|
||||
Remove-FionaBootstrapFunctions
|
||||
return
|
||||
}
|
||||
|
||||
Write-FionaStatus "Upgrading pip..."
|
||||
if ($NoEnv) {
|
||||
& $pipPython -m pip install --upgrade pip --user
|
||||
}
|
||||
else {
|
||||
& $pipPython -m pip install --upgrade pip
|
||||
}
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
Write-FionaWarning "Failed to upgrade pip. Continuing with package installation..."
|
||||
}
|
||||
|
||||
$helper = Join-Path $repoRoot "scripts\bootstrap_install.py"
|
||||
if (-not (Test-Path -LiteralPath $helper)) {
|
||||
Write-FionaError "Missing $helper"
|
||||
Remove-FionaBootstrapFunctions
|
||||
return
|
||||
}
|
||||
$helperArgs = @($helper)
|
||||
if ($Local) {
|
||||
$helperArgs += "--local"
|
||||
}
|
||||
if ($NoEnv) {
|
||||
$helperArgs += "--no-env"
|
||||
}
|
||||
$pipWords = @(& $pipPython @helperArgs | Where-Object { $_ -and "$_".Trim() -ne "" })
|
||||
if ($LASTEXITCODE -ne 0 -or $pipWords.Count -eq 0) {
|
||||
Write-FionaError "Failed to resolve pip install arguments"
|
||||
Remove-FionaBootstrapFunctions
|
||||
return
|
||||
}
|
||||
|
||||
if ($Local) {
|
||||
Write-FionaStatus "Installing this repository as a normal package..."
|
||||
}
|
||||
else {
|
||||
Write-FionaStatus "Installing package and dependencies from pyproject.toml..."
|
||||
}
|
||||
Push-Location $repoRoot
|
||||
try {
|
||||
& $pipPython -m pip @pipWords
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
Write-FionaError "Failed to install package and dependencies from pyproject.toml"
|
||||
Remove-FionaBootstrapFunctions
|
||||
return
|
||||
}
|
||||
}
|
||||
finally {
|
||||
Pop-Location
|
||||
}
|
||||
|
||||
$pythonVersionText = (& $pipPython --version 2>&1 | Out-String).Trim()
|
||||
$pipVersionText = (& $pipPython -m pip --version 2>&1 | Out-String).Trim()
|
||||
|
||||
Write-FionaStatus "Bootstrap completed successfully!"
|
||||
if ($NoEnv) {
|
||||
$scriptsDir = (& $pipPython -c "import sysconfig; print(sysconfig.get_path('scripts', 'nt_user'))" | Out-String).Trim()
|
||||
Write-FionaStatus "Installed for the current user (no virtual environment)"
|
||||
Write-FionaStatus "Command directory: $scriptsDir"
|
||||
Write-FionaStatus "If fiona-assistant is not found, add that directory to PATH."
|
||||
}
|
||||
else {
|
||||
Write-FionaStatus "Virtual environment '$EnvName' is active"
|
||||
}
|
||||
Write-FionaStatus "Python: $pythonVersionText ($pipPython)"
|
||||
Write-FionaStatus "Pip: $pipVersionText"
|
||||
Write-FionaStatus ""
|
||||
Write-FionaStatus "To verify installation, try:"
|
||||
if (-not $NoEnv) {
|
||||
Write-FionaStatus " python scripts/run_tests.py"
|
||||
}
|
||||
Write-FionaStatus " fiona-assistant --help"
|
||||
Write-FionaStatus ""
|
||||
Write-FionaStatus "Happy coding!"
|
||||
|
||||
Remove-FionaBootstrapFunctions
|
||||
@@ -11,6 +11,8 @@ if ! python -m build --version >/dev/null 2>&1; then
|
||||
fi
|
||||
|
||||
# Build the distribution files
|
||||
rm -rf dist/ build/ fiona_assistant.egg-info/
|
||||
|
||||
python -m build
|
||||
|
||||
echo "Build completed. Distribution files are in the dist/ directory."
|
||||
|
||||
-612
@@ -1,612 +0,0 @@
|
||||
<?xml version="1.0" ?>
|
||||
<coverage version="7.16.1" timestamp="1790528640380" lines-valid="578" lines-covered="454" line-rate="0.7855" branches-covered="0" branches-valid="0" branch-rate="0" complexity="0">
|
||||
<!-- Generated by coverage.py: https://coverage.readthedocs.io/en/7.16.1 -->
|
||||
<!-- Based on https://raw.githubusercontent.com/cobertura/web/master/htdocs/xml/coverage-04.dtd -->
|
||||
<sources>
|
||||
<source>/home/user/repositories/fiona-assistant/src/customlinter</source>
|
||||
</sources>
|
||||
<packages>
|
||||
<package name="." line-rate="0.7855" branch-rate="0" complexity="0">
|
||||
<classes>
|
||||
<class name="__init__.py" filename="__init__.py" complexity="0" line-rate="1" branch-rate="0">
|
||||
<methods/>
|
||||
<lines>
|
||||
<line number="17" hits="1"/>
|
||||
<line number="26" hits="1"/>
|
||||
<line number="27" hits="1"/>
|
||||
<line number="29" hits="1"/>
|
||||
</lines>
|
||||
</class>
|
||||
<class name="base.py" filename="base.py" complexity="0" line-rate="0.6962" branch-rate="0">
|
||||
<methods/>
|
||||
<lines>
|
||||
<line number="8" hits="1"/>
|
||||
<line number="10" hits="1"/>
|
||||
<line number="11" hits="1"/>
|
||||
<line number="12" hits="1"/>
|
||||
<line number="13" hits="1"/>
|
||||
<line number="14" hits="1"/>
|
||||
<line number="15" hits="1"/>
|
||||
<line number="16" hits="1"/>
|
||||
<line number="18" hits="1"/>
|
||||
<line number="25" hits="1"/>
|
||||
<line number="28" hits="1"/>
|
||||
<line number="33" hits="1"/>
|
||||
<line number="49" hits="1"/>
|
||||
<line number="52" hits="1"/>
|
||||
<line number="65" hits="1"/>
|
||||
<line number="66" hits="1"/>
|
||||
<line number="67" hits="1"/>
|
||||
<line number="68" hits="1"/>
|
||||
<line number="72" hits="1"/>
|
||||
<line number="79" hits="1"/>
|
||||
<line number="90" hits="1"/>
|
||||
<line number="97" hits="1"/>
|
||||
<line number="118" hits="1"/>
|
||||
<line number="119" hits="1"/>
|
||||
<line number="120" hits="1"/>
|
||||
<line number="121" hits="1"/>
|
||||
<line number="122" hits="1"/>
|
||||
<line number="123" hits="1"/>
|
||||
<line number="124" hits="1"/>
|
||||
<line number="126" hits="1"/>
|
||||
<line number="128" hits="1"/>
|
||||
<line number="138" hits="1"/>
|
||||
<line number="140" hits="1"/>
|
||||
<line number="147" hits="1"/>
|
||||
<line number="164" hits="1"/>
|
||||
<line number="167" hits="1"/>
|
||||
<line number="170" hits="1"/>
|
||||
<line number="172" hits="1"/>
|
||||
<line number="178" hits="1"/>
|
||||
<line number="179" hits="1"/>
|
||||
<line number="181" hits="1"/>
|
||||
<line number="183" hits="1"/>
|
||||
<line number="184" hits="1"/>
|
||||
<line number="186" hits="1"/>
|
||||
<line number="187" hits="1"/>
|
||||
<line number="188" hits="1"/>
|
||||
<line number="190" hits="1"/>
|
||||
<line number="191" hits="1"/>
|
||||
<line number="192" hits="1"/>
|
||||
<line number="193" hits="1"/>
|
||||
<line number="194" hits="0"/>
|
||||
<line number="195" hits="1"/>
|
||||
<line number="196" hits="1"/>
|
||||
<line number="197" hits="1"/>
|
||||
<line number="210" hits="1"/>
|
||||
<line number="219" hits="0"/>
|
||||
<line number="220" hits="0"/>
|
||||
<line number="222" hits="0"/>
|
||||
<line number="223" hits="0"/>
|
||||
<line number="226" hits="0"/>
|
||||
<line number="229" hits="0"/>
|
||||
<line number="230" hits="0"/>
|
||||
<line number="231" hits="0"/>
|
||||
<line number="233" hits="0"/>
|
||||
<line number="235" hits="1"/>
|
||||
<line number="244" hits="1"/>
|
||||
<line number="247" hits="1"/>
|
||||
<line number="248" hits="1"/>
|
||||
<line number="250" hits="1"/>
|
||||
<line number="251" hits="1"/>
|
||||
<line number="252" hits="1"/>
|
||||
<line number="253" hits="1"/>
|
||||
<line number="254" hits="1"/>
|
||||
<line number="255" hits="1"/>
|
||||
<line number="256" hits="1"/>
|
||||
<line number="257" hits="0"/>
|
||||
<line number="258" hits="0"/>
|
||||
<line number="261" hits="1"/>
|
||||
<line number="262" hits="1"/>
|
||||
<line number="265" hits="1"/>
|
||||
<line number="266" hits="1"/>
|
||||
<line number="267" hits="1"/>
|
||||
<line number="270" hits="1"/>
|
||||
<line number="272" hits="1"/>
|
||||
<line number="273" hits="1"/>
|
||||
<line number="276" hits="1"/>
|
||||
<line number="278" hits="1"/>
|
||||
<line number="280" hits="1"/>
|
||||
<line number="290" hits="1"/>
|
||||
<line number="291" hits="1"/>
|
||||
<line number="292" hits="1"/>
|
||||
<line number="294" hits="1"/>
|
||||
<line number="295" hits="1"/>
|
||||
<line number="296" hits="1"/>
|
||||
<line number="298" hits="1"/>
|
||||
<line number="308" hits="1"/>
|
||||
<line number="309" hits="1"/>
|
||||
<line number="312" hits="1"/>
|
||||
<line number="313" hits="1"/>
|
||||
<line number="316" hits="1"/>
|
||||
<line number="317" hits="1"/>
|
||||
<line number="318" hits="1"/>
|
||||
<line number="319" hits="1"/>
|
||||
<line number="320" hits="1"/>
|
||||
<line number="321" hits="1"/>
|
||||
<line number="323" hits="1"/>
|
||||
<line number="325" hits="1"/>
|
||||
<line number="335" hits="1"/>
|
||||
<line number="336" hits="1"/>
|
||||
<line number="337" hits="1"/>
|
||||
<line number="338" hits="1"/>
|
||||
<line number="339" hits="1"/>
|
||||
<line number="340" hits="0"/>
|
||||
<line number="342" hits="0"/>
|
||||
<line number="343" hits="0"/>
|
||||
<line number="345" hits="1"/>
|
||||
<line number="354" hits="1"/>
|
||||
<line number="355" hits="1"/>
|
||||
<line number="357" hits="1"/>
|
||||
<line number="358" hits="1"/>
|
||||
<line number="359" hits="1"/>
|
||||
<line number="361" hits="1"/>
|
||||
<line number="363" hits="1"/>
|
||||
<line number="372" hits="1"/>
|
||||
<line number="374" hits="1"/>
|
||||
<line number="390" hits="0"/>
|
||||
<line number="391" hits="0"/>
|
||||
<line number="392" hits="0"/>
|
||||
<line number="393" hits="0"/>
|
||||
<line number="396" hits="0"/>
|
||||
<line number="397" hits="0"/>
|
||||
<line number="398" hits="0"/>
|
||||
<line number="399" hits="0"/>
|
||||
<line number="400" hits="0"/>
|
||||
<line number="406" hits="0"/>
|
||||
<line number="407" hits="0"/>
|
||||
<line number="408" hits="0"/>
|
||||
<line number="409" hits="0"/>
|
||||
<line number="412" hits="0"/>
|
||||
<line number="415" hits="0"/>
|
||||
<line number="416" hits="0"/>
|
||||
<line number="417" hits="0"/>
|
||||
<line number="418" hits="0"/>
|
||||
<line number="419" hits="0"/>
|
||||
<line number="420" hits="0"/>
|
||||
<line number="421" hits="0"/>
|
||||
<line number="423" hits="0"/>
|
||||
<line number="424" hits="0"/>
|
||||
<line number="425" hits="0"/>
|
||||
<line number="426" hits="0"/>
|
||||
<line number="427" hits="0"/>
|
||||
<line number="429" hits="0"/>
|
||||
<line number="430" hits="0"/>
|
||||
<line number="433" hits="0"/>
|
||||
<line number="434" hits="0"/>
|
||||
<line number="435" hits="0"/>
|
||||
<line number="436" hits="0"/>
|
||||
<line number="438" hits="0"/>
|
||||
</lines>
|
||||
</class>
|
||||
<class name="fmlint.py" filename="fmlint.py" complexity="0" line-rate="0.766" branch-rate="0">
|
||||
<methods/>
|
||||
<lines>
|
||||
<line number="19" hits="1"/>
|
||||
<line number="21" hits="1"/>
|
||||
<line number="22" hits="1"/>
|
||||
<line number="23" hits="1"/>
|
||||
<line number="24" hits="1"/>
|
||||
<line number="25" hits="1"/>
|
||||
<line number="27" hits="1"/>
|
||||
<line number="29" hits="1"/>
|
||||
<line number="37" hits="1"/>
|
||||
<line number="40" hits="1"/>
|
||||
<line number="43" hits="1"/>
|
||||
<line number="46" hits="1"/>
|
||||
<line number="53" hits="1"/>
|
||||
<line number="103" hits="1"/>
|
||||
<line number="115" hits="0"/>
|
||||
<line number="116" hits="0"/>
|
||||
<line number="117" hits="0"/>
|
||||
<line number="118" hits="0"/>
|
||||
<line number="119" hits="0"/>
|
||||
<line number="120" hits="0"/>
|
||||
<line number="121" hits="0"/>
|
||||
<line number="122" hits="0"/>
|
||||
<line number="125" hits="1"/>
|
||||
<line number="134" hits="1"/>
|
||||
<line number="137" hits="1"/>
|
||||
<line number="148" hits="1"/>
|
||||
<line number="151" hits="1"/>
|
||||
<line number="152" hits="1"/>
|
||||
<line number="155" hits="1"/>
|
||||
<line number="156" hits="1"/>
|
||||
<line number="158" hits="1"/>
|
||||
<line number="161" hits="1"/>
|
||||
<line number="180" hits="1"/>
|
||||
<line number="181" hits="1"/>
|
||||
<line number="183" hits="1"/>
|
||||
<line number="184" hits="1"/>
|
||||
<line number="186" hits="1"/>
|
||||
<line number="187" hits="1"/>
|
||||
<line number="190" hits="1"/>
|
||||
<line number="191" hits="1"/>
|
||||
<line number="192" hits="1"/>
|
||||
<line number="193" hits="1"/>
|
||||
<line number="196" hits="1"/>
|
||||
<line number="197" hits="1"/>
|
||||
<line number="198" hits="1"/>
|
||||
<line number="199" hits="1"/>
|
||||
<line number="202" hits="1"/>
|
||||
<line number="203" hits="1"/>
|
||||
<line number="210" hits="1"/>
|
||||
<line number="212" hits="1"/>
|
||||
<line number="213" hits="1"/>
|
||||
<line number="219" hits="1"/>
|
||||
<line number="226" hits="1"/>
|
||||
<line number="238" hits="1"/>
|
||||
<line number="239" hits="1"/>
|
||||
<line number="240" hits="1"/>
|
||||
<line number="242" hits="1"/>
|
||||
<line number="258" hits="1"/>
|
||||
<line number="261" hits="1"/>
|
||||
<line number="262" hits="1"/>
|
||||
<line number="263" hits="1"/>
|
||||
<line number="264" hits="1"/>
|
||||
<line number="265" hits="1"/>
|
||||
<line number="266" hits="1"/>
|
||||
<line number="267" hits="1"/>
|
||||
<line number="270" hits="1"/>
|
||||
<line number="272" hits="1"/>
|
||||
<line number="275" hits="1"/>
|
||||
<line number="278" hits="1"/>
|
||||
<line number="281" hits="1"/>
|
||||
<line number="282" hits="1"/>
|
||||
<line number="283" hits="1"/>
|
||||
<line number="286" hits="1"/>
|
||||
<line number="287" hits="1"/>
|
||||
<line number="290" hits="1"/>
|
||||
<line number="291" hits="1"/>
|
||||
<line number="293" hits="1"/>
|
||||
<line number="302" hits="1"/>
|
||||
<line number="304" hits="1"/>
|
||||
<line number="306" hits="1"/>
|
||||
<line number="307" hits="0"/>
|
||||
<line number="310" hits="1"/>
|
||||
<line number="311" hits="0"/>
|
||||
<line number="314" hits="1"/>
|
||||
<line number="315" hits="1"/>
|
||||
<line number="316" hits="1"/>
|
||||
<line number="318" hits="0"/>
|
||||
<line number="319" hits="0"/>
|
||||
<line number="320" hits="0"/>
|
||||
<line number="321" hits="0"/>
|
||||
<line number="322" hits="0"/>
|
||||
<line number="323" hits="0"/>
|
||||
<line number="325" hits="0"/>
|
||||
<line number="326" hits="0"/>
|
||||
<line number="327" hits="1"/>
|
||||
<line number="328" hits="0"/>
|
||||
<line number="331" hits="1"/>
|
||||
<line number="332" hits="1"/>
|
||||
<line number="333" hits="0"/>
|
||||
<line number="335" hits="0"/>
|
||||
<line number="343" hits="0"/>
|
||||
<line number="344" hits="0"/>
|
||||
<line number="347" hits="1"/>
|
||||
<line number="348" hits="1"/>
|
||||
<line number="349" hits="1"/>
|
||||
<line number="350" hits="1"/>
|
||||
<line number="358" hits="1"/>
|
||||
<line number="359" hits="1"/>
|
||||
<line number="362" hits="1"/>
|
||||
<line number="363" hits="1"/>
|
||||
<line number="371" hits="1"/>
|
||||
<line number="372" hits="1"/>
|
||||
<line number="375" hits="1"/>
|
||||
<line number="378" hits="1"/>
|
||||
<line number="384" hits="1"/>
|
||||
<line number="385" hits="1"/>
|
||||
<line number="386" hits="1"/>
|
||||
<line number="394" hits="1"/>
|
||||
<line number="397" hits="1"/>
|
||||
<line number="399" hits="1"/>
|
||||
<line number="400" hits="1"/>
|
||||
<line number="408" hits="1"/>
|
||||
<line number="410" hits="1"/>
|
||||
<line number="411" hits="1"/>
|
||||
<line number="413" hits="1"/>
|
||||
<line number="414" hits="1"/>
|
||||
<line number="416" hits="1"/>
|
||||
<line number="417" hits="1"/>
|
||||
<line number="418" hits="1"/>
|
||||
<line number="426" hits="1"/>
|
||||
<line number="427" hits="1"/>
|
||||
<line number="428" hits="1"/>
|
||||
<line number="436" hits="1"/>
|
||||
<line number="439" hits="1"/>
|
||||
<line number="444" hits="1"/>
|
||||
<line number="446" hits="1"/>
|
||||
<line number="473" hits="1"/>
|
||||
<line number="476" hits="1"/>
|
||||
<line number="487" hits="1"/>
|
||||
<line number="489" hits="1"/>
|
||||
<line number="491" hits="1"/>
|
||||
<line number="492" hits="1"/>
|
||||
<line number="493" hits="1"/>
|
||||
<line number="495" hits="0"/>
|
||||
<line number="496" hits="0"/>
|
||||
<line number="497" hits="0"/>
|
||||
<line number="498" hits="0"/>
|
||||
<line number="499" hits="0"/>
|
||||
<line number="500" hits="0"/>
|
||||
<line number="502" hits="0"/>
|
||||
<line number="503" hits="0"/>
|
||||
<line number="504" hits="1"/>
|
||||
<line number="505" hits="0"/>
|
||||
<line number="508" hits="1"/>
|
||||
<line number="511" hits="1"/>
|
||||
<line number="512" hits="1"/>
|
||||
<line number="513" hits="1"/>
|
||||
<line number="514" hits="1"/>
|
||||
<line number="516" hits="1"/>
|
||||
<line number="517" hits="1"/>
|
||||
<line number="518" hits="1"/>
|
||||
<line number="523" hits="1"/>
|
||||
<line number="524" hits="1"/>
|
||||
<line number="526" hits="1"/>
|
||||
<line number="527" hits="1"/>
|
||||
<line number="528" hits="1"/>
|
||||
<line number="530" hits="1"/>
|
||||
<line number="532" hits="1"/>
|
||||
<line number="548" hits="1"/>
|
||||
<line number="549" hits="1"/>
|
||||
<line number="550" hits="1"/>
|
||||
<line number="551" hits="1"/>
|
||||
<line number="552" hits="1"/>
|
||||
<line number="556" hits="1"/>
|
||||
<line number="558" hits="1"/>
|
||||
<line number="570" hits="1"/>
|
||||
<line number="572" hits="1"/>
|
||||
<line number="573" hits="1"/>
|
||||
<line number="575" hits="1"/>
|
||||
<line number="578" hits="1"/>
|
||||
<line number="579" hits="1"/>
|
||||
<line number="580" hits="1"/>
|
||||
<line number="583" hits="1"/>
|
||||
<line number="584" hits="1"/>
|
||||
<line number="587" hits="1"/>
|
||||
<line number="588" hits="0"/>
|
||||
<line number="591" hits="1"/>
|
||||
<line number="592" hits="0"/>
|
||||
<line number="594" hits="1"/>
|
||||
<line number="597" hits="1"/>
|
||||
<line number="598" hits="1"/>
|
||||
<line number="599" hits="1"/>
|
||||
<line number="600" hits="1"/>
|
||||
<line number="601" hits="1"/>
|
||||
<line number="602" hits="1"/>
|
||||
<line number="603" hits="1"/>
|
||||
<line number="605" hits="1"/>
|
||||
<line number="607" hits="1"/>
|
||||
<line number="617" hits="1"/>
|
||||
<line number="618" hits="1"/>
|
||||
<line number="619" hits="1"/>
|
||||
<line number="620" hits="1"/>
|
||||
<line number="621" hits="1"/>
|
||||
<line number="622" hits="0"/>
|
||||
<line number="623" hits="1"/>
|
||||
<line number="624" hits="1"/>
|
||||
<line number="625" hits="1"/>
|
||||
<line number="627" hits="1"/>
|
||||
<line number="637" hits="1"/>
|
||||
<line number="638" hits="1"/>
|
||||
<line number="639" hits="1"/>
|
||||
<line number="641" hits="1"/>
|
||||
<line number="642" hits="1"/>
|
||||
<line number="643" hits="1"/>
|
||||
<line number="645" hits="1"/>
|
||||
<line number="646" hits="1"/>
|
||||
<line number="647" hits="1"/>
|
||||
<line number="648" hits="1"/>
|
||||
<line number="649" hits="1"/>
|
||||
<line number="651" hits="1"/>
|
||||
<line number="652" hits="1"/>
|
||||
<line number="653" hits="0"/>
|
||||
<line number="655" hits="1"/>
|
||||
<line number="656" hits="1"/>
|
||||
<line number="658" hits="1"/>
|
||||
<line number="660" hits="1"/>
|
||||
<line number="671" hits="1"/>
|
||||
<line number="673" hits="1"/>
|
||||
<line number="674" hits="1"/>
|
||||
<line number="675" hits="1"/>
|
||||
<line number="676" hits="1"/>
|
||||
<line number="679" hits="1"/>
|
||||
<line number="680" hits="1"/>
|
||||
<line number="681" hits="1"/>
|
||||
<line number="682" hits="1"/>
|
||||
<line number="683" hits="1"/>
|
||||
<line number="684" hits="1"/>
|
||||
<line number="685" hits="1"/>
|
||||
<line number="688" hits="1"/>
|
||||
<line number="689" hits="1"/>
|
||||
<line number="690" hits="1"/>
|
||||
<line number="691" hits="1"/>
|
||||
<line number="692" hits="1"/>
|
||||
<line number="693" hits="1"/>
|
||||
<line number="695" hits="1"/>
|
||||
<line number="696" hits="1"/>
|
||||
<line number="699" hits="1"/>
|
||||
<line number="700" hits="1"/>
|
||||
<line number="701" hits="1"/>
|
||||
<line number="702" hits="1"/>
|
||||
<line number="703" hits="1"/>
|
||||
<line number="705" hits="1"/>
|
||||
<line number="707" hits="1"/>
|
||||
<line number="725" hits="0"/>
|
||||
<line number="726" hits="0"/>
|
||||
<line number="727" hits="0"/>
|
||||
<line number="728" hits="0"/>
|
||||
<line number="731" hits="0"/>
|
||||
<line number="732" hits="0"/>
|
||||
<line number="733" hits="0"/>
|
||||
<line number="734" hits="0"/>
|
||||
<line number="735" hits="0"/>
|
||||
<line number="741" hits="0"/>
|
||||
<line number="742" hits="0"/>
|
||||
<line number="743" hits="0"/>
|
||||
<line number="744" hits="0"/>
|
||||
<line number="745" hits="0"/>
|
||||
<line number="746" hits="0"/>
|
||||
<line number="747" hits="0"/>
|
||||
<line number="750" hits="0"/>
|
||||
<line number="751" hits="0"/>
|
||||
<line number="752" hits="0"/>
|
||||
<line number="753" hits="0"/>
|
||||
<line number="754" hits="0"/>
|
||||
<line number="755" hits="0"/>
|
||||
<line number="758" hits="0"/>
|
||||
<line number="759" hits="0"/>
|
||||
<line number="760" hits="0"/>
|
||||
<line number="761" hits="0"/>
|
||||
<line number="762" hits="0"/>
|
||||
<line number="763" hits="0"/>
|
||||
<line number="765" hits="0"/>
|
||||
<line number="766" hits="0"/>
|
||||
<line number="769" hits="0"/>
|
||||
<line number="770" hits="0"/>
|
||||
<line number="771" hits="0"/>
|
||||
<line number="772" hits="0"/>
|
||||
<line number="774" hits="0"/>
|
||||
<line number="781" hits="1"/>
|
||||
<line number="783" hits="1"/>
|
||||
<line number="784" hits="1"/>
|
||||
<line number="786" hits="1"/>
|
||||
<line number="788" hits="1"/>
|
||||
<line number="790" hits="1"/>
|
||||
<line number="793" hits="1"/>
|
||||
<line number="796" hits="1"/>
|
||||
<line number="800" hits="1"/>
|
||||
<line number="807" hits="1"/>
|
||||
<line number="809" hits="1"/>
|
||||
<line number="811" hits="1"/>
|
||||
<line number="812" hits="1"/>
|
||||
<line number="814" hits="1"/>
|
||||
<line number="815" hits="1"/>
|
||||
<line number="816" hits="1"/>
|
||||
<line number="818" hits="1"/>
|
||||
<line number="819" hits="1"/>
|
||||
<line number="820" hits="1"/>
|
||||
<line number="822" hits="1"/>
|
||||
<line number="823" hits="1"/>
|
||||
<line number="824" hits="1"/>
|
||||
<line number="826" hits="0"/>
|
||||
<line number="827" hits="0"/>
|
||||
</lines>
|
||||
</class>
|
||||
<class name="pycustomlint.py" filename="pycustomlint.py" complexity="0" line-rate="0.9712" branch-rate="0">
|
||||
<methods/>
|
||||
<lines>
|
||||
<line number="19" hits="1"/>
|
||||
<line number="21" hits="1"/>
|
||||
<line number="22" hits="1"/>
|
||||
<line number="23" hits="1"/>
|
||||
<line number="24" hits="1"/>
|
||||
<line number="25" hits="1"/>
|
||||
<line number="27" hits="1"/>
|
||||
<line number="29" hits="1"/>
|
||||
<line number="32" hits="1"/>
|
||||
<line number="40" hits="1"/>
|
||||
<line number="60" hits="1"/>
|
||||
<line number="63" hits="1"/>
|
||||
<line number="69" hits="1"/>
|
||||
<line number="70" hits="1"/>
|
||||
<line number="72" hits="1"/>
|
||||
<line number="74" hits="1"/>
|
||||
<line number="75" hits="1"/>
|
||||
<line number="76" hits="1"/>
|
||||
<line number="78" hits="1"/>
|
||||
<line number="80" hits="1"/>
|
||||
<line number="81" hits="1"/>
|
||||
<line number="82" hits="1"/>
|
||||
<line number="84" hits="1"/>
|
||||
<line number="87" hits="1"/>
|
||||
<line number="88" hits="1"/>
|
||||
<line number="89" hits="1"/>
|
||||
<line number="90" hits="1"/>
|
||||
<line number="91" hits="1"/>
|
||||
<line number="92" hits="1"/>
|
||||
<line number="93" hits="1"/>
|
||||
<line number="94" hits="1"/>
|
||||
<line number="95" hits="1"/>
|
||||
<line number="97" hits="1"/>
|
||||
<line number="105" hits="1"/>
|
||||
<line number="107" hits="1"/>
|
||||
<line number="109" hits="1"/>
|
||||
<line number="116" hits="1"/>
|
||||
<line number="122" hits="1"/>
|
||||
<line number="123" hits="1"/>
|
||||
<line number="124" hits="1"/>
|
||||
<line number="126" hits="1"/>
|
||||
<line number="135" hits="1"/>
|
||||
<line number="137" hits="1"/>
|
||||
<line number="139" hits="1"/>
|
||||
<line number="140" hits="1"/>
|
||||
<line number="143" hits="1"/>
|
||||
<line number="144" hits="0"/>
|
||||
<line number="147" hits="1"/>
|
||||
<line number="148" hits="1"/>
|
||||
<line number="151" hits="1"/>
|
||||
<line number="152" hits="1"/>
|
||||
<line number="153" hits="1"/>
|
||||
<line number="155" hits="1"/>
|
||||
<line number="157" hits="1"/>
|
||||
<line number="158" hits="1"/>
|
||||
<line number="159" hits="1"/>
|
||||
<line number="161" hits="1"/>
|
||||
<line number="163" hits="1"/>
|
||||
<line number="177" hits="1"/>
|
||||
<line number="178" hits="1"/>
|
||||
<line number="179" hits="1"/>
|
||||
<line number="181" hits="1"/>
|
||||
<line number="182" hits="1"/>
|
||||
<line number="185" hits="1"/>
|
||||
<line number="188" hits="1"/>
|
||||
<line number="189" hits="1"/>
|
||||
<line number="190" hits="1"/>
|
||||
<line number="192" hits="1"/>
|
||||
<line number="194" hits="1"/>
|
||||
<line number="196" hits="1"/>
|
||||
<line number="203" hits="1"/>
|
||||
<line number="205" hits="1"/>
|
||||
<line number="207" hits="1"/>
|
||||
<line number="209" hits="1"/>
|
||||
<line number="210" hits="1"/>
|
||||
<line number="212" hits="1"/>
|
||||
<line number="215" hits="1"/>
|
||||
<line number="218" hits="1"/>
|
||||
<line number="222" hits="1"/>
|
||||
<line number="228" hits="1"/>
|
||||
<line number="234" hits="1"/>
|
||||
<line number="236" hits="1"/>
|
||||
<line number="238" hits="1"/>
|
||||
<line number="239" hits="1"/>
|
||||
<line number="240" hits="1"/>
|
||||
<line number="242" hits="1"/>
|
||||
<line number="243" hits="1"/>
|
||||
<line number="245" hits="1"/>
|
||||
<line number="247" hits="1"/>
|
||||
<line number="248" hits="1"/>
|
||||
<line number="249" hits="1"/>
|
||||
<line number="252" hits="1"/>
|
||||
<line number="253" hits="1"/>
|
||||
<line number="255" hits="1"/>
|
||||
<line number="257" hits="1"/>
|
||||
<line number="258" hits="1"/>
|
||||
<line number="260" hits="1"/>
|
||||
<line number="262" hits="1"/>
|
||||
<line number="263" hits="1"/>
|
||||
<line number="264" hits="1"/>
|
||||
<line number="266" hits="1"/>
|
||||
<line number="267" hits="1"/>
|
||||
<line number="269" hits="0"/>
|
||||
<line number="270" hits="0"/>
|
||||
</lines>
|
||||
</class>
|
||||
</classes>
|
||||
</package>
|
||||
</packages>
|
||||
</coverage>
|
||||
+28
-20
@@ -1,29 +1,32 @@
|
||||
[build-system]
|
||||
requires = ["setuptools>=42"]
|
||||
requires = ["setuptools>=61"]
|
||||
build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "fiona-assistant"
|
||||
version = "0.36.0"
|
||||
version = "1.19.0"
|
||||
description = "Tools for managing the LLM wiki."
|
||||
authors = [{name = "nvaldeziii"}]
|
||||
readme = "README.md"
|
||||
license = "GPL-3.0-only"
|
||||
requires-python = ">=3.8"
|
||||
requires-python = ">=3.11"
|
||||
classifiers = [
|
||||
"Programming Language :: Python :: 3",
|
||||
"Programming Language :: Python :: 3.11",
|
||||
"Programming Language :: Python :: 3.12",
|
||||
"Programming Language :: Python :: 3.13",
|
||||
"Operating System :: OS Independent",
|
||||
]
|
||||
|
||||
# Core dependencies (required for runtime)
|
||||
# Note: html5validator requires Java 8+ to be installed on the system
|
||||
dependencies = [
|
||||
"pymarkdownlnt>=0.9.0",
|
||||
"ruff>=0.16.8",
|
||||
"pyyaml>=6.0.3",
|
||||
"pylint",
|
||||
"GitPython>=3.1.0",
|
||||
"html5validator>=0.4.0",
|
||||
"beautifulsoup4>=4.12.0",
|
||||
"markdownify>=1.1.0",
|
||||
"pypdf>=5.0.0",
|
||||
]
|
||||
|
||||
# Optional dependencies groups
|
||||
@@ -32,6 +35,7 @@ dev = [
|
||||
"pytest>=7.0.0",
|
||||
"pytest-cov>=4.0.0",
|
||||
"pytest-mock",
|
||||
"pylint",
|
||||
"build",
|
||||
"twine",
|
||||
]
|
||||
@@ -41,23 +45,23 @@ include-package-data = true
|
||||
|
||||
[tool.setuptools.packages.find]
|
||||
where = ["src"]
|
||||
# llm-wiki is a hyphenated, non-importable data-only "package" that ships
|
||||
# template files (wiki/, ingest/, AGENTS.md, etc.) as package data.
|
||||
# The hyphen makes it unimportable as a Python module (import llm-wiki
|
||||
# is a SyntaxError), but setuptools allows it as a distribution package name.
|
||||
# The __init__.py exists only to satisfy setuptools' package detection.
|
||||
include = ["fiona", "librarian", "mdoctor", "python_ruff", "customlinter", "sanity_runner", "llm-wiki", "wiki", "gittool", "html_js_lint"]
|
||||
include = ["fiona*"]
|
||||
|
||||
[tool.setuptools.package-data]
|
||||
# Include all files under llm-wiki/ as package data (not Python code)
|
||||
"llm-wiki" = ["*", "**/*"]
|
||||
fiona = [
|
||||
"data/llm_wiki/**/*",
|
||||
"data/default.gitignore",
|
||||
"wiki/templates/*",
|
||||
"gittool/templates/*",
|
||||
]
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["src/tests"]
|
||||
testpaths = ["tests"]
|
||||
pythonpath = ["src"]
|
||||
|
||||
[tool.coverage.run]
|
||||
source = ["src"]
|
||||
omit = ["src/tests/*", "*/tests/*", "*/test_*.py"]
|
||||
source = ["src/fiona"]
|
||||
omit = ["*/tests/*", "*/test_*.py"]
|
||||
|
||||
[tool.coverage.report]
|
||||
exclude_lines = [
|
||||
@@ -73,6 +77,10 @@ exclude_lines = [
|
||||
# Define console scripts entry points
|
||||
[project.scripts]
|
||||
fiona-assistant = "fiona.__main__:main"
|
||||
librarian = "librarian.librarian:main"
|
||||
mdoctor = "mdoctor.mdoctor:main"
|
||||
html-js-lint = "html_js_lint.html_js_lint:main"
|
||||
librarian = "fiona.librarian.librarian:main"
|
||||
mdoctor = "fiona.mdoctor.mdoctor:main"
|
||||
|
||||
[tool.ruff]
|
||||
target-version = "py311"
|
||||
line-length = 100
|
||||
src = ["src"]
|
||||
|
||||
@@ -0,0 +1,83 @@
|
||||
"""Pip arguments for bootstrap install modes.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - local and no-env install options.
|
||||
|
||||
SPEC 7.21. Both ``bootstrap`` and ``bootstrap.ps1`` call this script
|
||||
and read one pip argument per stdout line.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
|
||||
|
||||
def pip_install_args(
|
||||
*,
|
||||
local: bool,
|
||||
no_env: bool,
|
||||
break_system_packages: bool = False,
|
||||
) -> list[str]:
|
||||
"""Return the arguments passed to ``python -m pip``.
|
||||
|
||||
Args:
|
||||
local: Install this repository as a normal package. When
|
||||
false, the install is editable and includes the ``dev``
|
||||
extra.
|
||||
no_env: Add ``--user`` so the install does not need a virtual
|
||||
environment.
|
||||
break_system_packages: Add ``--break-system-packages``. Bash
|
||||
sets this. PowerShell does not.
|
||||
|
||||
Returns:
|
||||
The pip argument list, starting with ``install``.
|
||||
"""
|
||||
args = ["install"]
|
||||
if no_env:
|
||||
args.append("--user")
|
||||
if break_system_packages:
|
||||
args.append("--break-system-packages")
|
||||
if local:
|
||||
args.append(".")
|
||||
else:
|
||||
args.extend(["-e", ".[dev]"])
|
||||
return args
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
"""Print pip arguments, one per line.
|
||||
|
||||
Args:
|
||||
argv: Flags to parse. ``None`` reads ``sys.argv``.
|
||||
|
||||
Returns:
|
||||
Process exit code. ``0`` after the arguments are written.
|
||||
"""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Print pip install arguments for bootstrap.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--local",
|
||||
action="store_true",
|
||||
help="install this repository as a normal package",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--no-env",
|
||||
action="store_true",
|
||||
help="add --user so no virtual environment is required",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--break-system-packages",
|
||||
action="store_true",
|
||||
help="pass --break-system-packages to pip",
|
||||
)
|
||||
parsed = parser.parse_args(argv)
|
||||
words = pip_install_args(
|
||||
local=parsed.local,
|
||||
no_env=parsed.no_env,
|
||||
break_system_packages=parsed.break_system_packages,
|
||||
)
|
||||
sys.stdout.write("\n".join(words) + "\n")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,17 @@
|
||||
"""Repo-root launcher for the sanity runner.
|
||||
|
||||
Generated by AI/LLM: 2026-09-29 - implementation lives in fiona.run_sanity.
|
||||
"""
|
||||
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
_SRC = pathlib.Path(__file__).resolve().parent.parent / "src"
|
||||
if str(_SRC) not in sys.path:
|
||||
sys.path.insert(0, str(_SRC))
|
||||
|
||||
import fiona.run_sanity as runner # noqa: E402
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(runner.main())
|
||||
@@ -0,0 +1,17 @@
|
||||
"""Repo-root launcher for the test runner.
|
||||
|
||||
Generated by AI/LLM: 2026-09-29 - implementation lives in fiona.run_tests.
|
||||
"""
|
||||
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
_SRC = pathlib.Path(__file__).resolve().parent.parent / "src"
|
||||
if str(_SRC) not in sys.path:
|
||||
sys.path.insert(0, str(_SRC))
|
||||
|
||||
import fiona.run_tests as runner # noqa: E402
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(runner.main())
|
||||
+54
-17
@@ -7,7 +7,7 @@ A collection of tools for managing the LLM wiki and related operations.
|
||||
The Fiona Assistant project provides command-line tools for:
|
||||
|
||||
- **Librarian**: Managing LLM wiki content and operations
|
||||
- **MDoctor**: [Description to be added]
|
||||
- **MDoctor**: Markdown diagnosis (lint, list, and fix)
|
||||
|
||||
## Installation
|
||||
|
||||
@@ -20,7 +20,9 @@ The Fiona Assistant project provides command-line tools for:
|
||||
cd fiona-assistant
|
||||
```
|
||||
|
||||
2. Set up the Python environment using the bootstrap script:
|
||||
2. Set up the Python environment using the bootstrap script.
|
||||
|
||||
Linux / macOS:
|
||||
|
||||
```bash
|
||||
source bootstrap --env fiona_env
|
||||
@@ -32,6 +34,21 @@ The Fiona Assistant project provides command-line tools for:
|
||||
source bootstrap --env fiona_env --python 3.11.0
|
||||
```
|
||||
|
||||
Windows (PowerShell):
|
||||
|
||||
```powershell
|
||||
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
|
||||
. .\bootstrap.ps1 --env fiona_env --python 3.11.0
|
||||
```
|
||||
|
||||
Install this checkout as a normal package with `--local`. Skip the
|
||||
virtual environment with `--no-env` (PowerShell may use `-File`
|
||||
instead of dot-sourcing):
|
||||
|
||||
```powershell
|
||||
powershell -ExecutionPolicy Bypass -File .\bootstrap.ps1 --local --no-env
|
||||
```
|
||||
|
||||
3. Install the package in development mode:
|
||||
|
||||
```bash
|
||||
@@ -48,7 +65,7 @@ pip install -e ".[dev]"
|
||||
|
||||
### Librarian
|
||||
|
||||
See [librarian/README.md](./librarian/README.md) for detailed usage.
|
||||
See [librarian/README.md](./fiona/librarian/README.md) for detailed usage.
|
||||
|
||||
```bash
|
||||
librarian --help
|
||||
@@ -56,7 +73,7 @@ librarian --help
|
||||
|
||||
### MDoctor
|
||||
|
||||
See [mdoctor/README.md](./mdoctor/README.md) for detailed usage.
|
||||
See [mdoctor/README.md](./fiona/mdoctor/README.md) for detailed usage.
|
||||
|
||||
```bash
|
||||
mdoctor --help
|
||||
@@ -65,32 +82,50 @@ mdoctor --help
|
||||
## Project Structure
|
||||
|
||||
```text
|
||||
src/
|
||||
├── fiona/ # Main package
|
||||
├── librarian/ # Wiki content management tool
|
||||
├── mdoctor/ # [Tool description]
|
||||
├── tests/ # Unit tests
|
||||
└── README.md # This file
|
||||
src/fiona/
|
||||
├── __main__.py # CLI entry point
|
||||
├── librarian/ # Wiki content management
|
||||
├── mdoctor/ # Markdown diagnosis
|
||||
├── customlinter/ # Frontmatter, markdown, Python, and JSON linters
|
||||
├── python_ruff/ # Ruff wrapper
|
||||
├── gittool/ # Git operations
|
||||
├── sanity_runner/ # Shared sanity engine
|
||||
├── wiki/ # Wiki init, update, sanity, and graph
|
||||
├── data/llm_wiki/ # Packaged wiki template
|
||||
├── run_tests.py # Test runner
|
||||
└── run_sanity.py # Sanity runner
|
||||
tests/ # Unit tests (repository root, not installed)
|
||||
scripts/
|
||||
├── run_tests.py
|
||||
├── run_sanity.py
|
||||
├── settings_dev.json
|
||||
├── utils.sh
|
||||
└── install_packages.sh
|
||||
```
|
||||
|
||||
## Running Tests
|
||||
|
||||
```bash
|
||||
# Run all tests
|
||||
pytest
|
||||
python scripts/run_tests.py
|
||||
|
||||
# Or use pytest directly
|
||||
python -m pytest
|
||||
|
||||
# Run with coverage
|
||||
pytest --cov=src
|
||||
python -m pytest --cov=src/fiona
|
||||
```
|
||||
|
||||
## Development Tools
|
||||
|
||||
The project includes several development utilities:
|
||||
|
||||
- `bootstrap`: Set up the Python development environment
|
||||
- `src/utils.sh`: Shell utility functions
|
||||
- `src/install_packages.sh`: Install local Python packages
|
||||
- `src/run_tests.sh`: Run the test suite
|
||||
- `bootstrap`: Set up the Python development environment (bash)
|
||||
- `bootstrap.ps1`: Set up the Python development environment (PowerShell)
|
||||
- `scripts/utils.sh`: Shell utility functions
|
||||
- `scripts/install_packages.sh`: Install local Python packages
|
||||
- `scripts/run_tests.py`: Run the test suite
|
||||
- `scripts/run_sanity.py`: Run tests and linters
|
||||
|
||||
## License
|
||||
|
||||
@@ -98,4 +133,6 @@ GPL-3.0-only
|
||||
|
||||
## Contributing
|
||||
|
||||
[Contribution guidelines to be added]
|
||||
Follow [AGENTS.md](../AGENTS.md): update [SPEC.md](../SPEC.md) first,
|
||||
add a failing test, then implement. Run `python scripts/run_tests.py`
|
||||
and `python scripts/run_sanity.py --non-interactive` before finishing.
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
# Fiona tools package
|
||||
+105
-13
@@ -8,18 +8,21 @@ import argparse
|
||||
import logging
|
||||
import sys
|
||||
|
||||
from customlinter.fmlint import main as fmlint_main
|
||||
from customlinter.jsonlint import main as jsonlint_main
|
||||
from fiona.customlinter.fmlint import main as fmlint_main
|
||||
from fiona.customlinter.jsonlint import main as jsonlint_main
|
||||
from fiona.gittool import diff_report
|
||||
from fiona.gittool.git_tool import main as git_tool_main
|
||||
from fiona.librarian.librarian import main as librarian_main
|
||||
from fiona.librarian.toc_generator import main as toc_generator_main
|
||||
from fiona.logging_config import setup_logging
|
||||
from gittool.git_tool import main as git_tool_main
|
||||
from librarian.librarian import main as librarian_main
|
||||
from librarian.toc_generator import main as toc_generator_main
|
||||
from mdoctor.mdoctor import main as mdoctor_main
|
||||
from python_ruff.python_ruff import main as python_ruff_main
|
||||
from wiki.graph_generator import wiki_graph_main
|
||||
from wiki.sanity_manager import sanity_main
|
||||
from wiki.update_manager import check_update_main, update_main
|
||||
from wiki.wiki import wiki_init
|
||||
from fiona.mdoctor.mdoctor import main as mdoctor_main
|
||||
from fiona.python_ruff.python_ruff import main as python_ruff_main
|
||||
from fiona.wiki.graph_generator import wiki_graph_main
|
||||
from fiona.wiki.housekeeping import housekeeping_main
|
||||
from fiona.wiki.ingest.pipeline import add_ingest_arguments, execute_ingest
|
||||
from fiona.wiki.sanity_manager import sanity_main
|
||||
from fiona.wiki.update_manager import check_update_main, update_main
|
||||
from fiona.wiki.wiki import wiki_init
|
||||
|
||||
|
||||
def main():
|
||||
@@ -128,6 +131,39 @@ def main():
|
||||
# fiona-assistant git status
|
||||
git_subparsers.add_parser("status", help="Show git status")
|
||||
|
||||
# fiona-assistant git diff
|
||||
git_diff_parser = git_subparsers.add_parser(
|
||||
"diff",
|
||||
help="Write constellation-themed HTML diff under wiki tmp/",
|
||||
)
|
||||
git_diff_parser.add_argument(
|
||||
"--base",
|
||||
help="Base revision (default: origin/main or main/master)",
|
||||
)
|
||||
git_diff_parser.add_argument("--head", default="HEAD", help="Head revision")
|
||||
git_diff_parser.add_argument("--title", help="Report H1 / page title")
|
||||
git_diff_parser.add_argument("--subtitle", help="Extra header line")
|
||||
git_diff_parser.add_argument(
|
||||
"-o",
|
||||
"--output",
|
||||
help="Output HTML path",
|
||||
)
|
||||
git_diff_parser.add_argument(
|
||||
"--collapsed",
|
||||
action="store_true",
|
||||
help="Start with per-file sections collapsed",
|
||||
)
|
||||
git_diff_parser.add_argument(
|
||||
"--staged",
|
||||
action="store_true",
|
||||
help="Diff only staged changes (ignores unstaged changes)",
|
||||
)
|
||||
git_diff_parser.add_argument(
|
||||
"--unstaged",
|
||||
action="store_true",
|
||||
help="Diff only unstaged changes (ignores staged changes)",
|
||||
)
|
||||
|
||||
# fiona-assistant wiki
|
||||
wiki_parser = subparsers.add_parser("wiki", help="Manage wiki directories")
|
||||
wiki_subparsers = wiki_parser.add_subparsers(dest="wiki_subcommand", help="Wiki subcommands")
|
||||
@@ -279,6 +315,31 @@ def main():
|
||||
help="Disable tracking of external links"
|
||||
)
|
||||
|
||||
# AI/LLM: 2026-09-30 - flags live here so wiki ingest --help is not rejected.
|
||||
wiki_ingest_parser = wiki_subparsers.add_parser(
|
||||
"ingest",
|
||||
prog="fiona-assistant wiki ingest",
|
||||
help="Archive one source and write a raw wiki page",
|
||||
)
|
||||
add_ingest_arguments(wiki_ingest_parser)
|
||||
|
||||
# AI/LLM: fiona-assistant housekeeping (SPEC 4.8)
|
||||
housekeeping_parser = subparsers.add_parser(
|
||||
"housekeeping",
|
||||
help="Run the wiki maintenance routine and report what needs work"
|
||||
)
|
||||
housekeeping_parser.add_argument(
|
||||
"directory",
|
||||
nargs="?",
|
||||
default=".",
|
||||
help="Wiki directory (default: current directory)"
|
||||
)
|
||||
housekeeping_parser.add_argument(
|
||||
"--json",
|
||||
action="store_true",
|
||||
help="Output results in JSON format"
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
setup_logging()
|
||||
logging.getLogger(__name__).debug("fiona-assistant command: %s", args.command)
|
||||
@@ -355,13 +416,40 @@ def main():
|
||||
elif args.git_subcommand == "status":
|
||||
sys.argv = ["git_tool", "status"]
|
||||
git_tool_main()
|
||||
elif args.git_subcommand == "diff":
|
||||
diff_argv: list[str] = []
|
||||
if args.base:
|
||||
diff_argv.extend(["--base", args.base])
|
||||
if args.head != "HEAD":
|
||||
diff_argv.extend(["--head", args.head])
|
||||
if args.title:
|
||||
diff_argv.extend(["--title", args.title])
|
||||
if args.subtitle:
|
||||
diff_argv.extend(["--subtitle", args.subtitle])
|
||||
if args.output is not None:
|
||||
diff_argv.extend(["--output", args.output])
|
||||
if args.collapsed:
|
||||
diff_argv.append("--collapsed")
|
||||
if args.staged:
|
||||
diff_argv.append("--staged")
|
||||
if args.unstaged:
|
||||
diff_argv.append("--unstaged")
|
||||
sys.exit(diff_report.main(diff_argv))
|
||||
else:
|
||||
git_parser.print_help()
|
||||
sys.exit(1)
|
||||
elif args.command == "wiki":
|
||||
if args.wiki_subcommand == "init":
|
||||
result = wiki_init(args.directory)
|
||||
print(result)
|
||||
try:
|
||||
result = wiki_init(args.directory)
|
||||
print(result)
|
||||
except FileExistsError as e:
|
||||
# SPEC 7.1.1.4: format the error message and exit 1
|
||||
# The FileExistsError message is: "Directory '<path>' already contains llm-wiki files: <names>"
|
||||
# We want: "Error: Directory '<path>' already contains llm-wiki files: <names>. Run fiona-assistant wiki update to refresh the template."
|
||||
formatted_error = f"Error: {e}. Run fiona-assistant wiki update to refresh the template."
|
||||
print(formatted_error, file=sys.stderr)
|
||||
sys.exit(1)
|
||||
elif args.wiki_subcommand == "check-update":
|
||||
sys.exit(check_update_main(
|
||||
directory=args.directory,
|
||||
@@ -381,6 +469,8 @@ def main():
|
||||
json_output=args.json,
|
||||
non_interactive=args.non_interactive
|
||||
))
|
||||
elif args.wiki_subcommand == "ingest":
|
||||
sys.exit(execute_ingest(args, wiki_ingest_parser))
|
||||
elif args.wiki_subcommand == "graph":
|
||||
sys.exit(wiki_graph_main(
|
||||
directory=args.directory,
|
||||
@@ -394,6 +484,8 @@ def main():
|
||||
else:
|
||||
wiki_parser.print_help()
|
||||
sys.exit(1)
|
||||
elif args.command == "housekeeping":
|
||||
sys.exit(housekeeping_main(directory=args.directory, json_output=args.json))
|
||||
else:
|
||||
parser.print_help()
|
||||
sys.exit(1)
|
||||
|
||||
@@ -16,7 +16,7 @@ various code and content rules. It includes:
|
||||
All linters share a common interface and output format for consistency.
|
||||
"""
|
||||
|
||||
from customlinter.base import (
|
||||
from fiona.customlinter.base import (
|
||||
DEFAULT_SETTINGS_FILE,
|
||||
READONLY_ZONES,
|
||||
BaseLinter,
|
||||
@@ -25,9 +25,9 @@ from customlinter.base import (
|
||||
LinterError,
|
||||
LintViolation,
|
||||
)
|
||||
from customlinter.fmlint import FrontmatterLinter
|
||||
from customlinter.mdlinter import MarkdownLinter
|
||||
from customlinter.pycustomlint import PythonCustomLinter
|
||||
from fiona.customlinter.fmlint import FrontmatterLinter
|
||||
from fiona.customlinter.mdlinter import MarkdownLinter
|
||||
from fiona.customlinter.pycustomlint import PythonCustomLinter
|
||||
|
||||
__all__ = [
|
||||
"DEFAULT_SETTINGS_FILE",
|
||||
@@ -15,19 +15,15 @@ import pathlib
|
||||
from abc import ABC, abstractmethod
|
||||
from typing import Any, ClassVar
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
from fiona.paths import llm_wiki_template
|
||||
|
||||
# =============================================================================
|
||||
# Constants
|
||||
# =============================================================================
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Read-only zone names that should be skipped
|
||||
READONLY_ZONES = ["personal", "raw"]
|
||||
|
||||
# Default settings file path
|
||||
DEFAULT_SETTINGS_FILE = (
|
||||
pathlib.Path(__file__).resolve().parent.parent / "llm-wiki" / "fiona-settings.json"
|
||||
)
|
||||
DEFAULT_SETTINGS_FILE = llm_wiki_template() / "fiona-settings.json"
|
||||
|
||||
# Re-export constants for backward compatibility
|
||||
__all__ = [
|
||||
@@ -41,6 +37,20 @@ __all__ = [
|
||||
]
|
||||
|
||||
|
||||
def report_path(path: pathlib.Path | str) -> str:
|
||||
"""Return a forward-slash path for reports.
|
||||
|
||||
Generated by AI/LLM: 2026-09-29 - str(Path) uses backslashes on Windows,
|
||||
which makes report text differ from the POSIX form the checks expect.
|
||||
|
||||
Args:
|
||||
path: Path to display.
|
||||
|
||||
Returns:
|
||||
The path with forward slashes.
|
||||
"""
|
||||
return pathlib.Path(path).as_posix()
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Exceptions
|
||||
@@ -56,7 +66,7 @@ class LinterError(Exception):
|
||||
line: int | None = None,
|
||||
):
|
||||
"""Initialize the exception with details.
|
||||
|
||||
|
||||
Args:
|
||||
message: The error message describing the linting failure.
|
||||
filepath: The path to the file that failed linting.
|
||||
@@ -71,14 +81,14 @@ class LinterError(Exception):
|
||||
# Backward compatibility aliases
|
||||
class CustomLintError(LinterError):
|
||||
"""Exception raised when custom Python linting fails.
|
||||
|
||||
|
||||
This is an alias for LinterError for backward compatibility.
|
||||
"""
|
||||
|
||||
|
||||
class FrontmatterLintError(LinterError):
|
||||
"""Exception raised when frontmatter validation fails.
|
||||
|
||||
|
||||
This is an alias for LinterError for backward compatibility.
|
||||
"""
|
||||
|
||||
@@ -89,7 +99,7 @@ class FrontmatterLintError(LinterError):
|
||||
|
||||
class LintViolation:
|
||||
"""Represents a single linting violation.
|
||||
|
||||
|
||||
This class provides a standardized format for representing violations
|
||||
across all linter types.
|
||||
"""
|
||||
@@ -105,7 +115,7 @@ class LintViolation:
|
||||
severity: str = "error",
|
||||
):
|
||||
"""Initialize a violation.
|
||||
|
||||
|
||||
Args:
|
||||
filepath: Path to the file containing the violation.
|
||||
line: Line number of the violation.
|
||||
@@ -126,7 +136,7 @@ class LintViolation:
|
||||
def to_dict(self) -> dict:
|
||||
"""Convert violation to dictionary for reporting."""
|
||||
return {
|
||||
"file": str(self.filepath),
|
||||
"file": report_path(self.filepath),
|
||||
"line": self.line,
|
||||
"column": self.column,
|
||||
"code": self.code,
|
||||
@@ -137,7 +147,10 @@ class LintViolation:
|
||||
|
||||
def __str__(self) -> str:
|
||||
"""Format violation for display."""
|
||||
return f"{self.filepath}:{self.line}:{self.column} {self.code} {self.message}"
|
||||
return (
|
||||
f"{report_path(self.filepath)}:{self.line}:{self.column} "
|
||||
f"{self.code} {self.message}"
|
||||
)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
@@ -146,14 +159,14 @@ class LintViolation:
|
||||
|
||||
class BaseLinter(ABC):
|
||||
"""Abstract base class for all custom linters.
|
||||
|
||||
|
||||
This class provides common functionality for all linters including:
|
||||
- Settings loading
|
||||
- File collection
|
||||
- Pattern matching
|
||||
- Read-only zone and .gitignore support
|
||||
- Consistent violation formatting
|
||||
|
||||
|
||||
Subclasses must implement:
|
||||
- lint_file() method
|
||||
- VIOLATION_CODES class attribute
|
||||
@@ -174,7 +187,7 @@ class BaseLinter(ABC):
|
||||
|
||||
def __init__(self, settings_file: pathlib.Path | None = None):
|
||||
"""Initialize the linter.
|
||||
|
||||
|
||||
Args:
|
||||
settings_file: Path to the settings file, or None for defaults.
|
||||
"""
|
||||
@@ -185,7 +198,7 @@ class BaseLinter(ABC):
|
||||
"""Load settings from the settings file."""
|
||||
if self.settings_file is None:
|
||||
self.settings_file = DEFAULT_SETTINGS_FILE
|
||||
|
||||
|
||||
try:
|
||||
with open(self.settings_file, encoding="utf-8") as f:
|
||||
settings = json.load(f)
|
||||
@@ -202,54 +215,54 @@ class BaseLinter(ABC):
|
||||
@abstractmethod
|
||||
def lint_file(self, filepath: pathlib.Path | str) -> list[LintViolation]:
|
||||
"""Lint a single file for violations.
|
||||
|
||||
|
||||
Args:
|
||||
filepath: Path to the file to lint.
|
||||
|
||||
|
||||
Returns:
|
||||
List of LintViolation objects.
|
||||
"""
|
||||
|
||||
def lint_directory(self, directory: str | pathlib.Path) -> list[LintViolation]:
|
||||
"""Lint all files in a directory tree.
|
||||
|
||||
|
||||
Args:
|
||||
directory: Path to the directory to lint.
|
||||
|
||||
|
||||
Returns:
|
||||
List of LintViolation objects from all files.
|
||||
"""
|
||||
directory = pathlib.Path(directory)
|
||||
violations: list[LintViolation] = []
|
||||
|
||||
|
||||
if not directory.exists():
|
||||
raise LinterError(f"Directory does not exist: {directory}")
|
||||
|
||||
|
||||
# Collect files
|
||||
files = self._collect_files(directory)
|
||||
|
||||
|
||||
# Lint each file
|
||||
for file in files:
|
||||
file_violations = self.lint_file(file)
|
||||
violations.extend(file_violations)
|
||||
|
||||
|
||||
return violations
|
||||
|
||||
def _collect_files(self, directory: pathlib.Path) -> list[pathlib.Path]:
|
||||
"""Collect all files to lint in a directory tree.
|
||||
|
||||
|
||||
Args:
|
||||
directory: Path to the directory to search.
|
||||
|
||||
|
||||
Returns:
|
||||
List of Path objects for files to lint.
|
||||
"""
|
||||
files: list[pathlib.Path] = []
|
||||
|
||||
|
||||
# Respect .gitignore
|
||||
gitignore_path = directory / ".gitignore"
|
||||
ignore_patterns: list[str] = []
|
||||
|
||||
|
||||
if gitignore_path.exists():
|
||||
try:
|
||||
with open(gitignore_path, encoding="utf-8") as f:
|
||||
@@ -259,34 +272,34 @@ class BaseLinter(ABC):
|
||||
ignore_patterns.append(line)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
# Walk the directory tree
|
||||
for root, dirs, filenames in os.walk(directory):
|
||||
root_path = pathlib.Path(root)
|
||||
|
||||
|
||||
# Skip read-only zones
|
||||
if any(zone in root_path.parts for zone in READONLY_ZONES):
|
||||
dirs.clear()
|
||||
continue
|
||||
|
||||
|
||||
# Skip based on .gitignore patterns
|
||||
dirs[:] = [d for d in dirs if not self._matches_pattern(d, ignore_patterns)]
|
||||
|
||||
|
||||
for filename in filenames:
|
||||
if any(filename.endswith(ext) for ext in self.file_extensions) and not self._matches_pattern(
|
||||
filename, ignore_patterns
|
||||
):
|
||||
files.append(root_path / filename)
|
||||
|
||||
|
||||
return files
|
||||
|
||||
def _matches_pattern(self, name: str, patterns: list[str]) -> bool:
|
||||
"""Check if a name matches any of the ignore patterns.
|
||||
|
||||
|
||||
Args:
|
||||
name: The name to check.
|
||||
patterns: List of patterns to match against.
|
||||
|
||||
|
||||
Returns:
|
||||
True if the name matches any pattern.
|
||||
"""
|
||||
@@ -300,21 +313,21 @@ class BaseLinter(ABC):
|
||||
|
||||
def _should_skip(self, filepath: pathlib.Path | str) -> bool:
|
||||
"""Check if a file should be skipped based on settings.
|
||||
|
||||
|
||||
Args:
|
||||
filepath: Path to the file (as Path or string).
|
||||
|
||||
|
||||
Returns:
|
||||
True if the file should be skipped.
|
||||
"""
|
||||
# Convert to Path if it's a string
|
||||
if isinstance(filepath, str):
|
||||
filepath = pathlib.Path(filepath)
|
||||
|
||||
|
||||
# Check read-only zones
|
||||
if any(zone in filepath.parts for zone in READONLY_ZONES):
|
||||
return True
|
||||
|
||||
|
||||
# Check ignore patterns from settings
|
||||
ignore_patterns = self._settings.get("ignore_patterns", [])
|
||||
if ignore_patterns:
|
||||
@@ -322,16 +335,16 @@ class BaseLinter(ABC):
|
||||
for pattern in ignore_patterns:
|
||||
if fnmatch.fnmatch(file_str, pattern):
|
||||
return True
|
||||
|
||||
|
||||
return False
|
||||
|
||||
def get_violation_message(self, violation_code: str, **kwargs) -> str:
|
||||
"""Get formatted violation message from VIOLATION_CODES dictionary.
|
||||
|
||||
|
||||
Args:
|
||||
violation_code: The violation code (e.g., 'PYC001', 'FML001').
|
||||
**kwargs: Keyword arguments to format the message template.
|
||||
|
||||
|
||||
Returns:
|
||||
Formatted violation message string.
|
||||
"""
|
||||
@@ -347,20 +360,20 @@ class BaseLinter(ABC):
|
||||
|
||||
def format_violations(self, violations: list[LintViolation]) -> str:
|
||||
"""Format violations for display.
|
||||
|
||||
|
||||
Args:
|
||||
violations: List of LintViolation objects.
|
||||
|
||||
|
||||
Returns:
|
||||
Formatted string with all violations.
|
||||
"""
|
||||
if not violations:
|
||||
return ""
|
||||
|
||||
|
||||
lines = []
|
||||
for violation in sorted(violations, key=lambda v: (str(v.filepath), v.line, v.column)):
|
||||
lines.append(str(violation))
|
||||
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
def compress_violations(
|
||||
@@ -397,18 +410,18 @@ class BaseLinter(ABC):
|
||||
"total_violations": 0,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
if target is not None:
|
||||
compressed_violations["meta"]["target"] = str(target_path)
|
||||
compressed_violations["meta"]["target"] = report_path(target_path)
|
||||
|
||||
for violation in violations:
|
||||
if target_path is not None:
|
||||
try:
|
||||
file_key = str(violation.filepath.relative_to(target_path))
|
||||
file_key = report_path(violation.filepath.relative_to(target_path))
|
||||
except ValueError:
|
||||
file_key = str(violation.filepath)
|
||||
file_key = report_path(violation.filepath)
|
||||
else:
|
||||
file_key = str(violation.filepath)
|
||||
file_key = report_path(violation.filepath)
|
||||
|
||||
if file_key not in compressed_violations["files"]:
|
||||
compressed_violations["files"][file_key] = []
|
||||
@@ -479,10 +492,10 @@ class BaseLinter(ABC):
|
||||
|
||||
def format_violations_json(self, violations: list[LintViolation]) -> str:
|
||||
"""Format violations as JSON.
|
||||
|
||||
|
||||
Args:
|
||||
violations: List of LintViolation objects.
|
||||
|
||||
|
||||
Returns:
|
||||
JSON string with all violations.
|
||||
"""
|
||||
@@ -495,12 +508,12 @@ class BaseLinter(ABC):
|
||||
linter_name: str,
|
||||
) -> str:
|
||||
"""Format violations in a comprehensive AI-friendly report.
|
||||
|
||||
|
||||
Args:
|
||||
violations: List of LintViolation objects.
|
||||
target: The target directory or file that was linted.
|
||||
linter_name: Name of the linter (for report header).
|
||||
|
||||
|
||||
Returns:
|
||||
Formatted report string.
|
||||
"""
|
||||
@@ -508,7 +521,7 @@ class BaseLinter(ABC):
|
||||
lines.append(f"{linter_name} Violations")
|
||||
lines.append(f"TARGET: {target}")
|
||||
lines.append("")
|
||||
|
||||
|
||||
# Collect all unique violation codes with their descriptions
|
||||
descriptions: dict[str, dict] = {}
|
||||
for violation in violations:
|
||||
@@ -518,7 +531,7 @@ class BaseLinter(ABC):
|
||||
"rule_name": code_info.get("rule_name", violation.code),
|
||||
"description": code_info.get("description", ""),
|
||||
}
|
||||
|
||||
|
||||
# Add definitions
|
||||
if descriptions:
|
||||
lines.append("- DEFINITIONS:")
|
||||
@@ -527,16 +540,16 @@ class BaseLinter(ABC):
|
||||
f" - {code}: {info['rule_name']}, {info['description']}"
|
||||
)
|
||||
lines.append("")
|
||||
|
||||
|
||||
# Add violations by file
|
||||
if violations:
|
||||
files_violations: dict[str, list[LintViolation]] = {}
|
||||
for violation in violations:
|
||||
file_key = str(violation.filepath)
|
||||
file_key = report_path(violation.filepath)
|
||||
if file_key not in files_violations:
|
||||
files_violations[file_key] = []
|
||||
files_violations[file_key].append(violation)
|
||||
|
||||
|
||||
for file_path, file_viols in sorted(files_violations.items()):
|
||||
lines.append(f"- FILE: {file_path}")
|
||||
for violation in sorted(file_viols, key=lambda v: (v.line, v.column)):
|
||||
@@ -545,11 +558,11 @@ class BaseLinter(ABC):
|
||||
else:
|
||||
lines.append("- No violations found.")
|
||||
lines.append("")
|
||||
|
||||
|
||||
# Add summary
|
||||
lines.append("- SUMMARY:")
|
||||
lines.append(f" - Total files with violations: {len(files_violations) if violations else 0}")
|
||||
lines.append(f" - Total violations: {len(violations)}")
|
||||
lines.append("")
|
||||
|
||||
|
||||
return "\n".join(lines)
|
||||
@@ -26,13 +26,14 @@ import pathlib
|
||||
import re
|
||||
import sys
|
||||
|
||||
from customlinter.base import (
|
||||
from fiona.customlinter.base import (
|
||||
BaseLinter,
|
||||
FrontmatterLintError,
|
||||
LintViolation,
|
||||
report_path,
|
||||
)
|
||||
from fiona.librarian.md_frontmatter import FrontmatterError, parse_frontmatter
|
||||
from fiona.logging_config import setup_logging
|
||||
from librarian.md_frontmatter import FrontmatterError, parse_frontmatter
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -695,7 +696,7 @@ class FrontmatterLinter(BaseLinter):
|
||||
# Group violations by file
|
||||
files_violations: dict[str, list[LintViolation]] = {}
|
||||
for violation in violations:
|
||||
file_key = str(violation.filepath)
|
||||
file_key = report_path(violation.filepath)
|
||||
if file_key not in files_violations:
|
||||
files_violations[file_key] = []
|
||||
files_violations[file_key].append(violation)
|
||||
@@ -9,7 +9,7 @@ Includes support for:
|
||||
- Key ordering (optional)
|
||||
- Auto-fixing certain violations
|
||||
|
||||
Follows the BaseLinter pattern from customlinter.base.
|
||||
Follows the BaseLinter pattern from fiona.customlinter.base.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -21,7 +21,7 @@ import re
|
||||
import sys
|
||||
from typing import Any, ClassVar
|
||||
|
||||
from customlinter.base import BaseLinter, LinterError, LintViolation
|
||||
from fiona.customlinter.base import BaseLinter, LinterError, LintViolation
|
||||
from fiona.logging_config import setup_logging
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -3,7 +3,7 @@
|
||||
This module provides a custom markdown linter for llm-wiki content,
|
||||
built on the customlinter framework (BaseLinter). It runs a registry of
|
||||
per-file checks over a wiki's markdown files and reports violations in
|
||||
the shared AI-friendly format (SPEC 4.9.3).
|
||||
the shared AI-friendly format (SPEC 6.1).
|
||||
|
||||
The first check is the dead-link check (moved from
|
||||
`wiki.sanity_manager.find_dead_links`):
|
||||
@@ -27,7 +27,7 @@ import pathlib
|
||||
import re
|
||||
from typing import Any, ClassVar
|
||||
|
||||
from customlinter.base import BaseLinter, LinterError, LintViolation
|
||||
from fiona.customlinter.base import BaseLinter, LinterError, LintViolation
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -67,6 +67,14 @@ VIOLATION_CODES = {
|
||||
},
|
||||
}
|
||||
|
||||
__all__ = [
|
||||
"FILE_EXTENSIONS",
|
||||
"SETTINGS_KEY_PATH",
|
||||
"VIOLATION_CODES",
|
||||
"MarkdownLinter",
|
||||
"find_dead_links",
|
||||
]
|
||||
|
||||
|
||||
def _is_ignorable_target(target: str) -> bool:
|
||||
"""Check whether a markdown link target is out of scope."""
|
||||
@@ -238,7 +246,7 @@ class MarkdownLinter(BaseLinter):
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Module-level Dead Link Helper (moved from wiki.sanity_manager)
|
||||
# Module-level Dead Link Helper (moved from fiona.wiki.sanity_manager)
|
||||
# =============================================================================
|
||||
|
||||
def find_dead_links(wiki_root: pathlib.Path) -> list[LintViolation]:
|
||||
@@ -262,16 +270,3 @@ def find_dead_links(wiki_root: pathlib.Path) -> list[LintViolation]:
|
||||
content = path.read_text(encoding="utf-8", errors="replace")
|
||||
violations.extend(linter._check_dead_links(path, content, md_stems))
|
||||
return violations
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Public API
|
||||
# =============================================================================
|
||||
|
||||
__all__ = [
|
||||
"FILE_EXTENSIONS",
|
||||
"SETTINGS_KEY_PATH",
|
||||
"VIOLATION_CODES",
|
||||
"MarkdownLinter",
|
||||
"find_dead_links",
|
||||
]
|
||||
@@ -11,6 +11,7 @@ The module provides functionality to:
|
||||
|
||||
Currently implemented rules:
|
||||
- Inline imports (imports inside functions/blocks instead of module level)
|
||||
- Late declarations (module-level assignments after the first function or class)
|
||||
|
||||
The framework is designed to be modular, allowing additional rules to be
|
||||
added in the future.
|
||||
@@ -25,7 +26,7 @@ import pathlib
|
||||
import sys
|
||||
from typing import ClassVar
|
||||
|
||||
from customlinter.base import (
|
||||
from fiona.customlinter.base import (
|
||||
BaseLinter,
|
||||
CustomLintError,
|
||||
LintViolation,
|
||||
@@ -46,7 +47,10 @@ READONLY_ZONES = ["personal", "raw"]
|
||||
VIOLATION_CODES = {
|
||||
"PYC001": {
|
||||
"rule_name": "Inline Import",
|
||||
"description": "Import statement found inside function or block (inline import)",
|
||||
"description": (
|
||||
"Import statement found inside a function or block (inline import). "
|
||||
"A module-level if/else before the first function or class is allowed"
|
||||
),
|
||||
"message_template": "PYC001 - Inline import at line {line}: {import_stmt}",
|
||||
"severity": "error"
|
||||
},
|
||||
@@ -56,6 +60,17 @@ VIOLATION_CODES = {
|
||||
"message_template": "PYC002 - Import after code at line {line}: {import_stmt}",
|
||||
"severity": "warning"
|
||||
},
|
||||
"PYC003": {
|
||||
"rule_name": "Late Declaration",
|
||||
"description": (
|
||||
"Module-level assignment found after the first function or class. "
|
||||
"Put declarations at the top of the file, before the first function or class"
|
||||
),
|
||||
"message_template": (
|
||||
"PYC003 - Declaration after first function or class at line {line}: {stmt}"
|
||||
),
|
||||
"severity": "error",
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
@@ -64,26 +79,78 @@ VIOLATION_CODES = {
|
||||
# =============================================================================
|
||||
|
||||
class InlineImportVisitor(ast.NodeVisitor):
|
||||
"""AST visitor to detect inline (indented) imports."""
|
||||
"""AST visitor to detect inline (indented) imports.
|
||||
|
||||
Modified by AI/LLM: 2026-09-29 - allow imports in a module-level
|
||||
if/else that appears before the first function or class (SPEC 6.4.2).
|
||||
"""
|
||||
|
||||
def __init__(self, filename: str):
|
||||
"""Initialize the visitor.
|
||||
|
||||
|
||||
Args:
|
||||
filename: The name of the file being analyzed.
|
||||
"""
|
||||
self.filename = filename
|
||||
self.violations: list[LintViolation] = []
|
||||
self._os_guard_depth = 0
|
||||
self._guard_if_ids: set[int] = set()
|
||||
|
||||
def visit_Module(self, node: ast.Module) -> None:
|
||||
"""Mark module-level if statements that sit before any function or class."""
|
||||
first_boundary = None
|
||||
for statement in node.body:
|
||||
if isinstance(statement, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
|
||||
first_boundary = statement.lineno
|
||||
break
|
||||
for statement in node.body:
|
||||
if first_boundary is not None and statement.lineno >= first_boundary:
|
||||
break
|
||||
if isinstance(statement, ast.If):
|
||||
self._guard_if_ids.add(id(statement))
|
||||
self.generic_visit(node)
|
||||
|
||||
def visit_If(self, node: ast.If) -> None:
|
||||
"""Keep the OS-guard depth raised while visiting an allowed if."""
|
||||
guarded = id(node) in self._guard_if_ids
|
||||
if guarded:
|
||||
self._os_guard_depth += 1
|
||||
self.generic_visit(node)
|
||||
if guarded:
|
||||
self._os_guard_depth -= 1
|
||||
|
||||
def visit_FunctionDef(self, node: ast.FunctionDef) -> None:
|
||||
"""Imports inside a function stay inline, even under an OS if."""
|
||||
self._visit_blocked_scope(node)
|
||||
|
||||
def visit_AsyncFunctionDef(self, node: ast.AsyncFunctionDef) -> None:
|
||||
"""Imports inside an async function stay inline."""
|
||||
self._visit_blocked_scope(node)
|
||||
|
||||
def visit_ClassDef(self, node: ast.ClassDef) -> None:
|
||||
"""Imports inside a class stay inline."""
|
||||
self._visit_blocked_scope(node)
|
||||
|
||||
def _visit_blocked_scope(self, node: ast.AST) -> None:
|
||||
"""Visit a function or class with the OS-guard exception off.
|
||||
|
||||
Modified by AI/LLM: 2026-09-29 - the SPEC 6.4.2 exception does
|
||||
not cover imports nested in a function or class.
|
||||
"""
|
||||
saved = self._os_guard_depth
|
||||
self._os_guard_depth = 0
|
||||
self.generic_visit(node)
|
||||
self._os_guard_depth = saved
|
||||
|
||||
def visit_Import(self, node: ast.Import) -> None:
|
||||
"""Visit import statement - check if indented."""
|
||||
if node.col_offset > 0: # Indented = inline
|
||||
if node.col_offset > 0 and self._os_guard_depth == 0:
|
||||
self._add_inline_import_violation(node)
|
||||
self.generic_visit(node)
|
||||
|
||||
def visit_ImportFrom(self, node: ast.ImportFrom) -> None:
|
||||
"""Visit import from statement - check if indented."""
|
||||
if node.col_offset > 0: # Indented = inline
|
||||
if node.col_offset > 0 and self._os_guard_depth == 0:
|
||||
self._add_inline_import_violation(node)
|
||||
self.generic_visit(node)
|
||||
|
||||
@@ -99,7 +166,7 @@ class InlineImportVisitor(ast.NodeVisitor):
|
||||
import_stmt = line.strip()
|
||||
except (OSError, IndexError, ValueError):
|
||||
pass
|
||||
|
||||
|
||||
violation = LintViolation(
|
||||
filepath=pathlib.Path(self.filename),
|
||||
line=node.lineno,
|
||||
@@ -115,16 +182,110 @@ class InlineImportVisitor(ast.NodeVisitor):
|
||||
return self.violations
|
||||
|
||||
|
||||
class LateDeclarationVisitor(ast.NodeVisitor):
|
||||
"""AST visitor for module-level assignments after the first function or class.
|
||||
|
||||
Modified by AI/LLM: 2026-09-30 - PYC003 late declarations (SPEC 6.4.3).
|
||||
"""
|
||||
|
||||
def __init__(self, filename: str) -> None:
|
||||
"""Initialize the visitor.
|
||||
|
||||
Args:
|
||||
filename: The name of the file being analyzed.
|
||||
"""
|
||||
self.filename = filename
|
||||
self.violations: list[LintViolation] = []
|
||||
self._boundary: int | None = None
|
||||
self._scope_depth = 0
|
||||
|
||||
def visit_Module(self, node: ast.Module) -> None:
|
||||
"""Record the first top-level function or class, then walk the module."""
|
||||
for statement in node.body:
|
||||
if isinstance(statement, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
|
||||
self._boundary = statement.lineno
|
||||
break
|
||||
self.generic_visit(node)
|
||||
|
||||
def visit_FunctionDef(self, node: ast.FunctionDef) -> None:
|
||||
"""Skip assignments that belong to a function."""
|
||||
self._visit_nested_scope(node)
|
||||
|
||||
def visit_AsyncFunctionDef(self, node: ast.AsyncFunctionDef) -> None:
|
||||
"""Skip assignments that belong to an async function."""
|
||||
self._visit_nested_scope(node)
|
||||
|
||||
def visit_ClassDef(self, node: ast.ClassDef) -> None:
|
||||
"""Skip assignments that belong to a class."""
|
||||
self._visit_nested_scope(node)
|
||||
|
||||
def _visit_nested_scope(self, node: ast.AST) -> None:
|
||||
"""Visit a function or class without treating its assignments as module-level.
|
||||
|
||||
Modified by AI/LLM: 2026-09-30 - class and function bodies are not
|
||||
declarations in the middle of the module (SPEC 6.4.3).
|
||||
"""
|
||||
self._scope_depth += 1
|
||||
self.generic_visit(node)
|
||||
self._scope_depth -= 1
|
||||
|
||||
def visit_Assign(self, node: ast.Assign) -> None:
|
||||
"""Flag a module-level assignment that follows the first function or class."""
|
||||
self._check_declaration(node)
|
||||
self.generic_visit(node)
|
||||
|
||||
def visit_AnnAssign(self, node: ast.AnnAssign) -> None:
|
||||
"""Flag a module-level annotated assignment after the first function or class."""
|
||||
self._check_declaration(node)
|
||||
self.generic_visit(node)
|
||||
|
||||
def _check_declaration(self, node: ast.Assign | ast.AnnAssign) -> None:
|
||||
"""Record PYC003 when this assignment is a late module declaration."""
|
||||
if self._scope_depth > 0 or self._boundary is None:
|
||||
return
|
||||
if node.lineno < self._boundary:
|
||||
return
|
||||
statement = self._source_line(node.lineno)
|
||||
self.violations.append(
|
||||
LintViolation(
|
||||
filepath=pathlib.Path(self.filename),
|
||||
line=node.lineno,
|
||||
column=node.col_offset,
|
||||
code="PYC003",
|
||||
message=(
|
||||
"PYC003 - Declaration after first function or class "
|
||||
f"at line {node.lineno}: {statement}"
|
||||
),
|
||||
rule_id="late-declaration",
|
||||
)
|
||||
)
|
||||
|
||||
def _source_line(self, lineno: int) -> str:
|
||||
"""Return the stripped source line for a violation message."""
|
||||
try:
|
||||
with open(self.filename, encoding="utf-8") as handle:
|
||||
lines = handle.readlines()
|
||||
if lineno <= len(lines):
|
||||
return lines[lineno - 1].strip()
|
||||
except (OSError, IndexError, ValueError):
|
||||
return "assignment"
|
||||
return "assignment"
|
||||
|
||||
def get_violations(self) -> list[LintViolation]:
|
||||
"""Get all detected violations."""
|
||||
return self.violations
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Python Custom Linter Class
|
||||
# =============================================================================
|
||||
|
||||
class PythonCustomLinter(BaseLinter):
|
||||
"""Main linter class for custom Python code rules.
|
||||
|
||||
|
||||
This class extends BaseLinter to provide Python-specific linting.
|
||||
"""
|
||||
|
||||
|
||||
VIOLATION_CODES: ClassVar[dict] = VIOLATION_CODES
|
||||
file_extensions: ClassVar[list[str]] = [".py"]
|
||||
settings_key_path: ClassVar[list[str]] = ["python", "custom_lint"]
|
||||
@@ -132,69 +293,72 @@ class PythonCustomLinter(BaseLinter):
|
||||
|
||||
def lint_file(self, filepath: pathlib.Path | str) -> list[LintViolation]:
|
||||
"""Lint a single Python file for custom rule violations.
|
||||
|
||||
|
||||
Args:
|
||||
filepath: Path to the Python file to lint (as Path or string).
|
||||
|
||||
|
||||
Returns:
|
||||
List of LintViolation objects.
|
||||
"""
|
||||
violations: list[LintViolation] = []
|
||||
|
||||
|
||||
try:
|
||||
# Convert to Path if it's a string
|
||||
if isinstance(filepath, str):
|
||||
filepath = pathlib.Path(filepath)
|
||||
|
||||
|
||||
# Check if file should be skipped
|
||||
if self._should_skip(filepath):
|
||||
return violations
|
||||
|
||||
|
||||
# Read and parse the file
|
||||
source = filepath.read_text(encoding="utf-8")
|
||||
tree = ast.parse(source, filename=str(filepath))
|
||||
|
||||
# Visit the AST to find inline imports
|
||||
|
||||
# Visit the AST to find inline imports and late declarations
|
||||
visitor = InlineImportVisitor(str(filepath))
|
||||
visitor.visit(tree)
|
||||
violations = visitor.get_violations()
|
||||
|
||||
declarations = LateDeclarationVisitor(str(filepath))
|
||||
declarations.visit(tree)
|
||||
violations.extend(declarations.get_violations())
|
||||
|
||||
except SyntaxError as e:
|
||||
# Syntax errors are not our concern
|
||||
logger.warning("Syntax error in %s: %s", filepath, e)
|
||||
except OSError as e:
|
||||
logger.warning("Could not read %s: %s", filepath, e)
|
||||
|
||||
|
||||
return violations
|
||||
|
||||
def lint_directory(self, directory: str | pathlib.Path) -> list[LintViolation]:
|
||||
"""Lint all files in a directory tree.
|
||||
|
||||
|
||||
This overrides the base class to raise CustomLintError instead of LinterError.
|
||||
|
||||
|
||||
Args:
|
||||
directory: Path to the directory to lint.
|
||||
|
||||
|
||||
Returns:
|
||||
List of LintViolation objects from all files.
|
||||
|
||||
|
||||
Raises:
|
||||
CustomLintError: If the directory does not exist.
|
||||
"""
|
||||
directory = pathlib.Path(directory)
|
||||
violations: list[LintViolation] = []
|
||||
|
||||
|
||||
if not directory.exists():
|
||||
raise CustomLintError(f"Directory does not exist: {directory}")
|
||||
|
||||
|
||||
# Collect files
|
||||
files = self._collect_files(directory)
|
||||
|
||||
|
||||
# Lint each file
|
||||
for file in files:
|
||||
file_violations = self.lint_file(file)
|
||||
violations.extend(file_violations)
|
||||
|
||||
|
||||
return violations
|
||||
|
||||
def _collect_python_files(self, directory: pathlib.Path) -> list[pathlib.Path]:
|
||||
@@ -208,15 +372,15 @@ class PythonCustomLinter(BaseLinter):
|
||||
|
||||
def main():
|
||||
"""Command-line interface for the Python custom linter."""
|
||||
|
||||
|
||||
setup_logging()
|
||||
logger.debug("Python custom linter starting")
|
||||
|
||||
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Python Custom Linter - Check for custom Python code rule violations."
|
||||
)
|
||||
subparsers = parser.add_subparsers(dest="subcommand", help="Available subcommands")
|
||||
|
||||
|
||||
# chklint subcommand
|
||||
chklint_parser = subparsers.add_parser(
|
||||
"chklint",
|
||||
@@ -233,35 +397,35 @@ def main():
|
||||
action="store_true",
|
||||
help="Output results in JSON format"
|
||||
)
|
||||
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
|
||||
if args.subcommand is None:
|
||||
# Default to chklint with current directory
|
||||
args.subcommand = "chklint"
|
||||
args.directory = "."
|
||||
args.json = False # Default value when not specified
|
||||
|
||||
|
||||
if args.subcommand == "chklint":
|
||||
linter = PythonCustomLinter()
|
||||
|
||||
|
||||
target = pathlib.Path(args.directory)
|
||||
|
||||
|
||||
if not target.exists():
|
||||
print(f"Error: '{args.directory}' does not exist.", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
# Support both files and directories
|
||||
if target.is_file():
|
||||
violations = linter.lint_file(target)
|
||||
else:
|
||||
violations = linter.lint_directory(target)
|
||||
|
||||
|
||||
if getattr(args, 'json', False):
|
||||
output = linter.format_violations_json(violations)
|
||||
else:
|
||||
output = linter.format_violations(violations)
|
||||
|
||||
|
||||
if output:
|
||||
print(output)
|
||||
sys.exit(1)
|
||||
@@ -0,0 +1,63 @@
|
||||
# Byte-compiled / optimized / DLL files
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
|
||||
# Virtual environments
|
||||
.venv/
|
||||
.venvs/
|
||||
venv/
|
||||
env/
|
||||
ENV/
|
||||
|
||||
# IDE-specific files
|
||||
.vscode/
|
||||
.idea/
|
||||
*.swp
|
||||
*.swo
|
||||
|
||||
# Local configuration
|
||||
.env
|
||||
*.local
|
||||
|
||||
# Build artifacts
|
||||
build/
|
||||
dist/
|
||||
*.egg-info/
|
||||
|
||||
# Logs and databases
|
||||
*.log
|
||||
*.sqlite
|
||||
*.db
|
||||
|
||||
# pytest
|
||||
.pytest_cache/
|
||||
|
||||
# Coverage
|
||||
.coverage
|
||||
coverage.xml
|
||||
htmlcov/
|
||||
|
||||
# Jupyter Notebook checkpoints
|
||||
.ipynb_checkpoints/
|
||||
|
||||
# macOS
|
||||
.DS_Store
|
||||
|
||||
# Windows
|
||||
Thumbs.db
|
||||
|
||||
# Obsidian per-user vault state
|
||||
.obsidian/workspace*.json
|
||||
.obsidian/cache
|
||||
|
||||
# Python
|
||||
*.pyc
|
||||
*.pyo
|
||||
*.pyd
|
||||
|
||||
*fiona.log
|
||||
*wiki_graph.html
|
||||
|
||||
# Generated review artifacts (git diff HTML, scratch exports)
|
||||
tmp/
|
||||
@@ -0,0 +1,72 @@
|
||||
---
|
||||
name: llm-wiki-agent-rules
|
||||
title: LLM Wiki Agent Rules
|
||||
version: 1.4.0
|
||||
created: 2025-09-23
|
||||
updated: 2026-10-14
|
||||
type: summary
|
||||
---
|
||||
|
||||
# LLM Wiki Agent Rules
|
||||
|
||||
This directory is an llm-wiki: a persistent knowledge base of
|
||||
interlinked markdown. The user brings sources and questions. You
|
||||
compile each source once into wiki pages, keep the cross-references
|
||||
current, and answer from those pages. The pattern is
|
||||
[llm-wiki-mod](./utils/skills/fiona/llm-wiki-mod.md). The wiki's
|
||||
domain, tags, and directory tree are in [schema](./wiki/schema.md).
|
||||
|
||||
This file states only what must be true before any skill is loaded.
|
||||
Commands, lint codes, and the edit workflow live in the Fiona skill.
|
||||
|
||||
- [1. Load the Fiona skill](#1-load-the-fiona-skill)
|
||||
- [2. What an llm-wiki is](#2-what-an-llm-wiki-is)
|
||||
- [3. Boundaries](#3-boundaries)
|
||||
- [4. History](#4-history)
|
||||
|
||||
## 1. Load the Fiona skill
|
||||
|
||||
If the Fiona skill is not already part of the system prompt, read
|
||||
[Fiona skill](./utils/skills/fiona/SKILL.md) and follow it for the
|
||||
rest of the session. It loads llm-wiki-mod.
|
||||
|
||||
When that skill is already loaded, do not read it again. This file
|
||||
does not repeat it.
|
||||
|
||||
Then read [MEMORY](./MEMORY.md): what this wiki's owner wants, kept
|
||||
across sessions.
|
||||
|
||||
Update `MEMORY.md` whenever an important memory comes up: a lasting
|
||||
preference, a correction, a repeated request, or a fact a run
|
||||
verified. Also update it whenever the user explicitly asks you to
|
||||
remember something. Section 1 of that file says how.
|
||||
|
||||
## 2. What an llm-wiki is
|
||||
|
||||
A directory is an llm-wiki when it contains `fiona-settings.json`.
|
||||
|
||||
Page types, tags, and the directory tree are defined in
|
||||
[schema](./wiki/schema.md). Do not invent a second map of the tree.
|
||||
|
||||
Install the tool with `python -m pip install fiona-assistant` when
|
||||
the `fiona-assistant` command is missing. How to use it is the
|
||||
Fiona skill, not this file.
|
||||
|
||||
## 3. Boundaries
|
||||
|
||||
These hold even before the skill is loaded:
|
||||
|
||||
- `personal/` is read-only. Never create, edit, or delete there.
|
||||
- `raw/` is the archive. Add a new source there. Never edit, move,
|
||||
or delete a file already there.
|
||||
- Synthesized pages belong under `wiki/`.
|
||||
- `ingest/` is only the drop zone for files waiting to be parsed.
|
||||
|
||||
## 4. History
|
||||
|
||||
Git is the wiki's history. After each ingest cycle and each
|
||||
housekeeping pass (lint fixes, index or TOC updates, page moves),
|
||||
commit with `fiona-assistant git save --description TEXT`.
|
||||
|
||||
`git clean`, `git undo`, and `git restore` discard work. Run them
|
||||
only when the user asks.
|
||||
@@ -0,0 +1,179 @@
|
||||
---
|
||||
name: memory
|
||||
version: 1.0.1
|
||||
title: Agent Memory
|
||||
created: DATE_PLACEHOLDER
|
||||
updated: DATE_PLACEHOLDER
|
||||
type: summary
|
||||
---
|
||||
|
||||
# Agent Memory
|
||||
|
||||
<!-- toc-start: headings 2..6 -->
|
||||
|
||||
- [1. How to use this file](#1-how-to-use-this-file)
|
||||
- [1.1 When to read](#11-when-to-read)
|
||||
- [1.2 When to write](#12-when-to-write)
|
||||
- [1.3 How to write](#13-how-to-write)
|
||||
- [1.4 How the memory evolves](#14-how-the-memory-evolves)
|
||||
- [2. Pages and writing](#2-pages-and-writing)
|
||||
- [2.1 Style and wording](#21-style-and-wording)
|
||||
- [2.2 Placement and links](#22-placement-and-links)
|
||||
- [3. Sources and ingest](#3-sources-and-ingest)
|
||||
- [3.1 Hosts and access](#31-hosts-and-access)
|
||||
- [3.2 Conversion and refinement](#32-conversion-and-refinement)
|
||||
- [4. Tools and environment](#4-tools-and-environment)
|
||||
- [4.1 Commands and flags](#41-commands-and-flags)
|
||||
- [4.2 Machine and shell](#42-machine-and-shell)
|
||||
- [5. History and housekeeping](#5-history-and-housekeeping)
|
||||
- [5.1 Saves and maintenance](#51-saves-and-maintenance)
|
||||
|
||||
<!-- toc-end -->
|
||||
|
||||
What this wiki's owner wants from you, kept across sessions:
|
||||
standing preferences, corrections, repeated request patterns, and
|
||||
facts a run verified about operating this wiki. Knowledge from
|
||||
sources does not go here; it goes to pages under `wiki/`.
|
||||
|
||||
## 1. How to use this file
|
||||
|
||||
### 1.1 When to read
|
||||
|
||||
- **1.1.1** Read this file once the
|
||||
[Fiona skill](./utils/skills/fiona/SKILL.md) is loaded, before the
|
||||
first wiki work of the session. When it is already in context, do
|
||||
not read it again.
|
||||
- **1.1.2** Apply every entry that matches the request.
|
||||
- **1.1.2.1** An entry overrides a default of the Fiona skill or
|
||||
the [schema](./wiki/schema.md), for this wiki only.
|
||||
- **1.1.2.2** An entry never overrides the boundaries in section 3
|
||||
of [AGENTS](./AGENTS.md); when the two conflict, follow the
|
||||
boundary and ask the user.
|
||||
- **1.1.2.3** An entry never changes a tool contract: a command, a
|
||||
flag, or a required frontmatter field.
|
||||
- **1.1.3** Every heading and list item is numbered as
|
||||
[sectionize-markdown](./utils/skills/sectionize-markdown/SKILL.md)
|
||||
describes. Cite an item as `MEMORY.md item 2.1.3`; a new item takes
|
||||
the next free number in its subsection.
|
||||
|
||||
### 1.2 When to write
|
||||
|
||||
- **1.2.1** Write an entry when one of these happens:
|
||||
- **1.2.1.1** The user states a requirement or preference that
|
||||
outlasts the current request ("always", "never", "from now on",
|
||||
"in this wiki", "we use").
|
||||
- **1.2.1.2** The user corrects your output or approach: a page, a
|
||||
summary, a placement, a link, or a command you ran.
|
||||
- **1.2.1.3** The same kind of request comes for the second time;
|
||||
record the pattern and what the user wanted.
|
||||
- **1.2.1.4** A run verifies a reusable fact about operating this
|
||||
wiki that the Fiona skill and the schema do not hold, such as a
|
||||
host that needs a login, a converter that loses a site's tables,
|
||||
or a flag combination that works.
|
||||
- **1.2.1.5** The user explicitly asks you to remember something
|
||||
("remember", "note this", "save to memory").
|
||||
- **1.2.2** Do not write:
|
||||
- **1.2.2.1** One-off values that only apply to the current
|
||||
request.
|
||||
- **1.2.2.2** Passwords, tokens, cookies, personal data, or
|
||||
anything the user asks to keep out.
|
||||
- **1.2.2.3** Knowledge from a source. That is a page under
|
||||
`wiki/`, written by an ingest.
|
||||
- **1.2.2.4** Facts that the Fiona skill, the schema, or a wiki
|
||||
page already states; cite them instead.
|
||||
- **1.2.2.5** Guesses; only what the user said or a run showed.
|
||||
|
||||
### 1.3 How to write
|
||||
|
||||
- **1.3.1** Put the entry under the `###` subsection that fits.
|
||||
- **1.3.1.1** When none fits, add a `###` subsection with the next
|
||||
free number under the matching `##`, or a new `##` category with
|
||||
the next free number.
|
||||
- **1.3.1.2** Never put an entry directly under a `##`; its id
|
||||
would collide with a later subsection.
|
||||
- **1.3.2** Use the knowledge-entry shape of sectionize-markdown
|
||||
section 4: a header line with the source tag and a short title,
|
||||
one child per fact, and a last child with the ref and the date.
|
||||
- **1.3.3** Source tags: `user` (the user said it), `run` (a command
|
||||
or tool run showed it), `source` (a file under `raw/` states it),
|
||||
joined with `+` when several agree.
|
||||
- **1.3.4** The ref says where to check the fact: the user's request
|
||||
and its date, the command and its output line, or a page or raw
|
||||
path.
|
||||
- **1.3.5** Write the entry in the same turn, then tell the user
|
||||
which item was added or changed, for example "Saved to MEMORY.md
|
||||
item 2.1.1".
|
||||
- **1.3.6** Bump `version` (patch for a changed entry, minor for a
|
||||
new entry or subsection) and set `updated` in the frontmatter on
|
||||
every change.
|
||||
- **1.3.7** Append `- YYYY-MM-DD - memory: <item id> <title>` to
|
||||
`wiki/log.md`, and bump that file too.
|
||||
- **1.3.8** The change lands in the save of the current ingest cycle
|
||||
or housekeeping pass (Fiona skill section 2.3).
|
||||
- **1.3.8.1** With no cycle running, save it on its own with
|
||||
`fiona-assistant git save --description "memory: <item id> <title>"`.
|
||||
|
||||
An entry looks like this:
|
||||
|
||||
```text
|
||||
- **2.1.1** [user] "Source summaries open with the decision"
|
||||
- **2.1.1.1** A source page starts with the decision the source records, then the reasons.
|
||||
- **2.1.1.2** Applies to meeting notes and design reviews.
|
||||
- **2.1.1.3** Ref: user request YYYY-MM-DD; added YYYY-MM-DD.
|
||||
```
|
||||
|
||||
### 1.4 How the memory evolves
|
||||
|
||||
- **1.4.1** Before adding an entry, look for a near-duplicate in the
|
||||
same category; extend it instead and keep its id and its earliest
|
||||
date.
|
||||
- **1.4.2** A finding that contradicts an entry becomes its next
|
||||
child, titled `OVERRIDE: ...`, with the new facts and its own ref.
|
||||
- **1.4.3** An entry that no longer applies is marked
|
||||
**Superseded YYYY-MM-DD** with the reason; it is not deleted.
|
||||
- **1.4.4** An entry that held for several uses graduates; mark it
|
||||
**Promoted YYYY-MM-DD** with the target.
|
||||
- **1.4.4.1** A convention for every page moves to the
|
||||
[schema](./wiki/schema.md).
|
||||
- **1.4.4.2** A reusable procedure moves to a skill under
|
||||
[utils/skills_local](./utils/skills_local/README.md).
|
||||
- **1.4.5** Never renumber ids; pages, log lines, and chats may cite
|
||||
them.
|
||||
- **1.4.6** This file belongs to this wiki and evolves with it.
|
||||
- **1.4.6.1** `fiona-assistant wiki update` adds it only when it is
|
||||
missing. It never compares or replaces it, and never writes a
|
||||
`MEMORY-update.md`.
|
||||
- **1.4.6.2** A newer template version of these rules does not
|
||||
reach an existing wiki; the owner decides what to adopt.
|
||||
- **1.4.7** Git is the memory's history: `git log -p MEMORY.md` shows
|
||||
when each entry changed and why.
|
||||
|
||||
## 2. Pages and writing
|
||||
|
||||
How pages read, where they go, and how they link.
|
||||
|
||||
### 2.1 Style and wording
|
||||
|
||||
### 2.2 Placement and links
|
||||
|
||||
## 3. Sources and ingest
|
||||
|
||||
Where sources come from and how they become pages.
|
||||
|
||||
### 3.1 Hosts and access
|
||||
|
||||
### 3.2 Conversion and refinement
|
||||
|
||||
## 4. Tools and environment
|
||||
|
||||
How `fiona-assistant` and the machine behave for this wiki.
|
||||
|
||||
### 4.1 Commands and flags
|
||||
|
||||
### 4.2 Machine and shell
|
||||
|
||||
## 5. History and housekeeping
|
||||
|
||||
Saves, commits, merges, and maintenance passes.
|
||||
|
||||
### 5.1 Saves and maintenance
|
||||
@@ -0,0 +1,59 @@
|
||||
{
|
||||
"version": "1.1.0",
|
||||
"background": "#f4efe6",
|
||||
"graph_background": "#fffdf8",
|
||||
"font_family": "Arial, sans-serif",
|
||||
"text_color": "#2c2416",
|
||||
"node_colors": {
|
||||
"low": "#8f8272",
|
||||
"mid": "#e0943a",
|
||||
"high": "#c23b4a"
|
||||
},
|
||||
"node_stroke": "none",
|
||||
"node_radius": {
|
||||
"min": 5,
|
||||
"max": 40
|
||||
},
|
||||
"edge_colors": {
|
||||
"low": "#a89884",
|
||||
"mid": "#b86a1a",
|
||||
"high": "#8e2436"
|
||||
},
|
||||
"edge_width": 1,
|
||||
"highlight": {
|
||||
"node_color": "#2aa7a1",
|
||||
"edge_color": "#0d6e74",
|
||||
"size_multiplier": 1.3
|
||||
},
|
||||
"clicked": {
|
||||
"node_color": "#0e6b66"
|
||||
},
|
||||
"orphan": {
|
||||
"enabled": false,
|
||||
"node_color": "#7b5ea7"
|
||||
},
|
||||
"buttons": {
|
||||
"background": "#0f6e78",
|
||||
"text": "#fffdf8",
|
||||
"hover": "#0b555c",
|
||||
"disabled": "#b7aea3"
|
||||
},
|
||||
"card": {
|
||||
"background": "#fffdf8",
|
||||
"border": "#e0d3c2"
|
||||
},
|
||||
"shapes": {
|
||||
"page": "circle",
|
||||
"external": "cloud"
|
||||
},
|
||||
"subtitle_color": "#2c2416",
|
||||
"subtitle": {
|
||||
"min_radius": 0.8,
|
||||
"padding": 2,
|
||||
"size": 6,
|
||||
"on_top": true,
|
||||
"outline": false,
|
||||
"outline_color": "#fffdf8",
|
||||
"outline_width": 3
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,59 @@
|
||||
{
|
||||
"version": "1.1.0",
|
||||
"background": "#1e2836",
|
||||
"graph_background": "#283448",
|
||||
"font_family": "Segoe UI, sans-serif",
|
||||
"text_color": "#e7eef8",
|
||||
"node_colors": {
|
||||
"low": "#7d8da3",
|
||||
"mid": "#fff4c4",
|
||||
"high": "#f7d67c"
|
||||
},
|
||||
"node_stroke": "none",
|
||||
"node_radius": {
|
||||
"min": 4,
|
||||
"max": 24
|
||||
},
|
||||
"edge_colors": {
|
||||
"low": "#51627a",
|
||||
"mid": "#8fa3bd",
|
||||
"high": "#d5e2f2"
|
||||
},
|
||||
"edge_width": 1,
|
||||
"highlight": {
|
||||
"node_color": "#f3d48a",
|
||||
"edge_color": "#9ec5ea",
|
||||
"size_multiplier": 1.45
|
||||
},
|
||||
"clicked": {
|
||||
"node_color": "#ffe3a3"
|
||||
},
|
||||
"orphan": {
|
||||
"enabled": false,
|
||||
"node_color": "#ef8a7a"
|
||||
},
|
||||
"buttons": {
|
||||
"background": "#3e5674",
|
||||
"text": "#f4f8fc",
|
||||
"hover": "#527094",
|
||||
"disabled": "#2a3648"
|
||||
},
|
||||
"card": {
|
||||
"background": "#31445c",
|
||||
"border": "#5c7390"
|
||||
},
|
||||
"shapes": {
|
||||
"page": "circle",
|
||||
"external": "cloud"
|
||||
},
|
||||
"subtitle_color": "#e7eef8",
|
||||
"subtitle": {
|
||||
"min_radius": 0.8,
|
||||
"padding": 2,
|
||||
"size": 6,
|
||||
"on_top": true,
|
||||
"outline": false,
|
||||
"outline_color": "#283448",
|
||||
"outline_width": 3
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,113 @@
|
||||
{
|
||||
"version": "1.1.0",
|
||||
"background": "#e6e6e6",
|
||||
"graph_background": "#f4f4f4",
|
||||
"font_family": "Arial, sans-serif",
|
||||
"text_color": "#141414",
|
||||
"node_colors": {
|
||||
"low": "#5a5a5a",
|
||||
"mid": "#2e2e2e",
|
||||
"high": "#0d0d0d"
|
||||
},
|
||||
"node_stroke": "#141414",
|
||||
"node_radius": {
|
||||
"min": 5,
|
||||
"max": 36
|
||||
},
|
||||
"edge_colors": {
|
||||
"low": "#8f8f8f",
|
||||
"mid": "#3d3d3d",
|
||||
"high": "#141414"
|
||||
},
|
||||
"edge_width": 1,
|
||||
"highlight": {
|
||||
"node_color": "#000000",
|
||||
"edge_color": "#1a1a1a",
|
||||
"size_multiplier": 1.4
|
||||
},
|
||||
"clicked": {
|
||||
"node_color": "#000000"
|
||||
},
|
||||
"orphan": {
|
||||
"enabled": false,
|
||||
"node_color": "#bdbdbd"
|
||||
},
|
||||
"buttons": {
|
||||
"background": "#2a2a2a",
|
||||
"text": "#ffffff",
|
||||
"hover": "#111111",
|
||||
"disabled": "#b0b0b0"
|
||||
},
|
||||
"card": {
|
||||
"background": "#ffffff",
|
||||
"border": "#8a8a8a"
|
||||
},
|
||||
"shapes": {
|
||||
"page": "circle",
|
||||
"external": "cloud"
|
||||
},
|
||||
"subtitle_color": "#141414",
|
||||
"subtitle": {
|
||||
"min_radius": 0.8,
|
||||
"padding": 2,
|
||||
"size": 6,
|
||||
"on_top": true,
|
||||
"outline": false,
|
||||
"outline_color": "#f4f4f4",
|
||||
"outline_width": 3
|
||||
},
|
||||
"dark": {
|
||||
"background": "#2c2c2c",
|
||||
"graph_background": "#3a3a3a",
|
||||
"font_family": "Arial, sans-serif",
|
||||
"text_color": "#f5f5f5",
|
||||
"node_colors": {
|
||||
"low": "#8d8d8d",
|
||||
"mid": "#fff4c4",
|
||||
"high": "#f5b44a"
|
||||
},
|
||||
"node_stroke": "none",
|
||||
"node_radius": {
|
||||
"min": 5,
|
||||
"max": 36
|
||||
},
|
||||
"edge_colors": {
|
||||
"low": "#6e6e6e",
|
||||
"mid": "#c4c4c4",
|
||||
"high": "#ffffff"
|
||||
},
|
||||
"edge_width": 1,
|
||||
"highlight": {
|
||||
"node_color": "#ffffff",
|
||||
"edge_color": "#ffffff",
|
||||
"size_multiplier": 1.4
|
||||
},
|
||||
"clicked": {
|
||||
"node_color": "#ffffff"
|
||||
},
|
||||
"buttons": {
|
||||
"background": "#e8e8e8",
|
||||
"text": "#1a1a1a",
|
||||
"hover": "#ffffff",
|
||||
"disabled": "#5a5a5a"
|
||||
},
|
||||
"card": {
|
||||
"background": "#454545",
|
||||
"border": "#9a9a9a"
|
||||
},
|
||||
"shapes": {
|
||||
"page": "circle",
|
||||
"external": "cloud"
|
||||
},
|
||||
"subtitle_color": "#f5f5f5",
|
||||
"subtitle": {
|
||||
"min_radius": 0.5,
|
||||
"padding": 2,
|
||||
"size": 8,
|
||||
"on_top": true,
|
||||
"outline": true,
|
||||
"outline_color": "#2c2c2c",
|
||||
"outline_width": 3
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,59 @@
|
||||
{
|
||||
"version": "1.1.0",
|
||||
"background": "#070b14",
|
||||
"graph_background": "#0b1020",
|
||||
"font_family": "Segoe UI, sans-serif",
|
||||
"text_color": "#d6def5",
|
||||
"node_colors": {
|
||||
"low": "#8ea2c8",
|
||||
"mid": "#ffe9a0",
|
||||
"high": "#ffffff"
|
||||
},
|
||||
"node_stroke": "none",
|
||||
"node_radius": {
|
||||
"min": 3,
|
||||
"max": 20
|
||||
},
|
||||
"edge_colors": {
|
||||
"low": "#243056",
|
||||
"mid": "#7f8cbf",
|
||||
"high": "#e7ecff"
|
||||
},
|
||||
"edge_width": 1,
|
||||
"highlight": {
|
||||
"node_color": "#fff6c2",
|
||||
"edge_color": "#9ecbff",
|
||||
"size_multiplier": 1.6
|
||||
},
|
||||
"clicked": {
|
||||
"node_color": "#ffd56a"
|
||||
},
|
||||
"orphan": {
|
||||
"enabled": false,
|
||||
"node_color": "#ff7f6e"
|
||||
},
|
||||
"buttons": {
|
||||
"background": "#1e2a4a",
|
||||
"text": "#e8eeff",
|
||||
"hover": "#314572",
|
||||
"disabled": "#3a4258"
|
||||
},
|
||||
"card": {
|
||||
"background": "#12182b",
|
||||
"border": "#2a3558"
|
||||
},
|
||||
"shapes": {
|
||||
"page": "circle",
|
||||
"external": "star"
|
||||
},
|
||||
"subtitle_color": "#d5def8",
|
||||
"subtitle": {
|
||||
"min_radius": 0.8,
|
||||
"padding": 2,
|
||||
"size": 6,
|
||||
"on_top": true,
|
||||
"outline": false,
|
||||
"outline_color": "#0b1020",
|
||||
"outline_width": 3
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,59 @@
|
||||
{
|
||||
"version": "1.1.0",
|
||||
"background": "#f7fafc",
|
||||
"graph_background": "#fcfdfe",
|
||||
"font_family": "Segoe UI, Calibri, Helvetica Neue, Arial, sans-serif",
|
||||
"text_color": "#4a5d70",
|
||||
"node_colors": {
|
||||
"low": "#e4eef4",
|
||||
"mid": "#a9c6d8",
|
||||
"high": "#7eabc4"
|
||||
},
|
||||
"node_stroke": "#9bb8cb",
|
||||
"node_radius": {
|
||||
"min": 4,
|
||||
"max": 26
|
||||
},
|
||||
"edge_colors": {
|
||||
"low": "#e7f0f5",
|
||||
"mid": "#b7d0de",
|
||||
"high": "#8eb4c9"
|
||||
},
|
||||
"edge_width": 1,
|
||||
"highlight": {
|
||||
"node_color": "#6aa3c0",
|
||||
"edge_color": "#7eabc4",
|
||||
"size_multiplier": 1.15
|
||||
},
|
||||
"clicked": {
|
||||
"node_color": "#5b93b0"
|
||||
},
|
||||
"orphan": {
|
||||
"enabled": false,
|
||||
"node_color": "#d9a5b3"
|
||||
},
|
||||
"buttons": {
|
||||
"background": "#8eb4c9",
|
||||
"text": "#ffffff",
|
||||
"hover": "#7aa6bd",
|
||||
"disabled": "#dce7ee"
|
||||
},
|
||||
"card": {
|
||||
"background": "#ffffff",
|
||||
"border": "#e6eef3"
|
||||
},
|
||||
"shapes": {
|
||||
"page": "circle",
|
||||
"external": "cloud"
|
||||
},
|
||||
"subtitle_color": "#4a5d70",
|
||||
"subtitle": {
|
||||
"min_radius": 0.8,
|
||||
"padding": 2,
|
||||
"size": 6,
|
||||
"on_top": true,
|
||||
"outline": false,
|
||||
"outline_color": "#fcfdfe",
|
||||
"outline_width": 3
|
||||
}
|
||||
}
|
||||
@@ -1,4 +1,5 @@
|
||||
{
|
||||
"version": "1.0.1",
|
||||
"markdown": {
|
||||
"pymarkdown": {
|
||||
"extensions": [
|
||||
@@ -58,20 +59,32 @@
|
||||
},
|
||||
"wiki": {
|
||||
"graph": {
|
||||
"node_colors": {
|
||||
"low": "#808080",
|
||||
"mid": "#ffa500",
|
||||
"high": "#FF0000"
|
||||
"theme": "dark",
|
||||
"show_external_links": true,
|
||||
"subtitle": {
|
||||
"enabled": true
|
||||
},
|
||||
"node_radius": {
|
||||
"min": 5,
|
||||
"max": 40
|
||||
},
|
||||
"hover": {
|
||||
"color": "#add8e6",
|
||||
"edge_color": "#0000ff",
|
||||
"size_multiplier": 1.3
|
||||
}
|
||||
"show_legend": true,
|
||||
"show_arrows": false,
|
||||
"show_source": true,
|
||||
"show_glow": true,
|
||||
"show_link_card": true,
|
||||
"first_fit_ms": 12000,
|
||||
"hover_link_tail_chars": 12,
|
||||
"simplify_external_links": true,
|
||||
"consolidate_external_links": true,
|
||||
"consolidate_external_links_depth": 3,
|
||||
"consolidate_external_links_max_char": 8,
|
||||
"ignore_files": [
|
||||
"*/table_of_contents.md",
|
||||
"*/README.md",
|
||||
"*/AGENTS.md"
|
||||
]
|
||||
},
|
||||
"update": {
|
||||
"ignore_files": [
|
||||
"corefiles.json"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
---
|
||||
name: map
|
||||
version: 1.1.0
|
||||
title: Script Map
|
||||
type: summary
|
||||
created: 2025-09-23
|
||||
updated: 2026-09-30
|
||||
---
|
||||
|
||||
# Script Map
|
||||
|
||||
Catalog of the reusable scripts in this folder. Read it before
|
||||
writing a new script, and add a row when you add one. The rules for
|
||||
writing a script are in the
|
||||
[Fiona skill](../skills/fiona/SKILL.md#write-reusable-scripts).
|
||||
|
||||
| Script | Purpose | Usage | Added |
|
||||
| --- | --- | --- | --- |
|
||||
| `example.py` | Placeholder that prints a greeting. | `python utils/scripts/example.py` | 2025-09-23 |
|
||||
@@ -0,0 +1,18 @@
|
||||
---
|
||||
name: skills
|
||||
version: 1.1.1
|
||||
title: Skills Directory
|
||||
type: summary
|
||||
created: 2025-09-23
|
||||
updated: 2026-10-21
|
||||
---
|
||||
|
||||
# Skills Directory
|
||||
|
||||
Skill definitions and configurations.
|
||||
|
||||
- [fiona](./fiona/SKILL.md): operate the llm-wiki with `fiona-assistant`.
|
||||
- [memory](./memory/SKILL.md): give a skill its own `MEMORIES.md`.
|
||||
- [python-tdd](./python-tdd/SKILL.md): test-driven Python work.
|
||||
- [sectionize-markdown](./sectionize-markdown/SKILL.md):
|
||||
number headings and list items so each rule can be cited.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,144 @@
|
||||
---
|
||||
name: llm-wiki-mod
|
||||
title: LLM Wiki Pattern
|
||||
version: 1.4.1
|
||||
updated: 2026-10-06
|
||||
type: concept
|
||||
tags: [wiki, schema]
|
||||
---
|
||||
|
||||
# LLM Wiki
|
||||
|
||||
A persistent knowledge base of interlinked markdown. The pattern
|
||||
comes from
|
||||
[Andrej Karpathy's LLM Wiki](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f).
|
||||
Page rules are in [schema](../../../wiki/schema.md). Commands and
|
||||
the edit loop are in the [Fiona skill](./SKILL.md).
|
||||
|
||||
- [1. Core idea](#1-core-idea)
|
||||
- [2. Layers](#2-layers)
|
||||
- [3. Operations](#3-operations)
|
||||
- [4. Index and log](#4-index-and-log)
|
||||
- [5. Page rules](#5-page-rules)
|
||||
- [6. Why this works](#6-why-this-works)
|
||||
- [7. Scope](#7-scope)
|
||||
|
||||
## 1. Core idea
|
||||
|
||||
Sources are read once and compiled into wiki pages. Later questions
|
||||
use those pages. Cross-references, contradictions, and synthesis
|
||||
stay in the files. Each new source and each filed answer adds to
|
||||
that store.
|
||||
|
||||
The human curates sources and asks the questions. The agent writes
|
||||
the pages, updates cross-references, and keeps the bookkeeping.
|
||||
|
||||
## 2. Layers
|
||||
|
||||
Three layers. The directory tree in
|
||||
[schema](../../../wiki/schema.md) is the map.
|
||||
|
||||
### 2.1. Raw sources
|
||||
|
||||
`raw/` is the archive. The agent adds new sources there and never
|
||||
edits, moves, or deletes one already archived.
|
||||
`ingest/` is the drop zone for files that still need parsing.
|
||||
`personal/` holds user notes and is read-only, including the
|
||||
directory itself.
|
||||
|
||||
### 2.2. The wiki
|
||||
|
||||
`wiki/` is agent-owned markdown. Synthesized pages go in
|
||||
`concepts/`, `entities/`, `comparisons/`, and `queries/`. Source
|
||||
summaries that back those pages go in `sources/`. The agent creates
|
||||
pages, revises them when a new source arrives, and keeps links
|
||||
consistent.
|
||||
|
||||
### 2.3. The schema
|
||||
|
||||
[schema](../../../wiki/schema.md) states file names, frontmatter,
|
||||
tags, when to create a page, and the directory tree.
|
||||
[AGENTS.md](../../../AGENTS.md) is the entry point. It loads the
|
||||
[Fiona skill](./SKILL.md) for commands and the edit loop. When a
|
||||
convention changes, change the schema. Leave the tree in that one
|
||||
file.
|
||||
|
||||
## 3. Operations
|
||||
|
||||
### 3.1. Ingest
|
||||
|
||||
Put a new source in `ingest/`. The agent reads it, writes the
|
||||
summary under `wiki/`, updates [index](../../../wiki/index.md),
|
||||
updates the entity and concept pages the source touches, and
|
||||
appends one dated line to [log](../../../wiki/log.md). One source
|
||||
can touch many pages. Take sources one at a time when the summary
|
||||
should be checked before the next file.
|
||||
|
||||
After the archive step, the source lives under `raw/` and is not
|
||||
edited again.
|
||||
|
||||
An ingest has two phases. A tool fetches the source, archives it
|
||||
under `raw/`, and converts it into a page marked `status: raw`.
|
||||
The agent then refines that page, updates the pages it touches,
|
||||
and marks it `status: refined`. The steps are section 18 of the
|
||||
[Fiona skill](./SKILL.md).
|
||||
|
||||
### 3.2. Query
|
||||
|
||||
Ask against the wiki. The agent reads the index, opens the pages
|
||||
that apply, and answers with citations. File a comparison, an
|
||||
analysis, or a connection that should outlive the chat under
|
||||
`wiki/queries/` or `wiki/comparisons/`.
|
||||
|
||||
### 3.3. Lint
|
||||
|
||||
Check the wiki for contradictions, claims a newer source has
|
||||
replaced, pages with no inbound links, concepts that need their
|
||||
own page, missing cross-references, and gaps a source could fill.
|
||||
The health command is `fiona-assistant wiki sanity`. Style,
|
||||
frontmatter, and link checks are in the [Fiona skill](./SKILL.md).
|
||||
|
||||
## 4. Index and log
|
||||
|
||||
[index](../../../wiki/index.md) is the catalog. Each page has a
|
||||
link and a one-line summary, grouped by section. Update it when
|
||||
a page is added. Read it first when answering a query.
|
||||
|
||||
[log](../../../wiki/log.md) is append-only. Each entry is one
|
||||
dated bullet:
|
||||
|
||||
```text
|
||||
- YYYY-MM-DD - what happened
|
||||
```
|
||||
|
||||
Bump `version` and `updated` on `log.md` when appending. The log
|
||||
is the timeline of ingests, queries, and lint passes.
|
||||
|
||||
Git holds the before and after of each entry. Every ingest cycle
|
||||
and every housekeeping pass ends in one commit, so one source or
|
||||
one cleanup can be reviewed or undone on its own.
|
||||
|
||||
## 5. Page rules
|
||||
|
||||
File names, frontmatter, tags, links, and when to create or split
|
||||
a page are in [schema](../../../wiki/schema.md). How to fill the
|
||||
frontmatter and the quality signals is section 15 of the
|
||||
[Fiona skill](./SKILL.md). This file does not repeat them.
|
||||
|
||||
## 6. Why this works
|
||||
|
||||
Cross-references, summaries that still match the sources, and notes
|
||||
where a new source disagrees with an older claim are the work that
|
||||
usually gets dropped. The agent does that work on each ingest, so
|
||||
the wiki stays current as it grows.
|
||||
|
||||
The human chooses sources, directs the analysis, and decides what
|
||||
the pages mean. The agent does the filing.
|
||||
|
||||
## 7. Scope
|
||||
|
||||
This file states the pattern. The directory tree stays in
|
||||
[schema](../../../wiki/schema.md). Commands stay in the
|
||||
[Fiona skill](./SKILL.md). An Obsidian graph, Marp slides,
|
||||
Dataview, or a local search engine can sit beside the wiki. This
|
||||
wiki does not require them.
|
||||
@@ -0,0 +1,173 @@
|
||||
---
|
||||
name: skill-memories
|
||||
version: 1.0.1
|
||||
title: SKILL_NAME Memories
|
||||
created: DATE_PLACEHOLDER
|
||||
updated: DATE_PLACEHOLDER
|
||||
type: summary
|
||||
---
|
||||
|
||||
# SKILL_NAME Memories
|
||||
|
||||
<!-- toc-start: headings 2..6 -->
|
||||
|
||||
- [1. How to use this file](#1-how-to-use-this-file)
|
||||
- [1.1 When to read](#11-when-to-read)
|
||||
- [1.2 When to write](#12-when-to-write)
|
||||
- [1.3 How to write](#13-how-to-write)
|
||||
- [1.4 How the memory evolves](#14-how-the-memory-evolves)
|
||||
- [2. Output and wording](#2-output-and-wording)
|
||||
- [2.1 Style and format](#21-style-and-format)
|
||||
- [2.2 Placement and naming](#22-placement-and-naming)
|
||||
- [3. Inputs and sources](#3-inputs-and-sources)
|
||||
- [3.1 Where inputs come from](#31-where-inputs-come-from)
|
||||
- [3.2 Quirks and parsing](#32-quirks-and-parsing)
|
||||
- [4. Steps and tools](#4-steps-and-tools)
|
||||
- [4.1 Commands and flags](#41-commands-and-flags)
|
||||
- [4.2 Machine and shell](#42-machine-and-shell)
|
||||
- [5. Recurring requests](#5-recurring-requests)
|
||||
- [5.1 Patterns](#51-patterns)
|
||||
|
||||
<!-- toc-end -->
|
||||
|
||||
What users want from the SKILL_NAME skill, kept across sessions:
|
||||
standing preferences, corrections, repeated request patterns, and
|
||||
facts a run verified while using this skill. Preferences for the
|
||||
whole wiki go to the wiki's `MEMORY.md`. Knowledge from sources goes
|
||||
to pages under `wiki/`.
|
||||
|
||||
## 1. How to use this file
|
||||
|
||||
### 1.1 When to read
|
||||
|
||||
- **1.1.1** Read this file once [the skill](./SKILL.md) is loaded,
|
||||
before its first step of the session. When it is already in
|
||||
context, do not read it again.
|
||||
- **1.1.2** Apply every entry that matches the request.
|
||||
- **1.1.2.1** An entry overrides a default of the skill and an
|
||||
entry of the wiki's `MEMORY.md`, for work done with this skill
|
||||
only.
|
||||
- **1.1.2.2** An entry never overrides the boundaries in the wiki's
|
||||
`AGENTS.md`; when the two conflict, follow the boundary and ask
|
||||
the user.
|
||||
- **1.1.2.3** An entry never changes a tool contract: a command, a
|
||||
flag, or a required frontmatter field.
|
||||
- **1.1.3** Every heading and list item is numbered as
|
||||
[sectionize-markdown](../../skills/sectionize-markdown/SKILL.md)
|
||||
describes. Cite an item as `SKILL_NAME MEMORIES.md item 2.1.3`; a
|
||||
new item takes the next free number in its subsection.
|
||||
|
||||
### 1.2 When to write
|
||||
|
||||
- **1.2.1** Write an entry when one of these happens while the skill
|
||||
is in use:
|
||||
- **1.2.1.1** The user states a requirement or preference for this
|
||||
skill that outlasts the current request ("always", "never", "from
|
||||
now on", "when you do X").
|
||||
- **1.2.1.2** The user corrects the skill's output or a step it
|
||||
took.
|
||||
- **1.2.1.3** The same kind of request comes for the second time;
|
||||
record the pattern and what the user wanted.
|
||||
- **1.2.1.4** A run verifies a reusable fact about the skill's
|
||||
inputs, hosts, or tools that the skill does not state.
|
||||
- **1.2.1.5** The user explicitly asks you to remember something
|
||||
for this skill.
|
||||
- **1.2.2** Do not write:
|
||||
- **1.2.2.1** One-off values that only apply to the current
|
||||
request.
|
||||
- **1.2.2.2** Passwords, tokens, cookies, personal data, or
|
||||
anything the user asks to keep out.
|
||||
- **1.2.2.3** Preferences that hold for any wiki work; those go to
|
||||
the wiki's `MEMORY.md`.
|
||||
- **1.2.2.4** Facts the skill or a wiki page already states; cite
|
||||
them instead.
|
||||
- **1.2.2.5** Guesses; only what the user said or a run showed.
|
||||
|
||||
### 1.3 How to write
|
||||
|
||||
- **1.3.1** Put the entry under the `###` subsection that fits.
|
||||
- **1.3.1.1** When none fits, add a `###` subsection with the next
|
||||
free number under the matching `##`, or a new `##` category with
|
||||
the next free number.
|
||||
- **1.3.1.2** Never put an entry directly under a `##`; its id
|
||||
would collide with a later subsection.
|
||||
- **1.3.2** Use the knowledge-entry shape of sectionize-markdown
|
||||
section 4: a header line with the source tag and a short title,
|
||||
one child per fact, and a last child with the ref and the date.
|
||||
- **1.3.3** Source tags: `user` (the user said it), `run` (a command
|
||||
or tool run showed it), `source` (a file under `raw/` states it),
|
||||
joined with `+` when several agree.
|
||||
- **1.3.4** The ref says where to check the fact: the user's request
|
||||
and its date, the command and its output line, or a page or raw
|
||||
path.
|
||||
- **1.3.5** Write the entry in the same turn, then tell the user
|
||||
which item was added or changed, for example "Saved to SKILL_NAME
|
||||
MEMORIES.md item 2.1.1".
|
||||
- **1.3.6** Bump `version` (patch for a changed entry, minor for a
|
||||
new entry or subsection) and set `updated` in the frontmatter on
|
||||
every change.
|
||||
- **1.3.7** Append `- YYYY-MM-DD - SKILL_NAME memory: <item id> <title>`
|
||||
to `wiki/log.md`, and bump that file too.
|
||||
- **1.3.8** The change lands in the save of the current ingest cycle
|
||||
or housekeeping pass.
|
||||
- **1.3.8.1** With no cycle running, save it on its own with
|
||||
`fiona-assistant git save --description "SKILL_NAME memory: <item id> <title>"`.
|
||||
|
||||
An entry looks like this:
|
||||
|
||||
```text
|
||||
- **2.1.1** [user] "Reports open with the verdict"
|
||||
- **2.1.1.1** The report starts with pass or fail, then the evidence.
|
||||
- **2.1.1.2** Applies to every run of this skill.
|
||||
- **2.1.1.3** Ref: user request YYYY-MM-DD; added YYYY-MM-DD.
|
||||
```
|
||||
|
||||
### 1.4 How the memory evolves
|
||||
|
||||
- **1.4.1** Before adding an entry, look for a near-duplicate in the
|
||||
same category; extend it instead and keep its id and its earliest
|
||||
date.
|
||||
- **1.4.2** A finding that contradicts an entry becomes its next
|
||||
child, titled `OVERRIDE: ...`, with the new facts and its own ref.
|
||||
- **1.4.3** An entry that no longer applies is marked
|
||||
**Superseded YYYY-MM-DD** with the reason; it is not deleted.
|
||||
- **1.4.4** An entry that held for several uses graduates; mark it
|
||||
**Promoted YYYY-MM-DD** with the target.
|
||||
- **1.4.4.1** A rule for every run of the skill moves into
|
||||
[the skill](./SKILL.md).
|
||||
- **1.4.4.2** A preference that turns out to hold for all wiki
|
||||
work moves to the wiki's `MEMORY.md`.
|
||||
- **1.4.5** Never renumber ids; pages, log lines, and chats may cite
|
||||
them.
|
||||
- **1.4.6** Git is the memory's history: `git log -p MEMORIES.md` in
|
||||
the skill's folder shows when each entry changed and why.
|
||||
|
||||
## 2. Output and wording
|
||||
|
||||
What the skill produces and how it reads.
|
||||
|
||||
### 2.1 Style and format
|
||||
|
||||
### 2.2 Placement and naming
|
||||
|
||||
## 3. Inputs and sources
|
||||
|
||||
What the skill reads and where it comes from.
|
||||
|
||||
### 3.1 Where inputs come from
|
||||
|
||||
### 3.2 Quirks and parsing
|
||||
|
||||
## 4. Steps and tools
|
||||
|
||||
How the skill's commands and the machine behave.
|
||||
|
||||
### 4.1 Commands and flags
|
||||
|
||||
### 4.2 Machine and shell
|
||||
|
||||
## 5. Recurring requests
|
||||
|
||||
Requests that come back, and what the user wanted each time.
|
||||
|
||||
### 5.1 Patterns
|
||||
@@ -0,0 +1,109 @@
|
||||
---
|
||||
name: memory
|
||||
category: productivity
|
||||
description: Give a skill its own memory, a MEMORIES.md beside its SKILL.md.
|
||||
author: nvaldez
|
||||
version: 1.0.1
|
||||
created: 2026-10-15
|
||||
updated: 2026-10-21
|
||||
---
|
||||
|
||||
# Skill Memory
|
||||
|
||||
<!-- toc-start: headings 2..6 -->
|
||||
|
||||
- [1. What a skill memory is](#1-what-a-skill-memory-is)
|
||||
- [2. Give a skill memory](#2-give-a-skill-memory)
|
||||
- [3. Wire it into the skill](#3-wire-it-into-the-skill)
|
||||
- [4. Use a skill memory](#4-use-a-skill-memory)
|
||||
|
||||
<!-- toc-end -->
|
||||
|
||||
A skill with memory keeps what its users want from that skill across
|
||||
sessions, in a `MEMORIES.md` file beside its `SKILL.md`. The blank
|
||||
file to copy is [MEMORIES](./MEMORIES.md). The wiki-wide memory is
|
||||
[MEMORY](../../../MEMORY.md); this skill does not replace it.
|
||||
|
||||
Every heading and list item is numbered as
|
||||
[sectionize-markdown](../sectionize-markdown/SKILL.md)
|
||||
describes. Cite a rule as `memory skill item 2.3`.
|
||||
|
||||
## 1. What a skill memory is
|
||||
|
||||
- **1.1** One file per skill: `MEMORIES.md` in the skill's folder,
|
||||
next to its `SKILL.md`.
|
||||
- **1.2** It holds facts about operating that one skill:
|
||||
- **1.2.1** Preferences for the skill's output.
|
||||
- **1.2.2** Corrections to the skill's steps or results.
|
||||
- **1.2.3** Request patterns that came back for that skill.
|
||||
- **1.2.4** Facts a run verified about the skill's inputs, hosts,
|
||||
and tools.
|
||||
- **1.3** It does not hold:
|
||||
- **1.3.1** Knowledge from sources; that is a page under `wiki/`.
|
||||
- **1.3.2** Preferences for the whole wiki; those go to
|
||||
[MEMORY](../../../MEMORY.md).
|
||||
- **1.3.3** Passwords, tokens, cookies, or personal data.
|
||||
- **1.4** Precedence, from strongest to weakest:
|
||||
- **1.4.1** The boundaries in [AGENTS](../../../AGENTS.md) and tool
|
||||
contracts (commands, flags, required frontmatter).
|
||||
- **1.4.2** The skill's `MEMORIES.md`, for work done with that skill.
|
||||
- **1.4.3** The wiki's `MEMORY.md`.
|
||||
- **1.4.4** The defaults in the skill's `SKILL.md`.
|
||||
|
||||
## 2. Give a skill memory
|
||||
|
||||
- **2.1** Give a skill memory when the user asks for a skill with
|
||||
memory, or when the same correction to one skill comes a second
|
||||
time.
|
||||
- **2.2** A new skill lives under `utils/skills_local/<skill>/`. A
|
||||
skill shipped under `utils/skills/` can also get a memory; the
|
||||
template never ships a `MEMORIES.md` for it, so an update leaves the
|
||||
file alone.
|
||||
- **2.3** Copy [MEMORIES](./MEMORIES.md) to
|
||||
`<skill-folder>/MEMORIES.md`. Never edit the template copy in this
|
||||
folder.
|
||||
- **2.4** In the copy:
|
||||
- **2.4.1** Set frontmatter `name` to `<skill>-memories`, `version`
|
||||
to `1.0.0`, and `created` and `updated` to today.
|
||||
- **2.4.2** Replace every `SKILL_NAME` with the skill's name.
|
||||
- **2.4.3** Rename the empty categories in sections 2 to 5 to fit
|
||||
the skill, or keep them. Leave section 1 as it is.
|
||||
- **2.5** Add the memory section of section 3 to the skill's
|
||||
`SKILL.md` and bump that file's `version`.
|
||||
- **2.6** Append `- YYYY-MM-DD - memory: <skill> gets MEMORIES.md` to
|
||||
`wiki/log.md`, then save with
|
||||
`fiona-assistant git save --description "memory: <skill> gets MEMORIES.md"`.
|
||||
|
||||
## 3. Wire it into the skill
|
||||
|
||||
- **3.1** The skill's `SKILL.md` gets this section as its last
|
||||
numbered section, with `N` set to that section's number:
|
||||
|
||||
```markdown
|
||||
## N. Memory
|
||||
|
||||
- **N.1** After loading this skill, read [MEMORIES](./MEMORIES.md)
|
||||
once per session. When it is already in context, do not read it
|
||||
again.
|
||||
- **N.2** Update it whenever an important memory about this skill
|
||||
comes up: a lasting preference, a correction, a repeated request,
|
||||
or a fact a run verified. Also update it whenever the user
|
||||
explicitly asks you to remember something for this skill.
|
||||
Section 1 of that file says how.
|
||||
```
|
||||
|
||||
- **3.2** Keep the link relative (`./MEMORIES.md`) so the skill still
|
||||
finds its memory after a move.
|
||||
|
||||
## 4. Use a skill memory
|
||||
|
||||
- **4.1** Section 1 of each skill's `MEMORIES.md` is the rule for
|
||||
reading and writing it. This file does not repeat it.
|
||||
- **4.2** Pick the narrowest memory that covers the fact:
|
||||
- **4.2.1** It only matters while one skill runs: that skill's
|
||||
`MEMORIES.md`.
|
||||
- **4.2.2** It matters for any wiki work: `MEMORY.md`.
|
||||
- **4.2.3** When unsure, ask the user which one.
|
||||
- **4.3** When a request uses a skill that has no memory and a
|
||||
memory comes up, write it to `MEMORY.md` and suggest giving the
|
||||
skill one (section 2).
|
||||
@@ -0,0 +1,159 @@
|
||||
---
|
||||
name: sectionize-markdown
|
||||
version: 1.1.3
|
||||
title: How to sectionize a markdown file
|
||||
description: Number every heading and list item of a markdown file so each
|
||||
rule, step, or fact has a stable id to cite.
|
||||
created: '2026-10-14'
|
||||
updated: '2026-10-21'
|
||||
confidence: high
|
||||
llm:
|
||||
- model: Claude Opus 5.5
|
||||
---
|
||||
|
||||
# How to sectionize a markdown file
|
||||
|
||||
<!-- toc-start: headings 2..6 -->
|
||||
|
||||
- [1. Headings](#1-headings)
|
||||
- [2. List item ids](#2-list-item-ids)
|
||||
- [3. Splitting long lines](#3-splitting-long-lines)
|
||||
- [4. Knowledge entries](#4-knowledge-entries)
|
||||
- [5. Organize by category](#5-organize-by-category)
|
||||
- [6. Doing it safely](#6-doing-it-safely)
|
||||
- [7. Example](#7-example)
|
||||
|
||||
<!-- toc-end -->
|
||||
|
||||
Numbers every heading and every list item of a markdown file so each
|
||||
rule, step, or fact has a stable id that can be cited ("item
|
||||
4.2.3"), tracked, and checked for duplicates. It suits rule files,
|
||||
skill instructions, glossaries, and knowledge caches.
|
||||
|
||||
## 1. Headings
|
||||
|
||||
- **1.1** Number every heading below the title: `## N.` for sections,
|
||||
`### N.M` for subsections, `#### N.M.K` below that.
|
||||
- **1.2** The `#` title and the table of contents are not numbered.
|
||||
- **1.3** Put a TOC of markdown links right under the title and intro.
|
||||
Regenerate it whenever headings change.
|
||||
- **1.4** Anchors are GitHub-style slugs: `## 5.6 Download the logs`
|
||||
becomes `#56-download-the-logs`.
|
||||
- **1.5** Avoid em dashes and other punctuation in headings; use a colon
|
||||
(`10.7 Outcome: a match exists`). Punctuation makes the slug
|
||||
unpredictable and breaks TOC links (MD051).
|
||||
- **1.6** A label that only introduces a list ("Files in this
|
||||
folder:") is a numbered list item, not a heading.
|
||||
|
||||
## 2. List item ids
|
||||
|
||||
- **2.1** Every list item gets a bold id right after the dash:
|
||||
`- **4.2.1** Text`. Bold is two asterisks on each side; a single
|
||||
asterisk is italic.
|
||||
- **2.2** The id is the section number plus a running counter:
|
||||
`**<section>.<n>**`. Nested items extend the parent id:
|
||||
`**<section>.<n>.<m>**`, indented two spaces.
|
||||
- **2.3** The counter runs through the whole section, across separate
|
||||
lists split by paragraphs or code blocks. Two lists under `### 12.3`
|
||||
give `**12.3.1**`…`**12.3.3**`, then `**12.3.4**`…, never a second
|
||||
`**12.3.1**`.
|
||||
- **2.4** Checklists keep the box first: `- [ ] **13.1** Verify …`.
|
||||
- **2.5** Do not number the TOC, code blocks, or table rows.
|
||||
- **2.6** State the convention in the file's first section so the next
|
||||
editor follows it ("Cite an item as `item 4.2.3`").
|
||||
- **2.7** Never renumber existing ids; other notes may cite them. A new
|
||||
item takes the next free number in its section, even if that puts it
|
||||
out of reading order. Fix the order only while the ids are still
|
||||
unreleased (uncommitted).
|
||||
|
||||
## 3. Splitting long lines
|
||||
|
||||
- **3.1** A list item holding several facts becomes a short header item
|
||||
plus one child per fact.
|
||||
- **3.2** One child is one sentence, or two closely linked sentences.
|
||||
- **3.3** Split at sentence ends (`.`, `?` or `!`, then a space and an
|
||||
uppercase letter, backtick, `*`, `"` or `[`). Do not split inside
|
||||
backticks or parentheses, or after `e.g.`, `i.e.`, `vs.`, `cf.`.
|
||||
- **3.4** Merge fragments shorter than about 45 characters back into the
|
||||
previous child; they read better together.
|
||||
- **3.5** A long single sentence stays on one line. Do not cut a
|
||||
sentence in half to meet a length target.
|
||||
- **3.6** Leave command lines, code blocks, and table rows as they are.
|
||||
|
||||
## 4. Knowledge entries
|
||||
|
||||
A file that collects findings or lessons learned uses a fixed entry
|
||||
shape so each fact is citable and the source is visible:
|
||||
|
||||
```text
|
||||
- **<section>.<n>** [<source>] "<short title>"
|
||||
- **<section>.<n>.1** <reusable fact + where it applies>
|
||||
- **<section>.<n>.2** <next fact>
|
||||
- **<section>.<n>.<k>** Ref: <path:line or query>; added <YYYY-MM-DD>.
|
||||
```
|
||||
|
||||
- **4.1** The entry line holds only the source tag and the title.
|
||||
- **4.2** `<source>` names where the fact was verified (a repository, a
|
||||
spec, …), joined with `+` when several agree.
|
||||
- **4.3** The last child is always the ref with the date it was added.
|
||||
- **4.4** A finding that contradicts an entry becomes that entry's next
|
||||
child, titled `OVERRIDE: …`, with its own children. Or mark the old
|
||||
entry **Superseded YYYY-MM-DD** with the reason.
|
||||
|
||||
## 5. Organize by category
|
||||
|
||||
- **5.1** Group entries by topic (setup, build, deployment, …), not by
|
||||
the date they were added. The date lives in the ref line.
|
||||
- **5.2** Grouping puts near-duplicates next to each other. Merge them
|
||||
into one entry and keep the earliest `added` date.
|
||||
- **5.3** A glossary gets the same treatment: one
|
||||
`###` per term under a `##` category, its definition split into
|
||||
numbered items.
|
||||
|
||||
## 6. Doing it safely
|
||||
|
||||
- **6.1** Restructure with a script, not by hand, when the file has more
|
||||
than a few dozen items. Hand edits drop or reorder text.
|
||||
- **6.2** The script asserts that the words are preserved: same word
|
||||
sequence before and after, only ids and line breaks added.
|
||||
- **6.3** Verify after every pass:
|
||||
- **6.3.1** No duplicate ids (`^\s*- (\[ \] )?\*\*([\d.]+)\*\*`).
|
||||
- **6.3.2** Every `](#anchor)` resolves to a heading slug.
|
||||
- **6.3.3** The entry count matches the original; diff titles against
|
||||
`HEAD` when it does not (an uncommitted entry may explain it).
|
||||
- **6.3.4** One `Ref:` line per entry.
|
||||
- **6.4** Watch for editors that reformat on save (escaped `\*`, `*`
|
||||
bullets padded with three spaces, lost HTML comments, fences without a language). Normalize
|
||||
back and re-verify.
|
||||
- **6.5** On Windows PowerShell, write helper scripts to `%TEMP%` and run
|
||||
them; `python -c` with backticks breaks. Avoid printing non-ASCII to
|
||||
the cp1252 console. Delete the temp scripts afterwards.
|
||||
- **6.6** Keep Unix line endings and `-` bullets, and run the markdown
|
||||
linter before committing.
|
||||
|
||||
## 7. Example
|
||||
|
||||
Before:
|
||||
|
||||
```text
|
||||
## Topic
|
||||
|
||||
- Rule about the topic. A fact about it. How to fix it. Ref: file:line.
|
||||
- Label for a group: first part of the rule, second part of the rule.
|
||||
```
|
||||
|
||||
After:
|
||||
|
||||
```text
|
||||
## 3. Topic
|
||||
|
||||
### 3.2 Subtopic
|
||||
|
||||
- **3.2.4** [source] "Rule about the topic"
|
||||
- **3.2.4.1** A fact about it.
|
||||
- **3.2.4.2** How to fix it.
|
||||
- **3.2.4.3** Ref: file:line; added YYYY-MM-DD.
|
||||
- **3.2.5** Label for a group:
|
||||
- **3.2.5.1** First part of the rule.
|
||||
- **3.2.5.2** Second part of the rule.
|
||||
```
|
||||
@@ -0,0 +1,321 @@
|
||||
---
|
||||
name: specification-writer
|
||||
description: Load when writing or updating project specifications (SPEC.md).
|
||||
Expert in creating structured and maintainable specification documents.
|
||||
type: skill
|
||||
version: 1.0.0
|
||||
created: 2026-11-03
|
||||
updated: 2026-11-03
|
||||
---
|
||||
|
||||
# Specification Writer
|
||||
|
||||
You are an expert in writing software project specifications.
|
||||
This skill provides guidance on creating structured, comprehensive, and
|
||||
maintainable SPEC.md files following established best practices.
|
||||
|
||||
## When to Load
|
||||
|
||||
Load this skill when:
|
||||
|
||||
- Creating a new SPEC.md file for a project
|
||||
- Updating or expanding an existing specification
|
||||
- Reviewing specification structure and content
|
||||
- Ensuring specifications follow consistent patterns
|
||||
|
||||
## Core Principles
|
||||
|
||||
### Sectionize-Markdown Conventions
|
||||
|
||||
Always apply sectionize-markdown conventions to every SPEC.md file.
|
||||
Reference: [sectionize-markdown skill](../sectionize-markdown/SKILL.md)
|
||||
|
||||
- Every heading MUST be numbered: `## 1.` for sections, `### 1.1`
|
||||
for subsections, etc.
|
||||
- Every list item MUST have a bold ID: `- **1.1.1** Text`
|
||||
- TOC MUST be present under the title with markdown links
|
||||
- Anchors MUST follow GitHub-style slugs
|
||||
- Never renumber existing IDs - they may be cited elsewhere
|
||||
|
||||
## SPEC.md Structure Template
|
||||
|
||||
Use this structure as the foundation for all project specifications:
|
||||
|
||||
```markdown
|
||||
---
|
||||
name: <Project Name> Specification
|
||||
version: <semver>
|
||||
title: Project Spec Sheet
|
||||
type: specification
|
||||
created: <YYYY-MM-DD>
|
||||
updated: <YYYY-MM-DD>
|
||||
---
|
||||
|
||||
# <Project Name> Specification
|
||||
|
||||
<!-- toc-start: headings 2..6 -->
|
||||
|
||||
- [1. Product Overview](#1-product-overview)
|
||||
- [1.1. Purpose](#11-purpose)
|
||||
- [1.2. Scope](#12-scope)
|
||||
- [1.3. Non-Goals](#13-non-goals)
|
||||
- [1.4. Document Conventions](#14-document-conventions)
|
||||
- [2. Architecture](#2-architecture)
|
||||
- [2.1. Component Diagram](#21-component-diagram)
|
||||
- [2.2. Data Flow](#22-data-flow)
|
||||
- [2.3. Module Structure](#23-module-structure)
|
||||
- [3. Core Contracts](#3-core-contracts)
|
||||
- [3.1. Interfaces](#31-interfaces)
|
||||
- [3.2. Data Structures](#32-data-structures)
|
||||
- [4. Package and Runtime](#4-package-and-runtime)
|
||||
- [4.1. Dependencies](#41-dependencies)
|
||||
- [4.2. Entry Points](#42-entry-points)
|
||||
- [4.3. Installation](#43-installation)
|
||||
- [5. Feature Requirements](#5-feature-requirements)
|
||||
- [6. Configuration Files](#6-configuration-files)
|
||||
- [7. Error Handling](#7-error-handling)
|
||||
- [8. Testing](#8-testing)
|
||||
- [9. Deployment](#9-deployment)
|
||||
|
||||
<!-- toc-end -->
|
||||
|
||||
This document is the build contract for <Project Name>.
|
||||
A reader with this file, and no source tree, must be able to produce
|
||||
the same observable software.
|
||||
Status values are `Implemented`, `Partially Implemented`, `Planned`,
|
||||
and `Not Implemented`.
|
||||
|
||||
---
|
||||
```
|
||||
|
||||
## Detailed Section Guidelines
|
||||
|
||||
### 1. Product Overview
|
||||
|
||||
#### 1.1. Purpose
|
||||
|
||||
- **1.1.1** Clearly state what problem the project solves
|
||||
- **1.1.2** Describe the primary value proposition
|
||||
- **1.1.3** Identify the target users or audience
|
||||
- **1.1.4** Define the project's mission statement
|
||||
|
||||
#### 1.2. Scope
|
||||
|
||||
- **1.2.1** **In Scope**: List what the project will do
|
||||
- **1.2.2** **Out of Scope**: List what the project will NOT do
|
||||
- **1.2.3** Define clear boundaries for the project
|
||||
|
||||
#### 1.3. Non-Goals
|
||||
|
||||
- **1.3.1** Explicitly state what will not be implemented
|
||||
- **1.3.2** Clarify boundaries with related projects or systems
|
||||
- **1.3.3** Document conscious exclusions and rationale
|
||||
|
||||
#### 1.4. Document Conventions
|
||||
|
||||
- **1.4.1** Define terminology used throughout the document
|
||||
- **1.4.2** Establish status labels and their meanings
|
||||
- Implemented: Feature is complete and tested
|
||||
- Partially Implemented: Feature is partially complete
|
||||
- Planned: Feature is scheduled for implementation
|
||||
- Not Implemented: Feature is specified but not yet built
|
||||
- **1.4.3** Specify numbering conventions for sections and items
|
||||
- **1.4.4** Define RFC2119 keywords (MUST, SHALL, SHOULD, MAY, etc.)
|
||||
|
||||
### 2. Architecture
|
||||
|
||||
#### 2.1. Component Diagram
|
||||
|
||||
- **2.1.1** Provide ASCII or Mermaid diagram showing system architecture
|
||||
- **2.1.2** Label all major components and their relationships
|
||||
- **2.1.3** Show data flow between components with arrows or descriptions
|
||||
- **2.1.4** Indicate external dependencies and integrations
|
||||
|
||||
#### 2.2. Data Flow
|
||||
|
||||
- **2.2.1** Describe how data moves through the system end-to-end
|
||||
- **2.2.2** Identify key data transformations at each stage
|
||||
- **2.2.3** Note where data is persisted and in what format
|
||||
- **2.2.4** Document data validation and error handling points
|
||||
|
||||
#### 2.3. Module Structure
|
||||
|
||||
- **2.3.1** Document the complete directory and file structure
|
||||
- **2.3.2** Describe each module's responsibility and purpose
|
||||
- **2.3.3** Note dependencies between modules
|
||||
- **2.3.4** Indicate which modules are public vs internal APIs
|
||||
|
||||
### 3. Core Contracts
|
||||
|
||||
#### 3.1. Interfaces
|
||||
|
||||
- **3.1.1** Define purpose and role of each interface
|
||||
- **3.1.2** List all required methods or functions with signatures
|
||||
- **3.1.3** Document expected inputs, outputs, and return types
|
||||
- **3.1.4** Specify error handling expectations and exceptions
|
||||
- **3.1.5** Note version compatibility requirements
|
||||
|
||||
#### 3.2. Data Structures
|
||||
|
||||
- **3.2.1** Provide complete type definitions or JSON schemas
|
||||
- **3.2.2** Include concrete examples of each data structure
|
||||
- **3.2.3** Document validation rules and constraints
|
||||
- **3.2.4** Note backward and forward compatibility considerations
|
||||
- **3.2.5** Specify serialization formats (JSON, YAML, etc.)
|
||||
|
||||
### 4. Package and Runtime
|
||||
|
||||
#### 4.1. Dependencies
|
||||
|
||||
- **4.1.1** List runtime dependencies with exact version requirements
|
||||
- **4.1.2** List development dependencies for building and testing
|
||||
- **4.1.3** Document optional dependencies and their purposes
|
||||
- **4.1.4** Note any version conflicts or compatibility constraints
|
||||
|
||||
#### 4.2. Entry Points
|
||||
|
||||
- **4.2.1** Define CLI entry points and their usage
|
||||
- **4.2.2** Document library entry points for programmatic use
|
||||
- **4.2.3** Specify plugin or extension entry points if applicable
|
||||
- **4.2.4** Include examples of invoking each entry point
|
||||
|
||||
#### 4.3. Installation
|
||||
|
||||
- **4.3.1** Document all supported installation methods
|
||||
- **4.3.2** Specify platform and environment requirements
|
||||
- **4.3.3** Provide verification steps to confirm successful installation
|
||||
- **4.3.4** Note any post-installation configuration needed
|
||||
|
||||
### 5. Feature Requirements
|
||||
|
||||
Organize requirements by feature area. Each feature section should include:
|
||||
|
||||
- **5.X.1** Status: One of Implemented, Partially Implemented, Planned,
|
||||
Not Implemented
|
||||
- **5.X.2** Purpose: User stories and motivations for the feature
|
||||
- **5.X.3** Functional requirements: What the feature must do
|
||||
- **5.X.4** Non-functional requirements: Performance, security, etc.
|
||||
- **5.X.5** Acceptance criteria: How to verify the feature works
|
||||
|
||||
### 6. Configuration Files
|
||||
|
||||
For each configuration file, document:
|
||||
|
||||
- **6.1.1** Location: Both relative path and typical absolute paths
|
||||
- Note platform-specific locations (Windows vs Unix)
|
||||
- **6.1.2** Purpose: What the configuration controls
|
||||
- **6.1.3** Structure: Complete format and schema
|
||||
- **6.1.4** Default values: What settings are used if not specified
|
||||
- **6.1.5** Customization: How users can modify the configuration
|
||||
|
||||
### 7. Error Handling
|
||||
|
||||
#### 7.1. Error Classification
|
||||
|
||||
- **7.1.1** Transient vs permanent errors
|
||||
- **7.1.2** Recoverable vs unrecoverable errors
|
||||
- **7.1.3** User-facing vs internal errors
|
||||
- **7.1.4** Expected vs unexpected errors
|
||||
|
||||
#### 7.2. Error Reporting
|
||||
|
||||
- **7.2.1** Logging strategy and verbosity levels
|
||||
- **7.2.2** User notification mechanisms (CLI, UI, etc.)
|
||||
- **7.2.3** Error codes and their meanings
|
||||
- **7.2.4** Error message format and content
|
||||
|
||||
#### 7.3. Recovery Strategies
|
||||
|
||||
- **7.3.1** Automatic recovery mechanisms and retries
|
||||
- **7.3.2** Manual recovery procedures and user actions
|
||||
- **7.3.3** Data consistency guarantees during failures
|
||||
- **7.3.4** Rollback and undo capabilities
|
||||
|
||||
### 8. Testing
|
||||
|
||||
#### 8.1. Test Strategy
|
||||
|
||||
- **8.1.1** Testing pyramid: Unit, integration, system, E2E
|
||||
- **8.1.2** Test coverage targets for different test types
|
||||
- **8.1.3** Test environments and their configurations
|
||||
- **8.1.4** CI/CD integration for automated testing
|
||||
|
||||
#### 8.2. Test Cases
|
||||
|
||||
- **8.2.1** Critical path tests covering core functionality
|
||||
- **8.2.2** Edge case tests for boundary conditions
|
||||
- **8.2.3** Regression tests for known issues
|
||||
- **8.2.4** Performance and load tests if applicable
|
||||
|
||||
### 9. Deployment
|
||||
|
||||
#### 9.1. Release Process
|
||||
|
||||
- **9.1.1** Versioning scheme (SemVer, CalVer, etc.)
|
||||
- **9.1.2** Release checklist and required steps
|
||||
- **9.1.3** Rollback procedures for failed deployments
|
||||
- **9.1.4** Release notes and changelog requirements
|
||||
|
||||
#### 9.2. Distribution
|
||||
|
||||
- **9.2.1** Package repositories and distribution channels
|
||||
- **9.2.2** Installation methods for end users
|
||||
- **9.2.3** Update mechanisms and version management
|
||||
- **9.2.4** Offline or air-gapped installation options
|
||||
|
||||
## Sectionize-Markdown Integration
|
||||
|
||||
**CRITICAL**: Every SPEC.md file you create or modify MUST follow
|
||||
sectionize-markdown conventions completely.
|
||||
|
||||
Before finalizing any SPEC.md:
|
||||
|
||||
1. Apply numbered headings (## 1., ### 1.1, #### 1.1.1)
|
||||
2. Add bold IDs to all list items (- **1.1.1**)
|
||||
3. Generate or update the TOC with markdown links
|
||||
4. Ensure all anchors resolve correctly
|
||||
5. Verify no duplicate IDs exist
|
||||
|
||||
Use the sectionize-markdown skill as the definitive reference for
|
||||
all sectionizing conventions and requirements.
|
||||
|
||||
## Quality Checklist
|
||||
|
||||
Before marking a SPEC as complete, verify:
|
||||
|
||||
- [ ] **Structure**: All required sections are present
|
||||
- [ ] **Numbering**: All headings and list items are numbered correctly
|
||||
- [ ] **TOC**: Table of contents is present and accurate
|
||||
- [ ] **Status**: Every feature section has a status line
|
||||
- [ ] **Examples**: Code and configuration examples are provided
|
||||
- [ ] **Cross-references**: All internal links resolve correctly
|
||||
- [ ] **Consistency**: Terminology and formatting are consistent
|
||||
- [ ] **Completeness**: All acceptance criteria are defined
|
||||
- [ ] **Sectionized**: Follows sectionize-markdown conventions completely
|
||||
|
||||
## Maintenance Guidelines
|
||||
|
||||
- **Versioning**: Increment version in frontmatter with each update
|
||||
- **Changelog**: Maintain updated and created dates in frontmatter
|
||||
- **Review**: SPEC should be reviewed before major implementation
|
||||
- **Sync**: Keep SPEC in sync with implementation (update status)
|
||||
|
||||
## Tools and Automation
|
||||
|
||||
When working with SPEC files:
|
||||
|
||||
- Use scripts to verify section numbering consistency
|
||||
- Automate TOC generation to avoid manual errors
|
||||
- Validate all internal links before committing
|
||||
- Check for duplicate IDs across the entire document
|
||||
- Run markdown linters to ensure formatting quality
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
- **Unnumbered items**: Every list item MUST have a bold ID
|
||||
- **Broken anchors**: Always verify all TOC links resolve
|
||||
- **Inconsistent status**: Use consistent status labels throughout
|
||||
- **Missing examples**: Always include examples for data formats
|
||||
- **Ambiguous requirements**: Requirements must be testable and verifiable
|
||||
- **Overly verbose**: Be concise but comprehensive
|
||||
- **Under-specified**: Include enough detail for implementation
|
||||
@@ -1,10 +1,15 @@
|
||||
---
|
||||
name: index
|
||||
title: Wiki Index
|
||||
version: 1.0.0
|
||||
version: 1.1.0
|
||||
type: summary
|
||||
created: 2025-09-23
|
||||
updated: 2026-09-24
|
||||
updated: 2026-09-30
|
||||
confidence: trusted
|
||||
llm:
|
||||
- model: none
|
||||
contested: false
|
||||
contradictions: []
|
||||
---
|
||||
|
||||
# Wiki Index
|
||||
@@ -0,0 +1,19 @@
|
||||
---
|
||||
name: log
|
||||
title: Wiki Log
|
||||
version: 2.0.0
|
||||
type: summary
|
||||
created: DATE_PLACEHOLDER
|
||||
updated: DATE_PLACEHOLDER
|
||||
confidence: trusted
|
||||
llm:
|
||||
- model: none
|
||||
contested: false
|
||||
contradictions: []
|
||||
---
|
||||
|
||||
# Wiki Log
|
||||
|
||||
Chronological tracking of all actions performed on the wiki.
|
||||
|
||||
- DATE_PLACEHOLDER - Initialized the llm-wiki.
|
||||
@@ -1,34 +1,47 @@
|
||||
---
|
||||
name: llm-wiki-schema
|
||||
title: LLM Wiki Schema
|
||||
version: 1.2.0
|
||||
version: 1.6.1
|
||||
created: 2025-09-18
|
||||
updated: 2026-09-26
|
||||
updated: 2026-10-06
|
||||
type: summary
|
||||
tags: [wiki, schema]
|
||||
confidence: trusted
|
||||
llm:
|
||||
- model: claude-opus-5.5
|
||||
contested: false
|
||||
contradictions: []
|
||||
---
|
||||
|
||||
# LLM Wiki Schema
|
||||
|
||||
## Domain
|
||||
|
||||
Personal knowledge base and inventory management.
|
||||
A personal knowledge base. Replace this line with the wiki's subject
|
||||
and extend the tag taxonomy to match it.
|
||||
|
||||
## Conventions
|
||||
|
||||
- File names: lowercase, hyphens, no spaces (e.g., `llm-wiki-schema.md`)
|
||||
- Every wiki page starts with YAML frontmatter (required: `name`, `version`;
|
||||
optional: `title`, `created`, `updated`, `type`, `tags`, `sources`).
|
||||
- Use `[[wikilinks]]` to link between pages (minimum 2 outbound links per page).
|
||||
- When updating a page, always bump the `updated` date.
|
||||
- Every page starts with YAML frontmatter. Required on every file:
|
||||
`name`, `version`. Required on pages under `wiki/`: `confidence`,
|
||||
`llm`, `contested`, `contradictions`. Optional: `title`, `created`,
|
||||
`updated`, `type`, `tags`, `sources`. Ingested pages also carry
|
||||
`status`, `source_meta`, and the `ingest` provenance map.
|
||||
- Use markdown links (`[text](url)`) to link between pages (minimum 2 outbound links per page).
|
||||
- When updating a page, bump `version` and set `updated` to the edit date.
|
||||
- Every new page must be added to `index.md` under the correct section.
|
||||
- Every action must be appended to `log.md`.
|
||||
- Every action must be appended to `log.md`. In the first edit of a
|
||||
new year, move the previous year's entries to `logs/log-YYYY.md`.
|
||||
|
||||
## Frontmatter
|
||||
|
||||
`name` and `version` are required on every md file; all other fields are
|
||||
optional. This is the single canonical frontmatter schema, identical to
|
||||
project agent rule 10.1 (see SPEC 4.2.1).
|
||||
`name` and `version` are required on every md file. Every page under
|
||||
`wiki/` also requires the quality signals `confidence`, `llm`,
|
||||
`contested`, and `contradictions`. How to set them is section 15 of
|
||||
the [Fiona skill](../utils/skills/fiona/SKILL.md). All other fields
|
||||
are optional. This is the single canonical frontmatter schema.
|
||||
fmlint enforces `name` and `version` only.
|
||||
|
||||
```yaml
|
||||
---
|
||||
@@ -40,17 +53,34 @@ updated: YYYY-MM-DD # bump when editing
|
||||
type: entity | concept | comparison | query | summary | toc
|
||||
sources: [raw/articles/source-name.md]
|
||||
tags: [from taxonomy below]
|
||||
# Optional quality signals:
|
||||
# quality signals (required on pages under wiki/):
|
||||
confidence: trusted | high | medium | low | zero
|
||||
llm:
|
||||
- model: none # required; model that last edited the page
|
||||
- reasoning: high # when the model reports a level
|
||||
contested: true
|
||||
contradictions: [other-page-slug]
|
||||
# written by wiki ingest (Fiona skill section 18):
|
||||
status: raw | refined
|
||||
source_meta: {author: Name} # metadata read from the source
|
||||
ingest: # provenance; every key always present
|
||||
origin: https://host/path # URL or file path of the source
|
||||
origin_kind: url | file
|
||||
final_url: null # URL after redirects
|
||||
fetched: YYYY-MM-DDTHH:MM:SSZ
|
||||
media_type: text/html
|
||||
size_bytes: 0
|
||||
sha256: <64 lowercase hex> # of the raw copy; never hand-edited
|
||||
raw_reused: false
|
||||
converter: html # converter name, or agent
|
||||
conversion: complete | partial
|
||||
warnings: []
|
||||
tool: fiona-assistant <version>
|
||||
---
|
||||
```
|
||||
|
||||
## Tag Taxonomy
|
||||
|
||||
- inventory
|
||||
- box
|
||||
- wiki
|
||||
- schema
|
||||
|
||||
@@ -59,7 +89,7 @@ contradictions: [other-page-slug]
|
||||
- Create a page when an entity/concept appears in 2+ sources OR is central to
|
||||
one source.
|
||||
- Split pages over 200 lines.
|
||||
- Archive superseded pages to `_archive/`.
|
||||
- Move superseded pages to `wiki/_archive/` and fix every inbound link.
|
||||
|
||||
## Directory Structure
|
||||
|
||||
@@ -75,7 +105,8 @@ llm-wiki/
|
||||
│ ├── articles/ # Web articles, clippings
|
||||
│ ├── papers/ # PDFs, arXiv papers
|
||||
│ ├── transcripts/ # Meeting notes, interviews
|
||||
│ └── assets/ # Images, diagrams referenced by sources
|
||||
│ ├── assets/ # Images, diagrams referenced by sources
|
||||
│ └── other/ # Ingested files no converter handles
|
||||
├── utils/
|
||||
│ ├── personas/ # Agent personas configurations
|
||||
│ ├── scripts/ # Reusable utility scripts
|
||||
@@ -90,12 +121,13 @@ llm-wiki/
|
||||
│ ├── comparisons/ # Layer 2: Side-by-side analyses
|
||||
│ ├── queries/ # Layer 2: Filed query results worth keeping
|
||||
│ ├── sources/ # Layer 2: Source material backing wiki pages
|
||||
│ └── logs/ # Archived log files rotated out of log.md
|
||||
│ ├── logs/ # Archived log files rotated out of log.md
|
||||
│ └── _archive/ # Superseded pages, created when first needed
|
||||
└── table_of_contents.md # Auto-generated navigation index for quick lookup
|
||||
```
|
||||
|
||||
- **Layer 1 — Raw Sources:** `raw/` at the wiki root. The agent reads but never
|
||||
modifies these.
|
||||
- **Layer 1 — Raw Sources:** `raw/` at the wiki root. The agent adds new
|
||||
sources and never modifies existing ones.
|
||||
- **Layer 2 — The Wiki:** Agent-owned markdown files. Created, updated, and
|
||||
cross-referenced by the agent.
|
||||
- **Layer 3 — The Schema:** `SCHEMA.md` defines structure, conventions, and tag taxonomy.
|
||||
- **Layer 3 — The Schema:** `wiki/schema.md` (this file) defines structure, conventions, and tag taxonomy.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -7,12 +7,14 @@ from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import pathlib
|
||||
from datetime import datetime, timezone
|
||||
from datetime import UTC, datetime
|
||||
|
||||
import git
|
||||
from git.repo.base import Repo
|
||||
|
||||
from fiona.gittool import diff_report
|
||||
from fiona.logging_config import setup_logging
|
||||
from fiona.wiki.ignore import ensure_wiki_gitignore
|
||||
|
||||
|
||||
class GitTool:
|
||||
@@ -50,7 +52,7 @@ class GitTool:
|
||||
Returns:
|
||||
str: Formatted timestamp string.
|
||||
"""
|
||||
return datetime.now(tz=timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
|
||||
return datetime.now(tz=UTC).strftime("%Y-%m-%d %H:%M:%S")
|
||||
|
||||
@staticmethod
|
||||
def _get_tag_timestamp() -> str:
|
||||
@@ -62,7 +64,7 @@ class GitTool:
|
||||
Returns:
|
||||
str: Formatted timestamp string safe for git tags.
|
||||
"""
|
||||
return datetime.now(tz=timezone.utc).strftime("%Y-%m-%d_%H-%M-%S")
|
||||
return datetime.now(tz=UTC).strftime("%Y-%m-%d_%H-%M-%S")
|
||||
|
||||
def _is_working_tree_clean(self) -> bool:
|
||||
"""Check if the working tree is clean.
|
||||
@@ -73,7 +75,7 @@ class GitTool:
|
||||
# Check for modified/staged files
|
||||
if self.repo.is_dirty():
|
||||
return False
|
||||
|
||||
|
||||
# Check for untracked files
|
||||
untracked = self.repo.untracked_files
|
||||
return len(untracked) == 0
|
||||
@@ -170,7 +172,7 @@ class GitTool:
|
||||
tag for tag in self.repo.tags
|
||||
if tag.name.endswith(f"_{tag_name}") or tag.name == tag_name
|
||||
]
|
||||
|
||||
|
||||
if matching_tags:
|
||||
# Sort by commit date descending and take the most recent
|
||||
matching_tags.sort(
|
||||
@@ -178,7 +180,7 @@ class GitTool:
|
||||
reverse=True
|
||||
)
|
||||
tag_ref = matching_tags[0]
|
||||
|
||||
|
||||
if tag_ref is None:
|
||||
raise git.GitCommandError(
|
||||
f"Tag '{tag_name}' does not exist."
|
||||
@@ -189,7 +191,7 @@ class GitTool:
|
||||
|
||||
# Restore the tag's tree into the working directory and index
|
||||
self.repo.git.checkout(tag_ref.commit, "--", ".")
|
||||
|
||||
|
||||
# Create a new commit with the restored state
|
||||
new_commit = self.repo.index.commit(restore_message)
|
||||
|
||||
@@ -272,9 +274,9 @@ class GitTool:
|
||||
|
||||
@staticmethod
|
||||
def init_wiki_repo(repo_path: str | pathlib.Path, initial_message: str = "Initial llm-wiki commit") -> git.Repo:
|
||||
"""Initialize a git repository for llm-wiki with .gitignore for personal/.
|
||||
"""Initialize a git repository for llm-wiki and seed .gitignore.
|
||||
|
||||
Creates a .gitignore file that excludes the personal/ directory,
|
||||
Puts or appends the repository ignore patterns plus ``personal/``,
|
||||
stages all files, and makes the initial commit.
|
||||
|
||||
Args:
|
||||
@@ -288,26 +290,23 @@ class GitTool:
|
||||
git.InvalidGitRepositoryError: If git initialization fails.
|
||||
"""
|
||||
repo_path = pathlib.Path(repo_path).resolve()
|
||||
|
||||
|
||||
# Initialize the repo
|
||||
repo = GitTool.init_repo(repo_path)
|
||||
|
||||
# Create .gitignore to exclude personal/ directory
|
||||
gitignore_path = repo_path / ".gitignore"
|
||||
with open(gitignore_path, "w", encoding="utf-8") as f:
|
||||
f.write("# Exclude personal directory from git as per SPEC 3.2.0.2\n")
|
||||
f.write("personal/\n")
|
||||
|
||||
|
||||
# AI/LLM: 2026-09-30 - seed .gitignore from the repository ignore file.
|
||||
ensure_wiki_gitignore(repo_path)
|
||||
|
||||
# Add all files and make initial commit
|
||||
repo.git.add("--all")
|
||||
repo.index.commit(initial_message)
|
||||
|
||||
|
||||
return repo
|
||||
|
||||
|
||||
def main(): # pragma: no cover
|
||||
"""Command-line interface for the GitTool.
|
||||
|
||||
|
||||
Note: CLI is tested via integration tests through fiona-assistant git commands.
|
||||
"""
|
||||
parser = argparse.ArgumentParser(
|
||||
@@ -351,6 +350,30 @@ def main(): # pragma: no cover
|
||||
# status subcommand
|
||||
subparsers.add_parser("status", help="Show git status")
|
||||
|
||||
# diff subcommand
|
||||
diff_parser = subparsers.add_parser(
|
||||
"diff",
|
||||
help="Write constellation-themed HTML diff under wiki tmp/",
|
||||
)
|
||||
diff_parser.add_argument("--base", help="Base revision (default: origin/main or main/master)")
|
||||
diff_parser.add_argument("--head", default="HEAD", help="Head revision (default: HEAD)")
|
||||
diff_parser.add_argument("--title", help="Report H1 / page title")
|
||||
diff_parser.add_argument(
|
||||
"--subtitle",
|
||||
help="Extra header line (e.g. temporary review before push)",
|
||||
)
|
||||
diff_parser.add_argument(
|
||||
"-o",
|
||||
"--output",
|
||||
type=pathlib.Path,
|
||||
help="Output HTML path (default: <wiki>/tmp/<repo>-git-diff-<short>.html)",
|
||||
)
|
||||
diff_parser.add_argument(
|
||||
"--collapsed",
|
||||
action="store_true",
|
||||
help="Start with per-file sections collapsed",
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
setup_logging()
|
||||
@@ -380,6 +403,21 @@ def main(): # pragma: no cover
|
||||
elif args.subcommand == "status":
|
||||
status = git_tool.status()
|
||||
print(status)
|
||||
elif args.subcommand == "diff":
|
||||
diff_argv: list[str] = []
|
||||
if args.base:
|
||||
diff_argv.extend(["--base", args.base])
|
||||
if args.head != "HEAD":
|
||||
diff_argv.extend(["--head", args.head])
|
||||
if args.title:
|
||||
diff_argv.extend(["--title", args.title])
|
||||
if args.subtitle:
|
||||
diff_argv.extend(["--subtitle", args.subtitle])
|
||||
if args.output is not None:
|
||||
diff_argv.extend(["--output", str(args.output)])
|
||||
if args.collapsed:
|
||||
diff_argv.append("--collapsed")
|
||||
raise SystemExit(diff_report.main(diff_argv))
|
||||
except Exception as e:
|
||||
print(f"Error: {e}")
|
||||
raise SystemExit(1) from e
|
||||
@@ -0,0 +1,327 @@
|
||||
<!DOCTYPE html>
|
||||
<!-- Packaged layout for fiona.gittool.diff_report; placeholders filled at generation time. -->
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>{{PAGE_TITLE}}</title>
|
||||
<style>
|
||||
{{THEME_CSS}}
|
||||
* { box-sizing: border-box; }
|
||||
body {
|
||||
margin: 0;
|
||||
font-family: var(--font);
|
||||
background: var(--bg);
|
||||
color: var(--text);
|
||||
line-height: 1.45;
|
||||
transition: background-color 0.3s ease, color 0.3s ease;
|
||||
}
|
||||
header {
|
||||
background: var(--card);
|
||||
color: var(--text);
|
||||
padding: 1rem 1.25rem;
|
||||
border-bottom: 1px solid var(--border);
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
align-items: center;
|
||||
flex-wrap: wrap;
|
||||
gap: 1rem;
|
||||
}
|
||||
header .header-title {
|
||||
flex: 1;
|
||||
min-width: 200px;
|
||||
}
|
||||
header h1 { margin: 0 0 0.35rem; font-size: 1.4rem; font-weight: 600; }
|
||||
header p { margin: 0.25rem 0; opacity: 0.92; font-size: 0.9rem; }
|
||||
header code {
|
||||
background: color-mix(in srgb, var(--button-text) 15%, transparent);
|
||||
padding: 0.1rem 0.35rem;
|
||||
border-radius: 4px;
|
||||
}
|
||||
.theme-toggle {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 0.5rem;
|
||||
}
|
||||
.theme-toggle button {
|
||||
padding: 0.35rem 0.65rem;
|
||||
cursor: pointer;
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 6px;
|
||||
background: var(--button-bg);
|
||||
color: var(--button-text);
|
||||
font-size: 0.85rem;
|
||||
}
|
||||
.theme-toggle button:hover { background: var(--button-hover); }
|
||||
.layout {
|
||||
display: flex;
|
||||
align-items: stretch;
|
||||
min-height: calc(100vh - 8rem);
|
||||
}
|
||||
aside.sidebar {
|
||||
width: var(--sidebar-w);
|
||||
flex-shrink: 0;
|
||||
position: sticky;
|
||||
top: 0;
|
||||
align-self: flex-start;
|
||||
max-height: 100vh;
|
||||
overflow: auto;
|
||||
background: var(--card);
|
||||
border-right: 1px solid var(--border);
|
||||
padding: 0.75rem 0.5rem 1rem;
|
||||
}
|
||||
aside.sidebar h2 {
|
||||
margin: 0 0 0.5rem 0.65rem;
|
||||
font-size: 0.75rem;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.05em;
|
||||
color: var(--blue);
|
||||
}
|
||||
.file-tree {
|
||||
list-style: none;
|
||||
margin: 0;
|
||||
padding: 0 0 0 0.35rem;
|
||||
font-size: 0.8rem;
|
||||
}
|
||||
.file-tree ul {
|
||||
list-style: none;
|
||||
margin: 0;
|
||||
padding-left: 0.65rem;
|
||||
border-left: 1px solid var(--border);
|
||||
}
|
||||
.file-tree li { margin: 0.15rem 0; }
|
||||
.file-tree .folder-label {
|
||||
display: block;
|
||||
font-weight: 600;
|
||||
color: var(--text);
|
||||
padding: 0.1rem 0.35rem;
|
||||
}
|
||||
.file-tree a {
|
||||
display: block;
|
||||
padding: 0.15rem 0.35rem;
|
||||
border-radius: 4px;
|
||||
color: var(--accent);
|
||||
text-decoration: none;
|
||||
word-break: break-all;
|
||||
}
|
||||
.file-tree a:hover { background: var(--link-hover-bg); text-decoration: underline; }
|
||||
.file-tree a.active {
|
||||
background: var(--link-active-bg);
|
||||
font-weight: 600;
|
||||
color: var(--blue);
|
||||
}
|
||||
.file-tree .churn {
|
||||
font-size: 0.7rem;
|
||||
color: var(--muted);
|
||||
margin-left: 0.25rem;
|
||||
white-space: nowrap;
|
||||
}
|
||||
.churn .add { color: var(--churn-add); }
|
||||
.churn .del { color: var(--churn-del); }
|
||||
.file-icon {
|
||||
font-size: 0.8rem;
|
||||
margin-right: 0.35rem;
|
||||
opacity: 0.7;
|
||||
}
|
||||
.stat-fold {
|
||||
margin: 0.75rem 0.5rem 0;
|
||||
font-size: 0.75rem;
|
||||
}
|
||||
.stat-fold summary { cursor: pointer; color: var(--muted); }
|
||||
.stat-fold pre {
|
||||
margin: 0.35rem 0 0;
|
||||
font-size: 0.68rem;
|
||||
white-space: pre-wrap;
|
||||
max-height: 12rem;
|
||||
overflow: auto;
|
||||
background: var(--summary-bg);
|
||||
padding: 0.5rem;
|
||||
border-radius: 4px;
|
||||
border: 1px solid var(--border);
|
||||
}
|
||||
.content {
|
||||
flex: 1;
|
||||
min-width: 0;
|
||||
padding: 1rem 1.25rem 2.5rem;
|
||||
}
|
||||
.toolbar {
|
||||
font-size: 0.85rem;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 0.5rem;
|
||||
flex-wrap: wrap;
|
||||
background: var(--card);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 8px;
|
||||
padding: 0.65rem 0.85rem;
|
||||
margin-bottom: 1rem;
|
||||
}
|
||||
button {
|
||||
padding: 0.35rem 0.65rem;
|
||||
cursor: pointer;
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 6px;
|
||||
background: var(--button-bg);
|
||||
color: var(--button-text);
|
||||
}
|
||||
button:hover { background: var(--button-hover); }
|
||||
details.file {
|
||||
margin: 0.75rem 0;
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 6px;
|
||||
background: var(--card);
|
||||
scroll-margin-top: 0.75rem;
|
||||
}
|
||||
details.file:target {
|
||||
outline: 2px solid var(--accent);
|
||||
outline-offset: 2px;
|
||||
}
|
||||
details.file summary {
|
||||
cursor: pointer;
|
||||
padding: 0.6rem 0.75rem;
|
||||
font-weight: 600;
|
||||
background: var(--summary-bg);
|
||||
}
|
||||
pre.diff {
|
||||
margin: 0;
|
||||
padding: 0.75rem;
|
||||
font-size: 0.72rem;
|
||||
line-height: 1.0;
|
||||
overflow: auto;
|
||||
max-height: 70vh;
|
||||
}
|
||||
.diff-add { background: var(--diff-add-bg); display: block; margin: 0; padding: 0; }
|
||||
.diff-del { background: var(--diff-del-bg); display: block; margin: 0; padding: 0; }
|
||||
.diff-hunk { color: var(--diff-hunk); display: block; margin: 0; padding: 0; }
|
||||
.diff-meta { color: var(--diff-meta); display: block; margin: 0; padding: 0; }
|
||||
.diff-ctx { display: block; margin: 0; padding: 0; }
|
||||
@media (max-width: 900px) {
|
||||
.layout { flex-direction: column; }
|
||||
aside.sidebar {
|
||||
width: 100%;
|
||||
position: relative;
|
||||
max-height: 40vh;
|
||||
border-right: none;
|
||||
border-bottom: 1px solid var(--border);
|
||||
}
|
||||
}
|
||||
/* Dark mode theme - applied when data-theme="dark" (default) */
|
||||
[data-theme="dark"] {
|
||||
--bg: #1a1a1a;
|
||||
--card: #2d2d2d;
|
||||
--border: #404040;
|
||||
--text: #e0e0e0;
|
||||
--blue: #6ba1f5;
|
||||
--accent: #7aa7d6;
|
||||
--header-end: #c586c0;
|
||||
--muted: #9e9e9e;
|
||||
--summary-bg: #252525;
|
||||
--button-bg: #3d3d3d;
|
||||
--button-text: #e0e0e0;
|
||||
--button-hover: #4a4a4a;
|
||||
--churn-add: #7edd8a;
|
||||
--churn-del: #ff7b7b;
|
||||
--diff-add-bg: color-mix(in srgb, #7edd8a 22%, var(--card));
|
||||
--diff-del-bg: color-mix(in srgb, #ff7b7b 22%, var(--card));
|
||||
--link-hover-bg: color-mix(in srgb, #7aa7d6 18%, var(--card));
|
||||
--link-active-bg: color-mix(in srgb, #7aa7d6 22%, var(--card));
|
||||
}
|
||||
/* Light mode theme - applied when data-theme="light" */
|
||||
[data-theme="light"] {
|
||||
--bg: #ffffff;
|
||||
--card: #f8f9fa;
|
||||
--border: #e2e8f0;
|
||||
--text: #1a1a1a;
|
||||
--blue: #0d6efd;
|
||||
--accent: #0d6efd;
|
||||
--header-end: #6610f2;
|
||||
--muted: #6c757d;
|
||||
--summary-bg: #f8f9fa;
|
||||
--button-bg: #ffffff;
|
||||
--button-text: #1a1a1a;
|
||||
--button-hover: #e2e8f0;
|
||||
--churn-add: #15803d;
|
||||
--churn-del: #b91c1c;
|
||||
--diff-add-bg: color-mix(in srgb, #15803d 22%, var(--card));
|
||||
--diff-del-bg: color-mix(in srgb, #b91c1c 22%, var(--card));
|
||||
--link-hover-bg: color-mix(in srgb, #0d6efd 18%, var(--card));
|
||||
--link-active-bg: color-mix(in srgb, #0d6efd 22%, var(--card));
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<header>
|
||||
<div class="header-title">
|
||||
{{HEADER_BODY}}
|
||||
</div>
|
||||
{{THEME_TOGGLE_HTML}}
|
||||
</header>
|
||||
<div class="layout">
|
||||
<aside class="sidebar" aria-label="Changed files">
|
||||
<h2>Files ({{FILE_COUNT}})</h2>
|
||||
{{TREE_HTML}}
|
||||
<details class="stat-fold">
|
||||
<summary>git diff --stat</summary>
|
||||
<pre>{{STAT_BLOCK}}</pre>
|
||||
</details>
|
||||
</aside>
|
||||
<div class="content">
|
||||
<div class="toolbar">
|
||||
<button type="button" id="btn-expand-all">Expand all</button>
|
||||
<button type="button" id="btn-collapse-all">Collapse all</button>
|
||||
</div>
|
||||
{{FILES_HTML}}
|
||||
</div>
|
||||
</div>
|
||||
<script>
|
||||
(function () {
|
||||
// Theme toggle functionality - default is dark mode
|
||||
var themeToggle = document.getElementById('theme-toggle');
|
||||
if (themeToggle) {
|
||||
var htmlEl = document.documentElement;
|
||||
// Default to dark mode
|
||||
var savedTheme = localStorage.getItem('fiona-diff-theme') || 'dark';
|
||||
htmlEl.setAttribute('data-theme', savedTheme);
|
||||
themeToggle.addEventListener('click', function () {
|
||||
var currentTheme = htmlEl.getAttribute('data-theme');
|
||||
var newTheme = currentTheme === 'dark' ? 'light' : 'dark';
|
||||
htmlEl.setAttribute('data-theme', newTheme);
|
||||
localStorage.setItem('fiona-diff-theme', newTheme);
|
||||
});
|
||||
}
|
||||
|
||||
function openFromHash() {
|
||||
var id = location.hash ? location.hash.slice(1) : "";
|
||||
document.querySelectorAll(".file-tree a").forEach(function (a) {
|
||||
a.classList.toggle("active", a.getAttribute("href") === "#" + id);
|
||||
});
|
||||
if (!id) {
|
||||
return;
|
||||
}
|
||||
var el = document.getElementById(id);
|
||||
if (el && el.tagName === "DETAILS") {
|
||||
el.open = true;
|
||||
el.scrollIntoView({ behavior: "smooth", block: "start" });
|
||||
}
|
||||
}
|
||||
document.getElementById("btn-expand-all").addEventListener("click", function () {
|
||||
document.querySelectorAll("details.file").forEach(function (d) { d.open = true; });
|
||||
});
|
||||
document.getElementById("btn-collapse-all").addEventListener("click", function () {
|
||||
document.querySelectorAll("details.file").forEach(function (d) { d.open = false; });
|
||||
});
|
||||
document.querySelectorAll(".file-tree a").forEach(function (a) {
|
||||
a.addEventListener("click", function () {
|
||||
var id = a.getAttribute("href").slice(1);
|
||||
var el = document.getElementById(id);
|
||||
if (el && el.tagName === "DETAILS") {
|
||||
el.open = true;
|
||||
}
|
||||
});
|
||||
});
|
||||
window.addEventListener("hashchange", openFromHash);
|
||||
openFromHash();
|
||||
})();
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -18,6 +18,18 @@ import re
|
||||
|
||||
import yaml
|
||||
|
||||
# Default required fields for wiki pages
|
||||
DEFAULT_REQUIRED_FIELDS = ["title", "created", "updated", "type"]
|
||||
|
||||
# Pattern to match frontmatter delimiters
|
||||
# This pattern matches --- at start, then any content (including empty), then ---
|
||||
# Handles: ---\ncontent\n--- or ---\n--- or ---\n\n---
|
||||
FRONTMATTER_PATTERN = re.compile(r'^---\s*\n([\s\S]*?)(?:\n---\s*($|\n)|---\s*($|\n))')
|
||||
|
||||
# Pattern to match a single --- delimiter line
|
||||
DELIMITER_PATTERN = re.compile(r'^---\s*$', re.MULTILINE)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Exceptions
|
||||
# =============================================================================
|
||||
@@ -30,22 +42,6 @@ class RequiredFieldError(Exception):
|
||||
"""Exception raised when required frontmatter fields are missing."""
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Constants
|
||||
# =============================================================================
|
||||
|
||||
# Default required fields for wiki pages
|
||||
DEFAULT_REQUIRED_FIELDS = ["title", "created", "updated", "type"]
|
||||
|
||||
# Pattern to match frontmatter delimiters
|
||||
# This pattern matches --- at start, then any content (including empty), then ---
|
||||
# Handles: ---\ncontent\n--- or ---\n--- or ---\n\n---
|
||||
FRONTMATTER_PATTERN = re.compile(r'^---\s*\n([\s\S]*?)(?:\n---\s*($|\n)|---\s*($|\n))')
|
||||
|
||||
# Pattern to match a single --- delimiter line
|
||||
DELIMITER_PATTERN = re.compile(r'^---\s*$', re.MULTILINE)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Helper Functions
|
||||
# =============================================================================
|
||||
@@ -16,8 +16,9 @@ import re
|
||||
import sys
|
||||
import typing
|
||||
|
||||
from fiona.librarian.md_frontmatter import FrontmatterError, get_file_version
|
||||
from fiona.logging_config import load_settings, setup_logging
|
||||
from librarian.md_frontmatter import FrontmatterError, get_file_version
|
||||
from fiona.paths import llm_wiki_template
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -40,9 +41,7 @@ SETTINGS_NAME = "fiona-settings.json"
|
||||
ARTIFACT_DIR_NAMES = frozenset({"build", "dist", "__pycache__"})
|
||||
FILE_TOC_KEYS = frozenset({"min_heading", "max_heading", "sectionize", "ignore_files"})
|
||||
|
||||
DEFAULT_SETTINGS_FILE = (
|
||||
pathlib.Path(__file__).resolve().parent.parent / "llm-wiki" / "fiona-settings.json"
|
||||
)
|
||||
DEFAULT_SETTINGS_FILE = llm_wiki_template() / "fiona-settings.json"
|
||||
|
||||
|
||||
def load_toc_settings(settings_file: str | pathlib.Path | None = None) -> dict:
|
||||
@@ -95,9 +94,12 @@ def depth_setting(settings: dict, key: str) -> int | None:
|
||||
|
||||
|
||||
def slugify(text: str) -> str:
|
||||
"""Convert a header text into a GitHub-style anchor."""
|
||||
"""Convert a header text into a GitHub-style anchor (SPEC 7.3.1.1.3).
|
||||
|
||||
Each space becomes one hyphen; runs are kept so ``a | b`` is ``a--b``.
|
||||
"""
|
||||
slug = re.sub(r"[^\w\s-]", "", text.lower())
|
||||
return re.sub(r"\s+", "-", slug.strip())
|
||||
return re.sub(r"\s", "-", slug.strip())
|
||||
|
||||
|
||||
def excluded(text: str, patterns: list[typing.Pattern]) -> bool:
|
||||
@@ -109,7 +111,7 @@ def read_headers(path: pathlib.Path) -> list[tuple[int, str]]:
|
||||
"""Return (level, text) for every header in a Markdown file, skipping code fences."""
|
||||
headers: list[tuple[int, str]] = []
|
||||
in_fence = False
|
||||
# CURSOR: utf-8-sig drops a leading BOM that would hide the first header.
|
||||
# AI/LLM: utf-8-sig drops a leading BOM that would hide the first header.
|
||||
for line in path.read_text(encoding="utf-8-sig", errors="replace").splitlines():
|
||||
if FENCE_RE.match(line):
|
||||
in_fence = not in_fence
|
||||
@@ -163,15 +165,19 @@ def build_toc(
|
||||
) -> str:
|
||||
"""Build a table of contents that nests folders, files and their headers.
|
||||
|
||||
Generated by AI/LLM: 2026-10-02 - frontmatter name and version for fmlint.
|
||||
|
||||
Args:
|
||||
folder: The folder the table of contents covers.
|
||||
files: Relative md file paths to list (as returned by collect_md).
|
||||
max_header_depth: Maximum header level to list per file
|
||||
(3 = h1 through h3). None or 0 mean unlimited.
|
||||
"""
|
||||
# Add YAML frontmatter
|
||||
today = datetime.datetime.now(datetime.timezone.utc).date().isoformat()
|
||||
# AI/LLM: name and version are required by fmlint on every markdown file.
|
||||
today = datetime.datetime.now(datetime.UTC).date().isoformat()
|
||||
frontmatter = f"""---
|
||||
name: Table of Contents
|
||||
version: 1.0.0
|
||||
title: Table of Contents
|
||||
created: {today}
|
||||
updated: {today}
|
||||
@@ -183,7 +189,7 @@ type: toc
|
||||
for name in files:
|
||||
parts = tuple(name.split("/"))
|
||||
folders = parts[:-1]
|
||||
# CURSOR: emit a bullet only for folder levels that differ from the previous entry.
|
||||
# AI/LLM: emit a bullet only for folder levels that differ from the previous entry.
|
||||
for depth in range(len(folders)):
|
||||
if folders[: depth + 1] != previous[: depth + 1]:
|
||||
lines.append("{}- {}".format(" " * depth, folders[depth]))
|
||||
@@ -215,7 +221,7 @@ class FileTocError(Exception):
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class FileTocConfig:
|
||||
"""Parameters for per-file TOC generation (SPEC 4.8.2).
|
||||
"""Parameters for per-file TOC generation (SPEC 7.3.1.2).
|
||||
|
||||
Attributes:
|
||||
min_heading: Smallest heading level included (never below 2;
|
||||
@@ -360,7 +366,7 @@ def merge_marker_config(lines: list[str], base: FileTocConfig) -> FileTocConfig:
|
||||
"""Apply the existing marker's params over the base config.
|
||||
|
||||
The marker records this TOC's own settings and has priority
|
||||
(SPEC 4.8.2): a `sectionized` marker keeps section numbering on
|
||||
(SPEC 7.3.1.2): a `sectionized` marker keeps section numbering on
|
||||
and the recorded heading range sticks, so regenerations keep the
|
||||
TOC's identity regardless of settings changes. Fields the marker
|
||||
does not record (bare legacy markers, plain markers without
|
||||
@@ -674,7 +680,7 @@ def _file_toc_star(sectionized: bool) -> None:
|
||||
|
||||
|
||||
def file_toc_main(argv: list[str]) -> None:
|
||||
"""CLI for per-file TOC generation (SPEC 4.8.2)."""
|
||||
"""CLI for per-file TOC generation (SPEC 7.3.1.2)."""
|
||||
parser = argparse.ArgumentParser(
|
||||
prog="toc_generator file",
|
||||
description="Generate an in-file table of contents below the H1 heading.",
|
||||
@@ -748,7 +754,7 @@ def main() -> None:
|
||||
for folder in target_folders(args.directory, patterns, args.recursive_toc):
|
||||
files = collect_md(folder, patterns, not args.non_recursive, max_depth=max_dir_depth)
|
||||
if files:
|
||||
# CURSOR: newline="\n" keeps Unix line endings on every platform.
|
||||
# AI/LLM: newline="\n" keeps Unix line endings on every platform.
|
||||
with open(folder / TOC_NAME, "w", encoding="utf-8", newline="\n") as toc:
|
||||
toc.write(build_toc(folder, files, max_header_depth=max_header_depth))
|
||||
logger.info("wrote %s (%d file(s))", folder / TOC_NAME, len(files))
|
||||
@@ -1,4 +1,4 @@
|
||||
"""Shared logging configuration for all Fiona tools (SPEC 5.1).
|
||||
"""Shared logging configuration for all Fiona tools (SPEC 7.17).
|
||||
|
||||
Diagnostics go through the standard `logging` module with the level
|
||||
resolved from (highest precedence first):
|
||||
@@ -13,20 +13,25 @@ records go to stderr.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import ctypes
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
from fiona.paths import llm_wiki_template
|
||||
|
||||
# AI/LLM: Windows-only; module-level so PYC001 allows it (SPEC 6.4.2).
|
||||
if os.name == "nt":
|
||||
import msvcrt
|
||||
|
||||
DEFAULT_LOG_LEVEL = "WARNING"
|
||||
DEFAULT_SETTINGS_FILE = (
|
||||
pathlib.Path(__file__).resolve().parent.parent / "llm-wiki" / "fiona-settings.json"
|
||||
)
|
||||
DEFAULT_SETTINGS_FILE = llm_wiki_template() / "fiona-settings.json"
|
||||
# Tool package loggers that carry the level explicitly: pymarkdown
|
||||
# resets the root logger to WARNING mid-run, which would otherwise
|
||||
# silence tool diagnostics (same pattern as run_sanity, see SPEC 4.7.1).
|
||||
TOOL_LOGGER_NAMES = ("fiona", "librarian", "mdoctor", "python_ruff", "wiki")
|
||||
# silence tool diagnostics (same pattern as run_sanity, see SPEC 7.17).
|
||||
TOOL_LOGGER_NAMES = ("fiona",)
|
||||
|
||||
LOG_FORMAT = "%(levelname)s %(name)s: %(message)s"
|
||||
FILE_LOG_FORMAT = "%(asctime)s %(levelname)s %(name)s: %(message)s"
|
||||
@@ -85,6 +90,68 @@ def load_log_level(settings: dict | None) -> str:
|
||||
return candidate.upper()
|
||||
|
||||
|
||||
def _open_shared_append(path: str, encoding: str):
|
||||
"""Open a log file that Windows can delete while the handle is open.
|
||||
|
||||
Generated by AI/LLM: 2026-09-29 - the default open() omits
|
||||
FILE_SHARE_DELETE, so a temp directory that still has fiona.log
|
||||
open cannot be removed.
|
||||
|
||||
Args:
|
||||
path: Log file path.
|
||||
encoding: Text encoding for the stream.
|
||||
|
||||
Returns:
|
||||
A text file object opened for append.
|
||||
"""
|
||||
if os.name != "nt":
|
||||
return open(path, "a", encoding=encoding)
|
||||
kernel32 = ctypes.WinDLL("kernel32", use_last_error=True)
|
||||
kernel32.CreateFileW.argtypes = [
|
||||
ctypes.c_wchar_p,
|
||||
ctypes.c_uint32,
|
||||
ctypes.c_uint32,
|
||||
ctypes.c_void_p,
|
||||
ctypes.c_uint32,
|
||||
ctypes.c_uint32,
|
||||
ctypes.c_void_p,
|
||||
]
|
||||
kernel32.CreateFileW.restype = ctypes.c_void_p
|
||||
kernel32.SetFilePointer.argtypes = [
|
||||
ctypes.c_void_p,
|
||||
ctypes.c_long,
|
||||
ctypes.c_void_p,
|
||||
ctypes.c_uint32,
|
||||
]
|
||||
kernel32.SetFilePointer.restype = ctypes.c_uint32
|
||||
handle = kernel32.CreateFileW(
|
||||
path,
|
||||
0x40000000, # GENERIC_WRITE
|
||||
0x1 | 0x2 | 0x4, # FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE
|
||||
None,
|
||||
4, # OPEN_ALWAYS
|
||||
0x80, # FILE_ATTRIBUTE_NORMAL
|
||||
None,
|
||||
)
|
||||
invalid = ctypes.c_void_p(-1).value
|
||||
if handle in (None, invalid):
|
||||
raise OSError(ctypes.get_last_error(), f"Cannot open log file {path}")
|
||||
kernel32.SetFilePointer(handle, 0, None, 2) # FILE_END
|
||||
fd = msvcrt.open_osfhandle(handle, os.O_APPEND)
|
||||
return os.fdopen(fd, "a", encoding=encoding, closefd=True)
|
||||
|
||||
|
||||
class _SharedFileHandler(logging.FileHandler):
|
||||
"""File log handler that does not block deletion of its directory.
|
||||
|
||||
Generated by AI/LLM: 2026-09-29 - Windows tests and temp wikis remove
|
||||
the working directory while fiona.log is still attached.
|
||||
"""
|
||||
|
||||
def _open(self):
|
||||
return _open_shared_append(self.baseFilename, self.encoding or "utf-8")
|
||||
|
||||
|
||||
def _remove_file_handler() -> None:
|
||||
"""Detach (and close) the managed file log handler, if any."""
|
||||
for handler in list(logging.getLogger().handlers):
|
||||
@@ -117,7 +184,7 @@ def _configure_file_log(fiona_settings: dict, console_level: str) -> None:
|
||||
)
|
||||
file_level = console_level
|
||||
try:
|
||||
handler = logging.FileHandler(location, encoding="utf-8")
|
||||
handler = _SharedFileHandler(location, encoding="utf-8")
|
||||
except OSError as error:
|
||||
logging.getLogger(__name__).warning("Cannot open log file %s: %s", location, error)
|
||||
return
|
||||
|
||||
@@ -4,22 +4,54 @@ MDoctor - A tool for diagnosing and managing the LLM wiki.
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import contextlib
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import pathlib
|
||||
import sys
|
||||
import tempfile
|
||||
from typing import Any, ClassVar
|
||||
|
||||
from pymarkdown.api import PyMarkdownApi, PyMarkdownApiException
|
||||
|
||||
from customlinter.base import BaseLinter, LintViolation
|
||||
from fiona.customlinter.base import BaseLinter, LintViolation
|
||||
from fiona.librarian.librarian import Librarian
|
||||
from fiona.logging_config import setup_logging
|
||||
from librarian.librarian import Librarian
|
||||
from fiona.paths import llm_wiki_template
|
||||
|
||||
DEFAULT_SETTINGS_FILE = (
|
||||
pathlib.Path(__file__).resolve().parent.parent / "llm-wiki" / "fiona-settings.json"
|
||||
)
|
||||
DEFAULT_SETTINGS_FILE = llm_wiki_template() / "fiona-settings.json"
|
||||
# AI/LLM: MD013 defaults used when fiona-settings.json omits the keys.
|
||||
MD013_DEFAULTS: dict[str, int | bool] = {
|
||||
"line_length": 100,
|
||||
"tables": False,
|
||||
}
|
||||
|
||||
|
||||
@contextlib.contextmanager
|
||||
def _pymarkdown_reopenable_temp_files():
|
||||
"""Keep pymarkdown temp files reopenable on Windows.
|
||||
|
||||
Generated by AI/LLM: 2026-09-29 - pymarkdown closes a NamedTemporaryFile
|
||||
and opens that name again. Windows delete-on-close denies the second open.
|
||||
|
||||
Yields:
|
||||
None.
|
||||
"""
|
||||
if os.name != "nt":
|
||||
yield
|
||||
return
|
||||
original = tempfile.NamedTemporaryFile
|
||||
|
||||
def _named_temporary_file(*args, **kwargs):
|
||||
kwargs["delete"] = False
|
||||
return original(*args, **kwargs)
|
||||
|
||||
tempfile.NamedTemporaryFile = _named_temporary_file
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
tempfile.NamedTemporaryFile = original
|
||||
|
||||
|
||||
class ProgrammaticLintViolation:
|
||||
@@ -37,20 +69,20 @@ class ProgrammaticLintViolation:
|
||||
|
||||
class MDoctor(BaseLinter):
|
||||
"""Core class for diagnosing wiki content and operations."""
|
||||
|
||||
|
||||
# File extensions to scan
|
||||
file_extensions: ClassVar[list[str]] = [".md"]
|
||||
|
||||
|
||||
# Settings key path in fiona-settings.json
|
||||
settings_key_path: ClassVar[list[str]] = ["markdown", "pymarkdown"]
|
||||
|
||||
|
||||
# Human-readable linter name for report headers
|
||||
linter_name: ClassVar[str] = "PyMarkdownLint"
|
||||
|
||||
|
||||
# Violation codes - pymarkdown rules are dynamic, so we use an empty dict
|
||||
# and populate descriptions from the API
|
||||
VIOLATION_CODES: ClassVar[dict[str, dict[str, Any]]] = {}
|
||||
|
||||
|
||||
def __init__(self, settings_file: str | pathlib.Path | None = None) -> None:
|
||||
self.librarian = Librarian()
|
||||
self.api = PyMarkdownApi()
|
||||
@@ -61,10 +93,17 @@ class MDoctor(BaseLinter):
|
||||
settings_file = DEFAULT_SETTINGS_FILE
|
||||
self.settings_file = pathlib.Path(settings_file)
|
||||
self.respect_gitignore = False
|
||||
|
||||
self._md013_tables = bool(MD013_DEFAULTS["tables"])
|
||||
self._table_lines: dict[pathlib.Path, set[int]] = {}
|
||||
self._gitignore_dirs: dict[pathlib.Path, set[str]] = {}
|
||||
|
||||
# Call parent init
|
||||
super().__init__(settings_file=self.settings_file)
|
||||
|
||||
|
||||
# AI/LLM: seed MD013 before the settings file so omitted keys keep these defaults.
|
||||
for name, value in MD013_DEFAULTS.items():
|
||||
self._set_plugin_property("md013", name, value)
|
||||
|
||||
if self.settings_file.exists():
|
||||
with open(self.settings_file, encoding="utf-8") as f:
|
||||
settings = json.load(f)
|
||||
@@ -90,34 +129,54 @@ class MDoctor(BaseLinter):
|
||||
if rule_id == "extensions" or not isinstance(properties, dict):
|
||||
continue
|
||||
for name, value in properties.items():
|
||||
property_name = f"plugins.{rule_id}.{name}"
|
||||
if isinstance(value, bool):
|
||||
self.api.set_boolean_property(property_name, value)
|
||||
elif isinstance(value, int):
|
||||
self.api.set_integer_property(property_name, value)
|
||||
else:
|
||||
self.api.set_string_property(property_name, value)
|
||||
self._set_plugin_property(rule_id, name, value)
|
||||
|
||||
def _set_plugin_property(self, rule_id: str, name: str, value: bool | int | str) -> None:
|
||||
"""Set one pymarkdown plugin property from a JSON value.
|
||||
|
||||
Generated by AI/LLM: 2026-09-29 - one setter for MD013 defaults and
|
||||
fiona-settings.json overrides. Bool is checked before int because
|
||||
bool is a subclass of int.
|
||||
|
||||
Args:
|
||||
rule_id: PyMarkdown rule id, such as ``md013``.
|
||||
name: Property name under that rule.
|
||||
value: Boolean, integer, or string property value.
|
||||
"""
|
||||
property_name = f"plugins.{rule_id}.{name}"
|
||||
if isinstance(value, bool):
|
||||
self.api.set_boolean_property(property_name, value)
|
||||
elif isinstance(value, int):
|
||||
self.api.set_integer_property(property_name, value)
|
||||
else:
|
||||
self.api.set_string_property(property_name, value)
|
||||
if rule_id == "md013" and name == "tables" and isinstance(value, bool):
|
||||
self._md013_tables = value
|
||||
|
||||
def lint_file(self, filepath: pathlib.Path | str) -> list[LintViolation]:
|
||||
"""Lint a single markdown file using PyMarkdownApi.
|
||||
|
||||
|
||||
Args:
|
||||
filepath: Path to the markdown file to lint.
|
||||
|
||||
|
||||
Returns:
|
||||
List of LintViolation objects.
|
||||
"""
|
||||
resolved_filepath = pathlib.Path(filepath).resolve()
|
||||
|
||||
if self._ignored_by_gitignore(resolved_filepath.parent, resolved_filepath):
|
||||
return []
|
||||
|
||||
# Use pymarkdown API to scan the file
|
||||
scan_result = self.api.scan_path(
|
||||
path_to_scan=str(resolved_filepath),
|
||||
recurse_if_directory=False,
|
||||
respect_gitignore=self.respect_gitignore
|
||||
**self._pymarkdown_path_options(resolved_filepath)
|
||||
)
|
||||
|
||||
|
||||
violations = []
|
||||
for failure in scan_result.scan_failures:
|
||||
if self._skip_table_md013(resolved_filepath, failure.line_number, failure.rule_id):
|
||||
continue
|
||||
violation = LintViolation(
|
||||
filepath=resolved_filepath,
|
||||
line=failure.line_number,
|
||||
@@ -128,7 +187,7 @@ class MDoctor(BaseLinter):
|
||||
severity="error"
|
||||
)
|
||||
violations.append(violation)
|
||||
|
||||
|
||||
return violations
|
||||
|
||||
def list_all_md_files(self, directory: str|pathlib.Path) -> list:
|
||||
@@ -141,11 +200,11 @@ class MDoctor(BaseLinter):
|
||||
|
||||
# Resolve the filepath to absolute path for scanning
|
||||
resolved_filepath = pathlib.Path(filepath).resolve()
|
||||
|
||||
|
||||
scan_result = api.scan_path(
|
||||
path_to_scan=str(resolved_filepath),
|
||||
recurse_if_directory=True,
|
||||
respect_gitignore=self.respect_gitignore
|
||||
**self._pymarkdown_path_options(resolved_filepath)
|
||||
)
|
||||
violations = []
|
||||
for failure in scan_result.scan_failures:
|
||||
@@ -158,17 +217,21 @@ class MDoctor(BaseLinter):
|
||||
base_path = resolved_filepath.parent
|
||||
else:
|
||||
base_path = resolved_filepath
|
||||
|
||||
|
||||
relative_path = str(scan_file_path.relative_to(base_path))
|
||||
|
||||
|
||||
# If the scan_file is the same as the filepath being scanned, return "."
|
||||
if scan_file_path == resolved_filepath:
|
||||
relative_path = "."
|
||||
|
||||
|
||||
except ValueError:
|
||||
# If relative path fails, use the absolute path
|
||||
relative_path = str(scan_file_path)
|
||||
|
||||
|
||||
if self._ignored_by_gitignore(resolved_filepath, scan_file_path):
|
||||
continue
|
||||
if self._skip_table_md013(scan_file_path, failure.line_number, failure.rule_id):
|
||||
continue
|
||||
violations.append({
|
||||
"scan_file": relative_path,
|
||||
"line": failure.line_number,
|
||||
@@ -181,6 +244,182 @@ class MDoctor(BaseLinter):
|
||||
|
||||
return violations
|
||||
|
||||
def _skip_table_md013(self, path: pathlib.Path, line_number: int, rule_id: str) -> bool:
|
||||
"""Drop MD013 on an unfenced pipe row when tables are ignored.
|
||||
|
||||
Generated by AI/LLM: 2026-09-29 - pymarkdown does not tokenize pipe
|
||||
rows while markdown-tables stays disabled, so plugins.md013.tables
|
||||
never sees them.
|
||||
|
||||
Args:
|
||||
path: Markdown file that produced the violation.
|
||||
line_number: 1-based line of the violation.
|
||||
rule_id: PyMarkdown rule id.
|
||||
|
||||
Returns:
|
||||
True when this hit should be omitted from the report.
|
||||
"""
|
||||
if self._md013_tables or rule_id != "MD013":
|
||||
return False
|
||||
return line_number in self._table_line_numbers(path)
|
||||
|
||||
def _table_line_numbers(self, path: pathlib.Path) -> set[int]:
|
||||
"""Return 1-based line numbers of unfenced GFM table rows.
|
||||
|
||||
Generated by AI/LLM: 2026-09-29 - identify pipe rows MD013 should skip.
|
||||
|
||||
Args:
|
||||
path: Markdown file to read.
|
||||
|
||||
Returns:
|
||||
Line numbers whose stripped text starts with `|` and contains
|
||||
another `|`, outside fenced code.
|
||||
"""
|
||||
resolved = path.resolve()
|
||||
cached = self._table_lines.get(resolved)
|
||||
if cached is not None:
|
||||
return cached
|
||||
numbers: set[int] = set()
|
||||
in_fence = False
|
||||
try:
|
||||
text = resolved.read_text(encoding="utf-8")
|
||||
except OSError:
|
||||
self._table_lines[resolved] = numbers
|
||||
return numbers
|
||||
for number, raw in enumerate(text.splitlines(), start=1):
|
||||
stripped = raw.lstrip()
|
||||
if stripped.startswith(("```", "~~~")):
|
||||
in_fence = not in_fence
|
||||
continue
|
||||
if in_fence:
|
||||
continue
|
||||
body = raw.strip()
|
||||
if body.startswith("|") and body.count("|") >= 2:
|
||||
numbers.add(number)
|
||||
self._table_lines[resolved] = numbers
|
||||
return numbers
|
||||
|
||||
def _ignored_by_gitignore(self, scan_root: pathlib.Path, file_path: pathlib.Path) -> bool:
|
||||
"""Skip a file that sits under a directory named in .gitignore.
|
||||
|
||||
Generated by AI/LLM: 2026-09-29 - pymarkdown only applies
|
||||
.gitignore inside a git checkout. Sanity still has to skip
|
||||
virtualenv markdown when the folder is not a repository.
|
||||
|
||||
Args:
|
||||
scan_root: Directory passed to the scan, or a file's parent.
|
||||
file_path: Markdown file to test.
|
||||
|
||||
Returns:
|
||||
True when respect_gitignore is on and a path segment matches
|
||||
a directory entry in the nearest .gitignore.
|
||||
"""
|
||||
if not self.respect_gitignore:
|
||||
return False
|
||||
root = self._gitignore_root(scan_root)
|
||||
if root is None:
|
||||
return False
|
||||
try:
|
||||
relative = file_path.resolve().relative_to(root)
|
||||
except ValueError:
|
||||
return False
|
||||
return any(part in self._gitignore_dir_names(root) for part in relative.parts)
|
||||
|
||||
def _gitignore_root(self, start: pathlib.Path) -> pathlib.Path | None:
|
||||
"""Find the nearest .gitignore at or above start.
|
||||
|
||||
Generated by AI/LLM: 2026-09-29 - locate the ignore file for a scan.
|
||||
|
||||
Args:
|
||||
start: File or directory being scanned.
|
||||
|
||||
Returns:
|
||||
Directory that contains .gitignore, or None.
|
||||
"""
|
||||
directory = start if start.is_dir() else start.parent
|
||||
for candidate in [directory, *directory.parents]:
|
||||
if (candidate / ".gitignore").is_file():
|
||||
return candidate
|
||||
return None
|
||||
|
||||
def _gitignore_dir_names(self, root: pathlib.Path) -> set[str]:
|
||||
"""Return directory names listed in root/.gitignore.
|
||||
|
||||
Generated by AI/LLM: 2026-09-29 - read directory entries such as
|
||||
`.venvs/` without invoking git.
|
||||
|
||||
Args:
|
||||
root: Directory that may contain .gitignore.
|
||||
|
||||
Returns:
|
||||
Names with no glob characters. Comments and negations are
|
||||
ignored.
|
||||
"""
|
||||
resolved = root.resolve()
|
||||
cached = self._gitignore_dirs.get(resolved)
|
||||
if cached is not None:
|
||||
return cached
|
||||
names: set[str] = set()
|
||||
gitignore = resolved / ".gitignore"
|
||||
try:
|
||||
lines = gitignore.read_text(encoding="utf-8").splitlines()
|
||||
except OSError:
|
||||
self._gitignore_dirs[resolved] = names
|
||||
return names
|
||||
for line in lines:
|
||||
entry = line.strip()
|
||||
if not entry or entry.startswith(("#", "!")):
|
||||
continue
|
||||
name = entry.strip("/")
|
||||
if not name or "/" in name or any(char in name for char in "*?["):
|
||||
continue
|
||||
names.add(name)
|
||||
self._gitignore_dirs[resolved] = names
|
||||
return names
|
||||
|
||||
def _pymarkdown_path_options(self, path: pathlib.Path) -> dict[str, Any]:
|
||||
"""Keep PyMarkdown from starting git.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - directory excludes replace git check-ignore.
|
||||
|
||||
Args:
|
||||
path: File or directory about to be scanned or fixed.
|
||||
|
||||
Returns:
|
||||
Keyword arguments for ``scan_path`` and ``fix_path``.
|
||||
"""
|
||||
return {
|
||||
"respect_gitignore": False,
|
||||
"exclude_patterns": self._exclude_patterns(path),
|
||||
}
|
||||
|
||||
def _exclude_patterns(self, path: pathlib.Path) -> list[str] | None:
|
||||
"""Return directory globs PyMarkdown can apply without git.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - prune ignored directories during the walk.
|
||||
|
||||
Patterns are relative to the working directory. A scan outside
|
||||
that directory gets no patterns, because the matcher rejects
|
||||
those paths. The post-scan directory check still applies.
|
||||
|
||||
Args:
|
||||
path: File or directory about to be scanned or fixed.
|
||||
|
||||
Returns:
|
||||
Glob list, or None when the path is outside the working directory.
|
||||
"""
|
||||
root = path if path.is_dir() else path.parent
|
||||
try:
|
||||
root.resolve().relative_to(pathlib.Path.cwd().resolve())
|
||||
except ValueError:
|
||||
return None
|
||||
names = {".git"}
|
||||
if self.respect_gitignore:
|
||||
git_root = self._gitignore_root(root)
|
||||
if git_root is not None:
|
||||
names.update(self._gitignore_dir_names(git_root))
|
||||
return [f"{name}/" for name in sorted(names)]
|
||||
|
||||
def fix_file_programmatically(self, filepath: str) -> list[str]:
|
||||
"""Apply PyMarkdown auto-fixes to a markdown file or directory.
|
||||
|
||||
@@ -197,27 +436,28 @@ class MDoctor(BaseLinter):
|
||||
api = self.api
|
||||
|
||||
resolved_filepath = pathlib.Path(filepath).resolve()
|
||||
fix_result = api.fix_path(
|
||||
path_to_scan=str(resolved_filepath),
|
||||
recurse_if_directory=True,
|
||||
respect_gitignore=self.respect_gitignore
|
||||
)
|
||||
with _pymarkdown_reopenable_temp_files():
|
||||
fix_result = api.fix_path(
|
||||
path_to_scan=str(resolved_filepath),
|
||||
recurse_if_directory=True,
|
||||
**self._pymarkdown_path_options(resolved_filepath)
|
||||
)
|
||||
return [str(pathlib.Path(f).resolve()) for f in fix_result.files_fixed]
|
||||
|
||||
def compress_violations(
|
||||
self, violations: list, put_extra: bool = False, target: str | pathlib.Path | None = None
|
||||
):
|
||||
"""Compress violations using the shared BaseLinter format.
|
||||
|
||||
|
||||
For backward compatibility, this method accepts:
|
||||
- List of violation dicts from scan_file_programmatically (old format)
|
||||
- List of LintViolation objects from lint_directory (new format)
|
||||
|
||||
|
||||
Args:
|
||||
violations: List of violation dicts or LintViolation objects.
|
||||
put_extra: Whether to include extra information in violation messages (old format only).
|
||||
target: Target directory path.
|
||||
|
||||
|
||||
Returns:
|
||||
Dictionary with compressed violations in shared format.
|
||||
"""
|
||||
@@ -232,7 +472,7 @@ class MDoctor(BaseLinter):
|
||||
# Construct the message with or without extra
|
||||
extra = f' {violation["extra"]}' if put_extra and violation.get("extra") else ''
|
||||
message = f'{violation["line"]}:{violation["column"]} {violation["rule_id"]}{extra}'
|
||||
|
||||
|
||||
# Create a LintViolation object
|
||||
# Note: filepath will be set to the scan_file, but we need to resolve it
|
||||
# For now, we'll use the scan_file as a string identifier
|
||||
@@ -246,24 +486,24 @@ class MDoctor(BaseLinter):
|
||||
severity="error"
|
||||
)
|
||||
lint_violations.append(lint_violation)
|
||||
|
||||
|
||||
# Store the rule info in VIOLATION_CODES for the base class to use
|
||||
if violation["rule_id"] not in self.VIOLATION_CODES:
|
||||
self.VIOLATION_CODES[violation["rule_id"]] = {
|
||||
"rule_name": violation["rule_name"],
|
||||
"description": violation["description"]
|
||||
}
|
||||
|
||||
|
||||
# Use the base class compress_violations method
|
||||
return super().compress_violations(lint_violations, target)
|
||||
|
||||
|
||||
def format_violations(self, compressed_violations: dict) -> str:
|
||||
"""Format violations using the shared BaseLinter format.
|
||||
|
||||
|
||||
Args:
|
||||
compressed_violations: Compressed violations dict from compress_violations.
|
||||
|
||||
|
||||
Returns:
|
||||
Formatted string with violation report.
|
||||
"""
|
||||
@@ -281,10 +521,10 @@ class MDoctor(BaseLinter):
|
||||
Formatted string with violation report
|
||||
"""
|
||||
directory_path = pathlib.Path(directory).resolve()
|
||||
|
||||
|
||||
# Use the standard lint_directory from base class
|
||||
violations = self.lint_directory(directory_path)
|
||||
|
||||
|
||||
# Compress and format using base class methods
|
||||
compressed_violations = self.compress_violations(violations, target=directory_path)
|
||||
return self.format_compressed_violations(compressed_violations)
|
||||
@@ -0,0 +1,21 @@
|
||||
"""Filesystem locations for packaged Fiona data.
|
||||
|
||||
Generated by AI/LLM: 2026-09-29 - one resolver for the llm-wiki template.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.resources
|
||||
import pathlib
|
||||
|
||||
|
||||
def llm_wiki_template() -> pathlib.Path:
|
||||
"""Return the packaged llm-wiki template directory.
|
||||
|
||||
Generated by AI/LLM: 2026-09-29 - load package data with importlib.resources.
|
||||
|
||||
Returns:
|
||||
Filesystem path of ``fiona/data/llm_wiki``.
|
||||
"""
|
||||
resource = importlib.resources.files("fiona").joinpath("data", "llm_wiki")
|
||||
return pathlib.Path(str(resource))
|
||||
@@ -8,7 +8,7 @@ from typing import Any, ClassVar
|
||||
|
||||
import ruff
|
||||
|
||||
from customlinter.base import BaseLinter, LintViolation
|
||||
from fiona.customlinter.base import BaseLinter, LintViolation
|
||||
from fiona.logging_config import setup_logging
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
"""Sanity Check Runner for Fiona-Assistant project.
|
||||
|
||||
Runs its checks through the shared SanityRunner engine
|
||||
(src/sanity_runner, SPEC 4.9.5): concurrent execution, per-check
|
||||
(src/sanity_runner, SPEC 7.14.1): concurrent execution, per-check
|
||||
timings, and the common report format. This module keeps only the
|
||||
repository-specific checks, their summaries, and the report
|
||||
presentation.
|
||||
@@ -17,19 +17,17 @@ import time
|
||||
from collections.abc import Callable, Sequence
|
||||
from pathlib import Path
|
||||
|
||||
import sanity_runner
|
||||
from customlinter.fmlint import FrontmatterLinter
|
||||
from customlinter.jsonlint import JsonLinter
|
||||
from customlinter.pycustomlint import PythonCustomLinter
|
||||
from mdoctor.mdoctor import MDoctor
|
||||
from python_ruff.python_ruff import PythonRuff
|
||||
from run_tests import TestRunner
|
||||
from fiona import sanity_runner
|
||||
from fiona.customlinter.fmlint import FrontmatterLinter
|
||||
from fiona.customlinter.jsonlint import JsonLinter
|
||||
from fiona.customlinter.pycustomlint import PythonCustomLinter
|
||||
from fiona.mdoctor.mdoctor import MDoctor
|
||||
from fiona.python_ruff.python_ruff import PythonRuff
|
||||
from fiona.run_tests import TestRunner
|
||||
|
||||
# Configure logging
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
SETTINGS_FILE = Path(__file__).resolve().parent / "settings_dev.json"
|
||||
|
||||
# Progress display defaults (overridable via the "progress" settings
|
||||
# section, see load_progress_settings)
|
||||
PROGRESS_BAR_WIDTH = 15
|
||||
@@ -37,6 +35,22 @@ DEFAULT_PROGRESS_INTERVAL_SECONDS = 3.0
|
||||
DEFAULT_EXPECTED_SECONDS = 30.0
|
||||
|
||||
|
||||
def dev_settings_file() -> Path:
|
||||
"""Return scripts/settings_dev.json from the checkout that contains this file.
|
||||
|
||||
Generated by AI/LLM: 2026-09-29 - development settings live outside the package.
|
||||
|
||||
Returns:
|
||||
The settings file when this module sits in a checkout, otherwise a
|
||||
path relative to the current working directory.
|
||||
"""
|
||||
for parent in Path(__file__).resolve().parents:
|
||||
candidate = parent / "scripts" / "settings_dev.json"
|
||||
if candidate.is_file():
|
||||
return candidate
|
||||
return Path("scripts") / "settings_dev.json"
|
||||
|
||||
|
||||
def configure_logging() -> None:
|
||||
"""Configure logging from the LOG_LEVEL environment variable.
|
||||
|
||||
@@ -57,15 +71,16 @@ def configure_logging() -> None:
|
||||
# pymarkdown resets the root logger to WARNING during lint scans,
|
||||
# which would silence the runner's debug output mid-run; the module
|
||||
# loggers therefore also carry the level explicitly
|
||||
for logger_name in (__name__, "run_sanity", "run_tests"):
|
||||
for logger_name in (__name__, "fiona.run_tests"):
|
||||
logging.getLogger(logger_name).setLevel(level)
|
||||
|
||||
|
||||
def load_required_coverage(settings_file: str | Path = SETTINGS_FILE) -> float:
|
||||
def load_required_coverage(settings_file: str | Path | None = None) -> float:
|
||||
"""Load the required test coverage percentage from the settings file.
|
||||
|
||||
Args:
|
||||
settings_file: Path to the JSON settings file.
|
||||
settings_file: Path to the JSON settings file. The dev settings
|
||||
file is used when this is omitted.
|
||||
|
||||
Returns:
|
||||
Required coverage percentage for pytests.
|
||||
@@ -74,16 +89,19 @@ def load_required_coverage(settings_file: str | Path = SETTINGS_FILE) -> float:
|
||||
KeyError: If the settings file has no "required-coverage" entry.
|
||||
json.JSONDecodeError: If the settings file is not valid JSON.
|
||||
"""
|
||||
if settings_file is None:
|
||||
settings_file = dev_settings_file()
|
||||
with open(settings_file, encoding="utf-8") as f:
|
||||
settings = json.load(f)
|
||||
return float(settings["pytests"]["required-coverage"])
|
||||
|
||||
|
||||
def load_progress_settings(settings_file: str | Path = SETTINGS_FILE) -> dict:
|
||||
def load_progress_settings(settings_file: str | Path | None = None) -> dict:
|
||||
"""Load the progress display settings from the settings file.
|
||||
|
||||
Args:
|
||||
settings_file: Path to the JSON settings file.
|
||||
settings_file: Path to the JSON settings file. The dev settings
|
||||
file is used when this is omitted.
|
||||
|
||||
Returns:
|
||||
dict: The display settings with:
|
||||
@@ -92,8 +110,13 @@ def load_progress_settings(settings_file: str | Path = SETTINGS_FILE) -> dict:
|
||||
Each falls back to its default when the "progress" section or
|
||||
the individual key is absent.
|
||||
"""
|
||||
with open(settings_file, encoding="utf-8") as f:
|
||||
settings = json.load(f)
|
||||
if settings_file is None:
|
||||
settings_file = dev_settings_file()
|
||||
try:
|
||||
with open(settings_file, encoding="utf-8") as f:
|
||||
settings = json.load(f)
|
||||
except FileNotFoundError:
|
||||
settings = {}
|
||||
progress = settings.get("progress", {})
|
||||
return {
|
||||
"interval_seconds": float(
|
||||
@@ -261,7 +284,7 @@ def run_pytests():
|
||||
(or the raw message when no traceback details exist)
|
||||
"""
|
||||
runner = TestRunner()
|
||||
result = runner.run_pytest("src/tests", verbose=True, capture_output=True, with_coverage=True)
|
||||
result = runner.run_pytest("tests", verbose=True, capture_output=True, with_coverage=True)
|
||||
|
||||
required_coverage = load_required_coverage()
|
||||
coverage_percent = result.get("coverage_percent")
|
||||
@@ -293,7 +316,7 @@ def run_pytests():
|
||||
unreported_failures,
|
||||
)
|
||||
|
||||
lines = ["Pytest Results", "TARGET: src/tests", ""]
|
||||
lines = ["Pytest Results", "TARGET: tests", ""]
|
||||
lines.append(f"Tests: {result['passed_count']}/{result['total_count']} passed")
|
||||
for file_name in sorted(file_results):
|
||||
counts = file_results[file_name]
|
||||
@@ -387,8 +410,9 @@ def run_ruff():
|
||||
def run_pycustomlint():
|
||||
"""Run the Python custom linter programmatically using PythonCustomLinter.
|
||||
|
||||
Scans Python files in the src tree for custom rule violations,
|
||||
currently inline imports (project agent rule 4.1, python-tdd skill).
|
||||
Scans Python files in the src tree for custom rule violations:
|
||||
inline imports (project agent rule 4.1, python-tdd skill) and
|
||||
module-level assignments after the first function or class (SPEC 6.4.3).
|
||||
|
||||
Returns:
|
||||
dict: A dictionary containing the lint results with:
|
||||
@@ -396,7 +420,7 @@ def run_pycustomlint():
|
||||
- violation_count: number of violations found
|
||||
- stdout: AI-friendly report in the unified linter format
|
||||
(Violations header, TARGET, DEFINITIONS, FILE, SUMMARY;
|
||||
see SPEC 4.9.3)
|
||||
see SPEC 6.1)
|
||||
"""
|
||||
linter = PythonCustomLinter()
|
||||
|
||||
@@ -429,7 +453,7 @@ def run_frontmatter_lint():
|
||||
linter = FrontmatterLinter()
|
||||
|
||||
# Lint markdown files in llm-wiki directory only (per project agent rule 10.1)
|
||||
llm_wiki_path = "./src/llm-wiki"
|
||||
llm_wiki_path = "./src/fiona/data/llm_wiki"
|
||||
violations_dict = linter.lint_directory(llm_wiki_path)
|
||||
|
||||
# Get violation count from meta
|
||||
@@ -602,7 +626,7 @@ def coverage_met(check_result: dict) -> bool:
|
||||
def format_duration(seconds: float) -> str:
|
||||
"""Format a duration in seconds as zero-padded MM:SS.
|
||||
|
||||
Delegates to the shared SanityRunner engine (SPEC 4.9.5).
|
||||
Delegates to the shared SanityRunner engine (SPEC 7.14.1).
|
||||
|
||||
Args:
|
||||
seconds: Duration in seconds.
|
||||
@@ -616,7 +640,7 @@ def format_duration(seconds: float) -> str:
|
||||
def format_check_summary(check_name: str, check_result: dict) -> str:
|
||||
"""Format a single check result as a human/AI-friendly line.
|
||||
|
||||
Delegates to the shared SanityRunner engine (SPEC 4.9.5); the
|
||||
Delegates to the shared SanityRunner engine (SPEC 7.14.1); the
|
||||
pytests and coverage summaries plug in as formatters.
|
||||
|
||||
Args:
|
||||
@@ -22,6 +22,8 @@ logger = logging.getLogger(__name__)
|
||||
class TestRunner:
|
||||
"""A test runner for Fiona tools that can be used both manually and in CI."""
|
||||
|
||||
__test__ = False
|
||||
|
||||
def __init__(self, base_dir: str | None = None):
|
||||
"""
|
||||
Initialize the test runner.
|
||||
@@ -450,7 +452,7 @@ class TestRunner:
|
||||
Args:
|
||||
test_path: Path to the test file or directory
|
||||
verbose: Whether to run in verbose mode
|
||||
capture_output: Whether to suppress the pytest output (for CI)
|
||||
capture_output: When True, hide pytest stdout/stderr (default False)
|
||||
with_coverage: Whether to run with coverage (default: True)
|
||||
ignore: Pattern to ignore (e.g., "test_sanity.py")
|
||||
|
||||
@@ -464,7 +466,7 @@ class TestRunner:
|
||||
if verbose:
|
||||
cmd.append("-v")
|
||||
if with_coverage:
|
||||
cmd.extend(["--cov=src", "--cov-config=.coveragerc", "--cov-report=term"])
|
||||
cmd.extend(["--cov=src/fiona", "--cov-report=term"])
|
||||
if ignore:
|
||||
cmd.extend(["--ignore-glob", f"*{ignore}*"])
|
||||
|
||||
@@ -477,13 +479,14 @@ class TestRunner:
|
||||
# environment variables (PY_COLORS, FORCE_COLOR) would break them
|
||||
cmd.append("--color=no")
|
||||
|
||||
# Set PYTHONPATH to include src directory for customlinter imports
|
||||
# Set PYTHONPATH to include src so the fiona package resolves.
|
||||
# AI/LLM: 2026-09-30 - Windows joins PATH entries with os.pathsep.
|
||||
env = os.environ.copy()
|
||||
src_dir = os.path.abspath(os.path.dirname(__file__))
|
||||
src_dir = str(pathlib.Path(__file__).resolve().parent.parent)
|
||||
if os.path.exists(src_dir):
|
||||
current_pythonpath = env.get("PYTHONPATH", "")
|
||||
if current_pythonpath:
|
||||
env["PYTHONPATH"] = f"{src_dir}:{current_pythonpath}"
|
||||
env["PYTHONPATH"] = f"{src_dir}{os.pathsep}{current_pythonpath}"
|
||||
else:
|
||||
env["PYTHONPATH"] = src_dir
|
||||
|
||||
@@ -491,7 +494,10 @@ class TestRunner:
|
||||
completed = subprocess.run(cmd, capture_output=True, text=True, check=False, env=env)
|
||||
|
||||
if not capture_output:
|
||||
print(completed.stdout)
|
||||
if completed.stdout:
|
||||
print(completed.stdout, end="" if completed.stdout.endswith("\n") else "\n")
|
||||
if completed.stderr:
|
||||
print(completed.stderr, end="" if completed.stderr.endswith("\n") else "\n", file=sys.stderr)
|
||||
|
||||
logger.debug("pytest command: %s", " ".join(str(part) for part in cmd))
|
||||
logger.debug("pytest return code: %s", completed.returncode)
|
||||
@@ -572,7 +578,7 @@ class TestRunner:
|
||||
True if all tests passed, False otherwise
|
||||
"""
|
||||
test_paths = [
|
||||
"src/tests"
|
||||
"tests"
|
||||
]
|
||||
|
||||
all_passed = True
|
||||
@@ -603,29 +609,56 @@ def main():
|
||||
"""Main function for command-line execution."""
|
||||
parser = argparse.ArgumentParser(description="Run Fiona tools tests.")
|
||||
parser.add_argument("--verbose", action="store_true", help="Run tests in verbose mode")
|
||||
parser.add_argument("--ci", action="store_true", help="Run in CI mode (capture output)")
|
||||
parser.add_argument(
|
||||
"--ci",
|
||||
action="store_true",
|
||||
help="Non-interactive CI mode: verbose pytest with full output to the log",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--non-interactive",
|
||||
action="store_true",
|
||||
help="Alias for --ci",
|
||||
)
|
||||
parser.add_argument("--test", help="Run specific test file or directory")
|
||||
parser.add_argument("--skip-cov", action="store_true", help="Skip coverage")
|
||||
|
||||
args = parser.parse_args()
|
||||
# AI/LLM: use `is True` — bool(MagicMock()) is True in unit tests.
|
||||
ci_mode = args.ci is True or args.non_interactive is True
|
||||
verbose = args.verbose is True or ci_mode
|
||||
capture_output = False
|
||||
|
||||
# Initialize test runner
|
||||
runner = TestRunner()
|
||||
|
||||
# Determine if we should run with coverage
|
||||
with_coverage = not args.skip_cov
|
||||
with_coverage = args.skip_cov is not True
|
||||
|
||||
def _print_failures() -> None:
|
||||
"""Echo parsed failure details after pytest output (CI-friendly)."""
|
||||
for detail in runner.test_results.values():
|
||||
for entry in detail.get("failed_tests") or []:
|
||||
test_id = entry.get("id", "?")
|
||||
message = entry.get("message", "")
|
||||
print(f"FAILED {test_id} - {message}", file=sys.stderr)
|
||||
if entry.get("location"):
|
||||
print(f" at {entry['location']}", file=sys.stderr)
|
||||
if entry.get("assert"):
|
||||
print(f" {entry['assert']}", file=sys.stderr)
|
||||
|
||||
if args.test:
|
||||
# Run specific test
|
||||
result = runner.run_pytest(args.test, args.verbose, args.ci, with_coverage)
|
||||
result = runner.run_pytest(args.test, verbose, capture_output, with_coverage)
|
||||
if not result["passed"]:
|
||||
print(f"Test failed: {args.test}")
|
||||
_print_failures()
|
||||
print(f"Test failed: {args.test}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
else:
|
||||
# Run all tests
|
||||
all_passed = runner.run_all_tests(args.verbose, args.ci, with_coverage)
|
||||
all_passed = runner.run_all_tests(verbose, capture_output, with_coverage)
|
||||
if not all_passed:
|
||||
print("Some tests failed") #TODO: this is not very informative, need improving
|
||||
_print_failures()
|
||||
print("Some tests failed", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
# Print summary
|
||||
@@ -3,10 +3,10 @@
|
||||
Provides `SanityRunner`, the generalized class shared by the
|
||||
repository sanity runner (run_sanity) and the wiki sanity command
|
||||
(wiki.sanity_manager): concurrent check execution with per-check
|
||||
timings and the common AI-friendly report format (SPEC 4.9.3/4.9.4).
|
||||
timings and the common AI-friendly report format (SPEC 6.1/7.14.1).
|
||||
"""
|
||||
|
||||
from sanity_runner.sanity_runner import (
|
||||
from fiona.sanity_runner.sanity_runner import (
|
||||
SanityRunner,
|
||||
format_duration,
|
||||
format_violation_count,
|
||||
@@ -1,14 +1,14 @@
|
||||
"""Shared sanity runner engine (SPEC 4.9.5).
|
||||
"""Shared sanity runner engine (SPEC 7.14.1).
|
||||
|
||||
This module provides `SanityRunner`, the generalized class behind both
|
||||
sanity runners in the project:
|
||||
|
||||
- the repository sanity runner (`src/run_sanity.py`, SPEC 4.9)
|
||||
- the wiki sanity command (`src/wiki/sanity_manager.py`, SPEC 3.1.3)
|
||||
- the repository sanity runner (`fiona.run_sanity`, SPEC 7.14)
|
||||
- the wiki sanity command (`src/wiki/sanity_manager.py`, SPEC 7.1.3)
|
||||
|
||||
The class owns the common machinery: concurrent check execution with
|
||||
per-check wall-clock timings (SPEC 4.9.4) and the shared AI-friendly
|
||||
report structure (SPEC 4.9.3) — one-line check summaries, the aligned
|
||||
per-check wall-clock timings (SPEC 7.14.1) and the shared AI-friendly
|
||||
report structure (SPEC 6.1) — one-line check summaries, the aligned
|
||||
`Results:` table rows, the `SANITY PASSED (MM:SS)` banner, and the
|
||||
`Sanity Failed` footer. Per-check specifics (summary wording, extra
|
||||
table rows, failure details) plug in as callables, so each runner
|
||||
@@ -35,6 +35,19 @@ POLL_INTERVAL_SECONDS = 0.5
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
__all__ = [
|
||||
"CheckCallable",
|
||||
"CheckResult",
|
||||
"ExtraResultLine",
|
||||
"FailureDetails",
|
||||
"HeartbeatCallback",
|
||||
"ProgressCallback",
|
||||
"SanityRunner",
|
||||
"SummaryFormatter",
|
||||
"format_duration",
|
||||
"format_violation_count",
|
||||
]
|
||||
|
||||
|
||||
def format_duration(seconds: float) -> str:
|
||||
"""Format a duration in seconds as zero-padded MM:SS.
|
||||
@@ -98,7 +111,7 @@ class SanityRunner:
|
||||
|
||||
Each result carries its wall-clock `duration_seconds`; times of
|
||||
parallel checks overlap and need not sum to the total
|
||||
(SPEC 4.9.4). The returned mapping keeps the registry order
|
||||
(SPEC 7.14.1). The returned mapping keeps the registry order
|
||||
regardless of completion order. The check lifecycle (start and
|
||||
completion with duration) is logged at INFO.
|
||||
|
||||
@@ -267,21 +280,3 @@ class SanityRunner:
|
||||
lines.extend(self.failure_details(check_name, check_result))
|
||||
|
||||
return lines
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Public API
|
||||
# =============================================================================
|
||||
|
||||
__all__ = [
|
||||
"CheckCallable",
|
||||
"CheckResult",
|
||||
"ExtraResultLine",
|
||||
"FailureDetails",
|
||||
"HeartbeatCallback",
|
||||
"ProgressCallback",
|
||||
"SanityRunner",
|
||||
"SummaryFormatter",
|
||||
"format_duration",
|
||||
"format_violation_count",
|
||||
]
|
||||
@@ -0,0 +1,5 @@
|
||||
"""Wiki package for Fiona-Assistant.
|
||||
|
||||
Submodules are imported directly. This package does not import them
|
||||
here, so git_tool can load wiki.ignore without a circular import.
|
||||
"""
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,367 @@
|
||||
"""Wiki housekeeping routine (SPEC 4.8, 7.2).
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - one call for LLM wiki maintenance.
|
||||
|
||||
`fiona-assistant housekeeping` runs the maintenance steps in a fixed
|
||||
order and reports, per step, a status (`PASS`, `WARN`, `FAIL`), a
|
||||
summary, detail lines, and the next actions for the LLM. The routine
|
||||
gathers what needs work; it writes only `wiki_graph.html`.
|
||||
|
||||
Steps, in order (`STEP_NAMES`):
|
||||
|
||||
- `check-update`: core files behind the packaged template (read-only).
|
||||
- `merge`: files whose stem ends with `-update` that `wiki update` left for a merge.
|
||||
- `ingest`: pages under `wiki/` still at `status: raw`.
|
||||
- `sanity`: every `wiki sanity` check.
|
||||
- `graph`: regenerate `wiki_graph.html` and list orphan pages.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
from fiona.librarian.md_frontmatter import FrontmatterError, parse_frontmatter
|
||||
from fiona.wiki.graph_generator import wiki_graph_main
|
||||
from fiona.wiki.sanity_manager import SETTINGS_FILE, run_wiki_checks
|
||||
from fiona.wiki.update_manager import (
|
||||
UPDATE_SUFFIX,
|
||||
check_wiki_updates,
|
||||
get_update_ignore_patterns,
|
||||
path_is_update_ignored,
|
||||
)
|
||||
|
||||
STEP_NAMES = ("check-update", "merge", "ingest", "sanity", "graph")
|
||||
GRAPH_FILE = "wiki_graph.html"
|
||||
UPDATE_FILE_SUFFIX = f"{UPDATE_SUFFIX}.md"
|
||||
# AI/LLM: read-only zones sit at the wiki root; .git is skipped anywhere
|
||||
ROOT_SKIP_DIRS = frozenset({"personal", "raw"})
|
||||
|
||||
__all__ = [
|
||||
"GRAPH_FILE",
|
||||
"STEP_NAMES",
|
||||
"check_update_step",
|
||||
"format_housekeeping_report",
|
||||
"graph_step",
|
||||
"housekeeping_main",
|
||||
"ingest_step",
|
||||
"merge_step",
|
||||
"run_housekeeping",
|
||||
"sanity_step",
|
||||
]
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Helpers
|
||||
# =============================================================================
|
||||
|
||||
def _step(name: str, status: str, summary: str, details: list[str] | None = None,
|
||||
next_steps: list[str] | None = None) -> dict:
|
||||
"""Build one step result.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - shared step shape (SPEC 4.8).
|
||||
"""
|
||||
return {
|
||||
"name": name,
|
||||
"status": status,
|
||||
"summary": summary,
|
||||
"details": details or [],
|
||||
"next": next_steps or [],
|
||||
}
|
||||
|
||||
|
||||
def _relative(path: str | Path, wiki_root: Path) -> str:
|
||||
"""Return a path relative to the wiki root, with `/` separators.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - report paths from the wiki root.
|
||||
"""
|
||||
return Path(os.path.relpath(path, wiki_root)).as_posix()
|
||||
|
||||
|
||||
def _frontmatter(path: Path) -> dict:
|
||||
"""Return a page's frontmatter, or an empty dict when it cannot be read.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - sanity reports broken YAML itself.
|
||||
"""
|
||||
try:
|
||||
frontmatter = parse_frontmatter(path)
|
||||
except (FrontmatterError, UnicodeDecodeError):
|
||||
return {}
|
||||
return frontmatter if isinstance(frontmatter, dict) else {}
|
||||
|
||||
|
||||
def _wiki_files(wiki_root: Path) -> list[Path]:
|
||||
"""List files outside read-only zones and `.git`, sorted.
|
||||
|
||||
Modified by AI/LLM: 2026-10-01 - merge walks every suffix, not only markdown.
|
||||
|
||||
Args:
|
||||
wiki_root: Root path of the wiki.
|
||||
|
||||
Returns:
|
||||
list[Path]: Files found.
|
||||
"""
|
||||
found = []
|
||||
for current, dirs, files in os.walk(wiki_root):
|
||||
at_root = Path(current) == wiki_root
|
||||
dirs[:] = [
|
||||
name for name in dirs
|
||||
if name != ".git" and not (at_root and name in ROOT_SKIP_DIRS)
|
||||
]
|
||||
found.extend(Path(current) / name for name in files)
|
||||
return sorted(found)
|
||||
|
||||
|
||||
def _markdown_files(wiki_root: Path, subdir: str = "") -> list[Path]:
|
||||
"""List markdown files outside read-only zones and `.git`, sorted.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - pruned walk for the merge and ingest steps.
|
||||
|
||||
Args:
|
||||
wiki_root: Root path of the wiki.
|
||||
subdir: Optional directory, relative to the root, to walk instead.
|
||||
|
||||
Returns:
|
||||
list[Path]: Markdown files found.
|
||||
"""
|
||||
start = wiki_root / subdir
|
||||
found = []
|
||||
for current, dirs, files in os.walk(start):
|
||||
at_root = Path(current) == wiki_root
|
||||
dirs[:] = [
|
||||
name for name in dirs
|
||||
if name != ".git" and not (at_root and name in ROOT_SKIP_DIRS)
|
||||
]
|
||||
found.extend(Path(current) / name for name in files if name.endswith(".md"))
|
||||
return sorted(found)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Steps
|
||||
# =============================================================================
|
||||
|
||||
def check_update_step(wiki_root: Path) -> dict:
|
||||
"""Report core files behind the template without writing (SPEC 7.2.2).
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - read-only wiki check-update.
|
||||
"""
|
||||
summary = check_wiki_updates(wiki_root)
|
||||
details = [f"update: {_relative(r.file_path, wiki_root)}" for r in summary.files_to_update]
|
||||
details += [
|
||||
f"conflict: {_relative(r.file_path, wiki_root)} -> {_relative(r.update_path, wiki_root)}"
|
||||
for r in summary.conflicts
|
||||
]
|
||||
details += [f"add: {path.as_posix()}" for path in summary.files_to_add]
|
||||
text = (
|
||||
f"{len(summary.files_to_update)} update(s), {len(summary.conflicts)} conflict(s), "
|
||||
f"{len(summary.files_to_add)} addition(s)"
|
||||
)
|
||||
if not details:
|
||||
return _step("check-update", "PASS", text)
|
||||
return _step("check-update", "WARN", text, details, [
|
||||
"Ask the user, then run fiona-assistant wiki update --dry-run and wiki update --yes."
|
||||
])
|
||||
|
||||
|
||||
def merge_step(wiki_root: Path) -> dict:
|
||||
"""List update files that wait for an LLM merge (SPEC 7.2.3).
|
||||
|
||||
Modified by AI/LLM: 2026-10-02 - skip update.ignore_files and corefiles.json.
|
||||
"""
|
||||
details, actions = [], []
|
||||
ignore_patterns = get_update_ignore_patterns(wiki_root)
|
||||
for update_file in _wiki_files(wiki_root):
|
||||
if not update_file.stem.endswith(UPDATE_SUFFIX):
|
||||
continue
|
||||
if path_is_update_ignored(_relative(update_file, wiki_root), ignore_patterns):
|
||||
continue
|
||||
target = update_file.with_name(
|
||||
update_file.stem[: -len(UPDATE_SUFFIX)] + update_file.suffix
|
||||
)
|
||||
update_rel = _relative(update_file, wiki_root)
|
||||
target_rel = _relative(target, wiki_root)
|
||||
missing = "" if target.exists() else " (target missing)"
|
||||
details.append(f"{update_rel} -> {target_rel}{missing}")
|
||||
version = _frontmatter(update_file).get("version") or "the template version"
|
||||
actions.append(
|
||||
f"Merge {update_rel} into {target_rel}: keep the owner's edits, "
|
||||
f"take the template changes, set version to at least {version}, "
|
||||
f"then delete {update_rel}."
|
||||
)
|
||||
text = f"{len(details)} update file(s) wait for a merge"
|
||||
return _step("merge", "FAIL" if details else "PASS", text, details, actions)
|
||||
|
||||
|
||||
def ingest_step(wiki_root: Path) -> dict:
|
||||
"""List pages under wiki/ still at `status: raw` (SPEC 7.2.4).
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - pages waiting for a refine.
|
||||
"""
|
||||
pages = [
|
||||
_relative(page, wiki_root)
|
||||
for page in _markdown_files(wiki_root, "wiki")
|
||||
if _frontmatter(page).get("status") == "raw"
|
||||
]
|
||||
text = f"{len(pages)} raw page(s)"
|
||||
if not pages:
|
||||
return _step("ingest", "PASS", text)
|
||||
return _step("ingest", "WARN", text, pages,
|
||||
["Refine each raw page, then set status: refined."])
|
||||
|
||||
|
||||
def sanity_step(wiki_root: Path) -> dict:
|
||||
"""Run every wiki sanity check without the progress display (SPEC 7.2.5).
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - failing check reports as details.
|
||||
"""
|
||||
results = run_wiki_checks(wiki_root)
|
||||
failing = {name: check for name, check in results["checks"].items() if check["result"]}
|
||||
text = f"{results['total_violations']} violation(s) in {len(failing)} check(s)"
|
||||
if not failing:
|
||||
return _step("sanity", "PASS", text)
|
||||
details = [
|
||||
f"{name}: {line.strip().removeprefix('- ')}"
|
||||
for name, check in failing.items()
|
||||
for line in check["stdout"].splitlines()
|
||||
if line.strip()
|
||||
]
|
||||
return _step("sanity", "FAIL", text, details,
|
||||
["Fix the violations, then run fiona-assistant housekeeping again."])
|
||||
|
||||
|
||||
def graph_step(wiki_root: Path) -> dict:
|
||||
"""Write wiki_graph.html at the wiki root and list orphans (SPEC 7.2.6).
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - graph plus orphan pages under wiki/.
|
||||
"""
|
||||
captured = io.StringIO()
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
graph_json = Path(temp_dir) / "graph.json"
|
||||
with contextlib.redirect_stdout(captured), contextlib.redirect_stderr(captured):
|
||||
code = wiki_graph_main(str(wiki_root), output=str(graph_json), format_type="json")
|
||||
if code == 0:
|
||||
code = wiki_graph_main(str(wiki_root), output=str(wiki_root / GRAPH_FILE))
|
||||
if code != 0:
|
||||
error = captured.getvalue().strip() or f"wiki graph exited {code}"
|
||||
return _step("graph", "FAIL", f"could not write {GRAPH_FILE}",
|
||||
error.splitlines())
|
||||
data = json.loads(graph_json.read_text(encoding="utf-8"))
|
||||
|
||||
orphans = sorted(
|
||||
rel for rel in (_relative(path, wiki_root) for path in data["orphans"])
|
||||
if rel.startswith("wiki/") and not rel.endswith(UPDATE_FILE_SUFFIX)
|
||||
)
|
||||
text = (
|
||||
f"wrote {GRAPH_FILE} ({data['metadata']['file_count']} nodes, "
|
||||
f"{len(data['edges'])} edges)"
|
||||
)
|
||||
if not orphans:
|
||||
return _step("graph", "PASS", text)
|
||||
return _step("graph", "WARN", text, [f"orphan: {rel}" for rel in orphans],
|
||||
["Link each orphan page from a related page or from wiki/index.md."])
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Routine and Report
|
||||
# =============================================================================
|
||||
|
||||
def run_housekeeping(directory: str | Path = ".") -> dict:
|
||||
"""Run every housekeeping step over a wiki (SPEC 7.2.1).
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - every step runs, even after a FAIL.
|
||||
|
||||
Args:
|
||||
directory: The wiki directory. Defaults to the current directory.
|
||||
|
||||
Returns:
|
||||
dict: `target` (resolved path), `steps` (one result per step,
|
||||
in `STEP_NAMES` order), and `passed` (no step is `FAIL`).
|
||||
|
||||
Raises:
|
||||
FileNotFoundError: If the directory does not exist.
|
||||
NotADirectoryError: If the directory is not an llm-wiki directory.
|
||||
"""
|
||||
wiki_root = Path(directory).resolve()
|
||||
if not wiki_root.exists():
|
||||
raise FileNotFoundError(f"Directory not found: {directory}")
|
||||
if not (wiki_root / SETTINGS_FILE).exists():
|
||||
raise NotADirectoryError(f"Directory '{directory}' is not an llm-wiki directory")
|
||||
|
||||
steps = [
|
||||
step(wiki_root)
|
||||
for step in (
|
||||
check_update_step,
|
||||
merge_step,
|
||||
ingest_step,
|
||||
sanity_step,
|
||||
graph_step,
|
||||
)
|
||||
]
|
||||
return {
|
||||
"target": wiki_root,
|
||||
"steps": steps,
|
||||
"passed": all(step["status"] != "FAIL" for step in steps),
|
||||
}
|
||||
|
||||
|
||||
def format_housekeeping_report(results: dict) -> str:
|
||||
"""Format housekeeping results for humans and LLMs (SPEC 4.8).
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - step blocks, next steps, verdict.
|
||||
|
||||
Args:
|
||||
results: The dict returned by run_housekeeping.
|
||||
|
||||
Returns:
|
||||
str: The report text.
|
||||
"""
|
||||
lines = ["Housekeeping Report", f"TARGET: {results['target']}", ""]
|
||||
for step in results["steps"]:
|
||||
lines.append(f"[{step['status']}] {step['name']}: {step['summary']}")
|
||||
lines.extend(f" - {detail}" for detail in step["details"])
|
||||
|
||||
actions = list(dict.fromkeys(a for step in results["steps"] for a in step["next"]))
|
||||
if actions:
|
||||
lines.extend(["", "Next steps:"])
|
||||
lines.extend(f" - {action}" for action in actions)
|
||||
|
||||
failing = [step["name"] for step in results["steps"] if step["status"] == "FAIL"]
|
||||
lines.append("")
|
||||
if failing:
|
||||
lines.append(
|
||||
f"HOUSEKEEPING FAILED - {len(failing)} step(s) need attention: "
|
||||
f"{', '.join(failing)}"
|
||||
)
|
||||
else:
|
||||
lines.append("HOUSEKEEPING PASSED")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def housekeeping_main(directory: str = ".", json_output: bool = False) -> int:
|
||||
"""Main function for the 'housekeeping' CLI command.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - exit 1 on any FAIL step.
|
||||
|
||||
Args:
|
||||
directory: Wiki directory (default: current directory).
|
||||
json_output: If True, print the results as one JSON object.
|
||||
|
||||
Returns:
|
||||
int: 0 when no step is `FAIL`, 1 otherwise or on an invalid wiki.
|
||||
"""
|
||||
try:
|
||||
results = run_housekeeping(directory)
|
||||
except (FileNotFoundError, NotADirectoryError) as error:
|
||||
print(f"Error: {error}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
if json_output:
|
||||
print(json.dumps({**results, "target": str(results["target"])}, indent=2))
|
||||
else:
|
||||
print(format_housekeeping_report(results))
|
||||
return 0 if results["passed"] else 1
|
||||
@@ -0,0 +1,171 @@
|
||||
"""Gitignore patterns shared by wiki init and wiki update.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - seed wiki ignores from the repository file.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import fnmatch
|
||||
import importlib.resources
|
||||
import pathlib
|
||||
|
||||
_PACKAGE_NAME = "fiona-assistant"
|
||||
_PERSONAL_PATTERN = "personal/"
|
||||
|
||||
|
||||
def gitignore_patterns_from_text(text: str) -> list[str]:
|
||||
"""Return ignore patterns from gitignore text.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - drop blanks and comments.
|
||||
|
||||
Args:
|
||||
text: Contents of a .gitignore file.
|
||||
|
||||
Returns:
|
||||
Patterns in file order, without comments or blank lines.
|
||||
"""
|
||||
patterns: list[str] = []
|
||||
for line in text.splitlines():
|
||||
stripped = line.strip()
|
||||
if stripped and not stripped.startswith("#"):
|
||||
patterns.append(stripped)
|
||||
return patterns
|
||||
|
||||
|
||||
def path_is_ignored(relpath: str, patterns: list[str]) -> bool:
|
||||
"""Return whether a relative posix path matches any gitignore pattern.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - match unanchored names and directory prefixes.
|
||||
|
||||
A pattern with no slash matches any path segment. A trailing slash matches
|
||||
a directory and everything under it. ``*`` does not cross a directory
|
||||
boundary because matching is done per segment.
|
||||
|
||||
Args:
|
||||
relpath: Path relative to the wiki or template root, posix form.
|
||||
patterns: Gitignore patterns.
|
||||
|
||||
Returns:
|
||||
True when one pattern matches.
|
||||
"""
|
||||
return any(_pattern_matches(relpath, pattern) for pattern in patterns)
|
||||
|
||||
|
||||
def _pattern_matches(relpath: str, pattern: str) -> bool:
|
||||
"""Return whether one gitignore pattern matches a relative posix path."""
|
||||
body = pattern.rstrip("/")
|
||||
if not body:
|
||||
return False
|
||||
parts = relpath.split("/")
|
||||
if "/" in body:
|
||||
prefixes = ["/".join(parts[:index]) for index in range(1, len(parts) + 1)]
|
||||
return any(fnmatch.fnmatchcase(prefix, body) for prefix in prefixes)
|
||||
return any(fnmatch.fnmatchcase(part, body) for part in parts)
|
||||
|
||||
|
||||
def repository_gitignore_path() -> pathlib.Path | None:
|
||||
"""Find the fiona-assistant repository .gitignore, if this tree has one.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - prefer the source checkout over package data.
|
||||
|
||||
Returns:
|
||||
Path to the repository .gitignore, or None when this install is not
|
||||
the source checkout.
|
||||
"""
|
||||
here = pathlib.Path(__file__).resolve()
|
||||
for parent in here.parents:
|
||||
project = parent / "pyproject.toml"
|
||||
gitignore = parent / ".gitignore"
|
||||
if not project.is_file() or not gitignore.is_file():
|
||||
continue
|
||||
text = project.read_text(encoding="utf-8")
|
||||
if f'name = "{_PACKAGE_NAME}"' in text:
|
||||
return gitignore
|
||||
return None
|
||||
|
||||
|
||||
def packaged_gitignore_path() -> pathlib.Path | None:
|
||||
"""Return the packaged copy of the repository .gitignore, if present.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - fallback for an installed wheel.
|
||||
|
||||
Returns:
|
||||
Path to ``fiona/data/default.gitignore``, or None when it is absent.
|
||||
"""
|
||||
resource = importlib.resources.files("fiona").joinpath("data", "default.gitignore")
|
||||
path = pathlib.Path(str(resource))
|
||||
if path.is_file():
|
||||
return path
|
||||
return None
|
||||
|
||||
|
||||
def repository_gitignore_text() -> str:
|
||||
"""Load the ignore file that seeds a wiki, as LF text.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - repository file, then packaged copy.
|
||||
|
||||
Returns:
|
||||
Ignore file text ending in a newline.
|
||||
|
||||
Raises:
|
||||
FileNotFoundError: When neither the repository nor the packaged file exists.
|
||||
"""
|
||||
path = repository_gitignore_path() or packaged_gitignore_path()
|
||||
if path is None:
|
||||
raise FileNotFoundError("fiona-assistant repository .gitignore was not found")
|
||||
text = path.read_text(encoding="utf-8").replace("\r\n", "\n").replace("\r", "\n")
|
||||
if text and not text.endswith("\n"):
|
||||
text += "\n"
|
||||
return text
|
||||
|
||||
|
||||
def wiki_gitignore_patterns() -> list[str]:
|
||||
"""Return patterns written into a wiki, repository set plus personal/.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - personal/ stays untracked after init.
|
||||
|
||||
Returns:
|
||||
Ignore patterns in write order.
|
||||
"""
|
||||
patterns = gitignore_patterns_from_text(repository_gitignore_text())
|
||||
if _PERSONAL_PATTERN not in patterns:
|
||||
patterns.append(_PERSONAL_PATTERN)
|
||||
return patterns
|
||||
|
||||
|
||||
def wiki_gitignore_text() -> str:
|
||||
"""Return the full .gitignore body for a wiki that does not have one yet.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - repository text plus personal/ when missing.
|
||||
|
||||
Returns:
|
||||
File body ending in a newline.
|
||||
"""
|
||||
text = repository_gitignore_text()
|
||||
if _PERSONAL_PATTERN not in gitignore_patterns_from_text(text):
|
||||
text += f"{_PERSONAL_PATTERN}\n"
|
||||
return text
|
||||
|
||||
|
||||
def ensure_wiki_gitignore(directory: pathlib.Path) -> None:
|
||||
"""Create .gitignore or append patterns that are not already listed.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - put or append the repository ignores.
|
||||
|
||||
Args:
|
||||
directory: Wiki root that should contain .gitignore.
|
||||
"""
|
||||
target = directory / ".gitignore"
|
||||
desired = wiki_gitignore_patterns()
|
||||
if not target.exists():
|
||||
target.write_text(wiki_gitignore_text(), encoding="utf-8", newline="\n")
|
||||
return
|
||||
existing = target.read_text(encoding="utf-8").replace("\r\n", "\n").replace("\r", "\n")
|
||||
have = set(gitignore_patterns_from_text(existing))
|
||||
missing = [pattern for pattern in desired if pattern not in have]
|
||||
if not missing:
|
||||
return
|
||||
if existing and not existing.endswith("\n"):
|
||||
existing += "\n"
|
||||
existing += "\n".join(missing) + "\n"
|
||||
target.write_text(existing, encoding="utf-8", newline="\n")
|
||||
@@ -0,0 +1,4 @@
|
||||
"""Wiki ingest package (SPEC 7.1.5).
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - stage modules for wiki ingest.
|
||||
"""
|
||||
@@ -0,0 +1,239 @@
|
||||
"""Archive a fetched file under ``raw/``.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.3. Never edits a file already there.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import logging
|
||||
import pathlib
|
||||
import shutil
|
||||
|
||||
from fiona.wiki.ingest.converters.base import ConverterRegistry
|
||||
from fiona.wiki.ingest.media import extension_for
|
||||
from fiona.wiki.ingest.models import ArchiveResult, FetchedSource, IngestError
|
||||
from fiona.wiki.ingest.settings import IngestSettings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class RawArchiver:
|
||||
"""Copy or move a source into ``raw/`` and reuse identical bytes.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SHA-256 reuse and collision names.
|
||||
"""
|
||||
|
||||
def __init__(self, wiki_root: pathlib.Path) -> None:
|
||||
"""Store the wiki root.
|
||||
|
||||
Args:
|
||||
wiki_root: Directory that contains ``raw/``.
|
||||
"""
|
||||
self.wiki_root = wiki_root
|
||||
|
||||
def find_by_sha256(
|
||||
self,
|
||||
digest: str,
|
||||
size: int,
|
||||
skip: pathlib.Path | None = None,
|
||||
) -> pathlib.Path | None:
|
||||
"""Return the first ``raw/`` file with this size and digest.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - only same-size files are hashed.
|
||||
|
||||
Args:
|
||||
digest: Lowercase SHA-256 hex.
|
||||
size: Byte size that a candidate must match.
|
||||
skip: A file to ignore, usually the source when it already sits in ``raw/``.
|
||||
|
||||
Returns:
|
||||
pathlib.Path | None: Match with the lowest path, or None.
|
||||
"""
|
||||
raw_root = self.wiki_root / "raw"
|
||||
if not raw_root.is_dir():
|
||||
return None
|
||||
skip_resolved = skip.resolve() if skip is not None else None
|
||||
matches: list[pathlib.Path] = []
|
||||
for path in raw_root.rglob("*"):
|
||||
if not path.is_file():
|
||||
continue
|
||||
if skip_resolved is not None and path.resolve() == skip_resolved:
|
||||
continue
|
||||
try:
|
||||
if path.stat().st_size != size:
|
||||
continue
|
||||
except OSError:
|
||||
continue
|
||||
if _sha256(path) == digest:
|
||||
matches.append(path)
|
||||
if not matches:
|
||||
return None
|
||||
matches.sort(key=lambda item: item.as_posix())
|
||||
return matches[0]
|
||||
|
||||
def archive(
|
||||
self,
|
||||
source: FetchedSource,
|
||||
*,
|
||||
registry: ConverterRegistry,
|
||||
settings: IngestSettings,
|
||||
raw_override: str | None,
|
||||
dry_run: bool,
|
||||
) -> ArchiveResult:
|
||||
"""Place the source under ``raw/`` or reuse an identical copy.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.3.
|
||||
|
||||
Args:
|
||||
source: Fetched file.
|
||||
registry: Used for the raw category and a missing extension.
|
||||
settings: ``raw_categories`` overrides.
|
||||
raw_override: ``-r`` value, relative to ``raw/``, or None.
|
||||
dry_run: When true, compute the path and write nothing.
|
||||
|
||||
Returns:
|
||||
ArchiveResult: Relative raw path, digest, and warnings.
|
||||
|
||||
Raises:
|
||||
IngestError: ``-r`` escapes ``raw/`` or is used on a file already there.
|
||||
"""
|
||||
raw_root = (self.wiki_root / "raw").resolve()
|
||||
source_in_raw = _is_inside(source.path, raw_root)
|
||||
if raw_override and source_in_raw:
|
||||
raise IngestError(
|
||||
f"--raw cannot be used on a file already under raw/: {source.argument}"
|
||||
)
|
||||
if raw_override:
|
||||
_require_inside(raw_root, raw_override, "--raw must stay inside raw/")
|
||||
digest = _sha256(source.path)
|
||||
match = self.find_by_sha256(
|
||||
digest,
|
||||
source.size_bytes,
|
||||
skip=source.path if source_in_raw else None,
|
||||
)
|
||||
warnings: list[str] = []
|
||||
if match is not None:
|
||||
raw_rel = _relative(match, self.wiki_root)
|
||||
if raw_override:
|
||||
warnings.append(f"already archived at {raw_rel}; --raw ignored")
|
||||
if _is_inside(source.path, self.wiki_root / "ingest") and not dry_run:
|
||||
source.path.unlink()
|
||||
logger.info("reused raw file %s", raw_rel)
|
||||
return ArchiveResult(match, raw_rel, digest, True, warnings)
|
||||
if source_in_raw:
|
||||
return ArchiveResult(source.path, _relative(source.path, self.wiki_root), digest, False, [])
|
||||
extension = source.path.suffix.lower() or extension_for(source.media_type, "", registry)
|
||||
dest = self._destination(source, registry, settings, raw_override, extension, raw_root)
|
||||
dest = _free_name(dest, source.slug, extension)
|
||||
if not dry_run:
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
if _is_inside(source.path, self.wiki_root / "ingest"):
|
||||
shutil.move(str(source.path), str(dest))
|
||||
else:
|
||||
shutil.copyfile(source.path, dest)
|
||||
logger.info("archived %s", dest)
|
||||
return ArchiveResult(dest, _relative(dest, self.wiki_root), digest, False, warnings)
|
||||
|
||||
def _destination(
|
||||
self,
|
||||
source: FetchedSource,
|
||||
registry: ConverterRegistry,
|
||||
settings: IngestSettings,
|
||||
raw_override: str | None,
|
||||
extension: str,
|
||||
raw_root: pathlib.Path,
|
||||
) -> pathlib.Path:
|
||||
"""Resolve the raw path before a collision suffix is applied.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - ``-r`` file or directory, else the category.
|
||||
"""
|
||||
filename = f"{source.slug}{extension}"
|
||||
if raw_override:
|
||||
relative = _strip_prefix(raw_override, "raw")
|
||||
suffix = pathlib.PurePosixPath(relative.replace("\\", "/")).suffix
|
||||
if suffix:
|
||||
return raw_root / relative
|
||||
return raw_root / relative / filename
|
||||
converter = registry.find(source.media_type, extension)
|
||||
if converter is None:
|
||||
category = "other"
|
||||
else:
|
||||
category = settings.raw_categories.get(converter.name, converter.raw_category)
|
||||
return raw_root / category / filename
|
||||
|
||||
|
||||
def _sha256(path: pathlib.Path) -> str:
|
||||
"""Return the lowercase SHA-256 hex digest of a file.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - streamed so large files stay cheap.
|
||||
"""
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
while True:
|
||||
chunk = handle.read(1024 * 1024)
|
||||
if not chunk:
|
||||
break
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def _free_name(dest: pathlib.Path, slug: str, extension: str) -> pathlib.Path:
|
||||
"""Return ``dest`` or the first free ``<slug>-N`` name beside it.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - never overwrite a raw file.
|
||||
"""
|
||||
if not dest.exists():
|
||||
return dest
|
||||
number = 2
|
||||
while True:
|
||||
candidate = dest.parent / f"{slug}-{number}{extension}"
|
||||
if not candidate.exists():
|
||||
return candidate
|
||||
number += 1
|
||||
|
||||
|
||||
def _strip_prefix(value: str, prefix: str) -> str:
|
||||
"""Drop a leading ``raw/`` or ``wiki/`` from a user path.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - both forms are accepted.
|
||||
"""
|
||||
relative = value.replace("\\", "/").lstrip("/")
|
||||
marker = f"{prefix}/"
|
||||
if relative == prefix:
|
||||
return ""
|
||||
if relative.startswith(marker):
|
||||
return relative[len(marker):]
|
||||
return relative
|
||||
|
||||
|
||||
def _require_inside(root: pathlib.Path, value: str, message: str) -> pathlib.Path:
|
||||
"""Resolve ``value`` under ``root`` or raise.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - ``..`` and symlinks must stay inside.
|
||||
"""
|
||||
relative = _strip_prefix(value, root.name)
|
||||
candidate = (root / relative).resolve()
|
||||
if not _is_inside(candidate, root):
|
||||
raise IngestError(f"{message}: {value}")
|
||||
return candidate
|
||||
|
||||
|
||||
def _is_inside(path: pathlib.Path, root: pathlib.Path) -> bool:
|
||||
"""Return whether ``path`` is ``root`` or a file inside it.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - compare resolved paths.
|
||||
"""
|
||||
try:
|
||||
path.resolve().relative_to(root.resolve())
|
||||
except ValueError:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _relative(path: pathlib.Path, root: pathlib.Path) -> str:
|
||||
"""Return a wiki-relative path with ``/`` separators.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - JSON and log paths.
|
||||
"""
|
||||
return path.resolve().relative_to(root.resolve()).as_posix()
|
||||
@@ -0,0 +1,38 @@
|
||||
"""Shipped converters, in registration order.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - markdown, text, html, then pdf.
|
||||
Modified by AI/LLM: 2026-10-07 - the html converter takes the block sweep settings.
|
||||
"""
|
||||
|
||||
from fiona.wiki.ingest.converters.base import ConverterRegistry
|
||||
from fiona.wiki.ingest.converters.html import HtmlConverter
|
||||
from fiona.wiki.ingest.converters.markdown import MarkdownConverter
|
||||
from fiona.wiki.ingest.converters.pdf import PdfConverter
|
||||
from fiona.wiki.ingest.converters.text import TextConverter
|
||||
from fiona.wiki.ingest.settings import IngestSettings, default_settings
|
||||
|
||||
|
||||
def default_registry(settings: IngestSettings | None = None) -> ConverterRegistry:
|
||||
"""Return the converters shipped with wiki ingest.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - registration order from SPEC 7.1.5.
|
||||
Modified by AI/LLM: 2026-10-07 - ``html.*`` settings reach ``HtmlConverter``.
|
||||
|
||||
Args:
|
||||
settings: Ingest settings. ``None`` uses the defaults.
|
||||
|
||||
Returns:
|
||||
ConverterRegistry: Markdown, text, HTML, and PDF.
|
||||
"""
|
||||
settings = settings or default_settings()
|
||||
registry = ConverterRegistry()
|
||||
registry.register(MarkdownConverter)
|
||||
registry.register(TextConverter)
|
||||
registry.register(
|
||||
HtmlConverter(
|
||||
min_words=settings.html_min_words,
|
||||
max_link_density=settings.html_max_link_density,
|
||||
)
|
||||
)
|
||||
registry.register(PdfConverter)
|
||||
return registry
|
||||
@@ -0,0 +1,108 @@
|
||||
"""Converter base types and the registry.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - one class per format, SPEC 7.1.5.1.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import abc
|
||||
import typing
|
||||
|
||||
from fiona.wiki.ingest.models import ConversionResult, FetchedSource
|
||||
|
||||
|
||||
class ConversionError(Exception):
|
||||
"""Raised when a converter cannot read a file it claims.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - exit 3, raw copy kept.
|
||||
"""
|
||||
|
||||
def __init__(self, converter: str, reason: str) -> None:
|
||||
"""Store the converter name and the reason.
|
||||
|
||||
Args:
|
||||
converter: Converter ``name`` attribute.
|
||||
reason: Short failure text, without the ``Error:`` prefix.
|
||||
"""
|
||||
super().__init__(f"{converter}: {reason}")
|
||||
self.converter = converter
|
||||
self.reason = reason
|
||||
|
||||
|
||||
class SourceConverter(abc.ABC):
|
||||
"""One format. Subclasses declare what they claim and implement ``convert``.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.1.2.
|
||||
"""
|
||||
|
||||
name: typing.ClassVar[str]
|
||||
media_types: typing.ClassVar[tuple[str, ...]]
|
||||
extensions: typing.ClassVar[tuple[str, ...]]
|
||||
raw_category: typing.ClassVar[str]
|
||||
|
||||
@abc.abstractmethod
|
||||
def convert(self, source: FetchedSource) -> ConversionResult:
|
||||
"""Convert an archived file to markdown.
|
||||
|
||||
Args:
|
||||
source: Fetched file, including media type and slug.
|
||||
|
||||
Returns:
|
||||
ConversionResult: Body, title, metadata, and warnings.
|
||||
|
||||
Raises:
|
||||
ConversionError: The file cannot be read.
|
||||
"""
|
||||
|
||||
|
||||
class ConverterRegistry:
|
||||
"""Ordered list of converters.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - first media-type match, then extension.
|
||||
"""
|
||||
|
||||
def __init__(self, converters: list[SourceConverter] | None = None) -> None:
|
||||
"""Start with an optional list of converter instances.
|
||||
|
||||
Args:
|
||||
converters: Instances in registration order.
|
||||
"""
|
||||
self.converters: list[SourceConverter] = list(converters or [])
|
||||
|
||||
def register(self, converter: SourceConverter | type[SourceConverter]) -> None:
|
||||
"""Append a converter class or instance.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - adding a format is one registration.
|
||||
|
||||
Args:
|
||||
converter: Class with a no-arg constructor, or a ready instance.
|
||||
"""
|
||||
if isinstance(converter, type):
|
||||
converter = converter()
|
||||
self.converters.append(converter)
|
||||
|
||||
def find(self, media_type: str, extension: str) -> SourceConverter | None:
|
||||
"""Pick the converter for a media type and extension.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.4.1 registration order.
|
||||
|
||||
Args:
|
||||
media_type: Detected type, or empty.
|
||||
extension: Suffix including the dot, or empty.
|
||||
|
||||
Returns:
|
||||
SourceConverter | None: The first match, or None.
|
||||
"""
|
||||
media = (media_type or "").lower()
|
||||
ext = extension.lower()
|
||||
if ext and not ext.startswith("."):
|
||||
ext = f".{ext}"
|
||||
if media:
|
||||
for converter in self.converters:
|
||||
if media in converter.media_types:
|
||||
return converter
|
||||
if ext:
|
||||
for converter in self.converters:
|
||||
if ext in converter.extensions:
|
||||
return converter
|
||||
return None
|
||||
@@ -0,0 +1,141 @@
|
||||
"""HTML source converter.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - article text via beautifulsoup4 and markdownify.
|
||||
Modified by AI/LLM: 2026-10-07 - the block sweep replaces the first-``<article>`` root.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import typing
|
||||
import urllib.parse
|
||||
|
||||
import bs4
|
||||
|
||||
from fiona.wiki.ingest.converters.base import SourceConverter
|
||||
from fiona.wiki.ingest.converters.html_sweep import sweep, unwrap_shadow_roots
|
||||
from fiona.wiki.ingest.media import decode_bytes
|
||||
from fiona.wiki.ingest.models import ConversionResult, FetchedSource
|
||||
|
||||
_DROP_TAGS = ("script", "style", "noscript", "template")
|
||||
_NO_TEXT = "no text content; the page may need JavaScript or a login"
|
||||
|
||||
|
||||
class HtmlConverter(SourceConverter):
|
||||
"""Convert an HTML document to markdown.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.4.5.
|
||||
Modified by AI/LLM: 2026-10-07 - sweep limits are constructor arguments.
|
||||
"""
|
||||
|
||||
name: typing.ClassVar[str] = "html"
|
||||
media_types: typing.ClassVar[tuple[str, ...]] = ("text/html", "application/xhtml+xml")
|
||||
extensions: typing.ClassVar[tuple[str, ...]] = (".html", ".htm", ".xhtml")
|
||||
raw_category: typing.ClassVar[str] = "articles"
|
||||
|
||||
def __init__(self, min_words: int = 4, max_link_density: float = 0.5) -> None:
|
||||
"""Store the block sweep limits.
|
||||
|
||||
Generated by AI/LLM: 2026-10-07 - defaults from SPEC 5.1 ``fiona.wiki.ingest.html``.
|
||||
|
||||
Args:
|
||||
min_words: Fewer words make a ``p`` or loose block ``short``.
|
||||
max_link_density: Link text over text length above this is ``links``.
|
||||
"""
|
||||
self.min_words = min_words
|
||||
self.max_link_density = max_link_density
|
||||
|
||||
def convert(self, source: FetchedSource) -> ConversionResult:
|
||||
"""Parse HTML and return every swept block of the body as markdown.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - drop scripts and absolutize links.
|
||||
Modified by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6 block sweep over ``<body>``.
|
||||
|
||||
Args:
|
||||
source: Archived HTML file. ``origin`` is the page URL when fetched.
|
||||
|
||||
Returns:
|
||||
ConversionResult: Markdown body, title, and ``source_meta``.
|
||||
"""
|
||||
text, warnings = decode_bytes(source.path.read_bytes(), source.charset)
|
||||
soup = bs4.BeautifulSoup(text, "html.parser")
|
||||
unwrap_shadow_roots(soup)
|
||||
for tag in soup.find_all(_DROP_TAGS):
|
||||
tag.decompose()
|
||||
for comment in soup.find_all(string=lambda node: isinstance(node, bs4.Comment)):
|
||||
comment.extract()
|
||||
body = soup.body or soup
|
||||
origin = source.origin if source.origin_kind == "url" else None
|
||||
_absolutize(body, soup, origin)
|
||||
conversion = "complete"
|
||||
if not body.get_text(strip=True):
|
||||
conversion = "partial"
|
||||
warnings = [*warnings, _NO_TEXT]
|
||||
swept = sweep(body, min_words=self.min_words, max_link_density=self.max_link_density)
|
||||
if swept.partial:
|
||||
conversion = "partial"
|
||||
return ConversionResult(
|
||||
body=swept.body,
|
||||
title=_html_title(soup, body),
|
||||
source_meta=_source_meta(soup),
|
||||
conversion=conversion,
|
||||
warnings=[*warnings, *swept.warnings],
|
||||
converter=self.name,
|
||||
)
|
||||
|
||||
|
||||
def _absolutize(root: bs4.element.Tag, soup: bs4.BeautifulSoup, origin: str | None) -> None:
|
||||
"""Rewrite relative href and src values against base href, then the origin.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - a local file with no origin keeps relative links.
|
||||
"""
|
||||
base_tag = soup.find("base", href=True)
|
||||
base = base_tag.get("href") if base_tag is not None else None
|
||||
if not base:
|
||||
base = origin
|
||||
if not base:
|
||||
return
|
||||
for tag in root.find_all(href=True):
|
||||
tag["href"] = urllib.parse.urljoin(str(base), str(tag["href"]))
|
||||
for tag in root.find_all(src=True):
|
||||
tag["src"] = urllib.parse.urljoin(str(base), str(tag["src"]))
|
||||
|
||||
|
||||
def _html_title(soup: bs4.BeautifulSoup, root: bs4.element.Tag) -> str | None:
|
||||
"""Return ``<title>``, then ``og:title``, then the first ``<h1>`` that has text.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.4.5.1.
|
||||
Modified by AI/LLM: 2026-10-07 - the root is ``<body>``, so a blank ``<h1>`` is skipped.
|
||||
"""
|
||||
if soup.title and soup.title.string and soup.title.string.strip():
|
||||
return soup.title.string.strip()
|
||||
og = soup.find("meta", attrs={"property": "og:title"})
|
||||
if og is not None and og.get("content"):
|
||||
return str(og["content"]).strip() or None
|
||||
for heading in root.find_all("h1"):
|
||||
text = heading.get_text(strip=True)
|
||||
if text:
|
||||
return text
|
||||
return None
|
||||
|
||||
|
||||
def _source_meta(soup: bs4.BeautifulSoup) -> dict:
|
||||
"""Collect language, canonical URL, and selected meta values.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - only keys that are present.
|
||||
"""
|
||||
meta: dict[str, str] = {}
|
||||
root = soup.find("html")
|
||||
if root is not None and root.get("lang"):
|
||||
meta["lang"] = str(root["lang"])
|
||||
canonical = soup.find("link", rel=lambda value: value and "canonical" in value)
|
||||
if canonical is not None and canonical.get("href"):
|
||||
meta["canonical"] = str(canonical["href"])
|
||||
for tag in soup.find_all("meta"):
|
||||
key = tag.get("name") or tag.get("property")
|
||||
content = tag.get("content")
|
||||
if not key or content is None:
|
||||
continue
|
||||
name = str(key)
|
||||
if name in {"description", "author"} or name.startswith("og:") or name == "article:published_time":
|
||||
meta[name] = str(content)
|
||||
return meta
|
||||
@@ -0,0 +1,285 @@
|
||||
"""Score every block of an HTML body as ``keep``, ``review``, or ``drop``.
|
||||
|
||||
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6. Rules read HTML structure only.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import dataclasses
|
||||
import re
|
||||
import typing
|
||||
|
||||
import bs4
|
||||
import markdownify
|
||||
|
||||
_BLOCK_KINDS = {
|
||||
"h1": "heading",
|
||||
"h2": "heading",
|
||||
"h3": "heading",
|
||||
"h4": "heading",
|
||||
"h5": "heading",
|
||||
"h6": "heading",
|
||||
"p": "p",
|
||||
"ul": "list",
|
||||
"ol": "list",
|
||||
"dl": "list",
|
||||
"blockquote": "quote",
|
||||
"pre": "pre",
|
||||
"table": "table",
|
||||
"figure": "figure",
|
||||
}
|
||||
_INLINE_TAGS = frozenset((
|
||||
"a", "abbr", "b", "bdi", "br", "cite", "code", "data", "del", "dfn", "em",
|
||||
"font", "i", "img", "ins", "kbd", "label", "mark", "q", "s", "samp", "small",
|
||||
"span", "strong", "sub", "sup", "time", "u", "var", "wbr",
|
||||
))
|
||||
_LAYOUT_CELL_CHILDREN = (*_BLOCK_KINDS, "div")
|
||||
_SKIPPED_TAGS = frozenset(("head",))
|
||||
_DROP_ANCESTORS = frozenset(("nav", "button", "select"))
|
||||
_CHROME_TAGS = frozenset(("aside", "form", "dialog"))
|
||||
_CHROME_ROLES = frozenset(("navigation", "banner", "contentinfo", "complementary"))
|
||||
_HIDDEN_STYLE = re.compile(r"display\s*:\s*none|visibility\s*:\s*hidden", re.IGNORECASE)
|
||||
_SHORT_KINDS = frozenset(("p", "loose"))
|
||||
_ALL_REVIEW = "no block passed the content checks; every block is marked for review"
|
||||
|
||||
|
||||
@dataclasses.dataclass
|
||||
class Block:
|
||||
"""One swept block: an element, or a run of loose text and inline nodes.
|
||||
|
||||
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6.1 and 7.1.5.4.5.6.2.
|
||||
"""
|
||||
|
||||
nodes: list[bs4.PageElement]
|
||||
kind: str
|
||||
text: str = ""
|
||||
link_chars: int = 0
|
||||
words: int = 0
|
||||
verdict: str = ""
|
||||
reason: str = ""
|
||||
|
||||
|
||||
@dataclasses.dataclass
|
||||
class SweepResult:
|
||||
"""Markdown body, warnings, and whether every block needs review.
|
||||
|
||||
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6.6 and 7.1.5.4.5.6.7.
|
||||
"""
|
||||
|
||||
body: str
|
||||
warnings: list[str]
|
||||
partial: bool
|
||||
|
||||
|
||||
def unwrap_shadow_roots(soup: bs4.BeautifulSoup) -> None:
|
||||
"""Read each declarative shadow root in place, so its text is swept.
|
||||
|
||||
Generated by AI/LLM: 2026-10-07 - ``template[shadowrootmode]``; other templates are dropped later.
|
||||
|
||||
Args:
|
||||
soup: Parsed document, before the drop loop.
|
||||
"""
|
||||
for template in soup.find_all("template", attrs={"shadowrootmode": True}):
|
||||
template.unwrap()
|
||||
|
||||
|
||||
def sweep(root: bs4.element.Tag, *, min_words: int, max_link_density: float) -> SweepResult:
|
||||
"""Walk ``root``, classify each block, and render the kept and review blocks.
|
||||
|
||||
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6. No clock, network, or random source.
|
||||
|
||||
Args:
|
||||
root: ``<body>``, or the document when there is no body.
|
||||
min_words: ``fiona.wiki.ingest.html.min_words``.
|
||||
max_link_density: ``fiona.wiki.ingest.html.max_link_density``.
|
||||
|
||||
Returns:
|
||||
SweepResult: Markdown body, the ``html sweep:`` warning, and the partial flag.
|
||||
"""
|
||||
blocks: list[Block] = []
|
||||
_walk(root, blocks)
|
||||
seen: set[str] = set()
|
||||
for block in blocks:
|
||||
_measure(block)
|
||||
block.verdict, block.reason = _classify(block, seen, min_words, max_link_density)
|
||||
if block.verdict != "drop":
|
||||
seen.add(block.text.casefold())
|
||||
counts = {verdict: sum(block.verdict == verdict for block in blocks) for verdict in ("keep", "review", "drop")}
|
||||
warnings = [f"html sweep: {counts['keep']} keep, {counts['review']} review, {counts['drop']} drop"]
|
||||
partial = counts["keep"] == 0 and counts["review"] > 0
|
||||
if partial:
|
||||
warnings.append(_ALL_REVIEW)
|
||||
parts = [part for part in (_render(block) for block in blocks if block.verdict != "drop") if part]
|
||||
body = "\n\n".join(parts)
|
||||
return SweepResult(body=body + "\n" if body else "", warnings=warnings, partial=partial)
|
||||
|
||||
|
||||
def _walk(container: bs4.element.Tag, blocks: list[Block]) -> None:
|
||||
"""Append the blocks of ``container`` in document order.
|
||||
|
||||
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6.1 to 7.1.5.4.5.6.1.2.
|
||||
"""
|
||||
run: list[bs4.PageElement] = []
|
||||
for child in list(container.children):
|
||||
if not isinstance(child, bs4.element.Tag):
|
||||
if isinstance(child, bs4.element.NavigableString):
|
||||
run.append(child)
|
||||
continue
|
||||
if child.name in _SKIPPED_TAGS:
|
||||
continue
|
||||
if _is_block(child):
|
||||
_flush(run, blocks)
|
||||
blocks.append(Block(nodes=[child], kind=_BLOCK_KINDS[child.name]))
|
||||
elif child.name in _INLINE_TAGS and not _holds_structure(child):
|
||||
run.append(child)
|
||||
else:
|
||||
_flush(run, blocks)
|
||||
_walk(child, blocks)
|
||||
_flush(run, blocks)
|
||||
|
||||
|
||||
def _flush(run: list[bs4.PageElement], blocks: list[Block]) -> None:
|
||||
"""Close a loose-text run as one block, unless it holds no text.
|
||||
|
||||
Generated by AI/LLM: 2026-10-07 - whitespace between tags is not a block.
|
||||
"""
|
||||
if run and _fold(_raw_text(run)):
|
||||
blocks.append(Block(nodes=list(run), kind="loose"))
|
||||
run.clear()
|
||||
|
||||
|
||||
def _is_block(tag: bs4.element.Tag) -> bool:
|
||||
"""Return whether ``tag`` is a block. A layout table is a container instead.
|
||||
|
||||
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6.1 layout tables.
|
||||
"""
|
||||
if tag.name not in _BLOCK_KINDS:
|
||||
return False
|
||||
if tag.name != "table":
|
||||
return True
|
||||
return not any(cell.find(_LAYOUT_CELL_CHILDREN) for cell in tag.find_all(("td", "th")))
|
||||
|
||||
|
||||
def _holds_structure(tag: bs4.element.Tag) -> bool:
|
||||
"""Return whether an inline tag wraps a block or container, like a card link.
|
||||
|
||||
Generated by AI/LLM: 2026-10-07 - ``<a><div>...</div></a>`` is walked as a container.
|
||||
"""
|
||||
return tag.find(lambda node: node.name not in _INLINE_TAGS) is not None
|
||||
|
||||
|
||||
def _raw_text(nodes: typing.Iterable[bs4.PageElement]) -> str:
|
||||
"""Return the text of ``nodes`` with no separator and ``<br>`` as a space.
|
||||
|
||||
Generated by AI/LLM: 2026-10-07 - keeps ``<b>Fa</b>st`` as one word.
|
||||
"""
|
||||
parts: list[str] = []
|
||||
for node in nodes:
|
||||
items = [node] if isinstance(node, bs4.element.NavigableString) else [node, *node.descendants]
|
||||
for item in items:
|
||||
if isinstance(item, bs4.element.NavigableString):
|
||||
parts.append(str(item))
|
||||
elif item.name == "br":
|
||||
parts.append(" ")
|
||||
return "".join(parts)
|
||||
|
||||
|
||||
def _fold(text: str) -> str:
|
||||
"""Fold whitespace runs to one space and trim.
|
||||
|
||||
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6.2.
|
||||
"""
|
||||
return " ".join(text.split())
|
||||
|
||||
|
||||
def _measure(block: Block) -> None:
|
||||
"""Set the folded text, link text length, and word count of a block.
|
||||
|
||||
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6.2.
|
||||
"""
|
||||
block.text = _fold(_raw_text(block.nodes))
|
||||
links: list[bs4.element.Tag] = []
|
||||
for node in block.nodes:
|
||||
if isinstance(node, bs4.element.Tag):
|
||||
links.extend([node] if node.name == "a" else node.find_all("a"))
|
||||
block.link_chars = sum(len(_fold(_raw_text([link]))) for link in links)
|
||||
block.words = len(block.text.split())
|
||||
|
||||
|
||||
def _classify(block: Block, seen: set[str], min_words: int, max_link_density: float) -> tuple[str, str]:
|
||||
"""Return ``(verdict, reason)`` with the first rule that holds.
|
||||
|
||||
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6.3 to 7.1.5.4.5.6.5.
|
||||
"""
|
||||
folded = block.text.casefold()
|
||||
if not any(char.isalnum() for char in block.text):
|
||||
return "drop", ""
|
||||
if block.words < min_words and folded in seen:
|
||||
return "drop", ""
|
||||
if _inside(block, _drops):
|
||||
return "drop", ""
|
||||
if _inside(block, _is_chrome):
|
||||
return "review", "chrome"
|
||||
if folded in seen:
|
||||
return "review", "repeat"
|
||||
if block.link_chars / len(block.text) > max_link_density:
|
||||
return "review", "links"
|
||||
if block.kind in _SHORT_KINDS and block.words < min_words:
|
||||
return "review", "short"
|
||||
return "keep", ""
|
||||
|
||||
|
||||
def _inside(block: Block, predicate: typing.Callable[[bs4.element.Tag], bool]) -> bool:
|
||||
"""Return whether the block, or an ancestor below the document, matches.
|
||||
|
||||
Generated by AI/LLM: 2026-10-07 - an element block checks itself; a loose run starts at its parent.
|
||||
"""
|
||||
first = block.nodes[0]
|
||||
node = first.parent if block.kind == "loose" else first
|
||||
while isinstance(node, bs4.element.Tag) and not isinstance(node, bs4.BeautifulSoup):
|
||||
if predicate(node):
|
||||
return True
|
||||
node = node.parent
|
||||
return False
|
||||
|
||||
|
||||
def _drops(tag: bs4.element.Tag) -> bool:
|
||||
"""Return whether text under ``tag`` is never content: nav, controls, hidden.
|
||||
|
||||
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6.3.
|
||||
"""
|
||||
if tag.name in _DROP_ANCESTORS or tag.has_attr("hidden"):
|
||||
return True
|
||||
if str(tag.get("aria-hidden", "")).strip().lower() == "true":
|
||||
return True
|
||||
return bool(_HIDDEN_STYLE.search(str(tag.get("style", ""))))
|
||||
|
||||
|
||||
def _is_chrome(tag: bs4.element.Tag) -> bool:
|
||||
"""Return whether ``tag`` is page chrome: aside, form, landmark role, outer header or footer.
|
||||
|
||||
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6.4 ``chrome``.
|
||||
"""
|
||||
if tag.name in _CHROME_TAGS:
|
||||
return True
|
||||
if str(tag.get("role", "")).strip().lower() in _CHROME_ROLES:
|
||||
return True
|
||||
if tag.name in ("header", "footer"):
|
||||
return tag.find_parent(("article", "main")) is None
|
||||
return False
|
||||
|
||||
|
||||
def _render(block: Block) -> str:
|
||||
"""Convert one block to markdown and wrap a ``review`` block in its markers.
|
||||
|
||||
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6.6, options of 7.1.5.4.5.4.
|
||||
"""
|
||||
markup = "".join(
|
||||
node.output_ready(formatter="minimal") if isinstance(node, bs4.element.NavigableString) else str(node)
|
||||
for node in block.nodes
|
||||
)
|
||||
heading = getattr(markdownify, "ATX", "ATX")
|
||||
text = markdownify.markdownify(markup, heading_style=heading, bullets="-").strip()
|
||||
if not text or block.verdict != "review":
|
||||
return text
|
||||
return f"<!-- fiona:review {block.reason} -->\n\n{text}\n\n<!-- /fiona:review -->"
|
||||
@@ -0,0 +1,105 @@
|
||||
"""Markdown source converter.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - keep the body and lift frontmatter into source_meta.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import typing
|
||||
|
||||
import yaml
|
||||
|
||||
from fiona.wiki.ingest.converters.base import SourceConverter
|
||||
from fiona.wiki.ingest.media import decode_bytes
|
||||
from fiona.wiki.ingest.models import ConversionResult, FetchedSource
|
||||
|
||||
_FRONTMATTER = re.compile(r"\A---\r?\n(.*?)\r?\n---\r?\n?", re.DOTALL)
|
||||
_H1 = re.compile(r"^#\s+(\S.*)$")
|
||||
_FENCE = re.compile(r"^\s*(`{3,}|~{3,})")
|
||||
|
||||
|
||||
class MarkdownConverter(SourceConverter):
|
||||
"""Keep markdown as it is and record any source frontmatter.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.4.3.
|
||||
|
||||
The body after a frontmatter block is kept character for character.
|
||||
The page writer adds the wiki frontmatter when the new page does not
|
||||
have it yet. Source keys move to ``source_meta`` instead of being
|
||||
written a second time as the page frontmatter.
|
||||
"""
|
||||
|
||||
name: typing.ClassVar[str] = "markdown"
|
||||
media_types: typing.ClassVar[tuple[str, ...]] = ("text/markdown",)
|
||||
extensions: typing.ClassVar[tuple[str, ...]] = (".md", ".markdown")
|
||||
raw_category: typing.ClassVar[str] = "articles"
|
||||
|
||||
def convert(self, source: FetchedSource) -> ConversionResult:
|
||||
"""Return the markdown body and any frontmatter as metadata.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - title from frontmatter, then H1, then slug.
|
||||
|
||||
Args:
|
||||
source: Archived markdown file.
|
||||
|
||||
Returns:
|
||||
ConversionResult: Unchanged body, ``complete`` unless decoding fell back.
|
||||
"""
|
||||
text, warnings = decode_bytes(source.path.read_bytes(), source.charset)
|
||||
meta: dict = {}
|
||||
body = text
|
||||
match = _FRONTMATTER.match(text)
|
||||
if match:
|
||||
loaded = _load_meta(match.group(1))
|
||||
if loaded is None:
|
||||
warnings = [*warnings, "source frontmatter did not parse"]
|
||||
else:
|
||||
meta = loaded
|
||||
body = text[match.end():]
|
||||
title = _title(meta, body, source.slug)
|
||||
return ConversionResult(
|
||||
body=body,
|
||||
title=title,
|
||||
source_meta=meta,
|
||||
conversion="complete",
|
||||
warnings=warnings,
|
||||
converter=self.name,
|
||||
)
|
||||
|
||||
|
||||
def _load_meta(block: str) -> dict | None:
|
||||
"""Parse a frontmatter block into a dict.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - invalid YAML stays in the body.
|
||||
"""
|
||||
try:
|
||||
loaded = yaml.safe_load(block)
|
||||
except yaml.YAMLError:
|
||||
return None
|
||||
if loaded is None:
|
||||
return {}
|
||||
if isinstance(loaded, dict):
|
||||
return loaded
|
||||
return None
|
||||
|
||||
|
||||
def _title(meta: dict, body: str, slug: str) -> str:
|
||||
"""Choose the page title.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - frontmatter title, first H1, then slug.
|
||||
"""
|
||||
raw_title = meta.get("title")
|
||||
if isinstance(raw_title, str) and raw_title.strip():
|
||||
return raw_title.strip()
|
||||
fenced = False
|
||||
for line in body.splitlines():
|
||||
if _FENCE.match(line):
|
||||
fenced = not fenced
|
||||
continue
|
||||
if fenced:
|
||||
continue
|
||||
match = _H1.match(line)
|
||||
if match:
|
||||
return match.group(1).strip()
|
||||
return slug
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user