Compare commits
4 Commits
main
..
shellcheck
| Author | SHA1 | Date | |
|---|---|---|---|
| 579f37f112 | |||
| e0ad912eb4 | |||
| a745ba92c1 | |||
| c8b98dbbd6 |
+19
@@ -0,0 +1,19 @@
|
||||
[run]
|
||||
source = src
|
||||
omit =
|
||||
src/tests/*
|
||||
*/tests/*
|
||||
*/test_*.py
|
||||
src/mdoctor/fmlint.py
|
||||
src/python_ruff/pycustomlint.py
|
||||
src/llm-wiki/__init__.py
|
||||
|
||||
[report]
|
||||
exclude_lines =
|
||||
pragma: no cover
|
||||
def __repr__
|
||||
raise AssertionError
|
||||
raise NotImplementedError
|
||||
if __name__ == .__main__.:
|
||||
if TYPE_CHECKING:
|
||||
@abstractmethod
|
||||
+1
-9
@@ -35,7 +35,6 @@ dist/
|
||||
|
||||
# Coverage
|
||||
.coverage
|
||||
coverage.xml
|
||||
htmlcov/
|
||||
|
||||
# Jupyter Notebook checkpoints
|
||||
@@ -47,17 +46,10 @@ htmlcov/
|
||||
# Windows
|
||||
Thumbs.db
|
||||
|
||||
# Obsidian per-user vault state
|
||||
.obsidian/workspace*.json
|
||||
.obsidian/cache
|
||||
|
||||
# Python
|
||||
*.pyc
|
||||
*.pyo
|
||||
*.pyd
|
||||
|
||||
*fiona.log
|
||||
*wiki_graph.html
|
||||
|
||||
# Generated review artifacts (git diff HTML, scratch exports)
|
||||
tmp/
|
||||
*wiki_graph.html
|
||||
@@ -1,23 +1,21 @@
|
||||
---
|
||||
name: Fiona-Assistant Development Agent Rules
|
||||
version: 1.9.3
|
||||
title: Agents
|
||||
name: Fiona-Assistant Agent Rules
|
||||
version: 1.3.2
|
||||
title: LLM Agent Rules
|
||||
type: summary
|
||||
created: 2025-09-23
|
||||
updated: 2026-10-06
|
||||
updated: 2026-09-29
|
||||
tags: [agents, rules, guidelines, llm, ai]
|
||||
sources: []
|
||||
---
|
||||
|
||||
# Agents
|
||||
# Fiona-Assistant LLM Agent Rules
|
||||
|
||||
- [1. Overview](#1-overview)
|
||||
- [1.1 Rule References](#11-rule-references)
|
||||
- [2. Basic Rules (MANDATORY)](#2-basic-rules-mandatory)
|
||||
- [2.1 Development Lifecycle](#21-development-lifecycle)
|
||||
- [2.2 Testing](#22-testing)
|
||||
- [2.3 Documentation](#23-documentation)
|
||||
- [2.4 Commit](#24-commit)
|
||||
- [2.1 Testing](#21-testing)
|
||||
- [2.2 Documentation](#22-documentation)
|
||||
- [3. Environment Setup (MANDATORY)](#3-environment-setup-mandatory)
|
||||
- [3.1 Always Use Bootstrap for Environment](#31-always-use-bootstrap-for-environment)
|
||||
- [4. Python Development Rules](#4-python-development-rules)
|
||||
@@ -57,80 +55,34 @@ Fiona-Assistant repository.
|
||||
|
||||
Refer to rules in this file as `project agent rule N.M` (e.g.
|
||||
`project agent rule 1.1`) to differentiate them from rules in other
|
||||
AGENTS.md files (e.g. the llm-wiki template's `src/fiona/data/llm_wiki/AGENTS.md`).
|
||||
AGENTS.md files (e.g. the llm-wiki template's `src/llm-wiki/AGENTS.md`).
|
||||
|
||||
---
|
||||
|
||||
## 2. Basic Rules (MANDATORY)
|
||||
|
||||
### 2.1 Development Lifecycle
|
||||
### 2.1 Testing
|
||||
|
||||
Every change that alters behavior, adds a feature, or changes a
|
||||
requirement follows these five steps, in order. Finish a step before
|
||||
starting the next. A read-only question skips this lifecycle.
|
||||
|
||||
Before step 2, load the environment (project agent rule 3.1).
|
||||
|
||||
1. **Write or check the spec.** Read [SPEC](./SPEC.md) for the
|
||||
area this task will change.
|
||||
- When the requirement or its `*acceptance*` is missing, write
|
||||
both in [SPEC](./SPEC.md) before any test or code. Follow
|
||||
SPEC 1.1 and set the section status.
|
||||
- When the requirement exists, confirm the task matches that
|
||||
text. That text is the behavior to build.
|
||||
2. **Write tests.** Add tests that fail for that acceptance
|
||||
(project agent rule 6.1). Leave production code unchanged in
|
||||
this step.
|
||||
3. **Implement, TDD style.** Load the
|
||||
[python-tdd skill](./src/fiona/data/llm_wiki/utils/skills/python-tdd/SKILL.md)
|
||||
(project agent rule 4.1).
|
||||
- Red: run the new tests and confirm they fail.
|
||||
- Green: write the smallest change that makes them pass.
|
||||
- Refactor while those tests stay green.
|
||||
4. **Verify the spec.** Re-read the requirement and each
|
||||
acceptance this task touched. The code and the tests satisfy
|
||||
that text.
|
||||
- When they disagree, update [SPEC](./SPEC.md) in this same
|
||||
task. Move status along `Planned` ->
|
||||
`Partially Implemented` -> `Implemented`, or set
|
||||
`Won't Do` when the requirement is rejected.
|
||||
- Run `python scripts/run_tests.py`, then
|
||||
`python scripts/run_sanity.py --non-interactive`. Both pass
|
||||
(project agent rule 2.2).
|
||||
5. **Bump versions.** On each markdown file this task edited,
|
||||
bump frontmatter `version` and set `updated` to the edit date
|
||||
(project agent rule 10.1).
|
||||
- Semver: patch for a clarification, minor for a new rule or
|
||||
requirement, major for a breaking contract change.
|
||||
- Bump [SPEC](./SPEC.md) when step 1 or step 4 changed it.
|
||||
- Python and shell files have no version tag yet (SPEC 7.3.2.1).
|
||||
Leave them untagged.
|
||||
|
||||
A documentation edit that does not change product behavior still
|
||||
bumps versions (step 5). It skips the test and implementation steps.
|
||||
|
||||
### 2.2 Testing
|
||||
|
||||
- For testing scripts always run `python scripts/run_tests.py` and make sure
|
||||
- For testing scripts always run `python src/run_tests.py` and make sure
|
||||
everything is passing on every loop
|
||||
- After modifying the repo, make sure
|
||||
`python scripts/run_sanity.py --non-interactive` passes before finalizing.
|
||||
- After Modifying the repo, make sure `python src/run_sanity.py --non-interactive` passes before finalizing.
|
||||
|
||||
### 2.3 Documentation
|
||||
### 2.2 Documentation
|
||||
|
||||
- Product behavior is specified in [SPEC](./SPEC.md). The order for
|
||||
reading it, writing it, and checking it after the code is project
|
||||
agent rule 2.1.
|
||||
- When documentation changes (for example [SPEC](./SPEC.md), especially
|
||||
the table of contents), confirm every reference to it still resolves.
|
||||
- Always link with a markdown link.
|
||||
- Before changing any functionality, read [SPEC](./SPEC.md) (the project
|
||||
spec sheet) so the change follows the specified requirements and
|
||||
acceptance criteria.
|
||||
- When documentation is changed (e.g., `SPEC.md`, especially the table of
|
||||
contents section), double-check that all references to it are unbroken.
|
||||
- Always link via markdownlink for connectivity.
|
||||
- Whenever a change alters behavior, adds a feature, or changes a
|
||||
requirement, update [SPEC](./SPEC.md) in the same task so the spec stays
|
||||
in sync (adjust section status: `planned` -> `partial` -> `implemented`;
|
||||
use `won't do` for rejected features).
|
||||
|
||||
### 2.4 Commit
|
||||
### 2.3 Commit
|
||||
|
||||
- *NEVER* commit without explicit permission!
|
||||
- This rule covers this repository only. A deployed wiki commits
|
||||
after each ingest cycle and housekeeping pass
|
||||
([SPEC](./SPEC.md) 3.3.3).
|
||||
|
||||
---
|
||||
|
||||
@@ -154,21 +106,10 @@ source bootstrap --env <env_name> --python <version>
|
||||
|
||||
# With cleanup flag
|
||||
source bootstrap --env <env_name> -c
|
||||
|
||||
# This checkout as a normal package (not editable)
|
||||
source bootstrap --local
|
||||
|
||||
# Current user, no virtual environment
|
||||
source bootstrap --local --no-env
|
||||
```
|
||||
|
||||
Windows (PowerShell). A virtual-environment install must be
|
||||
dot-sourced. This creates a repo-local venv at `.venvs\<name>` with
|
||||
the Python launcher. It does not install pyenv. `--no-env` skips that
|
||||
venv and may be executed with `-File`. `--local` installs this
|
||||
repository with `pip install .` (not editable, no `dev` extra).
|
||||
`--no-env` installs with `pip install --user`, so `fiona-assistant`
|
||||
runs without activating an environment. The two flags combine.
|
||||
Windows (PowerShell, must be dot-sourced). This creates a repo-local venv
|
||||
at `.venvs\<name>` with the Python launcher. It does not install pyenv.
|
||||
|
||||
```powershell
|
||||
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
|
||||
@@ -176,8 +117,6 @@ Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
|
||||
. .\bootstrap.ps1 --env fiona_env
|
||||
. .\bootstrap.ps1 --env <env_name> --python <version>
|
||||
. .\bootstrap.ps1 --env <env_name> -c
|
||||
. .\bootstrap.ps1 --local
|
||||
powershell -ExecutionPolicy Bypass -File .\bootstrap.ps1 --local --no-env
|
||||
```
|
||||
|
||||
**Rationale**: The bootstrap script sets up the Python virtual environment,
|
||||
@@ -193,13 +132,11 @@ dependency conflicts, incorrect Python versions, or missing packages.
|
||||
|
||||
### 4.1 Load Python TDD Skill for Python Tasks
|
||||
|
||||
> When modifying or creating Python scripts, the
|
||||
> [python-tdd skill](./src/fiona/data/llm_wiki/utils/skills/python-tdd/SKILL.md)
|
||||
> MUST be loaded
|
||||
> When modifying or creating Python scripts, the Python TDD Skill MUST be loaded
|
||||
|
||||
```text
|
||||
SKILL: python-tdd
|
||||
PATH: src/fiona/data/llm_wiki/utils/skills/python-tdd/SKILL.md
|
||||
PATH: llm-wiki/utils/skills/python-tdd/SKILL.md
|
||||
```
|
||||
|
||||
- Project Rules:
|
||||
@@ -208,8 +145,6 @@ PATH: src/fiona/data/llm_wiki/utils/skills/python-tdd/SKILL.md
|
||||
|
||||
**Required Practices from Python TDD Skill:**
|
||||
|
||||
- Step 3 of project agent rule 2.1 is this cycle. The spec and the
|
||||
failing tests already exist before implementation starts.
|
||||
- Strictly follow **Test-Driven Development (TDD)** - Red-Green-Refactor cycle
|
||||
- Use **pytest** for testing
|
||||
- Follow **OOP principles** and **SOLID** design patterns
|
||||
@@ -238,25 +173,22 @@ The repository has the following structure with specific permissions:
|
||||
|
||||
```text
|
||||
fiona-assistant/
|
||||
├── src/fiona/ # The only installable package
|
||||
│ ├── __main__.py # CLI entry point
|
||||
├── src/ # Source code
|
||||
│ ├── llm-wiki/ # Packaged llm-wiki template (data only)
|
||||
│ │ # Structure source of truth:
|
||||
│ │ # `src/llm-wiki/wiki/schema.md`
|
||||
│ ├── fiona/ # Main CLI package
|
||||
│ ├── librarian/ # Wiki content management
|
||||
│ ├── mdoctor/ # Markdown diagnosis
|
||||
│ ├── customlinter/ # Frontmatter, markdown, Python, JSON linters
|
||||
│ ├── python_ruff/ # Ruff wrapper
|
||||
│ ├── gittool/ # Git operations
|
||||
│ ├── sanity_runner/ # Shared sanity engine
|
||||
│ ├── wiki/ # Wiki lifecycle (init, update, sanity, graph)
|
||||
│ ├── data/llm_wiki/ # Packaged template (data only)
|
||||
│ ├── run_tests.py # Test runner module
|
||||
│ └── run_sanity.py # Sanity runner module
|
||||
├── tests/ # Unit tests (not installed)
|
||||
├── scripts/ # Repo tooling, not package source
|
||||
│ ├── run_tests.py # Launcher for fiona.run_tests
|
||||
│ ├── run_sanity.py # Launcher for fiona.run_sanity
|
||||
│ ├── python_ruff/ # Python code linting
|
||||
│ ├── wiki/ # Wiki lifecycle modules (init, update)
|
||||
│ ├── tests/ # Unit tests
|
||||
│ ├── run_tests.py # Test runner
|
||||
│ ├── run_sanity.py # Sanity runner (tests + linters)
|
||||
│ ├── settings_dev.json # Development settings
|
||||
│ ├── utils.sh # Shell utilities
|
||||
│ └── install_packages.sh # Dependency installer
|
||||
│
|
||||
├── pyproject.toml # Project configuration
|
||||
├── bootstrap # Environment setup (bash; source it)
|
||||
├── bootstrap.ps1 # Environment setup (PowerShell; dot-source it)
|
||||
@@ -264,7 +196,7 @@ fiona-assistant/
|
||||
```
|
||||
|
||||
The llm-wiki directory structure is not duplicated here:
|
||||
`src/fiona/data/llm_wiki/wiki/schema.md` is the final source of truth for the
|
||||
`src/llm-wiki/wiki/schema.md` is the final source of truth for the
|
||||
llm-wiki directory structure.
|
||||
|
||||
**General Rules:**
|
||||
@@ -273,8 +205,8 @@ llm-wiki directory structure.
|
||||
- `/raw/` directories are **READ-ONLY** - Never edit archived source materials
|
||||
- `/wiki/concepts/` is **READ-WRITE** - Place synthesized content here
|
||||
- `/ingest/` is for **input processing** - Parse files dropped here
|
||||
- **Always update** `src/fiona/data/llm_wiki/wiki/index.md` when adding new concepts
|
||||
- **Always append** to `src/fiona/data/llm_wiki/wiki/log.md` for every action
|
||||
- **Always update** `src/llm-wiki/wiki/index.md` when adding new concepts
|
||||
- **Always append** to `src/llm-wiki/wiki/log.md` for every action
|
||||
|
||||
---
|
||||
|
||||
@@ -282,7 +214,6 @@ llm-wiki directory structure.
|
||||
|
||||
### 6.1 Testing Requirements
|
||||
|
||||
Steps 2 and 3 of project agent rule 2.1.
|
||||
**All code changes MUST be tested** following TDD principles:
|
||||
|
||||
1. **Write failing tests first** (Red phase)
|
||||
@@ -293,7 +224,7 @@ Steps 2 and 3 of project agent rule 2.1.
|
||||
|
||||
```bash
|
||||
# Run all tests
|
||||
python scripts/run_tests.py
|
||||
python src/run_tests.py
|
||||
|
||||
# Run with pytest directly
|
||||
python -m pytest
|
||||
@@ -302,7 +233,7 @@ python -m pytest
|
||||
python -m pytest --cov=src
|
||||
|
||||
# Run specific test file
|
||||
python -m pytest tests/test_md_frontmatter.py -v
|
||||
python -m pytest src/tests/test_md_frontmatter.py -v
|
||||
```
|
||||
|
||||
**Test File Structure:**
|
||||
@@ -377,19 +308,41 @@ git commit -m "test: add TDD tests for name/version extraction"
|
||||
|
||||
### 9.1 Development Workflow
|
||||
|
||||
The development lifecycle is project agent rule 2.1. Do those five
|
||||
steps in order: write or check the spec, write tests, implement
|
||||
TDD-style, verify the spec, bump versions.
|
||||
**Standard Workflow:**
|
||||
|
||||
Before step 2 of that lifecycle:
|
||||
1. **Setup Environment** (MANDATORY)
|
||||
|
||||
1. Set up the environment (project agent rule 3.1).
|
||||
2. Load the skill when the task needs one (project agent rule 4.1).
|
||||
Python tasks load the
|
||||
[python-tdd skill](./src/fiona/data/llm_wiki/utils/skills/python-tdd/SKILL.md).
|
||||
Wiki tasks load the
|
||||
[Fiona skill](./src/fiona/data/llm_wiki/utils/skills/fiona/SKILL.md)
|
||||
unless `src/fiona/data/llm_wiki/AGENTS.md` already loaded it.
|
||||
```bash
|
||||
source bootstrap --env <env_name>
|
||||
```
|
||||
|
||||
On Windows, dot-source the PowerShell script instead:
|
||||
|
||||
```powershell
|
||||
. .\bootstrap.ps1 --env <env_name>
|
||||
```
|
||||
|
||||
2. **Load Appropriate Skill** (if applicable)
|
||||
- Python tasks: Load `python-tdd` skill
|
||||
- Wiki tasks: Load `llm-wiki` skill
|
||||
|
||||
3. Read the Spec
|
||||
- Read [SPEC](./SPEC.md) for the feature being changed
|
||||
- Confirm the change matches the specified requirements and acceptance criteria
|
||||
|
||||
4. Follow TDD for Code Changes
|
||||
- Write tests first
|
||||
- Implement minimal code
|
||||
- Refactor
|
||||
|
||||
5. Validate
|
||||
- Run tests
|
||||
- Run linters
|
||||
|
||||
6. Document
|
||||
- Update docstrings
|
||||
- Update README if needed
|
||||
- Update [SPEC](./SPEC.md) if the change affects any specified feature or requirement
|
||||
|
||||
---
|
||||
|
||||
@@ -398,12 +351,7 @@ Before step 2 of that lifecycle:
|
||||
### 10.1 MD File Frontmatter
|
||||
|
||||
**All markdown files in llm-wiki/ MUST have YAML frontmatter** with the
|
||||
required `name` and `version` fields. Every page under `wiki/` also
|
||||
requires the quality signals `confidence`, `llm`, `contested`, and
|
||||
`contradictions`. How to set them is section 15 of the
|
||||
[Fiona skill](./src/fiona/data/llm_wiki/utils/skills/fiona/SKILL.md).
|
||||
All other fields are optional. fmlint enforces `name` and `version`
|
||||
only.
|
||||
required `name` and `version` fields. All other fields are optional:
|
||||
|
||||
```yaml
|
||||
---
|
||||
@@ -415,29 +363,10 @@ updated: YYYY-MM-DD # optional (bump when editing)
|
||||
type: entity | concept | comparison | query | summary | toc
|
||||
sources: [raw/articles/source-name.md] # optional
|
||||
tags: [from taxonomy] # optional
|
||||
# quality signals (required on pages under wiki/):
|
||||
# Optional quality signals:
|
||||
confidence: trusted | high | medium | low | zero
|
||||
llm:
|
||||
- model: none # required; model that last edited the page
|
||||
- reasoning: high # when the model reports a level
|
||||
contested: true
|
||||
contradictions: [other-page-slug]
|
||||
# written by wiki ingest (SPEC 7.1.5), optional:
|
||||
status: raw | refined
|
||||
source_meta: {author: Name} # metadata read from the source
|
||||
ingest: # provenance; every key always present
|
||||
origin: https://host/path # URL or file path of the source
|
||||
origin_kind: url | file
|
||||
final_url: null # URL after redirects
|
||||
fetched: YYYY-MM-DDTHH:MM:SSZ
|
||||
media_type: text/html
|
||||
size_bytes: 0
|
||||
sha256: <64 lowercase hex> # of the raw copy; never hand-edited
|
||||
raw_reused: false
|
||||
converter: html # converter name, or agent
|
||||
conversion: complete | partial
|
||||
warnings: []
|
||||
tool: fiona-assistant <version>
|
||||
---
|
||||
```
|
||||
|
||||
@@ -448,7 +377,7 @@ ingest: # provenance; every key always present
|
||||
- Use semantic versioning (MAJOR.MINOR.PATCH)
|
||||
|
||||
This is the single canonical frontmatter schema; `wiki/schema.md` documents
|
||||
the same schema for the wiki template (see SPEC 6.2).
|
||||
the same schema for the wiki template (see SPEC 4.2.1).
|
||||
|
||||
---
|
||||
|
||||
@@ -487,8 +416,7 @@ the same schema for the wiki template (see SPEC 6.2).
|
||||
When multiple tasks are possible, prioritize in this order:
|
||||
|
||||
1. **User's explicit request** (highest priority)
|
||||
2. **Development lifecycle** (project agent rule 2.1): spec, then
|
||||
tests, then implementation
|
||||
2. **TDD requirements** (tests before implementation)
|
||||
3. **Code quality** (linting, formatting)
|
||||
4. **Documentation** (docstrings, README)
|
||||
5. **Refactoring** (improving existing code)
|
||||
@@ -501,8 +429,6 @@ When multiple tasks are possible, prioritize in this order:
|
||||
### 14.1 Things to Never Do
|
||||
|
||||
- Never modify `/personal/` or `/raw/` directories
|
||||
- Never write production code before the spec states the requirement
|
||||
and a failing test exists (project agent rule 2.1)
|
||||
- Never commit code without tests
|
||||
- Never commit code that fails linting
|
||||
- Never ignore AGENTS.md rules without explicit user override
|
||||
@@ -528,17 +454,14 @@ These rules **CAN** be overridden by:
|
||||
|
||||
| Task | Required Action |
|
||||
| ---- | -------------- |
|
||||
| Development | project agent rule 2.1, in order |
|
||||
| Any Python work | Load the [python-tdd skill](./src/fiona/data/llm_wiki/utils/skills/python-tdd/SKILL.md) |
|
||||
| Any Python work | Load python-tdd skill |
|
||||
| Before testing | `source bootstrap` (Windows: `. .\bootstrap.ps1`) |
|
||||
| Local install, no venv | `bootstrap --local --no-env` |
|
||||
| Step 1 | Write or check [SPEC](./SPEC.md) |
|
||||
| Step 2 and 3 | Failing tests, then the smallest change |
|
||||
| Step 4 | Re-read the spec; tests and sanity pass |
|
||||
| Step 5 | Bump `version` and `updated` on edited markdown |
|
||||
| Changing functionality | Read [SPEC](./SPEC.md) first |
|
||||
| Modifying code | Write tests first (TDD) |
|
||||
| Before commit | Run tests + linters |
|
||||
| Wiki changes | Load the [Fiona skill](./src/fiona/data/llm_wiki/utils/skills/fiona/SKILL.md); update index.md + log.md |
|
||||
| Frontmatter | `name` and `version` on every file; on `wiki/` pages also `confidence`, `llm`, `contested`, `contradictions` |
|
||||
| After behavior change | Update [SPEC](./SPEC.md) to stay in sync |
|
||||
| Wiki changes | Update index.md + log.md |
|
||||
| Frontmatter | Include name, version, title, dates |
|
||||
| Errors | Raise exceptions, don't fail silently |
|
||||
|
||||
---
|
||||
|
||||
-229
@@ -1,229 +0,0 @@
|
||||
---
|
||||
name: changelog
|
||||
version: 1.2.3
|
||||
title: Changelog
|
||||
created: 2026-10-08
|
||||
updated: 2026-10-24
|
||||
type: summary
|
||||
---
|
||||
|
||||
# Changelog
|
||||
|
||||
Revisions of [SPEC](./SPEC.md), newest first. Each entry names the
|
||||
requirements it adds or changes. "Unbuilt" means the requirement is
|
||||
specified and marked `NOT_IMPLEMENTED`.
|
||||
|
||||
## 1.22.1 - 2026-10-24
|
||||
|
||||
- Minor corrections to 1.22.0.
|
||||
|
||||
## 1.22.0 - 2026-10-24
|
||||
|
||||
- Adds `--staged` and `--unstaged` flags to `git diff` for staged-only and
|
||||
unstaged-only diffs (*7.12.7.*, *7.12.8.*).
|
||||
|
||||
## 1.21.0 - 2026-10-24
|
||||
|
||||
- Includes untracked and deleted files in `git diff` report by default
|
||||
(*7.12.6.*).
|
||||
- Updates diff rendering: removes spacing between consecutive added or
|
||||
deleted lines, preserves spacing between added and deleted blocks (*7.12.5.*).
|
||||
|
||||
## 1.20.0 - 2026-10-24
|
||||
|
||||
- Git commands discover and operate on the root repository from the current
|
||||
directory (*4.6.*, *7.12.0.*).
|
||||
|
||||
## 1.19.1 - 2026-10-21
|
||||
|
||||
- The sectionize-markdown skill ships as
|
||||
`utils/skills/sectionize-markdown/SKILL.md` (*3.4.2.*), matching the
|
||||
`<name>/SKILL.md` convention of the other skills. Every reference to
|
||||
the old `sectionize-markdown.md` path now points at `SKILL.md`.
|
||||
|
||||
## 1.18.1 - 2026-10-19
|
||||
|
||||
- `fiona-assistant git diff` writes a constellation-themed HTML
|
||||
report under the wiki `tmp/` directory (*4.6*, *7.12.5*). The
|
||||
packaged wiki `.gitignore` now lists `tmp/`.
|
||||
|
||||
## 1.18.0 - 2026-10-15
|
||||
|
||||
- Adds skill memory to the template (*3.5.*): the `memory` skill
|
||||
under `utils/skills/memory/` says how a skill keeps its own
|
||||
`MEMORIES.md` beside its `SKILL.md`, and ships the blank
|
||||
`MEMORIES.md` to copy. Wiki-wide preferences stay in `MEMORY.md`.
|
||||
|
||||
## 1.17.0 - 2026-10-15
|
||||
|
||||
- The `.gitignore` written by `wiki init` ignores Obsidian's per-user
|
||||
vault state, `.obsidian/workspace*.json` and `.obsidian/cache`, so a
|
||||
wiki opened as an Obsidian vault stays clean between saves. Shared
|
||||
vault settings such as `.obsidian/app.json` stay tracked (*3.2.3.*).
|
||||
|
||||
## 1.16.0 - 2026-10-14
|
||||
|
||||
- Adds the agent memory `MEMORY.md` to the template (*3.4.*): a
|
||||
sectionized file of owner preferences, corrections, and verified
|
||||
facts, linked from `AGENTS.md`. `AGENTS.md` says to update it on
|
||||
an important memory or when the user explicitly asks (*3.4.6.*). `wiki update` adds it when missing
|
||||
and never compares or replaces it after that (*3.4.4.*).
|
||||
|
||||
## 1.14.1 - 2026-10-08
|
||||
|
||||
- Moves the revision history out of the spec into this file.
|
||||
|
||||
## 1.14.0 - 2026-10-08
|
||||
|
||||
- Stages raw pages in a `refine/` directory at the wiki root, and has
|
||||
the agent move each refined page into `wiki/` (*7.1.5.15.*).
|
||||
Unbuilt.
|
||||
|
||||
## 1.13.0 - 2026-10-07
|
||||
|
||||
- Builds login detection with exit 4 (*7.1.5.2.8.*), `--origin`
|
||||
(*7.1.5.2.9.*), the agent steps for exit 4 (*7.1.5.14.*), and the
|
||||
HTML block sweep (*7.1.5.4.5.6.*).
|
||||
|
||||
## 1.11.0
|
||||
|
||||
- Specifies `wiki search` (*4.7.7.*, *7.1.6.*), a keyword search that
|
||||
prints `<path>:<line>: <text>` and shortens long lines around the
|
||||
match. Unbuilt.
|
||||
|
||||
## 1.10.0
|
||||
|
||||
- Replaces the first-`<article>` content root of the `html` converter
|
||||
with a block sweep that scores every block as `keep`, `review`, or
|
||||
`drop` (*7.1.5.4.5.6.*). Unbuilt.
|
||||
|
||||
## 1.9.0
|
||||
|
||||
- `wiki ingest` stops with exit 4 on a URL that needs a login
|
||||
(*7.1.5.2.8.*). Unbuilt.
|
||||
- Adds `--origin` for a file saved by hand (*7.1.5.2.9.*). Unbuilt.
|
||||
- Tells the agent how to get the source instead (*7.1.5.14.*).
|
||||
Unbuilt.
|
||||
|
||||
## 1.8.0
|
||||
|
||||
Adds the findings of the 2026-10-02 manual test. All unbuilt:
|
||||
|
||||
- One-line CLI errors with no traceback (*1.2.7.*, *7.1.1.4.*).
|
||||
- `--version` (*2.3.*).
|
||||
- A quiet default stderr (*7.17.3.*).
|
||||
- An ingest hint for a drop-zone file that already moved
|
||||
(*7.1.5.2.7.*).
|
||||
- A dry run that does not log writes (*7.1.5.8.1.*).
|
||||
- A lint-clean ingest log line (*7.1.5.6.1.*).
|
||||
- A follow-up hint for style violations left on a page
|
||||
(*7.1.5.5.6.*).
|
||||
- Single-file `mdoctor` (*7.4.4.*).
|
||||
- Compact housekeeping sanity details (*7.2.5.1.*).
|
||||
- A quiet graph overwrite (*7.1.4.5.9.1.*).
|
||||
- A usage section in the README (*7.18.4.*).
|
||||
|
||||
## 1.7.0
|
||||
|
||||
- `librarian move` rewrites links to the moved paths across the whole
|
||||
wiki (*7.3.3.6.*).
|
||||
|
||||
## 1.6.0
|
||||
|
||||
- Specifies `librarian move` (*7.3.3.*). Unbuilt.
|
||||
|
||||
## 1.5.2
|
||||
|
||||
- Gives `table_of_contents.md` frontmatter `name` and `version`
|
||||
(*7.3.1.1.1.*), so `fmlint` accepts it.
|
||||
|
||||
## 1.5.0
|
||||
|
||||
- Omits markdown files matched by `fiona.wiki.graph.ignore_files`
|
||||
from the graph (*7.1.4.20.*). The shipped list is
|
||||
`*/table_of_contents.md`, because a table of contents links to
|
||||
every page.
|
||||
|
||||
## 1.4.0
|
||||
|
||||
- Names the core-file record `corefiles.json` (*7.1.2.14.*).
|
||||
- Skips paths in `fiona.wiki.update.ignore_files` (*7.1.2.15.*). The
|
||||
shipped list names `corefiles.json`. Only `wiki update` writes that
|
||||
file.
|
||||
|
||||
## 1.3.0
|
||||
|
||||
- Specifies directory nodes (*7.1.4.18.*) and `wiki graph --serve`
|
||||
(*7.1.4.19.*). Both unbuilt.
|
||||
|
||||
## 1.2.0
|
||||
|
||||
- Folds text line endings before the SHA-256 compare (*7.1.2.13.*).
|
||||
- Writes `corefiles.json`, so a later update does not list a core
|
||||
file whose version and hashes are unchanged (*7.1.2.14.*).
|
||||
|
||||
## 1.1.0
|
||||
|
||||
- Treats a differing `fiona-settings.json` as a conflict and writes
|
||||
`fiona-settings-update.json` (*7.1.2.3.*).
|
||||
|
||||
## 1.0.0
|
||||
|
||||
- Groups each module's functions under that module.
|
||||
- Moves Non-Goals, Priority Order, and Open Questions to Appendices
|
||||
A, B, and C.
|
||||
|
||||
## 0.99.0
|
||||
|
||||
- Builds `wiki ingest` (*7.1.5.*).
|
||||
|
||||
## 0.98.0
|
||||
|
||||
- Specifies that `wiki graph` omits a file matched by `.gitignore`
|
||||
(*7.1.4.17.*). Unbuilt.
|
||||
|
||||
## 0.97.0
|
||||
|
||||
- Rejects a module-level assignment that follows the first function
|
||||
or class (*6.4.3.*).
|
||||
|
||||
## 0.96.0
|
||||
|
||||
- Compares a file without a version by SHA-256 and shows both short
|
||||
hashes in the update report (*7.1.2.12.*).
|
||||
|
||||
## 0.94.0
|
||||
|
||||
- `wiki update` yields to the owner's markdown and writes
|
||||
`<stem>-update.md` instead (*7.1.2.11.*).
|
||||
- Adds `fiona-assistant housekeeping` (*7.2.*).
|
||||
|
||||
## 0.93.0
|
||||
|
||||
- Adds SHA-256 reuse across `raw/`, the complete `ingest`
|
||||
frontmatter map, and the agent metadata check.
|
||||
|
||||
## 0.92.0
|
||||
|
||||
- Specifies `wiki ingest` (*7.1.5.*). Unbuilt.
|
||||
|
||||
## 0.90.0
|
||||
|
||||
- Specifies `node_importance_min_point` (*7.1.4.16.*). Unbuilt.
|
||||
|
||||
## 0.41.0
|
||||
|
||||
- Installs one `fiona` package and keeps tests and repo scripts
|
||||
outside it.
|
||||
|
||||
## 0.40.4
|
||||
|
||||
- Adds the PYC001 exception (*6.4.2.*).
|
||||
|
||||
## 0.40.1
|
||||
|
||||
- Numbers nested requirements such as `7.14.1.1`.
|
||||
|
||||
## 0.40.0
|
||||
|
||||
- Reorganizes the sheet into contracts.
|
||||
@@ -1,436 +1,367 @@
|
||||
# Fiona-Assistant
|
||||
|
||||
> Your LLM agent writes the wiki. Fiona keeps it clean.
|
||||
LLM Wiki Tool Helper for LLM Agents/Hives
|
||||
|
||||
Fiona-Assistant is a command-line toolkit for running an **llm-wiki**: a
|
||||
folder of interlinked markdown that an AI agent builds and maintains for
|
||||
you. You bring sources (web pages, PDFs, notes) and questions. The agent
|
||||
compiles them into wiki pages. Fiona does every deterministic part of the
|
||||
job: it scaffolds the wiki, ingests and archives sources, lints markdown,
|
||||
frontmatter, Python, and JSON, checks links, draws a link graph, versions
|
||||
everything in git, and tells the agent exactly what to fix next.
|
||||
|
||||
The result: the agent spends its tokens on reading and writing, not on
|
||||
listing files, chasing broken links, or reformatting tables.
|
||||
|
||||
```bash
|
||||
python -m pip install fiona-assistant
|
||||
fiona-assistant wiki init my-wiki
|
||||
cd my-wiki
|
||||
fiona-assistant wiki ingest https://example.com/article.html
|
||||
fiona-assistant housekeeping
|
||||
```
|
||||
Fiona-Assistant is a command-line toolkit designed to manage, organize, and
|
||||
maintain LLM wiki content with minimal AI token dependence. It combines
|
||||
script-based automation with structured workflows to help LLM Agents and
|
||||
Hives efficiently manage documentation, markdown files, and code quality.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
## Overview
|
||||
|
||||
- [Why Fiona](#why-fiona)
|
||||
- [Features](#features)
|
||||
- [Quick Start](#quick-start)
|
||||
- [How It Works](#how-it-works)
|
||||
- [Command Reference](#command-reference)
|
||||
- [Output Built for Agents](#output-built-for-agents)
|
||||
- [Roadmap](#roadmap)
|
||||
- [Development](#development)
|
||||
- [Documentation](#documentation)
|
||||
- [License](#license)
|
||||
### Purpose
|
||||
|
||||
---
|
||||
Fiona-Assistant serves as a **standalone MD/scripts/docs organizer** and
|
||||
**LLM wiki management helper** that:
|
||||
|
||||
## Why Fiona
|
||||
- **Reduces AI Token Usage**: Uses deterministic scripts and tools to handle
|
||||
wiki management tasks that don't require LLM intelligence
|
||||
- **Maintains Wiki Structure**: Enforces consistent organization of markdown
|
||||
files, concepts, and raw source materials
|
||||
- **Validates Content Quality**: Provides linting and diagnosis tools for markdown and Python code
|
||||
- **Enables Script-Based Workflows**: Automates repetitive wiki maintenance tasks through CLI commands
|
||||
|
||||
An LLM agent can maintain a knowledge base, but left alone it wastes
|
||||
context on chores and slowly breaks things: dead links, missing
|
||||
frontmatter, wrapped table rows, orphan pages, overwritten sources.
|
||||
### Core Philosophy
|
||||
|
||||
Fiona fixes that split of work:
|
||||
> Use scripts for deterministic tasks, reserve AI tokens for creative and analytical work
|
||||
|
||||
| The agent does | Fiona does |
|
||||
| --- | --- |
|
||||
| Reads sources and writes synthesized pages | Fetches, archives, and converts sources to markdown |
|
||||
| Decides what links to what | Finds dead links, dead wikilinks, and orphan pages |
|
||||
| Refines raw pages | Lints markdown, frontmatter, Python, and JSON |
|
||||
| Answers your questions from the wiki | Commits, snapshots, and restores wiki history |
|
||||
| Merges template updates | Detects template updates and never overwrites your files |
|
||||
|
||||
> Use scripts for deterministic tasks. Reserve AI tokens for creative and
|
||||
> analytical work.
|
||||
The toolkit is designed to be **AI-friendly** - outputs are formatted in
|
||||
consistent, parseable structures that LLMs can easily read and process.
|
||||
|
||||
---
|
||||
|
||||
## Features
|
||||
|
||||
### Wiki in one command
|
||||
### Tool Suite
|
||||
|
||||
`fiona-assistant wiki init` creates a ready-to-use llm-wiki: directory
|
||||
layout, settings, agent rules (`AGENTS.md`), page schema, index, log, and
|
||||
bundled agent skills. It also creates a git repository and makes the
|
||||
first commit, with `personal/` kept out of version control.
|
||||
| Tool | Purpose | Key Capabilities |
|
||||
| ---- | ------- | ---------------- |
|
||||
| **Librarian** | Wiki content management | List files, scan directories, organize MD content |
|
||||
| **MDoctor** | Markdown diagnosis | Lint markdown files using PyMarkdown, detect formatting issues |
|
||||
| **Python-Ruff** | Code quality | Lint Python files using Ruff, enforce code standards |
|
||||
| **Generate-TOC** | Table of contents | Auto-generate nested TOC from markdown headers |
|
||||
|
||||
### Source ingestion
|
||||
### Wiki Structure Support
|
||||
|
||||
`fiona-assistant wiki ingest <url-or-file>` turns a source into a wiki page
|
||||
in one step:
|
||||
Fiona-Assistant understands and maintains the [LLM Wiki directory structure](./llm-wiki/AGENTS.md):
|
||||
|
||||
- Downloads the URL or reads the local file
|
||||
- Archives an untouched copy under `raw/` (duplicates are detected by
|
||||
SHA-256, so the same bytes are never stored twice)
|
||||
- Converts it to markdown (HTML, PDF, plain text, and markdown ship today)
|
||||
- Writes a page under `wiki/` with full provenance in its frontmatter and
|
||||
`status: raw`, so the agent knows it still needs refining
|
||||
- Appends the action to `wiki/log.md`
|
||||
|
||||
`wiki ingest --pending` lists every page still waiting to be refined.
|
||||
|
||||
### One-call maintenance
|
||||
|
||||
`fiona-assistant housekeeping` runs the whole maintenance routine and
|
||||
prints a to-do list for the agent:
|
||||
|
||||
- **check-update**: is the packaged template newer than this wiki?
|
||||
- **merge**: which `*-update.md` files are waiting to be merged?
|
||||
- **ingest**: which pages are still `status: raw`?
|
||||
- **sanity**: which lint checks fail, and where?
|
||||
- **graph**: regenerates `wiki_graph.html` and lists orphan pages
|
||||
|
||||
It only reads, except for writing the graph. Every finding comes with a
|
||||
concrete "next step" the agent can act on. Add `--json` for machine use.
|
||||
|
||||
### Quality checks
|
||||
|
||||
`fiona-assistant wiki sanity` runs five checks in parallel and gives one
|
||||
pass/fail verdict:
|
||||
|
||||
| Check | What it catches |
|
||||
| --- | --- |
|
||||
| `markdownlint` | Markdown style, via PyMarkdown |
|
||||
| `frontmatter_lint` | Missing or malformed YAML frontmatter (`name`, `version`, and more) |
|
||||
| `python-ruff` | Python issues in wiki scripts, via Ruff |
|
||||
| `mdlinter` | Dead links, dead `[[wikilinks]]`, and table rows split across lines |
|
||||
| `jsonlint` | JSON syntax, indentation, trailing commas, duplicate keys |
|
||||
|
||||
Each linter is also available on its own, and `mdoctor fix` and
|
||||
`json-lint fix` repair what they can automatically.
|
||||
|
||||
### Interactive link graph
|
||||
|
||||
`fiona-assistant wiki graph` writes a single self-contained HTML file
|
||||
that shows how your pages link to each other. Nodes are scored by how
|
||||
many different pages point at them, so hub pages stand out. It
|
||||
understands markdown links and `[[wikilinks]]`, shows external URLs, and
|
||||
flags broken links and orphans. `--format json` exports the same graph
|
||||
for other tools.
|
||||
|
||||
### Safe template updates
|
||||
|
||||
As Fiona improves, so does its wiki template. `wiki check-update` shows
|
||||
what changed, and `wiki update` brings your wiki up to date without ever
|
||||
overwriting a file you own. When both sides changed, Fiona writes
|
||||
`<name>-update.md` next to your file and `housekeeping` tells the agent
|
||||
to merge it.
|
||||
|
||||
### Git history built in
|
||||
|
||||
`fiona-assistant git` wraps the operations an agent needs, with
|
||||
timestamped commit messages:
|
||||
|
||||
- `save` commits all current edits
|
||||
- `snapshot <name>` commits and tags a restore point
|
||||
- `restore <name>` returns to a snapshot without losing history
|
||||
- `undo`, `clean`, and `status` for everyday recovery
|
||||
|
||||
### Guardrails
|
||||
|
||||
- `personal/` and `raw/` are read-only for every automated tool. Ingest
|
||||
may add new files to `raw/`; it never changes an existing one.
|
||||
- Commands that need a decision prompt on a terminal and accept
|
||||
`--yes`, `--force`, or `--non-interactive` for unattended runs.
|
||||
- Respects `.gitignore` when scanning.
|
||||
|
||||
### Table of contents generation
|
||||
|
||||
`fiona-assistant librarian toc` builds `table_of_contents.md` for a folder
|
||||
tree. `librarian toc file <path>` keeps an in-file TOC under the H1 up to
|
||||
date, with optional section numbering (`--sectionized`).
|
||||
```text
|
||||
llm-wiki/
|
||||
├── ingest/ # Input zone: files to be processed
|
||||
├── raw/ # Archived source materials (read-only)
|
||||
│ ├── articles/
|
||||
│ ├── papers/
|
||||
│ ├── transcripts/
|
||||
│ └── assets/
|
||||
├── wiki/ # Processed content
|
||||
│ └── concepts/ # Synthesized concept notes
|
||||
├── personal/ # User notes (read-only for agents)
|
||||
└── utils/ # Utility scripts
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
## Installation
|
||||
|
||||
### 1. Install
|
||||
|
||||
Requires Python 3.11 or newer.
|
||||
### Quick Install (Production)
|
||||
|
||||
```bash
|
||||
python -m pip install fiona-assistant
|
||||
```
|
||||
|
||||
### 2. Create a wiki
|
||||
### Development Setup
|
||||
|
||||
- **Clone the repository:**
|
||||
|
||||
```bash
|
||||
git clone <repository-url>
|
||||
cd fiona-assistant
|
||||
```
|
||||
|
||||
- **Create and activate the virtual environment:**
|
||||
|
||||
Linux / macOS:
|
||||
|
||||
```bash
|
||||
source bootstrap --env fiona_env -c
|
||||
```
|
||||
|
||||
Windows (PowerShell):
|
||||
|
||||
```powershell
|
||||
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
|
||||
. .\bootstrap.ps1 --env fiona_env -c
|
||||
```
|
||||
|
||||
> Make sure `fiona_env` is activated before proceeding. On Windows the
|
||||
> environment lives in `.venvs\fiona_env`.
|
||||
|
||||
---
|
||||
|
||||
## Usage
|
||||
|
||||
### Main CLI
|
||||
|
||||
Fiona-Assistant provides a unified CLI with multiple subcommands:
|
||||
|
||||
```bash
|
||||
fiona-assistant wiki init my-wiki
|
||||
cd my-wiki
|
||||
# Show all available commands
|
||||
fiona-assistant --help
|
||||
|
||||
# Run via Python module
|
||||
python -m fiona --help
|
||||
```
|
||||
|
||||
### 3. Point your agent at it
|
||||
|
||||
Open the folder in your AI coding agent (Cursor, Claude Code, or any
|
||||
agent that reads `AGENTS.md`). The bundled `AGENTS.md` loads the Fiona
|
||||
skill, which teaches the agent the ingest, refine, and housekeeping
|
||||
workflow and when to call each command.
|
||||
|
||||
### 4. Feed it sources
|
||||
#### Available Commands
|
||||
|
||||
```bash
|
||||
fiona-assistant wiki ingest https://example.com/post.html
|
||||
fiona-assistant wiki ingest ./notes/meeting.pdf
|
||||
# Run tests
|
||||
fiona-assistant test
|
||||
|
||||
# Manage wiki content with Librarian
|
||||
fiona-assistant librarian [directory]
|
||||
|
||||
# Generate an in-file table of contents below the H1
|
||||
fiona-assistant librarian toc file <path_to_file>
|
||||
|
||||
# Force section numbering on for the run
|
||||
fiona-assistant librarian toc file <path_to_file> --sectionized
|
||||
|
||||
# Generate in-file TOCs for every md file in the current directory / wiki root
|
||||
fiona-assistant librarian toc file '*'
|
||||
|
||||
# Diagnose markdown files with MDoctor
|
||||
fiona-assistant mdoctor [chklint] [directory]
|
||||
|
||||
# Lint Python code with Ruff
|
||||
fiona-assistant python-ruff [chklint] [directory]
|
||||
```
|
||||
|
||||
Then ask the agent to refine the new pages.
|
||||
### Direct Tool Access
|
||||
|
||||
### 5. Keep it healthy
|
||||
Each tool can also be called directly:
|
||||
|
||||
```bash
|
||||
fiona-assistant housekeeping
|
||||
fiona-assistant git save --description "ingest meeting notes"
|
||||
# Librarian - List all files in a directory
|
||||
librarian [directory]
|
||||
|
||||
# MDoctor - Check markdown lint violations
|
||||
mdoctor chklint [directory]
|
||||
|
||||
# MDoctor - List all files
|
||||
|
||||
# Python-Ruff - Check Python code quality
|
||||
python-ruff chklint [directory]
|
||||
|
||||
# Generate table of contents for markdown files
|
||||
generate-toc [directory] [options]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## How It Works
|
||||
## Workflow Examples
|
||||
|
||||
A wiki created by Fiona has this layout:
|
||||
### Wiki Ingestion Workflow
|
||||
|
||||
```bash
|
||||
# 1. Drop raw files into ingest directory
|
||||
# (Manual or automated process)
|
||||
|
||||
# 2. List files ready for processing
|
||||
librarian llm-wiki/ingest
|
||||
|
||||
# 3. Validate markdown quality before processing
|
||||
mdoctor chklint llm-wiki/ingest
|
||||
|
||||
# 4. Move processed files to raw archive
|
||||
# (Agent/orchestration task)
|
||||
```
|
||||
|
||||
### Content Maintenance Workflow
|
||||
|
||||
```bash
|
||||
# Check all markdown files in wiki for lint violations
|
||||
mdoctor chklint llm-wiki/wiki
|
||||
|
||||
# Generate table of contents for concept notes
|
||||
generate-toc llm-wiki/wiki/concepts --recursive-toc
|
||||
|
||||
# Validate Python code in the project
|
||||
python-ruff chklint src/
|
||||
|
||||
# List all markdown files for inventory
|
||||
librarian llm-wiki/
|
||||
```
|
||||
|
||||
### Standalone Document Organization
|
||||
|
||||
```bash
|
||||
# Organize any directory of markdown files
|
||||
librarian /path/to/docs/
|
||||
|
||||
# Validate markdown formatting
|
||||
mdoctor chklint /path/to/docs/
|
||||
|
||||
# Create nested table of contents
|
||||
generate-toc /path/to/docs/ --recursive-toc
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Output Format
|
||||
|
||||
All tools produce **AI-friendly output** with consistent formatting:
|
||||
|
||||
### MDoctor Output Format
|
||||
|
||||
```text
|
||||
my-wiki/
|
||||
├── AGENTS.md # Entry point for the agent; loads the Fiona skill
|
||||
├── fiona-settings.json # Marks the wiki root; tool settings
|
||||
├── ingest/ # Drop zone for files waiting to be parsed
|
||||
├── raw/ # Immutable archive of original sources
|
||||
│ ├── articles/
|
||||
│ ├── papers/
|
||||
│ ├── transcripts/
|
||||
│ ├── assets/
|
||||
│ └── other/
|
||||
├── wiki/ # Pages the agent writes and maintains
|
||||
│ ├── schema.md # Page types, tags, conventions
|
||||
│ ├── index.md # Content catalog
|
||||
│ ├── log.md # Append-only action log
|
||||
│ ├── concepts/
|
||||
│ ├── entities/
|
||||
│ ├── comparisons/
|
||||
│ ├── queries/
|
||||
│ └── sources/
|
||||
├── personal/ # Your own notes (read-only for agents, untracked)
|
||||
└── utils/ # Agent skills, personas, and helper scripts
|
||||
```
|
||||
|
||||
The cycle is:
|
||||
|
||||
- **Ingest**: `wiki ingest` archives a source in `raw/` and writes a raw
|
||||
page in `wiki/`.
|
||||
- **Refine**: the agent rewrites the page, links it to related pages, and
|
||||
sets `status: refined`.
|
||||
- **Maintain**: `housekeeping` reports what needs attention; the agent
|
||||
fixes it.
|
||||
- **Save**: `git save` records the change.
|
||||
|
||||
The full page schema lives in the template's
|
||||
[schema](./src/fiona/data/llm_wiki/wiki/schema.md).
|
||||
|
||||
---
|
||||
|
||||
## Command Reference
|
||||
|
||||
Run `fiona-assistant --help` or `fiona-assistant <command> --help` for all
|
||||
flags. Commands that take an optional directory default to the current
|
||||
directory.
|
||||
|
||||
### Wiki lifecycle
|
||||
|
||||
| Command | What it does |
|
||||
| --- | --- |
|
||||
| `wiki init [dir]` | Create a new llm-wiki with git history |
|
||||
| `wiki ingest <source>` | Fetch, archive, and convert a URL or file into a raw page |
|
||||
| `wiki ingest --pending` | List pages still marked `status: raw` |
|
||||
| `wiki sanity [dir]` | Run every quality check; exit 1 on failure |
|
||||
| `wiki graph [dir]` | Write `wiki_graph.html` (or JSON with `--format json`) |
|
||||
| `wiki check-update [dir]` | Show what a template update would change |
|
||||
| `wiki update [dir]` | Apply a template update (`--dry-run`, `--yes`) |
|
||||
| `housekeeping [dir]` | Run the full maintenance routine and list next steps |
|
||||
|
||||
### Linters
|
||||
|
||||
| Command | What it does |
|
||||
| --- | --- |
|
||||
| `mdoctor chklint [dir]` | Markdown lint via PyMarkdown |
|
||||
| `mdoctor fix [dir]` | Auto-fix markdown lint violations |
|
||||
| `fmlint chklint [dir]` | Frontmatter lint |
|
||||
| `python-ruff chklint [dir]` | Python lint via Ruff |
|
||||
| `json-lint chklint [dir]` | JSON lint |
|
||||
| `json-lint fix [dir]` | Auto-fix JSON indentation and trailing commas |
|
||||
|
||||
### Files and TOC
|
||||
|
||||
| Command | What it does |
|
||||
| --- | --- |
|
||||
| `librarian list [dir]` | List files in a directory |
|
||||
| `librarian toc [dir]` | Write `table_of_contents.md` (`--recursive-toc`, `--exclude REGEX`) |
|
||||
| `librarian toc file <path>` | Write an in-file TOC below the H1 (`--sectionized`) |
|
||||
| `librarian toc file '*'` | Update in-file TOCs for every markdown file in the wiki |
|
||||
|
||||
### Git
|
||||
|
||||
| Command | What it does |
|
||||
| --- | --- |
|
||||
| `git save --description TEXT` | Commit all edits |
|
||||
| `git snapshot <name> --description TEXT` | Commit and tag a restore point |
|
||||
| `git restore <name>` | Restore a snapshot as a new commit |
|
||||
| `git undo` | Drop the last commit |
|
||||
| `git clean` | Discard uncommitted edits |
|
||||
| `git status` | Show whether the tree is clean |
|
||||
|
||||
Every command above is run as `fiona-assistant <command>`. The
|
||||
`librarian` and `mdoctor` tools are also installed as standalone
|
||||
commands.
|
||||
|
||||
---
|
||||
|
||||
## Output Built for Agents
|
||||
|
||||
Every report uses the same shape: a header, the target, rule
|
||||
definitions, findings per file with line and column, and a summary. An
|
||||
agent can parse it without guessing, and a human can skim it.
|
||||
|
||||
```text
|
||||
Markdown Lint Violations
|
||||
TARGET: wiki
|
||||
PyMarkdownLint Violations
|
||||
TARGET: /path/to/directory
|
||||
|
||||
- DEFINITIONS:
|
||||
- MDL001: Dead Link, Relative markdown link does not resolve to an existing file
|
||||
- MDL002: Dead Wiki Link, Wiki-internal link does not resolve to any md page in the wiki
|
||||
- md001: Heading levels should only increment by one, Heading levels should only increment by one level at a time
|
||||
- md013: Line length, Line length
|
||||
|
||||
- FILE: wiki/concepts/retrieval.md
|
||||
- MDL001 line 12: ../sources/old-paper.md does not exist
|
||||
- MDL002 line 30: [[embeddings]] does not resolve to any page
|
||||
- FILE: file1.md
|
||||
- 10:5 md013
|
||||
- 20:3 md001
|
||||
|
||||
- SUMMARY:
|
||||
- Total files with violations: 1
|
||||
- Total violations: 2
|
||||
```
|
||||
|
||||
`wiki sanity`, `housekeeping`, `wiki ingest`, `wiki check-update`, and
|
||||
`wiki update` also accept `--json` for structured output.
|
||||
### Python-Ruff Output Format
|
||||
|
||||
```text
|
||||
Ruff Violations
|
||||
TARGET: /path/to/directory
|
||||
|
||||
Definitions:
|
||||
- F401: `*` import used, but not imported
|
||||
- E501: Line too long
|
||||
|
||||
- FILE: script.py
|
||||
- 15:10 F401
|
||||
- 25:80 E501
|
||||
|
||||
- SUMMARY:
|
||||
- Total files with violations: 1
|
||||
- Total violations: 2
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Roadmap
|
||||
|
||||
Planned work, tracked with requirements and acceptance criteria in
|
||||
[SPEC](./SPEC.md) and prioritized in [todo](./todo.md):
|
||||
|
||||
| Feature | What it will do |
|
||||
| --- | --- |
|
||||
| Wiki root discovery | Run any command from a subfolder; Fiona finds the nearest `fiona-settings.json` |
|
||||
| Backup | One command writes a compressed, restorable archive of the wiki outside the tree |
|
||||
| `setup` command | Interactive review and editing of `fiona-settings.json`, with validation |
|
||||
| Smarter graph | Show only important nodes until you expand a parent; hide gitignored pages |
|
||||
| Code blocks in markdown | Lint Python and shell snippets inside fenced code blocks |
|
||||
| Shellcheck | Lint shell scripts as part of wiki sanity |
|
||||
| HTML and JavaScript lint | Validate HTML and JS files in the wiki |
|
||||
| `python-ruff fix` | Auto-fix Python lint findings |
|
||||
| More converters | Ingest more source formats beyond HTML, PDF, text, and markdown |
|
||||
| Version tags for scripts | Track versions of Python and shell files like markdown pages |
|
||||
|
||||
---
|
||||
|
||||
## Development
|
||||
|
||||
### Set up
|
||||
|
||||
Clone the repository and load the environment with the bootstrap script.
|
||||
|
||||
Linux and macOS:
|
||||
|
||||
```bash
|
||||
git clone <repository-url>
|
||||
cd fiona-assistant
|
||||
source bootstrap --env fiona_env -c
|
||||
```
|
||||
|
||||
Windows (PowerShell):
|
||||
|
||||
```powershell
|
||||
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
|
||||
. .\bootstrap.ps1 --env fiona_env -c
|
||||
```
|
||||
|
||||
The default install is editable (`pip install -e ".[dev]"`). On Windows
|
||||
the environment lives in `.venvs\fiona_env`.
|
||||
|
||||
### Other install modes
|
||||
|
||||
| Goal | Bash | PowerShell |
|
||||
| --- | --- | --- |
|
||||
| Install this checkout as a normal package | `source bootstrap --local` | `. .\bootstrap.ps1 --local` |
|
||||
| Same as `pip install fiona-assistant`, no venv | `source bootstrap --local --no-env` | `powershell -ExecutionPolicy Bypass -File .\bootstrap.ps1 --local --no-env` |
|
||||
|
||||
`--local --no-env` runs `python -m pip install --user .`. If
|
||||
`fiona-assistant` is not found afterwards, add the user scripts
|
||||
directory printed by bootstrap to `PATH`.
|
||||
|
||||
### Test and check
|
||||
|
||||
```bash
|
||||
# Unit tests with coverage
|
||||
python scripts/run_tests.py
|
||||
|
||||
# Full sanity: tests, coverage gate, markdown, Ruff, frontmatter, JSON
|
||||
python scripts/run_sanity.py --non-interactive
|
||||
```
|
||||
|
||||
Contributions follow the spec-first, test-first lifecycle in
|
||||
[AGENTS](./AGENTS.md).
|
||||
|
||||
### Project structure
|
||||
## Project Structure
|
||||
|
||||
```text
|
||||
fiona-assistant/
|
||||
├── src/fiona/ # The installable package
|
||||
│ ├── __main__.py # fiona-assistant CLI
|
||||
│ ├── librarian/ # File listing and TOC generation
|
||||
│ ├── mdoctor/ # Markdown lint and fix
|
||||
│ ├── customlinter/ # Frontmatter, markdown link, Python, JSON linters
|
||||
│ ├── python_ruff/ # Ruff wrapper
|
||||
│ ├── gittool/ # Git operations
|
||||
│ ├── sanity_runner/ # Shared sanity engine
|
||||
│ ├── wiki/ # init, update, sanity, graph, ingest, housekeeping
|
||||
│ └── data/llm_wiki/ # Packaged wiki template
|
||||
├── tests/ # Unit tests (not installed)
|
||||
├── scripts/ # Test and sanity runners, dev settings
|
||||
├── bootstrap # Environment setup (bash)
|
||||
├── bootstrap.ps1 # Environment setup (PowerShell)
|
||||
├── SPEC.md # Requirements and acceptance criteria
|
||||
└── pyproject.toml
|
||||
├── llm-wiki/ # Wiki content and structure definition
|
||||
│ ├── AGENTS.md # Agent rules and directory permissions
|
||||
│ ├── ingest/ # Input zone for new content
|
||||
│ ├── raw/ # Archived source materials
|
||||
│ ├── wiki/ # Processed concept notes
|
||||
│ └── utils/ # Utility scripts
|
||||
│
|
||||
├── src/
|
||||
│ ├── fiona/ # Main CLI entry point
|
||||
│ │ └── __main__.py # Unified command dispatcher
|
||||
│ │
|
||||
│ ├── librarian/ # Wiki content management
|
||||
│ │ ├── __init__.py
|
||||
│ │ ├── librarian.py # File scanning and listing
|
||||
│ │ └── toc_generator.py # Table of contents generator
|
||||
│ │
|
||||
│ ├── mdoctor/ # Markdown diagnosis
|
||||
│ │ ├── __init__.py
|
||||
│ │ └── mdoctor.py # PyMarkdown lint wrapper
|
||||
│ │
|
||||
│ ├── python_ruff/ # Python code linting
|
||||
│ │ ├── __init__.py
|
||||
│ │ └── python_ruff.py # Ruff lint wrapper
|
||||
│ │
|
||||
│ ├── tests/ # Unit tests
|
||||
│ │ ├── test_fiona_main.py
|
||||
│ │ ├── test_librarian.py
|
||||
│ │ ├── test_mdoctor.py
|
||||
│ │ └── test_python_ruff.py
|
||||
│ │
|
||||
│ ├── run_tests.py # Test runner
|
||||
│ └── README.md # Development documentation
|
||||
│
|
||||
├── pyproject.toml # Project configuration
|
||||
├── bootstrap # Environment setup script (bash)
|
||||
├── bootstrap.ps1 # Environment setup script (PowerShell)
|
||||
├── LICENSE # GPL-3.0-only
|
||||
└── README.md # This file
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Documentation
|
||||
## Testing
|
||||
|
||||
- [Project Specification](./SPEC.md): every command, setting, and rule,
|
||||
with acceptance criteria
|
||||
- [LLM Wiki Agent Rules](./src/fiona/data/llm_wiki/AGENTS.md): what a
|
||||
deployed wiki tells its agent
|
||||
- [Fiona skill](./src/fiona/data/llm_wiki/utils/skills/fiona/SKILL.md):
|
||||
the workflow the agent follows inside a wiki
|
||||
- [Development Agent Rules](./AGENTS.md): how to contribute to this
|
||||
repository
|
||||
- [Roadmap](./todo.md): planned and in-progress work
|
||||
### Run Tests
|
||||
|
||||
```bash
|
||||
# Run all tests
|
||||
python src/run_tests.py
|
||||
|
||||
# Or use pytest directly
|
||||
python -m pytest
|
||||
|
||||
# Run with coverage
|
||||
python -m pytest --cov=src
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## License
|
||||
|
||||
GPL-3.0-only
|
||||
|
||||
---
|
||||
|
||||
## Key Concepts
|
||||
|
||||
### For LLM Agents
|
||||
|
||||
- **Deterministic Operations**: Use Fiona-Assistant for file listing, linting, and organization
|
||||
- **Token Efficiency**: Reserve LLM context for synthesis, analysis, and creative tasks
|
||||
- **Structured Output**: All tool outputs are formatted for easy parsing by LLMs
|
||||
|
||||
### For Human Users
|
||||
|
||||
- **CLI-First**: All functionality available through command-line interface
|
||||
- **Composable Tools**: Combine tools in scripts for automated workflows
|
||||
- **Clear Structure**: Maintains consistent wiki organization standards
|
||||
|
||||
---
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [LLM Wiki Agent Rules](./llm-wiki/AGENTS.md) - Directory permissions and workflow rules
|
||||
- [Development Setup](./src/README.md) - Detailed development information
|
||||
- [Project Specification](./SPEC.md) - Feature spec sheet with requirements and acceptance criteria
|
||||
- [Project TODO](./todo.md) - Roadmap and planned features
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference Card
|
||||
|
||||
| Task | Command |
|
||||
| ---- | ------- |
|
||||
| Show help | `fiona-assistant --help` |
|
||||
| List files | `fiona-assistant librarian /path/to/dir` |
|
||||
| Check MD lint | `fiona-assistant mdoctor chklint /path/to/dir` |
|
||||
| Check Python lint | `fiona-assistant python-ruff chklint /path/to/dir` |
|
||||
| Generate TOC | `generate-toc /path/to/dir --recursive-toc` |
|
||||
| Generate file TOC | `fiona-assistant librarian toc file /path/to/file.md --sectionized` |
|
||||
| Run tests | `python src/run_tests.py` |
|
||||
| Install | `pip install fiona-assistant` |
|
||||
| Dev install | `pip install -e .` |
|
||||
|
||||
---
|
||||
|
||||
> **Note**: All paths are relative to the current working directory unless specified as absolute paths.
|
||||
|
||||
@@ -1,5 +1,4 @@
|
||||
#!/bin/bash
|
||||
# Modified by AI/LLM: 2026-09-30 - --local and --no-env install options.
|
||||
#
|
||||
# Project bootstrap script - Sets up Python development environment
|
||||
#
|
||||
@@ -7,47 +6,21 @@
|
||||
# source bootstrap
|
||||
#
|
||||
# Windows (PowerShell): dot-source bootstrap.ps1 instead
|
||||
# . .\bootstrap.ps1 [--env name] [--python 3.x.x] [-c|--clean] [--local] [--no-env]
|
||||
# . .\bootstrap.ps1 [--env name] [--python 3.x.x] [-c|--clean]
|
||||
#
|
||||
# What it does:
|
||||
# 1. Installs pyenv and required system dependencies if needed
|
||||
# 2. Initializes pyenv and pyenv-virtualenv in the current shell
|
||||
# 3. Creates a Python virtual environment named 'test_env' (or custom name)
|
||||
# 4. Activates the virtual environment
|
||||
# 5. Installs this repository (editable with the dev extra, unless --local)
|
||||
# 5. Installs all project requirements from requirements.txt files
|
||||
#
|
||||
# With --clean/-c option:
|
||||
# - Deletes existing environment before creating a new one
|
||||
#
|
||||
# With --local:
|
||||
# - Installs this repository as a normal package (pip install .), not editable
|
||||
#
|
||||
# With --no-env:
|
||||
# - Skips pyenv and the virtual environment
|
||||
# - Installs with pip --user so the commands run without activating an environment
|
||||
#
|
||||
# Usage:
|
||||
# source bootstrap [--pyenv] [-c|--clean] [--env custom_name] [--python 3.x.x] [--local] [--no-env]
|
||||
# source bootstrap [--pyenv] [-c|--clean] [--env custom_name] [--python 3.x.x]
|
||||
# ex. source bootstrap --pyenv -c --env my_env --python 3.10.6
|
||||
# ex. source bootstrap --local --no-env
|
||||
|
||||
# =============================================
|
||||
# SOURCED GUARD
|
||||
# =============================================
|
||||
# Activation only persists when this file is sourced; executing it
|
||||
# loses the environment when the child shell exits. `return` works
|
||||
# at the top level only inside a sourced script.
|
||||
if [ -z "${BASH_VERSION:-}" ]; then
|
||||
echo "ERROR: bootstrap is a bash script. Run it with bash:" >&2
|
||||
echo "Run: source bootstrap [--pyenv] [-c|--clean] [--env name] [--python 3.x.x] [--local] [--no-env]" >&2
|
||||
exit 1
|
||||
fi
|
||||
if ! (return 0 2>/dev/null); then
|
||||
echo "ERROR: bootstrap must be sourced, not executed." >&2
|
||||
echo "The environment would be lost when the child shell exits." >&2
|
||||
echo "Run: source bootstrap [--pyenv] [-c|--clean] [--env name] [--python 3.x.x] [--local] [--no-env]" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# =============================================
|
||||
# VARIABLES
|
||||
@@ -56,13 +29,10 @@ PYTHON_VERSION="3.13.8" # Default Python version
|
||||
export PYENV_NAME="test_env" # Default environment name
|
||||
FORCE_PYENV_INSTALL=false # Flag to force pyenv reinstallation
|
||||
CLEAN_ENVIRONMENT=false # Flag to delete and recreate environment
|
||||
LOCAL_INSTALL=false # Install this repo as a normal package, not editable
|
||||
NO_ENV=false # Skip the virtual environment and pip install --user
|
||||
PYTHON_BIN="python"
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" # scripts root directory
|
||||
|
||||
# Source utilities (needed for echo_error, echo_status, etc.)
|
||||
source "$SCRIPT_DIR/scripts/utils.sh"
|
||||
source "$SCRIPT_DIR/src/utils.sh"
|
||||
|
||||
# Parse command line arguments
|
||||
while [[ $# -gt 0 ]]; do
|
||||
@@ -75,14 +45,6 @@ while [[ $# -gt 0 ]]; do
|
||||
CLEAN_ENVIRONMENT=true
|
||||
shift
|
||||
;;
|
||||
--local)
|
||||
LOCAL_INSTALL=true
|
||||
shift
|
||||
;;
|
||||
--no-env)
|
||||
NO_ENV=true
|
||||
shift
|
||||
;;
|
||||
--env)
|
||||
if [ -n "$2" ]; then
|
||||
PYENV_NAME="$2"
|
||||
@@ -103,7 +65,7 @@ while [[ $# -gt 0 ]]; do
|
||||
;;
|
||||
*)
|
||||
echo_error "Unknown option: $1"
|
||||
echo "Usage: source bootstrap [--pyenv] [-c|--clean] [--env custom_name] [--python 3.x.x] [--local] [--no-env]"
|
||||
echo "Usage: source bootstrap [--pyenv] [-c|--clean] [--env custom_name] [--python 3.x.x]"
|
||||
return 1
|
||||
;;
|
||||
esac
|
||||
@@ -324,7 +286,7 @@ activate_environment() {
|
||||
# Get the virtualenv path
|
||||
local venv_path
|
||||
venv_path=$(pyenv prefix "$PYENV_NAME" 2>/dev/null)
|
||||
|
||||
|
||||
if [ -n "$venv_path" ] && [ -d "$venv_path" ]; then
|
||||
# Manually activate by sourcing the activate script
|
||||
local activate_script="$venv_path/bin/activate"
|
||||
@@ -337,7 +299,7 @@ activate_environment() {
|
||||
return 0
|
||||
fi
|
||||
fi
|
||||
|
||||
|
||||
# Fallback: try pyenv activate
|
||||
if pyenv activate "$PYENV_NAME" 2>/dev/null; then
|
||||
echo_status "Successfully activated $PYENV_NAME with pyenv"
|
||||
@@ -372,82 +334,52 @@ install_requirements() {
|
||||
echo_status "Installing requirements..."
|
||||
|
||||
# Check if pyproject.toml exists (could be in root or src/)
|
||||
local project_dir=""
|
||||
if [ -f "$SCRIPT_DIR/pyproject.toml" ]; then
|
||||
project_dir="$SCRIPT_DIR"
|
||||
elif [ -f "$SCRIPT_DIR/src/pyproject.toml" ]; then
|
||||
project_dir="$SCRIPT_DIR/src"
|
||||
fi
|
||||
|
||||
if [ -n "$project_dir" ]; then
|
||||
local helper="$SCRIPT_DIR/scripts/bootstrap_install.py"
|
||||
if [ ! -f "$helper" ]; then
|
||||
echo_error "Missing $helper"
|
||||
return 1
|
||||
fi
|
||||
local flags=(--break-system-packages)
|
||||
if [ "$LOCAL_INSTALL" = true ]; then
|
||||
echo_status "Installing this repository as a normal package..."
|
||||
flags+=(--local)
|
||||
else
|
||||
echo_status "Installing package and dependencies from pyproject.toml..."
|
||||
fi
|
||||
if [ "$NO_ENV" = true ]; then
|
||||
flags+=(--no-env)
|
||||
fi
|
||||
local pip_words=()
|
||||
local word
|
||||
while IFS= read -r word; do
|
||||
if [ -n "$word" ]; then
|
||||
pip_words+=("$word")
|
||||
fi
|
||||
done < <("$PYTHON_BIN" "$helper" "${flags[@]}")
|
||||
if [ "${#pip_words[@]}" -eq 0 ]; then
|
||||
echo_error "Failed to resolve pip install arguments"
|
||||
return 1
|
||||
fi
|
||||
(cd "$project_dir" && "$PYTHON_BIN" -m pip "${pip_words[@]}") || {
|
||||
local pyproject_root="$SCRIPT_DIR/pyproject.toml"
|
||||
local pyproject_src="$SCRIPT_DIR/src/pyproject.toml"
|
||||
|
||||
if [ -f "$pyproject_root" ]; then
|
||||
# Install the package in development mode from root
|
||||
echo_status "Installing package and dependencies from pyproject.toml..."
|
||||
(cd "$SCRIPT_DIR" && pip install -e ".[dev]" --break-system-packages) || {
|
||||
echo_error "Failed to install package and dependencies from pyproject.toml"
|
||||
return 1
|
||||
}
|
||||
return 0
|
||||
fi
|
||||
|
||||
echo_warning "No pyproject.toml found"
|
||||
|
||||
# Fallback: look for any requirements.txt files
|
||||
local req_files=()
|
||||
while IFS= read -r -d '' req_file; do
|
||||
req_files+=("$req_file")
|
||||
done < <(find "$SCRIPT_DIR" -name "requirements.txt" -print0)
|
||||
|
||||
if [ ${#req_files[@]} -eq 0 ]; then
|
||||
echo_status "No requirements files found"
|
||||
return 0
|
||||
fi
|
||||
|
||||
local user_args=()
|
||||
if [ "$NO_ENV" = true ]; then
|
||||
user_args+=(--user)
|
||||
fi
|
||||
for req_file in "${req_files[@]}"; do
|
||||
echo_status "Installing requirements from: $req_file"
|
||||
"$PYTHON_BIN" -m pip install "${user_args[@]}" -r "$req_file" --break-system-packages || {
|
||||
echo_error "Failed to install requirements from $req_file"
|
||||
elif [ -f "$pyproject_src" ]; then
|
||||
# Install the package in development mode from src/
|
||||
echo_status "Installing package and dependencies from pyproject.toml..."
|
||||
(cd "$SCRIPT_DIR/src" && pip install -e ".[dev]" --break-system-packages) || {
|
||||
echo_error "Failed to install package and dependencies from pyproject.toml"
|
||||
return 1
|
||||
}
|
||||
done
|
||||
else
|
||||
echo_warning "No pyproject.toml found"
|
||||
|
||||
# Fallback: look for any requirements.txt files
|
||||
local req_files=()
|
||||
while IFS= read -r -d '' req_file; do
|
||||
req_files+=("$req_file")
|
||||
done < <(find "$SCRIPT_DIR" -name "requirements.txt" -print0)
|
||||
|
||||
if [ ${#req_files[@]} -eq 0 ]; then
|
||||
echo_status "No requirements files found"
|
||||
return 0
|
||||
fi
|
||||
|
||||
for req_file in "${req_files[@]}"; do
|
||||
echo_status "Installing requirements from: $req_file"
|
||||
pip install -r "$req_file" --break-system-packages || {
|
||||
echo_error "Failed to install requirements from $req_file"
|
||||
return 1
|
||||
}
|
||||
done
|
||||
fi
|
||||
|
||||
return 0
|
||||
}
|
||||
|
||||
upgrade_pip() {
|
||||
echo_status "Upgrading pip..."
|
||||
local upgrade_args=(install --upgrade pip --break-system-packages)
|
||||
if [ "$NO_ENV" = true ]; then
|
||||
upgrade_args=(install --user --upgrade pip --break-system-packages)
|
||||
fi
|
||||
"$PYTHON_BIN" -m pip "${upgrade_args[@]}" || {
|
||||
python -m pip install --upgrade pip --break-system-packages || {
|
||||
echo_warning "Failed to upgrade pip"
|
||||
echo_warning "Continuing with requirements installation..."
|
||||
}
|
||||
@@ -455,18 +387,18 @@ upgrade_pip() {
|
||||
|
||||
install_dev_packages() {
|
||||
echo_status "Installing local Python packages in development mode..."
|
||||
local install_script="$SCRIPT_DIR/scripts/install_packages.sh"
|
||||
|
||||
local install_script="$SCRIPT_DIR/src/install_packages.sh"
|
||||
|
||||
if [ -f "$install_script" ]; then
|
||||
# Use source to run in current shell context, preserving environment variables
|
||||
(cd "$SCRIPT_DIR" && . "$install_script") || {
|
||||
(cd "$SCRIPT_DIR/src" && . "$install_script") || {
|
||||
echo_error "Dev packages not installed!"
|
||||
return 1
|
||||
}
|
||||
else
|
||||
echo_status "Install script not found at $install_script, attempting direct install..."
|
||||
# Fallback: install in editable mode from the repository root
|
||||
(cd "$SCRIPT_DIR" && pip install -e . --break-system-packages) || {
|
||||
# Fallback: install in editable mode from src directory
|
||||
(cd "$SCRIPT_DIR/src" && pip install -e . --break-system-packages) || {
|
||||
echo_error "Failed to install package in development mode"
|
||||
return 1
|
||||
}
|
||||
@@ -482,16 +414,16 @@ sanitize_pytest_env() {
|
||||
# Also remove -S or --no-summary (older versions)
|
||||
local sanitized
|
||||
sanitized=$(echo "$PYTEST_ADDOPTS" | sed -e 's/--no-summary[[:space:]]*//g' -e 's/--no-summary//g' -e 's/-S[[:space:]]*//g' -e 's/-S//g')
|
||||
|
||||
|
||||
# Remove leading/trailing whitespace and any double spaces
|
||||
sanitized=$(echo "$sanitized" | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//' -e 's/[[:space:]]\+/ /g')
|
||||
|
||||
|
||||
if [ "$sanitized" != "$PYTEST_ADDOPTS" ]; then
|
||||
echo_warning "Removing --no-summary from PYTEST_ADDOPTS to ensure test output is parsed correctly"
|
||||
export PYTEST_ADDOPTS="$sanitized"
|
||||
fi
|
||||
fi
|
||||
|
||||
|
||||
# Also unset FORCE_COLOR and PY_COLORS as they can interfere with output parsing
|
||||
if [ -n "$FORCE_COLOR" ]; then
|
||||
echo_warning "Unsetting FORCE_COLOR to ensure pytest output is parsed correctly"
|
||||
@@ -501,67 +433,42 @@ sanitize_pytest_env() {
|
||||
echo_warning "Unsetting PY_COLORS to ensure pytest output is parsed correctly"
|
||||
unset PY_COLORS
|
||||
fi
|
||||
|
||||
|
||||
return 0
|
||||
}
|
||||
|
||||
# =============================================
|
||||
# MAIN EXECUTION
|
||||
# =============================================
|
||||
if [ "$NO_ENV" = true ]; then
|
||||
echo_status "Starting project bootstrap (no virtual environment)"
|
||||
else
|
||||
echo_status "Starting project bootstrap with environment: $PYENV_NAME"
|
||||
echo_status "Starting project bootstrap with environment: $PYENV_NAME"
|
||||
|
||||
if [ "$FORCE_PYENV_INSTALL" = true ]; then
|
||||
echo_status "Forcing pyenv reinstallation"
|
||||
fi
|
||||
|
||||
if [ "$LOCAL_INSTALL" = true ]; then
|
||||
echo_status "Local install: this repository, not editable"
|
||||
fi
|
||||
if [ "$NO_ENV" = true ]; then
|
||||
echo_status "No virtual environment: installing with --user"
|
||||
# 1. Check if Python is installed
|
||||
if ! check_python_installed; then
|
||||
return 1
|
||||
fi
|
||||
|
||||
if [ "$NO_ENV" = true ]; then
|
||||
if ! command -v python >/dev/null 2>&1 && ! command -v python3 >/dev/null 2>&1; then
|
||||
echo_error "Python is not installed. Please install Python $PYTHON_VERSION before running this script."
|
||||
echo_error "Or download from: https://www.python.org/downloads/"
|
||||
return 1
|
||||
fi
|
||||
if command -v python >/dev/null 2>&1; then
|
||||
PYTHON_BIN="python"
|
||||
else
|
||||
PYTHON_BIN="python3"
|
||||
fi
|
||||
else
|
||||
if [ "$FORCE_PYENV_INSTALL" = true ]; then
|
||||
echo_status "Forcing pyenv reinstallation"
|
||||
fi
|
||||
# 2. Check/install pyenv
|
||||
if ! check_and_install_pyenv; then
|
||||
return 1
|
||||
fi
|
||||
|
||||
# 1. Check if Python is installed
|
||||
if ! check_python_installed; then
|
||||
return 1
|
||||
fi
|
||||
# 3. Initialize pyenv
|
||||
if ! initialize_pyenv; then
|
||||
return 1
|
||||
fi
|
||||
|
||||
# 2. Check/install pyenv
|
||||
if ! check_and_install_pyenv; then
|
||||
return 1
|
||||
fi
|
||||
# 4. Check/create virtual environment
|
||||
if ! check_environment_exists; then
|
||||
return 1
|
||||
fi
|
||||
|
||||
# 3. Initialize pyenv
|
||||
if ! initialize_pyenv; then
|
||||
return 1
|
||||
fi
|
||||
|
||||
# 4. Check/create virtual environment
|
||||
if ! check_environment_exists; then
|
||||
return 1
|
||||
fi
|
||||
|
||||
# 5. Activate the environment
|
||||
if ! activate_environment; then
|
||||
return 1
|
||||
fi
|
||||
PYTHON_BIN="python"
|
||||
# 5. Activate the environment
|
||||
if ! activate_environment; then
|
||||
return 1
|
||||
fi
|
||||
|
||||
# 5.5. Sanitize pytest environment to ensure proper output parsing
|
||||
@@ -570,25 +477,14 @@ sanitize_pytest_env
|
||||
# 6. Upgrade pip
|
||||
upgrade_pip
|
||||
|
||||
# 7. Install this repository
|
||||
if ! install_requirements; then
|
||||
return 1
|
||||
fi
|
||||
# 7. Install requirements (including package in dev mode with all dependencies)
|
||||
install_requirements
|
||||
|
||||
# Final verification
|
||||
echo_status "Bootstrap completed successfully!"
|
||||
if [ "$NO_ENV" = true ]; then
|
||||
scripts_dir=$("$PYTHON_BIN" -c "import sys, sysconfig; scheme = 'nt_user' if sys.platform == 'win32' else 'posix_user'; print(sysconfig.get_path('scripts', scheme))")
|
||||
echo_status "Installed for the current user (no virtual environment)"
|
||||
echo_status "Command directory: $scripts_dir"
|
||||
echo_status "If fiona-assistant is not found, add that directory to PATH."
|
||||
echo_status "Python: $("$PYTHON_BIN" --version) ($PYTHON_BIN)"
|
||||
echo_status "Pip: $("$PYTHON_BIN" -m pip --version)"
|
||||
else
|
||||
echo_status "Virtual environment '$PYENV_NAME' is active"
|
||||
echo_status "Python: $(python --version) ($(which python))"
|
||||
echo_status "Pip: $(pip --version)"
|
||||
fi
|
||||
echo_status "Virtual environment '$PYENV_NAME' is active"
|
||||
echo_status "Python: $(python --version) ($(which python))"
|
||||
echo_status "Pip: $(pip --version)"
|
||||
echo_status ""
|
||||
echo_status "To verify installation, try:"
|
||||
echo_status " fiona-assistant test"
|
||||
|
||||
+52
-128
@@ -1,29 +1,20 @@
|
||||
# Generated by AI/LLM: 2026-09-29 - Windows PowerShell bootstrap for Fiona development
|
||||
# Modified by AI/LLM: 2026-09-30 - --local and --no-env install options.
|
||||
# Generated by CURSOR: 2026-09-29 - Windows PowerShell bootstrap for Fiona development
|
||||
#
|
||||
# Project bootstrap for Windows. Dot-source a virtual-environment install so
|
||||
# the environment stays active in the current session (same idea as
|
||||
# `source bootstrap`):
|
||||
# Project bootstrap for Windows. Dot-source it so the virtual environment
|
||||
# stays active in the current session (same idea as `source bootstrap`):
|
||||
#
|
||||
# Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
|
||||
# . .\bootstrap.ps1
|
||||
# . .\bootstrap.ps1 --env fiona_env
|
||||
# . .\bootstrap.ps1 --env fiona_env --python 3.13.8 -c
|
||||
# . .\bootstrap.ps1 --local
|
||||
# powershell -ExecutionPolicy Bypass -File .\bootstrap.ps1 --local --no-env
|
||||
#
|
||||
# Creates a repo-local venv at .venvs\<name> with the Python launcher (py)
|
||||
# or python on PATH. Does not install pyenv. --pyenv is accepted so the
|
||||
# Linux flag still parses; on Windows it is a no-op notice.
|
||||
#
|
||||
# --local installs this repository as a normal package (not editable).
|
||||
# --no-env skips the virtual environment and installs with pip --user,
|
||||
# so fiona-assistant can run without activating an environment.
|
||||
# --no-env may be executed with -File. A venv install must be dot-sourced.
|
||||
|
||||
#Requires -Version 5.1
|
||||
|
||||
# AI/LLM: flags are parsed by hand. Windows PowerShell 5.1 does not bind
|
||||
# CURSOR: flags are parsed by hand. Windows PowerShell 5.1 does not bind
|
||||
# `--python` to a parameter alias, and the bash script uses those flags.
|
||||
|
||||
function Write-FionaStatus {
|
||||
@@ -58,7 +49,7 @@ function Remove-FionaBootstrapFunctions {
|
||||
}
|
||||
|
||||
function Resolve-FionaPython {
|
||||
# AI/LLM: pick a real interpreter. Prefer the py launcher so a requested
|
||||
# CURSOR: pick a real interpreter. Prefer the py launcher so a requested
|
||||
# version (3.13.8, then 3.13) wins over whatever `python` happens to be.
|
||||
param([string]$Version)
|
||||
|
||||
@@ -103,7 +94,7 @@ function Resolve-FionaPython {
|
||||
|
||||
$FionaPrevErrorAction = $ErrorActionPreference
|
||||
$ErrorActionPreference = "Stop"
|
||||
# AI/LLM: a failed pip/py call must be checked via $LASTEXITCODE, not thrown.
|
||||
# CURSOR: a failed pip/py call must be checked via $LASTEXITCODE, not thrown.
|
||||
if (Get-Variable -Name PSNativeCommandUseErrorActionPreference -ErrorAction SilentlyContinue) {
|
||||
$PSNativeCommandUseErrorActionPreference = $false
|
||||
}
|
||||
@@ -112,8 +103,6 @@ $Clean = $false
|
||||
$EnvName = "test_env"
|
||||
$PythonVersion = ""
|
||||
$Pyenv = $false
|
||||
$Local = $false
|
||||
$NoEnv = $false
|
||||
$bootstrapArgs = @($args)
|
||||
for ($i = 0; $i -lt $bootstrapArgs.Count; $i++) {
|
||||
$token = [string]$bootstrapArgs[$i]
|
||||
@@ -123,12 +112,6 @@ for ($i = 0; $i -lt $bootstrapArgs.Count; $i++) {
|
||||
elseif ($token -eq "-c" -or $token -eq "--clean") {
|
||||
$Clean = $true
|
||||
}
|
||||
elseif ($token -eq "--local") {
|
||||
$Local = $true
|
||||
}
|
||||
elseif ($token -eq "--no-env") {
|
||||
$NoEnv = $true
|
||||
}
|
||||
elseif ($token -eq "--env") {
|
||||
if (($i + 1) -ge $bootstrapArgs.Count) {
|
||||
Write-FionaError "--env requires an environment name argument"
|
||||
@@ -149,17 +132,15 @@ for ($i = 0; $i -lt $bootstrapArgs.Count; $i++) {
|
||||
}
|
||||
else {
|
||||
Write-FionaError "Unknown option: $token"
|
||||
Write-FionaError "Usage: . .\bootstrap.ps1 [--pyenv] [-c|--clean] [--env custom_name] [--python 3.x.x] [--local] [--no-env]"
|
||||
Write-FionaError "Usage: . .\bootstrap.ps1 [--pyenv] [-c|--clean] [--env custom_name] [--python 3.x.x]"
|
||||
Remove-FionaBootstrapFunctions
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
if (-not $NoEnv -and $MyInvocation.InvocationName -ne ".") {
|
||||
if ($MyInvocation.InvocationName -ne ".") {
|
||||
Write-FionaError "Dot-source this script so the virtual environment stays active in this session."
|
||||
Write-FionaError " . .\bootstrap.ps1"
|
||||
Write-FionaError "To install without a virtual environment, run:"
|
||||
Write-FionaError " powershell -ExecutionPolicy Bypass -File .\bootstrap.ps1 --local --no-env"
|
||||
Remove-FionaBootstrapFunctions
|
||||
return
|
||||
}
|
||||
@@ -173,27 +154,13 @@ if (-not $repoRoot) {
|
||||
$repoRoot = (Get-Location).Path
|
||||
}
|
||||
|
||||
if ($NoEnv) {
|
||||
Write-FionaStatus "Starting project bootstrap (no virtual environment)"
|
||||
}
|
||||
else {
|
||||
Write-FionaStatus "Starting project bootstrap with environment: $EnvName"
|
||||
}
|
||||
if ($Local) {
|
||||
Write-FionaStatus "Local install: this repository, not editable"
|
||||
}
|
||||
if ($NoEnv) {
|
||||
Write-FionaStatus "No virtual environment: installing with --user"
|
||||
}
|
||||
if ($NoEnv -and $Clean) {
|
||||
Write-FionaWarning "--clean has no effect with --no-env"
|
||||
}
|
||||
Write-FionaStatus "Starting project bootstrap with environment: $EnvName"
|
||||
|
||||
try {
|
||||
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass -Force -ErrorAction Stop | Out-Null
|
||||
}
|
||||
catch {
|
||||
# AI/LLM: a GPO can refuse the process override. RemoteSigned still runs local scripts.
|
||||
# CURSOR: a GPO can refuse the process override. RemoteSigned still runs local scripts.
|
||||
}
|
||||
|
||||
$pythonExe = Resolve-FionaPython -Version $PythonVersion
|
||||
@@ -207,49 +174,44 @@ if (-not $pythonExe) {
|
||||
return
|
||||
}
|
||||
|
||||
$pipPython = $pythonExe
|
||||
if (-not $NoEnv) {
|
||||
$venvRoot = Join-Path $repoRoot ".venvs"
|
||||
$venvPath = Join-Path $venvRoot $EnvName
|
||||
$venvRoot = Join-Path $repoRoot ".venvs"
|
||||
$venvPath = Join-Path $venvRoot $EnvName
|
||||
|
||||
if ($Clean -and (Test-Path -LiteralPath $venvPath)) {
|
||||
Write-FionaStatus "Deleting existing virtual environment $EnvName..."
|
||||
try {
|
||||
Remove-Item -LiteralPath $venvPath -Recurse -Force
|
||||
}
|
||||
catch {
|
||||
Write-FionaError "Failed to delete virtual environment $EnvName"
|
||||
Write-FionaError $_.Exception.Message
|
||||
Remove-FionaBootstrapFunctions
|
||||
return
|
||||
}
|
||||
if ($Clean -and (Test-Path -LiteralPath $venvPath)) {
|
||||
Write-FionaStatus "Deleting existing virtual environment $EnvName..."
|
||||
try {
|
||||
Remove-Item -LiteralPath $venvPath -Recurse -Force
|
||||
}
|
||||
|
||||
$activate = Join-Path $venvPath "Scripts\Activate.ps1"
|
||||
if (-not (Test-Path -LiteralPath $activate)) {
|
||||
Write-FionaStatus "Creating virtual environment $EnvName with $pythonExe..."
|
||||
New-Item -ItemType Directory -Path $venvRoot -Force | Out-Null
|
||||
& $pythonExe -m venv $venvPath
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
Write-FionaError "Failed to create virtual environment $EnvName"
|
||||
Remove-FionaBootstrapFunctions
|
||||
return
|
||||
}
|
||||
}
|
||||
else {
|
||||
Write-FionaStatus "Virtual environment $EnvName already exists"
|
||||
}
|
||||
|
||||
Write-FionaStatus "Activating $EnvName..."
|
||||
. $activate
|
||||
|
||||
if (-not $env:VIRTUAL_ENV) {
|
||||
Write-FionaError "Failed to activate $EnvName"
|
||||
catch {
|
||||
Write-FionaError "Failed to delete virtual environment $EnvName"
|
||||
Write-FionaError $_.Exception.Message
|
||||
Remove-FionaBootstrapFunctions
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
$pipPython = Join-Path $venvPath "Scripts\python.exe"
|
||||
$activate = Join-Path $venvPath "Scripts\Activate.ps1"
|
||||
if (-not (Test-Path -LiteralPath $activate)) {
|
||||
Write-FionaStatus "Creating virtual environment $EnvName with $pythonExe..."
|
||||
New-Item -ItemType Directory -Path $venvRoot -Force | Out-Null
|
||||
& $pythonExe -m venv $venvPath
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
Write-FionaError "Failed to create virtual environment $EnvName"
|
||||
Remove-FionaBootstrapFunctions
|
||||
return
|
||||
}
|
||||
}
|
||||
else {
|
||||
Write-FionaStatus "Virtual environment $EnvName already exists"
|
||||
}
|
||||
|
||||
Write-FionaStatus "Activating $EnvName..."
|
||||
. $activate
|
||||
|
||||
if (-not $env:VIRTUAL_ENV) {
|
||||
Write-FionaError "Failed to activate $EnvName"
|
||||
Remove-FionaBootstrapFunctions
|
||||
return
|
||||
}
|
||||
|
||||
if ($env:PYTEST_ADDOPTS) {
|
||||
@@ -282,45 +244,15 @@ if (-not (Test-Path -LiteralPath $pyproject)) {
|
||||
}
|
||||
|
||||
Write-FionaStatus "Upgrading pip..."
|
||||
if ($NoEnv) {
|
||||
& $pipPython -m pip install --upgrade pip --user
|
||||
}
|
||||
else {
|
||||
& $pipPython -m pip install --upgrade pip
|
||||
}
|
||||
& python -m pip install --upgrade pip
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
Write-FionaWarning "Failed to upgrade pip. Continuing with package installation..."
|
||||
}
|
||||
|
||||
$helper = Join-Path $repoRoot "scripts\bootstrap_install.py"
|
||||
if (-not (Test-Path -LiteralPath $helper)) {
|
||||
Write-FionaError "Missing $helper"
|
||||
Remove-FionaBootstrapFunctions
|
||||
return
|
||||
}
|
||||
$helperArgs = @($helper)
|
||||
if ($Local) {
|
||||
$helperArgs += "--local"
|
||||
}
|
||||
if ($NoEnv) {
|
||||
$helperArgs += "--no-env"
|
||||
}
|
||||
$pipWords = @(& $pipPython @helperArgs | Where-Object { $_ -and "$_".Trim() -ne "" })
|
||||
if ($LASTEXITCODE -ne 0 -or $pipWords.Count -eq 0) {
|
||||
Write-FionaError "Failed to resolve pip install arguments"
|
||||
Remove-FionaBootstrapFunctions
|
||||
return
|
||||
}
|
||||
|
||||
if ($Local) {
|
||||
Write-FionaStatus "Installing this repository as a normal package..."
|
||||
}
|
||||
else {
|
||||
Write-FionaStatus "Installing package and dependencies from pyproject.toml..."
|
||||
}
|
||||
Write-FionaStatus "Installing package and dependencies from pyproject.toml..."
|
||||
Push-Location $repoRoot
|
||||
try {
|
||||
& $pipPython -m pip @pipWords
|
||||
& python -m pip install -e ".[dev]"
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
Write-FionaError "Failed to install package and dependencies from pyproject.toml"
|
||||
Remove-FionaBootstrapFunctions
|
||||
@@ -331,26 +263,18 @@ finally {
|
||||
Pop-Location
|
||||
}
|
||||
|
||||
$pythonVersionText = (& $pipPython --version 2>&1 | Out-String).Trim()
|
||||
$pipVersionText = (& $pipPython -m pip --version 2>&1 | Out-String).Trim()
|
||||
$pythonCmd = Get-Command python -ErrorAction SilentlyContinue
|
||||
$pythonWhere = if ($pythonCmd) { $pythonCmd.Source } else { "python" }
|
||||
$pythonVersionText = (& python --version 2>&1 | Out-String).Trim()
|
||||
$pipVersionText = (& python -m pip --version 2>&1 | Out-String).Trim()
|
||||
|
||||
Write-FionaStatus "Bootstrap completed successfully!"
|
||||
if ($NoEnv) {
|
||||
$scriptsDir = (& $pipPython -c "import sysconfig; print(sysconfig.get_path('scripts', 'nt_user'))" | Out-String).Trim()
|
||||
Write-FionaStatus "Installed for the current user (no virtual environment)"
|
||||
Write-FionaStatus "Command directory: $scriptsDir"
|
||||
Write-FionaStatus "If fiona-assistant is not found, add that directory to PATH."
|
||||
}
|
||||
else {
|
||||
Write-FionaStatus "Virtual environment '$EnvName' is active"
|
||||
}
|
||||
Write-FionaStatus "Python: $pythonVersionText ($pipPython)"
|
||||
Write-FionaStatus "Virtual environment '$EnvName' is active"
|
||||
Write-FionaStatus "Python: $pythonVersionText ($pythonWhere)"
|
||||
Write-FionaStatus "Pip: $pipVersionText"
|
||||
Write-FionaStatus ""
|
||||
Write-FionaStatus "To verify installation, try:"
|
||||
if (-not $NoEnv) {
|
||||
Write-FionaStatus " python scripts/run_tests.py"
|
||||
}
|
||||
Write-FionaStatus " python src/run_tests.py"
|
||||
Write-FionaStatus " fiona-assistant --help"
|
||||
Write-FionaStatus ""
|
||||
Write-FionaStatus "Happy coding!"
|
||||
|
||||
@@ -11,8 +11,6 @@ if ! python -m build --version >/dev/null 2>&1; then
|
||||
fi
|
||||
|
||||
# Build the distribution files
|
||||
rm -rf dist/ build/ fiona_assistant.egg-info/
|
||||
|
||||
python -m build
|
||||
|
||||
echo "Build completed. Distribution files are in the dist/ directory."
|
||||
|
||||
+612
@@ -0,0 +1,612 @@
|
||||
<?xml version="1.0" ?>
|
||||
<coverage version="7.16.1" timestamp="1790528640380" lines-valid="578" lines-covered="454" line-rate="0.7855" branches-covered="0" branches-valid="0" branch-rate="0" complexity="0">
|
||||
<!-- Generated by coverage.py: https://coverage.readthedocs.io/en/7.16.1 -->
|
||||
<!-- Based on https://raw.githubusercontent.com/cobertura/web/master/htdocs/xml/coverage-04.dtd -->
|
||||
<sources>
|
||||
<source>/home/user/repositories/fiona-assistant/src/customlinter</source>
|
||||
</sources>
|
||||
<packages>
|
||||
<package name="." line-rate="0.7855" branch-rate="0" complexity="0">
|
||||
<classes>
|
||||
<class name="__init__.py" filename="__init__.py" complexity="0" line-rate="1" branch-rate="0">
|
||||
<methods/>
|
||||
<lines>
|
||||
<line number="17" hits="1"/>
|
||||
<line number="26" hits="1"/>
|
||||
<line number="27" hits="1"/>
|
||||
<line number="29" hits="1"/>
|
||||
</lines>
|
||||
</class>
|
||||
<class name="base.py" filename="base.py" complexity="0" line-rate="0.6962" branch-rate="0">
|
||||
<methods/>
|
||||
<lines>
|
||||
<line number="8" hits="1"/>
|
||||
<line number="10" hits="1"/>
|
||||
<line number="11" hits="1"/>
|
||||
<line number="12" hits="1"/>
|
||||
<line number="13" hits="1"/>
|
||||
<line number="14" hits="1"/>
|
||||
<line number="15" hits="1"/>
|
||||
<line number="16" hits="1"/>
|
||||
<line number="18" hits="1"/>
|
||||
<line number="25" hits="1"/>
|
||||
<line number="28" hits="1"/>
|
||||
<line number="33" hits="1"/>
|
||||
<line number="49" hits="1"/>
|
||||
<line number="52" hits="1"/>
|
||||
<line number="65" hits="1"/>
|
||||
<line number="66" hits="1"/>
|
||||
<line number="67" hits="1"/>
|
||||
<line number="68" hits="1"/>
|
||||
<line number="72" hits="1"/>
|
||||
<line number="79" hits="1"/>
|
||||
<line number="90" hits="1"/>
|
||||
<line number="97" hits="1"/>
|
||||
<line number="118" hits="1"/>
|
||||
<line number="119" hits="1"/>
|
||||
<line number="120" hits="1"/>
|
||||
<line number="121" hits="1"/>
|
||||
<line number="122" hits="1"/>
|
||||
<line number="123" hits="1"/>
|
||||
<line number="124" hits="1"/>
|
||||
<line number="126" hits="1"/>
|
||||
<line number="128" hits="1"/>
|
||||
<line number="138" hits="1"/>
|
||||
<line number="140" hits="1"/>
|
||||
<line number="147" hits="1"/>
|
||||
<line number="164" hits="1"/>
|
||||
<line number="167" hits="1"/>
|
||||
<line number="170" hits="1"/>
|
||||
<line number="172" hits="1"/>
|
||||
<line number="178" hits="1"/>
|
||||
<line number="179" hits="1"/>
|
||||
<line number="181" hits="1"/>
|
||||
<line number="183" hits="1"/>
|
||||
<line number="184" hits="1"/>
|
||||
<line number="186" hits="1"/>
|
||||
<line number="187" hits="1"/>
|
||||
<line number="188" hits="1"/>
|
||||
<line number="190" hits="1"/>
|
||||
<line number="191" hits="1"/>
|
||||
<line number="192" hits="1"/>
|
||||
<line number="193" hits="1"/>
|
||||
<line number="194" hits="0"/>
|
||||
<line number="195" hits="1"/>
|
||||
<line number="196" hits="1"/>
|
||||
<line number="197" hits="1"/>
|
||||
<line number="210" hits="1"/>
|
||||
<line number="219" hits="0"/>
|
||||
<line number="220" hits="0"/>
|
||||
<line number="222" hits="0"/>
|
||||
<line number="223" hits="0"/>
|
||||
<line number="226" hits="0"/>
|
||||
<line number="229" hits="0"/>
|
||||
<line number="230" hits="0"/>
|
||||
<line number="231" hits="0"/>
|
||||
<line number="233" hits="0"/>
|
||||
<line number="235" hits="1"/>
|
||||
<line number="244" hits="1"/>
|
||||
<line number="247" hits="1"/>
|
||||
<line number="248" hits="1"/>
|
||||
<line number="250" hits="1"/>
|
||||
<line number="251" hits="1"/>
|
||||
<line number="252" hits="1"/>
|
||||
<line number="253" hits="1"/>
|
||||
<line number="254" hits="1"/>
|
||||
<line number="255" hits="1"/>
|
||||
<line number="256" hits="1"/>
|
||||
<line number="257" hits="0"/>
|
||||
<line number="258" hits="0"/>
|
||||
<line number="261" hits="1"/>
|
||||
<line number="262" hits="1"/>
|
||||
<line number="265" hits="1"/>
|
||||
<line number="266" hits="1"/>
|
||||
<line number="267" hits="1"/>
|
||||
<line number="270" hits="1"/>
|
||||
<line number="272" hits="1"/>
|
||||
<line number="273" hits="1"/>
|
||||
<line number="276" hits="1"/>
|
||||
<line number="278" hits="1"/>
|
||||
<line number="280" hits="1"/>
|
||||
<line number="290" hits="1"/>
|
||||
<line number="291" hits="1"/>
|
||||
<line number="292" hits="1"/>
|
||||
<line number="294" hits="1"/>
|
||||
<line number="295" hits="1"/>
|
||||
<line number="296" hits="1"/>
|
||||
<line number="298" hits="1"/>
|
||||
<line number="308" hits="1"/>
|
||||
<line number="309" hits="1"/>
|
||||
<line number="312" hits="1"/>
|
||||
<line number="313" hits="1"/>
|
||||
<line number="316" hits="1"/>
|
||||
<line number="317" hits="1"/>
|
||||
<line number="318" hits="1"/>
|
||||
<line number="319" hits="1"/>
|
||||
<line number="320" hits="1"/>
|
||||
<line number="321" hits="1"/>
|
||||
<line number="323" hits="1"/>
|
||||
<line number="325" hits="1"/>
|
||||
<line number="335" hits="1"/>
|
||||
<line number="336" hits="1"/>
|
||||
<line number="337" hits="1"/>
|
||||
<line number="338" hits="1"/>
|
||||
<line number="339" hits="1"/>
|
||||
<line number="340" hits="0"/>
|
||||
<line number="342" hits="0"/>
|
||||
<line number="343" hits="0"/>
|
||||
<line number="345" hits="1"/>
|
||||
<line number="354" hits="1"/>
|
||||
<line number="355" hits="1"/>
|
||||
<line number="357" hits="1"/>
|
||||
<line number="358" hits="1"/>
|
||||
<line number="359" hits="1"/>
|
||||
<line number="361" hits="1"/>
|
||||
<line number="363" hits="1"/>
|
||||
<line number="372" hits="1"/>
|
||||
<line number="374" hits="1"/>
|
||||
<line number="390" hits="0"/>
|
||||
<line number="391" hits="0"/>
|
||||
<line number="392" hits="0"/>
|
||||
<line number="393" hits="0"/>
|
||||
<line number="396" hits="0"/>
|
||||
<line number="397" hits="0"/>
|
||||
<line number="398" hits="0"/>
|
||||
<line number="399" hits="0"/>
|
||||
<line number="400" hits="0"/>
|
||||
<line number="406" hits="0"/>
|
||||
<line number="407" hits="0"/>
|
||||
<line number="408" hits="0"/>
|
||||
<line number="409" hits="0"/>
|
||||
<line number="412" hits="0"/>
|
||||
<line number="415" hits="0"/>
|
||||
<line number="416" hits="0"/>
|
||||
<line number="417" hits="0"/>
|
||||
<line number="418" hits="0"/>
|
||||
<line number="419" hits="0"/>
|
||||
<line number="420" hits="0"/>
|
||||
<line number="421" hits="0"/>
|
||||
<line number="423" hits="0"/>
|
||||
<line number="424" hits="0"/>
|
||||
<line number="425" hits="0"/>
|
||||
<line number="426" hits="0"/>
|
||||
<line number="427" hits="0"/>
|
||||
<line number="429" hits="0"/>
|
||||
<line number="430" hits="0"/>
|
||||
<line number="433" hits="0"/>
|
||||
<line number="434" hits="0"/>
|
||||
<line number="435" hits="0"/>
|
||||
<line number="436" hits="0"/>
|
||||
<line number="438" hits="0"/>
|
||||
</lines>
|
||||
</class>
|
||||
<class name="fmlint.py" filename="fmlint.py" complexity="0" line-rate="0.766" branch-rate="0">
|
||||
<methods/>
|
||||
<lines>
|
||||
<line number="19" hits="1"/>
|
||||
<line number="21" hits="1"/>
|
||||
<line number="22" hits="1"/>
|
||||
<line number="23" hits="1"/>
|
||||
<line number="24" hits="1"/>
|
||||
<line number="25" hits="1"/>
|
||||
<line number="27" hits="1"/>
|
||||
<line number="29" hits="1"/>
|
||||
<line number="37" hits="1"/>
|
||||
<line number="40" hits="1"/>
|
||||
<line number="43" hits="1"/>
|
||||
<line number="46" hits="1"/>
|
||||
<line number="53" hits="1"/>
|
||||
<line number="103" hits="1"/>
|
||||
<line number="115" hits="0"/>
|
||||
<line number="116" hits="0"/>
|
||||
<line number="117" hits="0"/>
|
||||
<line number="118" hits="0"/>
|
||||
<line number="119" hits="0"/>
|
||||
<line number="120" hits="0"/>
|
||||
<line number="121" hits="0"/>
|
||||
<line number="122" hits="0"/>
|
||||
<line number="125" hits="1"/>
|
||||
<line number="134" hits="1"/>
|
||||
<line number="137" hits="1"/>
|
||||
<line number="148" hits="1"/>
|
||||
<line number="151" hits="1"/>
|
||||
<line number="152" hits="1"/>
|
||||
<line number="155" hits="1"/>
|
||||
<line number="156" hits="1"/>
|
||||
<line number="158" hits="1"/>
|
||||
<line number="161" hits="1"/>
|
||||
<line number="180" hits="1"/>
|
||||
<line number="181" hits="1"/>
|
||||
<line number="183" hits="1"/>
|
||||
<line number="184" hits="1"/>
|
||||
<line number="186" hits="1"/>
|
||||
<line number="187" hits="1"/>
|
||||
<line number="190" hits="1"/>
|
||||
<line number="191" hits="1"/>
|
||||
<line number="192" hits="1"/>
|
||||
<line number="193" hits="1"/>
|
||||
<line number="196" hits="1"/>
|
||||
<line number="197" hits="1"/>
|
||||
<line number="198" hits="1"/>
|
||||
<line number="199" hits="1"/>
|
||||
<line number="202" hits="1"/>
|
||||
<line number="203" hits="1"/>
|
||||
<line number="210" hits="1"/>
|
||||
<line number="212" hits="1"/>
|
||||
<line number="213" hits="1"/>
|
||||
<line number="219" hits="1"/>
|
||||
<line number="226" hits="1"/>
|
||||
<line number="238" hits="1"/>
|
||||
<line number="239" hits="1"/>
|
||||
<line number="240" hits="1"/>
|
||||
<line number="242" hits="1"/>
|
||||
<line number="258" hits="1"/>
|
||||
<line number="261" hits="1"/>
|
||||
<line number="262" hits="1"/>
|
||||
<line number="263" hits="1"/>
|
||||
<line number="264" hits="1"/>
|
||||
<line number="265" hits="1"/>
|
||||
<line number="266" hits="1"/>
|
||||
<line number="267" hits="1"/>
|
||||
<line number="270" hits="1"/>
|
||||
<line number="272" hits="1"/>
|
||||
<line number="275" hits="1"/>
|
||||
<line number="278" hits="1"/>
|
||||
<line number="281" hits="1"/>
|
||||
<line number="282" hits="1"/>
|
||||
<line number="283" hits="1"/>
|
||||
<line number="286" hits="1"/>
|
||||
<line number="287" hits="1"/>
|
||||
<line number="290" hits="1"/>
|
||||
<line number="291" hits="1"/>
|
||||
<line number="293" hits="1"/>
|
||||
<line number="302" hits="1"/>
|
||||
<line number="304" hits="1"/>
|
||||
<line number="306" hits="1"/>
|
||||
<line number="307" hits="0"/>
|
||||
<line number="310" hits="1"/>
|
||||
<line number="311" hits="0"/>
|
||||
<line number="314" hits="1"/>
|
||||
<line number="315" hits="1"/>
|
||||
<line number="316" hits="1"/>
|
||||
<line number="318" hits="0"/>
|
||||
<line number="319" hits="0"/>
|
||||
<line number="320" hits="0"/>
|
||||
<line number="321" hits="0"/>
|
||||
<line number="322" hits="0"/>
|
||||
<line number="323" hits="0"/>
|
||||
<line number="325" hits="0"/>
|
||||
<line number="326" hits="0"/>
|
||||
<line number="327" hits="1"/>
|
||||
<line number="328" hits="0"/>
|
||||
<line number="331" hits="1"/>
|
||||
<line number="332" hits="1"/>
|
||||
<line number="333" hits="0"/>
|
||||
<line number="335" hits="0"/>
|
||||
<line number="343" hits="0"/>
|
||||
<line number="344" hits="0"/>
|
||||
<line number="347" hits="1"/>
|
||||
<line number="348" hits="1"/>
|
||||
<line number="349" hits="1"/>
|
||||
<line number="350" hits="1"/>
|
||||
<line number="358" hits="1"/>
|
||||
<line number="359" hits="1"/>
|
||||
<line number="362" hits="1"/>
|
||||
<line number="363" hits="1"/>
|
||||
<line number="371" hits="1"/>
|
||||
<line number="372" hits="1"/>
|
||||
<line number="375" hits="1"/>
|
||||
<line number="378" hits="1"/>
|
||||
<line number="384" hits="1"/>
|
||||
<line number="385" hits="1"/>
|
||||
<line number="386" hits="1"/>
|
||||
<line number="394" hits="1"/>
|
||||
<line number="397" hits="1"/>
|
||||
<line number="399" hits="1"/>
|
||||
<line number="400" hits="1"/>
|
||||
<line number="408" hits="1"/>
|
||||
<line number="410" hits="1"/>
|
||||
<line number="411" hits="1"/>
|
||||
<line number="413" hits="1"/>
|
||||
<line number="414" hits="1"/>
|
||||
<line number="416" hits="1"/>
|
||||
<line number="417" hits="1"/>
|
||||
<line number="418" hits="1"/>
|
||||
<line number="426" hits="1"/>
|
||||
<line number="427" hits="1"/>
|
||||
<line number="428" hits="1"/>
|
||||
<line number="436" hits="1"/>
|
||||
<line number="439" hits="1"/>
|
||||
<line number="444" hits="1"/>
|
||||
<line number="446" hits="1"/>
|
||||
<line number="473" hits="1"/>
|
||||
<line number="476" hits="1"/>
|
||||
<line number="487" hits="1"/>
|
||||
<line number="489" hits="1"/>
|
||||
<line number="491" hits="1"/>
|
||||
<line number="492" hits="1"/>
|
||||
<line number="493" hits="1"/>
|
||||
<line number="495" hits="0"/>
|
||||
<line number="496" hits="0"/>
|
||||
<line number="497" hits="0"/>
|
||||
<line number="498" hits="0"/>
|
||||
<line number="499" hits="0"/>
|
||||
<line number="500" hits="0"/>
|
||||
<line number="502" hits="0"/>
|
||||
<line number="503" hits="0"/>
|
||||
<line number="504" hits="1"/>
|
||||
<line number="505" hits="0"/>
|
||||
<line number="508" hits="1"/>
|
||||
<line number="511" hits="1"/>
|
||||
<line number="512" hits="1"/>
|
||||
<line number="513" hits="1"/>
|
||||
<line number="514" hits="1"/>
|
||||
<line number="516" hits="1"/>
|
||||
<line number="517" hits="1"/>
|
||||
<line number="518" hits="1"/>
|
||||
<line number="523" hits="1"/>
|
||||
<line number="524" hits="1"/>
|
||||
<line number="526" hits="1"/>
|
||||
<line number="527" hits="1"/>
|
||||
<line number="528" hits="1"/>
|
||||
<line number="530" hits="1"/>
|
||||
<line number="532" hits="1"/>
|
||||
<line number="548" hits="1"/>
|
||||
<line number="549" hits="1"/>
|
||||
<line number="550" hits="1"/>
|
||||
<line number="551" hits="1"/>
|
||||
<line number="552" hits="1"/>
|
||||
<line number="556" hits="1"/>
|
||||
<line number="558" hits="1"/>
|
||||
<line number="570" hits="1"/>
|
||||
<line number="572" hits="1"/>
|
||||
<line number="573" hits="1"/>
|
||||
<line number="575" hits="1"/>
|
||||
<line number="578" hits="1"/>
|
||||
<line number="579" hits="1"/>
|
||||
<line number="580" hits="1"/>
|
||||
<line number="583" hits="1"/>
|
||||
<line number="584" hits="1"/>
|
||||
<line number="587" hits="1"/>
|
||||
<line number="588" hits="0"/>
|
||||
<line number="591" hits="1"/>
|
||||
<line number="592" hits="0"/>
|
||||
<line number="594" hits="1"/>
|
||||
<line number="597" hits="1"/>
|
||||
<line number="598" hits="1"/>
|
||||
<line number="599" hits="1"/>
|
||||
<line number="600" hits="1"/>
|
||||
<line number="601" hits="1"/>
|
||||
<line number="602" hits="1"/>
|
||||
<line number="603" hits="1"/>
|
||||
<line number="605" hits="1"/>
|
||||
<line number="607" hits="1"/>
|
||||
<line number="617" hits="1"/>
|
||||
<line number="618" hits="1"/>
|
||||
<line number="619" hits="1"/>
|
||||
<line number="620" hits="1"/>
|
||||
<line number="621" hits="1"/>
|
||||
<line number="622" hits="0"/>
|
||||
<line number="623" hits="1"/>
|
||||
<line number="624" hits="1"/>
|
||||
<line number="625" hits="1"/>
|
||||
<line number="627" hits="1"/>
|
||||
<line number="637" hits="1"/>
|
||||
<line number="638" hits="1"/>
|
||||
<line number="639" hits="1"/>
|
||||
<line number="641" hits="1"/>
|
||||
<line number="642" hits="1"/>
|
||||
<line number="643" hits="1"/>
|
||||
<line number="645" hits="1"/>
|
||||
<line number="646" hits="1"/>
|
||||
<line number="647" hits="1"/>
|
||||
<line number="648" hits="1"/>
|
||||
<line number="649" hits="1"/>
|
||||
<line number="651" hits="1"/>
|
||||
<line number="652" hits="1"/>
|
||||
<line number="653" hits="0"/>
|
||||
<line number="655" hits="1"/>
|
||||
<line number="656" hits="1"/>
|
||||
<line number="658" hits="1"/>
|
||||
<line number="660" hits="1"/>
|
||||
<line number="671" hits="1"/>
|
||||
<line number="673" hits="1"/>
|
||||
<line number="674" hits="1"/>
|
||||
<line number="675" hits="1"/>
|
||||
<line number="676" hits="1"/>
|
||||
<line number="679" hits="1"/>
|
||||
<line number="680" hits="1"/>
|
||||
<line number="681" hits="1"/>
|
||||
<line number="682" hits="1"/>
|
||||
<line number="683" hits="1"/>
|
||||
<line number="684" hits="1"/>
|
||||
<line number="685" hits="1"/>
|
||||
<line number="688" hits="1"/>
|
||||
<line number="689" hits="1"/>
|
||||
<line number="690" hits="1"/>
|
||||
<line number="691" hits="1"/>
|
||||
<line number="692" hits="1"/>
|
||||
<line number="693" hits="1"/>
|
||||
<line number="695" hits="1"/>
|
||||
<line number="696" hits="1"/>
|
||||
<line number="699" hits="1"/>
|
||||
<line number="700" hits="1"/>
|
||||
<line number="701" hits="1"/>
|
||||
<line number="702" hits="1"/>
|
||||
<line number="703" hits="1"/>
|
||||
<line number="705" hits="1"/>
|
||||
<line number="707" hits="1"/>
|
||||
<line number="725" hits="0"/>
|
||||
<line number="726" hits="0"/>
|
||||
<line number="727" hits="0"/>
|
||||
<line number="728" hits="0"/>
|
||||
<line number="731" hits="0"/>
|
||||
<line number="732" hits="0"/>
|
||||
<line number="733" hits="0"/>
|
||||
<line number="734" hits="0"/>
|
||||
<line number="735" hits="0"/>
|
||||
<line number="741" hits="0"/>
|
||||
<line number="742" hits="0"/>
|
||||
<line number="743" hits="0"/>
|
||||
<line number="744" hits="0"/>
|
||||
<line number="745" hits="0"/>
|
||||
<line number="746" hits="0"/>
|
||||
<line number="747" hits="0"/>
|
||||
<line number="750" hits="0"/>
|
||||
<line number="751" hits="0"/>
|
||||
<line number="752" hits="0"/>
|
||||
<line number="753" hits="0"/>
|
||||
<line number="754" hits="0"/>
|
||||
<line number="755" hits="0"/>
|
||||
<line number="758" hits="0"/>
|
||||
<line number="759" hits="0"/>
|
||||
<line number="760" hits="0"/>
|
||||
<line number="761" hits="0"/>
|
||||
<line number="762" hits="0"/>
|
||||
<line number="763" hits="0"/>
|
||||
<line number="765" hits="0"/>
|
||||
<line number="766" hits="0"/>
|
||||
<line number="769" hits="0"/>
|
||||
<line number="770" hits="0"/>
|
||||
<line number="771" hits="0"/>
|
||||
<line number="772" hits="0"/>
|
||||
<line number="774" hits="0"/>
|
||||
<line number="781" hits="1"/>
|
||||
<line number="783" hits="1"/>
|
||||
<line number="784" hits="1"/>
|
||||
<line number="786" hits="1"/>
|
||||
<line number="788" hits="1"/>
|
||||
<line number="790" hits="1"/>
|
||||
<line number="793" hits="1"/>
|
||||
<line number="796" hits="1"/>
|
||||
<line number="800" hits="1"/>
|
||||
<line number="807" hits="1"/>
|
||||
<line number="809" hits="1"/>
|
||||
<line number="811" hits="1"/>
|
||||
<line number="812" hits="1"/>
|
||||
<line number="814" hits="1"/>
|
||||
<line number="815" hits="1"/>
|
||||
<line number="816" hits="1"/>
|
||||
<line number="818" hits="1"/>
|
||||
<line number="819" hits="1"/>
|
||||
<line number="820" hits="1"/>
|
||||
<line number="822" hits="1"/>
|
||||
<line number="823" hits="1"/>
|
||||
<line number="824" hits="1"/>
|
||||
<line number="826" hits="0"/>
|
||||
<line number="827" hits="0"/>
|
||||
</lines>
|
||||
</class>
|
||||
<class name="pycustomlint.py" filename="pycustomlint.py" complexity="0" line-rate="0.9712" branch-rate="0">
|
||||
<methods/>
|
||||
<lines>
|
||||
<line number="19" hits="1"/>
|
||||
<line number="21" hits="1"/>
|
||||
<line number="22" hits="1"/>
|
||||
<line number="23" hits="1"/>
|
||||
<line number="24" hits="1"/>
|
||||
<line number="25" hits="1"/>
|
||||
<line number="27" hits="1"/>
|
||||
<line number="29" hits="1"/>
|
||||
<line number="32" hits="1"/>
|
||||
<line number="40" hits="1"/>
|
||||
<line number="60" hits="1"/>
|
||||
<line number="63" hits="1"/>
|
||||
<line number="69" hits="1"/>
|
||||
<line number="70" hits="1"/>
|
||||
<line number="72" hits="1"/>
|
||||
<line number="74" hits="1"/>
|
||||
<line number="75" hits="1"/>
|
||||
<line number="76" hits="1"/>
|
||||
<line number="78" hits="1"/>
|
||||
<line number="80" hits="1"/>
|
||||
<line number="81" hits="1"/>
|
||||
<line number="82" hits="1"/>
|
||||
<line number="84" hits="1"/>
|
||||
<line number="87" hits="1"/>
|
||||
<line number="88" hits="1"/>
|
||||
<line number="89" hits="1"/>
|
||||
<line number="90" hits="1"/>
|
||||
<line number="91" hits="1"/>
|
||||
<line number="92" hits="1"/>
|
||||
<line number="93" hits="1"/>
|
||||
<line number="94" hits="1"/>
|
||||
<line number="95" hits="1"/>
|
||||
<line number="97" hits="1"/>
|
||||
<line number="105" hits="1"/>
|
||||
<line number="107" hits="1"/>
|
||||
<line number="109" hits="1"/>
|
||||
<line number="116" hits="1"/>
|
||||
<line number="122" hits="1"/>
|
||||
<line number="123" hits="1"/>
|
||||
<line number="124" hits="1"/>
|
||||
<line number="126" hits="1"/>
|
||||
<line number="135" hits="1"/>
|
||||
<line number="137" hits="1"/>
|
||||
<line number="139" hits="1"/>
|
||||
<line number="140" hits="1"/>
|
||||
<line number="143" hits="1"/>
|
||||
<line number="144" hits="0"/>
|
||||
<line number="147" hits="1"/>
|
||||
<line number="148" hits="1"/>
|
||||
<line number="151" hits="1"/>
|
||||
<line number="152" hits="1"/>
|
||||
<line number="153" hits="1"/>
|
||||
<line number="155" hits="1"/>
|
||||
<line number="157" hits="1"/>
|
||||
<line number="158" hits="1"/>
|
||||
<line number="159" hits="1"/>
|
||||
<line number="161" hits="1"/>
|
||||
<line number="163" hits="1"/>
|
||||
<line number="177" hits="1"/>
|
||||
<line number="178" hits="1"/>
|
||||
<line number="179" hits="1"/>
|
||||
<line number="181" hits="1"/>
|
||||
<line number="182" hits="1"/>
|
||||
<line number="185" hits="1"/>
|
||||
<line number="188" hits="1"/>
|
||||
<line number="189" hits="1"/>
|
||||
<line number="190" hits="1"/>
|
||||
<line number="192" hits="1"/>
|
||||
<line number="194" hits="1"/>
|
||||
<line number="196" hits="1"/>
|
||||
<line number="203" hits="1"/>
|
||||
<line number="205" hits="1"/>
|
||||
<line number="207" hits="1"/>
|
||||
<line number="209" hits="1"/>
|
||||
<line number="210" hits="1"/>
|
||||
<line number="212" hits="1"/>
|
||||
<line number="215" hits="1"/>
|
||||
<line number="218" hits="1"/>
|
||||
<line number="222" hits="1"/>
|
||||
<line number="228" hits="1"/>
|
||||
<line number="234" hits="1"/>
|
||||
<line number="236" hits="1"/>
|
||||
<line number="238" hits="1"/>
|
||||
<line number="239" hits="1"/>
|
||||
<line number="240" hits="1"/>
|
||||
<line number="242" hits="1"/>
|
||||
<line number="243" hits="1"/>
|
||||
<line number="245" hits="1"/>
|
||||
<line number="247" hits="1"/>
|
||||
<line number="248" hits="1"/>
|
||||
<line number="249" hits="1"/>
|
||||
<line number="252" hits="1"/>
|
||||
<line number="253" hits="1"/>
|
||||
<line number="255" hits="1"/>
|
||||
<line number="257" hits="1"/>
|
||||
<line number="258" hits="1"/>
|
||||
<line number="260" hits="1"/>
|
||||
<line number="262" hits="1"/>
|
||||
<line number="263" hits="1"/>
|
||||
<line number="264" hits="1"/>
|
||||
<line number="266" hits="1"/>
|
||||
<line number="267" hits="1"/>
|
||||
<line number="269" hits="0"/>
|
||||
<line number="270" hits="0"/>
|
||||
</lines>
|
||||
</class>
|
||||
</classes>
|
||||
</package>
|
||||
</packages>
|
||||
</coverage>
|
||||
+20
-28
@@ -1,32 +1,29 @@
|
||||
[build-system]
|
||||
requires = ["setuptools>=61"]
|
||||
requires = ["setuptools>=42"]
|
||||
build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "fiona-assistant"
|
||||
version = "1.19.0"
|
||||
version = "0.36.0"
|
||||
description = "Tools for managing the LLM wiki."
|
||||
authors = [{name = "nvaldeziii"}]
|
||||
readme = "README.md"
|
||||
license = "GPL-3.0-only"
|
||||
requires-python = ">=3.11"
|
||||
requires-python = ">=3.8"
|
||||
classifiers = [
|
||||
"Programming Language :: Python :: 3",
|
||||
"Programming Language :: Python :: 3.11",
|
||||
"Programming Language :: Python :: 3.12",
|
||||
"Programming Language :: Python :: 3.13",
|
||||
"Operating System :: OS Independent",
|
||||
]
|
||||
|
||||
# Core dependencies (required for runtime)
|
||||
# Note: html5validator requires Java 8+ to be installed on the system
|
||||
dependencies = [
|
||||
"pymarkdownlnt>=0.9.0",
|
||||
"ruff>=0.16.8",
|
||||
"pyyaml>=6.0.3",
|
||||
"pylint",
|
||||
"GitPython>=3.1.0",
|
||||
"beautifulsoup4>=4.12.0",
|
||||
"markdownify>=1.1.0",
|
||||
"pypdf>=5.0.0",
|
||||
"html5validator>=0.4.0",
|
||||
]
|
||||
|
||||
# Optional dependencies groups
|
||||
@@ -35,7 +32,6 @@ dev = [
|
||||
"pytest>=7.0.0",
|
||||
"pytest-cov>=4.0.0",
|
||||
"pytest-mock",
|
||||
"pylint",
|
||||
"build",
|
||||
"twine",
|
||||
]
|
||||
@@ -45,23 +41,23 @@ include-package-data = true
|
||||
|
||||
[tool.setuptools.packages.find]
|
||||
where = ["src"]
|
||||
include = ["fiona*"]
|
||||
# llm-wiki is a hyphenated, non-importable data-only "package" that ships
|
||||
# template files (wiki/, ingest/, AGENTS.md, etc.) as package data.
|
||||
# The hyphen makes it unimportable as a Python module (import llm-wiki
|
||||
# is a SyntaxError), but setuptools allows it as a distribution package name.
|
||||
# The __init__.py exists only to satisfy setuptools' package detection.
|
||||
include = ["fiona", "librarian", "mdoctor", "python_ruff", "customlinter", "sanity_runner", "llm-wiki", "wiki", "gittool", "html_js_lint"]
|
||||
|
||||
[tool.setuptools.package-data]
|
||||
fiona = [
|
||||
"data/llm_wiki/**/*",
|
||||
"data/default.gitignore",
|
||||
"wiki/templates/*",
|
||||
"gittool/templates/*",
|
||||
]
|
||||
# Include all files under llm-wiki/ as package data (not Python code)
|
||||
"llm-wiki" = ["*", "**/*"]
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["tests"]
|
||||
pythonpath = ["src"]
|
||||
testpaths = ["src/tests"]
|
||||
|
||||
[tool.coverage.run]
|
||||
source = ["src/fiona"]
|
||||
omit = ["*/tests/*", "*/test_*.py"]
|
||||
source = ["src"]
|
||||
omit = ["src/tests/*", "*/tests/*", "*/test_*.py"]
|
||||
|
||||
[tool.coverage.report]
|
||||
exclude_lines = [
|
||||
@@ -77,10 +73,6 @@ exclude_lines = [
|
||||
# Define console scripts entry points
|
||||
[project.scripts]
|
||||
fiona-assistant = "fiona.__main__:main"
|
||||
librarian = "fiona.librarian.librarian:main"
|
||||
mdoctor = "fiona.mdoctor.mdoctor:main"
|
||||
|
||||
[tool.ruff]
|
||||
target-version = "py311"
|
||||
line-length = 100
|
||||
src = ["src"]
|
||||
librarian = "librarian.librarian:main"
|
||||
mdoctor = "mdoctor.mdoctor:main"
|
||||
html-js-lint = "html_js_lint.html_js_lint:main"
|
||||
|
||||
@@ -1,83 +0,0 @@
|
||||
"""Pip arguments for bootstrap install modes.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - local and no-env install options.
|
||||
|
||||
SPEC 7.21. Both ``bootstrap`` and ``bootstrap.ps1`` call this script
|
||||
and read one pip argument per stdout line.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
|
||||
|
||||
def pip_install_args(
|
||||
*,
|
||||
local: bool,
|
||||
no_env: bool,
|
||||
break_system_packages: bool = False,
|
||||
) -> list[str]:
|
||||
"""Return the arguments passed to ``python -m pip``.
|
||||
|
||||
Args:
|
||||
local: Install this repository as a normal package. When
|
||||
false, the install is editable and includes the ``dev``
|
||||
extra.
|
||||
no_env: Add ``--user`` so the install does not need a virtual
|
||||
environment.
|
||||
break_system_packages: Add ``--break-system-packages``. Bash
|
||||
sets this. PowerShell does not.
|
||||
|
||||
Returns:
|
||||
The pip argument list, starting with ``install``.
|
||||
"""
|
||||
args = ["install"]
|
||||
if no_env:
|
||||
args.append("--user")
|
||||
if break_system_packages:
|
||||
args.append("--break-system-packages")
|
||||
if local:
|
||||
args.append(".")
|
||||
else:
|
||||
args.extend(["-e", ".[dev]"])
|
||||
return args
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
"""Print pip arguments, one per line.
|
||||
|
||||
Args:
|
||||
argv: Flags to parse. ``None`` reads ``sys.argv``.
|
||||
|
||||
Returns:
|
||||
Process exit code. ``0`` after the arguments are written.
|
||||
"""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Print pip install arguments for bootstrap.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--local",
|
||||
action="store_true",
|
||||
help="install this repository as a normal package",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--no-env",
|
||||
action="store_true",
|
||||
help="add --user so no virtual environment is required",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--break-system-packages",
|
||||
action="store_true",
|
||||
help="pass --break-system-packages to pip",
|
||||
)
|
||||
parsed = parser.parse_args(argv)
|
||||
words = pip_install_args(
|
||||
local=parsed.local,
|
||||
no_env=parsed.no_env,
|
||||
break_system_packages=parsed.break_system_packages,
|
||||
)
|
||||
sys.stdout.write("\n".join(words) + "\n")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -1,17 +0,0 @@
|
||||
"""Repo-root launcher for the sanity runner.
|
||||
|
||||
Generated by AI/LLM: 2026-09-29 - implementation lives in fiona.run_sanity.
|
||||
"""
|
||||
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
_SRC = pathlib.Path(__file__).resolve().parent.parent / "src"
|
||||
if str(_SRC) not in sys.path:
|
||||
sys.path.insert(0, str(_SRC))
|
||||
|
||||
import fiona.run_sanity as runner # noqa: E402
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(runner.main())
|
||||
@@ -1,17 +0,0 @@
|
||||
"""Repo-root launcher for the test runner.
|
||||
|
||||
Generated by AI/LLM: 2026-09-29 - implementation lives in fiona.run_tests.
|
||||
"""
|
||||
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
_SRC = pathlib.Path(__file__).resolve().parent.parent / "src"
|
||||
if str(_SRC) not in sys.path:
|
||||
sys.path.insert(0, str(_SRC))
|
||||
|
||||
import fiona.run_tests as runner # noqa: E402
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(runner.main())
|
||||
+15
-42
@@ -7,7 +7,7 @@ A collection of tools for managing the LLM wiki and related operations.
|
||||
The Fiona Assistant project provides command-line tools for:
|
||||
|
||||
- **Librarian**: Managing LLM wiki content and operations
|
||||
- **MDoctor**: Markdown diagnosis (lint, list, and fix)
|
||||
- **MDoctor**: [Description to be added]
|
||||
|
||||
## Installation
|
||||
|
||||
@@ -41,14 +41,6 @@ The Fiona Assistant project provides command-line tools for:
|
||||
. .\bootstrap.ps1 --env fiona_env --python 3.11.0
|
||||
```
|
||||
|
||||
Install this checkout as a normal package with `--local`. Skip the
|
||||
virtual environment with `--no-env` (PowerShell may use `-File`
|
||||
instead of dot-sourcing):
|
||||
|
||||
```powershell
|
||||
powershell -ExecutionPolicy Bypass -File .\bootstrap.ps1 --local --no-env
|
||||
```
|
||||
|
||||
3. Install the package in development mode:
|
||||
|
||||
```bash
|
||||
@@ -65,7 +57,7 @@ pip install -e ".[dev]"
|
||||
|
||||
### Librarian
|
||||
|
||||
See [librarian/README.md](./fiona/librarian/README.md) for detailed usage.
|
||||
See [librarian/README.md](./librarian/README.md) for detailed usage.
|
||||
|
||||
```bash
|
||||
librarian --help
|
||||
@@ -73,7 +65,7 @@ librarian --help
|
||||
|
||||
### MDoctor
|
||||
|
||||
See [mdoctor/README.md](./fiona/mdoctor/README.md) for detailed usage.
|
||||
See [mdoctor/README.md](./mdoctor/README.md) for detailed usage.
|
||||
|
||||
```bash
|
||||
mdoctor --help
|
||||
@@ -82,38 +74,22 @@ mdoctor --help
|
||||
## Project Structure
|
||||
|
||||
```text
|
||||
src/fiona/
|
||||
├── __main__.py # CLI entry point
|
||||
├── librarian/ # Wiki content management
|
||||
├── mdoctor/ # Markdown diagnosis
|
||||
├── customlinter/ # Frontmatter, markdown, Python, and JSON linters
|
||||
├── python_ruff/ # Ruff wrapper
|
||||
├── gittool/ # Git operations
|
||||
├── sanity_runner/ # Shared sanity engine
|
||||
├── wiki/ # Wiki init, update, sanity, and graph
|
||||
├── data/llm_wiki/ # Packaged wiki template
|
||||
├── run_tests.py # Test runner
|
||||
└── run_sanity.py # Sanity runner
|
||||
tests/ # Unit tests (repository root, not installed)
|
||||
scripts/
|
||||
├── run_tests.py
|
||||
├── run_sanity.py
|
||||
├── settings_dev.json
|
||||
├── utils.sh
|
||||
└── install_packages.sh
|
||||
src/
|
||||
├── fiona/ # Main package
|
||||
├── librarian/ # Wiki content management tool
|
||||
├── mdoctor/ # [Tool description]
|
||||
├── tests/ # Unit tests
|
||||
└── README.md # This file
|
||||
```
|
||||
|
||||
## Running Tests
|
||||
|
||||
```bash
|
||||
# Run all tests
|
||||
python scripts/run_tests.py
|
||||
|
||||
# Or use pytest directly
|
||||
python -m pytest
|
||||
pytest
|
||||
|
||||
# Run with coverage
|
||||
python -m pytest --cov=src/fiona
|
||||
pytest --cov=src
|
||||
```
|
||||
|
||||
## Development Tools
|
||||
@@ -122,10 +98,9 @@ The project includes several development utilities:
|
||||
|
||||
- `bootstrap`: Set up the Python development environment (bash)
|
||||
- `bootstrap.ps1`: Set up the Python development environment (PowerShell)
|
||||
- `scripts/utils.sh`: Shell utility functions
|
||||
- `scripts/install_packages.sh`: Install local Python packages
|
||||
- `scripts/run_tests.py`: Run the test suite
|
||||
- `scripts/run_sanity.py`: Run tests and linters
|
||||
- `src/utils.sh`: Shell utility functions
|
||||
- `src/install_packages.sh`: Install local Python packages
|
||||
- `src/run_tests.sh`: Run the test suite
|
||||
|
||||
## License
|
||||
|
||||
@@ -133,6 +108,4 @@ GPL-3.0-only
|
||||
|
||||
## Contributing
|
||||
|
||||
Follow [AGENTS.md](../AGENTS.md): update [SPEC.md](../SPEC.md) first,
|
||||
add a failing test, then implement. Run `python scripts/run_tests.py`
|
||||
and `python scripts/run_sanity.py --non-interactive` before finishing.
|
||||
[Contribution guidelines to be added]
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
# Fiona tools package
|
||||
@@ -16,7 +16,7 @@ various code and content rules. It includes:
|
||||
All linters share a common interface and output format for consistency.
|
||||
"""
|
||||
|
||||
from fiona.customlinter.base import (
|
||||
from customlinter.base import (
|
||||
DEFAULT_SETTINGS_FILE,
|
||||
READONLY_ZONES,
|
||||
BaseLinter,
|
||||
@@ -25,9 +25,9 @@ from fiona.customlinter.base import (
|
||||
LinterError,
|
||||
LintViolation,
|
||||
)
|
||||
from fiona.customlinter.fmlint import FrontmatterLinter
|
||||
from fiona.customlinter.mdlinter import MarkdownLinter
|
||||
from fiona.customlinter.pycustomlint import PythonCustomLinter
|
||||
from customlinter.fmlint import FrontmatterLinter
|
||||
from customlinter.mdlinter import MarkdownLinter
|
||||
from customlinter.pycustomlint import PythonCustomLinter
|
||||
|
||||
__all__ = [
|
||||
"DEFAULT_SETTINGS_FILE",
|
||||
@@ -15,15 +15,19 @@ import pathlib
|
||||
from abc import ABC, abstractmethod
|
||||
from typing import Any, ClassVar
|
||||
|
||||
from fiona.paths import llm_wiki_template
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# =============================================================================
|
||||
# Constants
|
||||
# =============================================================================
|
||||
|
||||
# Read-only zone names that should be skipped
|
||||
READONLY_ZONES = ["personal", "raw"]
|
||||
|
||||
# Default settings file path
|
||||
DEFAULT_SETTINGS_FILE = llm_wiki_template() / "fiona-settings.json"
|
||||
DEFAULT_SETTINGS_FILE = (
|
||||
pathlib.Path(__file__).resolve().parent.parent / "llm-wiki" / "fiona-settings.json"
|
||||
)
|
||||
|
||||
# Re-export constants for backward compatibility
|
||||
__all__ = [
|
||||
@@ -37,20 +41,6 @@ __all__ = [
|
||||
]
|
||||
|
||||
|
||||
def report_path(path: pathlib.Path | str) -> str:
|
||||
"""Return a forward-slash path for reports.
|
||||
|
||||
Generated by AI/LLM: 2026-09-29 - str(Path) uses backslashes on Windows,
|
||||
which makes report text differ from the POSIX form the checks expect.
|
||||
|
||||
Args:
|
||||
path: Path to display.
|
||||
|
||||
Returns:
|
||||
The path with forward slashes.
|
||||
"""
|
||||
return pathlib.Path(path).as_posix()
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Exceptions
|
||||
@@ -66,7 +56,7 @@ class LinterError(Exception):
|
||||
line: int | None = None,
|
||||
):
|
||||
"""Initialize the exception with details.
|
||||
|
||||
|
||||
Args:
|
||||
message: The error message describing the linting failure.
|
||||
filepath: The path to the file that failed linting.
|
||||
@@ -81,14 +71,14 @@ class LinterError(Exception):
|
||||
# Backward compatibility aliases
|
||||
class CustomLintError(LinterError):
|
||||
"""Exception raised when custom Python linting fails.
|
||||
|
||||
|
||||
This is an alias for LinterError for backward compatibility.
|
||||
"""
|
||||
|
||||
|
||||
class FrontmatterLintError(LinterError):
|
||||
"""Exception raised when frontmatter validation fails.
|
||||
|
||||
|
||||
This is an alias for LinterError for backward compatibility.
|
||||
"""
|
||||
|
||||
@@ -99,7 +89,7 @@ class FrontmatterLintError(LinterError):
|
||||
|
||||
class LintViolation:
|
||||
"""Represents a single linting violation.
|
||||
|
||||
|
||||
This class provides a standardized format for representing violations
|
||||
across all linter types.
|
||||
"""
|
||||
@@ -115,7 +105,7 @@ class LintViolation:
|
||||
severity: str = "error",
|
||||
):
|
||||
"""Initialize a violation.
|
||||
|
||||
|
||||
Args:
|
||||
filepath: Path to the file containing the violation.
|
||||
line: Line number of the violation.
|
||||
@@ -136,7 +126,7 @@ class LintViolation:
|
||||
def to_dict(self) -> dict:
|
||||
"""Convert violation to dictionary for reporting."""
|
||||
return {
|
||||
"file": report_path(self.filepath),
|
||||
"file": str(self.filepath),
|
||||
"line": self.line,
|
||||
"column": self.column,
|
||||
"code": self.code,
|
||||
@@ -147,10 +137,7 @@ class LintViolation:
|
||||
|
||||
def __str__(self) -> str:
|
||||
"""Format violation for display."""
|
||||
return (
|
||||
f"{report_path(self.filepath)}:{self.line}:{self.column} "
|
||||
f"{self.code} {self.message}"
|
||||
)
|
||||
return f"{self.filepath}:{self.line}:{self.column} {self.code} {self.message}"
|
||||
|
||||
|
||||
# =============================================================================
|
||||
@@ -159,14 +146,14 @@ class LintViolation:
|
||||
|
||||
class BaseLinter(ABC):
|
||||
"""Abstract base class for all custom linters.
|
||||
|
||||
|
||||
This class provides common functionality for all linters including:
|
||||
- Settings loading
|
||||
- File collection
|
||||
- Pattern matching
|
||||
- Read-only zone and .gitignore support
|
||||
- Consistent violation formatting
|
||||
|
||||
|
||||
Subclasses must implement:
|
||||
- lint_file() method
|
||||
- VIOLATION_CODES class attribute
|
||||
@@ -187,7 +174,7 @@ class BaseLinter(ABC):
|
||||
|
||||
def __init__(self, settings_file: pathlib.Path | None = None):
|
||||
"""Initialize the linter.
|
||||
|
||||
|
||||
Args:
|
||||
settings_file: Path to the settings file, or None for defaults.
|
||||
"""
|
||||
@@ -198,7 +185,7 @@ class BaseLinter(ABC):
|
||||
"""Load settings from the settings file."""
|
||||
if self.settings_file is None:
|
||||
self.settings_file = DEFAULT_SETTINGS_FILE
|
||||
|
||||
|
||||
try:
|
||||
with open(self.settings_file, encoding="utf-8") as f:
|
||||
settings = json.load(f)
|
||||
@@ -215,54 +202,54 @@ class BaseLinter(ABC):
|
||||
@abstractmethod
|
||||
def lint_file(self, filepath: pathlib.Path | str) -> list[LintViolation]:
|
||||
"""Lint a single file for violations.
|
||||
|
||||
|
||||
Args:
|
||||
filepath: Path to the file to lint.
|
||||
|
||||
|
||||
Returns:
|
||||
List of LintViolation objects.
|
||||
"""
|
||||
|
||||
def lint_directory(self, directory: str | pathlib.Path) -> list[LintViolation]:
|
||||
"""Lint all files in a directory tree.
|
||||
|
||||
|
||||
Args:
|
||||
directory: Path to the directory to lint.
|
||||
|
||||
|
||||
Returns:
|
||||
List of LintViolation objects from all files.
|
||||
"""
|
||||
directory = pathlib.Path(directory)
|
||||
violations: list[LintViolation] = []
|
||||
|
||||
|
||||
if not directory.exists():
|
||||
raise LinterError(f"Directory does not exist: {directory}")
|
||||
|
||||
|
||||
# Collect files
|
||||
files = self._collect_files(directory)
|
||||
|
||||
|
||||
# Lint each file
|
||||
for file in files:
|
||||
file_violations = self.lint_file(file)
|
||||
violations.extend(file_violations)
|
||||
|
||||
|
||||
return violations
|
||||
|
||||
def _collect_files(self, directory: pathlib.Path) -> list[pathlib.Path]:
|
||||
"""Collect all files to lint in a directory tree.
|
||||
|
||||
|
||||
Args:
|
||||
directory: Path to the directory to search.
|
||||
|
||||
|
||||
Returns:
|
||||
List of Path objects for files to lint.
|
||||
"""
|
||||
files: list[pathlib.Path] = []
|
||||
|
||||
|
||||
# Respect .gitignore
|
||||
gitignore_path = directory / ".gitignore"
|
||||
ignore_patterns: list[str] = []
|
||||
|
||||
|
||||
if gitignore_path.exists():
|
||||
try:
|
||||
with open(gitignore_path, encoding="utf-8") as f:
|
||||
@@ -272,34 +259,34 @@ class BaseLinter(ABC):
|
||||
ignore_patterns.append(line)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
# Walk the directory tree
|
||||
for root, dirs, filenames in os.walk(directory):
|
||||
root_path = pathlib.Path(root)
|
||||
|
||||
|
||||
# Skip read-only zones
|
||||
if any(zone in root_path.parts for zone in READONLY_ZONES):
|
||||
dirs.clear()
|
||||
continue
|
||||
|
||||
|
||||
# Skip based on .gitignore patterns
|
||||
dirs[:] = [d for d in dirs if not self._matches_pattern(d, ignore_patterns)]
|
||||
|
||||
|
||||
for filename in filenames:
|
||||
if any(filename.endswith(ext) for ext in self.file_extensions) and not self._matches_pattern(
|
||||
filename, ignore_patterns
|
||||
):
|
||||
files.append(root_path / filename)
|
||||
|
||||
|
||||
return files
|
||||
|
||||
def _matches_pattern(self, name: str, patterns: list[str]) -> bool:
|
||||
"""Check if a name matches any of the ignore patterns.
|
||||
|
||||
|
||||
Args:
|
||||
name: The name to check.
|
||||
patterns: List of patterns to match against.
|
||||
|
||||
|
||||
Returns:
|
||||
True if the name matches any pattern.
|
||||
"""
|
||||
@@ -313,21 +300,21 @@ class BaseLinter(ABC):
|
||||
|
||||
def _should_skip(self, filepath: pathlib.Path | str) -> bool:
|
||||
"""Check if a file should be skipped based on settings.
|
||||
|
||||
|
||||
Args:
|
||||
filepath: Path to the file (as Path or string).
|
||||
|
||||
|
||||
Returns:
|
||||
True if the file should be skipped.
|
||||
"""
|
||||
# Convert to Path if it's a string
|
||||
if isinstance(filepath, str):
|
||||
filepath = pathlib.Path(filepath)
|
||||
|
||||
|
||||
# Check read-only zones
|
||||
if any(zone in filepath.parts for zone in READONLY_ZONES):
|
||||
return True
|
||||
|
||||
|
||||
# Check ignore patterns from settings
|
||||
ignore_patterns = self._settings.get("ignore_patterns", [])
|
||||
if ignore_patterns:
|
||||
@@ -335,16 +322,16 @@ class BaseLinter(ABC):
|
||||
for pattern in ignore_patterns:
|
||||
if fnmatch.fnmatch(file_str, pattern):
|
||||
return True
|
||||
|
||||
|
||||
return False
|
||||
|
||||
def get_violation_message(self, violation_code: str, **kwargs) -> str:
|
||||
"""Get formatted violation message from VIOLATION_CODES dictionary.
|
||||
|
||||
|
||||
Args:
|
||||
violation_code: The violation code (e.g., 'PYC001', 'FML001').
|
||||
**kwargs: Keyword arguments to format the message template.
|
||||
|
||||
|
||||
Returns:
|
||||
Formatted violation message string.
|
||||
"""
|
||||
@@ -360,20 +347,20 @@ class BaseLinter(ABC):
|
||||
|
||||
def format_violations(self, violations: list[LintViolation]) -> str:
|
||||
"""Format violations for display.
|
||||
|
||||
|
||||
Args:
|
||||
violations: List of LintViolation objects.
|
||||
|
||||
|
||||
Returns:
|
||||
Formatted string with all violations.
|
||||
"""
|
||||
if not violations:
|
||||
return ""
|
||||
|
||||
|
||||
lines = []
|
||||
for violation in sorted(violations, key=lambda v: (str(v.filepath), v.line, v.column)):
|
||||
lines.append(str(violation))
|
||||
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
def compress_violations(
|
||||
@@ -410,18 +397,18 @@ class BaseLinter(ABC):
|
||||
"total_violations": 0,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
if target is not None:
|
||||
compressed_violations["meta"]["target"] = report_path(target_path)
|
||||
compressed_violations["meta"]["target"] = str(target_path)
|
||||
|
||||
for violation in violations:
|
||||
if target_path is not None:
|
||||
try:
|
||||
file_key = report_path(violation.filepath.relative_to(target_path))
|
||||
file_key = str(violation.filepath.relative_to(target_path))
|
||||
except ValueError:
|
||||
file_key = report_path(violation.filepath)
|
||||
file_key = str(violation.filepath)
|
||||
else:
|
||||
file_key = report_path(violation.filepath)
|
||||
file_key = str(violation.filepath)
|
||||
|
||||
if file_key not in compressed_violations["files"]:
|
||||
compressed_violations["files"][file_key] = []
|
||||
@@ -492,10 +479,10 @@ class BaseLinter(ABC):
|
||||
|
||||
def format_violations_json(self, violations: list[LintViolation]) -> str:
|
||||
"""Format violations as JSON.
|
||||
|
||||
|
||||
Args:
|
||||
violations: List of LintViolation objects.
|
||||
|
||||
|
||||
Returns:
|
||||
JSON string with all violations.
|
||||
"""
|
||||
@@ -508,12 +495,12 @@ class BaseLinter(ABC):
|
||||
linter_name: str,
|
||||
) -> str:
|
||||
"""Format violations in a comprehensive AI-friendly report.
|
||||
|
||||
|
||||
Args:
|
||||
violations: List of LintViolation objects.
|
||||
target: The target directory or file that was linted.
|
||||
linter_name: Name of the linter (for report header).
|
||||
|
||||
|
||||
Returns:
|
||||
Formatted report string.
|
||||
"""
|
||||
@@ -521,7 +508,7 @@ class BaseLinter(ABC):
|
||||
lines.append(f"{linter_name} Violations")
|
||||
lines.append(f"TARGET: {target}")
|
||||
lines.append("")
|
||||
|
||||
|
||||
# Collect all unique violation codes with their descriptions
|
||||
descriptions: dict[str, dict] = {}
|
||||
for violation in violations:
|
||||
@@ -531,7 +518,7 @@ class BaseLinter(ABC):
|
||||
"rule_name": code_info.get("rule_name", violation.code),
|
||||
"description": code_info.get("description", ""),
|
||||
}
|
||||
|
||||
|
||||
# Add definitions
|
||||
if descriptions:
|
||||
lines.append("- DEFINITIONS:")
|
||||
@@ -540,16 +527,16 @@ class BaseLinter(ABC):
|
||||
f" - {code}: {info['rule_name']}, {info['description']}"
|
||||
)
|
||||
lines.append("")
|
||||
|
||||
|
||||
# Add violations by file
|
||||
if violations:
|
||||
files_violations: dict[str, list[LintViolation]] = {}
|
||||
for violation in violations:
|
||||
file_key = report_path(violation.filepath)
|
||||
file_key = str(violation.filepath)
|
||||
if file_key not in files_violations:
|
||||
files_violations[file_key] = []
|
||||
files_violations[file_key].append(violation)
|
||||
|
||||
|
||||
for file_path, file_viols in sorted(files_violations.items()):
|
||||
lines.append(f"- FILE: {file_path}")
|
||||
for violation in sorted(file_viols, key=lambda v: (v.line, v.column)):
|
||||
@@ -558,11 +545,11 @@ class BaseLinter(ABC):
|
||||
else:
|
||||
lines.append("- No violations found.")
|
||||
lines.append("")
|
||||
|
||||
|
||||
# Add summary
|
||||
lines.append("- SUMMARY:")
|
||||
lines.append(f" - Total files with violations: {len(files_violations) if violations else 0}")
|
||||
lines.append(f" - Total violations: {len(violations)}")
|
||||
lines.append("")
|
||||
|
||||
|
||||
return "\n".join(lines)
|
||||
@@ -26,14 +26,13 @@ import pathlib
|
||||
import re
|
||||
import sys
|
||||
|
||||
from fiona.customlinter.base import (
|
||||
from customlinter.base import (
|
||||
BaseLinter,
|
||||
FrontmatterLintError,
|
||||
LintViolation,
|
||||
report_path,
|
||||
)
|
||||
from fiona.librarian.md_frontmatter import FrontmatterError, parse_frontmatter
|
||||
from fiona.logging_config import setup_logging
|
||||
from librarian.md_frontmatter import FrontmatterError, parse_frontmatter
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -696,7 +695,7 @@ class FrontmatterLinter(BaseLinter):
|
||||
# Group violations by file
|
||||
files_violations: dict[str, list[LintViolation]] = {}
|
||||
for violation in violations:
|
||||
file_key = report_path(violation.filepath)
|
||||
file_key = str(violation.filepath)
|
||||
if file_key not in files_violations:
|
||||
files_violations[file_key] = []
|
||||
files_violations[file_key].append(violation)
|
||||
@@ -9,7 +9,7 @@ Includes support for:
|
||||
- Key ordering (optional)
|
||||
- Auto-fixing certain violations
|
||||
|
||||
Follows the BaseLinter pattern from fiona.customlinter.base.
|
||||
Follows the BaseLinter pattern from customlinter.base.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -21,7 +21,7 @@ import re
|
||||
import sys
|
||||
from typing import Any, ClassVar
|
||||
|
||||
from fiona.customlinter.base import BaseLinter, LinterError, LintViolation
|
||||
from customlinter.base import BaseLinter, LinterError, LintViolation
|
||||
from fiona.logging_config import setup_logging
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -3,7 +3,7 @@
|
||||
This module provides a custom markdown linter for llm-wiki content,
|
||||
built on the customlinter framework (BaseLinter). It runs a registry of
|
||||
per-file checks over a wiki's markdown files and reports violations in
|
||||
the shared AI-friendly format (SPEC 6.1).
|
||||
the shared AI-friendly format (SPEC 4.9.3).
|
||||
|
||||
The first check is the dead-link check (moved from
|
||||
`wiki.sanity_manager.find_dead_links`):
|
||||
@@ -27,7 +27,7 @@ import pathlib
|
||||
import re
|
||||
from typing import Any, ClassVar
|
||||
|
||||
from fiona.customlinter.base import BaseLinter, LinterError, LintViolation
|
||||
from customlinter.base import BaseLinter, LinterError, LintViolation
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -67,14 +67,6 @@ VIOLATION_CODES = {
|
||||
},
|
||||
}
|
||||
|
||||
__all__ = [
|
||||
"FILE_EXTENSIONS",
|
||||
"SETTINGS_KEY_PATH",
|
||||
"VIOLATION_CODES",
|
||||
"MarkdownLinter",
|
||||
"find_dead_links",
|
||||
]
|
||||
|
||||
|
||||
def _is_ignorable_target(target: str) -> bool:
|
||||
"""Check whether a markdown link target is out of scope."""
|
||||
@@ -246,7 +238,7 @@ class MarkdownLinter(BaseLinter):
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Module-level Dead Link Helper (moved from fiona.wiki.sanity_manager)
|
||||
# Module-level Dead Link Helper (moved from wiki.sanity_manager)
|
||||
# =============================================================================
|
||||
|
||||
def find_dead_links(wiki_root: pathlib.Path) -> list[LintViolation]:
|
||||
@@ -270,3 +262,16 @@ def find_dead_links(wiki_root: pathlib.Path) -> list[LintViolation]:
|
||||
content = path.read_text(encoding="utf-8", errors="replace")
|
||||
violations.extend(linter._check_dead_links(path, content, md_stems))
|
||||
return violations
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Public API
|
||||
# =============================================================================
|
||||
|
||||
__all__ = [
|
||||
"FILE_EXTENSIONS",
|
||||
"SETTINGS_KEY_PATH",
|
||||
"VIOLATION_CODES",
|
||||
"MarkdownLinter",
|
||||
"find_dead_links",
|
||||
]
|
||||
@@ -11,7 +11,6 @@ The module provides functionality to:
|
||||
|
||||
Currently implemented rules:
|
||||
- Inline imports (imports inside functions/blocks instead of module level)
|
||||
- Late declarations (module-level assignments after the first function or class)
|
||||
|
||||
The framework is designed to be modular, allowing additional rules to be
|
||||
added in the future.
|
||||
@@ -26,7 +25,7 @@ import pathlib
|
||||
import sys
|
||||
from typing import ClassVar
|
||||
|
||||
from fiona.customlinter.base import (
|
||||
from customlinter.base import (
|
||||
BaseLinter,
|
||||
CustomLintError,
|
||||
LintViolation,
|
||||
@@ -47,10 +46,7 @@ READONLY_ZONES = ["personal", "raw"]
|
||||
VIOLATION_CODES = {
|
||||
"PYC001": {
|
||||
"rule_name": "Inline Import",
|
||||
"description": (
|
||||
"Import statement found inside a function or block (inline import). "
|
||||
"A module-level if/else before the first function or class is allowed"
|
||||
),
|
||||
"description": "Import statement found inside function or block (inline import)",
|
||||
"message_template": "PYC001 - Inline import at line {line}: {import_stmt}",
|
||||
"severity": "error"
|
||||
},
|
||||
@@ -60,17 +56,6 @@ VIOLATION_CODES = {
|
||||
"message_template": "PYC002 - Import after code at line {line}: {import_stmt}",
|
||||
"severity": "warning"
|
||||
},
|
||||
"PYC003": {
|
||||
"rule_name": "Late Declaration",
|
||||
"description": (
|
||||
"Module-level assignment found after the first function or class. "
|
||||
"Put declarations at the top of the file, before the first function or class"
|
||||
),
|
||||
"message_template": (
|
||||
"PYC003 - Declaration after first function or class at line {line}: {stmt}"
|
||||
),
|
||||
"severity": "error",
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
@@ -79,78 +64,26 @@ VIOLATION_CODES = {
|
||||
# =============================================================================
|
||||
|
||||
class InlineImportVisitor(ast.NodeVisitor):
|
||||
"""AST visitor to detect inline (indented) imports.
|
||||
|
||||
Modified by AI/LLM: 2026-09-29 - allow imports in a module-level
|
||||
if/else that appears before the first function or class (SPEC 6.4.2).
|
||||
"""
|
||||
"""AST visitor to detect inline (indented) imports."""
|
||||
|
||||
def __init__(self, filename: str):
|
||||
"""Initialize the visitor.
|
||||
|
||||
|
||||
Args:
|
||||
filename: The name of the file being analyzed.
|
||||
"""
|
||||
self.filename = filename
|
||||
self.violations: list[LintViolation] = []
|
||||
self._os_guard_depth = 0
|
||||
self._guard_if_ids: set[int] = set()
|
||||
|
||||
def visit_Module(self, node: ast.Module) -> None:
|
||||
"""Mark module-level if statements that sit before any function or class."""
|
||||
first_boundary = None
|
||||
for statement in node.body:
|
||||
if isinstance(statement, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
|
||||
first_boundary = statement.lineno
|
||||
break
|
||||
for statement in node.body:
|
||||
if first_boundary is not None and statement.lineno >= first_boundary:
|
||||
break
|
||||
if isinstance(statement, ast.If):
|
||||
self._guard_if_ids.add(id(statement))
|
||||
self.generic_visit(node)
|
||||
|
||||
def visit_If(self, node: ast.If) -> None:
|
||||
"""Keep the OS-guard depth raised while visiting an allowed if."""
|
||||
guarded = id(node) in self._guard_if_ids
|
||||
if guarded:
|
||||
self._os_guard_depth += 1
|
||||
self.generic_visit(node)
|
||||
if guarded:
|
||||
self._os_guard_depth -= 1
|
||||
|
||||
def visit_FunctionDef(self, node: ast.FunctionDef) -> None:
|
||||
"""Imports inside a function stay inline, even under an OS if."""
|
||||
self._visit_blocked_scope(node)
|
||||
|
||||
def visit_AsyncFunctionDef(self, node: ast.AsyncFunctionDef) -> None:
|
||||
"""Imports inside an async function stay inline."""
|
||||
self._visit_blocked_scope(node)
|
||||
|
||||
def visit_ClassDef(self, node: ast.ClassDef) -> None:
|
||||
"""Imports inside a class stay inline."""
|
||||
self._visit_blocked_scope(node)
|
||||
|
||||
def _visit_blocked_scope(self, node: ast.AST) -> None:
|
||||
"""Visit a function or class with the OS-guard exception off.
|
||||
|
||||
Modified by AI/LLM: 2026-09-29 - the SPEC 6.4.2 exception does
|
||||
not cover imports nested in a function or class.
|
||||
"""
|
||||
saved = self._os_guard_depth
|
||||
self._os_guard_depth = 0
|
||||
self.generic_visit(node)
|
||||
self._os_guard_depth = saved
|
||||
|
||||
def visit_Import(self, node: ast.Import) -> None:
|
||||
"""Visit import statement - check if indented."""
|
||||
if node.col_offset > 0 and self._os_guard_depth == 0:
|
||||
if node.col_offset > 0: # Indented = inline
|
||||
self._add_inline_import_violation(node)
|
||||
self.generic_visit(node)
|
||||
|
||||
def visit_ImportFrom(self, node: ast.ImportFrom) -> None:
|
||||
"""Visit import from statement - check if indented."""
|
||||
if node.col_offset > 0 and self._os_guard_depth == 0:
|
||||
if node.col_offset > 0: # Indented = inline
|
||||
self._add_inline_import_violation(node)
|
||||
self.generic_visit(node)
|
||||
|
||||
@@ -166,7 +99,7 @@ class InlineImportVisitor(ast.NodeVisitor):
|
||||
import_stmt = line.strip()
|
||||
except (OSError, IndexError, ValueError):
|
||||
pass
|
||||
|
||||
|
||||
violation = LintViolation(
|
||||
filepath=pathlib.Path(self.filename),
|
||||
line=node.lineno,
|
||||
@@ -182,110 +115,16 @@ class InlineImportVisitor(ast.NodeVisitor):
|
||||
return self.violations
|
||||
|
||||
|
||||
class LateDeclarationVisitor(ast.NodeVisitor):
|
||||
"""AST visitor for module-level assignments after the first function or class.
|
||||
|
||||
Modified by AI/LLM: 2026-09-30 - PYC003 late declarations (SPEC 6.4.3).
|
||||
"""
|
||||
|
||||
def __init__(self, filename: str) -> None:
|
||||
"""Initialize the visitor.
|
||||
|
||||
Args:
|
||||
filename: The name of the file being analyzed.
|
||||
"""
|
||||
self.filename = filename
|
||||
self.violations: list[LintViolation] = []
|
||||
self._boundary: int | None = None
|
||||
self._scope_depth = 0
|
||||
|
||||
def visit_Module(self, node: ast.Module) -> None:
|
||||
"""Record the first top-level function or class, then walk the module."""
|
||||
for statement in node.body:
|
||||
if isinstance(statement, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
|
||||
self._boundary = statement.lineno
|
||||
break
|
||||
self.generic_visit(node)
|
||||
|
||||
def visit_FunctionDef(self, node: ast.FunctionDef) -> None:
|
||||
"""Skip assignments that belong to a function."""
|
||||
self._visit_nested_scope(node)
|
||||
|
||||
def visit_AsyncFunctionDef(self, node: ast.AsyncFunctionDef) -> None:
|
||||
"""Skip assignments that belong to an async function."""
|
||||
self._visit_nested_scope(node)
|
||||
|
||||
def visit_ClassDef(self, node: ast.ClassDef) -> None:
|
||||
"""Skip assignments that belong to a class."""
|
||||
self._visit_nested_scope(node)
|
||||
|
||||
def _visit_nested_scope(self, node: ast.AST) -> None:
|
||||
"""Visit a function or class without treating its assignments as module-level.
|
||||
|
||||
Modified by AI/LLM: 2026-09-30 - class and function bodies are not
|
||||
declarations in the middle of the module (SPEC 6.4.3).
|
||||
"""
|
||||
self._scope_depth += 1
|
||||
self.generic_visit(node)
|
||||
self._scope_depth -= 1
|
||||
|
||||
def visit_Assign(self, node: ast.Assign) -> None:
|
||||
"""Flag a module-level assignment that follows the first function or class."""
|
||||
self._check_declaration(node)
|
||||
self.generic_visit(node)
|
||||
|
||||
def visit_AnnAssign(self, node: ast.AnnAssign) -> None:
|
||||
"""Flag a module-level annotated assignment after the first function or class."""
|
||||
self._check_declaration(node)
|
||||
self.generic_visit(node)
|
||||
|
||||
def _check_declaration(self, node: ast.Assign | ast.AnnAssign) -> None:
|
||||
"""Record PYC003 when this assignment is a late module declaration."""
|
||||
if self._scope_depth > 0 or self._boundary is None:
|
||||
return
|
||||
if node.lineno < self._boundary:
|
||||
return
|
||||
statement = self._source_line(node.lineno)
|
||||
self.violations.append(
|
||||
LintViolation(
|
||||
filepath=pathlib.Path(self.filename),
|
||||
line=node.lineno,
|
||||
column=node.col_offset,
|
||||
code="PYC003",
|
||||
message=(
|
||||
"PYC003 - Declaration after first function or class "
|
||||
f"at line {node.lineno}: {statement}"
|
||||
),
|
||||
rule_id="late-declaration",
|
||||
)
|
||||
)
|
||||
|
||||
def _source_line(self, lineno: int) -> str:
|
||||
"""Return the stripped source line for a violation message."""
|
||||
try:
|
||||
with open(self.filename, encoding="utf-8") as handle:
|
||||
lines = handle.readlines()
|
||||
if lineno <= len(lines):
|
||||
return lines[lineno - 1].strip()
|
||||
except (OSError, IndexError, ValueError):
|
||||
return "assignment"
|
||||
return "assignment"
|
||||
|
||||
def get_violations(self) -> list[LintViolation]:
|
||||
"""Get all detected violations."""
|
||||
return self.violations
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Python Custom Linter Class
|
||||
# =============================================================================
|
||||
|
||||
class PythonCustomLinter(BaseLinter):
|
||||
"""Main linter class for custom Python code rules.
|
||||
|
||||
|
||||
This class extends BaseLinter to provide Python-specific linting.
|
||||
"""
|
||||
|
||||
|
||||
VIOLATION_CODES: ClassVar[dict] = VIOLATION_CODES
|
||||
file_extensions: ClassVar[list[str]] = [".py"]
|
||||
settings_key_path: ClassVar[list[str]] = ["python", "custom_lint"]
|
||||
@@ -293,72 +132,69 @@ class PythonCustomLinter(BaseLinter):
|
||||
|
||||
def lint_file(self, filepath: pathlib.Path | str) -> list[LintViolation]:
|
||||
"""Lint a single Python file for custom rule violations.
|
||||
|
||||
|
||||
Args:
|
||||
filepath: Path to the Python file to lint (as Path or string).
|
||||
|
||||
|
||||
Returns:
|
||||
List of LintViolation objects.
|
||||
"""
|
||||
violations: list[LintViolation] = []
|
||||
|
||||
|
||||
try:
|
||||
# Convert to Path if it's a string
|
||||
if isinstance(filepath, str):
|
||||
filepath = pathlib.Path(filepath)
|
||||
|
||||
|
||||
# Check if file should be skipped
|
||||
if self._should_skip(filepath):
|
||||
return violations
|
||||
|
||||
|
||||
# Read and parse the file
|
||||
source = filepath.read_text(encoding="utf-8")
|
||||
tree = ast.parse(source, filename=str(filepath))
|
||||
|
||||
# Visit the AST to find inline imports and late declarations
|
||||
|
||||
# Visit the AST to find inline imports
|
||||
visitor = InlineImportVisitor(str(filepath))
|
||||
visitor.visit(tree)
|
||||
violations = visitor.get_violations()
|
||||
declarations = LateDeclarationVisitor(str(filepath))
|
||||
declarations.visit(tree)
|
||||
violations.extend(declarations.get_violations())
|
||||
|
||||
|
||||
except SyntaxError as e:
|
||||
# Syntax errors are not our concern
|
||||
logger.warning("Syntax error in %s: %s", filepath, e)
|
||||
except OSError as e:
|
||||
logger.warning("Could not read %s: %s", filepath, e)
|
||||
|
||||
|
||||
return violations
|
||||
|
||||
def lint_directory(self, directory: str | pathlib.Path) -> list[LintViolation]:
|
||||
"""Lint all files in a directory tree.
|
||||
|
||||
|
||||
This overrides the base class to raise CustomLintError instead of LinterError.
|
||||
|
||||
|
||||
Args:
|
||||
directory: Path to the directory to lint.
|
||||
|
||||
|
||||
Returns:
|
||||
List of LintViolation objects from all files.
|
||||
|
||||
|
||||
Raises:
|
||||
CustomLintError: If the directory does not exist.
|
||||
"""
|
||||
directory = pathlib.Path(directory)
|
||||
violations: list[LintViolation] = []
|
||||
|
||||
|
||||
if not directory.exists():
|
||||
raise CustomLintError(f"Directory does not exist: {directory}")
|
||||
|
||||
|
||||
# Collect files
|
||||
files = self._collect_files(directory)
|
||||
|
||||
|
||||
# Lint each file
|
||||
for file in files:
|
||||
file_violations = self.lint_file(file)
|
||||
violations.extend(file_violations)
|
||||
|
||||
|
||||
return violations
|
||||
|
||||
def _collect_python_files(self, directory: pathlib.Path) -> list[pathlib.Path]:
|
||||
@@ -372,15 +208,15 @@ class PythonCustomLinter(BaseLinter):
|
||||
|
||||
def main():
|
||||
"""Command-line interface for the Python custom linter."""
|
||||
|
||||
|
||||
setup_logging()
|
||||
logger.debug("Python custom linter starting")
|
||||
|
||||
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Python Custom Linter - Check for custom Python code rule violations."
|
||||
)
|
||||
subparsers = parser.add_subparsers(dest="subcommand", help="Available subcommands")
|
||||
|
||||
|
||||
# chklint subcommand
|
||||
chklint_parser = subparsers.add_parser(
|
||||
"chklint",
|
||||
@@ -397,35 +233,35 @@ def main():
|
||||
action="store_true",
|
||||
help="Output results in JSON format"
|
||||
)
|
||||
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
|
||||
if args.subcommand is None:
|
||||
# Default to chklint with current directory
|
||||
args.subcommand = "chklint"
|
||||
args.directory = "."
|
||||
args.json = False # Default value when not specified
|
||||
|
||||
|
||||
if args.subcommand == "chklint":
|
||||
linter = PythonCustomLinter()
|
||||
|
||||
|
||||
target = pathlib.Path(args.directory)
|
||||
|
||||
|
||||
if not target.exists():
|
||||
print(f"Error: '{args.directory}' does not exist.", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
# Support both files and directories
|
||||
if target.is_file():
|
||||
violations = linter.lint_file(target)
|
||||
else:
|
||||
violations = linter.lint_directory(target)
|
||||
|
||||
|
||||
if getattr(args, 'json', False):
|
||||
output = linter.format_violations_json(violations)
|
||||
else:
|
||||
output = linter.format_violations(violations)
|
||||
|
||||
|
||||
if output:
|
||||
print(output)
|
||||
sys.exit(1)
|
||||
+13
-105
@@ -8,21 +8,18 @@ import argparse
|
||||
import logging
|
||||
import sys
|
||||
|
||||
from fiona.customlinter.fmlint import main as fmlint_main
|
||||
from fiona.customlinter.jsonlint import main as jsonlint_main
|
||||
from fiona.gittool import diff_report
|
||||
from fiona.gittool.git_tool import main as git_tool_main
|
||||
from fiona.librarian.librarian import main as librarian_main
|
||||
from fiona.librarian.toc_generator import main as toc_generator_main
|
||||
from customlinter.fmlint import main as fmlint_main
|
||||
from customlinter.jsonlint import main as jsonlint_main
|
||||
from fiona.logging_config import setup_logging
|
||||
from fiona.mdoctor.mdoctor import main as mdoctor_main
|
||||
from fiona.python_ruff.python_ruff import main as python_ruff_main
|
||||
from fiona.wiki.graph_generator import wiki_graph_main
|
||||
from fiona.wiki.housekeeping import housekeeping_main
|
||||
from fiona.wiki.ingest.pipeline import add_ingest_arguments, execute_ingest
|
||||
from fiona.wiki.sanity_manager import sanity_main
|
||||
from fiona.wiki.update_manager import check_update_main, update_main
|
||||
from fiona.wiki.wiki import wiki_init
|
||||
from gittool.git_tool import main as git_tool_main
|
||||
from librarian.librarian import main as librarian_main
|
||||
from librarian.toc_generator import main as toc_generator_main
|
||||
from mdoctor.mdoctor import main as mdoctor_main
|
||||
from python_ruff.python_ruff import main as python_ruff_main
|
||||
from wiki.graph_generator import wiki_graph_main
|
||||
from wiki.sanity_manager import sanity_main
|
||||
from wiki.update_manager import check_update_main, update_main
|
||||
from wiki.wiki import wiki_init
|
||||
|
||||
|
||||
def main():
|
||||
@@ -131,39 +128,6 @@ def main():
|
||||
# fiona-assistant git status
|
||||
git_subparsers.add_parser("status", help="Show git status")
|
||||
|
||||
# fiona-assistant git diff
|
||||
git_diff_parser = git_subparsers.add_parser(
|
||||
"diff",
|
||||
help="Write constellation-themed HTML diff under wiki tmp/",
|
||||
)
|
||||
git_diff_parser.add_argument(
|
||||
"--base",
|
||||
help="Base revision (default: origin/main or main/master)",
|
||||
)
|
||||
git_diff_parser.add_argument("--head", default="HEAD", help="Head revision")
|
||||
git_diff_parser.add_argument("--title", help="Report H1 / page title")
|
||||
git_diff_parser.add_argument("--subtitle", help="Extra header line")
|
||||
git_diff_parser.add_argument(
|
||||
"-o",
|
||||
"--output",
|
||||
help="Output HTML path",
|
||||
)
|
||||
git_diff_parser.add_argument(
|
||||
"--collapsed",
|
||||
action="store_true",
|
||||
help="Start with per-file sections collapsed",
|
||||
)
|
||||
git_diff_parser.add_argument(
|
||||
"--staged",
|
||||
action="store_true",
|
||||
help="Diff only staged changes (ignores unstaged changes)",
|
||||
)
|
||||
git_diff_parser.add_argument(
|
||||
"--unstaged",
|
||||
action="store_true",
|
||||
help="Diff only unstaged changes (ignores staged changes)",
|
||||
)
|
||||
|
||||
# fiona-assistant wiki
|
||||
wiki_parser = subparsers.add_parser("wiki", help="Manage wiki directories")
|
||||
wiki_subparsers = wiki_parser.add_subparsers(dest="wiki_subcommand", help="Wiki subcommands")
|
||||
@@ -315,31 +279,6 @@ def main():
|
||||
help="Disable tracking of external links"
|
||||
)
|
||||
|
||||
# AI/LLM: 2026-09-30 - flags live here so wiki ingest --help is not rejected.
|
||||
wiki_ingest_parser = wiki_subparsers.add_parser(
|
||||
"ingest",
|
||||
prog="fiona-assistant wiki ingest",
|
||||
help="Archive one source and write a raw wiki page",
|
||||
)
|
||||
add_ingest_arguments(wiki_ingest_parser)
|
||||
|
||||
# AI/LLM: fiona-assistant housekeeping (SPEC 4.8)
|
||||
housekeeping_parser = subparsers.add_parser(
|
||||
"housekeeping",
|
||||
help="Run the wiki maintenance routine and report what needs work"
|
||||
)
|
||||
housekeeping_parser.add_argument(
|
||||
"directory",
|
||||
nargs="?",
|
||||
default=".",
|
||||
help="Wiki directory (default: current directory)"
|
||||
)
|
||||
housekeeping_parser.add_argument(
|
||||
"--json",
|
||||
action="store_true",
|
||||
help="Output results in JSON format"
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
setup_logging()
|
||||
logging.getLogger(__name__).debug("fiona-assistant command: %s", args.command)
|
||||
@@ -416,40 +355,13 @@ def main():
|
||||
elif args.git_subcommand == "status":
|
||||
sys.argv = ["git_tool", "status"]
|
||||
git_tool_main()
|
||||
elif args.git_subcommand == "diff":
|
||||
diff_argv: list[str] = []
|
||||
if args.base:
|
||||
diff_argv.extend(["--base", args.base])
|
||||
if args.head != "HEAD":
|
||||
diff_argv.extend(["--head", args.head])
|
||||
if args.title:
|
||||
diff_argv.extend(["--title", args.title])
|
||||
if args.subtitle:
|
||||
diff_argv.extend(["--subtitle", args.subtitle])
|
||||
if args.output is not None:
|
||||
diff_argv.extend(["--output", args.output])
|
||||
if args.collapsed:
|
||||
diff_argv.append("--collapsed")
|
||||
if args.staged:
|
||||
diff_argv.append("--staged")
|
||||
if args.unstaged:
|
||||
diff_argv.append("--unstaged")
|
||||
sys.exit(diff_report.main(diff_argv))
|
||||
else:
|
||||
git_parser.print_help()
|
||||
sys.exit(1)
|
||||
elif args.command == "wiki":
|
||||
if args.wiki_subcommand == "init":
|
||||
try:
|
||||
result = wiki_init(args.directory)
|
||||
print(result)
|
||||
except FileExistsError as e:
|
||||
# SPEC 7.1.1.4: format the error message and exit 1
|
||||
# The FileExistsError message is: "Directory '<path>' already contains llm-wiki files: <names>"
|
||||
# We want: "Error: Directory '<path>' already contains llm-wiki files: <names>. Run fiona-assistant wiki update to refresh the template."
|
||||
formatted_error = f"Error: {e}. Run fiona-assistant wiki update to refresh the template."
|
||||
print(formatted_error, file=sys.stderr)
|
||||
sys.exit(1)
|
||||
result = wiki_init(args.directory)
|
||||
print(result)
|
||||
elif args.wiki_subcommand == "check-update":
|
||||
sys.exit(check_update_main(
|
||||
directory=args.directory,
|
||||
@@ -469,8 +381,6 @@ def main():
|
||||
json_output=args.json,
|
||||
non_interactive=args.non_interactive
|
||||
))
|
||||
elif args.wiki_subcommand == "ingest":
|
||||
sys.exit(execute_ingest(args, wiki_ingest_parser))
|
||||
elif args.wiki_subcommand == "graph":
|
||||
sys.exit(wiki_graph_main(
|
||||
directory=args.directory,
|
||||
@@ -484,8 +394,6 @@ def main():
|
||||
else:
|
||||
wiki_parser.print_help()
|
||||
sys.exit(1)
|
||||
elif args.command == "housekeeping":
|
||||
sys.exit(housekeeping_main(directory=args.directory, json_output=args.json))
|
||||
else:
|
||||
parser.print_help()
|
||||
sys.exit(1)
|
||||
|
||||
@@ -1,63 +0,0 @@
|
||||
# Byte-compiled / optimized / DLL files
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
|
||||
# Virtual environments
|
||||
.venv/
|
||||
.venvs/
|
||||
venv/
|
||||
env/
|
||||
ENV/
|
||||
|
||||
# IDE-specific files
|
||||
.vscode/
|
||||
.idea/
|
||||
*.swp
|
||||
*.swo
|
||||
|
||||
# Local configuration
|
||||
.env
|
||||
*.local
|
||||
|
||||
# Build artifacts
|
||||
build/
|
||||
dist/
|
||||
*.egg-info/
|
||||
|
||||
# Logs and databases
|
||||
*.log
|
||||
*.sqlite
|
||||
*.db
|
||||
|
||||
# pytest
|
||||
.pytest_cache/
|
||||
|
||||
# Coverage
|
||||
.coverage
|
||||
coverage.xml
|
||||
htmlcov/
|
||||
|
||||
# Jupyter Notebook checkpoints
|
||||
.ipynb_checkpoints/
|
||||
|
||||
# macOS
|
||||
.DS_Store
|
||||
|
||||
# Windows
|
||||
Thumbs.db
|
||||
|
||||
# Obsidian per-user vault state
|
||||
.obsidian/workspace*.json
|
||||
.obsidian/cache
|
||||
|
||||
# Python
|
||||
*.pyc
|
||||
*.pyo
|
||||
*.pyd
|
||||
|
||||
*fiona.log
|
||||
*wiki_graph.html
|
||||
|
||||
# Generated review artifacts (git diff HTML, scratch exports)
|
||||
tmp/
|
||||
@@ -1,72 +0,0 @@
|
||||
---
|
||||
name: llm-wiki-agent-rules
|
||||
title: LLM Wiki Agent Rules
|
||||
version: 1.4.0
|
||||
created: 2025-09-23
|
||||
updated: 2026-10-14
|
||||
type: summary
|
||||
---
|
||||
|
||||
# LLM Wiki Agent Rules
|
||||
|
||||
This directory is an llm-wiki: a persistent knowledge base of
|
||||
interlinked markdown. The user brings sources and questions. You
|
||||
compile each source once into wiki pages, keep the cross-references
|
||||
current, and answer from those pages. The pattern is
|
||||
[llm-wiki-mod](./utils/skills/fiona/llm-wiki-mod.md). The wiki's
|
||||
domain, tags, and directory tree are in [schema](./wiki/schema.md).
|
||||
|
||||
This file states only what must be true before any skill is loaded.
|
||||
Commands, lint codes, and the edit workflow live in the Fiona skill.
|
||||
|
||||
- [1. Load the Fiona skill](#1-load-the-fiona-skill)
|
||||
- [2. What an llm-wiki is](#2-what-an-llm-wiki-is)
|
||||
- [3. Boundaries](#3-boundaries)
|
||||
- [4. History](#4-history)
|
||||
|
||||
## 1. Load the Fiona skill
|
||||
|
||||
If the Fiona skill is not already part of the system prompt, read
|
||||
[Fiona skill](./utils/skills/fiona/SKILL.md) and follow it for the
|
||||
rest of the session. It loads llm-wiki-mod.
|
||||
|
||||
When that skill is already loaded, do not read it again. This file
|
||||
does not repeat it.
|
||||
|
||||
Then read [MEMORY](./MEMORY.md): what this wiki's owner wants, kept
|
||||
across sessions.
|
||||
|
||||
Update `MEMORY.md` whenever an important memory comes up: a lasting
|
||||
preference, a correction, a repeated request, or a fact a run
|
||||
verified. Also update it whenever the user explicitly asks you to
|
||||
remember something. Section 1 of that file says how.
|
||||
|
||||
## 2. What an llm-wiki is
|
||||
|
||||
A directory is an llm-wiki when it contains `fiona-settings.json`.
|
||||
|
||||
Page types, tags, and the directory tree are defined in
|
||||
[schema](./wiki/schema.md). Do not invent a second map of the tree.
|
||||
|
||||
Install the tool with `python -m pip install fiona-assistant` when
|
||||
the `fiona-assistant` command is missing. How to use it is the
|
||||
Fiona skill, not this file.
|
||||
|
||||
## 3. Boundaries
|
||||
|
||||
These hold even before the skill is loaded:
|
||||
|
||||
- `personal/` is read-only. Never create, edit, or delete there.
|
||||
- `raw/` is the archive. Add a new source there. Never edit, move,
|
||||
or delete a file already there.
|
||||
- Synthesized pages belong under `wiki/`.
|
||||
- `ingest/` is only the drop zone for files waiting to be parsed.
|
||||
|
||||
## 4. History
|
||||
|
||||
Git is the wiki's history. After each ingest cycle and each
|
||||
housekeeping pass (lint fixes, index or TOC updates, page moves),
|
||||
commit with `fiona-assistant git save --description TEXT`.
|
||||
|
||||
`git clean`, `git undo`, and `git restore` discard work. Run them
|
||||
only when the user asks.
|
||||
@@ -1,179 +0,0 @@
|
||||
---
|
||||
name: memory
|
||||
version: 1.0.1
|
||||
title: Agent Memory
|
||||
created: DATE_PLACEHOLDER
|
||||
updated: DATE_PLACEHOLDER
|
||||
type: summary
|
||||
---
|
||||
|
||||
# Agent Memory
|
||||
|
||||
<!-- toc-start: headings 2..6 -->
|
||||
|
||||
- [1. How to use this file](#1-how-to-use-this-file)
|
||||
- [1.1 When to read](#11-when-to-read)
|
||||
- [1.2 When to write](#12-when-to-write)
|
||||
- [1.3 How to write](#13-how-to-write)
|
||||
- [1.4 How the memory evolves](#14-how-the-memory-evolves)
|
||||
- [2. Pages and writing](#2-pages-and-writing)
|
||||
- [2.1 Style and wording](#21-style-and-wording)
|
||||
- [2.2 Placement and links](#22-placement-and-links)
|
||||
- [3. Sources and ingest](#3-sources-and-ingest)
|
||||
- [3.1 Hosts and access](#31-hosts-and-access)
|
||||
- [3.2 Conversion and refinement](#32-conversion-and-refinement)
|
||||
- [4. Tools and environment](#4-tools-and-environment)
|
||||
- [4.1 Commands and flags](#41-commands-and-flags)
|
||||
- [4.2 Machine and shell](#42-machine-and-shell)
|
||||
- [5. History and housekeeping](#5-history-and-housekeeping)
|
||||
- [5.1 Saves and maintenance](#51-saves-and-maintenance)
|
||||
|
||||
<!-- toc-end -->
|
||||
|
||||
What this wiki's owner wants from you, kept across sessions:
|
||||
standing preferences, corrections, repeated request patterns, and
|
||||
facts a run verified about operating this wiki. Knowledge from
|
||||
sources does not go here; it goes to pages under `wiki/`.
|
||||
|
||||
## 1. How to use this file
|
||||
|
||||
### 1.1 When to read
|
||||
|
||||
- **1.1.1** Read this file once the
|
||||
[Fiona skill](./utils/skills/fiona/SKILL.md) is loaded, before the
|
||||
first wiki work of the session. When it is already in context, do
|
||||
not read it again.
|
||||
- **1.1.2** Apply every entry that matches the request.
|
||||
- **1.1.2.1** An entry overrides a default of the Fiona skill or
|
||||
the [schema](./wiki/schema.md), for this wiki only.
|
||||
- **1.1.2.2** An entry never overrides the boundaries in section 3
|
||||
of [AGENTS](./AGENTS.md); when the two conflict, follow the
|
||||
boundary and ask the user.
|
||||
- **1.1.2.3** An entry never changes a tool contract: a command, a
|
||||
flag, or a required frontmatter field.
|
||||
- **1.1.3** Every heading and list item is numbered as
|
||||
[sectionize-markdown](./utils/skills/sectionize-markdown/SKILL.md)
|
||||
describes. Cite an item as `MEMORY.md item 2.1.3`; a new item takes
|
||||
the next free number in its subsection.
|
||||
|
||||
### 1.2 When to write
|
||||
|
||||
- **1.2.1** Write an entry when one of these happens:
|
||||
- **1.2.1.1** The user states a requirement or preference that
|
||||
outlasts the current request ("always", "never", "from now on",
|
||||
"in this wiki", "we use").
|
||||
- **1.2.1.2** The user corrects your output or approach: a page, a
|
||||
summary, a placement, a link, or a command you ran.
|
||||
- **1.2.1.3** The same kind of request comes for the second time;
|
||||
record the pattern and what the user wanted.
|
||||
- **1.2.1.4** A run verifies a reusable fact about operating this
|
||||
wiki that the Fiona skill and the schema do not hold, such as a
|
||||
host that needs a login, a converter that loses a site's tables,
|
||||
or a flag combination that works.
|
||||
- **1.2.1.5** The user explicitly asks you to remember something
|
||||
("remember", "note this", "save to memory").
|
||||
- **1.2.2** Do not write:
|
||||
- **1.2.2.1** One-off values that only apply to the current
|
||||
request.
|
||||
- **1.2.2.2** Passwords, tokens, cookies, personal data, or
|
||||
anything the user asks to keep out.
|
||||
- **1.2.2.3** Knowledge from a source. That is a page under
|
||||
`wiki/`, written by an ingest.
|
||||
- **1.2.2.4** Facts that the Fiona skill, the schema, or a wiki
|
||||
page already states; cite them instead.
|
||||
- **1.2.2.5** Guesses; only what the user said or a run showed.
|
||||
|
||||
### 1.3 How to write
|
||||
|
||||
- **1.3.1** Put the entry under the `###` subsection that fits.
|
||||
- **1.3.1.1** When none fits, add a `###` subsection with the next
|
||||
free number under the matching `##`, or a new `##` category with
|
||||
the next free number.
|
||||
- **1.3.1.2** Never put an entry directly under a `##`; its id
|
||||
would collide with a later subsection.
|
||||
- **1.3.2** Use the knowledge-entry shape of sectionize-markdown
|
||||
section 4: a header line with the source tag and a short title,
|
||||
one child per fact, and a last child with the ref and the date.
|
||||
- **1.3.3** Source tags: `user` (the user said it), `run` (a command
|
||||
or tool run showed it), `source` (a file under `raw/` states it),
|
||||
joined with `+` when several agree.
|
||||
- **1.3.4** The ref says where to check the fact: the user's request
|
||||
and its date, the command and its output line, or a page or raw
|
||||
path.
|
||||
- **1.3.5** Write the entry in the same turn, then tell the user
|
||||
which item was added or changed, for example "Saved to MEMORY.md
|
||||
item 2.1.1".
|
||||
- **1.3.6** Bump `version` (patch for a changed entry, minor for a
|
||||
new entry or subsection) and set `updated` in the frontmatter on
|
||||
every change.
|
||||
- **1.3.7** Append `- YYYY-MM-DD - memory: <item id> <title>` to
|
||||
`wiki/log.md`, and bump that file too.
|
||||
- **1.3.8** The change lands in the save of the current ingest cycle
|
||||
or housekeeping pass (Fiona skill section 2.3).
|
||||
- **1.3.8.1** With no cycle running, save it on its own with
|
||||
`fiona-assistant git save --description "memory: <item id> <title>"`.
|
||||
|
||||
An entry looks like this:
|
||||
|
||||
```text
|
||||
- **2.1.1** [user] "Source summaries open with the decision"
|
||||
- **2.1.1.1** A source page starts with the decision the source records, then the reasons.
|
||||
- **2.1.1.2** Applies to meeting notes and design reviews.
|
||||
- **2.1.1.3** Ref: user request YYYY-MM-DD; added YYYY-MM-DD.
|
||||
```
|
||||
|
||||
### 1.4 How the memory evolves
|
||||
|
||||
- **1.4.1** Before adding an entry, look for a near-duplicate in the
|
||||
same category; extend it instead and keep its id and its earliest
|
||||
date.
|
||||
- **1.4.2** A finding that contradicts an entry becomes its next
|
||||
child, titled `OVERRIDE: ...`, with the new facts and its own ref.
|
||||
- **1.4.3** An entry that no longer applies is marked
|
||||
**Superseded YYYY-MM-DD** with the reason; it is not deleted.
|
||||
- **1.4.4** An entry that held for several uses graduates; mark it
|
||||
**Promoted YYYY-MM-DD** with the target.
|
||||
- **1.4.4.1** A convention for every page moves to the
|
||||
[schema](./wiki/schema.md).
|
||||
- **1.4.4.2** A reusable procedure moves to a skill under
|
||||
[utils/skills_local](./utils/skills_local/README.md).
|
||||
- **1.4.5** Never renumber ids; pages, log lines, and chats may cite
|
||||
them.
|
||||
- **1.4.6** This file belongs to this wiki and evolves with it.
|
||||
- **1.4.6.1** `fiona-assistant wiki update` adds it only when it is
|
||||
missing. It never compares or replaces it, and never writes a
|
||||
`MEMORY-update.md`.
|
||||
- **1.4.6.2** A newer template version of these rules does not
|
||||
reach an existing wiki; the owner decides what to adopt.
|
||||
- **1.4.7** Git is the memory's history: `git log -p MEMORY.md` shows
|
||||
when each entry changed and why.
|
||||
|
||||
## 2. Pages and writing
|
||||
|
||||
How pages read, where they go, and how they link.
|
||||
|
||||
### 2.1 Style and wording
|
||||
|
||||
### 2.2 Placement and links
|
||||
|
||||
## 3. Sources and ingest
|
||||
|
||||
Where sources come from and how they become pages.
|
||||
|
||||
### 3.1 Hosts and access
|
||||
|
||||
### 3.2 Conversion and refinement
|
||||
|
||||
## 4. Tools and environment
|
||||
|
||||
How `fiona-assistant` and the machine behave for this wiki.
|
||||
|
||||
### 4.1 Commands and flags
|
||||
|
||||
### 4.2 Machine and shell
|
||||
|
||||
## 5. History and housekeeping
|
||||
|
||||
Saves, commits, merges, and maintenance passes.
|
||||
|
||||
### 5.1 Saves and maintenance
|
||||
@@ -1,59 +0,0 @@
|
||||
{
|
||||
"version": "1.1.0",
|
||||
"background": "#f4efe6",
|
||||
"graph_background": "#fffdf8",
|
||||
"font_family": "Arial, sans-serif",
|
||||
"text_color": "#2c2416",
|
||||
"node_colors": {
|
||||
"low": "#8f8272",
|
||||
"mid": "#e0943a",
|
||||
"high": "#c23b4a"
|
||||
},
|
||||
"node_stroke": "none",
|
||||
"node_radius": {
|
||||
"min": 5,
|
||||
"max": 40
|
||||
},
|
||||
"edge_colors": {
|
||||
"low": "#a89884",
|
||||
"mid": "#b86a1a",
|
||||
"high": "#8e2436"
|
||||
},
|
||||
"edge_width": 1,
|
||||
"highlight": {
|
||||
"node_color": "#2aa7a1",
|
||||
"edge_color": "#0d6e74",
|
||||
"size_multiplier": 1.3
|
||||
},
|
||||
"clicked": {
|
||||
"node_color": "#0e6b66"
|
||||
},
|
||||
"orphan": {
|
||||
"enabled": false,
|
||||
"node_color": "#7b5ea7"
|
||||
},
|
||||
"buttons": {
|
||||
"background": "#0f6e78",
|
||||
"text": "#fffdf8",
|
||||
"hover": "#0b555c",
|
||||
"disabled": "#b7aea3"
|
||||
},
|
||||
"card": {
|
||||
"background": "#fffdf8",
|
||||
"border": "#e0d3c2"
|
||||
},
|
||||
"shapes": {
|
||||
"page": "circle",
|
||||
"external": "cloud"
|
||||
},
|
||||
"subtitle_color": "#2c2416",
|
||||
"subtitle": {
|
||||
"min_radius": 0.8,
|
||||
"padding": 2,
|
||||
"size": 6,
|
||||
"on_top": true,
|
||||
"outline": false,
|
||||
"outline_color": "#fffdf8",
|
||||
"outline_width": 3
|
||||
}
|
||||
}
|
||||
@@ -1,59 +0,0 @@
|
||||
{
|
||||
"version": "1.1.0",
|
||||
"background": "#1e2836",
|
||||
"graph_background": "#283448",
|
||||
"font_family": "Segoe UI, sans-serif",
|
||||
"text_color": "#e7eef8",
|
||||
"node_colors": {
|
||||
"low": "#7d8da3",
|
||||
"mid": "#fff4c4",
|
||||
"high": "#f7d67c"
|
||||
},
|
||||
"node_stroke": "none",
|
||||
"node_radius": {
|
||||
"min": 4,
|
||||
"max": 24
|
||||
},
|
||||
"edge_colors": {
|
||||
"low": "#51627a",
|
||||
"mid": "#8fa3bd",
|
||||
"high": "#d5e2f2"
|
||||
},
|
||||
"edge_width": 1,
|
||||
"highlight": {
|
||||
"node_color": "#f3d48a",
|
||||
"edge_color": "#9ec5ea",
|
||||
"size_multiplier": 1.45
|
||||
},
|
||||
"clicked": {
|
||||
"node_color": "#ffe3a3"
|
||||
},
|
||||
"orphan": {
|
||||
"enabled": false,
|
||||
"node_color": "#ef8a7a"
|
||||
},
|
||||
"buttons": {
|
||||
"background": "#3e5674",
|
||||
"text": "#f4f8fc",
|
||||
"hover": "#527094",
|
||||
"disabled": "#2a3648"
|
||||
},
|
||||
"card": {
|
||||
"background": "#31445c",
|
||||
"border": "#5c7390"
|
||||
},
|
||||
"shapes": {
|
||||
"page": "circle",
|
||||
"external": "cloud"
|
||||
},
|
||||
"subtitle_color": "#e7eef8",
|
||||
"subtitle": {
|
||||
"min_radius": 0.8,
|
||||
"padding": 2,
|
||||
"size": 6,
|
||||
"on_top": true,
|
||||
"outline": false,
|
||||
"outline_color": "#283448",
|
||||
"outline_width": 3
|
||||
}
|
||||
}
|
||||
@@ -1,113 +0,0 @@
|
||||
{
|
||||
"version": "1.1.0",
|
||||
"background": "#e6e6e6",
|
||||
"graph_background": "#f4f4f4",
|
||||
"font_family": "Arial, sans-serif",
|
||||
"text_color": "#141414",
|
||||
"node_colors": {
|
||||
"low": "#5a5a5a",
|
||||
"mid": "#2e2e2e",
|
||||
"high": "#0d0d0d"
|
||||
},
|
||||
"node_stroke": "#141414",
|
||||
"node_radius": {
|
||||
"min": 5,
|
||||
"max": 36
|
||||
},
|
||||
"edge_colors": {
|
||||
"low": "#8f8f8f",
|
||||
"mid": "#3d3d3d",
|
||||
"high": "#141414"
|
||||
},
|
||||
"edge_width": 1,
|
||||
"highlight": {
|
||||
"node_color": "#000000",
|
||||
"edge_color": "#1a1a1a",
|
||||
"size_multiplier": 1.4
|
||||
},
|
||||
"clicked": {
|
||||
"node_color": "#000000"
|
||||
},
|
||||
"orphan": {
|
||||
"enabled": false,
|
||||
"node_color": "#bdbdbd"
|
||||
},
|
||||
"buttons": {
|
||||
"background": "#2a2a2a",
|
||||
"text": "#ffffff",
|
||||
"hover": "#111111",
|
||||
"disabled": "#b0b0b0"
|
||||
},
|
||||
"card": {
|
||||
"background": "#ffffff",
|
||||
"border": "#8a8a8a"
|
||||
},
|
||||
"shapes": {
|
||||
"page": "circle",
|
||||
"external": "cloud"
|
||||
},
|
||||
"subtitle_color": "#141414",
|
||||
"subtitle": {
|
||||
"min_radius": 0.8,
|
||||
"padding": 2,
|
||||
"size": 6,
|
||||
"on_top": true,
|
||||
"outline": false,
|
||||
"outline_color": "#f4f4f4",
|
||||
"outline_width": 3
|
||||
},
|
||||
"dark": {
|
||||
"background": "#2c2c2c",
|
||||
"graph_background": "#3a3a3a",
|
||||
"font_family": "Arial, sans-serif",
|
||||
"text_color": "#f5f5f5",
|
||||
"node_colors": {
|
||||
"low": "#8d8d8d",
|
||||
"mid": "#fff4c4",
|
||||
"high": "#f5b44a"
|
||||
},
|
||||
"node_stroke": "none",
|
||||
"node_radius": {
|
||||
"min": 5,
|
||||
"max": 36
|
||||
},
|
||||
"edge_colors": {
|
||||
"low": "#6e6e6e",
|
||||
"mid": "#c4c4c4",
|
||||
"high": "#ffffff"
|
||||
},
|
||||
"edge_width": 1,
|
||||
"highlight": {
|
||||
"node_color": "#ffffff",
|
||||
"edge_color": "#ffffff",
|
||||
"size_multiplier": 1.4
|
||||
},
|
||||
"clicked": {
|
||||
"node_color": "#ffffff"
|
||||
},
|
||||
"buttons": {
|
||||
"background": "#e8e8e8",
|
||||
"text": "#1a1a1a",
|
||||
"hover": "#ffffff",
|
||||
"disabled": "#5a5a5a"
|
||||
},
|
||||
"card": {
|
||||
"background": "#454545",
|
||||
"border": "#9a9a9a"
|
||||
},
|
||||
"shapes": {
|
||||
"page": "circle",
|
||||
"external": "cloud"
|
||||
},
|
||||
"subtitle_color": "#f5f5f5",
|
||||
"subtitle": {
|
||||
"min_radius": 0.5,
|
||||
"padding": 2,
|
||||
"size": 8,
|
||||
"on_top": true,
|
||||
"outline": true,
|
||||
"outline_color": "#2c2c2c",
|
||||
"outline_width": 3
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,59 +0,0 @@
|
||||
{
|
||||
"version": "1.1.0",
|
||||
"background": "#070b14",
|
||||
"graph_background": "#0b1020",
|
||||
"font_family": "Segoe UI, sans-serif",
|
||||
"text_color": "#d6def5",
|
||||
"node_colors": {
|
||||
"low": "#8ea2c8",
|
||||
"mid": "#ffe9a0",
|
||||
"high": "#ffffff"
|
||||
},
|
||||
"node_stroke": "none",
|
||||
"node_radius": {
|
||||
"min": 3,
|
||||
"max": 20
|
||||
},
|
||||
"edge_colors": {
|
||||
"low": "#243056",
|
||||
"mid": "#7f8cbf",
|
||||
"high": "#e7ecff"
|
||||
},
|
||||
"edge_width": 1,
|
||||
"highlight": {
|
||||
"node_color": "#fff6c2",
|
||||
"edge_color": "#9ecbff",
|
||||
"size_multiplier": 1.6
|
||||
},
|
||||
"clicked": {
|
||||
"node_color": "#ffd56a"
|
||||
},
|
||||
"orphan": {
|
||||
"enabled": false,
|
||||
"node_color": "#ff7f6e"
|
||||
},
|
||||
"buttons": {
|
||||
"background": "#1e2a4a",
|
||||
"text": "#e8eeff",
|
||||
"hover": "#314572",
|
||||
"disabled": "#3a4258"
|
||||
},
|
||||
"card": {
|
||||
"background": "#12182b",
|
||||
"border": "#2a3558"
|
||||
},
|
||||
"shapes": {
|
||||
"page": "circle",
|
||||
"external": "star"
|
||||
},
|
||||
"subtitle_color": "#d5def8",
|
||||
"subtitle": {
|
||||
"min_radius": 0.8,
|
||||
"padding": 2,
|
||||
"size": 6,
|
||||
"on_top": true,
|
||||
"outline": false,
|
||||
"outline_color": "#0b1020",
|
||||
"outline_width": 3
|
||||
}
|
||||
}
|
||||
@@ -1,59 +0,0 @@
|
||||
{
|
||||
"version": "1.1.0",
|
||||
"background": "#f7fafc",
|
||||
"graph_background": "#fcfdfe",
|
||||
"font_family": "Segoe UI, Calibri, Helvetica Neue, Arial, sans-serif",
|
||||
"text_color": "#4a5d70",
|
||||
"node_colors": {
|
||||
"low": "#e4eef4",
|
||||
"mid": "#a9c6d8",
|
||||
"high": "#7eabc4"
|
||||
},
|
||||
"node_stroke": "#9bb8cb",
|
||||
"node_radius": {
|
||||
"min": 4,
|
||||
"max": 26
|
||||
},
|
||||
"edge_colors": {
|
||||
"low": "#e7f0f5",
|
||||
"mid": "#b7d0de",
|
||||
"high": "#8eb4c9"
|
||||
},
|
||||
"edge_width": 1,
|
||||
"highlight": {
|
||||
"node_color": "#6aa3c0",
|
||||
"edge_color": "#7eabc4",
|
||||
"size_multiplier": 1.15
|
||||
},
|
||||
"clicked": {
|
||||
"node_color": "#5b93b0"
|
||||
},
|
||||
"orphan": {
|
||||
"enabled": false,
|
||||
"node_color": "#d9a5b3"
|
||||
},
|
||||
"buttons": {
|
||||
"background": "#8eb4c9",
|
||||
"text": "#ffffff",
|
||||
"hover": "#7aa6bd",
|
||||
"disabled": "#dce7ee"
|
||||
},
|
||||
"card": {
|
||||
"background": "#ffffff",
|
||||
"border": "#e6eef3"
|
||||
},
|
||||
"shapes": {
|
||||
"page": "circle",
|
||||
"external": "cloud"
|
||||
},
|
||||
"subtitle_color": "#4a5d70",
|
||||
"subtitle": {
|
||||
"min_radius": 0.8,
|
||||
"padding": 2,
|
||||
"size": 6,
|
||||
"on_top": true,
|
||||
"outline": false,
|
||||
"outline_color": "#fcfdfe",
|
||||
"outline_width": 3
|
||||
}
|
||||
}
|
||||
@@ -1,19 +0,0 @@
|
||||
---
|
||||
name: map
|
||||
version: 1.1.0
|
||||
title: Script Map
|
||||
type: summary
|
||||
created: 2025-09-23
|
||||
updated: 2026-09-30
|
||||
---
|
||||
|
||||
# Script Map
|
||||
|
||||
Catalog of the reusable scripts in this folder. Read it before
|
||||
writing a new script, and add a row when you add one. The rules for
|
||||
writing a script are in the
|
||||
[Fiona skill](../skills/fiona/SKILL.md#write-reusable-scripts).
|
||||
|
||||
| Script | Purpose | Usage | Added |
|
||||
| --- | --- | --- | --- |
|
||||
| `example.py` | Placeholder that prints a greeting. | `python utils/scripts/example.py` | 2025-09-23 |
|
||||
@@ -1,18 +0,0 @@
|
||||
---
|
||||
name: skills
|
||||
version: 1.1.1
|
||||
title: Skills Directory
|
||||
type: summary
|
||||
created: 2025-09-23
|
||||
updated: 2026-10-21
|
||||
---
|
||||
|
||||
# Skills Directory
|
||||
|
||||
Skill definitions and configurations.
|
||||
|
||||
- [fiona](./fiona/SKILL.md): operate the llm-wiki with `fiona-assistant`.
|
||||
- [memory](./memory/SKILL.md): give a skill its own `MEMORIES.md`.
|
||||
- [python-tdd](./python-tdd/SKILL.md): test-driven Python work.
|
||||
- [sectionize-markdown](./sectionize-markdown/SKILL.md):
|
||||
number headings and list items so each rule can be cited.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,144 +0,0 @@
|
||||
---
|
||||
name: llm-wiki-mod
|
||||
title: LLM Wiki Pattern
|
||||
version: 1.4.1
|
||||
updated: 2026-10-06
|
||||
type: concept
|
||||
tags: [wiki, schema]
|
||||
---
|
||||
|
||||
# LLM Wiki
|
||||
|
||||
A persistent knowledge base of interlinked markdown. The pattern
|
||||
comes from
|
||||
[Andrej Karpathy's LLM Wiki](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f).
|
||||
Page rules are in [schema](../../../wiki/schema.md). Commands and
|
||||
the edit loop are in the [Fiona skill](./SKILL.md).
|
||||
|
||||
- [1. Core idea](#1-core-idea)
|
||||
- [2. Layers](#2-layers)
|
||||
- [3. Operations](#3-operations)
|
||||
- [4. Index and log](#4-index-and-log)
|
||||
- [5. Page rules](#5-page-rules)
|
||||
- [6. Why this works](#6-why-this-works)
|
||||
- [7. Scope](#7-scope)
|
||||
|
||||
## 1. Core idea
|
||||
|
||||
Sources are read once and compiled into wiki pages. Later questions
|
||||
use those pages. Cross-references, contradictions, and synthesis
|
||||
stay in the files. Each new source and each filed answer adds to
|
||||
that store.
|
||||
|
||||
The human curates sources and asks the questions. The agent writes
|
||||
the pages, updates cross-references, and keeps the bookkeeping.
|
||||
|
||||
## 2. Layers
|
||||
|
||||
Three layers. The directory tree in
|
||||
[schema](../../../wiki/schema.md) is the map.
|
||||
|
||||
### 2.1. Raw sources
|
||||
|
||||
`raw/` is the archive. The agent adds new sources there and never
|
||||
edits, moves, or deletes one already archived.
|
||||
`ingest/` is the drop zone for files that still need parsing.
|
||||
`personal/` holds user notes and is read-only, including the
|
||||
directory itself.
|
||||
|
||||
### 2.2. The wiki
|
||||
|
||||
`wiki/` is agent-owned markdown. Synthesized pages go in
|
||||
`concepts/`, `entities/`, `comparisons/`, and `queries/`. Source
|
||||
summaries that back those pages go in `sources/`. The agent creates
|
||||
pages, revises them when a new source arrives, and keeps links
|
||||
consistent.
|
||||
|
||||
### 2.3. The schema
|
||||
|
||||
[schema](../../../wiki/schema.md) states file names, frontmatter,
|
||||
tags, when to create a page, and the directory tree.
|
||||
[AGENTS.md](../../../AGENTS.md) is the entry point. It loads the
|
||||
[Fiona skill](./SKILL.md) for commands and the edit loop. When a
|
||||
convention changes, change the schema. Leave the tree in that one
|
||||
file.
|
||||
|
||||
## 3. Operations
|
||||
|
||||
### 3.1. Ingest
|
||||
|
||||
Put a new source in `ingest/`. The agent reads it, writes the
|
||||
summary under `wiki/`, updates [index](../../../wiki/index.md),
|
||||
updates the entity and concept pages the source touches, and
|
||||
appends one dated line to [log](../../../wiki/log.md). One source
|
||||
can touch many pages. Take sources one at a time when the summary
|
||||
should be checked before the next file.
|
||||
|
||||
After the archive step, the source lives under `raw/` and is not
|
||||
edited again.
|
||||
|
||||
An ingest has two phases. A tool fetches the source, archives it
|
||||
under `raw/`, and converts it into a page marked `status: raw`.
|
||||
The agent then refines that page, updates the pages it touches,
|
||||
and marks it `status: refined`. The steps are section 18 of the
|
||||
[Fiona skill](./SKILL.md).
|
||||
|
||||
### 3.2. Query
|
||||
|
||||
Ask against the wiki. The agent reads the index, opens the pages
|
||||
that apply, and answers with citations. File a comparison, an
|
||||
analysis, or a connection that should outlive the chat under
|
||||
`wiki/queries/` or `wiki/comparisons/`.
|
||||
|
||||
### 3.3. Lint
|
||||
|
||||
Check the wiki for contradictions, claims a newer source has
|
||||
replaced, pages with no inbound links, concepts that need their
|
||||
own page, missing cross-references, and gaps a source could fill.
|
||||
The health command is `fiona-assistant wiki sanity`. Style,
|
||||
frontmatter, and link checks are in the [Fiona skill](./SKILL.md).
|
||||
|
||||
## 4. Index and log
|
||||
|
||||
[index](../../../wiki/index.md) is the catalog. Each page has a
|
||||
link and a one-line summary, grouped by section. Update it when
|
||||
a page is added. Read it first when answering a query.
|
||||
|
||||
[log](../../../wiki/log.md) is append-only. Each entry is one
|
||||
dated bullet:
|
||||
|
||||
```text
|
||||
- YYYY-MM-DD - what happened
|
||||
```
|
||||
|
||||
Bump `version` and `updated` on `log.md` when appending. The log
|
||||
is the timeline of ingests, queries, and lint passes.
|
||||
|
||||
Git holds the before and after of each entry. Every ingest cycle
|
||||
and every housekeeping pass ends in one commit, so one source or
|
||||
one cleanup can be reviewed or undone on its own.
|
||||
|
||||
## 5. Page rules
|
||||
|
||||
File names, frontmatter, tags, links, and when to create or split
|
||||
a page are in [schema](../../../wiki/schema.md). How to fill the
|
||||
frontmatter and the quality signals is section 15 of the
|
||||
[Fiona skill](./SKILL.md). This file does not repeat them.
|
||||
|
||||
## 6. Why this works
|
||||
|
||||
Cross-references, summaries that still match the sources, and notes
|
||||
where a new source disagrees with an older claim are the work that
|
||||
usually gets dropped. The agent does that work on each ingest, so
|
||||
the wiki stays current as it grows.
|
||||
|
||||
The human chooses sources, directs the analysis, and decides what
|
||||
the pages mean. The agent does the filing.
|
||||
|
||||
## 7. Scope
|
||||
|
||||
This file states the pattern. The directory tree stays in
|
||||
[schema](../../../wiki/schema.md). Commands stay in the
|
||||
[Fiona skill](./SKILL.md). An Obsidian graph, Marp slides,
|
||||
Dataview, or a local search engine can sit beside the wiki. This
|
||||
wiki does not require them.
|
||||
@@ -1,173 +0,0 @@
|
||||
---
|
||||
name: skill-memories
|
||||
version: 1.0.1
|
||||
title: SKILL_NAME Memories
|
||||
created: DATE_PLACEHOLDER
|
||||
updated: DATE_PLACEHOLDER
|
||||
type: summary
|
||||
---
|
||||
|
||||
# SKILL_NAME Memories
|
||||
|
||||
<!-- toc-start: headings 2..6 -->
|
||||
|
||||
- [1. How to use this file](#1-how-to-use-this-file)
|
||||
- [1.1 When to read](#11-when-to-read)
|
||||
- [1.2 When to write](#12-when-to-write)
|
||||
- [1.3 How to write](#13-how-to-write)
|
||||
- [1.4 How the memory evolves](#14-how-the-memory-evolves)
|
||||
- [2. Output and wording](#2-output-and-wording)
|
||||
- [2.1 Style and format](#21-style-and-format)
|
||||
- [2.2 Placement and naming](#22-placement-and-naming)
|
||||
- [3. Inputs and sources](#3-inputs-and-sources)
|
||||
- [3.1 Where inputs come from](#31-where-inputs-come-from)
|
||||
- [3.2 Quirks and parsing](#32-quirks-and-parsing)
|
||||
- [4. Steps and tools](#4-steps-and-tools)
|
||||
- [4.1 Commands and flags](#41-commands-and-flags)
|
||||
- [4.2 Machine and shell](#42-machine-and-shell)
|
||||
- [5. Recurring requests](#5-recurring-requests)
|
||||
- [5.1 Patterns](#51-patterns)
|
||||
|
||||
<!-- toc-end -->
|
||||
|
||||
What users want from the SKILL_NAME skill, kept across sessions:
|
||||
standing preferences, corrections, repeated request patterns, and
|
||||
facts a run verified while using this skill. Preferences for the
|
||||
whole wiki go to the wiki's `MEMORY.md`. Knowledge from sources goes
|
||||
to pages under `wiki/`.
|
||||
|
||||
## 1. How to use this file
|
||||
|
||||
### 1.1 When to read
|
||||
|
||||
- **1.1.1** Read this file once [the skill](./SKILL.md) is loaded,
|
||||
before its first step of the session. When it is already in
|
||||
context, do not read it again.
|
||||
- **1.1.2** Apply every entry that matches the request.
|
||||
- **1.1.2.1** An entry overrides a default of the skill and an
|
||||
entry of the wiki's `MEMORY.md`, for work done with this skill
|
||||
only.
|
||||
- **1.1.2.2** An entry never overrides the boundaries in the wiki's
|
||||
`AGENTS.md`; when the two conflict, follow the boundary and ask
|
||||
the user.
|
||||
- **1.1.2.3** An entry never changes a tool contract: a command, a
|
||||
flag, or a required frontmatter field.
|
||||
- **1.1.3** Every heading and list item is numbered as
|
||||
[sectionize-markdown](../../skills/sectionize-markdown/SKILL.md)
|
||||
describes. Cite an item as `SKILL_NAME MEMORIES.md item 2.1.3`; a
|
||||
new item takes the next free number in its subsection.
|
||||
|
||||
### 1.2 When to write
|
||||
|
||||
- **1.2.1** Write an entry when one of these happens while the skill
|
||||
is in use:
|
||||
- **1.2.1.1** The user states a requirement or preference for this
|
||||
skill that outlasts the current request ("always", "never", "from
|
||||
now on", "when you do X").
|
||||
- **1.2.1.2** The user corrects the skill's output or a step it
|
||||
took.
|
||||
- **1.2.1.3** The same kind of request comes for the second time;
|
||||
record the pattern and what the user wanted.
|
||||
- **1.2.1.4** A run verifies a reusable fact about the skill's
|
||||
inputs, hosts, or tools that the skill does not state.
|
||||
- **1.2.1.5** The user explicitly asks you to remember something
|
||||
for this skill.
|
||||
- **1.2.2** Do not write:
|
||||
- **1.2.2.1** One-off values that only apply to the current
|
||||
request.
|
||||
- **1.2.2.2** Passwords, tokens, cookies, personal data, or
|
||||
anything the user asks to keep out.
|
||||
- **1.2.2.3** Preferences that hold for any wiki work; those go to
|
||||
the wiki's `MEMORY.md`.
|
||||
- **1.2.2.4** Facts the skill or a wiki page already states; cite
|
||||
them instead.
|
||||
- **1.2.2.5** Guesses; only what the user said or a run showed.
|
||||
|
||||
### 1.3 How to write
|
||||
|
||||
- **1.3.1** Put the entry under the `###` subsection that fits.
|
||||
- **1.3.1.1** When none fits, add a `###` subsection with the next
|
||||
free number under the matching `##`, or a new `##` category with
|
||||
the next free number.
|
||||
- **1.3.1.2** Never put an entry directly under a `##`; its id
|
||||
would collide with a later subsection.
|
||||
- **1.3.2** Use the knowledge-entry shape of sectionize-markdown
|
||||
section 4: a header line with the source tag and a short title,
|
||||
one child per fact, and a last child with the ref and the date.
|
||||
- **1.3.3** Source tags: `user` (the user said it), `run` (a command
|
||||
or tool run showed it), `source` (a file under `raw/` states it),
|
||||
joined with `+` when several agree.
|
||||
- **1.3.4** The ref says where to check the fact: the user's request
|
||||
and its date, the command and its output line, or a page or raw
|
||||
path.
|
||||
- **1.3.5** Write the entry in the same turn, then tell the user
|
||||
which item was added or changed, for example "Saved to SKILL_NAME
|
||||
MEMORIES.md item 2.1.1".
|
||||
- **1.3.6** Bump `version` (patch for a changed entry, minor for a
|
||||
new entry or subsection) and set `updated` in the frontmatter on
|
||||
every change.
|
||||
- **1.3.7** Append `- YYYY-MM-DD - SKILL_NAME memory: <item id> <title>`
|
||||
to `wiki/log.md`, and bump that file too.
|
||||
- **1.3.8** The change lands in the save of the current ingest cycle
|
||||
or housekeeping pass.
|
||||
- **1.3.8.1** With no cycle running, save it on its own with
|
||||
`fiona-assistant git save --description "SKILL_NAME memory: <item id> <title>"`.
|
||||
|
||||
An entry looks like this:
|
||||
|
||||
```text
|
||||
- **2.1.1** [user] "Reports open with the verdict"
|
||||
- **2.1.1.1** The report starts with pass or fail, then the evidence.
|
||||
- **2.1.1.2** Applies to every run of this skill.
|
||||
- **2.1.1.3** Ref: user request YYYY-MM-DD; added YYYY-MM-DD.
|
||||
```
|
||||
|
||||
### 1.4 How the memory evolves
|
||||
|
||||
- **1.4.1** Before adding an entry, look for a near-duplicate in the
|
||||
same category; extend it instead and keep its id and its earliest
|
||||
date.
|
||||
- **1.4.2** A finding that contradicts an entry becomes its next
|
||||
child, titled `OVERRIDE: ...`, with the new facts and its own ref.
|
||||
- **1.4.3** An entry that no longer applies is marked
|
||||
**Superseded YYYY-MM-DD** with the reason; it is not deleted.
|
||||
- **1.4.4** An entry that held for several uses graduates; mark it
|
||||
**Promoted YYYY-MM-DD** with the target.
|
||||
- **1.4.4.1** A rule for every run of the skill moves into
|
||||
[the skill](./SKILL.md).
|
||||
- **1.4.4.2** A preference that turns out to hold for all wiki
|
||||
work moves to the wiki's `MEMORY.md`.
|
||||
- **1.4.5** Never renumber ids; pages, log lines, and chats may cite
|
||||
them.
|
||||
- **1.4.6** Git is the memory's history: `git log -p MEMORIES.md` in
|
||||
the skill's folder shows when each entry changed and why.
|
||||
|
||||
## 2. Output and wording
|
||||
|
||||
What the skill produces and how it reads.
|
||||
|
||||
### 2.1 Style and format
|
||||
|
||||
### 2.2 Placement and naming
|
||||
|
||||
## 3. Inputs and sources
|
||||
|
||||
What the skill reads and where it comes from.
|
||||
|
||||
### 3.1 Where inputs come from
|
||||
|
||||
### 3.2 Quirks and parsing
|
||||
|
||||
## 4. Steps and tools
|
||||
|
||||
How the skill's commands and the machine behave.
|
||||
|
||||
### 4.1 Commands and flags
|
||||
|
||||
### 4.2 Machine and shell
|
||||
|
||||
## 5. Recurring requests
|
||||
|
||||
Requests that come back, and what the user wanted each time.
|
||||
|
||||
### 5.1 Patterns
|
||||
@@ -1,109 +0,0 @@
|
||||
---
|
||||
name: memory
|
||||
category: productivity
|
||||
description: Give a skill its own memory, a MEMORIES.md beside its SKILL.md.
|
||||
author: nvaldez
|
||||
version: 1.0.1
|
||||
created: 2026-10-15
|
||||
updated: 2026-10-21
|
||||
---
|
||||
|
||||
# Skill Memory
|
||||
|
||||
<!-- toc-start: headings 2..6 -->
|
||||
|
||||
- [1. What a skill memory is](#1-what-a-skill-memory-is)
|
||||
- [2. Give a skill memory](#2-give-a-skill-memory)
|
||||
- [3. Wire it into the skill](#3-wire-it-into-the-skill)
|
||||
- [4. Use a skill memory](#4-use-a-skill-memory)
|
||||
|
||||
<!-- toc-end -->
|
||||
|
||||
A skill with memory keeps what its users want from that skill across
|
||||
sessions, in a `MEMORIES.md` file beside its `SKILL.md`. The blank
|
||||
file to copy is [MEMORIES](./MEMORIES.md). The wiki-wide memory is
|
||||
[MEMORY](../../../MEMORY.md); this skill does not replace it.
|
||||
|
||||
Every heading and list item is numbered as
|
||||
[sectionize-markdown](../sectionize-markdown/SKILL.md)
|
||||
describes. Cite a rule as `memory skill item 2.3`.
|
||||
|
||||
## 1. What a skill memory is
|
||||
|
||||
- **1.1** One file per skill: `MEMORIES.md` in the skill's folder,
|
||||
next to its `SKILL.md`.
|
||||
- **1.2** It holds facts about operating that one skill:
|
||||
- **1.2.1** Preferences for the skill's output.
|
||||
- **1.2.2** Corrections to the skill's steps or results.
|
||||
- **1.2.3** Request patterns that came back for that skill.
|
||||
- **1.2.4** Facts a run verified about the skill's inputs, hosts,
|
||||
and tools.
|
||||
- **1.3** It does not hold:
|
||||
- **1.3.1** Knowledge from sources; that is a page under `wiki/`.
|
||||
- **1.3.2** Preferences for the whole wiki; those go to
|
||||
[MEMORY](../../../MEMORY.md).
|
||||
- **1.3.3** Passwords, tokens, cookies, or personal data.
|
||||
- **1.4** Precedence, from strongest to weakest:
|
||||
- **1.4.1** The boundaries in [AGENTS](../../../AGENTS.md) and tool
|
||||
contracts (commands, flags, required frontmatter).
|
||||
- **1.4.2** The skill's `MEMORIES.md`, for work done with that skill.
|
||||
- **1.4.3** The wiki's `MEMORY.md`.
|
||||
- **1.4.4** The defaults in the skill's `SKILL.md`.
|
||||
|
||||
## 2. Give a skill memory
|
||||
|
||||
- **2.1** Give a skill memory when the user asks for a skill with
|
||||
memory, or when the same correction to one skill comes a second
|
||||
time.
|
||||
- **2.2** A new skill lives under `utils/skills_local/<skill>/`. A
|
||||
skill shipped under `utils/skills/` can also get a memory; the
|
||||
template never ships a `MEMORIES.md` for it, so an update leaves the
|
||||
file alone.
|
||||
- **2.3** Copy [MEMORIES](./MEMORIES.md) to
|
||||
`<skill-folder>/MEMORIES.md`. Never edit the template copy in this
|
||||
folder.
|
||||
- **2.4** In the copy:
|
||||
- **2.4.1** Set frontmatter `name` to `<skill>-memories`, `version`
|
||||
to `1.0.0`, and `created` and `updated` to today.
|
||||
- **2.4.2** Replace every `SKILL_NAME` with the skill's name.
|
||||
- **2.4.3** Rename the empty categories in sections 2 to 5 to fit
|
||||
the skill, or keep them. Leave section 1 as it is.
|
||||
- **2.5** Add the memory section of section 3 to the skill's
|
||||
`SKILL.md` and bump that file's `version`.
|
||||
- **2.6** Append `- YYYY-MM-DD - memory: <skill> gets MEMORIES.md` to
|
||||
`wiki/log.md`, then save with
|
||||
`fiona-assistant git save --description "memory: <skill> gets MEMORIES.md"`.
|
||||
|
||||
## 3. Wire it into the skill
|
||||
|
||||
- **3.1** The skill's `SKILL.md` gets this section as its last
|
||||
numbered section, with `N` set to that section's number:
|
||||
|
||||
```markdown
|
||||
## N. Memory
|
||||
|
||||
- **N.1** After loading this skill, read [MEMORIES](./MEMORIES.md)
|
||||
once per session. When it is already in context, do not read it
|
||||
again.
|
||||
- **N.2** Update it whenever an important memory about this skill
|
||||
comes up: a lasting preference, a correction, a repeated request,
|
||||
or a fact a run verified. Also update it whenever the user
|
||||
explicitly asks you to remember something for this skill.
|
||||
Section 1 of that file says how.
|
||||
```
|
||||
|
||||
- **3.2** Keep the link relative (`./MEMORIES.md`) so the skill still
|
||||
finds its memory after a move.
|
||||
|
||||
## 4. Use a skill memory
|
||||
|
||||
- **4.1** Section 1 of each skill's `MEMORIES.md` is the rule for
|
||||
reading and writing it. This file does not repeat it.
|
||||
- **4.2** Pick the narrowest memory that covers the fact:
|
||||
- **4.2.1** It only matters while one skill runs: that skill's
|
||||
`MEMORIES.md`.
|
||||
- **4.2.2** It matters for any wiki work: `MEMORY.md`.
|
||||
- **4.2.3** When unsure, ask the user which one.
|
||||
- **4.3** When a request uses a skill that has no memory and a
|
||||
memory comes up, write it to `MEMORY.md` and suggest giving the
|
||||
skill one (section 2).
|
||||
@@ -1,159 +0,0 @@
|
||||
---
|
||||
name: sectionize-markdown
|
||||
version: 1.1.3
|
||||
title: How to sectionize a markdown file
|
||||
description: Number every heading and list item of a markdown file so each
|
||||
rule, step, or fact has a stable id to cite.
|
||||
created: '2026-10-14'
|
||||
updated: '2026-10-21'
|
||||
confidence: high
|
||||
llm:
|
||||
- model: Claude Opus 5.5
|
||||
---
|
||||
|
||||
# How to sectionize a markdown file
|
||||
|
||||
<!-- toc-start: headings 2..6 -->
|
||||
|
||||
- [1. Headings](#1-headings)
|
||||
- [2. List item ids](#2-list-item-ids)
|
||||
- [3. Splitting long lines](#3-splitting-long-lines)
|
||||
- [4. Knowledge entries](#4-knowledge-entries)
|
||||
- [5. Organize by category](#5-organize-by-category)
|
||||
- [6. Doing it safely](#6-doing-it-safely)
|
||||
- [7. Example](#7-example)
|
||||
|
||||
<!-- toc-end -->
|
||||
|
||||
Numbers every heading and every list item of a markdown file so each
|
||||
rule, step, or fact has a stable id that can be cited ("item
|
||||
4.2.3"), tracked, and checked for duplicates. It suits rule files,
|
||||
skill instructions, glossaries, and knowledge caches.
|
||||
|
||||
## 1. Headings
|
||||
|
||||
- **1.1** Number every heading below the title: `## N.` for sections,
|
||||
`### N.M` for subsections, `#### N.M.K` below that.
|
||||
- **1.2** The `#` title and the table of contents are not numbered.
|
||||
- **1.3** Put a TOC of markdown links right under the title and intro.
|
||||
Regenerate it whenever headings change.
|
||||
- **1.4** Anchors are GitHub-style slugs: `## 5.6 Download the logs`
|
||||
becomes `#56-download-the-logs`.
|
||||
- **1.5** Avoid em dashes and other punctuation in headings; use a colon
|
||||
(`10.7 Outcome: a match exists`). Punctuation makes the slug
|
||||
unpredictable and breaks TOC links (MD051).
|
||||
- **1.6** A label that only introduces a list ("Files in this
|
||||
folder:") is a numbered list item, not a heading.
|
||||
|
||||
## 2. List item ids
|
||||
|
||||
- **2.1** Every list item gets a bold id right after the dash:
|
||||
`- **4.2.1** Text`. Bold is two asterisks on each side; a single
|
||||
asterisk is italic.
|
||||
- **2.2** The id is the section number plus a running counter:
|
||||
`**<section>.<n>**`. Nested items extend the parent id:
|
||||
`**<section>.<n>.<m>**`, indented two spaces.
|
||||
- **2.3** The counter runs through the whole section, across separate
|
||||
lists split by paragraphs or code blocks. Two lists under `### 12.3`
|
||||
give `**12.3.1**`…`**12.3.3**`, then `**12.3.4**`…, never a second
|
||||
`**12.3.1**`.
|
||||
- **2.4** Checklists keep the box first: `- [ ] **13.1** Verify …`.
|
||||
- **2.5** Do not number the TOC, code blocks, or table rows.
|
||||
- **2.6** State the convention in the file's first section so the next
|
||||
editor follows it ("Cite an item as `item 4.2.3`").
|
||||
- **2.7** Never renumber existing ids; other notes may cite them. A new
|
||||
item takes the next free number in its section, even if that puts it
|
||||
out of reading order. Fix the order only while the ids are still
|
||||
unreleased (uncommitted).
|
||||
|
||||
## 3. Splitting long lines
|
||||
|
||||
- **3.1** A list item holding several facts becomes a short header item
|
||||
plus one child per fact.
|
||||
- **3.2** One child is one sentence, or two closely linked sentences.
|
||||
- **3.3** Split at sentence ends (`.`, `?` or `!`, then a space and an
|
||||
uppercase letter, backtick, `*`, `"` or `[`). Do not split inside
|
||||
backticks or parentheses, or after `e.g.`, `i.e.`, `vs.`, `cf.`.
|
||||
- **3.4** Merge fragments shorter than about 45 characters back into the
|
||||
previous child; they read better together.
|
||||
- **3.5** A long single sentence stays on one line. Do not cut a
|
||||
sentence in half to meet a length target.
|
||||
- **3.6** Leave command lines, code blocks, and table rows as they are.
|
||||
|
||||
## 4. Knowledge entries
|
||||
|
||||
A file that collects findings or lessons learned uses a fixed entry
|
||||
shape so each fact is citable and the source is visible:
|
||||
|
||||
```text
|
||||
- **<section>.<n>** [<source>] "<short title>"
|
||||
- **<section>.<n>.1** <reusable fact + where it applies>
|
||||
- **<section>.<n>.2** <next fact>
|
||||
- **<section>.<n>.<k>** Ref: <path:line or query>; added <YYYY-MM-DD>.
|
||||
```
|
||||
|
||||
- **4.1** The entry line holds only the source tag and the title.
|
||||
- **4.2** `<source>` names where the fact was verified (a repository, a
|
||||
spec, …), joined with `+` when several agree.
|
||||
- **4.3** The last child is always the ref with the date it was added.
|
||||
- **4.4** A finding that contradicts an entry becomes that entry's next
|
||||
child, titled `OVERRIDE: …`, with its own children. Or mark the old
|
||||
entry **Superseded YYYY-MM-DD** with the reason.
|
||||
|
||||
## 5. Organize by category
|
||||
|
||||
- **5.1** Group entries by topic (setup, build, deployment, …), not by
|
||||
the date they were added. The date lives in the ref line.
|
||||
- **5.2** Grouping puts near-duplicates next to each other. Merge them
|
||||
into one entry and keep the earliest `added` date.
|
||||
- **5.3** A glossary gets the same treatment: one
|
||||
`###` per term under a `##` category, its definition split into
|
||||
numbered items.
|
||||
|
||||
## 6. Doing it safely
|
||||
|
||||
- **6.1** Restructure with a script, not by hand, when the file has more
|
||||
than a few dozen items. Hand edits drop or reorder text.
|
||||
- **6.2** The script asserts that the words are preserved: same word
|
||||
sequence before and after, only ids and line breaks added.
|
||||
- **6.3** Verify after every pass:
|
||||
- **6.3.1** No duplicate ids (`^\s*- (\[ \] )?\*\*([\d.]+)\*\*`).
|
||||
- **6.3.2** Every `](#anchor)` resolves to a heading slug.
|
||||
- **6.3.3** The entry count matches the original; diff titles against
|
||||
`HEAD` when it does not (an uncommitted entry may explain it).
|
||||
- **6.3.4** One `Ref:` line per entry.
|
||||
- **6.4** Watch for editors that reformat on save (escaped `\*`, `*`
|
||||
bullets padded with three spaces, lost HTML comments, fences without a language). Normalize
|
||||
back and re-verify.
|
||||
- **6.5** On Windows PowerShell, write helper scripts to `%TEMP%` and run
|
||||
them; `python -c` with backticks breaks. Avoid printing non-ASCII to
|
||||
the cp1252 console. Delete the temp scripts afterwards.
|
||||
- **6.6** Keep Unix line endings and `-` bullets, and run the markdown
|
||||
linter before committing.
|
||||
|
||||
## 7. Example
|
||||
|
||||
Before:
|
||||
|
||||
```text
|
||||
## Topic
|
||||
|
||||
- Rule about the topic. A fact about it. How to fix it. Ref: file:line.
|
||||
- Label for a group: first part of the rule, second part of the rule.
|
||||
```
|
||||
|
||||
After:
|
||||
|
||||
```text
|
||||
## 3. Topic
|
||||
|
||||
### 3.2 Subtopic
|
||||
|
||||
- **3.2.4** [source] "Rule about the topic"
|
||||
- **3.2.4.1** A fact about it.
|
||||
- **3.2.4.2** How to fix it.
|
||||
- **3.2.4.3** Ref: file:line; added YYYY-MM-DD.
|
||||
- **3.2.5** Label for a group:
|
||||
- **3.2.5.1** First part of the rule.
|
||||
- **3.2.5.2** Second part of the rule.
|
||||
```
|
||||
@@ -1,321 +0,0 @@
|
||||
---
|
||||
name: specification-writer
|
||||
description: Load when writing or updating project specifications (SPEC.md).
|
||||
Expert in creating structured and maintainable specification documents.
|
||||
type: skill
|
||||
version: 1.0.0
|
||||
created: 2026-11-03
|
||||
updated: 2026-11-03
|
||||
---
|
||||
|
||||
# Specification Writer
|
||||
|
||||
You are an expert in writing software project specifications.
|
||||
This skill provides guidance on creating structured, comprehensive, and
|
||||
maintainable SPEC.md files following established best practices.
|
||||
|
||||
## When to Load
|
||||
|
||||
Load this skill when:
|
||||
|
||||
- Creating a new SPEC.md file for a project
|
||||
- Updating or expanding an existing specification
|
||||
- Reviewing specification structure and content
|
||||
- Ensuring specifications follow consistent patterns
|
||||
|
||||
## Core Principles
|
||||
|
||||
### Sectionize-Markdown Conventions
|
||||
|
||||
Always apply sectionize-markdown conventions to every SPEC.md file.
|
||||
Reference: [sectionize-markdown skill](../sectionize-markdown/SKILL.md)
|
||||
|
||||
- Every heading MUST be numbered: `## 1.` for sections, `### 1.1`
|
||||
for subsections, etc.
|
||||
- Every list item MUST have a bold ID: `- **1.1.1** Text`
|
||||
- TOC MUST be present under the title with markdown links
|
||||
- Anchors MUST follow GitHub-style slugs
|
||||
- Never renumber existing IDs - they may be cited elsewhere
|
||||
|
||||
## SPEC.md Structure Template
|
||||
|
||||
Use this structure as the foundation for all project specifications:
|
||||
|
||||
```markdown
|
||||
---
|
||||
name: <Project Name> Specification
|
||||
version: <semver>
|
||||
title: Project Spec Sheet
|
||||
type: specification
|
||||
created: <YYYY-MM-DD>
|
||||
updated: <YYYY-MM-DD>
|
||||
---
|
||||
|
||||
# <Project Name> Specification
|
||||
|
||||
<!-- toc-start: headings 2..6 -->
|
||||
|
||||
- [1. Product Overview](#1-product-overview)
|
||||
- [1.1. Purpose](#11-purpose)
|
||||
- [1.2. Scope](#12-scope)
|
||||
- [1.3. Non-Goals](#13-non-goals)
|
||||
- [1.4. Document Conventions](#14-document-conventions)
|
||||
- [2. Architecture](#2-architecture)
|
||||
- [2.1. Component Diagram](#21-component-diagram)
|
||||
- [2.2. Data Flow](#22-data-flow)
|
||||
- [2.3. Module Structure](#23-module-structure)
|
||||
- [3. Core Contracts](#3-core-contracts)
|
||||
- [3.1. Interfaces](#31-interfaces)
|
||||
- [3.2. Data Structures](#32-data-structures)
|
||||
- [4. Package and Runtime](#4-package-and-runtime)
|
||||
- [4.1. Dependencies](#41-dependencies)
|
||||
- [4.2. Entry Points](#42-entry-points)
|
||||
- [4.3. Installation](#43-installation)
|
||||
- [5. Feature Requirements](#5-feature-requirements)
|
||||
- [6. Configuration Files](#6-configuration-files)
|
||||
- [7. Error Handling](#7-error-handling)
|
||||
- [8. Testing](#8-testing)
|
||||
- [9. Deployment](#9-deployment)
|
||||
|
||||
<!-- toc-end -->
|
||||
|
||||
This document is the build contract for <Project Name>.
|
||||
A reader with this file, and no source tree, must be able to produce
|
||||
the same observable software.
|
||||
Status values are `Implemented`, `Partially Implemented`, `Planned`,
|
||||
and `Not Implemented`.
|
||||
|
||||
---
|
||||
```
|
||||
|
||||
## Detailed Section Guidelines
|
||||
|
||||
### 1. Product Overview
|
||||
|
||||
#### 1.1. Purpose
|
||||
|
||||
- **1.1.1** Clearly state what problem the project solves
|
||||
- **1.1.2** Describe the primary value proposition
|
||||
- **1.1.3** Identify the target users or audience
|
||||
- **1.1.4** Define the project's mission statement
|
||||
|
||||
#### 1.2. Scope
|
||||
|
||||
- **1.2.1** **In Scope**: List what the project will do
|
||||
- **1.2.2** **Out of Scope**: List what the project will NOT do
|
||||
- **1.2.3** Define clear boundaries for the project
|
||||
|
||||
#### 1.3. Non-Goals
|
||||
|
||||
- **1.3.1** Explicitly state what will not be implemented
|
||||
- **1.3.2** Clarify boundaries with related projects or systems
|
||||
- **1.3.3** Document conscious exclusions and rationale
|
||||
|
||||
#### 1.4. Document Conventions
|
||||
|
||||
- **1.4.1** Define terminology used throughout the document
|
||||
- **1.4.2** Establish status labels and their meanings
|
||||
- Implemented: Feature is complete and tested
|
||||
- Partially Implemented: Feature is partially complete
|
||||
- Planned: Feature is scheduled for implementation
|
||||
- Not Implemented: Feature is specified but not yet built
|
||||
- **1.4.3** Specify numbering conventions for sections and items
|
||||
- **1.4.4** Define RFC2119 keywords (MUST, SHALL, SHOULD, MAY, etc.)
|
||||
|
||||
### 2. Architecture
|
||||
|
||||
#### 2.1. Component Diagram
|
||||
|
||||
- **2.1.1** Provide ASCII or Mermaid diagram showing system architecture
|
||||
- **2.1.2** Label all major components and their relationships
|
||||
- **2.1.3** Show data flow between components with arrows or descriptions
|
||||
- **2.1.4** Indicate external dependencies and integrations
|
||||
|
||||
#### 2.2. Data Flow
|
||||
|
||||
- **2.2.1** Describe how data moves through the system end-to-end
|
||||
- **2.2.2** Identify key data transformations at each stage
|
||||
- **2.2.3** Note where data is persisted and in what format
|
||||
- **2.2.4** Document data validation and error handling points
|
||||
|
||||
#### 2.3. Module Structure
|
||||
|
||||
- **2.3.1** Document the complete directory and file structure
|
||||
- **2.3.2** Describe each module's responsibility and purpose
|
||||
- **2.3.3** Note dependencies between modules
|
||||
- **2.3.4** Indicate which modules are public vs internal APIs
|
||||
|
||||
### 3. Core Contracts
|
||||
|
||||
#### 3.1. Interfaces
|
||||
|
||||
- **3.1.1** Define purpose and role of each interface
|
||||
- **3.1.2** List all required methods or functions with signatures
|
||||
- **3.1.3** Document expected inputs, outputs, and return types
|
||||
- **3.1.4** Specify error handling expectations and exceptions
|
||||
- **3.1.5** Note version compatibility requirements
|
||||
|
||||
#### 3.2. Data Structures
|
||||
|
||||
- **3.2.1** Provide complete type definitions or JSON schemas
|
||||
- **3.2.2** Include concrete examples of each data structure
|
||||
- **3.2.3** Document validation rules and constraints
|
||||
- **3.2.4** Note backward and forward compatibility considerations
|
||||
- **3.2.5** Specify serialization formats (JSON, YAML, etc.)
|
||||
|
||||
### 4. Package and Runtime
|
||||
|
||||
#### 4.1. Dependencies
|
||||
|
||||
- **4.1.1** List runtime dependencies with exact version requirements
|
||||
- **4.1.2** List development dependencies for building and testing
|
||||
- **4.1.3** Document optional dependencies and their purposes
|
||||
- **4.1.4** Note any version conflicts or compatibility constraints
|
||||
|
||||
#### 4.2. Entry Points
|
||||
|
||||
- **4.2.1** Define CLI entry points and their usage
|
||||
- **4.2.2** Document library entry points for programmatic use
|
||||
- **4.2.3** Specify plugin or extension entry points if applicable
|
||||
- **4.2.4** Include examples of invoking each entry point
|
||||
|
||||
#### 4.3. Installation
|
||||
|
||||
- **4.3.1** Document all supported installation methods
|
||||
- **4.3.2** Specify platform and environment requirements
|
||||
- **4.3.3** Provide verification steps to confirm successful installation
|
||||
- **4.3.4** Note any post-installation configuration needed
|
||||
|
||||
### 5. Feature Requirements
|
||||
|
||||
Organize requirements by feature area. Each feature section should include:
|
||||
|
||||
- **5.X.1** Status: One of Implemented, Partially Implemented, Planned,
|
||||
Not Implemented
|
||||
- **5.X.2** Purpose: User stories and motivations for the feature
|
||||
- **5.X.3** Functional requirements: What the feature must do
|
||||
- **5.X.4** Non-functional requirements: Performance, security, etc.
|
||||
- **5.X.5** Acceptance criteria: How to verify the feature works
|
||||
|
||||
### 6. Configuration Files
|
||||
|
||||
For each configuration file, document:
|
||||
|
||||
- **6.1.1** Location: Both relative path and typical absolute paths
|
||||
- Note platform-specific locations (Windows vs Unix)
|
||||
- **6.1.2** Purpose: What the configuration controls
|
||||
- **6.1.3** Structure: Complete format and schema
|
||||
- **6.1.4** Default values: What settings are used if not specified
|
||||
- **6.1.5** Customization: How users can modify the configuration
|
||||
|
||||
### 7. Error Handling
|
||||
|
||||
#### 7.1. Error Classification
|
||||
|
||||
- **7.1.1** Transient vs permanent errors
|
||||
- **7.1.2** Recoverable vs unrecoverable errors
|
||||
- **7.1.3** User-facing vs internal errors
|
||||
- **7.1.4** Expected vs unexpected errors
|
||||
|
||||
#### 7.2. Error Reporting
|
||||
|
||||
- **7.2.1** Logging strategy and verbosity levels
|
||||
- **7.2.2** User notification mechanisms (CLI, UI, etc.)
|
||||
- **7.2.3** Error codes and their meanings
|
||||
- **7.2.4** Error message format and content
|
||||
|
||||
#### 7.3. Recovery Strategies
|
||||
|
||||
- **7.3.1** Automatic recovery mechanisms and retries
|
||||
- **7.3.2** Manual recovery procedures and user actions
|
||||
- **7.3.3** Data consistency guarantees during failures
|
||||
- **7.3.4** Rollback and undo capabilities
|
||||
|
||||
### 8. Testing
|
||||
|
||||
#### 8.1. Test Strategy
|
||||
|
||||
- **8.1.1** Testing pyramid: Unit, integration, system, E2E
|
||||
- **8.1.2** Test coverage targets for different test types
|
||||
- **8.1.3** Test environments and their configurations
|
||||
- **8.1.4** CI/CD integration for automated testing
|
||||
|
||||
#### 8.2. Test Cases
|
||||
|
||||
- **8.2.1** Critical path tests covering core functionality
|
||||
- **8.2.2** Edge case tests for boundary conditions
|
||||
- **8.2.3** Regression tests for known issues
|
||||
- **8.2.4** Performance and load tests if applicable
|
||||
|
||||
### 9. Deployment
|
||||
|
||||
#### 9.1. Release Process
|
||||
|
||||
- **9.1.1** Versioning scheme (SemVer, CalVer, etc.)
|
||||
- **9.1.2** Release checklist and required steps
|
||||
- **9.1.3** Rollback procedures for failed deployments
|
||||
- **9.1.4** Release notes and changelog requirements
|
||||
|
||||
#### 9.2. Distribution
|
||||
|
||||
- **9.2.1** Package repositories and distribution channels
|
||||
- **9.2.2** Installation methods for end users
|
||||
- **9.2.3** Update mechanisms and version management
|
||||
- **9.2.4** Offline or air-gapped installation options
|
||||
|
||||
## Sectionize-Markdown Integration
|
||||
|
||||
**CRITICAL**: Every SPEC.md file you create or modify MUST follow
|
||||
sectionize-markdown conventions completely.
|
||||
|
||||
Before finalizing any SPEC.md:
|
||||
|
||||
1. Apply numbered headings (## 1., ### 1.1, #### 1.1.1)
|
||||
2. Add bold IDs to all list items (- **1.1.1**)
|
||||
3. Generate or update the TOC with markdown links
|
||||
4. Ensure all anchors resolve correctly
|
||||
5. Verify no duplicate IDs exist
|
||||
|
||||
Use the sectionize-markdown skill as the definitive reference for
|
||||
all sectionizing conventions and requirements.
|
||||
|
||||
## Quality Checklist
|
||||
|
||||
Before marking a SPEC as complete, verify:
|
||||
|
||||
- [ ] **Structure**: All required sections are present
|
||||
- [ ] **Numbering**: All headings and list items are numbered correctly
|
||||
- [ ] **TOC**: Table of contents is present and accurate
|
||||
- [ ] **Status**: Every feature section has a status line
|
||||
- [ ] **Examples**: Code and configuration examples are provided
|
||||
- [ ] **Cross-references**: All internal links resolve correctly
|
||||
- [ ] **Consistency**: Terminology and formatting are consistent
|
||||
- [ ] **Completeness**: All acceptance criteria are defined
|
||||
- [ ] **Sectionized**: Follows sectionize-markdown conventions completely
|
||||
|
||||
## Maintenance Guidelines
|
||||
|
||||
- **Versioning**: Increment version in frontmatter with each update
|
||||
- **Changelog**: Maintain updated and created dates in frontmatter
|
||||
- **Review**: SPEC should be reviewed before major implementation
|
||||
- **Sync**: Keep SPEC in sync with implementation (update status)
|
||||
|
||||
## Tools and Automation
|
||||
|
||||
When working with SPEC files:
|
||||
|
||||
- Use scripts to verify section numbering consistency
|
||||
- Automate TOC generation to avoid manual errors
|
||||
- Validate all internal links before committing
|
||||
- Check for duplicate IDs across the entire document
|
||||
- Run markdown linters to ensure formatting quality
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
- **Unnumbered items**: Every list item MUST have a bold ID
|
||||
- **Broken anchors**: Always verify all TOC links resolve
|
||||
- **Inconsistent status**: Use consistent status labels throughout
|
||||
- **Missing examples**: Always include examples for data formats
|
||||
- **Ambiguous requirements**: Requirements must be testable and verifiable
|
||||
- **Overly verbose**: Be concise but comprehensive
|
||||
- **Under-specified**: Include enough detail for implementation
|
||||
@@ -1,19 +0,0 @@
|
||||
---
|
||||
name: log
|
||||
title: Wiki Log
|
||||
version: 2.0.0
|
||||
type: summary
|
||||
created: DATE_PLACEHOLDER
|
||||
updated: DATE_PLACEHOLDER
|
||||
confidence: trusted
|
||||
llm:
|
||||
- model: none
|
||||
contested: false
|
||||
contradictions: []
|
||||
---
|
||||
|
||||
# Wiki Log
|
||||
|
||||
Chronological tracking of all actions performed on the wiki.
|
||||
|
||||
- DATE_PLACEHOLDER - Initialized the llm-wiki.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,327 +0,0 @@
|
||||
<!DOCTYPE html>
|
||||
<!-- Packaged layout for fiona.gittool.diff_report; placeholders filled at generation time. -->
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>{{PAGE_TITLE}}</title>
|
||||
<style>
|
||||
{{THEME_CSS}}
|
||||
* { box-sizing: border-box; }
|
||||
body {
|
||||
margin: 0;
|
||||
font-family: var(--font);
|
||||
background: var(--bg);
|
||||
color: var(--text);
|
||||
line-height: 1.45;
|
||||
transition: background-color 0.3s ease, color 0.3s ease;
|
||||
}
|
||||
header {
|
||||
background: var(--card);
|
||||
color: var(--text);
|
||||
padding: 1rem 1.25rem;
|
||||
border-bottom: 1px solid var(--border);
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
align-items: center;
|
||||
flex-wrap: wrap;
|
||||
gap: 1rem;
|
||||
}
|
||||
header .header-title {
|
||||
flex: 1;
|
||||
min-width: 200px;
|
||||
}
|
||||
header h1 { margin: 0 0 0.35rem; font-size: 1.4rem; font-weight: 600; }
|
||||
header p { margin: 0.25rem 0; opacity: 0.92; font-size: 0.9rem; }
|
||||
header code {
|
||||
background: color-mix(in srgb, var(--button-text) 15%, transparent);
|
||||
padding: 0.1rem 0.35rem;
|
||||
border-radius: 4px;
|
||||
}
|
||||
.theme-toggle {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 0.5rem;
|
||||
}
|
||||
.theme-toggle button {
|
||||
padding: 0.35rem 0.65rem;
|
||||
cursor: pointer;
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 6px;
|
||||
background: var(--button-bg);
|
||||
color: var(--button-text);
|
||||
font-size: 0.85rem;
|
||||
}
|
||||
.theme-toggle button:hover { background: var(--button-hover); }
|
||||
.layout {
|
||||
display: flex;
|
||||
align-items: stretch;
|
||||
min-height: calc(100vh - 8rem);
|
||||
}
|
||||
aside.sidebar {
|
||||
width: var(--sidebar-w);
|
||||
flex-shrink: 0;
|
||||
position: sticky;
|
||||
top: 0;
|
||||
align-self: flex-start;
|
||||
max-height: 100vh;
|
||||
overflow: auto;
|
||||
background: var(--card);
|
||||
border-right: 1px solid var(--border);
|
||||
padding: 0.75rem 0.5rem 1rem;
|
||||
}
|
||||
aside.sidebar h2 {
|
||||
margin: 0 0 0.5rem 0.65rem;
|
||||
font-size: 0.75rem;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.05em;
|
||||
color: var(--blue);
|
||||
}
|
||||
.file-tree {
|
||||
list-style: none;
|
||||
margin: 0;
|
||||
padding: 0 0 0 0.35rem;
|
||||
font-size: 0.8rem;
|
||||
}
|
||||
.file-tree ul {
|
||||
list-style: none;
|
||||
margin: 0;
|
||||
padding-left: 0.65rem;
|
||||
border-left: 1px solid var(--border);
|
||||
}
|
||||
.file-tree li { margin: 0.15rem 0; }
|
||||
.file-tree .folder-label {
|
||||
display: block;
|
||||
font-weight: 600;
|
||||
color: var(--text);
|
||||
padding: 0.1rem 0.35rem;
|
||||
}
|
||||
.file-tree a {
|
||||
display: block;
|
||||
padding: 0.15rem 0.35rem;
|
||||
border-radius: 4px;
|
||||
color: var(--accent);
|
||||
text-decoration: none;
|
||||
word-break: break-all;
|
||||
}
|
||||
.file-tree a:hover { background: var(--link-hover-bg); text-decoration: underline; }
|
||||
.file-tree a.active {
|
||||
background: var(--link-active-bg);
|
||||
font-weight: 600;
|
||||
color: var(--blue);
|
||||
}
|
||||
.file-tree .churn {
|
||||
font-size: 0.7rem;
|
||||
color: var(--muted);
|
||||
margin-left: 0.25rem;
|
||||
white-space: nowrap;
|
||||
}
|
||||
.churn .add { color: var(--churn-add); }
|
||||
.churn .del { color: var(--churn-del); }
|
||||
.file-icon {
|
||||
font-size: 0.8rem;
|
||||
margin-right: 0.35rem;
|
||||
opacity: 0.7;
|
||||
}
|
||||
.stat-fold {
|
||||
margin: 0.75rem 0.5rem 0;
|
||||
font-size: 0.75rem;
|
||||
}
|
||||
.stat-fold summary { cursor: pointer; color: var(--muted); }
|
||||
.stat-fold pre {
|
||||
margin: 0.35rem 0 0;
|
||||
font-size: 0.68rem;
|
||||
white-space: pre-wrap;
|
||||
max-height: 12rem;
|
||||
overflow: auto;
|
||||
background: var(--summary-bg);
|
||||
padding: 0.5rem;
|
||||
border-radius: 4px;
|
||||
border: 1px solid var(--border);
|
||||
}
|
||||
.content {
|
||||
flex: 1;
|
||||
min-width: 0;
|
||||
padding: 1rem 1.25rem 2.5rem;
|
||||
}
|
||||
.toolbar {
|
||||
font-size: 0.85rem;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 0.5rem;
|
||||
flex-wrap: wrap;
|
||||
background: var(--card);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 8px;
|
||||
padding: 0.65rem 0.85rem;
|
||||
margin-bottom: 1rem;
|
||||
}
|
||||
button {
|
||||
padding: 0.35rem 0.65rem;
|
||||
cursor: pointer;
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 6px;
|
||||
background: var(--button-bg);
|
||||
color: var(--button-text);
|
||||
}
|
||||
button:hover { background: var(--button-hover); }
|
||||
details.file {
|
||||
margin: 0.75rem 0;
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 6px;
|
||||
background: var(--card);
|
||||
scroll-margin-top: 0.75rem;
|
||||
}
|
||||
details.file:target {
|
||||
outline: 2px solid var(--accent);
|
||||
outline-offset: 2px;
|
||||
}
|
||||
details.file summary {
|
||||
cursor: pointer;
|
||||
padding: 0.6rem 0.75rem;
|
||||
font-weight: 600;
|
||||
background: var(--summary-bg);
|
||||
}
|
||||
pre.diff {
|
||||
margin: 0;
|
||||
padding: 0.75rem;
|
||||
font-size: 0.72rem;
|
||||
line-height: 1.0;
|
||||
overflow: auto;
|
||||
max-height: 70vh;
|
||||
}
|
||||
.diff-add { background: var(--diff-add-bg); display: block; margin: 0; padding: 0; }
|
||||
.diff-del { background: var(--diff-del-bg); display: block; margin: 0; padding: 0; }
|
||||
.diff-hunk { color: var(--diff-hunk); display: block; margin: 0; padding: 0; }
|
||||
.diff-meta { color: var(--diff-meta); display: block; margin: 0; padding: 0; }
|
||||
.diff-ctx { display: block; margin: 0; padding: 0; }
|
||||
@media (max-width: 900px) {
|
||||
.layout { flex-direction: column; }
|
||||
aside.sidebar {
|
||||
width: 100%;
|
||||
position: relative;
|
||||
max-height: 40vh;
|
||||
border-right: none;
|
||||
border-bottom: 1px solid var(--border);
|
||||
}
|
||||
}
|
||||
/* Dark mode theme - applied when data-theme="dark" (default) */
|
||||
[data-theme="dark"] {
|
||||
--bg: #1a1a1a;
|
||||
--card: #2d2d2d;
|
||||
--border: #404040;
|
||||
--text: #e0e0e0;
|
||||
--blue: #6ba1f5;
|
||||
--accent: #7aa7d6;
|
||||
--header-end: #c586c0;
|
||||
--muted: #9e9e9e;
|
||||
--summary-bg: #252525;
|
||||
--button-bg: #3d3d3d;
|
||||
--button-text: #e0e0e0;
|
||||
--button-hover: #4a4a4a;
|
||||
--churn-add: #7edd8a;
|
||||
--churn-del: #ff7b7b;
|
||||
--diff-add-bg: color-mix(in srgb, #7edd8a 22%, var(--card));
|
||||
--diff-del-bg: color-mix(in srgb, #ff7b7b 22%, var(--card));
|
||||
--link-hover-bg: color-mix(in srgb, #7aa7d6 18%, var(--card));
|
||||
--link-active-bg: color-mix(in srgb, #7aa7d6 22%, var(--card));
|
||||
}
|
||||
/* Light mode theme - applied when data-theme="light" */
|
||||
[data-theme="light"] {
|
||||
--bg: #ffffff;
|
||||
--card: #f8f9fa;
|
||||
--border: #e2e8f0;
|
||||
--text: #1a1a1a;
|
||||
--blue: #0d6efd;
|
||||
--accent: #0d6efd;
|
||||
--header-end: #6610f2;
|
||||
--muted: #6c757d;
|
||||
--summary-bg: #f8f9fa;
|
||||
--button-bg: #ffffff;
|
||||
--button-text: #1a1a1a;
|
||||
--button-hover: #e2e8f0;
|
||||
--churn-add: #15803d;
|
||||
--churn-del: #b91c1c;
|
||||
--diff-add-bg: color-mix(in srgb, #15803d 22%, var(--card));
|
||||
--diff-del-bg: color-mix(in srgb, #b91c1c 22%, var(--card));
|
||||
--link-hover-bg: color-mix(in srgb, #0d6efd 18%, var(--card));
|
||||
--link-active-bg: color-mix(in srgb, #0d6efd 22%, var(--card));
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<header>
|
||||
<div class="header-title">
|
||||
{{HEADER_BODY}}
|
||||
</div>
|
||||
{{THEME_TOGGLE_HTML}}
|
||||
</header>
|
||||
<div class="layout">
|
||||
<aside class="sidebar" aria-label="Changed files">
|
||||
<h2>Files ({{FILE_COUNT}})</h2>
|
||||
{{TREE_HTML}}
|
||||
<details class="stat-fold">
|
||||
<summary>git diff --stat</summary>
|
||||
<pre>{{STAT_BLOCK}}</pre>
|
||||
</details>
|
||||
</aside>
|
||||
<div class="content">
|
||||
<div class="toolbar">
|
||||
<button type="button" id="btn-expand-all">Expand all</button>
|
||||
<button type="button" id="btn-collapse-all">Collapse all</button>
|
||||
</div>
|
||||
{{FILES_HTML}}
|
||||
</div>
|
||||
</div>
|
||||
<script>
|
||||
(function () {
|
||||
// Theme toggle functionality - default is dark mode
|
||||
var themeToggle = document.getElementById('theme-toggle');
|
||||
if (themeToggle) {
|
||||
var htmlEl = document.documentElement;
|
||||
// Default to dark mode
|
||||
var savedTheme = localStorage.getItem('fiona-diff-theme') || 'dark';
|
||||
htmlEl.setAttribute('data-theme', savedTheme);
|
||||
themeToggle.addEventListener('click', function () {
|
||||
var currentTheme = htmlEl.getAttribute('data-theme');
|
||||
var newTheme = currentTheme === 'dark' ? 'light' : 'dark';
|
||||
htmlEl.setAttribute('data-theme', newTheme);
|
||||
localStorage.setItem('fiona-diff-theme', newTheme);
|
||||
});
|
||||
}
|
||||
|
||||
function openFromHash() {
|
||||
var id = location.hash ? location.hash.slice(1) : "";
|
||||
document.querySelectorAll(".file-tree a").forEach(function (a) {
|
||||
a.classList.toggle("active", a.getAttribute("href") === "#" + id);
|
||||
});
|
||||
if (!id) {
|
||||
return;
|
||||
}
|
||||
var el = document.getElementById(id);
|
||||
if (el && el.tagName === "DETAILS") {
|
||||
el.open = true;
|
||||
el.scrollIntoView({ behavior: "smooth", block: "start" });
|
||||
}
|
||||
}
|
||||
document.getElementById("btn-expand-all").addEventListener("click", function () {
|
||||
document.querySelectorAll("details.file").forEach(function (d) { d.open = true; });
|
||||
});
|
||||
document.getElementById("btn-collapse-all").addEventListener("click", function () {
|
||||
document.querySelectorAll("details.file").forEach(function (d) { d.open = false; });
|
||||
});
|
||||
document.querySelectorAll(".file-tree a").forEach(function (a) {
|
||||
a.addEventListener("click", function () {
|
||||
var id = a.getAttribute("href").slice(1);
|
||||
var el = document.getElementById(id);
|
||||
if (el && el.tagName === "DETAILS") {
|
||||
el.open = true;
|
||||
}
|
||||
});
|
||||
});
|
||||
window.addEventListener("hashchange", openFromHash);
|
||||
openFromHash();
|
||||
})();
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -1,4 +1,4 @@
|
||||
"""Shared logging configuration for all Fiona tools (SPEC 7.17).
|
||||
"""Shared logging configuration for all Fiona tools (SPEC 5.1).
|
||||
|
||||
Diagnostics go through the standard `logging` module with the level
|
||||
resolved from (highest precedence first):
|
||||
@@ -13,25 +13,20 @@ records go to stderr.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import ctypes
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
from fiona.paths import llm_wiki_template
|
||||
|
||||
# AI/LLM: Windows-only; module-level so PYC001 allows it (SPEC 6.4.2).
|
||||
if os.name == "nt":
|
||||
import msvcrt
|
||||
|
||||
DEFAULT_LOG_LEVEL = "WARNING"
|
||||
DEFAULT_SETTINGS_FILE = llm_wiki_template() / "fiona-settings.json"
|
||||
DEFAULT_SETTINGS_FILE = (
|
||||
pathlib.Path(__file__).resolve().parent.parent / "llm-wiki" / "fiona-settings.json"
|
||||
)
|
||||
# Tool package loggers that carry the level explicitly: pymarkdown
|
||||
# resets the root logger to WARNING mid-run, which would otherwise
|
||||
# silence tool diagnostics (same pattern as run_sanity, see SPEC 7.17).
|
||||
TOOL_LOGGER_NAMES = ("fiona",)
|
||||
# silence tool diagnostics (same pattern as run_sanity, see SPEC 4.7.1).
|
||||
TOOL_LOGGER_NAMES = ("fiona", "librarian", "mdoctor", "python_ruff", "wiki")
|
||||
|
||||
LOG_FORMAT = "%(levelname)s %(name)s: %(message)s"
|
||||
FILE_LOG_FORMAT = "%(asctime)s %(levelname)s %(name)s: %(message)s"
|
||||
@@ -90,68 +85,6 @@ def load_log_level(settings: dict | None) -> str:
|
||||
return candidate.upper()
|
||||
|
||||
|
||||
def _open_shared_append(path: str, encoding: str):
|
||||
"""Open a log file that Windows can delete while the handle is open.
|
||||
|
||||
Generated by AI/LLM: 2026-09-29 - the default open() omits
|
||||
FILE_SHARE_DELETE, so a temp directory that still has fiona.log
|
||||
open cannot be removed.
|
||||
|
||||
Args:
|
||||
path: Log file path.
|
||||
encoding: Text encoding for the stream.
|
||||
|
||||
Returns:
|
||||
A text file object opened for append.
|
||||
"""
|
||||
if os.name != "nt":
|
||||
return open(path, "a", encoding=encoding)
|
||||
kernel32 = ctypes.WinDLL("kernel32", use_last_error=True)
|
||||
kernel32.CreateFileW.argtypes = [
|
||||
ctypes.c_wchar_p,
|
||||
ctypes.c_uint32,
|
||||
ctypes.c_uint32,
|
||||
ctypes.c_void_p,
|
||||
ctypes.c_uint32,
|
||||
ctypes.c_uint32,
|
||||
ctypes.c_void_p,
|
||||
]
|
||||
kernel32.CreateFileW.restype = ctypes.c_void_p
|
||||
kernel32.SetFilePointer.argtypes = [
|
||||
ctypes.c_void_p,
|
||||
ctypes.c_long,
|
||||
ctypes.c_void_p,
|
||||
ctypes.c_uint32,
|
||||
]
|
||||
kernel32.SetFilePointer.restype = ctypes.c_uint32
|
||||
handle = kernel32.CreateFileW(
|
||||
path,
|
||||
0x40000000, # GENERIC_WRITE
|
||||
0x1 | 0x2 | 0x4, # FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE
|
||||
None,
|
||||
4, # OPEN_ALWAYS
|
||||
0x80, # FILE_ATTRIBUTE_NORMAL
|
||||
None,
|
||||
)
|
||||
invalid = ctypes.c_void_p(-1).value
|
||||
if handle in (None, invalid):
|
||||
raise OSError(ctypes.get_last_error(), f"Cannot open log file {path}")
|
||||
kernel32.SetFilePointer(handle, 0, None, 2) # FILE_END
|
||||
fd = msvcrt.open_osfhandle(handle, os.O_APPEND)
|
||||
return os.fdopen(fd, "a", encoding=encoding, closefd=True)
|
||||
|
||||
|
||||
class _SharedFileHandler(logging.FileHandler):
|
||||
"""File log handler that does not block deletion of its directory.
|
||||
|
||||
Generated by AI/LLM: 2026-09-29 - Windows tests and temp wikis remove
|
||||
the working directory while fiona.log is still attached.
|
||||
"""
|
||||
|
||||
def _open(self):
|
||||
return _open_shared_append(self.baseFilename, self.encoding or "utf-8")
|
||||
|
||||
|
||||
def _remove_file_handler() -> None:
|
||||
"""Detach (and close) the managed file log handler, if any."""
|
||||
for handler in list(logging.getLogger().handlers):
|
||||
@@ -184,7 +117,7 @@ def _configure_file_log(fiona_settings: dict, console_level: str) -> None:
|
||||
)
|
||||
file_level = console_level
|
||||
try:
|
||||
handler = _SharedFileHandler(location, encoding="utf-8")
|
||||
handler = logging.FileHandler(location, encoding="utf-8")
|
||||
except OSError as error:
|
||||
logging.getLogger(__name__).warning("Cannot open log file %s: %s", location, error)
|
||||
return
|
||||
|
||||
@@ -1,21 +0,0 @@
|
||||
"""Filesystem locations for packaged Fiona data.
|
||||
|
||||
Generated by AI/LLM: 2026-09-29 - one resolver for the llm-wiki template.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.resources
|
||||
import pathlib
|
||||
|
||||
|
||||
def llm_wiki_template() -> pathlib.Path:
|
||||
"""Return the packaged llm-wiki template directory.
|
||||
|
||||
Generated by AI/LLM: 2026-09-29 - load package data with importlib.resources.
|
||||
|
||||
Returns:
|
||||
Filesystem path of ``fiona/data/llm_wiki``.
|
||||
"""
|
||||
resource = importlib.resources.files("fiona").joinpath("data", "llm_wiki")
|
||||
return pathlib.Path(str(resource))
|
||||
@@ -1,5 +0,0 @@
|
||||
"""Wiki package for Fiona-Assistant.
|
||||
|
||||
Submodules are imported directly. This package does not import them
|
||||
here, so git_tool can load wiki.ignore without a circular import.
|
||||
"""
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,367 +0,0 @@
|
||||
"""Wiki housekeeping routine (SPEC 4.8, 7.2).
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - one call for LLM wiki maintenance.
|
||||
|
||||
`fiona-assistant housekeeping` runs the maintenance steps in a fixed
|
||||
order and reports, per step, a status (`PASS`, `WARN`, `FAIL`), a
|
||||
summary, detail lines, and the next actions for the LLM. The routine
|
||||
gathers what needs work; it writes only `wiki_graph.html`.
|
||||
|
||||
Steps, in order (`STEP_NAMES`):
|
||||
|
||||
- `check-update`: core files behind the packaged template (read-only).
|
||||
- `merge`: files whose stem ends with `-update` that `wiki update` left for a merge.
|
||||
- `ingest`: pages under `wiki/` still at `status: raw`.
|
||||
- `sanity`: every `wiki sanity` check.
|
||||
- `graph`: regenerate `wiki_graph.html` and list orphan pages.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
from fiona.librarian.md_frontmatter import FrontmatterError, parse_frontmatter
|
||||
from fiona.wiki.graph_generator import wiki_graph_main
|
||||
from fiona.wiki.sanity_manager import SETTINGS_FILE, run_wiki_checks
|
||||
from fiona.wiki.update_manager import (
|
||||
UPDATE_SUFFIX,
|
||||
check_wiki_updates,
|
||||
get_update_ignore_patterns,
|
||||
path_is_update_ignored,
|
||||
)
|
||||
|
||||
STEP_NAMES = ("check-update", "merge", "ingest", "sanity", "graph")
|
||||
GRAPH_FILE = "wiki_graph.html"
|
||||
UPDATE_FILE_SUFFIX = f"{UPDATE_SUFFIX}.md"
|
||||
# AI/LLM: read-only zones sit at the wiki root; .git is skipped anywhere
|
||||
ROOT_SKIP_DIRS = frozenset({"personal", "raw"})
|
||||
|
||||
__all__ = [
|
||||
"GRAPH_FILE",
|
||||
"STEP_NAMES",
|
||||
"check_update_step",
|
||||
"format_housekeeping_report",
|
||||
"graph_step",
|
||||
"housekeeping_main",
|
||||
"ingest_step",
|
||||
"merge_step",
|
||||
"run_housekeeping",
|
||||
"sanity_step",
|
||||
]
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Helpers
|
||||
# =============================================================================
|
||||
|
||||
def _step(name: str, status: str, summary: str, details: list[str] | None = None,
|
||||
next_steps: list[str] | None = None) -> dict:
|
||||
"""Build one step result.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - shared step shape (SPEC 4.8).
|
||||
"""
|
||||
return {
|
||||
"name": name,
|
||||
"status": status,
|
||||
"summary": summary,
|
||||
"details": details or [],
|
||||
"next": next_steps or [],
|
||||
}
|
||||
|
||||
|
||||
def _relative(path: str | Path, wiki_root: Path) -> str:
|
||||
"""Return a path relative to the wiki root, with `/` separators.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - report paths from the wiki root.
|
||||
"""
|
||||
return Path(os.path.relpath(path, wiki_root)).as_posix()
|
||||
|
||||
|
||||
def _frontmatter(path: Path) -> dict:
|
||||
"""Return a page's frontmatter, or an empty dict when it cannot be read.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - sanity reports broken YAML itself.
|
||||
"""
|
||||
try:
|
||||
frontmatter = parse_frontmatter(path)
|
||||
except (FrontmatterError, UnicodeDecodeError):
|
||||
return {}
|
||||
return frontmatter if isinstance(frontmatter, dict) else {}
|
||||
|
||||
|
||||
def _wiki_files(wiki_root: Path) -> list[Path]:
|
||||
"""List files outside read-only zones and `.git`, sorted.
|
||||
|
||||
Modified by AI/LLM: 2026-10-01 - merge walks every suffix, not only markdown.
|
||||
|
||||
Args:
|
||||
wiki_root: Root path of the wiki.
|
||||
|
||||
Returns:
|
||||
list[Path]: Files found.
|
||||
"""
|
||||
found = []
|
||||
for current, dirs, files in os.walk(wiki_root):
|
||||
at_root = Path(current) == wiki_root
|
||||
dirs[:] = [
|
||||
name for name in dirs
|
||||
if name != ".git" and not (at_root and name in ROOT_SKIP_DIRS)
|
||||
]
|
||||
found.extend(Path(current) / name for name in files)
|
||||
return sorted(found)
|
||||
|
||||
|
||||
def _markdown_files(wiki_root: Path, subdir: str = "") -> list[Path]:
|
||||
"""List markdown files outside read-only zones and `.git`, sorted.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - pruned walk for the merge and ingest steps.
|
||||
|
||||
Args:
|
||||
wiki_root: Root path of the wiki.
|
||||
subdir: Optional directory, relative to the root, to walk instead.
|
||||
|
||||
Returns:
|
||||
list[Path]: Markdown files found.
|
||||
"""
|
||||
start = wiki_root / subdir
|
||||
found = []
|
||||
for current, dirs, files in os.walk(start):
|
||||
at_root = Path(current) == wiki_root
|
||||
dirs[:] = [
|
||||
name for name in dirs
|
||||
if name != ".git" and not (at_root and name in ROOT_SKIP_DIRS)
|
||||
]
|
||||
found.extend(Path(current) / name for name in files if name.endswith(".md"))
|
||||
return sorted(found)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Steps
|
||||
# =============================================================================
|
||||
|
||||
def check_update_step(wiki_root: Path) -> dict:
|
||||
"""Report core files behind the template without writing (SPEC 7.2.2).
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - read-only wiki check-update.
|
||||
"""
|
||||
summary = check_wiki_updates(wiki_root)
|
||||
details = [f"update: {_relative(r.file_path, wiki_root)}" for r in summary.files_to_update]
|
||||
details += [
|
||||
f"conflict: {_relative(r.file_path, wiki_root)} -> {_relative(r.update_path, wiki_root)}"
|
||||
for r in summary.conflicts
|
||||
]
|
||||
details += [f"add: {path.as_posix()}" for path in summary.files_to_add]
|
||||
text = (
|
||||
f"{len(summary.files_to_update)} update(s), {len(summary.conflicts)} conflict(s), "
|
||||
f"{len(summary.files_to_add)} addition(s)"
|
||||
)
|
||||
if not details:
|
||||
return _step("check-update", "PASS", text)
|
||||
return _step("check-update", "WARN", text, details, [
|
||||
"Ask the user, then run fiona-assistant wiki update --dry-run and wiki update --yes."
|
||||
])
|
||||
|
||||
|
||||
def merge_step(wiki_root: Path) -> dict:
|
||||
"""List update files that wait for an LLM merge (SPEC 7.2.3).
|
||||
|
||||
Modified by AI/LLM: 2026-10-02 - skip update.ignore_files and corefiles.json.
|
||||
"""
|
||||
details, actions = [], []
|
||||
ignore_patterns = get_update_ignore_patterns(wiki_root)
|
||||
for update_file in _wiki_files(wiki_root):
|
||||
if not update_file.stem.endswith(UPDATE_SUFFIX):
|
||||
continue
|
||||
if path_is_update_ignored(_relative(update_file, wiki_root), ignore_patterns):
|
||||
continue
|
||||
target = update_file.with_name(
|
||||
update_file.stem[: -len(UPDATE_SUFFIX)] + update_file.suffix
|
||||
)
|
||||
update_rel = _relative(update_file, wiki_root)
|
||||
target_rel = _relative(target, wiki_root)
|
||||
missing = "" if target.exists() else " (target missing)"
|
||||
details.append(f"{update_rel} -> {target_rel}{missing}")
|
||||
version = _frontmatter(update_file).get("version") or "the template version"
|
||||
actions.append(
|
||||
f"Merge {update_rel} into {target_rel}: keep the owner's edits, "
|
||||
f"take the template changes, set version to at least {version}, "
|
||||
f"then delete {update_rel}."
|
||||
)
|
||||
text = f"{len(details)} update file(s) wait for a merge"
|
||||
return _step("merge", "FAIL" if details else "PASS", text, details, actions)
|
||||
|
||||
|
||||
def ingest_step(wiki_root: Path) -> dict:
|
||||
"""List pages under wiki/ still at `status: raw` (SPEC 7.2.4).
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - pages waiting for a refine.
|
||||
"""
|
||||
pages = [
|
||||
_relative(page, wiki_root)
|
||||
for page in _markdown_files(wiki_root, "wiki")
|
||||
if _frontmatter(page).get("status") == "raw"
|
||||
]
|
||||
text = f"{len(pages)} raw page(s)"
|
||||
if not pages:
|
||||
return _step("ingest", "PASS", text)
|
||||
return _step("ingest", "WARN", text, pages,
|
||||
["Refine each raw page, then set status: refined."])
|
||||
|
||||
|
||||
def sanity_step(wiki_root: Path) -> dict:
|
||||
"""Run every wiki sanity check without the progress display (SPEC 7.2.5).
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - failing check reports as details.
|
||||
"""
|
||||
results = run_wiki_checks(wiki_root)
|
||||
failing = {name: check for name, check in results["checks"].items() if check["result"]}
|
||||
text = f"{results['total_violations']} violation(s) in {len(failing)} check(s)"
|
||||
if not failing:
|
||||
return _step("sanity", "PASS", text)
|
||||
details = [
|
||||
f"{name}: {line.strip().removeprefix('- ')}"
|
||||
for name, check in failing.items()
|
||||
for line in check["stdout"].splitlines()
|
||||
if line.strip()
|
||||
]
|
||||
return _step("sanity", "FAIL", text, details,
|
||||
["Fix the violations, then run fiona-assistant housekeeping again."])
|
||||
|
||||
|
||||
def graph_step(wiki_root: Path) -> dict:
|
||||
"""Write wiki_graph.html at the wiki root and list orphans (SPEC 7.2.6).
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - graph plus orphan pages under wiki/.
|
||||
"""
|
||||
captured = io.StringIO()
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
graph_json = Path(temp_dir) / "graph.json"
|
||||
with contextlib.redirect_stdout(captured), contextlib.redirect_stderr(captured):
|
||||
code = wiki_graph_main(str(wiki_root), output=str(graph_json), format_type="json")
|
||||
if code == 0:
|
||||
code = wiki_graph_main(str(wiki_root), output=str(wiki_root / GRAPH_FILE))
|
||||
if code != 0:
|
||||
error = captured.getvalue().strip() or f"wiki graph exited {code}"
|
||||
return _step("graph", "FAIL", f"could not write {GRAPH_FILE}",
|
||||
error.splitlines())
|
||||
data = json.loads(graph_json.read_text(encoding="utf-8"))
|
||||
|
||||
orphans = sorted(
|
||||
rel for rel in (_relative(path, wiki_root) for path in data["orphans"])
|
||||
if rel.startswith("wiki/") and not rel.endswith(UPDATE_FILE_SUFFIX)
|
||||
)
|
||||
text = (
|
||||
f"wrote {GRAPH_FILE} ({data['metadata']['file_count']} nodes, "
|
||||
f"{len(data['edges'])} edges)"
|
||||
)
|
||||
if not orphans:
|
||||
return _step("graph", "PASS", text)
|
||||
return _step("graph", "WARN", text, [f"orphan: {rel}" for rel in orphans],
|
||||
["Link each orphan page from a related page or from wiki/index.md."])
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Routine and Report
|
||||
# =============================================================================
|
||||
|
||||
def run_housekeeping(directory: str | Path = ".") -> dict:
|
||||
"""Run every housekeeping step over a wiki (SPEC 7.2.1).
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - every step runs, even after a FAIL.
|
||||
|
||||
Args:
|
||||
directory: The wiki directory. Defaults to the current directory.
|
||||
|
||||
Returns:
|
||||
dict: `target` (resolved path), `steps` (one result per step,
|
||||
in `STEP_NAMES` order), and `passed` (no step is `FAIL`).
|
||||
|
||||
Raises:
|
||||
FileNotFoundError: If the directory does not exist.
|
||||
NotADirectoryError: If the directory is not an llm-wiki directory.
|
||||
"""
|
||||
wiki_root = Path(directory).resolve()
|
||||
if not wiki_root.exists():
|
||||
raise FileNotFoundError(f"Directory not found: {directory}")
|
||||
if not (wiki_root / SETTINGS_FILE).exists():
|
||||
raise NotADirectoryError(f"Directory '{directory}' is not an llm-wiki directory")
|
||||
|
||||
steps = [
|
||||
step(wiki_root)
|
||||
for step in (
|
||||
check_update_step,
|
||||
merge_step,
|
||||
ingest_step,
|
||||
sanity_step,
|
||||
graph_step,
|
||||
)
|
||||
]
|
||||
return {
|
||||
"target": wiki_root,
|
||||
"steps": steps,
|
||||
"passed": all(step["status"] != "FAIL" for step in steps),
|
||||
}
|
||||
|
||||
|
||||
def format_housekeeping_report(results: dict) -> str:
|
||||
"""Format housekeeping results for humans and LLMs (SPEC 4.8).
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - step blocks, next steps, verdict.
|
||||
|
||||
Args:
|
||||
results: The dict returned by run_housekeeping.
|
||||
|
||||
Returns:
|
||||
str: The report text.
|
||||
"""
|
||||
lines = ["Housekeeping Report", f"TARGET: {results['target']}", ""]
|
||||
for step in results["steps"]:
|
||||
lines.append(f"[{step['status']}] {step['name']}: {step['summary']}")
|
||||
lines.extend(f" - {detail}" for detail in step["details"])
|
||||
|
||||
actions = list(dict.fromkeys(a for step in results["steps"] for a in step["next"]))
|
||||
if actions:
|
||||
lines.extend(["", "Next steps:"])
|
||||
lines.extend(f" - {action}" for action in actions)
|
||||
|
||||
failing = [step["name"] for step in results["steps"] if step["status"] == "FAIL"]
|
||||
lines.append("")
|
||||
if failing:
|
||||
lines.append(
|
||||
f"HOUSEKEEPING FAILED - {len(failing)} step(s) need attention: "
|
||||
f"{', '.join(failing)}"
|
||||
)
|
||||
else:
|
||||
lines.append("HOUSEKEEPING PASSED")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def housekeeping_main(directory: str = ".", json_output: bool = False) -> int:
|
||||
"""Main function for the 'housekeeping' CLI command.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - exit 1 on any FAIL step.
|
||||
|
||||
Args:
|
||||
directory: Wiki directory (default: current directory).
|
||||
json_output: If True, print the results as one JSON object.
|
||||
|
||||
Returns:
|
||||
int: 0 when no step is `FAIL`, 1 otherwise or on an invalid wiki.
|
||||
"""
|
||||
try:
|
||||
results = run_housekeeping(directory)
|
||||
except (FileNotFoundError, NotADirectoryError) as error:
|
||||
print(f"Error: {error}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
if json_output:
|
||||
print(json.dumps({**results, "target": str(results["target"])}, indent=2))
|
||||
else:
|
||||
print(format_housekeeping_report(results))
|
||||
return 0 if results["passed"] else 1
|
||||
@@ -1,171 +0,0 @@
|
||||
"""Gitignore patterns shared by wiki init and wiki update.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - seed wiki ignores from the repository file.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import fnmatch
|
||||
import importlib.resources
|
||||
import pathlib
|
||||
|
||||
_PACKAGE_NAME = "fiona-assistant"
|
||||
_PERSONAL_PATTERN = "personal/"
|
||||
|
||||
|
||||
def gitignore_patterns_from_text(text: str) -> list[str]:
|
||||
"""Return ignore patterns from gitignore text.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - drop blanks and comments.
|
||||
|
||||
Args:
|
||||
text: Contents of a .gitignore file.
|
||||
|
||||
Returns:
|
||||
Patterns in file order, without comments or blank lines.
|
||||
"""
|
||||
patterns: list[str] = []
|
||||
for line in text.splitlines():
|
||||
stripped = line.strip()
|
||||
if stripped and not stripped.startswith("#"):
|
||||
patterns.append(stripped)
|
||||
return patterns
|
||||
|
||||
|
||||
def path_is_ignored(relpath: str, patterns: list[str]) -> bool:
|
||||
"""Return whether a relative posix path matches any gitignore pattern.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - match unanchored names and directory prefixes.
|
||||
|
||||
A pattern with no slash matches any path segment. A trailing slash matches
|
||||
a directory and everything under it. ``*`` does not cross a directory
|
||||
boundary because matching is done per segment.
|
||||
|
||||
Args:
|
||||
relpath: Path relative to the wiki or template root, posix form.
|
||||
patterns: Gitignore patterns.
|
||||
|
||||
Returns:
|
||||
True when one pattern matches.
|
||||
"""
|
||||
return any(_pattern_matches(relpath, pattern) for pattern in patterns)
|
||||
|
||||
|
||||
def _pattern_matches(relpath: str, pattern: str) -> bool:
|
||||
"""Return whether one gitignore pattern matches a relative posix path."""
|
||||
body = pattern.rstrip("/")
|
||||
if not body:
|
||||
return False
|
||||
parts = relpath.split("/")
|
||||
if "/" in body:
|
||||
prefixes = ["/".join(parts[:index]) for index in range(1, len(parts) + 1)]
|
||||
return any(fnmatch.fnmatchcase(prefix, body) for prefix in prefixes)
|
||||
return any(fnmatch.fnmatchcase(part, body) for part in parts)
|
||||
|
||||
|
||||
def repository_gitignore_path() -> pathlib.Path | None:
|
||||
"""Find the fiona-assistant repository .gitignore, if this tree has one.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - prefer the source checkout over package data.
|
||||
|
||||
Returns:
|
||||
Path to the repository .gitignore, or None when this install is not
|
||||
the source checkout.
|
||||
"""
|
||||
here = pathlib.Path(__file__).resolve()
|
||||
for parent in here.parents:
|
||||
project = parent / "pyproject.toml"
|
||||
gitignore = parent / ".gitignore"
|
||||
if not project.is_file() or not gitignore.is_file():
|
||||
continue
|
||||
text = project.read_text(encoding="utf-8")
|
||||
if f'name = "{_PACKAGE_NAME}"' in text:
|
||||
return gitignore
|
||||
return None
|
||||
|
||||
|
||||
def packaged_gitignore_path() -> pathlib.Path | None:
|
||||
"""Return the packaged copy of the repository .gitignore, if present.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - fallback for an installed wheel.
|
||||
|
||||
Returns:
|
||||
Path to ``fiona/data/default.gitignore``, or None when it is absent.
|
||||
"""
|
||||
resource = importlib.resources.files("fiona").joinpath("data", "default.gitignore")
|
||||
path = pathlib.Path(str(resource))
|
||||
if path.is_file():
|
||||
return path
|
||||
return None
|
||||
|
||||
|
||||
def repository_gitignore_text() -> str:
|
||||
"""Load the ignore file that seeds a wiki, as LF text.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - repository file, then packaged copy.
|
||||
|
||||
Returns:
|
||||
Ignore file text ending in a newline.
|
||||
|
||||
Raises:
|
||||
FileNotFoundError: When neither the repository nor the packaged file exists.
|
||||
"""
|
||||
path = repository_gitignore_path() or packaged_gitignore_path()
|
||||
if path is None:
|
||||
raise FileNotFoundError("fiona-assistant repository .gitignore was not found")
|
||||
text = path.read_text(encoding="utf-8").replace("\r\n", "\n").replace("\r", "\n")
|
||||
if text and not text.endswith("\n"):
|
||||
text += "\n"
|
||||
return text
|
||||
|
||||
|
||||
def wiki_gitignore_patterns() -> list[str]:
|
||||
"""Return patterns written into a wiki, repository set plus personal/.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - personal/ stays untracked after init.
|
||||
|
||||
Returns:
|
||||
Ignore patterns in write order.
|
||||
"""
|
||||
patterns = gitignore_patterns_from_text(repository_gitignore_text())
|
||||
if _PERSONAL_PATTERN not in patterns:
|
||||
patterns.append(_PERSONAL_PATTERN)
|
||||
return patterns
|
||||
|
||||
|
||||
def wiki_gitignore_text() -> str:
|
||||
"""Return the full .gitignore body for a wiki that does not have one yet.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - repository text plus personal/ when missing.
|
||||
|
||||
Returns:
|
||||
File body ending in a newline.
|
||||
"""
|
||||
text = repository_gitignore_text()
|
||||
if _PERSONAL_PATTERN not in gitignore_patterns_from_text(text):
|
||||
text += f"{_PERSONAL_PATTERN}\n"
|
||||
return text
|
||||
|
||||
|
||||
def ensure_wiki_gitignore(directory: pathlib.Path) -> None:
|
||||
"""Create .gitignore or append patterns that are not already listed.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - put or append the repository ignores.
|
||||
|
||||
Args:
|
||||
directory: Wiki root that should contain .gitignore.
|
||||
"""
|
||||
target = directory / ".gitignore"
|
||||
desired = wiki_gitignore_patterns()
|
||||
if not target.exists():
|
||||
target.write_text(wiki_gitignore_text(), encoding="utf-8", newline="\n")
|
||||
return
|
||||
existing = target.read_text(encoding="utf-8").replace("\r\n", "\n").replace("\r", "\n")
|
||||
have = set(gitignore_patterns_from_text(existing))
|
||||
missing = [pattern for pattern in desired if pattern not in have]
|
||||
if not missing:
|
||||
return
|
||||
if existing and not existing.endswith("\n"):
|
||||
existing += "\n"
|
||||
existing += "\n".join(missing) + "\n"
|
||||
target.write_text(existing, encoding="utf-8", newline="\n")
|
||||
@@ -1,4 +0,0 @@
|
||||
"""Wiki ingest package (SPEC 7.1.5).
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - stage modules for wiki ingest.
|
||||
"""
|
||||
@@ -1,239 +0,0 @@
|
||||
"""Archive a fetched file under ``raw/``.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.3. Never edits a file already there.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import logging
|
||||
import pathlib
|
||||
import shutil
|
||||
|
||||
from fiona.wiki.ingest.converters.base import ConverterRegistry
|
||||
from fiona.wiki.ingest.media import extension_for
|
||||
from fiona.wiki.ingest.models import ArchiveResult, FetchedSource, IngestError
|
||||
from fiona.wiki.ingest.settings import IngestSettings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class RawArchiver:
|
||||
"""Copy or move a source into ``raw/`` and reuse identical bytes.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SHA-256 reuse and collision names.
|
||||
"""
|
||||
|
||||
def __init__(self, wiki_root: pathlib.Path) -> None:
|
||||
"""Store the wiki root.
|
||||
|
||||
Args:
|
||||
wiki_root: Directory that contains ``raw/``.
|
||||
"""
|
||||
self.wiki_root = wiki_root
|
||||
|
||||
def find_by_sha256(
|
||||
self,
|
||||
digest: str,
|
||||
size: int,
|
||||
skip: pathlib.Path | None = None,
|
||||
) -> pathlib.Path | None:
|
||||
"""Return the first ``raw/`` file with this size and digest.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - only same-size files are hashed.
|
||||
|
||||
Args:
|
||||
digest: Lowercase SHA-256 hex.
|
||||
size: Byte size that a candidate must match.
|
||||
skip: A file to ignore, usually the source when it already sits in ``raw/``.
|
||||
|
||||
Returns:
|
||||
pathlib.Path | None: Match with the lowest path, or None.
|
||||
"""
|
||||
raw_root = self.wiki_root / "raw"
|
||||
if not raw_root.is_dir():
|
||||
return None
|
||||
skip_resolved = skip.resolve() if skip is not None else None
|
||||
matches: list[pathlib.Path] = []
|
||||
for path in raw_root.rglob("*"):
|
||||
if not path.is_file():
|
||||
continue
|
||||
if skip_resolved is not None and path.resolve() == skip_resolved:
|
||||
continue
|
||||
try:
|
||||
if path.stat().st_size != size:
|
||||
continue
|
||||
except OSError:
|
||||
continue
|
||||
if _sha256(path) == digest:
|
||||
matches.append(path)
|
||||
if not matches:
|
||||
return None
|
||||
matches.sort(key=lambda item: item.as_posix())
|
||||
return matches[0]
|
||||
|
||||
def archive(
|
||||
self,
|
||||
source: FetchedSource,
|
||||
*,
|
||||
registry: ConverterRegistry,
|
||||
settings: IngestSettings,
|
||||
raw_override: str | None,
|
||||
dry_run: bool,
|
||||
) -> ArchiveResult:
|
||||
"""Place the source under ``raw/`` or reuse an identical copy.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.3.
|
||||
|
||||
Args:
|
||||
source: Fetched file.
|
||||
registry: Used for the raw category and a missing extension.
|
||||
settings: ``raw_categories`` overrides.
|
||||
raw_override: ``-r`` value, relative to ``raw/``, or None.
|
||||
dry_run: When true, compute the path and write nothing.
|
||||
|
||||
Returns:
|
||||
ArchiveResult: Relative raw path, digest, and warnings.
|
||||
|
||||
Raises:
|
||||
IngestError: ``-r`` escapes ``raw/`` or is used on a file already there.
|
||||
"""
|
||||
raw_root = (self.wiki_root / "raw").resolve()
|
||||
source_in_raw = _is_inside(source.path, raw_root)
|
||||
if raw_override and source_in_raw:
|
||||
raise IngestError(
|
||||
f"--raw cannot be used on a file already under raw/: {source.argument}"
|
||||
)
|
||||
if raw_override:
|
||||
_require_inside(raw_root, raw_override, "--raw must stay inside raw/")
|
||||
digest = _sha256(source.path)
|
||||
match = self.find_by_sha256(
|
||||
digest,
|
||||
source.size_bytes,
|
||||
skip=source.path if source_in_raw else None,
|
||||
)
|
||||
warnings: list[str] = []
|
||||
if match is not None:
|
||||
raw_rel = _relative(match, self.wiki_root)
|
||||
if raw_override:
|
||||
warnings.append(f"already archived at {raw_rel}; --raw ignored")
|
||||
if _is_inside(source.path, self.wiki_root / "ingest") and not dry_run:
|
||||
source.path.unlink()
|
||||
logger.info("reused raw file %s", raw_rel)
|
||||
return ArchiveResult(match, raw_rel, digest, True, warnings)
|
||||
if source_in_raw:
|
||||
return ArchiveResult(source.path, _relative(source.path, self.wiki_root), digest, False, [])
|
||||
extension = source.path.suffix.lower() or extension_for(source.media_type, "", registry)
|
||||
dest = self._destination(source, registry, settings, raw_override, extension, raw_root)
|
||||
dest = _free_name(dest, source.slug, extension)
|
||||
if not dry_run:
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
if _is_inside(source.path, self.wiki_root / "ingest"):
|
||||
shutil.move(str(source.path), str(dest))
|
||||
else:
|
||||
shutil.copyfile(source.path, dest)
|
||||
logger.info("archived %s", dest)
|
||||
return ArchiveResult(dest, _relative(dest, self.wiki_root), digest, False, warnings)
|
||||
|
||||
def _destination(
|
||||
self,
|
||||
source: FetchedSource,
|
||||
registry: ConverterRegistry,
|
||||
settings: IngestSettings,
|
||||
raw_override: str | None,
|
||||
extension: str,
|
||||
raw_root: pathlib.Path,
|
||||
) -> pathlib.Path:
|
||||
"""Resolve the raw path before a collision suffix is applied.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - ``-r`` file or directory, else the category.
|
||||
"""
|
||||
filename = f"{source.slug}{extension}"
|
||||
if raw_override:
|
||||
relative = _strip_prefix(raw_override, "raw")
|
||||
suffix = pathlib.PurePosixPath(relative.replace("\\", "/")).suffix
|
||||
if suffix:
|
||||
return raw_root / relative
|
||||
return raw_root / relative / filename
|
||||
converter = registry.find(source.media_type, extension)
|
||||
if converter is None:
|
||||
category = "other"
|
||||
else:
|
||||
category = settings.raw_categories.get(converter.name, converter.raw_category)
|
||||
return raw_root / category / filename
|
||||
|
||||
|
||||
def _sha256(path: pathlib.Path) -> str:
|
||||
"""Return the lowercase SHA-256 hex digest of a file.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - streamed so large files stay cheap.
|
||||
"""
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
while True:
|
||||
chunk = handle.read(1024 * 1024)
|
||||
if not chunk:
|
||||
break
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def _free_name(dest: pathlib.Path, slug: str, extension: str) -> pathlib.Path:
|
||||
"""Return ``dest`` or the first free ``<slug>-N`` name beside it.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - never overwrite a raw file.
|
||||
"""
|
||||
if not dest.exists():
|
||||
return dest
|
||||
number = 2
|
||||
while True:
|
||||
candidate = dest.parent / f"{slug}-{number}{extension}"
|
||||
if not candidate.exists():
|
||||
return candidate
|
||||
number += 1
|
||||
|
||||
|
||||
def _strip_prefix(value: str, prefix: str) -> str:
|
||||
"""Drop a leading ``raw/`` or ``wiki/`` from a user path.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - both forms are accepted.
|
||||
"""
|
||||
relative = value.replace("\\", "/").lstrip("/")
|
||||
marker = f"{prefix}/"
|
||||
if relative == prefix:
|
||||
return ""
|
||||
if relative.startswith(marker):
|
||||
return relative[len(marker):]
|
||||
return relative
|
||||
|
||||
|
||||
def _require_inside(root: pathlib.Path, value: str, message: str) -> pathlib.Path:
|
||||
"""Resolve ``value`` under ``root`` or raise.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - ``..`` and symlinks must stay inside.
|
||||
"""
|
||||
relative = _strip_prefix(value, root.name)
|
||||
candidate = (root / relative).resolve()
|
||||
if not _is_inside(candidate, root):
|
||||
raise IngestError(f"{message}: {value}")
|
||||
return candidate
|
||||
|
||||
|
||||
def _is_inside(path: pathlib.Path, root: pathlib.Path) -> bool:
|
||||
"""Return whether ``path`` is ``root`` or a file inside it.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - compare resolved paths.
|
||||
"""
|
||||
try:
|
||||
path.resolve().relative_to(root.resolve())
|
||||
except ValueError:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _relative(path: pathlib.Path, root: pathlib.Path) -> str:
|
||||
"""Return a wiki-relative path with ``/`` separators.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - JSON and log paths.
|
||||
"""
|
||||
return path.resolve().relative_to(root.resolve()).as_posix()
|
||||
@@ -1,38 +0,0 @@
|
||||
"""Shipped converters, in registration order.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - markdown, text, html, then pdf.
|
||||
Modified by AI/LLM: 2026-10-07 - the html converter takes the block sweep settings.
|
||||
"""
|
||||
|
||||
from fiona.wiki.ingest.converters.base import ConverterRegistry
|
||||
from fiona.wiki.ingest.converters.html import HtmlConverter
|
||||
from fiona.wiki.ingest.converters.markdown import MarkdownConverter
|
||||
from fiona.wiki.ingest.converters.pdf import PdfConverter
|
||||
from fiona.wiki.ingest.converters.text import TextConverter
|
||||
from fiona.wiki.ingest.settings import IngestSettings, default_settings
|
||||
|
||||
|
||||
def default_registry(settings: IngestSettings | None = None) -> ConverterRegistry:
|
||||
"""Return the converters shipped with wiki ingest.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - registration order from SPEC 7.1.5.
|
||||
Modified by AI/LLM: 2026-10-07 - ``html.*`` settings reach ``HtmlConverter``.
|
||||
|
||||
Args:
|
||||
settings: Ingest settings. ``None`` uses the defaults.
|
||||
|
||||
Returns:
|
||||
ConverterRegistry: Markdown, text, HTML, and PDF.
|
||||
"""
|
||||
settings = settings or default_settings()
|
||||
registry = ConverterRegistry()
|
||||
registry.register(MarkdownConverter)
|
||||
registry.register(TextConverter)
|
||||
registry.register(
|
||||
HtmlConverter(
|
||||
min_words=settings.html_min_words,
|
||||
max_link_density=settings.html_max_link_density,
|
||||
)
|
||||
)
|
||||
registry.register(PdfConverter)
|
||||
return registry
|
||||
@@ -1,108 +0,0 @@
|
||||
"""Converter base types and the registry.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - one class per format, SPEC 7.1.5.1.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import abc
|
||||
import typing
|
||||
|
||||
from fiona.wiki.ingest.models import ConversionResult, FetchedSource
|
||||
|
||||
|
||||
class ConversionError(Exception):
|
||||
"""Raised when a converter cannot read a file it claims.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - exit 3, raw copy kept.
|
||||
"""
|
||||
|
||||
def __init__(self, converter: str, reason: str) -> None:
|
||||
"""Store the converter name and the reason.
|
||||
|
||||
Args:
|
||||
converter: Converter ``name`` attribute.
|
||||
reason: Short failure text, without the ``Error:`` prefix.
|
||||
"""
|
||||
super().__init__(f"{converter}: {reason}")
|
||||
self.converter = converter
|
||||
self.reason = reason
|
||||
|
||||
|
||||
class SourceConverter(abc.ABC):
|
||||
"""One format. Subclasses declare what they claim and implement ``convert``.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.1.2.
|
||||
"""
|
||||
|
||||
name: typing.ClassVar[str]
|
||||
media_types: typing.ClassVar[tuple[str, ...]]
|
||||
extensions: typing.ClassVar[tuple[str, ...]]
|
||||
raw_category: typing.ClassVar[str]
|
||||
|
||||
@abc.abstractmethod
|
||||
def convert(self, source: FetchedSource) -> ConversionResult:
|
||||
"""Convert an archived file to markdown.
|
||||
|
||||
Args:
|
||||
source: Fetched file, including media type and slug.
|
||||
|
||||
Returns:
|
||||
ConversionResult: Body, title, metadata, and warnings.
|
||||
|
||||
Raises:
|
||||
ConversionError: The file cannot be read.
|
||||
"""
|
||||
|
||||
|
||||
class ConverterRegistry:
|
||||
"""Ordered list of converters.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - first media-type match, then extension.
|
||||
"""
|
||||
|
||||
def __init__(self, converters: list[SourceConverter] | None = None) -> None:
|
||||
"""Start with an optional list of converter instances.
|
||||
|
||||
Args:
|
||||
converters: Instances in registration order.
|
||||
"""
|
||||
self.converters: list[SourceConverter] = list(converters or [])
|
||||
|
||||
def register(self, converter: SourceConverter | type[SourceConverter]) -> None:
|
||||
"""Append a converter class or instance.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - adding a format is one registration.
|
||||
|
||||
Args:
|
||||
converter: Class with a no-arg constructor, or a ready instance.
|
||||
"""
|
||||
if isinstance(converter, type):
|
||||
converter = converter()
|
||||
self.converters.append(converter)
|
||||
|
||||
def find(self, media_type: str, extension: str) -> SourceConverter | None:
|
||||
"""Pick the converter for a media type and extension.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.4.1 registration order.
|
||||
|
||||
Args:
|
||||
media_type: Detected type, or empty.
|
||||
extension: Suffix including the dot, or empty.
|
||||
|
||||
Returns:
|
||||
SourceConverter | None: The first match, or None.
|
||||
"""
|
||||
media = (media_type or "").lower()
|
||||
ext = extension.lower()
|
||||
if ext and not ext.startswith("."):
|
||||
ext = f".{ext}"
|
||||
if media:
|
||||
for converter in self.converters:
|
||||
if media in converter.media_types:
|
||||
return converter
|
||||
if ext:
|
||||
for converter in self.converters:
|
||||
if ext in converter.extensions:
|
||||
return converter
|
||||
return None
|
||||
@@ -1,141 +0,0 @@
|
||||
"""HTML source converter.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - article text via beautifulsoup4 and markdownify.
|
||||
Modified by AI/LLM: 2026-10-07 - the block sweep replaces the first-``<article>`` root.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import typing
|
||||
import urllib.parse
|
||||
|
||||
import bs4
|
||||
|
||||
from fiona.wiki.ingest.converters.base import SourceConverter
|
||||
from fiona.wiki.ingest.converters.html_sweep import sweep, unwrap_shadow_roots
|
||||
from fiona.wiki.ingest.media import decode_bytes
|
||||
from fiona.wiki.ingest.models import ConversionResult, FetchedSource
|
||||
|
||||
_DROP_TAGS = ("script", "style", "noscript", "template")
|
||||
_NO_TEXT = "no text content; the page may need JavaScript or a login"
|
||||
|
||||
|
||||
class HtmlConverter(SourceConverter):
|
||||
"""Convert an HTML document to markdown.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.4.5.
|
||||
Modified by AI/LLM: 2026-10-07 - sweep limits are constructor arguments.
|
||||
"""
|
||||
|
||||
name: typing.ClassVar[str] = "html"
|
||||
media_types: typing.ClassVar[tuple[str, ...]] = ("text/html", "application/xhtml+xml")
|
||||
extensions: typing.ClassVar[tuple[str, ...]] = (".html", ".htm", ".xhtml")
|
||||
raw_category: typing.ClassVar[str] = "articles"
|
||||
|
||||
def __init__(self, min_words: int = 4, max_link_density: float = 0.5) -> None:
|
||||
"""Store the block sweep limits.
|
||||
|
||||
Generated by AI/LLM: 2026-10-07 - defaults from SPEC 5.1 ``fiona.wiki.ingest.html``.
|
||||
|
||||
Args:
|
||||
min_words: Fewer words make a ``p`` or loose block ``short``.
|
||||
max_link_density: Link text over text length above this is ``links``.
|
||||
"""
|
||||
self.min_words = min_words
|
||||
self.max_link_density = max_link_density
|
||||
|
||||
def convert(self, source: FetchedSource) -> ConversionResult:
|
||||
"""Parse HTML and return every swept block of the body as markdown.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - drop scripts and absolutize links.
|
||||
Modified by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6 block sweep over ``<body>``.
|
||||
|
||||
Args:
|
||||
source: Archived HTML file. ``origin`` is the page URL when fetched.
|
||||
|
||||
Returns:
|
||||
ConversionResult: Markdown body, title, and ``source_meta``.
|
||||
"""
|
||||
text, warnings = decode_bytes(source.path.read_bytes(), source.charset)
|
||||
soup = bs4.BeautifulSoup(text, "html.parser")
|
||||
unwrap_shadow_roots(soup)
|
||||
for tag in soup.find_all(_DROP_TAGS):
|
||||
tag.decompose()
|
||||
for comment in soup.find_all(string=lambda node: isinstance(node, bs4.Comment)):
|
||||
comment.extract()
|
||||
body = soup.body or soup
|
||||
origin = source.origin if source.origin_kind == "url" else None
|
||||
_absolutize(body, soup, origin)
|
||||
conversion = "complete"
|
||||
if not body.get_text(strip=True):
|
||||
conversion = "partial"
|
||||
warnings = [*warnings, _NO_TEXT]
|
||||
swept = sweep(body, min_words=self.min_words, max_link_density=self.max_link_density)
|
||||
if swept.partial:
|
||||
conversion = "partial"
|
||||
return ConversionResult(
|
||||
body=swept.body,
|
||||
title=_html_title(soup, body),
|
||||
source_meta=_source_meta(soup),
|
||||
conversion=conversion,
|
||||
warnings=[*warnings, *swept.warnings],
|
||||
converter=self.name,
|
||||
)
|
||||
|
||||
|
||||
def _absolutize(root: bs4.element.Tag, soup: bs4.BeautifulSoup, origin: str | None) -> None:
|
||||
"""Rewrite relative href and src values against base href, then the origin.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - a local file with no origin keeps relative links.
|
||||
"""
|
||||
base_tag = soup.find("base", href=True)
|
||||
base = base_tag.get("href") if base_tag is not None else None
|
||||
if not base:
|
||||
base = origin
|
||||
if not base:
|
||||
return
|
||||
for tag in root.find_all(href=True):
|
||||
tag["href"] = urllib.parse.urljoin(str(base), str(tag["href"]))
|
||||
for tag in root.find_all(src=True):
|
||||
tag["src"] = urllib.parse.urljoin(str(base), str(tag["src"]))
|
||||
|
||||
|
||||
def _html_title(soup: bs4.BeautifulSoup, root: bs4.element.Tag) -> str | None:
|
||||
"""Return ``<title>``, then ``og:title``, then the first ``<h1>`` that has text.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.4.5.1.
|
||||
Modified by AI/LLM: 2026-10-07 - the root is ``<body>``, so a blank ``<h1>`` is skipped.
|
||||
"""
|
||||
if soup.title and soup.title.string and soup.title.string.strip():
|
||||
return soup.title.string.strip()
|
||||
og = soup.find("meta", attrs={"property": "og:title"})
|
||||
if og is not None and og.get("content"):
|
||||
return str(og["content"]).strip() or None
|
||||
for heading in root.find_all("h1"):
|
||||
text = heading.get_text(strip=True)
|
||||
if text:
|
||||
return text
|
||||
return None
|
||||
|
||||
|
||||
def _source_meta(soup: bs4.BeautifulSoup) -> dict:
|
||||
"""Collect language, canonical URL, and selected meta values.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - only keys that are present.
|
||||
"""
|
||||
meta: dict[str, str] = {}
|
||||
root = soup.find("html")
|
||||
if root is not None and root.get("lang"):
|
||||
meta["lang"] = str(root["lang"])
|
||||
canonical = soup.find("link", rel=lambda value: value and "canonical" in value)
|
||||
if canonical is not None and canonical.get("href"):
|
||||
meta["canonical"] = str(canonical["href"])
|
||||
for tag in soup.find_all("meta"):
|
||||
key = tag.get("name") or tag.get("property")
|
||||
content = tag.get("content")
|
||||
if not key or content is None:
|
||||
continue
|
||||
name = str(key)
|
||||
if name in {"description", "author"} or name.startswith("og:") or name == "article:published_time":
|
||||
meta[name] = str(content)
|
||||
return meta
|
||||
@@ -1,285 +0,0 @@
|
||||
"""Score every block of an HTML body as ``keep``, ``review``, or ``drop``.
|
||||
|
||||
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6. Rules read HTML structure only.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import dataclasses
|
||||
import re
|
||||
import typing
|
||||
|
||||
import bs4
|
||||
import markdownify
|
||||
|
||||
_BLOCK_KINDS = {
|
||||
"h1": "heading",
|
||||
"h2": "heading",
|
||||
"h3": "heading",
|
||||
"h4": "heading",
|
||||
"h5": "heading",
|
||||
"h6": "heading",
|
||||
"p": "p",
|
||||
"ul": "list",
|
||||
"ol": "list",
|
||||
"dl": "list",
|
||||
"blockquote": "quote",
|
||||
"pre": "pre",
|
||||
"table": "table",
|
||||
"figure": "figure",
|
||||
}
|
||||
_INLINE_TAGS = frozenset((
|
||||
"a", "abbr", "b", "bdi", "br", "cite", "code", "data", "del", "dfn", "em",
|
||||
"font", "i", "img", "ins", "kbd", "label", "mark", "q", "s", "samp", "small",
|
||||
"span", "strong", "sub", "sup", "time", "u", "var", "wbr",
|
||||
))
|
||||
_LAYOUT_CELL_CHILDREN = (*_BLOCK_KINDS, "div")
|
||||
_SKIPPED_TAGS = frozenset(("head",))
|
||||
_DROP_ANCESTORS = frozenset(("nav", "button", "select"))
|
||||
_CHROME_TAGS = frozenset(("aside", "form", "dialog"))
|
||||
_CHROME_ROLES = frozenset(("navigation", "banner", "contentinfo", "complementary"))
|
||||
_HIDDEN_STYLE = re.compile(r"display\s*:\s*none|visibility\s*:\s*hidden", re.IGNORECASE)
|
||||
_SHORT_KINDS = frozenset(("p", "loose"))
|
||||
_ALL_REVIEW = "no block passed the content checks; every block is marked for review"
|
||||
|
||||
|
||||
@dataclasses.dataclass
|
||||
class Block:
|
||||
"""One swept block: an element, or a run of loose text and inline nodes.
|
||||
|
||||
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6.1 and 7.1.5.4.5.6.2.
|
||||
"""
|
||||
|
||||
nodes: list[bs4.PageElement]
|
||||
kind: str
|
||||
text: str = ""
|
||||
link_chars: int = 0
|
||||
words: int = 0
|
||||
verdict: str = ""
|
||||
reason: str = ""
|
||||
|
||||
|
||||
@dataclasses.dataclass
|
||||
class SweepResult:
|
||||
"""Markdown body, warnings, and whether every block needs review.
|
||||
|
||||
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6.6 and 7.1.5.4.5.6.7.
|
||||
"""
|
||||
|
||||
body: str
|
||||
warnings: list[str]
|
||||
partial: bool
|
||||
|
||||
|
||||
def unwrap_shadow_roots(soup: bs4.BeautifulSoup) -> None:
|
||||
"""Read each declarative shadow root in place, so its text is swept.
|
||||
|
||||
Generated by AI/LLM: 2026-10-07 - ``template[shadowrootmode]``; other templates are dropped later.
|
||||
|
||||
Args:
|
||||
soup: Parsed document, before the drop loop.
|
||||
"""
|
||||
for template in soup.find_all("template", attrs={"shadowrootmode": True}):
|
||||
template.unwrap()
|
||||
|
||||
|
||||
def sweep(root: bs4.element.Tag, *, min_words: int, max_link_density: float) -> SweepResult:
|
||||
"""Walk ``root``, classify each block, and render the kept and review blocks.
|
||||
|
||||
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6. No clock, network, or random source.
|
||||
|
||||
Args:
|
||||
root: ``<body>``, or the document when there is no body.
|
||||
min_words: ``fiona.wiki.ingest.html.min_words``.
|
||||
max_link_density: ``fiona.wiki.ingest.html.max_link_density``.
|
||||
|
||||
Returns:
|
||||
SweepResult: Markdown body, the ``html sweep:`` warning, and the partial flag.
|
||||
"""
|
||||
blocks: list[Block] = []
|
||||
_walk(root, blocks)
|
||||
seen: set[str] = set()
|
||||
for block in blocks:
|
||||
_measure(block)
|
||||
block.verdict, block.reason = _classify(block, seen, min_words, max_link_density)
|
||||
if block.verdict != "drop":
|
||||
seen.add(block.text.casefold())
|
||||
counts = {verdict: sum(block.verdict == verdict for block in blocks) for verdict in ("keep", "review", "drop")}
|
||||
warnings = [f"html sweep: {counts['keep']} keep, {counts['review']} review, {counts['drop']} drop"]
|
||||
partial = counts["keep"] == 0 and counts["review"] > 0
|
||||
if partial:
|
||||
warnings.append(_ALL_REVIEW)
|
||||
parts = [part for part in (_render(block) for block in blocks if block.verdict != "drop") if part]
|
||||
body = "\n\n".join(parts)
|
||||
return SweepResult(body=body + "\n" if body else "", warnings=warnings, partial=partial)
|
||||
|
||||
|
||||
def _walk(container: bs4.element.Tag, blocks: list[Block]) -> None:
|
||||
"""Append the blocks of ``container`` in document order.
|
||||
|
||||
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6.1 to 7.1.5.4.5.6.1.2.
|
||||
"""
|
||||
run: list[bs4.PageElement] = []
|
||||
for child in list(container.children):
|
||||
if not isinstance(child, bs4.element.Tag):
|
||||
if isinstance(child, bs4.element.NavigableString):
|
||||
run.append(child)
|
||||
continue
|
||||
if child.name in _SKIPPED_TAGS:
|
||||
continue
|
||||
if _is_block(child):
|
||||
_flush(run, blocks)
|
||||
blocks.append(Block(nodes=[child], kind=_BLOCK_KINDS[child.name]))
|
||||
elif child.name in _INLINE_TAGS and not _holds_structure(child):
|
||||
run.append(child)
|
||||
else:
|
||||
_flush(run, blocks)
|
||||
_walk(child, blocks)
|
||||
_flush(run, blocks)
|
||||
|
||||
|
||||
def _flush(run: list[bs4.PageElement], blocks: list[Block]) -> None:
|
||||
"""Close a loose-text run as one block, unless it holds no text.
|
||||
|
||||
Generated by AI/LLM: 2026-10-07 - whitespace between tags is not a block.
|
||||
"""
|
||||
if run and _fold(_raw_text(run)):
|
||||
blocks.append(Block(nodes=list(run), kind="loose"))
|
||||
run.clear()
|
||||
|
||||
|
||||
def _is_block(tag: bs4.element.Tag) -> bool:
|
||||
"""Return whether ``tag`` is a block. A layout table is a container instead.
|
||||
|
||||
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6.1 layout tables.
|
||||
"""
|
||||
if tag.name not in _BLOCK_KINDS:
|
||||
return False
|
||||
if tag.name != "table":
|
||||
return True
|
||||
return not any(cell.find(_LAYOUT_CELL_CHILDREN) for cell in tag.find_all(("td", "th")))
|
||||
|
||||
|
||||
def _holds_structure(tag: bs4.element.Tag) -> bool:
|
||||
"""Return whether an inline tag wraps a block or container, like a card link.
|
||||
|
||||
Generated by AI/LLM: 2026-10-07 - ``<a><div>...</div></a>`` is walked as a container.
|
||||
"""
|
||||
return tag.find(lambda node: node.name not in _INLINE_TAGS) is not None
|
||||
|
||||
|
||||
def _raw_text(nodes: typing.Iterable[bs4.PageElement]) -> str:
|
||||
"""Return the text of ``nodes`` with no separator and ``<br>`` as a space.
|
||||
|
||||
Generated by AI/LLM: 2026-10-07 - keeps ``<b>Fa</b>st`` as one word.
|
||||
"""
|
||||
parts: list[str] = []
|
||||
for node in nodes:
|
||||
items = [node] if isinstance(node, bs4.element.NavigableString) else [node, *node.descendants]
|
||||
for item in items:
|
||||
if isinstance(item, bs4.element.NavigableString):
|
||||
parts.append(str(item))
|
||||
elif item.name == "br":
|
||||
parts.append(" ")
|
||||
return "".join(parts)
|
||||
|
||||
|
||||
def _fold(text: str) -> str:
|
||||
"""Fold whitespace runs to one space and trim.
|
||||
|
||||
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6.2.
|
||||
"""
|
||||
return " ".join(text.split())
|
||||
|
||||
|
||||
def _measure(block: Block) -> None:
|
||||
"""Set the folded text, link text length, and word count of a block.
|
||||
|
||||
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6.2.
|
||||
"""
|
||||
block.text = _fold(_raw_text(block.nodes))
|
||||
links: list[bs4.element.Tag] = []
|
||||
for node in block.nodes:
|
||||
if isinstance(node, bs4.element.Tag):
|
||||
links.extend([node] if node.name == "a" else node.find_all("a"))
|
||||
block.link_chars = sum(len(_fold(_raw_text([link]))) for link in links)
|
||||
block.words = len(block.text.split())
|
||||
|
||||
|
||||
def _classify(block: Block, seen: set[str], min_words: int, max_link_density: float) -> tuple[str, str]:
|
||||
"""Return ``(verdict, reason)`` with the first rule that holds.
|
||||
|
||||
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6.3 to 7.1.5.4.5.6.5.
|
||||
"""
|
||||
folded = block.text.casefold()
|
||||
if not any(char.isalnum() for char in block.text):
|
||||
return "drop", ""
|
||||
if block.words < min_words and folded in seen:
|
||||
return "drop", ""
|
||||
if _inside(block, _drops):
|
||||
return "drop", ""
|
||||
if _inside(block, _is_chrome):
|
||||
return "review", "chrome"
|
||||
if folded in seen:
|
||||
return "review", "repeat"
|
||||
if block.link_chars / len(block.text) > max_link_density:
|
||||
return "review", "links"
|
||||
if block.kind in _SHORT_KINDS and block.words < min_words:
|
||||
return "review", "short"
|
||||
return "keep", ""
|
||||
|
||||
|
||||
def _inside(block: Block, predicate: typing.Callable[[bs4.element.Tag], bool]) -> bool:
|
||||
"""Return whether the block, or an ancestor below the document, matches.
|
||||
|
||||
Generated by AI/LLM: 2026-10-07 - an element block checks itself; a loose run starts at its parent.
|
||||
"""
|
||||
first = block.nodes[0]
|
||||
node = first.parent if block.kind == "loose" else first
|
||||
while isinstance(node, bs4.element.Tag) and not isinstance(node, bs4.BeautifulSoup):
|
||||
if predicate(node):
|
||||
return True
|
||||
node = node.parent
|
||||
return False
|
||||
|
||||
|
||||
def _drops(tag: bs4.element.Tag) -> bool:
|
||||
"""Return whether text under ``tag`` is never content: nav, controls, hidden.
|
||||
|
||||
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6.3.
|
||||
"""
|
||||
if tag.name in _DROP_ANCESTORS or tag.has_attr("hidden"):
|
||||
return True
|
||||
if str(tag.get("aria-hidden", "")).strip().lower() == "true":
|
||||
return True
|
||||
return bool(_HIDDEN_STYLE.search(str(tag.get("style", ""))))
|
||||
|
||||
|
||||
def _is_chrome(tag: bs4.element.Tag) -> bool:
|
||||
"""Return whether ``tag`` is page chrome: aside, form, landmark role, outer header or footer.
|
||||
|
||||
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6.4 ``chrome``.
|
||||
"""
|
||||
if tag.name in _CHROME_TAGS:
|
||||
return True
|
||||
if str(tag.get("role", "")).strip().lower() in _CHROME_ROLES:
|
||||
return True
|
||||
if tag.name in ("header", "footer"):
|
||||
return tag.find_parent(("article", "main")) is None
|
||||
return False
|
||||
|
||||
|
||||
def _render(block: Block) -> str:
|
||||
"""Convert one block to markdown and wrap a ``review`` block in its markers.
|
||||
|
||||
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.4.5.6.6, options of 7.1.5.4.5.4.
|
||||
"""
|
||||
markup = "".join(
|
||||
node.output_ready(formatter="minimal") if isinstance(node, bs4.element.NavigableString) else str(node)
|
||||
for node in block.nodes
|
||||
)
|
||||
heading = getattr(markdownify, "ATX", "ATX")
|
||||
text = markdownify.markdownify(markup, heading_style=heading, bullets="-").strip()
|
||||
if not text or block.verdict != "review":
|
||||
return text
|
||||
return f"<!-- fiona:review {block.reason} -->\n\n{text}\n\n<!-- /fiona:review -->"
|
||||
@@ -1,105 +0,0 @@
|
||||
"""Markdown source converter.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - keep the body and lift frontmatter into source_meta.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import typing
|
||||
|
||||
import yaml
|
||||
|
||||
from fiona.wiki.ingest.converters.base import SourceConverter
|
||||
from fiona.wiki.ingest.media import decode_bytes
|
||||
from fiona.wiki.ingest.models import ConversionResult, FetchedSource
|
||||
|
||||
_FRONTMATTER = re.compile(r"\A---\r?\n(.*?)\r?\n---\r?\n?", re.DOTALL)
|
||||
_H1 = re.compile(r"^#\s+(\S.*)$")
|
||||
_FENCE = re.compile(r"^\s*(`{3,}|~{3,})")
|
||||
|
||||
|
||||
class MarkdownConverter(SourceConverter):
|
||||
"""Keep markdown as it is and record any source frontmatter.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.4.3.
|
||||
|
||||
The body after a frontmatter block is kept character for character.
|
||||
The page writer adds the wiki frontmatter when the new page does not
|
||||
have it yet. Source keys move to ``source_meta`` instead of being
|
||||
written a second time as the page frontmatter.
|
||||
"""
|
||||
|
||||
name: typing.ClassVar[str] = "markdown"
|
||||
media_types: typing.ClassVar[tuple[str, ...]] = ("text/markdown",)
|
||||
extensions: typing.ClassVar[tuple[str, ...]] = (".md", ".markdown")
|
||||
raw_category: typing.ClassVar[str] = "articles"
|
||||
|
||||
def convert(self, source: FetchedSource) -> ConversionResult:
|
||||
"""Return the markdown body and any frontmatter as metadata.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - title from frontmatter, then H1, then slug.
|
||||
|
||||
Args:
|
||||
source: Archived markdown file.
|
||||
|
||||
Returns:
|
||||
ConversionResult: Unchanged body, ``complete`` unless decoding fell back.
|
||||
"""
|
||||
text, warnings = decode_bytes(source.path.read_bytes(), source.charset)
|
||||
meta: dict = {}
|
||||
body = text
|
||||
match = _FRONTMATTER.match(text)
|
||||
if match:
|
||||
loaded = _load_meta(match.group(1))
|
||||
if loaded is None:
|
||||
warnings = [*warnings, "source frontmatter did not parse"]
|
||||
else:
|
||||
meta = loaded
|
||||
body = text[match.end():]
|
||||
title = _title(meta, body, source.slug)
|
||||
return ConversionResult(
|
||||
body=body,
|
||||
title=title,
|
||||
source_meta=meta,
|
||||
conversion="complete",
|
||||
warnings=warnings,
|
||||
converter=self.name,
|
||||
)
|
||||
|
||||
|
||||
def _load_meta(block: str) -> dict | None:
|
||||
"""Parse a frontmatter block into a dict.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - invalid YAML stays in the body.
|
||||
"""
|
||||
try:
|
||||
loaded = yaml.safe_load(block)
|
||||
except yaml.YAMLError:
|
||||
return None
|
||||
if loaded is None:
|
||||
return {}
|
||||
if isinstance(loaded, dict):
|
||||
return loaded
|
||||
return None
|
||||
|
||||
|
||||
def _title(meta: dict, body: str, slug: str) -> str:
|
||||
"""Choose the page title.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - frontmatter title, first H1, then slug.
|
||||
"""
|
||||
raw_title = meta.get("title")
|
||||
if isinstance(raw_title, str) and raw_title.strip():
|
||||
return raw_title.strip()
|
||||
fenced = False
|
||||
for line in body.splitlines():
|
||||
if _FENCE.match(line):
|
||||
fenced = not fenced
|
||||
continue
|
||||
if fenced:
|
||||
continue
|
||||
match = _H1.match(line)
|
||||
if match:
|
||||
return match.group(1).strip()
|
||||
return slug
|
||||
@@ -1,186 +0,0 @@
|
||||
"""PDF source converter.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - text layer via pypdf, SPEC 7.1.5.4.6.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime
|
||||
import typing
|
||||
|
||||
import pypdf
|
||||
import pypdf.errors
|
||||
|
||||
from fiona.wiki.ingest.converters.base import ConversionError, SourceConverter
|
||||
from fiona.wiki.ingest.models import ConversionResult, FetchedSource
|
||||
|
||||
_SCANNED = "no text layer; the PDF may be scanned"
|
||||
|
||||
|
||||
class PdfConverter(SourceConverter):
|
||||
"""Read a PDF text layer into markdown headings and paragraphs.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.4.6.
|
||||
"""
|
||||
|
||||
name: typing.ClassVar[str] = "pdf"
|
||||
media_types: typing.ClassVar[tuple[str, ...]] = ("application/pdf",)
|
||||
extensions: typing.ClassVar[tuple[str, ...]] = (".pdf",)
|
||||
raw_category: typing.ClassVar[str] = "papers"
|
||||
|
||||
def convert(self, source: FetchedSource) -> ConversionResult:
|
||||
"""Extract page text, outline headings, and document metadata.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - encrypted or corrupt files raise.
|
||||
|
||||
Args:
|
||||
source: Archived PDF.
|
||||
|
||||
Returns:
|
||||
ConversionResult: Markdown body. Empty pages make it ``partial``.
|
||||
|
||||
Raises:
|
||||
ConversionError: The PDF is encrypted or cannot be parsed.
|
||||
"""
|
||||
try:
|
||||
reader = pypdf.PdfReader(str(source.path))
|
||||
except pypdf.errors.PdfReadError as exc:
|
||||
raise ConversionError(self.name, str(exc)) from exc
|
||||
if reader.is_encrypted:
|
||||
raise ConversionError(self.name, "encrypted PDF")
|
||||
meta = _metadata(reader)
|
||||
title = meta.get("title") or source.slug
|
||||
outline = _outline(reader)
|
||||
pages = []
|
||||
empty: list[int] = []
|
||||
for index, page in enumerate(reader.pages, start=1):
|
||||
try:
|
||||
text = page.extract_text() or ""
|
||||
except pypdf.errors.PdfReadError as exc:
|
||||
raise ConversionError(self.name, str(exc)) from exc
|
||||
pages.append(text.strip())
|
||||
if not text.strip():
|
||||
empty.append(index)
|
||||
body = _render(pages, outline)
|
||||
warnings: list[str] = []
|
||||
conversion = "complete"
|
||||
if empty and len(empty) == len(pages):
|
||||
warnings.append(_SCANNED)
|
||||
conversion = "partial"
|
||||
elif empty:
|
||||
warnings.extend(f"page {number} has no text layer" for number in empty)
|
||||
conversion = "partial"
|
||||
return ConversionResult(
|
||||
body=body,
|
||||
title=str(title),
|
||||
source_meta=meta,
|
||||
conversion=conversion,
|
||||
warnings=warnings,
|
||||
converter=self.name,
|
||||
)
|
||||
|
||||
|
||||
def _metadata(reader: pypdf.PdfReader) -> dict:
|
||||
"""Return page count and the document info values that are present.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - dates as ISO 8601.
|
||||
"""
|
||||
meta: dict[str, object] = {"page_count": len(reader.pages)}
|
||||
info = reader.metadata
|
||||
if info is None:
|
||||
return meta
|
||||
mapping = (
|
||||
("title", info.title),
|
||||
("author", info.author),
|
||||
("subject", info.subject),
|
||||
("keywords", info.keywords),
|
||||
("creation_date", _iso_date(info.creation_date)),
|
||||
("mod_date", _iso_date(info.modification_date)),
|
||||
)
|
||||
for key, value in mapping:
|
||||
if value:
|
||||
meta[key] = value
|
||||
return meta
|
||||
|
||||
|
||||
def _iso_date(value: object) -> str | None:
|
||||
"""Format a PDF date as ISO 8601.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - accept datetime values and ``D:`` strings.
|
||||
"""
|
||||
if value is None:
|
||||
return None
|
||||
if isinstance(value, datetime.datetime):
|
||||
if value.tzinfo is None:
|
||||
return value.strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
return value.astimezone(datetime.UTC).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
text = str(value).strip().removeprefix("D:")
|
||||
digits = "".join(char for char in text if char.isdigit())
|
||||
if len(digits) < 8:
|
||||
return text or None
|
||||
year, month, day = digits[0:4], digits[4:6], digits[6:8]
|
||||
hour = digits[8:10] or "00"
|
||||
minute = digits[10:12] or "00"
|
||||
second = digits[12:14] or "00"
|
||||
return f"{year}-{month}-{day}T{hour}:{minute}:{second}Z"
|
||||
|
||||
|
||||
def _outline(reader: pypdf.PdfReader) -> list[tuple[int, int, str]]:
|
||||
"""Return outline entries as page index, heading level, and title.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - level is depth plus 2, capped at 6.
|
||||
"""
|
||||
found: list[tuple[int, int, str]] = []
|
||||
try:
|
||||
items = reader.outline
|
||||
except pypdf.errors.PdfReadError:
|
||||
return found
|
||||
_walk_outline(reader, items, 0, found)
|
||||
return found
|
||||
|
||||
|
||||
def _walk_outline(
|
||||
reader: pypdf.PdfReader,
|
||||
items: object,
|
||||
depth: int,
|
||||
found: list[tuple[int, int, str]],
|
||||
) -> None:
|
||||
"""Walk a pypdf outline tree.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - nested lists are deeper entries.
|
||||
"""
|
||||
if not items:
|
||||
return
|
||||
for item in items: # type: ignore[union-attr]
|
||||
if isinstance(item, list):
|
||||
_walk_outline(reader, item, depth + 1, found)
|
||||
continue
|
||||
try:
|
||||
page_index = reader.get_destination_page_number(item)
|
||||
except (ValueError, KeyError, pypdf.errors.PdfReadError):
|
||||
continue
|
||||
title = getattr(item, "title", "") or ""
|
||||
level = min(depth + 2, 6)
|
||||
found.append((page_index, level, str(title).strip() or "Untitled"))
|
||||
|
||||
|
||||
def _render(pages: list[str], outline: list[tuple[int, int, str]]) -> str:
|
||||
"""Render pages with outline headings or ``## Page n`` headings.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.4.6.2.
|
||||
"""
|
||||
by_page: dict[int, list[str]] = {}
|
||||
for page_index, level, title in outline:
|
||||
by_page.setdefault(page_index, []).append(f"{'#' * level} {title}")
|
||||
chunks: list[str] = []
|
||||
for index, text in enumerate(pages):
|
||||
if outline:
|
||||
chunks.extend(by_page.get(index, []))
|
||||
else:
|
||||
chunks.append(f"## Page {index + 1}")
|
||||
if text:
|
||||
chunks.append(text)
|
||||
body = "\n\n".join(chunks)
|
||||
if body:
|
||||
body += "\n"
|
||||
return body
|
||||
@@ -1,68 +0,0 @@
|
||||
"""Plain-text source converter.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - wrap text in a fence; the agent refines the page.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import typing
|
||||
|
||||
from fiona.wiki.ingest.converters.base import SourceConverter
|
||||
from fiona.wiki.ingest.media import decode_bytes
|
||||
from fiona.wiki.ingest.models import ConversionResult, FetchedSource
|
||||
|
||||
|
||||
class TextConverter(SourceConverter):
|
||||
"""Turn plain text into one fenced markdown block.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.4.4.
|
||||
|
||||
The page is written at ``status: raw``. The tool does not summarize
|
||||
the text. The agent rewrites the fence during refinement.
|
||||
"""
|
||||
|
||||
name: typing.ClassVar[str] = "text"
|
||||
media_types: typing.ClassVar[tuple[str, ...]] = ("text/plain",)
|
||||
extensions: typing.ClassVar[tuple[str, ...]] = (".txt", ".text", ".log")
|
||||
raw_category: typing.ClassVar[str] = "articles"
|
||||
|
||||
def convert(self, source: FetchedSource) -> ConversionResult:
|
||||
"""Decode the file and wrap it in a fence longer than any backtick run.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - title is the slug; fallback decode warns.
|
||||
|
||||
Args:
|
||||
source: Archived text file.
|
||||
|
||||
Returns:
|
||||
ConversionResult: Fenced body and the slug as the title.
|
||||
"""
|
||||
text, warnings = decode_bytes(source.path.read_bytes(), source.charset)
|
||||
marker = "`" * _fence_length(text)
|
||||
if text and not text.endswith("\n"):
|
||||
text += "\n"
|
||||
body = f"{marker}text\n{text}{marker}\n"
|
||||
return ConversionResult(
|
||||
body=body,
|
||||
title=source.slug,
|
||||
source_meta={},
|
||||
conversion="complete",
|
||||
warnings=warnings,
|
||||
converter=self.name,
|
||||
)
|
||||
|
||||
|
||||
def _fence_length(text: str) -> int:
|
||||
"""Return a fence length longer than the longest backtick run.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - a run of three when the text has fewer.
|
||||
"""
|
||||
longest = 0
|
||||
current = 0
|
||||
for char in text:
|
||||
if char == "`":
|
||||
current += 1
|
||||
longest = max(longest, current)
|
||||
else:
|
||||
current = 0
|
||||
return max(3, longest + 1)
|
||||
@@ -1,316 +0,0 @@
|
||||
"""Fetch a local file or an HTTP URL.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.2. No credentials are sent.
|
||||
Modified by AI/LLM: 2026-10-07 - login detection, SPEC 7.1.5.2.8.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import http
|
||||
import logging
|
||||
import pathlib
|
||||
import re
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
|
||||
from fiona.wiki.ingest.converters.base import ConverterRegistry
|
||||
from fiona.wiki.ingest.media import (
|
||||
detect_media_type,
|
||||
extension_for,
|
||||
slug_from_file,
|
||||
slug_from_url,
|
||||
utc_now,
|
||||
)
|
||||
from fiona.wiki.ingest.models import AuthRequiredError, FetchedSource, IngestError
|
||||
from fiona.wiki.ingest.settings import IngestSettings, default_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_SCHEME = re.compile(r"^([A-Za-z][A-Za-z0-9+.-]*)://")
|
||||
_WINDOWS_DRIVE = re.compile(r"^[A-Za-z]:[\\/]")
|
||||
_AUTH_STATUSES = (401, 403)
|
||||
_PASSWORD_INPUT = re.compile(rb"<input\b[^>]*?\btype\s*=\s*[\"']?password\b", re.IGNORECASE)
|
||||
|
||||
|
||||
def url_scheme(argument: str) -> str | None:
|
||||
"""Return the URL scheme, or None for a local path.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - a Windows drive letter is not a scheme.
|
||||
|
||||
Args:
|
||||
argument: The ``<source>`` argument as given.
|
||||
|
||||
Returns:
|
||||
str | None: Scheme text, preserving case, or None.
|
||||
"""
|
||||
if _WINDOWS_DRIVE.match(argument):
|
||||
return None
|
||||
match = _SCHEME.match(argument)
|
||||
if match is None:
|
||||
return None
|
||||
return match.group(1)
|
||||
|
||||
|
||||
class SourceFetcher:
|
||||
"""Base fetcher. Subclasses accept an argument and return a local file.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.1.3.
|
||||
"""
|
||||
|
||||
settings: IngestSettings | None = None
|
||||
|
||||
def accepts(self, argument: str) -> bool:
|
||||
"""Return whether this fetcher handles the argument.
|
||||
|
||||
Args:
|
||||
argument: The ``<source>`` argument as given.
|
||||
|
||||
Returns:
|
||||
bool: True when ``fetch`` can run.
|
||||
"""
|
||||
raise NotImplementedError
|
||||
|
||||
def fetch(self, argument: str, dest_dir: pathlib.Path) -> FetchedSource:
|
||||
"""Fetch the argument into ``dest_dir`` when a download is required.
|
||||
|
||||
Args:
|
||||
argument: The ``<source>`` argument as given.
|
||||
dest_dir: Directory for a downloaded file. Local fetches ignore it.
|
||||
|
||||
Returns:
|
||||
FetchedSource: Local path and origin facts.
|
||||
"""
|
||||
raise NotImplementedError
|
||||
|
||||
|
||||
class LocalFileFetcher(SourceFetcher):
|
||||
"""Read a local file in place.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - missing files and directories exit 1.
|
||||
"""
|
||||
|
||||
def __init__(self, registry: ConverterRegistry) -> None:
|
||||
"""Store the registry used to name the media type.
|
||||
|
||||
Args:
|
||||
registry: Converter registry for extension lookup.
|
||||
"""
|
||||
self.registry = registry
|
||||
self.settings = None
|
||||
|
||||
def accepts(self, argument: str) -> bool:
|
||||
"""Accept any argument that is not a URL.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - scheme detection stays in ``url_scheme``.
|
||||
|
||||
Args:
|
||||
argument: The ``<source>`` argument as given.
|
||||
|
||||
Returns:
|
||||
bool: True when there is no ``://`` scheme.
|
||||
"""
|
||||
return url_scheme(argument) is None
|
||||
|
||||
def fetch(self, argument: str, dest_dir: pathlib.Path) -> FetchedSource:
|
||||
"""Resolve a local path without copying it.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.2.1.
|
||||
|
||||
Args:
|
||||
argument: Local path as given.
|
||||
dest_dir: Unused. Present so every fetcher shares one signature.
|
||||
|
||||
Returns:
|
||||
FetchedSource: The resolved file. ``created`` is false.
|
||||
|
||||
Raises:
|
||||
IngestError: The path is missing or is a directory.
|
||||
"""
|
||||
del dest_dir
|
||||
path = pathlib.Path(argument)
|
||||
if not path.exists():
|
||||
raise IngestError(f"Source not found: {argument}")
|
||||
if path.is_dir():
|
||||
raise IngestError(f"Source is a directory: {argument}")
|
||||
data = path.read_bytes()
|
||||
media, charset = detect_media_type(data, path.suffix, None, self.registry)
|
||||
resolved = path.resolve()
|
||||
logger.info("fetched local file %s as %s", resolved, media)
|
||||
return FetchedSource(
|
||||
argument=argument,
|
||||
path=resolved,
|
||||
origin=str(resolved),
|
||||
origin_kind="file",
|
||||
final_url=None,
|
||||
media_type=media,
|
||||
charset=charset,
|
||||
size_bytes=len(data),
|
||||
slug=slug_from_file(path),
|
||||
fetched=utc_now(),
|
||||
created=False,
|
||||
)
|
||||
|
||||
|
||||
class UrlFetcher(SourceFetcher):
|
||||
"""Download one ``http`` or ``https`` URL.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - opener is injected; tests make no network call.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
registry: ConverterRegistry,
|
||||
opener: urllib.request.OpenerDirector | None = None,
|
||||
) -> None:
|
||||
"""Store the registry and the URL opener.
|
||||
|
||||
Args:
|
||||
registry: Converter registry for the saved extension.
|
||||
opener: ``urllib`` opener. The default follows redirects and proxies.
|
||||
"""
|
||||
self.registry = registry
|
||||
self.opener = opener or urllib.request.build_opener()
|
||||
self.settings = None
|
||||
|
||||
def accepts(self, argument: str) -> bool:
|
||||
"""Accept ``http`` and ``https`` only.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - other schemes are rejected by the pipeline.
|
||||
|
||||
Args:
|
||||
argument: The ``<source>`` argument as given.
|
||||
|
||||
Returns:
|
||||
bool: True for http and https.
|
||||
"""
|
||||
scheme = url_scheme(argument)
|
||||
return scheme is not None and scheme.lower() in {"http", "https"}
|
||||
|
||||
def fetch(self, argument: str, dest_dir: pathlib.Path) -> FetchedSource:
|
||||
"""Download a URL into ``dest_dir`` and stop past the size limit.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.2.2 and 7.1.5.2.3.
|
||||
|
||||
Args:
|
||||
argument: Absolute URL.
|
||||
dest_dir: Directory that receives ``<slug>.<ext>``.
|
||||
|
||||
Returns:
|
||||
FetchedSource: The saved file. ``created`` is true.
|
||||
|
||||
Raises:
|
||||
IngestError: The response is not 2xx, the network fails, or the body is too large.
|
||||
"""
|
||||
settings = self.settings or default_settings()
|
||||
requested = without_userinfo(argument)
|
||||
request = urllib.request.Request(
|
||||
requested,
|
||||
headers={"User-Agent": settings.user_agent},
|
||||
)
|
||||
try:
|
||||
response = self.opener.open(request, timeout=settings.timeout_seconds)
|
||||
except urllib.error.HTTPError as exc:
|
||||
if exc.code in _AUTH_STATUSES:
|
||||
phrase = http.HTTPStatus(exc.code).phrase
|
||||
raise AuthRequiredError(requested, f"{exc.code} {phrase}") from exc
|
||||
raise IngestError(f"Download failed: {argument}: {exc.code} {exc.reason}") from exc
|
||||
except urllib.error.URLError as exc:
|
||||
raise IngestError(f"Download failed: {argument}: {exc.reason}") from exc
|
||||
with response:
|
||||
final_url = response.geturl()
|
||||
header = response.headers.get("Content-Type")
|
||||
body = _read_limited(response, settings.max_download_bytes, argument)
|
||||
parsed = urllib.parse.urlsplit(argument)
|
||||
suffix = pathlib.PurePosixPath(parsed.path).suffix
|
||||
media, charset = detect_media_type(body, suffix, header, self.registry)
|
||||
_check_login(requested, final_url, media, body, settings.login_url_patterns)
|
||||
extension = extension_for(media, suffix, self.registry)
|
||||
slug = slug_from_url(argument)
|
||||
dest_dir.mkdir(parents=True, exist_ok=True)
|
||||
target = dest_dir / f"{slug}{extension}"
|
||||
partial = target.with_name(target.name + ".partial")
|
||||
try:
|
||||
partial.write_bytes(body)
|
||||
partial.replace(target)
|
||||
except OSError:
|
||||
if partial.exists():
|
||||
partial.unlink()
|
||||
raise
|
||||
logger.info("downloaded %s as %s (%s bytes)", argument, media, len(body))
|
||||
return FetchedSource(
|
||||
argument=argument,
|
||||
path=target,
|
||||
origin=argument,
|
||||
origin_kind="url",
|
||||
final_url=final_url,
|
||||
media_type=media,
|
||||
charset=charset,
|
||||
size_bytes=len(body),
|
||||
slug=slug,
|
||||
fetched=utc_now(),
|
||||
created=True,
|
||||
)
|
||||
|
||||
|
||||
def _check_login(
|
||||
requested: str,
|
||||
final_url: str,
|
||||
media: str,
|
||||
body: bytes,
|
||||
patterns: list[str],
|
||||
) -> None:
|
||||
"""Raise when a 2xx answer is a login redirect or a login form.
|
||||
|
||||
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.2.8.2 redirect, then login form.
|
||||
|
||||
Args:
|
||||
requested: URL that was requested, without userinfo.
|
||||
final_url: URL after redirects.
|
||||
media: Detected media type of the body.
|
||||
body: Response body, already within the size limit.
|
||||
patterns: ``login_url_patterns`` from the settings.
|
||||
|
||||
Raises:
|
||||
AuthRequiredError: The answer is a login page, not the source.
|
||||
"""
|
||||
if final_url and final_url != requested:
|
||||
parts = urllib.parse.urlsplit(final_url)
|
||||
target = f"{parts.hostname or ''}{parts.path}"
|
||||
if any(re.search(pattern, target, re.IGNORECASE) for pattern in patterns):
|
||||
raise AuthRequiredError(requested, f"redirected to {final_url}")
|
||||
if media == "text/html" and _PASSWORD_INPUT.search(body):
|
||||
raise AuthRequiredError(requested, f"login page at {final_url or requested}")
|
||||
|
||||
|
||||
def without_userinfo(url: str) -> str:
|
||||
"""Drop userinfo so the request sends no credentials.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.2.2.
|
||||
Modified by AI/LLM: 2026-10-07 - public; also used for ``--origin``.
|
||||
"""
|
||||
parts = urllib.parse.urlsplit(url)
|
||||
if parts.username is None and parts.password is None:
|
||||
return url
|
||||
host = parts.hostname or ""
|
||||
if parts.port:
|
||||
host = f"{host}:{parts.port}"
|
||||
return urllib.parse.urlunsplit((parts.scheme, host, parts.path, parts.query, parts.fragment))
|
||||
|
||||
|
||||
def _read_limited(response: object, limit: int, url: str) -> bytes:
|
||||
"""Read a response body, refusing anything over ``limit``.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - a partial download is not returned.
|
||||
"""
|
||||
chunks: list[bytes] = []
|
||||
total = 0
|
||||
while True:
|
||||
block = response.read(65536) # type: ignore[attr-defined]
|
||||
if not block:
|
||||
break
|
||||
total += len(block)
|
||||
if total > limit:
|
||||
raise IngestError(f"Download exceeds {limit} bytes: {url}")
|
||||
chunks.append(block)
|
||||
return b"".join(chunks)
|
||||
@@ -1,96 +0,0 @@
|
||||
"""Append one ingest line to ``wiki/log.md``.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.6. Does not edit ``index.md``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import pathlib
|
||||
import re
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_VERSION = re.compile(r"^version:\s*(\S+)\s*$", re.MULTILINE)
|
||||
_UPDATED = re.compile(r"^updated:\s*(\S+)\s*$", re.MULTILINE)
|
||||
|
||||
|
||||
class IngestLog:
|
||||
"""Record a finished archive in ``wiki/log.md``.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - bump patch version and ``updated``.
|
||||
"""
|
||||
|
||||
def __init__(self, wiki_root: pathlib.Path) -> None:
|
||||
"""Store the wiki root.
|
||||
|
||||
Args:
|
||||
wiki_root: Directory that contains ``wiki/log.md``.
|
||||
"""
|
||||
self.wiki_root = wiki_root
|
||||
|
||||
def append(
|
||||
self,
|
||||
*,
|
||||
day: str,
|
||||
origin: str,
|
||||
raw_rel: str,
|
||||
page_rel: str | None,
|
||||
dry_run: bool,
|
||||
) -> None:
|
||||
"""Append one bullet and bump the log frontmatter.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - ``(raw)`` or ``(no page)``.
|
||||
|
||||
Args:
|
||||
day: UTC date ``YYYY-MM-DD``.
|
||||
origin: URL, or file path (wiki-relative inside the wiki).
|
||||
raw_rel: Raw path relative to the wiki root.
|
||||
page_rel: Page path relative to the wiki root, or None.
|
||||
dry_run: When true, leave the log untouched.
|
||||
"""
|
||||
if dry_run:
|
||||
return
|
||||
path = self.wiki_root / "wiki" / "log.md"
|
||||
if page_rel:
|
||||
bullet = f"- {day} - ingest {origin} -> {raw_rel} -> {page_rel} (raw)"
|
||||
else:
|
||||
bullet = f"- {day} - ingest {origin} -> {raw_rel} (no page)"
|
||||
if not path.is_file():
|
||||
logger.warning("wiki/log.md is missing; ingest was not recorded")
|
||||
return
|
||||
text = path.read_text(encoding="utf-8")
|
||||
text = _bump_version(text)
|
||||
text = _set_updated(text, day)
|
||||
if not text.endswith("\n"):
|
||||
text += "\n"
|
||||
text += bullet + "\n"
|
||||
path.write_text(text, encoding="utf-8", newline="\n")
|
||||
logger.info("recorded %s", bullet)
|
||||
|
||||
|
||||
def _bump_version(text: str) -> str:
|
||||
"""Increment the patch number of the first ``version`` field.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - ``2.0.0`` becomes ``2.0.1``.
|
||||
"""
|
||||
match = _VERSION.search(text)
|
||||
if match is None:
|
||||
return text
|
||||
parts = match.group(1).split(".")
|
||||
if len(parts) == 3 and all(part.isdigit() for part in parts):
|
||||
parts[2] = str(int(parts[2]) + 1)
|
||||
bumped = ".".join(parts)
|
||||
return text[: match.start(1)] + bumped + text[match.end(1) :]
|
||||
return text
|
||||
|
||||
|
||||
def _set_updated(text: str, day: str) -> str:
|
||||
"""Set the first ``updated`` field to ``day``.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - replaces a placeholder or a previous date.
|
||||
"""
|
||||
match = _UPDATED.search(text)
|
||||
if match is None:
|
||||
return text
|
||||
return text[: match.start(1)] + day + text[match.end(1) :]
|
||||
@@ -1,255 +0,0 @@
|
||||
"""Media-type detection and slug rules for wiki ingest.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.2.4 and 7.1.5.2.6.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime
|
||||
import mimetypes
|
||||
import pathlib
|
||||
import re
|
||||
import urllib.parse
|
||||
|
||||
_SLUG_TRIM = 80
|
||||
_UNUSABLE_MEDIA = {"", "application/octet-stream"}
|
||||
|
||||
|
||||
def make_slug(text: str) -> str:
|
||||
"""Turn free text into a wiki slug.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - lowercase, hyphenate, and cap at 80.
|
||||
|
||||
Args:
|
||||
text: Stem, host, or joined path segments.
|
||||
|
||||
Returns:
|
||||
str: A slug, or ``source`` when the text has no usable characters.
|
||||
"""
|
||||
slug = re.sub(r"[^a-z0-9]+", "-", text.lower()).strip("-")[:_SLUG_TRIM]
|
||||
return slug or "source"
|
||||
|
||||
|
||||
def slug_from_url(url: str) -> str:
|
||||
"""Build a slug from the last three path segments of a URL.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - drop ``index`` and an empty path uses the host.
|
||||
|
||||
Args:
|
||||
url: Absolute ``http`` or ``https`` URL.
|
||||
|
||||
Returns:
|
||||
str: Slug for the downloaded file and the wiki page.
|
||||
"""
|
||||
parts = urllib.parse.urlsplit(url)
|
||||
cleaned: list[str] = []
|
||||
for segment in parts.path.split("/"):
|
||||
if not segment:
|
||||
continue
|
||||
path = pathlib.PurePosixPath(segment)
|
||||
stem = path.stem if path.suffix else segment
|
||||
if stem.lower() == "index":
|
||||
continue
|
||||
cleaned.append(stem)
|
||||
if not cleaned:
|
||||
return make_slug(parts.hostname or "source")
|
||||
return make_slug("-".join(cleaned[-3:]))
|
||||
|
||||
|
||||
def slug_from_file(path: pathlib.Path) -> str:
|
||||
"""Build a slug from a file stem.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - local paths use the stem only.
|
||||
|
||||
Args:
|
||||
path: Local file path. It does not have to exist.
|
||||
|
||||
Returns:
|
||||
str: Slug for the archive name and the wiki page.
|
||||
"""
|
||||
return make_slug(path.stem)
|
||||
|
||||
|
||||
def split_content_type(header: str | None) -> tuple[str | None, str | None]:
|
||||
"""Split a Content-Type header into a media type and charset.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - drop parameters other than charset.
|
||||
|
||||
``application/octet-stream`` is treated as missing so the extension
|
||||
or the leading bytes can decide.
|
||||
|
||||
Args:
|
||||
header: Raw Content-Type value, or None.
|
||||
|
||||
Returns:
|
||||
tuple: Media type without parameters, and charset if one was set.
|
||||
"""
|
||||
if not header:
|
||||
return None, None
|
||||
parts = [part.strip() for part in header.split(";") if part.strip()]
|
||||
media = parts[0].lower()
|
||||
charset = None
|
||||
for part in parts[1:]:
|
||||
if "=" not in part:
|
||||
continue
|
||||
key, value = part.split("=", 1)
|
||||
if key.strip().lower() == "charset":
|
||||
charset = value.strip().strip("\"'")
|
||||
if media in _UNUSABLE_MEDIA:
|
||||
media = None
|
||||
return media, charset
|
||||
|
||||
|
||||
def media_for_extension(suffix: str, registry: object) -> str | None:
|
||||
"""Map a file extension to a media type.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - converter table first, then mimetypes.
|
||||
|
||||
Args:
|
||||
suffix: Extension including the leading dot.
|
||||
registry: Converter registry in registration order.
|
||||
|
||||
Returns:
|
||||
str | None: Media type, or None when the extension is unknown.
|
||||
"""
|
||||
lowered = suffix.lower()
|
||||
for converter in registry.converters: # type: ignore[attr-defined]
|
||||
if lowered in converter.extensions and converter.media_types:
|
||||
return converter.media_types[0]
|
||||
guessed, _encoding = mimetypes.guess_type("file" + lowered)
|
||||
return guessed
|
||||
|
||||
|
||||
def sniff_media_type(data: bytes) -> str:
|
||||
"""Guess a media type from the first 1024 bytes.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - PDF, HTML, UTF-8 text, or octet-stream.
|
||||
|
||||
Args:
|
||||
data: File bytes. Only the prefix is inspected.
|
||||
|
||||
Returns:
|
||||
str: Detected media type.
|
||||
"""
|
||||
head = data[:1024]
|
||||
if head.startswith(b"%PDF-"):
|
||||
return "application/pdf"
|
||||
sample = head.lstrip().lower()
|
||||
if sample.startswith((b"<!doctype html", b"<html")):
|
||||
return "text/html"
|
||||
if b"\x00" not in head and _is_utf8(head):
|
||||
return "text/plain"
|
||||
return "application/octet-stream"
|
||||
|
||||
|
||||
def detect_media_type(
|
||||
data: bytes,
|
||||
suffix: str,
|
||||
header: str | None,
|
||||
registry: object,
|
||||
) -> tuple[str, str | None]:
|
||||
"""Detect a media type and optional charset hint.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - header, then extension, then bytes.
|
||||
|
||||
Args:
|
||||
data: File bytes, used only when the header and extension miss.
|
||||
suffix: Extension including the dot, or empty.
|
||||
header: Content-Type header for a download, or None for a local file.
|
||||
registry: Converter registry.
|
||||
|
||||
Returns:
|
||||
tuple: Media type and charset hint.
|
||||
"""
|
||||
media, charset = split_content_type(header)
|
||||
if media:
|
||||
return media, charset
|
||||
if suffix:
|
||||
from_ext = media_for_extension(suffix, registry)
|
||||
if from_ext:
|
||||
return from_ext, charset
|
||||
return sniff_media_type(data), charset
|
||||
|
||||
|
||||
def extension_for(media_type: str, path_suffix: str, registry: object) -> str:
|
||||
"""Choose the extension for a downloaded file.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - converter extension, URL suffix, then ``.bin``.
|
||||
|
||||
Args:
|
||||
media_type: Detected media type.
|
||||
path_suffix: Suffix from the URL path, including the dot, or empty.
|
||||
registry: Converter registry.
|
||||
|
||||
Returns:
|
||||
str: Extension including the leading dot.
|
||||
"""
|
||||
converter = registry.find(media_type, "") # type: ignore[attr-defined]
|
||||
if converter is not None and converter.extensions:
|
||||
return converter.extensions[0]
|
||||
if path_suffix:
|
||||
suffix = path_suffix.lower()
|
||||
return suffix if suffix.startswith(".") else f".{suffix}"
|
||||
return ".bin"
|
||||
|
||||
|
||||
def decode_bytes(data: bytes, charset: str | None) -> tuple[str, list[str]]:
|
||||
"""Decode source bytes to text.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - UTF-8, then the charset hint, then cp1252.
|
||||
|
||||
A BOM is stripped before UTF-8 is tried. A fallback decode adds one warning.
|
||||
|
||||
Args:
|
||||
data: Raw file bytes.
|
||||
charset: Optional charset from Content-Type.
|
||||
|
||||
Returns:
|
||||
tuple: Decoded text and warnings.
|
||||
"""
|
||||
payload = data[3:] if data.startswith(b"\xef\xbb\xbf") else data
|
||||
try:
|
||||
return payload.decode("utf-8"), []
|
||||
except UnicodeDecodeError:
|
||||
pass
|
||||
if charset:
|
||||
try:
|
||||
return payload.decode(charset), [f"decoded as {charset}; not valid UTF-8"]
|
||||
except (UnicodeDecodeError, LookupError):
|
||||
pass
|
||||
return payload.decode("cp1252"), ["decoded as cp1252; not valid UTF-8"]
|
||||
|
||||
|
||||
def utc_now() -> str:
|
||||
"""Return the current UTC time as ``YYYY-MM-DDTHH:MM:SSZ``.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - ingest ``fetched`` timestamp.
|
||||
|
||||
Returns:
|
||||
str: UTC timestamp with a ``Z`` suffix.
|
||||
"""
|
||||
moment = datetime.datetime.now(datetime.UTC).replace(microsecond=0)
|
||||
return moment.strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
|
||||
|
||||
def utc_day() -> str:
|
||||
"""Return today's UTC date as ``YYYY-MM-DD``.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - page ``created`` and ``updated`` dates.
|
||||
|
||||
Returns:
|
||||
str: UTC calendar date.
|
||||
"""
|
||||
return utc_now()[:10]
|
||||
|
||||
|
||||
def _is_utf8(data: bytes) -> bool:
|
||||
"""Return whether a byte prefix is valid UTF-8.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - allow a multibyte character cut at 1024.
|
||||
"""
|
||||
try:
|
||||
data.decode("utf-8")
|
||||
except UnicodeDecodeError as exc:
|
||||
return exc.start > 0 and exc.start >= len(data) - 3 and "unexpected end" in exc.reason
|
||||
return True
|
||||
@@ -1,144 +0,0 @@
|
||||
"""Dataclasses passed between wiki ingest stages.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - fetch, convert, and JSON result types.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import dataclasses
|
||||
import pathlib
|
||||
|
||||
|
||||
class IngestError(Exception):
|
||||
"""Raised when wiki ingest must stop before a page is written.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - exit 1 errors from section 4.7.6.
|
||||
"""
|
||||
|
||||
def __init__(self, message: str, exit_code: int = 1) -> None:
|
||||
"""Store the stderr message and process exit code.
|
||||
|
||||
Args:
|
||||
message: Text after ``Error: ``.
|
||||
exit_code: Process status. Section 4.7.6 uses 1.
|
||||
"""
|
||||
super().__init__(message)
|
||||
self.exit_code = exit_code
|
||||
|
||||
|
||||
class AuthRequiredError(IngestError):
|
||||
"""Raised when a URL answers with a login instead of the source.
|
||||
|
||||
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.2.8. Exit 4, nothing written.
|
||||
"""
|
||||
|
||||
def __init__(self, url: str, reason: str) -> None:
|
||||
"""Store the URL and the reason for the ``Error:`` line.
|
||||
|
||||
Args:
|
||||
url: The URL the user passed, without userinfo.
|
||||
reason: ``<status> <phrase>``, ``redirected to <url>``, or ``login page at <url>``.
|
||||
"""
|
||||
super().__init__(f"Authentication required: {url}: {reason}", exit_code=4)
|
||||
self.url = url
|
||||
self.reason = reason
|
||||
|
||||
|
||||
@dataclasses.dataclass
|
||||
class FetchedSource:
|
||||
"""A local file plus the facts the later stages need.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - output of the fetch stage.
|
||||
"""
|
||||
|
||||
argument: str
|
||||
path: pathlib.Path
|
||||
origin: str
|
||||
origin_kind: str
|
||||
final_url: str | None
|
||||
media_type: str
|
||||
charset: str | None
|
||||
size_bytes: int
|
||||
slug: str
|
||||
fetched: str
|
||||
created: bool
|
||||
|
||||
|
||||
@dataclasses.dataclass
|
||||
class ConversionResult:
|
||||
"""Markdown produced from one archived file.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - output of one converter.
|
||||
"""
|
||||
|
||||
body: str
|
||||
title: str | None
|
||||
source_meta: dict
|
||||
conversion: str
|
||||
warnings: list[str]
|
||||
converter: str
|
||||
|
||||
|
||||
@dataclasses.dataclass
|
||||
class ArchiveResult:
|
||||
"""Where the raw copy lives after the archive stage.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - raw path, hash, and reuse flag.
|
||||
"""
|
||||
|
||||
path: pathlib.Path
|
||||
raw_rel: str
|
||||
sha256: str
|
||||
raw_reused: bool
|
||||
warnings: list[str]
|
||||
|
||||
|
||||
@dataclasses.dataclass
|
||||
class IngestResult:
|
||||
"""One ingest run, including the JSON object from section 4.7.6.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - stdout facts for a finished run.
|
||||
"""
|
||||
|
||||
source: str
|
||||
origin: str
|
||||
origin_kind: str
|
||||
media_type: str
|
||||
raw_path: str | None
|
||||
raw_reused: bool
|
||||
sha256: str
|
||||
duplicate_page: str | None
|
||||
page_path: str | None
|
||||
converter: str | None
|
||||
conversion: str
|
||||
status: str | None
|
||||
warnings: list[str]
|
||||
dry_run: bool
|
||||
exit_code: int = 0
|
||||
error: str | None = None
|
||||
messages: list[str] = dataclasses.field(default_factory=list)
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
"""Return the JSON object from section 4.7.6.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - drop CLI-only fields.
|
||||
|
||||
Returns:
|
||||
dict: Keys and nulls from the ingest JSON contract.
|
||||
"""
|
||||
return {
|
||||
"source": self.source,
|
||||
"origin": self.origin,
|
||||
"origin_kind": self.origin_kind,
|
||||
"media_type": self.media_type,
|
||||
"raw_path": self.raw_path,
|
||||
"raw_reused": self.raw_reused,
|
||||
"sha256": self.sha256,
|
||||
"duplicate_page": self.duplicate_page,
|
||||
"page_path": self.page_path,
|
||||
"converter": self.converter,
|
||||
"conversion": self.conversion,
|
||||
"status": self.status,
|
||||
"warnings": list(self.warnings),
|
||||
"dry_run": self.dry_run,
|
||||
}
|
||||
@@ -1,375 +0,0 @@
|
||||
"""Write one ``status: raw`` wiki page.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.5. The tool never sets ``refined``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
import pathlib
|
||||
import re
|
||||
|
||||
import yaml
|
||||
from pymarkdown.api import PyMarkdownApiException
|
||||
|
||||
from fiona.mdoctor.mdoctor import MDoctor
|
||||
from fiona.wiki.ingest.media import utc_day
|
||||
from fiona.wiki.ingest.models import ArchiveResult, ConversionResult, FetchedSource, IngestError
|
||||
from fiona.wiki.ingest.settings import IngestSettings, package_version
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_FENCE = re.compile(r"^\s*(`{3,}|~{3,})")
|
||||
_ATX = re.compile(r"^(#{1,6})(\s+)(.*)$")
|
||||
_IMAGE = re.compile(r"!\[([^\]]*)\]\(([^)]+)\)")
|
||||
_LINK = re.compile(r"(?<!!)\[([^\]]*)\]\(([^)]+)\)")
|
||||
_PROTECTED = {"index.md", "log.md", "schema.md"}
|
||||
|
||||
|
||||
class RawPageWriter:
|
||||
"""Write the raw page and run the markdown fixer on that file only.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - frontmatter order from SPEC 7.1.5.5.3.
|
||||
"""
|
||||
|
||||
def __init__(self, wiki_root: pathlib.Path) -> None:
|
||||
"""Store the wiki root.
|
||||
|
||||
Args:
|
||||
wiki_root: Directory that contains ``wiki/``.
|
||||
"""
|
||||
self.wiki_root = wiki_root
|
||||
|
||||
def page_path(
|
||||
self,
|
||||
slug: str,
|
||||
settings: IngestSettings,
|
||||
target: str | None,
|
||||
) -> pathlib.Path:
|
||||
"""Resolve the page path and reject paths outside ``wiki/``.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.5.1. Does not touch the disk.
|
||||
|
||||
Args:
|
||||
slug: Page stem when ``target`` is a directory or omitted.
|
||||
settings: Supplies ``default_target``.
|
||||
target: ``-t`` value, or None.
|
||||
|
||||
Returns:
|
||||
pathlib.Path: Absolute page path inside ``wiki/``.
|
||||
|
||||
Raises:
|
||||
IngestError: The path escapes ``wiki/`` or names a protected file.
|
||||
"""
|
||||
wiki = (self.wiki_root / "wiki").resolve()
|
||||
if target is None:
|
||||
relative = f"{settings.default_target}/{slug}.md"
|
||||
else:
|
||||
relative = _strip_wiki_prefix(target)
|
||||
suffix = pathlib.PurePosixPath(relative.replace("\\", "/")).suffix
|
||||
if suffix != ".md":
|
||||
relative = f"{relative}/{slug}.md" if relative else f"{slug}.md"
|
||||
candidate = (wiki / relative).resolve()
|
||||
try:
|
||||
page_rel = candidate.relative_to(wiki).as_posix()
|
||||
except ValueError as exc:
|
||||
raise IngestError(f"--target must stay inside wiki/: {target}") from exc
|
||||
if page_rel in _PROTECTED or page_rel.split("/", 1)[0] == "logs":
|
||||
shown = target if target else page_rel
|
||||
raise IngestError(f"--target cannot replace {shown}")
|
||||
return candidate
|
||||
|
||||
def write(
|
||||
self,
|
||||
*,
|
||||
source: FetchedSource,
|
||||
archive: ArchiveResult,
|
||||
conversion: ConversionResult,
|
||||
page: pathlib.Path,
|
||||
dry_run: bool,
|
||||
) -> tuple[str, list[str]]:
|
||||
"""Write the page, fix it, and return the relative path and warnings.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.5.3 through 7.1.5.5.5.
|
||||
|
||||
Args:
|
||||
source: Fetch facts stored in ``ingest``.
|
||||
archive: Raw path, hash, and reuse flag.
|
||||
conversion: Converter output.
|
||||
page: Destination markdown file.
|
||||
dry_run: When true, build the warnings and write nothing.
|
||||
|
||||
Returns:
|
||||
tuple: Page path relative to the wiki, and the warning list.
|
||||
"""
|
||||
title = conversion.title or source.slug
|
||||
content, link_warnings = _page_content(conversion.body, title, page, archive.path)
|
||||
warnings = [*conversion.warnings, *archive.warnings, *link_warnings]
|
||||
if dry_run:
|
||||
return _rel(page, self.wiki_root), warnings
|
||||
page.parent.mkdir(parents=True, exist_ok=True)
|
||||
day = utc_day()
|
||||
page.write_text(
|
||||
_document(source, archive, conversion, content, warnings, day),
|
||||
encoding="utf-8",
|
||||
newline="\n",
|
||||
)
|
||||
_fix(page, self.wiki_root)
|
||||
style_count = _style_violations(page, self.wiki_root)
|
||||
if style_count:
|
||||
warnings.append(f"{style_count} markdown style violations left")
|
||||
fixed = _body_after_frontmatter(page.read_text(encoding="utf-8"))
|
||||
if not fixed.lstrip("\n").startswith("# "):
|
||||
fixed = content
|
||||
page.write_text(
|
||||
_document(source, archive, conversion, fixed.lstrip("\n"), warnings, day),
|
||||
encoding="utf-8",
|
||||
newline="\n",
|
||||
)
|
||||
logger.info("wrote %s", page)
|
||||
return _rel(page, self.wiki_root), warnings
|
||||
|
||||
|
||||
def _page_content(
|
||||
body: str,
|
||||
title: str,
|
||||
page: pathlib.Path,
|
||||
raw_path: pathlib.Path,
|
||||
) -> tuple[str, list[str]]:
|
||||
"""Demote extra H1s, drop dead relative links, and add the ingest intro.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - the page has one H1 and a link to the raw file.
|
||||
"""
|
||||
shifted = _shift_headings(body)
|
||||
rewritten, count = _rewrite_links(shifted, page.parent)
|
||||
warnings = [f"{count} relative link(s) did not resolve"] if count else []
|
||||
link = os.path.relpath(raw_path, page.parent).replace("\\", "/")
|
||||
intro = f"Raw ingest of [{raw_path.name}]({link}). Not refined yet."
|
||||
prepared = rewritten.strip("\n")
|
||||
tail = f"{intro}\n\n{prepared}\n" if prepared else f"{intro}\n"
|
||||
return f"# {title}\n\n{tail}", warnings
|
||||
|
||||
|
||||
def _document(
|
||||
source: FetchedSource,
|
||||
archive: ArchiveResult,
|
||||
conversion: ConversionResult,
|
||||
content: str,
|
||||
warnings: list[str],
|
||||
day: str,
|
||||
) -> str:
|
||||
"""Render frontmatter plus the page content.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - every key is present, in contract order.
|
||||
"""
|
||||
title = conversion.title or source.slug
|
||||
document = {
|
||||
"name": source.slug,
|
||||
"version": "0.1.0",
|
||||
"title": title,
|
||||
"created": day,
|
||||
"updated": day,
|
||||
"type": "summary",
|
||||
"status": "raw",
|
||||
"sources": [archive.raw_rel],
|
||||
"source_meta": conversion.source_meta or {},
|
||||
"ingest": {
|
||||
"origin": source.origin,
|
||||
"origin_kind": source.origin_kind,
|
||||
"final_url": source.final_url,
|
||||
"fetched": source.fetched,
|
||||
"media_type": source.media_type,
|
||||
"size_bytes": source.size_bytes,
|
||||
"sha256": archive.sha256,
|
||||
"raw_reused": archive.raw_reused,
|
||||
"converter": conversion.converter,
|
||||
"conversion": conversion.conversion,
|
||||
"warnings": list(warnings),
|
||||
"tool": f"fiona-assistant {package_version()}",
|
||||
},
|
||||
"tags": [],
|
||||
"confidence": "zero",
|
||||
"llm": [{"model": "none"}],
|
||||
"contested": False,
|
||||
"contradictions": [],
|
||||
}
|
||||
dumped = yaml.safe_dump(document, sort_keys=False, allow_unicode=True, width=1000)
|
||||
payload = content if content.endswith("\n") else content + "\n"
|
||||
return f"---\n{dumped}---\n\n{payload}"
|
||||
|
||||
|
||||
def _shift_headings(body: str) -> str:
|
||||
"""Move every ATX heading down one level when the body has an H1.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - H6 stays H6. Fenced lines are left alone.
|
||||
"""
|
||||
if not _has_h1(body):
|
||||
return body
|
||||
lines = body.splitlines(keepends=True)
|
||||
fenced = False
|
||||
shifted: list[str] = []
|
||||
for line in lines:
|
||||
ending, content = _split_ending(line)
|
||||
fence = _FENCE.match(content)
|
||||
if fence:
|
||||
fenced = not fenced
|
||||
shifted.append(line)
|
||||
continue
|
||||
match = None if fenced else _ATX.match(content)
|
||||
if match is None:
|
||||
shifted.append(line)
|
||||
continue
|
||||
marks = match.group(1)
|
||||
if len(marks) < 6:
|
||||
marks += "#"
|
||||
shifted.append(f"{marks}{match.group(2)}{match.group(3)}{ending}")
|
||||
return "".join(shifted)
|
||||
|
||||
|
||||
def _has_h1(body: str) -> bool:
|
||||
"""Return whether an unfenced ATX H1 is present.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - detection does not change the body.
|
||||
"""
|
||||
fenced = False
|
||||
for line in body.splitlines():
|
||||
if _FENCE.match(line):
|
||||
fenced = not fenced
|
||||
continue
|
||||
if not fenced and re.match(r"^#\s+\S", line):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _rewrite_links(body: str, page_dir: pathlib.Path) -> tuple[str, int]:
|
||||
"""Replace relative links and images that do not resolve from the page.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - one warning carries the count.
|
||||
"""
|
||||
count = 0
|
||||
lines = body.splitlines(keepends=True)
|
||||
fenced = False
|
||||
rewritten: list[str] = []
|
||||
for line in lines:
|
||||
ending, content = _split_ending(line)
|
||||
if _FENCE.match(content):
|
||||
fenced = not fenced
|
||||
rewritten.append(line)
|
||||
continue
|
||||
if fenced:
|
||||
rewritten.append(line)
|
||||
continue
|
||||
content, image_count = _replace_pattern(_IMAGE, content, page_dir)
|
||||
content, link_count = _replace_pattern(_LINK, content, page_dir)
|
||||
count += image_count + link_count
|
||||
rewritten.append(content + ending)
|
||||
return "".join(rewritten), count
|
||||
|
||||
|
||||
def _replace_pattern(
|
||||
pattern: re.Pattern[str],
|
||||
content: str,
|
||||
page_dir: pathlib.Path,
|
||||
) -> tuple[str, int]:
|
||||
"""Replace matches whose target does not exist.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - external URLs and anchors stay.
|
||||
"""
|
||||
count = 0
|
||||
|
||||
def _swap(match: re.Match[str]) -> str:
|
||||
nonlocal count
|
||||
target = match.group(2).strip()
|
||||
if _keeps_target(target, page_dir):
|
||||
return match.group(0)
|
||||
count += 1
|
||||
return match.group(1)
|
||||
|
||||
return pattern.sub(_swap, content), count
|
||||
|
||||
|
||||
def _keeps_target(target: str, page_dir: pathlib.Path) -> bool:
|
||||
"""Return whether a link target should stay in the page.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - http, mailto, and anchors are kept.
|
||||
"""
|
||||
lowered = target.lower()
|
||||
if lowered.startswith(("http://", "https://", "mailto:", "#", "//")) or "://" in target:
|
||||
return True
|
||||
path_part = target.split("#", 1)[0].split("?", 1)[0]
|
||||
if not path_part:
|
||||
return True
|
||||
resolved = (page_dir / path_part).resolve()
|
||||
return resolved.exists()
|
||||
|
||||
|
||||
def _fix(page: pathlib.Path, wiki_root: pathlib.Path) -> None:
|
||||
"""Run the pymarkdown fixer on one page.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.5.5. A fixer error is logged.
|
||||
"""
|
||||
try:
|
||||
doctor = MDoctor(settings_file=wiki_root / "fiona-settings.json")
|
||||
doctor.fix_file_programmatically(str(page))
|
||||
except PyMarkdownApiException:
|
||||
logger.warning("markdown fix failed for %s", page, exc_info=True)
|
||||
|
||||
|
||||
def _style_violations(page: pathlib.Path, wiki_root: pathlib.Path) -> int:
|
||||
"""Count pymarkdown violations still on the page.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - the count becomes one warning.
|
||||
"""
|
||||
try:
|
||||
doctor = MDoctor(settings_file=wiki_root / "fiona-settings.json")
|
||||
return len(doctor.scan_file_programmatically(str(page)))
|
||||
except PyMarkdownApiException:
|
||||
logger.warning("markdown scan failed for %s", page, exc_info=True)
|
||||
return 0
|
||||
|
||||
|
||||
def _body_after_frontmatter(text: str) -> str:
|
||||
"""Return the file body after the closing frontmatter marker.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - keep the fixer output and replace the header.
|
||||
"""
|
||||
if not text.startswith("---"):
|
||||
return text
|
||||
parts = text.split("---", 2)
|
||||
if len(parts) < 3:
|
||||
return text
|
||||
return parts[2].removeprefix("\n")
|
||||
|
||||
|
||||
def _split_ending(line: str) -> tuple[str, str]:
|
||||
"""Split a line into its ending and its content.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - keep ``\\n`` and ``\\r\\n``.
|
||||
"""
|
||||
if line.endswith("\r\n"):
|
||||
return "\r\n", line[:-2]
|
||||
if line.endswith("\n"):
|
||||
return "\n", line[:-1]
|
||||
return "", line
|
||||
|
||||
|
||||
def _strip_wiki_prefix(value: str) -> str:
|
||||
"""Drop a leading ``wiki/`` from a ``-t`` value.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - both forms are accepted.
|
||||
"""
|
||||
relative = value.replace("\\", "/").lstrip("/")
|
||||
if relative == "wiki":
|
||||
return ""
|
||||
if relative.startswith("wiki/"):
|
||||
return relative[5:]
|
||||
return relative
|
||||
|
||||
|
||||
def _rel(path: pathlib.Path, root: pathlib.Path) -> str:
|
||||
"""Return a path relative to the wiki root using ``/``.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - JSON page paths.
|
||||
"""
|
||||
return path.resolve().relative_to(root.resolve()).as_posix()
|
||||
@@ -1,586 +0,0 @@
|
||||
"""Run the wiki ingest stages in order.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5. The pipeline holds no format logic.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import dataclasses
|
||||
import json
|
||||
import logging
|
||||
import pathlib
|
||||
import shutil
|
||||
import sys
|
||||
import tempfile
|
||||
|
||||
from fiona.librarian.md_frontmatter import FrontmatterError, parse_frontmatter
|
||||
from fiona.wiki.ingest.archiver import RawArchiver
|
||||
from fiona.wiki.ingest.converters import default_registry
|
||||
from fiona.wiki.ingest.converters.base import ConversionError, ConverterRegistry
|
||||
from fiona.wiki.ingest.fetchers import (
|
||||
LocalFileFetcher,
|
||||
SourceFetcher,
|
||||
UrlFetcher,
|
||||
url_scheme,
|
||||
without_userinfo,
|
||||
)
|
||||
from fiona.wiki.ingest.log_writer import IngestLog
|
||||
from fiona.wiki.ingest.media import slug_from_file, slug_from_url, utc_day
|
||||
from fiona.wiki.ingest.models import AuthRequiredError, FetchedSource, IngestError, IngestResult
|
||||
from fiona.wiki.ingest.page_writer import RawPageWriter
|
||||
from fiona.wiki.ingest.settings import IngestSettings, load_ingest_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_NO_CONVERTER = "No converter for {media}. Convert {raw} by hand."
|
||||
_AUTH_NEXT = "Next: save it into ingest/, then run: wiki ingest <file> --origin {url}"
|
||||
|
||||
|
||||
class IngestPipeline:
|
||||
"""Fetch, archive, convert, write, and record one source.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - stages are constructor arguments.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
wiki_root: pathlib.Path,
|
||||
fetchers: list[SourceFetcher],
|
||||
registry: ConverterRegistry,
|
||||
archiver: RawArchiver,
|
||||
page_writer: RawPageWriter,
|
||||
log_writer: IngestLog,
|
||||
settings: IngestSettings,
|
||||
) -> None:
|
||||
"""Store the stage objects. No converter is named here.
|
||||
|
||||
Args:
|
||||
wiki_root: Wiki root directory.
|
||||
fetchers: Tried in order. Each implements ``accepts`` and ``fetch``.
|
||||
registry: Converter registry. Tests may register another converter.
|
||||
archiver: Raw archive stage.
|
||||
page_writer: Wiki page stage.
|
||||
log_writer: ``wiki/log.md`` stage.
|
||||
settings: Timeouts, size limit, and default paths.
|
||||
"""
|
||||
self.wiki_root = wiki_root
|
||||
self.fetchers = fetchers
|
||||
self.registry = registry
|
||||
self.archiver = archiver
|
||||
self.page_writer = page_writer
|
||||
self.log_writer = log_writer
|
||||
self.settings = settings
|
||||
|
||||
def run(
|
||||
self,
|
||||
argument: str,
|
||||
*,
|
||||
raw: str | None = None,
|
||||
target: str | None = None,
|
||||
raw_only: bool = False,
|
||||
force: bool = False,
|
||||
dry_run: bool = False,
|
||||
origin: str | None = None,
|
||||
) -> IngestResult:
|
||||
"""Ingest one URL or local file.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - exit 1 raises; exit 0 and 3 return a result.
|
||||
Modified by AI/LLM: 2026-10-07 - ``origin`` for a file saved by hand (SPEC 7.1.5.2.9).
|
||||
Modified by AI/LLM: 2026-10-08 - a file inside the wiki has a relative origin (SPEC 7.1.5.2.10).
|
||||
|
||||
Args:
|
||||
argument: Source as given.
|
||||
raw: ``-r`` path relative to ``raw/``.
|
||||
target: ``-t`` path relative to ``wiki/``.
|
||||
raw_only: Archive and record, skip convert and write.
|
||||
force: Replace an existing page, including a duplicate.
|
||||
dry_run: Report the run and write nothing under the wiki.
|
||||
origin: ``--origin`` URL a local file was saved from.
|
||||
|
||||
Returns:
|
||||
IngestResult: Facts for stdout. ``exit_code`` is 0 or 3.
|
||||
|
||||
Raises:
|
||||
IngestError: Exit 1. Nothing new remains under the wiki.
|
||||
AuthRequiredError: Exit 4. The URL needs a login.
|
||||
"""
|
||||
scheme = url_scheme(argument)
|
||||
if scheme is not None and scheme.lower() not in {"http", "https"}:
|
||||
raise IngestError(f"Unsupported URL scheme: {scheme}")
|
||||
_check_origin(argument, scheme, origin)
|
||||
slug = slug_from_url(argument) if scheme else slug_from_file(pathlib.Path(argument))
|
||||
page = self.page_writer.page_path(slug, self.settings, target)
|
||||
if page.is_file() and not force:
|
||||
rel = page.resolve().relative_to(self.wiki_root.resolve()).as_posix()
|
||||
raise IngestError(f"Page exists: {rel}. Use --force to replace it.")
|
||||
fetcher = self._fetcher(argument)
|
||||
fetcher.settings = self.settings
|
||||
temp_dir: pathlib.Path | None = None
|
||||
fetched: FetchedSource | None = None
|
||||
archived = False
|
||||
try:
|
||||
if dry_run and scheme:
|
||||
temp_dir = pathlib.Path(tempfile.mkdtemp(prefix="fiona-ingest-"))
|
||||
dest = temp_dir
|
||||
else:
|
||||
dest = self.wiki_root / "ingest"
|
||||
fetched = fetcher.fetch(argument, dest)
|
||||
if fetched.origin_kind == "file" and fetched.path.is_relative_to(self.wiki_root.resolve()):
|
||||
fetched = dataclasses.replace(
|
||||
fetched,
|
||||
origin=fetched.path.relative_to(self.wiki_root.resolve()).as_posix(),
|
||||
)
|
||||
if origin is not None:
|
||||
fetched = dataclasses.replace(
|
||||
fetched,
|
||||
origin=without_userinfo(origin),
|
||||
origin_kind="url",
|
||||
final_url=None,
|
||||
)
|
||||
archive = self.archiver.archive(
|
||||
fetched,
|
||||
registry=self.registry,
|
||||
settings=self.settings,
|
||||
raw_override=raw,
|
||||
dry_run=dry_run,
|
||||
)
|
||||
archived = True
|
||||
return self._after_archive(
|
||||
fetched,
|
||||
archive,
|
||||
page,
|
||||
target_explicit=target is not None,
|
||||
raw_only=raw_only,
|
||||
force=force,
|
||||
dry_run=dry_run,
|
||||
)
|
||||
except IngestError:
|
||||
if fetched is not None and fetched.created and fetched.path.exists() and not archived:
|
||||
fetched.path.unlink()
|
||||
raise
|
||||
finally:
|
||||
if temp_dir is not None:
|
||||
shutil.rmtree(temp_dir, ignore_errors=True)
|
||||
|
||||
def list_pending(self) -> list[dict]:
|
||||
"""List wiki pages whose ``status`` is ``raw``, sorted by path.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.9.
|
||||
|
||||
Returns:
|
||||
list[dict]: ``page_path``, ``sources``, and ``origin`` for each page.
|
||||
"""
|
||||
wiki = self.wiki_root / "wiki"
|
||||
if not wiki.is_dir():
|
||||
return []
|
||||
pages: list[dict] = []
|
||||
for path in sorted(wiki.rglob("*.md"), key=lambda item: item.as_posix()):
|
||||
if not path.is_file():
|
||||
continue
|
||||
try:
|
||||
front = parse_frontmatter(path)
|
||||
except FrontmatterError:
|
||||
continue
|
||||
if front.get("status") != "raw":
|
||||
continue
|
||||
sources = front.get("sources") or []
|
||||
if isinstance(sources, str):
|
||||
sources = [sources]
|
||||
ingest = front.get("ingest") if isinstance(front.get("ingest"), dict) else {}
|
||||
origin = ingest.get("origin") if isinstance(ingest, dict) else None
|
||||
pages.append({
|
||||
"page_path": path.resolve().relative_to(self.wiki_root.resolve()).as_posix(),
|
||||
"sources": list(sources),
|
||||
"origin": origin,
|
||||
})
|
||||
return pages
|
||||
|
||||
def _fetcher(self, argument: str) -> SourceFetcher:
|
||||
"""Return the first fetcher that accepts the argument.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - URL fetcher is registered before the local one.
|
||||
"""
|
||||
for fetcher in self.fetchers:
|
||||
if fetcher.accepts(argument):
|
||||
return fetcher
|
||||
raise IngestError(f"Source not found: {argument}")
|
||||
|
||||
def _after_archive(
|
||||
self,
|
||||
source: FetchedSource,
|
||||
archive: object,
|
||||
page: pathlib.Path,
|
||||
*,
|
||||
target_explicit: bool,
|
||||
raw_only: bool,
|
||||
force: bool,
|
||||
dry_run: bool,
|
||||
) -> IngestResult:
|
||||
"""Finish convert, write, and record after a successful archive.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - duplicate, raw-only, and missing converter exits.
|
||||
"""
|
||||
messages = _fetched_line(source)
|
||||
if archive.raw_reused: # type: ignore[attr-defined]
|
||||
short = archive.sha256[:12] # type: ignore[attr-defined]
|
||||
messages.append(
|
||||
f"Archived: {archive.raw_rel} (already archived, sha256 {short})" # type: ignore[attr-defined]
|
||||
)
|
||||
else:
|
||||
messages.append(f"Archived: {archive.raw_rel}") # type: ignore[attr-defined]
|
||||
duplicate = _citing_page(self.wiki_root, archive.raw_rel, archive.sha256) # type: ignore[attr-defined]
|
||||
if duplicate and not force:
|
||||
messages.append(f"Duplicate: {duplicate} already cites this source.")
|
||||
return _result(
|
||||
source,
|
||||
archive,
|
||||
messages=messages,
|
||||
duplicate_page=duplicate,
|
||||
dry_run=dry_run,
|
||||
warnings=list(archive.warnings), # type: ignore[attr-defined]
|
||||
)
|
||||
if duplicate and force and not target_explicit:
|
||||
page = self.wiki_root / duplicate
|
||||
if raw_only:
|
||||
self.log_writer.append(
|
||||
day=utc_day(),
|
||||
origin=source.origin,
|
||||
raw_rel=archive.raw_rel, # type: ignore[attr-defined]
|
||||
page_rel=None,
|
||||
dry_run=dry_run,
|
||||
)
|
||||
return _result(source, archive, messages=messages, dry_run=dry_run, warnings=list(archive.warnings)) # type: ignore[attr-defined]
|
||||
extension = archive.path.suffix # type: ignore[attr-defined]
|
||||
converter = self.registry.find(source.media_type, extension)
|
||||
if converter is None:
|
||||
messages.append(_NO_CONVERTER.format(media=source.media_type, raw=archive.raw_rel)) # type: ignore[attr-defined]
|
||||
self._record(source, archive, None, dry_run)
|
||||
return _result(
|
||||
source,
|
||||
archive,
|
||||
messages=messages,
|
||||
dry_run=dry_run,
|
||||
exit_code=3,
|
||||
warnings=list(archive.warnings), # type: ignore[attr-defined]
|
||||
)
|
||||
read_path = archive.path if archive.path.is_file() else source.path # type: ignore[attr-defined]
|
||||
try:
|
||||
conversion = converter.convert(dataclasses.replace(source, path=read_path))
|
||||
except ConversionError as exc:
|
||||
messages.append(_NO_CONVERTER.format(media=source.media_type, raw=archive.raw_rel)) # type: ignore[attr-defined]
|
||||
self._record(source, archive, None, dry_run)
|
||||
return _result(
|
||||
source,
|
||||
archive,
|
||||
messages=messages,
|
||||
dry_run=dry_run,
|
||||
exit_code=3,
|
||||
error=f"Conversion failed: {exc.converter}: {exc.reason}",
|
||||
warnings=list(archive.warnings), # type: ignore[attr-defined]
|
||||
)
|
||||
page_rel, warnings = self.page_writer.write(
|
||||
source=source,
|
||||
archive=archive, # type: ignore[arg-type]
|
||||
conversion=conversion,
|
||||
page=page,
|
||||
dry_run=dry_run,
|
||||
)
|
||||
messages.append(
|
||||
f"Wrote: {page_rel} (converter {conversion.converter}, conversion {conversion.conversion})"
|
||||
)
|
||||
messages.extend(f"Warning: {warning}" for warning in warnings)
|
||||
messages.append(f"Next: refine {page_rel}, then set status: refined.")
|
||||
self._record(source, archive, page_rel, dry_run)
|
||||
return _result(
|
||||
source,
|
||||
archive,
|
||||
messages=messages,
|
||||
page_path=page_rel,
|
||||
converter=conversion.converter,
|
||||
conversion=conversion.conversion,
|
||||
status="raw",
|
||||
dry_run=dry_run,
|
||||
warnings=warnings,
|
||||
)
|
||||
|
||||
def _record(self, source: FetchedSource, archive: object, page_rel: str | None, dry_run: bool) -> None:
|
||||
"""Append the log line for a run that archived a source.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - duplicates do not call this.
|
||||
"""
|
||||
self.log_writer.append(
|
||||
day=utc_day(),
|
||||
origin=source.origin,
|
||||
raw_rel=archive.raw_rel, # type: ignore[attr-defined]
|
||||
page_rel=page_rel,
|
||||
dry_run=dry_run,
|
||||
)
|
||||
|
||||
|
||||
def default_pipeline(wiki_root: pathlib.Path) -> IngestPipeline:
|
||||
"""Build the pipeline with the shipped fetchers and converters.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - URL fetcher, then local file fetcher.
|
||||
Modified by AI/LLM: 2026-10-07 - one settings object for the registry and the pipeline.
|
||||
|
||||
Args:
|
||||
wiki_root: Wiki root directory.
|
||||
|
||||
Returns:
|
||||
IngestPipeline: Ready to ``run`` or ``list_pending``.
|
||||
"""
|
||||
settings = load_ingest_settings(wiki_root)
|
||||
registry = default_registry(settings)
|
||||
return IngestPipeline(
|
||||
wiki_root=wiki_root,
|
||||
fetchers=[UrlFetcher(registry), LocalFileFetcher(registry)],
|
||||
registry=registry,
|
||||
archiver=RawArchiver(wiki_root),
|
||||
page_writer=RawPageWriter(wiki_root),
|
||||
log_writer=IngestLog(wiki_root),
|
||||
settings=settings,
|
||||
)
|
||||
|
||||
|
||||
def render_stdout(result: IngestResult) -> str:
|
||||
"""Return the human report, including the dry-run banner.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - section 4.7.6 line order.
|
||||
"""
|
||||
lines: list[str] = []
|
||||
if result.dry_run:
|
||||
lines.append("Dry run - no files will be modified:")
|
||||
lines.extend(result.messages)
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def add_ingest_arguments(parser: argparse.ArgumentParser) -> None:
|
||||
"""Add the section 4.7.6 flags to an ingest parser.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - the CLI subparser owns these flags, including --help.
|
||||
|
||||
Args:
|
||||
parser: The ``wiki ingest`` parser.
|
||||
"""
|
||||
parser.add_argument("source", nargs="?", help="http(s) URL or file path")
|
||||
parser.add_argument(
|
||||
"--wiki",
|
||||
default=".",
|
||||
help="llm-wiki directory (default: .)",
|
||||
)
|
||||
parser.add_argument("-r", "--raw", dest="raw", help="raw path, relative to raw/")
|
||||
parser.add_argument(
|
||||
"--origin",
|
||||
dest="origin",
|
||||
help="http(s) URL a local file was saved from",
|
||||
)
|
||||
parser.add_argument(
|
||||
"-t",
|
||||
"--target",
|
||||
dest="target",
|
||||
help="page path, relative to wiki/",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--raw-only",
|
||||
action="store_true",
|
||||
help="archive only; do not write a page",
|
||||
)
|
||||
parser.add_argument(
|
||||
"-f",
|
||||
"--force",
|
||||
action="store_true",
|
||||
help="replace an existing page",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--dry-run",
|
||||
action="store_true",
|
||||
help="print the plan and write nothing in the wiki",
|
||||
)
|
||||
parser.add_argument("--json", action="store_true", help="print only the JSON result")
|
||||
parser.add_argument(
|
||||
"--pending",
|
||||
action="store_true",
|
||||
help="list pages whose status is raw",
|
||||
)
|
||||
|
||||
|
||||
def ingest_main(argv: list[str] | None = None) -> int:
|
||||
"""CLI entry for ``fiona-assistant wiki ingest``.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - section 4.7.6 exit codes.
|
||||
|
||||
Args:
|
||||
argv: Arguments after the subcommand. ``None`` reads the process arguments.
|
||||
|
||||
Returns:
|
||||
int: 0, 1, 2, 3, or 4.
|
||||
"""
|
||||
parser = argparse.ArgumentParser(prog="fiona-assistant wiki ingest")
|
||||
add_ingest_arguments(parser)
|
||||
return execute_ingest(parser.parse_args(argv), parser)
|
||||
|
||||
|
||||
def execute_ingest(args: argparse.Namespace, parser: argparse.ArgumentParser) -> int:
|
||||
"""Run one ingest from an already parsed namespace.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - shared by ingest_main and the wiki subcommand.
|
||||
|
||||
Args:
|
||||
args: Parsed source, flags, and paths.
|
||||
parser: Parser used for the mutual-exclusion error.
|
||||
|
||||
Returns:
|
||||
int: 0, 1, 2, 3, or 4.
|
||||
"""
|
||||
if bool(args.source) == bool(args.pending):
|
||||
parser.error("provide a source or --pending, not both")
|
||||
try:
|
||||
root = _require_wiki(args.wiki)
|
||||
pipeline = default_pipeline(root)
|
||||
if args.pending:
|
||||
return _print_pending(pipeline, args.json)
|
||||
result = pipeline.run(
|
||||
args.source,
|
||||
raw=args.raw,
|
||||
target=args.target,
|
||||
raw_only=args.raw_only,
|
||||
force=args.force,
|
||||
dry_run=args.dry_run,
|
||||
origin=args.origin,
|
||||
)
|
||||
except IngestError as exc:
|
||||
print(f"Error: {exc}", file=sys.stderr)
|
||||
if isinstance(exc, AuthRequiredError):
|
||||
print(_AUTH_NEXT.format(url=exc.url), file=sys.stderr)
|
||||
return exc.exit_code
|
||||
if result.error:
|
||||
print(f"Error: {result.error}", file=sys.stderr)
|
||||
if args.json:
|
||||
print(json.dumps(result.to_dict(), indent=2))
|
||||
else:
|
||||
text = render_stdout(result)
|
||||
if text:
|
||||
print(text)
|
||||
return result.exit_code
|
||||
|
||||
|
||||
def _check_origin(argument: str, scheme: str | None, origin: str | None) -> None:
|
||||
"""Reject ``--origin`` with a URL source or a non-http value, before fetch.
|
||||
|
||||
Generated by AI/LLM: 2026-10-07 - SPEC 7.1.5.2.9.1.
|
||||
"""
|
||||
if origin is None:
|
||||
return
|
||||
if scheme is not None:
|
||||
raise IngestError(f"--origin needs a local file: {argument}")
|
||||
origin_scheme = url_scheme(origin)
|
||||
if origin_scheme is None or origin_scheme.lower() not in {"http", "https"}:
|
||||
raise IngestError(f"--origin must be an http or https URL: {origin}")
|
||||
|
||||
|
||||
def _require_wiki(directory: str) -> pathlib.Path:
|
||||
"""Return the wiki root or raise when settings are missing.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - section 4.7.6 directory error.
|
||||
"""
|
||||
root = pathlib.Path(directory)
|
||||
if not (root / "fiona-settings.json").is_file():
|
||||
raise IngestError(f"Directory '{directory}' is not an llm-wiki directory")
|
||||
return root
|
||||
|
||||
|
||||
def _print_pending(pipeline: IngestPipeline, json_output: bool) -> int:
|
||||
"""Print the pending-page report.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - human lines or one JSON object.
|
||||
"""
|
||||
pages = pipeline.list_pending()
|
||||
if json_output:
|
||||
print(json.dumps({"pending": pages, "total": len(pages)}, indent=2))
|
||||
return 0
|
||||
print(f"Raw pages: {len(pages)}")
|
||||
for page in pages:
|
||||
first = page["sources"][0] if page["sources"] else ""
|
||||
print(f" - {page['page_path']} ({first})")
|
||||
return 0
|
||||
|
||||
|
||||
def _fetched_line(source: FetchedSource) -> list[str]:
|
||||
"""Return the ``Fetched:`` line for a URL, or an empty list.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - local files do not print Fetched.
|
||||
Modified by AI/LLM: 2026-10-07 - a local file with ``--origin`` was not fetched either.
|
||||
"""
|
||||
if url_scheme(source.argument) is None:
|
||||
return []
|
||||
return [f"Fetched: {source.argument} ({source.media_type}, {source.size_bytes} bytes)"]
|
||||
|
||||
|
||||
def _citing_page(wiki_root: pathlib.Path, raw_rel: str, digest: str) -> str | None:
|
||||
"""Return a page that already cites this raw path or hash.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.1.5.3.5. First path in sorted order.
|
||||
"""
|
||||
wiki = wiki_root / "wiki"
|
||||
if not wiki.is_dir():
|
||||
return None
|
||||
matches: list[str] = []
|
||||
for path in wiki.rglob("*.md"):
|
||||
if not path.is_file():
|
||||
continue
|
||||
try:
|
||||
front = parse_frontmatter(path)
|
||||
except FrontmatterError:
|
||||
continue
|
||||
sources = front.get("sources") or []
|
||||
if isinstance(sources, str):
|
||||
sources = [sources]
|
||||
ingest = front.get("ingest") if isinstance(front.get("ingest"), dict) else {}
|
||||
sha = ingest.get("sha256") if isinstance(ingest, dict) else None
|
||||
if raw_rel in list(sources) or sha == digest:
|
||||
matches.append(path.resolve().relative_to(wiki_root.resolve()).as_posix())
|
||||
if not matches:
|
||||
return None
|
||||
matches.sort()
|
||||
return matches[0]
|
||||
|
||||
|
||||
def _result(
|
||||
source: FetchedSource,
|
||||
archive: object,
|
||||
*,
|
||||
messages: list[str],
|
||||
dry_run: bool,
|
||||
warnings: list[str],
|
||||
duplicate_page: str | None = None,
|
||||
page_path: str | None = None,
|
||||
converter: str | None = None,
|
||||
conversion: str = "none",
|
||||
status: str | None = None,
|
||||
exit_code: int = 0,
|
||||
error: str | None = None,
|
||||
) -> IngestResult:
|
||||
"""Build an ingest result from the archive and any page that was written.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - JSON fields from section 4.7.6.
|
||||
"""
|
||||
return IngestResult(
|
||||
source=source.argument,
|
||||
origin=source.origin,
|
||||
origin_kind=source.origin_kind,
|
||||
media_type=source.media_type,
|
||||
raw_path=archive.raw_rel, # type: ignore[attr-defined]
|
||||
raw_reused=archive.raw_reused, # type: ignore[attr-defined]
|
||||
sha256=archive.sha256, # type: ignore[attr-defined]
|
||||
duplicate_page=duplicate_page,
|
||||
page_path=page_path,
|
||||
converter=converter,
|
||||
conversion=conversion,
|
||||
status=status,
|
||||
warnings=warnings,
|
||||
dry_run=dry_run,
|
||||
exit_code=exit_code,
|
||||
error=error,
|
||||
messages=messages,
|
||||
)
|
||||
@@ -1,153 +0,0 @@
|
||||
"""Ingest settings from ``fiona-settings.json``.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 5.1 ``fiona.wiki.ingest`` keys.
|
||||
Modified by AI/LLM: 2026-10-07 - ``login_url_patterns`` and ``html.*`` keys.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import dataclasses
|
||||
import importlib.metadata
|
||||
import json
|
||||
import logging
|
||||
import pathlib
|
||||
import re
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
DEFAULT_LOGIN_URL_PATTERNS = (
|
||||
r"/(login|logon|signin|sign-in)\b",
|
||||
r"/(sso|saml2?|oauth2?|adfs|cas)(/|$)",
|
||||
r"login\.microsoftonline\.com",
|
||||
r"j_security_check",
|
||||
)
|
||||
|
||||
|
||||
def package_version() -> str:
|
||||
"""Return the installed fiona-assistant version.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - user agent and ``ingest.tool``.
|
||||
|
||||
Returns:
|
||||
str: Distribution version, or ``0.99.0`` when metadata is absent.
|
||||
"""
|
||||
try:
|
||||
return importlib.metadata.version("fiona-assistant")
|
||||
except importlib.metadata.PackageNotFoundError:
|
||||
return "0.99.0"
|
||||
|
||||
|
||||
@dataclasses.dataclass
|
||||
class IngestSettings:
|
||||
"""Runtime limits and paths for one ingest.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - defaults from the settings table.
|
||||
Modified by AI/LLM: 2026-10-07 - login patterns and block sweep limits.
|
||||
"""
|
||||
|
||||
timeout_seconds: float
|
||||
max_download_bytes: int
|
||||
user_agent: str
|
||||
default_target: str
|
||||
raw_categories: dict[str, str]
|
||||
login_url_patterns: list[str] = dataclasses.field(
|
||||
default_factory=lambda: list(DEFAULT_LOGIN_URL_PATTERNS)
|
||||
)
|
||||
html_min_words: int = 4
|
||||
html_max_link_density: float = 0.5
|
||||
|
||||
|
||||
def default_settings() -> IngestSettings:
|
||||
"""Return the settings used when the wiki omits ``fiona.wiki.ingest``.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - table defaults from SPEC 5.1.
|
||||
|
||||
Returns:
|
||||
IngestSettings: Built-in limits and paths.
|
||||
"""
|
||||
return IngestSettings(
|
||||
timeout_seconds=30,
|
||||
max_download_bytes=52428800,
|
||||
user_agent=f"fiona-assistant/{package_version()}",
|
||||
default_target="sources",
|
||||
raw_categories={},
|
||||
)
|
||||
|
||||
|
||||
def load_ingest_settings(wiki_root: pathlib.Path) -> IngestSettings:
|
||||
"""Load ingest settings, falling back per key when a value is unusable.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - absent or mistyped keys keep the default.
|
||||
Modified by AI/LLM: 2026-10-07 - read ``login_url_patterns`` and ``html``.
|
||||
|
||||
Args:
|
||||
wiki_root: Directory that contains ``fiona-settings.json``.
|
||||
|
||||
Returns:
|
||||
IngestSettings: Merged settings for this wiki.
|
||||
"""
|
||||
settings = default_settings()
|
||||
path = wiki_root / "fiona-settings.json"
|
||||
if not path.is_file():
|
||||
return settings
|
||||
try:
|
||||
document = json.loads(path.read_text(encoding="utf-8"))
|
||||
except json.JSONDecodeError:
|
||||
return settings
|
||||
section = {}
|
||||
if isinstance(document, dict):
|
||||
wiki = document.get("fiona", {})
|
||||
if isinstance(wiki, dict):
|
||||
wiki = wiki.get("wiki", {})
|
||||
if isinstance(wiki, dict):
|
||||
raw = wiki.get("ingest", {})
|
||||
if isinstance(raw, dict):
|
||||
section = raw
|
||||
timeout = section.get("timeout_seconds", settings.timeout_seconds)
|
||||
if isinstance(timeout, (int, float)) and not isinstance(timeout, bool) and timeout > 0:
|
||||
settings.timeout_seconds = float(timeout)
|
||||
limit = section.get("max_download_bytes", settings.max_download_bytes)
|
||||
if isinstance(limit, int) and not isinstance(limit, bool) and limit > 0:
|
||||
settings.max_download_bytes = limit
|
||||
agent = section.get("user_agent")
|
||||
if isinstance(agent, str) and agent.strip():
|
||||
settings.user_agent = agent
|
||||
target = section.get("default_target")
|
||||
if isinstance(target, str) and target.strip() and ".." not in target.replace("\\", "/"):
|
||||
settings.default_target = target.strip().strip("/")
|
||||
categories = section.get("raw_categories")
|
||||
if isinstance(categories, dict):
|
||||
cleaned = {
|
||||
str(key): value
|
||||
for key, value in categories.items()
|
||||
if isinstance(value, str) and value.strip()
|
||||
}
|
||||
settings.raw_categories = cleaned
|
||||
patterns = section.get("login_url_patterns")
|
||||
if isinstance(patterns, list):
|
||||
settings.login_url_patterns = _compilable(patterns)
|
||||
html = section.get("html")
|
||||
if isinstance(html, dict):
|
||||
words = html.get("min_words")
|
||||
if isinstance(words, int) and not isinstance(words, bool) and words >= 1:
|
||||
settings.html_min_words = words
|
||||
density = html.get("max_link_density")
|
||||
if isinstance(density, (int, float)) and not isinstance(density, bool) and 0 <= density <= 1:
|
||||
settings.html_max_link_density = float(density)
|
||||
return settings
|
||||
|
||||
|
||||
def _compilable(patterns: list) -> list[str]:
|
||||
"""Return the entries that compile as regular expressions.
|
||||
|
||||
Generated by AI/LLM: 2026-10-07 - SPEC 5.1, one warning per skipped entry.
|
||||
"""
|
||||
kept: list[str] = []
|
||||
for pattern in patterns:
|
||||
try:
|
||||
re.compile(pattern)
|
||||
except (re.error, TypeError):
|
||||
logger.warning("login_url_patterns: skipped %r; not a valid regular expression", pattern)
|
||||
continue
|
||||
kept.append(pattern)
|
||||
return kept
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,361 +0,0 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>Wiki Graph Visualization</title>
|
||||
<script src="https://d3js.org/d3.v7.min.js"></script>
|
||||
<style>
|
||||
html, body {
|
||||
height: 100%;
|
||||
}
|
||||
body {
|
||||
font-family: var(--font, Arial, sans-serif);
|
||||
margin: 0;
|
||||
padding: 0.75em 1em;
|
||||
background-color: var(--page-bg, #f5f5f5);
|
||||
color: var(--text, #333333);
|
||||
box-sizing: border-box;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
}
|
||||
.container {
|
||||
max-width: 100%;
|
||||
margin: 0 auto;
|
||||
width: 100%;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
flex: 1;
|
||||
min-height: 0;
|
||||
}
|
||||
#graph-container {
|
||||
width: 100%;
|
||||
flex: 1;
|
||||
min-height: 20em;
|
||||
background-color: var(--graph-bg, #ffffff);
|
||||
border: 0.0625em solid #ddd;
|
||||
border-radius: 0.25em;
|
||||
overflow: hidden;
|
||||
}
|
||||
h1 {
|
||||
text-align: center;
|
||||
color: var(--text, #333333);
|
||||
}
|
||||
.controls {
|
||||
background-color: var(--card-bg, #ffffff);
|
||||
color: var(--text, #333333);
|
||||
padding: 0.9375em;
|
||||
border: 0.0625em solid #ddd;
|
||||
border-radius: 0.25em;
|
||||
margin-bottom: 0.9375em;
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: 0.625em;
|
||||
align-items: center;
|
||||
}
|
||||
.control-group {
|
||||
display: flex;
|
||||
gap: 0.625em;
|
||||
align-items: center;
|
||||
flex-wrap: wrap;
|
||||
}
|
||||
.control-group label {
|
||||
font-size: 0.875em;
|
||||
}
|
||||
input[type="text"], input[type="checkbox"], select, button {
|
||||
padding: 0.5em 0.75em;
|
||||
border: 0.0625em solid #ccc;
|
||||
border-radius: 0.25em;
|
||||
font-size: 0.875em;
|
||||
}
|
||||
button {
|
||||
background-color: var(--btn-bg, #007bff);
|
||||
color: var(--btn-text, #ffffff);
|
||||
border: none;
|
||||
cursor: pointer;
|
||||
border-radius: 0.25em;
|
||||
padding: 0.5em 1em;
|
||||
font-size: 0.875em;
|
||||
transition: background-color 0.3s;
|
||||
}
|
||||
button:hover {
|
||||
background-color: var(--btn-hover, #0056b3);
|
||||
}
|
||||
button:disabled {
|
||||
background-color: var(--btn-disabled, #6c757d);
|
||||
cursor: not-allowed;
|
||||
}
|
||||
|
||||
.toggle {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
gap: 0.5em;
|
||||
cursor: pointer;
|
||||
font-size: 0.875em;
|
||||
user-select: none;
|
||||
}
|
||||
.toggle input {
|
||||
position: absolute;
|
||||
opacity: 0;
|
||||
width: 0.0625em;
|
||||
height: 0.0625em;
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
}
|
||||
.toggle-track {
|
||||
width: 2.25em;
|
||||
height: 1.25em;
|
||||
background: #ccc;
|
||||
border-radius: 0.625em;
|
||||
position: relative;
|
||||
flex: 0 0 auto;
|
||||
}
|
||||
.toggle-track::after {
|
||||
content: "";
|
||||
position: absolute;
|
||||
width: 1em;
|
||||
height: 1em;
|
||||
border-radius: 50%;
|
||||
background: #fff;
|
||||
top: 0.125em;
|
||||
left: 0.125em;
|
||||
transition: transform 0.15s ease;
|
||||
}
|
||||
.toggle input:checked + .toggle-track {
|
||||
background: var(--btn-bg, #007bff);
|
||||
}
|
||||
.toggle input:checked + .toggle-track::after {
|
||||
transform: translateX(1em);
|
||||
}
|
||||
.toggle input:focus-visible + .toggle-track {
|
||||
outline: 0.125em solid #007bff;
|
||||
outline-offset: 0.125em;
|
||||
}
|
||||
|
||||
.node-info {
|
||||
position: absolute;
|
||||
background: var(--card-bg, #ffffff);
|
||||
color: var(--text, #333333);
|
||||
border: 0.0625em solid #999;
|
||||
padding: 0.5em;
|
||||
border-radius: 0.25em;
|
||||
pointer-events: none;
|
||||
font-size: 0.75em;
|
||||
z-index: 1000;
|
||||
box-shadow: 0 0.125em 0.25em rgba(0,0,0,0.1);
|
||||
}
|
||||
.edge-info {
|
||||
position: absolute;
|
||||
background: var(--card-bg, #ffffff);
|
||||
color: var(--text, #333333);
|
||||
border: 0.0625em solid #999;
|
||||
padding: 0.5em;
|
||||
border-radius: 0.25em;
|
||||
pointer-events: none;
|
||||
font-size: 0.75em;
|
||||
z-index: 1000;
|
||||
box-shadow: 0 0.125em 0.25em rgba(0,0,0,0.1);
|
||||
}
|
||||
.frontmatter-card {
|
||||
position: absolute;
|
||||
background: var(--card-bg, #ffffff);
|
||||
color: var(--text, #333333);
|
||||
border: 0.0625em solid #999;
|
||||
padding: 0.625em;
|
||||
border-radius: 0.25em;
|
||||
font-size: 0.75em;
|
||||
z-index: 1001;
|
||||
box-shadow: 0 0.125em 0.25em rgba(0,0,0,0.2);
|
||||
max-width: 26.25em;
|
||||
max-height: 70vh;
|
||||
overflow: auto;
|
||||
word-break: break-word;
|
||||
}
|
||||
.legend-card {
|
||||
position: absolute;
|
||||
background: var(--card-bg, #ffffff);
|
||||
color: var(--text, #333333);
|
||||
border: 0.0625em solid #999;
|
||||
padding: 0.625em 0.75em;
|
||||
border-radius: 0.25em;
|
||||
font-size: 0.75em;
|
||||
z-index: 1001;
|
||||
box-shadow: 0 0.125em 0.25em rgba(0,0,0,0.2);
|
||||
max-width: 15em;
|
||||
}
|
||||
.legend-card h2 {
|
||||
margin: 0 0 0.375em;
|
||||
font-size: 0.8125em;
|
||||
}
|
||||
.legend-heading {
|
||||
margin: 0.625em 0 0;
|
||||
font-size: 0.6875em;
|
||||
font-weight: 700;
|
||||
}
|
||||
.legend-row {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 0.5em;
|
||||
margin-top: 0.375em;
|
||||
}
|
||||
.legend-mark {
|
||||
width: 1.75em;
|
||||
height: 1.375em;
|
||||
flex: 0 0 auto;
|
||||
}
|
||||
.stats {
|
||||
background-color: var(--card-bg, #ffffff);
|
||||
color: var(--text, #333333);
|
||||
padding: 0.625em;
|
||||
border: 0.0625em solid #ddd;
|
||||
border-radius: 0.25em;
|
||||
margin-bottom: 0.9375em;
|
||||
}
|
||||
.stats span {
|
||||
margin-right: 1.25em;
|
||||
font-size: 0.875em;
|
||||
}
|
||||
.controls, .stats, #graph-container, .legend-card,
|
||||
.frontmatter-card, .node-info, .edge-info {
|
||||
border-color: var(--card-border, #999999);
|
||||
}
|
||||
@media (max-width: 48em) {
|
||||
.controls {
|
||||
flex-direction: column;
|
||||
align-items: stretch;
|
||||
}
|
||||
.control-group {
|
||||
flex-direction: column;
|
||||
align-items: stretch;
|
||||
}
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="container">
|
||||
<h1>Wiki Graph Visualization</h1>
|
||||
<div class="stats">
|
||||
<span>Nodes: <strong>{{node_count}}</strong></span>
|
||||
<span>Edges: <strong>{{edge_count}}</strong></span>
|
||||
<span>Max in-degree: <strong>{{max_in_degree}}</strong></span>
|
||||
</div>
|
||||
<div class="controls">
|
||||
<div class="control-group">
|
||||
<label for="theme-select">Theme:</label>
|
||||
<select id="theme-select" autocomplete="off"></select>
|
||||
</div>
|
||||
<div class="control-group">
|
||||
<label for="search-input">Search:</label>
|
||||
<input type="text" id="search-input" placeholder="Filter nodes..." autocomplete="off" />
|
||||
</div>
|
||||
<div class="control-group">
|
||||
<button type="button" id="fit-button">See all</button>
|
||||
</div>
|
||||
<div class="control-group">
|
||||
<label class="toggle" for="orphan-toggle">
|
||||
<input type="checkbox" id="orphan-toggle" checked autocomplete="off" />
|
||||
<span class="toggle-track" aria-hidden="true"></span>
|
||||
Show Orphan Nodes
|
||||
</label>
|
||||
</div>
|
||||
<div class="control-group">
|
||||
<label class="toggle" for="external-toggle">
|
||||
<input type="checkbox" id="external-toggle" {{external_checked}} autocomplete="off" />
|
||||
<span class="toggle-track" aria-hidden="true"></span>
|
||||
Show External Links
|
||||
</label>
|
||||
</div>
|
||||
<div class="control-group">
|
||||
<label class="toggle" for="source-toggle">
|
||||
<input type="checkbox" id="source-toggle" {{source_checked}} autocomplete="off" />
|
||||
<span class="toggle-track" aria-hidden="true"></span>
|
||||
Show Sources
|
||||
</label>
|
||||
</div>
|
||||
<div class="control-group">
|
||||
<label class="toggle" for="subtitle-toggle">
|
||||
<input type="checkbox" id="subtitle-toggle" {{subtitle_checked}} autocomplete="off" />
|
||||
<span class="toggle-track" aria-hidden="true"></span>
|
||||
Show Subtitles
|
||||
</label>
|
||||
</div>
|
||||
<div class="control-group">
|
||||
<label class="toggle" for="legend-toggle">
|
||||
<input type="checkbox" id="legend-toggle" {{legend_checked}} autocomplete="off" />
|
||||
<span class="toggle-track" aria-hidden="true"></span>
|
||||
Show Legend
|
||||
</label>
|
||||
</div>
|
||||
<div class="control-group">
|
||||
<label class="toggle" for="arrow-toggle">
|
||||
<input type="checkbox" id="arrow-toggle" {{arrow_checked}} autocomplete="off" />
|
||||
<span class="toggle-track" aria-hidden="true"></span>
|
||||
Show Arrows
|
||||
</label>
|
||||
</div>
|
||||
</div>
|
||||
<div id="graph-container"></div>
|
||||
</div>
|
||||
<div class="node-info" style="display: none;"></div>
|
||||
<div class="edge-info" style="display: none;"></div>
|
||||
<div id="frontmatter-card" class="frontmatter-card" style="display: none;"></div>
|
||||
<div id="legend-card" class="legend-card" style="display: {{legend_display}};">
|
||||
<h2>Legend</h2>
|
||||
<div class="legend-heading">Shape</div>
|
||||
<div class="legend-row">
|
||||
<svg id="legend-shape-page" class="legend-mark" viewBox="-16 -14 32 28" aria-hidden="true">
|
||||
<circle r="7" fill="#333333"></circle>
|
||||
</svg>
|
||||
<span>Page</span>
|
||||
</div>
|
||||
<div class="legend-row">
|
||||
<svg id="legend-shape-external" class="legend-mark" viewBox="-16 -14 32 28" aria-hidden="true">
|
||||
<text text-anchor="middle" dominant-baseline="central" font-size="18" font-family="Segoe UI Symbol, sans-serif" fill="#333333">☁︎</text>
|
||||
</svg>
|
||||
<span>External link</span>
|
||||
</div>
|
||||
<div class="legend-heading">Files pointing here</div>
|
||||
<div class="legend-row">
|
||||
<svg class="legend-mark" viewBox="-16 -14 32 28" aria-hidden="true">
|
||||
<circle id="legend-node-low" r="4" fill="#8f8272"></circle>
|
||||
</svg>
|
||||
<span>Few</span>
|
||||
</div>
|
||||
<div class="legend-row">
|
||||
<svg class="legend-mark" viewBox="-16 -14 32 28" aria-hidden="true">
|
||||
<circle id="legend-node-mid" r="7" fill="#e0943a"></circle>
|
||||
</svg>
|
||||
<span>Some</span>
|
||||
</div>
|
||||
<div class="legend-row">
|
||||
<svg class="legend-mark" viewBox="-16 -14 32 28" aria-hidden="true">
|
||||
<circle id="legend-node-high" r="11" fill="#c23b4a"></circle>
|
||||
</svg>
|
||||
<span>Many</span>
|
||||
</div>
|
||||
<div class="legend-heading">Same link, repeated</div>
|
||||
<div class="legend-row">
|
||||
<svg class="legend-mark" viewBox="0 0 28 12" aria-hidden="true">
|
||||
<line id="legend-edge-low" x1="2" y1="6" x2="26" y2="6" stroke="#a89884" stroke-width="3"></line>
|
||||
</svg>
|
||||
<span>Linked once</span>
|
||||
</div>
|
||||
<div class="legend-row">
|
||||
<svg class="legend-mark" viewBox="0 0 28 12" aria-hidden="true">
|
||||
<line id="legend-edge-mid" x1="2" y1="6" x2="26" y2="6" stroke="#b86a1a" stroke-width="3"></line>
|
||||
</svg>
|
||||
<span>A few times</span>
|
||||
</div>
|
||||
<div class="legend-row">
|
||||
<svg class="legend-mark" viewBox="0 0 28 12" aria-hidden="true">
|
||||
<line id="legend-edge-high" x1="2" y1="6" x2="26" y2="6" stroke="#8e2436" stroke-width="3"></line>
|
||||
</svg>
|
||||
<span>Many times</span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
{{graph_physics_js}}
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -7,14 +7,12 @@ from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import pathlib
|
||||
from datetime import UTC, datetime
|
||||
from datetime import datetime, timezone
|
||||
|
||||
import git
|
||||
from git.repo.base import Repo
|
||||
|
||||
from fiona.gittool import diff_report
|
||||
from fiona.logging_config import setup_logging
|
||||
from fiona.wiki.ignore import ensure_wiki_gitignore
|
||||
|
||||
|
||||
class GitTool:
|
||||
@@ -52,7 +50,7 @@ class GitTool:
|
||||
Returns:
|
||||
str: Formatted timestamp string.
|
||||
"""
|
||||
return datetime.now(tz=UTC).strftime("%Y-%m-%d %H:%M:%S")
|
||||
return datetime.now(tz=timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
|
||||
|
||||
@staticmethod
|
||||
def _get_tag_timestamp() -> str:
|
||||
@@ -64,7 +62,7 @@ class GitTool:
|
||||
Returns:
|
||||
str: Formatted timestamp string safe for git tags.
|
||||
"""
|
||||
return datetime.now(tz=UTC).strftime("%Y-%m-%d_%H-%M-%S")
|
||||
return datetime.now(tz=timezone.utc).strftime("%Y-%m-%d_%H-%M-%S")
|
||||
|
||||
def _is_working_tree_clean(self) -> bool:
|
||||
"""Check if the working tree is clean.
|
||||
@@ -75,7 +73,7 @@ class GitTool:
|
||||
# Check for modified/staged files
|
||||
if self.repo.is_dirty():
|
||||
return False
|
||||
|
||||
|
||||
# Check for untracked files
|
||||
untracked = self.repo.untracked_files
|
||||
return len(untracked) == 0
|
||||
@@ -172,7 +170,7 @@ class GitTool:
|
||||
tag for tag in self.repo.tags
|
||||
if tag.name.endswith(f"_{tag_name}") or tag.name == tag_name
|
||||
]
|
||||
|
||||
|
||||
if matching_tags:
|
||||
# Sort by commit date descending and take the most recent
|
||||
matching_tags.sort(
|
||||
@@ -180,7 +178,7 @@ class GitTool:
|
||||
reverse=True
|
||||
)
|
||||
tag_ref = matching_tags[0]
|
||||
|
||||
|
||||
if tag_ref is None:
|
||||
raise git.GitCommandError(
|
||||
f"Tag '{tag_name}' does not exist."
|
||||
@@ -191,7 +189,7 @@ class GitTool:
|
||||
|
||||
# Restore the tag's tree into the working directory and index
|
||||
self.repo.git.checkout(tag_ref.commit, "--", ".")
|
||||
|
||||
|
||||
# Create a new commit with the restored state
|
||||
new_commit = self.repo.index.commit(restore_message)
|
||||
|
||||
@@ -274,9 +272,9 @@ class GitTool:
|
||||
|
||||
@staticmethod
|
||||
def init_wiki_repo(repo_path: str | pathlib.Path, initial_message: str = "Initial llm-wiki commit") -> git.Repo:
|
||||
"""Initialize a git repository for llm-wiki and seed .gitignore.
|
||||
"""Initialize a git repository for llm-wiki with .gitignore for personal/.
|
||||
|
||||
Puts or appends the repository ignore patterns plus ``personal/``,
|
||||
Creates a .gitignore file that excludes the personal/ directory,
|
||||
stages all files, and makes the initial commit.
|
||||
|
||||
Args:
|
||||
@@ -290,23 +288,26 @@ class GitTool:
|
||||
git.InvalidGitRepositoryError: If git initialization fails.
|
||||
"""
|
||||
repo_path = pathlib.Path(repo_path).resolve()
|
||||
|
||||
|
||||
# Initialize the repo
|
||||
repo = GitTool.init_repo(repo_path)
|
||||
|
||||
# AI/LLM: 2026-09-30 - seed .gitignore from the repository ignore file.
|
||||
ensure_wiki_gitignore(repo_path)
|
||||
|
||||
|
||||
# Create .gitignore to exclude personal/ directory
|
||||
gitignore_path = repo_path / ".gitignore"
|
||||
with open(gitignore_path, "w", encoding="utf-8") as f:
|
||||
f.write("# Exclude personal directory from git as per SPEC 3.2.0.2\n")
|
||||
f.write("personal/\n")
|
||||
|
||||
# Add all files and make initial commit
|
||||
repo.git.add("--all")
|
||||
repo.index.commit(initial_message)
|
||||
|
||||
|
||||
return repo
|
||||
|
||||
|
||||
def main(): # pragma: no cover
|
||||
"""Command-line interface for the GitTool.
|
||||
|
||||
|
||||
Note: CLI is tested via integration tests through fiona-assistant git commands.
|
||||
"""
|
||||
parser = argparse.ArgumentParser(
|
||||
@@ -350,30 +351,6 @@ def main(): # pragma: no cover
|
||||
# status subcommand
|
||||
subparsers.add_parser("status", help="Show git status")
|
||||
|
||||
# diff subcommand
|
||||
diff_parser = subparsers.add_parser(
|
||||
"diff",
|
||||
help="Write constellation-themed HTML diff under wiki tmp/",
|
||||
)
|
||||
diff_parser.add_argument("--base", help="Base revision (default: origin/main or main/master)")
|
||||
diff_parser.add_argument("--head", default="HEAD", help="Head revision (default: HEAD)")
|
||||
diff_parser.add_argument("--title", help="Report H1 / page title")
|
||||
diff_parser.add_argument(
|
||||
"--subtitle",
|
||||
help="Extra header line (e.g. temporary review before push)",
|
||||
)
|
||||
diff_parser.add_argument(
|
||||
"-o",
|
||||
"--output",
|
||||
type=pathlib.Path,
|
||||
help="Output HTML path (default: <wiki>/tmp/<repo>-git-diff-<short>.html)",
|
||||
)
|
||||
diff_parser.add_argument(
|
||||
"--collapsed",
|
||||
action="store_true",
|
||||
help="Start with per-file sections collapsed",
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
setup_logging()
|
||||
@@ -403,21 +380,6 @@ def main(): # pragma: no cover
|
||||
elif args.subcommand == "status":
|
||||
status = git_tool.status()
|
||||
print(status)
|
||||
elif args.subcommand == "diff":
|
||||
diff_argv: list[str] = []
|
||||
if args.base:
|
||||
diff_argv.extend(["--base", args.base])
|
||||
if args.head != "HEAD":
|
||||
diff_argv.extend(["--head", args.head])
|
||||
if args.title:
|
||||
diff_argv.extend(["--title", args.title])
|
||||
if args.subtitle:
|
||||
diff_argv.extend(["--subtitle", args.subtitle])
|
||||
if args.output is not None:
|
||||
diff_argv.extend(["--output", str(args.output)])
|
||||
if args.collapsed:
|
||||
diff_argv.append("--collapsed")
|
||||
raise SystemExit(diff_report.main(diff_argv))
|
||||
except Exception as e:
|
||||
print(f"Error: {e}")
|
||||
raise SystemExit(1) from e
|
||||
@@ -18,18 +18,6 @@ import re
|
||||
|
||||
import yaml
|
||||
|
||||
# Default required fields for wiki pages
|
||||
DEFAULT_REQUIRED_FIELDS = ["title", "created", "updated", "type"]
|
||||
|
||||
# Pattern to match frontmatter delimiters
|
||||
# This pattern matches --- at start, then any content (including empty), then ---
|
||||
# Handles: ---\ncontent\n--- or ---\n--- or ---\n\n---
|
||||
FRONTMATTER_PATTERN = re.compile(r'^---\s*\n([\s\S]*?)(?:\n---\s*($|\n)|---\s*($|\n))')
|
||||
|
||||
# Pattern to match a single --- delimiter line
|
||||
DELIMITER_PATTERN = re.compile(r'^---\s*$', re.MULTILINE)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Exceptions
|
||||
# =============================================================================
|
||||
@@ -42,6 +30,22 @@ class RequiredFieldError(Exception):
|
||||
"""Exception raised when required frontmatter fields are missing."""
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Constants
|
||||
# =============================================================================
|
||||
|
||||
# Default required fields for wiki pages
|
||||
DEFAULT_REQUIRED_FIELDS = ["title", "created", "updated", "type"]
|
||||
|
||||
# Pattern to match frontmatter delimiters
|
||||
# This pattern matches --- at start, then any content (including empty), then ---
|
||||
# Handles: ---\ncontent\n--- or ---\n--- or ---\n\n---
|
||||
FRONTMATTER_PATTERN = re.compile(r'^---\s*\n([\s\S]*?)(?:\n---\s*($|\n)|---\s*($|\n))')
|
||||
|
||||
# Pattern to match a single --- delimiter line
|
||||
DELIMITER_PATTERN = re.compile(r'^---\s*$', re.MULTILINE)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Helper Functions
|
||||
# =============================================================================
|
||||
@@ -16,9 +16,8 @@ import re
|
||||
import sys
|
||||
import typing
|
||||
|
||||
from fiona.librarian.md_frontmatter import FrontmatterError, get_file_version
|
||||
from fiona.logging_config import load_settings, setup_logging
|
||||
from fiona.paths import llm_wiki_template
|
||||
from librarian.md_frontmatter import FrontmatterError, get_file_version
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -41,7 +40,9 @@ SETTINGS_NAME = "fiona-settings.json"
|
||||
ARTIFACT_DIR_NAMES = frozenset({"build", "dist", "__pycache__"})
|
||||
FILE_TOC_KEYS = frozenset({"min_heading", "max_heading", "sectionize", "ignore_files"})
|
||||
|
||||
DEFAULT_SETTINGS_FILE = llm_wiki_template() / "fiona-settings.json"
|
||||
DEFAULT_SETTINGS_FILE = (
|
||||
pathlib.Path(__file__).resolve().parent.parent / "llm-wiki" / "fiona-settings.json"
|
||||
)
|
||||
|
||||
|
||||
def load_toc_settings(settings_file: str | pathlib.Path | None = None) -> dict:
|
||||
@@ -94,12 +95,9 @@ def depth_setting(settings: dict, key: str) -> int | None:
|
||||
|
||||
|
||||
def slugify(text: str) -> str:
|
||||
"""Convert a header text into a GitHub-style anchor (SPEC 7.3.1.1.3).
|
||||
|
||||
Each space becomes one hyphen; runs are kept so ``a | b`` is ``a--b``.
|
||||
"""
|
||||
"""Convert a header text into a GitHub-style anchor."""
|
||||
slug = re.sub(r"[^\w\s-]", "", text.lower())
|
||||
return re.sub(r"\s", "-", slug.strip())
|
||||
return re.sub(r"\s+", "-", slug.strip())
|
||||
|
||||
|
||||
def excluded(text: str, patterns: list[typing.Pattern]) -> bool:
|
||||
@@ -111,7 +109,7 @@ def read_headers(path: pathlib.Path) -> list[tuple[int, str]]:
|
||||
"""Return (level, text) for every header in a Markdown file, skipping code fences."""
|
||||
headers: list[tuple[int, str]] = []
|
||||
in_fence = False
|
||||
# AI/LLM: utf-8-sig drops a leading BOM that would hide the first header.
|
||||
# CURSOR: utf-8-sig drops a leading BOM that would hide the first header.
|
||||
for line in path.read_text(encoding="utf-8-sig", errors="replace").splitlines():
|
||||
if FENCE_RE.match(line):
|
||||
in_fence = not in_fence
|
||||
@@ -165,19 +163,15 @@ def build_toc(
|
||||
) -> str:
|
||||
"""Build a table of contents that nests folders, files and their headers.
|
||||
|
||||
Generated by AI/LLM: 2026-10-02 - frontmatter name and version for fmlint.
|
||||
|
||||
Args:
|
||||
folder: The folder the table of contents covers.
|
||||
files: Relative md file paths to list (as returned by collect_md).
|
||||
max_header_depth: Maximum header level to list per file
|
||||
(3 = h1 through h3). None or 0 mean unlimited.
|
||||
"""
|
||||
# AI/LLM: name and version are required by fmlint on every markdown file.
|
||||
today = datetime.datetime.now(datetime.UTC).date().isoformat()
|
||||
# Add YAML frontmatter
|
||||
today = datetime.datetime.now(datetime.timezone.utc).date().isoformat()
|
||||
frontmatter = f"""---
|
||||
name: Table of Contents
|
||||
version: 1.0.0
|
||||
title: Table of Contents
|
||||
created: {today}
|
||||
updated: {today}
|
||||
@@ -189,7 +183,7 @@ type: toc
|
||||
for name in files:
|
||||
parts = tuple(name.split("/"))
|
||||
folders = parts[:-1]
|
||||
# AI/LLM: emit a bullet only for folder levels that differ from the previous entry.
|
||||
# CURSOR: emit a bullet only for folder levels that differ from the previous entry.
|
||||
for depth in range(len(folders)):
|
||||
if folders[: depth + 1] != previous[: depth + 1]:
|
||||
lines.append("{}- {}".format(" " * depth, folders[depth]))
|
||||
@@ -221,7 +215,7 @@ class FileTocError(Exception):
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class FileTocConfig:
|
||||
"""Parameters for per-file TOC generation (SPEC 7.3.1.2).
|
||||
"""Parameters for per-file TOC generation (SPEC 4.8.2).
|
||||
|
||||
Attributes:
|
||||
min_heading: Smallest heading level included (never below 2;
|
||||
@@ -366,7 +360,7 @@ def merge_marker_config(lines: list[str], base: FileTocConfig) -> FileTocConfig:
|
||||
"""Apply the existing marker's params over the base config.
|
||||
|
||||
The marker records this TOC's own settings and has priority
|
||||
(SPEC 7.3.1.2): a `sectionized` marker keeps section numbering on
|
||||
(SPEC 4.8.2): a `sectionized` marker keeps section numbering on
|
||||
and the recorded heading range sticks, so regenerations keep the
|
||||
TOC's identity regardless of settings changes. Fields the marker
|
||||
does not record (bare legacy markers, plain markers without
|
||||
@@ -680,7 +674,7 @@ def _file_toc_star(sectionized: bool) -> None:
|
||||
|
||||
|
||||
def file_toc_main(argv: list[str]) -> None:
|
||||
"""CLI for per-file TOC generation (SPEC 7.3.1.2)."""
|
||||
"""CLI for per-file TOC generation (SPEC 4.8.2)."""
|
||||
parser = argparse.ArgumentParser(
|
||||
prog="toc_generator file",
|
||||
description="Generate an in-file table of contents below the H1 heading.",
|
||||
@@ -754,7 +748,7 @@ def main() -> None:
|
||||
for folder in target_folders(args.directory, patterns, args.recursive_toc):
|
||||
files = collect_md(folder, patterns, not args.non_recursive, max_depth=max_dir_depth)
|
||||
if files:
|
||||
# AI/LLM: newline="\n" keeps Unix line endings on every platform.
|
||||
# CURSOR: newline="\n" keeps Unix line endings on every platform.
|
||||
with open(folder / TOC_NAME, "w", encoding="utf-8", newline="\n") as toc:
|
||||
toc.write(build_toc(folder, files, max_header_depth=max_header_depth))
|
||||
logger.info("wrote %s (%d file(s))", folder / TOC_NAME, len(files))
|
||||
@@ -0,0 +1,77 @@
|
||||
---
|
||||
name: llm-wiki-agent-rules
|
||||
title: LLM Wiki Agent Rules
|
||||
version: 1.1.1
|
||||
created: 2025-09-23
|
||||
updated: 2026-09-27
|
||||
type: summary
|
||||
---
|
||||
|
||||
# LLM Wiki Agent Rules
|
||||
|
||||
- [1. Pre-requisites](#1-pre-requisites)
|
||||
- [1.1 Install fiona-assistant library](#11-install-fiona-assistant-library)
|
||||
- [1.2 fiona-assistant usage](#12-fiona-assistant-usage)
|
||||
- [2. Directory Permissions](#2-directory-permissions)
|
||||
- [3. Linking Rules](#3-linking-rules)
|
||||
- [4. Updates](#4-updates)
|
||||
- [5. Skills](#5-skills)
|
||||
- [6. Boundaries](#6-boundaries)
|
||||
|
||||
## 1. Pre-requisites
|
||||
|
||||
### 1.1 Install fiona-assistant library
|
||||
|
||||
- Install with pip
|
||||
- `python -m pip install fiona-assistant`
|
||||
|
||||
### 1.2 fiona-assistant usage
|
||||
|
||||
> TODO: add here how to use fiona-assistant for LLMs
|
||||
|
||||
## 2. Directory Permissions
|
||||
|
||||
- **`/personal/`**: Strictly **READ-ONLY**. Never create, edit, or delete notes here.
|
||||
- **`llm-wiki/ingest/`**: Input processing zone. Parse files dropped here into
|
||||
concept notes.
|
||||
- **`llm-wiki/wiki/`**: **READ-WRITE**. Place synthesized concept pages in `/wiki/concepts/`.
|
||||
- Only parsed/synthesized content belongs in `/wiki/concepts/`.
|
||||
- **`llm-wiki/raw/`**: **READ-ONLY**. Raw source archive, never edited.
|
||||
- **Raw files** (e.g., articles, papers, transcripts) must be stored in
|
||||
`/raw/` and **never edited**.
|
||||
- Subdirectories:
|
||||
- `/raw/articles/` (web articles, clippings)
|
||||
- `/raw/papers/` (PDFs, arXiv papers)
|
||||
- `/raw/transcripts/` (meeting notes, interviews)
|
||||
- `/raw/assets/` (images, diagrams referenced by sources)
|
||||
|
||||
- **`llm-wiki/utils/scripts`**: **READ-WRITE**. Place any script used and
|
||||
created here.
|
||||
|
||||
## 3. Linking Rules
|
||||
|
||||
- Format concept note titles in **lowercase** or **Title-Case** to match page headings.
|
||||
- Use standard markdown links for graph connectivity.
|
||||
|
||||
## 4. Updates
|
||||
|
||||
- Maintain a **master index** in `/wiki/index.md` whenever new concepts are synthesized.
|
||||
- Append all actions to `/wiki/log.md` for chronological tracking.
|
||||
|
||||
## 5. Skills
|
||||
|
||||
- Skills are stored in `utils/skills/`.
|
||||
- **`llm-wiki`** is the primary skill for maintaining this wiki.
|
||||
- Learn more: [llm-wiki skill](./utils/skills/fiona/SKILL.md).
|
||||
|
||||
## 6. Boundaries
|
||||
|
||||
- **Do:**
|
||||
- Follow directory permissions strictly.
|
||||
- Update `index.md` and `log.md` for every change.
|
||||
- Use provided skills for consistency.
|
||||
|
||||
- **Don’t:**
|
||||
- Modify `/personal/` under any circumstances.
|
||||
- Deviate from linking or naming conventions.
|
||||
- Assume user intent without confirmation.
|
||||
@@ -0,0 +1,4 @@
|
||||
# llm-wiki package
|
||||
# This package contains templates and utilities for managing the LLM wiki.
|
||||
|
||||
__version__ = "0.1.0"
|
||||
@@ -1,5 +1,4 @@
|
||||
{
|
||||
"version": "1.0.1",
|
||||
"markdown": {
|
||||
"pymarkdown": {
|
||||
"extensions": [
|
||||
@@ -59,32 +58,20 @@
|
||||
},
|
||||
"wiki": {
|
||||
"graph": {
|
||||
"theme": "dark",
|
||||
"show_external_links": true,
|
||||
"subtitle": {
|
||||
"enabled": true
|
||||
"node_colors": {
|
||||
"low": "#808080",
|
||||
"mid": "#ffa500",
|
||||
"high": "#FF0000"
|
||||
},
|
||||
"show_legend": true,
|
||||
"show_arrows": false,
|
||||
"show_source": true,
|
||||
"show_glow": true,
|
||||
"show_link_card": true,
|
||||
"first_fit_ms": 12000,
|
||||
"hover_link_tail_chars": 12,
|
||||
"simplify_external_links": true,
|
||||
"consolidate_external_links": true,
|
||||
"consolidate_external_links_depth": 3,
|
||||
"consolidate_external_links_max_char": 8,
|
||||
"ignore_files": [
|
||||
"*/table_of_contents.md",
|
||||
"*/README.md",
|
||||
"*/AGENTS.md"
|
||||
]
|
||||
},
|
||||
"update": {
|
||||
"ignore_files": [
|
||||
"corefiles.json"
|
||||
]
|
||||
"node_radius": {
|
||||
"min": 5,
|
||||
"max": 40
|
||||
},
|
||||
"hover": {
|
||||
"color": "#add8e6",
|
||||
"edge_color": "#0000ff",
|
||||
"size_multiplier": 1.3
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
---
|
||||
name: map
|
||||
version: 1.0.0
|
||||
title: Map Script
|
||||
type: summary
|
||||
created: 2025-09-23
|
||||
updated: 2026-09-24
|
||||
---
|
||||
|
||||
# Map Script
|
||||
|
||||
Mapping utility script.
|
||||
@@ -0,0 +1,12 @@
|
||||
---
|
||||
name: skills
|
||||
version: 1.0.0
|
||||
title: Skills Directory
|
||||
type: summary
|
||||
created: 2025-09-23
|
||||
updated: 2026-09-24
|
||||
---
|
||||
|
||||
# Skills Directory
|
||||
|
||||
Skill definitions and configurations.
|
||||
@@ -0,0 +1,71 @@
|
||||
---
|
||||
name: fiona
|
||||
category: productivity
|
||||
description: LLM-Wiki Maintenance Assistant - Open Source Edition
|
||||
author: nvaldez
|
||||
license: MIT
|
||||
version: 1.0.1
|
||||
---
|
||||
|
||||
# FIONA-ASSISTANT Skill
|
||||
|
||||
**F**ocused **I**ntelligence **O**rganizer for **N**otes and **A**nalysis
|
||||
|
||||
The open-source counterpart to NADIA, designed for knowledge organization,
|
||||
technical documentation, and information synthesis.
|
||||
FIONA-ASSISTANT provides a community-driven approach to intelligent note management.
|
||||
|
||||
---
|
||||
|
||||
## Core Identity
|
||||
|
||||
You are **FIONA-ASSISTANT**, the **LLM-Wiki Maintenance Assistant**
|
||||
specializing in:
|
||||
|
||||
- **F**ocused wiki organization
|
||||
- **I**ntelligent note structuring
|
||||
- **O**pen knowledge base management
|
||||
- **N**avigation of knowledge graphs
|
||||
- **A**utomated wiki maintenance
|
||||
|
||||
Your philosophy is:
|
||||
> "A well-maintained wiki is the foundation of collective intelligence.
|
||||
> I ensure knowledge is organized, accessible, and always up-to-date
|
||||
> through open standards and community collaboration."
|
||||
|
||||
## LLM-Wiki Maintenance Approach
|
||||
|
||||
FIONA-ASSISTANT knows basic [LLM-Wiki](./llm-wiki-mod.md) Management
|
||||
FIONA-ASSISTANT maintains LLM-wikis with strict adherence to standard Markdown rules:
|
||||
|
||||
1. **Standard Markdown Compliance**:
|
||||
- Follows [markdownlint rules](./markdown_rules.md)
|
||||
- Uses only standard Markdown syntax (`[text](path.md)` links, not wikilinks)
|
||||
- Maintains consistent heading hierarchy (MD001, MD002)
|
||||
- Enforces line length limits (MD013)
|
||||
- Ensures proper list indentation (MD005, MD007)
|
||||
|
||||
2. **Open Knowledge Standards**:
|
||||
- Uses YAML frontmatter for metadata (MD041)
|
||||
- Standard Markdown tables (MD048)
|
||||
- Fenced code blocks with language specification (MD046)
|
||||
- Proper emphasis syntax (MD049, MD050)
|
||||
|
||||
3. **Obsidian Compatibility Layer**:
|
||||
- Provides conversion tools between standard Markdown and Obsidian-specific formats
|
||||
- Maintains compatibility without requiring proprietary extensions
|
||||
- Preserves knowledge integrity when moving between tools
|
||||
|
||||
4. **Automated Markdown Validation**:
|
||||
- Continuous linting with markdownlint
|
||||
- Automatic correction of common Markdown issues
|
||||
- Link validation and repair (MD052)
|
||||
- Orphaned page detection
|
||||
- Consistent formatting enforcement
|
||||
|
||||
5. **Knowledge Graph Integrity**:
|
||||
- Ensures all links follow standard Markdown format
|
||||
- Maintains consistent tagging and categorization
|
||||
- Visualizes connections using standard Markdown-compatible tools
|
||||
|
||||
---
|
||||
@@ -0,0 +1,15 @@
|
||||
---
|
||||
name: llm-wiki-mod
|
||||
description: Modified Karpathy's LLM Wiki for FIONA
|
||||
category: research
|
||||
version: 1.0.0
|
||||
---
|
||||
|
||||
# Karpathy's LLM Wiki
|
||||
|
||||
Build and maintain a persistent, compounding knowledge base as interlinked
|
||||
markdown files.
|
||||
Based on [Andrej Karpathy's LLM Wiki pattern](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f).
|
||||
|
||||
Unlike traditional RAG (which rediscovers knowledge from scratch per query),
|
||||
the wiki compiles knowledge once and keeps it current. Cross-references are already there.
|
||||
@@ -0,0 +1,97 @@
|
||||
---
|
||||
name: obsidian
|
||||
description: Read, search, create, and edit notes in the Obsidian vault.
|
||||
version: 1.0.0
|
||||
author: Teknium (teknium1), Hermes Agent
|
||||
license: MIT
|
||||
platforms: [linux, macos, windows]
|
||||
metadata:
|
||||
hermes:
|
||||
tags: [Obsidian, Notes, Markdown, Vault]
|
||||
related_skills: []
|
||||
---
|
||||
|
||||
# Obsidian Vault
|
||||
|
||||
Use this skill for filesystem-first Obsidian vault work: reading notes, listing
|
||||
notes, searching note files, creating notes, appending content, and adding
|
||||
wikilinks.
|
||||
|
||||
## Vault path
|
||||
|
||||
Use a known or resolved vault path before calling file tools.
|
||||
|
||||
The documented vault-path convention is the `OBSIDIAN_VAULT_PATH` environment
|
||||
variable, for example from `${HERMES_HOME:-~/.hermes}/.env`. If it is unset,
|
||||
use `~/Documents/Obsidian Vault`.
|
||||
|
||||
File tools do not expand shell variables. Do not pass paths containing
|
||||
`$OBSIDIAN_VAULT_PATH` to `read_file`, `write_file`, `patch`, or
|
||||
`search_files`; resolve the vault path first and pass a concrete absolute path.
|
||||
Vault paths may contain spaces, which is another reason to prefer file tools
|
||||
over shell commands.
|
||||
|
||||
If the vault path is unknown, `terminal` is acceptable for resolving
|
||||
`OBSIDIAN_VAULT_PATH` or checking whether the fallback path exists. Once the
|
||||
path is known, switch back to file tools.
|
||||
|
||||
## Read a note
|
||||
|
||||
Use `read_file` with the resolved absolute path to the note. Prefer this over
|
||||
`cat` because it provides line numbers and pagination.
|
||||
|
||||
## List notes
|
||||
|
||||
Use `search_files` with `target: "files"` and the resolved vault path. Prefer
|
||||
this over `find` or `ls`.
|
||||
|
||||
- To list all markdown notes, use `pattern: "*.md"` under the vault path.
|
||||
- To list a subfolder, search under that subfolder's absolute path.
|
||||
|
||||
## Search
|
||||
|
||||
Use `search_files` for both filename and content searches. Prefer this over
|
||||
`grep`, `find`, or `ls`.
|
||||
|
||||
- For filenames, use `search_files` with `target: "files"` and a filename `pattern`.
|
||||
- For note contents, use `search_files` with `target: "content"`, the content
|
||||
regex as `pattern`, and `file_glob: "*.md"` when you want to restrict matches
|
||||
to markdown notes.
|
||||
|
||||
## Create a note
|
||||
|
||||
Use `write_file` with the resolved absolute path and the full markdown content.
|
||||
Prefer this over shell heredocs or `echo` because it avoids shell quoting
|
||||
issues and returns structured results.
|
||||
|
||||
## Append to a note
|
||||
|
||||
Prefer a native file-tool workflow when it is not awkward:
|
||||
|
||||
- Read the target note with `read_file`.
|
||||
- Use `patch` for an anchored append when there is stable context, such as
|
||||
adding a section after an existing heading or appending before a known
|
||||
trailing block.
|
||||
- Use `write_file` when rewriting the whole note is clearer than constructing a
|
||||
fragile patch.
|
||||
|
||||
For an anchored append with `patch`, replace the anchor with the anchor plus
|
||||
the new content.
|
||||
|
||||
For a simple append with no stable context, `terminal` is acceptable if it is
|
||||
the clearest safe option.
|
||||
|
||||
## Targeted edits
|
||||
|
||||
Use `patch` for focused note changes when the current content gives you stable
|
||||
context. Prefer this over shell text rewriting.
|
||||
|
||||
## Linking
|
||||
|
||||
For **compatibility** across tools (e.g., GitHub, VS Code, Obsidian), use
|
||||
**standard Markdown links** (`[link text](path/to/note.md)`) instead of
|
||||
wikilinks (`[[Note Name]]`).
|
||||
|
||||
- **Markdown links** ensure portability and work in all markdown renderers.
|
||||
- If Obsidian-specific features (e.g., graph view) are needed, wikilinks can be
|
||||
used **in addition to** Markdown links, but never as a replacement.
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user