ingest implemented

This commit is contained in:
nvaldez
2026-10-01 07:04:55 +02:00
parent 2d08b71b55
commit 482b2cb2e0
25 changed files with 4095 additions and 143 deletions
+57 -23
View File
@@ -1,6 +1,6 @@
---
name: Fiona-Assistant Specification
version: 0.97.0
version: 0.99.0
title: Project Spec Sheet
created: 2026-09-23
updated: 2026-09-30
@@ -99,7 +99,10 @@ yield to the owner's markdown and write `<stem>-update.md` instead
Revision 0.96.0 compares a file without a version by SHA-256 and
shows both short hashes in the update report (*7.2.12.*).
Revision 0.97.0 rejects a module-level assignment that follows the
first function or class (*6.4.3.*).
first function or class (*6.4.3.*). Revision 0.98.0 specifies that
`wiki graph` omits a file matched by `.gitignore` (*7.4.17.*) and
leaves that requirement unbuilt. Revision 0.99.0 builds `wiki ingest`
(*7.25.*).
Each feature section starts with a `Status` line. Requirements use
`- *N.M.K.*`. Each `*acceptance*` sits under the requirement it tests.
@@ -277,7 +280,6 @@ utils/skills/README.md
utils/skills/fiona/SKILL.md
utils/skills/fiona/markdown_rules.md
utils/skills/fiona/llm-wiki-mod.md
utils/skills/obsidian/SKILL.md
utils/skills/python-tdd/SKILL.md
utils/personas/default/SOUL.md
utils/personas/template/SOUL.md
@@ -732,7 +734,7 @@ extra untimed `coverage:` row under pytests. Details are section 7.16.
### 4.14. wiki ingest
Not built yet. The contract is section 7.25.
The contract is section 7.25.
```text
fiona-assistant wiki ingest <source> [--wiki DIR] [-r|--raw PATH]
@@ -1088,7 +1090,7 @@ directory relative to `wiki/`. `raw_categories` maps a converter
name to a directory under `raw/`, such as `{"text": "transcripts"}`.
A value that is absent, of the wrong type, or out of range uses the
default in the table. The packaged `fiona-settings.json` omits
`fiona.wiki.ingest` until section 7.25 is built.
`fiona.wiki.ingest`. Absent keys use the defaults in the table.
A mapping key under `toc_generator.file` whose name is not one of
`min_heading`, `max_heading`, `sectionize`, `ignore_files` is an
@@ -1593,8 +1595,8 @@ another `(name, runner)` pair in the wiki sanity list.
> Status: Partially Implemented
*7.4.16.* is planned. Every other requirement in this section is
built.
*7.4.16.* and *7.4.17.* are planned. Every other requirement in
this section is built.
The renderer is D3 v7, loaded from `https://d3js.org/d3.v7.min.js`,
with the visualization embedded in one HTML file. vis.js is not
@@ -1991,6 +1993,37 @@ used.
HTML draw only.
- *acceptance*: JSON includes a node whose
`unique_points` is below the setting.
- *7.4.17.* `NOT_IMPLEMENTED` A markdown file that `.gitignore`
matches is not a graph node. HTML and JSON both omit it. An
edge that would start or end at that file is omitted. The file
does not count toward `max_files`. Housekeeping's orphan list
(*7.26.6.*) uses this graph, so an omitted file is not an
orphan. There is no setting that turns this off.
- *acceptance*: The scanned `.gitignore` lists `secret.md`.
`secret.md` is absent from JSON `nodes`, `edges`, and
`orphans`. A page that links to it has no edge to it.
- *acceptance*: `max_files` is 1. The directory holds `kept.md`
and a gitignored `secret.md`. JSON `file_count` is 1 and the
only node is `kept.md`.
- *7.4.17.1.* `NOT_IMPLEMENTED` Patterns are the non-comment,
non-blank lines of the `.gitignore` in the scanned directory,
and of each `.gitignore` in a subdirectory. A subdirectory
file's patterns match only paths under that subdirectory.
Matching is *7.2.10.*: a pattern with no slash matches any
path segment, and a pattern that ends with `/` matches that
directory and everything under it.
- *acceptance*: The scanned `.gitignore` lists `drafts/`.
`drafts/note.md` is absent. `wiki/note.md` is present.
- *acceptance*: A line `# secret.md` does not omit
`secret.md`.
- *acceptance*: `notes/.gitignore` lists `scratch.md`.
`notes/scratch.md` is absent. `scratch.md` at the scan
root stays.
- *7.4.17.2.* `NOT_IMPLEMENTED` A path under `.git/` is omitted
even when no pattern names it.
- *acceptance*: With no `.gitignore`, a markdown file under
the scan root stays, and a markdown file under `.git/` is
absent.
### 7.5. Git Operations
@@ -2512,7 +2545,7 @@ is the pip-argument source both scripts call.
### 7.25. wiki ingest
> Status: Planned
> Status: Implemented
Implements section 4.14. An ingest has two phases. The tool runs
the first: it turns one source into a file on disk, archives a copy
@@ -2547,7 +2580,7 @@ A source that no converter claims is archived under `raw/other/`.
Pure-Python libraries are preferred so the install stays light.
Candidates for later formats are in section 10.
- *7.25.1.* `NOT_IMPLEMENTED` The pipeline is built from parts.
- *7.25.1.* The pipeline is built from parts.
- *7.25.1.1.* The pipeline receives its fetchers, converter
registry, archiver, page writer, and log writer as constructor
arguments. It holds no format-specific code.
@@ -2563,7 +2596,7 @@ Candidates for later formats are in section 10.
an argument.
- *acceptance*: Tests run the URL path with a fake opener and
make no network call.
- *7.25.2.* `NOT_IMPLEMENTED` Fetch.
- *7.25.2.* Fetch.
- *7.25.2.1.* An argument that starts with `http://` or `https://`
is a URL. Any other `<scheme>://` prefix exits 1 with
`Error: Unsupported URL scheme: <scheme>`. Anything else is a
@@ -2607,7 +2640,7 @@ Candidates for later formats are in section 10.
- *acceptance*:
`https://host/job/performance_TL153/2108/console` has the slug
`performance-tl153-2108-console`.
- *7.25.3.* `NOT_IMPLEMENTED` Archive.
- *7.25.3.* Archive.
- *7.25.3.1.* The default raw path is
`raw/<category>/<slug>.<ext>`. The category is the raw category
of the converter that claims the media type, unless
@@ -2661,7 +2694,7 @@ Candidates for later formats are in section 10.
- *acceptance*: After ingesting `ingest/a.pdf`, that path is gone
and `raw/papers/a.pdf` holds its bytes. A file ingested from
`personal/` is copied and the original is unchanged.
- *7.25.4.* `NOT_IMPLEMENTED` Convert.
- *7.25.4.* Convert.
- *7.25.4.1.* The registry picks the first converter, in
registration order, whose media types contain the detected type,
then the first whose extensions contain the file extension. With
@@ -2723,7 +2756,7 @@ Candidates for later formats are in section 10.
- *acceptance*: A two-page fixture with no outline has
`## Page 1` and `## Page 2`. A fixture with one blank page is
`partial` and names that page.
- *7.25.5.* `NOT_IMPLEMENTED` Write.
- *7.25.5.* Write.
- *7.25.5.1.* The default page is
`wiki/<default_target>/<slug>.md`. `-t/--target PATH` is
relative to `wiki/`, and a leading `wiki/` is accepted. A value
@@ -2799,7 +2832,7 @@ Candidates for later formats are in section 10.
`<N> markdown style violations left`.
- *acceptance*: Every fixture page passes fmlint with no
violation.
- *7.25.6.* `NOT_IMPLEMENTED` Record. A run that archives appends one
- *7.25.6.* Record. A run that archives appends one
bullet to `wiki/log.md` and bumps that file's patch `version` and
`updated`. The bullet is
`- <YYYY-MM-DD> - ingest <origin> -> <raw path> -> <page path> (raw)`,
@@ -2807,10 +2840,10 @@ Candidates for later formats are in section 10.
does not edit `wiki/index.md` or any other page.
- *acceptance*: One successful run adds one line to `log.md` and
leaves `index.md` unchanged.
- *7.25.7.* `NOT_IMPLEMENTED` `--raw-only` runs fetch, archive, and
- *7.25.7.* `--raw-only` runs fetch, archive, and
record, and skips convert and write. It exits 0.
- *acceptance*: No page is written and `conversion` is `none`.
- *7.25.8.* `NOT_IMPLEMENTED` `--dry-run` fetches into a temporary
- *7.25.8.* `--dry-run` fetches into a temporary
directory outside the wiki, so a URL is still downloaded. It
prints `Dry run - no files will be modified:` and then the lines
a real run would print. It writes nothing under the wiki, does not
@@ -2818,12 +2851,12 @@ Candidates for later formats are in section 10.
true in JSON.
- *acceptance*: File hashes under the wiki match before and after
a dry run, and an `ingest/` source is still in place.
- *7.25.9.* `NOT_IMPLEMENTED` `--pending` lists every page under
- *7.25.9.* `--pending` lists every page under
`wiki/` whose frontmatter `status` is `raw`, sorted by path, and
exits 0.
- *acceptance*: A page changed to `status: refined` is no longer
listed.
- *7.25.10.* `NOT_IMPLEMENTED` Refinement is the agent's phase. The
- *7.25.10.* Refinement is the agent's phase. The
tool never writes `status: refined`. A page is refined when
`status` is `refined`, `version` is `1.0.0` or later, the quality
signals are set and `llm.model` is not `none`, the page is listed
@@ -2832,7 +2865,7 @@ Candidates for later formats are in section 10.
- *acceptance*: Fiona skill section 16 lists those conditions and
the steps that meet them, and says when to convert a source by
hand.
- *7.25.11.* `NOT_IMPLEMENTED` The agent checks the frontmatter the
- *7.25.11.* The agent checks the frontmatter the
tool wrote before it refines the body. It may correct `title`,
`type`, `tags`, `source_meta`, `ingest.conversion`, and
`ingest.warnings` when the source shows they are wrong. It does
@@ -2842,11 +2875,11 @@ Candidates for later formats are in section 10.
the fault to the user.
- *acceptance*: Fiona skill section 16 has a metadata check step
that names the fields it may edit and the fields it may not.
- *7.25.12.* `NOT_IMPLEMENTED` Archived sources do not fail
- *7.25.12.* Archived sources do not fail
`wiki sanity`. Every check skips `raw/`.
- *acceptance*: After ingesting a markdown file with style
violations, `wiki sanity` reports none from its raw copy.
- *7.25.13.* `NOT_IMPLEMENTED` `--json` prints only the JSON object
- *7.25.13.* `--json` prints only the JSON object
from section 4.14 on stdout, on exit 0 and on exit 3. On exit 1,
stdout is empty.
- *acceptance*: The stdout of an exit 3 run parses with
@@ -2968,10 +3001,11 @@ Paths in details and actions are relative to the wiki root.
| P2 | Wiki root discovery | 7.23 | Planned |
| P2 | settings setup | 7.19 | Planned |
| P2 | Bootstrap install modes | 7.24 | Implemented |
| P2 | wiki ingest | 7.25 | Planned |
| P2 | wiki ingest | 7.25 | Implemented |
| P3 | fmlint enforcement of quality signals | 6.2 | Planned |
| P3 | Backup archive | 7.6 | Planned |
| P3 | Documentation refresh | 7.21 | Partially Implemented |
| P2 | Graph skips gitignored files | 7.4.17 | Planned |
| P3 | Node importance filter | 7.4.16 | Planned |
| P4 | Non-markdown version tags | 7.14 | Planned |
| P4 | Tool-role table | 7.22 | Planned |
@@ -3034,7 +3068,7 @@ These notes are not requirements. They point at the current tree.
| launchers | `scripts/run_tests.py`, `scripts/run_sanity.py` |
| bootstrap | `bootstrap`, `bootstrap.ps1`, `scripts/bootstrap_install.py` |
Planned layout for section 7.25, under `src/fiona/wiki/ingest/`:
Layout for section 7.25, under `src/fiona/wiki/ingest/`:
| Module | Classes | Role |
| --- | --- | --- |
+4 -1
View File
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "fiona-assistant"
version = "0.96.0"
version = "0.99.0"
description = "Tools for managing the LLM wiki."
authors = [{name = "nvaldeziii"}]
readme = "README.md"
@@ -24,6 +24,9 @@ dependencies = [
"ruff>=0.16.8",
"pyyaml>=6.0.3",
"GitPython>=3.1.0",
"beautifulsoup4>=4.12.0",
"markdownify>=1.1.0",
"pypdf>=5.0.0",
]
# Optional dependencies groups
+11
View File
@@ -18,6 +18,7 @@ from fiona.mdoctor.mdoctor import main as mdoctor_main
from fiona.python_ruff.python_ruff import main as python_ruff_main
from fiona.wiki.graph_generator import wiki_graph_main
from fiona.wiki.housekeeping import housekeeping_main
from fiona.wiki.ingest.pipeline import add_ingest_arguments, execute_ingest
from fiona.wiki.sanity_manager import sanity_main
from fiona.wiki.update_manager import check_update_main, update_main
from fiona.wiki.wiki import wiki_init
@@ -280,6 +281,14 @@ def main():
help="Disable tracking of external links"
)
# AI/LLM: 2026-09-30 - flags live here so wiki ingest --help is not rejected.
wiki_ingest_parser = wiki_subparsers.add_parser(
"ingest",
prog="fiona-assistant wiki ingest",
help="Archive one source and write a raw wiki page",
)
add_ingest_arguments(wiki_ingest_parser)
# AI/LLM: fiona-assistant housekeeping (SPEC 4.15)
housekeeping_parser = subparsers.add_parser(
"housekeeping",
@@ -399,6 +408,8 @@ def main():
json_output=args.json,
non_interactive=args.non_interactive
))
elif args.wiki_subcommand == "ingest":
sys.exit(execute_ingest(args, wiki_ingest_parser))
elif args.wiki_subcommand == "graph":
sys.exit(wiki_graph_main(
directory=args.directory,
@@ -4,7 +4,7 @@ category: productivity
description: Operate an llm-wiki with the fiona-assistant CLI.
author: nvaldez
license: MIT
version: 1.9.1
version: 1.11.0
updated: 2026-09-30
---
@@ -104,7 +104,19 @@ fiona-assistant git save --description "ingest <slug>: <what the source is>"
```
After a housekeeping pass (lint fixes, index, log, or TOC updates,
page moves, template updates), with sanity passing, run:
page moves, template updates), with sanity passing, read `git diff`
for the files `fiona-assistant git status` lists.
- **Metadata.** On each added or modified page, frontmatter matches
section 13. `name`, `version`, and `updated` are set. A page under
`wiki/` also has `confidence`, `llm`, `contested`, and
`contradictions`.
- **Integrity.** The body still holds the claims, links, tables, and
index or log entries from before this pass. A lint fix, a move, or
a template merge did not drop or garble them.
When a check fails, fix the page and run sanity again. When you
cannot tell whether a page is intact, ask before saving. Then run:
```text
fiona-assistant git save --description "housekeeping: <what changed>"
@@ -127,13 +139,14 @@ It is the step that makes the wiki's history usable:
- **Proof behind the log.** `wiki/log.md` says what happened. The
commit holds the before and after state that proves it.
Before saving, run `fiona-assistant git status`. `save` stages
Before either save, run `fiona-assistant git status`. `save` stages
everything in the tree. When it shows changes that are not from
this cycle, ask the user before saving. When sanity fails, fix it
first, or ask. A duplicate ingest writes nothing, so `save`
refusing a clean tree is expected. This standing instruction
covers `save` only. `snapshot`, `restore`, `clean`, and `undo`
still need the user's request.
this cycle, ask the user before saving. A housekeeping pass also
reads `git diff` and applies the metadata and integrity checks
above. When sanity fails, fix it first, or ask. A duplicate ingest
writes nothing, so `save` refusing a clean tree is expected. This
standing instruction covers `save` only. `snapshot`, `restore`,
`clean`, and `undo` still need the user's request.
## 1. Before a command
@@ -323,10 +336,6 @@ Also: `--include-pattern`, `--exclude-pattern`,
### 4.6. ingest
Planned. Run `fiona-assistant wiki ingest --help`
first. When the subcommand does not exist, do every step of
section 16 by hand.
```text
fiona-assistant wiki ingest <source> [--wiki DIR] [-r|--raw PATH] [-t|--target PATH] [--raw-only] [-f|--force] [--dry-run] [--json]
fiona-assistant wiki ingest --pending [--wiki DIR] [--json]
@@ -706,10 +715,6 @@ These are not implemented. Do not invent flags for them.
- `fiona-assistant pycustomlint`
- `fiona-assistant python-ruff fix`
`fiona-assistant wiki ingest` is planned. Call it only after
`fiona-assistant wiki ingest --help` succeeds. Otherwise follow
section 16 by hand.
## 16. Ingest cycle
An ingest has two phases. `wiki ingest` (section 4.6) converts the
@@ -776,7 +781,10 @@ the user. Do not move or delete anything under `raw/`.
The tool fills every key in section 16.5, but it reads the source
mechanically. Check the frontmatter against the source before you
refine, and fix what is wrong.
refine, and fix what is wrong. You may edit `title`, `type`, `tags`,
`source_meta`, `ingest.conversion`, and `ingest.warnings`. You may
not hand-edit `ingest.sha256`, `ingest.size_bytes`, `ingest.fetched`,
`ingest.media_type`, or `ingest.tool`.
| Key | Check | May edit |
| --- | --- | --- |
@@ -800,6 +808,11 @@ with `--force` or tell the user. Do not hand-edit it. Bump the page
1. `fiona-assistant wiki ingest --pending --json` lists the pages
still at `status: raw`. Without the tool, search `wiki/` for
`status: raw`. Check each page's metadata first (section 16.3).
A markdown source keeps its body. The tool writes the page
frontmatter when the source did not already have it. Source
frontmatter is in `source_meta`. A text source (`.txt`, `.text`,
`.log`) is one fenced `text` block plus that frontmatter, still
at `status: raw`. Rewrite that fence. The tool does not refine it.
2. Read the page and the source it cites. Open the raw file when
`ingest.conversion` is `partial` or a claim looks garbled.
3. Rewrite the page as a source summary:
@@ -826,8 +839,11 @@ with `--force` or tell the user. Do not hand-edit it. Bump the page
`wiki/log.md`.
9. Run the work loop in section 12, ending with `wiki sanity`.
The ingest is done when no page is left at `status: raw` and sanity
passes. Then save it
A page is refined when `status` is `refined`, `version` is `1.0.0`
or later, the quality signals are set and `llm.model` is not `none`,
the page is listed in `wiki/index.md`, and `wiki/log.md` has a line
for the refinement. The ingest is done when every page from the run
meets that and `wiki sanity` passes. Then save it
([save rule](#save-after-every-ingest-cycle-and-housekeeping-pass)).
### 16.5. Raw page frontmatter
@@ -1,97 +0,0 @@
---
name: obsidian
description: Read, search, create, and edit notes in the Obsidian vault.
version: 1.0.0
author: Teknium (teknium1), Hermes Agent
license: MIT
platforms: [linux, macos, windows]
metadata:
hermes:
tags: [Obsidian, Notes, Markdown, Vault]
related_skills: []
---
# Obsidian Vault
Use this skill for filesystem-first Obsidian vault work: reading notes, listing
notes, searching note files, creating notes, appending content, and adding
wikilinks.
## Vault path
Use a known or resolved vault path before calling file tools.
The documented vault-path convention is the `OBSIDIAN_VAULT_PATH` environment
variable, for example from `${HERMES_HOME:-~/.hermes}/.env`. If it is unset,
use `~/Documents/Obsidian Vault`.
File tools do not expand shell variables. Do not pass paths containing
`$OBSIDIAN_VAULT_PATH` to `read_file`, `write_file`, `patch`, or
`search_files`; resolve the vault path first and pass a concrete absolute path.
Vault paths may contain spaces, which is another reason to prefer file tools
over shell commands.
If the vault path is unknown, `terminal` is acceptable for resolving
`OBSIDIAN_VAULT_PATH` or checking whether the fallback path exists. Once the
path is known, switch back to file tools.
## Read a note
Use `read_file` with the resolved absolute path to the note. Prefer this over
`cat` because it provides line numbers and pagination.
## List notes
Use `search_files` with `target: "files"` and the resolved vault path. Prefer
this over `find` or `ls`.
- To list all markdown notes, use `pattern: "*.md"` under the vault path.
- To list a subfolder, search under that subfolder's absolute path.
## Search
Use `search_files` for both filename and content searches. Prefer this over
`grep`, `find`, or `ls`.
- For filenames, use `search_files` with `target: "files"` and a filename `pattern`.
- For note contents, use `search_files` with `target: "content"`, the content
regex as `pattern`, and `file_glob: "*.md"` when you want to restrict matches
to markdown notes.
## Create a note
Use `write_file` with the resolved absolute path and the full markdown content.
Prefer this over shell heredocs or `echo` because it avoids shell quoting
issues and returns structured results.
## Append to a note
Prefer a native file-tool workflow when it is not awkward:
- Read the target note with `read_file`.
- Use `patch` for an anchored append when there is stable context, such as
adding a section after an existing heading or appending before a known
trailing block.
- Use `write_file` when rewriting the whole note is clearer than constructing a
fragile patch.
For an anchored append with `patch`, replace the anchor with the anchor plus
the new content.
For a simple append with no stable context, `terminal` is acceptable if it is
the clearest safe option.
## Targeted edits
Use `patch` for focused note changes when the current content gives you stable
context. Prefer this over shell text rewriting.
## Linking
For **compatibility** across tools (e.g., GitHub, VS Code, Obsidian), use
**standard Markdown links** (`[link text](path/to/note.md)`) instead of
wikilinks (`[[Note Name]]`).
- **Markdown links** ensure portability and work in all markdown renderers.
- If Obsidian-specific features (e.g., graph view) are needed, wikilinks can be
used **in addition to** Markdown links, but never as a replacement.
+4
View File
@@ -0,0 +1,4 @@
"""Wiki ingest package (SPEC 7.25).
Generated by AI/LLM: 2026-09-30 - stage modules for wiki ingest.
"""
+239
View File
@@ -0,0 +1,239 @@
"""Archive a fetched file under ``raw/``.
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.3. Never edits a file already there.
"""
from __future__ import annotations
import hashlib
import logging
import pathlib
import shutil
from fiona.wiki.ingest.converters.base import ConverterRegistry
from fiona.wiki.ingest.media import extension_for
from fiona.wiki.ingest.models import ArchiveResult, FetchedSource, IngestError
from fiona.wiki.ingest.settings import IngestSettings
logger = logging.getLogger(__name__)
class RawArchiver:
"""Copy or move a source into ``raw/`` and reuse identical bytes.
Generated by AI/LLM: 2026-09-30 - SHA-256 reuse and collision names.
"""
def __init__(self, wiki_root: pathlib.Path) -> None:
"""Store the wiki root.
Args:
wiki_root: Directory that contains ``raw/``.
"""
self.wiki_root = wiki_root
def find_by_sha256(
self,
digest: str,
size: int,
skip: pathlib.Path | None = None,
) -> pathlib.Path | None:
"""Return the first ``raw/`` file with this size and digest.
Generated by AI/LLM: 2026-09-30 - only same-size files are hashed.
Args:
digest: Lowercase SHA-256 hex.
size: Byte size that a candidate must match.
skip: A file to ignore, usually the source when it already sits in ``raw/``.
Returns:
pathlib.Path | None: Match with the lowest path, or None.
"""
raw_root = self.wiki_root / "raw"
if not raw_root.is_dir():
return None
skip_resolved = skip.resolve() if skip is not None else None
matches: list[pathlib.Path] = []
for path in raw_root.rglob("*"):
if not path.is_file():
continue
if skip_resolved is not None and path.resolve() == skip_resolved:
continue
try:
if path.stat().st_size != size:
continue
except OSError:
continue
if _sha256(path) == digest:
matches.append(path)
if not matches:
return None
matches.sort(key=lambda item: item.as_posix())
return matches[0]
def archive(
self,
source: FetchedSource,
*,
registry: ConverterRegistry,
settings: IngestSettings,
raw_override: str | None,
dry_run: bool,
) -> ArchiveResult:
"""Place the source under ``raw/`` or reuse an identical copy.
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.3.
Args:
source: Fetched file.
registry: Used for the raw category and a missing extension.
settings: ``raw_categories`` overrides.
raw_override: ``-r`` value, relative to ``raw/``, or None.
dry_run: When true, compute the path and write nothing.
Returns:
ArchiveResult: Relative raw path, digest, and warnings.
Raises:
IngestError: ``-r`` escapes ``raw/`` or is used on a file already there.
"""
raw_root = (self.wiki_root / "raw").resolve()
source_in_raw = _is_inside(source.path, raw_root)
if raw_override and source_in_raw:
raise IngestError(
f"--raw cannot be used on a file already under raw/: {source.argument}"
)
if raw_override:
_require_inside(raw_root, raw_override, "--raw must stay inside raw/")
digest = _sha256(source.path)
match = self.find_by_sha256(
digest,
source.size_bytes,
skip=source.path if source_in_raw else None,
)
warnings: list[str] = []
if match is not None:
raw_rel = _relative(match, self.wiki_root)
if raw_override:
warnings.append(f"already archived at {raw_rel}; --raw ignored")
if _is_inside(source.path, self.wiki_root / "ingest") and not dry_run:
source.path.unlink()
logger.info("reused raw file %s", raw_rel)
return ArchiveResult(match, raw_rel, digest, True, warnings)
if source_in_raw:
return ArchiveResult(source.path, _relative(source.path, self.wiki_root), digest, False, [])
extension = source.path.suffix.lower() or extension_for(source.media_type, "", registry)
dest = self._destination(source, registry, settings, raw_override, extension, raw_root)
dest = _free_name(dest, source.slug, extension)
if not dry_run:
dest.parent.mkdir(parents=True, exist_ok=True)
if _is_inside(source.path, self.wiki_root / "ingest"):
shutil.move(str(source.path), str(dest))
else:
shutil.copyfile(source.path, dest)
logger.info("archived %s", dest)
return ArchiveResult(dest, _relative(dest, self.wiki_root), digest, False, warnings)
def _destination(
self,
source: FetchedSource,
registry: ConverterRegistry,
settings: IngestSettings,
raw_override: str | None,
extension: str,
raw_root: pathlib.Path,
) -> pathlib.Path:
"""Resolve the raw path before a collision suffix is applied.
Generated by AI/LLM: 2026-09-30 - ``-r`` file or directory, else the category.
"""
filename = f"{source.slug}{extension}"
if raw_override:
relative = _strip_prefix(raw_override, "raw")
suffix = pathlib.PurePosixPath(relative.replace("\\", "/")).suffix
if suffix:
return raw_root / relative
return raw_root / relative / filename
converter = registry.find(source.media_type, extension)
if converter is None:
category = "other"
else:
category = settings.raw_categories.get(converter.name, converter.raw_category)
return raw_root / category / filename
def _sha256(path: pathlib.Path) -> str:
"""Return the lowercase SHA-256 hex digest of a file.
Generated by AI/LLM: 2026-09-30 - streamed so large files stay cheap.
"""
digest = hashlib.sha256()
with path.open("rb") as handle:
while True:
chunk = handle.read(1024 * 1024)
if not chunk:
break
digest.update(chunk)
return digest.hexdigest()
def _free_name(dest: pathlib.Path, slug: str, extension: str) -> pathlib.Path:
"""Return ``dest`` or the first free ``<slug>-N`` name beside it.
Generated by AI/LLM: 2026-09-30 - never overwrite a raw file.
"""
if not dest.exists():
return dest
number = 2
while True:
candidate = dest.parent / f"{slug}-{number}{extension}"
if not candidate.exists():
return candidate
number += 1
def _strip_prefix(value: str, prefix: str) -> str:
"""Drop a leading ``raw/`` or ``wiki/`` from a user path.
Generated by AI/LLM: 2026-09-30 - both forms are accepted.
"""
relative = value.replace("\\", "/").lstrip("/")
marker = f"{prefix}/"
if relative == prefix:
return ""
if relative.startswith(marker):
return relative[len(marker):]
return relative
def _require_inside(root: pathlib.Path, value: str, message: str) -> pathlib.Path:
"""Resolve ``value`` under ``root`` or raise.
Generated by AI/LLM: 2026-09-30 - ``..`` and symlinks must stay inside.
"""
relative = _strip_prefix(value, root.name)
candidate = (root / relative).resolve()
if not _is_inside(candidate, root):
raise IngestError(f"{message}: {value}")
return candidate
def _is_inside(path: pathlib.Path, root: pathlib.Path) -> bool:
"""Return whether ``path`` is ``root`` or a file inside it.
Generated by AI/LLM: 2026-09-30 - compare resolved paths.
"""
try:
path.resolve().relative_to(root.resolve())
except ValueError:
return False
return True
def _relative(path: pathlib.Path, root: pathlib.Path) -> str:
"""Return a wiki-relative path with ``/`` separators.
Generated by AI/LLM: 2026-09-30 - JSON and log paths.
"""
return path.resolve().relative_to(root.resolve()).as_posix()
@@ -0,0 +1,26 @@
"""Shipped converters, in registration order.
Generated by AI/LLM: 2026-09-30 - markdown, text, html, then pdf.
"""
from fiona.wiki.ingest.converters.base import ConverterRegistry
from fiona.wiki.ingest.converters.html import HtmlConverter
from fiona.wiki.ingest.converters.markdown import MarkdownConverter
from fiona.wiki.ingest.converters.pdf import PdfConverter
from fiona.wiki.ingest.converters.text import TextConverter
def default_registry() -> ConverterRegistry:
"""Return the converters shipped with wiki ingest.
Generated by AI/LLM: 2026-09-30 - registration order from SPEC 7.25.
Returns:
ConverterRegistry: Markdown, text, HTML, and PDF.
"""
registry = ConverterRegistry()
registry.register(MarkdownConverter)
registry.register(TextConverter)
registry.register(HtmlConverter)
registry.register(PdfConverter)
return registry
+108
View File
@@ -0,0 +1,108 @@
"""Converter base types and the registry.
Generated by AI/LLM: 2026-09-30 - one class per format, SPEC 7.25.1.
"""
from __future__ import annotations
import abc
import typing
from fiona.wiki.ingest.models import ConversionResult, FetchedSource
class ConversionError(Exception):
"""Raised when a converter cannot read a file it claims.
Generated by AI/LLM: 2026-09-30 - exit 3, raw copy kept.
"""
def __init__(self, converter: str, reason: str) -> None:
"""Store the converter name and the reason.
Args:
converter: Converter ``name`` attribute.
reason: Short failure text, without the ``Error:`` prefix.
"""
super().__init__(f"{converter}: {reason}")
self.converter = converter
self.reason = reason
class SourceConverter(abc.ABC):
"""One format. Subclasses declare what they claim and implement ``convert``.
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.1.2.
"""
name: typing.ClassVar[str]
media_types: typing.ClassVar[tuple[str, ...]]
extensions: typing.ClassVar[tuple[str, ...]]
raw_category: typing.ClassVar[str]
@abc.abstractmethod
def convert(self, source: FetchedSource) -> ConversionResult:
"""Convert an archived file to markdown.
Args:
source: Fetched file, including media type and slug.
Returns:
ConversionResult: Body, title, metadata, and warnings.
Raises:
ConversionError: The file cannot be read.
"""
class ConverterRegistry:
"""Ordered list of converters.
Generated by AI/LLM: 2026-09-30 - first media-type match, then extension.
"""
def __init__(self, converters: list[SourceConverter] | None = None) -> None:
"""Start with an optional list of converter instances.
Args:
converters: Instances in registration order.
"""
self.converters: list[SourceConverter] = list(converters or [])
def register(self, converter: SourceConverter | type[SourceConverter]) -> None:
"""Append a converter class or instance.
Generated by AI/LLM: 2026-09-30 - adding a format is one registration.
Args:
converter: Class with a no-arg constructor, or a ready instance.
"""
if isinstance(converter, type):
converter = converter()
self.converters.append(converter)
def find(self, media_type: str, extension: str) -> SourceConverter | None:
"""Pick the converter for a media type and extension.
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.4.1 registration order.
Args:
media_type: Detected type, or empty.
extension: Suffix including the dot, or empty.
Returns:
SourceConverter | None: The first match, or None.
"""
media = (media_type or "").lower()
ext = extension.lower()
if ext and not ext.startswith("."):
ext = f".{ext}"
if media:
for converter in self.converters:
if media in converter.media_types:
return converter
if ext:
for converter in self.converters:
if ext in converter.extensions:
return converter
return None
+128
View File
@@ -0,0 +1,128 @@
"""HTML source converter.
Generated by AI/LLM: 2026-09-30 - article text via beautifulsoup4 and markdownify.
"""
from __future__ import annotations
import typing
import urllib.parse
import bs4
import markdownify
from fiona.wiki.ingest.converters.base import SourceConverter
from fiona.wiki.ingest.media import decode_bytes
from fiona.wiki.ingest.models import ConversionResult, FetchedSource
_DROP_TAGS = ("script", "style", "noscript", "template")
_NO_TEXT = "no text content; the page may need JavaScript or a login"
class HtmlConverter(SourceConverter):
"""Convert an HTML document to markdown.
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.4.5.
"""
name: typing.ClassVar[str] = "html"
media_types: typing.ClassVar[tuple[str, ...]] = ("text/html", "application/xhtml+xml")
extensions: typing.ClassVar[tuple[str, ...]] = (".html", ".htm", ".xhtml")
raw_category: typing.ClassVar[str] = "articles"
def convert(self, source: FetchedSource) -> ConversionResult:
"""Parse HTML and return the article body as markdown.
Generated by AI/LLM: 2026-09-30 - drop scripts and absolutize links.
Args:
source: Archived HTML file. ``origin`` is the page URL when fetched.
Returns:
ConversionResult: Markdown body, title, and ``source_meta``.
"""
text, warnings = decode_bytes(source.path.read_bytes(), source.charset)
soup = bs4.BeautifulSoup(text, "html.parser")
for tag in soup.find_all(_DROP_TAGS):
tag.decompose()
for comment in soup.find_all(string=lambda node: isinstance(node, bs4.Comment)):
comment.extract()
root = soup.find("article") or soup.find("main") or soup.body or soup
origin = source.origin if source.origin_kind == "url" else None
_absolutize(root, soup, origin)
visible = root.get_text(strip=True)
heading = getattr(markdownify, "ATX", "ATX")
body = markdownify.markdownify(str(root), heading_style=heading, bullets="-").strip()
if body:
body += "\n"
meta = _source_meta(soup)
title = _html_title(soup, root)
conversion = "complete"
if not visible:
conversion = "partial"
warnings = [*warnings, _NO_TEXT]
return ConversionResult(
body=body,
title=title,
source_meta=meta,
conversion=conversion,
warnings=warnings,
converter=self.name,
)
def _absolutize(root: bs4.element.Tag, soup: bs4.BeautifulSoup, origin: str | None) -> None:
"""Rewrite relative href and src values against base href, then the origin.
Generated by AI/LLM: 2026-09-30 - a local file with no origin keeps relative links.
"""
base_tag = soup.find("base", href=True)
base = base_tag.get("href") if base_tag is not None else None
if not base:
base = origin
if not base:
return
for tag in root.find_all(href=True):
tag["href"] = urllib.parse.urljoin(str(base), str(tag["href"]))
for tag in root.find_all(src=True):
tag["src"] = urllib.parse.urljoin(str(base), str(tag["src"]))
def _html_title(soup: bs4.BeautifulSoup, root: bs4.element.Tag) -> str | None:
"""Return ``<title>``, then ``og:title``, then the first ``<h1>``.
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.4.5.1.
"""
if soup.title and soup.title.string and soup.title.string.strip():
return soup.title.string.strip()
og = soup.find("meta", attrs={"property": "og:title"})
if og is not None and og.get("content"):
return str(og["content"]).strip() or None
heading = root.find("h1")
if heading is not None:
text = heading.get_text(strip=True)
return text or None
return None
def _source_meta(soup: bs4.BeautifulSoup) -> dict:
"""Collect language, canonical URL, and selected meta values.
Generated by AI/LLM: 2026-09-30 - only keys that are present.
"""
meta: dict[str, str] = {}
root = soup.find("html")
if root is not None and root.get("lang"):
meta["lang"] = str(root["lang"])
canonical = soup.find("link", rel=lambda value: value and "canonical" in value)
if canonical is not None and canonical.get("href"):
meta["canonical"] = str(canonical["href"])
for tag in soup.find_all("meta"):
key = tag.get("name") or tag.get("property")
content = tag.get("content")
if not key or content is None:
continue
name = str(key)
if name in {"description", "author"} or name.startswith("og:") or name == "article:published_time":
meta[name] = str(content)
return meta
@@ -0,0 +1,105 @@
"""Markdown source converter.
Generated by AI/LLM: 2026-09-30 - keep the body and lift frontmatter into source_meta.
"""
from __future__ import annotations
import re
import typing
import yaml
from fiona.wiki.ingest.converters.base import SourceConverter
from fiona.wiki.ingest.media import decode_bytes
from fiona.wiki.ingest.models import ConversionResult, FetchedSource
_FRONTMATTER = re.compile(r"\A---\r?\n(.*?)\r?\n---\r?\n?", re.DOTALL)
_H1 = re.compile(r"^#\s+(\S.*)$")
_FENCE = re.compile(r"^\s*(`{3,}|~{3,})")
class MarkdownConverter(SourceConverter):
"""Keep markdown as it is and record any source frontmatter.
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.4.3.
The body after a frontmatter block is kept character for character.
The page writer adds the wiki frontmatter when the new page does not
have it yet. Source keys move to ``source_meta`` instead of being
written a second time as the page frontmatter.
"""
name: typing.ClassVar[str] = "markdown"
media_types: typing.ClassVar[tuple[str, ...]] = ("text/markdown",)
extensions: typing.ClassVar[tuple[str, ...]] = (".md", ".markdown")
raw_category: typing.ClassVar[str] = "articles"
def convert(self, source: FetchedSource) -> ConversionResult:
"""Return the markdown body and any frontmatter as metadata.
Generated by AI/LLM: 2026-09-30 - title from frontmatter, then H1, then slug.
Args:
source: Archived markdown file.
Returns:
ConversionResult: Unchanged body, ``complete`` unless decoding fell back.
"""
text, warnings = decode_bytes(source.path.read_bytes(), source.charset)
meta: dict = {}
body = text
match = _FRONTMATTER.match(text)
if match:
loaded = _load_meta(match.group(1))
if loaded is None:
warnings = [*warnings, "source frontmatter did not parse"]
else:
meta = loaded
body = text[match.end():]
title = _title(meta, body, source.slug)
return ConversionResult(
body=body,
title=title,
source_meta=meta,
conversion="complete",
warnings=warnings,
converter=self.name,
)
def _load_meta(block: str) -> dict | None:
"""Parse a frontmatter block into a dict.
Generated by AI/LLM: 2026-09-30 - invalid YAML stays in the body.
"""
try:
loaded = yaml.safe_load(block)
except yaml.YAMLError:
return None
if loaded is None:
return {}
if isinstance(loaded, dict):
return loaded
return None
def _title(meta: dict, body: str, slug: str) -> str:
"""Choose the page title.
Generated by AI/LLM: 2026-09-30 - frontmatter title, first H1, then slug.
"""
raw_title = meta.get("title")
if isinstance(raw_title, str) and raw_title.strip():
return raw_title.strip()
fenced = False
for line in body.splitlines():
if _FENCE.match(line):
fenced = not fenced
continue
if fenced:
continue
match = _H1.match(line)
if match:
return match.group(1).strip()
return slug
+186
View File
@@ -0,0 +1,186 @@
"""PDF source converter.
Generated by AI/LLM: 2026-09-30 - text layer via pypdf, SPEC 7.25.4.6.
"""
from __future__ import annotations
import datetime
import typing
import pypdf
import pypdf.errors
from fiona.wiki.ingest.converters.base import ConversionError, SourceConverter
from fiona.wiki.ingest.models import ConversionResult, FetchedSource
_SCANNED = "no text layer; the PDF may be scanned"
class PdfConverter(SourceConverter):
"""Read a PDF text layer into markdown headings and paragraphs.
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.4.6.
"""
name: typing.ClassVar[str] = "pdf"
media_types: typing.ClassVar[tuple[str, ...]] = ("application/pdf",)
extensions: typing.ClassVar[tuple[str, ...]] = (".pdf",)
raw_category: typing.ClassVar[str] = "papers"
def convert(self, source: FetchedSource) -> ConversionResult:
"""Extract page text, outline headings, and document metadata.
Generated by AI/LLM: 2026-09-30 - encrypted or corrupt files raise.
Args:
source: Archived PDF.
Returns:
ConversionResult: Markdown body. Empty pages make it ``partial``.
Raises:
ConversionError: The PDF is encrypted or cannot be parsed.
"""
try:
reader = pypdf.PdfReader(str(source.path))
except pypdf.errors.PdfReadError as exc:
raise ConversionError(self.name, str(exc)) from exc
if reader.is_encrypted:
raise ConversionError(self.name, "encrypted PDF")
meta = _metadata(reader)
title = meta.get("title") or source.slug
outline = _outline(reader)
pages = []
empty: list[int] = []
for index, page in enumerate(reader.pages, start=1):
try:
text = page.extract_text() or ""
except pypdf.errors.PdfReadError as exc:
raise ConversionError(self.name, str(exc)) from exc
pages.append(text.strip())
if not text.strip():
empty.append(index)
body = _render(pages, outline)
warnings: list[str] = []
conversion = "complete"
if empty and len(empty) == len(pages):
warnings.append(_SCANNED)
conversion = "partial"
elif empty:
warnings.extend(f"page {number} has no text layer" for number in empty)
conversion = "partial"
return ConversionResult(
body=body,
title=str(title),
source_meta=meta,
conversion=conversion,
warnings=warnings,
converter=self.name,
)
def _metadata(reader: pypdf.PdfReader) -> dict:
"""Return page count and the document info values that are present.
Generated by AI/LLM: 2026-09-30 - dates as ISO 8601.
"""
meta: dict[str, object] = {"page_count": len(reader.pages)}
info = reader.metadata
if info is None:
return meta
mapping = (
("title", info.title),
("author", info.author),
("subject", info.subject),
("keywords", info.keywords),
("creation_date", _iso_date(info.creation_date)),
("mod_date", _iso_date(info.modification_date)),
)
for key, value in mapping:
if value:
meta[key] = value
return meta
def _iso_date(value: object) -> str | None:
"""Format a PDF date as ISO 8601.
Generated by AI/LLM: 2026-09-30 - accept datetime values and ``D:`` strings.
"""
if value is None:
return None
if isinstance(value, datetime.datetime):
if value.tzinfo is None:
return value.strftime("%Y-%m-%dT%H:%M:%SZ")
return value.astimezone(datetime.UTC).strftime("%Y-%m-%dT%H:%M:%SZ")
text = str(value).strip().removeprefix("D:")
digits = "".join(char for char in text if char.isdigit())
if len(digits) < 8:
return text or None
year, month, day = digits[0:4], digits[4:6], digits[6:8]
hour = digits[8:10] or "00"
minute = digits[10:12] or "00"
second = digits[12:14] or "00"
return f"{year}-{month}-{day}T{hour}:{minute}:{second}Z"
def _outline(reader: pypdf.PdfReader) -> list[tuple[int, int, str]]:
"""Return outline entries as page index, heading level, and title.
Generated by AI/LLM: 2026-09-30 - level is depth plus 2, capped at 6.
"""
found: list[tuple[int, int, str]] = []
try:
items = reader.outline
except pypdf.errors.PdfReadError:
return found
_walk_outline(reader, items, 0, found)
return found
def _walk_outline(
reader: pypdf.PdfReader,
items: object,
depth: int,
found: list[tuple[int, int, str]],
) -> None:
"""Walk a pypdf outline tree.
Generated by AI/LLM: 2026-09-30 - nested lists are deeper entries.
"""
if not items:
return
for item in items: # type: ignore[union-attr]
if isinstance(item, list):
_walk_outline(reader, item, depth + 1, found)
continue
try:
page_index = reader.get_destination_page_number(item)
except (ValueError, KeyError, pypdf.errors.PdfReadError):
continue
title = getattr(item, "title", "") or ""
level = min(depth + 2, 6)
found.append((page_index, level, str(title).strip() or "Untitled"))
def _render(pages: list[str], outline: list[tuple[int, int, str]]) -> str:
"""Render pages with outline headings or ``## Page n`` headings.
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.4.6.2.
"""
by_page: dict[int, list[str]] = {}
for page_index, level, title in outline:
by_page.setdefault(page_index, []).append(f"{'#' * level} {title}")
chunks: list[str] = []
for index, text in enumerate(pages):
if outline:
chunks.extend(by_page.get(index, []))
else:
chunks.append(f"## Page {index + 1}")
if text:
chunks.append(text)
body = "\n\n".join(chunks)
if body:
body += "\n"
return body
+68
View File
@@ -0,0 +1,68 @@
"""Plain-text source converter.
Generated by AI/LLM: 2026-09-30 - wrap text in a fence; the agent refines the page.
"""
from __future__ import annotations
import typing
from fiona.wiki.ingest.converters.base import SourceConverter
from fiona.wiki.ingest.media import decode_bytes
from fiona.wiki.ingest.models import ConversionResult, FetchedSource
class TextConverter(SourceConverter):
"""Turn plain text into one fenced markdown block.
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.4.4.
The page is written at ``status: raw``. The tool does not summarize
the text. The agent rewrites the fence during refinement.
"""
name: typing.ClassVar[str] = "text"
media_types: typing.ClassVar[tuple[str, ...]] = ("text/plain",)
extensions: typing.ClassVar[tuple[str, ...]] = (".txt", ".text", ".log")
raw_category: typing.ClassVar[str] = "articles"
def convert(self, source: FetchedSource) -> ConversionResult:
"""Decode the file and wrap it in a fence longer than any backtick run.
Generated by AI/LLM: 2026-09-30 - title is the slug; fallback decode warns.
Args:
source: Archived text file.
Returns:
ConversionResult: Fenced body and the slug as the title.
"""
text, warnings = decode_bytes(source.path.read_bytes(), source.charset)
marker = "`" * _fence_length(text)
if text and not text.endswith("\n"):
text += "\n"
body = f"{marker}text\n{text}{marker}\n"
return ConversionResult(
body=body,
title=source.slug,
source_meta={},
conversion="complete",
warnings=warnings,
converter=self.name,
)
def _fence_length(text: str) -> int:
"""Return a fence length longer than the longest backtick run.
Generated by AI/LLM: 2026-09-30 - a run of three when the text has fewer.
"""
longest = 0
current = 0
for char in text:
if char == "`":
current += 1
longest = max(longest, current)
else:
current = 0
return max(3, longest + 1)
+276
View File
@@ -0,0 +1,276 @@
"""Fetch a local file or an HTTP URL.
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.2. No credentials are sent.
"""
from __future__ import annotations
import logging
import pathlib
import re
import urllib.error
import urllib.parse
import urllib.request
from fiona.wiki.ingest.converters.base import ConverterRegistry
from fiona.wiki.ingest.media import (
detect_media_type,
extension_for,
slug_from_file,
slug_from_url,
utc_now,
)
from fiona.wiki.ingest.models import FetchedSource, IngestError
from fiona.wiki.ingest.settings import IngestSettings, default_settings
logger = logging.getLogger(__name__)
_SCHEME = re.compile(r"^([A-Za-z][A-Za-z0-9+.-]*)://")
_WINDOWS_DRIVE = re.compile(r"^[A-Za-z]:[\\/]")
def url_scheme(argument: str) -> str | None:
"""Return the URL scheme, or None for a local path.
Generated by AI/LLM: 2026-09-30 - a Windows drive letter is not a scheme.
Args:
argument: The ``<source>`` argument as given.
Returns:
str | None: Scheme text, preserving case, or None.
"""
if _WINDOWS_DRIVE.match(argument):
return None
match = _SCHEME.match(argument)
if match is None:
return None
return match.group(1)
class SourceFetcher:
"""Base fetcher. Subclasses accept an argument and return a local file.
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.1.3.
"""
settings: IngestSettings | None = None
def accepts(self, argument: str) -> bool:
"""Return whether this fetcher handles the argument.
Args:
argument: The ``<source>`` argument as given.
Returns:
bool: True when ``fetch`` can run.
"""
raise NotImplementedError
def fetch(self, argument: str, dest_dir: pathlib.Path) -> FetchedSource:
"""Fetch the argument into ``dest_dir`` when a download is required.
Args:
argument: The ``<source>`` argument as given.
dest_dir: Directory for a downloaded file. Local fetches ignore it.
Returns:
FetchedSource: Local path and origin facts.
"""
raise NotImplementedError
class LocalFileFetcher(SourceFetcher):
"""Read a local file in place.
Generated by AI/LLM: 2026-09-30 - missing files and directories exit 1.
"""
def __init__(self, registry: ConverterRegistry) -> None:
"""Store the registry used to name the media type.
Args:
registry: Converter registry for extension lookup.
"""
self.registry = registry
self.settings = None
def accepts(self, argument: str) -> bool:
"""Accept any argument that is not a URL.
Generated by AI/LLM: 2026-09-30 - scheme detection stays in ``url_scheme``.
Args:
argument: The ``<source>`` argument as given.
Returns:
bool: True when there is no ``://`` scheme.
"""
return url_scheme(argument) is None
def fetch(self, argument: str, dest_dir: pathlib.Path) -> FetchedSource:
"""Resolve a local path without copying it.
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.2.1.
Args:
argument: Local path as given.
dest_dir: Unused. Present so every fetcher shares one signature.
Returns:
FetchedSource: The resolved file. ``created`` is false.
Raises:
IngestError: The path is missing or is a directory.
"""
del dest_dir
path = pathlib.Path(argument)
if not path.exists():
raise IngestError(f"Source not found: {argument}")
if path.is_dir():
raise IngestError(f"Source is a directory: {argument}")
data = path.read_bytes()
media, charset = detect_media_type(data, path.suffix, None, self.registry)
resolved = path.resolve()
logger.info("fetched local file %s as %s", resolved, media)
return FetchedSource(
argument=argument,
path=resolved,
origin=str(resolved),
origin_kind="file",
final_url=None,
media_type=media,
charset=charset,
size_bytes=len(data),
slug=slug_from_file(path),
fetched=utc_now(),
created=False,
)
class UrlFetcher(SourceFetcher):
"""Download one ``http`` or ``https`` URL.
Generated by AI/LLM: 2026-09-30 - opener is injected; tests make no network call.
"""
def __init__(
self,
registry: ConverterRegistry,
opener: urllib.request.OpenerDirector | None = None,
) -> None:
"""Store the registry and the URL opener.
Args:
registry: Converter registry for the saved extension.
opener: ``urllib`` opener. The default follows redirects and proxies.
"""
self.registry = registry
self.opener = opener or urllib.request.build_opener()
self.settings = None
def accepts(self, argument: str) -> bool:
"""Accept ``http`` and ``https`` only.
Generated by AI/LLM: 2026-09-30 - other schemes are rejected by the pipeline.
Args:
argument: The ``<source>`` argument as given.
Returns:
bool: True for http and https.
"""
scheme = url_scheme(argument)
return scheme is not None and scheme.lower() in {"http", "https"}
def fetch(self, argument: str, dest_dir: pathlib.Path) -> FetchedSource:
"""Download a URL into ``dest_dir`` and stop past the size limit.
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.2.2 and 7.25.2.3.
Args:
argument: Absolute URL.
dest_dir: Directory that receives ``<slug>.<ext>``.
Returns:
FetchedSource: The saved file. ``created`` is true.
Raises:
IngestError: The response is not 2xx, the network fails, or the body is too large.
"""
settings = self.settings or default_settings()
request = urllib.request.Request(
_without_userinfo(argument),
headers={"User-Agent": settings.user_agent},
)
try:
response = self.opener.open(request, timeout=settings.timeout_seconds)
except urllib.error.HTTPError as exc:
raise IngestError(f"Download failed: {argument}: {exc.code} {exc.reason}") from exc
except urllib.error.URLError as exc:
raise IngestError(f"Download failed: {argument}: {exc.reason}") from exc
with response:
final_url = response.geturl()
header = response.headers.get("Content-Type")
body = _read_limited(response, settings.max_download_bytes, argument)
parsed = urllib.parse.urlsplit(argument)
suffix = pathlib.PurePosixPath(parsed.path).suffix
media, charset = detect_media_type(body, suffix, header, self.registry)
extension = extension_for(media, suffix, self.registry)
slug = slug_from_url(argument)
dest_dir.mkdir(parents=True, exist_ok=True)
target = dest_dir / f"{slug}{extension}"
partial = target.with_name(target.name + ".partial")
try:
partial.write_bytes(body)
partial.replace(target)
except OSError:
if partial.exists():
partial.unlink()
raise
logger.info("downloaded %s as %s (%s bytes)", argument, media, len(body))
return FetchedSource(
argument=argument,
path=target,
origin=argument,
origin_kind="url",
final_url=final_url,
media_type=media,
charset=charset,
size_bytes=len(body),
slug=slug,
fetched=utc_now(),
created=True,
)
def _without_userinfo(url: str) -> str:
"""Drop userinfo so the request sends no credentials.
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.2.2.
"""
parts = urllib.parse.urlsplit(url)
if parts.username is None and parts.password is None:
return url
host = parts.hostname or ""
if parts.port:
host = f"{host}:{parts.port}"
return urllib.parse.urlunsplit((parts.scheme, host, parts.path, parts.query, parts.fragment))
def _read_limited(response: object, limit: int, url: str) -> bytes:
"""Read a response body, refusing anything over ``limit``.
Generated by AI/LLM: 2026-09-30 - a partial download is not returned.
"""
chunks: list[bytes] = []
total = 0
while True:
block = response.read(65536) # type: ignore[attr-defined]
if not block:
break
total += len(block)
if total > limit:
raise IngestError(f"Download exceeds {limit} bytes: {url}")
chunks.append(block)
return b"".join(chunks)
+96
View File
@@ -0,0 +1,96 @@
"""Append one ingest line to ``wiki/log.md``.
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.6. Does not edit ``index.md``.
"""
from __future__ import annotations
import logging
import pathlib
import re
logger = logging.getLogger(__name__)
_VERSION = re.compile(r"^version:\s*(\S+)\s*$", re.MULTILINE)
_UPDATED = re.compile(r"^updated:\s*(\S+)\s*$", re.MULTILINE)
class IngestLog:
"""Record a finished archive in ``wiki/log.md``.
Generated by AI/LLM: 2026-09-30 - bump patch version and ``updated``.
"""
def __init__(self, wiki_root: pathlib.Path) -> None:
"""Store the wiki root.
Args:
wiki_root: Directory that contains ``wiki/log.md``.
"""
self.wiki_root = wiki_root
def append(
self,
*,
day: str,
origin: str,
raw_rel: str,
page_rel: str | None,
dry_run: bool,
) -> None:
"""Append one bullet and bump the log frontmatter.
Generated by AI/LLM: 2026-09-30 - ``(raw)`` or ``(no page)``.
Args:
day: UTC date ``YYYY-MM-DD``.
origin: URL or resolved file path.
raw_rel: Raw path relative to the wiki root.
page_rel: Page path relative to the wiki root, or None.
dry_run: When true, leave the log untouched.
"""
if dry_run:
return
path = self.wiki_root / "wiki" / "log.md"
if page_rel:
bullet = f"- {day} - ingest {origin} -> {raw_rel} -> {page_rel} (raw)"
else:
bullet = f"- {day} - ingest {origin} -> {raw_rel} (no page)"
if not path.is_file():
logger.warning("wiki/log.md is missing; ingest was not recorded")
return
text = path.read_text(encoding="utf-8")
text = _bump_version(text)
text = _set_updated(text, day)
if not text.endswith("\n"):
text += "\n"
text += bullet + "\n"
path.write_text(text, encoding="utf-8", newline="\n")
logger.info("recorded %s", bullet)
def _bump_version(text: str) -> str:
"""Increment the patch number of the first ``version`` field.
Generated by AI/LLM: 2026-09-30 - ``2.0.0`` becomes ``2.0.1``.
"""
match = _VERSION.search(text)
if match is None:
return text
parts = match.group(1).split(".")
if len(parts) == 3 and all(part.isdigit() for part in parts):
parts[2] = str(int(parts[2]) + 1)
bumped = ".".join(parts)
return text[: match.start(1)] + bumped + text[match.end(1) :]
return text
def _set_updated(text: str, day: str) -> str:
"""Set the first ``updated`` field to ``day``.
Generated by AI/LLM: 2026-09-30 - replaces a placeholder or a previous date.
"""
match = _UPDATED.search(text)
if match is None:
return text
return text[: match.start(1)] + day + text[match.end(1) :]
+255
View File
@@ -0,0 +1,255 @@
"""Media-type detection and slug rules for wiki ingest.
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.2.4 and 7.25.2.6.
"""
from __future__ import annotations
import datetime
import mimetypes
import pathlib
import re
import urllib.parse
_SLUG_TRIM = 80
_UNUSABLE_MEDIA = {"", "application/octet-stream"}
def make_slug(text: str) -> str:
"""Turn free text into a wiki slug.
Generated by AI/LLM: 2026-09-30 - lowercase, hyphenate, and cap at 80.
Args:
text: Stem, host, or joined path segments.
Returns:
str: A slug, or ``source`` when the text has no usable characters.
"""
slug = re.sub(r"[^a-z0-9]+", "-", text.lower()).strip("-")[:_SLUG_TRIM]
return slug or "source"
def slug_from_url(url: str) -> str:
"""Build a slug from the last three path segments of a URL.
Generated by AI/LLM: 2026-09-30 - drop ``index`` and an empty path uses the host.
Args:
url: Absolute ``http`` or ``https`` URL.
Returns:
str: Slug for the downloaded file and the wiki page.
"""
parts = urllib.parse.urlsplit(url)
cleaned: list[str] = []
for segment in parts.path.split("/"):
if not segment:
continue
path = pathlib.PurePosixPath(segment)
stem = path.stem if path.suffix else segment
if stem.lower() == "index":
continue
cleaned.append(stem)
if not cleaned:
return make_slug(parts.hostname or "source")
return make_slug("-".join(cleaned[-3:]))
def slug_from_file(path: pathlib.Path) -> str:
"""Build a slug from a file stem.
Generated by AI/LLM: 2026-09-30 - local paths use the stem only.
Args:
path: Local file path. It does not have to exist.
Returns:
str: Slug for the archive name and the wiki page.
"""
return make_slug(path.stem)
def split_content_type(header: str | None) -> tuple[str | None, str | None]:
"""Split a Content-Type header into a media type and charset.
Generated by AI/LLM: 2026-09-30 - drop parameters other than charset.
``application/octet-stream`` is treated as missing so the extension
or the leading bytes can decide.
Args:
header: Raw Content-Type value, or None.
Returns:
tuple: Media type without parameters, and charset if one was set.
"""
if not header:
return None, None
parts = [part.strip() for part in header.split(";") if part.strip()]
media = parts[0].lower()
charset = None
for part in parts[1:]:
if "=" not in part:
continue
key, value = part.split("=", 1)
if key.strip().lower() == "charset":
charset = value.strip().strip("\"'")
if media in _UNUSABLE_MEDIA:
media = None
return media, charset
def media_for_extension(suffix: str, registry: object) -> str | None:
"""Map a file extension to a media type.
Generated by AI/LLM: 2026-09-30 - converter table first, then mimetypes.
Args:
suffix: Extension including the leading dot.
registry: Converter registry in registration order.
Returns:
str | None: Media type, or None when the extension is unknown.
"""
lowered = suffix.lower()
for converter in registry.converters: # type: ignore[attr-defined]
if lowered in converter.extensions and converter.media_types:
return converter.media_types[0]
guessed, _encoding = mimetypes.guess_type("file" + lowered)
return guessed
def sniff_media_type(data: bytes) -> str:
"""Guess a media type from the first 1024 bytes.
Generated by AI/LLM: 2026-09-30 - PDF, HTML, UTF-8 text, or octet-stream.
Args:
data: File bytes. Only the prefix is inspected.
Returns:
str: Detected media type.
"""
head = data[:1024]
if head.startswith(b"%PDF-"):
return "application/pdf"
sample = head.lstrip().lower()
if sample.startswith((b"<!doctype html", b"<html")):
return "text/html"
if b"\x00" not in head and _is_utf8(head):
return "text/plain"
return "application/octet-stream"
def detect_media_type(
data: bytes,
suffix: str,
header: str | None,
registry: object,
) -> tuple[str, str | None]:
"""Detect a media type and optional charset hint.
Generated by AI/LLM: 2026-09-30 - header, then extension, then bytes.
Args:
data: File bytes, used only when the header and extension miss.
suffix: Extension including the dot, or empty.
header: Content-Type header for a download, or None for a local file.
registry: Converter registry.
Returns:
tuple: Media type and charset hint.
"""
media, charset = split_content_type(header)
if media:
return media, charset
if suffix:
from_ext = media_for_extension(suffix, registry)
if from_ext:
return from_ext, charset
return sniff_media_type(data), charset
def extension_for(media_type: str, path_suffix: str, registry: object) -> str:
"""Choose the extension for a downloaded file.
Generated by AI/LLM: 2026-09-30 - converter extension, URL suffix, then ``.bin``.
Args:
media_type: Detected media type.
path_suffix: Suffix from the URL path, including the dot, or empty.
registry: Converter registry.
Returns:
str: Extension including the leading dot.
"""
converter = registry.find(media_type, "") # type: ignore[attr-defined]
if converter is not None and converter.extensions:
return converter.extensions[0]
if path_suffix:
suffix = path_suffix.lower()
return suffix if suffix.startswith(".") else f".{suffix}"
return ".bin"
def decode_bytes(data: bytes, charset: str | None) -> tuple[str, list[str]]:
"""Decode source bytes to text.
Generated by AI/LLM: 2026-09-30 - UTF-8, then the charset hint, then cp1252.
A BOM is stripped before UTF-8 is tried. A fallback decode adds one warning.
Args:
data: Raw file bytes.
charset: Optional charset from Content-Type.
Returns:
tuple: Decoded text and warnings.
"""
payload = data[3:] if data.startswith(b"\xef\xbb\xbf") else data
try:
return payload.decode("utf-8"), []
except UnicodeDecodeError:
pass
if charset:
try:
return payload.decode(charset), [f"decoded as {charset}; not valid UTF-8"]
except (UnicodeDecodeError, LookupError):
pass
return payload.decode("cp1252"), ["decoded as cp1252; not valid UTF-8"]
def utc_now() -> str:
"""Return the current UTC time as ``YYYY-MM-DDTHH:MM:SSZ``.
Generated by AI/LLM: 2026-09-30 - ingest ``fetched`` timestamp.
Returns:
str: UTC timestamp with a ``Z`` suffix.
"""
moment = datetime.datetime.now(datetime.UTC).replace(microsecond=0)
return moment.strftime("%Y-%m-%dT%H:%M:%SZ")
def utc_day() -> str:
"""Return today's UTC date as ``YYYY-MM-DD``.
Generated by AI/LLM: 2026-09-30 - page ``created`` and ``updated`` dates.
Returns:
str: UTC calendar date.
"""
return utc_now()[:10]
def _is_utf8(data: bytes) -> bool:
"""Return whether a byte prefix is valid UTF-8.
Generated by AI/LLM: 2026-09-30 - allow a multibyte character cut at 1024.
"""
try:
data.decode("utf-8")
except UnicodeDecodeError as exc:
return exc.start > 0 and exc.start >= len(data) - 3 and "unexpected end" in exc.reason
return True
+126
View File
@@ -0,0 +1,126 @@
"""Dataclasses passed between wiki ingest stages.
Generated by AI/LLM: 2026-09-30 - fetch, convert, and JSON result types.
"""
from __future__ import annotations
import dataclasses
import pathlib
class IngestError(Exception):
"""Raised when wiki ingest must stop before a page is written.
Generated by AI/LLM: 2026-09-30 - exit 1 errors from section 4.14.
"""
def __init__(self, message: str, exit_code: int = 1) -> None:
"""Store the stderr message and process exit code.
Args:
message: Text after ``Error: ``.
exit_code: Process status. Section 4.14 uses 1.
"""
super().__init__(message)
self.exit_code = exit_code
@dataclasses.dataclass
class FetchedSource:
"""A local file plus the facts the later stages need.
Generated by AI/LLM: 2026-09-30 - output of the fetch stage.
"""
argument: str
path: pathlib.Path
origin: str
origin_kind: str
final_url: str | None
media_type: str
charset: str | None
size_bytes: int
slug: str
fetched: str
created: bool
@dataclasses.dataclass
class ConversionResult:
"""Markdown produced from one archived file.
Generated by AI/LLM: 2026-09-30 - output of one converter.
"""
body: str
title: str | None
source_meta: dict
conversion: str
warnings: list[str]
converter: str
@dataclasses.dataclass
class ArchiveResult:
"""Where the raw copy lives after the archive stage.
Generated by AI/LLM: 2026-09-30 - raw path, hash, and reuse flag.
"""
path: pathlib.Path
raw_rel: str
sha256: str
raw_reused: bool
warnings: list[str]
@dataclasses.dataclass
class IngestResult:
"""One ingest run, including the JSON object from section 4.14.
Generated by AI/LLM: 2026-09-30 - stdout facts for a finished run.
"""
source: str
origin: str
origin_kind: str
media_type: str
raw_path: str | None
raw_reused: bool
sha256: str
duplicate_page: str | None
page_path: str | None
converter: str | None
conversion: str
status: str | None
warnings: list[str]
dry_run: bool
exit_code: int = 0
error: str | None = None
messages: list[str] = dataclasses.field(default_factory=list)
def to_dict(self) -> dict:
"""Return the JSON object from section 4.14.
Generated by AI/LLM: 2026-09-30 - drop CLI-only fields.
Returns:
dict: Keys and nulls from the ingest JSON contract.
"""
return {
"source": self.source,
"origin": self.origin,
"origin_kind": self.origin_kind,
"media_type": self.media_type,
"raw_path": self.raw_path,
"raw_reused": self.raw_reused,
"sha256": self.sha256,
"duplicate_page": self.duplicate_page,
"page_path": self.page_path,
"converter": self.converter,
"conversion": self.conversion,
"status": self.status,
"warnings": list(self.warnings),
"dry_run": self.dry_run,
}
+375
View File
@@ -0,0 +1,375 @@
"""Write one ``status: raw`` wiki page.
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.5. The tool never sets ``refined``.
"""
from __future__ import annotations
import logging
import os
import pathlib
import re
import yaml
from pymarkdown.api import PyMarkdownApiException
from fiona.mdoctor.mdoctor import MDoctor
from fiona.wiki.ingest.media import utc_day
from fiona.wiki.ingest.models import ArchiveResult, ConversionResult, FetchedSource, IngestError
from fiona.wiki.ingest.settings import IngestSettings, package_version
logger = logging.getLogger(__name__)
_FENCE = re.compile(r"^\s*(`{3,}|~{3,})")
_ATX = re.compile(r"^(#{1,6})(\s+)(.*)$")
_IMAGE = re.compile(r"!\[([^\]]*)\]\(([^)]+)\)")
_LINK = re.compile(r"(?<!!)\[([^\]]*)\]\(([^)]+)\)")
_PROTECTED = {"index.md", "log.md", "schema.md"}
class RawPageWriter:
"""Write the raw page and run the markdown fixer on that file only.
Generated by AI/LLM: 2026-09-30 - frontmatter order from SPEC 7.25.5.3.
"""
def __init__(self, wiki_root: pathlib.Path) -> None:
"""Store the wiki root.
Args:
wiki_root: Directory that contains ``wiki/``.
"""
self.wiki_root = wiki_root
def page_path(
self,
slug: str,
settings: IngestSettings,
target: str | None,
) -> pathlib.Path:
"""Resolve the page path and reject paths outside ``wiki/``.
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.5.1. Does not touch the disk.
Args:
slug: Page stem when ``target`` is a directory or omitted.
settings: Supplies ``default_target``.
target: ``-t`` value, or None.
Returns:
pathlib.Path: Absolute page path inside ``wiki/``.
Raises:
IngestError: The path escapes ``wiki/`` or names a protected file.
"""
wiki = (self.wiki_root / "wiki").resolve()
if target is None:
relative = f"{settings.default_target}/{slug}.md"
else:
relative = _strip_wiki_prefix(target)
suffix = pathlib.PurePosixPath(relative.replace("\\", "/")).suffix
if suffix != ".md":
relative = f"{relative}/{slug}.md" if relative else f"{slug}.md"
candidate = (wiki / relative).resolve()
try:
page_rel = candidate.relative_to(wiki).as_posix()
except ValueError as exc:
raise IngestError(f"--target must stay inside wiki/: {target}") from exc
if page_rel in _PROTECTED or page_rel.split("/", 1)[0] == "logs":
shown = target if target else page_rel
raise IngestError(f"--target cannot replace {shown}")
return candidate
def write(
self,
*,
source: FetchedSource,
archive: ArchiveResult,
conversion: ConversionResult,
page: pathlib.Path,
dry_run: bool,
) -> tuple[str, list[str]]:
"""Write the page, fix it, and return the relative path and warnings.
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.5.3 through 7.25.5.5.
Args:
source: Fetch facts stored in ``ingest``.
archive: Raw path, hash, and reuse flag.
conversion: Converter output.
page: Destination markdown file.
dry_run: When true, build the warnings and write nothing.
Returns:
tuple: Page path relative to the wiki, and the warning list.
"""
title = conversion.title or source.slug
content, link_warnings = _page_content(conversion.body, title, page, archive.path)
warnings = [*conversion.warnings, *archive.warnings, *link_warnings]
if dry_run:
return _rel(page, self.wiki_root), warnings
page.parent.mkdir(parents=True, exist_ok=True)
day = utc_day()
page.write_text(
_document(source, archive, conversion, content, warnings, day),
encoding="utf-8",
newline="\n",
)
_fix(page, self.wiki_root)
style_count = _style_violations(page, self.wiki_root)
if style_count:
warnings.append(f"{style_count} markdown style violations left")
fixed = _body_after_frontmatter(page.read_text(encoding="utf-8"))
if not fixed.lstrip("\n").startswith("# "):
fixed = content
page.write_text(
_document(source, archive, conversion, fixed.lstrip("\n"), warnings, day),
encoding="utf-8",
newline="\n",
)
logger.info("wrote %s", page)
return _rel(page, self.wiki_root), warnings
def _page_content(
body: str,
title: str,
page: pathlib.Path,
raw_path: pathlib.Path,
) -> tuple[str, list[str]]:
"""Demote extra H1s, drop dead relative links, and add the ingest intro.
Generated by AI/LLM: 2026-09-30 - the page has one H1 and a link to the raw file.
"""
shifted = _shift_headings(body)
rewritten, count = _rewrite_links(shifted, page.parent)
warnings = [f"{count} relative link(s) did not resolve"] if count else []
link = os.path.relpath(raw_path, page.parent).replace("\\", "/")
intro = f"Raw ingest of [{raw_path.name}]({link}). Not refined yet."
prepared = rewritten.strip("\n")
tail = f"{intro}\n\n{prepared}\n" if prepared else f"{intro}\n"
return f"# {title}\n\n{tail}", warnings
def _document(
source: FetchedSource,
archive: ArchiveResult,
conversion: ConversionResult,
content: str,
warnings: list[str],
day: str,
) -> str:
"""Render frontmatter plus the page content.
Generated by AI/LLM: 2026-09-30 - every key is present, in contract order.
"""
title = conversion.title or source.slug
document = {
"name": source.slug,
"version": "0.1.0",
"title": title,
"created": day,
"updated": day,
"type": "summary",
"status": "raw",
"sources": [archive.raw_rel],
"source_meta": conversion.source_meta or {},
"ingest": {
"origin": source.origin,
"origin_kind": source.origin_kind,
"final_url": source.final_url,
"fetched": source.fetched,
"media_type": source.media_type,
"size_bytes": source.size_bytes,
"sha256": archive.sha256,
"raw_reused": archive.raw_reused,
"converter": conversion.converter,
"conversion": conversion.conversion,
"warnings": list(warnings),
"tool": f"fiona-assistant {package_version()}",
},
"tags": [],
"confidence": "zero",
"llm": [{"model": "none"}],
"contested": False,
"contradictions": [],
}
dumped = yaml.safe_dump(document, sort_keys=False, allow_unicode=True, width=1000)
payload = content if content.endswith("\n") else content + "\n"
return f"---\n{dumped}---\n\n{payload}"
def _shift_headings(body: str) -> str:
"""Move every ATX heading down one level when the body has an H1.
Generated by AI/LLM: 2026-09-30 - H6 stays H6. Fenced lines are left alone.
"""
if not _has_h1(body):
return body
lines = body.splitlines(keepends=True)
fenced = False
shifted: list[str] = []
for line in lines:
ending, content = _split_ending(line)
fence = _FENCE.match(content)
if fence:
fenced = not fenced
shifted.append(line)
continue
match = None if fenced else _ATX.match(content)
if match is None:
shifted.append(line)
continue
marks = match.group(1)
if len(marks) < 6:
marks += "#"
shifted.append(f"{marks}{match.group(2)}{match.group(3)}{ending}")
return "".join(shifted)
def _has_h1(body: str) -> bool:
"""Return whether an unfenced ATX H1 is present.
Generated by AI/LLM: 2026-09-30 - detection does not change the body.
"""
fenced = False
for line in body.splitlines():
if _FENCE.match(line):
fenced = not fenced
continue
if not fenced and re.match(r"^#\s+\S", line):
return True
return False
def _rewrite_links(body: str, page_dir: pathlib.Path) -> tuple[str, int]:
"""Replace relative links and images that do not resolve from the page.
Generated by AI/LLM: 2026-09-30 - one warning carries the count.
"""
count = 0
lines = body.splitlines(keepends=True)
fenced = False
rewritten: list[str] = []
for line in lines:
ending, content = _split_ending(line)
if _FENCE.match(content):
fenced = not fenced
rewritten.append(line)
continue
if fenced:
rewritten.append(line)
continue
content, image_count = _replace_pattern(_IMAGE, content, page_dir)
content, link_count = _replace_pattern(_LINK, content, page_dir)
count += image_count + link_count
rewritten.append(content + ending)
return "".join(rewritten), count
def _replace_pattern(
pattern: re.Pattern[str],
content: str,
page_dir: pathlib.Path,
) -> tuple[str, int]:
"""Replace matches whose target does not exist.
Generated by AI/LLM: 2026-09-30 - external URLs and anchors stay.
"""
count = 0
def _swap(match: re.Match[str]) -> str:
nonlocal count
target = match.group(2).strip()
if _keeps_target(target, page_dir):
return match.group(0)
count += 1
return match.group(1)
return pattern.sub(_swap, content), count
def _keeps_target(target: str, page_dir: pathlib.Path) -> bool:
"""Return whether a link target should stay in the page.
Generated by AI/LLM: 2026-09-30 - http, mailto, and anchors are kept.
"""
lowered = target.lower()
if lowered.startswith(("http://", "https://", "mailto:", "#", "//")) or "://" in target:
return True
path_part = target.split("#", 1)[0].split("?", 1)[0]
if not path_part:
return True
resolved = (page_dir / path_part).resolve()
return resolved.exists()
def _fix(page: pathlib.Path, wiki_root: pathlib.Path) -> None:
"""Run the pymarkdown fixer on one page.
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.5.5. A fixer error is logged.
"""
try:
doctor = MDoctor(settings_file=wiki_root / "fiona-settings.json")
doctor.fix_file_programmatically(str(page))
except PyMarkdownApiException:
logger.warning("markdown fix failed for %s", page, exc_info=True)
def _style_violations(page: pathlib.Path, wiki_root: pathlib.Path) -> int:
"""Count pymarkdown violations still on the page.
Generated by AI/LLM: 2026-09-30 - the count becomes one warning.
"""
try:
doctor = MDoctor(settings_file=wiki_root / "fiona-settings.json")
return len(doctor.scan_file_programmatically(str(page)))
except PyMarkdownApiException:
logger.warning("markdown scan failed for %s", page, exc_info=True)
return 0
def _body_after_frontmatter(text: str) -> str:
"""Return the file body after the closing frontmatter marker.
Generated by AI/LLM: 2026-09-30 - keep the fixer output and replace the header.
"""
if not text.startswith("---"):
return text
parts = text.split("---", 2)
if len(parts) < 3:
return text
return parts[2].removeprefix("\n")
def _split_ending(line: str) -> tuple[str, str]:
"""Split a line into its ending and its content.
Generated by AI/LLM: 2026-09-30 - keep ``\\n`` and ``\\r\\n``.
"""
if line.endswith("\r\n"):
return "\r\n", line[:-2]
if line.endswith("\n"):
return "\n", line[:-1]
return "", line
def _strip_wiki_prefix(value: str) -> str:
"""Drop a leading ``wiki/`` from a ``-t`` value.
Generated by AI/LLM: 2026-09-30 - both forms are accepted.
"""
relative = value.replace("\\", "/").lstrip("/")
if relative == "wiki":
return ""
if relative.startswith("wiki/"):
return relative[5:]
return relative
def _rel(path: pathlib.Path, root: pathlib.Path) -> str:
"""Return a path relative to the wiki root using ``/``.
Generated by AI/LLM: 2026-09-30 - JSON page paths.
"""
return path.resolve().relative_to(root.resolve()).as_posix()
+536
View File
@@ -0,0 +1,536 @@
"""Run the wiki ingest stages in order.
Generated by AI/LLM: 2026-09-30 - SPEC 7.25. The pipeline holds no format logic.
"""
from __future__ import annotations
import argparse
import dataclasses
import json
import logging
import pathlib
import shutil
import sys
import tempfile
from fiona.librarian.md_frontmatter import FrontmatterError, parse_frontmatter
from fiona.wiki.ingest.archiver import RawArchiver
from fiona.wiki.ingest.converters import default_registry
from fiona.wiki.ingest.converters.base import ConversionError, ConverterRegistry
from fiona.wiki.ingest.fetchers import LocalFileFetcher, SourceFetcher, UrlFetcher, url_scheme
from fiona.wiki.ingest.log_writer import IngestLog
from fiona.wiki.ingest.media import slug_from_file, slug_from_url, utc_day
from fiona.wiki.ingest.models import FetchedSource, IngestError, IngestResult
from fiona.wiki.ingest.page_writer import RawPageWriter
from fiona.wiki.ingest.settings import IngestSettings, load_ingest_settings
logger = logging.getLogger(__name__)
_NO_CONVERTER = "No converter for {media}. Convert {raw} by hand."
class IngestPipeline:
"""Fetch, archive, convert, write, and record one source.
Generated by AI/LLM: 2026-09-30 - stages are constructor arguments.
"""
def __init__(
self,
wiki_root: pathlib.Path,
fetchers: list[SourceFetcher],
registry: ConverterRegistry,
archiver: RawArchiver,
page_writer: RawPageWriter,
log_writer: IngestLog,
settings: IngestSettings,
) -> None:
"""Store the stage objects. No converter is named here.
Args:
wiki_root: Wiki root directory.
fetchers: Tried in order. Each implements ``accepts`` and ``fetch``.
registry: Converter registry. Tests may register another converter.
archiver: Raw archive stage.
page_writer: Wiki page stage.
log_writer: ``wiki/log.md`` stage.
settings: Timeouts, size limit, and default paths.
"""
self.wiki_root = wiki_root
self.fetchers = fetchers
self.registry = registry
self.archiver = archiver
self.page_writer = page_writer
self.log_writer = log_writer
self.settings = settings
def run(
self,
argument: str,
*,
raw: str | None = None,
target: str | None = None,
raw_only: bool = False,
force: bool = False,
dry_run: bool = False,
) -> IngestResult:
"""Ingest one URL or local file.
Generated by AI/LLM: 2026-09-30 - exit 1 raises; exit 0 and 3 return a result.
Args:
argument: Source as given.
raw: ``-r`` path relative to ``raw/``.
target: ``-t`` path relative to ``wiki/``.
raw_only: Archive and record, skip convert and write.
force: Replace an existing page, including a duplicate.
dry_run: Report the run and write nothing under the wiki.
Returns:
IngestResult: Facts for stdout. ``exit_code`` is 0 or 3.
Raises:
IngestError: Exit 1. Nothing new remains under the wiki.
"""
scheme = url_scheme(argument)
if scheme is not None and scheme.lower() not in {"http", "https"}:
raise IngestError(f"Unsupported URL scheme: {scheme}")
slug = slug_from_url(argument) if scheme else slug_from_file(pathlib.Path(argument))
page = self.page_writer.page_path(slug, self.settings, target)
if page.is_file() and not force:
rel = page.resolve().relative_to(self.wiki_root.resolve()).as_posix()
raise IngestError(f"Page exists: {rel}. Use --force to replace it.")
fetcher = self._fetcher(argument)
fetcher.settings = self.settings
temp_dir: pathlib.Path | None = None
fetched: FetchedSource | None = None
archived = False
try:
if dry_run and scheme:
temp_dir = pathlib.Path(tempfile.mkdtemp(prefix="fiona-ingest-"))
dest = temp_dir
else:
dest = self.wiki_root / "ingest"
fetched = fetcher.fetch(argument, dest)
archive = self.archiver.archive(
fetched,
registry=self.registry,
settings=self.settings,
raw_override=raw,
dry_run=dry_run,
)
archived = True
return self._after_archive(
fetched,
archive,
page,
target_explicit=target is not None,
raw_only=raw_only,
force=force,
dry_run=dry_run,
)
except IngestError:
if fetched is not None and fetched.created and fetched.path.exists() and not archived:
fetched.path.unlink()
raise
finally:
if temp_dir is not None:
shutil.rmtree(temp_dir, ignore_errors=True)
def list_pending(self) -> list[dict]:
"""List wiki pages whose ``status`` is ``raw``, sorted by path.
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.9.
Returns:
list[dict]: ``page_path``, ``sources``, and ``origin`` for each page.
"""
wiki = self.wiki_root / "wiki"
if not wiki.is_dir():
return []
pages: list[dict] = []
for path in sorted(wiki.rglob("*.md"), key=lambda item: item.as_posix()):
if not path.is_file():
continue
try:
front = parse_frontmatter(path)
except FrontmatterError:
continue
if front.get("status") != "raw":
continue
sources = front.get("sources") or []
if isinstance(sources, str):
sources = [sources]
ingest = front.get("ingest") if isinstance(front.get("ingest"), dict) else {}
origin = ingest.get("origin") if isinstance(ingest, dict) else None
pages.append({
"page_path": path.resolve().relative_to(self.wiki_root.resolve()).as_posix(),
"sources": list(sources),
"origin": origin,
})
return pages
def _fetcher(self, argument: str) -> SourceFetcher:
"""Return the first fetcher that accepts the argument.
Generated by AI/LLM: 2026-09-30 - URL fetcher is registered before the local one.
"""
for fetcher in self.fetchers:
if fetcher.accepts(argument):
return fetcher
raise IngestError(f"Source not found: {argument}")
def _after_archive(
self,
source: FetchedSource,
archive: object,
page: pathlib.Path,
*,
target_explicit: bool,
raw_only: bool,
force: bool,
dry_run: bool,
) -> IngestResult:
"""Finish convert, write, and record after a successful archive.
Generated by AI/LLM: 2026-09-30 - duplicate, raw-only, and missing converter exits.
"""
messages = _fetched_line(source)
if archive.raw_reused: # type: ignore[attr-defined]
short = archive.sha256[:12] # type: ignore[attr-defined]
messages.append(
f"Archived: {archive.raw_rel} (already archived, sha256 {short})" # type: ignore[attr-defined]
)
else:
messages.append(f"Archived: {archive.raw_rel}") # type: ignore[attr-defined]
duplicate = _citing_page(self.wiki_root, archive.raw_rel, archive.sha256) # type: ignore[attr-defined]
if duplicate and not force:
messages.append(f"Duplicate: {duplicate} already cites this source.")
return _result(
source,
archive,
messages=messages,
duplicate_page=duplicate,
dry_run=dry_run,
warnings=list(archive.warnings), # type: ignore[attr-defined]
)
if duplicate and force and not target_explicit:
page = self.wiki_root / duplicate
if raw_only:
self.log_writer.append(
day=utc_day(),
origin=source.origin,
raw_rel=archive.raw_rel, # type: ignore[attr-defined]
page_rel=None,
dry_run=dry_run,
)
return _result(source, archive, messages=messages, dry_run=dry_run, warnings=list(archive.warnings)) # type: ignore[attr-defined]
extension = archive.path.suffix # type: ignore[attr-defined]
converter = self.registry.find(source.media_type, extension)
if converter is None:
messages.append(_NO_CONVERTER.format(media=source.media_type, raw=archive.raw_rel)) # type: ignore[attr-defined]
self._record(source, archive, None, dry_run)
return _result(
source,
archive,
messages=messages,
dry_run=dry_run,
exit_code=3,
warnings=list(archive.warnings), # type: ignore[attr-defined]
)
read_path = archive.path if archive.path.is_file() else source.path # type: ignore[attr-defined]
try:
conversion = converter.convert(dataclasses.replace(source, path=read_path))
except ConversionError as exc:
messages.append(_NO_CONVERTER.format(media=source.media_type, raw=archive.raw_rel)) # type: ignore[attr-defined]
self._record(source, archive, None, dry_run)
return _result(
source,
archive,
messages=messages,
dry_run=dry_run,
exit_code=3,
error=f"Conversion failed: {exc.converter}: {exc.reason}",
warnings=list(archive.warnings), # type: ignore[attr-defined]
)
page_rel, warnings = self.page_writer.write(
source=source,
archive=archive, # type: ignore[arg-type]
conversion=conversion,
page=page,
dry_run=dry_run,
)
messages.append(
f"Wrote: {page_rel} (converter {conversion.converter}, conversion {conversion.conversion})"
)
messages.extend(f"Warning: {warning}" for warning in warnings)
messages.append(f"Next: refine {page_rel}, then set status: refined.")
self._record(source, archive, page_rel, dry_run)
return _result(
source,
archive,
messages=messages,
page_path=page_rel,
converter=conversion.converter,
conversion=conversion.conversion,
status="raw",
dry_run=dry_run,
warnings=warnings,
)
def _record(self, source: FetchedSource, archive: object, page_rel: str | None, dry_run: bool) -> None:
"""Append the log line for a run that archived a source.
Generated by AI/LLM: 2026-09-30 - duplicates do not call this.
"""
self.log_writer.append(
day=utc_day(),
origin=source.origin,
raw_rel=archive.raw_rel, # type: ignore[attr-defined]
page_rel=page_rel,
dry_run=dry_run,
)
def default_pipeline(wiki_root: pathlib.Path) -> IngestPipeline:
"""Build the pipeline with the shipped fetchers and converters.
Generated by AI/LLM: 2026-09-30 - URL fetcher, then local file fetcher.
Args:
wiki_root: Wiki root directory.
Returns:
IngestPipeline: Ready to ``run`` or ``list_pending``.
"""
registry = default_registry()
return IngestPipeline(
wiki_root=wiki_root,
fetchers=[UrlFetcher(registry), LocalFileFetcher(registry)],
registry=registry,
archiver=RawArchiver(wiki_root),
page_writer=RawPageWriter(wiki_root),
log_writer=IngestLog(wiki_root),
settings=load_ingest_settings(wiki_root),
)
def render_stdout(result: IngestResult) -> str:
"""Return the human report, including the dry-run banner.
Generated by AI/LLM: 2026-09-30 - section 4.14 line order.
"""
lines: list[str] = []
if result.dry_run:
lines.append("Dry run - no files will be modified:")
lines.extend(result.messages)
return "\n".join(lines)
def add_ingest_arguments(parser: argparse.ArgumentParser) -> None:
"""Add the section 4.14 flags to an ingest parser.
Generated by AI/LLM: 2026-09-30 - the CLI subparser owns these flags, including --help.
Args:
parser: The ``wiki ingest`` parser.
"""
parser.add_argument("source", nargs="?", help="http(s) URL or file path")
parser.add_argument(
"--wiki",
default=".",
help="llm-wiki directory (default: .)",
)
parser.add_argument("-r", "--raw", dest="raw", help="raw path, relative to raw/")
parser.add_argument(
"-t",
"--target",
dest="target",
help="page path, relative to wiki/",
)
parser.add_argument(
"--raw-only",
action="store_true",
help="archive only; do not write a page",
)
parser.add_argument(
"-f",
"--force",
action="store_true",
help="replace an existing page",
)
parser.add_argument(
"--dry-run",
action="store_true",
help="print the plan and write nothing in the wiki",
)
parser.add_argument("--json", action="store_true", help="print only the JSON result")
parser.add_argument(
"--pending",
action="store_true",
help="list pages whose status is raw",
)
def ingest_main(argv: list[str] | None = None) -> int:
"""CLI entry for ``fiona-assistant wiki ingest``.
Generated by AI/LLM: 2026-09-30 - section 4.14 exit codes.
Args:
argv: Arguments after the subcommand. ``None`` reads the process arguments.
Returns:
int: 0, 1, 2, or 3.
"""
parser = argparse.ArgumentParser(prog="fiona-assistant wiki ingest")
add_ingest_arguments(parser)
return execute_ingest(parser.parse_args(argv), parser)
def execute_ingest(args: argparse.Namespace, parser: argparse.ArgumentParser) -> int:
"""Run one ingest from an already parsed namespace.
Generated by AI/LLM: 2026-09-30 - shared by ingest_main and the wiki subcommand.
Args:
args: Parsed source, flags, and paths.
parser: Parser used for the mutual-exclusion error.
Returns:
int: 0, 1, 2, or 3.
"""
if bool(args.source) == bool(args.pending):
parser.error("provide a source or --pending, not both")
try:
root = _require_wiki(args.wiki)
pipeline = default_pipeline(root)
if args.pending:
return _print_pending(pipeline, args.json)
result = pipeline.run(
args.source,
raw=args.raw,
target=args.target,
raw_only=args.raw_only,
force=args.force,
dry_run=args.dry_run,
)
except IngestError as exc:
print(f"Error: {exc}", file=sys.stderr)
return exc.exit_code
if result.error:
print(f"Error: {result.error}", file=sys.stderr)
if args.json:
print(json.dumps(result.to_dict(), indent=2))
else:
text = render_stdout(result)
if text:
print(text)
return result.exit_code
def _require_wiki(directory: str) -> pathlib.Path:
"""Return the wiki root or raise when settings are missing.
Generated by AI/LLM: 2026-09-30 - section 4.14 directory error.
"""
root = pathlib.Path(directory)
if not (root / "fiona-settings.json").is_file():
raise IngestError(f"Directory '{directory}' is not an llm-wiki directory")
return root
def _print_pending(pipeline: IngestPipeline, json_output: bool) -> int:
"""Print the pending-page report.
Generated by AI/LLM: 2026-09-30 - human lines or one JSON object.
"""
pages = pipeline.list_pending()
if json_output:
print(json.dumps({"pending": pages, "total": len(pages)}, indent=2))
return 0
print(f"Raw pages: {len(pages)}")
for page in pages:
first = page["sources"][0] if page["sources"] else ""
print(f" - {page['page_path']} ({first})")
return 0
def _fetched_line(source: FetchedSource) -> list[str]:
"""Return the ``Fetched:`` line for a URL, or an empty list.
Generated by AI/LLM: 2026-09-30 - local files do not print Fetched.
"""
if source.origin_kind != "url":
return []
return [f"Fetched: {source.argument} ({source.media_type}, {source.size_bytes} bytes)"]
def _citing_page(wiki_root: pathlib.Path, raw_rel: str, digest: str) -> str | None:
"""Return a page that already cites this raw path or hash.
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.3.5. First path in sorted order.
"""
wiki = wiki_root / "wiki"
if not wiki.is_dir():
return None
matches: list[str] = []
for path in wiki.rglob("*.md"):
if not path.is_file():
continue
try:
front = parse_frontmatter(path)
except FrontmatterError:
continue
sources = front.get("sources") or []
if isinstance(sources, str):
sources = [sources]
ingest = front.get("ingest") if isinstance(front.get("ingest"), dict) else {}
sha = ingest.get("sha256") if isinstance(ingest, dict) else None
if raw_rel in list(sources) or sha == digest:
matches.append(path.resolve().relative_to(wiki_root.resolve()).as_posix())
if not matches:
return None
matches.sort()
return matches[0]
def _result(
source: FetchedSource,
archive: object,
*,
messages: list[str],
dry_run: bool,
warnings: list[str],
duplicate_page: str | None = None,
page_path: str | None = None,
converter: str | None = None,
conversion: str = "none",
status: str | None = None,
exit_code: int = 0,
error: str | None = None,
) -> IngestResult:
"""Build an ingest result from the archive and any page that was written.
Generated by AI/LLM: 2026-09-30 - JSON fields from section 4.14.
"""
return IngestResult(
source=source.argument,
origin=source.origin,
origin_kind=source.origin_kind,
media_type=source.media_type,
raw_path=archive.raw_rel, # type: ignore[attr-defined]
raw_reused=archive.raw_reused, # type: ignore[attr-defined]
sha256=archive.sha256, # type: ignore[attr-defined]
duplicate_page=duplicate_page,
page_path=page_path,
converter=converter,
conversion=conversion,
status=status,
warnings=warnings,
dry_run=dry_run,
exit_code=exit_code,
error=error,
messages=messages,
)
+107
View File
@@ -0,0 +1,107 @@
"""Ingest settings from ``fiona-settings.json``.
Generated by AI/LLM: 2026-09-30 - SPEC 5.1 ``fiona.wiki.ingest`` keys.
"""
from __future__ import annotations
import dataclasses
import importlib.metadata
import json
import pathlib
def package_version() -> str:
"""Return the installed fiona-assistant version.
Generated by AI/LLM: 2026-09-30 - user agent and ``ingest.tool``.
Returns:
str: Distribution version, or ``0.99.0`` when metadata is absent.
"""
try:
return importlib.metadata.version("fiona-assistant")
except importlib.metadata.PackageNotFoundError:
return "0.99.0"
@dataclasses.dataclass
class IngestSettings:
"""Runtime limits and paths for one ingest.
Generated by AI/LLM: 2026-09-30 - defaults from the settings table.
"""
timeout_seconds: float
max_download_bytes: int
user_agent: str
default_target: str
raw_categories: dict[str, str]
def default_settings() -> IngestSettings:
"""Return the settings used when the wiki omits ``fiona.wiki.ingest``.
Generated by AI/LLM: 2026-09-30 - table defaults from SPEC 5.1.
Returns:
IngestSettings: Built-in limits and paths.
"""
return IngestSettings(
timeout_seconds=30,
max_download_bytes=52428800,
user_agent=f"fiona-assistant/{package_version()}",
default_target="sources",
raw_categories={},
)
def load_ingest_settings(wiki_root: pathlib.Path) -> IngestSettings:
"""Load ingest settings, falling back per key when a value is unusable.
Generated by AI/LLM: 2026-09-30 - absent or mistyped keys keep the default.
Args:
wiki_root: Directory that contains ``fiona-settings.json``.
Returns:
IngestSettings: Merged settings for this wiki.
"""
settings = default_settings()
path = wiki_root / "fiona-settings.json"
if not path.is_file():
return settings
try:
document = json.loads(path.read_text(encoding="utf-8"))
except json.JSONDecodeError:
return settings
section = {}
if isinstance(document, dict):
wiki = document.get("fiona", {})
if isinstance(wiki, dict):
wiki = wiki.get("wiki", {})
if isinstance(wiki, dict):
raw = wiki.get("ingest", {})
if isinstance(raw, dict):
section = raw
timeout = section.get("timeout_seconds", settings.timeout_seconds)
if isinstance(timeout, (int, float)) and not isinstance(timeout, bool) and timeout > 0:
settings.timeout_seconds = float(timeout)
limit = section.get("max_download_bytes", settings.max_download_bytes)
if isinstance(limit, int) and not isinstance(limit, bool) and limit > 0:
settings.max_download_bytes = limit
agent = section.get("user_agent")
if isinstance(agent, str) and agent.strip():
settings.user_agent = agent
target = section.get("default_target")
if isinstance(target, str) and target.strip() and ".." not in target.replace("\\", "/"):
settings.default_target = target.strip().strip("/")
categories = section.get("raw_categories")
if isinstance(categories, dict):
cleaned = {
str(key): value
for key, value in categories.items()
if isinstance(value, str) and value.strip()
}
settings.raw_categories = cleaned
return settings
+31 -2
View File
@@ -51,6 +51,29 @@ __all__ = [
# Check Runners
# =============================================================================
def _outside_raw(wiki_root: Path, path: str) -> bool:
"""Return whether a reported path is outside the raw archive.
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.12. Sanity checks skip ``raw/``.
Args:
wiki_root: Wiki root directory.
path: Violation path, absolute or relative to the wiki.
Returns:
bool: True when the path is not under ``raw/``.
"""
text = path.replace("\\", "/")
if text == "raw" or text.startswith("raw/"):
return False
candidate = Path(path)
try:
relative = candidate.resolve().relative_to(wiki_root.resolve())
except (ValueError, OSError):
return True
return not relative.parts or relative.parts[0] != "raw"
def run_markdown_lint(wiki_root: Path) -> dict:
"""Run markdownlint (mdoctor) over the wiki.
@@ -62,7 +85,10 @@ def run_markdown_lint(wiki_root: Path) -> dict:
the raw mdoctor violations.
"""
mdoctor = MDoctor(settings_file=wiki_root / SETTINGS_FILE)
violations = mdoctor.scan_file_programmatically(str(wiki_root))
violations = [
item for item in mdoctor.scan_file_programmatically(str(wiki_root))
if _outside_raw(wiki_root, str(item.get("scan_file", "")))
]
report = mdoctor.format_violations(
mdoctor.compress_violations(violations, target=str(wiki_root))
)
@@ -85,7 +111,10 @@ def run_python_lint(wiki_root: Path) -> dict:
the raw ruff violations.
"""
ruff_linter = PythonRuff()
violations = ruff_linter.lint(str(wiki_root))
violations = [
item for item in ruff_linter.lint(str(wiki_root))
if _outside_raw(wiki_root, str(item.get("filename", "")))
]
report = ruff_linter.format_violations(violations, str(wiki_root))
return {
"result": 0 if not violations else 1,
+18 -1
View File
@@ -94,6 +94,23 @@ class TestFionaMain:
captured = capsys.readouterr()
assert "usage" in captured.out.lower()
def test_main_wiki_ingest_help(self, capsys):
"""Test wiki ingest --help is printed by the ingest parser.
Generated by AI/LLM: 2026-09-30 - SPEC 4.14 help stays on this command.
"""
with (
patch("sys.argv", ["fiona-assistant", "wiki", "ingest", "--help"]),
pytest.raises(SystemExit) as exc_info,
):
main()
assert exc_info.value.code == 0
captured = capsys.readouterr()
assert "fiona-assistant wiki ingest" in captured.out
assert "--pending" in captured.out
assert "--raw-only" in captured.out
def test_main_mdoctor_default(self, capsys):
"""Test main() with mdoctor command and default arguments."""
with tempfile.TemporaryDirectory() as temp_dir:
@@ -440,7 +457,7 @@ class TestFionaMain:
# Create test markdown files with valid frontmatter
with open(os.path.join(temp_dir, "test1.md"), 'w') as f:
f.write("---\nname: Test1\nversion: 1.0.0\n---\n# Test1\n")
with open(os.path.join(temp_dir, "test2.md"), 'w') as f:
f.write("---\nname: Test2\nversion: 2.0.0\n---\n# Test2\n")
+729
View File
@@ -0,0 +1,729 @@
"""Tests for wiki ingest (SPEC 7.25).
Generated by AI/LLM: 2026-09-30 - acceptance checks for fetch, archive, convert, and write.
"""
from __future__ import annotations
import contextlib
import email.message
import hashlib
import io
import json
import pathlib
import shutil
import typing
import urllib.error
import urllib.request
import pypdf
import pypdf.generic
import pytest
import yaml
from fiona.customlinter.fmlint import FrontmatterLinter
from fiona.customlinter.mdlinter import MarkdownLinter
from fiona.paths import llm_wiki_template
from fiona.wiki.ingest.converters.base import SourceConverter
from fiona.wiki.ingest.fetchers import UrlFetcher
from fiona.wiki.ingest.media import slug_from_url
from fiona.wiki.ingest.models import ConversionResult, FetchedSource, IngestError
from fiona.wiki.ingest.pipeline import default_pipeline, ingest_main
from fiona.wiki.sanity_manager import run_wiki_checks
_FRONTMATTER_KEYS = (
"name",
"version",
"title",
"created",
"updated",
"type",
"status",
"sources",
"source_meta",
"ingest",
"tags",
"confidence",
"llm",
"contested",
"contradictions",
)
_INGEST_KEYS = (
"origin",
"origin_kind",
"final_url",
"fetched",
"media_type",
"size_bytes",
"sha256",
"raw_reused",
"converter",
"conversion",
"warnings",
"tool",
)
class _FakeResponse:
"""Minimal urllib response for one canned body."""
def __init__(self, body: bytes, header: str, url: str) -> None:
self._body = body
self._offset = 0
self.headers = email.message.Message()
self.headers["Content-Type"] = header
self._url = url
self.status = 200
def read(self, amount: int = -1) -> bytes:
"""Return the next chunk of the canned body."""
if amount is None or amount < 0:
amount = len(self._body) - self._offset
chunk = self._body[self._offset:self._offset + amount]
self._offset += len(chunk)
return chunk
def geturl(self) -> str:
"""Return the URL after redirects."""
return self._url
def __enter__(self) -> typing.Self:
return self
def __exit__(self, *_args: object) -> bool:
return False
class _FakeOpener:
"""URL opener that never touches the network."""
def __init__(self, body: bytes, header: str, *, status: int = 200, final_url: str | None = None) -> None:
self.body = body
self.header = header
self.status = status
self.final_url = final_url
self.requests: list[urllib.request.Request] = []
self.timeout: float | None = None
self.calls = 0
def open(self, request: urllib.request.Request, timeout: float | None = None) -> _FakeResponse:
"""Record the request and return the canned response or a 404."""
self.calls += 1
self.requests.append(request)
self.timeout = timeout
if self.status != 200:
raise urllib.error.HTTPError(
request.full_url,
self.status,
"Not Found",
email.message.Message(),
io.BytesIO(b""),
)
final = self.final_url or request.full_url
return _FakeResponse(self.body, self.header, final)
class _XTestConverter(SourceConverter):
"""Converter registered only by the test, for ``application/x-test``."""
name = "xtest"
media_types = ("application/x-test",)
extensions = (".xtest",)
raw_category = "articles"
def convert(self, source: FetchedSource) -> ConversionResult:
"""Return a fixed body so the test can see the registration."""
return ConversionResult(
body="xtest body\n",
title="X Test",
source_meta={},
conversion="complete",
warnings=[],
converter=self.name,
)
def _wiki(tmp_path: pathlib.Path) -> pathlib.Path:
"""Create a small llm-wiki with settings, a log, and an index."""
root = tmp_path / "wiki"
(root / "wiki").mkdir(parents=True)
(root / "ingest").mkdir()
(root / "raw").mkdir()
(root / "personal").mkdir()
shutil.copy2(llm_wiki_template() / "fiona-settings.json", root / "fiona-settings.json")
(root / "wiki" / "log.md").write_text(
"---\n"
"name: log\n"
"title: Wiki Log\n"
"version: 2.0.0\n"
"type: summary\n"
"created: 2026-09-30\n"
"updated: 2026-09-30\n"
"confidence: trusted\n"
"llm:\n"
" - model: none\n"
"contested: false\n"
"contradictions: []\n"
"---\n\n"
"# Wiki Log\n\n"
"- 2026-09-30 - Initialized the llm-wiki.\n",
encoding="utf-8",
newline="\n",
)
(root / "wiki" / "index.md").write_text(
"---\nname: index\nversion: 1.0.0\n---\n\n# Index\n",
encoding="utf-8",
newline="\n",
)
return root
def _run(root: pathlib.Path, *args: str) -> tuple[int, str, str]:
"""Run the ingest CLI against ``root`` and capture both streams."""
stdout = io.StringIO()
stderr = io.StringIO()
with contextlib.redirect_stdout(stdout), contextlib.redirect_stderr(stderr):
code = ingest_main(["--wiki", str(root), *args])
return code, stdout.getvalue(), stderr.getvalue()
def _front(path: pathlib.Path) -> dict:
"""Parse page frontmatter with types left intact."""
text = path.read_text(encoding="utf-8")
return yaml.safe_load(text.split("---", 2)[1])
def _pdf(texts: list[str], *, title: str | None = None, password: str | None = None) -> bytes:
"""Build a PDF whose pages contain the given strings."""
writer = pypdf.PdfWriter()
for text in texts:
page = writer.add_blank_page(width=612, height=792)
if text:
_pdf_text(writer, page, text)
if title:
writer.add_metadata({"/Title": title})
if password:
writer.encrypt(password)
buffer = io.BytesIO()
writer.write(buffer)
return buffer.getvalue()
def _pdf_text(writer: pypdf.PdfWriter, page: pypdf.PageObject, text: str) -> None:
"""Draw one Helvetica string on a page."""
font = pypdf.generic.DictionaryObject({
pypdf.generic.NameObject("/Type"): pypdf.generic.NameObject("/Font"),
pypdf.generic.NameObject("/Subtype"): pypdf.generic.NameObject("/Type1"),
pypdf.generic.NameObject("/BaseFont"): pypdf.generic.NameObject("/Helvetica"),
})
font_ref = writer._add_object(font)
fonts = pypdf.generic.DictionaryObject({pypdf.generic.NameObject("/F1"): font_ref})
page[pypdf.generic.NameObject("/Resources")] = pypdf.generic.DictionaryObject({
pypdf.generic.NameObject("/Font"): fonts,
})
stream = pypdf.generic.DecodedStreamObject()
stream.set_data(f"BT /F1 12 Tf 72 100 Td ({text}) Tj ET".encode("latin-1"))
page[pypdf.generic.NameObject("/Contents")] = writer._add_object(stream)
def _hashes(root: pathlib.Path) -> dict[str, str]:
"""Hash every file under a wiki."""
found: dict[str, str] = {}
for path in root.rglob("*"):
if path.is_file():
found[path.relative_to(root).as_posix()] = hashlib.sha256(path.read_bytes()).hexdigest()
return found
def test_slug_from_console_url() -> None:
"""SPEC 7.25.2.6: the last three segments become one hyphenated slug."""
assert slug_from_url("https://host/job/performance_TL153/2108/console") == (
"performance-tl153-2108-console"
)
def test_markdown_gets_frontmatter_and_keeps_body(tmp_path: pathlib.Path) -> None:
"""A markdown file keeps its wording. Missing page frontmatter is added."""
root = _wiki(tmp_path)
source = root / "ingest" / "Notes.md"
source.write_text("# Kept\n\nThe body stays.\n", encoding="utf-8", newline="\n")
code, _stdout, stderr = _run(root, str(source))
assert code == 0, stderr
assert not source.exists()
raw = root / "raw" / "articles" / "notes.md"
assert raw.read_text(encoding="utf-8") == "# Kept\n\nThe body stays.\n"
page = root / "wiki" / "sources" / "notes.md"
front = _front(page)
assert front["status"] == "raw"
assert front["source_meta"] == {}
assert "The body stays." in page.read_text(encoding="utf-8")
assert front["title"] == "Kept"
text = page.read_text(encoding="utf-8")
assert [line for line in text.splitlines() if line.startswith("# ")] == ["# Kept"]
def test_markdown_frontmatter_moves_to_source_meta(tmp_path: pathlib.Path) -> None:
"""Source frontmatter is metadata. The page still receives ingest frontmatter."""
root = _wiki(tmp_path)
source = tmp_path / "ready.md"
source.write_text(
"---\ntitle: Real Title\nauthor: Ada\n---\n\nParagraph.\n",
encoding="utf-8",
newline="\n",
)
code, _stdout, stderr = _run(root, str(source))
assert code == 0, stderr
page = root / "wiki" / "sources" / "ready.md"
front = _front(page)
assert front["title"] == "Real Title"
assert front["source_meta"]["author"] == "Ada"
assert "Paragraph." in page.read_text(encoding="utf-8")
assert source.read_text(encoding="utf-8").startswith("---\n")
def test_text_becomes_markdown_for_later_refinement(tmp_path: pathlib.Path) -> None:
"""A text file becomes a fenced page at status raw. The tool does not summarize it."""
root = _wiki(tmp_path)
source = tmp_path / "trace.log"
source.write_text("alpha\n```\nbeta\n", encoding="utf-8", newline="\n")
code, stdout, stderr = _run(root, str(source))
assert code == 0, stderr
page = root / "wiki" / "sources" / "trace.md"
front = _front(page)
assert front["status"] == "raw"
assert front["ingest"]["converter"] == "text"
assert front["ingest"]["conversion"] == "complete"
body = page.read_text(encoding="utf-8")
assert "````text" in body
assert "alpha" in body
assert "Not refined yet." in body
assert "Next: refine" in stdout
code, pending, _stderr = _run(root, "--pending")
assert code == 0
assert "wiki/sources/trace.md" in pending
def test_text_fallback_decode_warns(tmp_path: pathlib.Path) -> None:
"""Bytes that are not UTF-8 are decoded with a warning."""
root = _wiki(tmp_path)
source = tmp_path / "legacy.txt"
source.write_bytes(b"caf\x80")
code, _stdout, stderr = _run(root, str(source))
assert code == 0, stderr
front = _front(root / "wiki" / "sources" / "legacy.md")
assert any("cp1252" in warning for warning in front["ingest"]["warnings"])
def test_register_converter_without_pipeline_edits(tmp_path: pathlib.Path) -> None:
"""SPEC 7.25.1.2: a new format is one class and one registration."""
root = _wiki(tmp_path)
sample = tmp_path / "sample.xtest"
sample.write_bytes(b"hello")
pipeline = default_pipeline(root)
pipeline.registry.register(_XTestConverter)
result = pipeline.run(str(sample))
assert result.exit_code == 0
assert result.converter == "xtest"
assert (root / "raw" / "articles" / "sample.xtest").is_file()
assert "xtest body" in (root / "wiki" / "sources" / "sample.md").read_text(encoding="utf-8")
def test_ftp_scheme_writes_nothing(tmp_path: pathlib.Path) -> None:
"""SPEC 7.25.2.1: an unsupported scheme exits 1 and writes nothing."""
root = _wiki(tmp_path)
before = _hashes(root)
code, stdout, stderr = _run(root, "ftp://host/a.pdf")
assert code == 1
assert stdout == ""
assert "Unsupported URL scheme: ftp" in stderr
assert _hashes(root) == before
def test_missing_file_and_directory(tmp_path: pathlib.Path) -> None:
"""A missing path and a directory each exit 1."""
root = _wiki(tmp_path)
code, stdout, stderr = _run(root, str(tmp_path / "missing.txt"))
assert code == 1
assert stdout == ""
assert "Source not found:" in stderr
code, stdout, stderr = _run(root, str(tmp_path))
assert code == 1
assert stdout == ""
assert "Source is a directory:" in stderr
def test_download_uses_content_type_extension(tmp_path: pathlib.Path) -> None:
"""SPEC 7.25.2.5: a PDF response with no suffix is saved as ``.pdf``."""
root = _wiki(tmp_path)
body = _pdf(["Hello"])
opener = _FakeOpener(body, "application/pdf", final_url="https://host/report")
pipeline = default_pipeline(root)
pipeline.fetchers[0] = UrlFetcher(pipeline.registry, opener=opener)
result = pipeline.run("https://host/report")
assert result.exit_code == 0
assert result.raw_path == "raw/papers/report.pdf"
assert (root / "raw" / "papers" / "report.pdf").read_bytes() == body
assert not (root / "ingest" / "report.pdf").exists()
assert opener.requests[0].get_header("Authorization") is None
assert "fiona-assistant/" in opener.requests[0].get_header("User-agent")
def test_download_404_leaves_ingest_unchanged(tmp_path: pathlib.Path) -> None:
"""SPEC 7.25.2.3: a 404 exits 1 and does not add files."""
root = _wiki(tmp_path)
(root / "ingest" / "keep.txt").write_text("stay", encoding="utf-8")
before = sorted(path.name for path in (root / "ingest").iterdir())
opener = _FakeOpener(b"", "text/plain", status=404)
pipeline = default_pipeline(root)
pipeline.fetchers[0] = UrlFetcher(pipeline.registry, opener=opener)
with pytest.raises(IngestError, match="404"):
pipeline.run("https://host/missing")
assert sorted(path.name for path in (root / "ingest").iterdir()) == before
def test_download_over_limit_is_deleted(tmp_path: pathlib.Path) -> None:
"""A body over the limit exits 1 and leaves no partial file."""
root = _wiki(tmp_path)
pipeline = default_pipeline(root)
pipeline.settings.max_download_bytes = 4
opener = _FakeOpener(b"0123456789", "text/plain")
pipeline.fetchers[0] = UrlFetcher(pipeline.registry, opener=opener)
with pytest.raises(IngestError, match="exceeds 4 bytes"):
pipeline.run("https://host/big")
assert list((root / "ingest").iterdir()) == []
def test_extensionless_pdf_uses_pdf_converter(tmp_path: pathlib.Path) -> None:
"""SPEC 7.25.2.4: leading ``%PDF-`` bytes select the pdf converter."""
root = _wiki(tmp_path)
source = tmp_path / "notes"
source.write_bytes(_pdf(["Page text"]))
code, _stdout, stderr = _run(root, str(source))
assert code == 0, stderr
page = root / "wiki" / "sources" / "notes.md"
assert _front(page)["ingest"]["converter"] == "pdf"
assert (root / "raw" / "papers" / "notes.pdf").is_file()
def test_raw_override_and_escape(tmp_path: pathlib.Path) -> None:
"""SPEC 7.25.3.2: ``-r`` places the file, and ``..`` exits 1."""
root = _wiki(tmp_path)
html = tmp_path / "page.html"
html.write_text("<html><body><article><p>Hi</p></article></body></html>", encoding="utf-8")
code, _stdout, stderr = _run(root, str(html), "-r", "jenkins/tl153")
assert code == 0, stderr
assert (root / "raw" / "jenkins" / "tl153" / "page.html").is_file()
other = tmp_path / "other.html"
other.write_text("<html><body><p>Other</p></body></html>", encoding="utf-8")
before = _hashes(root)
code, stdout, stderr = _run(root, str(other), "-r", "../wiki/x")
assert code == 1
assert stdout == ""
assert "--raw must stay inside raw/" in stderr
assert _hashes(root) == before
def test_reuses_identical_bytes(tmp_path: pathlib.Path) -> None:
"""SPEC 7.25.3.3: a second name and ``-r`` reuse the first raw copy."""
root = _wiki(tmp_path)
first = tmp_path / "one.txt"
first.write_text("same bytes\n", encoding="utf-8", newline="\n")
code, _stdout, stderr = _run(root, str(first))
assert code == 0, stderr
second = tmp_path / "two.txt"
second.write_text("same bytes\n", encoding="utf-8", newline="\n")
pipeline = default_pipeline(root)
result = pipeline.run(str(second), raw="other/dir")
assert result.raw_reused is True
assert result.raw_path == "raw/articles/one.txt"
assert result.duplicate_page == "wiki/sources/one.md"
copies = [path for path in (root / "raw").rglob("*") if path.is_file() and path.suffix == ".txt"]
assert copies == [root / "raw" / "articles" / "one.txt"]
assert "already archived at raw/articles/one.txt; --raw ignored" in result.warnings
def test_changed_file_gets_a_second_raw_name(tmp_path: pathlib.Path) -> None:
"""SPEC 7.25.3.4: a changed file of the same name is stored as ``slug-2``."""
root = _wiki(tmp_path)
source = tmp_path / "notes.txt"
source.write_text("first\n", encoding="utf-8", newline="\n")
code, _stdout, stderr = _run(root, str(source), "--raw-only")
assert code == 0, stderr
original = (root / "raw" / "articles" / "notes.txt").read_bytes()
source.write_text("second\n", encoding="utf-8", newline="\n")
code, _stdout, stderr = _run(root, str(source))
assert code == 0, stderr
assert (root / "raw" / "articles" / "notes.txt").read_bytes() == original
assert (root / "raw" / "articles" / "notes-2.txt").read_text(encoding="utf-8") == "second\n"
def test_duplicate_exits_zero_and_skips_the_log(tmp_path: pathlib.Path) -> None:
"""SPEC 7.25.3.5: the same bytes report the first page and do not rewrite it."""
root = _wiki(tmp_path)
source = root / "ingest" / "a.txt"
source.write_text("alpha\n", encoding="utf-8", newline="\n")
code, _stdout, stderr = _run(root, str(source))
assert code == 0, stderr
page = root / "wiki" / "sources" / "a.md"
page_bytes = page.read_bytes()
log_before = (root / "wiki" / "log.md").read_text(encoding="utf-8")
again = root / "ingest" / "b.txt"
again.write_text("alpha\n", encoding="utf-8", newline="\n")
code, stdout, stderr = _run(root, str(again), "--json")
assert code == 0, stderr
payload = json.loads(stdout)
assert payload["duplicate_page"] == "wiki/sources/a.md"
assert payload["page_path"] is None
assert not again.exists()
assert page.read_bytes() == page_bytes
assert (root / "wiki" / "log.md").read_text(encoding="utf-8") == log_before
def test_ingest_drop_moves_and_personal_is_copied(tmp_path: pathlib.Path) -> None:
"""SPEC 7.25.3.6: ``ingest/`` is moved. A file outside it is copied."""
root = _wiki(tmp_path)
dropped = root / "ingest" / "a.pdf"
dropped.write_bytes(_pdf(["Moved"]))
code, _stdout, stderr = _run(root, str(dropped))
assert code == 0, stderr
assert not dropped.exists()
archived = root / "raw" / "papers" / "a.pdf"
assert archived.read_bytes() == _pdf(["Moved"])
personal = root / "personal" / "note.txt"
personal.write_text("keep me\n", encoding="utf-8", newline="\n")
original = personal.read_bytes()
code, _stdout, stderr = _run(root, str(personal))
assert code == 0, stderr
assert personal.read_bytes() == original
assert (root / "raw" / "articles" / "note.txt").read_bytes() == original
def test_unknown_bytes_archive_without_a_page(tmp_path: pathlib.Path) -> None:
"""SPEC 7.25.4.1: random bytes exit 3 under ``raw/other``."""
root = _wiki(tmp_path)
source = tmp_path / "blob.xyz"
source.write_bytes(b"\x00\x01\x02\xff")
code, stdout, _stderr = _run(root, str(source), "--json")
assert code == 3
payload = json.loads(stdout)
assert payload["conversion"] == "none"
assert payload["page_path"] is None
assert payload["raw_path"] == "raw/other/blob.xyz"
assert (root / "raw" / "other" / "blob.xyz").is_file()
assert not (root / "wiki" / "sources" / "blob.md").exists()
human_source = tmp_path / "other.xyz"
human_source.write_bytes(b"\x00\x02")
code, human, _stderr = _run(root, str(human_source))
assert code == 3
assert human.strip().splitlines()[-1].startswith("No converter for")
def test_encrypted_pdf_keeps_raw_and_skips_the_page(tmp_path: pathlib.Path) -> None:
"""SPEC 7.25.4.2: an encrypted PDF exits 3 and the raw copy stays."""
root = _wiki(tmp_path)
source = tmp_path / "secret.pdf"
source.write_bytes(_pdf(["hidden"], password="secret"))
code, stdout, stderr = _run(root, str(source))
assert code == 3
assert "Conversion failed: pdf:" in stderr
assert (root / "raw" / "papers" / "secret.pdf").is_file()
assert not (root / "wiki" / "sources" / "secret.md").exists()
assert stdout.strip().splitlines()[-1].startswith("No converter for")
def test_html_keeps_article_link_and_table(tmp_path: pathlib.Path) -> None:
"""SPEC 7.25.4.5: article text, no script, an absolute link, and a pipe table."""
root = _wiki(tmp_path)
html = """<!doctype html>
<html lang="en"><head><title>Page Title</title></head>
<body>
<nav>NAVSECRET</nav>
<article>
<p>ARTICLETEX</p>
<a href="/a">Go</a>
<table><tr><th>Col</th></tr><tr><td>Cell</td></tr></table>
</article>
<script>SCRIPTSECRET</script>
</body></html>
"""
opener = _FakeOpener(html.encode("utf-8"), "text/html")
pipeline = default_pipeline(root)
pipeline.fetchers[0] = UrlFetcher(pipeline.registry, opener=opener)
result = pipeline.run("https://host/p")
assert result.exit_code == 0
body = (root / result.page_path).read_text(encoding="utf-8")
assert "ARTICLETEX" in body
assert "SCRIPTSECRET" not in body
assert "https://host/a" in body
assert "|" in body
def test_pdf_page_headings_and_blank_page(tmp_path: pathlib.Path) -> None:
"""SPEC 7.25.4.6: pages are numbered, and a blank page is partial."""
root = _wiki(tmp_path)
numbered = tmp_path / "two.pdf"
numbered.write_bytes(_pdf(["Alpha", "Beta"]))
code, _stdout, stderr = _run(root, str(numbered))
assert code == 0, stderr
body = (root / "wiki" / "sources" / "two.md").read_text(encoding="utf-8")
assert "## Page 1" in body
assert "## Page 2" in body
blank = tmp_path / "gap.pdf"
blank.write_bytes(_pdf(["Alpha", ""]))
code, _stdout, stderr = _run(root, str(blank))
assert code == 0, stderr
front = _front(root / "wiki" / "sources" / "gap.md")
assert front["ingest"]["conversion"] == "partial"
assert "page 2 has no text layer" in front["ingest"]["warnings"]
def test_target_paths(tmp_path: pathlib.Path) -> None:
"""SPEC 7.25.5.1: ``-t concepts`` writes under concepts. Escapes exit 1."""
root = _wiki(tmp_path)
source = tmp_path / "idea.txt"
source.write_text("idea\n", encoding="utf-8", newline="\n")
code, _stdout, stderr = _run(root, str(source), "-t", "concepts")
assert code == 0, stderr
assert (root / "wiki" / "concepts" / "idea.md").is_file()
code, stdout, stderr = _run(root, str(source), "-t", "../raw/x.md")
assert code == 1
assert stdout == ""
assert "--target must stay inside wiki/" in stderr
other = tmp_path / "other.txt"
other.write_text("x\n", encoding="utf-8", newline="\n")
code, stdout, stderr = _run(root, str(other), "-t", "index.md")
assert code == 1
assert "--target cannot replace" in stderr
def test_existing_page_refuses_until_force(tmp_path: pathlib.Path) -> None:
"""SPEC 7.25.5.2: a second run exits 1. ``--force`` replaces the page."""
root = _wiki(tmp_path)
source = tmp_path / "item.txt"
source.write_text("one\n", encoding="utf-8", newline="\n")
code, _stdout, stderr = _run(root, str(source))
assert code == 0, stderr
page = root / "wiki" / "sources" / "item.md"
before = page.read_bytes()
opener = _FakeOpener(b"nope", "text/plain")
pipeline = default_pipeline(root)
pipeline.fetchers[0] = UrlFetcher(pipeline.registry, opener=opener)
with pytest.raises(IngestError, match="Page exists:"):
pipeline.run("https://host/item")
assert opener.calls == 0
assert page.read_bytes() == before
source.write_text("two\n", encoding="utf-8", newline="\n")
code, _stdout, stderr = _run(root, str(source), "--force")
assert code == 0, stderr
assert page.read_bytes() != before
assert "two" in (root / "raw" / "articles" / "item-2.txt").read_text(encoding="utf-8")
def test_page_frontmatter_matches_raw_hash(tmp_path: pathlib.Path) -> None:
"""SPEC 7.25.5.3 and 7.25.5.4: frontmatter is complete and the page has one H1."""
root = _wiki(tmp_path)
index_before = (root / "wiki" / "index.md").read_bytes()
source = tmp_path / "story.md"
source.write_text("# Story\n\nSee [gone](missing.md).\n", encoding="utf-8", newline="\n")
pipeline = default_pipeline(root)
result = pipeline.run(str(source))
assert result.exit_code == 0
page = root / result.page_path
front = _front(page)
for key in _FRONTMATTER_KEYS:
assert key in front
for key in _INGEST_KEYS:
assert key in front["ingest"]
raw = root / front["sources"][0]
assert front["ingest"]["sha256"] == hashlib.sha256(raw.read_bytes()).hexdigest()
assert front["ingest"]["warnings"] == result.warnings
assert front["status"] == "raw"
assert front["confidence"] == "zero"
assert front["llm"] == [{"model": "none"}]
headings = [line for line in page.read_text(encoding="utf-8").splitlines() if line.startswith("# ")]
assert headings == ["# Story"]
linter = FrontmatterLinter(settings_file=root / "fiona-settings.json")
assert linter.lint_file(page) == []
links = MarkdownLinter(settings_file=root / "fiona-settings.json")
assert [item for item in links.lint_file(page) if item.code == "MDL001"] == []
assert (root / "wiki" / "index.md").read_bytes() == index_before
log = (root / "wiki" / "log.md").read_text(encoding="utf-8")
assert "version: 2.0.1" in log
assert f"ingest {source.resolve()} -> raw/articles/story.md -> wiki/sources/story.md (raw)" in log
def test_style_violations_become_one_warning(tmp_path: pathlib.Path) -> None:
"""SPEC 7.25.5.5: violations the fixer leaves are counted in one warning."""
root = _wiki(tmp_path)
source = tmp_path / "long.md"
source.write_text("# Title\n\n" + ("word " * 40) + "\n", encoding="utf-8", newline="\n")
code, _stdout, stderr = _run(root, str(source))
assert code == 0, stderr
warnings = _front(root / "wiki" / "sources" / "long.md")["ingest"]["warnings"]
assert any(warning.endswith("markdown style violations left") for warning in warnings)
def test_raw_only_and_dry_run(tmp_path: pathlib.Path) -> None:
"""SPEC 7.25.7 and 7.25.8: raw-only skips the page. A dry run writes nothing."""
root = _wiki(tmp_path)
source = root / "ingest" / "only.txt"
source.write_text("archive me\n", encoding="utf-8", newline="\n")
code, stdout, stderr = _run(root, str(source), "--raw-only", "--json")
assert code == 0, stderr
payload = json.loads(stdout)
assert payload["conversion"] == "none"
assert payload["page_path"] is None
assert not source.exists()
assert (root / "raw" / "articles" / "only.txt").is_file()
kept = root / "ingest" / "stay.txt"
kept.write_text("stay\n", encoding="utf-8", newline="\n")
before = _hashes(root)
code, stdout, stderr = _run(root, str(kept), "--dry-run")
assert code == 0, stderr
assert stdout.startswith("Dry run - no files will be modified:")
assert kept.is_file()
assert _hashes(root) == before
code, stdout, _stderr = _run(root, str(kept), "--dry-run", "--json")
assert json.loads(stdout)["dry_run"] is True
def test_pending_hides_refined_pages(tmp_path: pathlib.Path) -> None:
"""SPEC 7.25.9: ``--pending`` lists raw pages and drops refined ones."""
root = _wiki(tmp_path)
source = tmp_path / "open.txt"
source.write_text("open\n", encoding="utf-8", newline="\n")
assert _run(root, str(source))[0] == 0
code, stdout, _stderr = _run(root, "--pending", "--json")
payload = json.loads(stdout)
assert code == 0
assert payload["total"] == 1
page = root / "wiki" / "sources" / "open.md"
text = page.read_text(encoding="utf-8").replace("status: raw", "status: refined", 1)
page.write_text(text, encoding="utf-8", newline="\n")
code, stdout, _stderr = _run(root, "--pending", "--json")
assert json.loads(stdout)["pending"] == []
def test_not_a_wiki_and_mutually_exclusive_args(tmp_path: pathlib.Path) -> None:
"""A directory without settings exits 1. Source and ``--pending`` cannot combine."""
code, stdout, stderr = _run(tmp_path, str(tmp_path / "a.txt"))
assert code == 1
assert stdout == ""
assert "is not an llm-wiki directory" in stderr
with pytest.raises(SystemExit) as caught:
ingest_main(["--pending", "file.txt"])
assert caught.value.code == 2
def test_sanity_ignores_raw_markdown(tmp_path: pathlib.Path) -> None:
"""SPEC 7.25.12: a raw markdown file with style violations does not fail sanity."""
root = _wiki(tmp_path)
raw = root / "raw" / "articles"
raw.mkdir(parents=True)
(raw / "bad-style.md").write_text(
"# Bad\n\n" + ("x" * 180) + "\n\n[gone](missing-file.md)\n",
encoding="utf-8",
newline="\n",
)
results = run_wiki_checks(root)
assert "bad-style.md" not in str(results)
+552
View File
@@ -0,0 +1,552 @@
"""Branch coverage for wiki ingest stages.
Generated by AI/LLM: 2026-09-30 - error paths and converter edges from SPEC 7.25.
"""
from __future__ import annotations
import contextlib
import datetime
import email.message
import importlib.metadata
import io
import json
import pathlib
import shutil
import types
import typing
import urllib.error
import urllib.request
import pypdf
import pytest
from pymarkdown.api import PyMarkdownApiException
from fiona.mdoctor.mdoctor import MDoctor
from fiona.paths import llm_wiki_template
from fiona.wiki.ingest import log_writer as log_mod
from fiona.wiki.ingest import media
from fiona.wiki.ingest import page_writer as pages
from fiona.wiki.ingest import settings as settings_mod
from fiona.wiki.ingest.archiver import RawArchiver, _strip_prefix
from fiona.wiki.ingest.converters import default_registry
from fiona.wiki.ingest.converters.base import ConversionError, ConverterRegistry
from fiona.wiki.ingest.converters.html import HtmlConverter
from fiona.wiki.ingest.converters.markdown import MarkdownConverter
from fiona.wiki.ingest.converters.pdf import (
PdfConverter,
_iso_date,
_metadata,
_outline,
_render,
_walk_outline,
)
from fiona.wiki.ingest.fetchers import UrlFetcher, _without_userinfo, url_scheme
from fiona.wiki.ingest.log_writer import IngestLog
from fiona.wiki.ingest.models import ArchiveResult, ConversionResult, FetchedSource, IngestError
from fiona.wiki.ingest.pipeline import (
IngestPipeline,
_citing_page,
_print_pending,
default_pipeline,
)
from fiona.wiki.ingest.settings import default_settings, load_ingest_settings
def _wiki(tmp_path: pathlib.Path) -> pathlib.Path:
"""Create a small llm-wiki."""
root = tmp_path / "wiki"
(root / "wiki").mkdir(parents=True)
(root / "ingest").mkdir()
(root / "raw").mkdir()
shutil.copy2(llm_wiki_template() / "fiona-settings.json", root / "fiona-settings.json")
(root / "wiki" / "log.md").write_text(
"---\nname: log\nversion: 2.0.0\nupdated: 2026-09-30\n---\n\n# Log\n",
encoding="utf-8",
newline="\n",
)
(root / "wiki" / "index.md").write_text(
"---\nname: index\nversion: 1.0.0\n---\n\n# Index\n",
encoding="utf-8",
newline="\n",
)
return root
def _source(path: pathlib.Path, *, origin_kind: str = "file", origin: str | None = None) -> FetchedSource:
"""Build a fetched-source record for a file that already exists."""
return FetchedSource(
argument=str(path),
path=path,
origin=origin or str(path.resolve()),
origin_kind=origin_kind,
final_url=None,
media_type="text/plain",
charset=None,
size_bytes=path.stat().st_size,
slug=path.stem,
fetched="2026-09-30T00:00:00Z",
created=False,
)
class _FakeResponse:
"""One canned download body."""
def __init__(self, body: bytes, url: str) -> None:
self._body = body
self._offset = 0
self.headers = email.message.Message()
self.headers["Content-Type"] = "text/plain"
self._url = url
def read(self, amount: int = -1) -> bytes:
"""Return the next slice of the body."""
if amount < 0:
amount = len(self._body) - self._offset
chunk = self._body[self._offset:self._offset + amount]
self._offset += len(chunk)
return chunk
def geturl(self) -> str:
"""Return the final URL."""
return self._url
def __enter__(self) -> typing.Self:
return self
def __exit__(self, *_args: object) -> bool:
return False
class _Opener:
"""Opener that returns one body or raises URLError."""
def __init__(self, body: bytes | None = b"hello\n", *, error: str | None = None) -> None:
self.body = body or b""
self.error = error
def open(self, request: urllib.request.Request, timeout: float | None = None) -> _FakeResponse:
"""Return the canned body or fail the download."""
del timeout
if self.error:
raise urllib.error.URLError(self.error)
return _FakeResponse(self.body, request.full_url)
def test_slug_and_media_edges() -> None:
"""Slugs, sniffing, extensions, and decode fallbacks."""
assert url_scheme("note.txt") is None
assert media.slug_from_url("https://host/index") == "host"
assert media.slug_from_url("https://host/") == "host"
assert media.split_content_type("text/plain; boundary; charset=\"utf-8\"") == ("text/plain", "utf-8")
assert media.split_content_type("application/octet-stream") == (None, None)
registry = default_registry()
assert media.sniff_media_type(b" <!DOCTYPE html><html>") == "text/html"
assert media.sniff_media_type(b"\x00\xff") == "application/octet-stream"
assert media.sniff_media_type(b"ab\xe2\x82") == "text/plain"
assert media.sniff_media_type(b"plain text") == "text/plain"
assert media.extension_for("application/zip", "pdf", registry) == ".pdf"
assert media.extension_for("application/zip", "", registry) == ".bin"
assert media.decode_bytes(b"\xef\xbb\xbfhi", None) == ("hi", [])
text, warnings = media.decode_bytes(b"\xff", "latin-1")
assert text == "ÿ"
assert warnings == ["decoded as latin-1; not valid UTF-8"]
_text, fallback = media.decode_bytes(b"\xff", "not-a-charset")
assert "cp1252" in fallback[0]
assert registry.find("nope", "md").name == "markdown"
def test_settings_fallbacks(tmp_path: pathlib.Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""Missing, broken, and partial ingest settings keep usable values."""
def missing(_name: str) -> str:
raise importlib.metadata.PackageNotFoundError
monkeypatch.setattr(settings_mod.importlib.metadata, "version", missing)
assert settings_mod.package_version() == "0.99.0"
assert load_ingest_settings(tmp_path).timeout_seconds == 30
broken = tmp_path / "bad"
broken.mkdir()
(broken / "fiona-settings.json").write_text("{", encoding="utf-8")
assert load_ingest_settings(broken).max_download_bytes == 52428800
good = tmp_path / "good"
good.mkdir()
(good / "fiona-settings.json").write_text(json.dumps({
"fiona": {"wiki": {"ingest": {
"user_agent": "agent",
"default_target": "concepts",
"raw_categories": {"pdf": "papers", "html": 1},
}}},
}), encoding="utf-8")
loaded = load_ingest_settings(good)
assert loaded.user_agent == "agent"
assert loaded.default_target == "concepts"
assert loaded.raw_categories == {"pdf": "papers"}
def test_markdown_frontmatter_edges(tmp_path: pathlib.Path) -> None:
"""Invalid, empty, and non-mapping frontmatter, and a fenced heading."""
converter = MarkdownConverter()
cases = {
"bad.md": "---\n[\n---\n# Kept\n",
"empty.md": "---\n\n---\n# Kept\n",
"list.md": "---\n- a\n---\n# Kept\n",
"fenced.md": "```\n# Hidden\n```\n",
}
for name, text in cases.items():
path = tmp_path / name
path.write_text(text, encoding="utf-8")
result = converter.convert(_source(path))
assert result.conversion == "complete"
fenced = converter.convert(_source(tmp_path / "fenced.md"))
assert fenced.title == "fenced"
assert fenced.body.startswith("```")
assert converter.convert(_source(tmp_path / "bad.md")).warnings
def test_html_edges(tmp_path: pathlib.Path) -> None:
"""Comments, empty pages, og:title, images, and meta tags."""
converter = HtmlConverter()
html = """<!DOCTYPE html>
<html lang="en"><head>
<base href="https://host/base/">
<meta property="og:title" content="">
<meta name="author" content="Ada">
<meta property="og:site_name" content="Site">
<meta property="article:published_time" content="2020-01-01">
<meta name="unused" content="no">
<meta name="description">
<link rel="canonical" href="https://host/c">
<!-- secret -->
</head><body>
<h1> </h1>
<article><h1>Real</h1><img src="pic.png" alt="pic"></article>
</body></html>
"""
path = tmp_path / "page.html"
path.write_text(html, encoding="utf-8")
source = _source(path, origin_kind="url", origin="https://host/p")
result = converter.convert(source)
assert "secret" not in result.body
assert "https://host/base/pic.png" in result.body
assert result.source_meta["author"] == "Ada"
assert result.source_meta["canonical"] == "https://host/c"
assert result.title == "Real"
empty = tmp_path / "empty.html"
empty.write_text("<html><body><script>only</script></body></html>", encoding="utf-8")
partial = converter.convert(_source(empty, origin_kind="url", origin="https://host/e"))
assert partial.conversion == "partial"
og = tmp_path / "og.html"
og.write_text(
'<html><head><meta property="og:title" content="Og"></head><body><p>x</p></body></html>',
encoding="utf-8",
)
assert converter.convert(_source(og)).title == "Og"
bare = tmp_path / "bare.html"
bare.write_text("<html><body><p>x</p></body></html>", encoding="utf-8")
assert converter.convert(_source(bare)).title is None
blank_h1 = tmp_path / "blankh1.html"
blank_h1.write_text("<html><body><h1> </h1><p>x</p></body></html>", encoding="utf-8")
assert converter.convert(_source(blank_h1)).title is None
def test_pdf_metadata_outline_and_failures(tmp_path: pathlib.Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""PDF dates, outline headings, a scanned file, and read failures."""
aware = datetime.datetime(2020, 1, 2, tzinfo=datetime.UTC)
naive = datetime.datetime(2020, 1, 1) # noqa: DTZ001
assert _iso_date(None) is None
assert _iso_date(naive).startswith("2020-01-01T")
assert _iso_date(aware).startswith("2020-01-02T")
assert _iso_date("D:20200103") == "2020-01-03T00:00:00Z"
assert _iso_date("nope") == "nope"
assert _iso_date("") is None
info_obj = types.SimpleNamespace(
title="T",
author="",
subject="S",
keywords=None,
creation_date=naive,
modification_date=aware,
)
empty = types.SimpleNamespace(pages=[1], metadata=None)
info_reader = types.SimpleNamespace(pages=[1, 2], metadata=info_obj)
assert _metadata(empty) == {"page_count": 1} # type: ignore[arg-type]
info = _metadata(info_reader) # type: ignore[arg-type]
assert info["title"] == "T"
assert "author" not in info
class _OutlineReader:
"""Fake reader whose first outline item has no destination."""
def get_destination_page_number(self, item: object) -> int:
if getattr(item, "title", None) == "":
raise ValueError("bad")
return 0
found: list[tuple[int, int, str]] = []
bad = types.SimpleNamespace(title="")
good = types.SimpleNamespace(title=None)
_walk_outline(_OutlineReader(), [bad, [good]], 5, found) # type: ignore[arg-type]
assert found == [(0, 6, "Untitled")]
assert _render(["body"], [(0, 2, "Chapter")]) == "## Chapter\n\nbody\n"
assert "## Page" not in _render([""], [(1, 2, "Later")])
class _BrokenOutline:
"""Fake reader whose outline cannot be read."""
@property
def outline(self) -> list[object]:
raise pypdf.errors.PdfReadError("no outline")
assert _outline(_BrokenOutline()) == [] # type: ignore[arg-type]
writer = pypdf.PdfWriter()
writer.add_blank_page(width=200, height=200)
parent = writer.add_outline_item("Parent", 0)
writer.add_outline_item("Child", 0, parent=parent)
writer.add_metadata({"/Title": "Outlined", "/CreationDate": "D:20200101120000"})
buffer = io.BytesIO()
writer.write(buffer)
outlined = tmp_path / "outlined.pdf"
outlined.write_bytes(buffer.getvalue())
converted = PdfConverter().convert(_source(outlined))
assert "Parent" in converted.body
assert converted.title == "Outlined"
blank = tmp_path / "blank.pdf"
blank_writer = pypdf.PdfWriter()
blank_writer.add_blank_page(width=200, height=200)
blank_buffer = io.BytesIO()
blank_writer.write(blank_buffer)
blank.write_bytes(blank_buffer.getvalue())
scanned = PdfConverter().convert(_source(blank))
assert scanned.conversion == "partial"
assert any("scanned" in warning for warning in scanned.warnings)
corrupt = tmp_path / "corrupt.pdf"
corrupt.write_bytes(b"%PDF-1.7 not a real file")
with pytest.raises(ConversionError):
PdfConverter().convert(_source(corrupt))
def extract_text() -> str:
raise pypdf.errors.PdfReadError("page")
reader = types.SimpleNamespace(
is_encrypted=False,
metadata=None,
pages=[types.SimpleNamespace(extract_text=extract_text)],
outline=[],
)
monkeypatch.setattr(pypdf, "PdfReader", lambda _path: reader)
with pytest.raises(ConversionError, match="page"):
PdfConverter().convert(_source(outlined))
def test_archive_edges(tmp_path: pathlib.Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""Reuse skips, raw escapes, named raw files, and collision suffixes."""
root = _wiki(tmp_path)
archiver = RawArchiver(tmp_path / "empty")
assert archiver.find_by_sha256("ab" * 32, 1) is None
assert _strip_prefix("raw", "raw") == ""
assert _strip_prefix("raw/notes/a.txt", "raw") == "notes/a.txt"
registry = default_registry()
settings = default_settings()
already = root / "raw" / "articles" / "already.md"
already.parent.mkdir(parents=True)
already.write_text("# Already\n", encoding="utf-8")
placed = archiver_on(root).archive(
_source(already),
registry=registry,
settings=settings,
raw_override=None,
dry_run=False,
)
assert placed.raw_reused is False
with pytest.raises(IngestError, match="already under raw"):
archiver_on(root).archive(
_source(already),
registry=registry,
settings=settings,
raw_override="other",
dry_run=False,
)
original_stat = pathlib.Path.stat
seen: dict[str, int] = {}
def flaky(self: pathlib.Path, *args: object, **kwargs: object) -> object:
if self.name == "bad.bin":
seen[str(self)] = seen.get(str(self), 0) + 1
if seen[str(self)] > 1:
raise OSError("stat failed")
return original_stat(self, *args, **kwargs)
(root / "raw" / "bad.bin").write_bytes(b"x")
monkeypatch.setattr(pathlib.Path, "stat", flaky)
assert archiver_on(root).find_by_sha256("ab" * 32, 1) is None
monkeypatch.undo()
pipeline = default_pipeline(root)
named = tmp_path / "fresh.txt"
named.write_text("fresh\n", encoding="utf-8")
pipeline.run(str(named), raw="notes/kept.html", raw_only=True)
assert (root / "raw" / "notes" / "kept.html").is_file()
for index, text in enumerate(("one\n", "two\n", "three\n"), start=1):
path = tmp_path / f"dir{index}" / "note.txt"
path.parent.mkdir()
path.write_text(text, encoding="utf-8")
pipeline.run(str(path), raw_only=True)
assert (root / "raw" / "articles" / "note-3.txt").is_file()
def archiver_on(root: pathlib.Path) -> RawArchiver:
"""Return an archiver bound to ``root``."""
return RawArchiver(root)
def test_download_and_log_edges(tmp_path: pathlib.Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""URL errors, a failed save, userinfo, and a log that cannot be bumped."""
root = _wiki(tmp_path)
registry = default_registry()
fetcher = UrlFetcher(registry, opener=_Opener(error="timed out"))
with pytest.raises(IngestError, match="timed out"):
fetcher.fetch("https://host/a.txt", root / "ingest")
assert _without_userinfo("https://user:secret@host:443/a.txt") == "https://host:443/a.txt"
assert _without_userinfo("https://host/a.txt") == "https://host/a.txt"
original_write = pathlib.Path.write_bytes
def fail_write(self: pathlib.Path, data: bytes) -> int:
original_write(self, data)
raise OSError("disk")
monkeypatch.setattr(pathlib.Path, "write_bytes", fail_write)
fetcher = UrlFetcher(registry, opener=_Opener(b"hello\n"))
with pytest.raises(OSError, match="disk"):
fetcher.fetch("https://host/b.txt", root / "ingest")
monkeypatch.undo()
assert not list((root / "ingest").glob("*.partial"))
log = IngestLog(root)
log.append(day="2026-09-30", origin="o", raw_rel="raw/a", page_rel=None, dry_run=True)
(root / "wiki" / "log.md").unlink()
log.append(day="2026-09-30", origin="o", raw_rel="raw/a", page_rel="wiki/a.md", dry_run=False)
odd = root / "wiki" / "log.md"
odd.write_text("version: beta\n# Log", encoding="utf-8", newline="\n")
log.append(day="2026-09-30", origin="o", raw_rel="raw/a", page_rel=None, dry_run=False)
text = odd.read_text(encoding="utf-8")
assert "version: beta" in text
assert text.endswith("\n")
assert log_mod._set_updated("no updated here\n", "2026-09-30") == "no updated here\n"
assert log_mod._bump_version("version: 1\n") == "version: 1\n"
assert log_mod._bump_version("no version here\n") == "no version here\n"
def test_page_writer_edges(tmp_path: pathlib.Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""Heading fences, links, fixer failures, and target prefixes."""
body = "# Title\n\n```\n# inside\n```\n\n###### Six\n"
shifted = pages._shift_headings(body)
assert "```\n# inside\n```" in shifted
assert "###### Six" in shifted
assert pages._keeps_target("?q", tmp_path) is True
assert pages._keeps_target("missing.md", tmp_path) is False
assert pages._body_after_frontmatter("no front\n") == "no front\n"
assert pages._body_after_frontmatter("---\nonly\n") == "---\nonly\n"
assert pages._split_ending("a\r\n") == ("\r\n", "a")
assert pages._split_ending("a") == ("", "a")
assert pages._strip_wiki_prefix("wiki") == ""
assert pages._strip_wiki_prefix("wiki/concepts") == "concepts"
root = _wiki(tmp_path)
writer = pages.RawPageWriter(root)
source_path = tmp_path / "note.txt"
source_path.write_text("note\n", encoding="utf-8")
source = _source(source_path)
archive_path = root / "raw" / "articles" / "note.txt"
archive_path.parent.mkdir(parents=True, exist_ok=True)
archive_path.write_text("note\n", encoding="utf-8")
archive = ArchiveResult(archive_path, "raw/articles/note.txt", "ab" * 32, False, [])
conversion = ConversionResult("plain\n", "Note", {}, "complete", [], "text")
def broken_fix(self: MDoctor, path: str) -> None:
raise PyMarkdownApiException("fix")
def broken_scan(self: MDoctor, path: str) -> list:
raise PyMarkdownApiException("scan")
monkeypatch.setattr(MDoctor, "fix_file_programmatically", broken_fix)
monkeypatch.setattr(MDoctor, "scan_file_programmatically", broken_scan)
monkeypatch.setattr(pages, "_body_after_frontmatter", lambda _text: "no heading\n")
page = root / "wiki" / "sources" / "note.md"
writer.write(
source=source,
archive=archive,
conversion=conversion,
page=page,
dry_run=False,
)
assert page.is_file()
assert page.read_text(encoding="utf-8").count("# Note") == 1
def test_pipeline_edges(tmp_path: pathlib.Path) -> None:
"""Dry-run downloads, pending pages, and a forced duplicate."""
root = _wiki(tmp_path)
pipeline = default_pipeline(root)
pipeline.fetchers[0] = UrlFetcher(pipeline.registry, opener=_Opener(b"hello\n"))
result = pipeline.run("https://user:secret@host/hello.txt", dry_run=True)
assert result.dry_run is True
assert list((root / "ingest").iterdir()) == []
with pytest.raises(IngestError, match="stay inside raw"):
pipeline.run("https://host/escape.txt", raw="../outside")
assert list((root / "ingest").glob("*.txt")) == []
bare = IngestPipeline(
root,
[],
ConverterRegistry(),
pipeline.archiver,
pipeline.page_writer,
pipeline.log_writer,
default_settings(),
)
local = tmp_path / "local.txt"
local.write_text("local\n", encoding="utf-8")
with pytest.raises(IngestError, match="Source not found"):
bare.run(str(local))
(root / "wiki" / "skip.md").mkdir()
(root / "wiki" / "bad.md").write_text("---\n[\n---\n", encoding="utf-8")
(root / "wiki" / "sources").mkdir(exist_ok=True)
(root / "wiki" / "sources" / "stringy.md").write_text(
"---\nname: stringy\nstatus: raw\nsources: raw/a.txt\ningest: plain\n---\n\n# S\n",
encoding="utf-8",
newline="\n",
)
pending = pipeline.list_pending()
assert any(item["page_path"].endswith("stringy.md") for item in pending)
assert _citing_page(root, "raw/a.txt", "nope") == "wiki/sources/stringy.md"
captured = io_capture(_print_pending, pipeline, False)
assert "stringy.md" in captured
shutil.rmtree(root / "wiki")
assert pipeline.list_pending() == []
assert _citing_page(root, "raw/a.txt", "nope") is None
root = _wiki(tmp_path / "second")
pipeline = default_pipeline(root)
first = tmp_path / "first.txt"
second = tmp_path / "other" / "second.txt"
first.write_text("same\n", encoding="utf-8")
second.parent.mkdir()
second.write_text("same\n", encoding="utf-8")
pipeline.run(str(first))
forced = pipeline.run(str(second), force=True)
assert forced.page_path == "wiki/sources/first.md"
def io_capture(func: typing.Callable, *args: object) -> str:
"""Return stdout from a function that prints."""
buffer = io.StringIO()
with contextlib.redirect_stdout(buffer):
func(*args)
return buffer.getvalue()
+23
View File
@@ -680,6 +680,29 @@ WARNING run_tests: pytest short summary missing from output; failure details com
- [ ] `0`, or a missing key, draws every node
- [ ] A positive integer hides a node whose `unique_points` is
below that integer until one of its parents is clicked
- [ ] Omit gitignored markdown from the wiki graph (SPEC 7.4.17)
- [ ] Failing tests in `tests/test_graph_generator.py` for the
acceptances in SPEC 7.4.17, 7.4.17.1, and 7.4.17.2:
`secret.md` absent from JSON nodes, edges, and orphans;
a link to it creates no edge; `max_files` 1 still keeps
`kept.md`; `drafts/` omits `drafts/note.md` and keeps
`wiki/note.md`; a `# secret.md` comment does not omit;
`notes/.gitignore` omits `notes/scratch.md` only; with no
`.gitignore`, `.git/*.md` is absent and a root page stays
- [ ] In `wiki_graph_main` (`src/fiona/wiki/graph_generator.py`),
drop a match before the `max_files` cut. Read patterns with
`gitignore_patterns_from_text` and test them with
`path_is_ignored` (`src/fiona/wiki/ignore.py`), the same
rules as SPEC 7.2.10. The scan-root `.gitignore` applies to
the whole tree. A subdirectory `.gitignore` applies only
under that subdirectory. Also drop any path under `.git/`
- [ ] Do not parse an omitted file. Drop an edge whose target
resolves to an omitted file, so it never becomes a node
- [ ] HTML uses that same graph. No new settings key: the filter
always runs
- [ ] Housekeeping's orphan list follows automatically, because
`graph_step` builds the graph through `wiki_graph_main`
(SPEC 7.26.6)
- [ ] in spec, refine the sectioning by compiling the modules,
and their functions as subsections
- [ ] eg 4.1. librarian then 4.1.1 librarian toc, to make it cleaner