diff --git a/SPEC.md b/SPEC.md index a6f421e..034e885 100644 --- a/SPEC.md +++ b/SPEC.md @@ -1,6 +1,6 @@ --- name: Fiona-Assistant Specification -version: 0.97.0 +version: 0.99.0 title: Project Spec Sheet created: 2026-09-23 updated: 2026-09-30 @@ -99,7 +99,10 @@ yield to the owner's markdown and write `-update.md` instead Revision 0.96.0 compares a file without a version by SHA-256 and shows both short hashes in the update report (*7.2.12.*). Revision 0.97.0 rejects a module-level assignment that follows the -first function or class (*6.4.3.*). +first function or class (*6.4.3.*). Revision 0.98.0 specifies that +`wiki graph` omits a file matched by `.gitignore` (*7.4.17.*) and +leaves that requirement unbuilt. Revision 0.99.0 builds `wiki ingest` +(*7.25.*). Each feature section starts with a `Status` line. Requirements use `- *N.M.K.*`. Each `*acceptance*` sits under the requirement it tests. @@ -277,7 +280,6 @@ utils/skills/README.md utils/skills/fiona/SKILL.md utils/skills/fiona/markdown_rules.md utils/skills/fiona/llm-wiki-mod.md -utils/skills/obsidian/SKILL.md utils/skills/python-tdd/SKILL.md utils/personas/default/SOUL.md utils/personas/template/SOUL.md @@ -732,7 +734,7 @@ extra untimed `coverage:` row under pytests. Details are section 7.16. ### 4.14. wiki ingest -Not built yet. The contract is section 7.25. +The contract is section 7.25. ```text fiona-assistant wiki ingest [--wiki DIR] [-r|--raw PATH] @@ -1088,7 +1090,7 @@ directory relative to `wiki/`. `raw_categories` maps a converter name to a directory under `raw/`, such as `{"text": "transcripts"}`. A value that is absent, of the wrong type, or out of range uses the default in the table. The packaged `fiona-settings.json` omits -`fiona.wiki.ingest` until section 7.25 is built. +`fiona.wiki.ingest`. Absent keys use the defaults in the table. A mapping key under `toc_generator.file` whose name is not one of `min_heading`, `max_heading`, `sectionize`, `ignore_files` is an @@ -1593,8 +1595,8 @@ another `(name, runner)` pair in the wiki sanity list. > Status: Partially Implemented -*7.4.16.* is planned. Every other requirement in this section is -built. +*7.4.16.* and *7.4.17.* are planned. Every other requirement in +this section is built. The renderer is D3 v7, loaded from `https://d3js.org/d3.v7.min.js`, with the visualization embedded in one HTML file. vis.js is not @@ -1991,6 +1993,37 @@ used. HTML draw only. - *acceptance*: JSON includes a node whose `unique_points` is below the setting. +- *7.4.17.* `NOT_IMPLEMENTED` A markdown file that `.gitignore` + matches is not a graph node. HTML and JSON both omit it. An + edge that would start or end at that file is omitted. The file + does not count toward `max_files`. Housekeeping's orphan list + (*7.26.6.*) uses this graph, so an omitted file is not an + orphan. There is no setting that turns this off. + - *acceptance*: The scanned `.gitignore` lists `secret.md`. + `secret.md` is absent from JSON `nodes`, `edges`, and + `orphans`. A page that links to it has no edge to it. + - *acceptance*: `max_files` is 1. The directory holds `kept.md` + and a gitignored `secret.md`. JSON `file_count` is 1 and the + only node is `kept.md`. + - *7.4.17.1.* `NOT_IMPLEMENTED` Patterns are the non-comment, + non-blank lines of the `.gitignore` in the scanned directory, + and of each `.gitignore` in a subdirectory. A subdirectory + file's patterns match only paths under that subdirectory. + Matching is *7.2.10.*: a pattern with no slash matches any + path segment, and a pattern that ends with `/` matches that + directory and everything under it. + - *acceptance*: The scanned `.gitignore` lists `drafts/`. + `drafts/note.md` is absent. `wiki/note.md` is present. + - *acceptance*: A line `# secret.md` does not omit + `secret.md`. + - *acceptance*: `notes/.gitignore` lists `scratch.md`. + `notes/scratch.md` is absent. `scratch.md` at the scan + root stays. + - *7.4.17.2.* `NOT_IMPLEMENTED` A path under `.git/` is omitted + even when no pattern names it. + - *acceptance*: With no `.gitignore`, a markdown file under + the scan root stays, and a markdown file under `.git/` is + absent. ### 7.5. Git Operations @@ -2512,7 +2545,7 @@ is the pip-argument source both scripts call. ### 7.25. wiki ingest -> Status: Planned +> Status: Implemented Implements section 4.14. An ingest has two phases. The tool runs the first: it turns one source into a file on disk, archives a copy @@ -2547,7 +2580,7 @@ A source that no converter claims is archived under `raw/other/`. Pure-Python libraries are preferred so the install stays light. Candidates for later formats are in section 10. -- *7.25.1.* `NOT_IMPLEMENTED` The pipeline is built from parts. +- *7.25.1.* The pipeline is built from parts. - *7.25.1.1.* The pipeline receives its fetchers, converter registry, archiver, page writer, and log writer as constructor arguments. It holds no format-specific code. @@ -2563,7 +2596,7 @@ Candidates for later formats are in section 10. an argument. - *acceptance*: Tests run the URL path with a fake opener and make no network call. -- *7.25.2.* `NOT_IMPLEMENTED` Fetch. +- *7.25.2.* Fetch. - *7.25.2.1.* An argument that starts with `http://` or `https://` is a URL. Any other `://` prefix exits 1 with `Error: Unsupported URL scheme: `. Anything else is a @@ -2607,7 +2640,7 @@ Candidates for later formats are in section 10. - *acceptance*: `https://host/job/performance_TL153/2108/console` has the slug `performance-tl153-2108-console`. -- *7.25.3.* `NOT_IMPLEMENTED` Archive. +- *7.25.3.* Archive. - *7.25.3.1.* The default raw path is `raw//.`. The category is the raw category of the converter that claims the media type, unless @@ -2661,7 +2694,7 @@ Candidates for later formats are in section 10. - *acceptance*: After ingesting `ingest/a.pdf`, that path is gone and `raw/papers/a.pdf` holds its bytes. A file ingested from `personal/` is copied and the original is unchanged. -- *7.25.4.* `NOT_IMPLEMENTED` Convert. +- *7.25.4.* Convert. - *7.25.4.1.* The registry picks the first converter, in registration order, whose media types contain the detected type, then the first whose extensions contain the file extension. With @@ -2723,7 +2756,7 @@ Candidates for later formats are in section 10. - *acceptance*: A two-page fixture with no outline has `## Page 1` and `## Page 2`. A fixture with one blank page is `partial` and names that page. -- *7.25.5.* `NOT_IMPLEMENTED` Write. +- *7.25.5.* Write. - *7.25.5.1.* The default page is `wiki//.md`. `-t/--target PATH` is relative to `wiki/`, and a leading `wiki/` is accepted. A value @@ -2799,7 +2832,7 @@ Candidates for later formats are in section 10. ` markdown style violations left`. - *acceptance*: Every fixture page passes fmlint with no violation. -- *7.25.6.* `NOT_IMPLEMENTED` Record. A run that archives appends one +- *7.25.6.* Record. A run that archives appends one bullet to `wiki/log.md` and bumps that file's patch `version` and `updated`. The bullet is `- - ingest -> -> (raw)`, @@ -2807,10 +2840,10 @@ Candidates for later formats are in section 10. does not edit `wiki/index.md` or any other page. - *acceptance*: One successful run adds one line to `log.md` and leaves `index.md` unchanged. -- *7.25.7.* `NOT_IMPLEMENTED` `--raw-only` runs fetch, archive, and +- *7.25.7.* `--raw-only` runs fetch, archive, and record, and skips convert and write. It exits 0. - *acceptance*: No page is written and `conversion` is `none`. -- *7.25.8.* `NOT_IMPLEMENTED` `--dry-run` fetches into a temporary +- *7.25.8.* `--dry-run` fetches into a temporary directory outside the wiki, so a URL is still downloaded. It prints `Dry run - no files will be modified:` and then the lines a real run would print. It writes nothing under the wiki, does not @@ -2818,12 +2851,12 @@ Candidates for later formats are in section 10. true in JSON. - *acceptance*: File hashes under the wiki match before and after a dry run, and an `ingest/` source is still in place. -- *7.25.9.* `NOT_IMPLEMENTED` `--pending` lists every page under +- *7.25.9.* `--pending` lists every page under `wiki/` whose frontmatter `status` is `raw`, sorted by path, and exits 0. - *acceptance*: A page changed to `status: refined` is no longer listed. -- *7.25.10.* `NOT_IMPLEMENTED` Refinement is the agent's phase. The +- *7.25.10.* Refinement is the agent's phase. The tool never writes `status: refined`. A page is refined when `status` is `refined`, `version` is `1.0.0` or later, the quality signals are set and `llm.model` is not `none`, the page is listed @@ -2832,7 +2865,7 @@ Candidates for later formats are in section 10. - *acceptance*: Fiona skill section 16 lists those conditions and the steps that meet them, and says when to convert a source by hand. -- *7.25.11.* `NOT_IMPLEMENTED` The agent checks the frontmatter the +- *7.25.11.* The agent checks the frontmatter the tool wrote before it refines the body. It may correct `title`, `type`, `tags`, `source_meta`, `ingest.conversion`, and `ingest.warnings` when the source shows they are wrong. It does @@ -2842,11 +2875,11 @@ Candidates for later formats are in section 10. the fault to the user. - *acceptance*: Fiona skill section 16 has a metadata check step that names the fields it may edit and the fields it may not. -- *7.25.12.* `NOT_IMPLEMENTED` Archived sources do not fail +- *7.25.12.* Archived sources do not fail `wiki sanity`. Every check skips `raw/`. - *acceptance*: After ingesting a markdown file with style violations, `wiki sanity` reports none from its raw copy. -- *7.25.13.* `NOT_IMPLEMENTED` `--json` prints only the JSON object +- *7.25.13.* `--json` prints only the JSON object from section 4.14 on stdout, on exit 0 and on exit 3. On exit 1, stdout is empty. - *acceptance*: The stdout of an exit 3 run parses with @@ -2968,10 +3001,11 @@ Paths in details and actions are relative to the wiki root. | P2 | Wiki root discovery | 7.23 | Planned | | P2 | settings setup | 7.19 | Planned | | P2 | Bootstrap install modes | 7.24 | Implemented | -| P2 | wiki ingest | 7.25 | Planned | +| P2 | wiki ingest | 7.25 | Implemented | | P3 | fmlint enforcement of quality signals | 6.2 | Planned | | P3 | Backup archive | 7.6 | Planned | | P3 | Documentation refresh | 7.21 | Partially Implemented | +| P2 | Graph skips gitignored files | 7.4.17 | Planned | | P3 | Node importance filter | 7.4.16 | Planned | | P4 | Non-markdown version tags | 7.14 | Planned | | P4 | Tool-role table | 7.22 | Planned | @@ -3034,7 +3068,7 @@ These notes are not requirements. They point at the current tree. | launchers | `scripts/run_tests.py`, `scripts/run_sanity.py` | | bootstrap | `bootstrap`, `bootstrap.ps1`, `scripts/bootstrap_install.py` | -Planned layout for section 7.25, under `src/fiona/wiki/ingest/`: +Layout for section 7.25, under `src/fiona/wiki/ingest/`: | Module | Classes | Role | | --- | --- | --- | diff --git a/pyproject.toml b/pyproject.toml index e28a5c7..b8a718d 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "fiona-assistant" -version = "0.96.0" +version = "0.99.0" description = "Tools for managing the LLM wiki." authors = [{name = "nvaldeziii"}] readme = "README.md" @@ -24,6 +24,9 @@ dependencies = [ "ruff>=0.16.8", "pyyaml>=6.0.3", "GitPython>=3.1.0", + "beautifulsoup4>=4.12.0", + "markdownify>=1.1.0", + "pypdf>=5.0.0", ] # Optional dependencies groups diff --git a/src/fiona/__main__.py b/src/fiona/__main__.py index 7a4dc98..078c110 100644 --- a/src/fiona/__main__.py +++ b/src/fiona/__main__.py @@ -18,6 +18,7 @@ from fiona.mdoctor.mdoctor import main as mdoctor_main from fiona.python_ruff.python_ruff import main as python_ruff_main from fiona.wiki.graph_generator import wiki_graph_main from fiona.wiki.housekeeping import housekeeping_main +from fiona.wiki.ingest.pipeline import add_ingest_arguments, execute_ingest from fiona.wiki.sanity_manager import sanity_main from fiona.wiki.update_manager import check_update_main, update_main from fiona.wiki.wiki import wiki_init @@ -280,6 +281,14 @@ def main(): help="Disable tracking of external links" ) + # AI/LLM: 2026-09-30 - flags live here so wiki ingest --help is not rejected. + wiki_ingest_parser = wiki_subparsers.add_parser( + "ingest", + prog="fiona-assistant wiki ingest", + help="Archive one source and write a raw wiki page", + ) + add_ingest_arguments(wiki_ingest_parser) + # AI/LLM: fiona-assistant housekeeping (SPEC 4.15) housekeeping_parser = subparsers.add_parser( "housekeeping", @@ -399,6 +408,8 @@ def main(): json_output=args.json, non_interactive=args.non_interactive )) + elif args.wiki_subcommand == "ingest": + sys.exit(execute_ingest(args, wiki_ingest_parser)) elif args.wiki_subcommand == "graph": sys.exit(wiki_graph_main( directory=args.directory, diff --git a/src/fiona/data/llm_wiki/utils/skills/fiona/SKILL.md b/src/fiona/data/llm_wiki/utils/skills/fiona/SKILL.md index daa29db..38ce49b 100644 --- a/src/fiona/data/llm_wiki/utils/skills/fiona/SKILL.md +++ b/src/fiona/data/llm_wiki/utils/skills/fiona/SKILL.md @@ -4,7 +4,7 @@ category: productivity description: Operate an llm-wiki with the fiona-assistant CLI. author: nvaldez license: MIT -version: 1.9.1 +version: 1.11.0 updated: 2026-09-30 --- @@ -104,7 +104,19 @@ fiona-assistant git save --description "ingest : " ``` After a housekeeping pass (lint fixes, index, log, or TOC updates, -page moves, template updates), with sanity passing, run: +page moves, template updates), with sanity passing, read `git diff` +for the files `fiona-assistant git status` lists. + +- **Metadata.** On each added or modified page, frontmatter matches + section 13. `name`, `version`, and `updated` are set. A page under + `wiki/` also has `confidence`, `llm`, `contested`, and + `contradictions`. +- **Integrity.** The body still holds the claims, links, tables, and + index or log entries from before this pass. A lint fix, a move, or + a template merge did not drop or garble them. + +When a check fails, fix the page and run sanity again. When you +cannot tell whether a page is intact, ask before saving. Then run: ```text fiona-assistant git save --description "housekeeping: " @@ -127,13 +139,14 @@ It is the step that makes the wiki's history usable: - **Proof behind the log.** `wiki/log.md` says what happened. The commit holds the before and after state that proves it. -Before saving, run `fiona-assistant git status`. `save` stages +Before either save, run `fiona-assistant git status`. `save` stages everything in the tree. When it shows changes that are not from -this cycle, ask the user before saving. When sanity fails, fix it -first, or ask. A duplicate ingest writes nothing, so `save` -refusing a clean tree is expected. This standing instruction -covers `save` only. `snapshot`, `restore`, `clean`, and `undo` -still need the user's request. +this cycle, ask the user before saving. A housekeeping pass also +reads `git diff` and applies the metadata and integrity checks +above. When sanity fails, fix it first, or ask. A duplicate ingest +writes nothing, so `save` refusing a clean tree is expected. This +standing instruction covers `save` only. `snapshot`, `restore`, +`clean`, and `undo` still need the user's request. ## 1. Before a command @@ -323,10 +336,6 @@ Also: `--include-pattern`, `--exclude-pattern`, ### 4.6. ingest -Planned. Run `fiona-assistant wiki ingest --help` -first. When the subcommand does not exist, do every step of -section 16 by hand. - ```text fiona-assistant wiki ingest [--wiki DIR] [-r|--raw PATH] [-t|--target PATH] [--raw-only] [-f|--force] [--dry-run] [--json] fiona-assistant wiki ingest --pending [--wiki DIR] [--json] @@ -706,10 +715,6 @@ These are not implemented. Do not invent flags for them. - `fiona-assistant pycustomlint` - `fiona-assistant python-ruff fix` -`fiona-assistant wiki ingest` is planned. Call it only after -`fiona-assistant wiki ingest --help` succeeds. Otherwise follow -section 16 by hand. - ## 16. Ingest cycle An ingest has two phases. `wiki ingest` (section 4.6) converts the @@ -776,7 +781,10 @@ the user. Do not move or delete anything under `raw/`. The tool fills every key in section 16.5, but it reads the source mechanically. Check the frontmatter against the source before you -refine, and fix what is wrong. +refine, and fix what is wrong. You may edit `title`, `type`, `tags`, +`source_meta`, `ingest.conversion`, and `ingest.warnings`. You may +not hand-edit `ingest.sha256`, `ingest.size_bytes`, `ingest.fetched`, +`ingest.media_type`, or `ingest.tool`. | Key | Check | May edit | | --- | --- | --- | @@ -800,6 +808,11 @@ with `--force` or tell the user. Do not hand-edit it. Bump the page 1. `fiona-assistant wiki ingest --pending --json` lists the pages still at `status: raw`. Without the tool, search `wiki/` for `status: raw`. Check each page's metadata first (section 16.3). + A markdown source keeps its body. The tool writes the page + frontmatter when the source did not already have it. Source + frontmatter is in `source_meta`. A text source (`.txt`, `.text`, + `.log`) is one fenced `text` block plus that frontmatter, still + at `status: raw`. Rewrite that fence. The tool does not refine it. 2. Read the page and the source it cites. Open the raw file when `ingest.conversion` is `partial` or a claim looks garbled. 3. Rewrite the page as a source summary: @@ -826,8 +839,11 @@ with `--force` or tell the user. Do not hand-edit it. Bump the page `wiki/log.md`. 9. Run the work loop in section 12, ending with `wiki sanity`. -The ingest is done when no page is left at `status: raw` and sanity -passes. Then save it +A page is refined when `status` is `refined`, `version` is `1.0.0` +or later, the quality signals are set and `llm.model` is not `none`, +the page is listed in `wiki/index.md`, and `wiki/log.md` has a line +for the refinement. The ingest is done when every page from the run +meets that and `wiki sanity` passes. Then save it ([save rule](#save-after-every-ingest-cycle-and-housekeeping-pass)). ### 16.5. Raw page frontmatter diff --git a/src/fiona/data/llm_wiki/utils/skills/obsidian/SKILL.md b/src/fiona/data/llm_wiki/utils/skills/obsidian/SKILL.md deleted file mode 100644 index 3e7e72b..0000000 --- a/src/fiona/data/llm_wiki/utils/skills/obsidian/SKILL.md +++ /dev/null @@ -1,97 +0,0 @@ ---- -name: obsidian -description: Read, search, create, and edit notes in the Obsidian vault. -version: 1.0.0 -author: Teknium (teknium1), Hermes Agent -license: MIT -platforms: [linux, macos, windows] -metadata: -hermes: -tags: [Obsidian, Notes, Markdown, Vault] -related_skills: [] ---- - -# Obsidian Vault - -Use this skill for filesystem-first Obsidian vault work: reading notes, listing -notes, searching note files, creating notes, appending content, and adding -wikilinks. - -## Vault path - -Use a known or resolved vault path before calling file tools. - -The documented vault-path convention is the `OBSIDIAN_VAULT_PATH` environment -variable, for example from `${HERMES_HOME:-~/.hermes}/.env`. If it is unset, -use `~/Documents/Obsidian Vault`. - -File tools do not expand shell variables. Do not pass paths containing -`$OBSIDIAN_VAULT_PATH` to `read_file`, `write_file`, `patch`, or -`search_files`; resolve the vault path first and pass a concrete absolute path. -Vault paths may contain spaces, which is another reason to prefer file tools -over shell commands. - -If the vault path is unknown, `terminal` is acceptable for resolving -`OBSIDIAN_VAULT_PATH` or checking whether the fallback path exists. Once the -path is known, switch back to file tools. - -## Read a note - -Use `read_file` with the resolved absolute path to the note. Prefer this over -`cat` because it provides line numbers and pagination. - -## List notes - -Use `search_files` with `target: "files"` and the resolved vault path. Prefer -this over `find` or `ls`. - -- To list all markdown notes, use `pattern: "*.md"` under the vault path. -- To list a subfolder, search under that subfolder's absolute path. - -## Search - -Use `search_files` for both filename and content searches. Prefer this over -`grep`, `find`, or `ls`. - -- For filenames, use `search_files` with `target: "files"` and a filename `pattern`. -- For note contents, use `search_files` with `target: "content"`, the content - regex as `pattern`, and `file_glob: "*.md"` when you want to restrict matches - to markdown notes. - -## Create a note - -Use `write_file` with the resolved absolute path and the full markdown content. -Prefer this over shell heredocs or `echo` because it avoids shell quoting -issues and returns structured results. - -## Append to a note - -Prefer a native file-tool workflow when it is not awkward: - -- Read the target note with `read_file`. -- Use `patch` for an anchored append when there is stable context, such as - adding a section after an existing heading or appending before a known - trailing block. -- Use `write_file` when rewriting the whole note is clearer than constructing a - fragile patch. - -For an anchored append with `patch`, replace the anchor with the anchor plus -the new content. - -For a simple append with no stable context, `terminal` is acceptable if it is -the clearest safe option. - -## Targeted edits - -Use `patch` for focused note changes when the current content gives you stable -context. Prefer this over shell text rewriting. - -## Linking - -For **compatibility** across tools (e.g., GitHub, VS Code, Obsidian), use -**standard Markdown links** (`[link text](path/to/note.md)`) instead of -wikilinks (`[[Note Name]]`). - -- **Markdown links** ensure portability and work in all markdown renderers. -- If Obsidian-specific features (e.g., graph view) are needed, wikilinks can be - used **in addition to** Markdown links, but never as a replacement. diff --git a/src/fiona/wiki/ingest/__init__.py b/src/fiona/wiki/ingest/__init__.py new file mode 100644 index 0000000..715e3d9 --- /dev/null +++ b/src/fiona/wiki/ingest/__init__.py @@ -0,0 +1,4 @@ +"""Wiki ingest package (SPEC 7.25). + +Generated by AI/LLM: 2026-09-30 - stage modules for wiki ingest. +""" diff --git a/src/fiona/wiki/ingest/archiver.py b/src/fiona/wiki/ingest/archiver.py new file mode 100644 index 0000000..d58747f --- /dev/null +++ b/src/fiona/wiki/ingest/archiver.py @@ -0,0 +1,239 @@ +"""Archive a fetched file under ``raw/``. + +Generated by AI/LLM: 2026-09-30 - SPEC 7.25.3. Never edits a file already there. +""" + +from __future__ import annotations + +import hashlib +import logging +import pathlib +import shutil + +from fiona.wiki.ingest.converters.base import ConverterRegistry +from fiona.wiki.ingest.media import extension_for +from fiona.wiki.ingest.models import ArchiveResult, FetchedSource, IngestError +from fiona.wiki.ingest.settings import IngestSettings + +logger = logging.getLogger(__name__) + + +class RawArchiver: + """Copy or move a source into ``raw/`` and reuse identical bytes. + + Generated by AI/LLM: 2026-09-30 - SHA-256 reuse and collision names. + """ + + def __init__(self, wiki_root: pathlib.Path) -> None: + """Store the wiki root. + + Args: + wiki_root: Directory that contains ``raw/``. + """ + self.wiki_root = wiki_root + + def find_by_sha256( + self, + digest: str, + size: int, + skip: pathlib.Path | None = None, + ) -> pathlib.Path | None: + """Return the first ``raw/`` file with this size and digest. + + Generated by AI/LLM: 2026-09-30 - only same-size files are hashed. + + Args: + digest: Lowercase SHA-256 hex. + size: Byte size that a candidate must match. + skip: A file to ignore, usually the source when it already sits in ``raw/``. + + Returns: + pathlib.Path | None: Match with the lowest path, or None. + """ + raw_root = self.wiki_root / "raw" + if not raw_root.is_dir(): + return None + skip_resolved = skip.resolve() if skip is not None else None + matches: list[pathlib.Path] = [] + for path in raw_root.rglob("*"): + if not path.is_file(): + continue + if skip_resolved is not None and path.resolve() == skip_resolved: + continue + try: + if path.stat().st_size != size: + continue + except OSError: + continue + if _sha256(path) == digest: + matches.append(path) + if not matches: + return None + matches.sort(key=lambda item: item.as_posix()) + return matches[0] + + def archive( + self, + source: FetchedSource, + *, + registry: ConverterRegistry, + settings: IngestSettings, + raw_override: str | None, + dry_run: bool, + ) -> ArchiveResult: + """Place the source under ``raw/`` or reuse an identical copy. + + Generated by AI/LLM: 2026-09-30 - SPEC 7.25.3. + + Args: + source: Fetched file. + registry: Used for the raw category and a missing extension. + settings: ``raw_categories`` overrides. + raw_override: ``-r`` value, relative to ``raw/``, or None. + dry_run: When true, compute the path and write nothing. + + Returns: + ArchiveResult: Relative raw path, digest, and warnings. + + Raises: + IngestError: ``-r`` escapes ``raw/`` or is used on a file already there. + """ + raw_root = (self.wiki_root / "raw").resolve() + source_in_raw = _is_inside(source.path, raw_root) + if raw_override and source_in_raw: + raise IngestError( + f"--raw cannot be used on a file already under raw/: {source.argument}" + ) + if raw_override: + _require_inside(raw_root, raw_override, "--raw must stay inside raw/") + digest = _sha256(source.path) + match = self.find_by_sha256( + digest, + source.size_bytes, + skip=source.path if source_in_raw else None, + ) + warnings: list[str] = [] + if match is not None: + raw_rel = _relative(match, self.wiki_root) + if raw_override: + warnings.append(f"already archived at {raw_rel}; --raw ignored") + if _is_inside(source.path, self.wiki_root / "ingest") and not dry_run: + source.path.unlink() + logger.info("reused raw file %s", raw_rel) + return ArchiveResult(match, raw_rel, digest, True, warnings) + if source_in_raw: + return ArchiveResult(source.path, _relative(source.path, self.wiki_root), digest, False, []) + extension = source.path.suffix.lower() or extension_for(source.media_type, "", registry) + dest = self._destination(source, registry, settings, raw_override, extension, raw_root) + dest = _free_name(dest, source.slug, extension) + if not dry_run: + dest.parent.mkdir(parents=True, exist_ok=True) + if _is_inside(source.path, self.wiki_root / "ingest"): + shutil.move(str(source.path), str(dest)) + else: + shutil.copyfile(source.path, dest) + logger.info("archived %s", dest) + return ArchiveResult(dest, _relative(dest, self.wiki_root), digest, False, warnings) + + def _destination( + self, + source: FetchedSource, + registry: ConverterRegistry, + settings: IngestSettings, + raw_override: str | None, + extension: str, + raw_root: pathlib.Path, + ) -> pathlib.Path: + """Resolve the raw path before a collision suffix is applied. + + Generated by AI/LLM: 2026-09-30 - ``-r`` file or directory, else the category. + """ + filename = f"{source.slug}{extension}" + if raw_override: + relative = _strip_prefix(raw_override, "raw") + suffix = pathlib.PurePosixPath(relative.replace("\\", "/")).suffix + if suffix: + return raw_root / relative + return raw_root / relative / filename + converter = registry.find(source.media_type, extension) + if converter is None: + category = "other" + else: + category = settings.raw_categories.get(converter.name, converter.raw_category) + return raw_root / category / filename + + +def _sha256(path: pathlib.Path) -> str: + """Return the lowercase SHA-256 hex digest of a file. + + Generated by AI/LLM: 2026-09-30 - streamed so large files stay cheap. + """ + digest = hashlib.sha256() + with path.open("rb") as handle: + while True: + chunk = handle.read(1024 * 1024) + if not chunk: + break + digest.update(chunk) + return digest.hexdigest() + + +def _free_name(dest: pathlib.Path, slug: str, extension: str) -> pathlib.Path: + """Return ``dest`` or the first free ``-N`` name beside it. + + Generated by AI/LLM: 2026-09-30 - never overwrite a raw file. + """ + if not dest.exists(): + return dest + number = 2 + while True: + candidate = dest.parent / f"{slug}-{number}{extension}" + if not candidate.exists(): + return candidate + number += 1 + + +def _strip_prefix(value: str, prefix: str) -> str: + """Drop a leading ``raw/`` or ``wiki/`` from a user path. + + Generated by AI/LLM: 2026-09-30 - both forms are accepted. + """ + relative = value.replace("\\", "/").lstrip("/") + marker = f"{prefix}/" + if relative == prefix: + return "" + if relative.startswith(marker): + return relative[len(marker):] + return relative + + +def _require_inside(root: pathlib.Path, value: str, message: str) -> pathlib.Path: + """Resolve ``value`` under ``root`` or raise. + + Generated by AI/LLM: 2026-09-30 - ``..`` and symlinks must stay inside. + """ + relative = _strip_prefix(value, root.name) + candidate = (root / relative).resolve() + if not _is_inside(candidate, root): + raise IngestError(f"{message}: {value}") + return candidate + + +def _is_inside(path: pathlib.Path, root: pathlib.Path) -> bool: + """Return whether ``path`` is ``root`` or a file inside it. + + Generated by AI/LLM: 2026-09-30 - compare resolved paths. + """ + try: + path.resolve().relative_to(root.resolve()) + except ValueError: + return False + return True + + +def _relative(path: pathlib.Path, root: pathlib.Path) -> str: + """Return a wiki-relative path with ``/`` separators. + + Generated by AI/LLM: 2026-09-30 - JSON and log paths. + """ + return path.resolve().relative_to(root.resolve()).as_posix() diff --git a/src/fiona/wiki/ingest/converters/__init__.py b/src/fiona/wiki/ingest/converters/__init__.py new file mode 100644 index 0000000..63a00e7 --- /dev/null +++ b/src/fiona/wiki/ingest/converters/__init__.py @@ -0,0 +1,26 @@ +"""Shipped converters, in registration order. + +Generated by AI/LLM: 2026-09-30 - markdown, text, html, then pdf. +""" + +from fiona.wiki.ingest.converters.base import ConverterRegistry +from fiona.wiki.ingest.converters.html import HtmlConverter +from fiona.wiki.ingest.converters.markdown import MarkdownConverter +from fiona.wiki.ingest.converters.pdf import PdfConverter +from fiona.wiki.ingest.converters.text import TextConverter + + +def default_registry() -> ConverterRegistry: + """Return the converters shipped with wiki ingest. + + Generated by AI/LLM: 2026-09-30 - registration order from SPEC 7.25. + + Returns: + ConverterRegistry: Markdown, text, HTML, and PDF. + """ + registry = ConverterRegistry() + registry.register(MarkdownConverter) + registry.register(TextConverter) + registry.register(HtmlConverter) + registry.register(PdfConverter) + return registry diff --git a/src/fiona/wiki/ingest/converters/base.py b/src/fiona/wiki/ingest/converters/base.py new file mode 100644 index 0000000..5c7f979 --- /dev/null +++ b/src/fiona/wiki/ingest/converters/base.py @@ -0,0 +1,108 @@ +"""Converter base types and the registry. + +Generated by AI/LLM: 2026-09-30 - one class per format, SPEC 7.25.1. +""" + +from __future__ import annotations + +import abc +import typing + +from fiona.wiki.ingest.models import ConversionResult, FetchedSource + + +class ConversionError(Exception): + """Raised when a converter cannot read a file it claims. + + Generated by AI/LLM: 2026-09-30 - exit 3, raw copy kept. + """ + + def __init__(self, converter: str, reason: str) -> None: + """Store the converter name and the reason. + + Args: + converter: Converter ``name`` attribute. + reason: Short failure text, without the ``Error:`` prefix. + """ + super().__init__(f"{converter}: {reason}") + self.converter = converter + self.reason = reason + + +class SourceConverter(abc.ABC): + """One format. Subclasses declare what they claim and implement ``convert``. + + Generated by AI/LLM: 2026-09-30 - SPEC 7.25.1.2. + """ + + name: typing.ClassVar[str] + media_types: typing.ClassVar[tuple[str, ...]] + extensions: typing.ClassVar[tuple[str, ...]] + raw_category: typing.ClassVar[str] + + @abc.abstractmethod + def convert(self, source: FetchedSource) -> ConversionResult: + """Convert an archived file to markdown. + + Args: + source: Fetched file, including media type and slug. + + Returns: + ConversionResult: Body, title, metadata, and warnings. + + Raises: + ConversionError: The file cannot be read. + """ + + +class ConverterRegistry: + """Ordered list of converters. + + Generated by AI/LLM: 2026-09-30 - first media-type match, then extension. + """ + + def __init__(self, converters: list[SourceConverter] | None = None) -> None: + """Start with an optional list of converter instances. + + Args: + converters: Instances in registration order. + """ + self.converters: list[SourceConverter] = list(converters or []) + + def register(self, converter: SourceConverter | type[SourceConverter]) -> None: + """Append a converter class or instance. + + Generated by AI/LLM: 2026-09-30 - adding a format is one registration. + + Args: + converter: Class with a no-arg constructor, or a ready instance. + """ + if isinstance(converter, type): + converter = converter() + self.converters.append(converter) + + def find(self, media_type: str, extension: str) -> SourceConverter | None: + """Pick the converter for a media type and extension. + + Generated by AI/LLM: 2026-09-30 - SPEC 7.25.4.1 registration order. + + Args: + media_type: Detected type, or empty. + extension: Suffix including the dot, or empty. + + Returns: + SourceConverter | None: The first match, or None. + """ + media = (media_type or "").lower() + ext = extension.lower() + if ext and not ext.startswith("."): + ext = f".{ext}" + if media: + for converter in self.converters: + if media in converter.media_types: + return converter + if ext: + for converter in self.converters: + if ext in converter.extensions: + return converter + return None diff --git a/src/fiona/wiki/ingest/converters/html.py b/src/fiona/wiki/ingest/converters/html.py new file mode 100644 index 0000000..7b8c039 --- /dev/null +++ b/src/fiona/wiki/ingest/converters/html.py @@ -0,0 +1,128 @@ +"""HTML source converter. + +Generated by AI/LLM: 2026-09-30 - article text via beautifulsoup4 and markdownify. +""" + +from __future__ import annotations + +import typing +import urllib.parse + +import bs4 +import markdownify + +from fiona.wiki.ingest.converters.base import SourceConverter +from fiona.wiki.ingest.media import decode_bytes +from fiona.wiki.ingest.models import ConversionResult, FetchedSource + +_DROP_TAGS = ("script", "style", "noscript", "template") +_NO_TEXT = "no text content; the page may need JavaScript or a login" + + +class HtmlConverter(SourceConverter): + """Convert an HTML document to markdown. + + Generated by AI/LLM: 2026-09-30 - SPEC 7.25.4.5. + """ + + name: typing.ClassVar[str] = "html" + media_types: typing.ClassVar[tuple[str, ...]] = ("text/html", "application/xhtml+xml") + extensions: typing.ClassVar[tuple[str, ...]] = (".html", ".htm", ".xhtml") + raw_category: typing.ClassVar[str] = "articles" + + def convert(self, source: FetchedSource) -> ConversionResult: + """Parse HTML and return the article body as markdown. + + Generated by AI/LLM: 2026-09-30 - drop scripts and absolutize links. + + Args: + source: Archived HTML file. ``origin`` is the page URL when fetched. + + Returns: + ConversionResult: Markdown body, title, and ``source_meta``. + """ + text, warnings = decode_bytes(source.path.read_bytes(), source.charset) + soup = bs4.BeautifulSoup(text, "html.parser") + for tag in soup.find_all(_DROP_TAGS): + tag.decompose() + for comment in soup.find_all(string=lambda node: isinstance(node, bs4.Comment)): + comment.extract() + root = soup.find("article") or soup.find("main") or soup.body or soup + origin = source.origin if source.origin_kind == "url" else None + _absolutize(root, soup, origin) + visible = root.get_text(strip=True) + heading = getattr(markdownify, "ATX", "ATX") + body = markdownify.markdownify(str(root), heading_style=heading, bullets="-").strip() + if body: + body += "\n" + meta = _source_meta(soup) + title = _html_title(soup, root) + conversion = "complete" + if not visible: + conversion = "partial" + warnings = [*warnings, _NO_TEXT] + return ConversionResult( + body=body, + title=title, + source_meta=meta, + conversion=conversion, + warnings=warnings, + converter=self.name, + ) + + +def _absolutize(root: bs4.element.Tag, soup: bs4.BeautifulSoup, origin: str | None) -> None: + """Rewrite relative href and src values against base href, then the origin. + + Generated by AI/LLM: 2026-09-30 - a local file with no origin keeps relative links. + """ + base_tag = soup.find("base", href=True) + base = base_tag.get("href") if base_tag is not None else None + if not base: + base = origin + if not base: + return + for tag in root.find_all(href=True): + tag["href"] = urllib.parse.urljoin(str(base), str(tag["href"])) + for tag in root.find_all(src=True): + tag["src"] = urllib.parse.urljoin(str(base), str(tag["src"])) + + +def _html_title(soup: bs4.BeautifulSoup, root: bs4.element.Tag) -> str | None: + """Return ````, then ``og:title``, then the first ``<h1>``. + + Generated by AI/LLM: 2026-09-30 - SPEC 7.25.4.5.1. + """ + if soup.title and soup.title.string and soup.title.string.strip(): + return soup.title.string.strip() + og = soup.find("meta", attrs={"property": "og:title"}) + if og is not None and og.get("content"): + return str(og["content"]).strip() or None + heading = root.find("h1") + if heading is not None: + text = heading.get_text(strip=True) + return text or None + return None + + +def _source_meta(soup: bs4.BeautifulSoup) -> dict: + """Collect language, canonical URL, and selected meta values. + + Generated by AI/LLM: 2026-09-30 - only keys that are present. + """ + meta: dict[str, str] = {} + root = soup.find("html") + if root is not None and root.get("lang"): + meta["lang"] = str(root["lang"]) + canonical = soup.find("link", rel=lambda value: value and "canonical" in value) + if canonical is not None and canonical.get("href"): + meta["canonical"] = str(canonical["href"]) + for tag in soup.find_all("meta"): + key = tag.get("name") or tag.get("property") + content = tag.get("content") + if not key or content is None: + continue + name = str(key) + if name in {"description", "author"} or name.startswith("og:") or name == "article:published_time": + meta[name] = str(content) + return meta diff --git a/src/fiona/wiki/ingest/converters/markdown.py b/src/fiona/wiki/ingest/converters/markdown.py new file mode 100644 index 0000000..ba515b6 --- /dev/null +++ b/src/fiona/wiki/ingest/converters/markdown.py @@ -0,0 +1,105 @@ +"""Markdown source converter. + +Generated by AI/LLM: 2026-09-30 - keep the body and lift frontmatter into source_meta. +""" + +from __future__ import annotations + +import re +import typing + +import yaml + +from fiona.wiki.ingest.converters.base import SourceConverter +from fiona.wiki.ingest.media import decode_bytes +from fiona.wiki.ingest.models import ConversionResult, FetchedSource + +_FRONTMATTER = re.compile(r"\A---\r?\n(.*?)\r?\n---\r?\n?", re.DOTALL) +_H1 = re.compile(r"^#\s+(\S.*)$") +_FENCE = re.compile(r"^\s*(`{3,}|~{3,})") + + +class MarkdownConverter(SourceConverter): + """Keep markdown as it is and record any source frontmatter. + + Generated by AI/LLM: 2026-09-30 - SPEC 7.25.4.3. + + The body after a frontmatter block is kept character for character. + The page writer adds the wiki frontmatter when the new page does not + have it yet. Source keys move to ``source_meta`` instead of being + written a second time as the page frontmatter. + """ + + name: typing.ClassVar[str] = "markdown" + media_types: typing.ClassVar[tuple[str, ...]] = ("text/markdown",) + extensions: typing.ClassVar[tuple[str, ...]] = (".md", ".markdown") + raw_category: typing.ClassVar[str] = "articles" + + def convert(self, source: FetchedSource) -> ConversionResult: + """Return the markdown body and any frontmatter as metadata. + + Generated by AI/LLM: 2026-09-30 - title from frontmatter, then H1, then slug. + + Args: + source: Archived markdown file. + + Returns: + ConversionResult: Unchanged body, ``complete`` unless decoding fell back. + """ + text, warnings = decode_bytes(source.path.read_bytes(), source.charset) + meta: dict = {} + body = text + match = _FRONTMATTER.match(text) + if match: + loaded = _load_meta(match.group(1)) + if loaded is None: + warnings = [*warnings, "source frontmatter did not parse"] + else: + meta = loaded + body = text[match.end():] + title = _title(meta, body, source.slug) + return ConversionResult( + body=body, + title=title, + source_meta=meta, + conversion="complete", + warnings=warnings, + converter=self.name, + ) + + +def _load_meta(block: str) -> dict | None: + """Parse a frontmatter block into a dict. + + Generated by AI/LLM: 2026-09-30 - invalid YAML stays in the body. + """ + try: + loaded = yaml.safe_load(block) + except yaml.YAMLError: + return None + if loaded is None: + return {} + if isinstance(loaded, dict): + return loaded + return None + + +def _title(meta: dict, body: str, slug: str) -> str: + """Choose the page title. + + Generated by AI/LLM: 2026-09-30 - frontmatter title, first H1, then slug. + """ + raw_title = meta.get("title") + if isinstance(raw_title, str) and raw_title.strip(): + return raw_title.strip() + fenced = False + for line in body.splitlines(): + if _FENCE.match(line): + fenced = not fenced + continue + if fenced: + continue + match = _H1.match(line) + if match: + return match.group(1).strip() + return slug diff --git a/src/fiona/wiki/ingest/converters/pdf.py b/src/fiona/wiki/ingest/converters/pdf.py new file mode 100644 index 0000000..0236b57 --- /dev/null +++ b/src/fiona/wiki/ingest/converters/pdf.py @@ -0,0 +1,186 @@ +"""PDF source converter. + +Generated by AI/LLM: 2026-09-30 - text layer via pypdf, SPEC 7.25.4.6. +""" + +from __future__ import annotations + +import datetime +import typing + +import pypdf +import pypdf.errors + +from fiona.wiki.ingest.converters.base import ConversionError, SourceConverter +from fiona.wiki.ingest.models import ConversionResult, FetchedSource + +_SCANNED = "no text layer; the PDF may be scanned" + + +class PdfConverter(SourceConverter): + """Read a PDF text layer into markdown headings and paragraphs. + + Generated by AI/LLM: 2026-09-30 - SPEC 7.25.4.6. + """ + + name: typing.ClassVar[str] = "pdf" + media_types: typing.ClassVar[tuple[str, ...]] = ("application/pdf",) + extensions: typing.ClassVar[tuple[str, ...]] = (".pdf",) + raw_category: typing.ClassVar[str] = "papers" + + def convert(self, source: FetchedSource) -> ConversionResult: + """Extract page text, outline headings, and document metadata. + + Generated by AI/LLM: 2026-09-30 - encrypted or corrupt files raise. + + Args: + source: Archived PDF. + + Returns: + ConversionResult: Markdown body. Empty pages make it ``partial``. + + Raises: + ConversionError: The PDF is encrypted or cannot be parsed. + """ + try: + reader = pypdf.PdfReader(str(source.path)) + except pypdf.errors.PdfReadError as exc: + raise ConversionError(self.name, str(exc)) from exc + if reader.is_encrypted: + raise ConversionError(self.name, "encrypted PDF") + meta = _metadata(reader) + title = meta.get("title") or source.slug + outline = _outline(reader) + pages = [] + empty: list[int] = [] + for index, page in enumerate(reader.pages, start=1): + try: + text = page.extract_text() or "" + except pypdf.errors.PdfReadError as exc: + raise ConversionError(self.name, str(exc)) from exc + pages.append(text.strip()) + if not text.strip(): + empty.append(index) + body = _render(pages, outline) + warnings: list[str] = [] + conversion = "complete" + if empty and len(empty) == len(pages): + warnings.append(_SCANNED) + conversion = "partial" + elif empty: + warnings.extend(f"page {number} has no text layer" for number in empty) + conversion = "partial" + return ConversionResult( + body=body, + title=str(title), + source_meta=meta, + conversion=conversion, + warnings=warnings, + converter=self.name, + ) + + +def _metadata(reader: pypdf.PdfReader) -> dict: + """Return page count and the document info values that are present. + + Generated by AI/LLM: 2026-09-30 - dates as ISO 8601. + """ + meta: dict[str, object] = {"page_count": len(reader.pages)} + info = reader.metadata + if info is None: + return meta + mapping = ( + ("title", info.title), + ("author", info.author), + ("subject", info.subject), + ("keywords", info.keywords), + ("creation_date", _iso_date(info.creation_date)), + ("mod_date", _iso_date(info.modification_date)), + ) + for key, value in mapping: + if value: + meta[key] = value + return meta + + +def _iso_date(value: object) -> str | None: + """Format a PDF date as ISO 8601. + + Generated by AI/LLM: 2026-09-30 - accept datetime values and ``D:`` strings. + """ + if value is None: + return None + if isinstance(value, datetime.datetime): + if value.tzinfo is None: + return value.strftime("%Y-%m-%dT%H:%M:%SZ") + return value.astimezone(datetime.UTC).strftime("%Y-%m-%dT%H:%M:%SZ") + text = str(value).strip().removeprefix("D:") + digits = "".join(char for char in text if char.isdigit()) + if len(digits) < 8: + return text or None + year, month, day = digits[0:4], digits[4:6], digits[6:8] + hour = digits[8:10] or "00" + minute = digits[10:12] or "00" + second = digits[12:14] or "00" + return f"{year}-{month}-{day}T{hour}:{minute}:{second}Z" + + +def _outline(reader: pypdf.PdfReader) -> list[tuple[int, int, str]]: + """Return outline entries as page index, heading level, and title. + + Generated by AI/LLM: 2026-09-30 - level is depth plus 2, capped at 6. + """ + found: list[tuple[int, int, str]] = [] + try: + items = reader.outline + except pypdf.errors.PdfReadError: + return found + _walk_outline(reader, items, 0, found) + return found + + +def _walk_outline( + reader: pypdf.PdfReader, + items: object, + depth: int, + found: list[tuple[int, int, str]], +) -> None: + """Walk a pypdf outline tree. + + Generated by AI/LLM: 2026-09-30 - nested lists are deeper entries. + """ + if not items: + return + for item in items: # type: ignore[union-attr] + if isinstance(item, list): + _walk_outline(reader, item, depth + 1, found) + continue + try: + page_index = reader.get_destination_page_number(item) + except (ValueError, KeyError, pypdf.errors.PdfReadError): + continue + title = getattr(item, "title", "") or "" + level = min(depth + 2, 6) + found.append((page_index, level, str(title).strip() or "Untitled")) + + +def _render(pages: list[str], outline: list[tuple[int, int, str]]) -> str: + """Render pages with outline headings or ``## Page n`` headings. + + Generated by AI/LLM: 2026-09-30 - SPEC 7.25.4.6.2. + """ + by_page: dict[int, list[str]] = {} + for page_index, level, title in outline: + by_page.setdefault(page_index, []).append(f"{'#' * level} {title}") + chunks: list[str] = [] + for index, text in enumerate(pages): + if outline: + chunks.extend(by_page.get(index, [])) + else: + chunks.append(f"## Page {index + 1}") + if text: + chunks.append(text) + body = "\n\n".join(chunks) + if body: + body += "\n" + return body diff --git a/src/fiona/wiki/ingest/converters/text.py b/src/fiona/wiki/ingest/converters/text.py new file mode 100644 index 0000000..041ef04 --- /dev/null +++ b/src/fiona/wiki/ingest/converters/text.py @@ -0,0 +1,68 @@ +"""Plain-text source converter. + +Generated by AI/LLM: 2026-09-30 - wrap text in a fence; the agent refines the page. +""" + +from __future__ import annotations + +import typing + +from fiona.wiki.ingest.converters.base import SourceConverter +from fiona.wiki.ingest.media import decode_bytes +from fiona.wiki.ingest.models import ConversionResult, FetchedSource + + +class TextConverter(SourceConverter): + """Turn plain text into one fenced markdown block. + + Generated by AI/LLM: 2026-09-30 - SPEC 7.25.4.4. + + The page is written at ``status: raw``. The tool does not summarize + the text. The agent rewrites the fence during refinement. + """ + + name: typing.ClassVar[str] = "text" + media_types: typing.ClassVar[tuple[str, ...]] = ("text/plain",) + extensions: typing.ClassVar[tuple[str, ...]] = (".txt", ".text", ".log") + raw_category: typing.ClassVar[str] = "articles" + + def convert(self, source: FetchedSource) -> ConversionResult: + """Decode the file and wrap it in a fence longer than any backtick run. + + Generated by AI/LLM: 2026-09-30 - title is the slug; fallback decode warns. + + Args: + source: Archived text file. + + Returns: + ConversionResult: Fenced body and the slug as the title. + """ + text, warnings = decode_bytes(source.path.read_bytes(), source.charset) + marker = "`" * _fence_length(text) + if text and not text.endswith("\n"): + text += "\n" + body = f"{marker}text\n{text}{marker}\n" + return ConversionResult( + body=body, + title=source.slug, + source_meta={}, + conversion="complete", + warnings=warnings, + converter=self.name, + ) + + +def _fence_length(text: str) -> int: + """Return a fence length longer than the longest backtick run. + + Generated by AI/LLM: 2026-09-30 - a run of three when the text has fewer. + """ + longest = 0 + current = 0 + for char in text: + if char == "`": + current += 1 + longest = max(longest, current) + else: + current = 0 + return max(3, longest + 1) diff --git a/src/fiona/wiki/ingest/fetchers.py b/src/fiona/wiki/ingest/fetchers.py new file mode 100644 index 0000000..80d0e02 --- /dev/null +++ b/src/fiona/wiki/ingest/fetchers.py @@ -0,0 +1,276 @@ +"""Fetch a local file or an HTTP URL. + +Generated by AI/LLM: 2026-09-30 - SPEC 7.25.2. No credentials are sent. +""" + +from __future__ import annotations + +import logging +import pathlib +import re +import urllib.error +import urllib.parse +import urllib.request + +from fiona.wiki.ingest.converters.base import ConverterRegistry +from fiona.wiki.ingest.media import ( + detect_media_type, + extension_for, + slug_from_file, + slug_from_url, + utc_now, +) +from fiona.wiki.ingest.models import FetchedSource, IngestError +from fiona.wiki.ingest.settings import IngestSettings, default_settings + +logger = logging.getLogger(__name__) + +_SCHEME = re.compile(r"^([A-Za-z][A-Za-z0-9+.-]*)://") +_WINDOWS_DRIVE = re.compile(r"^[A-Za-z]:[\\/]") + + +def url_scheme(argument: str) -> str | None: + """Return the URL scheme, or None for a local path. + + Generated by AI/LLM: 2026-09-30 - a Windows drive letter is not a scheme. + + Args: + argument: The ``<source>`` argument as given. + + Returns: + str | None: Scheme text, preserving case, or None. + """ + if _WINDOWS_DRIVE.match(argument): + return None + match = _SCHEME.match(argument) + if match is None: + return None + return match.group(1) + + +class SourceFetcher: + """Base fetcher. Subclasses accept an argument and return a local file. + + Generated by AI/LLM: 2026-09-30 - SPEC 7.25.1.3. + """ + + settings: IngestSettings | None = None + + def accepts(self, argument: str) -> bool: + """Return whether this fetcher handles the argument. + + Args: + argument: The ``<source>`` argument as given. + + Returns: + bool: True when ``fetch`` can run. + """ + raise NotImplementedError + + def fetch(self, argument: str, dest_dir: pathlib.Path) -> FetchedSource: + """Fetch the argument into ``dest_dir`` when a download is required. + + Args: + argument: The ``<source>`` argument as given. + dest_dir: Directory for a downloaded file. Local fetches ignore it. + + Returns: + FetchedSource: Local path and origin facts. + """ + raise NotImplementedError + + +class LocalFileFetcher(SourceFetcher): + """Read a local file in place. + + Generated by AI/LLM: 2026-09-30 - missing files and directories exit 1. + """ + + def __init__(self, registry: ConverterRegistry) -> None: + """Store the registry used to name the media type. + + Args: + registry: Converter registry for extension lookup. + """ + self.registry = registry + self.settings = None + + def accepts(self, argument: str) -> bool: + """Accept any argument that is not a URL. + + Generated by AI/LLM: 2026-09-30 - scheme detection stays in ``url_scheme``. + + Args: + argument: The ``<source>`` argument as given. + + Returns: + bool: True when there is no ``://`` scheme. + """ + return url_scheme(argument) is None + + def fetch(self, argument: str, dest_dir: pathlib.Path) -> FetchedSource: + """Resolve a local path without copying it. + + Generated by AI/LLM: 2026-09-30 - SPEC 7.25.2.1. + + Args: + argument: Local path as given. + dest_dir: Unused. Present so every fetcher shares one signature. + + Returns: + FetchedSource: The resolved file. ``created`` is false. + + Raises: + IngestError: The path is missing or is a directory. + """ + del dest_dir + path = pathlib.Path(argument) + if not path.exists(): + raise IngestError(f"Source not found: {argument}") + if path.is_dir(): + raise IngestError(f"Source is a directory: {argument}") + data = path.read_bytes() + media, charset = detect_media_type(data, path.suffix, None, self.registry) + resolved = path.resolve() + logger.info("fetched local file %s as %s", resolved, media) + return FetchedSource( + argument=argument, + path=resolved, + origin=str(resolved), + origin_kind="file", + final_url=None, + media_type=media, + charset=charset, + size_bytes=len(data), + slug=slug_from_file(path), + fetched=utc_now(), + created=False, + ) + + +class UrlFetcher(SourceFetcher): + """Download one ``http`` or ``https`` URL. + + Generated by AI/LLM: 2026-09-30 - opener is injected; tests make no network call. + """ + + def __init__( + self, + registry: ConverterRegistry, + opener: urllib.request.OpenerDirector | None = None, + ) -> None: + """Store the registry and the URL opener. + + Args: + registry: Converter registry for the saved extension. + opener: ``urllib`` opener. The default follows redirects and proxies. + """ + self.registry = registry + self.opener = opener or urllib.request.build_opener() + self.settings = None + + def accepts(self, argument: str) -> bool: + """Accept ``http`` and ``https`` only. + + Generated by AI/LLM: 2026-09-30 - other schemes are rejected by the pipeline. + + Args: + argument: The ``<source>`` argument as given. + + Returns: + bool: True for http and https. + """ + scheme = url_scheme(argument) + return scheme is not None and scheme.lower() in {"http", "https"} + + def fetch(self, argument: str, dest_dir: pathlib.Path) -> FetchedSource: + """Download a URL into ``dest_dir`` and stop past the size limit. + + Generated by AI/LLM: 2026-09-30 - SPEC 7.25.2.2 and 7.25.2.3. + + Args: + argument: Absolute URL. + dest_dir: Directory that receives ``<slug>.<ext>``. + + Returns: + FetchedSource: The saved file. ``created`` is true. + + Raises: + IngestError: The response is not 2xx, the network fails, or the body is too large. + """ + settings = self.settings or default_settings() + request = urllib.request.Request( + _without_userinfo(argument), + headers={"User-Agent": settings.user_agent}, + ) + try: + response = self.opener.open(request, timeout=settings.timeout_seconds) + except urllib.error.HTTPError as exc: + raise IngestError(f"Download failed: {argument}: {exc.code} {exc.reason}") from exc + except urllib.error.URLError as exc: + raise IngestError(f"Download failed: {argument}: {exc.reason}") from exc + with response: + final_url = response.geturl() + header = response.headers.get("Content-Type") + body = _read_limited(response, settings.max_download_bytes, argument) + parsed = urllib.parse.urlsplit(argument) + suffix = pathlib.PurePosixPath(parsed.path).suffix + media, charset = detect_media_type(body, suffix, header, self.registry) + extension = extension_for(media, suffix, self.registry) + slug = slug_from_url(argument) + dest_dir.mkdir(parents=True, exist_ok=True) + target = dest_dir / f"{slug}{extension}" + partial = target.with_name(target.name + ".partial") + try: + partial.write_bytes(body) + partial.replace(target) + except OSError: + if partial.exists(): + partial.unlink() + raise + logger.info("downloaded %s as %s (%s bytes)", argument, media, len(body)) + return FetchedSource( + argument=argument, + path=target, + origin=argument, + origin_kind="url", + final_url=final_url, + media_type=media, + charset=charset, + size_bytes=len(body), + slug=slug, + fetched=utc_now(), + created=True, + ) + + +def _without_userinfo(url: str) -> str: + """Drop userinfo so the request sends no credentials. + + Generated by AI/LLM: 2026-09-30 - SPEC 7.25.2.2. + """ + parts = urllib.parse.urlsplit(url) + if parts.username is None and parts.password is None: + return url + host = parts.hostname or "" + if parts.port: + host = f"{host}:{parts.port}" + return urllib.parse.urlunsplit((parts.scheme, host, parts.path, parts.query, parts.fragment)) + + +def _read_limited(response: object, limit: int, url: str) -> bytes: + """Read a response body, refusing anything over ``limit``. + + Generated by AI/LLM: 2026-09-30 - a partial download is not returned. + """ + chunks: list[bytes] = [] + total = 0 + while True: + block = response.read(65536) # type: ignore[attr-defined] + if not block: + break + total += len(block) + if total > limit: + raise IngestError(f"Download exceeds {limit} bytes: {url}") + chunks.append(block) + return b"".join(chunks) diff --git a/src/fiona/wiki/ingest/log_writer.py b/src/fiona/wiki/ingest/log_writer.py new file mode 100644 index 0000000..db23020 --- /dev/null +++ b/src/fiona/wiki/ingest/log_writer.py @@ -0,0 +1,96 @@ +"""Append one ingest line to ``wiki/log.md``. + +Generated by AI/LLM: 2026-09-30 - SPEC 7.25.6. Does not edit ``index.md``. +""" + +from __future__ import annotations + +import logging +import pathlib +import re + +logger = logging.getLogger(__name__) + +_VERSION = re.compile(r"^version:\s*(\S+)\s*$", re.MULTILINE) +_UPDATED = re.compile(r"^updated:\s*(\S+)\s*$", re.MULTILINE) + + +class IngestLog: + """Record a finished archive in ``wiki/log.md``. + + Generated by AI/LLM: 2026-09-30 - bump patch version and ``updated``. + """ + + def __init__(self, wiki_root: pathlib.Path) -> None: + """Store the wiki root. + + Args: + wiki_root: Directory that contains ``wiki/log.md``. + """ + self.wiki_root = wiki_root + + def append( + self, + *, + day: str, + origin: str, + raw_rel: str, + page_rel: str | None, + dry_run: bool, + ) -> None: + """Append one bullet and bump the log frontmatter. + + Generated by AI/LLM: 2026-09-30 - ``(raw)`` or ``(no page)``. + + Args: + day: UTC date ``YYYY-MM-DD``. + origin: URL or resolved file path. + raw_rel: Raw path relative to the wiki root. + page_rel: Page path relative to the wiki root, or None. + dry_run: When true, leave the log untouched. + """ + if dry_run: + return + path = self.wiki_root / "wiki" / "log.md" + if page_rel: + bullet = f"- {day} - ingest {origin} -> {raw_rel} -> {page_rel} (raw)" + else: + bullet = f"- {day} - ingest {origin} -> {raw_rel} (no page)" + if not path.is_file(): + logger.warning("wiki/log.md is missing; ingest was not recorded") + return + text = path.read_text(encoding="utf-8") + text = _bump_version(text) + text = _set_updated(text, day) + if not text.endswith("\n"): + text += "\n" + text += bullet + "\n" + path.write_text(text, encoding="utf-8", newline="\n") + logger.info("recorded %s", bullet) + + +def _bump_version(text: str) -> str: + """Increment the patch number of the first ``version`` field. + + Generated by AI/LLM: 2026-09-30 - ``2.0.0`` becomes ``2.0.1``. + """ + match = _VERSION.search(text) + if match is None: + return text + parts = match.group(1).split(".") + if len(parts) == 3 and all(part.isdigit() for part in parts): + parts[2] = str(int(parts[2]) + 1) + bumped = ".".join(parts) + return text[: match.start(1)] + bumped + text[match.end(1) :] + return text + + +def _set_updated(text: str, day: str) -> str: + """Set the first ``updated`` field to ``day``. + + Generated by AI/LLM: 2026-09-30 - replaces a placeholder or a previous date. + """ + match = _UPDATED.search(text) + if match is None: + return text + return text[: match.start(1)] + day + text[match.end(1) :] diff --git a/src/fiona/wiki/ingest/media.py b/src/fiona/wiki/ingest/media.py new file mode 100644 index 0000000..43436b7 --- /dev/null +++ b/src/fiona/wiki/ingest/media.py @@ -0,0 +1,255 @@ +"""Media-type detection and slug rules for wiki ingest. + +Generated by AI/LLM: 2026-09-30 - SPEC 7.25.2.4 and 7.25.2.6. +""" + +from __future__ import annotations + +import datetime +import mimetypes +import pathlib +import re +import urllib.parse + +_SLUG_TRIM = 80 +_UNUSABLE_MEDIA = {"", "application/octet-stream"} + + +def make_slug(text: str) -> str: + """Turn free text into a wiki slug. + + Generated by AI/LLM: 2026-09-30 - lowercase, hyphenate, and cap at 80. + + Args: + text: Stem, host, or joined path segments. + + Returns: + str: A slug, or ``source`` when the text has no usable characters. + """ + slug = re.sub(r"[^a-z0-9]+", "-", text.lower()).strip("-")[:_SLUG_TRIM] + return slug or "source" + + +def slug_from_url(url: str) -> str: + """Build a slug from the last three path segments of a URL. + + Generated by AI/LLM: 2026-09-30 - drop ``index`` and an empty path uses the host. + + Args: + url: Absolute ``http`` or ``https`` URL. + + Returns: + str: Slug for the downloaded file and the wiki page. + """ + parts = urllib.parse.urlsplit(url) + cleaned: list[str] = [] + for segment in parts.path.split("/"): + if not segment: + continue + path = pathlib.PurePosixPath(segment) + stem = path.stem if path.suffix else segment + if stem.lower() == "index": + continue + cleaned.append(stem) + if not cleaned: + return make_slug(parts.hostname or "source") + return make_slug("-".join(cleaned[-3:])) + + +def slug_from_file(path: pathlib.Path) -> str: + """Build a slug from a file stem. + + Generated by AI/LLM: 2026-09-30 - local paths use the stem only. + + Args: + path: Local file path. It does not have to exist. + + Returns: + str: Slug for the archive name and the wiki page. + """ + return make_slug(path.stem) + + +def split_content_type(header: str | None) -> tuple[str | None, str | None]: + """Split a Content-Type header into a media type and charset. + + Generated by AI/LLM: 2026-09-30 - drop parameters other than charset. + + ``application/octet-stream`` is treated as missing so the extension + or the leading bytes can decide. + + Args: + header: Raw Content-Type value, or None. + + Returns: + tuple: Media type without parameters, and charset if one was set. + """ + if not header: + return None, None + parts = [part.strip() for part in header.split(";") if part.strip()] + media = parts[0].lower() + charset = None + for part in parts[1:]: + if "=" not in part: + continue + key, value = part.split("=", 1) + if key.strip().lower() == "charset": + charset = value.strip().strip("\"'") + if media in _UNUSABLE_MEDIA: + media = None + return media, charset + + +def media_for_extension(suffix: str, registry: object) -> str | None: + """Map a file extension to a media type. + + Generated by AI/LLM: 2026-09-30 - converter table first, then mimetypes. + + Args: + suffix: Extension including the leading dot. + registry: Converter registry in registration order. + + Returns: + str | None: Media type, or None when the extension is unknown. + """ + lowered = suffix.lower() + for converter in registry.converters: # type: ignore[attr-defined] + if lowered in converter.extensions and converter.media_types: + return converter.media_types[0] + guessed, _encoding = mimetypes.guess_type("file" + lowered) + return guessed + + +def sniff_media_type(data: bytes) -> str: + """Guess a media type from the first 1024 bytes. + + Generated by AI/LLM: 2026-09-30 - PDF, HTML, UTF-8 text, or octet-stream. + + Args: + data: File bytes. Only the prefix is inspected. + + Returns: + str: Detected media type. + """ + head = data[:1024] + if head.startswith(b"%PDF-"): + return "application/pdf" + sample = head.lstrip().lower() + if sample.startswith((b"<!doctype html", b"<html")): + return "text/html" + if b"\x00" not in head and _is_utf8(head): + return "text/plain" + return "application/octet-stream" + + +def detect_media_type( + data: bytes, + suffix: str, + header: str | None, + registry: object, +) -> tuple[str, str | None]: + """Detect a media type and optional charset hint. + + Generated by AI/LLM: 2026-09-30 - header, then extension, then bytes. + + Args: + data: File bytes, used only when the header and extension miss. + suffix: Extension including the dot, or empty. + header: Content-Type header for a download, or None for a local file. + registry: Converter registry. + + Returns: + tuple: Media type and charset hint. + """ + media, charset = split_content_type(header) + if media: + return media, charset + if suffix: + from_ext = media_for_extension(suffix, registry) + if from_ext: + return from_ext, charset + return sniff_media_type(data), charset + + +def extension_for(media_type: str, path_suffix: str, registry: object) -> str: + """Choose the extension for a downloaded file. + + Generated by AI/LLM: 2026-09-30 - converter extension, URL suffix, then ``.bin``. + + Args: + media_type: Detected media type. + path_suffix: Suffix from the URL path, including the dot, or empty. + registry: Converter registry. + + Returns: + str: Extension including the leading dot. + """ + converter = registry.find(media_type, "") # type: ignore[attr-defined] + if converter is not None and converter.extensions: + return converter.extensions[0] + if path_suffix: + suffix = path_suffix.lower() + return suffix if suffix.startswith(".") else f".{suffix}" + return ".bin" + + +def decode_bytes(data: bytes, charset: str | None) -> tuple[str, list[str]]: + """Decode source bytes to text. + + Generated by AI/LLM: 2026-09-30 - UTF-8, then the charset hint, then cp1252. + + A BOM is stripped before UTF-8 is tried. A fallback decode adds one warning. + + Args: + data: Raw file bytes. + charset: Optional charset from Content-Type. + + Returns: + tuple: Decoded text and warnings. + """ + payload = data[3:] if data.startswith(b"\xef\xbb\xbf") else data + try: + return payload.decode("utf-8"), [] + except UnicodeDecodeError: + pass + if charset: + try: + return payload.decode(charset), [f"decoded as {charset}; not valid UTF-8"] + except (UnicodeDecodeError, LookupError): + pass + return payload.decode("cp1252"), ["decoded as cp1252; not valid UTF-8"] + + +def utc_now() -> str: + """Return the current UTC time as ``YYYY-MM-DDTHH:MM:SSZ``. + + Generated by AI/LLM: 2026-09-30 - ingest ``fetched`` timestamp. + + Returns: + str: UTC timestamp with a ``Z`` suffix. + """ + moment = datetime.datetime.now(datetime.UTC).replace(microsecond=0) + return moment.strftime("%Y-%m-%dT%H:%M:%SZ") + + +def utc_day() -> str: + """Return today's UTC date as ``YYYY-MM-DD``. + + Generated by AI/LLM: 2026-09-30 - page ``created`` and ``updated`` dates. + + Returns: + str: UTC calendar date. + """ + return utc_now()[:10] + + +def _is_utf8(data: bytes) -> bool: + """Return whether a byte prefix is valid UTF-8. + + Generated by AI/LLM: 2026-09-30 - allow a multibyte character cut at 1024. + """ + try: + data.decode("utf-8") + except UnicodeDecodeError as exc: + return exc.start > 0 and exc.start >= len(data) - 3 and "unexpected end" in exc.reason + return True diff --git a/src/fiona/wiki/ingest/models.py b/src/fiona/wiki/ingest/models.py new file mode 100644 index 0000000..62a0967 --- /dev/null +++ b/src/fiona/wiki/ingest/models.py @@ -0,0 +1,126 @@ +"""Dataclasses passed between wiki ingest stages. + +Generated by AI/LLM: 2026-09-30 - fetch, convert, and JSON result types. +""" + +from __future__ import annotations + +import dataclasses +import pathlib + + +class IngestError(Exception): + """Raised when wiki ingest must stop before a page is written. + + Generated by AI/LLM: 2026-09-30 - exit 1 errors from section 4.14. + """ + + def __init__(self, message: str, exit_code: int = 1) -> None: + """Store the stderr message and process exit code. + + Args: + message: Text after ``Error: ``. + exit_code: Process status. Section 4.14 uses 1. + """ + super().__init__(message) + self.exit_code = exit_code + + +@dataclasses.dataclass +class FetchedSource: + """A local file plus the facts the later stages need. + + Generated by AI/LLM: 2026-09-30 - output of the fetch stage. + """ + + argument: str + path: pathlib.Path + origin: str + origin_kind: str + final_url: str | None + media_type: str + charset: str | None + size_bytes: int + slug: str + fetched: str + created: bool + + +@dataclasses.dataclass +class ConversionResult: + """Markdown produced from one archived file. + + Generated by AI/LLM: 2026-09-30 - output of one converter. + """ + + body: str + title: str | None + source_meta: dict + conversion: str + warnings: list[str] + converter: str + + +@dataclasses.dataclass +class ArchiveResult: + """Where the raw copy lives after the archive stage. + + Generated by AI/LLM: 2026-09-30 - raw path, hash, and reuse flag. + """ + + path: pathlib.Path + raw_rel: str + sha256: str + raw_reused: bool + warnings: list[str] + + +@dataclasses.dataclass +class IngestResult: + """One ingest run, including the JSON object from section 4.14. + + Generated by AI/LLM: 2026-09-30 - stdout facts for a finished run. + """ + + source: str + origin: str + origin_kind: str + media_type: str + raw_path: str | None + raw_reused: bool + sha256: str + duplicate_page: str | None + page_path: str | None + converter: str | None + conversion: str + status: str | None + warnings: list[str] + dry_run: bool + exit_code: int = 0 + error: str | None = None + messages: list[str] = dataclasses.field(default_factory=list) + + def to_dict(self) -> dict: + """Return the JSON object from section 4.14. + + Generated by AI/LLM: 2026-09-30 - drop CLI-only fields. + + Returns: + dict: Keys and nulls from the ingest JSON contract. + """ + return { + "source": self.source, + "origin": self.origin, + "origin_kind": self.origin_kind, + "media_type": self.media_type, + "raw_path": self.raw_path, + "raw_reused": self.raw_reused, + "sha256": self.sha256, + "duplicate_page": self.duplicate_page, + "page_path": self.page_path, + "converter": self.converter, + "conversion": self.conversion, + "status": self.status, + "warnings": list(self.warnings), + "dry_run": self.dry_run, + } diff --git a/src/fiona/wiki/ingest/page_writer.py b/src/fiona/wiki/ingest/page_writer.py new file mode 100644 index 0000000..16c318a --- /dev/null +++ b/src/fiona/wiki/ingest/page_writer.py @@ -0,0 +1,375 @@ +"""Write one ``status: raw`` wiki page. + +Generated by AI/LLM: 2026-09-30 - SPEC 7.25.5. The tool never sets ``refined``. +""" + +from __future__ import annotations + +import logging +import os +import pathlib +import re + +import yaml +from pymarkdown.api import PyMarkdownApiException + +from fiona.mdoctor.mdoctor import MDoctor +from fiona.wiki.ingest.media import utc_day +from fiona.wiki.ingest.models import ArchiveResult, ConversionResult, FetchedSource, IngestError +from fiona.wiki.ingest.settings import IngestSettings, package_version + +logger = logging.getLogger(__name__) + +_FENCE = re.compile(r"^\s*(`{3,}|~{3,})") +_ATX = re.compile(r"^(#{1,6})(\s+)(.*)$") +_IMAGE = re.compile(r"!\[([^\]]*)\]\(([^)]+)\)") +_LINK = re.compile(r"(?<!!)\[([^\]]*)\]\(([^)]+)\)") +_PROTECTED = {"index.md", "log.md", "schema.md"} + + +class RawPageWriter: + """Write the raw page and run the markdown fixer on that file only. + + Generated by AI/LLM: 2026-09-30 - frontmatter order from SPEC 7.25.5.3. + """ + + def __init__(self, wiki_root: pathlib.Path) -> None: + """Store the wiki root. + + Args: + wiki_root: Directory that contains ``wiki/``. + """ + self.wiki_root = wiki_root + + def page_path( + self, + slug: str, + settings: IngestSettings, + target: str | None, + ) -> pathlib.Path: + """Resolve the page path and reject paths outside ``wiki/``. + + Generated by AI/LLM: 2026-09-30 - SPEC 7.25.5.1. Does not touch the disk. + + Args: + slug: Page stem when ``target`` is a directory or omitted. + settings: Supplies ``default_target``. + target: ``-t`` value, or None. + + Returns: + pathlib.Path: Absolute page path inside ``wiki/``. + + Raises: + IngestError: The path escapes ``wiki/`` or names a protected file. + """ + wiki = (self.wiki_root / "wiki").resolve() + if target is None: + relative = f"{settings.default_target}/{slug}.md" + else: + relative = _strip_wiki_prefix(target) + suffix = pathlib.PurePosixPath(relative.replace("\\", "/")).suffix + if suffix != ".md": + relative = f"{relative}/{slug}.md" if relative else f"{slug}.md" + candidate = (wiki / relative).resolve() + try: + page_rel = candidate.relative_to(wiki).as_posix() + except ValueError as exc: + raise IngestError(f"--target must stay inside wiki/: {target}") from exc + if page_rel in _PROTECTED or page_rel.split("/", 1)[0] == "logs": + shown = target if target else page_rel + raise IngestError(f"--target cannot replace {shown}") + return candidate + + def write( + self, + *, + source: FetchedSource, + archive: ArchiveResult, + conversion: ConversionResult, + page: pathlib.Path, + dry_run: bool, + ) -> tuple[str, list[str]]: + """Write the page, fix it, and return the relative path and warnings. + + Generated by AI/LLM: 2026-09-30 - SPEC 7.25.5.3 through 7.25.5.5. + + Args: + source: Fetch facts stored in ``ingest``. + archive: Raw path, hash, and reuse flag. + conversion: Converter output. + page: Destination markdown file. + dry_run: When true, build the warnings and write nothing. + + Returns: + tuple: Page path relative to the wiki, and the warning list. + """ + title = conversion.title or source.slug + content, link_warnings = _page_content(conversion.body, title, page, archive.path) + warnings = [*conversion.warnings, *archive.warnings, *link_warnings] + if dry_run: + return _rel(page, self.wiki_root), warnings + page.parent.mkdir(parents=True, exist_ok=True) + day = utc_day() + page.write_text( + _document(source, archive, conversion, content, warnings, day), + encoding="utf-8", + newline="\n", + ) + _fix(page, self.wiki_root) + style_count = _style_violations(page, self.wiki_root) + if style_count: + warnings.append(f"{style_count} markdown style violations left") + fixed = _body_after_frontmatter(page.read_text(encoding="utf-8")) + if not fixed.lstrip("\n").startswith("# "): + fixed = content + page.write_text( + _document(source, archive, conversion, fixed.lstrip("\n"), warnings, day), + encoding="utf-8", + newline="\n", + ) + logger.info("wrote %s", page) + return _rel(page, self.wiki_root), warnings + + +def _page_content( + body: str, + title: str, + page: pathlib.Path, + raw_path: pathlib.Path, +) -> tuple[str, list[str]]: + """Demote extra H1s, drop dead relative links, and add the ingest intro. + + Generated by AI/LLM: 2026-09-30 - the page has one H1 and a link to the raw file. + """ + shifted = _shift_headings(body) + rewritten, count = _rewrite_links(shifted, page.parent) + warnings = [f"{count} relative link(s) did not resolve"] if count else [] + link = os.path.relpath(raw_path, page.parent).replace("\\", "/") + intro = f"Raw ingest of [{raw_path.name}]({link}). Not refined yet." + prepared = rewritten.strip("\n") + tail = f"{intro}\n\n{prepared}\n" if prepared else f"{intro}\n" + return f"# {title}\n\n{tail}", warnings + + +def _document( + source: FetchedSource, + archive: ArchiveResult, + conversion: ConversionResult, + content: str, + warnings: list[str], + day: str, +) -> str: + """Render frontmatter plus the page content. + + Generated by AI/LLM: 2026-09-30 - every key is present, in contract order. + """ + title = conversion.title or source.slug + document = { + "name": source.slug, + "version": "0.1.0", + "title": title, + "created": day, + "updated": day, + "type": "summary", + "status": "raw", + "sources": [archive.raw_rel], + "source_meta": conversion.source_meta or {}, + "ingest": { + "origin": source.origin, + "origin_kind": source.origin_kind, + "final_url": source.final_url, + "fetched": source.fetched, + "media_type": source.media_type, + "size_bytes": source.size_bytes, + "sha256": archive.sha256, + "raw_reused": archive.raw_reused, + "converter": conversion.converter, + "conversion": conversion.conversion, + "warnings": list(warnings), + "tool": f"fiona-assistant {package_version()}", + }, + "tags": [], + "confidence": "zero", + "llm": [{"model": "none"}], + "contested": False, + "contradictions": [], + } + dumped = yaml.safe_dump(document, sort_keys=False, allow_unicode=True, width=1000) + payload = content if content.endswith("\n") else content + "\n" + return f"---\n{dumped}---\n\n{payload}" + + +def _shift_headings(body: str) -> str: + """Move every ATX heading down one level when the body has an H1. + + Generated by AI/LLM: 2026-09-30 - H6 stays H6. Fenced lines are left alone. + """ + if not _has_h1(body): + return body + lines = body.splitlines(keepends=True) + fenced = False + shifted: list[str] = [] + for line in lines: + ending, content = _split_ending(line) + fence = _FENCE.match(content) + if fence: + fenced = not fenced + shifted.append(line) + continue + match = None if fenced else _ATX.match(content) + if match is None: + shifted.append(line) + continue + marks = match.group(1) + if len(marks) < 6: + marks += "#" + shifted.append(f"{marks}{match.group(2)}{match.group(3)}{ending}") + return "".join(shifted) + + +def _has_h1(body: str) -> bool: + """Return whether an unfenced ATX H1 is present. + + Generated by AI/LLM: 2026-09-30 - detection does not change the body. + """ + fenced = False + for line in body.splitlines(): + if _FENCE.match(line): + fenced = not fenced + continue + if not fenced and re.match(r"^#\s+\S", line): + return True + return False + + +def _rewrite_links(body: str, page_dir: pathlib.Path) -> tuple[str, int]: + """Replace relative links and images that do not resolve from the page. + + Generated by AI/LLM: 2026-09-30 - one warning carries the count. + """ + count = 0 + lines = body.splitlines(keepends=True) + fenced = False + rewritten: list[str] = [] + for line in lines: + ending, content = _split_ending(line) + if _FENCE.match(content): + fenced = not fenced + rewritten.append(line) + continue + if fenced: + rewritten.append(line) + continue + content, image_count = _replace_pattern(_IMAGE, content, page_dir) + content, link_count = _replace_pattern(_LINK, content, page_dir) + count += image_count + link_count + rewritten.append(content + ending) + return "".join(rewritten), count + + +def _replace_pattern( + pattern: re.Pattern[str], + content: str, + page_dir: pathlib.Path, +) -> tuple[str, int]: + """Replace matches whose target does not exist. + + Generated by AI/LLM: 2026-09-30 - external URLs and anchors stay. + """ + count = 0 + + def _swap(match: re.Match[str]) -> str: + nonlocal count + target = match.group(2).strip() + if _keeps_target(target, page_dir): + return match.group(0) + count += 1 + return match.group(1) + + return pattern.sub(_swap, content), count + + +def _keeps_target(target: str, page_dir: pathlib.Path) -> bool: + """Return whether a link target should stay in the page. + + Generated by AI/LLM: 2026-09-30 - http, mailto, and anchors are kept. + """ + lowered = target.lower() + if lowered.startswith(("http://", "https://", "mailto:", "#", "//")) or "://" in target: + return True + path_part = target.split("#", 1)[0].split("?", 1)[0] + if not path_part: + return True + resolved = (page_dir / path_part).resolve() + return resolved.exists() + + +def _fix(page: pathlib.Path, wiki_root: pathlib.Path) -> None: + """Run the pymarkdown fixer on one page. + + Generated by AI/LLM: 2026-09-30 - SPEC 7.25.5.5. A fixer error is logged. + """ + try: + doctor = MDoctor(settings_file=wiki_root / "fiona-settings.json") + doctor.fix_file_programmatically(str(page)) + except PyMarkdownApiException: + logger.warning("markdown fix failed for %s", page, exc_info=True) + + +def _style_violations(page: pathlib.Path, wiki_root: pathlib.Path) -> int: + """Count pymarkdown violations still on the page. + + Generated by AI/LLM: 2026-09-30 - the count becomes one warning. + """ + try: + doctor = MDoctor(settings_file=wiki_root / "fiona-settings.json") + return len(doctor.scan_file_programmatically(str(page))) + except PyMarkdownApiException: + logger.warning("markdown scan failed for %s", page, exc_info=True) + return 0 + + +def _body_after_frontmatter(text: str) -> str: + """Return the file body after the closing frontmatter marker. + + Generated by AI/LLM: 2026-09-30 - keep the fixer output and replace the header. + """ + if not text.startswith("---"): + return text + parts = text.split("---", 2) + if len(parts) < 3: + return text + return parts[2].removeprefix("\n") + + +def _split_ending(line: str) -> tuple[str, str]: + """Split a line into its ending and its content. + + Generated by AI/LLM: 2026-09-30 - keep ``\\n`` and ``\\r\\n``. + """ + if line.endswith("\r\n"): + return "\r\n", line[:-2] + if line.endswith("\n"): + return "\n", line[:-1] + return "", line + + +def _strip_wiki_prefix(value: str) -> str: + """Drop a leading ``wiki/`` from a ``-t`` value. + + Generated by AI/LLM: 2026-09-30 - both forms are accepted. + """ + relative = value.replace("\\", "/").lstrip("/") + if relative == "wiki": + return "" + if relative.startswith("wiki/"): + return relative[5:] + return relative + + +def _rel(path: pathlib.Path, root: pathlib.Path) -> str: + """Return a path relative to the wiki root using ``/``. + + Generated by AI/LLM: 2026-09-30 - JSON page paths. + """ + return path.resolve().relative_to(root.resolve()).as_posix() diff --git a/src/fiona/wiki/ingest/pipeline.py b/src/fiona/wiki/ingest/pipeline.py new file mode 100644 index 0000000..11f59bf --- /dev/null +++ b/src/fiona/wiki/ingest/pipeline.py @@ -0,0 +1,536 @@ +"""Run the wiki ingest stages in order. + +Generated by AI/LLM: 2026-09-30 - SPEC 7.25. The pipeline holds no format logic. +""" + +from __future__ import annotations + +import argparse +import dataclasses +import json +import logging +import pathlib +import shutil +import sys +import tempfile + +from fiona.librarian.md_frontmatter import FrontmatterError, parse_frontmatter +from fiona.wiki.ingest.archiver import RawArchiver +from fiona.wiki.ingest.converters import default_registry +from fiona.wiki.ingest.converters.base import ConversionError, ConverterRegistry +from fiona.wiki.ingest.fetchers import LocalFileFetcher, SourceFetcher, UrlFetcher, url_scheme +from fiona.wiki.ingest.log_writer import IngestLog +from fiona.wiki.ingest.media import slug_from_file, slug_from_url, utc_day +from fiona.wiki.ingest.models import FetchedSource, IngestError, IngestResult +from fiona.wiki.ingest.page_writer import RawPageWriter +from fiona.wiki.ingest.settings import IngestSettings, load_ingest_settings + +logger = logging.getLogger(__name__) + +_NO_CONVERTER = "No converter for {media}. Convert {raw} by hand." + + +class IngestPipeline: + """Fetch, archive, convert, write, and record one source. + + Generated by AI/LLM: 2026-09-30 - stages are constructor arguments. + """ + + def __init__( + self, + wiki_root: pathlib.Path, + fetchers: list[SourceFetcher], + registry: ConverterRegistry, + archiver: RawArchiver, + page_writer: RawPageWriter, + log_writer: IngestLog, + settings: IngestSettings, + ) -> None: + """Store the stage objects. No converter is named here. + + Args: + wiki_root: Wiki root directory. + fetchers: Tried in order. Each implements ``accepts`` and ``fetch``. + registry: Converter registry. Tests may register another converter. + archiver: Raw archive stage. + page_writer: Wiki page stage. + log_writer: ``wiki/log.md`` stage. + settings: Timeouts, size limit, and default paths. + """ + self.wiki_root = wiki_root + self.fetchers = fetchers + self.registry = registry + self.archiver = archiver + self.page_writer = page_writer + self.log_writer = log_writer + self.settings = settings + + def run( + self, + argument: str, + *, + raw: str | None = None, + target: str | None = None, + raw_only: bool = False, + force: bool = False, + dry_run: bool = False, + ) -> IngestResult: + """Ingest one URL or local file. + + Generated by AI/LLM: 2026-09-30 - exit 1 raises; exit 0 and 3 return a result. + + Args: + argument: Source as given. + raw: ``-r`` path relative to ``raw/``. + target: ``-t`` path relative to ``wiki/``. + raw_only: Archive and record, skip convert and write. + force: Replace an existing page, including a duplicate. + dry_run: Report the run and write nothing under the wiki. + + Returns: + IngestResult: Facts for stdout. ``exit_code`` is 0 or 3. + + Raises: + IngestError: Exit 1. Nothing new remains under the wiki. + """ + scheme = url_scheme(argument) + if scheme is not None and scheme.lower() not in {"http", "https"}: + raise IngestError(f"Unsupported URL scheme: {scheme}") + slug = slug_from_url(argument) if scheme else slug_from_file(pathlib.Path(argument)) + page = self.page_writer.page_path(slug, self.settings, target) + if page.is_file() and not force: + rel = page.resolve().relative_to(self.wiki_root.resolve()).as_posix() + raise IngestError(f"Page exists: {rel}. Use --force to replace it.") + fetcher = self._fetcher(argument) + fetcher.settings = self.settings + temp_dir: pathlib.Path | None = None + fetched: FetchedSource | None = None + archived = False + try: + if dry_run and scheme: + temp_dir = pathlib.Path(tempfile.mkdtemp(prefix="fiona-ingest-")) + dest = temp_dir + else: + dest = self.wiki_root / "ingest" + fetched = fetcher.fetch(argument, dest) + archive = self.archiver.archive( + fetched, + registry=self.registry, + settings=self.settings, + raw_override=raw, + dry_run=dry_run, + ) + archived = True + return self._after_archive( + fetched, + archive, + page, + target_explicit=target is not None, + raw_only=raw_only, + force=force, + dry_run=dry_run, + ) + except IngestError: + if fetched is not None and fetched.created and fetched.path.exists() and not archived: + fetched.path.unlink() + raise + finally: + if temp_dir is not None: + shutil.rmtree(temp_dir, ignore_errors=True) + + def list_pending(self) -> list[dict]: + """List wiki pages whose ``status`` is ``raw``, sorted by path. + + Generated by AI/LLM: 2026-09-30 - SPEC 7.25.9. + + Returns: + list[dict]: ``page_path``, ``sources``, and ``origin`` for each page. + """ + wiki = self.wiki_root / "wiki" + if not wiki.is_dir(): + return [] + pages: list[dict] = [] + for path in sorted(wiki.rglob("*.md"), key=lambda item: item.as_posix()): + if not path.is_file(): + continue + try: + front = parse_frontmatter(path) + except FrontmatterError: + continue + if front.get("status") != "raw": + continue + sources = front.get("sources") or [] + if isinstance(sources, str): + sources = [sources] + ingest = front.get("ingest") if isinstance(front.get("ingest"), dict) else {} + origin = ingest.get("origin") if isinstance(ingest, dict) else None + pages.append({ + "page_path": path.resolve().relative_to(self.wiki_root.resolve()).as_posix(), + "sources": list(sources), + "origin": origin, + }) + return pages + + def _fetcher(self, argument: str) -> SourceFetcher: + """Return the first fetcher that accepts the argument. + + Generated by AI/LLM: 2026-09-30 - URL fetcher is registered before the local one. + """ + for fetcher in self.fetchers: + if fetcher.accepts(argument): + return fetcher + raise IngestError(f"Source not found: {argument}") + + def _after_archive( + self, + source: FetchedSource, + archive: object, + page: pathlib.Path, + *, + target_explicit: bool, + raw_only: bool, + force: bool, + dry_run: bool, + ) -> IngestResult: + """Finish convert, write, and record after a successful archive. + + Generated by AI/LLM: 2026-09-30 - duplicate, raw-only, and missing converter exits. + """ + messages = _fetched_line(source) + if archive.raw_reused: # type: ignore[attr-defined] + short = archive.sha256[:12] # type: ignore[attr-defined] + messages.append( + f"Archived: {archive.raw_rel} (already archived, sha256 {short})" # type: ignore[attr-defined] + ) + else: + messages.append(f"Archived: {archive.raw_rel}") # type: ignore[attr-defined] + duplicate = _citing_page(self.wiki_root, archive.raw_rel, archive.sha256) # type: ignore[attr-defined] + if duplicate and not force: + messages.append(f"Duplicate: {duplicate} already cites this source.") + return _result( + source, + archive, + messages=messages, + duplicate_page=duplicate, + dry_run=dry_run, + warnings=list(archive.warnings), # type: ignore[attr-defined] + ) + if duplicate and force and not target_explicit: + page = self.wiki_root / duplicate + if raw_only: + self.log_writer.append( + day=utc_day(), + origin=source.origin, + raw_rel=archive.raw_rel, # type: ignore[attr-defined] + page_rel=None, + dry_run=dry_run, + ) + return _result(source, archive, messages=messages, dry_run=dry_run, warnings=list(archive.warnings)) # type: ignore[attr-defined] + extension = archive.path.suffix # type: ignore[attr-defined] + converter = self.registry.find(source.media_type, extension) + if converter is None: + messages.append(_NO_CONVERTER.format(media=source.media_type, raw=archive.raw_rel)) # type: ignore[attr-defined] + self._record(source, archive, None, dry_run) + return _result( + source, + archive, + messages=messages, + dry_run=dry_run, + exit_code=3, + warnings=list(archive.warnings), # type: ignore[attr-defined] + ) + read_path = archive.path if archive.path.is_file() else source.path # type: ignore[attr-defined] + try: + conversion = converter.convert(dataclasses.replace(source, path=read_path)) + except ConversionError as exc: + messages.append(_NO_CONVERTER.format(media=source.media_type, raw=archive.raw_rel)) # type: ignore[attr-defined] + self._record(source, archive, None, dry_run) + return _result( + source, + archive, + messages=messages, + dry_run=dry_run, + exit_code=3, + error=f"Conversion failed: {exc.converter}: {exc.reason}", + warnings=list(archive.warnings), # type: ignore[attr-defined] + ) + page_rel, warnings = self.page_writer.write( + source=source, + archive=archive, # type: ignore[arg-type] + conversion=conversion, + page=page, + dry_run=dry_run, + ) + messages.append( + f"Wrote: {page_rel} (converter {conversion.converter}, conversion {conversion.conversion})" + ) + messages.extend(f"Warning: {warning}" for warning in warnings) + messages.append(f"Next: refine {page_rel}, then set status: refined.") + self._record(source, archive, page_rel, dry_run) + return _result( + source, + archive, + messages=messages, + page_path=page_rel, + converter=conversion.converter, + conversion=conversion.conversion, + status="raw", + dry_run=dry_run, + warnings=warnings, + ) + + def _record(self, source: FetchedSource, archive: object, page_rel: str | None, dry_run: bool) -> None: + """Append the log line for a run that archived a source. + + Generated by AI/LLM: 2026-09-30 - duplicates do not call this. + """ + self.log_writer.append( + day=utc_day(), + origin=source.origin, + raw_rel=archive.raw_rel, # type: ignore[attr-defined] + page_rel=page_rel, + dry_run=dry_run, + ) + + +def default_pipeline(wiki_root: pathlib.Path) -> IngestPipeline: + """Build the pipeline with the shipped fetchers and converters. + + Generated by AI/LLM: 2026-09-30 - URL fetcher, then local file fetcher. + + Args: + wiki_root: Wiki root directory. + + Returns: + IngestPipeline: Ready to ``run`` or ``list_pending``. + """ + registry = default_registry() + return IngestPipeline( + wiki_root=wiki_root, + fetchers=[UrlFetcher(registry), LocalFileFetcher(registry)], + registry=registry, + archiver=RawArchiver(wiki_root), + page_writer=RawPageWriter(wiki_root), + log_writer=IngestLog(wiki_root), + settings=load_ingest_settings(wiki_root), + ) + + +def render_stdout(result: IngestResult) -> str: + """Return the human report, including the dry-run banner. + + Generated by AI/LLM: 2026-09-30 - section 4.14 line order. + """ + lines: list[str] = [] + if result.dry_run: + lines.append("Dry run - no files will be modified:") + lines.extend(result.messages) + return "\n".join(lines) + + +def add_ingest_arguments(parser: argparse.ArgumentParser) -> None: + """Add the section 4.14 flags to an ingest parser. + + Generated by AI/LLM: 2026-09-30 - the CLI subparser owns these flags, including --help. + + Args: + parser: The ``wiki ingest`` parser. + """ + parser.add_argument("source", nargs="?", help="http(s) URL or file path") + parser.add_argument( + "--wiki", + default=".", + help="llm-wiki directory (default: .)", + ) + parser.add_argument("-r", "--raw", dest="raw", help="raw path, relative to raw/") + parser.add_argument( + "-t", + "--target", + dest="target", + help="page path, relative to wiki/", + ) + parser.add_argument( + "--raw-only", + action="store_true", + help="archive only; do not write a page", + ) + parser.add_argument( + "-f", + "--force", + action="store_true", + help="replace an existing page", + ) + parser.add_argument( + "--dry-run", + action="store_true", + help="print the plan and write nothing in the wiki", + ) + parser.add_argument("--json", action="store_true", help="print only the JSON result") + parser.add_argument( + "--pending", + action="store_true", + help="list pages whose status is raw", + ) + + +def ingest_main(argv: list[str] | None = None) -> int: + """CLI entry for ``fiona-assistant wiki ingest``. + + Generated by AI/LLM: 2026-09-30 - section 4.14 exit codes. + + Args: + argv: Arguments after the subcommand. ``None`` reads the process arguments. + + Returns: + int: 0, 1, 2, or 3. + """ + parser = argparse.ArgumentParser(prog="fiona-assistant wiki ingest") + add_ingest_arguments(parser) + return execute_ingest(parser.parse_args(argv), parser) + + +def execute_ingest(args: argparse.Namespace, parser: argparse.ArgumentParser) -> int: + """Run one ingest from an already parsed namespace. + + Generated by AI/LLM: 2026-09-30 - shared by ingest_main and the wiki subcommand. + + Args: + args: Parsed source, flags, and paths. + parser: Parser used for the mutual-exclusion error. + + Returns: + int: 0, 1, 2, or 3. + """ + if bool(args.source) == bool(args.pending): + parser.error("provide a source or --pending, not both") + try: + root = _require_wiki(args.wiki) + pipeline = default_pipeline(root) + if args.pending: + return _print_pending(pipeline, args.json) + result = pipeline.run( + args.source, + raw=args.raw, + target=args.target, + raw_only=args.raw_only, + force=args.force, + dry_run=args.dry_run, + ) + except IngestError as exc: + print(f"Error: {exc}", file=sys.stderr) + return exc.exit_code + if result.error: + print(f"Error: {result.error}", file=sys.stderr) + if args.json: + print(json.dumps(result.to_dict(), indent=2)) + else: + text = render_stdout(result) + if text: + print(text) + return result.exit_code + + +def _require_wiki(directory: str) -> pathlib.Path: + """Return the wiki root or raise when settings are missing. + + Generated by AI/LLM: 2026-09-30 - section 4.14 directory error. + """ + root = pathlib.Path(directory) + if not (root / "fiona-settings.json").is_file(): + raise IngestError(f"Directory '{directory}' is not an llm-wiki directory") + return root + + +def _print_pending(pipeline: IngestPipeline, json_output: bool) -> int: + """Print the pending-page report. + + Generated by AI/LLM: 2026-09-30 - human lines or one JSON object. + """ + pages = pipeline.list_pending() + if json_output: + print(json.dumps({"pending": pages, "total": len(pages)}, indent=2)) + return 0 + print(f"Raw pages: {len(pages)}") + for page in pages: + first = page["sources"][0] if page["sources"] else "" + print(f" - {page['page_path']} ({first})") + return 0 + + +def _fetched_line(source: FetchedSource) -> list[str]: + """Return the ``Fetched:`` line for a URL, or an empty list. + + Generated by AI/LLM: 2026-09-30 - local files do not print Fetched. + """ + if source.origin_kind != "url": + return [] + return [f"Fetched: {source.argument} ({source.media_type}, {source.size_bytes} bytes)"] + + +def _citing_page(wiki_root: pathlib.Path, raw_rel: str, digest: str) -> str | None: + """Return a page that already cites this raw path or hash. + + Generated by AI/LLM: 2026-09-30 - SPEC 7.25.3.5. First path in sorted order. + """ + wiki = wiki_root / "wiki" + if not wiki.is_dir(): + return None + matches: list[str] = [] + for path in wiki.rglob("*.md"): + if not path.is_file(): + continue + try: + front = parse_frontmatter(path) + except FrontmatterError: + continue + sources = front.get("sources") or [] + if isinstance(sources, str): + sources = [sources] + ingest = front.get("ingest") if isinstance(front.get("ingest"), dict) else {} + sha = ingest.get("sha256") if isinstance(ingest, dict) else None + if raw_rel in list(sources) or sha == digest: + matches.append(path.resolve().relative_to(wiki_root.resolve()).as_posix()) + if not matches: + return None + matches.sort() + return matches[0] + + +def _result( + source: FetchedSource, + archive: object, + *, + messages: list[str], + dry_run: bool, + warnings: list[str], + duplicate_page: str | None = None, + page_path: str | None = None, + converter: str | None = None, + conversion: str = "none", + status: str | None = None, + exit_code: int = 0, + error: str | None = None, +) -> IngestResult: + """Build an ingest result from the archive and any page that was written. + + Generated by AI/LLM: 2026-09-30 - JSON fields from section 4.14. + """ + return IngestResult( + source=source.argument, + origin=source.origin, + origin_kind=source.origin_kind, + media_type=source.media_type, + raw_path=archive.raw_rel, # type: ignore[attr-defined] + raw_reused=archive.raw_reused, # type: ignore[attr-defined] + sha256=archive.sha256, # type: ignore[attr-defined] + duplicate_page=duplicate_page, + page_path=page_path, + converter=converter, + conversion=conversion, + status=status, + warnings=warnings, + dry_run=dry_run, + exit_code=exit_code, + error=error, + messages=messages, + ) diff --git a/src/fiona/wiki/ingest/settings.py b/src/fiona/wiki/ingest/settings.py new file mode 100644 index 0000000..7e7538e --- /dev/null +++ b/src/fiona/wiki/ingest/settings.py @@ -0,0 +1,107 @@ +"""Ingest settings from ``fiona-settings.json``. + +Generated by AI/LLM: 2026-09-30 - SPEC 5.1 ``fiona.wiki.ingest`` keys. +""" + +from __future__ import annotations + +import dataclasses +import importlib.metadata +import json +import pathlib + + +def package_version() -> str: + """Return the installed fiona-assistant version. + + Generated by AI/LLM: 2026-09-30 - user agent and ``ingest.tool``. + + Returns: + str: Distribution version, or ``0.99.0`` when metadata is absent. + """ + try: + return importlib.metadata.version("fiona-assistant") + except importlib.metadata.PackageNotFoundError: + return "0.99.0" + + +@dataclasses.dataclass +class IngestSettings: + """Runtime limits and paths for one ingest. + + Generated by AI/LLM: 2026-09-30 - defaults from the settings table. + """ + + timeout_seconds: float + max_download_bytes: int + user_agent: str + default_target: str + raw_categories: dict[str, str] + + +def default_settings() -> IngestSettings: + """Return the settings used when the wiki omits ``fiona.wiki.ingest``. + + Generated by AI/LLM: 2026-09-30 - table defaults from SPEC 5.1. + + Returns: + IngestSettings: Built-in limits and paths. + """ + return IngestSettings( + timeout_seconds=30, + max_download_bytes=52428800, + user_agent=f"fiona-assistant/{package_version()}", + default_target="sources", + raw_categories={}, + ) + + +def load_ingest_settings(wiki_root: pathlib.Path) -> IngestSettings: + """Load ingest settings, falling back per key when a value is unusable. + + Generated by AI/LLM: 2026-09-30 - absent or mistyped keys keep the default. + + Args: + wiki_root: Directory that contains ``fiona-settings.json``. + + Returns: + IngestSettings: Merged settings for this wiki. + """ + settings = default_settings() + path = wiki_root / "fiona-settings.json" + if not path.is_file(): + return settings + try: + document = json.loads(path.read_text(encoding="utf-8")) + except json.JSONDecodeError: + return settings + section = {} + if isinstance(document, dict): + wiki = document.get("fiona", {}) + if isinstance(wiki, dict): + wiki = wiki.get("wiki", {}) + if isinstance(wiki, dict): + raw = wiki.get("ingest", {}) + if isinstance(raw, dict): + section = raw + timeout = section.get("timeout_seconds", settings.timeout_seconds) + if isinstance(timeout, (int, float)) and not isinstance(timeout, bool) and timeout > 0: + settings.timeout_seconds = float(timeout) + limit = section.get("max_download_bytes", settings.max_download_bytes) + if isinstance(limit, int) and not isinstance(limit, bool) and limit > 0: + settings.max_download_bytes = limit + agent = section.get("user_agent") + if isinstance(agent, str) and agent.strip(): + settings.user_agent = agent + target = section.get("default_target") + if isinstance(target, str) and target.strip() and ".." not in target.replace("\\", "/"): + settings.default_target = target.strip().strip("/") + categories = section.get("raw_categories") + if isinstance(categories, dict): + cleaned = { + str(key): value + for key, value in categories.items() + if isinstance(value, str) and value.strip() + } + settings.raw_categories = cleaned + return settings diff --git a/src/fiona/wiki/sanity_manager.py b/src/fiona/wiki/sanity_manager.py index 1d07476..af98c5d 100644 --- a/src/fiona/wiki/sanity_manager.py +++ b/src/fiona/wiki/sanity_manager.py @@ -51,6 +51,29 @@ __all__ = [ # Check Runners # ============================================================================= +def _outside_raw(wiki_root: Path, path: str) -> bool: + """Return whether a reported path is outside the raw archive. + + Generated by AI/LLM: 2026-09-30 - SPEC 7.25.12. Sanity checks skip ``raw/``. + + Args: + wiki_root: Wiki root directory. + path: Violation path, absolute or relative to the wiki. + + Returns: + bool: True when the path is not under ``raw/``. + """ + text = path.replace("\\", "/") + if text == "raw" or text.startswith("raw/"): + return False + candidate = Path(path) + try: + relative = candidate.resolve().relative_to(wiki_root.resolve()) + except (ValueError, OSError): + return True + return not relative.parts or relative.parts[0] != "raw" + + def run_markdown_lint(wiki_root: Path) -> dict: """Run markdownlint (mdoctor) over the wiki. @@ -62,7 +85,10 @@ def run_markdown_lint(wiki_root: Path) -> dict: the raw mdoctor violations. """ mdoctor = MDoctor(settings_file=wiki_root / SETTINGS_FILE) - violations = mdoctor.scan_file_programmatically(str(wiki_root)) + violations = [ + item for item in mdoctor.scan_file_programmatically(str(wiki_root)) + if _outside_raw(wiki_root, str(item.get("scan_file", ""))) + ] report = mdoctor.format_violations( mdoctor.compress_violations(violations, target=str(wiki_root)) ) @@ -85,7 +111,10 @@ def run_python_lint(wiki_root: Path) -> dict: the raw ruff violations. """ ruff_linter = PythonRuff() - violations = ruff_linter.lint(str(wiki_root)) + violations = [ + item for item in ruff_linter.lint(str(wiki_root)) + if _outside_raw(wiki_root, str(item.get("filename", ""))) + ] report = ruff_linter.format_violations(violations, str(wiki_root)) return { "result": 0 if not violations else 1, diff --git a/tests/test_fiona_main.py b/tests/test_fiona_main.py index 93c1065..481c55f 100644 --- a/tests/test_fiona_main.py +++ b/tests/test_fiona_main.py @@ -94,6 +94,23 @@ class TestFionaMain: captured = capsys.readouterr() assert "usage" in captured.out.lower() + def test_main_wiki_ingest_help(self, capsys): + """Test wiki ingest --help is printed by the ingest parser. + + Generated by AI/LLM: 2026-09-30 - SPEC 4.14 help stays on this command. + """ + with ( + patch("sys.argv", ["fiona-assistant", "wiki", "ingest", "--help"]), + pytest.raises(SystemExit) as exc_info, + ): + main() + + assert exc_info.value.code == 0 + captured = capsys.readouterr() + assert "fiona-assistant wiki ingest" in captured.out + assert "--pending" in captured.out + assert "--raw-only" in captured.out + def test_main_mdoctor_default(self, capsys): """Test main() with mdoctor command and default arguments.""" with tempfile.TemporaryDirectory() as temp_dir: @@ -440,7 +457,7 @@ class TestFionaMain: # Create test markdown files with valid frontmatter with open(os.path.join(temp_dir, "test1.md"), 'w') as f: f.write("---\nname: Test1\nversion: 1.0.0\n---\n# Test1\n") - + with open(os.path.join(temp_dir, "test2.md"), 'w') as f: f.write("---\nname: Test2\nversion: 2.0.0\n---\n# Test2\n") diff --git a/tests/test_ingest.py b/tests/test_ingest.py new file mode 100644 index 0000000..b7ad2cc --- /dev/null +++ b/tests/test_ingest.py @@ -0,0 +1,729 @@ +"""Tests for wiki ingest (SPEC 7.25). + +Generated by AI/LLM: 2026-09-30 - acceptance checks for fetch, archive, convert, and write. +""" + +from __future__ import annotations + +import contextlib +import email.message +import hashlib +import io +import json +import pathlib +import shutil +import typing +import urllib.error +import urllib.request + +import pypdf +import pypdf.generic +import pytest +import yaml + +from fiona.customlinter.fmlint import FrontmatterLinter +from fiona.customlinter.mdlinter import MarkdownLinter +from fiona.paths import llm_wiki_template +from fiona.wiki.ingest.converters.base import SourceConverter +from fiona.wiki.ingest.fetchers import UrlFetcher +from fiona.wiki.ingest.media import slug_from_url +from fiona.wiki.ingest.models import ConversionResult, FetchedSource, IngestError +from fiona.wiki.ingest.pipeline import default_pipeline, ingest_main +from fiona.wiki.sanity_manager import run_wiki_checks + +_FRONTMATTER_KEYS = ( + "name", + "version", + "title", + "created", + "updated", + "type", + "status", + "sources", + "source_meta", + "ingest", + "tags", + "confidence", + "llm", + "contested", + "contradictions", +) +_INGEST_KEYS = ( + "origin", + "origin_kind", + "final_url", + "fetched", + "media_type", + "size_bytes", + "sha256", + "raw_reused", + "converter", + "conversion", + "warnings", + "tool", +) + + +class _FakeResponse: + """Minimal urllib response for one canned body.""" + + def __init__(self, body: bytes, header: str, url: str) -> None: + self._body = body + self._offset = 0 + self.headers = email.message.Message() + self.headers["Content-Type"] = header + self._url = url + self.status = 200 + + def read(self, amount: int = -1) -> bytes: + """Return the next chunk of the canned body.""" + if amount is None or amount < 0: + amount = len(self._body) - self._offset + chunk = self._body[self._offset:self._offset + amount] + self._offset += len(chunk) + return chunk + + def geturl(self) -> str: + """Return the URL after redirects.""" + return self._url + + def __enter__(self) -> typing.Self: + return self + + def __exit__(self, *_args: object) -> bool: + return False + + +class _FakeOpener: + """URL opener that never touches the network.""" + + def __init__(self, body: bytes, header: str, *, status: int = 200, final_url: str | None = None) -> None: + self.body = body + self.header = header + self.status = status + self.final_url = final_url + self.requests: list[urllib.request.Request] = [] + self.timeout: float | None = None + self.calls = 0 + + def open(self, request: urllib.request.Request, timeout: float | None = None) -> _FakeResponse: + """Record the request and return the canned response or a 404.""" + self.calls += 1 + self.requests.append(request) + self.timeout = timeout + if self.status != 200: + raise urllib.error.HTTPError( + request.full_url, + self.status, + "Not Found", + email.message.Message(), + io.BytesIO(b""), + ) + final = self.final_url or request.full_url + return _FakeResponse(self.body, self.header, final) + + +class _XTestConverter(SourceConverter): + """Converter registered only by the test, for ``application/x-test``.""" + + name = "xtest" + media_types = ("application/x-test",) + extensions = (".xtest",) + raw_category = "articles" + + def convert(self, source: FetchedSource) -> ConversionResult: + """Return a fixed body so the test can see the registration.""" + return ConversionResult( + body="xtest body\n", + title="X Test", + source_meta={}, + conversion="complete", + warnings=[], + converter=self.name, + ) + + +def _wiki(tmp_path: pathlib.Path) -> pathlib.Path: + """Create a small llm-wiki with settings, a log, and an index.""" + root = tmp_path / "wiki" + (root / "wiki").mkdir(parents=True) + (root / "ingest").mkdir() + (root / "raw").mkdir() + (root / "personal").mkdir() + shutil.copy2(llm_wiki_template() / "fiona-settings.json", root / "fiona-settings.json") + (root / "wiki" / "log.md").write_text( + "---\n" + "name: log\n" + "title: Wiki Log\n" + "version: 2.0.0\n" + "type: summary\n" + "created: 2026-09-30\n" + "updated: 2026-09-30\n" + "confidence: trusted\n" + "llm:\n" + " - model: none\n" + "contested: false\n" + "contradictions: []\n" + "---\n\n" + "# Wiki Log\n\n" + "- 2026-09-30 - Initialized the llm-wiki.\n", + encoding="utf-8", + newline="\n", + ) + (root / "wiki" / "index.md").write_text( + "---\nname: index\nversion: 1.0.0\n---\n\n# Index\n", + encoding="utf-8", + newline="\n", + ) + return root + + +def _run(root: pathlib.Path, *args: str) -> tuple[int, str, str]: + """Run the ingest CLI against ``root`` and capture both streams.""" + stdout = io.StringIO() + stderr = io.StringIO() + with contextlib.redirect_stdout(stdout), contextlib.redirect_stderr(stderr): + code = ingest_main(["--wiki", str(root), *args]) + return code, stdout.getvalue(), stderr.getvalue() + + +def _front(path: pathlib.Path) -> dict: + """Parse page frontmatter with types left intact.""" + text = path.read_text(encoding="utf-8") + return yaml.safe_load(text.split("---", 2)[1]) + + +def _pdf(texts: list[str], *, title: str | None = None, password: str | None = None) -> bytes: + """Build a PDF whose pages contain the given strings.""" + writer = pypdf.PdfWriter() + for text in texts: + page = writer.add_blank_page(width=612, height=792) + if text: + _pdf_text(writer, page, text) + if title: + writer.add_metadata({"/Title": title}) + if password: + writer.encrypt(password) + buffer = io.BytesIO() + writer.write(buffer) + return buffer.getvalue() + + +def _pdf_text(writer: pypdf.PdfWriter, page: pypdf.PageObject, text: str) -> None: + """Draw one Helvetica string on a page.""" + font = pypdf.generic.DictionaryObject({ + pypdf.generic.NameObject("/Type"): pypdf.generic.NameObject("/Font"), + pypdf.generic.NameObject("/Subtype"): pypdf.generic.NameObject("/Type1"), + pypdf.generic.NameObject("/BaseFont"): pypdf.generic.NameObject("/Helvetica"), + }) + font_ref = writer._add_object(font) + fonts = pypdf.generic.DictionaryObject({pypdf.generic.NameObject("/F1"): font_ref}) + page[pypdf.generic.NameObject("/Resources")] = pypdf.generic.DictionaryObject({ + pypdf.generic.NameObject("/Font"): fonts, + }) + stream = pypdf.generic.DecodedStreamObject() + stream.set_data(f"BT /F1 12 Tf 72 100 Td ({text}) Tj ET".encode("latin-1")) + page[pypdf.generic.NameObject("/Contents")] = writer._add_object(stream) + + +def _hashes(root: pathlib.Path) -> dict[str, str]: + """Hash every file under a wiki.""" + found: dict[str, str] = {} + for path in root.rglob("*"): + if path.is_file(): + found[path.relative_to(root).as_posix()] = hashlib.sha256(path.read_bytes()).hexdigest() + return found + + +def test_slug_from_console_url() -> None: + """SPEC 7.25.2.6: the last three segments become one hyphenated slug.""" + assert slug_from_url("https://host/job/performance_TL153/2108/console") == ( + "performance-tl153-2108-console" + ) + + +def test_markdown_gets_frontmatter_and_keeps_body(tmp_path: pathlib.Path) -> None: + """A markdown file keeps its wording. Missing page frontmatter is added.""" + root = _wiki(tmp_path) + source = root / "ingest" / "Notes.md" + source.write_text("# Kept\n\nThe body stays.\n", encoding="utf-8", newline="\n") + code, _stdout, stderr = _run(root, str(source)) + assert code == 0, stderr + assert not source.exists() + raw = root / "raw" / "articles" / "notes.md" + assert raw.read_text(encoding="utf-8") == "# Kept\n\nThe body stays.\n" + page = root / "wiki" / "sources" / "notes.md" + front = _front(page) + assert front["status"] == "raw" + assert front["source_meta"] == {} + assert "The body stays." in page.read_text(encoding="utf-8") + assert front["title"] == "Kept" + text = page.read_text(encoding="utf-8") + assert [line for line in text.splitlines() if line.startswith("# ")] == ["# Kept"] + + +def test_markdown_frontmatter_moves_to_source_meta(tmp_path: pathlib.Path) -> None: + """Source frontmatter is metadata. The page still receives ingest frontmatter.""" + root = _wiki(tmp_path) + source = tmp_path / "ready.md" + source.write_text( + "---\ntitle: Real Title\nauthor: Ada\n---\n\nParagraph.\n", + encoding="utf-8", + newline="\n", + ) + code, _stdout, stderr = _run(root, str(source)) + assert code == 0, stderr + page = root / "wiki" / "sources" / "ready.md" + front = _front(page) + assert front["title"] == "Real Title" + assert front["source_meta"]["author"] == "Ada" + assert "Paragraph." in page.read_text(encoding="utf-8") + assert source.read_text(encoding="utf-8").startswith("---\n") + + +def test_text_becomes_markdown_for_later_refinement(tmp_path: pathlib.Path) -> None: + """A text file becomes a fenced page at status raw. The tool does not summarize it.""" + root = _wiki(tmp_path) + source = tmp_path / "trace.log" + source.write_text("alpha\n```\nbeta\n", encoding="utf-8", newline="\n") + code, stdout, stderr = _run(root, str(source)) + assert code == 0, stderr + page = root / "wiki" / "sources" / "trace.md" + front = _front(page) + assert front["status"] == "raw" + assert front["ingest"]["converter"] == "text" + assert front["ingest"]["conversion"] == "complete" + body = page.read_text(encoding="utf-8") + assert "````text" in body + assert "alpha" in body + assert "Not refined yet." in body + assert "Next: refine" in stdout + code, pending, _stderr = _run(root, "--pending") + assert code == 0 + assert "wiki/sources/trace.md" in pending + + +def test_text_fallback_decode_warns(tmp_path: pathlib.Path) -> None: + """Bytes that are not UTF-8 are decoded with a warning.""" + root = _wiki(tmp_path) + source = tmp_path / "legacy.txt" + source.write_bytes(b"caf\x80") + code, _stdout, stderr = _run(root, str(source)) + assert code == 0, stderr + front = _front(root / "wiki" / "sources" / "legacy.md") + assert any("cp1252" in warning for warning in front["ingest"]["warnings"]) + + +def test_register_converter_without_pipeline_edits(tmp_path: pathlib.Path) -> None: + """SPEC 7.25.1.2: a new format is one class and one registration.""" + root = _wiki(tmp_path) + sample = tmp_path / "sample.xtest" + sample.write_bytes(b"hello") + pipeline = default_pipeline(root) + pipeline.registry.register(_XTestConverter) + result = pipeline.run(str(sample)) + assert result.exit_code == 0 + assert result.converter == "xtest" + assert (root / "raw" / "articles" / "sample.xtest").is_file() + assert "xtest body" in (root / "wiki" / "sources" / "sample.md").read_text(encoding="utf-8") + + +def test_ftp_scheme_writes_nothing(tmp_path: pathlib.Path) -> None: + """SPEC 7.25.2.1: an unsupported scheme exits 1 and writes nothing.""" + root = _wiki(tmp_path) + before = _hashes(root) + code, stdout, stderr = _run(root, "ftp://host/a.pdf") + assert code == 1 + assert stdout == "" + assert "Unsupported URL scheme: ftp" in stderr + assert _hashes(root) == before + + +def test_missing_file_and_directory(tmp_path: pathlib.Path) -> None: + """A missing path and a directory each exit 1.""" + root = _wiki(tmp_path) + code, stdout, stderr = _run(root, str(tmp_path / "missing.txt")) + assert code == 1 + assert stdout == "" + assert "Source not found:" in stderr + code, stdout, stderr = _run(root, str(tmp_path)) + assert code == 1 + assert stdout == "" + assert "Source is a directory:" in stderr + + +def test_download_uses_content_type_extension(tmp_path: pathlib.Path) -> None: + """SPEC 7.25.2.5: a PDF response with no suffix is saved as ``.pdf``.""" + root = _wiki(tmp_path) + body = _pdf(["Hello"]) + opener = _FakeOpener(body, "application/pdf", final_url="https://host/report") + pipeline = default_pipeline(root) + pipeline.fetchers[0] = UrlFetcher(pipeline.registry, opener=opener) + result = pipeline.run("https://host/report") + assert result.exit_code == 0 + assert result.raw_path == "raw/papers/report.pdf" + assert (root / "raw" / "papers" / "report.pdf").read_bytes() == body + assert not (root / "ingest" / "report.pdf").exists() + assert opener.requests[0].get_header("Authorization") is None + assert "fiona-assistant/" in opener.requests[0].get_header("User-agent") + + +def test_download_404_leaves_ingest_unchanged(tmp_path: pathlib.Path) -> None: + """SPEC 7.25.2.3: a 404 exits 1 and does not add files.""" + root = _wiki(tmp_path) + (root / "ingest" / "keep.txt").write_text("stay", encoding="utf-8") + before = sorted(path.name for path in (root / "ingest").iterdir()) + opener = _FakeOpener(b"", "text/plain", status=404) + pipeline = default_pipeline(root) + pipeline.fetchers[0] = UrlFetcher(pipeline.registry, opener=opener) + with pytest.raises(IngestError, match="404"): + pipeline.run("https://host/missing") + assert sorted(path.name for path in (root / "ingest").iterdir()) == before + + +def test_download_over_limit_is_deleted(tmp_path: pathlib.Path) -> None: + """A body over the limit exits 1 and leaves no partial file.""" + root = _wiki(tmp_path) + pipeline = default_pipeline(root) + pipeline.settings.max_download_bytes = 4 + opener = _FakeOpener(b"0123456789", "text/plain") + pipeline.fetchers[0] = UrlFetcher(pipeline.registry, opener=opener) + with pytest.raises(IngestError, match="exceeds 4 bytes"): + pipeline.run("https://host/big") + assert list((root / "ingest").iterdir()) == [] + + +def test_extensionless_pdf_uses_pdf_converter(tmp_path: pathlib.Path) -> None: + """SPEC 7.25.2.4: leading ``%PDF-`` bytes select the pdf converter.""" + root = _wiki(tmp_path) + source = tmp_path / "notes" + source.write_bytes(_pdf(["Page text"])) + code, _stdout, stderr = _run(root, str(source)) + assert code == 0, stderr + page = root / "wiki" / "sources" / "notes.md" + assert _front(page)["ingest"]["converter"] == "pdf" + assert (root / "raw" / "papers" / "notes.pdf").is_file() + + +def test_raw_override_and_escape(tmp_path: pathlib.Path) -> None: + """SPEC 7.25.3.2: ``-r`` places the file, and ``..`` exits 1.""" + root = _wiki(tmp_path) + html = tmp_path / "page.html" + html.write_text("<html><body><article><p>Hi</p></article></body></html>", encoding="utf-8") + code, _stdout, stderr = _run(root, str(html), "-r", "jenkins/tl153") + assert code == 0, stderr + assert (root / "raw" / "jenkins" / "tl153" / "page.html").is_file() + other = tmp_path / "other.html" + other.write_text("<html><body><p>Other</p></body></html>", encoding="utf-8") + before = _hashes(root) + code, stdout, stderr = _run(root, str(other), "-r", "../wiki/x") + assert code == 1 + assert stdout == "" + assert "--raw must stay inside raw/" in stderr + assert _hashes(root) == before + + +def test_reuses_identical_bytes(tmp_path: pathlib.Path) -> None: + """SPEC 7.25.3.3: a second name and ``-r`` reuse the first raw copy.""" + root = _wiki(tmp_path) + first = tmp_path / "one.txt" + first.write_text("same bytes\n", encoding="utf-8", newline="\n") + code, _stdout, stderr = _run(root, str(first)) + assert code == 0, stderr + second = tmp_path / "two.txt" + second.write_text("same bytes\n", encoding="utf-8", newline="\n") + pipeline = default_pipeline(root) + result = pipeline.run(str(second), raw="other/dir") + assert result.raw_reused is True + assert result.raw_path == "raw/articles/one.txt" + assert result.duplicate_page == "wiki/sources/one.md" + copies = [path for path in (root / "raw").rglob("*") if path.is_file() and path.suffix == ".txt"] + assert copies == [root / "raw" / "articles" / "one.txt"] + assert "already archived at raw/articles/one.txt; --raw ignored" in result.warnings + + +def test_changed_file_gets_a_second_raw_name(tmp_path: pathlib.Path) -> None: + """SPEC 7.25.3.4: a changed file of the same name is stored as ``slug-2``.""" + root = _wiki(tmp_path) + source = tmp_path / "notes.txt" + source.write_text("first\n", encoding="utf-8", newline="\n") + code, _stdout, stderr = _run(root, str(source), "--raw-only") + assert code == 0, stderr + original = (root / "raw" / "articles" / "notes.txt").read_bytes() + source.write_text("second\n", encoding="utf-8", newline="\n") + code, _stdout, stderr = _run(root, str(source)) + assert code == 0, stderr + assert (root / "raw" / "articles" / "notes.txt").read_bytes() == original + assert (root / "raw" / "articles" / "notes-2.txt").read_text(encoding="utf-8") == "second\n" + + +def test_duplicate_exits_zero_and_skips_the_log(tmp_path: pathlib.Path) -> None: + """SPEC 7.25.3.5: the same bytes report the first page and do not rewrite it.""" + root = _wiki(tmp_path) + source = root / "ingest" / "a.txt" + source.write_text("alpha\n", encoding="utf-8", newline="\n") + code, _stdout, stderr = _run(root, str(source)) + assert code == 0, stderr + page = root / "wiki" / "sources" / "a.md" + page_bytes = page.read_bytes() + log_before = (root / "wiki" / "log.md").read_text(encoding="utf-8") + again = root / "ingest" / "b.txt" + again.write_text("alpha\n", encoding="utf-8", newline="\n") + code, stdout, stderr = _run(root, str(again), "--json") + assert code == 0, stderr + payload = json.loads(stdout) + assert payload["duplicate_page"] == "wiki/sources/a.md" + assert payload["page_path"] is None + assert not again.exists() + assert page.read_bytes() == page_bytes + assert (root / "wiki" / "log.md").read_text(encoding="utf-8") == log_before + + +def test_ingest_drop_moves_and_personal_is_copied(tmp_path: pathlib.Path) -> None: + """SPEC 7.25.3.6: ``ingest/`` is moved. A file outside it is copied.""" + root = _wiki(tmp_path) + dropped = root / "ingest" / "a.pdf" + dropped.write_bytes(_pdf(["Moved"])) + code, _stdout, stderr = _run(root, str(dropped)) + assert code == 0, stderr + assert not dropped.exists() + archived = root / "raw" / "papers" / "a.pdf" + assert archived.read_bytes() == _pdf(["Moved"]) + personal = root / "personal" / "note.txt" + personal.write_text("keep me\n", encoding="utf-8", newline="\n") + original = personal.read_bytes() + code, _stdout, stderr = _run(root, str(personal)) + assert code == 0, stderr + assert personal.read_bytes() == original + assert (root / "raw" / "articles" / "note.txt").read_bytes() == original + + +def test_unknown_bytes_archive_without_a_page(tmp_path: pathlib.Path) -> None: + """SPEC 7.25.4.1: random bytes exit 3 under ``raw/other``.""" + root = _wiki(tmp_path) + source = tmp_path / "blob.xyz" + source.write_bytes(b"\x00\x01\x02\xff") + code, stdout, _stderr = _run(root, str(source), "--json") + assert code == 3 + payload = json.loads(stdout) + assert payload["conversion"] == "none" + assert payload["page_path"] is None + assert payload["raw_path"] == "raw/other/blob.xyz" + assert (root / "raw" / "other" / "blob.xyz").is_file() + assert not (root / "wiki" / "sources" / "blob.md").exists() + human_source = tmp_path / "other.xyz" + human_source.write_bytes(b"\x00\x02") + code, human, _stderr = _run(root, str(human_source)) + assert code == 3 + assert human.strip().splitlines()[-1].startswith("No converter for") + + +def test_encrypted_pdf_keeps_raw_and_skips_the_page(tmp_path: pathlib.Path) -> None: + """SPEC 7.25.4.2: an encrypted PDF exits 3 and the raw copy stays.""" + root = _wiki(tmp_path) + source = tmp_path / "secret.pdf" + source.write_bytes(_pdf(["hidden"], password="secret")) + code, stdout, stderr = _run(root, str(source)) + assert code == 3 + assert "Conversion failed: pdf:" in stderr + assert (root / "raw" / "papers" / "secret.pdf").is_file() + assert not (root / "wiki" / "sources" / "secret.md").exists() + assert stdout.strip().splitlines()[-1].startswith("No converter for") + + +def test_html_keeps_article_link_and_table(tmp_path: pathlib.Path) -> None: + """SPEC 7.25.4.5: article text, no script, an absolute link, and a pipe table.""" + root = _wiki(tmp_path) + html = """<!doctype html> +<html lang="en"><head><title>Page Title + + +
+

ARTICLETEX

+Go +
Col
Cell
+
+ + +""" + opener = _FakeOpener(html.encode("utf-8"), "text/html") + pipeline = default_pipeline(root) + pipeline.fetchers[0] = UrlFetcher(pipeline.registry, opener=opener) + result = pipeline.run("https://host/p") + assert result.exit_code == 0 + body = (root / result.page_path).read_text(encoding="utf-8") + assert "ARTICLETEX" in body + assert "SCRIPTSECRET" not in body + assert "https://host/a" in body + assert "|" in body + + +def test_pdf_page_headings_and_blank_page(tmp_path: pathlib.Path) -> None: + """SPEC 7.25.4.6: pages are numbered, and a blank page is partial.""" + root = _wiki(tmp_path) + numbered = tmp_path / "two.pdf" + numbered.write_bytes(_pdf(["Alpha", "Beta"])) + code, _stdout, stderr = _run(root, str(numbered)) + assert code == 0, stderr + body = (root / "wiki" / "sources" / "two.md").read_text(encoding="utf-8") + assert "## Page 1" in body + assert "## Page 2" in body + blank = tmp_path / "gap.pdf" + blank.write_bytes(_pdf(["Alpha", ""])) + code, _stdout, stderr = _run(root, str(blank)) + assert code == 0, stderr + front = _front(root / "wiki" / "sources" / "gap.md") + assert front["ingest"]["conversion"] == "partial" + assert "page 2 has no text layer" in front["ingest"]["warnings"] + + +def test_target_paths(tmp_path: pathlib.Path) -> None: + """SPEC 7.25.5.1: ``-t concepts`` writes under concepts. Escapes exit 1.""" + root = _wiki(tmp_path) + source = tmp_path / "idea.txt" + source.write_text("idea\n", encoding="utf-8", newline="\n") + code, _stdout, stderr = _run(root, str(source), "-t", "concepts") + assert code == 0, stderr + assert (root / "wiki" / "concepts" / "idea.md").is_file() + code, stdout, stderr = _run(root, str(source), "-t", "../raw/x.md") + assert code == 1 + assert stdout == "" + assert "--target must stay inside wiki/" in stderr + other = tmp_path / "other.txt" + other.write_text("x\n", encoding="utf-8", newline="\n") + code, stdout, stderr = _run(root, str(other), "-t", "index.md") + assert code == 1 + assert "--target cannot replace" in stderr + + +def test_existing_page_refuses_until_force(tmp_path: pathlib.Path) -> None: + """SPEC 7.25.5.2: a second run exits 1. ``--force`` replaces the page.""" + root = _wiki(tmp_path) + source = tmp_path / "item.txt" + source.write_text("one\n", encoding="utf-8", newline="\n") + code, _stdout, stderr = _run(root, str(source)) + assert code == 0, stderr + page = root / "wiki" / "sources" / "item.md" + before = page.read_bytes() + opener = _FakeOpener(b"nope", "text/plain") + pipeline = default_pipeline(root) + pipeline.fetchers[0] = UrlFetcher(pipeline.registry, opener=opener) + with pytest.raises(IngestError, match="Page exists:"): + pipeline.run("https://host/item") + assert opener.calls == 0 + assert page.read_bytes() == before + source.write_text("two\n", encoding="utf-8", newline="\n") + code, _stdout, stderr = _run(root, str(source), "--force") + assert code == 0, stderr + assert page.read_bytes() != before + assert "two" in (root / "raw" / "articles" / "item-2.txt").read_text(encoding="utf-8") + + +def test_page_frontmatter_matches_raw_hash(tmp_path: pathlib.Path) -> None: + """SPEC 7.25.5.3 and 7.25.5.4: frontmatter is complete and the page has one H1.""" + root = _wiki(tmp_path) + index_before = (root / "wiki" / "index.md").read_bytes() + source = tmp_path / "story.md" + source.write_text("# Story\n\nSee [gone](missing.md).\n", encoding="utf-8", newline="\n") + pipeline = default_pipeline(root) + result = pipeline.run(str(source)) + assert result.exit_code == 0 + page = root / result.page_path + front = _front(page) + for key in _FRONTMATTER_KEYS: + assert key in front + for key in _INGEST_KEYS: + assert key in front["ingest"] + raw = root / front["sources"][0] + assert front["ingest"]["sha256"] == hashlib.sha256(raw.read_bytes()).hexdigest() + assert front["ingest"]["warnings"] == result.warnings + assert front["status"] == "raw" + assert front["confidence"] == "zero" + assert front["llm"] == [{"model": "none"}] + headings = [line for line in page.read_text(encoding="utf-8").splitlines() if line.startswith("# ")] + assert headings == ["# Story"] + linter = FrontmatterLinter(settings_file=root / "fiona-settings.json") + assert linter.lint_file(page) == [] + links = MarkdownLinter(settings_file=root / "fiona-settings.json") + assert [item for item in links.lint_file(page) if item.code == "MDL001"] == [] + assert (root / "wiki" / "index.md").read_bytes() == index_before + log = (root / "wiki" / "log.md").read_text(encoding="utf-8") + assert "version: 2.0.1" in log + assert f"ingest {source.resolve()} -> raw/articles/story.md -> wiki/sources/story.md (raw)" in log + + +def test_style_violations_become_one_warning(tmp_path: pathlib.Path) -> None: + """SPEC 7.25.5.5: violations the fixer leaves are counted in one warning.""" + root = _wiki(tmp_path) + source = tmp_path / "long.md" + source.write_text("# Title\n\n" + ("word " * 40) + "\n", encoding="utf-8", newline="\n") + code, _stdout, stderr = _run(root, str(source)) + assert code == 0, stderr + warnings = _front(root / "wiki" / "sources" / "long.md")["ingest"]["warnings"] + assert any(warning.endswith("markdown style violations left") for warning in warnings) + + +def test_raw_only_and_dry_run(tmp_path: pathlib.Path) -> None: + """SPEC 7.25.7 and 7.25.8: raw-only skips the page. A dry run writes nothing.""" + root = _wiki(tmp_path) + source = root / "ingest" / "only.txt" + source.write_text("archive me\n", encoding="utf-8", newline="\n") + code, stdout, stderr = _run(root, str(source), "--raw-only", "--json") + assert code == 0, stderr + payload = json.loads(stdout) + assert payload["conversion"] == "none" + assert payload["page_path"] is None + assert not source.exists() + assert (root / "raw" / "articles" / "only.txt").is_file() + kept = root / "ingest" / "stay.txt" + kept.write_text("stay\n", encoding="utf-8", newline="\n") + before = _hashes(root) + code, stdout, stderr = _run(root, str(kept), "--dry-run") + assert code == 0, stderr + assert stdout.startswith("Dry run - no files will be modified:") + assert kept.is_file() + assert _hashes(root) == before + code, stdout, _stderr = _run(root, str(kept), "--dry-run", "--json") + assert json.loads(stdout)["dry_run"] is True + + +def test_pending_hides_refined_pages(tmp_path: pathlib.Path) -> None: + """SPEC 7.25.9: ``--pending`` lists raw pages and drops refined ones.""" + root = _wiki(tmp_path) + source = tmp_path / "open.txt" + source.write_text("open\n", encoding="utf-8", newline="\n") + assert _run(root, str(source))[0] == 0 + code, stdout, _stderr = _run(root, "--pending", "--json") + payload = json.loads(stdout) + assert code == 0 + assert payload["total"] == 1 + page = root / "wiki" / "sources" / "open.md" + text = page.read_text(encoding="utf-8").replace("status: raw", "status: refined", 1) + page.write_text(text, encoding="utf-8", newline="\n") + code, stdout, _stderr = _run(root, "--pending", "--json") + assert json.loads(stdout)["pending"] == [] + + +def test_not_a_wiki_and_mutually_exclusive_args(tmp_path: pathlib.Path) -> None: + """A directory without settings exits 1. Source and ``--pending`` cannot combine.""" + code, stdout, stderr = _run(tmp_path, str(tmp_path / "a.txt")) + assert code == 1 + assert stdout == "" + assert "is not an llm-wiki directory" in stderr + with pytest.raises(SystemExit) as caught: + ingest_main(["--pending", "file.txt"]) + assert caught.value.code == 2 + + +def test_sanity_ignores_raw_markdown(tmp_path: pathlib.Path) -> None: + """SPEC 7.25.12: a raw markdown file with style violations does not fail sanity.""" + root = _wiki(tmp_path) + raw = root / "raw" / "articles" + raw.mkdir(parents=True) + (raw / "bad-style.md").write_text( + "# Bad\n\n" + ("x" * 180) + "\n\n[gone](missing-file.md)\n", + encoding="utf-8", + newline="\n", + ) + results = run_wiki_checks(root) + assert "bad-style.md" not in str(results) diff --git a/tests/test_ingest_edges.py b/tests/test_ingest_edges.py new file mode 100644 index 0000000..e31f4f7 --- /dev/null +++ b/tests/test_ingest_edges.py @@ -0,0 +1,552 @@ +"""Branch coverage for wiki ingest stages. + +Generated by AI/LLM: 2026-09-30 - error paths and converter edges from SPEC 7.25. +""" + +from __future__ import annotations + +import contextlib +import datetime +import email.message +import importlib.metadata +import io +import json +import pathlib +import shutil +import types +import typing +import urllib.error +import urllib.request + +import pypdf +import pytest +from pymarkdown.api import PyMarkdownApiException + +from fiona.mdoctor.mdoctor import MDoctor +from fiona.paths import llm_wiki_template +from fiona.wiki.ingest import log_writer as log_mod +from fiona.wiki.ingest import media +from fiona.wiki.ingest import page_writer as pages +from fiona.wiki.ingest import settings as settings_mod +from fiona.wiki.ingest.archiver import RawArchiver, _strip_prefix +from fiona.wiki.ingest.converters import default_registry +from fiona.wiki.ingest.converters.base import ConversionError, ConverterRegistry +from fiona.wiki.ingest.converters.html import HtmlConverter +from fiona.wiki.ingest.converters.markdown import MarkdownConverter +from fiona.wiki.ingest.converters.pdf import ( + PdfConverter, + _iso_date, + _metadata, + _outline, + _render, + _walk_outline, +) +from fiona.wiki.ingest.fetchers import UrlFetcher, _without_userinfo, url_scheme +from fiona.wiki.ingest.log_writer import IngestLog +from fiona.wiki.ingest.models import ArchiveResult, ConversionResult, FetchedSource, IngestError +from fiona.wiki.ingest.pipeline import ( + IngestPipeline, + _citing_page, + _print_pending, + default_pipeline, +) +from fiona.wiki.ingest.settings import default_settings, load_ingest_settings + + +def _wiki(tmp_path: pathlib.Path) -> pathlib.Path: + """Create a small llm-wiki.""" + root = tmp_path / "wiki" + (root / "wiki").mkdir(parents=True) + (root / "ingest").mkdir() + (root / "raw").mkdir() + shutil.copy2(llm_wiki_template() / "fiona-settings.json", root / "fiona-settings.json") + (root / "wiki" / "log.md").write_text( + "---\nname: log\nversion: 2.0.0\nupdated: 2026-09-30\n---\n\n# Log\n", + encoding="utf-8", + newline="\n", + ) + (root / "wiki" / "index.md").write_text( + "---\nname: index\nversion: 1.0.0\n---\n\n# Index\n", + encoding="utf-8", + newline="\n", + ) + return root + + +def _source(path: pathlib.Path, *, origin_kind: str = "file", origin: str | None = None) -> FetchedSource: + """Build a fetched-source record for a file that already exists.""" + return FetchedSource( + argument=str(path), + path=path, + origin=origin or str(path.resolve()), + origin_kind=origin_kind, + final_url=None, + media_type="text/plain", + charset=None, + size_bytes=path.stat().st_size, + slug=path.stem, + fetched="2026-09-30T00:00:00Z", + created=False, + ) + + +class _FakeResponse: + """One canned download body.""" + + def __init__(self, body: bytes, url: str) -> None: + self._body = body + self._offset = 0 + self.headers = email.message.Message() + self.headers["Content-Type"] = "text/plain" + self._url = url + + def read(self, amount: int = -1) -> bytes: + """Return the next slice of the body.""" + if amount < 0: + amount = len(self._body) - self._offset + chunk = self._body[self._offset:self._offset + amount] + self._offset += len(chunk) + return chunk + + def geturl(self) -> str: + """Return the final URL.""" + return self._url + + def __enter__(self) -> typing.Self: + return self + + def __exit__(self, *_args: object) -> bool: + return False + + +class _Opener: + """Opener that returns one body or raises URLError.""" + + def __init__(self, body: bytes | None = b"hello\n", *, error: str | None = None) -> None: + self.body = body or b"" + self.error = error + + def open(self, request: urllib.request.Request, timeout: float | None = None) -> _FakeResponse: + """Return the canned body or fail the download.""" + del timeout + if self.error: + raise urllib.error.URLError(self.error) + return _FakeResponse(self.body, request.full_url) + + +def test_slug_and_media_edges() -> None: + """Slugs, sniffing, extensions, and decode fallbacks.""" + assert url_scheme("note.txt") is None + assert media.slug_from_url("https://host/index") == "host" + assert media.slug_from_url("https://host/") == "host" + assert media.split_content_type("text/plain; boundary; charset=\"utf-8\"") == ("text/plain", "utf-8") + assert media.split_content_type("application/octet-stream") == (None, None) + registry = default_registry() + assert media.sniff_media_type(b" ") == "text/html" + assert media.sniff_media_type(b"\x00\xff") == "application/octet-stream" + assert media.sniff_media_type(b"ab\xe2\x82") == "text/plain" + assert media.sniff_media_type(b"plain text") == "text/plain" + assert media.extension_for("application/zip", "pdf", registry) == ".pdf" + assert media.extension_for("application/zip", "", registry) == ".bin" + assert media.decode_bytes(b"\xef\xbb\xbfhi", None) == ("hi", []) + text, warnings = media.decode_bytes(b"\xff", "latin-1") + assert text == "ΓΏ" + assert warnings == ["decoded as latin-1; not valid UTF-8"] + _text, fallback = media.decode_bytes(b"\xff", "not-a-charset") + assert "cp1252" in fallback[0] + assert registry.find("nope", "md").name == "markdown" + + +def test_settings_fallbacks(tmp_path: pathlib.Path, monkeypatch: pytest.MonkeyPatch) -> None: + """Missing, broken, and partial ingest settings keep usable values.""" + def missing(_name: str) -> str: + raise importlib.metadata.PackageNotFoundError + + monkeypatch.setattr(settings_mod.importlib.metadata, "version", missing) + assert settings_mod.package_version() == "0.99.0" + assert load_ingest_settings(tmp_path).timeout_seconds == 30 + broken = tmp_path / "bad" + broken.mkdir() + (broken / "fiona-settings.json").write_text("{", encoding="utf-8") + assert load_ingest_settings(broken).max_download_bytes == 52428800 + good = tmp_path / "good" + good.mkdir() + (good / "fiona-settings.json").write_text(json.dumps({ + "fiona": {"wiki": {"ingest": { + "user_agent": "agent", + "default_target": "concepts", + "raw_categories": {"pdf": "papers", "html": 1}, + }}}, + }), encoding="utf-8") + loaded = load_ingest_settings(good) + assert loaded.user_agent == "agent" + assert loaded.default_target == "concepts" + assert loaded.raw_categories == {"pdf": "papers"} + + +def test_markdown_frontmatter_edges(tmp_path: pathlib.Path) -> None: + """Invalid, empty, and non-mapping frontmatter, and a fenced heading.""" + converter = MarkdownConverter() + cases = { + "bad.md": "---\n[\n---\n# Kept\n", + "empty.md": "---\n\n---\n# Kept\n", + "list.md": "---\n- a\n---\n# Kept\n", + "fenced.md": "```\n# Hidden\n```\n", + } + for name, text in cases.items(): + path = tmp_path / name + path.write_text(text, encoding="utf-8") + result = converter.convert(_source(path)) + assert result.conversion == "complete" + fenced = converter.convert(_source(tmp_path / "fenced.md")) + assert fenced.title == "fenced" + assert fenced.body.startswith("```") + assert converter.convert(_source(tmp_path / "bad.md")).warnings + + +def test_html_edges(tmp_path: pathlib.Path) -> None: + """Comments, empty pages, og:title, images, and meta tags.""" + converter = HtmlConverter() + html = """ + + + + + + + + + + + +

+

Real

pic
+ + """ + path = tmp_path / "page.html" + path.write_text(html, encoding="utf-8") + source = _source(path, origin_kind="url", origin="https://host/p") + result = converter.convert(source) + assert "secret" not in result.body + assert "https://host/base/pic.png" in result.body + assert result.source_meta["author"] == "Ada" + assert result.source_meta["canonical"] == "https://host/c" + assert result.title == "Real" + empty = tmp_path / "empty.html" + empty.write_text("", encoding="utf-8") + partial = converter.convert(_source(empty, origin_kind="url", origin="https://host/e")) + assert partial.conversion == "partial" + og = tmp_path / "og.html" + og.write_text( + '

x

', + encoding="utf-8", + ) + assert converter.convert(_source(og)).title == "Og" + bare = tmp_path / "bare.html" + bare.write_text("

x

", encoding="utf-8") + assert converter.convert(_source(bare)).title is None + blank_h1 = tmp_path / "blankh1.html" + blank_h1.write_text("

x

", encoding="utf-8") + assert converter.convert(_source(blank_h1)).title is None + + +def test_pdf_metadata_outline_and_failures(tmp_path: pathlib.Path, monkeypatch: pytest.MonkeyPatch) -> None: + """PDF dates, outline headings, a scanned file, and read failures.""" + aware = datetime.datetime(2020, 1, 2, tzinfo=datetime.UTC) + naive = datetime.datetime(2020, 1, 1) # noqa: DTZ001 + assert _iso_date(None) is None + assert _iso_date(naive).startswith("2020-01-01T") + assert _iso_date(aware).startswith("2020-01-02T") + assert _iso_date("D:20200103") == "2020-01-03T00:00:00Z" + assert _iso_date("nope") == "nope" + assert _iso_date("") is None + + info_obj = types.SimpleNamespace( + title="T", + author="", + subject="S", + keywords=None, + creation_date=naive, + modification_date=aware, + ) + empty = types.SimpleNamespace(pages=[1], metadata=None) + info_reader = types.SimpleNamespace(pages=[1, 2], metadata=info_obj) + assert _metadata(empty) == {"page_count": 1} # type: ignore[arg-type] + info = _metadata(info_reader) # type: ignore[arg-type] + assert info["title"] == "T" + assert "author" not in info + + class _OutlineReader: + """Fake reader whose first outline item has no destination.""" + + def get_destination_page_number(self, item: object) -> int: + if getattr(item, "title", None) == "": + raise ValueError("bad") + return 0 + + found: list[tuple[int, int, str]] = [] + bad = types.SimpleNamespace(title="") + good = types.SimpleNamespace(title=None) + _walk_outline(_OutlineReader(), [bad, [good]], 5, found) # type: ignore[arg-type] + assert found == [(0, 6, "Untitled")] + assert _render(["body"], [(0, 2, "Chapter")]) == "## Chapter\n\nbody\n" + assert "## Page" not in _render([""], [(1, 2, "Later")]) + + class _BrokenOutline: + """Fake reader whose outline cannot be read.""" + + @property + def outline(self) -> list[object]: + raise pypdf.errors.PdfReadError("no outline") + + assert _outline(_BrokenOutline()) == [] # type: ignore[arg-type] + + writer = pypdf.PdfWriter() + writer.add_blank_page(width=200, height=200) + parent = writer.add_outline_item("Parent", 0) + writer.add_outline_item("Child", 0, parent=parent) + writer.add_metadata({"/Title": "Outlined", "/CreationDate": "D:20200101120000"}) + buffer = io.BytesIO() + writer.write(buffer) + outlined = tmp_path / "outlined.pdf" + outlined.write_bytes(buffer.getvalue()) + converted = PdfConverter().convert(_source(outlined)) + assert "Parent" in converted.body + assert converted.title == "Outlined" + + blank = tmp_path / "blank.pdf" + blank_writer = pypdf.PdfWriter() + blank_writer.add_blank_page(width=200, height=200) + blank_buffer = io.BytesIO() + blank_writer.write(blank_buffer) + blank.write_bytes(blank_buffer.getvalue()) + scanned = PdfConverter().convert(_source(blank)) + assert scanned.conversion == "partial" + assert any("scanned" in warning for warning in scanned.warnings) + + corrupt = tmp_path / "corrupt.pdf" + corrupt.write_bytes(b"%PDF-1.7 not a real file") + with pytest.raises(ConversionError): + PdfConverter().convert(_source(corrupt)) + + def extract_text() -> str: + raise pypdf.errors.PdfReadError("page") + + reader = types.SimpleNamespace( + is_encrypted=False, + metadata=None, + pages=[types.SimpleNamespace(extract_text=extract_text)], + outline=[], + ) + monkeypatch.setattr(pypdf, "PdfReader", lambda _path: reader) + with pytest.raises(ConversionError, match="page"): + PdfConverter().convert(_source(outlined)) + + +def test_archive_edges(tmp_path: pathlib.Path, monkeypatch: pytest.MonkeyPatch) -> None: + """Reuse skips, raw escapes, named raw files, and collision suffixes.""" + root = _wiki(tmp_path) + archiver = RawArchiver(tmp_path / "empty") + assert archiver.find_by_sha256("ab" * 32, 1) is None + assert _strip_prefix("raw", "raw") == "" + assert _strip_prefix("raw/notes/a.txt", "raw") == "notes/a.txt" + registry = default_registry() + settings = default_settings() + already = root / "raw" / "articles" / "already.md" + already.parent.mkdir(parents=True) + already.write_text("# Already\n", encoding="utf-8") + placed = archiver_on(root).archive( + _source(already), + registry=registry, + settings=settings, + raw_override=None, + dry_run=False, + ) + assert placed.raw_reused is False + with pytest.raises(IngestError, match="already under raw"): + archiver_on(root).archive( + _source(already), + registry=registry, + settings=settings, + raw_override="other", + dry_run=False, + ) + original_stat = pathlib.Path.stat + seen: dict[str, int] = {} + + def flaky(self: pathlib.Path, *args: object, **kwargs: object) -> object: + if self.name == "bad.bin": + seen[str(self)] = seen.get(str(self), 0) + 1 + if seen[str(self)] > 1: + raise OSError("stat failed") + return original_stat(self, *args, **kwargs) + + (root / "raw" / "bad.bin").write_bytes(b"x") + monkeypatch.setattr(pathlib.Path, "stat", flaky) + assert archiver_on(root).find_by_sha256("ab" * 32, 1) is None + monkeypatch.undo() + + pipeline = default_pipeline(root) + named = tmp_path / "fresh.txt" + named.write_text("fresh\n", encoding="utf-8") + pipeline.run(str(named), raw="notes/kept.html", raw_only=True) + assert (root / "raw" / "notes" / "kept.html").is_file() + for index, text in enumerate(("one\n", "two\n", "three\n"), start=1): + path = tmp_path / f"dir{index}" / "note.txt" + path.parent.mkdir() + path.write_text(text, encoding="utf-8") + pipeline.run(str(path), raw_only=True) + assert (root / "raw" / "articles" / "note-3.txt").is_file() + + +def archiver_on(root: pathlib.Path) -> RawArchiver: + """Return an archiver bound to ``root``.""" + return RawArchiver(root) + + +def test_download_and_log_edges(tmp_path: pathlib.Path, monkeypatch: pytest.MonkeyPatch) -> None: + """URL errors, a failed save, userinfo, and a log that cannot be bumped.""" + root = _wiki(tmp_path) + registry = default_registry() + fetcher = UrlFetcher(registry, opener=_Opener(error="timed out")) + with pytest.raises(IngestError, match="timed out"): + fetcher.fetch("https://host/a.txt", root / "ingest") + assert _without_userinfo("https://user:secret@host:443/a.txt") == "https://host:443/a.txt" + assert _without_userinfo("https://host/a.txt") == "https://host/a.txt" + + original_write = pathlib.Path.write_bytes + + def fail_write(self: pathlib.Path, data: bytes) -> int: + original_write(self, data) + raise OSError("disk") + + monkeypatch.setattr(pathlib.Path, "write_bytes", fail_write) + fetcher = UrlFetcher(registry, opener=_Opener(b"hello\n")) + with pytest.raises(OSError, match="disk"): + fetcher.fetch("https://host/b.txt", root / "ingest") + monkeypatch.undo() + assert not list((root / "ingest").glob("*.partial")) + + log = IngestLog(root) + log.append(day="2026-09-30", origin="o", raw_rel="raw/a", page_rel=None, dry_run=True) + (root / "wiki" / "log.md").unlink() + log.append(day="2026-09-30", origin="o", raw_rel="raw/a", page_rel="wiki/a.md", dry_run=False) + odd = root / "wiki" / "log.md" + odd.write_text("version: beta\n# Log", encoding="utf-8", newline="\n") + log.append(day="2026-09-30", origin="o", raw_rel="raw/a", page_rel=None, dry_run=False) + text = odd.read_text(encoding="utf-8") + assert "version: beta" in text + assert text.endswith("\n") + assert log_mod._set_updated("no updated here\n", "2026-09-30") == "no updated here\n" + assert log_mod._bump_version("version: 1\n") == "version: 1\n" + assert log_mod._bump_version("no version here\n") == "no version here\n" + + +def test_page_writer_edges(tmp_path: pathlib.Path, monkeypatch: pytest.MonkeyPatch) -> None: + """Heading fences, links, fixer failures, and target prefixes.""" + body = "# Title\n\n```\n# inside\n```\n\n###### Six\n" + shifted = pages._shift_headings(body) + assert "```\n# inside\n```" in shifted + assert "###### Six" in shifted + assert pages._keeps_target("?q", tmp_path) is True + assert pages._keeps_target("missing.md", tmp_path) is False + assert pages._body_after_frontmatter("no front\n") == "no front\n" + assert pages._body_after_frontmatter("---\nonly\n") == "---\nonly\n" + assert pages._split_ending("a\r\n") == ("\r\n", "a") + assert pages._split_ending("a") == ("", "a") + assert pages._strip_wiki_prefix("wiki") == "" + assert pages._strip_wiki_prefix("wiki/concepts") == "concepts" + + root = _wiki(tmp_path) + writer = pages.RawPageWriter(root) + source_path = tmp_path / "note.txt" + source_path.write_text("note\n", encoding="utf-8") + source = _source(source_path) + archive_path = root / "raw" / "articles" / "note.txt" + archive_path.parent.mkdir(parents=True, exist_ok=True) + archive_path.write_text("note\n", encoding="utf-8") + archive = ArchiveResult(archive_path, "raw/articles/note.txt", "ab" * 32, False, []) + conversion = ConversionResult("plain\n", "Note", {}, "complete", [], "text") + + def broken_fix(self: MDoctor, path: str) -> None: + raise PyMarkdownApiException("fix") + + def broken_scan(self: MDoctor, path: str) -> list: + raise PyMarkdownApiException("scan") + + monkeypatch.setattr(MDoctor, "fix_file_programmatically", broken_fix) + monkeypatch.setattr(MDoctor, "scan_file_programmatically", broken_scan) + monkeypatch.setattr(pages, "_body_after_frontmatter", lambda _text: "no heading\n") + page = root / "wiki" / "sources" / "note.md" + writer.write( + source=source, + archive=archive, + conversion=conversion, + page=page, + dry_run=False, + ) + assert page.is_file() + assert page.read_text(encoding="utf-8").count("# Note") == 1 + + +def test_pipeline_edges(tmp_path: pathlib.Path) -> None: + """Dry-run downloads, pending pages, and a forced duplicate.""" + root = _wiki(tmp_path) + pipeline = default_pipeline(root) + pipeline.fetchers[0] = UrlFetcher(pipeline.registry, opener=_Opener(b"hello\n")) + result = pipeline.run("https://user:secret@host/hello.txt", dry_run=True) + assert result.dry_run is True + assert list((root / "ingest").iterdir()) == [] + with pytest.raises(IngestError, match="stay inside raw"): + pipeline.run("https://host/escape.txt", raw="../outside") + assert list((root / "ingest").glob("*.txt")) == [] + + bare = IngestPipeline( + root, + [], + ConverterRegistry(), + pipeline.archiver, + pipeline.page_writer, + pipeline.log_writer, + default_settings(), + ) + local = tmp_path / "local.txt" + local.write_text("local\n", encoding="utf-8") + with pytest.raises(IngestError, match="Source not found"): + bare.run(str(local)) + + (root / "wiki" / "skip.md").mkdir() + (root / "wiki" / "bad.md").write_text("---\n[\n---\n", encoding="utf-8") + (root / "wiki" / "sources").mkdir(exist_ok=True) + (root / "wiki" / "sources" / "stringy.md").write_text( + "---\nname: stringy\nstatus: raw\nsources: raw/a.txt\ningest: plain\n---\n\n# S\n", + encoding="utf-8", + newline="\n", + ) + pending = pipeline.list_pending() + assert any(item["page_path"].endswith("stringy.md") for item in pending) + assert _citing_page(root, "raw/a.txt", "nope") == "wiki/sources/stringy.md" + captured = io_capture(_print_pending, pipeline, False) + assert "stringy.md" in captured + shutil.rmtree(root / "wiki") + assert pipeline.list_pending() == [] + assert _citing_page(root, "raw/a.txt", "nope") is None + + root = _wiki(tmp_path / "second") + pipeline = default_pipeline(root) + first = tmp_path / "first.txt" + second = tmp_path / "other" / "second.txt" + first.write_text("same\n", encoding="utf-8") + second.parent.mkdir() + second.write_text("same\n", encoding="utf-8") + pipeline.run(str(first)) + forced = pipeline.run(str(second), force=True) + assert forced.page_path == "wiki/sources/first.md" + + +def io_capture(func: typing.Callable, *args: object) -> str: + """Return stdout from a function that prints.""" + buffer = io.StringIO() + with contextlib.redirect_stdout(buffer): + func(*args) + return buffer.getvalue() diff --git a/todo.md b/todo.md index 0bd36db..9edc7e1 100644 --- a/todo.md +++ b/todo.md @@ -680,6 +680,29 @@ WARNING run_tests: pytest short summary missing from output; failure details com - [ ] `0`, or a missing key, draws every node - [ ] A positive integer hides a node whose `unique_points` is below that integer until one of its parents is clicked +- [ ] Omit gitignored markdown from the wiki graph (SPEC 7.4.17) + - [ ] Failing tests in `tests/test_graph_generator.py` for the + acceptances in SPEC 7.4.17, 7.4.17.1, and 7.4.17.2: + `secret.md` absent from JSON nodes, edges, and orphans; + a link to it creates no edge; `max_files` 1 still keeps + `kept.md`; `drafts/` omits `drafts/note.md` and keeps + `wiki/note.md`; a `# secret.md` comment does not omit; + `notes/.gitignore` omits `notes/scratch.md` only; with no + `.gitignore`, `.git/*.md` is absent and a root page stays + - [ ] In `wiki_graph_main` (`src/fiona/wiki/graph_generator.py`), + drop a match before the `max_files` cut. Read patterns with + `gitignore_patterns_from_text` and test them with + `path_is_ignored` (`src/fiona/wiki/ignore.py`), the same + rules as SPEC 7.2.10. The scan-root `.gitignore` applies to + the whole tree. A subdirectory `.gitignore` applies only + under that subdirectory. Also drop any path under `.git/` + - [ ] Do not parse an omitted file. Drop an edge whose target + resolves to an omitted file, so it never becomes a node + - [ ] HTML uses that same graph. No new settings key: the filter + always runs + - [ ] Housekeeping's orphan list follows automatically, because + `graph_step` builds the graph through `wiki_graph_main` + (SPEC 7.26.6) - [ ] in spec, refine the sectioning by compiling the modules, and their functions as subsections - [ ] eg 4.1. librarian then 4.1.1 librarian toc, to make it cleaner