ingest implemented
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: Fiona-Assistant Specification
|
||||
version: 0.97.0
|
||||
version: 0.99.0
|
||||
title: Project Spec Sheet
|
||||
created: 2026-09-23
|
||||
updated: 2026-09-30
|
||||
@@ -99,7 +99,10 @@ yield to the owner's markdown and write `<stem>-update.md` instead
|
||||
Revision 0.96.0 compares a file without a version by SHA-256 and
|
||||
shows both short hashes in the update report (*7.2.12.*).
|
||||
Revision 0.97.0 rejects a module-level assignment that follows the
|
||||
first function or class (*6.4.3.*).
|
||||
first function or class (*6.4.3.*). Revision 0.98.0 specifies that
|
||||
`wiki graph` omits a file matched by `.gitignore` (*7.4.17.*) and
|
||||
leaves that requirement unbuilt. Revision 0.99.0 builds `wiki ingest`
|
||||
(*7.25.*).
|
||||
|
||||
Each feature section starts with a `Status` line. Requirements use
|
||||
`- *N.M.K.*`. Each `*acceptance*` sits under the requirement it tests.
|
||||
@@ -277,7 +280,6 @@ utils/skills/README.md
|
||||
utils/skills/fiona/SKILL.md
|
||||
utils/skills/fiona/markdown_rules.md
|
||||
utils/skills/fiona/llm-wiki-mod.md
|
||||
utils/skills/obsidian/SKILL.md
|
||||
utils/skills/python-tdd/SKILL.md
|
||||
utils/personas/default/SOUL.md
|
||||
utils/personas/template/SOUL.md
|
||||
@@ -732,7 +734,7 @@ extra untimed `coverage:` row under pytests. Details are section 7.16.
|
||||
|
||||
### 4.14. wiki ingest
|
||||
|
||||
Not built yet. The contract is section 7.25.
|
||||
The contract is section 7.25.
|
||||
|
||||
```text
|
||||
fiona-assistant wiki ingest <source> [--wiki DIR] [-r|--raw PATH]
|
||||
@@ -1088,7 +1090,7 @@ directory relative to `wiki/`. `raw_categories` maps a converter
|
||||
name to a directory under `raw/`, such as `{"text": "transcripts"}`.
|
||||
A value that is absent, of the wrong type, or out of range uses the
|
||||
default in the table. The packaged `fiona-settings.json` omits
|
||||
`fiona.wiki.ingest` until section 7.25 is built.
|
||||
`fiona.wiki.ingest`. Absent keys use the defaults in the table.
|
||||
|
||||
A mapping key under `toc_generator.file` whose name is not one of
|
||||
`min_heading`, `max_heading`, `sectionize`, `ignore_files` is an
|
||||
@@ -1593,8 +1595,8 @@ another `(name, runner)` pair in the wiki sanity list.
|
||||
|
||||
> Status: Partially Implemented
|
||||
|
||||
*7.4.16.* is planned. Every other requirement in this section is
|
||||
built.
|
||||
*7.4.16.* and *7.4.17.* are planned. Every other requirement in
|
||||
this section is built.
|
||||
|
||||
The renderer is D3 v7, loaded from `https://d3js.org/d3.v7.min.js`,
|
||||
with the visualization embedded in one HTML file. vis.js is not
|
||||
@@ -1991,6 +1993,37 @@ used.
|
||||
HTML draw only.
|
||||
- *acceptance*: JSON includes a node whose
|
||||
`unique_points` is below the setting.
|
||||
- *7.4.17.* `NOT_IMPLEMENTED` A markdown file that `.gitignore`
|
||||
matches is not a graph node. HTML and JSON both omit it. An
|
||||
edge that would start or end at that file is omitted. The file
|
||||
does not count toward `max_files`. Housekeeping's orphan list
|
||||
(*7.26.6.*) uses this graph, so an omitted file is not an
|
||||
orphan. There is no setting that turns this off.
|
||||
- *acceptance*: The scanned `.gitignore` lists `secret.md`.
|
||||
`secret.md` is absent from JSON `nodes`, `edges`, and
|
||||
`orphans`. A page that links to it has no edge to it.
|
||||
- *acceptance*: `max_files` is 1. The directory holds `kept.md`
|
||||
and a gitignored `secret.md`. JSON `file_count` is 1 and the
|
||||
only node is `kept.md`.
|
||||
- *7.4.17.1.* `NOT_IMPLEMENTED` Patterns are the non-comment,
|
||||
non-blank lines of the `.gitignore` in the scanned directory,
|
||||
and of each `.gitignore` in a subdirectory. A subdirectory
|
||||
file's patterns match only paths under that subdirectory.
|
||||
Matching is *7.2.10.*: a pattern with no slash matches any
|
||||
path segment, and a pattern that ends with `/` matches that
|
||||
directory and everything under it.
|
||||
- *acceptance*: The scanned `.gitignore` lists `drafts/`.
|
||||
`drafts/note.md` is absent. `wiki/note.md` is present.
|
||||
- *acceptance*: A line `# secret.md` does not omit
|
||||
`secret.md`.
|
||||
- *acceptance*: `notes/.gitignore` lists `scratch.md`.
|
||||
`notes/scratch.md` is absent. `scratch.md` at the scan
|
||||
root stays.
|
||||
- *7.4.17.2.* `NOT_IMPLEMENTED` A path under `.git/` is omitted
|
||||
even when no pattern names it.
|
||||
- *acceptance*: With no `.gitignore`, a markdown file under
|
||||
the scan root stays, and a markdown file under `.git/` is
|
||||
absent.
|
||||
|
||||
### 7.5. Git Operations
|
||||
|
||||
@@ -2512,7 +2545,7 @@ is the pip-argument source both scripts call.
|
||||
|
||||
### 7.25. wiki ingest
|
||||
|
||||
> Status: Planned
|
||||
> Status: Implemented
|
||||
|
||||
Implements section 4.14. An ingest has two phases. The tool runs
|
||||
the first: it turns one source into a file on disk, archives a copy
|
||||
@@ -2547,7 +2580,7 @@ A source that no converter claims is archived under `raw/other/`.
|
||||
Pure-Python libraries are preferred so the install stays light.
|
||||
Candidates for later formats are in section 10.
|
||||
|
||||
- *7.25.1.* `NOT_IMPLEMENTED` The pipeline is built from parts.
|
||||
- *7.25.1.* The pipeline is built from parts.
|
||||
- *7.25.1.1.* The pipeline receives its fetchers, converter
|
||||
registry, archiver, page writer, and log writer as constructor
|
||||
arguments. It holds no format-specific code.
|
||||
@@ -2563,7 +2596,7 @@ Candidates for later formats are in section 10.
|
||||
an argument.
|
||||
- *acceptance*: Tests run the URL path with a fake opener and
|
||||
make no network call.
|
||||
- *7.25.2.* `NOT_IMPLEMENTED` Fetch.
|
||||
- *7.25.2.* Fetch.
|
||||
- *7.25.2.1.* An argument that starts with `http://` or `https://`
|
||||
is a URL. Any other `<scheme>://` prefix exits 1 with
|
||||
`Error: Unsupported URL scheme: <scheme>`. Anything else is a
|
||||
@@ -2607,7 +2640,7 @@ Candidates for later formats are in section 10.
|
||||
- *acceptance*:
|
||||
`https://host/job/performance_TL153/2108/console` has the slug
|
||||
`performance-tl153-2108-console`.
|
||||
- *7.25.3.* `NOT_IMPLEMENTED` Archive.
|
||||
- *7.25.3.* Archive.
|
||||
- *7.25.3.1.* The default raw path is
|
||||
`raw/<category>/<slug>.<ext>`. The category is the raw category
|
||||
of the converter that claims the media type, unless
|
||||
@@ -2661,7 +2694,7 @@ Candidates for later formats are in section 10.
|
||||
- *acceptance*: After ingesting `ingest/a.pdf`, that path is gone
|
||||
and `raw/papers/a.pdf` holds its bytes. A file ingested from
|
||||
`personal/` is copied and the original is unchanged.
|
||||
- *7.25.4.* `NOT_IMPLEMENTED` Convert.
|
||||
- *7.25.4.* Convert.
|
||||
- *7.25.4.1.* The registry picks the first converter, in
|
||||
registration order, whose media types contain the detected type,
|
||||
then the first whose extensions contain the file extension. With
|
||||
@@ -2723,7 +2756,7 @@ Candidates for later formats are in section 10.
|
||||
- *acceptance*: A two-page fixture with no outline has
|
||||
`## Page 1` and `## Page 2`. A fixture with one blank page is
|
||||
`partial` and names that page.
|
||||
- *7.25.5.* `NOT_IMPLEMENTED` Write.
|
||||
- *7.25.5.* Write.
|
||||
- *7.25.5.1.* The default page is
|
||||
`wiki/<default_target>/<slug>.md`. `-t/--target PATH` is
|
||||
relative to `wiki/`, and a leading `wiki/` is accepted. A value
|
||||
@@ -2799,7 +2832,7 @@ Candidates for later formats are in section 10.
|
||||
`<N> markdown style violations left`.
|
||||
- *acceptance*: Every fixture page passes fmlint with no
|
||||
violation.
|
||||
- *7.25.6.* `NOT_IMPLEMENTED` Record. A run that archives appends one
|
||||
- *7.25.6.* Record. A run that archives appends one
|
||||
bullet to `wiki/log.md` and bumps that file's patch `version` and
|
||||
`updated`. The bullet is
|
||||
`- <YYYY-MM-DD> - ingest <origin> -> <raw path> -> <page path> (raw)`,
|
||||
@@ -2807,10 +2840,10 @@ Candidates for later formats are in section 10.
|
||||
does not edit `wiki/index.md` or any other page.
|
||||
- *acceptance*: One successful run adds one line to `log.md` and
|
||||
leaves `index.md` unchanged.
|
||||
- *7.25.7.* `NOT_IMPLEMENTED` `--raw-only` runs fetch, archive, and
|
||||
- *7.25.7.* `--raw-only` runs fetch, archive, and
|
||||
record, and skips convert and write. It exits 0.
|
||||
- *acceptance*: No page is written and `conversion` is `none`.
|
||||
- *7.25.8.* `NOT_IMPLEMENTED` `--dry-run` fetches into a temporary
|
||||
- *7.25.8.* `--dry-run` fetches into a temporary
|
||||
directory outside the wiki, so a URL is still downloaded. It
|
||||
prints `Dry run - no files will be modified:` and then the lines
|
||||
a real run would print. It writes nothing under the wiki, does not
|
||||
@@ -2818,12 +2851,12 @@ Candidates for later formats are in section 10.
|
||||
true in JSON.
|
||||
- *acceptance*: File hashes under the wiki match before and after
|
||||
a dry run, and an `ingest/` source is still in place.
|
||||
- *7.25.9.* `NOT_IMPLEMENTED` `--pending` lists every page under
|
||||
- *7.25.9.* `--pending` lists every page under
|
||||
`wiki/` whose frontmatter `status` is `raw`, sorted by path, and
|
||||
exits 0.
|
||||
- *acceptance*: A page changed to `status: refined` is no longer
|
||||
listed.
|
||||
- *7.25.10.* `NOT_IMPLEMENTED` Refinement is the agent's phase. The
|
||||
- *7.25.10.* Refinement is the agent's phase. The
|
||||
tool never writes `status: refined`. A page is refined when
|
||||
`status` is `refined`, `version` is `1.0.0` or later, the quality
|
||||
signals are set and `llm.model` is not `none`, the page is listed
|
||||
@@ -2832,7 +2865,7 @@ Candidates for later formats are in section 10.
|
||||
- *acceptance*: Fiona skill section 16 lists those conditions and
|
||||
the steps that meet them, and says when to convert a source by
|
||||
hand.
|
||||
- *7.25.11.* `NOT_IMPLEMENTED` The agent checks the frontmatter the
|
||||
- *7.25.11.* The agent checks the frontmatter the
|
||||
tool wrote before it refines the body. It may correct `title`,
|
||||
`type`, `tags`, `source_meta`, `ingest.conversion`, and
|
||||
`ingest.warnings` when the source shows they are wrong. It does
|
||||
@@ -2842,11 +2875,11 @@ Candidates for later formats are in section 10.
|
||||
the fault to the user.
|
||||
- *acceptance*: Fiona skill section 16 has a metadata check step
|
||||
that names the fields it may edit and the fields it may not.
|
||||
- *7.25.12.* `NOT_IMPLEMENTED` Archived sources do not fail
|
||||
- *7.25.12.* Archived sources do not fail
|
||||
`wiki sanity`. Every check skips `raw/`.
|
||||
- *acceptance*: After ingesting a markdown file with style
|
||||
violations, `wiki sanity` reports none from its raw copy.
|
||||
- *7.25.13.* `NOT_IMPLEMENTED` `--json` prints only the JSON object
|
||||
- *7.25.13.* `--json` prints only the JSON object
|
||||
from section 4.14 on stdout, on exit 0 and on exit 3. On exit 1,
|
||||
stdout is empty.
|
||||
- *acceptance*: The stdout of an exit 3 run parses with
|
||||
@@ -2968,10 +3001,11 @@ Paths in details and actions are relative to the wiki root.
|
||||
| P2 | Wiki root discovery | 7.23 | Planned |
|
||||
| P2 | settings setup | 7.19 | Planned |
|
||||
| P2 | Bootstrap install modes | 7.24 | Implemented |
|
||||
| P2 | wiki ingest | 7.25 | Planned |
|
||||
| P2 | wiki ingest | 7.25 | Implemented |
|
||||
| P3 | fmlint enforcement of quality signals | 6.2 | Planned |
|
||||
| P3 | Backup archive | 7.6 | Planned |
|
||||
| P3 | Documentation refresh | 7.21 | Partially Implemented |
|
||||
| P2 | Graph skips gitignored files | 7.4.17 | Planned |
|
||||
| P3 | Node importance filter | 7.4.16 | Planned |
|
||||
| P4 | Non-markdown version tags | 7.14 | Planned |
|
||||
| P4 | Tool-role table | 7.22 | Planned |
|
||||
@@ -3034,7 +3068,7 @@ These notes are not requirements. They point at the current tree.
|
||||
| launchers | `scripts/run_tests.py`, `scripts/run_sanity.py` |
|
||||
| bootstrap | `bootstrap`, `bootstrap.ps1`, `scripts/bootstrap_install.py` |
|
||||
|
||||
Planned layout for section 7.25, under `src/fiona/wiki/ingest/`:
|
||||
Layout for section 7.25, under `src/fiona/wiki/ingest/`:
|
||||
|
||||
| Module | Classes | Role |
|
||||
| --- | --- | --- |
|
||||
|
||||
+4
-1
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "fiona-assistant"
|
||||
version = "0.96.0"
|
||||
version = "0.99.0"
|
||||
description = "Tools for managing the LLM wiki."
|
||||
authors = [{name = "nvaldeziii"}]
|
||||
readme = "README.md"
|
||||
@@ -24,6 +24,9 @@ dependencies = [
|
||||
"ruff>=0.16.8",
|
||||
"pyyaml>=6.0.3",
|
||||
"GitPython>=3.1.0",
|
||||
"beautifulsoup4>=4.12.0",
|
||||
"markdownify>=1.1.0",
|
||||
"pypdf>=5.0.0",
|
||||
]
|
||||
|
||||
# Optional dependencies groups
|
||||
|
||||
@@ -18,6 +18,7 @@ from fiona.mdoctor.mdoctor import main as mdoctor_main
|
||||
from fiona.python_ruff.python_ruff import main as python_ruff_main
|
||||
from fiona.wiki.graph_generator import wiki_graph_main
|
||||
from fiona.wiki.housekeeping import housekeeping_main
|
||||
from fiona.wiki.ingest.pipeline import add_ingest_arguments, execute_ingest
|
||||
from fiona.wiki.sanity_manager import sanity_main
|
||||
from fiona.wiki.update_manager import check_update_main, update_main
|
||||
from fiona.wiki.wiki import wiki_init
|
||||
@@ -280,6 +281,14 @@ def main():
|
||||
help="Disable tracking of external links"
|
||||
)
|
||||
|
||||
# AI/LLM: 2026-09-30 - flags live here so wiki ingest --help is not rejected.
|
||||
wiki_ingest_parser = wiki_subparsers.add_parser(
|
||||
"ingest",
|
||||
prog="fiona-assistant wiki ingest",
|
||||
help="Archive one source and write a raw wiki page",
|
||||
)
|
||||
add_ingest_arguments(wiki_ingest_parser)
|
||||
|
||||
# AI/LLM: fiona-assistant housekeeping (SPEC 4.15)
|
||||
housekeeping_parser = subparsers.add_parser(
|
||||
"housekeeping",
|
||||
@@ -399,6 +408,8 @@ def main():
|
||||
json_output=args.json,
|
||||
non_interactive=args.non_interactive
|
||||
))
|
||||
elif args.wiki_subcommand == "ingest":
|
||||
sys.exit(execute_ingest(args, wiki_ingest_parser))
|
||||
elif args.wiki_subcommand == "graph":
|
||||
sys.exit(wiki_graph_main(
|
||||
directory=args.directory,
|
||||
|
||||
@@ -4,7 +4,7 @@ category: productivity
|
||||
description: Operate an llm-wiki with the fiona-assistant CLI.
|
||||
author: nvaldez
|
||||
license: MIT
|
||||
version: 1.9.1
|
||||
version: 1.11.0
|
||||
updated: 2026-09-30
|
||||
---
|
||||
|
||||
@@ -104,7 +104,19 @@ fiona-assistant git save --description "ingest <slug>: <what the source is>"
|
||||
```
|
||||
|
||||
After a housekeeping pass (lint fixes, index, log, or TOC updates,
|
||||
page moves, template updates), with sanity passing, run:
|
||||
page moves, template updates), with sanity passing, read `git diff`
|
||||
for the files `fiona-assistant git status` lists.
|
||||
|
||||
- **Metadata.** On each added or modified page, frontmatter matches
|
||||
section 13. `name`, `version`, and `updated` are set. A page under
|
||||
`wiki/` also has `confidence`, `llm`, `contested`, and
|
||||
`contradictions`.
|
||||
- **Integrity.** The body still holds the claims, links, tables, and
|
||||
index or log entries from before this pass. A lint fix, a move, or
|
||||
a template merge did not drop or garble them.
|
||||
|
||||
When a check fails, fix the page and run sanity again. When you
|
||||
cannot tell whether a page is intact, ask before saving. Then run:
|
||||
|
||||
```text
|
||||
fiona-assistant git save --description "housekeeping: <what changed>"
|
||||
@@ -127,13 +139,14 @@ It is the step that makes the wiki's history usable:
|
||||
- **Proof behind the log.** `wiki/log.md` says what happened. The
|
||||
commit holds the before and after state that proves it.
|
||||
|
||||
Before saving, run `fiona-assistant git status`. `save` stages
|
||||
Before either save, run `fiona-assistant git status`. `save` stages
|
||||
everything in the tree. When it shows changes that are not from
|
||||
this cycle, ask the user before saving. When sanity fails, fix it
|
||||
first, or ask. A duplicate ingest writes nothing, so `save`
|
||||
refusing a clean tree is expected. This standing instruction
|
||||
covers `save` only. `snapshot`, `restore`, `clean`, and `undo`
|
||||
still need the user's request.
|
||||
this cycle, ask the user before saving. A housekeeping pass also
|
||||
reads `git diff` and applies the metadata and integrity checks
|
||||
above. When sanity fails, fix it first, or ask. A duplicate ingest
|
||||
writes nothing, so `save` refusing a clean tree is expected. This
|
||||
standing instruction covers `save` only. `snapshot`, `restore`,
|
||||
`clean`, and `undo` still need the user's request.
|
||||
|
||||
## 1. Before a command
|
||||
|
||||
@@ -323,10 +336,6 @@ Also: `--include-pattern`, `--exclude-pattern`,
|
||||
|
||||
### 4.6. ingest
|
||||
|
||||
Planned. Run `fiona-assistant wiki ingest --help`
|
||||
first. When the subcommand does not exist, do every step of
|
||||
section 16 by hand.
|
||||
|
||||
```text
|
||||
fiona-assistant wiki ingest <source> [--wiki DIR] [-r|--raw PATH] [-t|--target PATH] [--raw-only] [-f|--force] [--dry-run] [--json]
|
||||
fiona-assistant wiki ingest --pending [--wiki DIR] [--json]
|
||||
@@ -706,10 +715,6 @@ These are not implemented. Do not invent flags for them.
|
||||
- `fiona-assistant pycustomlint`
|
||||
- `fiona-assistant python-ruff fix`
|
||||
|
||||
`fiona-assistant wiki ingest` is planned. Call it only after
|
||||
`fiona-assistant wiki ingest --help` succeeds. Otherwise follow
|
||||
section 16 by hand.
|
||||
|
||||
## 16. Ingest cycle
|
||||
|
||||
An ingest has two phases. `wiki ingest` (section 4.6) converts the
|
||||
@@ -776,7 +781,10 @@ the user. Do not move or delete anything under `raw/`.
|
||||
|
||||
The tool fills every key in section 16.5, but it reads the source
|
||||
mechanically. Check the frontmatter against the source before you
|
||||
refine, and fix what is wrong.
|
||||
refine, and fix what is wrong. You may edit `title`, `type`, `tags`,
|
||||
`source_meta`, `ingest.conversion`, and `ingest.warnings`. You may
|
||||
not hand-edit `ingest.sha256`, `ingest.size_bytes`, `ingest.fetched`,
|
||||
`ingest.media_type`, or `ingest.tool`.
|
||||
|
||||
| Key | Check | May edit |
|
||||
| --- | --- | --- |
|
||||
@@ -800,6 +808,11 @@ with `--force` or tell the user. Do not hand-edit it. Bump the page
|
||||
1. `fiona-assistant wiki ingest --pending --json` lists the pages
|
||||
still at `status: raw`. Without the tool, search `wiki/` for
|
||||
`status: raw`. Check each page's metadata first (section 16.3).
|
||||
A markdown source keeps its body. The tool writes the page
|
||||
frontmatter when the source did not already have it. Source
|
||||
frontmatter is in `source_meta`. A text source (`.txt`, `.text`,
|
||||
`.log`) is one fenced `text` block plus that frontmatter, still
|
||||
at `status: raw`. Rewrite that fence. The tool does not refine it.
|
||||
2. Read the page and the source it cites. Open the raw file when
|
||||
`ingest.conversion` is `partial` or a claim looks garbled.
|
||||
3. Rewrite the page as a source summary:
|
||||
@@ -826,8 +839,11 @@ with `--force` or tell the user. Do not hand-edit it. Bump the page
|
||||
`wiki/log.md`.
|
||||
9. Run the work loop in section 12, ending with `wiki sanity`.
|
||||
|
||||
The ingest is done when no page is left at `status: raw` and sanity
|
||||
passes. Then save it
|
||||
A page is refined when `status` is `refined`, `version` is `1.0.0`
|
||||
or later, the quality signals are set and `llm.model` is not `none`,
|
||||
the page is listed in `wiki/index.md`, and `wiki/log.md` has a line
|
||||
for the refinement. The ingest is done when every page from the run
|
||||
meets that and `wiki sanity` passes. Then save it
|
||||
([save rule](#save-after-every-ingest-cycle-and-housekeeping-pass)).
|
||||
|
||||
### 16.5. Raw page frontmatter
|
||||
|
||||
@@ -1,97 +0,0 @@
|
||||
---
|
||||
name: obsidian
|
||||
description: Read, search, create, and edit notes in the Obsidian vault.
|
||||
version: 1.0.0
|
||||
author: Teknium (teknium1), Hermes Agent
|
||||
license: MIT
|
||||
platforms: [linux, macos, windows]
|
||||
metadata:
|
||||
hermes:
|
||||
tags: [Obsidian, Notes, Markdown, Vault]
|
||||
related_skills: []
|
||||
---
|
||||
|
||||
# Obsidian Vault
|
||||
|
||||
Use this skill for filesystem-first Obsidian vault work: reading notes, listing
|
||||
notes, searching note files, creating notes, appending content, and adding
|
||||
wikilinks.
|
||||
|
||||
## Vault path
|
||||
|
||||
Use a known or resolved vault path before calling file tools.
|
||||
|
||||
The documented vault-path convention is the `OBSIDIAN_VAULT_PATH` environment
|
||||
variable, for example from `${HERMES_HOME:-~/.hermes}/.env`. If it is unset,
|
||||
use `~/Documents/Obsidian Vault`.
|
||||
|
||||
File tools do not expand shell variables. Do not pass paths containing
|
||||
`$OBSIDIAN_VAULT_PATH` to `read_file`, `write_file`, `patch`, or
|
||||
`search_files`; resolve the vault path first and pass a concrete absolute path.
|
||||
Vault paths may contain spaces, which is another reason to prefer file tools
|
||||
over shell commands.
|
||||
|
||||
If the vault path is unknown, `terminal` is acceptable for resolving
|
||||
`OBSIDIAN_VAULT_PATH` or checking whether the fallback path exists. Once the
|
||||
path is known, switch back to file tools.
|
||||
|
||||
## Read a note
|
||||
|
||||
Use `read_file` with the resolved absolute path to the note. Prefer this over
|
||||
`cat` because it provides line numbers and pagination.
|
||||
|
||||
## List notes
|
||||
|
||||
Use `search_files` with `target: "files"` and the resolved vault path. Prefer
|
||||
this over `find` or `ls`.
|
||||
|
||||
- To list all markdown notes, use `pattern: "*.md"` under the vault path.
|
||||
- To list a subfolder, search under that subfolder's absolute path.
|
||||
|
||||
## Search
|
||||
|
||||
Use `search_files` for both filename and content searches. Prefer this over
|
||||
`grep`, `find`, or `ls`.
|
||||
|
||||
- For filenames, use `search_files` with `target: "files"` and a filename `pattern`.
|
||||
- For note contents, use `search_files` with `target: "content"`, the content
|
||||
regex as `pattern`, and `file_glob: "*.md"` when you want to restrict matches
|
||||
to markdown notes.
|
||||
|
||||
## Create a note
|
||||
|
||||
Use `write_file` with the resolved absolute path and the full markdown content.
|
||||
Prefer this over shell heredocs or `echo` because it avoids shell quoting
|
||||
issues and returns structured results.
|
||||
|
||||
## Append to a note
|
||||
|
||||
Prefer a native file-tool workflow when it is not awkward:
|
||||
|
||||
- Read the target note with `read_file`.
|
||||
- Use `patch` for an anchored append when there is stable context, such as
|
||||
adding a section after an existing heading or appending before a known
|
||||
trailing block.
|
||||
- Use `write_file` when rewriting the whole note is clearer than constructing a
|
||||
fragile patch.
|
||||
|
||||
For an anchored append with `patch`, replace the anchor with the anchor plus
|
||||
the new content.
|
||||
|
||||
For a simple append with no stable context, `terminal` is acceptable if it is
|
||||
the clearest safe option.
|
||||
|
||||
## Targeted edits
|
||||
|
||||
Use `patch` for focused note changes when the current content gives you stable
|
||||
context. Prefer this over shell text rewriting.
|
||||
|
||||
## Linking
|
||||
|
||||
For **compatibility** across tools (e.g., GitHub, VS Code, Obsidian), use
|
||||
**standard Markdown links** (`[link text](path/to/note.md)`) instead of
|
||||
wikilinks (`[[Note Name]]`).
|
||||
|
||||
- **Markdown links** ensure portability and work in all markdown renderers.
|
||||
- If Obsidian-specific features (e.g., graph view) are needed, wikilinks can be
|
||||
used **in addition to** Markdown links, but never as a replacement.
|
||||
@@ -0,0 +1,4 @@
|
||||
"""Wiki ingest package (SPEC 7.25).
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - stage modules for wiki ingest.
|
||||
"""
|
||||
@@ -0,0 +1,239 @@
|
||||
"""Archive a fetched file under ``raw/``.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.3. Never edits a file already there.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import logging
|
||||
import pathlib
|
||||
import shutil
|
||||
|
||||
from fiona.wiki.ingest.converters.base import ConverterRegistry
|
||||
from fiona.wiki.ingest.media import extension_for
|
||||
from fiona.wiki.ingest.models import ArchiveResult, FetchedSource, IngestError
|
||||
from fiona.wiki.ingest.settings import IngestSettings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class RawArchiver:
|
||||
"""Copy or move a source into ``raw/`` and reuse identical bytes.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SHA-256 reuse and collision names.
|
||||
"""
|
||||
|
||||
def __init__(self, wiki_root: pathlib.Path) -> None:
|
||||
"""Store the wiki root.
|
||||
|
||||
Args:
|
||||
wiki_root: Directory that contains ``raw/``.
|
||||
"""
|
||||
self.wiki_root = wiki_root
|
||||
|
||||
def find_by_sha256(
|
||||
self,
|
||||
digest: str,
|
||||
size: int,
|
||||
skip: pathlib.Path | None = None,
|
||||
) -> pathlib.Path | None:
|
||||
"""Return the first ``raw/`` file with this size and digest.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - only same-size files are hashed.
|
||||
|
||||
Args:
|
||||
digest: Lowercase SHA-256 hex.
|
||||
size: Byte size that a candidate must match.
|
||||
skip: A file to ignore, usually the source when it already sits in ``raw/``.
|
||||
|
||||
Returns:
|
||||
pathlib.Path | None: Match with the lowest path, or None.
|
||||
"""
|
||||
raw_root = self.wiki_root / "raw"
|
||||
if not raw_root.is_dir():
|
||||
return None
|
||||
skip_resolved = skip.resolve() if skip is not None else None
|
||||
matches: list[pathlib.Path] = []
|
||||
for path in raw_root.rglob("*"):
|
||||
if not path.is_file():
|
||||
continue
|
||||
if skip_resolved is not None and path.resolve() == skip_resolved:
|
||||
continue
|
||||
try:
|
||||
if path.stat().st_size != size:
|
||||
continue
|
||||
except OSError:
|
||||
continue
|
||||
if _sha256(path) == digest:
|
||||
matches.append(path)
|
||||
if not matches:
|
||||
return None
|
||||
matches.sort(key=lambda item: item.as_posix())
|
||||
return matches[0]
|
||||
|
||||
def archive(
|
||||
self,
|
||||
source: FetchedSource,
|
||||
*,
|
||||
registry: ConverterRegistry,
|
||||
settings: IngestSettings,
|
||||
raw_override: str | None,
|
||||
dry_run: bool,
|
||||
) -> ArchiveResult:
|
||||
"""Place the source under ``raw/`` or reuse an identical copy.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.3.
|
||||
|
||||
Args:
|
||||
source: Fetched file.
|
||||
registry: Used for the raw category and a missing extension.
|
||||
settings: ``raw_categories`` overrides.
|
||||
raw_override: ``-r`` value, relative to ``raw/``, or None.
|
||||
dry_run: When true, compute the path and write nothing.
|
||||
|
||||
Returns:
|
||||
ArchiveResult: Relative raw path, digest, and warnings.
|
||||
|
||||
Raises:
|
||||
IngestError: ``-r`` escapes ``raw/`` or is used on a file already there.
|
||||
"""
|
||||
raw_root = (self.wiki_root / "raw").resolve()
|
||||
source_in_raw = _is_inside(source.path, raw_root)
|
||||
if raw_override and source_in_raw:
|
||||
raise IngestError(
|
||||
f"--raw cannot be used on a file already under raw/: {source.argument}"
|
||||
)
|
||||
if raw_override:
|
||||
_require_inside(raw_root, raw_override, "--raw must stay inside raw/")
|
||||
digest = _sha256(source.path)
|
||||
match = self.find_by_sha256(
|
||||
digest,
|
||||
source.size_bytes,
|
||||
skip=source.path if source_in_raw else None,
|
||||
)
|
||||
warnings: list[str] = []
|
||||
if match is not None:
|
||||
raw_rel = _relative(match, self.wiki_root)
|
||||
if raw_override:
|
||||
warnings.append(f"already archived at {raw_rel}; --raw ignored")
|
||||
if _is_inside(source.path, self.wiki_root / "ingest") and not dry_run:
|
||||
source.path.unlink()
|
||||
logger.info("reused raw file %s", raw_rel)
|
||||
return ArchiveResult(match, raw_rel, digest, True, warnings)
|
||||
if source_in_raw:
|
||||
return ArchiveResult(source.path, _relative(source.path, self.wiki_root), digest, False, [])
|
||||
extension = source.path.suffix.lower() or extension_for(source.media_type, "", registry)
|
||||
dest = self._destination(source, registry, settings, raw_override, extension, raw_root)
|
||||
dest = _free_name(dest, source.slug, extension)
|
||||
if not dry_run:
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
if _is_inside(source.path, self.wiki_root / "ingest"):
|
||||
shutil.move(str(source.path), str(dest))
|
||||
else:
|
||||
shutil.copyfile(source.path, dest)
|
||||
logger.info("archived %s", dest)
|
||||
return ArchiveResult(dest, _relative(dest, self.wiki_root), digest, False, warnings)
|
||||
|
||||
def _destination(
|
||||
self,
|
||||
source: FetchedSource,
|
||||
registry: ConverterRegistry,
|
||||
settings: IngestSettings,
|
||||
raw_override: str | None,
|
||||
extension: str,
|
||||
raw_root: pathlib.Path,
|
||||
) -> pathlib.Path:
|
||||
"""Resolve the raw path before a collision suffix is applied.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - ``-r`` file or directory, else the category.
|
||||
"""
|
||||
filename = f"{source.slug}{extension}"
|
||||
if raw_override:
|
||||
relative = _strip_prefix(raw_override, "raw")
|
||||
suffix = pathlib.PurePosixPath(relative.replace("\\", "/")).suffix
|
||||
if suffix:
|
||||
return raw_root / relative
|
||||
return raw_root / relative / filename
|
||||
converter = registry.find(source.media_type, extension)
|
||||
if converter is None:
|
||||
category = "other"
|
||||
else:
|
||||
category = settings.raw_categories.get(converter.name, converter.raw_category)
|
||||
return raw_root / category / filename
|
||||
|
||||
|
||||
def _sha256(path: pathlib.Path) -> str:
|
||||
"""Return the lowercase SHA-256 hex digest of a file.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - streamed so large files stay cheap.
|
||||
"""
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
while True:
|
||||
chunk = handle.read(1024 * 1024)
|
||||
if not chunk:
|
||||
break
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def _free_name(dest: pathlib.Path, slug: str, extension: str) -> pathlib.Path:
|
||||
"""Return ``dest`` or the first free ``<slug>-N`` name beside it.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - never overwrite a raw file.
|
||||
"""
|
||||
if not dest.exists():
|
||||
return dest
|
||||
number = 2
|
||||
while True:
|
||||
candidate = dest.parent / f"{slug}-{number}{extension}"
|
||||
if not candidate.exists():
|
||||
return candidate
|
||||
number += 1
|
||||
|
||||
|
||||
def _strip_prefix(value: str, prefix: str) -> str:
|
||||
"""Drop a leading ``raw/`` or ``wiki/`` from a user path.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - both forms are accepted.
|
||||
"""
|
||||
relative = value.replace("\\", "/").lstrip("/")
|
||||
marker = f"{prefix}/"
|
||||
if relative == prefix:
|
||||
return ""
|
||||
if relative.startswith(marker):
|
||||
return relative[len(marker):]
|
||||
return relative
|
||||
|
||||
|
||||
def _require_inside(root: pathlib.Path, value: str, message: str) -> pathlib.Path:
|
||||
"""Resolve ``value`` under ``root`` or raise.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - ``..`` and symlinks must stay inside.
|
||||
"""
|
||||
relative = _strip_prefix(value, root.name)
|
||||
candidate = (root / relative).resolve()
|
||||
if not _is_inside(candidate, root):
|
||||
raise IngestError(f"{message}: {value}")
|
||||
return candidate
|
||||
|
||||
|
||||
def _is_inside(path: pathlib.Path, root: pathlib.Path) -> bool:
|
||||
"""Return whether ``path`` is ``root`` or a file inside it.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - compare resolved paths.
|
||||
"""
|
||||
try:
|
||||
path.resolve().relative_to(root.resolve())
|
||||
except ValueError:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _relative(path: pathlib.Path, root: pathlib.Path) -> str:
|
||||
"""Return a wiki-relative path with ``/`` separators.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - JSON and log paths.
|
||||
"""
|
||||
return path.resolve().relative_to(root.resolve()).as_posix()
|
||||
@@ -0,0 +1,26 @@
|
||||
"""Shipped converters, in registration order.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - markdown, text, html, then pdf.
|
||||
"""
|
||||
|
||||
from fiona.wiki.ingest.converters.base import ConverterRegistry
|
||||
from fiona.wiki.ingest.converters.html import HtmlConverter
|
||||
from fiona.wiki.ingest.converters.markdown import MarkdownConverter
|
||||
from fiona.wiki.ingest.converters.pdf import PdfConverter
|
||||
from fiona.wiki.ingest.converters.text import TextConverter
|
||||
|
||||
|
||||
def default_registry() -> ConverterRegistry:
|
||||
"""Return the converters shipped with wiki ingest.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - registration order from SPEC 7.25.
|
||||
|
||||
Returns:
|
||||
ConverterRegistry: Markdown, text, HTML, and PDF.
|
||||
"""
|
||||
registry = ConverterRegistry()
|
||||
registry.register(MarkdownConverter)
|
||||
registry.register(TextConverter)
|
||||
registry.register(HtmlConverter)
|
||||
registry.register(PdfConverter)
|
||||
return registry
|
||||
@@ -0,0 +1,108 @@
|
||||
"""Converter base types and the registry.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - one class per format, SPEC 7.25.1.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import abc
|
||||
import typing
|
||||
|
||||
from fiona.wiki.ingest.models import ConversionResult, FetchedSource
|
||||
|
||||
|
||||
class ConversionError(Exception):
|
||||
"""Raised when a converter cannot read a file it claims.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - exit 3, raw copy kept.
|
||||
"""
|
||||
|
||||
def __init__(self, converter: str, reason: str) -> None:
|
||||
"""Store the converter name and the reason.
|
||||
|
||||
Args:
|
||||
converter: Converter ``name`` attribute.
|
||||
reason: Short failure text, without the ``Error:`` prefix.
|
||||
"""
|
||||
super().__init__(f"{converter}: {reason}")
|
||||
self.converter = converter
|
||||
self.reason = reason
|
||||
|
||||
|
||||
class SourceConverter(abc.ABC):
|
||||
"""One format. Subclasses declare what they claim and implement ``convert``.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.1.2.
|
||||
"""
|
||||
|
||||
name: typing.ClassVar[str]
|
||||
media_types: typing.ClassVar[tuple[str, ...]]
|
||||
extensions: typing.ClassVar[tuple[str, ...]]
|
||||
raw_category: typing.ClassVar[str]
|
||||
|
||||
@abc.abstractmethod
|
||||
def convert(self, source: FetchedSource) -> ConversionResult:
|
||||
"""Convert an archived file to markdown.
|
||||
|
||||
Args:
|
||||
source: Fetched file, including media type and slug.
|
||||
|
||||
Returns:
|
||||
ConversionResult: Body, title, metadata, and warnings.
|
||||
|
||||
Raises:
|
||||
ConversionError: The file cannot be read.
|
||||
"""
|
||||
|
||||
|
||||
class ConverterRegistry:
|
||||
"""Ordered list of converters.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - first media-type match, then extension.
|
||||
"""
|
||||
|
||||
def __init__(self, converters: list[SourceConverter] | None = None) -> None:
|
||||
"""Start with an optional list of converter instances.
|
||||
|
||||
Args:
|
||||
converters: Instances in registration order.
|
||||
"""
|
||||
self.converters: list[SourceConverter] = list(converters or [])
|
||||
|
||||
def register(self, converter: SourceConverter | type[SourceConverter]) -> None:
|
||||
"""Append a converter class or instance.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - adding a format is one registration.
|
||||
|
||||
Args:
|
||||
converter: Class with a no-arg constructor, or a ready instance.
|
||||
"""
|
||||
if isinstance(converter, type):
|
||||
converter = converter()
|
||||
self.converters.append(converter)
|
||||
|
||||
def find(self, media_type: str, extension: str) -> SourceConverter | None:
|
||||
"""Pick the converter for a media type and extension.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.4.1 registration order.
|
||||
|
||||
Args:
|
||||
media_type: Detected type, or empty.
|
||||
extension: Suffix including the dot, or empty.
|
||||
|
||||
Returns:
|
||||
SourceConverter | None: The first match, or None.
|
||||
"""
|
||||
media = (media_type or "").lower()
|
||||
ext = extension.lower()
|
||||
if ext and not ext.startswith("."):
|
||||
ext = f".{ext}"
|
||||
if media:
|
||||
for converter in self.converters:
|
||||
if media in converter.media_types:
|
||||
return converter
|
||||
if ext:
|
||||
for converter in self.converters:
|
||||
if ext in converter.extensions:
|
||||
return converter
|
||||
return None
|
||||
@@ -0,0 +1,128 @@
|
||||
"""HTML source converter.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - article text via beautifulsoup4 and markdownify.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import typing
|
||||
import urllib.parse
|
||||
|
||||
import bs4
|
||||
import markdownify
|
||||
|
||||
from fiona.wiki.ingest.converters.base import SourceConverter
|
||||
from fiona.wiki.ingest.media import decode_bytes
|
||||
from fiona.wiki.ingest.models import ConversionResult, FetchedSource
|
||||
|
||||
_DROP_TAGS = ("script", "style", "noscript", "template")
|
||||
_NO_TEXT = "no text content; the page may need JavaScript or a login"
|
||||
|
||||
|
||||
class HtmlConverter(SourceConverter):
|
||||
"""Convert an HTML document to markdown.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.4.5.
|
||||
"""
|
||||
|
||||
name: typing.ClassVar[str] = "html"
|
||||
media_types: typing.ClassVar[tuple[str, ...]] = ("text/html", "application/xhtml+xml")
|
||||
extensions: typing.ClassVar[tuple[str, ...]] = (".html", ".htm", ".xhtml")
|
||||
raw_category: typing.ClassVar[str] = "articles"
|
||||
|
||||
def convert(self, source: FetchedSource) -> ConversionResult:
|
||||
"""Parse HTML and return the article body as markdown.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - drop scripts and absolutize links.
|
||||
|
||||
Args:
|
||||
source: Archived HTML file. ``origin`` is the page URL when fetched.
|
||||
|
||||
Returns:
|
||||
ConversionResult: Markdown body, title, and ``source_meta``.
|
||||
"""
|
||||
text, warnings = decode_bytes(source.path.read_bytes(), source.charset)
|
||||
soup = bs4.BeautifulSoup(text, "html.parser")
|
||||
for tag in soup.find_all(_DROP_TAGS):
|
||||
tag.decompose()
|
||||
for comment in soup.find_all(string=lambda node: isinstance(node, bs4.Comment)):
|
||||
comment.extract()
|
||||
root = soup.find("article") or soup.find("main") or soup.body or soup
|
||||
origin = source.origin if source.origin_kind == "url" else None
|
||||
_absolutize(root, soup, origin)
|
||||
visible = root.get_text(strip=True)
|
||||
heading = getattr(markdownify, "ATX", "ATX")
|
||||
body = markdownify.markdownify(str(root), heading_style=heading, bullets="-").strip()
|
||||
if body:
|
||||
body += "\n"
|
||||
meta = _source_meta(soup)
|
||||
title = _html_title(soup, root)
|
||||
conversion = "complete"
|
||||
if not visible:
|
||||
conversion = "partial"
|
||||
warnings = [*warnings, _NO_TEXT]
|
||||
return ConversionResult(
|
||||
body=body,
|
||||
title=title,
|
||||
source_meta=meta,
|
||||
conversion=conversion,
|
||||
warnings=warnings,
|
||||
converter=self.name,
|
||||
)
|
||||
|
||||
|
||||
def _absolutize(root: bs4.element.Tag, soup: bs4.BeautifulSoup, origin: str | None) -> None:
|
||||
"""Rewrite relative href and src values against base href, then the origin.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - a local file with no origin keeps relative links.
|
||||
"""
|
||||
base_tag = soup.find("base", href=True)
|
||||
base = base_tag.get("href") if base_tag is not None else None
|
||||
if not base:
|
||||
base = origin
|
||||
if not base:
|
||||
return
|
||||
for tag in root.find_all(href=True):
|
||||
tag["href"] = urllib.parse.urljoin(str(base), str(tag["href"]))
|
||||
for tag in root.find_all(src=True):
|
||||
tag["src"] = urllib.parse.urljoin(str(base), str(tag["src"]))
|
||||
|
||||
|
||||
def _html_title(soup: bs4.BeautifulSoup, root: bs4.element.Tag) -> str | None:
|
||||
"""Return ``<title>``, then ``og:title``, then the first ``<h1>``.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.4.5.1.
|
||||
"""
|
||||
if soup.title and soup.title.string and soup.title.string.strip():
|
||||
return soup.title.string.strip()
|
||||
og = soup.find("meta", attrs={"property": "og:title"})
|
||||
if og is not None and og.get("content"):
|
||||
return str(og["content"]).strip() or None
|
||||
heading = root.find("h1")
|
||||
if heading is not None:
|
||||
text = heading.get_text(strip=True)
|
||||
return text or None
|
||||
return None
|
||||
|
||||
|
||||
def _source_meta(soup: bs4.BeautifulSoup) -> dict:
|
||||
"""Collect language, canonical URL, and selected meta values.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - only keys that are present.
|
||||
"""
|
||||
meta: dict[str, str] = {}
|
||||
root = soup.find("html")
|
||||
if root is not None and root.get("lang"):
|
||||
meta["lang"] = str(root["lang"])
|
||||
canonical = soup.find("link", rel=lambda value: value and "canonical" in value)
|
||||
if canonical is not None and canonical.get("href"):
|
||||
meta["canonical"] = str(canonical["href"])
|
||||
for tag in soup.find_all("meta"):
|
||||
key = tag.get("name") or tag.get("property")
|
||||
content = tag.get("content")
|
||||
if not key or content is None:
|
||||
continue
|
||||
name = str(key)
|
||||
if name in {"description", "author"} or name.startswith("og:") or name == "article:published_time":
|
||||
meta[name] = str(content)
|
||||
return meta
|
||||
@@ -0,0 +1,105 @@
|
||||
"""Markdown source converter.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - keep the body and lift frontmatter into source_meta.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import typing
|
||||
|
||||
import yaml
|
||||
|
||||
from fiona.wiki.ingest.converters.base import SourceConverter
|
||||
from fiona.wiki.ingest.media import decode_bytes
|
||||
from fiona.wiki.ingest.models import ConversionResult, FetchedSource
|
||||
|
||||
_FRONTMATTER = re.compile(r"\A---\r?\n(.*?)\r?\n---\r?\n?", re.DOTALL)
|
||||
_H1 = re.compile(r"^#\s+(\S.*)$")
|
||||
_FENCE = re.compile(r"^\s*(`{3,}|~{3,})")
|
||||
|
||||
|
||||
class MarkdownConverter(SourceConverter):
|
||||
"""Keep markdown as it is and record any source frontmatter.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.4.3.
|
||||
|
||||
The body after a frontmatter block is kept character for character.
|
||||
The page writer adds the wiki frontmatter when the new page does not
|
||||
have it yet. Source keys move to ``source_meta`` instead of being
|
||||
written a second time as the page frontmatter.
|
||||
"""
|
||||
|
||||
name: typing.ClassVar[str] = "markdown"
|
||||
media_types: typing.ClassVar[tuple[str, ...]] = ("text/markdown",)
|
||||
extensions: typing.ClassVar[tuple[str, ...]] = (".md", ".markdown")
|
||||
raw_category: typing.ClassVar[str] = "articles"
|
||||
|
||||
def convert(self, source: FetchedSource) -> ConversionResult:
|
||||
"""Return the markdown body and any frontmatter as metadata.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - title from frontmatter, then H1, then slug.
|
||||
|
||||
Args:
|
||||
source: Archived markdown file.
|
||||
|
||||
Returns:
|
||||
ConversionResult: Unchanged body, ``complete`` unless decoding fell back.
|
||||
"""
|
||||
text, warnings = decode_bytes(source.path.read_bytes(), source.charset)
|
||||
meta: dict = {}
|
||||
body = text
|
||||
match = _FRONTMATTER.match(text)
|
||||
if match:
|
||||
loaded = _load_meta(match.group(1))
|
||||
if loaded is None:
|
||||
warnings = [*warnings, "source frontmatter did not parse"]
|
||||
else:
|
||||
meta = loaded
|
||||
body = text[match.end():]
|
||||
title = _title(meta, body, source.slug)
|
||||
return ConversionResult(
|
||||
body=body,
|
||||
title=title,
|
||||
source_meta=meta,
|
||||
conversion="complete",
|
||||
warnings=warnings,
|
||||
converter=self.name,
|
||||
)
|
||||
|
||||
|
||||
def _load_meta(block: str) -> dict | None:
|
||||
"""Parse a frontmatter block into a dict.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - invalid YAML stays in the body.
|
||||
"""
|
||||
try:
|
||||
loaded = yaml.safe_load(block)
|
||||
except yaml.YAMLError:
|
||||
return None
|
||||
if loaded is None:
|
||||
return {}
|
||||
if isinstance(loaded, dict):
|
||||
return loaded
|
||||
return None
|
||||
|
||||
|
||||
def _title(meta: dict, body: str, slug: str) -> str:
|
||||
"""Choose the page title.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - frontmatter title, first H1, then slug.
|
||||
"""
|
||||
raw_title = meta.get("title")
|
||||
if isinstance(raw_title, str) and raw_title.strip():
|
||||
return raw_title.strip()
|
||||
fenced = False
|
||||
for line in body.splitlines():
|
||||
if _FENCE.match(line):
|
||||
fenced = not fenced
|
||||
continue
|
||||
if fenced:
|
||||
continue
|
||||
match = _H1.match(line)
|
||||
if match:
|
||||
return match.group(1).strip()
|
||||
return slug
|
||||
@@ -0,0 +1,186 @@
|
||||
"""PDF source converter.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - text layer via pypdf, SPEC 7.25.4.6.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime
|
||||
import typing
|
||||
|
||||
import pypdf
|
||||
import pypdf.errors
|
||||
|
||||
from fiona.wiki.ingest.converters.base import ConversionError, SourceConverter
|
||||
from fiona.wiki.ingest.models import ConversionResult, FetchedSource
|
||||
|
||||
_SCANNED = "no text layer; the PDF may be scanned"
|
||||
|
||||
|
||||
class PdfConverter(SourceConverter):
|
||||
"""Read a PDF text layer into markdown headings and paragraphs.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.4.6.
|
||||
"""
|
||||
|
||||
name: typing.ClassVar[str] = "pdf"
|
||||
media_types: typing.ClassVar[tuple[str, ...]] = ("application/pdf",)
|
||||
extensions: typing.ClassVar[tuple[str, ...]] = (".pdf",)
|
||||
raw_category: typing.ClassVar[str] = "papers"
|
||||
|
||||
def convert(self, source: FetchedSource) -> ConversionResult:
|
||||
"""Extract page text, outline headings, and document metadata.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - encrypted or corrupt files raise.
|
||||
|
||||
Args:
|
||||
source: Archived PDF.
|
||||
|
||||
Returns:
|
||||
ConversionResult: Markdown body. Empty pages make it ``partial``.
|
||||
|
||||
Raises:
|
||||
ConversionError: The PDF is encrypted or cannot be parsed.
|
||||
"""
|
||||
try:
|
||||
reader = pypdf.PdfReader(str(source.path))
|
||||
except pypdf.errors.PdfReadError as exc:
|
||||
raise ConversionError(self.name, str(exc)) from exc
|
||||
if reader.is_encrypted:
|
||||
raise ConversionError(self.name, "encrypted PDF")
|
||||
meta = _metadata(reader)
|
||||
title = meta.get("title") or source.slug
|
||||
outline = _outline(reader)
|
||||
pages = []
|
||||
empty: list[int] = []
|
||||
for index, page in enumerate(reader.pages, start=1):
|
||||
try:
|
||||
text = page.extract_text() or ""
|
||||
except pypdf.errors.PdfReadError as exc:
|
||||
raise ConversionError(self.name, str(exc)) from exc
|
||||
pages.append(text.strip())
|
||||
if not text.strip():
|
||||
empty.append(index)
|
||||
body = _render(pages, outline)
|
||||
warnings: list[str] = []
|
||||
conversion = "complete"
|
||||
if empty and len(empty) == len(pages):
|
||||
warnings.append(_SCANNED)
|
||||
conversion = "partial"
|
||||
elif empty:
|
||||
warnings.extend(f"page {number} has no text layer" for number in empty)
|
||||
conversion = "partial"
|
||||
return ConversionResult(
|
||||
body=body,
|
||||
title=str(title),
|
||||
source_meta=meta,
|
||||
conversion=conversion,
|
||||
warnings=warnings,
|
||||
converter=self.name,
|
||||
)
|
||||
|
||||
|
||||
def _metadata(reader: pypdf.PdfReader) -> dict:
|
||||
"""Return page count and the document info values that are present.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - dates as ISO 8601.
|
||||
"""
|
||||
meta: dict[str, object] = {"page_count": len(reader.pages)}
|
||||
info = reader.metadata
|
||||
if info is None:
|
||||
return meta
|
||||
mapping = (
|
||||
("title", info.title),
|
||||
("author", info.author),
|
||||
("subject", info.subject),
|
||||
("keywords", info.keywords),
|
||||
("creation_date", _iso_date(info.creation_date)),
|
||||
("mod_date", _iso_date(info.modification_date)),
|
||||
)
|
||||
for key, value in mapping:
|
||||
if value:
|
||||
meta[key] = value
|
||||
return meta
|
||||
|
||||
|
||||
def _iso_date(value: object) -> str | None:
|
||||
"""Format a PDF date as ISO 8601.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - accept datetime values and ``D:`` strings.
|
||||
"""
|
||||
if value is None:
|
||||
return None
|
||||
if isinstance(value, datetime.datetime):
|
||||
if value.tzinfo is None:
|
||||
return value.strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
return value.astimezone(datetime.UTC).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
text = str(value).strip().removeprefix("D:")
|
||||
digits = "".join(char for char in text if char.isdigit())
|
||||
if len(digits) < 8:
|
||||
return text or None
|
||||
year, month, day = digits[0:4], digits[4:6], digits[6:8]
|
||||
hour = digits[8:10] or "00"
|
||||
minute = digits[10:12] or "00"
|
||||
second = digits[12:14] or "00"
|
||||
return f"{year}-{month}-{day}T{hour}:{minute}:{second}Z"
|
||||
|
||||
|
||||
def _outline(reader: pypdf.PdfReader) -> list[tuple[int, int, str]]:
|
||||
"""Return outline entries as page index, heading level, and title.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - level is depth plus 2, capped at 6.
|
||||
"""
|
||||
found: list[tuple[int, int, str]] = []
|
||||
try:
|
||||
items = reader.outline
|
||||
except pypdf.errors.PdfReadError:
|
||||
return found
|
||||
_walk_outline(reader, items, 0, found)
|
||||
return found
|
||||
|
||||
|
||||
def _walk_outline(
|
||||
reader: pypdf.PdfReader,
|
||||
items: object,
|
||||
depth: int,
|
||||
found: list[tuple[int, int, str]],
|
||||
) -> None:
|
||||
"""Walk a pypdf outline tree.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - nested lists are deeper entries.
|
||||
"""
|
||||
if not items:
|
||||
return
|
||||
for item in items: # type: ignore[union-attr]
|
||||
if isinstance(item, list):
|
||||
_walk_outline(reader, item, depth + 1, found)
|
||||
continue
|
||||
try:
|
||||
page_index = reader.get_destination_page_number(item)
|
||||
except (ValueError, KeyError, pypdf.errors.PdfReadError):
|
||||
continue
|
||||
title = getattr(item, "title", "") or ""
|
||||
level = min(depth + 2, 6)
|
||||
found.append((page_index, level, str(title).strip() or "Untitled"))
|
||||
|
||||
|
||||
def _render(pages: list[str], outline: list[tuple[int, int, str]]) -> str:
|
||||
"""Render pages with outline headings or ``## Page n`` headings.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.4.6.2.
|
||||
"""
|
||||
by_page: dict[int, list[str]] = {}
|
||||
for page_index, level, title in outline:
|
||||
by_page.setdefault(page_index, []).append(f"{'#' * level} {title}")
|
||||
chunks: list[str] = []
|
||||
for index, text in enumerate(pages):
|
||||
if outline:
|
||||
chunks.extend(by_page.get(index, []))
|
||||
else:
|
||||
chunks.append(f"## Page {index + 1}")
|
||||
if text:
|
||||
chunks.append(text)
|
||||
body = "\n\n".join(chunks)
|
||||
if body:
|
||||
body += "\n"
|
||||
return body
|
||||
@@ -0,0 +1,68 @@
|
||||
"""Plain-text source converter.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - wrap text in a fence; the agent refines the page.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import typing
|
||||
|
||||
from fiona.wiki.ingest.converters.base import SourceConverter
|
||||
from fiona.wiki.ingest.media import decode_bytes
|
||||
from fiona.wiki.ingest.models import ConversionResult, FetchedSource
|
||||
|
||||
|
||||
class TextConverter(SourceConverter):
|
||||
"""Turn plain text into one fenced markdown block.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.4.4.
|
||||
|
||||
The page is written at ``status: raw``. The tool does not summarize
|
||||
the text. The agent rewrites the fence during refinement.
|
||||
"""
|
||||
|
||||
name: typing.ClassVar[str] = "text"
|
||||
media_types: typing.ClassVar[tuple[str, ...]] = ("text/plain",)
|
||||
extensions: typing.ClassVar[tuple[str, ...]] = (".txt", ".text", ".log")
|
||||
raw_category: typing.ClassVar[str] = "articles"
|
||||
|
||||
def convert(self, source: FetchedSource) -> ConversionResult:
|
||||
"""Decode the file and wrap it in a fence longer than any backtick run.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - title is the slug; fallback decode warns.
|
||||
|
||||
Args:
|
||||
source: Archived text file.
|
||||
|
||||
Returns:
|
||||
ConversionResult: Fenced body and the slug as the title.
|
||||
"""
|
||||
text, warnings = decode_bytes(source.path.read_bytes(), source.charset)
|
||||
marker = "`" * _fence_length(text)
|
||||
if text and not text.endswith("\n"):
|
||||
text += "\n"
|
||||
body = f"{marker}text\n{text}{marker}\n"
|
||||
return ConversionResult(
|
||||
body=body,
|
||||
title=source.slug,
|
||||
source_meta={},
|
||||
conversion="complete",
|
||||
warnings=warnings,
|
||||
converter=self.name,
|
||||
)
|
||||
|
||||
|
||||
def _fence_length(text: str) -> int:
|
||||
"""Return a fence length longer than the longest backtick run.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - a run of three when the text has fewer.
|
||||
"""
|
||||
longest = 0
|
||||
current = 0
|
||||
for char in text:
|
||||
if char == "`":
|
||||
current += 1
|
||||
longest = max(longest, current)
|
||||
else:
|
||||
current = 0
|
||||
return max(3, longest + 1)
|
||||
@@ -0,0 +1,276 @@
|
||||
"""Fetch a local file or an HTTP URL.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.2. No credentials are sent.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import pathlib
|
||||
import re
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
|
||||
from fiona.wiki.ingest.converters.base import ConverterRegistry
|
||||
from fiona.wiki.ingest.media import (
|
||||
detect_media_type,
|
||||
extension_for,
|
||||
slug_from_file,
|
||||
slug_from_url,
|
||||
utc_now,
|
||||
)
|
||||
from fiona.wiki.ingest.models import FetchedSource, IngestError
|
||||
from fiona.wiki.ingest.settings import IngestSettings, default_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_SCHEME = re.compile(r"^([A-Za-z][A-Za-z0-9+.-]*)://")
|
||||
_WINDOWS_DRIVE = re.compile(r"^[A-Za-z]:[\\/]")
|
||||
|
||||
|
||||
def url_scheme(argument: str) -> str | None:
|
||||
"""Return the URL scheme, or None for a local path.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - a Windows drive letter is not a scheme.
|
||||
|
||||
Args:
|
||||
argument: The ``<source>`` argument as given.
|
||||
|
||||
Returns:
|
||||
str | None: Scheme text, preserving case, or None.
|
||||
"""
|
||||
if _WINDOWS_DRIVE.match(argument):
|
||||
return None
|
||||
match = _SCHEME.match(argument)
|
||||
if match is None:
|
||||
return None
|
||||
return match.group(1)
|
||||
|
||||
|
||||
class SourceFetcher:
|
||||
"""Base fetcher. Subclasses accept an argument and return a local file.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.1.3.
|
||||
"""
|
||||
|
||||
settings: IngestSettings | None = None
|
||||
|
||||
def accepts(self, argument: str) -> bool:
|
||||
"""Return whether this fetcher handles the argument.
|
||||
|
||||
Args:
|
||||
argument: The ``<source>`` argument as given.
|
||||
|
||||
Returns:
|
||||
bool: True when ``fetch`` can run.
|
||||
"""
|
||||
raise NotImplementedError
|
||||
|
||||
def fetch(self, argument: str, dest_dir: pathlib.Path) -> FetchedSource:
|
||||
"""Fetch the argument into ``dest_dir`` when a download is required.
|
||||
|
||||
Args:
|
||||
argument: The ``<source>`` argument as given.
|
||||
dest_dir: Directory for a downloaded file. Local fetches ignore it.
|
||||
|
||||
Returns:
|
||||
FetchedSource: Local path and origin facts.
|
||||
"""
|
||||
raise NotImplementedError
|
||||
|
||||
|
||||
class LocalFileFetcher(SourceFetcher):
|
||||
"""Read a local file in place.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - missing files and directories exit 1.
|
||||
"""
|
||||
|
||||
def __init__(self, registry: ConverterRegistry) -> None:
|
||||
"""Store the registry used to name the media type.
|
||||
|
||||
Args:
|
||||
registry: Converter registry for extension lookup.
|
||||
"""
|
||||
self.registry = registry
|
||||
self.settings = None
|
||||
|
||||
def accepts(self, argument: str) -> bool:
|
||||
"""Accept any argument that is not a URL.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - scheme detection stays in ``url_scheme``.
|
||||
|
||||
Args:
|
||||
argument: The ``<source>`` argument as given.
|
||||
|
||||
Returns:
|
||||
bool: True when there is no ``://`` scheme.
|
||||
"""
|
||||
return url_scheme(argument) is None
|
||||
|
||||
def fetch(self, argument: str, dest_dir: pathlib.Path) -> FetchedSource:
|
||||
"""Resolve a local path without copying it.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.2.1.
|
||||
|
||||
Args:
|
||||
argument: Local path as given.
|
||||
dest_dir: Unused. Present so every fetcher shares one signature.
|
||||
|
||||
Returns:
|
||||
FetchedSource: The resolved file. ``created`` is false.
|
||||
|
||||
Raises:
|
||||
IngestError: The path is missing or is a directory.
|
||||
"""
|
||||
del dest_dir
|
||||
path = pathlib.Path(argument)
|
||||
if not path.exists():
|
||||
raise IngestError(f"Source not found: {argument}")
|
||||
if path.is_dir():
|
||||
raise IngestError(f"Source is a directory: {argument}")
|
||||
data = path.read_bytes()
|
||||
media, charset = detect_media_type(data, path.suffix, None, self.registry)
|
||||
resolved = path.resolve()
|
||||
logger.info("fetched local file %s as %s", resolved, media)
|
||||
return FetchedSource(
|
||||
argument=argument,
|
||||
path=resolved,
|
||||
origin=str(resolved),
|
||||
origin_kind="file",
|
||||
final_url=None,
|
||||
media_type=media,
|
||||
charset=charset,
|
||||
size_bytes=len(data),
|
||||
slug=slug_from_file(path),
|
||||
fetched=utc_now(),
|
||||
created=False,
|
||||
)
|
||||
|
||||
|
||||
class UrlFetcher(SourceFetcher):
|
||||
"""Download one ``http`` or ``https`` URL.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - opener is injected; tests make no network call.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
registry: ConverterRegistry,
|
||||
opener: urllib.request.OpenerDirector | None = None,
|
||||
) -> None:
|
||||
"""Store the registry and the URL opener.
|
||||
|
||||
Args:
|
||||
registry: Converter registry for the saved extension.
|
||||
opener: ``urllib`` opener. The default follows redirects and proxies.
|
||||
"""
|
||||
self.registry = registry
|
||||
self.opener = opener or urllib.request.build_opener()
|
||||
self.settings = None
|
||||
|
||||
def accepts(self, argument: str) -> bool:
|
||||
"""Accept ``http`` and ``https`` only.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - other schemes are rejected by the pipeline.
|
||||
|
||||
Args:
|
||||
argument: The ``<source>`` argument as given.
|
||||
|
||||
Returns:
|
||||
bool: True for http and https.
|
||||
"""
|
||||
scheme = url_scheme(argument)
|
||||
return scheme is not None and scheme.lower() in {"http", "https"}
|
||||
|
||||
def fetch(self, argument: str, dest_dir: pathlib.Path) -> FetchedSource:
|
||||
"""Download a URL into ``dest_dir`` and stop past the size limit.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.2.2 and 7.25.2.3.
|
||||
|
||||
Args:
|
||||
argument: Absolute URL.
|
||||
dest_dir: Directory that receives ``<slug>.<ext>``.
|
||||
|
||||
Returns:
|
||||
FetchedSource: The saved file. ``created`` is true.
|
||||
|
||||
Raises:
|
||||
IngestError: The response is not 2xx, the network fails, or the body is too large.
|
||||
"""
|
||||
settings = self.settings or default_settings()
|
||||
request = urllib.request.Request(
|
||||
_without_userinfo(argument),
|
||||
headers={"User-Agent": settings.user_agent},
|
||||
)
|
||||
try:
|
||||
response = self.opener.open(request, timeout=settings.timeout_seconds)
|
||||
except urllib.error.HTTPError as exc:
|
||||
raise IngestError(f"Download failed: {argument}: {exc.code} {exc.reason}") from exc
|
||||
except urllib.error.URLError as exc:
|
||||
raise IngestError(f"Download failed: {argument}: {exc.reason}") from exc
|
||||
with response:
|
||||
final_url = response.geturl()
|
||||
header = response.headers.get("Content-Type")
|
||||
body = _read_limited(response, settings.max_download_bytes, argument)
|
||||
parsed = urllib.parse.urlsplit(argument)
|
||||
suffix = pathlib.PurePosixPath(parsed.path).suffix
|
||||
media, charset = detect_media_type(body, suffix, header, self.registry)
|
||||
extension = extension_for(media, suffix, self.registry)
|
||||
slug = slug_from_url(argument)
|
||||
dest_dir.mkdir(parents=True, exist_ok=True)
|
||||
target = dest_dir / f"{slug}{extension}"
|
||||
partial = target.with_name(target.name + ".partial")
|
||||
try:
|
||||
partial.write_bytes(body)
|
||||
partial.replace(target)
|
||||
except OSError:
|
||||
if partial.exists():
|
||||
partial.unlink()
|
||||
raise
|
||||
logger.info("downloaded %s as %s (%s bytes)", argument, media, len(body))
|
||||
return FetchedSource(
|
||||
argument=argument,
|
||||
path=target,
|
||||
origin=argument,
|
||||
origin_kind="url",
|
||||
final_url=final_url,
|
||||
media_type=media,
|
||||
charset=charset,
|
||||
size_bytes=len(body),
|
||||
slug=slug,
|
||||
fetched=utc_now(),
|
||||
created=True,
|
||||
)
|
||||
|
||||
|
||||
def _without_userinfo(url: str) -> str:
|
||||
"""Drop userinfo so the request sends no credentials.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.2.2.
|
||||
"""
|
||||
parts = urllib.parse.urlsplit(url)
|
||||
if parts.username is None and parts.password is None:
|
||||
return url
|
||||
host = parts.hostname or ""
|
||||
if parts.port:
|
||||
host = f"{host}:{parts.port}"
|
||||
return urllib.parse.urlunsplit((parts.scheme, host, parts.path, parts.query, parts.fragment))
|
||||
|
||||
|
||||
def _read_limited(response: object, limit: int, url: str) -> bytes:
|
||||
"""Read a response body, refusing anything over ``limit``.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - a partial download is not returned.
|
||||
"""
|
||||
chunks: list[bytes] = []
|
||||
total = 0
|
||||
while True:
|
||||
block = response.read(65536) # type: ignore[attr-defined]
|
||||
if not block:
|
||||
break
|
||||
total += len(block)
|
||||
if total > limit:
|
||||
raise IngestError(f"Download exceeds {limit} bytes: {url}")
|
||||
chunks.append(block)
|
||||
return b"".join(chunks)
|
||||
@@ -0,0 +1,96 @@
|
||||
"""Append one ingest line to ``wiki/log.md``.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.6. Does not edit ``index.md``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import pathlib
|
||||
import re
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_VERSION = re.compile(r"^version:\s*(\S+)\s*$", re.MULTILINE)
|
||||
_UPDATED = re.compile(r"^updated:\s*(\S+)\s*$", re.MULTILINE)
|
||||
|
||||
|
||||
class IngestLog:
|
||||
"""Record a finished archive in ``wiki/log.md``.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - bump patch version and ``updated``.
|
||||
"""
|
||||
|
||||
def __init__(self, wiki_root: pathlib.Path) -> None:
|
||||
"""Store the wiki root.
|
||||
|
||||
Args:
|
||||
wiki_root: Directory that contains ``wiki/log.md``.
|
||||
"""
|
||||
self.wiki_root = wiki_root
|
||||
|
||||
def append(
|
||||
self,
|
||||
*,
|
||||
day: str,
|
||||
origin: str,
|
||||
raw_rel: str,
|
||||
page_rel: str | None,
|
||||
dry_run: bool,
|
||||
) -> None:
|
||||
"""Append one bullet and bump the log frontmatter.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - ``(raw)`` or ``(no page)``.
|
||||
|
||||
Args:
|
||||
day: UTC date ``YYYY-MM-DD``.
|
||||
origin: URL or resolved file path.
|
||||
raw_rel: Raw path relative to the wiki root.
|
||||
page_rel: Page path relative to the wiki root, or None.
|
||||
dry_run: When true, leave the log untouched.
|
||||
"""
|
||||
if dry_run:
|
||||
return
|
||||
path = self.wiki_root / "wiki" / "log.md"
|
||||
if page_rel:
|
||||
bullet = f"- {day} - ingest {origin} -> {raw_rel} -> {page_rel} (raw)"
|
||||
else:
|
||||
bullet = f"- {day} - ingest {origin} -> {raw_rel} (no page)"
|
||||
if not path.is_file():
|
||||
logger.warning("wiki/log.md is missing; ingest was not recorded")
|
||||
return
|
||||
text = path.read_text(encoding="utf-8")
|
||||
text = _bump_version(text)
|
||||
text = _set_updated(text, day)
|
||||
if not text.endswith("\n"):
|
||||
text += "\n"
|
||||
text += bullet + "\n"
|
||||
path.write_text(text, encoding="utf-8", newline="\n")
|
||||
logger.info("recorded %s", bullet)
|
||||
|
||||
|
||||
def _bump_version(text: str) -> str:
|
||||
"""Increment the patch number of the first ``version`` field.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - ``2.0.0`` becomes ``2.0.1``.
|
||||
"""
|
||||
match = _VERSION.search(text)
|
||||
if match is None:
|
||||
return text
|
||||
parts = match.group(1).split(".")
|
||||
if len(parts) == 3 and all(part.isdigit() for part in parts):
|
||||
parts[2] = str(int(parts[2]) + 1)
|
||||
bumped = ".".join(parts)
|
||||
return text[: match.start(1)] + bumped + text[match.end(1) :]
|
||||
return text
|
||||
|
||||
|
||||
def _set_updated(text: str, day: str) -> str:
|
||||
"""Set the first ``updated`` field to ``day``.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - replaces a placeholder or a previous date.
|
||||
"""
|
||||
match = _UPDATED.search(text)
|
||||
if match is None:
|
||||
return text
|
||||
return text[: match.start(1)] + day + text[match.end(1) :]
|
||||
@@ -0,0 +1,255 @@
|
||||
"""Media-type detection and slug rules for wiki ingest.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.2.4 and 7.25.2.6.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime
|
||||
import mimetypes
|
||||
import pathlib
|
||||
import re
|
||||
import urllib.parse
|
||||
|
||||
_SLUG_TRIM = 80
|
||||
_UNUSABLE_MEDIA = {"", "application/octet-stream"}
|
||||
|
||||
|
||||
def make_slug(text: str) -> str:
|
||||
"""Turn free text into a wiki slug.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - lowercase, hyphenate, and cap at 80.
|
||||
|
||||
Args:
|
||||
text: Stem, host, or joined path segments.
|
||||
|
||||
Returns:
|
||||
str: A slug, or ``source`` when the text has no usable characters.
|
||||
"""
|
||||
slug = re.sub(r"[^a-z0-9]+", "-", text.lower()).strip("-")[:_SLUG_TRIM]
|
||||
return slug or "source"
|
||||
|
||||
|
||||
def slug_from_url(url: str) -> str:
|
||||
"""Build a slug from the last three path segments of a URL.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - drop ``index`` and an empty path uses the host.
|
||||
|
||||
Args:
|
||||
url: Absolute ``http`` or ``https`` URL.
|
||||
|
||||
Returns:
|
||||
str: Slug for the downloaded file and the wiki page.
|
||||
"""
|
||||
parts = urllib.parse.urlsplit(url)
|
||||
cleaned: list[str] = []
|
||||
for segment in parts.path.split("/"):
|
||||
if not segment:
|
||||
continue
|
||||
path = pathlib.PurePosixPath(segment)
|
||||
stem = path.stem if path.suffix else segment
|
||||
if stem.lower() == "index":
|
||||
continue
|
||||
cleaned.append(stem)
|
||||
if not cleaned:
|
||||
return make_slug(parts.hostname or "source")
|
||||
return make_slug("-".join(cleaned[-3:]))
|
||||
|
||||
|
||||
def slug_from_file(path: pathlib.Path) -> str:
|
||||
"""Build a slug from a file stem.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - local paths use the stem only.
|
||||
|
||||
Args:
|
||||
path: Local file path. It does not have to exist.
|
||||
|
||||
Returns:
|
||||
str: Slug for the archive name and the wiki page.
|
||||
"""
|
||||
return make_slug(path.stem)
|
||||
|
||||
|
||||
def split_content_type(header: str | None) -> tuple[str | None, str | None]:
|
||||
"""Split a Content-Type header into a media type and charset.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - drop parameters other than charset.
|
||||
|
||||
``application/octet-stream`` is treated as missing so the extension
|
||||
or the leading bytes can decide.
|
||||
|
||||
Args:
|
||||
header: Raw Content-Type value, or None.
|
||||
|
||||
Returns:
|
||||
tuple: Media type without parameters, and charset if one was set.
|
||||
"""
|
||||
if not header:
|
||||
return None, None
|
||||
parts = [part.strip() for part in header.split(";") if part.strip()]
|
||||
media = parts[0].lower()
|
||||
charset = None
|
||||
for part in parts[1:]:
|
||||
if "=" not in part:
|
||||
continue
|
||||
key, value = part.split("=", 1)
|
||||
if key.strip().lower() == "charset":
|
||||
charset = value.strip().strip("\"'")
|
||||
if media in _UNUSABLE_MEDIA:
|
||||
media = None
|
||||
return media, charset
|
||||
|
||||
|
||||
def media_for_extension(suffix: str, registry: object) -> str | None:
|
||||
"""Map a file extension to a media type.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - converter table first, then mimetypes.
|
||||
|
||||
Args:
|
||||
suffix: Extension including the leading dot.
|
||||
registry: Converter registry in registration order.
|
||||
|
||||
Returns:
|
||||
str | None: Media type, or None when the extension is unknown.
|
||||
"""
|
||||
lowered = suffix.lower()
|
||||
for converter in registry.converters: # type: ignore[attr-defined]
|
||||
if lowered in converter.extensions and converter.media_types:
|
||||
return converter.media_types[0]
|
||||
guessed, _encoding = mimetypes.guess_type("file" + lowered)
|
||||
return guessed
|
||||
|
||||
|
||||
def sniff_media_type(data: bytes) -> str:
|
||||
"""Guess a media type from the first 1024 bytes.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - PDF, HTML, UTF-8 text, or octet-stream.
|
||||
|
||||
Args:
|
||||
data: File bytes. Only the prefix is inspected.
|
||||
|
||||
Returns:
|
||||
str: Detected media type.
|
||||
"""
|
||||
head = data[:1024]
|
||||
if head.startswith(b"%PDF-"):
|
||||
return "application/pdf"
|
||||
sample = head.lstrip().lower()
|
||||
if sample.startswith((b"<!doctype html", b"<html")):
|
||||
return "text/html"
|
||||
if b"\x00" not in head and _is_utf8(head):
|
||||
return "text/plain"
|
||||
return "application/octet-stream"
|
||||
|
||||
|
||||
def detect_media_type(
|
||||
data: bytes,
|
||||
suffix: str,
|
||||
header: str | None,
|
||||
registry: object,
|
||||
) -> tuple[str, str | None]:
|
||||
"""Detect a media type and optional charset hint.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - header, then extension, then bytes.
|
||||
|
||||
Args:
|
||||
data: File bytes, used only when the header and extension miss.
|
||||
suffix: Extension including the dot, or empty.
|
||||
header: Content-Type header for a download, or None for a local file.
|
||||
registry: Converter registry.
|
||||
|
||||
Returns:
|
||||
tuple: Media type and charset hint.
|
||||
"""
|
||||
media, charset = split_content_type(header)
|
||||
if media:
|
||||
return media, charset
|
||||
if suffix:
|
||||
from_ext = media_for_extension(suffix, registry)
|
||||
if from_ext:
|
||||
return from_ext, charset
|
||||
return sniff_media_type(data), charset
|
||||
|
||||
|
||||
def extension_for(media_type: str, path_suffix: str, registry: object) -> str:
|
||||
"""Choose the extension for a downloaded file.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - converter extension, URL suffix, then ``.bin``.
|
||||
|
||||
Args:
|
||||
media_type: Detected media type.
|
||||
path_suffix: Suffix from the URL path, including the dot, or empty.
|
||||
registry: Converter registry.
|
||||
|
||||
Returns:
|
||||
str: Extension including the leading dot.
|
||||
"""
|
||||
converter = registry.find(media_type, "") # type: ignore[attr-defined]
|
||||
if converter is not None and converter.extensions:
|
||||
return converter.extensions[0]
|
||||
if path_suffix:
|
||||
suffix = path_suffix.lower()
|
||||
return suffix if suffix.startswith(".") else f".{suffix}"
|
||||
return ".bin"
|
||||
|
||||
|
||||
def decode_bytes(data: bytes, charset: str | None) -> tuple[str, list[str]]:
|
||||
"""Decode source bytes to text.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - UTF-8, then the charset hint, then cp1252.
|
||||
|
||||
A BOM is stripped before UTF-8 is tried. A fallback decode adds one warning.
|
||||
|
||||
Args:
|
||||
data: Raw file bytes.
|
||||
charset: Optional charset from Content-Type.
|
||||
|
||||
Returns:
|
||||
tuple: Decoded text and warnings.
|
||||
"""
|
||||
payload = data[3:] if data.startswith(b"\xef\xbb\xbf") else data
|
||||
try:
|
||||
return payload.decode("utf-8"), []
|
||||
except UnicodeDecodeError:
|
||||
pass
|
||||
if charset:
|
||||
try:
|
||||
return payload.decode(charset), [f"decoded as {charset}; not valid UTF-8"]
|
||||
except (UnicodeDecodeError, LookupError):
|
||||
pass
|
||||
return payload.decode("cp1252"), ["decoded as cp1252; not valid UTF-8"]
|
||||
|
||||
|
||||
def utc_now() -> str:
|
||||
"""Return the current UTC time as ``YYYY-MM-DDTHH:MM:SSZ``.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - ingest ``fetched`` timestamp.
|
||||
|
||||
Returns:
|
||||
str: UTC timestamp with a ``Z`` suffix.
|
||||
"""
|
||||
moment = datetime.datetime.now(datetime.UTC).replace(microsecond=0)
|
||||
return moment.strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
|
||||
|
||||
def utc_day() -> str:
|
||||
"""Return today's UTC date as ``YYYY-MM-DD``.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - page ``created`` and ``updated`` dates.
|
||||
|
||||
Returns:
|
||||
str: UTC calendar date.
|
||||
"""
|
||||
return utc_now()[:10]
|
||||
|
||||
|
||||
def _is_utf8(data: bytes) -> bool:
|
||||
"""Return whether a byte prefix is valid UTF-8.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - allow a multibyte character cut at 1024.
|
||||
"""
|
||||
try:
|
||||
data.decode("utf-8")
|
||||
except UnicodeDecodeError as exc:
|
||||
return exc.start > 0 and exc.start >= len(data) - 3 and "unexpected end" in exc.reason
|
||||
return True
|
||||
@@ -0,0 +1,126 @@
|
||||
"""Dataclasses passed between wiki ingest stages.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - fetch, convert, and JSON result types.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import dataclasses
|
||||
import pathlib
|
||||
|
||||
|
||||
class IngestError(Exception):
|
||||
"""Raised when wiki ingest must stop before a page is written.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - exit 1 errors from section 4.14.
|
||||
"""
|
||||
|
||||
def __init__(self, message: str, exit_code: int = 1) -> None:
|
||||
"""Store the stderr message and process exit code.
|
||||
|
||||
Args:
|
||||
message: Text after ``Error: ``.
|
||||
exit_code: Process status. Section 4.14 uses 1.
|
||||
"""
|
||||
super().__init__(message)
|
||||
self.exit_code = exit_code
|
||||
|
||||
|
||||
@dataclasses.dataclass
|
||||
class FetchedSource:
|
||||
"""A local file plus the facts the later stages need.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - output of the fetch stage.
|
||||
"""
|
||||
|
||||
argument: str
|
||||
path: pathlib.Path
|
||||
origin: str
|
||||
origin_kind: str
|
||||
final_url: str | None
|
||||
media_type: str
|
||||
charset: str | None
|
||||
size_bytes: int
|
||||
slug: str
|
||||
fetched: str
|
||||
created: bool
|
||||
|
||||
|
||||
@dataclasses.dataclass
|
||||
class ConversionResult:
|
||||
"""Markdown produced from one archived file.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - output of one converter.
|
||||
"""
|
||||
|
||||
body: str
|
||||
title: str | None
|
||||
source_meta: dict
|
||||
conversion: str
|
||||
warnings: list[str]
|
||||
converter: str
|
||||
|
||||
|
||||
@dataclasses.dataclass
|
||||
class ArchiveResult:
|
||||
"""Where the raw copy lives after the archive stage.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - raw path, hash, and reuse flag.
|
||||
"""
|
||||
|
||||
path: pathlib.Path
|
||||
raw_rel: str
|
||||
sha256: str
|
||||
raw_reused: bool
|
||||
warnings: list[str]
|
||||
|
||||
|
||||
@dataclasses.dataclass
|
||||
class IngestResult:
|
||||
"""One ingest run, including the JSON object from section 4.14.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - stdout facts for a finished run.
|
||||
"""
|
||||
|
||||
source: str
|
||||
origin: str
|
||||
origin_kind: str
|
||||
media_type: str
|
||||
raw_path: str | None
|
||||
raw_reused: bool
|
||||
sha256: str
|
||||
duplicate_page: str | None
|
||||
page_path: str | None
|
||||
converter: str | None
|
||||
conversion: str
|
||||
status: str | None
|
||||
warnings: list[str]
|
||||
dry_run: bool
|
||||
exit_code: int = 0
|
||||
error: str | None = None
|
||||
messages: list[str] = dataclasses.field(default_factory=list)
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
"""Return the JSON object from section 4.14.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - drop CLI-only fields.
|
||||
|
||||
Returns:
|
||||
dict: Keys and nulls from the ingest JSON contract.
|
||||
"""
|
||||
return {
|
||||
"source": self.source,
|
||||
"origin": self.origin,
|
||||
"origin_kind": self.origin_kind,
|
||||
"media_type": self.media_type,
|
||||
"raw_path": self.raw_path,
|
||||
"raw_reused": self.raw_reused,
|
||||
"sha256": self.sha256,
|
||||
"duplicate_page": self.duplicate_page,
|
||||
"page_path": self.page_path,
|
||||
"converter": self.converter,
|
||||
"conversion": self.conversion,
|
||||
"status": self.status,
|
||||
"warnings": list(self.warnings),
|
||||
"dry_run": self.dry_run,
|
||||
}
|
||||
@@ -0,0 +1,375 @@
|
||||
"""Write one ``status: raw`` wiki page.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.5. The tool never sets ``refined``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
import pathlib
|
||||
import re
|
||||
|
||||
import yaml
|
||||
from pymarkdown.api import PyMarkdownApiException
|
||||
|
||||
from fiona.mdoctor.mdoctor import MDoctor
|
||||
from fiona.wiki.ingest.media import utc_day
|
||||
from fiona.wiki.ingest.models import ArchiveResult, ConversionResult, FetchedSource, IngestError
|
||||
from fiona.wiki.ingest.settings import IngestSettings, package_version
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_FENCE = re.compile(r"^\s*(`{3,}|~{3,})")
|
||||
_ATX = re.compile(r"^(#{1,6})(\s+)(.*)$")
|
||||
_IMAGE = re.compile(r"!\[([^\]]*)\]\(([^)]+)\)")
|
||||
_LINK = re.compile(r"(?<!!)\[([^\]]*)\]\(([^)]+)\)")
|
||||
_PROTECTED = {"index.md", "log.md", "schema.md"}
|
||||
|
||||
|
||||
class RawPageWriter:
|
||||
"""Write the raw page and run the markdown fixer on that file only.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - frontmatter order from SPEC 7.25.5.3.
|
||||
"""
|
||||
|
||||
def __init__(self, wiki_root: pathlib.Path) -> None:
|
||||
"""Store the wiki root.
|
||||
|
||||
Args:
|
||||
wiki_root: Directory that contains ``wiki/``.
|
||||
"""
|
||||
self.wiki_root = wiki_root
|
||||
|
||||
def page_path(
|
||||
self,
|
||||
slug: str,
|
||||
settings: IngestSettings,
|
||||
target: str | None,
|
||||
) -> pathlib.Path:
|
||||
"""Resolve the page path and reject paths outside ``wiki/``.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.5.1. Does not touch the disk.
|
||||
|
||||
Args:
|
||||
slug: Page stem when ``target`` is a directory or omitted.
|
||||
settings: Supplies ``default_target``.
|
||||
target: ``-t`` value, or None.
|
||||
|
||||
Returns:
|
||||
pathlib.Path: Absolute page path inside ``wiki/``.
|
||||
|
||||
Raises:
|
||||
IngestError: The path escapes ``wiki/`` or names a protected file.
|
||||
"""
|
||||
wiki = (self.wiki_root / "wiki").resolve()
|
||||
if target is None:
|
||||
relative = f"{settings.default_target}/{slug}.md"
|
||||
else:
|
||||
relative = _strip_wiki_prefix(target)
|
||||
suffix = pathlib.PurePosixPath(relative.replace("\\", "/")).suffix
|
||||
if suffix != ".md":
|
||||
relative = f"{relative}/{slug}.md" if relative else f"{slug}.md"
|
||||
candidate = (wiki / relative).resolve()
|
||||
try:
|
||||
page_rel = candidate.relative_to(wiki).as_posix()
|
||||
except ValueError as exc:
|
||||
raise IngestError(f"--target must stay inside wiki/: {target}") from exc
|
||||
if page_rel in _PROTECTED or page_rel.split("/", 1)[0] == "logs":
|
||||
shown = target if target else page_rel
|
||||
raise IngestError(f"--target cannot replace {shown}")
|
||||
return candidate
|
||||
|
||||
def write(
|
||||
self,
|
||||
*,
|
||||
source: FetchedSource,
|
||||
archive: ArchiveResult,
|
||||
conversion: ConversionResult,
|
||||
page: pathlib.Path,
|
||||
dry_run: bool,
|
||||
) -> tuple[str, list[str]]:
|
||||
"""Write the page, fix it, and return the relative path and warnings.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.5.3 through 7.25.5.5.
|
||||
|
||||
Args:
|
||||
source: Fetch facts stored in ``ingest``.
|
||||
archive: Raw path, hash, and reuse flag.
|
||||
conversion: Converter output.
|
||||
page: Destination markdown file.
|
||||
dry_run: When true, build the warnings and write nothing.
|
||||
|
||||
Returns:
|
||||
tuple: Page path relative to the wiki, and the warning list.
|
||||
"""
|
||||
title = conversion.title or source.slug
|
||||
content, link_warnings = _page_content(conversion.body, title, page, archive.path)
|
||||
warnings = [*conversion.warnings, *archive.warnings, *link_warnings]
|
||||
if dry_run:
|
||||
return _rel(page, self.wiki_root), warnings
|
||||
page.parent.mkdir(parents=True, exist_ok=True)
|
||||
day = utc_day()
|
||||
page.write_text(
|
||||
_document(source, archive, conversion, content, warnings, day),
|
||||
encoding="utf-8",
|
||||
newline="\n",
|
||||
)
|
||||
_fix(page, self.wiki_root)
|
||||
style_count = _style_violations(page, self.wiki_root)
|
||||
if style_count:
|
||||
warnings.append(f"{style_count} markdown style violations left")
|
||||
fixed = _body_after_frontmatter(page.read_text(encoding="utf-8"))
|
||||
if not fixed.lstrip("\n").startswith("# "):
|
||||
fixed = content
|
||||
page.write_text(
|
||||
_document(source, archive, conversion, fixed.lstrip("\n"), warnings, day),
|
||||
encoding="utf-8",
|
||||
newline="\n",
|
||||
)
|
||||
logger.info("wrote %s", page)
|
||||
return _rel(page, self.wiki_root), warnings
|
||||
|
||||
|
||||
def _page_content(
|
||||
body: str,
|
||||
title: str,
|
||||
page: pathlib.Path,
|
||||
raw_path: pathlib.Path,
|
||||
) -> tuple[str, list[str]]:
|
||||
"""Demote extra H1s, drop dead relative links, and add the ingest intro.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - the page has one H1 and a link to the raw file.
|
||||
"""
|
||||
shifted = _shift_headings(body)
|
||||
rewritten, count = _rewrite_links(shifted, page.parent)
|
||||
warnings = [f"{count} relative link(s) did not resolve"] if count else []
|
||||
link = os.path.relpath(raw_path, page.parent).replace("\\", "/")
|
||||
intro = f"Raw ingest of [{raw_path.name}]({link}). Not refined yet."
|
||||
prepared = rewritten.strip("\n")
|
||||
tail = f"{intro}\n\n{prepared}\n" if prepared else f"{intro}\n"
|
||||
return f"# {title}\n\n{tail}", warnings
|
||||
|
||||
|
||||
def _document(
|
||||
source: FetchedSource,
|
||||
archive: ArchiveResult,
|
||||
conversion: ConversionResult,
|
||||
content: str,
|
||||
warnings: list[str],
|
||||
day: str,
|
||||
) -> str:
|
||||
"""Render frontmatter plus the page content.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - every key is present, in contract order.
|
||||
"""
|
||||
title = conversion.title or source.slug
|
||||
document = {
|
||||
"name": source.slug,
|
||||
"version": "0.1.0",
|
||||
"title": title,
|
||||
"created": day,
|
||||
"updated": day,
|
||||
"type": "summary",
|
||||
"status": "raw",
|
||||
"sources": [archive.raw_rel],
|
||||
"source_meta": conversion.source_meta or {},
|
||||
"ingest": {
|
||||
"origin": source.origin,
|
||||
"origin_kind": source.origin_kind,
|
||||
"final_url": source.final_url,
|
||||
"fetched": source.fetched,
|
||||
"media_type": source.media_type,
|
||||
"size_bytes": source.size_bytes,
|
||||
"sha256": archive.sha256,
|
||||
"raw_reused": archive.raw_reused,
|
||||
"converter": conversion.converter,
|
||||
"conversion": conversion.conversion,
|
||||
"warnings": list(warnings),
|
||||
"tool": f"fiona-assistant {package_version()}",
|
||||
},
|
||||
"tags": [],
|
||||
"confidence": "zero",
|
||||
"llm": [{"model": "none"}],
|
||||
"contested": False,
|
||||
"contradictions": [],
|
||||
}
|
||||
dumped = yaml.safe_dump(document, sort_keys=False, allow_unicode=True, width=1000)
|
||||
payload = content if content.endswith("\n") else content + "\n"
|
||||
return f"---\n{dumped}---\n\n{payload}"
|
||||
|
||||
|
||||
def _shift_headings(body: str) -> str:
|
||||
"""Move every ATX heading down one level when the body has an H1.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - H6 stays H6. Fenced lines are left alone.
|
||||
"""
|
||||
if not _has_h1(body):
|
||||
return body
|
||||
lines = body.splitlines(keepends=True)
|
||||
fenced = False
|
||||
shifted: list[str] = []
|
||||
for line in lines:
|
||||
ending, content = _split_ending(line)
|
||||
fence = _FENCE.match(content)
|
||||
if fence:
|
||||
fenced = not fenced
|
||||
shifted.append(line)
|
||||
continue
|
||||
match = None if fenced else _ATX.match(content)
|
||||
if match is None:
|
||||
shifted.append(line)
|
||||
continue
|
||||
marks = match.group(1)
|
||||
if len(marks) < 6:
|
||||
marks += "#"
|
||||
shifted.append(f"{marks}{match.group(2)}{match.group(3)}{ending}")
|
||||
return "".join(shifted)
|
||||
|
||||
|
||||
def _has_h1(body: str) -> bool:
|
||||
"""Return whether an unfenced ATX H1 is present.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - detection does not change the body.
|
||||
"""
|
||||
fenced = False
|
||||
for line in body.splitlines():
|
||||
if _FENCE.match(line):
|
||||
fenced = not fenced
|
||||
continue
|
||||
if not fenced and re.match(r"^#\s+\S", line):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _rewrite_links(body: str, page_dir: pathlib.Path) -> tuple[str, int]:
|
||||
"""Replace relative links and images that do not resolve from the page.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - one warning carries the count.
|
||||
"""
|
||||
count = 0
|
||||
lines = body.splitlines(keepends=True)
|
||||
fenced = False
|
||||
rewritten: list[str] = []
|
||||
for line in lines:
|
||||
ending, content = _split_ending(line)
|
||||
if _FENCE.match(content):
|
||||
fenced = not fenced
|
||||
rewritten.append(line)
|
||||
continue
|
||||
if fenced:
|
||||
rewritten.append(line)
|
||||
continue
|
||||
content, image_count = _replace_pattern(_IMAGE, content, page_dir)
|
||||
content, link_count = _replace_pattern(_LINK, content, page_dir)
|
||||
count += image_count + link_count
|
||||
rewritten.append(content + ending)
|
||||
return "".join(rewritten), count
|
||||
|
||||
|
||||
def _replace_pattern(
|
||||
pattern: re.Pattern[str],
|
||||
content: str,
|
||||
page_dir: pathlib.Path,
|
||||
) -> tuple[str, int]:
|
||||
"""Replace matches whose target does not exist.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - external URLs and anchors stay.
|
||||
"""
|
||||
count = 0
|
||||
|
||||
def _swap(match: re.Match[str]) -> str:
|
||||
nonlocal count
|
||||
target = match.group(2).strip()
|
||||
if _keeps_target(target, page_dir):
|
||||
return match.group(0)
|
||||
count += 1
|
||||
return match.group(1)
|
||||
|
||||
return pattern.sub(_swap, content), count
|
||||
|
||||
|
||||
def _keeps_target(target: str, page_dir: pathlib.Path) -> bool:
|
||||
"""Return whether a link target should stay in the page.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - http, mailto, and anchors are kept.
|
||||
"""
|
||||
lowered = target.lower()
|
||||
if lowered.startswith(("http://", "https://", "mailto:", "#", "//")) or "://" in target:
|
||||
return True
|
||||
path_part = target.split("#", 1)[0].split("?", 1)[0]
|
||||
if not path_part:
|
||||
return True
|
||||
resolved = (page_dir / path_part).resolve()
|
||||
return resolved.exists()
|
||||
|
||||
|
||||
def _fix(page: pathlib.Path, wiki_root: pathlib.Path) -> None:
|
||||
"""Run the pymarkdown fixer on one page.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.5.5. A fixer error is logged.
|
||||
"""
|
||||
try:
|
||||
doctor = MDoctor(settings_file=wiki_root / "fiona-settings.json")
|
||||
doctor.fix_file_programmatically(str(page))
|
||||
except PyMarkdownApiException:
|
||||
logger.warning("markdown fix failed for %s", page, exc_info=True)
|
||||
|
||||
|
||||
def _style_violations(page: pathlib.Path, wiki_root: pathlib.Path) -> int:
|
||||
"""Count pymarkdown violations still on the page.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - the count becomes one warning.
|
||||
"""
|
||||
try:
|
||||
doctor = MDoctor(settings_file=wiki_root / "fiona-settings.json")
|
||||
return len(doctor.scan_file_programmatically(str(page)))
|
||||
except PyMarkdownApiException:
|
||||
logger.warning("markdown scan failed for %s", page, exc_info=True)
|
||||
return 0
|
||||
|
||||
|
||||
def _body_after_frontmatter(text: str) -> str:
|
||||
"""Return the file body after the closing frontmatter marker.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - keep the fixer output and replace the header.
|
||||
"""
|
||||
if not text.startswith("---"):
|
||||
return text
|
||||
parts = text.split("---", 2)
|
||||
if len(parts) < 3:
|
||||
return text
|
||||
return parts[2].removeprefix("\n")
|
||||
|
||||
|
||||
def _split_ending(line: str) -> tuple[str, str]:
|
||||
"""Split a line into its ending and its content.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - keep ``\\n`` and ``\\r\\n``.
|
||||
"""
|
||||
if line.endswith("\r\n"):
|
||||
return "\r\n", line[:-2]
|
||||
if line.endswith("\n"):
|
||||
return "\n", line[:-1]
|
||||
return "", line
|
||||
|
||||
|
||||
def _strip_wiki_prefix(value: str) -> str:
|
||||
"""Drop a leading ``wiki/`` from a ``-t`` value.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - both forms are accepted.
|
||||
"""
|
||||
relative = value.replace("\\", "/").lstrip("/")
|
||||
if relative == "wiki":
|
||||
return ""
|
||||
if relative.startswith("wiki/"):
|
||||
return relative[5:]
|
||||
return relative
|
||||
|
||||
|
||||
def _rel(path: pathlib.Path, root: pathlib.Path) -> str:
|
||||
"""Return a path relative to the wiki root using ``/``.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - JSON page paths.
|
||||
"""
|
||||
return path.resolve().relative_to(root.resolve()).as_posix()
|
||||
@@ -0,0 +1,536 @@
|
||||
"""Run the wiki ingest stages in order.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.25. The pipeline holds no format logic.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import dataclasses
|
||||
import json
|
||||
import logging
|
||||
import pathlib
|
||||
import shutil
|
||||
import sys
|
||||
import tempfile
|
||||
|
||||
from fiona.librarian.md_frontmatter import FrontmatterError, parse_frontmatter
|
||||
from fiona.wiki.ingest.archiver import RawArchiver
|
||||
from fiona.wiki.ingest.converters import default_registry
|
||||
from fiona.wiki.ingest.converters.base import ConversionError, ConverterRegistry
|
||||
from fiona.wiki.ingest.fetchers import LocalFileFetcher, SourceFetcher, UrlFetcher, url_scheme
|
||||
from fiona.wiki.ingest.log_writer import IngestLog
|
||||
from fiona.wiki.ingest.media import slug_from_file, slug_from_url, utc_day
|
||||
from fiona.wiki.ingest.models import FetchedSource, IngestError, IngestResult
|
||||
from fiona.wiki.ingest.page_writer import RawPageWriter
|
||||
from fiona.wiki.ingest.settings import IngestSettings, load_ingest_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_NO_CONVERTER = "No converter for {media}. Convert {raw} by hand."
|
||||
|
||||
|
||||
class IngestPipeline:
|
||||
"""Fetch, archive, convert, write, and record one source.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - stages are constructor arguments.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
wiki_root: pathlib.Path,
|
||||
fetchers: list[SourceFetcher],
|
||||
registry: ConverterRegistry,
|
||||
archiver: RawArchiver,
|
||||
page_writer: RawPageWriter,
|
||||
log_writer: IngestLog,
|
||||
settings: IngestSettings,
|
||||
) -> None:
|
||||
"""Store the stage objects. No converter is named here.
|
||||
|
||||
Args:
|
||||
wiki_root: Wiki root directory.
|
||||
fetchers: Tried in order. Each implements ``accepts`` and ``fetch``.
|
||||
registry: Converter registry. Tests may register another converter.
|
||||
archiver: Raw archive stage.
|
||||
page_writer: Wiki page stage.
|
||||
log_writer: ``wiki/log.md`` stage.
|
||||
settings: Timeouts, size limit, and default paths.
|
||||
"""
|
||||
self.wiki_root = wiki_root
|
||||
self.fetchers = fetchers
|
||||
self.registry = registry
|
||||
self.archiver = archiver
|
||||
self.page_writer = page_writer
|
||||
self.log_writer = log_writer
|
||||
self.settings = settings
|
||||
|
||||
def run(
|
||||
self,
|
||||
argument: str,
|
||||
*,
|
||||
raw: str | None = None,
|
||||
target: str | None = None,
|
||||
raw_only: bool = False,
|
||||
force: bool = False,
|
||||
dry_run: bool = False,
|
||||
) -> IngestResult:
|
||||
"""Ingest one URL or local file.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - exit 1 raises; exit 0 and 3 return a result.
|
||||
|
||||
Args:
|
||||
argument: Source as given.
|
||||
raw: ``-r`` path relative to ``raw/``.
|
||||
target: ``-t`` path relative to ``wiki/``.
|
||||
raw_only: Archive and record, skip convert and write.
|
||||
force: Replace an existing page, including a duplicate.
|
||||
dry_run: Report the run and write nothing under the wiki.
|
||||
|
||||
Returns:
|
||||
IngestResult: Facts for stdout. ``exit_code`` is 0 or 3.
|
||||
|
||||
Raises:
|
||||
IngestError: Exit 1. Nothing new remains under the wiki.
|
||||
"""
|
||||
scheme = url_scheme(argument)
|
||||
if scheme is not None and scheme.lower() not in {"http", "https"}:
|
||||
raise IngestError(f"Unsupported URL scheme: {scheme}")
|
||||
slug = slug_from_url(argument) if scheme else slug_from_file(pathlib.Path(argument))
|
||||
page = self.page_writer.page_path(slug, self.settings, target)
|
||||
if page.is_file() and not force:
|
||||
rel = page.resolve().relative_to(self.wiki_root.resolve()).as_posix()
|
||||
raise IngestError(f"Page exists: {rel}. Use --force to replace it.")
|
||||
fetcher = self._fetcher(argument)
|
||||
fetcher.settings = self.settings
|
||||
temp_dir: pathlib.Path | None = None
|
||||
fetched: FetchedSource | None = None
|
||||
archived = False
|
||||
try:
|
||||
if dry_run and scheme:
|
||||
temp_dir = pathlib.Path(tempfile.mkdtemp(prefix="fiona-ingest-"))
|
||||
dest = temp_dir
|
||||
else:
|
||||
dest = self.wiki_root / "ingest"
|
||||
fetched = fetcher.fetch(argument, dest)
|
||||
archive = self.archiver.archive(
|
||||
fetched,
|
||||
registry=self.registry,
|
||||
settings=self.settings,
|
||||
raw_override=raw,
|
||||
dry_run=dry_run,
|
||||
)
|
||||
archived = True
|
||||
return self._after_archive(
|
||||
fetched,
|
||||
archive,
|
||||
page,
|
||||
target_explicit=target is not None,
|
||||
raw_only=raw_only,
|
||||
force=force,
|
||||
dry_run=dry_run,
|
||||
)
|
||||
except IngestError:
|
||||
if fetched is not None and fetched.created and fetched.path.exists() and not archived:
|
||||
fetched.path.unlink()
|
||||
raise
|
||||
finally:
|
||||
if temp_dir is not None:
|
||||
shutil.rmtree(temp_dir, ignore_errors=True)
|
||||
|
||||
def list_pending(self) -> list[dict]:
|
||||
"""List wiki pages whose ``status`` is ``raw``, sorted by path.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.9.
|
||||
|
||||
Returns:
|
||||
list[dict]: ``page_path``, ``sources``, and ``origin`` for each page.
|
||||
"""
|
||||
wiki = self.wiki_root / "wiki"
|
||||
if not wiki.is_dir():
|
||||
return []
|
||||
pages: list[dict] = []
|
||||
for path in sorted(wiki.rglob("*.md"), key=lambda item: item.as_posix()):
|
||||
if not path.is_file():
|
||||
continue
|
||||
try:
|
||||
front = parse_frontmatter(path)
|
||||
except FrontmatterError:
|
||||
continue
|
||||
if front.get("status") != "raw":
|
||||
continue
|
||||
sources = front.get("sources") or []
|
||||
if isinstance(sources, str):
|
||||
sources = [sources]
|
||||
ingest = front.get("ingest") if isinstance(front.get("ingest"), dict) else {}
|
||||
origin = ingest.get("origin") if isinstance(ingest, dict) else None
|
||||
pages.append({
|
||||
"page_path": path.resolve().relative_to(self.wiki_root.resolve()).as_posix(),
|
||||
"sources": list(sources),
|
||||
"origin": origin,
|
||||
})
|
||||
return pages
|
||||
|
||||
def _fetcher(self, argument: str) -> SourceFetcher:
|
||||
"""Return the first fetcher that accepts the argument.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - URL fetcher is registered before the local one.
|
||||
"""
|
||||
for fetcher in self.fetchers:
|
||||
if fetcher.accepts(argument):
|
||||
return fetcher
|
||||
raise IngestError(f"Source not found: {argument}")
|
||||
|
||||
def _after_archive(
|
||||
self,
|
||||
source: FetchedSource,
|
||||
archive: object,
|
||||
page: pathlib.Path,
|
||||
*,
|
||||
target_explicit: bool,
|
||||
raw_only: bool,
|
||||
force: bool,
|
||||
dry_run: bool,
|
||||
) -> IngestResult:
|
||||
"""Finish convert, write, and record after a successful archive.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - duplicate, raw-only, and missing converter exits.
|
||||
"""
|
||||
messages = _fetched_line(source)
|
||||
if archive.raw_reused: # type: ignore[attr-defined]
|
||||
short = archive.sha256[:12] # type: ignore[attr-defined]
|
||||
messages.append(
|
||||
f"Archived: {archive.raw_rel} (already archived, sha256 {short})" # type: ignore[attr-defined]
|
||||
)
|
||||
else:
|
||||
messages.append(f"Archived: {archive.raw_rel}") # type: ignore[attr-defined]
|
||||
duplicate = _citing_page(self.wiki_root, archive.raw_rel, archive.sha256) # type: ignore[attr-defined]
|
||||
if duplicate and not force:
|
||||
messages.append(f"Duplicate: {duplicate} already cites this source.")
|
||||
return _result(
|
||||
source,
|
||||
archive,
|
||||
messages=messages,
|
||||
duplicate_page=duplicate,
|
||||
dry_run=dry_run,
|
||||
warnings=list(archive.warnings), # type: ignore[attr-defined]
|
||||
)
|
||||
if duplicate and force and not target_explicit:
|
||||
page = self.wiki_root / duplicate
|
||||
if raw_only:
|
||||
self.log_writer.append(
|
||||
day=utc_day(),
|
||||
origin=source.origin,
|
||||
raw_rel=archive.raw_rel, # type: ignore[attr-defined]
|
||||
page_rel=None,
|
||||
dry_run=dry_run,
|
||||
)
|
||||
return _result(source, archive, messages=messages, dry_run=dry_run, warnings=list(archive.warnings)) # type: ignore[attr-defined]
|
||||
extension = archive.path.suffix # type: ignore[attr-defined]
|
||||
converter = self.registry.find(source.media_type, extension)
|
||||
if converter is None:
|
||||
messages.append(_NO_CONVERTER.format(media=source.media_type, raw=archive.raw_rel)) # type: ignore[attr-defined]
|
||||
self._record(source, archive, None, dry_run)
|
||||
return _result(
|
||||
source,
|
||||
archive,
|
||||
messages=messages,
|
||||
dry_run=dry_run,
|
||||
exit_code=3,
|
||||
warnings=list(archive.warnings), # type: ignore[attr-defined]
|
||||
)
|
||||
read_path = archive.path if archive.path.is_file() else source.path # type: ignore[attr-defined]
|
||||
try:
|
||||
conversion = converter.convert(dataclasses.replace(source, path=read_path))
|
||||
except ConversionError as exc:
|
||||
messages.append(_NO_CONVERTER.format(media=source.media_type, raw=archive.raw_rel)) # type: ignore[attr-defined]
|
||||
self._record(source, archive, None, dry_run)
|
||||
return _result(
|
||||
source,
|
||||
archive,
|
||||
messages=messages,
|
||||
dry_run=dry_run,
|
||||
exit_code=3,
|
||||
error=f"Conversion failed: {exc.converter}: {exc.reason}",
|
||||
warnings=list(archive.warnings), # type: ignore[attr-defined]
|
||||
)
|
||||
page_rel, warnings = self.page_writer.write(
|
||||
source=source,
|
||||
archive=archive, # type: ignore[arg-type]
|
||||
conversion=conversion,
|
||||
page=page,
|
||||
dry_run=dry_run,
|
||||
)
|
||||
messages.append(
|
||||
f"Wrote: {page_rel} (converter {conversion.converter}, conversion {conversion.conversion})"
|
||||
)
|
||||
messages.extend(f"Warning: {warning}" for warning in warnings)
|
||||
messages.append(f"Next: refine {page_rel}, then set status: refined.")
|
||||
self._record(source, archive, page_rel, dry_run)
|
||||
return _result(
|
||||
source,
|
||||
archive,
|
||||
messages=messages,
|
||||
page_path=page_rel,
|
||||
converter=conversion.converter,
|
||||
conversion=conversion.conversion,
|
||||
status="raw",
|
||||
dry_run=dry_run,
|
||||
warnings=warnings,
|
||||
)
|
||||
|
||||
def _record(self, source: FetchedSource, archive: object, page_rel: str | None, dry_run: bool) -> None:
|
||||
"""Append the log line for a run that archived a source.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - duplicates do not call this.
|
||||
"""
|
||||
self.log_writer.append(
|
||||
day=utc_day(),
|
||||
origin=source.origin,
|
||||
raw_rel=archive.raw_rel, # type: ignore[attr-defined]
|
||||
page_rel=page_rel,
|
||||
dry_run=dry_run,
|
||||
)
|
||||
|
||||
|
||||
def default_pipeline(wiki_root: pathlib.Path) -> IngestPipeline:
|
||||
"""Build the pipeline with the shipped fetchers and converters.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - URL fetcher, then local file fetcher.
|
||||
|
||||
Args:
|
||||
wiki_root: Wiki root directory.
|
||||
|
||||
Returns:
|
||||
IngestPipeline: Ready to ``run`` or ``list_pending``.
|
||||
"""
|
||||
registry = default_registry()
|
||||
return IngestPipeline(
|
||||
wiki_root=wiki_root,
|
||||
fetchers=[UrlFetcher(registry), LocalFileFetcher(registry)],
|
||||
registry=registry,
|
||||
archiver=RawArchiver(wiki_root),
|
||||
page_writer=RawPageWriter(wiki_root),
|
||||
log_writer=IngestLog(wiki_root),
|
||||
settings=load_ingest_settings(wiki_root),
|
||||
)
|
||||
|
||||
|
||||
def render_stdout(result: IngestResult) -> str:
|
||||
"""Return the human report, including the dry-run banner.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - section 4.14 line order.
|
||||
"""
|
||||
lines: list[str] = []
|
||||
if result.dry_run:
|
||||
lines.append("Dry run - no files will be modified:")
|
||||
lines.extend(result.messages)
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def add_ingest_arguments(parser: argparse.ArgumentParser) -> None:
|
||||
"""Add the section 4.14 flags to an ingest parser.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - the CLI subparser owns these flags, including --help.
|
||||
|
||||
Args:
|
||||
parser: The ``wiki ingest`` parser.
|
||||
"""
|
||||
parser.add_argument("source", nargs="?", help="http(s) URL or file path")
|
||||
parser.add_argument(
|
||||
"--wiki",
|
||||
default=".",
|
||||
help="llm-wiki directory (default: .)",
|
||||
)
|
||||
parser.add_argument("-r", "--raw", dest="raw", help="raw path, relative to raw/")
|
||||
parser.add_argument(
|
||||
"-t",
|
||||
"--target",
|
||||
dest="target",
|
||||
help="page path, relative to wiki/",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--raw-only",
|
||||
action="store_true",
|
||||
help="archive only; do not write a page",
|
||||
)
|
||||
parser.add_argument(
|
||||
"-f",
|
||||
"--force",
|
||||
action="store_true",
|
||||
help="replace an existing page",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--dry-run",
|
||||
action="store_true",
|
||||
help="print the plan and write nothing in the wiki",
|
||||
)
|
||||
parser.add_argument("--json", action="store_true", help="print only the JSON result")
|
||||
parser.add_argument(
|
||||
"--pending",
|
||||
action="store_true",
|
||||
help="list pages whose status is raw",
|
||||
)
|
||||
|
||||
|
||||
def ingest_main(argv: list[str] | None = None) -> int:
|
||||
"""CLI entry for ``fiona-assistant wiki ingest``.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - section 4.14 exit codes.
|
||||
|
||||
Args:
|
||||
argv: Arguments after the subcommand. ``None`` reads the process arguments.
|
||||
|
||||
Returns:
|
||||
int: 0, 1, 2, or 3.
|
||||
"""
|
||||
parser = argparse.ArgumentParser(prog="fiona-assistant wiki ingest")
|
||||
add_ingest_arguments(parser)
|
||||
return execute_ingest(parser.parse_args(argv), parser)
|
||||
|
||||
|
||||
def execute_ingest(args: argparse.Namespace, parser: argparse.ArgumentParser) -> int:
|
||||
"""Run one ingest from an already parsed namespace.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - shared by ingest_main and the wiki subcommand.
|
||||
|
||||
Args:
|
||||
args: Parsed source, flags, and paths.
|
||||
parser: Parser used for the mutual-exclusion error.
|
||||
|
||||
Returns:
|
||||
int: 0, 1, 2, or 3.
|
||||
"""
|
||||
if bool(args.source) == bool(args.pending):
|
||||
parser.error("provide a source or --pending, not both")
|
||||
try:
|
||||
root = _require_wiki(args.wiki)
|
||||
pipeline = default_pipeline(root)
|
||||
if args.pending:
|
||||
return _print_pending(pipeline, args.json)
|
||||
result = pipeline.run(
|
||||
args.source,
|
||||
raw=args.raw,
|
||||
target=args.target,
|
||||
raw_only=args.raw_only,
|
||||
force=args.force,
|
||||
dry_run=args.dry_run,
|
||||
)
|
||||
except IngestError as exc:
|
||||
print(f"Error: {exc}", file=sys.stderr)
|
||||
return exc.exit_code
|
||||
if result.error:
|
||||
print(f"Error: {result.error}", file=sys.stderr)
|
||||
if args.json:
|
||||
print(json.dumps(result.to_dict(), indent=2))
|
||||
else:
|
||||
text = render_stdout(result)
|
||||
if text:
|
||||
print(text)
|
||||
return result.exit_code
|
||||
|
||||
|
||||
def _require_wiki(directory: str) -> pathlib.Path:
|
||||
"""Return the wiki root or raise when settings are missing.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - section 4.14 directory error.
|
||||
"""
|
||||
root = pathlib.Path(directory)
|
||||
if not (root / "fiona-settings.json").is_file():
|
||||
raise IngestError(f"Directory '{directory}' is not an llm-wiki directory")
|
||||
return root
|
||||
|
||||
|
||||
def _print_pending(pipeline: IngestPipeline, json_output: bool) -> int:
|
||||
"""Print the pending-page report.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - human lines or one JSON object.
|
||||
"""
|
||||
pages = pipeline.list_pending()
|
||||
if json_output:
|
||||
print(json.dumps({"pending": pages, "total": len(pages)}, indent=2))
|
||||
return 0
|
||||
print(f"Raw pages: {len(pages)}")
|
||||
for page in pages:
|
||||
first = page["sources"][0] if page["sources"] else ""
|
||||
print(f" - {page['page_path']} ({first})")
|
||||
return 0
|
||||
|
||||
|
||||
def _fetched_line(source: FetchedSource) -> list[str]:
|
||||
"""Return the ``Fetched:`` line for a URL, or an empty list.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - local files do not print Fetched.
|
||||
"""
|
||||
if source.origin_kind != "url":
|
||||
return []
|
||||
return [f"Fetched: {source.argument} ({source.media_type}, {source.size_bytes} bytes)"]
|
||||
|
||||
|
||||
def _citing_page(wiki_root: pathlib.Path, raw_rel: str, digest: str) -> str | None:
|
||||
"""Return a page that already cites this raw path or hash.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.3.5. First path in sorted order.
|
||||
"""
|
||||
wiki = wiki_root / "wiki"
|
||||
if not wiki.is_dir():
|
||||
return None
|
||||
matches: list[str] = []
|
||||
for path in wiki.rglob("*.md"):
|
||||
if not path.is_file():
|
||||
continue
|
||||
try:
|
||||
front = parse_frontmatter(path)
|
||||
except FrontmatterError:
|
||||
continue
|
||||
sources = front.get("sources") or []
|
||||
if isinstance(sources, str):
|
||||
sources = [sources]
|
||||
ingest = front.get("ingest") if isinstance(front.get("ingest"), dict) else {}
|
||||
sha = ingest.get("sha256") if isinstance(ingest, dict) else None
|
||||
if raw_rel in list(sources) or sha == digest:
|
||||
matches.append(path.resolve().relative_to(wiki_root.resolve()).as_posix())
|
||||
if not matches:
|
||||
return None
|
||||
matches.sort()
|
||||
return matches[0]
|
||||
|
||||
|
||||
def _result(
|
||||
source: FetchedSource,
|
||||
archive: object,
|
||||
*,
|
||||
messages: list[str],
|
||||
dry_run: bool,
|
||||
warnings: list[str],
|
||||
duplicate_page: str | None = None,
|
||||
page_path: str | None = None,
|
||||
converter: str | None = None,
|
||||
conversion: str = "none",
|
||||
status: str | None = None,
|
||||
exit_code: int = 0,
|
||||
error: str | None = None,
|
||||
) -> IngestResult:
|
||||
"""Build an ingest result from the archive and any page that was written.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - JSON fields from section 4.14.
|
||||
"""
|
||||
return IngestResult(
|
||||
source=source.argument,
|
||||
origin=source.origin,
|
||||
origin_kind=source.origin_kind,
|
||||
media_type=source.media_type,
|
||||
raw_path=archive.raw_rel, # type: ignore[attr-defined]
|
||||
raw_reused=archive.raw_reused, # type: ignore[attr-defined]
|
||||
sha256=archive.sha256, # type: ignore[attr-defined]
|
||||
duplicate_page=duplicate_page,
|
||||
page_path=page_path,
|
||||
converter=converter,
|
||||
conversion=conversion,
|
||||
status=status,
|
||||
warnings=warnings,
|
||||
dry_run=dry_run,
|
||||
exit_code=exit_code,
|
||||
error=error,
|
||||
messages=messages,
|
||||
)
|
||||
@@ -0,0 +1,107 @@
|
||||
"""Ingest settings from ``fiona-settings.json``.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 5.1 ``fiona.wiki.ingest`` keys.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import dataclasses
|
||||
import importlib.metadata
|
||||
import json
|
||||
import pathlib
|
||||
|
||||
|
||||
def package_version() -> str:
|
||||
"""Return the installed fiona-assistant version.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - user agent and ``ingest.tool``.
|
||||
|
||||
Returns:
|
||||
str: Distribution version, or ``0.99.0`` when metadata is absent.
|
||||
"""
|
||||
try:
|
||||
return importlib.metadata.version("fiona-assistant")
|
||||
except importlib.metadata.PackageNotFoundError:
|
||||
return "0.99.0"
|
||||
|
||||
|
||||
@dataclasses.dataclass
|
||||
class IngestSettings:
|
||||
"""Runtime limits and paths for one ingest.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - defaults from the settings table.
|
||||
"""
|
||||
|
||||
timeout_seconds: float
|
||||
max_download_bytes: int
|
||||
user_agent: str
|
||||
default_target: str
|
||||
raw_categories: dict[str, str]
|
||||
|
||||
|
||||
def default_settings() -> IngestSettings:
|
||||
"""Return the settings used when the wiki omits ``fiona.wiki.ingest``.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - table defaults from SPEC 5.1.
|
||||
|
||||
Returns:
|
||||
IngestSettings: Built-in limits and paths.
|
||||
"""
|
||||
return IngestSettings(
|
||||
timeout_seconds=30,
|
||||
max_download_bytes=52428800,
|
||||
user_agent=f"fiona-assistant/{package_version()}",
|
||||
default_target="sources",
|
||||
raw_categories={},
|
||||
)
|
||||
|
||||
|
||||
def load_ingest_settings(wiki_root: pathlib.Path) -> IngestSettings:
|
||||
"""Load ingest settings, falling back per key when a value is unusable.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - absent or mistyped keys keep the default.
|
||||
|
||||
Args:
|
||||
wiki_root: Directory that contains ``fiona-settings.json``.
|
||||
|
||||
Returns:
|
||||
IngestSettings: Merged settings for this wiki.
|
||||
"""
|
||||
settings = default_settings()
|
||||
path = wiki_root / "fiona-settings.json"
|
||||
if not path.is_file():
|
||||
return settings
|
||||
try:
|
||||
document = json.loads(path.read_text(encoding="utf-8"))
|
||||
except json.JSONDecodeError:
|
||||
return settings
|
||||
section = {}
|
||||
if isinstance(document, dict):
|
||||
wiki = document.get("fiona", {})
|
||||
if isinstance(wiki, dict):
|
||||
wiki = wiki.get("wiki", {})
|
||||
if isinstance(wiki, dict):
|
||||
raw = wiki.get("ingest", {})
|
||||
if isinstance(raw, dict):
|
||||
section = raw
|
||||
timeout = section.get("timeout_seconds", settings.timeout_seconds)
|
||||
if isinstance(timeout, (int, float)) and not isinstance(timeout, bool) and timeout > 0:
|
||||
settings.timeout_seconds = float(timeout)
|
||||
limit = section.get("max_download_bytes", settings.max_download_bytes)
|
||||
if isinstance(limit, int) and not isinstance(limit, bool) and limit > 0:
|
||||
settings.max_download_bytes = limit
|
||||
agent = section.get("user_agent")
|
||||
if isinstance(agent, str) and agent.strip():
|
||||
settings.user_agent = agent
|
||||
target = section.get("default_target")
|
||||
if isinstance(target, str) and target.strip() and ".." not in target.replace("\\", "/"):
|
||||
settings.default_target = target.strip().strip("/")
|
||||
categories = section.get("raw_categories")
|
||||
if isinstance(categories, dict):
|
||||
cleaned = {
|
||||
str(key): value
|
||||
for key, value in categories.items()
|
||||
if isinstance(value, str) and value.strip()
|
||||
}
|
||||
settings.raw_categories = cleaned
|
||||
return settings
|
||||
@@ -51,6 +51,29 @@ __all__ = [
|
||||
# Check Runners
|
||||
# =============================================================================
|
||||
|
||||
def _outside_raw(wiki_root: Path, path: str) -> bool:
|
||||
"""Return whether a reported path is outside the raw archive.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 7.25.12. Sanity checks skip ``raw/``.
|
||||
|
||||
Args:
|
||||
wiki_root: Wiki root directory.
|
||||
path: Violation path, absolute or relative to the wiki.
|
||||
|
||||
Returns:
|
||||
bool: True when the path is not under ``raw/``.
|
||||
"""
|
||||
text = path.replace("\\", "/")
|
||||
if text == "raw" or text.startswith("raw/"):
|
||||
return False
|
||||
candidate = Path(path)
|
||||
try:
|
||||
relative = candidate.resolve().relative_to(wiki_root.resolve())
|
||||
except (ValueError, OSError):
|
||||
return True
|
||||
return not relative.parts or relative.parts[0] != "raw"
|
||||
|
||||
|
||||
def run_markdown_lint(wiki_root: Path) -> dict:
|
||||
"""Run markdownlint (mdoctor) over the wiki.
|
||||
|
||||
@@ -62,7 +85,10 @@ def run_markdown_lint(wiki_root: Path) -> dict:
|
||||
the raw mdoctor violations.
|
||||
"""
|
||||
mdoctor = MDoctor(settings_file=wiki_root / SETTINGS_FILE)
|
||||
violations = mdoctor.scan_file_programmatically(str(wiki_root))
|
||||
violations = [
|
||||
item for item in mdoctor.scan_file_programmatically(str(wiki_root))
|
||||
if _outside_raw(wiki_root, str(item.get("scan_file", "")))
|
||||
]
|
||||
report = mdoctor.format_violations(
|
||||
mdoctor.compress_violations(violations, target=str(wiki_root))
|
||||
)
|
||||
@@ -85,7 +111,10 @@ def run_python_lint(wiki_root: Path) -> dict:
|
||||
the raw ruff violations.
|
||||
"""
|
||||
ruff_linter = PythonRuff()
|
||||
violations = ruff_linter.lint(str(wiki_root))
|
||||
violations = [
|
||||
item for item in ruff_linter.lint(str(wiki_root))
|
||||
if _outside_raw(wiki_root, str(item.get("filename", "")))
|
||||
]
|
||||
report = ruff_linter.format_violations(violations, str(wiki_root))
|
||||
return {
|
||||
"result": 0 if not violations else 1,
|
||||
|
||||
@@ -94,6 +94,23 @@ class TestFionaMain:
|
||||
captured = capsys.readouterr()
|
||||
assert "usage" in captured.out.lower()
|
||||
|
||||
def test_main_wiki_ingest_help(self, capsys):
|
||||
"""Test wiki ingest --help is printed by the ingest parser.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - SPEC 4.14 help stays on this command.
|
||||
"""
|
||||
with (
|
||||
patch("sys.argv", ["fiona-assistant", "wiki", "ingest", "--help"]),
|
||||
pytest.raises(SystemExit) as exc_info,
|
||||
):
|
||||
main()
|
||||
|
||||
assert exc_info.value.code == 0
|
||||
captured = capsys.readouterr()
|
||||
assert "fiona-assistant wiki ingest" in captured.out
|
||||
assert "--pending" in captured.out
|
||||
assert "--raw-only" in captured.out
|
||||
|
||||
def test_main_mdoctor_default(self, capsys):
|
||||
"""Test main() with mdoctor command and default arguments."""
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
@@ -440,7 +457,7 @@ class TestFionaMain:
|
||||
# Create test markdown files with valid frontmatter
|
||||
with open(os.path.join(temp_dir, "test1.md"), 'w') as f:
|
||||
f.write("---\nname: Test1\nversion: 1.0.0\n---\n# Test1\n")
|
||||
|
||||
|
||||
with open(os.path.join(temp_dir, "test2.md"), 'w') as f:
|
||||
f.write("---\nname: Test2\nversion: 2.0.0\n---\n# Test2\n")
|
||||
|
||||
|
||||
@@ -0,0 +1,729 @@
|
||||
"""Tests for wiki ingest (SPEC 7.25).
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - acceptance checks for fetch, archive, convert, and write.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import email.message
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
import pathlib
|
||||
import shutil
|
||||
import typing
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
import pypdf
|
||||
import pypdf.generic
|
||||
import pytest
|
||||
import yaml
|
||||
|
||||
from fiona.customlinter.fmlint import FrontmatterLinter
|
||||
from fiona.customlinter.mdlinter import MarkdownLinter
|
||||
from fiona.paths import llm_wiki_template
|
||||
from fiona.wiki.ingest.converters.base import SourceConverter
|
||||
from fiona.wiki.ingest.fetchers import UrlFetcher
|
||||
from fiona.wiki.ingest.media import slug_from_url
|
||||
from fiona.wiki.ingest.models import ConversionResult, FetchedSource, IngestError
|
||||
from fiona.wiki.ingest.pipeline import default_pipeline, ingest_main
|
||||
from fiona.wiki.sanity_manager import run_wiki_checks
|
||||
|
||||
_FRONTMATTER_KEYS = (
|
||||
"name",
|
||||
"version",
|
||||
"title",
|
||||
"created",
|
||||
"updated",
|
||||
"type",
|
||||
"status",
|
||||
"sources",
|
||||
"source_meta",
|
||||
"ingest",
|
||||
"tags",
|
||||
"confidence",
|
||||
"llm",
|
||||
"contested",
|
||||
"contradictions",
|
||||
)
|
||||
_INGEST_KEYS = (
|
||||
"origin",
|
||||
"origin_kind",
|
||||
"final_url",
|
||||
"fetched",
|
||||
"media_type",
|
||||
"size_bytes",
|
||||
"sha256",
|
||||
"raw_reused",
|
||||
"converter",
|
||||
"conversion",
|
||||
"warnings",
|
||||
"tool",
|
||||
)
|
||||
|
||||
|
||||
class _FakeResponse:
|
||||
"""Minimal urllib response for one canned body."""
|
||||
|
||||
def __init__(self, body: bytes, header: str, url: str) -> None:
|
||||
self._body = body
|
||||
self._offset = 0
|
||||
self.headers = email.message.Message()
|
||||
self.headers["Content-Type"] = header
|
||||
self._url = url
|
||||
self.status = 200
|
||||
|
||||
def read(self, amount: int = -1) -> bytes:
|
||||
"""Return the next chunk of the canned body."""
|
||||
if amount is None or amount < 0:
|
||||
amount = len(self._body) - self._offset
|
||||
chunk = self._body[self._offset:self._offset + amount]
|
||||
self._offset += len(chunk)
|
||||
return chunk
|
||||
|
||||
def geturl(self) -> str:
|
||||
"""Return the URL after redirects."""
|
||||
return self._url
|
||||
|
||||
def __enter__(self) -> typing.Self:
|
||||
return self
|
||||
|
||||
def __exit__(self, *_args: object) -> bool:
|
||||
return False
|
||||
|
||||
|
||||
class _FakeOpener:
|
||||
"""URL opener that never touches the network."""
|
||||
|
||||
def __init__(self, body: bytes, header: str, *, status: int = 200, final_url: str | None = None) -> None:
|
||||
self.body = body
|
||||
self.header = header
|
||||
self.status = status
|
||||
self.final_url = final_url
|
||||
self.requests: list[urllib.request.Request] = []
|
||||
self.timeout: float | None = None
|
||||
self.calls = 0
|
||||
|
||||
def open(self, request: urllib.request.Request, timeout: float | None = None) -> _FakeResponse:
|
||||
"""Record the request and return the canned response or a 404."""
|
||||
self.calls += 1
|
||||
self.requests.append(request)
|
||||
self.timeout = timeout
|
||||
if self.status != 200:
|
||||
raise urllib.error.HTTPError(
|
||||
request.full_url,
|
||||
self.status,
|
||||
"Not Found",
|
||||
email.message.Message(),
|
||||
io.BytesIO(b""),
|
||||
)
|
||||
final = self.final_url or request.full_url
|
||||
return _FakeResponse(self.body, self.header, final)
|
||||
|
||||
|
||||
class _XTestConverter(SourceConverter):
|
||||
"""Converter registered only by the test, for ``application/x-test``."""
|
||||
|
||||
name = "xtest"
|
||||
media_types = ("application/x-test",)
|
||||
extensions = (".xtest",)
|
||||
raw_category = "articles"
|
||||
|
||||
def convert(self, source: FetchedSource) -> ConversionResult:
|
||||
"""Return a fixed body so the test can see the registration."""
|
||||
return ConversionResult(
|
||||
body="xtest body\n",
|
||||
title="X Test",
|
||||
source_meta={},
|
||||
conversion="complete",
|
||||
warnings=[],
|
||||
converter=self.name,
|
||||
)
|
||||
|
||||
|
||||
def _wiki(tmp_path: pathlib.Path) -> pathlib.Path:
|
||||
"""Create a small llm-wiki with settings, a log, and an index."""
|
||||
root = tmp_path / "wiki"
|
||||
(root / "wiki").mkdir(parents=True)
|
||||
(root / "ingest").mkdir()
|
||||
(root / "raw").mkdir()
|
||||
(root / "personal").mkdir()
|
||||
shutil.copy2(llm_wiki_template() / "fiona-settings.json", root / "fiona-settings.json")
|
||||
(root / "wiki" / "log.md").write_text(
|
||||
"---\n"
|
||||
"name: log\n"
|
||||
"title: Wiki Log\n"
|
||||
"version: 2.0.0\n"
|
||||
"type: summary\n"
|
||||
"created: 2026-09-30\n"
|
||||
"updated: 2026-09-30\n"
|
||||
"confidence: trusted\n"
|
||||
"llm:\n"
|
||||
" - model: none\n"
|
||||
"contested: false\n"
|
||||
"contradictions: []\n"
|
||||
"---\n\n"
|
||||
"# Wiki Log\n\n"
|
||||
"- 2026-09-30 - Initialized the llm-wiki.\n",
|
||||
encoding="utf-8",
|
||||
newline="\n",
|
||||
)
|
||||
(root / "wiki" / "index.md").write_text(
|
||||
"---\nname: index\nversion: 1.0.0\n---\n\n# Index\n",
|
||||
encoding="utf-8",
|
||||
newline="\n",
|
||||
)
|
||||
return root
|
||||
|
||||
|
||||
def _run(root: pathlib.Path, *args: str) -> tuple[int, str, str]:
|
||||
"""Run the ingest CLI against ``root`` and capture both streams."""
|
||||
stdout = io.StringIO()
|
||||
stderr = io.StringIO()
|
||||
with contextlib.redirect_stdout(stdout), contextlib.redirect_stderr(stderr):
|
||||
code = ingest_main(["--wiki", str(root), *args])
|
||||
return code, stdout.getvalue(), stderr.getvalue()
|
||||
|
||||
|
||||
def _front(path: pathlib.Path) -> dict:
|
||||
"""Parse page frontmatter with types left intact."""
|
||||
text = path.read_text(encoding="utf-8")
|
||||
return yaml.safe_load(text.split("---", 2)[1])
|
||||
|
||||
|
||||
def _pdf(texts: list[str], *, title: str | None = None, password: str | None = None) -> bytes:
|
||||
"""Build a PDF whose pages contain the given strings."""
|
||||
writer = pypdf.PdfWriter()
|
||||
for text in texts:
|
||||
page = writer.add_blank_page(width=612, height=792)
|
||||
if text:
|
||||
_pdf_text(writer, page, text)
|
||||
if title:
|
||||
writer.add_metadata({"/Title": title})
|
||||
if password:
|
||||
writer.encrypt(password)
|
||||
buffer = io.BytesIO()
|
||||
writer.write(buffer)
|
||||
return buffer.getvalue()
|
||||
|
||||
|
||||
def _pdf_text(writer: pypdf.PdfWriter, page: pypdf.PageObject, text: str) -> None:
|
||||
"""Draw one Helvetica string on a page."""
|
||||
font = pypdf.generic.DictionaryObject({
|
||||
pypdf.generic.NameObject("/Type"): pypdf.generic.NameObject("/Font"),
|
||||
pypdf.generic.NameObject("/Subtype"): pypdf.generic.NameObject("/Type1"),
|
||||
pypdf.generic.NameObject("/BaseFont"): pypdf.generic.NameObject("/Helvetica"),
|
||||
})
|
||||
font_ref = writer._add_object(font)
|
||||
fonts = pypdf.generic.DictionaryObject({pypdf.generic.NameObject("/F1"): font_ref})
|
||||
page[pypdf.generic.NameObject("/Resources")] = pypdf.generic.DictionaryObject({
|
||||
pypdf.generic.NameObject("/Font"): fonts,
|
||||
})
|
||||
stream = pypdf.generic.DecodedStreamObject()
|
||||
stream.set_data(f"BT /F1 12 Tf 72 100 Td ({text}) Tj ET".encode("latin-1"))
|
||||
page[pypdf.generic.NameObject("/Contents")] = writer._add_object(stream)
|
||||
|
||||
|
||||
def _hashes(root: pathlib.Path) -> dict[str, str]:
|
||||
"""Hash every file under a wiki."""
|
||||
found: dict[str, str] = {}
|
||||
for path in root.rglob("*"):
|
||||
if path.is_file():
|
||||
found[path.relative_to(root).as_posix()] = hashlib.sha256(path.read_bytes()).hexdigest()
|
||||
return found
|
||||
|
||||
|
||||
def test_slug_from_console_url() -> None:
|
||||
"""SPEC 7.25.2.6: the last three segments become one hyphenated slug."""
|
||||
assert slug_from_url("https://host/job/performance_TL153/2108/console") == (
|
||||
"performance-tl153-2108-console"
|
||||
)
|
||||
|
||||
|
||||
def test_markdown_gets_frontmatter_and_keeps_body(tmp_path: pathlib.Path) -> None:
|
||||
"""A markdown file keeps its wording. Missing page frontmatter is added."""
|
||||
root = _wiki(tmp_path)
|
||||
source = root / "ingest" / "Notes.md"
|
||||
source.write_text("# Kept\n\nThe body stays.\n", encoding="utf-8", newline="\n")
|
||||
code, _stdout, stderr = _run(root, str(source))
|
||||
assert code == 0, stderr
|
||||
assert not source.exists()
|
||||
raw = root / "raw" / "articles" / "notes.md"
|
||||
assert raw.read_text(encoding="utf-8") == "# Kept\n\nThe body stays.\n"
|
||||
page = root / "wiki" / "sources" / "notes.md"
|
||||
front = _front(page)
|
||||
assert front["status"] == "raw"
|
||||
assert front["source_meta"] == {}
|
||||
assert "The body stays." in page.read_text(encoding="utf-8")
|
||||
assert front["title"] == "Kept"
|
||||
text = page.read_text(encoding="utf-8")
|
||||
assert [line for line in text.splitlines() if line.startswith("# ")] == ["# Kept"]
|
||||
|
||||
|
||||
def test_markdown_frontmatter_moves_to_source_meta(tmp_path: pathlib.Path) -> None:
|
||||
"""Source frontmatter is metadata. The page still receives ingest frontmatter."""
|
||||
root = _wiki(tmp_path)
|
||||
source = tmp_path / "ready.md"
|
||||
source.write_text(
|
||||
"---\ntitle: Real Title\nauthor: Ada\n---\n\nParagraph.\n",
|
||||
encoding="utf-8",
|
||||
newline="\n",
|
||||
)
|
||||
code, _stdout, stderr = _run(root, str(source))
|
||||
assert code == 0, stderr
|
||||
page = root / "wiki" / "sources" / "ready.md"
|
||||
front = _front(page)
|
||||
assert front["title"] == "Real Title"
|
||||
assert front["source_meta"]["author"] == "Ada"
|
||||
assert "Paragraph." in page.read_text(encoding="utf-8")
|
||||
assert source.read_text(encoding="utf-8").startswith("---\n")
|
||||
|
||||
|
||||
def test_text_becomes_markdown_for_later_refinement(tmp_path: pathlib.Path) -> None:
|
||||
"""A text file becomes a fenced page at status raw. The tool does not summarize it."""
|
||||
root = _wiki(tmp_path)
|
||||
source = tmp_path / "trace.log"
|
||||
source.write_text("alpha\n```\nbeta\n", encoding="utf-8", newline="\n")
|
||||
code, stdout, stderr = _run(root, str(source))
|
||||
assert code == 0, stderr
|
||||
page = root / "wiki" / "sources" / "trace.md"
|
||||
front = _front(page)
|
||||
assert front["status"] == "raw"
|
||||
assert front["ingest"]["converter"] == "text"
|
||||
assert front["ingest"]["conversion"] == "complete"
|
||||
body = page.read_text(encoding="utf-8")
|
||||
assert "````text" in body
|
||||
assert "alpha" in body
|
||||
assert "Not refined yet." in body
|
||||
assert "Next: refine" in stdout
|
||||
code, pending, _stderr = _run(root, "--pending")
|
||||
assert code == 0
|
||||
assert "wiki/sources/trace.md" in pending
|
||||
|
||||
|
||||
def test_text_fallback_decode_warns(tmp_path: pathlib.Path) -> None:
|
||||
"""Bytes that are not UTF-8 are decoded with a warning."""
|
||||
root = _wiki(tmp_path)
|
||||
source = tmp_path / "legacy.txt"
|
||||
source.write_bytes(b"caf\x80")
|
||||
code, _stdout, stderr = _run(root, str(source))
|
||||
assert code == 0, stderr
|
||||
front = _front(root / "wiki" / "sources" / "legacy.md")
|
||||
assert any("cp1252" in warning for warning in front["ingest"]["warnings"])
|
||||
|
||||
|
||||
def test_register_converter_without_pipeline_edits(tmp_path: pathlib.Path) -> None:
|
||||
"""SPEC 7.25.1.2: a new format is one class and one registration."""
|
||||
root = _wiki(tmp_path)
|
||||
sample = tmp_path / "sample.xtest"
|
||||
sample.write_bytes(b"hello")
|
||||
pipeline = default_pipeline(root)
|
||||
pipeline.registry.register(_XTestConverter)
|
||||
result = pipeline.run(str(sample))
|
||||
assert result.exit_code == 0
|
||||
assert result.converter == "xtest"
|
||||
assert (root / "raw" / "articles" / "sample.xtest").is_file()
|
||||
assert "xtest body" in (root / "wiki" / "sources" / "sample.md").read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def test_ftp_scheme_writes_nothing(tmp_path: pathlib.Path) -> None:
|
||||
"""SPEC 7.25.2.1: an unsupported scheme exits 1 and writes nothing."""
|
||||
root = _wiki(tmp_path)
|
||||
before = _hashes(root)
|
||||
code, stdout, stderr = _run(root, "ftp://host/a.pdf")
|
||||
assert code == 1
|
||||
assert stdout == ""
|
||||
assert "Unsupported URL scheme: ftp" in stderr
|
||||
assert _hashes(root) == before
|
||||
|
||||
|
||||
def test_missing_file_and_directory(tmp_path: pathlib.Path) -> None:
|
||||
"""A missing path and a directory each exit 1."""
|
||||
root = _wiki(tmp_path)
|
||||
code, stdout, stderr = _run(root, str(tmp_path / "missing.txt"))
|
||||
assert code == 1
|
||||
assert stdout == ""
|
||||
assert "Source not found:" in stderr
|
||||
code, stdout, stderr = _run(root, str(tmp_path))
|
||||
assert code == 1
|
||||
assert stdout == ""
|
||||
assert "Source is a directory:" in stderr
|
||||
|
||||
|
||||
def test_download_uses_content_type_extension(tmp_path: pathlib.Path) -> None:
|
||||
"""SPEC 7.25.2.5: a PDF response with no suffix is saved as ``.pdf``."""
|
||||
root = _wiki(tmp_path)
|
||||
body = _pdf(["Hello"])
|
||||
opener = _FakeOpener(body, "application/pdf", final_url="https://host/report")
|
||||
pipeline = default_pipeline(root)
|
||||
pipeline.fetchers[0] = UrlFetcher(pipeline.registry, opener=opener)
|
||||
result = pipeline.run("https://host/report")
|
||||
assert result.exit_code == 0
|
||||
assert result.raw_path == "raw/papers/report.pdf"
|
||||
assert (root / "raw" / "papers" / "report.pdf").read_bytes() == body
|
||||
assert not (root / "ingest" / "report.pdf").exists()
|
||||
assert opener.requests[0].get_header("Authorization") is None
|
||||
assert "fiona-assistant/" in opener.requests[0].get_header("User-agent")
|
||||
|
||||
|
||||
def test_download_404_leaves_ingest_unchanged(tmp_path: pathlib.Path) -> None:
|
||||
"""SPEC 7.25.2.3: a 404 exits 1 and does not add files."""
|
||||
root = _wiki(tmp_path)
|
||||
(root / "ingest" / "keep.txt").write_text("stay", encoding="utf-8")
|
||||
before = sorted(path.name for path in (root / "ingest").iterdir())
|
||||
opener = _FakeOpener(b"", "text/plain", status=404)
|
||||
pipeline = default_pipeline(root)
|
||||
pipeline.fetchers[0] = UrlFetcher(pipeline.registry, opener=opener)
|
||||
with pytest.raises(IngestError, match="404"):
|
||||
pipeline.run("https://host/missing")
|
||||
assert sorted(path.name for path in (root / "ingest").iterdir()) == before
|
||||
|
||||
|
||||
def test_download_over_limit_is_deleted(tmp_path: pathlib.Path) -> None:
|
||||
"""A body over the limit exits 1 and leaves no partial file."""
|
||||
root = _wiki(tmp_path)
|
||||
pipeline = default_pipeline(root)
|
||||
pipeline.settings.max_download_bytes = 4
|
||||
opener = _FakeOpener(b"0123456789", "text/plain")
|
||||
pipeline.fetchers[0] = UrlFetcher(pipeline.registry, opener=opener)
|
||||
with pytest.raises(IngestError, match="exceeds 4 bytes"):
|
||||
pipeline.run("https://host/big")
|
||||
assert list((root / "ingest").iterdir()) == []
|
||||
|
||||
|
||||
def test_extensionless_pdf_uses_pdf_converter(tmp_path: pathlib.Path) -> None:
|
||||
"""SPEC 7.25.2.4: leading ``%PDF-`` bytes select the pdf converter."""
|
||||
root = _wiki(tmp_path)
|
||||
source = tmp_path / "notes"
|
||||
source.write_bytes(_pdf(["Page text"]))
|
||||
code, _stdout, stderr = _run(root, str(source))
|
||||
assert code == 0, stderr
|
||||
page = root / "wiki" / "sources" / "notes.md"
|
||||
assert _front(page)["ingest"]["converter"] == "pdf"
|
||||
assert (root / "raw" / "papers" / "notes.pdf").is_file()
|
||||
|
||||
|
||||
def test_raw_override_and_escape(tmp_path: pathlib.Path) -> None:
|
||||
"""SPEC 7.25.3.2: ``-r`` places the file, and ``..`` exits 1."""
|
||||
root = _wiki(tmp_path)
|
||||
html = tmp_path / "page.html"
|
||||
html.write_text("<html><body><article><p>Hi</p></article></body></html>", encoding="utf-8")
|
||||
code, _stdout, stderr = _run(root, str(html), "-r", "jenkins/tl153")
|
||||
assert code == 0, stderr
|
||||
assert (root / "raw" / "jenkins" / "tl153" / "page.html").is_file()
|
||||
other = tmp_path / "other.html"
|
||||
other.write_text("<html><body><p>Other</p></body></html>", encoding="utf-8")
|
||||
before = _hashes(root)
|
||||
code, stdout, stderr = _run(root, str(other), "-r", "../wiki/x")
|
||||
assert code == 1
|
||||
assert stdout == ""
|
||||
assert "--raw must stay inside raw/" in stderr
|
||||
assert _hashes(root) == before
|
||||
|
||||
|
||||
def test_reuses_identical_bytes(tmp_path: pathlib.Path) -> None:
|
||||
"""SPEC 7.25.3.3: a second name and ``-r`` reuse the first raw copy."""
|
||||
root = _wiki(tmp_path)
|
||||
first = tmp_path / "one.txt"
|
||||
first.write_text("same bytes\n", encoding="utf-8", newline="\n")
|
||||
code, _stdout, stderr = _run(root, str(first))
|
||||
assert code == 0, stderr
|
||||
second = tmp_path / "two.txt"
|
||||
second.write_text("same bytes\n", encoding="utf-8", newline="\n")
|
||||
pipeline = default_pipeline(root)
|
||||
result = pipeline.run(str(second), raw="other/dir")
|
||||
assert result.raw_reused is True
|
||||
assert result.raw_path == "raw/articles/one.txt"
|
||||
assert result.duplicate_page == "wiki/sources/one.md"
|
||||
copies = [path for path in (root / "raw").rglob("*") if path.is_file() and path.suffix == ".txt"]
|
||||
assert copies == [root / "raw" / "articles" / "one.txt"]
|
||||
assert "already archived at raw/articles/one.txt; --raw ignored" in result.warnings
|
||||
|
||||
|
||||
def test_changed_file_gets_a_second_raw_name(tmp_path: pathlib.Path) -> None:
|
||||
"""SPEC 7.25.3.4: a changed file of the same name is stored as ``slug-2``."""
|
||||
root = _wiki(tmp_path)
|
||||
source = tmp_path / "notes.txt"
|
||||
source.write_text("first\n", encoding="utf-8", newline="\n")
|
||||
code, _stdout, stderr = _run(root, str(source), "--raw-only")
|
||||
assert code == 0, stderr
|
||||
original = (root / "raw" / "articles" / "notes.txt").read_bytes()
|
||||
source.write_text("second\n", encoding="utf-8", newline="\n")
|
||||
code, _stdout, stderr = _run(root, str(source))
|
||||
assert code == 0, stderr
|
||||
assert (root / "raw" / "articles" / "notes.txt").read_bytes() == original
|
||||
assert (root / "raw" / "articles" / "notes-2.txt").read_text(encoding="utf-8") == "second\n"
|
||||
|
||||
|
||||
def test_duplicate_exits_zero_and_skips_the_log(tmp_path: pathlib.Path) -> None:
|
||||
"""SPEC 7.25.3.5: the same bytes report the first page and do not rewrite it."""
|
||||
root = _wiki(tmp_path)
|
||||
source = root / "ingest" / "a.txt"
|
||||
source.write_text("alpha\n", encoding="utf-8", newline="\n")
|
||||
code, _stdout, stderr = _run(root, str(source))
|
||||
assert code == 0, stderr
|
||||
page = root / "wiki" / "sources" / "a.md"
|
||||
page_bytes = page.read_bytes()
|
||||
log_before = (root / "wiki" / "log.md").read_text(encoding="utf-8")
|
||||
again = root / "ingest" / "b.txt"
|
||||
again.write_text("alpha\n", encoding="utf-8", newline="\n")
|
||||
code, stdout, stderr = _run(root, str(again), "--json")
|
||||
assert code == 0, stderr
|
||||
payload = json.loads(stdout)
|
||||
assert payload["duplicate_page"] == "wiki/sources/a.md"
|
||||
assert payload["page_path"] is None
|
||||
assert not again.exists()
|
||||
assert page.read_bytes() == page_bytes
|
||||
assert (root / "wiki" / "log.md").read_text(encoding="utf-8") == log_before
|
||||
|
||||
|
||||
def test_ingest_drop_moves_and_personal_is_copied(tmp_path: pathlib.Path) -> None:
|
||||
"""SPEC 7.25.3.6: ``ingest/`` is moved. A file outside it is copied."""
|
||||
root = _wiki(tmp_path)
|
||||
dropped = root / "ingest" / "a.pdf"
|
||||
dropped.write_bytes(_pdf(["Moved"]))
|
||||
code, _stdout, stderr = _run(root, str(dropped))
|
||||
assert code == 0, stderr
|
||||
assert not dropped.exists()
|
||||
archived = root / "raw" / "papers" / "a.pdf"
|
||||
assert archived.read_bytes() == _pdf(["Moved"])
|
||||
personal = root / "personal" / "note.txt"
|
||||
personal.write_text("keep me\n", encoding="utf-8", newline="\n")
|
||||
original = personal.read_bytes()
|
||||
code, _stdout, stderr = _run(root, str(personal))
|
||||
assert code == 0, stderr
|
||||
assert personal.read_bytes() == original
|
||||
assert (root / "raw" / "articles" / "note.txt").read_bytes() == original
|
||||
|
||||
|
||||
def test_unknown_bytes_archive_without_a_page(tmp_path: pathlib.Path) -> None:
|
||||
"""SPEC 7.25.4.1: random bytes exit 3 under ``raw/other``."""
|
||||
root = _wiki(tmp_path)
|
||||
source = tmp_path / "blob.xyz"
|
||||
source.write_bytes(b"\x00\x01\x02\xff")
|
||||
code, stdout, _stderr = _run(root, str(source), "--json")
|
||||
assert code == 3
|
||||
payload = json.loads(stdout)
|
||||
assert payload["conversion"] == "none"
|
||||
assert payload["page_path"] is None
|
||||
assert payload["raw_path"] == "raw/other/blob.xyz"
|
||||
assert (root / "raw" / "other" / "blob.xyz").is_file()
|
||||
assert not (root / "wiki" / "sources" / "blob.md").exists()
|
||||
human_source = tmp_path / "other.xyz"
|
||||
human_source.write_bytes(b"\x00\x02")
|
||||
code, human, _stderr = _run(root, str(human_source))
|
||||
assert code == 3
|
||||
assert human.strip().splitlines()[-1].startswith("No converter for")
|
||||
|
||||
|
||||
def test_encrypted_pdf_keeps_raw_and_skips_the_page(tmp_path: pathlib.Path) -> None:
|
||||
"""SPEC 7.25.4.2: an encrypted PDF exits 3 and the raw copy stays."""
|
||||
root = _wiki(tmp_path)
|
||||
source = tmp_path / "secret.pdf"
|
||||
source.write_bytes(_pdf(["hidden"], password="secret"))
|
||||
code, stdout, stderr = _run(root, str(source))
|
||||
assert code == 3
|
||||
assert "Conversion failed: pdf:" in stderr
|
||||
assert (root / "raw" / "papers" / "secret.pdf").is_file()
|
||||
assert not (root / "wiki" / "sources" / "secret.md").exists()
|
||||
assert stdout.strip().splitlines()[-1].startswith("No converter for")
|
||||
|
||||
|
||||
def test_html_keeps_article_link_and_table(tmp_path: pathlib.Path) -> None:
|
||||
"""SPEC 7.25.4.5: article text, no script, an absolute link, and a pipe table."""
|
||||
root = _wiki(tmp_path)
|
||||
html = """<!doctype html>
|
||||
<html lang="en"><head><title>Page Title</title></head>
|
||||
<body>
|
||||
<nav>NAVSECRET</nav>
|
||||
<article>
|
||||
<p>ARTICLETEX</p>
|
||||
<a href="/a">Go</a>
|
||||
<table><tr><th>Col</th></tr><tr><td>Cell</td></tr></table>
|
||||
</article>
|
||||
<script>SCRIPTSECRET</script>
|
||||
</body></html>
|
||||
"""
|
||||
opener = _FakeOpener(html.encode("utf-8"), "text/html")
|
||||
pipeline = default_pipeline(root)
|
||||
pipeline.fetchers[0] = UrlFetcher(pipeline.registry, opener=opener)
|
||||
result = pipeline.run("https://host/p")
|
||||
assert result.exit_code == 0
|
||||
body = (root / result.page_path).read_text(encoding="utf-8")
|
||||
assert "ARTICLETEX" in body
|
||||
assert "SCRIPTSECRET" not in body
|
||||
assert "https://host/a" in body
|
||||
assert "|" in body
|
||||
|
||||
|
||||
def test_pdf_page_headings_and_blank_page(tmp_path: pathlib.Path) -> None:
|
||||
"""SPEC 7.25.4.6: pages are numbered, and a blank page is partial."""
|
||||
root = _wiki(tmp_path)
|
||||
numbered = tmp_path / "two.pdf"
|
||||
numbered.write_bytes(_pdf(["Alpha", "Beta"]))
|
||||
code, _stdout, stderr = _run(root, str(numbered))
|
||||
assert code == 0, stderr
|
||||
body = (root / "wiki" / "sources" / "two.md").read_text(encoding="utf-8")
|
||||
assert "## Page 1" in body
|
||||
assert "## Page 2" in body
|
||||
blank = tmp_path / "gap.pdf"
|
||||
blank.write_bytes(_pdf(["Alpha", ""]))
|
||||
code, _stdout, stderr = _run(root, str(blank))
|
||||
assert code == 0, stderr
|
||||
front = _front(root / "wiki" / "sources" / "gap.md")
|
||||
assert front["ingest"]["conversion"] == "partial"
|
||||
assert "page 2 has no text layer" in front["ingest"]["warnings"]
|
||||
|
||||
|
||||
def test_target_paths(tmp_path: pathlib.Path) -> None:
|
||||
"""SPEC 7.25.5.1: ``-t concepts`` writes under concepts. Escapes exit 1."""
|
||||
root = _wiki(tmp_path)
|
||||
source = tmp_path / "idea.txt"
|
||||
source.write_text("idea\n", encoding="utf-8", newline="\n")
|
||||
code, _stdout, stderr = _run(root, str(source), "-t", "concepts")
|
||||
assert code == 0, stderr
|
||||
assert (root / "wiki" / "concepts" / "idea.md").is_file()
|
||||
code, stdout, stderr = _run(root, str(source), "-t", "../raw/x.md")
|
||||
assert code == 1
|
||||
assert stdout == ""
|
||||
assert "--target must stay inside wiki/" in stderr
|
||||
other = tmp_path / "other.txt"
|
||||
other.write_text("x\n", encoding="utf-8", newline="\n")
|
||||
code, stdout, stderr = _run(root, str(other), "-t", "index.md")
|
||||
assert code == 1
|
||||
assert "--target cannot replace" in stderr
|
||||
|
||||
|
||||
def test_existing_page_refuses_until_force(tmp_path: pathlib.Path) -> None:
|
||||
"""SPEC 7.25.5.2: a second run exits 1. ``--force`` replaces the page."""
|
||||
root = _wiki(tmp_path)
|
||||
source = tmp_path / "item.txt"
|
||||
source.write_text("one\n", encoding="utf-8", newline="\n")
|
||||
code, _stdout, stderr = _run(root, str(source))
|
||||
assert code == 0, stderr
|
||||
page = root / "wiki" / "sources" / "item.md"
|
||||
before = page.read_bytes()
|
||||
opener = _FakeOpener(b"nope", "text/plain")
|
||||
pipeline = default_pipeline(root)
|
||||
pipeline.fetchers[0] = UrlFetcher(pipeline.registry, opener=opener)
|
||||
with pytest.raises(IngestError, match="Page exists:"):
|
||||
pipeline.run("https://host/item")
|
||||
assert opener.calls == 0
|
||||
assert page.read_bytes() == before
|
||||
source.write_text("two\n", encoding="utf-8", newline="\n")
|
||||
code, _stdout, stderr = _run(root, str(source), "--force")
|
||||
assert code == 0, stderr
|
||||
assert page.read_bytes() != before
|
||||
assert "two" in (root / "raw" / "articles" / "item-2.txt").read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def test_page_frontmatter_matches_raw_hash(tmp_path: pathlib.Path) -> None:
|
||||
"""SPEC 7.25.5.3 and 7.25.5.4: frontmatter is complete and the page has one H1."""
|
||||
root = _wiki(tmp_path)
|
||||
index_before = (root / "wiki" / "index.md").read_bytes()
|
||||
source = tmp_path / "story.md"
|
||||
source.write_text("# Story\n\nSee [gone](missing.md).\n", encoding="utf-8", newline="\n")
|
||||
pipeline = default_pipeline(root)
|
||||
result = pipeline.run(str(source))
|
||||
assert result.exit_code == 0
|
||||
page = root / result.page_path
|
||||
front = _front(page)
|
||||
for key in _FRONTMATTER_KEYS:
|
||||
assert key in front
|
||||
for key in _INGEST_KEYS:
|
||||
assert key in front["ingest"]
|
||||
raw = root / front["sources"][0]
|
||||
assert front["ingest"]["sha256"] == hashlib.sha256(raw.read_bytes()).hexdigest()
|
||||
assert front["ingest"]["warnings"] == result.warnings
|
||||
assert front["status"] == "raw"
|
||||
assert front["confidence"] == "zero"
|
||||
assert front["llm"] == [{"model": "none"}]
|
||||
headings = [line for line in page.read_text(encoding="utf-8").splitlines() if line.startswith("# ")]
|
||||
assert headings == ["# Story"]
|
||||
linter = FrontmatterLinter(settings_file=root / "fiona-settings.json")
|
||||
assert linter.lint_file(page) == []
|
||||
links = MarkdownLinter(settings_file=root / "fiona-settings.json")
|
||||
assert [item for item in links.lint_file(page) if item.code == "MDL001"] == []
|
||||
assert (root / "wiki" / "index.md").read_bytes() == index_before
|
||||
log = (root / "wiki" / "log.md").read_text(encoding="utf-8")
|
||||
assert "version: 2.0.1" in log
|
||||
assert f"ingest {source.resolve()} -> raw/articles/story.md -> wiki/sources/story.md (raw)" in log
|
||||
|
||||
|
||||
def test_style_violations_become_one_warning(tmp_path: pathlib.Path) -> None:
|
||||
"""SPEC 7.25.5.5: violations the fixer leaves are counted in one warning."""
|
||||
root = _wiki(tmp_path)
|
||||
source = tmp_path / "long.md"
|
||||
source.write_text("# Title\n\n" + ("word " * 40) + "\n", encoding="utf-8", newline="\n")
|
||||
code, _stdout, stderr = _run(root, str(source))
|
||||
assert code == 0, stderr
|
||||
warnings = _front(root / "wiki" / "sources" / "long.md")["ingest"]["warnings"]
|
||||
assert any(warning.endswith("markdown style violations left") for warning in warnings)
|
||||
|
||||
|
||||
def test_raw_only_and_dry_run(tmp_path: pathlib.Path) -> None:
|
||||
"""SPEC 7.25.7 and 7.25.8: raw-only skips the page. A dry run writes nothing."""
|
||||
root = _wiki(tmp_path)
|
||||
source = root / "ingest" / "only.txt"
|
||||
source.write_text("archive me\n", encoding="utf-8", newline="\n")
|
||||
code, stdout, stderr = _run(root, str(source), "--raw-only", "--json")
|
||||
assert code == 0, stderr
|
||||
payload = json.loads(stdout)
|
||||
assert payload["conversion"] == "none"
|
||||
assert payload["page_path"] is None
|
||||
assert not source.exists()
|
||||
assert (root / "raw" / "articles" / "only.txt").is_file()
|
||||
kept = root / "ingest" / "stay.txt"
|
||||
kept.write_text("stay\n", encoding="utf-8", newline="\n")
|
||||
before = _hashes(root)
|
||||
code, stdout, stderr = _run(root, str(kept), "--dry-run")
|
||||
assert code == 0, stderr
|
||||
assert stdout.startswith("Dry run - no files will be modified:")
|
||||
assert kept.is_file()
|
||||
assert _hashes(root) == before
|
||||
code, stdout, _stderr = _run(root, str(kept), "--dry-run", "--json")
|
||||
assert json.loads(stdout)["dry_run"] is True
|
||||
|
||||
|
||||
def test_pending_hides_refined_pages(tmp_path: pathlib.Path) -> None:
|
||||
"""SPEC 7.25.9: ``--pending`` lists raw pages and drops refined ones."""
|
||||
root = _wiki(tmp_path)
|
||||
source = tmp_path / "open.txt"
|
||||
source.write_text("open\n", encoding="utf-8", newline="\n")
|
||||
assert _run(root, str(source))[0] == 0
|
||||
code, stdout, _stderr = _run(root, "--pending", "--json")
|
||||
payload = json.loads(stdout)
|
||||
assert code == 0
|
||||
assert payload["total"] == 1
|
||||
page = root / "wiki" / "sources" / "open.md"
|
||||
text = page.read_text(encoding="utf-8").replace("status: raw", "status: refined", 1)
|
||||
page.write_text(text, encoding="utf-8", newline="\n")
|
||||
code, stdout, _stderr = _run(root, "--pending", "--json")
|
||||
assert json.loads(stdout)["pending"] == []
|
||||
|
||||
|
||||
def test_not_a_wiki_and_mutually_exclusive_args(tmp_path: pathlib.Path) -> None:
|
||||
"""A directory without settings exits 1. Source and ``--pending`` cannot combine."""
|
||||
code, stdout, stderr = _run(tmp_path, str(tmp_path / "a.txt"))
|
||||
assert code == 1
|
||||
assert stdout == ""
|
||||
assert "is not an llm-wiki directory" in stderr
|
||||
with pytest.raises(SystemExit) as caught:
|
||||
ingest_main(["--pending", "file.txt"])
|
||||
assert caught.value.code == 2
|
||||
|
||||
|
||||
def test_sanity_ignores_raw_markdown(tmp_path: pathlib.Path) -> None:
|
||||
"""SPEC 7.25.12: a raw markdown file with style violations does not fail sanity."""
|
||||
root = _wiki(tmp_path)
|
||||
raw = root / "raw" / "articles"
|
||||
raw.mkdir(parents=True)
|
||||
(raw / "bad-style.md").write_text(
|
||||
"# Bad\n\n" + ("x" * 180) + "\n\n[gone](missing-file.md)\n",
|
||||
encoding="utf-8",
|
||||
newline="\n",
|
||||
)
|
||||
results = run_wiki_checks(root)
|
||||
assert "bad-style.md" not in str(results)
|
||||
@@ -0,0 +1,552 @@
|
||||
"""Branch coverage for wiki ingest stages.
|
||||
|
||||
Generated by AI/LLM: 2026-09-30 - error paths and converter edges from SPEC 7.25.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import datetime
|
||||
import email.message
|
||||
import importlib.metadata
|
||||
import io
|
||||
import json
|
||||
import pathlib
|
||||
import shutil
|
||||
import types
|
||||
import typing
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
import pypdf
|
||||
import pytest
|
||||
from pymarkdown.api import PyMarkdownApiException
|
||||
|
||||
from fiona.mdoctor.mdoctor import MDoctor
|
||||
from fiona.paths import llm_wiki_template
|
||||
from fiona.wiki.ingest import log_writer as log_mod
|
||||
from fiona.wiki.ingest import media
|
||||
from fiona.wiki.ingest import page_writer as pages
|
||||
from fiona.wiki.ingest import settings as settings_mod
|
||||
from fiona.wiki.ingest.archiver import RawArchiver, _strip_prefix
|
||||
from fiona.wiki.ingest.converters import default_registry
|
||||
from fiona.wiki.ingest.converters.base import ConversionError, ConverterRegistry
|
||||
from fiona.wiki.ingest.converters.html import HtmlConverter
|
||||
from fiona.wiki.ingest.converters.markdown import MarkdownConverter
|
||||
from fiona.wiki.ingest.converters.pdf import (
|
||||
PdfConverter,
|
||||
_iso_date,
|
||||
_metadata,
|
||||
_outline,
|
||||
_render,
|
||||
_walk_outline,
|
||||
)
|
||||
from fiona.wiki.ingest.fetchers import UrlFetcher, _without_userinfo, url_scheme
|
||||
from fiona.wiki.ingest.log_writer import IngestLog
|
||||
from fiona.wiki.ingest.models import ArchiveResult, ConversionResult, FetchedSource, IngestError
|
||||
from fiona.wiki.ingest.pipeline import (
|
||||
IngestPipeline,
|
||||
_citing_page,
|
||||
_print_pending,
|
||||
default_pipeline,
|
||||
)
|
||||
from fiona.wiki.ingest.settings import default_settings, load_ingest_settings
|
||||
|
||||
|
||||
def _wiki(tmp_path: pathlib.Path) -> pathlib.Path:
|
||||
"""Create a small llm-wiki."""
|
||||
root = tmp_path / "wiki"
|
||||
(root / "wiki").mkdir(parents=True)
|
||||
(root / "ingest").mkdir()
|
||||
(root / "raw").mkdir()
|
||||
shutil.copy2(llm_wiki_template() / "fiona-settings.json", root / "fiona-settings.json")
|
||||
(root / "wiki" / "log.md").write_text(
|
||||
"---\nname: log\nversion: 2.0.0\nupdated: 2026-09-30\n---\n\n# Log\n",
|
||||
encoding="utf-8",
|
||||
newline="\n",
|
||||
)
|
||||
(root / "wiki" / "index.md").write_text(
|
||||
"---\nname: index\nversion: 1.0.0\n---\n\n# Index\n",
|
||||
encoding="utf-8",
|
||||
newline="\n",
|
||||
)
|
||||
return root
|
||||
|
||||
|
||||
def _source(path: pathlib.Path, *, origin_kind: str = "file", origin: str | None = None) -> FetchedSource:
|
||||
"""Build a fetched-source record for a file that already exists."""
|
||||
return FetchedSource(
|
||||
argument=str(path),
|
||||
path=path,
|
||||
origin=origin or str(path.resolve()),
|
||||
origin_kind=origin_kind,
|
||||
final_url=None,
|
||||
media_type="text/plain",
|
||||
charset=None,
|
||||
size_bytes=path.stat().st_size,
|
||||
slug=path.stem,
|
||||
fetched="2026-09-30T00:00:00Z",
|
||||
created=False,
|
||||
)
|
||||
|
||||
|
||||
class _FakeResponse:
|
||||
"""One canned download body."""
|
||||
|
||||
def __init__(self, body: bytes, url: str) -> None:
|
||||
self._body = body
|
||||
self._offset = 0
|
||||
self.headers = email.message.Message()
|
||||
self.headers["Content-Type"] = "text/plain"
|
||||
self._url = url
|
||||
|
||||
def read(self, amount: int = -1) -> bytes:
|
||||
"""Return the next slice of the body."""
|
||||
if amount < 0:
|
||||
amount = len(self._body) - self._offset
|
||||
chunk = self._body[self._offset:self._offset + amount]
|
||||
self._offset += len(chunk)
|
||||
return chunk
|
||||
|
||||
def geturl(self) -> str:
|
||||
"""Return the final URL."""
|
||||
return self._url
|
||||
|
||||
def __enter__(self) -> typing.Self:
|
||||
return self
|
||||
|
||||
def __exit__(self, *_args: object) -> bool:
|
||||
return False
|
||||
|
||||
|
||||
class _Opener:
|
||||
"""Opener that returns one body or raises URLError."""
|
||||
|
||||
def __init__(self, body: bytes | None = b"hello\n", *, error: str | None = None) -> None:
|
||||
self.body = body or b""
|
||||
self.error = error
|
||||
|
||||
def open(self, request: urllib.request.Request, timeout: float | None = None) -> _FakeResponse:
|
||||
"""Return the canned body or fail the download."""
|
||||
del timeout
|
||||
if self.error:
|
||||
raise urllib.error.URLError(self.error)
|
||||
return _FakeResponse(self.body, request.full_url)
|
||||
|
||||
|
||||
def test_slug_and_media_edges() -> None:
|
||||
"""Slugs, sniffing, extensions, and decode fallbacks."""
|
||||
assert url_scheme("note.txt") is None
|
||||
assert media.slug_from_url("https://host/index") == "host"
|
||||
assert media.slug_from_url("https://host/") == "host"
|
||||
assert media.split_content_type("text/plain; boundary; charset=\"utf-8\"") == ("text/plain", "utf-8")
|
||||
assert media.split_content_type("application/octet-stream") == (None, None)
|
||||
registry = default_registry()
|
||||
assert media.sniff_media_type(b" <!DOCTYPE html><html>") == "text/html"
|
||||
assert media.sniff_media_type(b"\x00\xff") == "application/octet-stream"
|
||||
assert media.sniff_media_type(b"ab\xe2\x82") == "text/plain"
|
||||
assert media.sniff_media_type(b"plain text") == "text/plain"
|
||||
assert media.extension_for("application/zip", "pdf", registry) == ".pdf"
|
||||
assert media.extension_for("application/zip", "", registry) == ".bin"
|
||||
assert media.decode_bytes(b"\xef\xbb\xbfhi", None) == ("hi", [])
|
||||
text, warnings = media.decode_bytes(b"\xff", "latin-1")
|
||||
assert text == "ÿ"
|
||||
assert warnings == ["decoded as latin-1; not valid UTF-8"]
|
||||
_text, fallback = media.decode_bytes(b"\xff", "not-a-charset")
|
||||
assert "cp1252" in fallback[0]
|
||||
assert registry.find("nope", "md").name == "markdown"
|
||||
|
||||
|
||||
def test_settings_fallbacks(tmp_path: pathlib.Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Missing, broken, and partial ingest settings keep usable values."""
|
||||
def missing(_name: str) -> str:
|
||||
raise importlib.metadata.PackageNotFoundError
|
||||
|
||||
monkeypatch.setattr(settings_mod.importlib.metadata, "version", missing)
|
||||
assert settings_mod.package_version() == "0.99.0"
|
||||
assert load_ingest_settings(tmp_path).timeout_seconds == 30
|
||||
broken = tmp_path / "bad"
|
||||
broken.mkdir()
|
||||
(broken / "fiona-settings.json").write_text("{", encoding="utf-8")
|
||||
assert load_ingest_settings(broken).max_download_bytes == 52428800
|
||||
good = tmp_path / "good"
|
||||
good.mkdir()
|
||||
(good / "fiona-settings.json").write_text(json.dumps({
|
||||
"fiona": {"wiki": {"ingest": {
|
||||
"user_agent": "agent",
|
||||
"default_target": "concepts",
|
||||
"raw_categories": {"pdf": "papers", "html": 1},
|
||||
}}},
|
||||
}), encoding="utf-8")
|
||||
loaded = load_ingest_settings(good)
|
||||
assert loaded.user_agent == "agent"
|
||||
assert loaded.default_target == "concepts"
|
||||
assert loaded.raw_categories == {"pdf": "papers"}
|
||||
|
||||
|
||||
def test_markdown_frontmatter_edges(tmp_path: pathlib.Path) -> None:
|
||||
"""Invalid, empty, and non-mapping frontmatter, and a fenced heading."""
|
||||
converter = MarkdownConverter()
|
||||
cases = {
|
||||
"bad.md": "---\n[\n---\n# Kept\n",
|
||||
"empty.md": "---\n\n---\n# Kept\n",
|
||||
"list.md": "---\n- a\n---\n# Kept\n",
|
||||
"fenced.md": "```\n# Hidden\n```\n",
|
||||
}
|
||||
for name, text in cases.items():
|
||||
path = tmp_path / name
|
||||
path.write_text(text, encoding="utf-8")
|
||||
result = converter.convert(_source(path))
|
||||
assert result.conversion == "complete"
|
||||
fenced = converter.convert(_source(tmp_path / "fenced.md"))
|
||||
assert fenced.title == "fenced"
|
||||
assert fenced.body.startswith("```")
|
||||
assert converter.convert(_source(tmp_path / "bad.md")).warnings
|
||||
|
||||
|
||||
def test_html_edges(tmp_path: pathlib.Path) -> None:
|
||||
"""Comments, empty pages, og:title, images, and meta tags."""
|
||||
converter = HtmlConverter()
|
||||
html = """<!DOCTYPE html>
|
||||
<html lang="en"><head>
|
||||
<base href="https://host/base/">
|
||||
<meta property="og:title" content="">
|
||||
<meta name="author" content="Ada">
|
||||
<meta property="og:site_name" content="Site">
|
||||
<meta property="article:published_time" content="2020-01-01">
|
||||
<meta name="unused" content="no">
|
||||
<meta name="description">
|
||||
<link rel="canonical" href="https://host/c">
|
||||
<!-- secret -->
|
||||
</head><body>
|
||||
<h1> </h1>
|
||||
<article><h1>Real</h1><img src="pic.png" alt="pic"></article>
|
||||
</body></html>
|
||||
"""
|
||||
path = tmp_path / "page.html"
|
||||
path.write_text(html, encoding="utf-8")
|
||||
source = _source(path, origin_kind="url", origin="https://host/p")
|
||||
result = converter.convert(source)
|
||||
assert "secret" not in result.body
|
||||
assert "https://host/base/pic.png" in result.body
|
||||
assert result.source_meta["author"] == "Ada"
|
||||
assert result.source_meta["canonical"] == "https://host/c"
|
||||
assert result.title == "Real"
|
||||
empty = tmp_path / "empty.html"
|
||||
empty.write_text("<html><body><script>only</script></body></html>", encoding="utf-8")
|
||||
partial = converter.convert(_source(empty, origin_kind="url", origin="https://host/e"))
|
||||
assert partial.conversion == "partial"
|
||||
og = tmp_path / "og.html"
|
||||
og.write_text(
|
||||
'<html><head><meta property="og:title" content="Og"></head><body><p>x</p></body></html>',
|
||||
encoding="utf-8",
|
||||
)
|
||||
assert converter.convert(_source(og)).title == "Og"
|
||||
bare = tmp_path / "bare.html"
|
||||
bare.write_text("<html><body><p>x</p></body></html>", encoding="utf-8")
|
||||
assert converter.convert(_source(bare)).title is None
|
||||
blank_h1 = tmp_path / "blankh1.html"
|
||||
blank_h1.write_text("<html><body><h1> </h1><p>x</p></body></html>", encoding="utf-8")
|
||||
assert converter.convert(_source(blank_h1)).title is None
|
||||
|
||||
|
||||
def test_pdf_metadata_outline_and_failures(tmp_path: pathlib.Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""PDF dates, outline headings, a scanned file, and read failures."""
|
||||
aware = datetime.datetime(2020, 1, 2, tzinfo=datetime.UTC)
|
||||
naive = datetime.datetime(2020, 1, 1) # noqa: DTZ001
|
||||
assert _iso_date(None) is None
|
||||
assert _iso_date(naive).startswith("2020-01-01T")
|
||||
assert _iso_date(aware).startswith("2020-01-02T")
|
||||
assert _iso_date("D:20200103") == "2020-01-03T00:00:00Z"
|
||||
assert _iso_date("nope") == "nope"
|
||||
assert _iso_date("") is None
|
||||
|
||||
info_obj = types.SimpleNamespace(
|
||||
title="T",
|
||||
author="",
|
||||
subject="S",
|
||||
keywords=None,
|
||||
creation_date=naive,
|
||||
modification_date=aware,
|
||||
)
|
||||
empty = types.SimpleNamespace(pages=[1], metadata=None)
|
||||
info_reader = types.SimpleNamespace(pages=[1, 2], metadata=info_obj)
|
||||
assert _metadata(empty) == {"page_count": 1} # type: ignore[arg-type]
|
||||
info = _metadata(info_reader) # type: ignore[arg-type]
|
||||
assert info["title"] == "T"
|
||||
assert "author" not in info
|
||||
|
||||
class _OutlineReader:
|
||||
"""Fake reader whose first outline item has no destination."""
|
||||
|
||||
def get_destination_page_number(self, item: object) -> int:
|
||||
if getattr(item, "title", None) == "":
|
||||
raise ValueError("bad")
|
||||
return 0
|
||||
|
||||
found: list[tuple[int, int, str]] = []
|
||||
bad = types.SimpleNamespace(title="")
|
||||
good = types.SimpleNamespace(title=None)
|
||||
_walk_outline(_OutlineReader(), [bad, [good]], 5, found) # type: ignore[arg-type]
|
||||
assert found == [(0, 6, "Untitled")]
|
||||
assert _render(["body"], [(0, 2, "Chapter")]) == "## Chapter\n\nbody\n"
|
||||
assert "## Page" not in _render([""], [(1, 2, "Later")])
|
||||
|
||||
class _BrokenOutline:
|
||||
"""Fake reader whose outline cannot be read."""
|
||||
|
||||
@property
|
||||
def outline(self) -> list[object]:
|
||||
raise pypdf.errors.PdfReadError("no outline")
|
||||
|
||||
assert _outline(_BrokenOutline()) == [] # type: ignore[arg-type]
|
||||
|
||||
writer = pypdf.PdfWriter()
|
||||
writer.add_blank_page(width=200, height=200)
|
||||
parent = writer.add_outline_item("Parent", 0)
|
||||
writer.add_outline_item("Child", 0, parent=parent)
|
||||
writer.add_metadata({"/Title": "Outlined", "/CreationDate": "D:20200101120000"})
|
||||
buffer = io.BytesIO()
|
||||
writer.write(buffer)
|
||||
outlined = tmp_path / "outlined.pdf"
|
||||
outlined.write_bytes(buffer.getvalue())
|
||||
converted = PdfConverter().convert(_source(outlined))
|
||||
assert "Parent" in converted.body
|
||||
assert converted.title == "Outlined"
|
||||
|
||||
blank = tmp_path / "blank.pdf"
|
||||
blank_writer = pypdf.PdfWriter()
|
||||
blank_writer.add_blank_page(width=200, height=200)
|
||||
blank_buffer = io.BytesIO()
|
||||
blank_writer.write(blank_buffer)
|
||||
blank.write_bytes(blank_buffer.getvalue())
|
||||
scanned = PdfConverter().convert(_source(blank))
|
||||
assert scanned.conversion == "partial"
|
||||
assert any("scanned" in warning for warning in scanned.warnings)
|
||||
|
||||
corrupt = tmp_path / "corrupt.pdf"
|
||||
corrupt.write_bytes(b"%PDF-1.7 not a real file")
|
||||
with pytest.raises(ConversionError):
|
||||
PdfConverter().convert(_source(corrupt))
|
||||
|
||||
def extract_text() -> str:
|
||||
raise pypdf.errors.PdfReadError("page")
|
||||
|
||||
reader = types.SimpleNamespace(
|
||||
is_encrypted=False,
|
||||
metadata=None,
|
||||
pages=[types.SimpleNamespace(extract_text=extract_text)],
|
||||
outline=[],
|
||||
)
|
||||
monkeypatch.setattr(pypdf, "PdfReader", lambda _path: reader)
|
||||
with pytest.raises(ConversionError, match="page"):
|
||||
PdfConverter().convert(_source(outlined))
|
||||
|
||||
|
||||
def test_archive_edges(tmp_path: pathlib.Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Reuse skips, raw escapes, named raw files, and collision suffixes."""
|
||||
root = _wiki(tmp_path)
|
||||
archiver = RawArchiver(tmp_path / "empty")
|
||||
assert archiver.find_by_sha256("ab" * 32, 1) is None
|
||||
assert _strip_prefix("raw", "raw") == ""
|
||||
assert _strip_prefix("raw/notes/a.txt", "raw") == "notes/a.txt"
|
||||
registry = default_registry()
|
||||
settings = default_settings()
|
||||
already = root / "raw" / "articles" / "already.md"
|
||||
already.parent.mkdir(parents=True)
|
||||
already.write_text("# Already\n", encoding="utf-8")
|
||||
placed = archiver_on(root).archive(
|
||||
_source(already),
|
||||
registry=registry,
|
||||
settings=settings,
|
||||
raw_override=None,
|
||||
dry_run=False,
|
||||
)
|
||||
assert placed.raw_reused is False
|
||||
with pytest.raises(IngestError, match="already under raw"):
|
||||
archiver_on(root).archive(
|
||||
_source(already),
|
||||
registry=registry,
|
||||
settings=settings,
|
||||
raw_override="other",
|
||||
dry_run=False,
|
||||
)
|
||||
original_stat = pathlib.Path.stat
|
||||
seen: dict[str, int] = {}
|
||||
|
||||
def flaky(self: pathlib.Path, *args: object, **kwargs: object) -> object:
|
||||
if self.name == "bad.bin":
|
||||
seen[str(self)] = seen.get(str(self), 0) + 1
|
||||
if seen[str(self)] > 1:
|
||||
raise OSError("stat failed")
|
||||
return original_stat(self, *args, **kwargs)
|
||||
|
||||
(root / "raw" / "bad.bin").write_bytes(b"x")
|
||||
monkeypatch.setattr(pathlib.Path, "stat", flaky)
|
||||
assert archiver_on(root).find_by_sha256("ab" * 32, 1) is None
|
||||
monkeypatch.undo()
|
||||
|
||||
pipeline = default_pipeline(root)
|
||||
named = tmp_path / "fresh.txt"
|
||||
named.write_text("fresh\n", encoding="utf-8")
|
||||
pipeline.run(str(named), raw="notes/kept.html", raw_only=True)
|
||||
assert (root / "raw" / "notes" / "kept.html").is_file()
|
||||
for index, text in enumerate(("one\n", "two\n", "three\n"), start=1):
|
||||
path = tmp_path / f"dir{index}" / "note.txt"
|
||||
path.parent.mkdir()
|
||||
path.write_text(text, encoding="utf-8")
|
||||
pipeline.run(str(path), raw_only=True)
|
||||
assert (root / "raw" / "articles" / "note-3.txt").is_file()
|
||||
|
||||
|
||||
def archiver_on(root: pathlib.Path) -> RawArchiver:
|
||||
"""Return an archiver bound to ``root``."""
|
||||
return RawArchiver(root)
|
||||
|
||||
|
||||
def test_download_and_log_edges(tmp_path: pathlib.Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""URL errors, a failed save, userinfo, and a log that cannot be bumped."""
|
||||
root = _wiki(tmp_path)
|
||||
registry = default_registry()
|
||||
fetcher = UrlFetcher(registry, opener=_Opener(error="timed out"))
|
||||
with pytest.raises(IngestError, match="timed out"):
|
||||
fetcher.fetch("https://host/a.txt", root / "ingest")
|
||||
assert _without_userinfo("https://user:secret@host:443/a.txt") == "https://host:443/a.txt"
|
||||
assert _without_userinfo("https://host/a.txt") == "https://host/a.txt"
|
||||
|
||||
original_write = pathlib.Path.write_bytes
|
||||
|
||||
def fail_write(self: pathlib.Path, data: bytes) -> int:
|
||||
original_write(self, data)
|
||||
raise OSError("disk")
|
||||
|
||||
monkeypatch.setattr(pathlib.Path, "write_bytes", fail_write)
|
||||
fetcher = UrlFetcher(registry, opener=_Opener(b"hello\n"))
|
||||
with pytest.raises(OSError, match="disk"):
|
||||
fetcher.fetch("https://host/b.txt", root / "ingest")
|
||||
monkeypatch.undo()
|
||||
assert not list((root / "ingest").glob("*.partial"))
|
||||
|
||||
log = IngestLog(root)
|
||||
log.append(day="2026-09-30", origin="o", raw_rel="raw/a", page_rel=None, dry_run=True)
|
||||
(root / "wiki" / "log.md").unlink()
|
||||
log.append(day="2026-09-30", origin="o", raw_rel="raw/a", page_rel="wiki/a.md", dry_run=False)
|
||||
odd = root / "wiki" / "log.md"
|
||||
odd.write_text("version: beta\n# Log", encoding="utf-8", newline="\n")
|
||||
log.append(day="2026-09-30", origin="o", raw_rel="raw/a", page_rel=None, dry_run=False)
|
||||
text = odd.read_text(encoding="utf-8")
|
||||
assert "version: beta" in text
|
||||
assert text.endswith("\n")
|
||||
assert log_mod._set_updated("no updated here\n", "2026-09-30") == "no updated here\n"
|
||||
assert log_mod._bump_version("version: 1\n") == "version: 1\n"
|
||||
assert log_mod._bump_version("no version here\n") == "no version here\n"
|
||||
|
||||
|
||||
def test_page_writer_edges(tmp_path: pathlib.Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Heading fences, links, fixer failures, and target prefixes."""
|
||||
body = "# Title\n\n```\n# inside\n```\n\n###### Six\n"
|
||||
shifted = pages._shift_headings(body)
|
||||
assert "```\n# inside\n```" in shifted
|
||||
assert "###### Six" in shifted
|
||||
assert pages._keeps_target("?q", tmp_path) is True
|
||||
assert pages._keeps_target("missing.md", tmp_path) is False
|
||||
assert pages._body_after_frontmatter("no front\n") == "no front\n"
|
||||
assert pages._body_after_frontmatter("---\nonly\n") == "---\nonly\n"
|
||||
assert pages._split_ending("a\r\n") == ("\r\n", "a")
|
||||
assert pages._split_ending("a") == ("", "a")
|
||||
assert pages._strip_wiki_prefix("wiki") == ""
|
||||
assert pages._strip_wiki_prefix("wiki/concepts") == "concepts"
|
||||
|
||||
root = _wiki(tmp_path)
|
||||
writer = pages.RawPageWriter(root)
|
||||
source_path = tmp_path / "note.txt"
|
||||
source_path.write_text("note\n", encoding="utf-8")
|
||||
source = _source(source_path)
|
||||
archive_path = root / "raw" / "articles" / "note.txt"
|
||||
archive_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
archive_path.write_text("note\n", encoding="utf-8")
|
||||
archive = ArchiveResult(archive_path, "raw/articles/note.txt", "ab" * 32, False, [])
|
||||
conversion = ConversionResult("plain\n", "Note", {}, "complete", [], "text")
|
||||
|
||||
def broken_fix(self: MDoctor, path: str) -> None:
|
||||
raise PyMarkdownApiException("fix")
|
||||
|
||||
def broken_scan(self: MDoctor, path: str) -> list:
|
||||
raise PyMarkdownApiException("scan")
|
||||
|
||||
monkeypatch.setattr(MDoctor, "fix_file_programmatically", broken_fix)
|
||||
monkeypatch.setattr(MDoctor, "scan_file_programmatically", broken_scan)
|
||||
monkeypatch.setattr(pages, "_body_after_frontmatter", lambda _text: "no heading\n")
|
||||
page = root / "wiki" / "sources" / "note.md"
|
||||
writer.write(
|
||||
source=source,
|
||||
archive=archive,
|
||||
conversion=conversion,
|
||||
page=page,
|
||||
dry_run=False,
|
||||
)
|
||||
assert page.is_file()
|
||||
assert page.read_text(encoding="utf-8").count("# Note") == 1
|
||||
|
||||
|
||||
def test_pipeline_edges(tmp_path: pathlib.Path) -> None:
|
||||
"""Dry-run downloads, pending pages, and a forced duplicate."""
|
||||
root = _wiki(tmp_path)
|
||||
pipeline = default_pipeline(root)
|
||||
pipeline.fetchers[0] = UrlFetcher(pipeline.registry, opener=_Opener(b"hello\n"))
|
||||
result = pipeline.run("https://user:secret@host/hello.txt", dry_run=True)
|
||||
assert result.dry_run is True
|
||||
assert list((root / "ingest").iterdir()) == []
|
||||
with pytest.raises(IngestError, match="stay inside raw"):
|
||||
pipeline.run("https://host/escape.txt", raw="../outside")
|
||||
assert list((root / "ingest").glob("*.txt")) == []
|
||||
|
||||
bare = IngestPipeline(
|
||||
root,
|
||||
[],
|
||||
ConverterRegistry(),
|
||||
pipeline.archiver,
|
||||
pipeline.page_writer,
|
||||
pipeline.log_writer,
|
||||
default_settings(),
|
||||
)
|
||||
local = tmp_path / "local.txt"
|
||||
local.write_text("local\n", encoding="utf-8")
|
||||
with pytest.raises(IngestError, match="Source not found"):
|
||||
bare.run(str(local))
|
||||
|
||||
(root / "wiki" / "skip.md").mkdir()
|
||||
(root / "wiki" / "bad.md").write_text("---\n[\n---\n", encoding="utf-8")
|
||||
(root / "wiki" / "sources").mkdir(exist_ok=True)
|
||||
(root / "wiki" / "sources" / "stringy.md").write_text(
|
||||
"---\nname: stringy\nstatus: raw\nsources: raw/a.txt\ningest: plain\n---\n\n# S\n",
|
||||
encoding="utf-8",
|
||||
newline="\n",
|
||||
)
|
||||
pending = pipeline.list_pending()
|
||||
assert any(item["page_path"].endswith("stringy.md") for item in pending)
|
||||
assert _citing_page(root, "raw/a.txt", "nope") == "wiki/sources/stringy.md"
|
||||
captured = io_capture(_print_pending, pipeline, False)
|
||||
assert "stringy.md" in captured
|
||||
shutil.rmtree(root / "wiki")
|
||||
assert pipeline.list_pending() == []
|
||||
assert _citing_page(root, "raw/a.txt", "nope") is None
|
||||
|
||||
root = _wiki(tmp_path / "second")
|
||||
pipeline = default_pipeline(root)
|
||||
first = tmp_path / "first.txt"
|
||||
second = tmp_path / "other" / "second.txt"
|
||||
first.write_text("same\n", encoding="utf-8")
|
||||
second.parent.mkdir()
|
||||
second.write_text("same\n", encoding="utf-8")
|
||||
pipeline.run(str(first))
|
||||
forced = pipeline.run(str(second), force=True)
|
||||
assert forced.page_path == "wiki/sources/first.md"
|
||||
|
||||
|
||||
def io_capture(func: typing.Callable, *args: object) -> str:
|
||||
"""Return stdout from a function that prints."""
|
||||
buffer = io.StringIO()
|
||||
with contextlib.redirect_stdout(buffer):
|
||||
func(*args)
|
||||
return buffer.getvalue()
|
||||
@@ -680,6 +680,29 @@ WARNING run_tests: pytest short summary missing from output; failure details com
|
||||
- [ ] `0`, or a missing key, draws every node
|
||||
- [ ] A positive integer hides a node whose `unique_points` is
|
||||
below that integer until one of its parents is clicked
|
||||
- [ ] Omit gitignored markdown from the wiki graph (SPEC 7.4.17)
|
||||
- [ ] Failing tests in `tests/test_graph_generator.py` for the
|
||||
acceptances in SPEC 7.4.17, 7.4.17.1, and 7.4.17.2:
|
||||
`secret.md` absent from JSON nodes, edges, and orphans;
|
||||
a link to it creates no edge; `max_files` 1 still keeps
|
||||
`kept.md`; `drafts/` omits `drafts/note.md` and keeps
|
||||
`wiki/note.md`; a `# secret.md` comment does not omit;
|
||||
`notes/.gitignore` omits `notes/scratch.md` only; with no
|
||||
`.gitignore`, `.git/*.md` is absent and a root page stays
|
||||
- [ ] In `wiki_graph_main` (`src/fiona/wiki/graph_generator.py`),
|
||||
drop a match before the `max_files` cut. Read patterns with
|
||||
`gitignore_patterns_from_text` and test them with
|
||||
`path_is_ignored` (`src/fiona/wiki/ignore.py`), the same
|
||||
rules as SPEC 7.2.10. The scan-root `.gitignore` applies to
|
||||
the whole tree. A subdirectory `.gitignore` applies only
|
||||
under that subdirectory. Also drop any path under `.git/`
|
||||
- [ ] Do not parse an omitted file. Drop an edge whose target
|
||||
resolves to an omitted file, so it never becomes a node
|
||||
- [ ] HTML uses that same graph. No new settings key: the filter
|
||||
always runs
|
||||
- [ ] Housekeeping's orphan list follows automatically, because
|
||||
`graph_step` builds the graph through `wiki_graph_main`
|
||||
(SPEC 7.26.6)
|
||||
- [ ] in spec, refine the sectioning by compiling the modules,
|
||||
and their functions as subsections
|
||||
- [ ] eg 4.1. librarian then 4.1.1 librarian toc, to make it cleaner
|
||||
|
||||
Reference in New Issue
Block a user