diff --git a/SPEC.md b/SPEC.md index 5004747..290c07c 100644 --- a/SPEC.md +++ b/SPEC.md @@ -1,9 +1,9 @@ --- name: Fiona-Assistant Specification -version: 1.8.0 +version: 1.9.0 title: Project Spec Sheet created: 2026-09-23 -updated: 2026-10-02 +updated: 2026-10-05 type: summary --- @@ -137,7 +137,11 @@ already moved (*7.1.5.2.7.*), a dry run that does not log writes follow-up hint for style violations left on a page (*7.1.5.5.6.*), single-file `mdoctor` (*7.4.4.*), compact housekeeping sanity details (*7.2.5.1.*), a quiet graph overwrite (*7.1.4.5.9.1.*), and a usage -section in the README (*7.18.4.*). +section in the README (*7.18.4.*). Revision 1.9.0 makes +`wiki ingest` stop with exit 4 on a URL that needs a login +(*7.1.5.2.8.*), adds `--origin` for a file saved by hand +(*7.1.5.2.9.*), and tells the agent how to get the source instead +(*7.1.5.14.*). All three are unbuilt. Each feature section starts with a `Status` line. Requirements use `- *N.M.K.*`. Each `*acceptance*` sits under the requirement it tests. @@ -844,7 +848,8 @@ The contract is section 7.1.5. ```text fiona-assistant wiki ingest [--wiki DIR] [-r|--raw PATH] - [-t|--target PATH] [--raw-only] [-f|--force] [--dry-run] [--json] + [-t|--target PATH] [--origin URL] [--raw-only] [-f|--force] + [--dry-run] [--json] fiona-assistant wiki ingest --pending [--wiki DIR] [--json] ``` @@ -854,6 +859,8 @@ fiona-assistant wiki ingest --pending [--wiki DIR] [--json] `Error: Directory '' is not an llm-wiki directory` and exit 1. `` and `--pending` are mutually exclusive, and one of them is required. argparse reports a violation and exits 2. +`NOT_IMPLEMENTED` `--origin URL` names the URL a local file was saved +from (*7.1.5.2.9.*). Human stdout on success, the lines that apply, in this order: @@ -917,6 +924,7 @@ prints `{"pending": [{"page_path": ..., "sources": [...], | 1 | Error on stderr. Nothing written under the wiki. | | 2 | argparse usage error. | | 3 | Raw copy archived. No page: no converter, or conversion failed. | +| 4 | `NOT_IMPLEMENTED` The URL needs a login (*7.1.5.2.8.*). Error on stderr. Nothing written under the wiki. | Errors, each on stderr with exit 1: @@ -932,8 +940,22 @@ Error: --raw cannot be used on a file already under raw/: Error: --target must stay inside wiki/: Error: --target cannot replace Error: Page exists: . Use --force to replace it. +Error: --origin needs a local file: +Error: --origin must be an http or https URL: ``` +`NOT_IMPLEMENTED` A URL that needs a login prints two lines on stderr +and exits 4 (*7.1.5.2.8.*): + +```text +Error: Authentication required: : +Next: save it into ingest/, then run: wiki ingest --origin +``` + +The `Next:` line names the command without the `fiona-assistant` +prefix so it fits one line. The agent fills `` with the path +it saved (*7.1.5.14.*). + A failed conversion prints `Error: Conversion failed: : ` on stderr and exits 3, not 1, because the raw copy stays. `NOT_IMPLEMENTED` The `(moved to ...)` form is *7.1.5.2.7.*. @@ -1152,6 +1174,7 @@ Shipped document (logical content; `wiki init` copies the file): | `fiona.wiki.ingest.user_agent` | string | `fiona-assistant/` | wiki ingest | | `fiona.wiki.ingest.default_target` | string | `sources` | wiki ingest | | `fiona.wiki.ingest.raw_categories` | object of strings | `{}` | wiki ingest | +| `fiona.wiki.ingest.login_url_patterns` | list of regex strings | see below | wiki ingest | Bool pymarkdown properties use `set_boolean_property`, ints use `set_integer_property`, strings use `set_string_property`, applied as @@ -1236,6 +1259,19 @@ name to a directory under `raw/`, such as `{"text": "transcripts"}`. A value that is absent, of the wrong type, or out of range uses the default in the table. The packaged `fiona-settings.json` omits `fiona.wiki.ingest`. Absent keys use the defaults in the table. +`login_url_patterns` is read by *7.1.5.2.8.3.* and is not built +yet. A list replaces the default list, it does not extend it. An +entry that does not compile is skipped with one warning. The +default list: + +```json +[ + "/(login|logon|signin|sign-in)\\b", + "/(sso|saml2?|oauth2?|adfs|cas)(/|$)", + "login\\.microsoftonline\\.com", + "j_security_check" +] +``` A mapping key under `toc_generator.file` whose name is not one of `min_heading`, `max_heading`, `sectionize`, `ignore_files` is an @@ -2593,7 +2629,8 @@ Candidates for later formats are in Appendix C. the environment. The request sends `user_agent` and waits up to `timeout_seconds`. No credentials are sent. - *7.1.5.2.3.* A non-2xx response or a network error exits 1 with - `Error: Download failed: : `. The reason names the + `Error: Download failed: : `. A response that needs + a login is *7.1.5.2.8.* instead. The reason names the HTTP status when there is one. A body larger than `max_download_bytes` stops the download and exits 1. In each case the partial file is deleted and `ingest/` is unchanged. @@ -2636,6 +2673,82 @@ Candidates for later formats are in Appendix C. `.\ingest\a.md` exits 0, then exits 1 with `Error: Source not found: .\ingest\a.md (moved to raw/articles/a.md by an earlier ingest)` on one line. + - *7.1.5.2.8.* `NOT_IMPLEMENTED` A URL that needs a login is an + error, not a source. The tool cannot log in, so it stops and + leaves the next step to the agent (*7.1.5.14.*). A login page + must never reach `raw/` as if it were the source. + - *7.1.5.2.8.1.* The tool sends no credentials, cookies, or + tokens, and reads none from the environment or from a file + (*7.1.5.2.2.*). It never asks for a password. + - *7.1.5.2.8.2.* Three checks run in this order. The first hit + decides the reason: + + | Check | Hit when | Reason text | + | --- | --- | --- | + | Status | The response is HTTP 401 or 403 | ` `, e.g. `401 Unauthorized` | + | Redirect | The final URL differs from the requested URL and its host and path match one of `login_url_patterns` | `redirected to ` | + | Login form | The media type is `text/html` and the body has an `` whose `type` is `password`, in any case or quoting | `login page at ` | + + - *7.1.5.2.8.3.* `login_url_patterns` is a list of regular + expressions, searched case-insensitively in + `` of the final URL. The query string is not + searched. The default list is in section 5.1. The redirect + check is skipped when there was no redirect, so a user can + still ingest a page whose own path holds `login`. + - *7.1.5.2.8.4.* On a hit, stderr has the `Error: + Authentication required` line and the `Next:` line from + section 4.7.6, and the command exits 4. The body is read + within `max_download_bytes` and then dropped. Nothing is + written under `ingest/`, `raw/`, or `wiki/`, and `wiki/log.md` + is unchanged. The URL in both lines is the one the user + passed, with userinfo removed. + - *7.1.5.2.8.5.* `--dry-run` runs the same checks and exits 4 + the same way. `--json` prints nothing on stdout + (*7.1.5.13.*). + - *7.1.5.2.8.6.* An HTML page that passes the three checks but + has no text still converts as `partial` with the warning in + *7.1.5.4.5.5.*. Login detection does not replace that warning. + - *acceptance*: A fake opener that answers 401 exits 4. Stderr + has `Authentication required`, `401`, and the `Next:` line. + `ingest/`, `raw/`, `wiki/`, and `wiki/log.md` have the same + hashes as before. + - *acceptance*: A fake opener that redirects + `https://host/job/x/console` to + `https://host/login?from=%2Fjob%2Fx%2Fconsole` and answers 200 + exits 4 with the reason `redirected to https://host/login?...`. + - *acceptance*: A fake opener that answers 200 with an HTML form + holding `` exits 4 with the reason + `login page at `. + - *acceptance*: A 200 HTML page at `https://host/docs/login`, + with no redirect and no password input, is ingested with + exit 0. + - *acceptance*: A 404 still exits 1 with `Download failed`. + - *7.1.5.2.9.* `NOT_IMPLEMENTED` `--origin URL` records where a + local file came from, so a source saved by hand after + *7.1.5.2.8.* keeps its real provenance. + - *7.1.5.2.9.1.* `--origin` is valid only when `` is a + local path. With a URL source it exits 1 with + `Error: --origin needs a local file: `. A value that + is not `http://` or `https://` exits 1 with + `Error: --origin must be an http or https URL: `. Both + checks run before fetch. + - *7.1.5.2.9.2.* The file is fetched, archived, and moved as any + local file. The page frontmatter then has `ingest.origin` set + to the URL without userinfo, `origin_kind: url`, and + `final_url: null`. The `html` converter uses that URL to make + relative links absolute (*7.1.5.4.5.3.*). The log bullet names + that URL (*7.1.5.6.*). JSON `origin` and `origin_kind` match + the frontmatter. + - *7.1.5.2.9.3.* The slug and default raw name still come from + the file stem (*7.1.5.2.6.*), so the name the agent chose in + `ingest/` is kept. + - *acceptance*: `wiki ingest ingest/console.html --origin + https://host/job/x/console` exits 0. The page has + `origin: https://host/job/x/console`, `origin_kind: url`, + and a relative link `/a` in the file becomes + `https://host/a`. No network call is made. + - *acceptance*: `wiki ingest https://host/a --origin + https://host/b` exits 1 and makes no network call. - *7.1.5.3.* Archive. - *7.1.5.3.1.* The default raw path is `raw//.`. The category is the raw category @@ -2907,10 +3020,36 @@ Candidates for later formats are in Appendix C. - *acceptance*: After ingesting a markdown file with style violations, `wiki sanity` reports none from its raw copy. - *7.1.5.13.* `--json` prints only the JSON object - from section 4.7.6 on stdout, on exit 0 and on exit 3. On exit 1, - stdout is empty. + from section 4.7.6 on stdout, on exit 0 and on exit 3. On exit 1 + and exit 4, stdout is empty. - *acceptance*: The stdout of an exit 3 run parses with `json.loads` and has `conversion: none`. +- *7.1.5.14.* `NOT_IMPLEMENTED` On exit 4 the agent gets the source + another way, or asks the user for it. The Fiona skill teaches + this in section 16.2. The steps, in order: + - *7.1.5.14.1.* Use a tool the agent already has that can read + that host with the user's access, such as an MCP server for + Confluence, SharePoint, Jenkins, or a ticket system, or an + authenticated API script in the wiki's `utils/scripts/`. Save + what it returns into `ingest/.` in its own format: + HTML as `.html`, a PDF as `.pdf`, plain text as `.txt` or `.md`. + Then run the `Next:` command with that file and + `--origin `. + - *7.1.5.14.2.* When no tool has access, stop and ask the user to + save the page into `ingest/`, for example with the browser's + Save As or Print to PDF. Name the URL, the folder, and the + command the agent will run once the file is there. Do not + retry the URL. + - *7.1.5.14.3.* Never ask the user for a password, token, or + cookie. Never put credentials in a URL, a command line, a + page, or the log. + - *7.1.5.14.4.* A tool that returns a summary instead of the + source is not the source. Archive the content the tool + returned as it is, and say in `ingest.warnings` that it came + through that tool. + - *acceptance*: Fiona skill section 4.6 lists exit 4. Section 16.2 + has the three cases above (a tool with access, ask the user, + never ask for credentials) and shows the `--origin` command. ### 7.2. housekeeping @@ -3747,6 +3886,7 @@ is the pip-argument source both scripts call. | P2 | settings setup | 7.16 | Planned | | P2 | Bootstrap install modes | 7.21 | Implemented | | P2 | wiki ingest | 7.1.5 | Partially Implemented | +| P2 | Ingest login detection and `--origin` | 7.1.5.2.8, 7.1.5.2.9, 7.1.5.14 | Planned | | P3 | `--version` | 2.3 | Planned | | P3 | fmlint enforcement of quality signals | 6.2 | Planned | | P3 | Backup archive | 7.13 | Planned | @@ -3801,6 +3941,13 @@ is the pip-argument source both scripts call. 2026-10-02 manual test lost a `git save` commit this way. A TTY prompt with `--yes` (*1.2.2.*) would make agents pass `--yes` on every call. Not decided. +- `wiki ingest` sends no credentials (*7.1.5.2.8.1.*). A cookie + file, a bearer token from an environment variable, or a + `--header` option would let it read some login-only hosts + itself. Each one puts secrets near the wiki. Not decided. +- On exit 4, `--json` prints nothing (*7.1.5.13.*). A JSON error + object such as `{"error": "auth_required", "url": ..., + "reason": ...}` would save the agent parsing stderr. Not decided. --- diff --git a/todo.md b/todo.md index 80abf5a..bb21158 100644 --- a/todo.md +++ b/todo.md @@ -4,6 +4,7 @@ - [P0 — Packaging and Structure Fixes](#p0--packaging-and-structure-fixes) - [Code defects — 2026-09-29 audit](#code-defects--2026-09-29-audit) - [Manual test findings — 2026-10-02](#manual-test-findings--2026-10-02) +- [Ingest of a URL that needs a login](#ingest-of-a-url-that-needs-a-login) - [P1 — Wiki Lifecycle and Version Control](#p1--wiki-lifecycle-and-version-control) - [P2 — Quality Tooling](#p2--quality-tooling) - [P3 — Metadata, Backup, and Documentation](#p3--metadata-backup-and-documentation) @@ -424,6 +425,70 @@ SPEC 1.8.0 holds the requirements. Order is the build order. (SPEC Appendix C). The tester lost a `git save` commit with `git undo`. Listed under P4 as well +## Ingest of a URL that needs a login + +Users paste links to Jenkins, Confluence, SharePoint, and ticket +pages that need a login. Today a 401 or 403 exits 1 with the generic +`Download failed`, and an SSO redirect that ends on a 200 login page +is archived under `raw/` and converted as if it were the source. +SPEC 1.9.0 holds the requirements (7.1.5.2.8, 7.1.5.2.9, 7.1.5.14). +Spec written 2026-10-05; nothing below is built. Order is the build +order. + +- [ ] Detect a login in the URL fetcher (SPEC 7.1.5.2.8). In + `src/fiona/wiki/ingest/fetchers.py`, raise a new + `AuthRequiredError(IngestError)` that carries the URL and the + reason, before anything is written to `ingest/` + - [ ] HTTP 401 and 403 from `HTTPError`: reason ` `. + Other non-2xx stay `Download failed` (SPEC 7.1.5.2.3) + - [ ] Redirect check: final URL differs from the requested URL + and `` matches `login_url_patterns` + (case-insensitive, query string ignored) + - [ ] Login form check: `text/html` body with an `` whose + `type` is `password`, any case or quoting. Run it on the + bytes already read, so the size limit still applies first + - [ ] Tests with a fake opener, no network: 401, 403, redirect to + `/login?from=...`, 200 page with ``, + a `/docs/login` page with no redirect that still ingests, + and a 404 that still exits 1. Each auth case leaves + `ingest/`, `raw/`, `wiki/`, and `wiki/log.md` unchanged +- [ ] `login_url_patterns` setting (SPEC 5.1). Add it to + `src/fiona/wiki/ingest/settings.py` with the default list from + SPEC 5.1. A user list replaces the default. A regex that does + not compile is skipped with one warning + - [ ] Tests: default list, user override, one bad regex +- [ ] Exit 4 and the two stderr lines (SPEC 4.7.6). Catch + `AuthRequiredError` in the ingest CLI handler, print the + `Error: Authentication required: : ` line and the + `Next:` line, exit 4. URL without userinfo. `--json` stdout + stays empty (SPEC 7.1.5.13); `--dry-run` exits 4 the same way + - [ ] Tests: exit code, both stderr lines, empty stdout with + `--json`, dry run, no `Traceback` (SPEC 1.2.7) +- [ ] `--origin URL` (SPEC 7.1.5.2.9). Valid only with a local file, + `http`/`https` only, both checks before fetch. Sets + `ingest.origin`, `origin_kind: url`, `final_url: null`, feeds + the `html` converter's base URL and the log bullet. Slug stays + the file stem + - [ ] Tests: saved HTML with a relative link `/a` becomes + `https://host/a`; URL source plus `--origin` exits 1; bad + scheme exits 1; no network call in any case +- [ ] Fiona skill (SPEC 7.1.5.14). In + `src/fiona/data/llm_wiki/utils/skills/fiona/SKILL.md`: add exit + 4 to the section 4.6 exit table and `--origin` to its options; + rewrite the authentication bullet in section 16.2 into the + three steps (use a tool with access, else ask the user to save + it into `ingest/`, never ask for credentials) with the + `--origin` command. Bump the skill version +- [ ] README ingest notes (SPEC 7.18.4): one line on exit 4 and + `--origin` +- [ ] Verify the spec (project agent rule 2.1 step 4): drop the + `NOT_IMPLEMENTED` markers on 4.7.6, 7.1.5.2.8, 7.1.5.2.9, + 7.1.5.14, and the `login_url_patterns` prose in 5.1; set the + Appendix B row to `Implemented`; bump SPEC +- [ ] Open questions left in SPEC Appendix C, not part of this + work: credential support (cookie file, token, `--header`) and + a JSON error object on exit 4 + ## P1 — Wiki Lifecycle and Version Control - [x] `wiki init` hardening (SPEC 7.1.1)