wiki ingest additional spec for auth required links

This commit is contained in:
nvaldez
2026-10-05 09:07:02 +02:00
committed by nvaldez
parent 9ff5af588a
commit dcc0536f6e
2 changed files with 219 additions and 7 deletions
+154 -7
View File
@@ -1,9 +1,9 @@
---
name: Fiona-Assistant Specification
version: 1.8.0
version: 1.9.0
title: Project Spec Sheet
created: 2026-09-23
updated: 2026-10-02
updated: 2026-10-05
type: summary
---
@@ -137,7 +137,11 @@ already moved (*7.1.5.2.7.*), a dry run that does not log writes
follow-up hint for style violations left on a page (*7.1.5.5.6.*),
single-file `mdoctor` (*7.4.4.*), compact housekeeping sanity details
(*7.2.5.1.*), a quiet graph overwrite (*7.1.4.5.9.1.*), and a usage
section in the README (*7.18.4.*).
section in the README (*7.18.4.*). Revision 1.9.0 makes
`wiki ingest` stop with exit 4 on a URL that needs a login
(*7.1.5.2.8.*), adds `--origin` for a file saved by hand
(*7.1.5.2.9.*), and tells the agent how to get the source instead
(*7.1.5.14.*). All three are unbuilt.
Each feature section starts with a `Status` line. Requirements use
`- *N.M.K.*`. Each `*acceptance*` sits under the requirement it tests.
@@ -844,7 +848,8 @@ The contract is section 7.1.5.
```text
fiona-assistant wiki ingest <source> [--wiki DIR] [-r|--raw PATH]
[-t|--target PATH] [--raw-only] [-f|--force] [--dry-run] [--json]
[-t|--target PATH] [--origin URL] [--raw-only] [-f|--force]
[--dry-run] [--json]
fiona-assistant wiki ingest --pending [--wiki DIR] [--json]
```
@@ -854,6 +859,8 @@ fiona-assistant wiki ingest --pending [--wiki DIR] [--json]
`Error: Directory '<path>' is not an llm-wiki directory` and exit 1.
`<source>` and `--pending` are mutually exclusive, and one of them is
required. argparse reports a violation and exits 2.
`NOT_IMPLEMENTED` `--origin URL` names the URL a local file was saved
from (*7.1.5.2.9.*).
Human stdout on success, the lines that apply, in this order:
@@ -917,6 +924,7 @@ prints `{"pending": [{"page_path": ..., "sources": [...],
| 1 | Error on stderr. Nothing written under the wiki. |
| 2 | argparse usage error. |
| 3 | Raw copy archived. No page: no converter, or conversion failed. |
| 4 | `NOT_IMPLEMENTED` The URL needs a login (*7.1.5.2.8.*). Error on stderr. Nothing written under the wiki. |
Errors, each on stderr with exit 1:
@@ -932,8 +940,22 @@ Error: --raw cannot be used on a file already under raw/: <path>
Error: --target must stay inside wiki/: <value>
Error: --target cannot replace <path>
Error: Page exists: <path>. Use --force to replace it.
Error: --origin needs a local file: <source>
Error: --origin must be an http or https URL: <value>
```
`NOT_IMPLEMENTED` A URL that needs a login prints two lines on stderr
and exits 4 (*7.1.5.2.8.*):
```text
Error: Authentication required: <url>: <reason>
Next: save it into ingest/, then run: wiki ingest <file> --origin <url>
```
The `Next:` line names the command without the `fiona-assistant`
prefix so it fits one line. The agent fills `<file>` with the path
it saved (*7.1.5.14.*).
A failed conversion prints `Error: Conversion failed: <converter>:
<reason>` on stderr and exits 3, not 1, because the raw copy stays.
`NOT_IMPLEMENTED` The `(moved to ...)` form is *7.1.5.2.7.*.
@@ -1152,6 +1174,7 @@ Shipped document (logical content; `wiki init` copies the file):
| `fiona.wiki.ingest.user_agent` | string | `fiona-assistant/<package version>` | wiki ingest |
| `fiona.wiki.ingest.default_target` | string | `sources` | wiki ingest |
| `fiona.wiki.ingest.raw_categories` | object of strings | `{}` | wiki ingest |
| `fiona.wiki.ingest.login_url_patterns` | list of regex strings | see below | wiki ingest |
Bool pymarkdown properties use `set_boolean_property`, ints use
`set_integer_property`, strings use `set_string_property`, applied as
@@ -1236,6 +1259,19 @@ name to a directory under `raw/`, such as `{"text": "transcripts"}`.
A value that is absent, of the wrong type, or out of range uses the
default in the table. The packaged `fiona-settings.json` omits
`fiona.wiki.ingest`. Absent keys use the defaults in the table.
`login_url_patterns` is read by *7.1.5.2.8.3.* and is not built
yet. A list replaces the default list, it does not extend it. An
entry that does not compile is skipped with one warning. The
default list:
```json
[
"/(login|logon|signin|sign-in)\\b",
"/(sso|saml2?|oauth2?|adfs|cas)(/|$)",
"login\\.microsoftonline\\.com",
"j_security_check"
]
```
A mapping key under `toc_generator.file` whose name is not one of
`min_heading`, `max_heading`, `sectionize`, `ignore_files` is an
@@ -2593,7 +2629,8 @@ Candidates for later formats are in Appendix C.
the environment. The request sends `user_agent` and waits up to
`timeout_seconds`. No credentials are sent.
- *7.1.5.2.3.* A non-2xx response or a network error exits 1 with
`Error: Download failed: <url>: <reason>`. The reason names the
`Error: Download failed: <url>: <reason>`. A response that needs
a login is *7.1.5.2.8.* instead. The reason names the
HTTP status when there is one. A body larger than
`max_download_bytes` stops the download and exits 1. In each
case the partial file is deleted and `ingest/` is unchanged.
@@ -2636,6 +2673,82 @@ Candidates for later formats are in Appendix C.
`.\ingest\a.md` exits 0, then exits 1 with
`Error: Source not found: .\ingest\a.md (moved to
raw/articles/a.md by an earlier ingest)` on one line.
- *7.1.5.2.8.* `NOT_IMPLEMENTED` A URL that needs a login is an
error, not a source. The tool cannot log in, so it stops and
leaves the next step to the agent (*7.1.5.14.*). A login page
must never reach `raw/` as if it were the source.
- *7.1.5.2.8.1.* The tool sends no credentials, cookies, or
tokens, and reads none from the environment or from a file
(*7.1.5.2.2.*). It never asks for a password.
- *7.1.5.2.8.2.* Three checks run in this order. The first hit
decides the reason:
| Check | Hit when | Reason text |
| --- | --- | --- |
| Status | The response is HTTP 401 or 403 | `<status> <phrase>`, e.g. `401 Unauthorized` |
| Redirect | The final URL differs from the requested URL and its host and path match one of `login_url_patterns` | `redirected to <final url>` |
| Login form | The media type is `text/html` and the body has an `<input>` whose `type` is `password`, in any case or quoting | `login page at <final url>` |
- *7.1.5.2.8.3.* `login_url_patterns` is a list of regular
expressions, searched case-insensitively in
`<host><path>` of the final URL. The query string is not
searched. The default list is in section 5.1. The redirect
check is skipped when there was no redirect, so a user can
still ingest a page whose own path holds `login`.
- *7.1.5.2.8.4.* On a hit, stderr has the `Error:
Authentication required` line and the `Next:` line from
section 4.7.6, and the command exits 4. The body is read
within `max_download_bytes` and then dropped. Nothing is
written under `ingest/`, `raw/`, or `wiki/`, and `wiki/log.md`
is unchanged. The URL in both lines is the one the user
passed, with userinfo removed.
- *7.1.5.2.8.5.* `--dry-run` runs the same checks and exits 4
the same way. `--json` prints nothing on stdout
(*7.1.5.13.*).
- *7.1.5.2.8.6.* An HTML page that passes the three checks but
has no text still converts as `partial` with the warning in
*7.1.5.4.5.5.*. Login detection does not replace that warning.
- *acceptance*: A fake opener that answers 401 exits 4. Stderr
has `Authentication required`, `401`, and the `Next:` line.
`ingest/`, `raw/`, `wiki/`, and `wiki/log.md` have the same
hashes as before.
- *acceptance*: A fake opener that redirects
`https://host/job/x/console` to
`https://host/login?from=%2Fjob%2Fx%2Fconsole` and answers 200
exits 4 with the reason `redirected to https://host/login?...`.
- *acceptance*: A fake opener that answers 200 with an HTML form
holding `<INPUT TYPE='password'>` exits 4 with the reason
`login page at <url>`.
- *acceptance*: A 200 HTML page at `https://host/docs/login`,
with no redirect and no password input, is ingested with
exit 0.
- *acceptance*: A 404 still exits 1 with `Download failed`.
- *7.1.5.2.9.* `NOT_IMPLEMENTED` `--origin URL` records where a
local file came from, so a source saved by hand after
*7.1.5.2.8.* keeps its real provenance.
- *7.1.5.2.9.1.* `--origin` is valid only when `<source>` is a
local path. With a URL source it exits 1 with
`Error: --origin needs a local file: <source>`. A value that
is not `http://` or `https://` exits 1 with
`Error: --origin must be an http or https URL: <value>`. Both
checks run before fetch.
- *7.1.5.2.9.2.* The file is fetched, archived, and moved as any
local file. The page frontmatter then has `ingest.origin` set
to the URL without userinfo, `origin_kind: url`, and
`final_url: null`. The `html` converter uses that URL to make
relative links absolute (*7.1.5.4.5.3.*). The log bullet names
that URL (*7.1.5.6.*). JSON `origin` and `origin_kind` match
the frontmatter.
- *7.1.5.2.9.3.* The slug and default raw name still come from
the file stem (*7.1.5.2.6.*), so the name the agent chose in
`ingest/` is kept.
- *acceptance*: `wiki ingest ingest/console.html --origin
https://host/job/x/console` exits 0. The page has
`origin: https://host/job/x/console`, `origin_kind: url`,
and a relative link `/a` in the file becomes
`https://host/a`. No network call is made.
- *acceptance*: `wiki ingest https://host/a --origin
https://host/b` exits 1 and makes no network call.
- *7.1.5.3.* Archive.
- *7.1.5.3.1.* The default raw path is
`raw/<category>/<slug>.<ext>`. The category is the raw category
@@ -2907,10 +3020,36 @@ Candidates for later formats are in Appendix C.
- *acceptance*: After ingesting a markdown file with style
violations, `wiki sanity` reports none from its raw copy.
- *7.1.5.13.* `--json` prints only the JSON object
from section 4.7.6 on stdout, on exit 0 and on exit 3. On exit 1,
stdout is empty.
from section 4.7.6 on stdout, on exit 0 and on exit 3. On exit 1
and exit 4, stdout is empty.
- *acceptance*: The stdout of an exit 3 run parses with
`json.loads` and has `conversion: none`.
- *7.1.5.14.* `NOT_IMPLEMENTED` On exit 4 the agent gets the source
another way, or asks the user for it. The Fiona skill teaches
this in section 16.2. The steps, in order:
- *7.1.5.14.1.* Use a tool the agent already has that can read
that host with the user's access, such as an MCP server for
Confluence, SharePoint, Jenkins, or a ticket system, or an
authenticated API script in the wiki's `utils/scripts/`. Save
what it returns into `ingest/<slug>.<ext>` in its own format:
HTML as `.html`, a PDF as `.pdf`, plain text as `.txt` or `.md`.
Then run the `Next:` command with that file and
`--origin <url>`.
- *7.1.5.14.2.* When no tool has access, stop and ask the user to
save the page into `ingest/`, for example with the browser's
Save As or Print to PDF. Name the URL, the folder, and the
command the agent will run once the file is there. Do not
retry the URL.
- *7.1.5.14.3.* Never ask the user for a password, token, or
cookie. Never put credentials in a URL, a command line, a
page, or the log.
- *7.1.5.14.4.* A tool that returns a summary instead of the
source is not the source. Archive the content the tool
returned as it is, and say in `ingest.warnings` that it came
through that tool.
- *acceptance*: Fiona skill section 4.6 lists exit 4. Section 16.2
has the three cases above (a tool with access, ask the user,
never ask for credentials) and shows the `--origin` command.
### 7.2. housekeeping
@@ -3747,6 +3886,7 @@ is the pip-argument source both scripts call.
| P2 | settings setup | 7.16 | Planned |
| P2 | Bootstrap install modes | 7.21 | Implemented |
| P2 | wiki ingest | 7.1.5 | Partially Implemented |
| P2 | Ingest login detection and `--origin` | 7.1.5.2.8, 7.1.5.2.9, 7.1.5.14 | Planned |
| P3 | `--version` | 2.3 | Planned |
| P3 | fmlint enforcement of quality signals | 6.2 | Planned |
| P3 | Backup archive | 7.13 | Planned |
@@ -3801,6 +3941,13 @@ is the pip-argument source both scripts call.
2026-10-02 manual test lost a `git save` commit this way. A TTY
prompt with `--yes` (*1.2.2.*) would make agents pass `--yes` on
every call. Not decided.
- `wiki ingest` sends no credentials (*7.1.5.2.8.1.*). A cookie
file, a bearer token from an environment variable, or a
`--header` option would let it read some login-only hosts
itself. Each one puts secrets near the wiki. Not decided.
- On exit 4, `--json` prints nothing (*7.1.5.13.*). A JSON error
object such as `{"error": "auth_required", "url": ...,
"reason": ...}` would save the agent parsing stderr. Not decided.
---
+65
View File
@@ -4,6 +4,7 @@
- [P0 — Packaging and Structure Fixes](#p0--packaging-and-structure-fixes)
- [Code defects — 2026-09-29 audit](#code-defects--2026-09-29-audit)
- [Manual test findings — 2026-10-02](#manual-test-findings--2026-10-02)
- [Ingest of a URL that needs a login](#ingest-of-a-url-that-needs-a-login)
- [P1 — Wiki Lifecycle and Version Control](#p1--wiki-lifecycle-and-version-control)
- [P2 — Quality Tooling](#p2--quality-tooling)
- [P3 — Metadata, Backup, and Documentation](#p3--metadata-backup-and-documentation)
@@ -424,6 +425,70 @@ SPEC 1.8.0 holds the requirements. Order is the build order.
(SPEC Appendix C). The tester lost a `git save` commit with
`git undo`. Listed under P4 as well
## Ingest of a URL that needs a login
Users paste links to Jenkins, Confluence, SharePoint, and ticket
pages that need a login. Today a 401 or 403 exits 1 with the generic
`Download failed`, and an SSO redirect that ends on a 200 login page
is archived under `raw/` and converted as if it were the source.
SPEC 1.9.0 holds the requirements (7.1.5.2.8, 7.1.5.2.9, 7.1.5.14).
Spec written 2026-10-05; nothing below is built. Order is the build
order.
- [ ] Detect a login in the URL fetcher (SPEC 7.1.5.2.8). In
`src/fiona/wiki/ingest/fetchers.py`, raise a new
`AuthRequiredError(IngestError)` that carries the URL and the
reason, before anything is written to `ingest/`
- [ ] HTTP 401 and 403 from `HTTPError`: reason `<code> <phrase>`.
Other non-2xx stay `Download failed` (SPEC 7.1.5.2.3)
- [ ] Redirect check: final URL differs from the requested URL
and `<host><path>` matches `login_url_patterns`
(case-insensitive, query string ignored)
- [ ] Login form check: `text/html` body with an `<input>` whose
`type` is `password`, any case or quoting. Run it on the
bytes already read, so the size limit still applies first
- [ ] Tests with a fake opener, no network: 401, 403, redirect to
`/login?from=...`, 200 page with `<INPUT TYPE='password'>`,
a `/docs/login` page with no redirect that still ingests,
and a 404 that still exits 1. Each auth case leaves
`ingest/`, `raw/`, `wiki/`, and `wiki/log.md` unchanged
- [ ] `login_url_patterns` setting (SPEC 5.1). Add it to
`src/fiona/wiki/ingest/settings.py` with the default list from
SPEC 5.1. A user list replaces the default. A regex that does
not compile is skipped with one warning
- [ ] Tests: default list, user override, one bad regex
- [ ] Exit 4 and the two stderr lines (SPEC 4.7.6). Catch
`AuthRequiredError` in the ingest CLI handler, print the
`Error: Authentication required: <url>: <reason>` line and the
`Next:` line, exit 4. URL without userinfo. `--json` stdout
stays empty (SPEC 7.1.5.13); `--dry-run` exits 4 the same way
- [ ] Tests: exit code, both stderr lines, empty stdout with
`--json`, dry run, no `Traceback` (SPEC 1.2.7)
- [ ] `--origin URL` (SPEC 7.1.5.2.9). Valid only with a local file,
`http`/`https` only, both checks before fetch. Sets
`ingest.origin`, `origin_kind: url`, `final_url: null`, feeds
the `html` converter's base URL and the log bullet. Slug stays
the file stem
- [ ] Tests: saved HTML with a relative link `/a` becomes
`https://host/a`; URL source plus `--origin` exits 1; bad
scheme exits 1; no network call in any case
- [ ] Fiona skill (SPEC 7.1.5.14). In
`src/fiona/data/llm_wiki/utils/skills/fiona/SKILL.md`: add exit
4 to the section 4.6 exit table and `--origin` to its options;
rewrite the authentication bullet in section 16.2 into the
three steps (use a tool with access, else ask the user to save
it into `ingest/`, never ask for credentials) with the
`--origin` command. Bump the skill version
- [ ] README ingest notes (SPEC 7.18.4): one line on exit 4 and
`--origin`
- [ ] Verify the spec (project agent rule 2.1 step 4): drop the
`NOT_IMPLEMENTED` markers on 4.7.6, 7.1.5.2.8, 7.1.5.2.9,
7.1.5.14, and the `login_url_patterns` prose in 5.1; set the
Appendix B row to `Implemented`; bump SPEC
- [ ] Open questions left in SPEC Appendix C, not part of this
work: credential support (cookie file, token, `--header`) and
a JSON error object on exit 4
## P1 — Wiki Lifecycle and Version Control
- [x] `wiki init` hardening (SPEC 7.1.1)