wiki ingest additional spec for auth required links
This commit is contained in:
@@ -1,9 +1,9 @@
|
||||
---
|
||||
name: Fiona-Assistant Specification
|
||||
version: 1.8.0
|
||||
version: 1.9.0
|
||||
title: Project Spec Sheet
|
||||
created: 2026-09-23
|
||||
updated: 2026-10-02
|
||||
updated: 2026-10-05
|
||||
type: summary
|
||||
---
|
||||
|
||||
@@ -137,7 +137,11 @@ already moved (*7.1.5.2.7.*), a dry run that does not log writes
|
||||
follow-up hint for style violations left on a page (*7.1.5.5.6.*),
|
||||
single-file `mdoctor` (*7.4.4.*), compact housekeeping sanity details
|
||||
(*7.2.5.1.*), a quiet graph overwrite (*7.1.4.5.9.1.*), and a usage
|
||||
section in the README (*7.18.4.*).
|
||||
section in the README (*7.18.4.*). Revision 1.9.0 makes
|
||||
`wiki ingest` stop with exit 4 on a URL that needs a login
|
||||
(*7.1.5.2.8.*), adds `--origin` for a file saved by hand
|
||||
(*7.1.5.2.9.*), and tells the agent how to get the source instead
|
||||
(*7.1.5.14.*). All three are unbuilt.
|
||||
|
||||
Each feature section starts with a `Status` line. Requirements use
|
||||
`- *N.M.K.*`. Each `*acceptance*` sits under the requirement it tests.
|
||||
@@ -844,7 +848,8 @@ The contract is section 7.1.5.
|
||||
|
||||
```text
|
||||
fiona-assistant wiki ingest <source> [--wiki DIR] [-r|--raw PATH]
|
||||
[-t|--target PATH] [--raw-only] [-f|--force] [--dry-run] [--json]
|
||||
[-t|--target PATH] [--origin URL] [--raw-only] [-f|--force]
|
||||
[--dry-run] [--json]
|
||||
fiona-assistant wiki ingest --pending [--wiki DIR] [--json]
|
||||
```
|
||||
|
||||
@@ -854,6 +859,8 @@ fiona-assistant wiki ingest --pending [--wiki DIR] [--json]
|
||||
`Error: Directory '<path>' is not an llm-wiki directory` and exit 1.
|
||||
`<source>` and `--pending` are mutually exclusive, and one of them is
|
||||
required. argparse reports a violation and exits 2.
|
||||
`NOT_IMPLEMENTED` `--origin URL` names the URL a local file was saved
|
||||
from (*7.1.5.2.9.*).
|
||||
|
||||
Human stdout on success, the lines that apply, in this order:
|
||||
|
||||
@@ -917,6 +924,7 @@ prints `{"pending": [{"page_path": ..., "sources": [...],
|
||||
| 1 | Error on stderr. Nothing written under the wiki. |
|
||||
| 2 | argparse usage error. |
|
||||
| 3 | Raw copy archived. No page: no converter, or conversion failed. |
|
||||
| 4 | `NOT_IMPLEMENTED` The URL needs a login (*7.1.5.2.8.*). Error on stderr. Nothing written under the wiki. |
|
||||
|
||||
Errors, each on stderr with exit 1:
|
||||
|
||||
@@ -932,8 +940,22 @@ Error: --raw cannot be used on a file already under raw/: <path>
|
||||
Error: --target must stay inside wiki/: <value>
|
||||
Error: --target cannot replace <path>
|
||||
Error: Page exists: <path>. Use --force to replace it.
|
||||
Error: --origin needs a local file: <source>
|
||||
Error: --origin must be an http or https URL: <value>
|
||||
```
|
||||
|
||||
`NOT_IMPLEMENTED` A URL that needs a login prints two lines on stderr
|
||||
and exits 4 (*7.1.5.2.8.*):
|
||||
|
||||
```text
|
||||
Error: Authentication required: <url>: <reason>
|
||||
Next: save it into ingest/, then run: wiki ingest <file> --origin <url>
|
||||
```
|
||||
|
||||
The `Next:` line names the command without the `fiona-assistant`
|
||||
prefix so it fits one line. The agent fills `<file>` with the path
|
||||
it saved (*7.1.5.14.*).
|
||||
|
||||
A failed conversion prints `Error: Conversion failed: <converter>:
|
||||
<reason>` on stderr and exits 3, not 1, because the raw copy stays.
|
||||
`NOT_IMPLEMENTED` The `(moved to ...)` form is *7.1.5.2.7.*.
|
||||
@@ -1152,6 +1174,7 @@ Shipped document (logical content; `wiki init` copies the file):
|
||||
| `fiona.wiki.ingest.user_agent` | string | `fiona-assistant/<package version>` | wiki ingest |
|
||||
| `fiona.wiki.ingest.default_target` | string | `sources` | wiki ingest |
|
||||
| `fiona.wiki.ingest.raw_categories` | object of strings | `{}` | wiki ingest |
|
||||
| `fiona.wiki.ingest.login_url_patterns` | list of regex strings | see below | wiki ingest |
|
||||
|
||||
Bool pymarkdown properties use `set_boolean_property`, ints use
|
||||
`set_integer_property`, strings use `set_string_property`, applied as
|
||||
@@ -1236,6 +1259,19 @@ name to a directory under `raw/`, such as `{"text": "transcripts"}`.
|
||||
A value that is absent, of the wrong type, or out of range uses the
|
||||
default in the table. The packaged `fiona-settings.json` omits
|
||||
`fiona.wiki.ingest`. Absent keys use the defaults in the table.
|
||||
`login_url_patterns` is read by *7.1.5.2.8.3.* and is not built
|
||||
yet. A list replaces the default list, it does not extend it. An
|
||||
entry that does not compile is skipped with one warning. The
|
||||
default list:
|
||||
|
||||
```json
|
||||
[
|
||||
"/(login|logon|signin|sign-in)\\b",
|
||||
"/(sso|saml2?|oauth2?|adfs|cas)(/|$)",
|
||||
"login\\.microsoftonline\\.com",
|
||||
"j_security_check"
|
||||
]
|
||||
```
|
||||
|
||||
A mapping key under `toc_generator.file` whose name is not one of
|
||||
`min_heading`, `max_heading`, `sectionize`, `ignore_files` is an
|
||||
@@ -2593,7 +2629,8 @@ Candidates for later formats are in Appendix C.
|
||||
the environment. The request sends `user_agent` and waits up to
|
||||
`timeout_seconds`. No credentials are sent.
|
||||
- *7.1.5.2.3.* A non-2xx response or a network error exits 1 with
|
||||
`Error: Download failed: <url>: <reason>`. The reason names the
|
||||
`Error: Download failed: <url>: <reason>`. A response that needs
|
||||
a login is *7.1.5.2.8.* instead. The reason names the
|
||||
HTTP status when there is one. A body larger than
|
||||
`max_download_bytes` stops the download and exits 1. In each
|
||||
case the partial file is deleted and `ingest/` is unchanged.
|
||||
@@ -2636,6 +2673,82 @@ Candidates for later formats are in Appendix C.
|
||||
`.\ingest\a.md` exits 0, then exits 1 with
|
||||
`Error: Source not found: .\ingest\a.md (moved to
|
||||
raw/articles/a.md by an earlier ingest)` on one line.
|
||||
- *7.1.5.2.8.* `NOT_IMPLEMENTED` A URL that needs a login is an
|
||||
error, not a source. The tool cannot log in, so it stops and
|
||||
leaves the next step to the agent (*7.1.5.14.*). A login page
|
||||
must never reach `raw/` as if it were the source.
|
||||
- *7.1.5.2.8.1.* The tool sends no credentials, cookies, or
|
||||
tokens, and reads none from the environment or from a file
|
||||
(*7.1.5.2.2.*). It never asks for a password.
|
||||
- *7.1.5.2.8.2.* Three checks run in this order. The first hit
|
||||
decides the reason:
|
||||
|
||||
| Check | Hit when | Reason text |
|
||||
| --- | --- | --- |
|
||||
| Status | The response is HTTP 401 or 403 | `<status> <phrase>`, e.g. `401 Unauthorized` |
|
||||
| Redirect | The final URL differs from the requested URL and its host and path match one of `login_url_patterns` | `redirected to <final url>` |
|
||||
| Login form | The media type is `text/html` and the body has an `<input>` whose `type` is `password`, in any case or quoting | `login page at <final url>` |
|
||||
|
||||
- *7.1.5.2.8.3.* `login_url_patterns` is a list of regular
|
||||
expressions, searched case-insensitively in
|
||||
`<host><path>` of the final URL. The query string is not
|
||||
searched. The default list is in section 5.1. The redirect
|
||||
check is skipped when there was no redirect, so a user can
|
||||
still ingest a page whose own path holds `login`.
|
||||
- *7.1.5.2.8.4.* On a hit, stderr has the `Error:
|
||||
Authentication required` line and the `Next:` line from
|
||||
section 4.7.6, and the command exits 4. The body is read
|
||||
within `max_download_bytes` and then dropped. Nothing is
|
||||
written under `ingest/`, `raw/`, or `wiki/`, and `wiki/log.md`
|
||||
is unchanged. The URL in both lines is the one the user
|
||||
passed, with userinfo removed.
|
||||
- *7.1.5.2.8.5.* `--dry-run` runs the same checks and exits 4
|
||||
the same way. `--json` prints nothing on stdout
|
||||
(*7.1.5.13.*).
|
||||
- *7.1.5.2.8.6.* An HTML page that passes the three checks but
|
||||
has no text still converts as `partial` with the warning in
|
||||
*7.1.5.4.5.5.*. Login detection does not replace that warning.
|
||||
- *acceptance*: A fake opener that answers 401 exits 4. Stderr
|
||||
has `Authentication required`, `401`, and the `Next:` line.
|
||||
`ingest/`, `raw/`, `wiki/`, and `wiki/log.md` have the same
|
||||
hashes as before.
|
||||
- *acceptance*: A fake opener that redirects
|
||||
`https://host/job/x/console` to
|
||||
`https://host/login?from=%2Fjob%2Fx%2Fconsole` and answers 200
|
||||
exits 4 with the reason `redirected to https://host/login?...`.
|
||||
- *acceptance*: A fake opener that answers 200 with an HTML form
|
||||
holding `<INPUT TYPE='password'>` exits 4 with the reason
|
||||
`login page at <url>`.
|
||||
- *acceptance*: A 200 HTML page at `https://host/docs/login`,
|
||||
with no redirect and no password input, is ingested with
|
||||
exit 0.
|
||||
- *acceptance*: A 404 still exits 1 with `Download failed`.
|
||||
- *7.1.5.2.9.* `NOT_IMPLEMENTED` `--origin URL` records where a
|
||||
local file came from, so a source saved by hand after
|
||||
*7.1.5.2.8.* keeps its real provenance.
|
||||
- *7.1.5.2.9.1.* `--origin` is valid only when `<source>` is a
|
||||
local path. With a URL source it exits 1 with
|
||||
`Error: --origin needs a local file: <source>`. A value that
|
||||
is not `http://` or `https://` exits 1 with
|
||||
`Error: --origin must be an http or https URL: <value>`. Both
|
||||
checks run before fetch.
|
||||
- *7.1.5.2.9.2.* The file is fetched, archived, and moved as any
|
||||
local file. The page frontmatter then has `ingest.origin` set
|
||||
to the URL without userinfo, `origin_kind: url`, and
|
||||
`final_url: null`. The `html` converter uses that URL to make
|
||||
relative links absolute (*7.1.5.4.5.3.*). The log bullet names
|
||||
that URL (*7.1.5.6.*). JSON `origin` and `origin_kind` match
|
||||
the frontmatter.
|
||||
- *7.1.5.2.9.3.* The slug and default raw name still come from
|
||||
the file stem (*7.1.5.2.6.*), so the name the agent chose in
|
||||
`ingest/` is kept.
|
||||
- *acceptance*: `wiki ingest ingest/console.html --origin
|
||||
https://host/job/x/console` exits 0. The page has
|
||||
`origin: https://host/job/x/console`, `origin_kind: url`,
|
||||
and a relative link `/a` in the file becomes
|
||||
`https://host/a`. No network call is made.
|
||||
- *acceptance*: `wiki ingest https://host/a --origin
|
||||
https://host/b` exits 1 and makes no network call.
|
||||
- *7.1.5.3.* Archive.
|
||||
- *7.1.5.3.1.* The default raw path is
|
||||
`raw/<category>/<slug>.<ext>`. The category is the raw category
|
||||
@@ -2907,10 +3020,36 @@ Candidates for later formats are in Appendix C.
|
||||
- *acceptance*: After ingesting a markdown file with style
|
||||
violations, `wiki sanity` reports none from its raw copy.
|
||||
- *7.1.5.13.* `--json` prints only the JSON object
|
||||
from section 4.7.6 on stdout, on exit 0 and on exit 3. On exit 1,
|
||||
stdout is empty.
|
||||
from section 4.7.6 on stdout, on exit 0 and on exit 3. On exit 1
|
||||
and exit 4, stdout is empty.
|
||||
- *acceptance*: The stdout of an exit 3 run parses with
|
||||
`json.loads` and has `conversion: none`.
|
||||
- *7.1.5.14.* `NOT_IMPLEMENTED` On exit 4 the agent gets the source
|
||||
another way, or asks the user for it. The Fiona skill teaches
|
||||
this in section 16.2. The steps, in order:
|
||||
- *7.1.5.14.1.* Use a tool the agent already has that can read
|
||||
that host with the user's access, such as an MCP server for
|
||||
Confluence, SharePoint, Jenkins, or a ticket system, or an
|
||||
authenticated API script in the wiki's `utils/scripts/`. Save
|
||||
what it returns into `ingest/<slug>.<ext>` in its own format:
|
||||
HTML as `.html`, a PDF as `.pdf`, plain text as `.txt` or `.md`.
|
||||
Then run the `Next:` command with that file and
|
||||
`--origin <url>`.
|
||||
- *7.1.5.14.2.* When no tool has access, stop and ask the user to
|
||||
save the page into `ingest/`, for example with the browser's
|
||||
Save As or Print to PDF. Name the URL, the folder, and the
|
||||
command the agent will run once the file is there. Do not
|
||||
retry the URL.
|
||||
- *7.1.5.14.3.* Never ask the user for a password, token, or
|
||||
cookie. Never put credentials in a URL, a command line, a
|
||||
page, or the log.
|
||||
- *7.1.5.14.4.* A tool that returns a summary instead of the
|
||||
source is not the source. Archive the content the tool
|
||||
returned as it is, and say in `ingest.warnings` that it came
|
||||
through that tool.
|
||||
- *acceptance*: Fiona skill section 4.6 lists exit 4. Section 16.2
|
||||
has the three cases above (a tool with access, ask the user,
|
||||
never ask for credentials) and shows the `--origin` command.
|
||||
|
||||
### 7.2. housekeeping
|
||||
|
||||
@@ -3747,6 +3886,7 @@ is the pip-argument source both scripts call.
|
||||
| P2 | settings setup | 7.16 | Planned |
|
||||
| P2 | Bootstrap install modes | 7.21 | Implemented |
|
||||
| P2 | wiki ingest | 7.1.5 | Partially Implemented |
|
||||
| P2 | Ingest login detection and `--origin` | 7.1.5.2.8, 7.1.5.2.9, 7.1.5.14 | Planned |
|
||||
| P3 | `--version` | 2.3 | Planned |
|
||||
| P3 | fmlint enforcement of quality signals | 6.2 | Planned |
|
||||
| P3 | Backup archive | 7.13 | Planned |
|
||||
@@ -3801,6 +3941,13 @@ is the pip-argument source both scripts call.
|
||||
2026-10-02 manual test lost a `git save` commit this way. A TTY
|
||||
prompt with `--yes` (*1.2.2.*) would make agents pass `--yes` on
|
||||
every call. Not decided.
|
||||
- `wiki ingest` sends no credentials (*7.1.5.2.8.1.*). A cookie
|
||||
file, a bearer token from an environment variable, or a
|
||||
`--header` option would let it read some login-only hosts
|
||||
itself. Each one puts secrets near the wiki. Not decided.
|
||||
- On exit 4, `--json` prints nothing (*7.1.5.13.*). A JSON error
|
||||
object such as `{"error": "auth_required", "url": ...,
|
||||
"reason": ...}` would save the agent parsing stderr. Not decided.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -4,6 +4,7 @@
|
||||
- [P0 — Packaging and Structure Fixes](#p0--packaging-and-structure-fixes)
|
||||
- [Code defects — 2026-09-29 audit](#code-defects--2026-09-29-audit)
|
||||
- [Manual test findings — 2026-10-02](#manual-test-findings--2026-10-02)
|
||||
- [Ingest of a URL that needs a login](#ingest-of-a-url-that-needs-a-login)
|
||||
- [P1 — Wiki Lifecycle and Version Control](#p1--wiki-lifecycle-and-version-control)
|
||||
- [P2 — Quality Tooling](#p2--quality-tooling)
|
||||
- [P3 — Metadata, Backup, and Documentation](#p3--metadata-backup-and-documentation)
|
||||
@@ -424,6 +425,70 @@ SPEC 1.8.0 holds the requirements. Order is the build order.
|
||||
(SPEC Appendix C). The tester lost a `git save` commit with
|
||||
`git undo`. Listed under P4 as well
|
||||
|
||||
## Ingest of a URL that needs a login
|
||||
|
||||
Users paste links to Jenkins, Confluence, SharePoint, and ticket
|
||||
pages that need a login. Today a 401 or 403 exits 1 with the generic
|
||||
`Download failed`, and an SSO redirect that ends on a 200 login page
|
||||
is archived under `raw/` and converted as if it were the source.
|
||||
SPEC 1.9.0 holds the requirements (7.1.5.2.8, 7.1.5.2.9, 7.1.5.14).
|
||||
Spec written 2026-10-05; nothing below is built. Order is the build
|
||||
order.
|
||||
|
||||
- [ ] Detect a login in the URL fetcher (SPEC 7.1.5.2.8). In
|
||||
`src/fiona/wiki/ingest/fetchers.py`, raise a new
|
||||
`AuthRequiredError(IngestError)` that carries the URL and the
|
||||
reason, before anything is written to `ingest/`
|
||||
- [ ] HTTP 401 and 403 from `HTTPError`: reason `<code> <phrase>`.
|
||||
Other non-2xx stay `Download failed` (SPEC 7.1.5.2.3)
|
||||
- [ ] Redirect check: final URL differs from the requested URL
|
||||
and `<host><path>` matches `login_url_patterns`
|
||||
(case-insensitive, query string ignored)
|
||||
- [ ] Login form check: `text/html` body with an `<input>` whose
|
||||
`type` is `password`, any case or quoting. Run it on the
|
||||
bytes already read, so the size limit still applies first
|
||||
- [ ] Tests with a fake opener, no network: 401, 403, redirect to
|
||||
`/login?from=...`, 200 page with `<INPUT TYPE='password'>`,
|
||||
a `/docs/login` page with no redirect that still ingests,
|
||||
and a 404 that still exits 1. Each auth case leaves
|
||||
`ingest/`, `raw/`, `wiki/`, and `wiki/log.md` unchanged
|
||||
- [ ] `login_url_patterns` setting (SPEC 5.1). Add it to
|
||||
`src/fiona/wiki/ingest/settings.py` with the default list from
|
||||
SPEC 5.1. A user list replaces the default. A regex that does
|
||||
not compile is skipped with one warning
|
||||
- [ ] Tests: default list, user override, one bad regex
|
||||
- [ ] Exit 4 and the two stderr lines (SPEC 4.7.6). Catch
|
||||
`AuthRequiredError` in the ingest CLI handler, print the
|
||||
`Error: Authentication required: <url>: <reason>` line and the
|
||||
`Next:` line, exit 4. URL without userinfo. `--json` stdout
|
||||
stays empty (SPEC 7.1.5.13); `--dry-run` exits 4 the same way
|
||||
- [ ] Tests: exit code, both stderr lines, empty stdout with
|
||||
`--json`, dry run, no `Traceback` (SPEC 1.2.7)
|
||||
- [ ] `--origin URL` (SPEC 7.1.5.2.9). Valid only with a local file,
|
||||
`http`/`https` only, both checks before fetch. Sets
|
||||
`ingest.origin`, `origin_kind: url`, `final_url: null`, feeds
|
||||
the `html` converter's base URL and the log bullet. Slug stays
|
||||
the file stem
|
||||
- [ ] Tests: saved HTML with a relative link `/a` becomes
|
||||
`https://host/a`; URL source plus `--origin` exits 1; bad
|
||||
scheme exits 1; no network call in any case
|
||||
- [ ] Fiona skill (SPEC 7.1.5.14). In
|
||||
`src/fiona/data/llm_wiki/utils/skills/fiona/SKILL.md`: add exit
|
||||
4 to the section 4.6 exit table and `--origin` to its options;
|
||||
rewrite the authentication bullet in section 16.2 into the
|
||||
three steps (use a tool with access, else ask the user to save
|
||||
it into `ingest/`, never ask for credentials) with the
|
||||
`--origin` command. Bump the skill version
|
||||
- [ ] README ingest notes (SPEC 7.18.4): one line on exit 4 and
|
||||
`--origin`
|
||||
- [ ] Verify the spec (project agent rule 2.1 step 4): drop the
|
||||
`NOT_IMPLEMENTED` markers on 4.7.6, 7.1.5.2.8, 7.1.5.2.9,
|
||||
7.1.5.14, and the `login_url_patterns` prose in 5.1; set the
|
||||
Appendix B row to `Implemented`; bump SPEC
|
||||
- [ ] Open questions left in SPEC Appendix C, not part of this
|
||||
work: credential support (cookie file, token, `--header`) and
|
||||
a JSON error object on exit 4
|
||||
|
||||
## P1 — Wiki Lifecycle and Version Control
|
||||
|
||||
- [x] `wiki init` hardening (SPEC 7.1.1)
|
||||
|
||||
Reference in New Issue
Block a user