From 9ff5af588ac70aaac8453cb397d6db15de61463f Mon Sep 17 00:00:00 2001 From: nvaldez Date: Mon, 5 Oct 2026 07:30:51 +0200 Subject: [PATCH] test result changed spec --- SPEC.md | 156 +++++++++++++++++++++++++++++++++++++++++++++++++++----- todo.md | 81 ++++++++++++++++++++++++++++- 2 files changed, 223 insertions(+), 14 deletions(-) diff --git a/SPEC.md b/SPEC.md index 8e8b731..5004747 100644 --- a/SPEC.md +++ b/SPEC.md @@ -1,6 +1,6 @@ --- name: Fiona-Assistant Specification -version: 1.7.0 +version: 1.8.0 title: Project Spec Sheet created: 2026-09-23 updated: 2026-10-02 @@ -128,7 +128,16 @@ gives that file frontmatter `name` and `version` (*7.3.1.1.1.*) so `fmlint` accepts it. Revision 1.6.0 specifies `librarian move` (*7.3.3.*) and leaves it unbuilt. Revision 1.7.0 makes that command rewrite links to the moved paths across the whole wiki -(*7.3.3.6.*). +(*7.3.3.6.*). Revision 1.8.0 adds the findings of the 2026-10-02 +manual test and leaves them unbuilt: one-line CLI errors with no +traceback (*1.2.7.*, *7.1.1.4.*), `--version` (*2.3.*), a quiet +default stderr (*7.17.3.*), an ingest hint for a drop-zone file that +already moved (*7.1.5.2.7.*), a dry run that does not log writes +(*7.1.5.8.1.*), a lint-clean ingest log line (*7.1.5.6.1.*), a +follow-up hint for style violations left on a page (*7.1.5.5.6.*), +single-file `mdoctor` (*7.4.4.*), compact housekeeping sanity details +(*7.2.5.1.*), a quiet graph overwrite (*7.1.4.5.9.1.*), and a usage +section in the README (*7.18.4.*). Each feature section starts with a `Status` line. Requirements use `- *N.M.K.*`. Each `*acceptance*` sits under the requirement it tests. @@ -203,12 +212,18 @@ These apply to every feature. A position read from the layout engine is written back in `rem`. A theme length number uses the same scale: 16 is `1em`. - *acceptance*: The graph template contains no `px` length. +- *1.2.7.* `NOT_IMPLEMENTED` An expected failure, such as a missing + path, a refusal, or bad input, prints one `Error: ` line on + stderr and exits 1. It prints no Python traceback. A traceback is + logged only at `DEBUG`. + - *acceptance*: A second `wiki init` on the same directory writes + no line containing `Traceback` on stderr (*7.1.1.4.*). --- ## 2. Package and Runtime -> Status: Implemented +> Status: Partially Implemented Python `>= 3.11` (also 3.12 and 3.13). The default development install from the repository root is `pip install -e ".[dev]"`. @@ -273,6 +288,12 @@ pymarkdown resets the root logger mid-run. exists. - *acceptance*: A fresh install does not fail at import because of a missing `html_js_lint` package. +- *2.3.* `NOT_IMPLEMENTED` `fiona-assistant --version` prints + `fiona-assistant ` on stdout and exits 0. The text + is the same string `wiki ingest` writes to `ingest.tool` + (*7.1.5.5.3.*). + - *acceptance*: The stdout of `--version` equals the `ingest.tool` + value of a page ingested by the same install. --- @@ -537,6 +558,10 @@ A pymarkdown API failure prints `Error: pymarkdown fix failed: ` on stderr and exits 1. The command does not skip `personal/` or `raw/`. +`NOT_IMPLEMENTED` `chklint` and `fix` also take one markdown file in +place of `[directory]` (*7.4.4.*). Today a file prints +`Error: '' is not a valid directory.` and exits 1. + ### 4.3. fmlint `fiona-assistant fmlint` with no subcommand means `chklint .`. @@ -798,7 +823,7 @@ section 7.14. `fiona-assistant wiki graph [directory] [-o FILE] [--format html|json] [--include-pattern GLOB] [--exclude-pattern GLOB] [--wikilinks|--no-wikilinks] [--json] -[--external-links|--no-external-links] +[--external-links|--no-external-links] [-f|--force] [--serve [--host HOST] [--port PORT] [--no-open]]`. Defaults: output `wiki_graph.html` in the current directory, format @@ -809,7 +834,9 @@ A path in `fiona.wiki.graph.ignore_files` is not a node also matches `table_of_contents.md` at the scan root. `NOT_IMPLEMENTED` `--serve` runs a local live view instead of writing once (*7.1.4.19.*). `--host`, `--port`, and `--no-open` -are accepted only with `--serve`. +are accepted only with `--serve`. `NOT_IMPLEMENTED` `-f/--force` +overwrites an existing output file without the prompt of +*7.1.4.5.9.* (*7.1.4.5.9.1.*). #### 4.7.6. wiki ingest @@ -895,6 +922,7 @@ Errors, each on stderr with exit 1: ```text Error: Source not found: +Error: Source not found: (moved to by an earlier ingest) Error: Source is a directory: Error: Unsupported URL scheme: Error: Download failed: : @@ -908,6 +936,7 @@ Error: Page exists: . Use --force to replace it. A failed conversion prints `Error: Conversion failed: : ` on stderr and exits 3, not 1, because the raw copy stays. +`NOT_IMPLEMENTED` The `(moved to ...)` form is *7.1.5.2.7.*. ### 4.8. housekeeping @@ -1590,7 +1619,7 @@ Requirements for the `wiki` subcommands. #### 7.1.1. wiki init -> Status: Implemented +> Status: Partially Implemented Implements sections 3 and 4.7.1. @@ -1610,6 +1639,14 @@ Implements sections 3 and 4.7.1. - *acceptance*: After init, `.gitignore` contains `__pycache__/` and `personal/`. A line `keep-me` written before init is still in the file. +- *7.1.1.4.* `NOT_IMPLEMENTED` The CLI turns the refusal of *3.2.2.* + into one stderr line and exit 1 (*1.2.7.*): + `Error: Directory '' already contains llm-wiki files: + . Run fiona-assistant wiki update to refresh the template.` + The line is one line, even though it is wrapped here. `` is + the comma-separated list from the `FileExistsError` message. + - *acceptance*: A second `wiki init` exits 1, stderr is exactly that + line, and no file changes. #### 7.1.2. wiki update @@ -1967,6 +2004,15 @@ used. or hides orphans. It starts checked. The label does not change. - *7.1.4.5.9.* Overwrite of an existing output file asks for confirmation on a TTY. + - *7.1.4.5.9.1.* `NOT_IMPLEMENTED` `-f/--force` skips that + prompt. Without a TTY and without `--force`, the file is still + overwritten and one `WARNING` record names it. The housekeeping + graph step (*7.2.6.*) passes `--force`, because + `wiki_graph.html` is the one file it owns (*7.2.7.*), and logs + that overwrite at `INFO`. + - *acceptance*: Housekeeping on a wiki that already has + `wiki_graph.html`, at `LOG_LEVEL=WARNING`, writes no + `Overwriting` line on stderr. - *7.1.4.5.10.* Files above `max_files` are skipped after a warning. The first `max_files` files are drawn. - *7.1.4.5.11.* The run completes in under 10 seconds for 1000 files @@ -2484,7 +2530,7 @@ used. #### 7.1.5. wiki ingest -> Status: Implemented +> Status: Partially Implemented Implements section 4.7.6. An ingest has two phases. The tool runs the first: it turns one source into a file on disk, archives a copy @@ -2579,6 +2625,17 @@ Candidates for later formats are in Appendix C. - *acceptance*: `https://host/job/performance_TL153/2108/console` has the slug `performance-tl153-2108-console`. + - *7.1.5.2.7.* `NOT_IMPLEMENTED` A local path is resolved against + the current directory. `./`, `.\`, and either separator are + accepted on Windows. A missing path inside the wiki's `ingest/` + is often a file an earlier run already moved (*7.1.5.3.6.*). + When a file with the same name exists under `raw/`, the error + adds one space and `(moved to by an earlier ingest)`. + Several matches use the first in sorted path order. + - *acceptance*: Ingesting `./ingest/a.md` and then + `.\ingest\a.md` exits 0, then exits 1 with + `Error: Source not found: .\ingest\a.md (moved to + raw/articles/a.md by an earlier ingest)` on one line. - *7.1.5.3.* Archive. - *7.1.5.3.1.* The default raw path is `raw//.`. The category is the raw category @@ -2771,6 +2828,14 @@ Candidates for later formats are in Appendix C. ` markdown style violations left`. - *acceptance*: Every fixture page passes fmlint with no violation. + - *7.1.5.5.6.* `NOT_IMPLEMENTED` When violations are left, human + stdout prints, before the `Next: refine` line, + `Next: fix the style violations listed by fiona-assistant mdoctor chklint .` + The page must pass `wiki sanity` before it is refined. The JSON + `warnings` text does not change. + - *acceptance*: An HTML fixture with a bare URL that the fixer + leaves prints that line with the page path. A clean fixture + does not. - *7.1.5.6.* Record. A run that archives appends one bullet to `wiki/log.md` and bumps that file's patch `version` and `updated`. The bullet is @@ -2779,6 +2844,22 @@ Candidates for later formats are in Appendix C. does not edit `wiki/index.md` or any other page. - *acceptance*: One successful run adds one line to `log.md` and leaves `index.md` unchanged. + - *7.1.5.6.1.* `NOT_IMPLEMENTED` The bullet passes the wiki's own + markdownlint. The origin is a code span, so a URL is not a bare + URL (MD034). Each `->` part starts a continuation line indented + two spaces, so no line is longer than the bullet's longest path: + + ```text + - - ingest `` + -> + -> (raw) + ``` + + The bullet still counts as one entry for *3.3.4.* and the first + acceptance above. + - *acceptance*: After ingesting a URL of 90 characters into a + fresh wiki, `wiki sanity` reports no markdownlint hit in + `wiki/log.md`. - *7.1.5.7.* `--raw-only` runs fetch, archive, and record, and skips convert and write. It exits 0. - *acceptance*: No page is written and `conversion` is `none`. @@ -2790,6 +2871,13 @@ Candidates for later formats are in Appendix C. true in JSON. - *acceptance*: File hashes under the wiki match before and after a dry run, and an `ingest/` source is still in place. + - *7.1.5.8.1.* `NOT_IMPLEMENTED` A dry run logs no record that says + a file was archived, moved, or written. Those records read + `dry run: would archive ` and + `dry run: would write `. + - *acceptance*: A dry run at `LOG_LEVEL=INFO` has `would archive` + on stderr and no line that starts with `archived` after the + logger name. - *7.1.5.9.* `--pending` lists every page under `wiki/` whose frontmatter `status` is `raw`, sorted by path, and exits 0. @@ -2826,7 +2914,7 @@ Candidates for later formats are in Appendix C. ### 7.2. housekeeping -> Status: Implemented +> Status: Partially Implemented Implements section 4.8. Housekeeping is the routine an LLM runs to maintain a wiki. It gathers what needs work; the LLM does the work. @@ -2883,6 +2971,13 @@ Paths in details and actions are relative to the wiki root. `Fix the violations, then run fiona-assistant housekeeping again.` - *acceptance*: One dead link makes the step `FAIL` and a detail line names the file. + - *7.2.5.1.* `NOT_IMPLEMENTED` The details are one line per + violation, `: :: `, + with the path relative to the wiki root. Report headers, + DEFINITIONS, and SUMMARY lines are not details. The full report + stays with `wiki sanity`. + - *acceptance*: A wiki with three MD013 hits and one dead link + gives exactly four `sanity` detail lines. - *7.2.6.* `graph` writes `/wiki_graph.html` with the defaults of section 4.7.5 and the wiki's graph settings. The summary is `wrote wiki_graph.html ( nodes, edges)`. A @@ -3150,7 +3245,7 @@ in any case. ### 7.4. mdoctor -> Status: Implemented +> Status: Partially Implemented Implements section 4.2 and the pymarkdown keys in section 5.1. Settings load at construction. The default settings path is the @@ -3175,6 +3270,12 @@ pymarkdown defaults. - *acceptance*: `scan_path` is called with `respect_gitignore` false. Inside the working directory the exclude list contains `.git/` and each plain directory name from `.gitignore`. +- *7.4.4.* `NOT_IMPLEMENTED` `chklint` and `fix` accept one markdown + file as well as a directory, the same as `fmlint chklint`. A path + that does not exist prints `Error: '' does not exist.` on + stderr and exits 1. + - *acceptance*: `mdoctor fix wiki/sources/a.md` lists only that + file, and every other file hash is unchanged. ### 7.5. fmlint @@ -3463,7 +3564,7 @@ on an existing wiki. It does not create the wiki tree. ### 7.17. Logging -> Status: Implemented +> Status: Partially Implemented The contract is section 2. Reports stay on stdout. Diagnostics go through the logging module to stderr and, when configured, to the @@ -3474,6 +3575,16 @@ file sink. the human report stays on stdout. - *7.17.2.* Dropping `file_log_location` leaves no file handler. - *acceptance*: No log file is created for that run. +- *7.17.3.* `NOT_IMPLEMENTED` The shipped `fiona-settings.json` sets + `fiona.log_level` to `WARNING`, the same as the default in + section 5.1. `fiona.file_log_level` stays `INFO`, so `fiona.log` + keeps the detail. PowerShell shows every native stderr line as a + `NativeCommandError`, so an `INFO` record looks like a failure + even when the exit code is 0. When this is built, the shipped + document in section 5.1 changes to match. + - *acceptance*: In a fresh wiki, `wiki sanity` on the clean + template and `wiki ingest` of a local markdown file write nothing + on stderr, and `fiona.log` has their `INFO` lines. ### 7.18. Documentation @@ -3495,6 +3606,17 @@ are behind the current CLIs. - *7.18.3.* Documentation markdown passes `mdoctor chklint`. - *acceptance*: `README.md` contains a markdown link to `SPEC.md` and the lint report is clean. +- *7.18.4.* `NOT_IMPLEMENTED` The README Quick Start has a short + "After ingest" and "On Windows" part. It says that a file in + `ingest/` moves into `raw/` and a second run on it fails, that + `LOG_LEVEL=ERROR` hides diagnostics until *7.17.3.* ships, the loop + `housekeeping`, then fix style violations, then refine and set + `status: refined`, that `--version` (*2.3.*) or `pip show + fiona-assistant` gives the installed version, and that commands + run from the wiki root until section 7.20 is built. The Fiona skill + carries the same loop. + - *acceptance*: Each of those five points appears in `README.md`, + and the after-ingest loop appears in the Fiona skill. ### 7.19. Tool Roles @@ -3607,11 +3729,14 @@ is the pip-argument source both scripts call. | P1 | wiki update / check-update | 7.1.2 | Implemented | | P1 | Git save, snapshot, restore, clean, undo, status | 7.12 | Implemented | | P1 | wiki sanity, including dead links | 7.1.3 | Implemented | -| P1 | housekeeping | 7.2 | Implemented | +| P1 | housekeeping | 7.2 | Partially Implemented | +| P1 | wiki init | 7.1.1 | Partially Implemented | +| P1 | CLI errors without traceback | 1.2.7 | Planned | | P1 | wiki graph | 7.1.4 | Partially Implemented | | P2 | fmlint | 7.5 | Implemented | | P2 | json-lint | 7.11 | Implemented | -| P2 | mdoctor fix | 7.4 | Implemented | +| P2 | mdoctor fix | 7.4 | Partially Implemented | +| P2 | Logging | 7.17 | Partially Implemented | | P2 | Settings schema | 5 | Implemented | | P2 | Sanity reporting, coverage, progress | 7.14 | Implemented | | P2 | Shellcheck | 7.9 | Planned | @@ -3621,7 +3746,8 @@ is the pip-argument source both scripts call. | P2 | Wiki root discovery | 7.20 | Planned | | P2 | settings setup | 7.16 | Planned | | P2 | Bootstrap install modes | 7.21 | Implemented | -| P2 | wiki ingest | 7.1.5 | Implemented | +| P2 | wiki ingest | 7.1.5 | Partially Implemented | +| P3 | `--version` | 2.3 | Planned | | P3 | fmlint enforcement of quality signals | 6.2 | Planned | | P3 | Backup archive | 7.13 | Planned | | P3 | Documentation refresh | 7.18 | Partially Implemented | @@ -3671,6 +3797,10 @@ is the pip-argument source both scripts call. - `librarian move` (*7.3.3.4.4.*) does not rewrite links in HTML tags (``, ``). Whether it should is not decided. +- `git undo` and `git clean` discard work with no prompt. The + 2026-10-02 manual test lost a `git save` commit this way. A TTY + prompt with `--yes` (*1.2.2.*) would make agents pass `--yes` on + every call. Not decided. --- diff --git a/todo.md b/todo.md index f99a0f8..80abf5a 100644 --- a/todo.md +++ b/todo.md @@ -3,6 +3,7 @@ - [Done](#done) - [P0 — Packaging and Structure Fixes](#p0--packaging-and-structure-fixes) - [Code defects — 2026-09-29 audit](#code-defects--2026-09-29-audit) +- [Manual test findings — 2026-10-02](#manual-test-findings--2026-10-02) - [P1 — Wiki Lifecycle and Version Control](#p1--wiki-lifecycle-and-version-control) - [P2 — Quality Tooling](#p2--quality-tooling) - [P3 — Metadata, Backup, and Documentation](#p3--metadata-backup-and-documentation) @@ -348,6 +349,81 @@ items below are done. The command defects after them are still open. `main()`, which parses the arguments a second time. Call the library functions directly so each command is parsed once. +## Manual test findings — 2026-10-02 + +Black-box test from the README and SPEC only. Report: +`tests/wikitest/wiki/queries/fiona-assistant-manual-test-2026-10-02.md`. +Each item below was re-run on 2026-10-02 before it was written down. +SPEC 1.8.0 holds the requirements. Order is the build order. + +- [ ] Second `wiki init` prints a Python traceback (SPEC 1.2.7, + 7.1.1.4). Reproduced: the `FileExistsError` reaches the top + level. Catch it in the `wiki init` CLI handler, print the one + `Error:` line on stderr, exit 1. Then sweep the other handlers + in `src/fiona/__main__.py` for expected errors that escape + - [ ] Test: second init, stderr has no `Traceback`, exit 1, file + hashes unchanged +- [ ] Shipped `fiona.log_level` is `INFO`, the code default is + `WARNING` (SPEC 7.17.3). Every INFO record shows as a red + `NativeCommandError` in PowerShell even on exit 0. Ship + `WARNING`, keep `file_log_level` `INFO`, and update the shipped + document in SPEC 5.1 + - [ ] Test: fresh wiki, clean `wiki sanity` and a local + `wiki ingest` write nothing on stderr; `fiona.log` has INFO + - [ ] Existing wikis get this through `wiki update` as a + `fiona-settings-update.json` conflict. Check the housekeeping + merge action is clear enough for that one key +- [ ] Ingest log bullet fails the wiki's own lint (SPEC 7.1.5.6.1). + Reproduced in the test wiki: `wiki/log.md` has MD013 (112 and + 156 chars) and MD034 (bare origin URL). Put the origin in a + code span and each `-> ` on a two-space continuation + line in `log_writer.py` + - [ ] Test: 90-char URL ingest, then `wiki sanity` has no + markdownlint hit in `wiki/log.md`; still one bullet per run +- [ ] `mdoctor chklint` and `mdoctor fix` reject a file path + (SPEC 4.2, 7.4.4). Reproduced: both print + `is not a valid directory`. Accept a single `.md` file the way + `fmlint chklint` does + - [ ] Test: `fix` on one file lists only that file; other hashes + unchanged; missing path gives `does not exist` +- [ ] Post-ingest hint for leftover style violations + (SPEC 7.1.5.5.6). Depends on the `mdoctor` file item above. + Print `Next: fix the style violations listed by fiona-assistant mdoctor chklint .` + before `Next: refine` +- [ ] "Relative path fails on Windows" is a misdiagnosis + (SPEC 7.1.5.2.7). `./ingest/x.md` and `.\ingest\x.md` both + work from the wiki root. The tester's file was already moved + to `raw/articles/tester-note.md` by the earlier absolute-path + run (SPEC 7.1.5.3.6), so the second run failed. Make the error + say so: add `(moved to by an earlier ingest)` when + the missing path is under `ingest/` and `raw/` has that name + - [ ] Test: both separators ingest; second run gets the hint +- [ ] `--dry-run` logs `archived ` although nothing is + written (SPEC 7.1.5.8.1). Reproduced. Log + `dry run: would archive` / `dry run: would write` instead +- [ ] Housekeeping sanity details repeat the whole PyMarkdown report + (SPEC 7.2.5.1). One line per violation: + `: :: ` +- [ ] Housekeeping logs `WARNING Overwriting existing file: + wiki_graph.html` on every run (SPEC 7.1.4.5.9.1). Add + `wiki graph -f/--force`; housekeeping passes it and logs at + INFO +- [ ] `fiona-assistant --version` (SPEC 2.3). Reproduced: argparse + rejects it. Print the same string as `ingest.tool` +- [ ] README "After ingest" and "On Windows" notes (SPEC 7.18.4), and + the same after-ingest loop in the Fiona skill. Bump both + versions +- [ ] Wiki root discovery from `wiki/` (SPEC 7.20) is already tracked + under P2. The tester calls it daily friction for agents. It is + blocked on the Appendix C rule; decide that rule, then consider + moving SPEC 7.20 to P1 +- [ ] `librarian move` is already tracked under P1. The tester hit + `invalid choice`; no new action, since SPEC 4.1.2 marks it + `NOT_IMPLEMENTED` and the README does not list it +- [ ] Decide whether `git undo` and `git clean` prompt on a TTY + (SPEC Appendix C). The tester lost a `git save` commit with + `git undo`. Listed under P4 as well + ## P1 — Wiki Lifecycle and Version Control - [x] `wiki init` hardening (SPEC 7.1.1) @@ -464,7 +540,7 @@ items below are done. The command defects after them are still open. markdown file as bytes, detect a UTF-8 BOM, decode, call `rewrite_text`. Disk is read only - [ ] Inbound scan inside `plan` (SPEC 7.3.3.6): walk every - `*.md` under `wiki_root`, skip hidden folders (`.git` + `*.md` under `wiki_root`, skip hidden folders (`.git`\[\[[^|]*\|(.*)\]\] included) and the moved files. Call `rewrite_text(..., inbound=True)` on each. A file whose first part relative to the root is in `READONLY_DIRS` goes to @@ -1117,6 +1193,9 @@ WARNING run_tests: pytest short summary missing from output; failure details com ## P4 — Design Decisions Needed +- [ ] Should `git undo` and `git clean` prompt on a TTY, with `--yes` + to skip it? Agents would then pass `--yes` every time + (SPEC Appendix C, 2026-10-02 manual test) - [ ] Librarian: version tracking for non-MD files (SPEC 7.3.2) - [ ] Python files — format TBD - [ ] Bash scripts — format TBD