Port the docgen site generator (gen.py/check.py/palette.py) to Ludic #41

New issue

Closed

opened 2026-08-31 00:09:33 +02:00 by orkun · 1 comment

orkun commented

2026-08-31 00:09:33 +02:00

Owner

Follow-up to #31 (the doc/lint/grammar checks were ported to Ludic there; this is the remaining docgen piece).

Scope

Port the documentation site generator and its guards off Python:

tools/docgen/gen.py (462 lines) — the per-symbol → static-site generator (index/api/<id>/ns-*/color pages, ludic-highlight.js, symbols.json).
tools/docgen/check.py (148 lines) — the generated-site coverage/integrity guard.
tools/docgen/palette.py (298 lines) — the named-colour source of truth (emits selfhost/backend/stdlib/emit_color.ludic + palette.json). Also has a latent ptr/pointer drift to fix while here.

Building blocks already in place

tools/x/json.ludic — a JSON reader (objects/arrays/strings with \uXXXX+surrogates/numbers/literals) added in #31; reuse it for palette.json / site.json.
tools/x/checks.ludic — string helpers (s_index/sslice/s_trim/front-matter walking) and the fence/param parsing patterns.
The generator reads its CSS/HTML/JS from tools/docgen/assets/* and fills placeholders, so the templates themselves don't need reimplementing — only the parse → symbol-model → fill/emit logic and a small JSON serializer (for symbols.json + the highlighter embed).

Plan

x docs-gen [--out DIR] in Ludic: parse docs/language/** front-matter + bodies (fences, Parameters:), build the section/symbol model, read the asset templates, emit every page + symbols.json + .nojekyll.
x docs-check DIR: port check.py (required files, a page per inventory.json symbol, duplicate-token / one-dir-per-namespace guards, highlighter targets exist).
Port palette.py (move the palette table into Ludic; emit emit_color.ludic with pointer not ptr).
Verify against the Python oracle: run both gen.py and x docs-gen, diff the output trees; iterate to byte-identity (or, if a cosmetic diff is accepted, at least check.py-clean).
Swap CI: .forgejo/workflows/ci.yml and .forgejo/workflows/docs.yml call the Ludic generator; drop the python:3.12 container from docs.yml (this is what completes issue #31's criterion 3).

Why staged

The generator emits 465 files (~14 KB each) with Python-json.dumps formatting and many HTML-escaping/whitespace edge cases; a reimplementation shipped straight to the live pages branch risks a regression, so it wants its own change with an oracle-diff gate — which is why #31's acceptance explicitly allowed staging docgen ("docgen ported or a tracked plan for the remaining piece").

Follow-up to #31 (the doc/lint/grammar **checks** were ported to Ludic there; this is the remaining **docgen** piece). ## Scope Port the documentation **site generator** and its guards off Python: - `tools/docgen/gen.py` (462 lines) — the per-symbol → static-site generator (index/api/`<id>`/`ns-*`/color pages, `ludic-highlight.js`, `symbols.json`). - `tools/docgen/check.py` (148 lines) — the generated-site coverage/integrity guard. - `tools/docgen/palette.py` (298 lines) — the named-colour source of truth (emits `selfhost/backend/stdlib/emit_color.ludic` + `palette.json`). Also has a latent `ptr`/`pointer` drift to fix while here. ## Building blocks already in place - `tools/x/json.ludic` — a JSON reader (objects/arrays/strings with `\uXXXX`+surrogates/numbers/literals) added in #31; reuse it for `palette.json` / `site.json`. - `tools/x/checks.ludic` — string helpers (`s_index`/`sslice`/`s_trim`/front-matter walking) and the fence/param parsing patterns. - The generator reads its CSS/HTML/JS from `tools/docgen/assets/*` and fills placeholders, so the templates themselves don't need reimplementing — only the parse → symbol-model → fill/emit logic and a small JSON **serializer** (for `symbols.json` + the highlighter embed). ## Plan 1. `x docs-gen [--out DIR]` in Ludic: parse `docs/language/**` front-matter + bodies (fences, `Parameters:`), build the section/symbol model, read the asset templates, emit every page + `symbols.json` + `.nojekyll`. 2. `x docs-check DIR`: port `check.py` (required files, a page per `inventory.json` symbol, duplicate-token / one-dir-per-namespace guards, highlighter targets exist). 3. Port `palette.py` (move the palette table into Ludic; emit `emit_color.ludic` with `pointer` not `ptr`). 4. **Verify against the Python oracle**: run both `gen.py` and `x docs-gen`, diff the output trees; iterate to byte-identity (or, if a cosmetic diff is accepted, at least `check.py`-clean). 5. Swap CI: `.forgejo/workflows/ci.yml` and `.forgejo/workflows/docs.yml` call the Ludic generator; drop the `python:3.12` container from `docs.yml` (this is what completes issue #31's criterion 3). ## Why staged The generator emits 465 files (~14 KB each) with Python-`json.dumps` formatting and many HTML-escaping/whitespace edge cases; a reimplementation shipped straight to the **live** pages branch risks a regression, so it wants its own change with an oracle-diff gate — which is why #31's acceptance explicitly allowed staging docgen ("docgen ported *or a tracked plan for the remaining piece*").

orkun added the

area:tooling

priority:medium

labels 2026-08-31 00:10:10 +02:00

orkun referenced this issue

2026-08-31 00:10:10 +02:00

DX: replace Python/JS tooling with Ludic (docgen, checks, LSP/web drivers) #31

orkun referenced this issue from a commit

2026-08-31 01:13:40 +02:00

feat(tooling): port the docgen site generator to Ludic (no Python) (#41)

orkun commented

2026-08-31 01:15:20 +02:00

Author

Owner

Done in 9ed0070.

The whole docgen pipeline is Ludic now — no Python anywhere. Three new x subcommands, in Ludic and compiled by Ludic:

x docs-gen [--out DIR] — the static-site generator. Parses docs/language/** front-matter + bodies (```ludic fences, Parameters:), builds the section/symbol model, reads the assets/* templates, and emits every per-symbol page, the ns-*/ns-color/api pages, the landing page, ludic-highlight.js, symbols.json and .nojekyll.
x docs-check [DIR] — the coverage/integrity guard: required files, a page per inventory.json symbol, duplicate-token and one-dir-per-namespace guards, and highlighter link targets.
x docs-palette — the named-colour source of truth. The PALETTE table moved into tools/x/docgen.ludic (the pal_add(...) rows), emitting selfhost/backend/stdlib/emit_color.ludic (as pointer, not ptr) + palette.json.

Oracle-diff gate: x docs-gen reproduces the Python generator's output byte-for-byte across all 466 files, and x docs-check matches check.py's pass/fail output (verified on both a clean and a deliberately-broken tree). This needed a faithful Ludic port of json.dumps (ordered dicts, indent=2 vs compact separators, ensure_ascii \uXXXX with surrogate pairs), html.escape (quote on/off), the front-matter/fence/Parameters:/paragraph parsing, and codepoint-aware first-sentence truncation. Two subtleties that bit: cross-kind token collisions in sym["tips"] (e.g. exit/string/fixed) need dict-overwrite semantics, and plain_text's 160-char cap is codepoints, not bytes.

Wired into x test as a regression gate (docs-gen → docs-check on a fresh site; docs-palette stays byte-identical) — the suite is now 59 passed, 0 failed.

CI swap: ci.yml and docs.yml call the Ludic generator; docs.yml drops the python:3.12 container and bootstraps the toolchain from the IR seed with clang instead. tools/docgen/{gen,check,palette}.py deleted — only assets/ + inventory.json remain. Completes #31's criterion 3.

Done in 9ed0070. The whole docgen pipeline is Ludic now — no Python anywhere. Three new `x` subcommands, in Ludic and compiled by Ludic: - **`x docs-gen [--out DIR]`** — the static-site generator. Parses `docs/language/**` front-matter + bodies (```ludic fences, `Parameters:`), builds the section/symbol model, reads the `assets/*` templates, and emits every per-symbol page, the `ns-*`/`ns-color`/`api` pages, the landing page, `ludic-highlight.js`, `symbols.json` and `.nojekyll`. - **`x docs-check [DIR]`** — the coverage/integrity guard: required files, a page per `inventory.json` symbol, duplicate-token and one-dir-per-namespace guards, and highlighter link targets. - **`x docs-palette`** — the named-colour source of truth. The `PALETTE` table moved into `tools/x/docgen.ludic` (the `pal_add(...)` rows), emitting `selfhost/backend/stdlib/emit_color.ludic` (as `pointer`, not `ptr`) + `palette.json`. **Oracle-diff gate:** `x docs-gen` reproduces the Python generator's output **byte-for-byte across all 466 files**, and `x docs-check` matches `check.py`'s pass/fail output (verified on both a clean and a deliberately-broken tree). This needed a faithful Ludic port of `json.dumps` (ordered dicts, `indent=2` vs compact separators, `ensure_ascii` `\uXXXX` with surrogate pairs), `html.escape` (`quote` on/off), the front-matter/fence/`Parameters:`/paragraph parsing, and codepoint-aware first-sentence truncation. Two subtleties that bit: cross-kind token collisions in `sym["tips"]` (e.g. `exit`/`string`/`fixed`) need dict-overwrite semantics, and `plain_text`'s 160-char cap is codepoints, not bytes. Wired into `x test` as a regression gate (`docs-gen` → `docs-check` on a fresh site; `docs-palette` stays byte-identical) — the suite is now 59 passed, 0 failed. **CI swap:** `ci.yml` and `docs.yml` call the Ludic generator; `docs.yml` drops the `python:3.12` container and bootstraps the toolchain from the IR seed with clang instead. `tools/docgen/{gen,check,palette}.py` deleted — only `assets/` + `inventory.json` remain. Completes #31's criterion 3.

orkun closed this issue

2026-08-31 01:15:20 +02:00