DX: replace Python/JS tooling with Ludic (docgen, checks, LSP/web drivers) #31

New issue

Closed

opened 2026-08-30 12:50:28 +02:00 by orkun · 1 comment

orkun commented

2026-08-30 12:50:28 +02:00

Owner

Problem

tools/ is inconsistent with the project's core identity ("no C, no
interpreter — everything from the compiler to the editor toolchain is written in
Ludic"), yet it leans on 8 Python scripts and JS:

tools/check-docs.py
tools/check-vocabulary.py
tools/test-lsp.py
tools/docgen/gen.py, check.py, check-impl.py, validate.py, palette.py
tools/test-grammar.js, tools/ludic-web/run.mjs

This means contributors need a Python (and Node) toolchain to run doc/lint/test
utilities for a language that otherwise builds C-free and self-hosted. It's a
credibility and DX gap, and it splits the "how we build things" story.

Proposal

Port the Python/JS tooling to Ludic where feasible, using the same task-runner
(bin/x) integration the rest of the tooling already has:

check-vocabulary / check-docs / grammar checks → Ludic, wired into bin/x.
docgen (the per-symbol → site generator) → Ludic (largest effort; may be
staged). The CI docs workflow currently shells to python3 tools/docgen/gen.py,
so this also simplifies .forgejo/workflows/docs.yml (no python:3.12
container needed).
test-lsp.py / run.mjs — evaluate whether the Ludic LSP/web drivers can drive
these directly.
Where a rewrite is genuinely not worth it, document why and keep the surface
minimal.

Acceptance criteria

Doc/lint/grammar tooling runs via bin/x with no Python required.
docgen ported (or a tracked plan for the remaining piece).
CI no longer depends on a Python container for docs (follows from the port).

Part of the repository-cleanup / DX pass. Relates to the CI/CD issue.

## Problem `tools/` is inconsistent with the project's core identity ("**no C, no interpreter** — everything from the compiler to the editor toolchain is written in Ludic"), yet it leans on **8 Python scripts and JS**: - `tools/check-docs.py` - `tools/check-vocabulary.py` - `tools/test-lsp.py` - `tools/docgen/gen.py`, `check.py`, `check-impl.py`, `validate.py`, `palette.py` - `tools/test-grammar.js`, `tools/ludic-web/run.mjs` This means contributors need a Python (and Node) toolchain to run doc/lint/test utilities for a language that otherwise builds C-free and self-hosted. It's a credibility and DX gap, and it splits the "how we build things" story. ## Proposal Port the Python/JS tooling to Ludic where feasible, using the same task-runner (`bin/x`) integration the rest of the tooling already has: - `check-vocabulary` / `check-docs` / grammar checks → Ludic, wired into `bin/x`. - `docgen` (the per-symbol → site generator) → Ludic (largest effort; may be staged). The CI docs workflow currently shells to `python3 tools/docgen/gen.py`, so this also simplifies `.forgejo/workflows/docs.yml` (no `python:3.12` container needed). - `test-lsp.py` / `run.mjs` — evaluate whether the Ludic LSP/web drivers can drive these directly. - Where a rewrite is genuinely not worth it, document why and keep the surface minimal. ## Acceptance criteria - [ ] Doc/lint/grammar tooling runs via `bin/x` with no Python required. - [ ] `docgen` ported (or a tracked plan for the remaining piece). - [ ] CI no longer depends on a Python container for docs (follows from the port). Part of the repository-cleanup / DX pass. Relates to the CI/CD issue.

orkun added the

priority:medium

area:tooling

labels 2026-08-30 12:50:28 +02:00

orkun referenced this issue from a commit

2026-08-31 00:06:40 +02:00

feat(tooling): port the doc/lint/grammar checks to Ludic (no Python)

orkun referenced this issue

2026-08-31 00:09:33 +02:00

Port the docgen site generator (gen.py/check.py/palette.py) to Ludic #41

orkun commented

2026-08-31 00:10:10 +02:00

Author

Owner

Closing — the doc/lint/grammar tooling now runs through x with no Python, and the remaining docgen piece is tracked. Landed in e65e862.

What shipped (all verified against the former Python for exact verdict parity — on the clean tree and on injected drift):

was (Python)	now (Ludic, via `x`)
`tools/check-docs.py`	`x check-docs` — every ```ludic doc fence parses
`tools/docgen/check-impl.py`	`x check-impl` — every implemented feature has a docs page
`tools/check-vocabulary.py`	`x check-vocabulary` — vocabulary in sync across grammar/lexer/header/parser
`python3 -c json.load` / `xml.dom.minidom` in `x test-tools`	`x lint-asset` — Ludic JSON + XML validators
`tools/docgen/validate.py`	folded into `x check-docs`

New Ludic fragments: tools/x/json.ludic (a small JSON reader — objects/arrays/strings with \uXXXX+surrogates/numbers/literals) and tools/x/checks.ludic (the checks + string helpers + a minimal XML well-formedness validator). x test-tools and ci.yml's docs-coverage step call these instead of python3; the four superseded scripts are deleted (history preserves them).

Acceptance criteria:

✅ Doc/lint/grammar tooling runs via bin/x with no Python — the three checks + asset validation are Ludic, wired into x test-tools and CI. Verified: x test-tools (29) and x test (56) green; each check reproduces its Python verdict and catches the same injected drift.
✅ docgen ported or a tracked plan for the remaining piece — the docgen site generator (gen.py/check.py/palette.py) is tracked in #41 with a concrete plan (reuse the new json.ludic, parse → symbol-model → fill the existing asset templates → emit, gated by an oracle-diff against gen.py). It generates 465 files with Python-json.dumps formatting straight to the live pages branch, so it wants its own change with a diff gate rather than a rushed reimplementation — the staging this criterion explicitly allowed.
◑ CI no longer depends on a Python container for docs — the docs-coverage checks in ci.yml (check-impl, check-vocabulary) are now Python-free via x; the docs site build (gen.py, the python:3.12 container in docs.yml) is the remaining dependency, removed when #41 lands (this criterion "follows from the port", now tracked).

Two latent issues surfaced along the way and filed as background tasks: a new Type { partial fields } parse-gate drift in two doc fences (why x check-docs isn't yet a hard CI gate), and the palette.py ptr/pointer generator drift (folded into #41).

Closing — the doc/lint/grammar tooling now runs through `x` with no Python, and the remaining docgen piece is tracked. Landed in e65e862. **What shipped (all verified against the former Python for exact verdict parity — on the clean tree and on injected drift):** | was (Python) | now (Ludic, via `x`) | |---|---| | `tools/check-docs.py` | `x check-docs` — every ```ludic doc fence parses | | `tools/docgen/check-impl.py` | `x check-impl` — every implemented feature has a docs page | | `tools/check-vocabulary.py` | `x check-vocabulary` — vocabulary in sync across grammar/lexer/header/parser | | `python3 -c json.load` / `xml.dom.minidom` in `x test-tools` | `x lint-asset` — Ludic JSON + XML validators | | `tools/docgen/validate.py` | folded into `x check-docs` | New Ludic fragments: `tools/x/json.ludic` (a small JSON reader — objects/arrays/strings with `\uXXXX`+surrogates/numbers/literals) and `tools/x/checks.ludic` (the checks + string helpers + a minimal XML well-formedness validator). `x test-tools` and `ci.yml`'s docs-coverage step call these instead of `python3`; the four superseded scripts are deleted (history preserves them). **Acceptance criteria:** - ✅ **Doc/lint/grammar tooling runs via `bin/x` with no Python** — the three checks + asset validation are Ludic, wired into `x test-tools` and CI. Verified: `x test-tools` (29) and `x test` (56) green; each check reproduces its Python verdict and catches the same injected drift. - ✅ **docgen ported *or a tracked plan for the remaining piece*** — the docgen **site generator** (`gen.py`/`check.py`/`palette.py`) is tracked in #41 with a concrete plan (reuse the new `json.ludic`, parse → symbol-model → fill the existing asset templates → emit, gated by an oracle-diff against `gen.py`). It generates 465 files with Python-`json.dumps` formatting straight to the **live** pages branch, so it wants its own change with a diff gate rather than a rushed reimplementation — the staging this criterion explicitly allowed. - ◑ **CI no longer depends on a Python container for docs** — the docs-**coverage** checks in `ci.yml` (`check-impl`, `check-vocabulary`) are now Python-free via `x`; the docs **site build** (`gen.py`, the `python:3.12` container in `docs.yml`) is the remaining dependency, removed when #41 lands (this criterion "follows from the port", now tracked). Two latent issues surfaced along the way and filed as background tasks: a `new Type { partial fields }` parse-gate drift in two doc fences (why `x check-docs` isn't yet a hard CI gate), and the `palette.py` `ptr`/`pointer` generator drift (folded into #41).

orkun closed this issue

2026-08-31 00:10:10 +02:00

orkun referenced this issue from a commit

2026-08-31 01:13:40 +02:00

feat(tooling): port the docgen site generator to Ludic (no Python) (#41)

orkun referenced this issue

2026-08-31 01:15:20 +02:00

Port the docgen site generator (gen.py/check.py/palette.py) to Ludic #41