Baseline: Ludic compiler + toolchain, Phase 1 syntax fixes complete

Self-hosted compiler (selfhost/*.ludic), runtime, examples, editor tooling, and docs. Phase 1 of the syntax-redesign cohesion pass has landed: edge-system fix, signature-query, when-alias, and the documentation truth-pass. Suite green (14/14), C-free bootstrap fixpoint holds. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-27 15:15:35 +03:00 · 2026-08-27 15:15:35 +03:00 · 985f9ad8f2
commit 985f9ad8f2
418 changed files with 39065 additions and 0 deletions
--- a/SYNTAX-REDESIGN.md
+++ b/SYNTAX-REDESIGN.md
@ -0,0 +1,254 @@
+# Ludic Syntax Redesign — Cohesion Pass
+
+A plan to make Ludic's syntax internally consistent. It fixes the drift between
+the spec and the compiler, then unifies the grammar around two rules. Scope:
+**full redesign (Phases 0–5)**. Named-field direction: **colon everywhere**.
+
+> Status: **Phase 1 landed** (see below). Phases 2→5 are proposals. The phases
+> are ordered so the documentation never describes syntax the compiler rejects,
+> and every phase ends with the compiler still self-hosting to a fixpoint
+> (`./test.sh`).
+>
+> Coordinated with the toolchain agent (CLI front-end / `ludicc`+`ludic`
+> binaries) via serialized reseeds of `selfhost/ludicc.seed.ll`; Phase 1 rode in
+> alongside their `emit_*`/`main.ludic` work, combined suite **14/14 green**.
+
+---
+
+## Why (the findings)
+
+Verified against the self-hosted compiler ([selfhost/parse.ludic](selfhost/parse.ludic),
+[selfhost/parse_game.ludic](selfhost/parse_game.ludic), [selfhost/lex.ludic](selfhost/lex.ludic)):
+
+**Structural incoherence**
+1. **Five micro-syntaxes for named parts** — `name: type = d` (fields), `name: type`
+   (params), `Field = { k = v }` (spawn), `[Name, {Tag}]` (query), whitespace
+   `phase X reads [..]` (system clauses), `key=value` (ui props).
+2. **`=` means seven things, `:` means one** — assignment, default, record init,
+   ui prop, extern symbol, const value, `state X = N` all use `=`.
+3. **No statement terminators** — `\n` and `;` both lex to `TK_NL`
+   ([lex.ludic:45,121](selfhost/lex.ludic)) but the parser never requires a
+   separator, so `t.kind = k  t.text = x  t.ival = v` (three statements, spaces
+   only) is idiomatic.
+
+**Broken / dead syntax (compiler-verified)**
+4. `edge system` — **hard parse error** (documented at [LANGUAGE.md:188](LANGUAGE.md)). *(✅ fixed in Phase 1)*
+5. `pure fn` — parses, `pure` silently discarded ([parse.ludic:272](selfhost/parse.ludic)); undocumented.
+6. `@anno` + `reads/writes/needs/uses [..]` — parsed then thrown away
+   ([parse_game.ludic:15-31](selfhost/parse_game.ludic)); four synonyms, two undocumented.
+7. `scene`/`layer`/`on enter` — full LANGUAGE.md section + [examples/scenes.ludic](examples/scenes.ludic),
+   **does not compile** (`expected declaration`).
+8. `query (v) [..]` in a system signature — two LANGUAGE.md sections +
+   [examples/qdecl.ludic](examples/qdecl.ludic), **does not compile** (`parse error: {`). *(✅ implemented in Phase 1)*
+9. `when cond {}` — documented ([LANGUAGE.md:329](LANGUAGE.md)) + in all three editor
+   highlighters, **never parsed**. *(✅ implemented in Phase 1 as an if-without-else alias)*
+10. CLI `--emit-llvm`/`-o`/`--shared`/`--fmt` — documented, but `ludicc` only
+    accepts `--windowed`/`--headless` ([main.ludic:8-14](selfhost/main.ludic)).
+
+**Philosophical splits**
+11. Operators are words (`and`/`or`/`not`), symbols (`==`/`<=`), *and* functions
+    (`band`/`shl`) at once.
+12. Three overlapping control families — `if`/`when`, `match`, `machine`/`become`
+    — and `enter` reuses `become`'s AST node ([parse.ludic:176-177](selfhost/parse.ludic)).
+13. Typed components/structs exist, but real state lives in 64 untyped int
+    registers (`reg`/`setreg`), so `machine`/`match` dispatch on magic numbers.
+
+---
+
+## The two rules everything converges on
+
+**Rule A — `:` associates, `=` binds.**
+- `:` introduces a *named part* and its type or value in a declarative structure:
+  component/struct fields' types, record initializers, ui props, (future) named
+  call arguments.
+- `=` binds a value to a storage location or a constant: `let`, assignment
+  (`+=` …), `const` value, a field's **default**, and the extern symbol.
+- A field declaration uses both, unambiguously: `x: int = 0` reads "`x` *has type*
+  `int` (`:`), *defaulting to* `0` (`=`)" — same shape as Rust/TypeScript.
+- A record/spawn initializer is declarative, so it uses `:` — `Pos { x: 10 }`.
+
+**Rule B — a statement ends at a newline (or `;` or `}`).**
+- Newlines become significant. Two statements on one line require an explicit
+  `;`. `ludic-fmt` normalizes one statement per line and inserts/removes `;`.
+
+Everything below is these two rules applied construct by construct.
+
+---
+
+## Target grammar (before → after)
+
+### Records / spawn initializers
+```ludic
+# doc-check: skip — illustrative redesign snippet (proposed / partial syntax)
+# before
+spawn Hero { Pos = { x = 10, y = 5 }  Player = { } }
+# after
+spawn Hero {
+  Pos { x: 10, y: 5 }
+  Player {}
+}
+```
+`Field = { k = v }` → `Field { k: v }`. The component name is followed directly
+by a record; fields use `:`. (Record literals elsewhere read the same:
+`{ x: 10, y: 5 }`.)
+
+### UI props → named-argument form
+```ludic
+# doc-check: skip — illustrative redesign snippet (proposed / partial syntax)
+# before
+panel id=Root w=288 pad=16 gap=6 align=center { label text="HI" size=26 }
+# after
+panel(id: Root, w: 288, pad: 16, gap: 6, align: center) {
+  label(text: "HI", size: 26)
+}
+```
+A widget becomes "a constructor with named args, then an optional child block."
+This deletes the bespoke `key=value` dialect and reuses `:` + commas. (Lower-churn
+alternative if the paren form is disliked: keep whitespace separation but colonize
+— `panel id: Root w: 288` — still removes the `=` overload.)
+
+### System clauses & modifiers → one annotation channel
+```ludic
+# doc-check: skip — illustrative redesign snippet (proposed / partial syntax)
+# before
+edge system Move @deterministic reads [Vel] writes [Pos] phase FixedUpdate
+  query (p, v) [Pos, Vel] where a.x > 0 { … }
+# after
+@edge @deterministic
+system Move
+  phase FixedUpdate
+  reads [Vel] writes [Pos]
+  query (p, v) [Pos, Vel] where p.x > 0
+{ … }
+```
+- Prefix modifier words (`edge`, `pure`, `export`) are **retired**; all modifiers
+  become `@annotations`, parsed into a real list on the node (not skipped). This
+  fixes the `edge system` parse bug (#4) by construction.
+- `needs`/`uses` are dropped; `reads`/`writes` stay as the two structural clauses
+  and are **stored** (even if analysis is future work) rather than discarded.
+- The `query (v) [..]` signature clause is **actually implemented** in
+  `parse_system` (#8), lowering to the same `S_QUERY` node as the inline `for`.
+
+### extern
+```ludic
+# doc-check: skip — illustrative redesign snippet (proposed / partial syntax)
+extern fn c_hypot(a: fixed, b: fixed) -> fixed = "hypot_fx"   # unchanged
+```
+The `= "symbol"` is a binding under Rule A — it stays.
+
+### Statements
+```ludic
+# doc-check: skip — illustrative redesign snippet (proposed / partial syntax)
+# before (legal today)
+t.kind = kind  t.text = text  t.ival = ival
+# after
+t.kind = kind
+t.text = text
+t.ival = ival
+# or, explicitly, on one line:
+t.kind = kind; t.text = text; t.ival = ival
+```
+
+### Control flow (Phase 4)
+- **`when` vs `if`** — `when` is now a working `if`-without-else alias (Phase 1).
+  Phase 4 decides whether to keep both spellings or collapse to one; if collapsed,
+  remove `when` from docs, the parser, and all editor highlighters together.
+- **Typed states replace magic-int machines.** Introduce `enum`, and let
+  `machine` dispatch on a typed variable instead of a register:
+```ludic
+# doc-check: skip — illustrative redesign snippet (proposed / partial syntax)
+# before                                   # after
+const R_PHASE: int = 0                     enum Phase { KnightMenu, KnightResolve, MageMenu, EnemyTurn }
+machine R_PHASE {                          var phase: Phase = Phase.KnightMenu
+  state KnightMenu = 0 { … become … }      machine phase {
+  state KnightResolve = 1 { … }              state KnightMenu { … become KnightResolve }
+}                                            state KnightResolve { … }
+                                           }
+```
+  `state X = N` loses the magic `= N` (ordinal comes from the enum). `become`
+  and `enter` (scenes) keep one shared lowering but read from a typed slot.
+
+---
+
+## Phase sequence
+
+Each phase is independently shippable and ends green on `./test.sh` +
+`./selfhost/test.sh` (fixpoint).
+
+### Phase 0 — Doctrine (done here)
+Rules A and B above; colon-everywhere; `@`-annotations as the single modifier
+channel; typed enums for state. No code.
+
+### Phase 1 — Truth-in-documentation ✅ DONE
+Made spec ⇄ compiler agree **before** any grammar change. What landed:
+- ✅ **`edge system` crash fixed** (#4) — `parse_system` now consumes an optional
+  `edge` marker before `system` ([parse_game.ludic](selfhost/parse_game.ludic)).
+  (`edge` is a pure marker; the emitter never lowered it differently.)
+- ✅ **Signature-`query` implemented** (#8) — `query (vars) [terms] where c` in a
+  system header desugars to the same `S_QUERY` node the inline `for` builds, so
+  `examples/qdecl.ludic` compiles and runs. Also fixed multi-line clause parsing
+  (clauses may now span lines).
+- ✅ **`when c { }` implemented** (#9) — as an `if`-without-else alias in
+  [parse.ludic](selfhost/parse.ludic). Docs + editors already listed it; now the
+  compiler agrees, so no editor-vocab churn was needed.
+- ✅ **`scene`/`layer` marked not-yet-implemented** (#7) — prominent note in
+  LANGUAGE.md §"Scenes & layers" + a header on [examples/scenes.ludic](examples/scenes.ludic).
+  Full scene front-end + emission deferred (real work, out of Phase 1 scope).
+- ✅ **`reads`/`writes` honesty** (#6) + the stale "Not yet implemented" section
+  updated in [LANGUAGE.md](LANGUAGE.md); scenes/reads-writes/dropped-CLI-flags now
+  listed there.
+- ✅ **`test.sh` guards drift** — added a `qsmoke qdecl` compile check. Suite
+  green (14/14 incl. the toolchain agent's CLI smoke tests).
+- ✅ **CLI flags** (#10) — `--shared`/`--fmt`/wasm noted as dropped-with-the-C-driver
+  in LANGUAGE.md; `-o`/`--emit-llvm` were being re-added by the toolchain agent
+  (now real, verified in `test.sh`); COMPILING.md updated by that agent.
+- Deferred (intentionally): `pure`-is-ignored (#5) is undocumented and harmless;
+  it will be folded into `@pure` in Phase 3 rather than churned now.
+- **Not done / by design:** `scenes.ludic` is *not* added to `test.sh` (it can't
+  compile yet — a positive test would fail; the header note + LANGUAGE.md warning
+  cover the drift instead).
+
+### Phase 2 — Statement separation (Rule B)
+- Enforce a separator in `block()`/`stmt()` ([parse.ludic:130-199](selfhost/parse.ludic)):
+  after a statement, require `TK_NL` or `}`.
+- Teach `ludic-fmt` to normalize one-statement-per-line and insert `;` where two
+  share a line. Reformat the whole corpus with it.
+- Risk: the self-host sources themselves use space-juxtaposed statements heavily
+  — reformat them in the same commit, re-seed, re-verify fixpoint.
+
+### Phase 3 — Named-field unification (Rule A)
+- Migrate spawn/record init (`= {…=…}` → `{…:…}`) and ui props (`key=value` →
+  named-arg form) in the parser and emitters.
+- Collapse `reads/writes/needs/uses` → `reads`/`writes`, stored on the node.
+- Move `edge`/`pure`/`export` into `@`-annotations; parse annotations into a list.
+- Ship `ludic-fmt --migrate`: a mechanical codemod that rewrites old syntax to
+  new, run over `examples/`, `runtime/`, and `selfhost/`.
+- Gate: compiler still self-hosts; every golden render is byte-identical.
+
+### Phase 4 — Control-flow & operator consolidation (largest)
+- Remove `when`; confirm `if` covers all uses in the corpus.
+- Introduce `enum` + typed `machine`/state; migrate `combat.ludic`'s R_PHASE
+  machine and the register-driven `match reg(…)` sites.
+- Decide the bitwise story: keep `band/shl/…` as functions (document as a
+  deliberate "one spelling" choice) or promote to operators — pick one and state
+  it, don't leave it implicit.
+
+### Phase 5 — Single source of truth for keywords/grammar
+- Generate every editor plugin keyword list, `ludic_syntax.h`, and the LSP's
+  token set from **one** canonical list so a keyword can never again be
+  highlighted but unparsed.
+- Add a CI check (extend `tools/check-docs.py` / `check-vocabulary.py`) that
+  every ` ```ludic ` fence in the docs compiles, closing the doc-drift loop
+  permanently.
+
+---
+
+## Decision log
+- **Scope:** full redesign, Phases 0–5. *(chosen)*
+- **Named fields:** colon everywhere; `=` is binding-only. *(chosen)*
+- **Open — Phase 4 detail:** typed `enum` state vs. keep integer registers.
+  Recommended: typed enums (fixes #13), but it's the deepest change; can be
+  deferred without blocking Phases 1–3.
+- **Open — ui props:** paren named-args (`panel(id: Root)`) vs. colonized
+  whitespace (`panel id: Root`). Recommended: paren form for full cohesion.
+- **Open — bitwise ops:** functions (status quo, documented) vs. operators.