Proposal: built-in testing framework + coverage (Go-style) — one command, low ceremony for game devs #12

New issue

Closed

opened 2026-08-29 20:21:56 +02:00 by orkun · 1 comment

orkun commented

2026-08-29 20:21:56 +02:00

Owner

Summary

A built-in testing framework with coverage, in the spirit of Go's go test:
tests live next to code, run with one command, and report pass/fail + line
coverage. This is the quality backbone for every other library in this batch.

Why it matters

The stdlib libraries proposed here (Time, Fs, Regex, Crypto, Sorting…) need
tests to be trustworthy; we should build the harness first-class rather than
shell scripts.
For game developers, keep the entry price near zero: writing a test should
feel like writing a function, and you should be able to ignore it until you
want it. It's opt-in, never in the way of shipping a prototype.

Proposed design (illustrative)

# doc-check: skip — illustrative API sketch
test "player takes damage" {
  var p = Player { Stats { hp: 10 } }
  apply_damage(p, 3)
  expect(p.hp).to_equal(7)
}

test "save round-trips" {
  let w = sample_world()
  expect(load(save(w))).to_equal(w)
}

A test "name" { … } block (or @test fn) discovered automatically.
A tiny, readable assertion set: expect(x).to_equal/to_be_true/to_be_near/…
(to_be_near for fixed-point/float — games need tolerance-based checks).
One command via the bin/x task runner: bin/x test runs all, prints a
summary; --coverage reports per-file line coverage.

Considerations

Coverage needs compiler instrumentation (line hit counts) — the biggest
lift; design the IR hooks with the type-system/compiler work in mind.
Deterministic, reproducible runs (seedable RNG helper for property-ish tests).
Fast: compile-and-run natively; parallel test execution optional later.
Integrate with the existing bin/x runner and CI on push to main.
Golden/snapshot helper would pair well with the docs doc-check tooling.

Scope / acceptance

test blocks + auto-discovery in the compiler.
Assertion library incl. tolerance + collection matchers.
bin/x test runner with a clean summary; non-zero exit on failure.
--coverage with per-file line coverage.
Docs page + convert an existing selfhost/tests/* sample to the new form.

Related: error handling (asserting failures), and every stdlib issue in this batch.

## Summary A **built-in testing framework** with coverage, in the spirit of Go's `go test`: tests live next to code, run with one command, and report pass/fail + line coverage. This is the quality backbone for every other library in this batch. ## Why it matters - The stdlib libraries proposed here (Time, Fs, Regex, Crypto, Sorting…) **need tests** to be trustworthy; we should build the harness first-class rather than shell scripts. - For **game developers**, keep the entry price near zero: writing a test should feel like writing a function, and you should be able to ignore it until you want it. It's opt-in, never in the way of shipping a prototype. ## Proposed design (illustrative) ```ludic # doc-check: skip — illustrative API sketch test "player takes damage" { var p = Player { Stats { hp: 10 } } apply_damage(p, 3) expect(p.hp).to_equal(7) } test "save round-trips" { let w = sample_world() expect(load(save(w))).to_equal(w) } ``` - A `test "name" { … }` block (or `@test fn`) discovered automatically. - A tiny, readable assertion set: `expect(x).to_equal/to_be_true/to_be_near/…` (`to_be_near` for fixed-point/float — games need tolerance-based checks). - One command via the `bin/x` task runner: `bin/x test` runs all, prints a summary; `--coverage` reports per-file line coverage. ## Considerations - **Coverage** needs compiler instrumentation (line hit counts) — the biggest lift; design the IR hooks with the type-system/compiler work in mind. - Deterministic, reproducible runs (seedable RNG helper for property-ish tests). - Fast: compile-and-run natively; parallel test execution optional later. - Integrate with the existing `bin/x` runner and CI on push to `main`. - Golden/snapshot helper would pair well with the docs `doc-check` tooling. ## Scope / acceptance - [ ] `test` blocks + auto-discovery in the compiler. - [ ] Assertion library incl. tolerance + collection matchers. - [ ] `bin/x test` runner with a clean summary; non-zero exit on failure. - [ ] `--coverage` with per-file line coverage. - [ ] Docs page + convert an existing `selfhost/tests/*` sample to the new form. Related: error handling (asserting failures), and every stdlib issue in this batch.

orkun added the

proposal

priority:high

area:tooling

labels 2026-08-29 20:21:56 +02:00

orkun referenced this issue from a commit

2026-08-31 13:27:48 +02:00

feat(testing): built-in test blocks + expect assertions, auto-run with pass/fail (#12)

orkun commented

2026-08-31 13:28:05 +02:00

Author

Owner

Shipped in ea2c6ab — a built-in testing framework in the spirit of go test: tests live next to the code, run with one command, report pass/fail, and exit non-zero on failure.

What landed:

test "name" { … } blocks, discovered automatically. No entry to write, nothing to register — the compiler synthesises a runner @main that runs every test in the file.
Assertions: expect(cond), expect_eq(a, b), and expect_near(a, b, tol). expect_near carries the tolerance fixed-point and accumulated-integer game math need (the KAT-style to_be_near the proposal called out). On failure they print file:line: <what> failed (got G, want W) and mark the test failed without aborting, so one run reports every failure.
Runner output: ok - name / FAIL - name per test, a == N passed, M failed == summary, and a non-zero exit if anything failed — so ludic spec_test.ludic drops straight into bin/x and CI.

Example:

program MathSpec {
  function add(a: int, b: int) -> int { return a + b }
  test "addition adds" { expect_eq(add(2, 3), 5) }
  test "fixed math is close" { expect_near(fixed(1) / 3, 21845, 2) }
}

How it's built: a new test declaration keyword (parse_test → N_TEST, collected in g_tests); call nodes now carry their source line for the messages; emit_test_runner lowers each block to a @fn__test_i and generates the runner @main; expect* lower to a branch-print-flag tail. The compiler's own source has no test blocks, so its self-compiled IR is byte-identical and the C-free bootstrap fixpoint still holds (verified). test is wired into the vocabulary (ludic_syntax.h, the JetBrains lexer, the TextMate grammar — x test-tools confirms they're in sync) and documented at docs/language/testing/. A passing spec (examples/library/testing.ludic) is guarded in the regression suite by a new spec_case (build, run, require exit 0 + the expected summary); x test is now 67 checks.

Acceptance: ✅ test blocks + auto-discovery, ✅ assertion library incl. tolerance, ✅ one-command run with a clean summary + non-zero exit on failure, ✅ docs page + a converted-form spec. Deferred: --coverage (per-file line coverage) — the issue's own "biggest lift," it needs IR line-hit instrumentation and is best done as its own change so the fixpoint stays easy to reason about. Closing the framework; a follow-up will track coverage.

Shipped in ea2c6ab — a built-in testing framework in the spirit of `go test`: tests live next to the code, run with one command, report pass/fail, and exit non-zero on failure. **What landed:** - **`test "name" { … }` blocks**, discovered automatically. No `entry` to write, nothing to register — the compiler synthesises a runner `@main` that runs every test in the file. - **Assertions**: `expect(cond)`, `expect_eq(a, b)`, and `expect_near(a, b, tol)`. `expect_near` carries the tolerance fixed-point and accumulated-integer game math need (the KAT-style `to_be_near` the proposal called out). On failure they print `file:line: <what> failed (got G, want W)` and mark the test failed **without aborting**, so one run reports *every* failure. - **Runner output**: `ok - name` / `FAIL - name` per test, a `== N passed, M failed ==` summary, and a **non-zero exit** if anything failed — so `ludic spec_test.ludic` drops straight into `bin/x` and CI. Example: ```ludic program MathSpec { function add(a: int, b: int) -> int { return a + b } test "addition adds" { expect_eq(add(2, 3), 5) } test "fixed math is close" { expect_near(fixed(1) / 3, 21845, 2) } } ``` **How it's built:** a new `test` declaration keyword (`parse_test` → `N_TEST`, collected in `g_tests`); call nodes now carry their source line for the messages; `emit_test_runner` lowers each block to a `@fn__test_i` and generates the runner `@main`; `expect*` lower to a branch-print-flag tail. The compiler's own source has no `test` blocks, so its self-compiled IR is byte-identical and the **C-free bootstrap fixpoint still holds** (verified). `test` is wired into the vocabulary (`ludic_syntax.h`, the JetBrains lexer, the TextMate grammar — `x test-tools` confirms they're in sync) and documented at `docs/language/testing/`. A passing spec (`examples/library/testing.ludic`) is guarded in the regression suite by a new `spec_case` (build, run, require exit 0 + the expected summary); `x test` is now 67 checks. **Acceptance:** ✅ test blocks + auto-discovery, ✅ assertion library incl. tolerance, ✅ one-command run with a clean summary + non-zero exit on failure, ✅ docs page + a converted-form spec. **Deferred:** `--coverage` (per-file line coverage) — the issue's own "biggest lift," it needs IR line-hit instrumentation and is best done as its own change so the fixpoint stays easy to reason about. Closing the framework; a follow-up will track coverage.

orkun closed this issue

2026-08-31 13:28:05 +02:00

orkun referenced this issue

2026-08-31 13:34:32 +02:00

Follow-up to #12: line coverage for the testing framework (bin/x test --coverage) #45

orkun referenced this issue from a commit

2026-08-31 13:51:23 +02:00

feat(testing): line coverage via --coverage + bin/x test --coverage (#45)

orkun referenced this issue

2026-08-31 13:51:50 +02:00

Follow-up to #12: line coverage for the testing framework (bin/x test --coverage) #45