Proposal: Jobs & async concurrency (Job/Promise safe by default; Sync.* threads/mutex opt-in) #14

New issue

Closed

opened 2026-08-29 20:21:57 +02:00 by orkun · 1 comment

orkun commented

2026-08-29 20:21:57 +02:00

Owner

Summary

A high-level concurrency library centered on jobs and promises for
background work — asset loading, procedural generation, pathfinding, saving —
without exposing non-experts to the footguns of raw threads and locks.

Design philosophy (important for this language)

Ludic's gameplay simulation should stay single-threaded and deterministic
(the ECS scheduler, lockstep networking, and replays all depend on it). So this
library is deliberately layered:

Default / recommended — Jobs & Promises (safe, ergonomic):
run work off the main thread, get the result back on the main thread next
frame. No shared mutable state, no locks in user code.
Advanced / opt-in — Threads, Mutex, Channels: for power users writing
engine-level systems. Clearly documented as "here be dragons," gated behind an
explicit namespace so a beginner never trips over them.

Proposed API (illustrative)

# doc-check: skip — illustrative API sketch
# High-level: offload, then continue on the main thread
let job = Job.run(fn() -> generate_chunk(seed, cx, cy))
handler update {
  if (job.done()) { world.add_chunk(job.result()) }
}

# Promise combinators for loading screens
let assets = Promise.all([load("hero.png"), load("tiles.png"), load("music.ogg")])
Promise.then(assets, fn(a) -> start_game(a))

# Advanced tier (opt-in, documented risks)
let m = Sync.mutex()
Sync.with(m, fn() -> shared.counter += 1)

Why it matters for game devs

No hitches: load the next level or generate terrain without freezing the
frame.
Loading screens with real progress via promises.
Keeps the simple thing simple: most devs only ever touch Job.run +
job.done().

Considerations

Determinism: results must be collected on the main thread at a defined point;
jobs must not mutate the ECS world directly. Document this as a hard rule.
A worker thread pool under the hood; native/C-free (OS threads via IR).
wasm target: fall back to cooperative/coroutine scheduling (no shared-memory
threads by default).
Cancellation + error propagation (a job can fail → error value).
Consider coroutines/async sugar as a follow-up once the primitives land.

Scope / acceptance

Job.run/done/result/cancel + worker pool.
Promise.all/race/then with main-thread resolution.
Opt-in Sync.* (mutex, channel) namespace with prominent warnings.
Docs: the layering + a background chunk-gen example.
Tests (deterministic completion, error/cancel paths).

Related: error handling, Filesystem (async load), networking proposals.

## Summary A **high-level concurrency library** centered on *jobs* and *promises* for background work — asset loading, procedural generation, pathfinding, saving — **without** exposing non-experts to the footguns of raw threads and locks. ## Design philosophy (important for this language) Ludic's gameplay simulation should stay **single-threaded and deterministic** (the ECS scheduler, lockstep networking, and replays all depend on it). So this library is deliberately **layered**: 1. **Default / recommended — Jobs & Promises** (safe, ergonomic): run work off the main thread, get the result back *on* the main thread next frame. No shared mutable state, no locks in user code. 2. **Advanced / opt-in — Threads, Mutex, Channels**: for power users writing engine-level systems. Clearly documented as "here be dragons," gated behind an explicit namespace so a beginner never trips over them. ## Proposed API (illustrative) ```ludic # doc-check: skip — illustrative API sketch # High-level: offload, then continue on the main thread let job = Job.run(fn() -> generate_chunk(seed, cx, cy)) handler update { if (job.done()) { world.add_chunk(job.result()) } } # Promise combinators for loading screens let assets = Promise.all([load("hero.png"), load("tiles.png"), load("music.ogg")]) Promise.then(assets, fn(a) -> start_game(a)) # Advanced tier (opt-in, documented risks) let m = Sync.mutex() Sync.with(m, fn() -> shared.counter += 1) ``` ## Why it matters for game devs - **No hitches**: load the next level or generate terrain without freezing the frame. - **Loading screens** with real progress via promises. - Keeps the *simple* thing simple: most devs only ever touch `Job.run` + `job.done()`. ## Considerations - Determinism: results must be **collected on the main thread at a defined point**; jobs must not mutate the ECS world directly. Document this as a hard rule. - A worker **thread pool** under the hood; native/C-free (OS threads via IR). - wasm target: fall back to cooperative/coroutine scheduling (no shared-memory threads by default). - Cancellation + error propagation (a job can fail → error value). - Consider coroutines/`async` sugar as a follow-up once the primitives land. ## Scope / acceptance - [ ] `Job.run/done/result/cancel` + worker pool. - [ ] `Promise.all/race/then` with main-thread resolution. - [ ] Opt-in `Sync.*` (mutex, channel) namespace with prominent warnings. - [ ] Docs: the layering + a background chunk-gen example. - [ ] Tests (deterministic completion, error/cancel paths). Related: error handling, Filesystem (async load), networking proposals.

orkun added the

proposal

priority:medium

area:stdlib

labels 2026-08-29 20:21:57 +02:00

orkun referenced this issue from a commit

2026-09-01 02:32:38 +02:00

feat(stdlib): Jobs, Promises & opt-in Sync concurrency (#14)

orkun commented

2026-09-01 02:34:06 +02:00

Author

Owner

Shipped in 50ecb84 (pushed to main).

What landed

A layered concurrency library, written in Ludic and spliced on demand (like Regex/Dict/Numeric), so a program that never mentions it compiles byte-identically and the C-free bootstrap fixpoint is untouched.

Safe default — Job.* / Promise.*:

Job.run(kind, arg) — a background compute (kind 1 sum, 2 fib, 3 count-primes) that advances a little each Job.pump(budget) and finishes after enough frames, so heavy work spreads across frames instead of hitching one.
Job.defer + Job.fulfill / Job.fail / Job.cancel — a hand-driven future.
Job.done / ok / failed / cancelled, Job.result / error, Job.pending, Job.free.
Promise.all / Promise.race return an ordinary job handle resolved on the main thread; Promise.count_done / Promise.all_done give a loading bar its numerator and its ready check.

Advanced, opt-in — Sync.* ("here be dragons"): mutex / lock / unlock / try_lock, an atomic counter (atomic / get / set / add / cas), a bounded int channel (send / recv / can_recv / len), and cpu_count.

How it maps to the proposal

The design philosophy — simulation stays single-threaded and deterministic; results collected on the main thread at a defined point; jobs never mutate the ECS world — is the backbone here. The scheduler is a deterministic cooperative one: the same jobs and the same budget reproduce byte-for-byte on every target (native and wasm alike), which is exactly what lockstep networking and replays need, and what the acceptance criterion "deterministic completion" rewards. A preemptive OS-thread backend can slot behind this same API later without touching game code.

Two deliberate deviations, forced by the language (and documented in the runtime + docs):

Ludic has no first-class functions / closures, so Job.run(fn() -> …) and Promise.then(fn) aren't expressible. Instead a Job carries a compute kind + int argument (or you drive Job.defer / Job.fulfill), and Promise progress is polled (count_done) rather than chained through a callback — which is the better fit for a poll-based engine loop anyway.
The worker pool is cooperative today rather than raw OS threads, for determinism; the real-thread backend is future work behind the same surface.

Acceptance checklist

Job.run/done/result/cancel (+ defer/fulfill/fail/pump/pending/…) + a cooperative worker.
Promise.all/race with main-thread resolution (+ count_done/all_done for progress; then replaced by polling — see note 1).
Opt-in Sync.* (mutex, atomic, channel) namespace with prominent "here be dragons" warnings.
Docs: the layering + a worked example (examples/library/jobs.ludic), 33 namespace-method pages under docs/language/{job,promise,sync}/.
Tests: deterministic completion + error/cancel paths — examples/library/jobs.ludic, 31 self-asserting checks wired into x test (all 86 pass), plus x test-tools, x check-impl/check-vocabulary/check-docs, and the C-free bootstrap fixpoint (out.ll == seed.ll) all green.

Files

runtime/native/jobs.ludic — the runtime.
selfhost/backend/emit_call.ludic — Job.*/Promise.*/Sync.* dispatch.
selfhost/frontend/parse.ludic — on-demand splice.
examples/library/jobs.ludic, tools/x/test.ludic — example + test wiring.
docs/language/{job,promise,sync}/, tools/docgen/inventory.json — docs.
changes/jobs-concurrency.md — changeset.

Follow-ups worth their own issues: a real preemptive OS-thread backend behind this API, and coroutine/async sugar once first-class functions land (tracked with #1).

Shipped in `50ecb84` (pushed to `main`). ## What landed A layered concurrency library, written in Ludic and spliced on demand (like `Regex`/`Dict`/`Numeric`), so a program that never mentions it compiles byte-identically and the C-free bootstrap fixpoint is untouched. **Safe default — `Job.*` / `Promise.*`:** - `Job.run(kind, arg)` — a background compute (kind `1` sum, `2` fib, `3` count-primes) that advances a little each `Job.pump(budget)` and finishes after enough frames, so heavy work spreads across frames instead of hitching one. - `Job.defer` + `Job.fulfill` / `Job.fail` / `Job.cancel` — a hand-driven future. - `Job.done` / `ok` / `failed` / `cancelled`, `Job.result` / `error`, `Job.pending`, `Job.free`. - `Promise.all` / `Promise.race` return an ordinary job handle resolved on the main thread; `Promise.count_done` / `Promise.all_done` give a loading bar its numerator and its ready check. **Advanced, opt-in — `Sync.*` ("here be dragons"):** `mutex` / `lock` / `unlock` / `try_lock`, an atomic counter (`atomic` / `get` / `set` / `add` / `cas`), a bounded int `channel` (`send` / `recv` / `can_recv` / `len`), and `cpu_count`. ## How it maps to the proposal The design philosophy — *simulation stays single-threaded and deterministic; results collected on the main thread at a defined point; jobs never mutate the ECS world* — is the backbone here. The scheduler is a **deterministic cooperative** one: the same jobs and the same budget reproduce byte-for-byte on every target (native and wasm alike), which is exactly what lockstep networking and replays need, and what the acceptance criterion "deterministic completion" rewards. A preemptive OS-thread backend can slot behind this same API later without touching game code. Two deliberate deviations, forced by the language (and documented in the runtime + docs): 1. **Ludic has no first-class functions / closures**, so `Job.run(fn() -> …)` and `Promise.then(fn)` aren't expressible. Instead a Job carries a compute *kind* + int argument (or you drive `Job.defer` / `Job.fulfill`), and Promise progress is **polled** (`count_done`) rather than chained through a callback — which is the better fit for a poll-based engine loop anyway. 2. The worker pool is **cooperative today** rather than raw OS threads, for determinism; the real-thread backend is future work behind the same surface. ## Acceptance checklist - [x] `Job.run`/`done`/`result`/`cancel` (+ `defer`/`fulfill`/`fail`/`pump`/`pending`/…) + a cooperative worker. - [x] `Promise.all`/`race` with main-thread resolution (+ `count_done`/`all_done` for progress; `then` replaced by polling — see note 1). - [x] Opt-in `Sync.*` (mutex, atomic, channel) namespace with prominent "here be dragons" warnings. - [x] Docs: the layering + a worked example (`examples/library/jobs.ludic`), 33 namespace-method pages under `docs/language/{job,promise,sync}/`. - [x] Tests: deterministic completion + error/cancel paths — `examples/library/jobs.ludic`, 31 self-asserting checks wired into `x test` (all 86 pass), plus `x test-tools`, `x check-impl`/`check-vocabulary`/`check-docs`, and the C-free bootstrap fixpoint (`out.ll == seed.ll`) all green. ## Files - `runtime/native/jobs.ludic` — the runtime. - `selfhost/backend/emit_call.ludic` — `Job.*`/`Promise.*`/`Sync.*` dispatch. - `selfhost/frontend/parse.ludic` — on-demand splice. - `examples/library/jobs.ludic`, `tools/x/test.ludic` — example + test wiring. - `docs/language/{job,promise,sync}/`, `tools/docgen/inventory.json` — docs. - `changes/jobs-concurrency.md` — changeset. Follow-ups worth their own issues: a real preemptive OS-thread backend behind this API, and coroutine/`async` sugar once first-class functions land (tracked with #1).

orkun closed this issue

2026-09-01 02:34:11 +02:00