From 075a7b143037713337b6658dfa18fecc76b0fcdc Mon Sep 17 00:00:00 2001 From: Orkuncakilkaya Date: Tue, 1 Sep 2026 07:52:06 +0300 Subject: [PATCH] =?UTF-8?q?feat(pkg):=20x=20build-lib=20/=20prebuilt=20con?= =?UTF-8?q?sumption=20=E2=80=94=20binary=20packages=20end-to-end=20(#64)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The package-manager half of prebuilt binary packages, on top of the compiler foundation (dynamic system registration + --emit-module). - x build-lib [module.ludic]: compile a package's module to a per-target native dylib under lib//, with an @rpath install name so a consumer resolves it from the content-addressed store. - x link-flags: print the clang flags (the dylib, an rpath to its store dir, -export_dynamic) so any build system links a project's prebuilt module dylibs; x app splices them automatically for in-repo builds. - kind prebuilt is resolved + linked like any dependency; a missing build target stays a hard error. Proven hermetically (macOS-gated, since dylibs are native): a module exporting a component + an @System(Update) + a function is built with x build-lib, fetched as a prebuilt dep, and linked into a consumer game that never saw its source — the module's system mutates the shared world and its function is callable ("3 42"). Package suite 17/0; full suite 87/0. Co-Authored-By: Claude Opus 4.8 --- changes/prebuilt-packages.md | 3 ++ docs/PACKAGES.md | 64 +++++++++++++++++++++++++++----- tools/x/build.ludic | 6 ++- tools/x/main.ludic | 4 ++ tools/x/pkg.ludic | 71 ++++++++++++++++++++++++++++++++++++ tools/x/pkg_test.ludic | 36 ++++++++++++++++++ 6 files changed, 173 insertions(+), 11 deletions(-) create mode 100644 changes/prebuilt-packages.md diff --git a/changes/prebuilt-packages.md b/changes/prebuilt-packages.md new file mode 100644 index 00000000..6352a07a --- /dev/null +++ b/changes/prebuilt-packages.md @@ -0,0 +1,3 @@ +bump: minor +type: feat +Prebuilt binary packages (#64) — a package can now ship a compiled artifact whose exported functions, systems and components a consumer uses without the source, over the stable reflection C-ABI. `x build-lib ` compiles a package's module to a per-target native dylib (`lib//`); a `kind prebuilt` dependency is fetched and linked like any other, and `x link-flags` prints the clang flags to link the module dylibs into a game (or `x app` does it in-repo). A module registers its dynamic components (`world_register_prop`) and its `@System(Phase)` functions with the host at load through a constructor, and the host dispatches every registered system each frame — the systems analogue of dynamic components. Binary packages are native-only and second-class ECS by design (source packages remain the portable, first-class, deterministic path); missing a build target is a hard error. See docs/PACKAGES.md. diff --git a/docs/PACKAGES.md b/docs/PACKAGES.md index b9a24a44..48d1e7e4 100644 --- a/docs/PACKAGES.md +++ b/docs/PACKAGES.md @@ -102,16 +102,62 @@ still hold. Two packages may not register the same `Foo.*` namespace — a collision is a hard error naming both modules. -## Prebuilt libraries +## Prebuilt binary packages (issue #64) -A `kind prebuilt` package ships a compiled artifact per target it declares in -`targets`. Resolution selects the artifact for the build target -(`$LUDIC_TARGET`, else `native-` for the host). If a needed target is not -shipped it is a hard error — unless the package also ships source, in which case -the source path is used. Prebuilt libs are the escape hatch for closed-source or -other-language code over the engine's stable C-ABI; source packages are the -default because they keep cross-compilation (including the wasm target) and the -compile-time ECS first-class. +A `kind prebuilt` package ships a **compiled artifact** (a native dylib per +target it lists in `targets`) instead of source. A consumer uses its exported +**functions, systems and components** without ever seeing the source. This is +the escape hatch for closed-source or other-language code; source packages stay +the default because they keep cross-compilation (including wasm) and the +compile-time ECS first-class. Prebuilt packages are **native-only** and ride the +stable reflection C-ABI — second-class ECS (dynamic, by-name components; one +indirect call per registered system), not part of the deterministic/replay core. + +**How a binary module works.** The module is compiled with `--emit-module`: no +`main`, no world table (the consumer owns the single world). It carries a +load-time constructor that, when the dylib loads, registers its pieces against +the host through the C-ABI: + +- **Components** — `world_register_prop(name, nfields)` in a `module_init` + function; the host owns storage, the module reads/writes by name with + `world_get`/`world_set`/`world_has`/`world_attach_dyn`. +- **Systems** — a function marked `@System(Phase)` is registered with + `ludic_register_system`; the host's frame loop calls it every frame in that + phase, after its own handlers. Phases: `Input`, `FixedUpdate`, `Update`, + `LateUpdate`, `Render`, `Start`, `OnQuit`. +- **Functions** — plain functions become dylib symbols; a consumer binds them + with `extern function name(...) -> T = "fn_name"`. + +A module registers its `ludic_*` calls as undefined and binds them back to the +host image at load (`-undefined dynamic_lookup`); the host exports its ABI +(`-export_dynamic`). Every Ludic game is a capable host — the reflection ABI is +always emitted (unused parts dead-strip). + +**Publishing.** In the package repo: + +``` +x build-lib module.ludic # -> lib//lib.dylib +# add `kind prebuilt` and `targets ""` to package.ludic, commit lib/, git tag +``` + +**Consuming.** In the game project: + +``` +x add git.host/user/module # kind prebuilt is resolved + the dylib linked into the view +x get # links the artifact for the build target (hard error if the target is missing) +``` + +then link the module dylibs into the game build. `x link-flags` prints the exact +clang flags (the dylib, an rpath to the store, `-export_dynamic`) for any build +system to splice into its link step: + +``` +clang -O2 game.ll $(x link-flags) -o game +``` + +(`x app` links them automatically when building in-repo.) If the package does +not ship the build target, `x get` fails — build from source instead where the +package offers it. ## Offline / hermetic builds diff --git a/tools/x/build.ludic b/tools/x/build.ludic index 256a8fd9..da4953d2 100644 --- a/tools/x/build.ludic +++ b/tools/x/build.ludic @@ -62,12 +62,14 @@ function cmd_app() -> int { if (src == null) { err("usage: x app [--headless] [--save-temps]\n"); return 1 } let name = capture_line(`basename {src} .ludic`) + let pbf = prebuilt_link_flags() # #64: link resolved prebuilt module dylibs, if any + if (mode == 2) { let outbin = `build/{name}_headless` let ll = `{outbin}.ll` print(`ludicc: {src} -> {outbin} (renders the last frame to build/out.ppm)`) if not shq(`bin/ludicc --headless {src} > {ll}`) { return 1 } - if not shq(`{cc()} -O2 {ll} -o {outbin}`) { return 1 } + if not shq(`{cc()} -O2 {ll}{pbf} -o {outbin}`) { return 1 } if not save { run(`rm -f {ll}`) } print(`done. run: mkdir -p build && printf 'ddss' | ./{outbin} && open build/out.ppm`) return 0 @@ -79,7 +81,7 @@ function cmd_app() -> int { if not shq(`bin/ludicc --windowed {src} > {ll}`) { return 1 } # audio.ll (#22) is always linked here for the dev runner — unused snd_* are # dead-stripped; the canonical `ludicc -o` path links it only when Audio.* is used. - if not shq(`{cc()} -O2 {ll} runtime/native/cocoa.ll runtime/native/audio.ll -framework Cocoa -Wl,-needed_framework,GameController -Wl,-needed_framework,AVFoundation -Wl,-rpath,@loader_path -o {outbin}`) { return 1 } + if not shq(`{cc()} -O2 {ll} runtime/native/cocoa.ll runtime/native/audio.ll -framework Cocoa -Wl,-needed_framework,GameController -Wl,-needed_framework,AVFoundation -Wl,-rpath,@loader_path{pbf} -o {outbin}`) { return 1 } if not save { run(`rm -f {ll}`) } print(`done. run: ./{outbin} (from the repo root, so assets/ resolves)`) return 0 diff --git a/tools/x/main.ludic b/tools/x/main.ludic index a0434921..8f51eb17 100644 --- a/tools/x/main.ludic +++ b/tools/x/main.ludic @@ -43,6 +43,8 @@ program X { print(" x update [module] bump a dep (or all) to its latest published version, then relock") print(" x verify check every locked package against the store by content hash") print(" x vendor copy the resolved packages into ./vendor for offline builds") + print(" x build-lib (#64) compile a package's module to a prebuilt dylib in lib//") + print(" x link-flags (#64) print the clang flags to link this project's prebuilt module dylibs") print("") print("test:") print(" x test the full regression suite") @@ -121,6 +123,8 @@ program X { if (cmd == "update") { exit(cmd_pkg_update()) } if (cmd == "verify") { exit(cmd_pkg_verify()) } if (cmd == "vendor") { exit(cmd_pkg_vendor()) } + if (cmd == "build-lib") { exit(cmd_pkg_build_lib()) } + if (cmd == "link-flags") { exit(cmd_pkg_link_flags()) } if (cmd == "test-pkg") { exit(cmd_test_pkg()) } if (cmd == "version") or (cmd == "--version") or (cmd == "-v") { exit(cmd_version()) } if (cmd == "release") { exit(cmd_release()) } diff --git a/tools/x/pkg.ludic b/tools/x/pkg.ludic index df76d3a2..74932aab 100644 --- a/tools/x/pkg.ludic +++ b/tools/x/pkg.ludic @@ -638,6 +638,77 @@ function cmd_pkg_vendor() -> int { return 0 } +# ---- prebuilt binary packages (issue #64) ----------------------------------- + +# x build-lib [module.ludic] — compile a package's module to a prebuilt dylib for +# the current target, under lib//. The developer runs this in the package +# repo, commits the artifact, and `git tag`s — consumers fetch and link it. +# The compiler is $LUDICC (default bin/ludicc) so this works both in-repo and, +# with the flag set, from an installed toolchain. +function cmd_pkg_build_lib() -> int { + var src = argn(2, "") + if src == "" { src = capture_line("ls *.ludic 2>/dev/null | grep -v package.ludic | head -1") } + if src == "" or not file_exists(src) { err("usage: x build-lib (run in the package directory)\n"); return 1 } + let man = read_root_manifest() + var name = "" + if man.module != "" { name = capture_line(`basename {man.module}`) } + if name == "" { name = capture_line(`basename {src} .ludic`) } + let t = target_id() + run(`mkdir -p lib/{t}`) + let ludicc = getenv_or("LUDICC", "bin/ludicc") + let ll = "/tmp/x_buildlib.ll" + if not shq(`{ludicc} --emit-module {src} -o {ll} 2>/tmp/x_bl.err`) { + err(`x: build-lib compile failed — {capture_line("tail -1 /tmp/x_bl.err")}\n`); return 1 + } + let out = `lib/{t}/lib{name}.dylib` + # @rpath install name so a consumer resolves it via -rpath to the store dir + if not shq(`{cc()} -O2 -Wno-override-module -dynamiclib -undefined dynamic_lookup -Wl,-install_name,@rpath/lib{name}.dylib {ll} -o {out} 2>/tmp/x_bl.err`) { + err(`x: build-lib link failed — {capture_line("tail -1 /tmp/x_bl.err")}\n`); return 1 + } + run(`rm -f {ll}`) + print(`built {out} (target {t})`) + print(`add to package.ludic: kind prebuilt / targets "{t}" — then commit lib/ and git tag`) + return 0 +} + +# x link-flags — print the clang link flags a consumer build needs to pull in the +# project's prebuilt module dylibs (for the current target). Portable escape from +# `x app` (which is in-repo only): any build system can splice `$(x link-flags)` +# into its link step. +function cmd_pkg_link_flags() -> int { + let f = prebuilt_link_flags() + print(s_trim(f)) + return 0 +} + +# clang link flags that pull every locked prebuilt package's dylib (for the +# current target) into a consumer build: the dylib itself, an rpath to its store +# directory, and -export_dynamic so the module resolves the host's ludic_* ABI at +# load. Empty when the project has no prebuilt dependencies. +function prebuilt_link_flags() -> pointer { + let txt = read_file("package.lock.ludic") + if txt == null { return "" } + let locked = parse_lock(txt) + let t = target_id() + var flags = "" + var any = false + var i = 0 + while i < len(locked) { + let m = locked[i] + if m.kind == "prebuilt" { + let raw = strip_prefix(m.hash, "sha256:") + let dir = `{store_root()}{raw}/lib/{t}` + let libs = split_lines(capture(`ls {dir}/*.dylib 2>/dev/null`)) + var k = 0 + while k < len(libs) { flags = flags + " " + libs[k]; any = true; k = k + 1 } + if len(libs) > 0 { flags = flags + ` -Wl,-rpath,{dir}` } + } + i = i + 1 + } + if any { flags = flags + " -Wl,-export_dynamic" } + return flags +} + # drop a leading `prefix` from `s` if present function strip_prefix(s: pointer, prefix: pointer) -> pointer { let pn = slen(prefix) diff --git a/tools/x/pkg_test.ludic b/tools/x/pkg_test.ludic index 309951bf..60f1fe2a 100644 --- a/tools/x/pkg_test.ludic +++ b/tools/x/pkg_test.ludic @@ -170,5 +170,41 @@ function cmd_test_pkg() -> int { ok("x vendor copies the resolved packages into ./vendor") } else { bad2("x vendor failed", capture_line(`tail -1 {work}/vendor.out`)) } + # ---- prebuilt binary package (issue #64): exported function + system + component + # A module is compiled to a dylib (x build-lib), fetched as a `kind prebuilt` + # dependency, and linked into a consumer game that never saw its source. At load + # the module registers a dynamic component and an @System(Update); the consumer + # also calls an exported function via `extern`. Native/dylib only → macOS-gated. + if not is_darwin() { + skip("prebuilt binary packages (dylib) — native/macOS only") + return report() + } + let tgt = `native-{capture_line("uname -m")}` + let mp = `{proxy}/example.test/manapack` + run(`mkdir -p {mp} && git -C {mp} init -q`) + pt_write(`{mp}/package.ludic`, `package "example.test/manapack"` + nl() + `version "1.0.0"` + nl() + "kind prebuilt" + nl() + `provides "Mana"` + nl() + `targets "{tgt}"` + nl()) + pt_write(`{mp}/manapack.ludic`, "program ManaPack {\n function module_init() -> void { world_register_prop(\"Mana\", 2) }\n @System(Update) function mana_regen() -> void {\n let M = world_prop_id(\"Mana\")\n let e = world_query_next(M, 0)\n if e >= 0 { let v = world_get(e, M, 0); world_set(e, M, 0, v + 1) }\n }\n function mana_bonus() -> int { return 42 }\n}\n") + if shq(`( cd {mp} && LUDICC={root}/bin/ludicc {root}/bin/x build-lib manapack.ludic > {work}/bl.out 2>&1 )`) and file_exists(`{mp}/lib/{tgt}/libmanapack.dylib`) { + ok("x build-lib compiles a package module to a per-target dylib") + } else { bad2("x build-lib failed", capture_line(`tail -1 {work}/bl.out`)) } + pt_commit_tag(mp, "v1.0.0") + + let proj3 = `{work}/proj3` + pt_write(`{proj3}/package.ludic`, `package "game"` + nl() + `version "0.0.0"` + nl() + `require "example.test/manapack" "1.0.0"` + nl()) + pt_write(`{proj3}/game.ludic`, "program Game {\n property Dummy { x: int = 0 }\n model Ent { Dummy }\n @OnSpawn(Ent) handler Init { }\n extern function mana_bonus() -> int = \"fn_mana_bonus\"\n var frames: int = 0\n handler Boot phase Start {\n let e = world_spawn(world_model_id(\"Ent\"))\n let M = world_prop_id(\"Mana\")\n world_attach_dyn(e, M)\n world_set(e, M, 0, 0)\n }\n handler Run phase Update {\n frames = frames + 1\n if frames >= 4 {\n let M = world_prop_id(\"Mana\")\n let e = world_query_next(M, 0)\n print(world_get(e, M, 0))\n print(mana_bonus())\n quit()\n }\n }\n}\n") + if shq(`( cd {proj3} && {envp} {root}/bin/x get > {work}/get3.out 2>&1 )`) and file_exists(`{proj3}/ludic_modules/example.test/manapack/lib/{tgt}/libmanapack.dylib`) { + ok("x get fetches + links a prebuilt (kind prebuilt) package for the target") + } else { bad2("prebuilt x get failed", capture_line(`tail -1 {work}/get3.out`)) } + # compile the ECS consumer from the repo root (so runtime/native resolves), then + # link the module dylib via `x link-flags` (a real external build's link step). + run(`LUDIC_HOME={root} {root}/bin/ludicc --headless {proj3}/game.ludic --emit-llvm -o {proj3}/game.ll 2>{work}/gc.err`) + let flags = capture_line(`( cd {proj3} && LUDIC_STORE={store} {root}/bin/x link-flags )`) + if shq(`{cc()} -O2 -Wno-override-module {proj3}/game.ll {flags} -o {proj3}/game 2>{work}/gl.err`) { + let got = capture_line(`printf 'aaaaaa' | {proj3}/game`) + if got == "3 42" { + ok("a consumer links the prebuilt module: its system mutates the world + its function is callable (\"3 42\")") + } else { bad2("prebuilt consumer output mismatch", `got [{got}] want [3 42]`) } + } else { bad2("prebuilt consumer link failed", capture_line(`tail -1 {work}/gl.err`)) } + return report() }