workshopsoft/ludic

Fork 0

RFC: package manager for Ludic (fetch + resolve + namespace registration) #63

New issue

Closed

opened 2026-09-01 05:25:02 +02:00 by orkun · 1 comment

orkun commented

2026-09-01 05:25:02 +02:00

Owner

Summary

Ludic has no package manager yet. As the stdlib grows and external packages (e.g. the gameplay controllers roadmap) come online, we need a way to declare, fetch, resolve, and compile third-party Ludic packages. This issue started as an open RFC; the v1 direction is now decided (see below). The axis discussion is kept as rationale.

Two properties frame Ludic's situation:

We host on Forgejo (git.workshopsoft.io) and drive everything through one Ludic task runner (bin/x). The design slots into that — no new infra to run.
Packages come in two kinds (see "Package kinds" below): plain Ludic source, or prebuilt libs, each shipping their own Foo.* namespaces. The prebuilt-lib kind reintroduces an ABI/target concern that a source-only design would not have — called out explicitly in Open Questions.

Decisions (v1 direction)

Axis	Decision
Where source comes from	Go-style URL-as-identity, no central registry. The import path is the location (`git.workshopsoft.io/user/pkg`). `git tag` is publishing — no account, no publish step, no registry to run.
Version resolution	Go-style Minimum Version Selection (MVS). Pick the lowest version satisfying all constraints. Deterministic without heavy machinery, no SAT solver, implementable in Ludic in a few hundred lines.
Where deps live	pnpm-style content-addressed global store + links. One immutable global store keyed by content hash; each project gets a lightweight linked view (hardlink/symlink) instead of a full copy. No storage bloat across projects, no per-project duplication.
Manifest / lock	`package.ludic` (declared deps + provided namespaces) and `package.lock.ludic` (resolved versions + content hashes).
Package kinds	A package is either plain Ludic source or a prebuilt lib; both ship their own `Foo.*` namespaces.

Rationale — the decision axes

The design splits into four independent axes; the Go model is a bundle of specific choices we adopted for two of them.

Axis 1 — Where source comes from → URL-as-identity (Go)

Import path is the location. Zero infra given our Forgejo hosting. Accepted costs: no global namespace/search, name tied to host, URL typosquatting, dead host = dead package without a proxy. Rejected: central registry (must build+run infra, heavy at v0.2), hybrid index.

Axis 2 — Version resolution → MVS (Go)

Lowest-satisfying version, deterministic, no SAT solver — and critically, the resolver has to be written in Ludic, so a few-hundred-line MVS beats a full backtracking solver. Rejected: SemVer+backtracking (Cargo/npm machinery), pinned-commit-only (no diamond dedup).

Axis 3 — Where deps live → pnpm-style store + links

Global content-addressed store, projects link into it. Avoids the per-project duplication of a vendored/node_modules model while keeping the shared-immutable benefit of a global cache. Vendoring can still be offered as an opt-in for hermetic/offline builds.

Axis 4 — Manifest + lockfile → `package.ludic` + `package.lock.ludic`

Manifest lists deps and the namespace(s) the package provides; lockfile pins resolved versions + content hashes for reproducibility and integrity.

Package kinds & namespace registration (`Foo.*`)

A package must be able to register a stdlib-style Foo.* namespace (Regex., Grid., Math.* … are compiler-internal today; third parties should ship the same shape). Two kinds are in scope:

Source package — plain Ludic source, spliced and compiled together with the consumer via the existing splice-a-runtime pattern.
Prebuilt lib — a compiled artifact (the native backend already emits dylib; wasm has its own target) shipping a namespace over a stable surface.

package.ludic therefore declares, per package, the namespace(s) it provides, its kind, and (for prebuilt) which targets it ships.

Open sub-questions:

Manifest syntax for declaring a provided namespace + splice hook.
Collision policy when two packages claim the same Foo.*.
On-demand splice vs. always-linked, and how that interacts with the content-addressed store.
Prebuilt libs reintroduce an ABI + target matrix — see Open Questions.

Open questions

Prebuilt-lib ABI & target matrix. A prebuilt lib is target-specific (native arch, wasm). What's the stable ABI/calling surface between a prebuilt namespace and consumer code? How does the store key artifacts by target, and what happens when a needed target isn't shipped — hard error, or fall back to building from source if source is also present?
Manifest format details for package.ludic (declared deps, provided namespaces, kind, targets) and the exact lock schema for package.lock.ludic.
Store layout + linking strategy (hardlink vs symlink, cross-filesystem fallback, GC of unreferenced content).
Integrity: content hashes + a bin/x verify step; proxy/mirror for host-death resilience — day one or later?
v1 command surface on bin/x: x add / x get / x vendor / x update / x verify.
MVS + prebuilt interaction: does target availability participate in resolution, or only after a version is selected?

Non-goals (v1)

A public web registry UI / central index (URL-as-identity means none is needed).
General binary artifact distribution infrastructure beyond what the Forgejo git host + release attachments already provide (prebuilt libs ride on git/releases, we don't stand up a separate CDN).

## Summary Ludic has no package manager yet. As the stdlib grows and external packages (e.g. the gameplay controllers roadmap) come online, we need a way to declare, fetch, resolve, and compile third-party Ludic packages. This issue started as an open RFC; the **v1 direction is now decided** (see below). The axis discussion is kept as rationale. Two properties frame Ludic's situation: - **We host on Forgejo** (`git.workshopsoft.io`) and drive everything through one Ludic task runner (`bin/x`). The design slots into that — no new infra to run. - **Packages come in two kinds** (see "Package kinds" below): plain Ludic source, or **prebuilt libs**, each shipping their own `Foo.*` namespaces. The prebuilt-lib kind reintroduces an ABI/target concern that a source-only design would not have — called out explicitly in Open Questions. --- ## Decisions (v1 direction) | Axis | Decision | | --- | --- | | **Where source comes from** | **Go-style URL-as-identity, no central registry.** The import path *is* the location (`git.workshopsoft.io/user/pkg`). `git tag` is publishing — no account, no publish step, no registry to run. | | **Version resolution** | **Go-style Minimum Version Selection (MVS).** Pick the *lowest* version satisfying all constraints. Deterministic without heavy machinery, no SAT solver, implementable in Ludic in a few hundred lines. | | **Where deps live** | **pnpm-style content-addressed global store + links.** One immutable global store keyed by content hash; each project gets a lightweight linked view (hardlink/symlink) instead of a full copy. No storage bloat across projects, no per-project duplication. | | **Manifest / lock** | **`package.ludic`** (declared deps + provided namespaces) and **`package.lock.ludic`** (resolved versions + content hashes). | | **Package kinds** | A package is **either** plain Ludic source **or** a prebuilt lib; **both** ship their own `Foo.*` namespaces. | --- ## Rationale — the decision axes The design splits into four independent axes; the Go model is a bundle of specific choices we adopted for two of them. ### Axis 1 — Where source comes from → **URL-as-identity (Go)** Import path is the location. Zero infra given our Forgejo hosting. Accepted costs: no global namespace/search, name tied to host, URL typosquatting, dead host = dead package without a proxy. Rejected: central registry (must build+run infra, heavy at v0.2), hybrid index. ### Axis 2 — Version resolution → **MVS (Go)** Lowest-satisfying version, deterministic, no SAT solver — and critically, the resolver has to be written *in Ludic*, so a few-hundred-line MVS beats a full backtracking solver. Rejected: SemVer+backtracking (Cargo/npm machinery), pinned-commit-only (no diamond dedup). ### Axis 3 — Where deps live → **pnpm-style store + links** Global content-addressed store, projects link into it. Avoids the per-project duplication of a vendored/`node_modules` model while keeping the shared-immutable benefit of a global cache. Vendoring can still be offered as an opt-in for hermetic/offline builds. ### Axis 4 — Manifest + lockfile → **`package.ludic` + `package.lock.ludic`** Manifest lists deps and the namespace(s) the package provides; lockfile pins resolved versions + content hashes for reproducibility and integrity. --- ## Package kinds & namespace registration (`Foo.*`) A package must be able to register a stdlib-style `Foo.*` namespace (Regex.*, Grid.*, Math.* … are compiler-internal today; third parties should ship the same shape). Two kinds are in scope: 1. **Source package** — plain Ludic source, spliced and compiled together with the consumer via the existing splice-a-runtime pattern. 2. **Prebuilt lib** — a compiled artifact (the native backend already emits `dylib`; wasm has its own target) shipping a namespace over a stable surface. `package.ludic` therefore declares, per package, the namespace(s) it provides, its kind, and (for prebuilt) which targets it ships. Open sub-questions: - Manifest syntax for declaring a provided namespace + splice hook. - Collision policy when two packages claim the same `Foo.*`. - On-demand splice vs. always-linked, and how that interacts with the content-addressed store. - **Prebuilt libs reintroduce an ABI + target matrix** — see Open Questions. --- ## Open questions - **Prebuilt-lib ABI & target matrix.** A prebuilt lib is target-specific (native arch, wasm). What's the stable ABI/calling surface between a prebuilt namespace and consumer code? How does the store key artifacts by target, and what happens when a needed target isn't shipped — hard error, or fall back to building from source if source is also present? - Manifest format details for `package.ludic` (declared deps, provided namespaces, kind, targets) and the exact lock schema for `package.lock.ludic`. - Store layout + linking strategy (hardlink vs symlink, cross-filesystem fallback, GC of unreferenced content). - Integrity: content hashes + a `bin/x` verify step; proxy/mirror for host-death resilience — day one or later? - v1 command surface on `bin/x`: `x add` / `x get` / `x vendor` / `x update` / `x verify`. - MVS + prebuilt interaction: does target availability participate in resolution, or only after a version is selected? ## Non-goals (v1) - A public web registry UI / central index (URL-as-identity means none is needed). - General binary artifact *distribution infrastructure* beyond what the Forgejo git host + release attachments already provide (prebuilt libs ride on git/releases, we don't stand up a separate CDN).

orkun referenced this issue from a commit

2026-09-01 06:08:18 +02:00

feat(pkg): package manager — fetch + MVS resolve + namespace registration (#63)

orkun commented

2026-09-01 06:12:01 +02:00

Author

Owner

Implemented and shipped to main in 2c44bae.

The v1 direction from this RFC is now a working package manager — a set of x subcommands plus one small, contained compiler change. All four axes landed as decided:

URL-as-identity, no registry. A dependency is named by its git import path; a git tag vX.Y.Z publishes a version. Fetching drives plain git (a local $LUDIC_PKG_PROXY tree serves as a mirror/offline source and is what the test suite uses).

Minimum Version Selection. A require is a minimum; the resolver picks the greatest required minimum per module, then keeps the reachable closure at those versions. Deterministic, no SAT solver — a few hundred lines of Ludic in tools/x/pkg.ludic.

Content-addressed global store + per-project links (pnpm-style). Packages live once under ~/.ludic/store/<sha256> (keyed by a metadata-independent content hash); each project gets a symlinked view under ludic_modules/. $LUDIC_STORE overrides the location.

package.ludic + package.lock.ludic. The manifest declares deps, provided namespaces, kind (source/prebuilt) and prebuilt targets; the lock pins resolved versions + content hashes. x verify rehashes each store entry and confirms the project links to it.

Namespace registration. do_import now resolves a non-local, non-absolute import under $LUDIC_MODULES (default ludic_modules/), so a fetched source package's Ludic is spliced into the consumer's AOT build exactly the way the built-in stdlib namespaces are — no ABI seam, whole-program determinism preserved. Two packages claiming the same Foo.* is a hard error. Prebuilt libs declare shipped targets; a missing target for the build target is a hard error (falls back to source when the package also ships it) — the documented escape hatch over the engine C-ABI.

Command surface (all on bin/x): add, get, update, verify, vendor.

Open questions from the issue — resolved for v1:

Prebuilt ABI & target matrix — target id is $LUDIC_TARGET else native-<arch>; a package keys artifacts by target and errors when the needed one is absent (source fallback if present). Full cross-target artifact builds remain future work; the resolution/verification contract is in place.
Manifest/lock schema — line-oriented, documented in docs/PACKAGES.md.
Store layout & linking — symlink into an immutable content-addressed store; x vendor gives the hermetic/offline copy (LUDIC_MODULES=vendor).
Integrity — content hashes in the lock + x verify. A proxy/mirror rides on $LUDIC_PKG_PROXY (host-death resilience without standing up a CDN).
MVS + prebuilt interaction — target availability is checked after a version is selected (not part of version resolution).

Verification. New hermetic suite x test-pkg (throwaway git repos, fully offline) covers fetch → MVS → store → link → compile → run → verify → collision → prebuilt-target → vendor — 12/12 green — and is gated inside x test. The full regression suite is 87/87; the existing golden renders are byte-identical and the C-free bootstrap fixpoint holds (the import fallback only fires when the local path is absent, so existing programs compile unchanged; the seed was regenerated).

Docs: docs/PACKAGES.md (design + manifest/lock/commands), README quick-start, and a changeset for the next release.

This also satisfies the packaging prerequisite that the controllers roadmap (#57–#62) depends on: external, versioned source packages that compile into the consumer. Registry-izing the engine-system hooks (the other half of #62) is separate follow-up.

Implemented and shipped to `main` in 2c44bae. The v1 direction from this RFC is now a working package manager — a set of `x` subcommands plus one small, contained compiler change. All four axes landed as decided: **URL-as-identity, no registry.** A dependency is named by its git import path; a `git tag vX.Y.Z` publishes a version. Fetching drives plain `git` (a local `$LUDIC_PKG_PROXY` tree serves as a mirror/offline source and is what the test suite uses). **Minimum Version Selection.** A `require` is a minimum; the resolver picks the greatest required minimum per module, then keeps the reachable closure at those versions. Deterministic, no SAT solver — a few hundred lines of Ludic in `tools/x/pkg.ludic`. **Content-addressed global store + per-project links (pnpm-style).** Packages live once under `~/.ludic/store/<sha256>` (keyed by a metadata-independent content hash); each project gets a symlinked view under `ludic_modules/`. `$LUDIC_STORE` overrides the location. **`package.ludic` + `package.lock.ludic`.** The manifest declares deps, provided namespaces, kind (source/prebuilt) and prebuilt targets; the lock pins resolved versions + content hashes. `x verify` rehashes each store entry and confirms the project links to it. **Namespace registration.** `do_import` now resolves a non-local, non-absolute import under `$LUDIC_MODULES` (default `ludic_modules/`), so a fetched **source package**'s Ludic is spliced into the consumer's AOT build exactly the way the built-in stdlib namespaces are — no ABI seam, whole-program determinism preserved. Two packages claiming the same `Foo.*` is a hard error. **Prebuilt libs** declare shipped `targets`; a missing target for the build target is a hard error (falls back to source when the package also ships it) — the documented escape hatch over the engine C-ABI. **Command surface (all on `bin/x`):** `add`, `get`, `update`, `verify`, `vendor`. **Open questions from the issue — resolved for v1:** - *Prebuilt ABI & target matrix* — target id is `$LUDIC_TARGET` else `native-<arch>`; a package keys artifacts by target and errors when the needed one is absent (source fallback if present). Full cross-target artifact *builds* remain future work; the resolution/verification contract is in place. - *Manifest/lock schema* — line-oriented, documented in `docs/PACKAGES.md`. - *Store layout & linking* — symlink into an immutable content-addressed store; `x vendor` gives the hermetic/offline copy (`LUDIC_MODULES=vendor`). - *Integrity* — content hashes in the lock + `x verify`. A proxy/mirror rides on `$LUDIC_PKG_PROXY` (host-death resilience without standing up a CDN). - *MVS + prebuilt interaction* — target availability is checked after a version is selected (not part of version resolution). **Verification.** New hermetic suite `x test-pkg` (throwaway git repos, fully offline) covers fetch → MVS → store → link → compile → run → verify → collision → prebuilt-target → vendor — 12/12 green — and is gated inside `x test`. The full regression suite is 87/87; the existing golden renders are byte-identical and the C-free bootstrap fixpoint holds (the import fallback only fires when the local path is absent, so existing programs compile unchanged; the seed was regenerated). Docs: `docs/PACKAGES.md` (design + manifest/lock/commands), README quick-start, and a changeset for the next release. This also satisfies the packaging prerequisite that the controllers roadmap (#57–#62) depends on: external, versioned **source packages** that compile into the consumer. Registry-izing the *engine-system* hooks (the other half of #62) is separate follow-up.

orkun closed this issue

2026-09-01 06:12:01 +02:00

orkun referenced this issue

2026-09-01 06:29:51 +02:00

feat: prebuilt binary packages — closed-source functions/systems/components over the reflection ABI #64

orkun referenced this issue

2026-09-01 06:52:34 +02:00

feat: prebuilt binary packages — closed-source functions/systems/components over the reflection ABI #64

orkun referenced this issue

2026-09-01 11:32:36 +02:00

Proposal: package/module distribution + package-declarable components & engine-systems (prerequisite for controller libs #57) #62