wiki update (asset archiving, building instructions)

Asset-Archiving.md

@@ -0,0 +1,180 @@
+# Asset Archiving
+
+Terrarum uses the **TerranVirtualDisk (TEVD) Clustered format** to package game assets into a single archive file (`assets.tevd`) for distribution builds. Assets are read directly from the archive at runtime — no extraction step is needed.
+
+In development, the engine reads loose files from `./assets/`. The switch is automatic based on whether `assets.tevd` exists next to the game JAR.
+
+```
+Build time:  assets/ → assets_release/ → assets.tevd
+Runtime:     assets.tevd opened as ClusteredFormatDOM → files served on demand
+```
+
+## Overview
+
+### Two Modes of Operation
+
+| | Development | Distribution |
+|---|---|---|
+| **Assets source** | `./assets/` directory | `./assets.tevd` archive |
+| **Detection** | No `assets.tevd` present | `assets.tevd` exists next to the JAR |
+| **FileHandle type** | `Gdx.files.internal(...)` | `ClustfileHandle` (backed by `Clustfile`) |
+| **Used by** | Developers running from IDE | Release builds shipped to players |
+
+### Key Classes
+
+* `AssetCache` — Central accessor; opens the archive on startup, provides `FileHandle` instances
+* `ClustfileHandle` — GDX `FileHandle` subclass backed by a virtual disk file
+* `AssetArchiveBuilder` — Build-time tool that creates `assets.tevd` from `assets_release/`
+* `ModMgr` — Module loader; uses `AssetCache` to resolve internal mod assets transparently
+
+## Build Pipeline
+
+### Step 1: Prepare Release Assets
+
+The `assets_release/` directory is a processed copy of `assets/`, stripped of source-only files. This step is handled by `buildapp/make_assets_release.sh`.
+
+### Step 2: Create the Archive
+
+Run `make assets` from the `buildapp/` directory, or invoke the builder directly:
+
+```bash
+java -cp TerrarumBuild.jar:lib/TerranVirtualDisk.jar \
+    net.torvald.terrarum.AssetArchiveBuilderKt \
+    assets_release out/assets.tevd
+```
+
+The builder (`AssetArchiveBuilder.kt`) performs:
+
+1. **Scan** — Walks `assets_release/` to count files, directories, and total size
+2. **Allocate** — Creates a new TEVD archive with sufficient capacity (clusters + FAT entries)
+3. **Pre-grow FAT** — Expands the File Allocation Table upfront to avoid costly mid-import growth
+4. **Import** — Recursively imports all files and directories via `Clustfile.importFrom()`
+5. **Trim** — Removes unused trailing clusters to minimise file size
+
+### Build Script
+
+The build script (`buildapp/make_assets_archive.sh`) assembles the classpath from the project JAR and all library JARs, then invokes `AssetArchiveBuilderKt`:
+
+```bash
+# Simplified:
+CP="../out/TerrarumBuild.jar"
+for jar in ../lib/*.jar; do CP="$CP:$jar"; done
+java -cp "$CP" net.torvald.terrarum.AssetArchiveBuilderKt "$SRCDIR" "$OUTDIR/assets.tevd"
+```
+
+## Runtime Loading
+
+### Initialisation
+
+`AssetCache.init()` is called early in `App.main()`:
+
+```kotlin
+AssetCache.init()
+// or with an explicit path:
+AssetCache.init("/path/to/custom.tevd")
+```
+
+If `./assets.tevd` exists, the archive is opened as a read-only `ClusteredFormatDOM`. Otherwise, the engine falls back to reading loose files from `./assets/`.
+
+### Getting File Handles
+
+All asset access should go through `AssetCache` or `ModMgr`:
+
+```kotlin
+// Via AssetCache (for paths relative to assets root):
+val handle: FileHandle = AssetCache.getFileHandle("mods/basegame/audio/music/title.ogg")
+
+// Via ModMgr (for module-relative paths — preferred for mod content):
+val handle: FileHandle = ModMgr.getGdxFile("basegame", "audio/music/title.ogg")
+```
+
+Both return a standard GDX `FileHandle`. In distribution mode this is a `ClustfileHandle`; in development mode it is a regular `Gdx.files.internal(...)` handle. Callers do not need to know which.
+
+### What Works Transparently
+
+Because `ClustfileHandle` extends `FileHandle`, all standard GDX asset loading works without changes:
+
+* `Gdx.audio.newMusic(fileHandle)` — Streaming audio
+* `Texture(fileHandle)` — Texture loading
+* `Pixmap(fileHandle)` — Image processing
+* `JsonReader().parse(fileHandle.readString())` — JSON data
+* `Properties().load(fileHandle.read())` — Java properties files
+* `fileHandle.readBytes()` — Raw binary data
+* `fileHandle.list()` — Directory listing
+* `fileHandle.child("name")` — Child file access
+* `fileHandle.sibling("name")` — Sibling file access
+
+## ClustfileHandle
+
+`ClustfileHandle` bridges GDX's `FileHandle` API to TerranVirtualDisk's `Clustfile`:
+
+```kotlin
+class ClustfileHandle(private val clustfile: Clustfile) : FileHandle() {
+    override fun read(): InputStream = ClustfileInputStream(clustfile)
+    override fun readBytes(): ByteArray = clustfile.readBytes()
+    override fun exists(): Boolean = clustfile.exists()
+    override fun length(): Long = clustfile.length()
+    override fun isDirectory(): Boolean = clustfile.isDirectory
+    override fun name(): String = clustfile.name
+    override fun path(): String = clustfile.path
+    override fun list(): Array<FileHandle> = clustfile.listFiles()?.map { ClustfileHandle(it) }?.toTypedArray() ?: arrayOf()
+    override fun child(name: String): FileHandle = ClustfileHandle(Clustfile(clustfile.DOM, "$path/$name"))
+    override fun sibling(name: String?): FileHandle = ClustfileHandle(Clustfile(clustfile.DOM, "$parent/$name"))
+    override fun parent(): FileHandle = ClustfileHandle(clustfile.parentFile ?: Clustfile(clustfile.DOM, "/"))
+    // ...
+}
+```
+
+The `read()` method returns a `ClustfileInputStream` that reads data from the virtual disk's clusters or inline FAT entries, supporting all standard `InputStream` operations including `mark()`, `reset()`, and `skip()`.
+
+## AssetCache API Reference
+
+| Method | Returns | Description |
+|---|---|---|
+| `init()` | — | Opens `./assets.tevd` if present |
+| `init(path)` | — | Opens a specific `.tevd` archive |
+| `isDistribution` | `Boolean` | Whether running from an archive |
+| `getFileHandle(path)` | `FileHandle` | GDX handle for an asset (relative to assets root) |
+| `getClustfile(path)` | `Clustfile` | Raw virtual disk file handle (distribution only) |
+| `resolve(path)` | `String` | Filesystem path (development only; throws in distribution) |
+| `dispose()` | — | Closes the archive |
+
+## TEVD Archive Format
+
+The archive uses TerranVirtualDisk's **Clustered format** — the same format used for savegames. Key characteristics:
+
+* **Cluster size** — 4096 bytes
+* **FAT entries** — 256 bytes each, 16 per cluster
+* **Inline files** — Files under ~2 KB are stored directly in FAT entries (no cluster allocation)
+* **Directory listings** — Stored as arrays of 3-byte entry IDs, sorted by filename
+* **Read-only at runtime** — The archive is opened with `RandomAccessFile(file, "r")`
+
+For format details, see the [TerranVirtualDisk repository](https://github.com/minjaesong/TerranVirtualDisk).
+
+## For Module Developers
+
+If you are developing a module:
+
+* **Internal modules** (shipped with the game) — Assets are inside `assets.tevd` under `mods/<module>/`. Use `ModMgr.getGdxFile(module, path)` to access them.
+* **External modules** (installed by players) — Assets live on the real filesystem under `~/.Terrarum/Modules/<module>/`. These are **not** affected by the archive system. `ModMgr.getGdxFile()` returns a regular `FileHandle` for them.
+
+No changes to module code are needed. `ModMgr.getGdxFile()` handles both cases transparently.
+
+## Troubleshooting
+
+### Common Issues
+
+* **"No archive found, using loose assets"** — `assets.tevd` is not next to the game JAR. This is normal during development.
+* **"Cannot seek to inlined cluster"** — A `ClustfileInputStream` read went past the end of file. Ensure callers respect the stream's `available()` and return values.
+* **Files not found in archive** — Verify the file exists in `assets_release/` before building the archive. Use `AssetArchiveBuilder` with a small test directory to debug.
+
+### Verifying Archive Contents
+
+You can inspect the archive by writing a small test program:
+
+```kotlin
+val dom = ClusteredFormatDOM(RandomAccessFile(File("assets.tevd"), "r"))
+val root = Clustfile(dom, "/")
+root.listFiles()?.forEach { println(it.path) }
+dom.dispose()
+```

Building-the-App.md

@@ -0,0 +1,328 @@
+# Building the App
+
+This page covers how to build Terrarum for distribution — from preparing custom JVM runtimes and the asset archive, to packaging platform-specific executables.
+
+All five platform builds (Linux x86, Linux ARM, Windows x86, macOS Apple Silicon, macOS Intel) can be produced from a single x86 Linux workstation. The build scripts live in the `buildapp/` directory.
+
+## Quick Start
+
+### 1. Build the Project JAR
+
+Build the project from your IDE (IntelliJ IDEA). The output JAR should be at `out/TerrarumBuild.jar`.
+
+### 2. Run Prebuild
+
+Run `Prebuild.kt`'s `main()` from the IDE. This generates platform-specific metadata files in `out/`:
+
+* `build_autogen_buildinfo.properties` — Version number and build date
+* `build_autogen_linux.desktop` — Linux `.desktop` entry
+* `build_autogen_macos_Info.plist` — macOS `Info.plist`
+* `build_autogen_windows.rc` — Windows resource file (icon, version info)
+
+It also regenerates `assets/mods/basegame/metadata.properties` with the current version and date.
+
+### 3. Prepare JVM Runtimes
+
+Download JDK 21.0.2 for each target platform from [jdk.java.net/archive](https://jdk.java.net/archive/) and unzip them to `~/Documents/openjdk/` with these filenames:
+
+| Target | Filename |
+|---|---|
+| Linux AMD64 | `jdk-21.0.2-x86` |
+| Linux Aarch64 | `jdk-21.0.2-arm` |
+| Windows AMD64 | `jdk-21.0.2-windows` |
+| macOS Apple Silicon | `jdk-21.0.2.jdk-arm` |
+| macOS Intel | `jdk-21.0.2.jdk-x86` |
+
+Then create stripped-down runtimes using `jlink`:
+
+```bash
+# Linux x86
+jlink --module-path ~/Documents/openjdk/jdk-21.0.2-x86/jmods:mods \
+  --add-modules java.base,java.desktop,java.net.http,jdk.crypto.ec,java.logging,java.scripting,jdk.unsupported \
+  --output ~/Documents/Terrarum/out/runtime-linux-x86 \
+  --no-header-files --no-man-pages --strip-debug --compress=2
+
+# Linux ARM
+jlink --module-path ~/Documents/openjdk/jdk-21.0.2-arm/jmods:mods \
+  --add-modules java.base,java.desktop,java.net.http,jdk.crypto.ec,java.logging,java.scripting,jdk.unsupported \
+  --output ~/Documents/Terrarum/out/runtime-linux-arm \
+  --no-header-files --no-man-pages --strip-debug --compress=2
+
+# Windows x86
+jlink --module-path ~/Documents/openjdk/jdk-21.0.2-windows/jmods:mods \
+  --add-modules java.base,java.desktop,java.net.http,jdk.crypto.ec,java.logging,java.scripting,jdk.unsupported \
+  --output ~/Documents/Terrarum/out/runtime-windows-x86 \
+  --no-header-files --no-man-pages --strip-debug --compress=2
+
+# macOS Apple Silicon
+jlink --module-path ~/Documents/openjdk/jdk-21.0.2.jdk-arm/Contents/Home/jmods:mods \
+  --add-modules java.base,java.desktop,java.net.http,jdk.crypto.ec,java.logging,java.scripting,jdk.unsupported \
+  --output ~/Documents/Terrarum/out/runtime-osx-arm \
+  --no-header-files --no-man-pages --strip-debug --compress=2
+
+# macOS Intel
+jlink --module-path ~/Documents/openjdk/jdk-21.0.2.jdk-x86/Contents/Home/jmods:mods \
+  --add-modules java.base,java.desktop,java.net.http,jdk.crypto.ec,java.logging,java.scripting,jdk.unsupported \
+  --output ~/Documents/Terrarum/out/runtime-osx-x86 \
+  --no-header-files --no-man-pages --strip-debug --compress=2
+```
+
+The `jlink` runtimes include only the Java modules the game actually needs, significantly reducing the distribution size.
+
+> **Note:** The Linux Aarch64 runtime must be prepared on an actual ARM Linux system, then copied to your workstation.
+
+### 4. Build the Asset Archive
+
+```bash
+cd buildapp/
+make assets
+```
+
+This runs two scripts in sequence:
+
+1. `make_assets_release.sh` — Copies `assets/` to `assets_release/`, stripping development-only files
+2. `make_assets_archive.sh` — Packs `assets_release/` into `buildapp/out/assets.tevd` using the TEVD Clustered format
+
+See [[Asset Archiving]] for details on the archive format and runtime loading.
+
+### 5. Package for All Platforms
+
+```bash
+cd buildapp/
+make all
+```
+
+Or build individual targets:
+
+```bash
+make linux_x86     # Linux AMD64 AppImage
+make linux_arm     # Linux Aarch64 AppImage
+make mac           # macOS Apple Silicon .app (alias: make mac_arm)
+make mac_x86       # macOS Intel .app
+make windows       # Windows .exe + .zip
+```
+
+Output files appear in `buildapp/out/`.
+
+## Makefile Targets
+
+| Target | Output | Script |
+|---|---|---|
+| `what` (default) | Prints usage help | `make_print_description_then_exit.sh` |
+| `assets` | `out/assets.tevd` | `make_assets_release.sh` then `make_assets_archive.sh` |
+| `linux_x86` | `out/TerrarumLinux.x86.AppImage` | `build_app_linux_x86.sh` |
+| `linux_arm` | `out/TerrarumLinux.arm.AppImage` | `build_app_linux_arm.sh` |
+| `mac` / `mac_arm` | `out/TerrarumMac.arm.app.zip` | `build_app_mac_arm.sh` |
+| `mac_x86` | `out/TerrarumMac.x86.app.zip` | `build_app_mac_x86.sh` |
+| `windows` | `out/TerrarumWindows.x86.zip` | `build_app_windows_x86.sh` |
+| `all` | All five of the above | `build_app_all.sh` |
+
+## Asset Preparation
+
+### make_assets_release.sh
+
+Creates `assets_release/` from `assets/` by removing files that should not ship:
+
+* Test files (`loopey.wav`, `batchtest.txt`, `test_texture.tga`, etc.)
+* Development modules (`dwarventech`, `myawesomemod`)
+* Source/working files (`*.gz`, `*.txt`, `*.md`, `*.kra` in mod directories)
+* OS junk (`.DS_Store`, `Thumbs.db`, `.directory`)
+
+### make_assets_archive.sh
+
+Invokes `AssetArchiveBuilderKt` to pack `assets_release/` into a TEVD archive:
+
+```bash
+# Builds classpath from the project JAR and all library JARs
+CP="../out/TerrarumBuild.jar"
+for jar in ../lib/*.jar; do CP="$CP:$jar"; done
+
+java -cp "$CP" net.torvald.terrarum.AssetArchiveBuilderKt "$SRCDIR" "$OUTDIR/assets.tevd"
+```
+
+Requires `out/TerrarumBuild.jar` and `lib/TerranVirtualDisk.jar` to exist.
+
+## Prebuild (Autogenerated Files)
+
+`Prebuild.kt` (`src/net/torvald/terrarum/Prebuild.kt`) reads the version number from `App.VERSION_RAW` and generates platform-specific files. Run it before packaging.
+
+| Generated File | Purpose | Used By |
+|---|---|---|
+| `out/build_autogen_buildinfo.properties` | Version number and UTC build date | Runtime version display |
+| `out/build_autogen_linux.desktop` | FreeDesktop `.desktop` entry | Linux AppImage |
+| `out/build_autogen_macos_Info.plist` | macOS application bundle metadata | macOS `.app` |
+| `out/build_autogen_windows.rc` | Windows resource script (icon, version) | Windows `.exe` |
+| `assets/mods/basegame/metadata.properties` | Base game module metadata with current version | Module loader |
+
+## Platform-Specific Packaging
+
+Each platform build script creates a self-contained application bundle with the game JAR, the asset archive, and a custom JVM runtime. All scripts refuse to run as root.
+
+### Linux (AppImage)
+
+**Script:** `build_app_linux_x86.sh` (and `build_app_linux_arm.sh`)
+
+Packages the game as an [AppImage](https://appimage.org/) — a single executable file that runs on most Linux distributions without installation.
+
+**Structure:**
+
+```
+TerrarumLinux.x86/
+├── AppRun              ← Entry point (shell script)
+├── icns.png            ← Application icon
+├── build_autogen_linux.desktop
+├── assets.tevd         ← Game assets archive
+└── out/
+    ├── TerrarumBuild.jar
+    └── runtime-linux-x86/
+        └── bin/java    ← Custom JVM
+```
+
+**AppRun** is a shell script that launches the JVM:
+
+```bash
+./out/runtime-linux-x86/bin/java -Dswing.aatext=true -Dawt.useSystemAAFontSettings=lcd -jar ./out/TerrarumBuild.jar
+```
+
+The directory is then packed into an AppImage using `appimagetool-x86_64.AppImage`.
+
+**Requirements:** `appimagetool` must be present in `buildapp/`.
+
+### Windows (EXE + ZIP)
+
+**Script:** `build_app_windows_x86.sh`
+
+Cross-compiles a native Windows launcher using `x86_64-w64-mingw32-gcc`, then zips the entire package.
+
+**Launcher (`Terrarum.c`):** A small C program that uses `CreateProcessA` to launch the JVM with the console window hidden (`SW_HIDE`). This gives users a clean double-click experience without a visible command prompt.
+
+**Build steps:**
+
+1. Compile the Windows resource file (`.rc` → `.res`) with `x86_64-w64-mingw32-windres` for icon and version metadata
+2. Compile `Terrarum.c` with `x86_64-w64-mingw32-gcc`, linking the resource file
+3. Copy runtime, JAR, and assets into the output directory
+4. Zip into `out/TerrarumWindows.x86.zip`
+
+**Structure (inside zip):**
+
+```
+TerrarumWindows.x86/
+├── Terrarum.exe        ← Native launcher
+├── assets.tevd
+└── out/
+    ├── TerrarumBuild.jar
+    └── runtime-windows-x86/
+        └── bin/java.exe
+```
+
+**Requirements:** `mingw-w64` cross-compiler (`x86_64-w64-mingw32-gcc` and `x86_64-w64-mingw32-windres`).
+
+### macOS (.app Bundle)
+
+**Script:** `build_app_mac_arm.sh` (and `build_app_mac_x86.sh`)
+
+Creates a standard macOS `.app` bundle and compresses it with `7z`.
+
+**Structure:**
+
+```
+TerrarumMac.arm.app/
+└── Contents/
+    ├── Info.plist          ← Application metadata
+    ├── Resources/
+    │   └── AppIcon.icns    ← macOS icon
+    └── MacOS/
+        ├── Terrarum.sh     ← Entry point (shell script)
+        ├── assets.tevd
+        └── out/
+            ├── TerrarumBuild.jar
+            └── runtime-osx-arm/
+                └── bin/java
+```
+
+**Terrarum.sh** launches the JVM:
+
+```bash
+./out/runtime-osx-arm/bin/java -jar ./out/TerrarumBuild.jar
+```
+
+**Requirements:** `7z` for creating the zip archive.
+
+## Directory Layout
+
+### buildapp/
+
+```
+buildapp/
+├── Makefile
+├── make_print_description_then_exit.sh
+├── make_assets_release.sh
+├── make_assets_archive.sh
+├── build_app_all.sh
+├── build_app_linux_x86.sh
+├── build_app_linux_arm.sh
+├── build_app_mac_arm.sh
+├── build_app_mac_x86.sh
+├── build_app_windows_x86.sh
+├── appimagetool-x86_64.AppImage     ← Linux AppImage packer
+├── icns.png                         ← Linux/generic icon
+├── icns.ico                         ← Windows icon
+├── AppIcon.icns                     ← macOS icon
+├── terrarum.manifest                ← Windows application manifest
+├── terrarumlinux_x86/               ← Linux x86 template
+│   └── AppRun
+├── terrarumlinux_arm/               ← Linux ARM template
+│   └── AppRun
+├── terrarumwindows_x86/             ← Windows template
+│   └── Terrarum.c
+├── terrarummac_arm/                 ← macOS ARM template
+│   └── Terrarum.sh
+├── terrarummac_x86/                 ← macOS Intel template
+│   └── Terrarum.sh
+└── out/                             ← Build output
+    ├── assets.tevd
+    ├── TerrarumLinux.x86.AppImage
+    ├── TerrarumLinux.arm.AppImage
+    ├── TerrarumWindows.x86.zip
+    ├── TerrarumMac.arm.app.zip
+    └── TerrarumMac.x86.app.zip
+```
+
+### out/ (Project Root)
+
+```
+out/
+├── TerrarumBuild.jar                  ← Compiled game (fat JAR)
+├── build_autogen_buildinfo.properties ← Generated by Prebuild.kt
+├── build_autogen_linux.desktop
+├── build_autogen_macos_Info.plist
+├── build_autogen_windows.rc
+├── build_autogen_windows.rc.res       ← Compiled by mingw windres
+├── runtime-linux-x86/                 ← jlink runtimes
+├── runtime-linux-arm/
+├── runtime-windows-x86/
+├── runtime-osx-arm/
+└── runtime-osx-x86/
+```
+
+## Icon Creation
+
+### macOS .icns
+
+On a real macOS system:
+
+1. Create a folder called `icon.iconset`
+2. Copy the PNG icon as `icon_512x512.png` into that folder
+3. Run: `iconutil -c icns icon.iconset`
+
+### Windows .ico
+
+1. Open the PNG icon in GIMP
+2. Export As `.ico`, selecting 24 bpp (1-bit alpha) or 32 bpp (8-bit alpha)
+
+## Notes
+
+* The game does **not** use Java 9+ modules — all dependencies are fat-jar'd into `TerrarumBuild.jar`
+* The `jlink` runtimes contain only the modules listed in the `--add-modules` flag, keeping them small
+* All five executables can be built on a single x86 Linux workstation, but the ARM Linux runtime must be prepared on an actual ARM system
+* Build scripts all refuse to run as root as a safety measure

Developer-Portal.md

@@ -38,6 +38,10 @@ This is a place for developers, a hub for documentation and code conventions for
 
 **Advanced technical documentation for engine developers:**
 
+### Build & Distribution
+* [[Building the App]] — Packaging for all platforms, custom JVM runtimes, and build scripts
+* [[Asset Archiving]] — TEVD archive format, build pipeline, and runtime loading
+
 ### Rendering & Graphics
 * [[Tile Atlas System]] — Six-season atlas generation, subtiling, and dynamic texture management
 * [[Autotiling In-Depth]] — Connection algorithms, lookup tables, and subtiling patterns