From e936eae8a6a322e2c949232abc333f0ac1e63c7b Mon Sep 17 00:00:00 2001 From: minjaesong Date: Sat, 28 Feb 2026 10:53:51 +0900 Subject: [PATCH] wiki update (asset archiving, building instructions) --- Asset-Archiving.md | 180 ++++++++++++++++++++++++ Building-the-App.md | 328 ++++++++++++++++++++++++++++++++++++++++++++ Developer-Portal.md | 4 + 3 files changed, 512 insertions(+) create mode 100644 Asset-Archiving.md create mode 100644 Building-the-App.md diff --git a/Asset-Archiving.md b/Asset-Archiving.md new file mode 100644 index 0000000..119f14b --- /dev/null +++ b/Asset-Archiving.md @@ -0,0 +1,180 @@ +# Asset Archiving + +Terrarum uses the **TerranVirtualDisk (TEVD) Clustered format** to package game assets into a single archive file (`assets.tevd`) for distribution builds. Assets are read directly from the archive at runtime — no extraction step is needed. + +In development, the engine reads loose files from `./assets/`. The switch is automatic based on whether `assets.tevd` exists next to the game JAR. + +``` +Build time: assets/ → assets_release/ → assets.tevd +Runtime: assets.tevd opened as ClusteredFormatDOM → files served on demand +``` + +## Overview + +### Two Modes of Operation + +| | Development | Distribution | +|---|---|---| +| **Assets source** | `./assets/` directory | `./assets.tevd` archive | +| **Detection** | No `assets.tevd` present | `assets.tevd` exists next to the JAR | +| **FileHandle type** | `Gdx.files.internal(...)` | `ClustfileHandle` (backed by `Clustfile`) | +| **Used by** | Developers running from IDE | Release builds shipped to players | + +### Key Classes + +* `AssetCache` — Central accessor; opens the archive on startup, provides `FileHandle` instances +* `ClustfileHandle` — GDX `FileHandle` subclass backed by a virtual disk file +* `AssetArchiveBuilder` — Build-time tool that creates `assets.tevd` from `assets_release/` +* `ModMgr` — Module loader; uses `AssetCache` to resolve internal mod assets transparently + +## Build Pipeline + +### Step 1: Prepare Release Assets + +The `assets_release/` directory is a processed copy of `assets/`, stripped of source-only files. This step is handled by `buildapp/make_assets_release.sh`. + +### Step 2: Create the Archive + +Run `make assets` from the `buildapp/` directory, or invoke the builder directly: + +```bash +java -cp TerrarumBuild.jar:lib/TerranVirtualDisk.jar \ + net.torvald.terrarum.AssetArchiveBuilderKt \ + assets_release out/assets.tevd +``` + +The builder (`AssetArchiveBuilder.kt`) performs: + +1. **Scan** — Walks `assets_release/` to count files, directories, and total size +2. **Allocate** — Creates a new TEVD archive with sufficient capacity (clusters + FAT entries) +3. **Pre-grow FAT** — Expands the File Allocation Table upfront to avoid costly mid-import growth +4. **Import** — Recursively imports all files and directories via `Clustfile.importFrom()` +5. **Trim** — Removes unused trailing clusters to minimise file size + +### Build Script + +The build script (`buildapp/make_assets_archive.sh`) assembles the classpath from the project JAR and all library JARs, then invokes `AssetArchiveBuilderKt`: + +```bash +# Simplified: +CP="../out/TerrarumBuild.jar" +for jar in ../lib/*.jar; do CP="$CP:$jar"; done +java -cp "$CP" net.torvald.terrarum.AssetArchiveBuilderKt "$SRCDIR" "$OUTDIR/assets.tevd" +``` + +## Runtime Loading + +### Initialisation + +`AssetCache.init()` is called early in `App.main()`: + +```kotlin +AssetCache.init() +// or with an explicit path: +AssetCache.init("/path/to/custom.tevd") +``` + +If `./assets.tevd` exists, the archive is opened as a read-only `ClusteredFormatDOM`. Otherwise, the engine falls back to reading loose files from `./assets/`. + +### Getting File Handles + +All asset access should go through `AssetCache` or `ModMgr`: + +```kotlin +// Via AssetCache (for paths relative to assets root): +val handle: FileHandle = AssetCache.getFileHandle("mods/basegame/audio/music/title.ogg") + +// Via ModMgr (for module-relative paths — preferred for mod content): +val handle: FileHandle = ModMgr.getGdxFile("basegame", "audio/music/title.ogg") +``` + +Both return a standard GDX `FileHandle`. In distribution mode this is a `ClustfileHandle`; in development mode it is a regular `Gdx.files.internal(...)` handle. Callers do not need to know which. + +### What Works Transparently + +Because `ClustfileHandle` extends `FileHandle`, all standard GDX asset loading works without changes: + +* `Gdx.audio.newMusic(fileHandle)` — Streaming audio +* `Texture(fileHandle)` — Texture loading +* `Pixmap(fileHandle)` — Image processing +* `JsonReader().parse(fileHandle.readString())` — JSON data +* `Properties().load(fileHandle.read())` — Java properties files +* `fileHandle.readBytes()` — Raw binary data +* `fileHandle.list()` — Directory listing +* `fileHandle.child("name")` — Child file access +* `fileHandle.sibling("name")` — Sibling file access + +## ClustfileHandle + +`ClustfileHandle` bridges GDX's `FileHandle` API to TerranVirtualDisk's `Clustfile`: + +```kotlin +class ClustfileHandle(private val clustfile: Clustfile) : FileHandle() { + override fun read(): InputStream = ClustfileInputStream(clustfile) + override fun readBytes(): ByteArray = clustfile.readBytes() + override fun exists(): Boolean = clustfile.exists() + override fun length(): Long = clustfile.length() + override fun isDirectory(): Boolean = clustfile.isDirectory + override fun name(): String = clustfile.name + override fun path(): String = clustfile.path + override fun list(): Array = clustfile.listFiles()?.map { ClustfileHandle(it) }?.toTypedArray() ?: arrayOf() + override fun child(name: String): FileHandle = ClustfileHandle(Clustfile(clustfile.DOM, "$path/$name")) + override fun sibling(name: String?): FileHandle = ClustfileHandle(Clustfile(clustfile.DOM, "$parent/$name")) + override fun parent(): FileHandle = ClustfileHandle(clustfile.parentFile ?: Clustfile(clustfile.DOM, "/")) + // ... +} +``` + +The `read()` method returns a `ClustfileInputStream` that reads data from the virtual disk's clusters or inline FAT entries, supporting all standard `InputStream` operations including `mark()`, `reset()`, and `skip()`. + +## AssetCache API Reference + +| Method | Returns | Description | +|---|---|---| +| `init()` | — | Opens `./assets.tevd` if present | +| `init(path)` | — | Opens a specific `.tevd` archive | +| `isDistribution` | `Boolean` | Whether running from an archive | +| `getFileHandle(path)` | `FileHandle` | GDX handle for an asset (relative to assets root) | +| `getClustfile(path)` | `Clustfile` | Raw virtual disk file handle (distribution only) | +| `resolve(path)` | `String` | Filesystem path (development only; throws in distribution) | +| `dispose()` | — | Closes the archive | + +## TEVD Archive Format + +The archive uses TerranVirtualDisk's **Clustered format** — the same format used for savegames. Key characteristics: + +* **Cluster size** — 4096 bytes +* **FAT entries** — 256 bytes each, 16 per cluster +* **Inline files** — Files under ~2 KB are stored directly in FAT entries (no cluster allocation) +* **Directory listings** — Stored as arrays of 3-byte entry IDs, sorted by filename +* **Read-only at runtime** — The archive is opened with `RandomAccessFile(file, "r")` + +For format details, see the [TerranVirtualDisk repository](https://github.com/minjaesong/TerranVirtualDisk). + +## For Module Developers + +If you are developing a module: + +* **Internal modules** (shipped with the game) — Assets are inside `assets.tevd` under `mods//`. Use `ModMgr.getGdxFile(module, path)` to access them. +* **External modules** (installed by players) — Assets live on the real filesystem under `~/.Terrarum/Modules//`. These are **not** affected by the archive system. `ModMgr.getGdxFile()` returns a regular `FileHandle` for them. + +No changes to module code are needed. `ModMgr.getGdxFile()` handles both cases transparently. + +## Troubleshooting + +### Common Issues + +* **"No archive found, using loose assets"** — `assets.tevd` is not next to the game JAR. This is normal during development. +* **"Cannot seek to inlined cluster"** — A `ClustfileInputStream` read went past the end of file. Ensure callers respect the stream's `available()` and return values. +* **Files not found in archive** — Verify the file exists in `assets_release/` before building the archive. Use `AssetArchiveBuilder` with a small test directory to debug. + +### Verifying Archive Contents + +You can inspect the archive by writing a small test program: + +```kotlin +val dom = ClusteredFormatDOM(RandomAccessFile(File("assets.tevd"), "r")) +val root = Clustfile(dom, "/") +root.listFiles()?.forEach { println(it.path) } +dom.dispose() +``` diff --git a/Building-the-App.md b/Building-the-App.md new file mode 100644 index 0000000..9d649bb --- /dev/null +++ b/Building-the-App.md @@ -0,0 +1,328 @@ +# Building the App + +This page covers how to build Terrarum for distribution — from preparing custom JVM runtimes and the asset archive, to packaging platform-specific executables. + +All five platform builds (Linux x86, Linux ARM, Windows x86, macOS Apple Silicon, macOS Intel) can be produced from a single x86 Linux workstation. The build scripts live in the `buildapp/` directory. + +## Quick Start + +### 1. Build the Project JAR + +Build the project from your IDE (IntelliJ IDEA). The output JAR should be at `out/TerrarumBuild.jar`. + +### 2. Run Prebuild + +Run `Prebuild.kt`'s `main()` from the IDE. This generates platform-specific metadata files in `out/`: + +* `build_autogen_buildinfo.properties` — Version number and build date +* `build_autogen_linux.desktop` — Linux `.desktop` entry +* `build_autogen_macos_Info.plist` — macOS `Info.plist` +* `build_autogen_windows.rc` — Windows resource file (icon, version info) + +It also regenerates `assets/mods/basegame/metadata.properties` with the current version and date. + +### 3. Prepare JVM Runtimes + +Download JDK 21.0.2 for each target platform from [jdk.java.net/archive](https://jdk.java.net/archive/) and unzip them to `~/Documents/openjdk/` with these filenames: + +| Target | Filename | +|---|---| +| Linux AMD64 | `jdk-21.0.2-x86` | +| Linux Aarch64 | `jdk-21.0.2-arm` | +| Windows AMD64 | `jdk-21.0.2-windows` | +| macOS Apple Silicon | `jdk-21.0.2.jdk-arm` | +| macOS Intel | `jdk-21.0.2.jdk-x86` | + +Then create stripped-down runtimes using `jlink`: + +```bash +# Linux x86 +jlink --module-path ~/Documents/openjdk/jdk-21.0.2-x86/jmods:mods \ + --add-modules java.base,java.desktop,java.net.http,jdk.crypto.ec,java.logging,java.scripting,jdk.unsupported \ + --output ~/Documents/Terrarum/out/runtime-linux-x86 \ + --no-header-files --no-man-pages --strip-debug --compress=2 + +# Linux ARM +jlink --module-path ~/Documents/openjdk/jdk-21.0.2-arm/jmods:mods \ + --add-modules java.base,java.desktop,java.net.http,jdk.crypto.ec,java.logging,java.scripting,jdk.unsupported \ + --output ~/Documents/Terrarum/out/runtime-linux-arm \ + --no-header-files --no-man-pages --strip-debug --compress=2 + +# Windows x86 +jlink --module-path ~/Documents/openjdk/jdk-21.0.2-windows/jmods:mods \ + --add-modules java.base,java.desktop,java.net.http,jdk.crypto.ec,java.logging,java.scripting,jdk.unsupported \ + --output ~/Documents/Terrarum/out/runtime-windows-x86 \ + --no-header-files --no-man-pages --strip-debug --compress=2 + +# macOS Apple Silicon +jlink --module-path ~/Documents/openjdk/jdk-21.0.2.jdk-arm/Contents/Home/jmods:mods \ + --add-modules java.base,java.desktop,java.net.http,jdk.crypto.ec,java.logging,java.scripting,jdk.unsupported \ + --output ~/Documents/Terrarum/out/runtime-osx-arm \ + --no-header-files --no-man-pages --strip-debug --compress=2 + +# macOS Intel +jlink --module-path ~/Documents/openjdk/jdk-21.0.2.jdk-x86/Contents/Home/jmods:mods \ + --add-modules java.base,java.desktop,java.net.http,jdk.crypto.ec,java.logging,java.scripting,jdk.unsupported \ + --output ~/Documents/Terrarum/out/runtime-osx-x86 \ + --no-header-files --no-man-pages --strip-debug --compress=2 +``` + +The `jlink` runtimes include only the Java modules the game actually needs, significantly reducing the distribution size. + +> **Note:** The Linux Aarch64 runtime must be prepared on an actual ARM Linux system, then copied to your workstation. + +### 4. Build the Asset Archive + +```bash +cd buildapp/ +make assets +``` + +This runs two scripts in sequence: + +1. `make_assets_release.sh` — Copies `assets/` to `assets_release/`, stripping development-only files +2. `make_assets_archive.sh` — Packs `assets_release/` into `buildapp/out/assets.tevd` using the TEVD Clustered format + +See [[Asset Archiving]] for details on the archive format and runtime loading. + +### 5. Package for All Platforms + +```bash +cd buildapp/ +make all +``` + +Or build individual targets: + +```bash +make linux_x86 # Linux AMD64 AppImage +make linux_arm # Linux Aarch64 AppImage +make mac # macOS Apple Silicon .app (alias: make mac_arm) +make mac_x86 # macOS Intel .app +make windows # Windows .exe + .zip +``` + +Output files appear in `buildapp/out/`. + +## Makefile Targets + +| Target | Output | Script | +|---|---|---| +| `what` (default) | Prints usage help | `make_print_description_then_exit.sh` | +| `assets` | `out/assets.tevd` | `make_assets_release.sh` then `make_assets_archive.sh` | +| `linux_x86` | `out/TerrarumLinux.x86.AppImage` | `build_app_linux_x86.sh` | +| `linux_arm` | `out/TerrarumLinux.arm.AppImage` | `build_app_linux_arm.sh` | +| `mac` / `mac_arm` | `out/TerrarumMac.arm.app.zip` | `build_app_mac_arm.sh` | +| `mac_x86` | `out/TerrarumMac.x86.app.zip` | `build_app_mac_x86.sh` | +| `windows` | `out/TerrarumWindows.x86.zip` | `build_app_windows_x86.sh` | +| `all` | All five of the above | `build_app_all.sh` | + +## Asset Preparation + +### make_assets_release.sh + +Creates `assets_release/` from `assets/` by removing files that should not ship: + +* Test files (`loopey.wav`, `batchtest.txt`, `test_texture.tga`, etc.) +* Development modules (`dwarventech`, `myawesomemod`) +* Source/working files (`*.gz`, `*.txt`, `*.md`, `*.kra` in mod directories) +* OS junk (`.DS_Store`, `Thumbs.db`, `.directory`) + +### make_assets_archive.sh + +Invokes `AssetArchiveBuilderKt` to pack `assets_release/` into a TEVD archive: + +```bash +# Builds classpath from the project JAR and all library JARs +CP="../out/TerrarumBuild.jar" +for jar in ../lib/*.jar; do CP="$CP:$jar"; done + +java -cp "$CP" net.torvald.terrarum.AssetArchiveBuilderKt "$SRCDIR" "$OUTDIR/assets.tevd" +``` + +Requires `out/TerrarumBuild.jar` and `lib/TerranVirtualDisk.jar` to exist. + +## Prebuild (Autogenerated Files) + +`Prebuild.kt` (`src/net/torvald/terrarum/Prebuild.kt`) reads the version number from `App.VERSION_RAW` and generates platform-specific files. Run it before packaging. + +| Generated File | Purpose | Used By | +|---|---|---| +| `out/build_autogen_buildinfo.properties` | Version number and UTC build date | Runtime version display | +| `out/build_autogen_linux.desktop` | FreeDesktop `.desktop` entry | Linux AppImage | +| `out/build_autogen_macos_Info.plist` | macOS application bundle metadata | macOS `.app` | +| `out/build_autogen_windows.rc` | Windows resource script (icon, version) | Windows `.exe` | +| `assets/mods/basegame/metadata.properties` | Base game module metadata with current version | Module loader | + +## Platform-Specific Packaging + +Each platform build script creates a self-contained application bundle with the game JAR, the asset archive, and a custom JVM runtime. All scripts refuse to run as root. + +### Linux (AppImage) + +**Script:** `build_app_linux_x86.sh` (and `build_app_linux_arm.sh`) + +Packages the game as an [AppImage](https://appimage.org/) — a single executable file that runs on most Linux distributions without installation. + +**Structure:** + +``` +TerrarumLinux.x86/ +├── AppRun ← Entry point (shell script) +├── icns.png ← Application icon +├── build_autogen_linux.desktop +├── assets.tevd ← Game assets archive +└── out/ + ├── TerrarumBuild.jar + └── runtime-linux-x86/ + └── bin/java ← Custom JVM +``` + +**AppRun** is a shell script that launches the JVM: + +```bash +./out/runtime-linux-x86/bin/java -Dswing.aatext=true -Dawt.useSystemAAFontSettings=lcd -jar ./out/TerrarumBuild.jar +``` + +The directory is then packed into an AppImage using `appimagetool-x86_64.AppImage`. + +**Requirements:** `appimagetool` must be present in `buildapp/`. + +### Windows (EXE + ZIP) + +**Script:** `build_app_windows_x86.sh` + +Cross-compiles a native Windows launcher using `x86_64-w64-mingw32-gcc`, then zips the entire package. + +**Launcher (`Terrarum.c`):** A small C program that uses `CreateProcessA` to launch the JVM with the console window hidden (`SW_HIDE`). This gives users a clean double-click experience without a visible command prompt. + +**Build steps:** + +1. Compile the Windows resource file (`.rc` → `.res`) with `x86_64-w64-mingw32-windres` for icon and version metadata +2. Compile `Terrarum.c` with `x86_64-w64-mingw32-gcc`, linking the resource file +3. Copy runtime, JAR, and assets into the output directory +4. Zip into `out/TerrarumWindows.x86.zip` + +**Structure (inside zip):** + +``` +TerrarumWindows.x86/ +├── Terrarum.exe ← Native launcher +├── assets.tevd +└── out/ + ├── TerrarumBuild.jar + └── runtime-windows-x86/ + └── bin/java.exe +``` + +**Requirements:** `mingw-w64` cross-compiler (`x86_64-w64-mingw32-gcc` and `x86_64-w64-mingw32-windres`). + +### macOS (.app Bundle) + +**Script:** `build_app_mac_arm.sh` (and `build_app_mac_x86.sh`) + +Creates a standard macOS `.app` bundle and compresses it with `7z`. + +**Structure:** + +``` +TerrarumMac.arm.app/ +└── Contents/ + ├── Info.plist ← Application metadata + ├── Resources/ + │ └── AppIcon.icns ← macOS icon + └── MacOS/ + ├── Terrarum.sh ← Entry point (shell script) + ├── assets.tevd + └── out/ + ├── TerrarumBuild.jar + └── runtime-osx-arm/ + └── bin/java +``` + +**Terrarum.sh** launches the JVM: + +```bash +./out/runtime-osx-arm/bin/java -jar ./out/TerrarumBuild.jar +``` + +**Requirements:** `7z` for creating the zip archive. + +## Directory Layout + +### buildapp/ + +``` +buildapp/ +├── Makefile +├── make_print_description_then_exit.sh +├── make_assets_release.sh +├── make_assets_archive.sh +├── build_app_all.sh +├── build_app_linux_x86.sh +├── build_app_linux_arm.sh +├── build_app_mac_arm.sh +├── build_app_mac_x86.sh +├── build_app_windows_x86.sh +├── appimagetool-x86_64.AppImage ← Linux AppImage packer +├── icns.png ← Linux/generic icon +├── icns.ico ← Windows icon +├── AppIcon.icns ← macOS icon +├── terrarum.manifest ← Windows application manifest +├── terrarumlinux_x86/ ← Linux x86 template +│ └── AppRun +├── terrarumlinux_arm/ ← Linux ARM template +│ └── AppRun +├── terrarumwindows_x86/ ← Windows template +│ └── Terrarum.c +├── terrarummac_arm/ ← macOS ARM template +│ └── Terrarum.sh +├── terrarummac_x86/ ← macOS Intel template +│ └── Terrarum.sh +└── out/ ← Build output + ├── assets.tevd + ├── TerrarumLinux.x86.AppImage + ├── TerrarumLinux.arm.AppImage + ├── TerrarumWindows.x86.zip + ├── TerrarumMac.arm.app.zip + └── TerrarumMac.x86.app.zip +``` + +### out/ (Project Root) + +``` +out/ +├── TerrarumBuild.jar ← Compiled game (fat JAR) +├── build_autogen_buildinfo.properties ← Generated by Prebuild.kt +├── build_autogen_linux.desktop +├── build_autogen_macos_Info.plist +├── build_autogen_windows.rc +├── build_autogen_windows.rc.res ← Compiled by mingw windres +├── runtime-linux-x86/ ← jlink runtimes +├── runtime-linux-arm/ +├── runtime-windows-x86/ +├── runtime-osx-arm/ +└── runtime-osx-x86/ +``` + +## Icon Creation + +### macOS .icns + +On a real macOS system: + +1. Create a folder called `icon.iconset` +2. Copy the PNG icon as `icon_512x512.png` into that folder +3. Run: `iconutil -c icns icon.iconset` + +### Windows .ico + +1. Open the PNG icon in GIMP +2. Export As `.ico`, selecting 24 bpp (1-bit alpha) or 32 bpp (8-bit alpha) + +## Notes + +* The game does **not** use Java 9+ modules — all dependencies are fat-jar'd into `TerrarumBuild.jar` +* The `jlink` runtimes contain only the modules listed in the `--add-modules` flag, keeping them small +* All five executables can be built on a single x86 Linux workstation, but the ARM Linux runtime must be prepared on an actual ARM system +* Build scripts all refuse to run as root as a safety measure diff --git a/Developer-Portal.md b/Developer-Portal.md index 7c2e863..c61d9f0 100644 --- a/Developer-Portal.md +++ b/Developer-Portal.md @@ -38,6 +38,10 @@ This is a place for developers, a hub for documentation and code conventions for **Advanced technical documentation for engine developers:** +### Build & Distribution +* [[Building the App]] — Packaging for all platforms, custom JVM runtimes, and build scripts +* [[Asset Archiving]] — TEVD archive format, build pipeline, and runtime loading + ### Rendering & Graphics * [[Tile Atlas System]] — Six-season atlas generation, subtiling, and dynamic texture management * [[Autotiling In-Depth]] — Connection algorithms, lookup tables, and subtiling patterns