Yuanhao Feng

Back

You want AI to build a Minecraft castle. The most direct approach is to have the model output coordinates and materials, block by block.

A 30×30×30 building has roughly 27,000 voxel positions. Even listing only non-empty blocks, the output is thousands of lines of coordinates plus materials, each looking like this:

[12, 5, 8, "stone_bricks"]
plaintext

Beyond token cost, raw coordinates make editing and reuse difficult. When the user says “make the roof red,” there is no “roof” in a coordinate list. There are only blocks that happen to form one.

CubeMuse’s approach is to hand “what to build” to the LLM and “how to draw it” to a deterministic engine. The two systems communicate through a JSON instruction set. This article explains how that separation works and why it is designed this way.

LLM handles intent Code handles geometry Prompt Agent loop LLM + toolbox Search Vision DSL JSON DSL engine 18 ops → voxels Grid Render Export

What CubeMuse Is#

CubeMuse is a browser-based AI building generator for Minecraft players and creators. The user describes a building in natural language, the AI produces a structured build plan, a deterministic engine renders the plan into a voxel structure, and the result is exported as a Java Edition file (.litematic) or Bedrock Edition file (.mcstructure) for use in-game.

The architecture assigns natural language interpretation to the LLM and precise spatial computation to code.

The following terms appear throughout this article:

TermDefinition
GridThe voxel grid storing every block position. Single source of truth for rendering and export.
PaletteMaps 16-bit indices to concrete block types and states.
ChunkA 16×16×16 storage partition. The Grid only allocates non-empty chunks.
OpA single build instruction in the DSL, such as wall, roof, or door.
Op-logA complete DSL program (JSON). The “recipe” that deterministically produces a Grid.
BlueprintA build plan = Grid (result) + Op-log (recipe).

Coordinates and Structured Instructions#

Option A: raw coordinates Option B: structured ops [0, 0, 0, "stone_bricks"], [1, 0, 0, "stone_bricks"], [2, 0, 0, "stone_bricks"], [3, 0, 0, "stone_bricks"], ... (300+ lines) [8, 4, 5, "oak_planks"], [8, 4, 6, "oak_planks"] one line per non-air block, 300+ lines {"op":"room","id":"main", ...} {"op":"roof","anchor":{...},...} {"op":"door","anchor":{...},...} 3 ops; the engine computes all geometry Output tokens ~4,500 ~150 30:1 ✗ change material = regenerate everything ✗ reuse = copy-paste coordinates ✗ incremental edit = no concept of "roof" ✗ export = all semantics lost ✓ change material = edit one palette line ✓ reuse = group + repeat ✓ incremental edit = replace one op ✓ export = palette + named entities

Consider an 8×5×6 cottage expressed as coordinates:

[0,0,0,"stone_bricks"], [1,0,0,"stone_bricks"], [2,0,0,"stone_bricks"], ...
plaintext

For around 300 non-empty blocks, at roughly 15 tokens per line with mainstream BPE tokenizers, the output totals about 4,500 tokens.

The same building with structured instructions:

{ "op": "room", "id": "main", "box": { "from": [0,0,0], "to": [8,4,6] }, "wall_material": "wall" }
{ "op": "roof", "anchor": { "ref": "main.top_face" }, "size": [8,4,6], "shape": "gable", "material": "roof" }
{ "op": "door", "anchor": { "ref": "main.south_face", "align": "center" }, "size": [1,2,1], "facing": "south", "material": "door" }
json

Three ops, over an order of magnitude fewer tokens than the coordinate version.

Structured instructions also simplify later operations:

  1. Material editing. Changing the roof from dark to red requires updating one palette entry.
  2. Reuse. A row of windows can be defined with group and copied with repeat.
  3. Local edits. Changing the roof to hip style requires updating the roof op’s shape parameter, leaving the other ops intact.
  4. Format conversion. To export .litematic or .mcstructure, adapters convert the blocks in the Grid and their palette mappings into the data required by the target format.

These are not theoretical concerns. Early project testing confirmed that raw voxel output costs two orders of magnitude more tokens and cannot maintain spatial consistency. The ceiling for LLM-generated structured output in a single call is also limited (prior research ↗ found GPT-4 achieved only 38% completion rate). Another approach considered was Voyager ↗-style free code generation. Its visual ceiling might be higher, but the product loses named editing handles, palette reskinning, anchor alignment, deterministic testing, and the agent’s ability to do targeted revisions. CubeMuse needs structured blueprints that users can edit, share, and export after generation.

A JSON Instruction Set#

JSON and Two-Layer Validation#

JSON is a common format for structured LLM output and avoids the need for a custom text parser. JSON Schema provides structural validation (field types, enum values, required fields, additionalProperties: false to reject unknown fields), and a semantic validation layer on top checks that anchor refs point to existing named entities, that material roles are declared in the palette, that box.from ≤ box.to, and so on. The two layers together let the engine reject malformed ops before execution. Tool calls (tool_calls) return JSON objects directly, eliminating the step of extracting structure from free text and reducing format errors.

Three Levels of DSL Abstraction#

Abstraction Expressiveness Structure ops state intent, the engine computes geometry wall room roof staircase Parts: door window beam Model: building semantics. Engine: block states (facing / half / shape / hinge) Parametric geometry ops state parameters, the engine computes voxels shape (7 kinds) path fill sphere / dome / cylinder / cone / ellipsoid / arch / pyramid; path: Bresenham Free voxel op per-block placement (escape hatch) blocks (≤512) Composition ops reuse and transform across layers group repeat mirror rotate Post-processing ops decoration and components detail place_component scatter The LLM prefers the highest layer and drops down only when the layer above cannot express it. "What was missing was never freedom. It was the lower steps of the ladder." DSL v0.9.0 structure 7 + geometry 3 + free 1 + composition 4 + post-processing 3 = 18 ops

This article groups the eighteen ops into three tiers by abstraction level. Moving from structural instructions to individual voxels gives the model finer control. Generation should use high-level instructions first, reserving lower-level ops for details they cannot express.

Layer 1: Structure ops. State intent, let the engine compute geometry.

This layer includes wall, room, roof, staircase, plus members like door, window, and beam. Concepts that are not standalone ops are expressed through parameters: floors and ceilings are room’s floor_material and ceiling_material parameters; a pillar is beam with its axis set to y. The model describes architectural intent. It says “put a gable roof here,” and the engine computes each block’s facing, top or bottom half, and shape state.

Take roof as an example:

{ "op": "roof", "anchor": { "ref": "main.top_face", "offset": [0,1,0] }, "size": [10,5,8], "shape": "gable", "material": "roof" }
json

The model wrote one line. The engine determines the ridge line position, calculates the facing for the two slopes, and fills the gable triangle (gable_fill). Overhang is achieved by enlarging the roof footprint: the anchor offsets one block upward, and size adds 2 to each horizontal axis. A gable roof’s only varying state is facing (two values, one per slope); everything else is uniform across the surface.

Hip roofs best illustrate the value of structure ops. Each of the four corners needs a different corner shape (outer_left, outer_right), and facing and shape vary block by block across the entire surface. Having the model output these block states directly would produce a very high error rate. It would be handing the most error-prone part of the job to a probabilistic model.

Layer 2: Parametric geometry ops. State parameters, let the engine compute voxels.

This layer includes shape (sphere, dome, cylinder, cone, ellipsoid, arch, pyramid), path (Bresenham line voxelization, up to 64 waypoints), and fill (solid fill). These ops are not tied to architectural semantics. They are general-purpose geometric primitives. Shape supports hollow shells. A single op can produce a hollow dome. Path uses Bresenham’s algorithm for 3D polyline voxelization, useful for masts, tree branches, and cables.

Layer 3: Free voxel op. Direct block placement.

This tier contains one op, blocks, limited to 512 cells per invocation. Calls exceeding that limit are rejected. It directly specifies each block’s position and material. Lowest abstraction, highest expressiveness. Used for anything the upper two layers cannot express: facial features on a sculpture, trim curves, manual edit records.

The prompt instructs the model to prefer structural ops and use blocks only when needed. The schema rejects calls exceeding 512 cells, while evals monitor the share of blocks in the output to detect overuse of individual block placement.

Post-processing and component placement. These operations add details to existing structures or place prefabricated components.

detail scans the grid accumulated so far in the current scope and automatically adds decorative features like plinths, sills, and cornices. The engine auto-appends a detail op at the end of every AI-generated program, and the prompt accordingly forbids the model from manually placing such decorations. scatter randomly distributes vegetation or rubble on existing surfaces, with built-in support checks (only lands on solid surfaces, never overwrites, never floats). place_component places a prefabricated component at a specified position, functioning as a library call. These three operations are listed separately to complement the geometric instructions above.

Composition and Transformation Ops#

Four ops that produce no geometry of their own but reuse and transform other ops’ output. They work across all layers:

  • group packages a set of ops into a named component. Child ops render in an isolated sub-grid, then stamp onto the main grid. Think of it as a function definition.
  • repeat translates and copies a set of ops along a vector. Three identical windows along a wall do not need to be written three times.
  • mirror reflects across an x, y, or z plane. State-aware: it flips facing, half, hinge, and shape chirality. Stairs face the right way after mirroring; door hinges swap sides.
  • rotate rotates around the Y axis by 90°, 180°, or 270°, also state-aware. Rotation also swaps axis between x and z.

A symmetrical building does not need both halves described separately. One half plus mirror produces the full building and guarantees strict symmetry.

Material Role Decoupling#

The Palette maps role names to concrete blocks:

{
  "palette": {
    "primary": { "id": "minecraft:oak_planks" },
    "wall": { "id": "minecraft:stone_bricks" },
    "roof": { "id": "minecraft:dark_oak_stairs" },
    "accent": { "id": "minecraft:stripped_oak_log" },
    "glass": { "id": "minecraft:glass_pane" }
  }
}
json

Ops reference material roles through fields such as "material": "wall". Updating the corresponding palette entry changes every block that uses that role.

The palette separates ops from block IDs. The model still needs to know blocks: the prompt provides a block vocabulary list (wood families, stone types, stairs and slabs, glass, 16-color blocks, copper and its oxidation variants, etc.), and the model picks IDs from it to fill the palette. Each block ID is defined once in the palette, so material changes can be made in one place. The prompt requires each plan to use 3 to 5 material roles with clear contrast between them.

Anchor Positioning and Dependency Validation#

Once an operation is selected, it needs a position.

CubeMuse provides three positioning modes:

  1. Absolute coordinates box (from and to): closed interval, suitable for the first op placed.
  2. Anchors anchor (ref + align + offset + inset) (preferred by the AI): reference a named op’s face, center, or edge.
  3. Along arrays: place at regular intervals along an edge. Used for window strips, column rows, and other repeating structures.

A comparison illustrates why anchors matter.

Without anchors, the model computes door coordinates manually. The wall spans [0,0,0] to [8,4,6], so the door should be at [4,0,3]. After the wall is widened, the door is still at [4,0,3], no longer centered. After several rounds of modification, coordinates drift. Doors float in mid-air. Windows end up embedded in walls.

With anchors:

{ "op": "door", "anchor": { "ref": "main.south_face", "align": "center" }, "size": [1,2,1], "facing": "south", "material": "door" }
json

The door attaches to the center of main’s south face and stays centered as the wall moves or changes width.

Anchors are the DSL’s fundamental guarantee: doors stay grounded, windows are evenly spaced, roofs sit flush on walls.

Under the hood, named entities (ops carrying an id) register their bounding box after execution. Subsequent ops reference them via anchor.ref: "<name>.<part>". The <part> can be a face (south_face, top_face), corner (min and max), center (center, center_bottom), floor level (floor:N), or edge (edge:<dir>). These references form a dependency graph that keeps the entire building internally consistent across multiple rounds of modification. Anchor validity is not something JSON Schema can check (Schema only knows ref is a string); it is enforced by the semantic validation layer: the ref must point to the id of a previously executed op, and the part must be a legal value.

Grid and Op-log Representations#

Blueprint Grid (source of truth) 16³ 16³ empty 16³ ... 16-bit palette indices, sparse chunks only non-empty chunks are allocated Palette: 0→air 1→oak_planks 2→stone_bricks ... Op-log (recipe) op: room id: "main" ... op: roof anchor.ref: "main.top_face" op: door anchor.ref: "main.south_face" DSL JSON program, replayed → Grid replay Render / export AI reading / diffs The Grid is never stored. The database keeps only the Op-log (program JSON), replayed when a blueprint opens. Determinism: same op sequence → same Grid. Golden tests rely on this. The AI gets neither the Grid nor the full Op-log, but a Structure Summary (a 3–4 line text digest).

Grid: The Single Source of Truth#

Rendering, export, and statistics all read from the Grid.

  • 16×16×16 chunk sparse storage; only non-empty chunks are allocated
  • Each cell stores a 16-bit palette index, supporting up to 65,535 distinct blocks
  • Index 0 = air
  • The palette-and-index-array storage structure is shared with .litematic, .schem, and .mcstructure; import and export adapters handle index reordering and byte-order conversion as each format requires

Op-log: A Replayable Build Program#

The Op-log is a DSL program in JSON. Execute it from its starting state, and it deterministically produces the complete Grid. The starting state is usually an empty grid, but an imported building is encoded as a starting snapshot (bounding box + palette + RLE-compressed voxels) stored in the program’s base field, with subsequent ops appended on top. Imported and generated builds follow the same path through editing, undo, AI revision, and export.

Each program carries a dsl_version field. When the engine evolves (for example, changing roof’s gable_fill default from false to true), the migration layer explicitly patches stored programs with the old default, ensuring that existing blueprints reproduce the same output block-for-block while new programs benefit from the new default. The ability to replay recipes across versions is what makes the Op-log viable as a persistent representation.

State Access and Program Replay#

The Grid stores the executed result for direct access during rendering and export, avoiding a full replay each time. The Op-log preserves the build process and named structures for version comparisons and targeted edits such as replace_group. It also reduces the amount of information the AI needs to understand a build. A Blueprint contains both representations:

Blueprint
├── Grid           ← rendering / export reads this
│   ├── Palette    ← index → block type + state
│   └── Chunks[]   ← 16³ sparse partitions
└── Op-log         ← AI comprehension / version diff reads this
    └── DslProgram ← JSON, deterministic replay → Grid
plaintext

The Grid is not stored in the database. The database stores only the Op-log (the program JSON, including any base snapshot). Opening a blueprint replays the Op-log to produce the Grid. This guarantees the two are always consistent. Determinism guarantee: the same op sequence produces the same Grid. This is the foundation of golden tests.

The AI receives a Structure Summary, a 3-to-4-line text digest containing the bounding box, top-8 material distribution, and the list of named entities. Raw voxel data never enters the LLM context directly.

Dual-Edition Export: A Version-Neutral Block Model#

NeutralBlock Java-style names + block states Java adapter name mapping + NBT encoding Bedrock adapter state mapping + LE-NBT .litematic .mcstructure Cross-edition gaps are detected at export and listed in a substitution report

Internally, everything uses Java Edition naming plus block states as the neutral representation, called NeutralBlock. It reuses the Java Edition’s naming system because Java Edition has the most complete block name coverage and the most mature community toolchain.

At the import and export boundary, adapters handle format conversion:

  • Java side: .litematic (Litematica) import and export; .schem (Sponge v1-v3) import only
  • Bedrock side: .mcstructure (Structure Block)

Cross-edition discrepancies are detected at export time, producing a substitution report. For example, Java’s quartz_pillar maps to Bedrock’s quartz_block[chisel_type=lines], tagged renamed; stair shape states (corner geometry) in Java are dropped on Bedrock (Bedrock computes corners automatically); blocks outside the vocabulary are kept as-is and tagged unsupported. The mapping table covers only blocks that the DSL vocabulary actually produces, rather than embedding a full ten-thousand-row cross-edition dataset.

Closing#

The design assigns a specific role to each part:

  • The DSL expresses building intent. The engine computes geometry and the block states required by structural ops.
  • Anchors express relative positions through named entities. Structural internal consistency is guaranteed by the dependency graph.
  • The Palette manages material roles and block IDs in one place, making material replacement straightforward.
  • The Op-log preserves an editable build program; the Grid stores voxel results for rendering and export.

This pipeline forms the basis of building in CubeMuse. The next article covers how the model selects and arranges ops, and how the system controls the generation process.

References#

LLM Handles Intent, Code Handles Geometry
https://me.yuanhaofeng.com/en/blog/cubemuse-build-dsl
Author Yuanhao Feng
Published at September 9, 2026