The second article introduced CubeMuse’s Agent build loop: the model uses a set of tools to build voxel structures, deciding what to do next at each step. One of those tools is inspect, which returns block count, bounding box, material breakdown, and lint results. These text results give the model only part of the picture. inspect can tell the model “the roof uses 120 red blocks,” but not “the roof looks disproportionate” or “the wall and roof colors clash.” The model needs images to judge silhouette, proportion, and color harmony.
CubeMuse feeds visual information into the build process through two paths. The first is look: the Agent calls it during building, and the server-side soft renderer produces four screenshots of the current Grid. A vision model returns an appearance description, which is added to the Agent’s context for later steps. The second is critique: the user triggers it after building is complete. The browser captures four standard views, runs structural checks and a visual review, and returns a score with revision suggestions.
Visual Observation During Building#
In one Suzhou garden build, visual feedback changed what the Agent built next. At step 19, the Agent called look, and the vision model pointed out insufficient vertical layering. At step 20, the Agent executed add_ops, adding four named components: a tower, corridor roofing, flowers, and red ornaments. The visual observation was included in the next operation’s context, and the Agent used it to adjust the build.
look requires the vision model to return a description field containing two to four sentences about the build’s appearance. The parser extracts this field first; on failure it falls back to the raw text, truncated to 700 characters. This description enters the build model’s context as a tool result, and the build model decides whether changes are needed.
The build model and the vision model are separate model calls. The vision model receives images and returns text; the build model reads that text and continues generating DSL. They do not share a context window. The description passes visual observations from one model to the other.
Server-Side Soft Rendering#
look runs on the server, with no GPU available. CubeMuse uses a CPU-only soft renderer to generate screenshots.
Rendering has four stages. First, voxels inside the Grid’s bounding box are copied into a dense 3D array. Each ray is then clipped to the bounding box and stepped through the voxel grid using the DDA algorithm until it reaches the first non-empty voxel. On a hit, the pixel is shaded by the block family’s base color and a face-direction brightness coefficient. Finally, the pixel buffer is encoded as a PNG data URL.
The soft renderer outputs four 384×240 PNG images per call, sharing the 16:10 frame ratio and four standard view angles with the browser screenshots. Camera poses, labels, and framing distances come from a shared geometry module, poses.ts, covered later in this article.
Flat shading with face-direction brightness captures volume effectively, but stairs and other non-full blocks are rendered as full cubes, and transparent materials stop the ray at the first non-empty voxel. This means look primarily observes volume and silhouette; material detail and sub-block shapes require browser images.
look soft render | critique browser capture | |
|---|---|---|
| Output | 4× 384×240 PNG | 4× 640×400 JPEG |
| Environment | Node server, no GPU | Browser, Three.js |
| Shading | Face-direction flat color | Textures + vertex AO |
| Non-full blocks | Full-cube approximation | Full geometry |
Before the browser takes screenshots, fog is temporarily disabled, the draw buffer is resized to the target dimensions, and the camera aspect ratio is updated. A finally block restores fog, buffer size, DPR, and aspect ratio. On success, the camera position and rotation are also restored, and one frame is re-rendered. This ensures captures are independent of the current viewport state.
Observation Timing and Call Budgets#
The timing of look depends on how much has been built and how many steps remain. Too early, and the model sees a scatter of unconnected blocks with no actionable feedback. Too late, and the remaining build steps are not enough to make meaningful changes.
The prompt suggests calling look after the main volume has taken shape, around one-third of the budget. If actual build steps pass the halfway mark without a single look attempt, the loop appends a reminder.
look calls are capped per ScalePlan tier.
| ScalePlan | look limit | shared recon budget |
|---|---|---|
| S | 2 | 4 |
| M | 2 | 6 |
| L | 3 | 8 |
| XL | 4 | 10 |
look shares a reconnaissance budget with search_library, get_entry, and inspect. Once the recon budget is exhausted, further look calls consume build steps instead. The model can therefore divide its budget between scouting and building as needed.
The tool rejects look calls when the scene is empty. When rendering or the vision model call fails, the tool returns an explicit failure observation. The attempt still counts against the look budget, and the Agent can continue building.
For an S or M tier build, a full building-plus-review cycle produces at most 3 visual calls and 12 images: 2 look calls plus 1 critique. This assumes the look budget is fully used, the user triggers one review, and lint checks pass.
Finished-Build Review and Revision#
After building is complete, the user can click the review button on the version card to start the critique flow.
The review endpoint first replays the build program and runs category-aware lint. If lint raises warn-level issues, a structural fix list is generated directly, with zero vision model calls. Visual review proceeds only when all lint results are at info level. This allows structural problems to be addressed before vision tokens are spent.
If structural checks pass but the vision model is unavailable, the endpoint returns vlmSkipped: true with zero usage, explicitly noting that visual review was skipped.
The visual review receives the user’s intent, a structural summary, editable group ids, category criteria, and target dimensions. The vision model returns a score from 0 to 100 with an accept or revise verdict, along with revision suggestions containing target, issue, and fix fields. These suggestions can be forwarded to the Agent to modify the existing build program.
75 is the acceptance threshold in the review prompt and the fallback for deriving a verdict when the model omits one. The parser prioritizes an explicit verdict from the model. Scores are clamped to 0–100 and rounded to integers. If the model omits a score, accept defaults to 85 and revise to 55.
The UI allows up to two rounds of review-driven revision by default.
A manual spot-check examined review results for a clock tower, a dragon, and a courtyard. These three builds produced 7 findings in total, all three with accept verdicts and scores between 82 and 85. Two clock tower findings were confirmed by comparing against the south elevation screenshot: the stepped tower top and the white panel on the front face both matched the image. These results show the visual review can identify visible defects, but defect recall and post-revision quality improvement have not been systematically measured.
Four-View Camera Coverage#
During generation testing, I found that an AI-built cat had a face on both the front and the back. The original build program was not preserved, and z-axis mirroring causing a duplicate face remains a hypothesis. The investigation did confirm a problem with camera coverage: the old four-view setup placed all cameras on the south side of, or above, the model center. Three ground-level cameras shot from the southeast, southwest, and due south. None could observe the north face. The vision model could not see the north face, so it could not flag problems there.
The fix changed view index 1 from southwest aerial to northwest aerial. After the change, all five directions meet the coverage condition: east, west, south, north, and top.
The coverage condition requires at least one camera’s center-to-camera unit vector to have a dot product of at least 0.3 with the target face normal. The 0.3 threshold ensures the camera is not viewing the face at an extreme grazing angle. The top-down camera has a z-offset of 0.01, a tiny value that makes it pass only the top face check rather than counting as south coverage due to floating-point precision.
| Face | Old view count | Current view count |
|---|---|---|
| East | 1 | 1 |
| West | 1 | 1 |
| South | 3 | 2 |
| North | 0 | 1 |
| Top | 3 | 3 |
| Bottom | 0 | 0 |
The north face is now covered by the northwest aerial. The bottom face is intentionally omitted, since voxel buildings typically have no observable features on their underside.
Tests verify coverage in two ways. Geometric tests check each standard view’s normal coverage against the 0.3 threshold, including regression cases for the old incorrect poses. A pixel probe test places a red block on the north side of a stone box and checks for red-dominant pixels in the northwest aerial view. When the red block is buried inside a sealed box, no view should show red pixels, verifying that occlusion works correctly.
The fix still uses four images, with zero increase in image count or call budget.
Unified Framing#
The soft renderer and browser capture must use the same camera poses. With different coordinates, the browser and the vision model would see the build from different angles, making review results harder to check against the browser view.
Camera positions, labels, framing distances, and frame ratios are centralized in the geometry module poses.ts. Both the server-side soft renderer and the browser capture import it. poses.ts avoids Node-specific dependencies like node:zlib, so it can be imported directly in the browser. Framing distance is computed from the bounding box’s longest edge: d = 1.6m + 5, where m is that edge’s length.
critique generates its view descriptions from the actual image labels. look uses a generic description prompt, and the transport layer prepends the actual view label before each image. Both paths read labels from poses.ts, keeping text descriptions aligned with image content.
Unified framing also fixed an earlier problem with inconsistent screenshot sizes. Before captureSize was introduced, the browser screenshot height depended on the current workspace layout. The same build produced 640×1252 in one window configuration and 640×446 in another, a 2.8× difference in pixel count. The current captureSize computes height from width at the fixed 16:10 ratio, making capture dimensions independent of workspace layout.
Cover Image Framing#
Cover screenshots share the same capture entry point as visual reviews, but the framing varies by trigger.
| Trigger | Camera source | Typical use |
|---|---|---|
| Default | Standard view index 0 (SE aerial) | Most buildings |
| Door-facing vote | Side of the bounding box containing doors; ties resolved in order of south, east, north, then west | Buildings with doors |
| Current angle | Preserves the user’s camera position and orientation | User-selected front face |
When the user selects the current angle for the cover, the screenshot retains the camera position and orientation and adjusts horizontal framing to the unified frame. This lets users choose a suitable display angle for sculptures, animals, and other builds with no fixed front face.