Yuanhao Feng

Back

The second article introduced v3’s tool loop. Among the agent’s eight tools, two are for searching a reference library: search_library finds candidates, and get_entry reads their details. This article covers the retrieval mechanism behind those tools: 800 entries, five scoring layers, and why each layer exists.

Two-Step Retrieval#

The agent can query a reference library during a build. The library has two types of entries. A Part is a placeable voxel component, such as a window, tree, or windmill tower. Its details include instructions for placing it with place_component. A Ref is a whole-scene reference card that provides volume, palette, and technique information for design guidance. The current library holds 800 entries: 737 voxel stamps, 11 parametric components, 42 whole-scene reference cards, and 10 style cards.

Retrieval is a two-step process. The agent first calls search_library, which scores and ranks candidates, then returns two summary lists: up to 6 Parts and up to 2 Refs. Each result includes an entry ID, an evidence type, and a one-line summary. The agent then decides which entries are worth reading in full and calls get_entry on those. Style cards are injected separately with the build intent and do not go through retrieval ranking.

User intent search_library 5-layer scorer Part ≤6 id · evidence · summary get_entry Place component Ref ≤2 id · evidence · summary get_entry Reference design Style card Injected with build intent

With the library available, the model can search for and place existing components instead of describing known structures block by block.

The Five-Layer Scorer#

A single search produces a total score from five signals.

Query 800-entry pool Signal Algorithm Weight ID match Full or space-separated ID +3 Keywords Derived terms, each hit +2 / term BM25 body IDF-weighted · normalized 0 ~ 2 Condition Category +2 · Style +3 · Generic +1 variable Semantic cosine cos ≥ 0.32 · normalized 0 ~ 3 3I + 2K + 2B/B_max + cond + 3C/C_max Part ≤6 Ref ≤2 On zero hits: condition-score fallback (fallback)

ID match. If the query contains an entry’s full ID or its space-separated form, that entry gets +3. Querying windmill tower matches windmill-tower exactly.

Keywords. Each entry carries a set of keywords derived from its title, tokenized title, tags, and curated fields. The query is checked against these keywords; each hit adds +2, with no cap. English matching uses word boundaries and is case-insensitive.

BM25 body. The query and each entry’s indexed text (summary + body) are scored with BM25. BM25 uses document frequency to distinguish rare terms from common ones, applies term-frequency saturation, and normalizes for document length. The raw BM25 score is normalized to a 0–2 range against the highest score in the current candidate set. Why BM25 replaced the earlier substring matching is covered later.

Condition. Matching the query’s category adds +2, matching its style adds +3, and generic entries with no category get +1. Condition scores participate in ranking existing hits and in the zero-hit fallback.

Semantic cosine. Queries and entries are mapped into the same embedding space (1024 dimensions). After L2 normalization, cosine similarity is computed via dot product. Candidates below a 0.32 threshold are excluded from semantic scoring. Those above are normalized against the set’s maximum cosine, capped at 3. The design and experimental results for the semantic layer are covered in a later section.

The final weighted formula:

3I + 2K + 2B/Bmax + cond + 3C/Cmax
plaintext

When a denominator is zero, its term drops out. Ties are broken by entry type and ID.

Results carry one of four evidence labels: lexical hit, semantic hit, both, or condition fallback. Semantic-only parts prompt the agent to verify the component’s identity before placing it. Condition fallback results state explicitly that there is no direct query match. The Part and Ref lists are ranked independently but share a single query embedding and BM25 index. A list falls back to entries with positive condition scores only when it has no query-matched candidates at all.

Evaluation Set and Metrics#

The evaluation suite uses 50 hand-labeled queries divided into six mutually exclusive buckets.

direct 8 queries / 8 scored Name queries lexical 11 queries / 11 scored Body text vocabulary semantic 10 queries / 10 scored Phrasing differs from entries zh 6 queries / 6 scored Chinese queries conditioned 7 queries / 6 scored Category · style conditions negative 8 queries / 2 scored Banned results · empty set 50 queries 43 scored production: same 50 queries + production conditions (part fallback 3, ref fallback 8)

BucketQueriesScoredTarget
direct88Name queries
lexical1111Body text vocabulary
semantic1010Phrasing differs from entries
zh66Chinese queries
conditioned76Category and style conditions
negative82Banned results and empty sets

Of the 50 queries, 43 have relevance sets and contribute to aggregate metrics. hit@6 is the fraction of scored queries whose top-6 results contain at least one labeled relevant entry. MRR is the mean reciprocal rank of the first relevant result across the 43 scored queries, with misses counted as zero. Normalized recall uses the smaller of the relevance set size and the actual return count as the denominator.

The production view reruns the same 50 queries with production-environment category and style conditions, keeping the original relevance labels. This view covers condition-fallback behavior: 3 Part fallbacks and 8 Ref fallbacks across the 50 queries.

The suite also includes seven degraded retrievers: ignore-query, id-only, naive-tokenizer, parts-only, semantic-only, constant-vector, and compound-split. Each one verifies that the evaluation can detect the corresponding deficiency. The suite runs offline, reusing saved query vectors with no embedding-model calls.

From Substring Matching to BM25#

The original body-text layer checked each query term as a substring, added +1 per hit, and capped the total at 2. The problem was that common terms created large pools of tied candidates. The library has 232 entries whose indexed text contains glass and 167 containing window. Searching stained glass windows returned no labeled relevant entry in the top six.

BM25 breaks ties by distinguishing term rarity. It computes IDF across the full library, giving rare terms more weight than common ones. Term frequency saturates: ten occurrences of a term contribute only marginally more than three. Document length normalization prevents long texts from scoring high on sheer volume.

After switching to BM25, enchanting-pavilion, a relevant entry for stained glass windows, ranked first. The lexical bucket’s MRR rose from 0.700 to 0.894.

There was a trade-off. The direct bucket kept hit@6 at 100%, but MRR dropped from 1.000 to 0.838. BM25’s document-frequency weighting rearranged the ranking among already-matched entries. Recall rose from 89.6% to 91.7%, meaning more relevant entries appeared in the returned set overall.

The keyword layer also went through cleanup. The query a place of worship was once polluted by a single-character a produced by title tokenization, sending all six slots to irrelevant entries. Filtering single-character title fragments fixed this. A reference card and a stamp once shared the ID windmill, causing one entry to hide the other when fetching details. The reference card was renamed to windmill-farmstead, and the assembly step now merges on globally unique IDs.

Semantic Layer and Cosine Threshold#

BM25 depends on text overlap. When a query and an entry share no common terms, lexical retrieval misses entirely. a way to get across a river shares no word stems with diagonal-azalea-wood-bridge. Neither does vessel that travels under the sea with submarine.

The semantic layer maps queries and entries into the same embedding space. The model is OpenAI text-embedding-3-large, producing 1024-dimensional vectors. After L2 normalization, cosine similarity is computed via dot product, with a threshold at 0.32.

QueryRelevant entryNo vectorsWith vectorsCosineTypea way to get across a riverdiagonal-azalea-wood-bridge—20.326Recoverya place of worshipabbey—10.382Recoveryvessel that travels under the seasubmarine—20.476Recovery风车spartan-windmill—10.394Recoveryheavy machine for diggingexcavator510.508Rank liftThreshold boundary: F1 roller coaster ↔ ferris-wheel cos=0.398, D5 风车 ↔ windmill cos=0.394Raising the threshold to 0.40 would exclude both candidates

Four queries that went from no hit to a hit:

QueryRelevant entryCosineRank with vectors
a way to get across a riverdiagonal-azalea-wood-bridge0.3262
a place of worshipabbey0.3821
vessel that travels under the seasubmarine0.4762
风车spartan-windmill0.3941

heavy machine for digging already hit excavator without vectors, but at rank 5. Adding vectors pushed it to rank 1 with a cosine of 0.508.

The 0.32 threshold balances coverage against precision. roller coaster and colorful-ferris-wheel have a cosine of 0.398. 风车 (windmill in Chinese) and spartan-windmill have 0.394. Raising the threshold to 0.40 would exclude both.

For fusing lexical and semantic scores, weighted summation was chosen over RRF (reciprocal rank fusion). In the same experiment, RRF achieved a mixed MRR of 0.771 versus 0.760 for weighting, but the conditioned bucket’s hit@6 dropped from 100% to 83.3%. Condition scores add directly in the weighted scheme; RRF’s rank-based fusion diluted them.

After first adding vectors, the semantic bucket’s hit@6 rose from 30% to 60%, and mixed overall from 76.7% to 88.4%. A later adjustment from semantic weight 2 to 3 brought the semantic bucket to 70% and mixed overall to 90.7%.

0%20%40%60%80%100%hit@690.9%100.0%+9.1ppBM25 upgradelexical bucketMRR 0.700 → 0.89430.0%60.0%+30.0ppSemantic vectorssemantic bucketMRR 0.103 → 0.45088.4%90.7%+2.3ppWeight 2 → 3mixed overallMRR 0.771 → 0.806BeforeAfter

Results with semantic-only evidence are labeled accordingly. Semantic-matched parts prompt the agent to verify identity before placement. Reference cards are labeled as related inspiration. When Chinese queries fail the lexical layer, the system attaches a recovery prompt as long as the Part results are still useful.

Separate Part and Ref Lists#

Early on, the library had 137 Parts and 42 Refs, all ranked together. Refs made up 40.8% of returned results. As Parts grew to 748, Ref share fell to 14.1%. With Parts outnumbering Refs nearly 18 to 1, mixed ranking diluted the reference cards.

0%10%20%30%40%50%200400600800Part countRef share40.8%137 parts16.7%437 parts14.1%748 partsCurrent: Part ≤6 / Ref ≤2 separate lists

A guaranteed-slot scheme was tried. But guarantees have a cost: for the query heavy machine for digging, all relevant parts were pushed out of the top six when 2 Ref slots were reserved. That version later reduced the guarantee to 1, but the root problem was mixed ranking itself.

The current implementation uses separate lists. Parts return up to 6 results, Refs up to 2, each ranked independently. Results from the 50 evaluation queries under the current setup:

ViewScoredhit@6MRR
Mixed-compatible top-64390.7%0.806
Production Part@64292.9%0.819
Production Ref@25100%1.000

All 5 scored Ref queries hit. Part MRR under production conditions is slightly higher than the mixed-compatible view, since separate lists remove cross-type ranking interference.

Vector Storage and Compute Scale#

The vector table is stored as server-side JSON: 800 entries × 1024 dimensions, 4.2 MiB on disk. The raw float payload is 800 × 1024 × 4 = 3.3 MB; the rest is base64 encoding overhead and metadata. On load, the system checks the model name, declared dimensions, actual dimensions, and text fingerprint. Entries that fail validation keep lexical retrieval; the loader reports the reason.

Each full-library cosine computation is 819,200 element-wise multiply-adds. At 800 entries, computing scores for the full library in-process is practical. The ADR lists conditions for reconsidering a database: corpus reaching the 100,000-entry range, user-facing search, cross-instance sharing, or per-user filtering. The current implementation avoids that additional infrastructure until it is needed.

Within a single build session, query embeddings are cached for reuse. The evaluation runs offline with saved query vectors and makes no embedding-model calls. New models or new queries require rebuilding the vector fixture; existing scores correspond only to the saved vector space.

A follow-up keyword experiment capped keyword hits at 2, which lowered mixed MRR from 0.806 to 0.783. Dropping keywords with document frequency above 40 showed no benefit on scored queries. The final rule remains +2 per hit, uncapped.

The current evaluation uses 800 valid entries and vectors. Across the 43 scored queries in the 50-query suite, mixed hit@6 is 90.7% and MRR is 0.806. End-to-end build quality and token savings lack independent controls. A full ablation matrix adding the five signals one by one has not been documented; the existing evidence covers BM25, semantic, and weight comparisons in separate before-and-after comparisons, plus per-query case analysis.

From Keyword Scoring to Semantic Vectors
https://me.yuanhaofeng.com/en/blog/cubemuse-hybrid-retrieval
Author Yuanhao Feng
Published at September 25, 2026