Technology
Evaluation
GenomeBench is the test harness used to measure the genome. It is a simulation, and this page says what that means.
Read this first: every number here comes from a simulation
- The customers are six made-up personas. No real customer data was used.
- The “customer” that accepts or rejects an output is a rule-based checker, not a person.
- The comparison is against lexical top-3 retrieval (word overlap). It is not retrieval with embeddings.
- The no-memory arm scores 0% because of how the checker is built.
- Thirty units are not thirty independent samples.
- The author designed both the system and the benchmark.
What GenomeBench is
GenomeBench is an offline harness. It asks one question: does an agent need fewer customer corrections when it has a genome than when it has no memory, or a simple memory? It answers in a simulation, so it is cheap to repeat and fully reproducible. It cannot say anything about how real customers would feel.
The customers and the checker
There are six synthetic personas, which are simulated small-business customers: a chartered accountancy firm, an HR team, a D2C handloom brand,
a school office, a dental clinic and a textile trader. Each has 3 to 5 preferences that a program can check,
drawn from seven kinds: a maximum word count, bullet points, Hindi output, a required label (such as
GSTIN:), no greeting, banned words, and an exact sign-off line. Each persona has 10 task prompts.
Every preference applies to every task of that persona.
The simulated customer is a checker. It accepts an output only if it breaks none of the persona’s preferences. If the output is rejected, the feedback names only the first broken preference, in the customer’s words. It states a preference, never a fact.
The steps in words
- Pick a persona and a seed. The seed only changes the order of the persona’s tasks.
- The arm builds the prompt. Each arm adds its own memory to the same base prompt.
- The model writes an output. A live model is called with temperature 0.2 and a 1,024-token output limit.
- The checker reads it. The rule-based customer lists any broken preferences.
- Accept, or reject with feedback. No broken preference means accepted. Otherwise the first broken one is sent back as feedback.
- The arm updates its memory. The arm is told the verdict and any feedback. Memory arms may also work between sessions.
This repeats for 3 tasks in a session and 10 sessions: 30 outputs for each arm and persona.
The three arms
| Arm | What memory it has |
|---|---|
plain | None. |
retrieval | It stores every feedback text. For each task it adds the 3 stored texts that share the most words with the task (ties go to the most recent; identical texts count once). This is lexical retrieval, not embedding-based retrieval. |
genome | The genome described on this site, as used in the benchmark: an in-memory log, the benchmark’s own consolidation between sessions, the real fold and compiler, and removal of the model’s report block. |
All three get the same base prompt, the same persona and the same task order for a given seed. Two more arms
exist in the harness (memory_list, which puts every feedback text in the prompt, and a second genome
version). They were not part of any reported run.
The protocol
- 10 sessions of 3 tasks give 30 outputs for each arm and persona. The 10 tasks cycle three times.
- 5 seeds (7, 11, 13, 17 and 19) × 6 personas = 30 units per run set. That is 2,700 model calls per set.
- First-pass acceptance is the share of a unit’s 30 outputs that were accepted as written. It is averaged over all 10 sessions, so it includes the early sessions before any memory has helped.
- Corrections are the number of rejected outputs among the 30.
- Intervals are 95% percentile-bootstrap intervals of the mean over the 30 units. Differences are paired per unit. A second analysis resamples whole personas, which gives wider intervals. See Research.
The three live run sets are called A, B and C. They differ in code revision and sampling, and set B’s model is stated by the person who ran it. They are reported separately and never pooled.
Headline results
Measured M01 First-pass acceptance by arm, and the paired difference between the genome and lexical retrieval. The no-memory arm is 0% in every unit of every set.
| Figure | Value | n | 95% interval | Method |
|---|---|---|---|---|
| Set A, lexical top-3 retrieval: first-pass acceptance | 49.12% | 30 units | 35.67 to 62.68 | 95% percentile bootstrap of the mean over 30 units (4,000 resamples) |
| Set A, genome: first-pass acceptance | 81.79% | 30 units | 80.12 to 83.35 | 95% percentile bootstrap of the mean over 30 units (4,000 resamples) |
| Set A, genome minus retrieval: first-pass acceptance | +32.67 points | 30 units | 19.22 to 46.22 | 95% percentile bootstrap, paired per unit (4,000 resamples) |
| Set B, lexical top-3 retrieval: first-pass acceptance | 46.67% | 30 units | 35.22 to 58.01 | 95% percentile bootstrap of the mean over 30 units (4,000 resamples) |
| Set B, genome: first-pass acceptance | 74.22% | 30 units | 68.67 to 79.45 | 95% percentile bootstrap of the mean over 30 units (4,000 resamples) |
| Set B, genome minus retrieval: first-pass acceptance | +27.55 points | 30 units | 16.99 to 38.89 | 95% percentile bootstrap, paired per unit (4,000 resamples) |
| Set C, lexical top-3 retrieval: first-pass acceptance | 49.45% | 30 units | 35.34 to 63.34 | 95% percentile bootstrap of the mean over 30 units (4,000 resamples) |
| Set C, genome: first-pass acceptance | 82.00% | 30 units | 79.89 to 84.01 | 95% percentile bootstrap of the mean over 30 units (4,000 resamples) |
| Set C, genome minus retrieval: first-pass acceptance | +32.55 points | 30 units | 18.88 to 45.89 | 95% percentile bootstrap, paired per unit (4,000 resamples) |
Source: tools/genome_bench/baselines/phase1-live-2026-10-13/aggregate.json (IntelliNix repository); docs/benchmark_results_5seeds/aggregate.json (IntelliNix repository, untracked folder); tools/genome_bench/fixtures/replay/20261024_aab93e66/ and .kiro/specs/agent-genome-2/bench-log.md (IntelliNix repository).
In this simulation, the genome arm was accepted more often than the retrieval arm in all three sets. The advantage is not uniform. Retrieval did better than the genome for some personas (for example the HR team and the school office in sets A and C), and retrieval never produced an accepted output for the handloom brand in any seed of any set. Counting the 30 units of each set, the genome scored above, equal to and below retrieval in 17, 3 and 10 units (set A); 21, 5 and 4 units (set B); 16, 5 and 9 units (set C). A plausible explanation for the brand is that this persona has more preferences than the 3 texts retrieval can carry. That explanation has not been tested.
Intervals that resample whole personas are about twice as wide for the paired difference, and their lower ends stay above zero. The direction of the difference holds across the three sets. Its size is poorly pinned down. The persona-clustered intervals and the model and code revision of each set are on the Research page.
Why the no-memory arm scores 0%
The base prompt states none of a persona’s preferences. A model that has no memory cannot guess an exact label
such as GSTIN: or an exact sign-off line, so it fails at least one rule every time. The 0% is a
property of the checker and the persona design. It is a floor. It is not an empirical finding about the model,
and it says nothing about effect size beyond “memory is needed in this benchmark”.
What the results do not show
- Anything about real customers. The checker verifies format-like rules. It cannot judge tone, helpfulness or correctness.
- Anything about stronger retrieval. There is no embedding retrieval, no larger k, no reranking and no comparison with published memory systems.
- Which part of the genome does the work. There is no ablation that separates consolidation, strength and the prompt scaffold. About 1,100 characters of the genome’s prompt are a fixed scaffold, and there is no control for that.
- Whether context-dependent preferences work. Every persona applies all its preferences to every task.
- Whether strength decay, retirement and relearning work. No persona ever changes a preference.
- Whether the governance parts work. Rollback, erasure, tenant isolation and the immune screen are built but not evaluated by the benchmark.
- Anything about other model families. All three sets used Gemini Flash-class models.
- Money cost. Only prompt size and compile time were measured.
Replay, the CI gate and scorecards
- Replay. Run set C stored every model response. Replaying them offline reproduces every verdict: 2,700 calls, 2,700 matches, no verdict changed. Sets A and B cannot be replayed. Measured M08
- CI gate. A quality gate replays the recorded set and fails the build on a regression beyond fixed tolerances. A ratchet lets the baseline only improve unless a reviewed exception is added. Built and tested G79, G80
- Scorecards. The quality engine defines 13 named figures. Each has a value, unit, sample size, 95% interval, method, source and status, and there is no single composite score. This site follows the same rule: a figure without n and an interval is not shown. Built and tested G50 Live scorecards for running agents are Built, flag off by default G52 .
- The harness. The personas, checker, arms, runner and statistics have code and tests. Built and tested G76