Research

Summary

Measured M01 to M08 In GenomeBench, an agent with a genome had more of its outputs accepted on the first try than an agent that kept past feedback and pasted the 3 most word-similar items into its prompt. In the most recent run (set C, with every model response stored for replay), the genome arm scored 82.00% against 49.45% for the retrieval arm, a difference of +32.55 percentage points. The 95% interval for that difference over the 30 benchmark units is 18.88 to 45.89. Sets A and B, run earlier, point the same way.

That is the whole claim. It says that in this simulation, with this checker and this baseline, remembering feedback as a structured genome beat a simple word-overlap memory. It does not say the genome beats good retrieval and it does not say that real customers would be happier.

What is simulated

• The customers. Six synthetic personas (invented for the test, not real customers): an accountancy firm, an HR team, a handloom brand, a school office, a dental clinic and a textile trader. Each has 3 to 5 preferences that a program can check, such as a maximum word count, a required label or an exact sign-off line.
• The judge. A rule-based checker accepts an output only if it breaks none of the persona’s preferences. When it rejects, it sends back the first broken preference as feedback.
• The baseline. The retrieval arm stores every feedback text and adds the 3 stored texts that share the most words with the task. This is lexical retrieval.
• The model. The model calls are real calls to a hosted model. Everything around them is simulated.

A plain explanation of each part is on Technology: Evaluation.

Method

• One unit is one seed and one persona. A run set has 5 seeds (7, 11, 13, 17, 19) and 6 personas, so n = 30 units.
• Each unit runs 10 sessions of 3 tasks, which gives 30 outputs per arm. A run set makes 2,700 model calls.
• First-pass acceptance is the share of a unit’s outputs accepted as written, averaged over all 10 sessions. It includes the early sessions before any memory could help.
• Intervals are 95% percentile-bootstrap intervals over the 30 units, with 4,000 resamples. Differences between arms are paired per unit.
• The three run sets are reported separately and are never pooled.

The three run sets

Set C is the one to look at first, because its responses are stored and its verdicts can be reproduced. Sets A and B cannot be replayed. Set B’s model name rests on the word of the person who ran it. All three are Gemini Flash-class models, so nothing here applies to other model families.

Results

First-pass acceptance by arm, and the paired difference between the genome and lexical retrieval. “Points” means percentage points. n is the number of units.

The result files named under the table are in the IntelliNix repository. That repository is not public, so you cannot inspect them from this site yet. Treat the figures as the authors’ report until the files and the benchmark code are published.

A more cautious interval

The 30 units are not 30 independent samples. The 5 seeds of one persona are related, because the persona’s preferences and checker are the same. A fairer interval resamples whole personas, which leaves only 6 independent groups. That interval is much wider.

The direction of the difference holds in all three sets, since the lower end of each interval stays above zero. The size of the difference is poorly pinned down. With 6 personas, even this interval is a rough guide.

Where the genome did not win

The advantage is not uniform. In some personas, such as the HR team and the school office in sets A and C, lexical retrieval did better than the genome. The count of units where the genome was above, equal to and below retrieval is in the table of run sets. For the handloom brand, retrieval never produced an accepted output in any seed of any set. A plausible reason is that this persona has more preferences than the 3 retrieved texts can carry. That explanation has not been tested.

What this does not show

• Real customers. A rule checker verifies format-like rules. It cannot judge tone, usefulness or correctness. No real user has used the genome in a measured study.
• Strong retrieval. There is no embedding retrieval, no larger k, no reranking and no comparison with published memory systems. A better baseline could close the gap.
• The cause. No ablation separates consolidation, strength and the fixed reading-rules scaffold. The no-memory arm scores 0% by construction, because it cannot know a persona’s exact rules, so it says nothing about effect size.
• Changing preferences. No persona ever changes a preference, so strength decay, retirement and relearning were never exercised in a measured result.
• Governance. Rollback, erasure, tenant isolation and the immune screen are built and unit-tested, but the benchmark does not evaluate them.
• Other models. All three sets used Gemini Flash-class models. The benchmark can call only that client.
• Independence. The author built both the system and the benchmark, and the paper has not been peer reviewed.
• Packs. None of the packs in the catalog has been measured. Their pages say “Not measured”.

The Future page lists the studies that would address these limits. None of them has been run.

Replay and re-running

Measured M08 Set C stored all 2,700 model responses. Replaying them offline reproduces every verdict, so the scoring can be checked without calling a model again. Replay checks the scoring. It does not show that a new live run would give the same numbers, because a hosted model can change and sampling at temperature 0.2 is not seeded.

Built and tested G76 to G80 The harness, the record-and-replay step and a CI gate that fails on a regression have code and tests. A new live run on another model costs about 2,700 paid model calls. It would be reported as a new, labelled set and never pooled with A, B or C.

Related work

Learning preferences from feedback and storing them as text in a prompt is not new. The genome combines known ideas (memory, reflection, retrieval, learning from edits) with governance around them: layers, review, rollback and erasure. These papers are starting points. Links go to arXiv, an outside site.

• Packer et al. (2023). MemGPT: Towards LLMs as Operating Systems. arXiv:2310.08560. Memory management for long-running agents.
• Shinn et al. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. arXiv:2303.11366. Agents that keep written lessons from earlier attempts.
• Park et al. (2023). Generative Agents: Interactive Simulacra of Human Behavior. arXiv:2304.03442. Memory streams and reflection.
• Lewis et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv:2005.11401. The retrieval idea that the lexical baseline is a simple version of.
• Gao et al. (2024). Aligning LLM Agents by Learning Latent Preference from User Edits. arXiv:2404.15269. Learning preferences from user edits, without changing model weights.
• Salemi et al. (2024). LaMP: When Large Language Models Meet Personalization. arXiv:2304.11406. A benchmark for personalised language models.
• Chhikara et al. (2025). Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory. arXiv:2504.19413. A published agent memory system. It is not compared against here.

Statistics: Efron (1979), Bootstrap Methods: Another Look at the Jackknife, The Annals of Statistics.

The research paper has not been peer reviewed. Return to the technology overview or read the docs.