MemPABench: interaction-preference memory
A modified nanobot, memory adapters and an IX interaction wrapper test whether feedback changes how an assistant helps. These are research configurations, not commercial product results.
Jialin Li et al., NYU Abu Dhabi · Published 2026-10-21 · reviewed 2026-10-29
Scope and method
Four simulated users, 14 preferences across work/personal contexts; 99 sessions per user over a simulated four-month timeline. Each user has 33 probes, repeated three times.
PSA checks preference selection; CS requires both contexts correct; PET requires before/after change endpoints correct. Separate judges assess delivered behavior and simulated-user perception.
How PAEval uses it
Memory and personalization research reference; no scores transferred to default nanobot, memory vendors or finished PA products.
Evidence family: mempabench-v1. Shared tasks and tracks are not independent evidence votes.
Preference outcomes (%)
Paper aggregates across backbones and users; PSA, CS and PET measure different constructs. Reference conditions are not products.
| Memory condition | PSA | CS | PET |
|---|---|---|---|
| File memory | 59.2 | 14.4 | 25.4 |
| Simple RAG | 49.9 | 20.8 | 15.0 |
| Mem0 | 50.0 | 22.2 | 14.2 |
| MemOS | 50.0 | 19.9 | 10.8 |
| No Memory reference | 49.6 | 21.8 | 11.2 |
Limits and interpretation
- IX wrapper shapes delivered behavior and actions; results describe the full configuration.
- Memory conditions have different retrieval budgets and accumulated histories. Four synthetic users limit generalization.
- Simulated months do not establish production uptime. Repeating probes does not rerun the full accumulated life history.
Sources and reuse
PAEval summarizes existing published evidence and runs no new tests. Public readability does not establish permission to redistribute full datasets, conversations or images.
All benchmark sources → · How to read outcomes and intervention →