Braintrust: personal shopping assistants
Three finished assistants receive 25 shopping requests in a fixed WebShop catalog and on the live web. Product fit, answer quality and condition matching are separate outcomes.
Braintrust · Published 2026-10-12 · reviewed 2026-10-29
Scope and method
25 requests × 3 products × 2 environments; 150 planned product–request–environment cells, not 150 independent tasks.
Automatic condition checks plus one blinded reviewer ranking products and answers, allowing ties. Tests recommendations, not purchases or long-term personalization.
How PAEval uses it
Shopping-specific evidence; excluded from the general scoreboards.
Evidence family: braintrust-shopping-2026-10. Shared tasks and tracks are not independent evidence votes.
Fixed catalog
Condition matching is the share of predicates met among scored recommendations; mean product-fit rank is lower-is-better. Neither is a task success rate.
| Product | Condition match | Scored label | Product-fit mean rank ↓ | No recommendation /25 |
|---|---|---|---|---|
| Muse | 94.2% | 17/23 | 1.28 | 6 |
| Instinct | 91.7% | 22/23 | 1.04 | 1 |
| Dots | 88.3% | 22/23 | 1.00 | 1 |
Live web
All 25 requests enter the human ranking. Automatic scoring has missing-output constraints; no transaction occurred.
| Product | Condition match | Product-fit rank ↓ | Answer-quality rank ↓ | No recommendation /25 |
|---|---|---|---|---|
| Muse | 94.7% | 1.52 | 2.24 | 5 |
| Instinct | 88.3% | 1.40 | 1.44 | 6 |
| Dots | 82.1% | 1.28 | 2.04 | 5 |
Limits and interpretation
- Versions, plans and retry rules are incompletely disclosed; one human reviewer.
- Fixed-catalog automatic scores label 17/23, 22/23 and 22/23 scored despite 25 planned requests. The 23-item subset is unexplained; preserve both denominators.
- Live-web automatic checks use saved answers, not a fresh merchant verification. Missing recommendations and unscored outputs remain visible.
- Braintrust provides evaluation tooling and used its own platform. Full experiment redistribution permission is not established.
Sources and reuse
PAEval summarizes existing published evidence and runs no new tests. Public readability does not establish permission to redistribute full datasets, conversations or images.
Related product records
All benchmark sources → · How to read outcomes and intervention →