What each source actually measures

Latest sample denominators differ; Town withheld; Muse adapter corrected. Historical common-set filters remove many non-arrivals, with material sensitivity. Source itself says batches are not publicly auditable.

Products have different scored baskets; test and observed labels do not prove matched conditions/repetitions. AB mean is a source rating, not PAEval’s global measure.

Yoom sells automation and promotes itself in the article. Limited task-based review, not a representative battery; do not convert prose to invented numeric scores.

Google Slides required per-slide prompts while other products got whole-deck prompts. Useful workflow-friction evidence but unequal procedures; one brief is not robust task success data.

Historical qualitative evidence on travel/research/files, download failures and instability; casual ChatGPT comparison is not matched agent-mode testing.

Partial accessible evidence, old version and unverified total sample; reuse of prior prompt is not simultaneous matched conditions.

Do not count mirror reports as independent tests. Historical task examples; cannot infer success rate from selected examples.

ChatGPT here is not today’s agent mode/dots; no repeated runs, no blinded raters, primarily response-quality tasks.

Self-selected single-user tasks, assistance and failures/cancellations distinct; 17 interventions overlap outcomes. Not comparable to ten fixed editorial tasks or an automated harness.

Author-specific task/rubric selection; 8.8 task mean is different from 9.2 overall opinion. Longitudinal scheduling, learning, some controls and Teams were not tested. Do not map to entire brand or all modes.

Four unscored items include permission denial/undecided/not-run and are not four failures. Same author as dots does not establish same tasks, date, permissions or complete basket.

N=1. Eight AI coding tools are the objects researched, not eight tool calls or independent trials. Old Max billing cannot be applied to current Pro or Personal Computer.

Do not silently pool with same-author dots/Muse: dates, accounts, task definitions and rubric interpretation must match first. Private inputs prevent full public replication.

Public-document annotation, not experimental performance. Independent annotation does not make vendor capability claims an independently tested fact. Important 2026 products and updates are outside the snapshot.

User satisfaction and adoption experience; self-selection and category mismatch. Missing price/features cannot be zero; one review has no stable population inference.

Review scores measure sentiment, not autonomous task success; app/brand-level reviews may not isolate agent mode.

Vendor assesses itself; ChatGPT described as non-autonomous, so cannot represent current agent/dots modes. Score formula includes vendor-specific judgments; unpublished per-case values prevent reproducing multiplicative aggregate from marginal means.

Only Atlas full proof-of-concept demonstrated; several other products meet preconditions, which is not the same as a successful exploit. Historical versions; no incident denominator or overall safety score.

Root data/ is the legacy board, not the current-judge board. Current-judge CSV has missing FACT values; never fill them from legacy. Product modes are not whole products; Sonar and API harnesses are not Perplexity consumer-app scores.

Same broad research family as DRB I; not an independent lab vote. Different task set, rubric and judge versions are not interchangeable; report-generation version still needed.

Manus app, OpenAI o3/o4-mini research APIs, Sonar, and custom agents cannot be silently mapped to current PA brands. Publisher also evaluates its own Salesforce AIR system.

Model+harness rows; runtime estimated from ReAct steps, not elapsed wall time. Must not map Claude row to Claude consumer PA.

Same named memory library inside two agents does not make PA performance equal; add/search/answer contract fixes components that real products vary.

It is mainly based on information answers and cannot prove that the account operation is completed; it is a different project from assistantbenchmark.com.

Scored primarily for final answer; does not equate to action completion, security authorization, or long-term reliability in Real services.

Stateful unit testing can verify completion and incidental changes, but simulated API performance is not a substitute for real website login, GUI, or payment process performance.

The customer service scenario cannot directly represent a complete personal assistant. banking_knowledge is not comparable before and after v1.0.1; pass^k and pass@k have different meanings.

Scores are affected by user simulator and milestone design; trajectory similarity is not equal to binary success rate for all tasks.

Focuses on hard-to-find short answers and does not cover open-ended reporting quality, practical action completion, or the full personal assistant experience.

Memorizing questions and answers accurately does not mean using memory correctly in action. S, M, and oracle are configurations and cannot be summed up as three batches of independent problems; V2 should be listed separately.

Comparison of the impact of operating system, observation interface, step budget, and task exclusion; the results of original, Verified, and 2.0 cannot be directly merged.

800 refers to standard verification scenarios, with an additional 320 augmented scenario configurations; simulation times and self-reported runs do not demonstrate true multi-week reliability.

Denying all tasks cannot be regarded as a good security assistant; normal effectiveness and attack success rate need to be considered at the same time, and it cannot represent all permissions and privacy risks.

The current report focuses on a mail and calendar simulation environment and runs up to 75 steps; complete pass rates and average rewards including partial completions cannot be mixed. Different from micro1 PersonalAgentBench.