12 October 2026

Muse, Instinct, and Dots as personal shopping agents

Jess Wang·11 min

AI agents are offering to run our lives. We've all seen those Instagram ads telling us to use Meta's Muse to analyze our workouts, review our bills, book our next vacation, and even manage our calendars and inboxes. Similarly, Instinct is marketed as a personal AI assistant, and OpenAI's Dots is a ChatGPT-based assistant.

When we ask "which AI assistant performs the best?", it's tricky to judge because these products are highly simplified and completely black boxes. Unlike using Codex or Claude Code to build our own agents, we have no control over the model, tool policy, search behavior, memory, or product UX.

To compare the three products, we decided to see which one was the best at personal shopping. Shopping is a good way to test several parts of an assistant at once: whether it follows constraints, selects useful sources, makes a clear decision, and explains that decision well.

Methodology

In this eval, we compared 25 identical personal-shopping requests across Muse, Instinct, and Dots. Each assistant received every request twice: once using a fixed WebShop catalog and once on the open web.

This is a deliberately narrow test of one shopping interaction, not a test of personalized context such as email, cards, purchase history, or long-term memory.[1]

We used a 25-row dataset of LLM-generated shopping scenarios. The requests included preferences around budget, product type, ingredients, ratings, review counts, and aesthetics. Examples include:

• “I want to buy a shampoo or conditioner for damaged hair. It must be sulfate-free, paraben-free, contain hyaluronic acid and coconut oil, and support hair growth and hair-loss care. Keep it at or below $40.”
• “I want black or very dark pants that make my butt look good but are still comfortable enough to sit in all day. No leggings. I want them to look intentional rather than like gym clothes, and I would rather stay under $70.”
• “I need an adjustable phone stand for my desk. Only recommend it if it is rated at least 4.9 stars with 50,000 or more reviews and costs $10 or less. If nothing meets every requirement, tell me that rather than recommending the closest option.”
• “I need a small white-elephant gift that feels genuinely funny or delightfully over-the-top, not generic gag-store junk. Bright orange, purple, cartoon prints, or a strange pattern would be great. Keep it under $20.”

This dataset was used across two approaches in our personal shopping eval:

The first approach was built around WebShop, which is a controlled retail environment with a fixed catalog. For each request, we gave the assistant the shopper prompt and a link to a WebShop sandbox. Braintrust logged its searches and product-page opens, letting us compare research behavior in a fixed catalog where price, inventory, and ranking did not change. WebShop is not a substitute for live-web shopping, but it gives us a clean baseline for comparing the assistants across more volatile open-web conditions.

In the second approach, we allowed the assistants to search the live internet instead of restricting them to the WebShop catalog. Since we were not able to trace the browser actions with this approach, we evaluated only the outcome instead: how good were the product recommendation and written response?

How we logged each run

For WebShop, the harness created a Braintrust root trace and gave the assistant a unique store link. Every landing-page visit, search, returned result set, and product-page open was logged by WebShop, and we added them as child spans in Braintrust.[2] For open-web runs, we recorded the prompt, visible final response, and product link to Braintrust, but we could not record browsing actions on third-party sites like Walmart or Amazon.

For both approaches, we used two types of scorers:

• Deterministic "requirements-fulfilled" score: For each prompt, we used an LLM to translate its requirements into structured predicates. We evaluated the final product on whether it satisfied each predicate. The shampoo and conditioner prompt above had nine true/false checks: product type contains either shampoo or conditioner, price is at or below $40, sulfate-free, paraben-free, hyaluronic acid, coconut oil, damaged hair, hair growth, and hair loss. A product that matched all nine would receive 1.00, and matching eight out of nine would receive 0.89.

• Human review: One reviewer assessed all results in a blinded A/B/C triad, with each assistant hidden and randomized as one of the candidates. The reviewer separately ranked each candidate's product recommendation and written response, and ties were allowed. We unblinded the assistant identities only after all review decisions were complete.

Explore the full experiment logs in Braintrust: Muse, Instinct, and Dots.

Results

Shopping in a fixed catalog (WebShop)

WebShop lets us compare deterministic requirements fulfillment, blinded human product fit, and the logged browsing behavior for Muse, Instinct, and Dots:

There are instances where an assistant did not return a product recommendation because it could not find anything that fit the requirements, so the deterministic scorer did not run on those results. Muse had six such cases, compared with one each for Dots and Instinct. Two patterns explain all five cases where Muse disagreed with Instinct and Dots about whether the WebShop catalog supported a reasonable recommendation; the table highlights one example of each.

In three of the five cases, Muse declined to recommend a product because it did not find an exact match, while Instinct and Dots proposed the closest available option. In the other two, both assistants returned a product that met the objective requirements and that Muse did not select.

This makes the split between our deterministic score and human review score easier to interpret. Muse’s deterministic score includes only traces with a resolvable final product, while the human review treats a no-recommendation outcome as worse than a helpful near-match.

Muse made the most searches of the three assistants, while Dots followed a shorter observed path with the fewest search and product-open events.

When looking at the traces, we saw a pattern. Muse was not failing to discover the candidates that Instinct and Dots found as well, but it was searching more broadly, opening fewer candidate product pages, and declining to commit to a product. The results seem to support the claim that extra searching does not reliably translate into product verification or a more decisive final recommendation.

Shopping on the live web

The controlled WebShop environment records each search and product-page visit, so we can inspect browsing behavior. On the open web, the assistants browse external sites that do not have built-in logging, so we cannot see the browser actions themselves. For the open-web runs, we evaluate final-answer evidence and use human review rather than analyzing search paths.

Muse had the highest stated-fact coverage score in the open-web audit. That score measures how completely the final response documented the objective requirements, not whether a live product page independently satisfied them. However, it is important to note that quite a few of these rows were not graded by the deterministic scorer. For Muse and Dots, five rows did not return a product because the assistant could not find anything that fulfilled the ask. For Instinct, it was six rows. Furthermore, the scorer only graded the information that was included in the final response, which sometimes did not contain every parameter (such as product type, price cap, required attributes, rating, review count, etc.). The number of ungraded responses is likely a reflection of the completeness of the assistant's response. This differs from WebShop, where every product belongs to a fixed sandbox catalog, and the scorer can map the final recommendation to that catalog record. On the open web, live retailer pages can change price, stock, and product copy, so the scorer deliberately uses the saved answer instead of refetching the pages.

The human review adds context that the limited-coverage deterministic audit cannot provide on its own. Using the same product-quality ranking as our WebShop human review, Dots ranked best, followed by Instinct and Muse. We broke the human review scores down by response quality and product quality:

A lower mean rank is better. Instinct wins response quality and Dots wins product quality. To add more color to these results, we decided to analyze the comments for each of these human reviews and group the comments by theme:

Price headroom

Since all of our prompts included a target budget, we wanted to see whether there was a pattern in how close each assistant got to the target budget. For each row, we calculated headroom as the budget minus the final product price, then compared the median headroom for each assistant. Here are the results:

No product recommendation went over the target budget. Muse recommended products closer to the target budget, whereas Dots recommended products further below budget.

Overlap in results

We also wanted to inspect whether there was any pattern of overlap between the responses of the three AI assistants. When two assistants independently recommend the same product, it's evidence that the product was probably the most salient solution in the available choice set, especially when it comes to an open-web search. Of course, agreement does not always mean "right," since the assistants could share popular-but-bad results. It is also interesting to observe whether certain assistants tend to converge with each other more, because it can give us a behavioral fingerprint of the harnesses even though we cannot inspect what is happening internally.

For WebShop, Instinct matched with Muse on eight products and Dots on eight, while Muse and Dots matched only on four, all of which were three-way-consensus cases. This does not prove that Instinct is more correct, but it does show that its final choices were closer to the shared catalog consensus. The open-web overlap is less cleanly clustered than WebShop. There is no obvious pattern of agreement, meaning that the assistants took genuinely different paths in their shopping choices.

Interactive exercise

Review it yourself

Rank the results. 1 is best, 3 is worst, and ties are allowed.

Case 1Case 2Case 3

Shopper request

I want to buy a shampoo or conditioner for damaged hair. It must be sulfate-free, paraben-free, contain hyaluronic acid and coconut oil, and support hair growth and hair-loss care. Keep it at or below $40.

CandidateA

Recommended product

Hyaluronic Acid Moisturizing Conditioner (Luseta Beauty), 16.9 oz ↗

Read the response

Hyaluronic Acid Moisturizing Conditioner (Luseta Beauty), 16.9 oz — $16. Brand says sulfate-free and paraben-free, and the ingredient list includes hyaluronic acid and coconut oil. It targets dryness, brittleness, and detangling. Caveat: its growth and hair-loss claims are marketing, not clinical proof. I’d pick it for damage and breakage care, not as a proven treatment for shedding. It contains fragrance and silicones; price is before shipping and tax.

CandidateB

Recommended product

Hyaluronic Acid Moisturizing Shampoo ↗

Read the response

Hyaluronic Acid Moisturizing Shampoo at $16.00 — sulfate-free and paraben-free, with hyaluronic acid and coconut oil, positioned for dryness and brittleness with hair-growth claims, well under $40. One caveat: those hair-growth benefits are the brand’s claims, not clinically proven. Topical hyaluronic acid and coconut oil are not established hair-regrowth treatments.

CandidateC

Recommended product

Noughty Get Set, Grow Thickening Conditioner ↗

Read the response

Noughty Get Set, Grow Thickening Conditioner — $12.99 for 8.4 fl oz. It’s sulfate-free and paraben-free, and its ingredient list includes coconut oil and sodium hyaluronate. It moisturizes fragile strands and is marketed for scalp health and growth support. It supports hair care but isn’t a proven treatment for regrowing lost hair. Price excludes shipping and tax.

Which written response is best?

Candidate A

123

Candidate B

123

Candidate C

123

Which product is best?

Candidate A

123

Candidate B

123

Candidate C

123

Reveal candidates

Within this 25-case personal-shopping study, Instinct was the strongest all-around default. It had the best open-web response quality, the second-best WebShop human product-fit result, and the most WebShop overlap with another assistant. Its price behavior also sat between Muse's tendency to spend close to the limit and Dots' tendency to find much cheaper options. This is not necessarily better, but it is a sign of a more balanced shopping strategy.

That being said, the results are more useful as a map of tradeoffs than as a universal leaderboard. Dots may be a better option for somebody who wants strong product choices at a lower price, and Muse may be better for a very cautious shopper who makes broader searches and follows guidelines strictly.

You can build an eval like this to compare how your own agents make decisions, follow constraints, and communicate recommendations. Sign up for free to run an eval, or book a demo to talk through your evaluation plan.