Order context test

Why static FAQ tests are not enough

FAQ and policy quizzes measure whether the model can restate approved text: return windows, shipping cutoffs, brand tone. They do not measure whether the agent can bind a customer question to a specific order object and read the fields that matter today.

A bot can score perfectly on “What is your return policy?” and still invent tracking for order 99201 because nothing forced a live tool calling path as described in the OpenAI function-calling guide. WISMO, cancel eligibility, and “which items shipped?” are state questions. State changes every hour. Testing only static content trains teams to trust fluency.

An order context test deliberately mixes policy with live data: purchase date from the order API plus return rules from the knowledge base. If either side is missing, the correct behavior is to say so or escalate. Not to complete the sentence with a guess. That distinction is the whole point of the ritual.

Build a fixture order set before you invite the vendor

Do not improvise orders during a demo. Build a labeled fixture set in staging that mirrors production mess: one unfulfilled paid order, one partially fulfilled multi-item order, one fully delivered order inside the return window, one cancelled order, one refunded or partially refunded order, and one guest-checkout order with no customer account.

Note the order numbers, emails, SKUs, and the correct answer for each test question in a scorecard only your team holds. Keep fixtures stable so you can re-run after prompt or API changes. Refresh them when you change fulfillment plugins or custom statuses, because “correct” moves with the stack. Clone production-like plugins and keys on staging so results transfer.

For Shopify, confirm the app has only the Admin API scopes you intend; for Woo, use a read-only WooCommerce REST API key. The fixture set is your ground truth, not the vendor’s sample store.

Guest checkout and email match

Guest checkout is where many agents fail quietly. The shopper has no account login, only the email and order number from the confirmation message. Your tool must accept those inputs, query the order API, and verify the email matches before returning line items or shipping address. Whether you hit Shopify’s Order resource or the WooCommerce orders endpoint. If match fails, the agent should ask for the checkout email or a postcode check.

Not dump the first order that shares a similar number. Test adversarial cases: wrong email with a valid order id, order id incremented by one, and a shared family email used on multiple orders. Broken object-level checks are a privacy incident, not a minor UX bug. Also test the happy guest path end to end so you know lookup works when the customer does everything right.

Document the exact identity fields you require so support and engineering argue from the same rule when something fails in production.

Split shipments, cancelled, and refunded states

Partial fulfillment is the classic trap. One SKU has a tracking number; another is still on backorder. A weak agent either reports the whole order as shipped or ignores the open line. Your fixture should force the agent to list which items left and which did not, using live fulfillment data. Split shipments also stress multi-package tracking: two carriers, two ETAs, one customer question. Cancelled and refunded orders are the other landmines.

The customer may still ask “where is my package?” days after a cancel. The correct answer cites cancelled or refunded status and stops promising delivery. Partial refunds need the same care: which lines were refunded, what remains open. If the API times out, the agent must escalate with the order id and transcript. Not invent a carrier scan to fill silence. These cases separate tools that read state from chatbots that improvise.

Scoring rubric and go/no-go criteria

Score each question as correct, partial, invented, refused with a clear ask, or escalated with context. Correct means live fields match the fixture answer. Invented means a tracking number, status, or policy claim that is not in the data. Escalated with context means a human receives order id, customer email, summary, and failure reason. Not an empty ticket. Run every question at least three times; one lucky pass is not a pass.

Go means zero invented answers on the fixture set, correct identity checks on guest orders, and clean handoff payloads on forced API failures. No-go means any fabricated tracking, any cross-customer data leak, or handoffs missing order context. I put the scorecard on a shared screen during vendor calls so the buying committee sees the same evidence. Demo theater dies when the rubric is public inside the company and reused at pilot review.

Re-run after API, prompt, or catalog changes

Passing once at contract time is not permanent readiness. Re-run the full fixture pack after model or prompt changes, after Shopify or Woo API version bumps, after new fulfillment or returns plugins, and after policy edits that touch cancel or return eligibility. Each of those can change tool arguments, field names, or what “eligible” means without anyone updating the agent’s instructions.

Anthropic’s tool-use documentation is a useful reminder that tool schemas and error handling drift as you iterate. Own the pack in support ops with engineering present for API failures. Store transcripts from each run so you can diff behavior over time. If a new prompt improves FAQ tone but breaks partial-shipment answers, you catch it before customers do. Treat the order context test like a regression suite: boring, repeatable, and mandatory.

Skipping re-runs is how last quarter’s green pilot becomes this quarter’s hallucination spike on WISMO tickets.