V1rtucious/Ecom-Chatbot-Test-Set

--- license: mit language: - en tags: - e-commerce - chatbot - evaluation - synthetic - tool-calling - rag - escalation size_categories: - 1K<n<10K task_categories: - text-generation --- # Ecom Chatbot Synthetic Test Set A 2,000-sample fully synthetic test set for evaluating e-commerce chatbot models fine-tuned on [rescommons/Ecom-Chatbot-Finetuning-Dataset](https://huggingface.co/datasets/rescommons/Ecom-Chatbot-Finetuning-Dataset). Designed for zero-contamination evaluation — all products, orders, customer names, and responses are synthetically generated and do not overlap with the training data. ## Dataset Summary | Split | Samples | |-------|---------| | test | 2,000 | ## Group Distribution | Group | Count | Description | |-------|-------|-------------| | A | 667 | Tool-calling — order management (status, cancel, return, exchange, address, refund, reorder) | | B | 667 | RAG / Product info — product QA, similarity search, bundle suggestions, cross-sell, review QA | | C | 666 | Escalation / Edge cases — complaints, escalations, policy exceptions, repeat issues, edge cases | ## Schema Matches the original training dataset schema exactly: | Field | Type | Description | |-------|------|-------------| | `id` | string | Unique sample ID (`ecomm_XXXXXXXX`) | | `source` | string | Always `synthetic_v1` | | `group` | string | A, B, or C | | `difficulty` | int | Always 2 | | `system` | string | Aria system prompt | | `history` | list | Conversation history (empty for all samples) | | `prompt` | string | Customer message | | `context` | string | Retrieved context (product catalog / order data JSON) | | `tools` | string | JSON tool schemas (Group A only) | | `response_type` | string | `text`, `tool_call`, or `mixed` | | `response` | string | Aria's response | | `language` | string | Always `en` | | `locale` | string | Always `en-US` | | `annotator` | string | Always `synthetic_v1` | | `quality_score` | float | Always 0.91 | | `domain` | string | Product domain (office, baby, automotive, home, grocery, pets, sports_outdoors, electronics, fashion, general) | | `intent_category` | string | `order_management`, `product_discovery`, or `escalation` | | `intent` | string | Specific intent | | `sub_intent` | string | Sub-intent classification | ## Contamination Prevention - All product names, brands, and IDs are fully synthetic (NovaTech, AuraSound, TrailBlazer, etc.) - Customer names drawn from a diverse synthetic name pool - Order IDs and item IDs randomly generated - Response phrasing and structure is distinct from training data sources - Domains match the original dataset but product entries are novel