AI Data Harmonization
收藏资源简介：
# AI Data Harmonization AI-powered Master Data Management pipeline that runs natively on Snowflake. The system matches transaction items to a master product catalog using a 3-stage AI pipeline, enabling automated and accurate product classification at scale. ## How It Works AI Data Harmonization uses a multi-stage AI pipeline powered by Snowflake Cortex: 1. **Schema Discovery** — Automatically analyzes your master catalog to determine the domain and optimal matching strategies 2. **Attribute Extraction** — Uses LLM (Claude 3.5 Sonnet via Cortex) to extract structured attributes from product descriptions 3. **Candidate Search** — Hybrid vector + keyword search (Cortex Search) to find potential matches in your master catalog 4. **Classification** — 3-tier classification logic with configurable thresholds for auto-accept, auto-reject, and LLM-assisted decisions ## Features - **Multi-Catalog Architecture** — Connect and manage multiple product catalogs simultaneously, each with its own domain and matching configuration - **Schema-Agnostic Matching** — Works with any catalog structure; the AI automatically discovers the relevant schema and adapts - **Configurable Classification Presets** — Fine-tune auto-accept, auto-reject, and confidence thresholds per catalog - **Interactive Web Interface** — Full React-based UI for pipeline management, results review, and dashboard analytics - **Ground Truth Evaluation** — Compare pipeline results against labeled ground truth data to measure accuracy - **Admin Portal** — Streamlit-based administration for service management, catalog setup, and monitoring ## Use Cases **Product Matching & Harmonization**<br/>Match transaction-level product descriptions to a standardized master catalog. Ideal for CPG, retail, and distribution companies that need to reconcile disparate product naming across systems. **Spend Classification**<br/>Classify procurement and spend data against a standard taxonomy to enable spend analytics, supplier consolidation, and compliance reporting. ## Getting Started After installing the application: 1. Use the Admin Portal to configure your external tables and connect them to your catalogs 2. Start the application services via the Admin Portal or use available procedures `CALL app_public.start_application(...)` 3. Access the web interface via the service URL once it is up ## Requirements - Snowflake account with Snowpark Container Services (SPCS) enabled - Snowflake Cortex access (for LLM and vector search capabilities) - At least one master catalog table and one transactions table are needed
数据集概述：AI Data Harmonization
- 提供方：Blend360
- 类别：AI & ML，数据质量与清洗
- 交付方式：原生应用（Native App）
- 定价模型：标准版（Standard）
核心功能
这是一个运行在Snowflake上的AI驱动的主数据管理（Master Data Management）管道，主要用于自动化产品匹配与分类。
- 工作原理：采用多阶段AI流水线，利用Snowflake Cortex能力：
- Schema发现：自动分析主目录，确定领域和最优匹配策略。
- 属性提取：使用LLM（Claude 3.5 Sonnet via Cortex）从产品描述中提取结构化属性。
- 候选搜索：混合向量+关键词搜索（Cortex Search），在主目录中查找潜在匹配项。
- 分类：三级分类逻辑，包含可配置的自动接受、自动拒绝和LLM辅助决策阈值。
- 主要特性：
- 多目录架构：同时连接和管理多个产品目录。
- Schema无关匹配：适应任何目录结构，AI自动发现并适配相关Schema。
- 可配置分类预设：可针对每个目录微调自动接受、自动拒绝和置信度阈值。
- 交互式Web界面：基于React的UI，用于管道管理、结果审查和仪表盘分析。
- 真实值评估：将管道结果与标记的真实数据进行比较，衡量准确性。
- 管理门户：基于Streamlit的管理界面，用于服务管理、目录设置和监控。
应用场景
- 产品匹配与协调：将交易级别的产品描述与标准化主目录进行匹配，适用于CPG、零售和分销公司。
- 支出分类：根据标准分类法对采购和支出数据进行分类，支持支出分析、供应商整合和合规报告。
使用要求
- 需要一个启用了Snowpark容器服务（SPCS）的Snowflake账户。
- 需要访问Snowflake Cortex（用于LLM和向量搜索）。
- 至少需要一个主目录表和一个交易表。
安全与权限
- 已完成Snowflake安全审查。
- 基于角色的访问控制（RBAC）。
- 敏感数据保护。
- 建议授予连接和应用程序事件相关的账户级别权限。
