Orchestra Metadata App

## Orchestra Native App

## Use Metadata for Data Engineering Insights

<p><br/></p>

Orchestra provides a unified control plane for Data and AI workflows, tightly integrated with Snowflake. With the Orchestra Snowflake Native App, you can:

<p><br/></p>

- Monitor pipeline runs and individual task durations across Snowflake and orchestration tools
- Rapidly detect, diagnose, and resolve errors, with detailed visibility into failed steps
- Analyse time‑based error trends to optimise reliability and reduce incidents
- Reduce waste and save money by tracking query cost, bytes processed, and inefficient jobs
- Support other AI Workflows, enabling your team to build, run, and monitor AI‑powered data products from a single pane
- Access data in Snowflake securely, with built‑in metadata, data quality testing, anomaly detection (powered by Snowflake Cortex), and full lineage traces

<p><br/></p>

The Orchestra Snowflake Native Application pulls pipeline metadata from the Orchestra API into your Snowflake environment. This metadata is extremely valuable, and gives data teams comprehensive visibility into pipeline runs, task executions, and operational metrics to build analytics dashboards and monitor data pipeline performance.

## Sample Tables

**Pipeline Runs** - Track execution metadata:

id, pipeline_name, run_status, started_at, completed_at, branch, commit, message

**Task Runs** - Monitor task execution details:

id, pipeline_run_id, task_name, status, integration, external_status, number_of_attempts

**Operations** - Detailed operation metrics:

id, operation_name, operation_status, operation_duration, rows_affected, external_id

## Expected Workflow

- Install app and grant External Access Integration permissions
- Run CALL core.create_eai_objects() to initialize
- Use stored procedures to load data into tables: CALL core.load_pipeline_runs()
- Query with standard SQL for analytics and monitoring

**Results:** Real-time pipeline metadata with automatic deduplication and updates, enabling performance monitoring, SLA tracking, and operational insights.

## Data Sources

Metadata sourced from Orchestra API (app.getorchestra.io), which aggregates data from Snowflake, BigQuery, Databricks, dbt, and other data platforms through secure HTTPS connections.

<p><br/></p>

To make full use of the Orchestra Snowflake Data App you must be an existing customer of Orchestra. New customers can signup to Orchestra at [https://app.getorchestra.io](https://app.getorchestra.io) or by reaching out to **[[e-mail]](mailto:[e-mail])** to schedule a full demo.

原始信息汇总

Orchestra Metadata App 数据集概述

数据集基本信息

• 数据集名称: Orchestra Metadata App
• 提供商: Orchestra
• 试用信息: 提供1天无限试用
• 类别: AI & ML, 分析工具, 数据工程
• 数据刷新频率: 静态数据
• 云区域可用性: AWS EU (London)

数据集功能

• 从Orchestra提取元数据到Snowflake
• 监控管道运行和任务持续时间
• 快速检测、诊断和解决错误
• 分析基于时间的错误趋势以优化可靠性
• 跟踪查询成本、处理字节和低效作业以减少浪费
• 支持AI工作流，构建、运行和监控AI驱动的数据产品

样本表结构

Pipeline Runs

• 字段: id, pipeline_name, run_status, started_at, completed_at, branch, commit, message
• 用途: 跟踪执行元数据

Task Runs

• 字段: id, pipeline_run_id, task_name, status, integration, external_status, number_of_attempts
• 用途: 监控任务执行细节

Operations

• 字段: id, operation_name, operation_status, operation_duration, rows_affected, external_id
• 用途: 详细操作指标

预期工作流程

1. 安装应用并授予外部访问集成权限
2. 运行 CALL core.create_eai_objects() 进行初始化
3. 使用存储过程将数据加载到表中: CALL core.load_pipeline_runs()
4. 使用标准SQL查询进行分析和监控

数据来源

• 元数据来自Orchestra API (app.getorchestra.io)
• 聚合来自Snowflake、BigQuery、Databricks、dbt等数据平台的数据

业务需求

• 数据质量与治理: 确保数据管道准确、最新且产生相关信息
• 监控: 管道执行状态、任务性能和操作指标

使用示例

管道性能

sql -- 监控管道性能和成功率 SELECT pipeline_name, run_status, COUNT(*) as run_count, AVG(TIMESTAMPDIFF(minute, started_at, completed_at)) as avg_duration_minutes, MAX(created_at) as last_run FROM PUBLIC.pipeline_runs WHERE created_at >= DATEADD(day, -30, CURRENT_DATE()) GROUP BY pipeline_name, run_status ORDER BY pipeline_name, run_count DESC;

安全信息

• 安全审查: 已完成Snowflake安全审查
• 访问控制: 受Snowflake基于角色的访问控制保护
• 推荐权限: 根据需要授予账户级别权限和对象权限

联系方式

• 销售: [e-mail]
• 支持: [e-mail]
• 网站: https://getorchestra.io
• 门户: https://app.getorchestra.io/signup