Infostrux Loader

This application performs the "load" (L) portion of an "extract load transform" (ELT) data pipeline. It works on raw extracts that conform to Singer open-source standard.

The application provides a stored procedure that performs the load. The input is a table, view, or query string that returns the raw extracted data. The stored procedure reads the raw extracted data, creates target tables with proper schema, and populates them. The stored procedure manages target table schemas as needed. The stored procedure returns the state needed for the next incremental extraction.

The functionality of the app is similar to that of a Singer target.

The app works well with extractor applications that output data conforming to Singer standard (https://github.com/singer-io/getting-started), such as Infostrux Extractor Apps, but it can be used standalone.

搜集汇总

背景与挑战

背景概述

该数据集是一个用于ELT数据管道中加载环节的应用程序，专门处理符合Singer标准的原始提取数据。它通过存储过程自动创建目标表、管理架构并返回增量提取所需的状态，可与Singer兼容的提取器应用配合使用或独立运行。