Pull from Kafka - Connector

The Pull from Kafka Connector uses [Ockam](https://ockam.io/) to securely ingest Kafka messages coming from a private Kafka server and insert them to specific Snowflake tables. With Ockam, you can access Kafka topics, without having to create a PrivateLink VPC or expose internet ports, allowing you to be securely connected within minutes. <p><br/></p> **Features** The connector establishes a secure connection to a private Kafka server and: - Continuously queries a set of selected Kafka topics for new messages. - Extracts a chosen subset of fields from those messages and insert them into one or several Snowflake tables in your customer account. <p><br/></p> Several additional features are supported: - The Kafka messages can be encoded according to a schema provided by a private Schema Registry, also securely accessed via Ockam. - A dead-letter queue can be configured to store messages which couldn't be processed by the connector (if a field is missing for example). - Metadata fields can also be mapped to Snowflake table columns. - The Kafka client can be configured by adjusting: the commit period, the checkpoint limit, the replay of non-acknowledged messages, and several other properties. <br/>**Installation overview** - Enroll an Ockam node inside Snowflake. - Grant privileges to the application. - Map Kafka topics to Snowflake tables. - Start the application to create secure point-to-point connections.

搜集汇总

背景与挑战

背景概述

Pull from Kafka - Connector通过Ockam安全连接私有Kafka服务器，将消息数据提取并插入Snowflake表，无需复杂网络配置。它支持持续查询主题、字段提取、Schema Registry处理、死信队列和客户端属性调整，安装过程包括注册节点、授权和主题表映射。