Pull from Kafka - Connector

The Pull from Kafka Connector uses [Ockam](https://ockam.io/) to securely ingest Kafka messages coming from a private Kafka server and insert them to specific Snowflake tables.

With Ockam, you can access Kafka topics, without having to create a PrivateLink VPC or expose internet ports, allowing you to be securely connected within minutes.

<p><br/></p>

**Features**

The connector establishes a secure connection to a private Kafka server and:

- Continuously queries a set of selected Kafka topics for new messages.
- Extracts a chosen subset of fields from those messages and insert them into one or several Snowflake tables in your customer account.

<p><br/></p>

Several additional features are supported:

- The Kafka messages can be encoded according to a schema provided by a private Schema Registry, also securely accessed via Ockam.
- A dead-letter queue can be configured to store messages which couldn't be processed by the connector (if a field is missing for example).
- Metadata fields can also be mapped to Snowflake table columns.
- The Kafka client can be configured by adjusting: the commit period, the checkpoint limit, the replay of non-acknowledged messages, and several other properties.

<br/>**Installation overview**

- Enroll an Ockam node inside Snowflake.
- Grant privileges to the application.
- Map Kafka topics to Snowflake tables.
- Start the application to create secure point-to-point connections.

搜集汇总

背景与挑战

背景概述

Pull from Kafka - Connector通过Ockam安全连接私有Kafka服务器，将消息数据提取并插入Snowflake表，无需复杂网络配置。它支持持续查询主题、字段提取、Schema Registry处理、死信队列和客户端属性调整，安装过程包括注册节点、授权和主题表映射。