Overview
Integrate.io's ELT & CDC offering allows companies to replicate their data to Snowflake in real-time with industry leading replication speed of every 60 seconds. This is particularly useful for companies that are using Snowflake to power data products where real-time data is required, particularly if it is for customer-facing data products.
ELT & CDC uses Snowflake compute resource in auto suspend and auto-resume mode to run queries to load data from an external stage. You have an option to either run a ELT & CDC dedicated compute warehouse and provide a database for ELT & CDC to load data or choose an existing warehouse and database to load.
Using an existing warehouse is recommended to minimise compute resources usage and cost for loading data from any source to Snowflake
Requirements
- Any one of the ACCOUNTADMIN, SYSADMIN or SECURITYADMIN access for running the prerequisite/setup steps.
Setup script
Copy the user script and run on your snowflake worksheet.
Copy the role script and run on your snowflake worksheet.
Copy the integration script and run on your snowflake worksheet.
Register your Snowflake Account Identifier, Warehouse, Username, Password, Role, Database and Schema on the dashboard
Note that the all details here (username, password, etc) should be the same with the one you executed on your snowflake worksheet. Input your snowflake identifier per dashboard instruction.
How to get your Snowflake Account Identifier:
- Via classic console: from the url eg: https://xxxxxxx.us-east-1.snowflakecomputing.com)
xxxxxxx.us-east-1 --> this is the value you need to input in the dashboard.
- Via Snowsight web interface: go to your Account > Copy account URL
Snowflake Network Policy
If you have an existing Network Policy, please modify your network policy add ELT&CDC IP Addresses
Limitations
- The maximum allowed VARCHAR/TEXT length of single-byte characters is 16,777,216 and 4,194,304 for 4 byte characters (per snowflake's documentation).
- The maximum length of a VARIANT is 16 MB.
Required Privileges
| Name |
Description |
Enforced? |
Reference |
| CREATE SCHEMA (Database) |
|
✅ |
|
| MONITOR (Database) |
|
✅ |
|
| USAGE (Database) |
|
✅ |
|
| USAGE (Warehouse) |
|
✅ |
|
| CREATE EXTERNAL TABLE (Schema) |
|
✅ |
|
| CREATE FILE FORMAT (Schema) |
|
✅ |
|
| CREATE FUNCTION (Schema) |
|
✅ |
|
| CREATE PIPE (Schema) |
|
✅ |
|
| CREATE PROCEDURE (Schema) |
|
✅ |
|
| CREATE SEQUENCE (Schema) |
|
✅ |
|
| CREATE STAGE (Schema) |
|
✅ |
|
| CREATE STREAM (Schema) |
|
✅ |
|
| CREATE TABLE (Schema) |
|
✅ |
|
| CREATE TASK (Schema) |
|
✅ |
|
| CREATE TEMPORARY TABLE (Schema) |
|
✅ |
|
| CREATE VIEW (Schema) |
|
✅ |
|
| MODIFY (Schema) |
|
✅ |
|
| MONITOR (Schema) |
|
✅ |
|
| USAGE (Schema) |
|
✅ |
|