Data Replication to Snowflake with AWS DMS

 AWS DMS (Amazon Web Service Database Migration Service) is a cloud-based service where modern businesses can avail of data on an almost unlimited scale, currency, and performance. Organizations can combine their data into silos, share governed data, and carry out diverse and multiple workloads with AWS DMS Snowflake that delivers the Data Cloud on AWS.

The main benefit of Snowflake is that it provides a seamless single-point experience across various public clouds. Users, customers, and data providers can cross new frontiers in the Data Cloud with AWS DMS Snowflake. It is also possible to use the services of the AWS Cloud to process data continually and load it into a database of Snowflake.


How to execute this process?

Amazon Kinesis Firehose delivers data to Amazon S3 (Simple Storage Service). The Amazon Simple Notification Service (Amazon SNS) on receipt of this data sends notifications to Snowflake Snowpipe that in turn loads the data into a Snowflake database. The downside faced by users is that Snowflake Snowpipe does not connect directly to Kines is Data Firehose. 

The upside is that the AWS DMS Snowflake process is fully automated and users can access continually generated and updated data in seconds. They can also avoid manual COPY commands and avail the benefits of support to semi-structured data on load.

There are several pre-conditions for AWS DMS Snowflake.

First, the Snowflake account should have the necessary permissions to integrate storage with the data present concurrently in a table in Snowflake. Next, have an AWS account that has the required permissions to create AWS Identity and Access Management (I AM) policies and roles. Finally, have an Amazon S3 bucket that Data Wrangler can use for entering the transformed data.     


Comments

Post a Comment

Popular posts from this blog

The Change Data Capture of Microsoft SQL Server

 The Change Data Capture feature of Microsoft SQL Server was started in 2005 with the launch of its “after date”, “after delete”, and “after insert” capabilities. However, there were certain flaws that surfaced which were ironed out in a new version in 2008, giving rise to SQL Server change data capture in its present form. This can monitor, track, and capture any changes that are made to the SQL Server database.  SQL Server change data capture records all changes made to a SQL Server table like insert, update, and delete leading to details of all changes being available in a user-friendly relational format. Column information and metadata crucial to apply the changes to the target database are captured for the modified rows and stored in change tables that replicate the column structure of the tracked source tables. SQL Server change data capture has table-valued functions that allow consumers to have seamless access to the changed data. The source of SQL Server change data c...
Read more