Overall, there’s more management involved with Redshift than Snowflake.ġ. With Redshift, you have to manage specific servers even though the service is virtual. You can switch the data compute capacity as you see fit. Since compute and storage are separate in Snowflake, you don’t have to resort to copying the data to scale up or down. Redshift Resize operations can also become quite expensive, resulting in significant downtime. This poses a few challenges in Redshift, similar to the challenges faced while scaling up or down in Redshift. In terms of analyzing and vacuuming the tables regularly, Snowflake offers a reliable solution. This is super helpful and something we could not do in Redshift. Once we provided the permissions for teams to use the warehouse, it was easy to identify the cost associated with each application and business unit.
We built a dedicated warehouse for our major applications and named the warehouse so that it was easy to recognize who within the organization is using it. One of the big advantages of Snowflake architecture is that it provides the separation between storage and compute. All other transaction tables were copied from their respective source databases using HVR and Talend. The master tables were migrated from Redshift to S3 using the UNLOAD command and then loaded into Snowflake using the COPY command.
For all our ETL needs, we used the same Talend and Matillion tools along with Python scripting. We used HVR, a Snowflake partnered real time data replication solution, as a replacement for AWS DMS. Snowflake offers these capabilities, and we decided to use it as the consumption platform for all end user reporting and access needs. We needed a platform with a much more scalable database solution that could dynamically be scaled based on user requirements and SLA expectations.
#REDSHIFT DATEADD FULL#
Migrating to a different Redshift cluster configuration can take a full 48 hours, but we weren’t sure that it would resolve our speed issues. We had to keep aside all our development tasks and work on getting this latency issue resolved. Many times, we had to manually re-run the ETLs as the Redshift disk/CPU utilization crossed 90%, and all ETL and Tableau processes began to develop latency. Still, as our data started growing in hundreds of TBs and users querying it also increased, we started to encounter concurrency issues when multiple ETL jobs were loading the same table or users were trying to access it at the same time. This process was stable for a number of years, and it served its purpose at the time. A number of Tableau reports and dashboards were also made from this data, with Tableau connecting to Redshift. After data cleaning and transformation were complete, the Data analytics/science team used the same data for analysis and making business decisions. The existing data warehouse, i.e., Redshift, had a number of ETL (Talend, Matillion) and real-time data replicating jobs (AWS DMS) running on it. This blog talks about migrating our existing data warehouse from Redshift to Snowflake.
0 kommentar(er)
