Posted on

Salesforce and Snowflake Native Data Integration Options

Introduction

Salesforce and Snowflake became strong technology partners more than a year ago. That partnership fruited prebuilt, bi-directional integration options between the two leading platforms in CRM and Data domains. The solution offers easy-to-use, point-and-click integration to push CRM data into Snowflake Data Cloud and also receive analytics data from Snowflake into Salesforce. The native Salesforce and Snowflake integration is built on top of Salesforce Tableau CRM (recently renamed CRM Analytics).

Architecture

From a technical perspective, there are 4 options that the Salesforce-Snowflake native data integration features can offer:

Posted on

SaaS Galore: Integrating CockroachDB With Confluent Kafka, Fivetran, and Snowflake

Motivation

The problem this tutorial is trying to solve is the lack of a native Fivetran connector for CockroachDB. My customer has built their analytics pipeline based on Fivetran. Given there is no native integration, their next best guess was to set up a Postgres connector:

CockroachDB is PostgreSQL wire compatible, but it is not correct to assume it is 1:1. Let's attempt to configure the connector:

Posted on

Redshift vs. Snowflake: The Definitive Guide

What Is Snowflake?

At its core Snowflake is a data platform. It's not specifically based on any cloud service which means it can run any of the major cloud providers like Amazon Web Services (AWS), Microsoft Azure, Google Cloud Platform (GCP). As a SaaS (Software-as-a-Service) solution, it helps organizations consolidate data from different sources into a central repository for analytics purposes to help solve Business Intelligence use cases.

Once data is loaded into Snowflake, data scientists, engineers, and analysts can use business logic to transform and model that data in a way that makes sense for their company. With Snowflake users can easily query data using simple SQL. This information is then used to power reports and dashboards so business stakeholders can make key decisions based on relevant insights.

Posted on

Building an Enterprise CDC Solution

Introduction

This article is a follow-up to the Data Platform: Building an Enterprise CDC Solution, where Miguel García and I described:

  • Several Change Data Capture (CDC) use cases and common scenarios in an enterprise platform
  • A proposal using Debezium (as log-based CDC) to capture data from the relational databases, and Kafka as a channel that enables several consumers to propagate data changes for different use cases.

One of the common scenarios for this solution consists of data replication from OLTP Database to OLAP Database (from the operational database to the data warehouse).

Posted on

Databricks vs Snowflake: The Definitive Guide

There is a lot of discussion surrounding Snowflake and Databricks in determining which modern cloud solution is better for analytics. However, both solutions were purpose-built to handle different tasks, so neither should be compared from an “apples to apples” perspective.

With that in mind, I’ll do my best to break down some of the core differences between the two and share the pros/cons of each as unbiasedly as possible. Before diving into the weeds of Snowflake and Databricks though, it is important to understand the overall ecosystem.

Posted on

Azure Synapse vs Snowflake: The Definitive Guide

With the world on pace to reach 175 Zettabytes of data by 2025, it’s no wonder why organizations are placing such a high emphasis on building out their technology stacks. Now more than ever, companies need a way to collect and consolidate data into a single platform to derive insights quickly.

This is one of the core reasons that Snowflake and Azure Synapse Analytics have risen to such popularity. However, Synapse and Snowflake are different solutions and both should be analyzed from an unbiased lens. With that in mind, here are some of the core differences and pros/cons to Snowflake and Synapse.

Posted on

How to Generate Customer Success Analytics in Snowflake

As the distinction between data professionals and non-data professionals becomes smaller and smaller, the need for technology that bridges the gap between the two parties is crucial. The benefits of interacting with a data warehouse, especially with large amounts of data, are unquestionable, but as a peripheral member of the core technology team who might not be very technical, it is not always practical to generate SQL queries on the fly. 

This poses a problem, especially when departments such as sales, customer success, account management, etc., want the robust insights that could come from the vast amount of data that a company is storing, but they don’t necessarily know how to quickly gather these insights. 

Posted on

Best Practices for Transforming Data in Snowflake

The death of the star-schema is not exaggerated. Gone are the days of all-encompassing data warehouse models and the 24-month projects to build them.

We live in a highly disruptive, event-driven world. New analytics are required almost daily to understand how our customers, business, and markets shift. A modern data stack using the speed, flexibility, and scalability of Snowflake needs to allow an organization to “model as you go” to answer critical business questions on the fly.

Posted on

Migrating to Snowflake, Redshift, or BigQuery? Avoid these Common Pitfalls

The Drive to Migrate Data to the Cloud

With data being valued more than oil in recent years, many organizations feel the pressure to become innovative and cost-effective when it comes to consolidating, storing, and using data. Although most enterprises are aware of big data opportunities, their existing infrastructure isn’t always capable of handling massive amounts of data.

By migrating to modern cloud data warehouses, organizations can benefit from improved scalability, better price elasticity, and enhanced security. But even with all these benefits, many businesses are still reluctant to make the move.

Posted on

How to Migrate Your Data From Redshift to Snowflake

For decades, data warehousing solutions have been the backbone of enterprise reporting and business intelligence. But, in recent years, cloud-based data warehouses like Amazon Redshift and Snowflake have become extremely popular. So, why would someone want to migrate from one cloud-based data warehouse to another?

The answer is simple: More scale and flexibility. With Snowflake, users can quickly scale out data and compute resources independently by automatically adding nodes. Using the VARIANT data type, Snowflake also supports storing richer data such as objects, arrays, and JSON data. Debugging Redshift is not always straightforward as well, as Redshift users know. Sometimes it goes beyond feature differences that could trigger a desire to migrate. Maybe your team just knows how to work with Snowflake better than Redshift, or perhaps your organization wants to standardize on one particular technology.

Posted on

Cloud Data Warehouse Comparison: Redshift vs. BigQuery vs. Azure vs. Snowflake for Real-Time Workloads

Data helps companies take the guesswork out of decision-making. Teams can use data-driven evidence to decide which products to build, which features to add, and which growth initiatives to pursue. And, such insights-driven businesses grow at an annual rate of over 30%.

But, there’s a difference between being merely data-aware and insights-driven. Discovering insights requires finding a way to analyze data in near real-time, which is where cloud data warehouses play a vital role. As scalable repositories of data, warehouses allow businesses to find insights by storing and analyzing huge amounts of structured and semi-structured data.

Posted on

Serverless Kafka in a Cloud-Native Data Lake Architecture

Apache Kafka became the de facto standard for processing data in motion. Kafka is open, flexible, and scalable. Unfortunately, the latter makes operations a challenge for many teams. Ideally, teams can use a serverless Kafka SaaS offering to focus on business logic. However, hybrid scenarios require a cloud-native platform that provides automated and elastic tooling to reduce the operations burden. This blog post explores how to leverage cloud-native and serverless Kafka offerings in a hybrid cloud architecture. We start from the perspective of data at rest with a data lake and explore its relation to data in motion with Kafka.

Data at Rest - Still the Right Approach?

Data at Rest means to store data in a database, data warehouse, or data lake. This means that the data is processed too late in many use cases - even if a real-time streaming component (like Kafka) ingests the data. The data processing is still a web service call, SQL query, or map-reduce batch process away from providing a result to your problem.

Posted on

Snowflake Data Encryption and Data Masking Policies

Introduction

Snowflake architecture has been built with security in mind from the very beginning. A wide range of security features are made available from data encryption to advanced key management to versatile security policies to role-based data access and many more, at no additional cost. This post will describe data encryption and data masking functionalities.

Snowflake Security Reference Architecture

Snowflake Security Reference Architecture includes various state-of-the-art security techniques that offer multiple outstanding cloud security capabilities. It includes data encryption while data at rest, secure data transfers while data in transit, role-based table access, column and row-level access to a particular table, network access/IP range filtering, multi-factor authentication, Federated Single Single On, etc.

Posted on

Connecting Snowflake With MuleSoft Database Connector

Introduction

Snowflake is an analytics data warehouse provided as Software as a Service. It is faster, flexible, and easy to use and it is not built on the top of any BigData platform like Hadoop. It has many similarities to the Enterprise data warehouse but also some unique and additional capabilities. As Snowflake is a SaaS application, there is no need for any hardware need to set up virtually or physically, and no additional software required to install, configure, or manage.

MuleSoft Database Connector

MuleSoft database connectors have capabilities to connect any database like MS SQL, Oracle, MySQL, etc. The database connector provides many operations like Select, Update, Delete, Insert, Calling Stored Procedure, etc.

Posted on

API-First Product Managers’ Popular API Tools and API Metrics

We interviewed the product managers at a number of the larger API-first companies that are based in San Francisco. The companies are all publicly traded, have TTM revenue of more than $100M and are in the fields of billing, security, communications and workflow automation.

The PMs were asked what were their favorite tools and what API metrics they cared most about. Where possible we identified tools and metrics that were common across all market segments, excluding the (many) edge cases that you’d expect when your customer base numbered in the 1,000s.

Posted on

Snowflake External Functions

Introduction

Snowflake has recently announced external functions available in public preview. This allows developers to invoke external APIs from within their Snowflake SQL queries and blend the response into their query result, in the same way as if they were internal Snowflake functions.

In this article, we will demonstrate how to invoke an API via Amazon Web Services API Gateway that will trigger an AWS Lambda function. The Lambda function (written in Python) then  invokes a public API from to return the exchange rate for USD and multiple foreign currencies that can be used to calculate our sales values in USD and a number of selected currencies in SQL query running in our Snowflake warehouse. This solution eliminates the need for loading exchange rates into Snowflake regularly and also guarantees accurate, reliable real-time currency values.