DBT is one of the most popular solutions to write and execute dynamic SQL Statements. It’s quick and easy to convert complex SQL statements into a few lines of code in DBT, we have the flexibility to use different configurations to customize our models, we can use Jinja and Macros, to schedule jobs in DBT […]
We have a small project with four layers. When we added relationship tests between project layers, we started getting false test failures. The raw layer is a bit like dbt’s stage layer, we do renaming and some very simple changes here and have validation for duplicates and nulls. The validation layer we build dims and […]
Snowflake is one of the most popular Software-as-a-Service (SaaS) solutions to store and process the data of your whole organization. It’s a pioneer in data management, which is why Snowflake places a high value on security. Imagine you are pulling data from different databases and sources in your organization, and you are centralizing all this [...]
Event Storming is the creation of Alberto Brandolini, a lightweight but powerful workshop approach for software development and domain-driven design. Using sticky notes, experts like software developers and product designers work together at an accelerated pace to focus, rethink, reframe, and bounce ideas off each other. Event Storming is a methodology meant to: Be quick, [...]
Looking at Looker, Part 2 By Renee Miller and Victoria Davis Why should we talk about Looker? With Looker JOIN19 a week away, we thought we’d dive deeper into the Looker platform (See Part 1 here). Earlier this year, Google Cloud announced a plan to buy Looker for $2.6 billion, making it even more of a [...]
The easy way Before I learned how to load only the columns I needed from DynamoDB into Redshift, I created an entire glue job to load a single table. I’d like to prevent you from making the same mistake. But to understand how much easier the easy way is, you will first need to understand how [...]
This blog, Deploying Kafka Streams and KSQL with Gradle – Part 2: Managing KSQL Implementations, was originally posted on the Confluent Blog on May 29, 2019. See the original source here. In part 1, we discussed an event streaming architecture that we implemented for a customer using Apache Kafka®, KSQL from Confluent, and Kafka Streams. Now [...]
In Part 1 of our post about Migrating Data to Snowflake, we exported our data from on-premise, and uploaded it to S3. We are just about ready to start importing the data into Snowflake. First here is a quick recap of what we have done so far. Part 1 Recap Okay, to recap at [...]
How to use the Snowflake Data Warehouse destination in StreamSets Data Collector to replicate an OLTP database StreamSets Data Collector offers incredible flexibility while creating data integration pipelines by allowing developers to either create a custom stage (an origin, processor, executor, or destination) completely from scratch, tailor a universal stage to a specific requirement, or use [...]
In a recent blog titled Managing Snowflake Data Warehouse Compute in OBIEE, I explained how to assign application users and roles to use designated compute clusters (warehouses) in Snowflake. The idea of specifying a Snowflake warehouse on a per user (or groups of users) basis is not specific to OBIEE and can be achieved with [...]
