0:0020:40

Build Data Intense Applications using Google Cloud Platform

Pooja Kelgaonkar 20:40 2023

Choosing ETL, ELT or a data mesh starts with three pillars, not a tool

Every data platform on Google Cloud, whether warehouse, lake or data mesh, rests on three pillars: the foundation of the data system, how data is distributed, and how it is derived for analytics and AI. Pooja Kelgaonkar, a senior data architect at Rackspace Technology with 16 plus years in data, uses them to sort a crowded catalogue of services into reusable patterns. Which pattern fits your sources, consumers and governance needs?

See reference designs for ETL, ELT, warehouse, lake, data mesh and Dataplex governance on GCP

Watch on YouTube

In short

Pooja Kelgaonkar, Senior Data Architect at Rackspace Technology, shows how to design data-intensive applications on Google Cloud Platform starting from three pillars: the foundation of data systems, distributed data and derived data. She maps ETL, ELT and hybrid approaches to GCP services and walks through reference designs for a data warehouse, data lake, data mesh and Dataplex-based governance.

“Depending on the type of application you have and the type of consumers you have, and the agreement between your application and your consumers, you define the consistency policies.”
“So these are the three important pillars of the data intensive application design. So the first one is the foundation of data system, the second one is the distributed data, and the third one is the derived data.”

What you will learn

  • Evaluate a data application against three pillars: foundation of data systems, distributed data and derived data.
  • Choose between ETL, ELT and a hybrid approach based on source variety, available integrations and the data you process.
  • Map GCP services to patterns: Dataflow or Data Fusion for ETL, BigQuery or Dataproc for ELT, Vertex AI, Looker and AutoML for analytics.
  • Compare warehouse, data lake and data mesh platform designs built on BigQuery, Cloud Storage and domain-level producers.
  • Control consumer access with BigQuery authorized views, functions or datasets and govern data with Dataplex.

Key takeaways from the speaker

  • This session will help audience to understand deisgn patterns to build data intense applications using Google Cloud Platform

Details

Level
Advanced
Format
Tech talk
Technical depth
Practitioner
For
Mid-career, Senior individual contributor
Length
20:40
Published
June 2023

Questions this talk answers

What are the pillars of data-intensive application design?

The foundation of data systems (reliable, scalable, maintainable applications plus data models and storage), distributed data (replication, partitioning, transactions and consistency) and derived data (batch or streaming processing and the analytics built on it).

When should you use ETL, ELT or a hybrid approach?

ETL transforms data on the fly before it lands in the platform, ELT loads it to a raw or staging layer first and transforms it there, and hybrid mixes both depending on heterogeneous sources, available integrations and the kind of data processed.

Which Google Cloud services implement ETL and ELT patterns?

Dataflow and Data Fusion serve as ETL tools, while BigQuery is the most widely used ELT engine and Dataproc supports ELT for Hadoop migrations by running Spark jobs on ephemeral clusters.

How can machine learning be added to a BigQuery data warehouse?

Through Vertex AI, AutoML or BigQuery ML, where models are created with CREATE MODEL statements, trained and deployed in BigQuery and invoked with simple SELECT statements.

How do consumers access data in a data mesh on GCP?

Through BigQuery authorized views, functions or datasets so underlying data is not exposed, or less preferably through read APIs governed by role-based policies.

About this talk

This session covers various aspects of data application design and implementation. This will help to pick and choose from set of Data services offered by Google Cloud covering architecting and implementing data intense applications.

Topics

Related talks

Browse the library

What Is NoSQL?

Heather Maga Business Owner / Software Developer, Copper Owl Development

In NoSQL, the database stops checking your data and the developer takes over

Beginner AI and data 2020