Build Data Intense Applications using Google Cloud Platform
Pooja Kelgaonkar20:402023
Choosing ETL, ELT or a data mesh starts with three pillars, not a tool
Every data platform on Google Cloud, whether warehouse, lake or data mesh, rests on three pillars: the foundation of the data system, how data is distributed, and how it is derived for analytics and AI. Pooja Kelgaonkar, a senior data architect at Rackspace Technology with 16 plus years in data, uses them to sort a crowded catalogue of services into reusable patterns. Which pattern fits your sources, consumers and governance needs?
See reference designs for ETL, ELT, warehouse, lake, data mesh and Dataplex governance on GCP
Pooja Kelgaonkar, Senior Data Architect at Rackspace Technology, shows how to design data-intensive applications on Google Cloud Platform starting from three pillars: the foundation of data systems, distributed data and derived data. She maps ETL, ELT and hybrid approaches to GCP services and walks through reference designs for a data warehouse, data lake, data mesh and Dataplex-based governance.
“Depending on the type of application you have and the type of consumers you have, and the agreement between your application and your consumers, you define the consistency policies.”
“So these are the three important pillars of the data intensive application design. So the first one is the foundation of data system, the second one is the distributed data, and the third one is the derived data.”
What you will learn
Evaluate a data application against three pillars: foundation of data systems, distributed data and derived data.
Choose between ETL, ELT and a hybrid approach based on source variety, available integrations and the data you process.
Map GCP services to patterns: Dataflow or Data Fusion for ETL, BigQuery or Dataproc for ELT, Vertex AI, Looker and AutoML for analytics.
Compare warehouse, data lake and data mesh platform designs built on BigQuery, Cloud Storage and domain-level producers.
Control consumer access with BigQuery authorized views, functions or datasets and govern data with Dataplex.
Key takeaways from the speaker
This session will help audience to understand deisgn patterns to build data intense applications using Google Cloud Platform
Details
Level
Advanced
Format
Tech talk
Technical depth
Practitioner
For
Mid-career, Senior individual contributor
Length
20:40
Published
June 2023
Questions this talk answers
What are the pillars of data-intensive application design?
The foundation of data systems (reliable, scalable, maintainable applications plus data models and storage), distributed data (replication, partitioning, transactions and consistency) and derived data (batch or streaming processing and the analytics built on it).
When should you use ETL, ELT or a hybrid approach?
ETL transforms data on the fly before it lands in the platform, ELT loads it to a raw or staging layer first and transforms it there, and hybrid mixes both depending on heterogeneous sources, available integrations and the kind of data processed.
Which Google Cloud services implement ETL and ELT patterns?
Dataflow and Data Fusion serve as ETL tools, while BigQuery is the most widely used ELT engine and Dataproc supports ELT for Hadoop migrations by running Spark jobs on ephemeral clusters.
How can machine learning be added to a BigQuery data warehouse?
Through Vertex AI, AutoML or BigQuery ML, where models are created with CREATE MODEL statements, trained and deployed in BigQuery and invoked with simple SELECT statements.
How do consumers access data in a data mesh on GCP?
Through BigQuery authorized views, functions or datasets so underlying data is not exposed, or less preferably through read APIs governed by role-based policies.
About this talk
This session covers various aspects of data application design and implementation. This will help to pick and choose from set of Data services offered by Google Cloud covering architecting and implementing data intense applications.
The foundation and distributed data pillars (80 s)
ETL vs ELT vs hybrid in under a minute (55 s)
Data mesh: decentralizing data by domain (51 s)
Speaker 1: Hello everyone. Good morning. Welcome to the session. Thanks for joining. So this session we are going to talk about building data-intense applications with Google Cloud Platform. Before we start with the session, quickly about myself. So I'm Poohja Kelgaonkar. I have 16 plus years of experience in data domain. I'm currently working as a senior data architect with Racspace. I've recently been awarded and recognized as a Snowflake data superhero. So one of the 72 members across the globe. So in today's session, this is going to be the agenda. We are going to talk about what are the data-intense application pillars. So what are the pillars that contribute to the application design and how we can use them to design and data application? What are the GCP services available? What are the data application and analytics patterns that we can design with Google Cloud Platform? What are the various data platform designs like Data Warehouse, Data Lake, Data Mesh? How we can do it using GCP services? And what is
what is the data governance design model? What is the data governance design model that we can implement and what are the services from GCP can be liberates to build a data governance design? So let's get started with the application pillars. So when we talk about Google Cloud, so we know it's a cloud offering by Google. And when we talk about a cloud adoption, so these are some of the key reasons are the opportunities because of which we move to cloud. So cloud has a lot of opportunities in terms of advanced analytics, future development, and the perception change from how to what to be developed, like an innovation mindset.
The main important thing, let's talk about the what are the pillars of the application design. So the first pillar is the foundation of data systems. So there are three things that the three components contribute to the foundation of a data system. The reliable, scalable and maintainable applications. So how your application is reliable, how scalable it is, and how about the maintenance, whether it is a low cost or the high cost maintenance.
The second thing is data models and query languages. The second thing is data models and query languages. So what kind of databases are they like relational, non-relational, how they are queried, does they support as a SQL or no SQL models, how the storage and retrieval. So what kind of data is being stored and retrieved, whether they are retrieved by the consumer integration applications or the queries, and what kind of data it is, whether it is a transactional data or analytical data.
The second important pillar is distributed data. So what kind of data distribution is there, how data is being replicated, is it replicated and designed in such a way that it is always a high available. So talking about disaster recovery and disaster recovery planning. The next one is the partitioning. The next one is the partitioning, how data is being stored across the data platform and partition so that you get performance efficient access to the data systems. Next is a transaction. So what kind of operations been run on top of your data platform?
Are there analytical transactions? Are there transaction data where the role level operations are being done? How the data is being scanned? And the last important, last and most important thing is the consistency. How consistent your data is, whether it's a highly consistent, strong consistency implemented, or do you have eventual consistency? Depending on the type of application you have and the type of consumers you have, and the agreement between your application and your consumers, you define the consistency policies.
So last one is the derived data. So how your data is being derived and stored and processed in your platform. So once you bring in data, what kind of nature of the data it is, whether it is being considered as a batch, whether it is being considered as a streaming, whether it's an event stream or a database streams, and what is the future of your data? Like what kind of analytics is being done?
Considering the descriptive, or a predictive analytics. So these given the era of a generative AI. So how, what kind of AI models you can build, how your data is being categorized into the AIML integrations, and what kind of a predictive analytics you can build. So these are the three important pillars of the data intensive application design. So the first one is the foundation of data system, the second one is the distributed data, and the third one is the derived data. So these are the three important pillars using which we are going to talk about the various patterns, and how we are going to design the applications.
So let's talk about the approaches. So what kind of approach you can implement when we talk about the application design. So we know the ETL, ALT and the half grade approaches are there. So these are the three widely used application patterns when we design any of the data applications, whether you extract the data, transform it on the fly and load it before it leads to the data platform.
The second one is the second one is the extract load and transform, whether you end up loading your data to the raw layer or the staging layer on the database first on the data platform, and then you run a set of transformation jobs to transform your data and have it ready for the analytics to be done. And the third one is the hybrid where you end up using mix of both ETL and ELT, depending on the heterogeneous data sources you have depending on the integrations available with the data sources and the kind of data you want to process.
The third one is the third one is the third one is the third one is the third one is the data services. I'm sure most of you are aware of GCP and data services. I may not spend much time explaining each of these services, but yes, as you know, GCP is reaching data services, and there are various services available which supports the relational databases which supports the transactional operations which supports the NoSQL and analytical operations as well. So there are also various categories available which are formulated and categorized into the type of storages available, type of analytics being run, type of AI/ML being run, and the type of a serving model where which services can be leveraged to integrate with a consumer application. So be it your mobile applications or web app based applications, or any typical set of BI analytical applications where consumer can fetch data from
your platform in terms of your platform in terms of your platform in terms of reports and dashboards. Then the next one is, this is another level of a categorization where we can categorize the GCP services between the various compute offerings, storage offerings, big data, and the most importantly, the machine learning or the auto ML. So GCP also offer lots of auto ML services, where it is just a kind of a plug and play, you may not need to have, you know, ML models to be developed.
So these are just a kind of a plug and play depending on your application requirements. Now let's see how these patterns can be implemented using GCP. So this is a ETL pattern. So if you observe here, data flow is the service that is being used as a ETL tool here, whereas in the various sources here are integrated and used as a part of a ETL integration.
And then there are set of services can be used to storage and analyze the data. So similar to data flow, data fusion can also be used as a ETL service where you can integrate it with heterogeneous sources, bring in the data directly from the source or have it landed to the storage layer and read it from there and push it to the data lake or a data warehouse or the storage layer, depending on your application design and the requirements.
So these are the two ETL designs that you can implement using the data flow or data fusion on GCP. The second one is the ELT patterns. So how you can design the ELT. So ELT as the name itself says we have to extract the data first and then transform. So transforming leveraging the services part. So here is the one of the most commonly and widely used architecture that is using BigQuery for the ELT implementation.
The second one, the second one, the second one, the second one, the same, you can do it using data proc. So this is the reference architecture for the Hadoop migration use case where you design the ELT pattern using data proc. all your Hadoop Spark jobs are converted to run on the ephemeral clusters. on using the cloud data proc. And these are again kind of ELT pattern where you extract data from the sources, have it as well.
on the storage layer and then you read the data from storage and then you read the data from storage, process it using Spark jobs. using data proc all your hadoop spark jobs are converted to run on the ephemeral clusters on using the cloud data proc and these are again kind of elt pattern where you extract data from the sources have it on the storage layer and then you read the data from storage process it using spark jobs and once your data is been processed you store it here on the storage or you push it to the warehouse or a big table depending on the type of consumers you have and the accessibility required so these are the two patterns that these eyes that we have seen etl and elt now let's see
what analytical pattern we can implement so when i talk about data analytics so there are two types of analytics right so the predictive ones and the descriptive ones where all the dashboards and bi tools fall in place so this is the example of a predictive analytics where you can implement uh automated model training cycle using vertex ai where the data is being trained based on the new data coming in so as and when new data comes to the comes to the table the model gets automatically trained trained and deployed so that you have a continuous deployment training deployment cycle available the next one is the looker so gcp looker is the service that can be used to have a descriptive analytics implemented you can have dashboards and various reports made designed and made available to the
consumers the next try is the auto ml so auto ml are the services which are used uh widely if you don't want to have your customized models built using vertex ai you can very well use these models and some of the widely used models are the nlp the natural language processing model or the translation models or the vision models or text to speech so depending on the business requirements you can use these ml services now let's talk the most important part of the session is the platform designs so what are the different platform designs that we can implement using gcp services so the first one is the warehouse platform so as you know i believe you are very well aware of the data lake data warehouse data mesh and the lighthouse patterns
so this particular pattern talks about a data warehouse where you design and build a warehouse using gcp bigquery and there are various integrations available so if you see here these are various sources available and these sources can be integrated with a big query and the middle layer talks about the data processing layer where data can be processed in the big query leveraging the big query native power or leveraging the data flow data fusion or data proc services so here it talks about both the patterns elt and etl depending on the requirements you can have the warehouse design to follow either of them or both depending on the type of source you have and the ingestion integration requirements and then followed up by the machine learning model so as you all know gcp bigquery is one of the flagship service
of a gcp where it can be integrated with any other services on the google cloud platform so the same way it can be very well integrated with auto ml or a vertex ai but along with these two machine learning offerings bigquery also have built-in native support to the bigquery ml so it's uh similar to the dml uh dml uh dml statements so you can create a model using create statements and you can there are specific algorithms supported as a part of a big query ml you can create the models using create model statements have them trained and deployed on a bigquery and you can use simple select statements to invoke those models and have the predictions run so these are the three ways you can have the machine learning implemented
in a warehouse pattern using gcp bigquery and similar to any other consumer integrations you can have various consumer integrations available either through looker or any of the any of the third-party platforms even you can integrate it with tableau power bi or any of the open source available so bigquery has connectors available and you can very well connect it with a bigquery for a consumer application design the next pattern is the data lake pattern so data lake is different from the data warehouse where lake data lake talks about storing and maintaining your data at the uh at the storage level okay so even though you have various integrations for your batch streaming you have your intense data processing then you still have a storage where you use restore and your store data for your future
references and maintain the historical data the same way you can do it using in the warehouse as well but the lake in the lake design if you see it here the cloud storage is used as the core of the data lake and the data lake is being implemented using a cloud storage so uh the if you see heat here the storage is at the center and these are the various sources source level integrations there are two types of the processing available stream processing the stream can be implemented using pub sub and a data flow and a batch can be implemented using various loading jobs or the data can be directly pulled into the source to the source buckets on the cloud storage and then there are machine learning available using the vertex ai
or the data prep and the auto ml available and the same goes with them uh consumer layers so this is how you can have the lake design and when we talk about data lake so even though your data is stored at the storage level you can have various data marks defined either at a big query or a big table and the data data services available so that your data is made available to the consumer integrations via big query or a big table then the next one is the data mesh design so data mesh as you all know it talks about decentralizing the data at a domain level so instead of uh putting uh in a contrary to the warehouse design where you had a data centralized the data was at your center and the the consumers were revolving around
your data right so unlike to that data mesh talks about decentralizing the data at a domain level so if you see this design there are various domains here defined and these are the producers and these are the consumer layers so you can have the data segregated at the domain level and each of the domain would have its own integrations from source to the consumers have own processing and the transformation logic and your consumers would access the data to the uh to your producer or the at a domain level so these are uh there are two ways you can have the consumers integrated so one is through authorized access and authorized views so here if you see the the big query is being used to store all the curated data and big query has a
feature of authorized views it can be defined either authorized views authorized functions or authorized data set so if you give access at authorized data set label your consumer would be able to access various objects present within the data set unable to access it or you can do it at a view level where you define the target layer views and have the access given as authorized views and if you see it here the consumers are nothing but the various level of a consumer applications through which these data can be accessed so for example the reporting is the is a one level uh type of a consumer where data can be accessed through bi tools and as we know any of the bi or tool for example here we are talking about looker so it
internally runs a set of queries to fetch in the data from a big query so these authorized views can be used to build in these dashboards and data would be faced for through these authorized views so the underlying data won't be accessible to the consumer layer the same goes with any of the downstream analytical applications where any of the consumer applications running on let's say running on data prop or data flow or any other level of a business application which treat this as a source data it can be accessed through the authorized views or the data sets then the next one is through the read api so this may not be the most recommended way to do it but this is also one of the way where you define the read apis
and you give access to the users based on to the role based policies and the access will be given to the specific users to the apis so that they can hit in your big query underlying data and process it so this can be again used at a machine learning applications or the analytical layer application the next one is the design patterns uh the governance so this is the most important part of the application design where these five pillars of the governance can be implemented so gcp offers a service dataplex which helps to define a unified data management and a governance control so you can define uh the policies at a dataplex level which can be applied to the warehouses data lakes or data mods so
this is the sample uh use case to implement a data catalog so data catalog can be implemented at a technology level and a business level so this is how you can implement a data catalog using dataplex and data governance can be implemented using dataplex so this is all i had uh for today's session so i'll take a pause here that is there is still one more minute remaining i'll open up for any questions if you have please uh feel free to drop in the question in chat or a q and a box here i'll be happy to answer them okay thank you so much for joining in the session so gcp as i said gcp is rich in data services and the solutions so when i talk about implementing any of the data
applications gcp is uh one of the hyperscaler that i would definitely prefer to use given given the various uh support to the various types of data and the platform integrations like lake or warehouse or a data mesh
Transcribed automatically; names checked against the speaker record. Select a time to jump there.
Article based on this talk
How to pick the right Google Cloud pattern for a data-intensive application
Pooja Kelgaonkar turns three design pillars into reference architectures for ETL, ELT, warehouses, lakes, data mesh and governance on GCP.
In short
Pooja Kelgaonkar argues that designing a data-intensive application on Google Cloud Platform should start with three pillars, not with a product list: the foundation of the data system, distributed data and derived data. Once those are clear, she shows how ETL, ELT and hybrid approaches map onto specific GCP services, and how those services combine into reference designs for a data warehouse, a data lake, a data mesh and a governance layer built on Dataplex.
Key takeaways
This session helps the audience understand design patterns to build data-intensive applications using Google Cloud Platform.
Three pillars come before any service
Kelgaonkar, who brings 16 plus years in the data domain, opens with the questions an architect should answer before choosing tools. The first pillar is the foundation of data systems: whether the application is reliable, scalable and maintainable, what data models and query languages it relies on (relational or non-relational, SQL or NoSQL), and whether the data is transactional or analytical.
The second pillar is distributed data. Here she covers replication for high availability and disaster recovery, partitioning for performance-efficient access, the kinds of transactions running on the platform, and consistency. Consistency, she stresses, is a contract rather than a default: "Depending on the type of application you have and the type of consumers you have, and the agreement between your application and your consumers, you define the consistency policies." (3:17)
The third pillar is derived data: whether data arrives as batch, event streams or database streams, and what analytics it will feed, from descriptive reporting to predictive models in what she calls the era of generative AI. She sums up the framework plainly: "So these are the three important pillars of the data intensive application design. So the first one is the foundation of data system, the second one is the distributed data, and the third one is the derived data." (4:13)
ETL, ELT or both
With the pillars in place, Kelgaonkar describes the three approaches used most widely in data application design. ETL extracts data and transforms it on the fly before it lands in the platform. ELT loads data first into a raw or staging layer and then runs transformation jobs to prepare it for analytics. The third option combines the two: "And the third one is the hybrid where you end up using mix of both ETL and ELT, depending on the heterogeneous data sources you have depending on the integrations available with the data sources and the kind of data you want to process." (5:14)
She then groups GCP's data services by storage, analytics, AI/ML and serving, noting that the serving layer is what lets mobile, web and BI applications fetch data. For teams without data scientists, she points to managed machine learning: "So GCP also offer lots of auto ML services, where it is just a kind of a plug and play, you may not need to have, you know, ML models to be developed." (6:36)
Mapping patterns to GCP services
For ETL, Kelgaonkar shows Dataflow and Data Fusion as the transformation engines, integrating heterogeneous sources and pushing results to a lake, a warehouse or the storage layer. For ELT, she highlights BigQuery: "So here is the one of the most commonly and widely used architecture that is using BigQuery for the ELT implementation." (8:03) A second ELT reference comes from Hadoop migrations, where Spark jobs are converted to run on ephemeral Dataproc clusters that read from storage, process the data and write it back to storage, the warehouse or Bigtable.
On analytics, she separates predictive from descriptive work. Vertex AI can run an automated training cycle that retrains and redeploys a model as new data arrives, Looker handles dashboards and reports, and AutoML offers ready models for natural language processing, translation, vision and text to speech.
Warehouse, lake and mesh designs
The core of the session is platform design. In the warehouse pattern, BigQuery sits at the center, with processing done natively in BigQuery or through Dataflow, Data Fusion or Dataproc, so ETL and ELT can coexist. Machine learning can be added three ways: Vertex AI, AutoML or BigQuery ML, where models are created with CREATE MODEL statements and called with simple SELECT statements. Consumers connect through Looker or third-party tools such as Tableau and Power BI.
The data lake pattern moves Cloud Storage to the core, with streaming handled by Pub/Sub and Dataflow and batch data loaded into source buckets. Data marts in BigQuery or Bigtable then serve consumers.
The data mesh pattern breaks with centralization. In Kelgaonkar's words, "data mesh talks about decentralizing the data at a domain level" (15:21), with each domain owning its integrations, processing and transformation logic from source to consumer. Consumers reach curated data through BigQuery authorized views, functions or datasets, so dashboards and downstream applications get what they need while "the underlying data won't be accessible to the consumer layer" (17:00). Read APIs with role-based access are a second option, though she notes it may not be the most recommended way.
Governance as the final layer
Kelgaonkar calls governance the most important part of application design. Her answer on GCP is Dataplex, which she describes as a service that "helps to define a unified data management and a governance control" (17:46). Policies defined at the Dataplex level apply across warehouses, lakes and data marts, and the same service supports a data catalog at both a technology level and a business level. She closes by explaining why GCP is the hyperscaler she would prefer for data applications: its support for many data types and for lake, warehouse and data mesh platform integrations.
Watch the talk
Build Data Intense Applications using Google Cloud Platform, Pooja Kelgaonkar, 21 minutes: https://www.womentech.net/video/build-data-intense-applications-using-google-cloud-platform
About the speaker
Pooja Kelgaonkar is a Senior Data Architect at Rackspace Technology with strong data warehousing skills and a specialism in cloud migrations that implement DWBI solutions on cloud platforms. She designs and delivers legacy migrations to Google Cloud Platform and has experience with large projects and heterogeneous infrastructures.
Questions this talk answers
What are the pillars of data-intensive application design?
When should you use ETL, ELT or a hybrid approach?
Which Google Cloud services implement ETL and ELT patterns?
How can machine learning be added to a BigQuery data warehouse?
How do consumers access data in a data mesh on GCP?
Drafted with AI from the talk transcript. Editor review pending.
Frameworks and ideas mentioned
Three pillars of data-intensive application design
ETL
ELT
Hybrid ETL/ELT
Data warehouse
Data lake
Data mesh
Lakehouse
Cite this talk
Pooja Kelgaonkar. "Build Data Intense Applications using Google Cloud Platform." WomenTech Network, 2023. https://www.womentech.net/video/build-data-intense-applications-using-google-cloud-platform