Introduction to Data Engineering on Google Cloud
Module 00: Course Introduction
Full name: Introduction to Data Engineering on Google Cloud
Product code: T-IDEGC-B
These links are based on the current course version.
Online course options:
- Introduction to Data Engineering on Google Cloud - Google Cloud Skills Boost
- Introduction to Data Engineering on Google Cloud - Coursera
Module 01: Data Engineering Tasks and Components
Websites
- BigQuery overview https://cloud.google.com/bigquery/docs/introduction
- BigQuery documentation https://cloud.google.com/bigquery/docs
- Cloud Storage documentation https://cloud.google.com/storage/docs
- Cloud Storage classes https://cloud.google.com/storage/docs/storage-classes
- Database options on Google Cloud https://cloud.google.com/products/databases
- Knowledge Catalog (Dataplex) https://cloud.google.com/dataplex/docs/introduction
- Analytics Hub overview https://cloud.google.com/bigquery/docs/analytics-hub-introduction
- Data formats overview https://cloud.google.com/bigquery/docs/loading-data#data_formats
Videos
- What is BigQuery? https://www.youtube.com/watch?v=m8WqxLd1jSc
- Analytics Hub in a minute https://www.youtube.com/watch?v=h8_xCbTYtTU
Module 02: Data Replication and Migration
Websites
- Storage Transfer Service https://cloud.google.com/storage-transfer-service
- Transfer Appliance documentation https://cloud.google.com/transfer-appliance/docs/4.0/overview
- Datastream overview https://cloud.google.com/datastream/docs/overview
- Database Migration Service https://cloud.google.com/database-migration
- gcloud storage command reference https://cloud.google.com/sdk/gcloud/reference/storage
- Change data capture with Datastream https://cloud.google.com/datastream/docs/concepts
Videos
- Introduction to Datastream for BigQuery https://www.youtube.com/watch?v=vMo6Zgkvt40
- What is Storage Transfer Service? https://www.youtube.com/watch?v=1oIw2mT_4-M
Module 03: The Extract and Load Data Pipeline Pattern
Websites
- Loading data into BigQuery https://cloud.google.com/bigquery/docs/loading-data
- BigQuery Data Transfer Service https://cloud.google.com/bigquery/docs/dts-introduction
- BigLake overview https://cloud.google.com/biglake/docs/introduction
- External tables in BigQuery https://cloud.google.com/bigquery/docs/external-tables
- Lakehouse on Google Cloud https://cloud.google.com/solutions/lakehouse
- bq command-line tool reference https://cloud.google.com/bigquery/docs/bq-command-line-tool
- LOAD DATA statement https://cloud.google.com/bigquery/docs/reference/standard-sql/other-statements#load_data_statement
- Querying Google Sheets data https://cloud.google.com/bigquery/docs/external-data-drive
Videos
- BigLake in a minute https://www.youtube.com/watch?v=UdpNYk0EP0g
- BigQuery Migration Service: SQL and data transfer https://www.youtube.com/watch?v=vY_m9dM6WxE
- Open Analytics Lakehouse on Google Cloud https://www.youtube.com/watch?v=_HmbwjODKgo
Module 04: The Extract, Load, and Transform Data Pipeline Pattern
Websites
- Dataform overview https://cloud.google.com/dataform/docs/overview
- Dataform quickstart https://cloud.google.com/dataform/docs/quickstart
- BigQuery scripting (procedural language) https://cloud.google.com/bigquery/docs/reference/standard-sql/scripting
- User-defined functions (UDFs) https://cloud.google.com/bigquery/docs/reference/standard-sql/user-defined-functions
- Stored procedures https://cloud.google.com/bigquery/docs/procedures
- Scheduled queries in BigQuery https://cloud.google.com/bigquery/docs/scheduling-queries
- Remote functions https://cloud.google.com/bigquery/docs/remote-functions
- BigQuery DataFrames (bigframes) https://cloud.google.com/bigquery/docs/bigquery-dataframes-introduction
- Apache Spark procedures in BigQuery https://cloud.google.com/bigquery/docs/spark-procedures
Videos
- Streamline Your SQL Workflows with Dataform https://www.youtube.com/watch?v=8pVM6iXq4Jw
Module 05: The Extract, Transform, and Load Data Pipeline Pattern
Websites
- Dataflow documentation https://cloud.google.com/dataflow/docs
- Apache Beam overview https://beam.apache.org/documentation/
- Dataflow templates https://cloud.google.com/dataflow/docs/concepts/dataflow-templates
- Google-provided Dataflow templates https://cloud.google.com/dataflow/docs/guides/templates/provided-templates
- Pub/Sub documentation https://cloud.google.com/pubsub/docs/overview
- Managed Service for Apache Spark (Dataproc) https://cloud.google.com/dataproc/docs
- Dataproc Serverless https://cloud.google.com/dataproc-serverless/docs
- Cloud Data Fusion https://cloud.google.com/data-fusion/docs
- Dataprep by Trifacta https://cloud.google.com/dataprep/docs
- Bigtable documentation https://cloud.google.com/bigtable/docs
Videos
- What is Dataflow? https://www.youtube.com/watch?v=KalJ0VuEM7s
- What is Pub/Sub? https://www.youtube.com/watch?v=cvu53CnZmGI
- Dataproc in a minute https://www.youtube.com/watch?v=Jj6mp7Sam10
- Data Science Workloads on Dataproc Serverless https://www.youtube.com/watch?v=dLcDBs9ilf0
- What is Cloud Bigtable? https://www.youtube.com/watch?v=P4q4nqJAamo
- Data Fusion in a minute https://www.youtube.com/watch?v=xjsNWh1TLKo
Module 06: Automation Techniques
Websites
- Cloud Scheduler https://cloud.google.com/scheduler/docs
- Cloud Workflows https://cloud.google.com/workflows/docs
- Cloud Composer (managed Airflow) https://cloud.google.com/composer/docs
- Cloud Run functions https://cloud.google.com/functions/docs
- Eventarc overview https://cloud.google.com/eventarc/docs/overview
- Dataproc Workflow Templates https://cloud.google.com/dataproc/docs/concepts/workflows/overview
Videos
- Cloud Composer in a minute https://www.youtube.com/watch?v=pSjD5nHkXLU
- Cloud Run in a minute https://www.youtube.com/watch?v=AL2rAmWFZjM
- Event-driven orchestration with Eventarc and Workflows https://www.youtube.com/watch?v=2SI12QE-2DU
- Getting started with Cloud Tasks and Cloud Scheduler https://www.youtube.com/watch?v=P9MCC9KmM_8
- Cloud Workflows in a minute https://www.youtube.com/watch?v=K_Gn0pZysLg
Module 07: Course Summary
Websites
- Google Cloud Data Engineering Learning Path https://cloud.google.com/learn/training/data-engineering-and-analytics
- Google Cloud Certification - Professional Data Engineer https://cloud.google.com/learn/certification/data-engineer
- Google Cloud Architecture Center https://cloud.google.com/architecture
- Google Cloud blog - Data Analytics https://cloud.google.com/blog/products/data-analytics