Databricks & Apache Spark for Data Engineers: Build Production-Ready Data Pipelines
This course is designed for intermediate data engineers looking to master Apache Spark and the Databricks platform. You will learn to design, build, optimize, and deploy robust, production-ready data pipelines for batch and streaming data, leveraging the power of Delta Lake and best practices for performance and reliability.
Author:Ștefan Iliescu
You will learn:
- Basic Python programming knowledge
- Basic SQL knowledge
- Familiarity with databases and data pipelines
- Basic understanding of cloud computing concepts
- No prior Databricks or Apache Spark experience required
Course curriculum
- 1
Introduction to Databricks & Apache Spark for Data Engineering
4 lessons25 minutesExplore the core components of Apache Spark and the Databricks platform, understanding their roles in modern data engineering workflows and the Lakehouse architecture.
- The Role of Spark and Databricks in Data Engineering7 minutes
- Understanding the Apache Spark Ecosystem6 minutes
- Introduction to the Databricks Lakehouse Platform6 minutes
- Quiz: Spark & Databricks Fundamentals6 minutes
- 2
Databricks Platform Fundamentals and Workspace Management
4 lessons25 minutesNavigate the Databricks workspace, manage clusters, and utilize notebooks for data exploration, development, and collaborative engineering.
- Databricks Workspace Overview and Navigation7 minutes
- Cluster Management and Configuration Strategies6 minutes
- Working with Databricks Notebooks and Repos6 minutes
- Quiz: Databricks Interface & Tools6 minutes
- 3
Apache Spark Core Concepts for Data Engineers
4 lessons25 minutesDelve into Spark's fundamental concepts, including DataFrames, the Catalyst Optimizer, and execution strategies critical for efficient pipeline design.
- Spark Architecture and Execution Model Deep Dive7 minutes
- Understanding Spark DataFrames and Schema Enforcement6 minutes
- The Catalyst Optimizer and Tungsten Engine in Action6 minutes
- Quiz: Spark Core Internals6 minutes
- 4
Building Robust Batch Data Pipelines with Spark
4 lessons25 minutesLearn to design and implement robust batch ETL pipelines using Apache Spark, covering data ingestion, complex transformations, and efficient data loading.
- Ingesting Data from Various Source Systems7 minutes
- Common Data Transformation Patterns in Spark6 minutes
- Loading Processed Data to Different Sink Systems6 minutes
- Quiz: Batch Pipeline Development6 minutes
- 5
Introduction to Spark Structured Streaming for Real-time Data
4 lessons25 minutesExplore the principles of real-time data processing with Spark Structured Streaming, building continuous data pipelines for near real-time analytics.
- Streaming Data Concepts and Challenges7 minutes
- Spark Structured Streaming Fundamentals and Architecture6 minutes
- Building a Basic Structured Streaming Pipeline6 minutes
- Quiz: Structured Streaming Basics6 minutes
- 6
Delta Lake for Reliable and Performant Data Lakes
4 lessons25 minutesMaster Delta Lake's capabilities for building reliable, high-performance data lakes on Databricks, ensuring data quality, consistency, and governance.
- Introduction to Delta Lake and Lakehouse Principles7 minutes
- ACID Transactions, Schema Enforcement, and Evolution6 minutes
- Time Travel, Upserts, and Data Versioning with Delta6 minutes
- Quiz: Delta Lake Features6 minutes
- 7
Data Pipeline Deployment, Orchestration, and Monitoring
4 lessons25 minutesLearn strategies for deploying, scheduling, and orchestrating Spark data pipelines in a production environment using Databricks Jobs and external tools, along with effective monitoring techniques.
- Databricks Jobs: Scheduling and Automation for Production7 minutes
- Monitoring and Logging Spark Applications6 minutes
- Introduction to External Orchestration Tools (e.g., Airflow)6 minutes
- Quiz: Deployment & Orchestration6 minutes
- 8
Performance Tuning and Best Practices for Spark Pipelines
4 lessons25 minutesOptimize your Spark applications for performance and cost efficiency, identifying common bottlenecks, applying best practices, and leveraging advanced optimization techniques.
- Understanding Spark Shuffle and Data Skew Issues7 minutes
- Caching, Broadcast Joins, and Adaptive Query Execution6 minutes
- Resource Management and Cost Optimization on Databricks6 minutes
- Quiz: Performance Optimization6 minutes
Instructor:
Course rating:
No ratings or reviews yet.