Delta Lake: The Definitive Guide

Discover how Delta Lake simplifies the process of building data lakehouses and data pipelines at scale. With this practical guide, data engineers, data scientists, and data analysts will explore key data reliability challenges and learn to apply modern data engineering and management techniques. You'll also understand how ACID transactions bring reliability to data lakehouses at scale. Authors Denny Lee, Prashanth Babu, Tristen Wentling, and Scott Haines explain how to harness the power of Delta Lake to increase your data productivity at scale. You'll learn how to run batch and streaming jobs concurrently on your data lake and accelerate the usability of your data by building effective and high-quality end-to-end pipelines, from data ingestion to analytics. This book helps you: Understand key data reliability challenges Examine data management and engineering techniques using the modern data stack Realize data reliability improvements using Delta Lake Concurrently run streaming and batch jobs against your data lake Execute update, delete, and merge commands Use time travel to rollback and examine previous versions of your data Build a streaming data quality pipeline following the medallion construct About the authors: Denny Lee is a Delta Lake maintainer and Apache Spark and MLflow contributor. Prashanth Babu is a Delta practitioner who works at Databricks. Tristen Wentling is a Delta practitioner who works at Databricks. Scott Haines is an Apache Spark and Delta Lake contributor who works at Nike.