Efficient Upserts into Data Lakes with Databricks Delta

Posted Leave a commentPosted in Announcements, Change Data Capture, Company Blog, Databricks Delta, Delta, Merge, Product

Simplify building big data pipelines for change data capture (CDC) and GDPR use cases. Databricks Delta, the next-generation unified analytics engine built on top of Apache Spark™, now supports the MERGE command, which allows you to efficiently upsert and delete records in your data lakes. MERGE dramatically simplifies how a number of common data pipelines […]

Simplifying Change Data Capture with Databricks Delta

Posted Leave a commentPosted in Apache Spark, CDC, Change Data Capture, Company Blog, Databricks Delta, Education, Engineering Blog, Product

A common use case that we run into at Databricks is that customers looking to perform change data capture (CDC) from one or many sources into a set of Databricks Delta tables. These sources may be on-premises or in the cloud, operational transactional stores, or data warehouses. The common glue that binds them all is […]