Scaling Genomic Workflows with Spark SQL BGEN and VCF Readers

Posted Leave a commentPosted in Announcements, Apache Spark, BGEN, Ecosystem, Engineering Blog, Genomics, HLS, Spark SQL, VCF

In the past decade, the amount of available genomic data has exploded as the price of genome sequencing has dropped. Researchers are now able to scan for associations between genetic variation and diseases across cohorts of hundreds of thousands of individuals from projects such as the UK Biobank. These analyses will lead to a deeper […]

Accurately Building Genomic Cohorts at Scale with Delta Lake and Spark SQL

Posted Leave a commentPosted in Apache Spark, Delta, Delta Lake, Ecosystem, Engineering Blog, Genomics, HLS, Joint Genotyping, SparkSQL

This is the second post in our “Genomic Analysis at Scale”  series.  In our first post, we explored a simple problem: how to provide real-time aggregates when sequencing large volumes of genomes. We solved this problem by using Delta Lake and a streaming pipeline built using Spark SQL. In this blog, we focus on the more advanced […]

Simplifying Genomics Pipelines at Scale with Databricks Delta

Posted Leave a commentPosted in Apache Spark, data pipeline, Engineering Blog, Genomics, HLS, Machine Learning, Streaming

Try this notebook in Databricks This blog is the first blog in our “Genomics Analysis at Scale” series. In this series, we will demonstrate how the Databricks UAP4Genomics enables customers to analyze population-scale genomic data. Starting from the output of our genomics pipeline, this series will provide a tutorial on using Databricks to run sample […]