CIO Survey: Top 3 Challenges Adopting AI and How to Overcome Them

Posted Leave a commentPosted in AI, Announcements, CIO, Company Blog, data, Events, ML, Product, Survey, Unified analytics

  We recently hosted the webinar — CIO Survey: Enterprise Challenges to AI and How to Overcome Them — featuring Jen Garofalo, Research Director at IDG, the parent company to CIO.com, and Pat McDonough, VP of Customer Success at Databricks. This webinar covered key findings from a recent CIO.com survey of 200 executives on the […]

Announcing Databricks Runtime 5.0 – The Databricks Blog

Posted Leave a commentPosted in Announcements, Apache Spark, Company Blog, Product

We’re excited to announce the general availability of Databricks Runtime 5.0. Included in this release is Spark 2.4. This release offers substantial performance increases within key areas of the platform. Benchmarking workloads have shown a 16% improvement in total execution time and Databricks Delta benefits from substantial improvements to metadata caching, improving query latency by […]

Simplifying Change Data Capture with Databricks Delta

Posted Leave a commentPosted in Apache Spark, CDC, Change Data Capture, Company Blog, Databricks Delta, Education, Engineering Blog, Product

A common use case that we run into at Databricks is that customers looking to perform change data capture (CDC) from one or many sources into a set of Databricks Delta tables. These sources may be on-premises or in the cloud, operational transactional stores, or data warehouses. The common glue that binds them all is […]

Simplify Market Basket Analysis using FP-growth on Databricks

Posted Leave a commentPosted in Apache Spark, Company Blog, Education, Engineering Blog, FP-Growth, Machine Learning, Market Basket Analysis, Product

Try this notebook in Databricks When providing recommendations to shoppers on what to purchase, you are often looking for items that are frequently purchased together (e.g. peanut butter and jelly). A key technique to uncover associations between different items is known as market basket analysis. In your recommendation engine toolbox, the association rules generated by […]

Identify Suspicious Behavior in Video with Databricks Runtime for Machine Learning

Posted Leave a commentPosted in Apache Spark, Company Blog, Deep Learning, Deep Learning Pipelines, Education, Engineering Blog, Machine Learning, OpenCV, Platform, Product, TensorFlow, Video Analytics

Try this notebook series in Databricks With the exponential growth of cameras and visual recordings, it is becoming increasingly important to operationalize and automate the process of video identification and categorization. Applications ranging from identifying the correct cat video to visually categorizing objects are becoming more prevalent.  With millions of users around the world generating […]

MLflow On-Demand Webinar and FAQ Now Available!

Posted Leave a commentPosted in Data Science, Deep Learning, Ecosystem, Engineering Blog, Machine Learning, MLflow, Model Management, Platform, Product, Unified Analytics Platform

On August 30th, our team hosted a live webinar—Introducing MLflow: Infrastructure for a complete Machine Learning lifecycle—with Matei Zaharia, Co-Founder and Chief Technologist at Databricks. In this webinar, we walked you through MLflow, a new open source project from Databricks that aims to design an open ML platform where organizations can use any ML library […]

Introducing Cluster-scoped init scripts – The Databricks Blog

Posted Leave a commentPosted in Cluster Management, DBFS, Engineering Blog, Platform, Product, Summer Internship, Unified Analytics Platform

Introduction This summer, I worked at Databricks as a software engineering intern on the Clusters team. As part of my internship project, I designed and implemented Cluster-scoped init scripts, improving scalability and ease of use. In this blog, I will discuss various benefits of Cluster-scoped init scripts, followed by my internship experience at Databricks, and […]

Announcing Databricks Runtime 4.3 – The Databricks Blog

Posted Leave a commentPosted in Announcements, Company Blog, Engineering Blog, Platform, Product, Runtime

I’m pleased to announce the release of Databricks Runtime 4.3, powered by Apache Spark.  We’ve packed this release with an assortment of new features, performance improvements, and quality improvements to the platform.   We recommend moving to Databricks Runtime 4.3 in order to take advantage of these improvements. In our obsession to continually improve our platform’s […]

Building a Real-Time Attribution Pipeline with Databricks Delta

Posted Leave a commentPosted in Adhoc Analysis, Advertising Analytics, Apache Spark, bi, Company Blog, Databricks Delta, Ecosystem, Education, Engineering Blog, Kinesis, Machine Learning, Platform, Product, Spark Streaming, Streaming, Structured Streaming, Tableau

Try this notebook in Databricks In digital advertising, one of the most important things to be able to deliver to clients is information about how their advertising spend drove results.  The more quickly we can provide this, the better. To tie conversions or engagements to the impressions served in an advertising campaign, companies must perform […]

Loan Risk Analysis with XGBoost and Databricks Runtime for Machine Learning

Posted Leave a commentPosted in Apache Spark, Company Blog, data pipeline, Data Visualization, Ecosystem, Education, Engineering Blog, financial, Machine Learning, MLlib, Platform, Product, XGBoost

Try this notebook series in Databricks For companies that make money off of interest on loans held by their customer, it’s always about increasing the bottom line. Being able to assess the risk of loan applications can save a lender the cost of holding too many risky assets. It is the data scientist’s job to […]