Apache Spark 2.3 with Native Kubernetes Support

Posted Leave a commentPosted in Apache Spark, Ecosystem, Engineering Blog, Kubernetes, Machine Learning, Structured Streaming

This is a community blog from Anirudh Ramanathan and Palak Bhatia, software engineer and product manager respectively at Google, working in the Kubernetes team. They are part of the group of companies that contributed to native Kubernetes support for the Apache Spark 2.3. This post is cross-posted on blog.kubernetes.io Kubernetes and Big Data The open […]

Introducing Apache Spark 2.3 – The Databricks Blog

Posted Leave a commentPosted in Apache Spark, Databricks Runtime, Engineering Blog, Machine Learning, Streaming

Today we are happy to announce the availability of Apache Spark 2.3.0 on Databricks as part of its Databricks Runtime 4.0. We want to thank the Apache Spark community for all their valuable contributions to Spark 2.3 release. Continuing with the objectives to make Spark faster, easier, and smarter, Spark 2.3 marks a major milestone […]

Accelerate Innovation with Microsoft Azure Databricks

Posted Leave a commentPosted in Announcements, Azure, Company Blog, Engineering Blog, Events, Partners, Product, Webinar

It’s hard to believe that we are already three weeks into 2018. If you’re still struggling to get valuable insights from your data, now is the perfect time to try something new! We recently announced Azure Databricks, a fast, easy and collaborative Apache® Spark™ based analytics platform optimized for Azure. With Azure Databricks, you can help your […]

Basic components of Hadoop Architecture & Frameworks used for Data Science

Posted Leave a commentPosted in Data Science, Hadoop

Every business now recognizes the power of Big Data Analytics in developing deep actionable insights to enjoy business advantages. However, unlike before when businesses were required to deal with gigabytes of data, the present scenario requires to store and process huge piles of data that is measured in petabytes and terabytes as it is produced […]

Matei Zaharia’s 5 predictions about big data and AI in 2018

Posted Leave a commentPosted in Announcements, Company Blog, Engineering Blog, Events, Predictions, Webinar

Over the past few years, the demand for artificial intelligence (AI) and machine learning capabilities has surged with innovations in natural language processing, task automation, and predictions. From autonomous cars to a more personalized shopping experience, big data and artificial intelligence is at the forefront of new solutions that are delighting customers, improving business operations […]

Meltdown and Spectre: Exploits and Mitigation Strategies

Posted Leave a commentPosted in Apache Spark, Engineering Blog, Enterprise Security, Security

  In an earlier blog post, we analyzed the performance impact of Meltdown and Spectre on big data workloads in the cloud. In this blog post, we explain these exploits, their mitigation strategies and how they impact Databricks from a security and performance perspective. Meltdown Meltdown breaks a fundamental assumption in operating system security: an […]