What’s New for Apache Spark on Kubernetes in the Upcoming Apache Spark 2.4 Release

Posted Leave a commentPosted in Apache Spark, Ecosystem, Engineering Blog, Kubernetes

This is a community blog from Yinan Li, a software engineer at Google, working in the Kubernetes Engine team. He is part of the group of companies that have contributed to Kubernetes support in the upcoming Apache Spark 2.4. Since the Kubernetes cluster scheduler backend was initially introduced in Apache Spark 2.3, the community has […]

A Guide to Developer, Apache Spark Use Cases, and Deep Dives Talks at Spark + AI Summit

Posted Leave a commentPosted in Apache Spark, Company Blog, Events, Kubernetes, Machine Learning, PySpark, Spark + AI Summit, Structured Streaming

Apache Spark is tackling new frontiers through innovations by unifying new workloads. This enables developers to combine data and AI to develop intelligent applications. Developers come to this summit not just to hear about innovations from contributors. They come to share their use cases, experiences, and absorb knowledge. @matei_zaharia just announced support of #kubernetes in […]

Introducing Click: The Command Line Interactive Controller for Kubernetes

Posted Leave a commentPosted in Databricks, Engineering Blog, Infrastructure, Kubernetes, Open Source, Platform, Unified Analytics Platform

Click is an open-source tool that lets you quickly and easily run commands against Kubernetes resources, without copy/pasting all the time, and that easily integrates into your existing command line workflows. At Databricks we use Kubernetes, a lot. We deploy our services (of which there are many) in unique namespaces, across multiple clouds, in multiple […]

Apache Spark 2.3 with Native Kubernetes Support

Posted Leave a commentPosted in Apache Spark, Ecosystem, Engineering Blog, Kubernetes, Machine Learning, Structured Streaming

This is a community blog from Anirudh Ramanathan and Palak Bhatia, software engineer and product manager respectively at Google, working in the Kubernetes team. They are part of the group of companies that contributed to native Kubernetes support for the Apache Spark 2.3. This post is cross-posted on blog.kubernetes.io Kubernetes and Big Data The open […]