Introducing Databricks Optimized Auto-Scaling – The Databricks Blog

Posted Leave a commentPosted in Announcements, Auto-scaling, autoscaling, Company Blog, Data Engineering, Databricks, Engineering Blog, Product

Databricks is thrilled to announce our new optimized auto-scaling feature. The new Apache Spark™-aware resource manager leverages Spark shuffle and executor statistics to resize a cluster intelligently, improving resource utilization. When we tested long-running big data workloads, we observed cloud cost savings of up to 30%. What’s the problem with current state-of-the-art auto-scaling approaches? Today, […]