Analyze Games from European Soccer Leagues with Apache Spark and Databricks

Posted Leave a commentPosted in ad hoc analysis, Apache Spark, Data Engineering, Data Visualization, Education, Engineering Blog, ETL, Machine Learning, Platform, Product, Unified Analytics Platform

Try this notebook series in Databricks Introduction The global sports market is huge, comprised of players, teams, leagues, fan clubs, sponsors, etc., and all of these entities interact in myriad ways generating an enormous amount of data. Some of that data is used internally to help make better decisions, and there are a number of […]

Introducing Databricks Optimized Auto-Scaling – The Databricks Blog

Posted Leave a commentPosted in Announcements, Auto-scaling, autoscaling, Company Blog, Data Engineering, Databricks, Engineering Blog, Product

Databricks is thrilled to announce our new optimized auto-scaling feature. The new Apache Spark™-aware resource manager leverages Spark shuffle and executor statistics to resize a cluster intelligently, improving resource utilization. When we tested long-running big data workloads, we observed cloud cost savings of up to 30%. What’s the problem with current state-of-the-art auto-scaling approaches? Today, […]