Introducing Databricks Optimized Auto-Scaling – The Databricks Blog

Posted Leave a commentPosted in Announcements, Auto-scaling, autoscaling, Company Blog, Data Engineering, Databricks, Engineering Blog, Product

Databricks is thrilled to announce our new optimized auto-scaling feature. The new Apache Spark™-aware resource manager leverages Spark shuffle and executor statistics to resize a cluster intelligently, improving resource utilization. When we tested long-running big data workloads, we observed cloud cost savings of up to 30%. What’s the problem with current state-of-the-art auto-scaling approaches? Today, […]

How Big Data Can Help Improve the U.S. Airport System

Posted Leave a commentPosted in Analytics, Big Data, big data analytics, big data improvements

United States air transportation firms have earmarked $75.7 billion for the implementation of infrastructure upgrades through the year 2019. The Airports Council International North America (ACI-NA) report “Airport Capital Development Needs: 2015-2019” outlines the necessity for the upgrades that air transportation firms have designated funding for so far. The firms plan to use the funds […]

How Big Data Technology Impacts Investments and Trading

Posted Leave a commentPosted in Big Data, big data helping businesses, Business Intelligence, SmartData Collective Exclusive

The world of online trading has been growing year on year, and it now offers traders/investors the ability to invest in almost any global market of their choosing. As such, the technology surrounding trading and investment is constantly being developed and improved to help traders with investment decisions. In recent years, there have been significant […]

Are Cryptocurrencies Disrupting the Economy in 2018?

Posted Leave a commentPosted in Business Rules, cryptocurrency, cryptocurrency payment solutions, cryptocurrency sharing economy, IT, Market Research, News, SmartData Collective Exclusive

Despite the hype around bitcoin and other cryptocurrencies, the digital forms of payment have thus far failed to challenge traditional and well-established methods. In fact, the use of credit cards is actually on the rise in Canada, with 61 percent of online shoppers using that option in 2017 compared to 51 percent three years earlier. […]

GDPR Implications of Desktop and Mobile Online Collaboration Platforms

Posted Leave a commentPosted in #GDPR, Big Data, Data Storage, GDPR implications, News, SmartData Collective Exclusive

The GDPR is the newest data privacy directive in the European Union. It is a very complex policy that is meant to address the vast privacy concerns that have arisen in the 23 years since the previous data privacy law was passed. Data scientists and legal experts are both debating the nuances of the new […]

Government Agencies Are Gaining New Data Collection Tools in 2018

Posted Leave a commentPosted in Big Data, data collection, Data Management, Policy and Governance

After 9/11, the way that the United States government handled data shifted. At first, all ears were directed outside of the country. Now, increasingly, we are seeing a shift to domestic data surveillance. This was highlighted in Edward Snowden’s massive data leak from the National Security Agency. It has been further confirmed by the discovery […]

How Tesla Is Using Big Data And AI

Posted Leave a commentPosted in Artificial Intelligence, Big Data

Elon Musk’s Tesla has long enjoyed a well-deserved spotlight in global headlines, largely thanks to the eccentric nature of the company’s founder and the unique allure of its electric vehicles. Despite all the hubbub surrounding recent gimmicks like sending a Tesla roadster hurtling towards Mars, however, most investors and tech enthusiast have a different reason […]