Improving Big Data Analytics To Address Cybersecurity Challenges

Posted Leave a commentPosted in Big Data, cybersecurity, cybersecurity challenges, cybersecurity threats, Data Management, improving big data, Privacy, Risk Management, SmartData Collective Exclusive

Advances in mass storage and mobile computing brought about the phenomenon we now know as “big data.” These developments then ushered in solutions and tools that can process vast amounts of information — think terabytes of it or more — in real-time. That is how “big” the need for big data analytics came to be. […]

7 Powerful Open Source Tools For Data Projects

Posted Leave a commentPosted in Big Data, data projects, open source tools, SmartData Collective Exclusive

Regardless of if you’re a data science professional or an IT department who wants to help your company have more successful data science projects, it’s essential to have some data science tools under your belt to avail of when needed. Here are some open-source options to consider. Ludwig is a tool that allows people to […]

Leveraging Big Data With State-Of-The-Art Business Dashboards

Posted Leave a commentPosted in Big Data, business dashboards, Business Intelligence, dashboards, leveraging big data, SmartData Collective Exclusive

There are a lot of ways that organizations can leverage big data. Most of them don’t have difficulty collecting the data they need to make more informed decisions. However, they often struggle to conceptualize the data and present it in a format that supports their conclusions. This is one of the areas where a business […]

How Informatica Goes Hadoop-less with Databricks

Posted Leave a commentPosted in Apache Spark, Company Blog, Delta Lake, Hadoop, Hadoop migration, Partners

Back in May, we announced our partnership with Informatica to build out a rich set of integrations between our two platforms. It’s been exciting work for the team because of what we can do for joint customers that combine our Managed Delta Lake with Informatica’s Big Data Management and Enterprise Data Catalog.  The vision led […]

Democratizing Financial Time Series Analysis with Databricks

Posted Leave a commentPosted in Apache Spark, Data and ML Industry Use Case, Data Science and Machine Learning, Engineering Blog, Koalas, Machine Learning, MLflow, Time Series

Try this notebook in Databricks Introduction The role of data scientists, data engineers, and analysts at financial institutions includes (but is not limited to) protecting hundreds of billions of dollars worth of assets and protecting investors from trillion-dollar impacts, say from a flash crash. One of the biggest technical challenges underlying these problems is scaling […]