Data Scalability Raises Considerable Risk Management Concerns

Posted Leave a commentPosted in Analytics, Best Practices, Big Data, Data Management, Data Quality, data scalability, Risk Management, SmartData Collective Exclusive

Are you frustrated by an increase in the quantity of the data that your organization handles? Well, you aren’t alone! Many businesses globally are dealing with big data which brings along a mix of benefits and challenges. A report by China’s International Data Corporation showed that global data would rise to 175 Zettabyte by 2025. […]

Accurately Building Genomic Cohorts at Scale with Delta Lake and Spark SQL

Posted Leave a commentPosted in Apache Spark, Delta, Delta Lake, Ecosystem, Engineering Blog, Genomics, HLS, Joint Genotyping, SparkSQL

This is the second post in our “Genomic Analysis at Scale”  series.  In our first post, we explored a simple problem: how to provide real-time aggregates when sequencing large volumes of genomes. We solved this problem by using Delta Lake and a streaming pipeline built using Spark SQL. In this blog, we focus on the more advanced […]

VR Websites: A Reality?

Posted Leave a commentPosted in AR / VR

Virtual reality or VR refers to using computer technology for creating a simulated environment. Virtual reality places the user inside experience, and instead of watching on a screen, the users get immersed and can carry out interactions with the 3D world. Through virtual reality, many senses are simulated, including the sense of vision, hearing, touch […]

Big Data Advances Lead to More Optimal SEO-Predicated Hosting

Posted Leave a commentPosted in Big Data, big data helping hosting companies, Hadoop, SmartData Collective Exclusive

Big data is a very important part of any digital marketing strategy. There are a number of reasons that machine learning, data analytics and Hadoop technology are changing SEO: Machine learning is becoming more widely used in search engine algorithms. SEOs that use machine learning can partially reverse engineer these algorithms. Big data helps SEO […]

Simplifying Streaming Stock Analysis using Delta Lake and Apache Spark: On-Demand Webinar and FAQ Now Available!

Posted Leave a commentPosted in ACID Transactions, Apache Spark, Company Blog, Delta Lake, Education, Engineering Blog, Financial Services, Product, Streaming, Structured Streaming, Time Travel, Unified Batch and Streaming Sync

On June 13th, we hosted a live webinar — Simplifying Streaming Stock Analysis using Delta Lake and Apache Spark — with Junta Nakai, Industry Leader – Financial Services at Databricks, John O’Dwyer, Solution Architect at Databricks, and Denny Lee, Technical Product Marketing Manager at Databricks. This is the first webinar in a series of financial […]

Connecting MongoDB to Ruby with Self-Signed Certificates for SSL

Posted Leave a commentPosted in Cloud, Technical

Given the popularity of our post on connecting MongoDB SSL with Self-Signed Certificates in Node.js, we decided to write a tutorial on connecting MongoDB with Ruby. In this blog, we’ll show you how to connect to a MongoDB server configured with self-signed certificates for SSL using both the Ruby MongoDB driver and the popular Object-Document-Mapper (ODM) mongoid. ScaleGrid currently uses […]

How to Increase Diversity in the Tech Workplace

Posted Leave a commentPosted in Big Data

Diversity in the workplace is something that all tech companies should strive for. When appropriately embraced in the tech sector, diversity has been shown to increase financial performance, increase employee retention, foster innovation, and help teams to develop better products. For example, data marketing teams that have equitable hiring practices in regards to gender exemplify […]

Detecting Bias with SHAP – The Databricks Blog

Posted Leave a commentPosted in Apache Spark, Bias, Deep Learning, Education, Engineering Blog, Machine Learning, MLflow, SHAP, Stack Overflow

StackOverflow’s annual developer survey concluded earlier this year, and they have graciously published the (anonymized) 2019 results for analysis. They’re a rich view into the experience of software developers around the world — what’s their favorite editor? how many years of experience? tabs or spaces? and crucially, salary. Software engineers’ salaries are good, and sometimes […]

Data Scalability Leads To New Evolutions In Smart Technology

Posted Leave a commentPosted in Big Data, data scalability, SmartData Collective Exclusive, technology

Big data is changing the nature of the world we live in. It has created a number of new forms of smart technology, which are simplifying our lives in wonderful ways. The classic example is with smart homes. You can probably remember watching movies about smart houses on the Disney Channel and other networks over […]