Introducing New Built-in and Higher-Order Functions for Complex Data Types in Apache Spark 2.4

Posted Leave a commentPosted in Apache Spark, Apache Spark 2.4, Engineering Blog, SparkSQL, sql

Try this notebook in Databricks Apache Spark 2.4 introduces 29 new built-in functions for manipulating complex types (e.g., array type), including higher-order functions. Before Spark 2.4, for manipulating the complex types directly, there were two typical solutions: 1) Exploding the nested structure into individual rows, and applying some functions, and then creating the structure again […]

SQL Pivot: Converting Rows to Columns

Posted Leave a commentPosted in Apache Spark, DataFrames, Engineering Blog, Spark SQL, sql, Unified Analytics Platform

Try this notebook in Databricks Pivot was first introduced in Apache Spark 1.6 as a new DataFrame feature that allows users to rotate a table-valued expression by turning the unique values from one column into individual columns. The upcoming Apache Spark 2.4 release extends this powerful functionality of pivoting data to our SQL users as […]

Assessing the Severity of SQL Injection Threats to IoT Security

Posted Leave a commentPosted in Internet of Things, iot security, Security, SEO injection threats, SmartData Collective Exclusive, sql

The evolution of the IOT has changed the world in countless ways. Many people are still struggling to adapt to it. One of the biggest learning curves that most people face is trying to understand the security vulnerabilities that the IOT network faces. Unfortunately, SQL injections can be an even bigger danger to the IOT […]

A Standard Graph Query Language Could Be Coming—Here’s What To Know

Posted Leave a commentPosted in gql, graph query language, News, Programming, SmartData Collective Exclusive, sql

IT standards make everything better. It is not a stretch to say that coming together as a community and developing, evolving, and committing to standards make better, cheaper, and more secure software and systems. The International Standards Organization is considering a standard for Graph Query Language (GQL) which would ultimately allow for interoperability not only […]