Stephen Skinner, CIO, First Team Real Estate

The exponential growth of big data has led to significant challenges when it comes to finding meaning hidden within the patterns of large, unstructured datasets. By placing data into a visual context, data visualization technologies have the potential to help one explore and discover the structure and patterns of today’s seemingly endless streams of new data.

As a method of visual communication, data visualization is not new. There are many widely published examples from the 18th-19th centuries including the Periodic Table of the Elements, Charles Darwin’s Tree of Life, Charles Minard’s statistical graphic of Napoleon’s Russian Campaign and others. Automated data visualization technologies are relatively new, and have come into broad use in recent years. Early examples of automated data visualization include 3D Finite Element Modeling and Computational Fluid Dynamics. Processing large datasets of this type requires significant technical expertise and fast, large-scale computing systems such as massively parallel supercomputing arrays. National supercomputing centers were established in the 1980’s to encourage the development and adoption of data visualization. Many of the early projects in this area were related to engineering, defense, molecular modeling, simulation and the production of 3D computer animation. Educational institutions supporting these facilities were often responsible for the original development of supercomputer visualization software as well. The University of Illinois at Urbana-Champaign was one of these centers and continues to be a leading educational center for Data Visualization. 

It goes without saying that in the late-twentieth century, data visualization was still a very costly and exotic discipline practiced quite literally by rocket scientists and Hollywood visual effects artists.

We are now in the Big Data Century and one of today’s key business challenges is to make effective use of the data that is available from various automated systems and sources. Machine Data and the Internet of Things are driving the rapid growth of automated data generation from a multitude of sources. While business intelligence systems and data dashboards have the potential to help interpret and present these seemingly endless data streams, effective implementation of data visualization strategies can be complex and challenging. Large complex datasets are cumbersome and challenging to work with, data format standards remain elusive, and presentation standards for data visualization are varied. Data visualization is a form of visual communication, and despite the fact that machine data by its very nature is automatically generated some of the best examples of effective data visualization are the result of digital media collaboration between skilled infographic designers and data scientists.

Enter the modern era, circa 2016. Data visualization is everywhere and nowhere. Despite the fact that infographics and data dashboards are common in some enterprise applications, the widespread use of data visualization to support business and operational intelligence remains elusive. Today most visualization projects require skilled data science and sophisticated visual design. Many, if not most organizations simply do not have these capabilities in-house.

With the advent of cloud computing, the processing challenges of visualizing big data can be readily addressed. Large scale computing resources are now available in the cloud and can be sized to accomplish complex computational tasks at scale. Cloud services from leading vendors provide applications and infrastructure as a service that can readily ingest, compute and visualize machine data from a variety of sources as well. 

With the advent of cloud computing, the processing challenges of visualizing big data can be readily addressed .

Examples of this include solutions in Network Performance Monitoring and the Security Information and Event Management space.

In daily life, some of the best examples of data visualization appear in devices and applications in the consumer sector. The Internet of Things and gamification have resulted in many devices, apps and services for consumers that use simple, sophisticated visual formats to present complex information in an easy to comprehend manner. Combined with the consumerization of IT, many of these technologies and solutions have found their way into the enterprise. While this development has generally been positive, enterprise applications commonly lag far behind the consumer space when it comes to effective, easy to use data visualization applications and services. That said, it is encouraging to note that many ‘next generation’ cloud services for the enterprise have begun to incorporate new visual standards for the presentation of complex data.

In practical use, the human element remains a key factor in the effective use of data visualization technologies. Rarely are tasks accomplished on a purely automatic basis, and even advanced data visualization systems and services can require a labor intensive setup and customization. Software developers, data scientists and visual designers can be the key to an effective data visualization strategy in order to resolve issues with data format, statistical analysis and information design.

There is great potential for the continued adoption and development of data visualization systems that bring order to the seemingly chaotic and unstructured landscape that is the dark side of big data. Continued advances in Artificial Intelligence, Machine Communications and Machine Learning will likely result in simpler solutions for data visualization and greater adoption in the enterprise. Or, to put it another way: A picture is still worth a thousand words. Data visualization in the enterprise has a great deal untapped potential to help reveal the big picture hidden within big data.