Cut through Big Data hype and extract real business value from new data sources

Today, Big Data is generally being defined by the volumes of data coming from the Internet, with social media now a primary source.  The amount of data in this case is way beyond “big”.  Adding unstructured data to the mix has driven the gigabyte/terabyte world to zettabytes and beyond. Now we are talking really “big.”

In the utility industry, our world is a byte or two smaller.  Big Data here is defined by the data coming from new sources in the Smart Grid.  Volumes here are large compared to our industry’s experience, but nowhere near public Internet data volumes.  Further, most of this data is structured in nature. 

Nevertheless, our challenge is significant.  Legacy systems and approaches are not sufficient to the task.  Major investments in technologies and processes are required to assure that timely, accurate data is managed and accessible to the enterprise.

Sampling rates for data acquisition are much higher than in the past. Synchro-Phasor data is sub-cycle and generated by hundreds of phasor measurement units.  Each quantity is necessarily time stamped, so time-series data bases will supplant relational databases for this reason. Similarly, Smart Meter data, from millions of meters, will dramatically add volumes to the “meter-to-cash” data flows of today and will also demand new approaches to warehousing these new databases for utilities.

While these examples and others from the Smart Grid generate significantly larger data volumes than our legacy sources, the volumes are simply not “Big Data”.  Our industry over-hyped “Smart Grid” while under communicating its value proposition to customers.  As a result, we have experienced cases of both regulatory and customer push-back that may limit our ability to fully exploit the true value of the Smart Grid.  Let’s not do that this time with “Big Data”.  Leave that hype to other arenas.

Our focus in the utility industry should be upon extracting real business value from our new data sources.  So how do we do that?

First, data management today must move beyond yesterday’s data silos.  Data is an enterprise resource.  Thus, data will be integrated across multiple sources and processes within a utility in order to fully unlock information and value from the data.  Also, data quality and cleansing considerations have to be addressed before the data is stored and made accessible to the enterprise.

Second, new “analytics” will be deployed as the newly enriched data resources have much hidden information to be unlocked.  While horizontal industry business intelligence (BI) tools, such as dashboards, will do much for enterprise situational understanding, new utility industry vertical analytics will be the real key to converting data into information, knowledge, and empowerment for the enterprise staff.  Processes will be transformed leading to improved and faster decision making.

The “Smart Enterprise” is before us.  Let’s undertake and fulfill the mission without the hype.  Better to deliver results well above expectations than under deliver against inflated promises. This works well for industries just as it has for individuals.