As indicated in the Deloitte 2017 annual Technology Trends report dark analytics is recognized as one of the disruptive technologies that will disrupt businesses in the next 18-24 months.
Data and the insights derived from them are multiplying at an incredible rate. It is estimated that 90% of all existing data is generated during the last five years. The digital universe with the data we create and copy is doubling in size every 12 months. Its size is expected to be 44 zettabytes in 2020. This results in a digital universe that almost contains as many digital bits as stars in our universe (Krambles, Roma, Mittel & Sharma, 2017).
According to Tom Coughlin (2017) we can divide data into light, grey and dark data. Light data consists all the data that is well-known, structured and readily available. Grey data is data that is only accessible to those player with the means and contact to access. Lastly, dark data includes all the data that is not readily accessible and requires modern data analysis tools to be exposed.
Dark analytics is especially focusing on the last form of data, in which they mine unstructured and inaccessible data sets with the use of modern data analysis tools. By tapping into this data, companies have the opportunity to turn unknown patterns and connections into forceful insights for the development of their market intelligence (Mittel, 2017). Therefore, the purpose of dark analytics is not to catalogue fast quantities of unstructured data but to derive actual insights which can be used in one’s own advantage (Deloitte UK, 2017).
In general, dark analytics is focusing on three dimensions of data (Krambles et al., 2017):
- The untapped data already in the possession of a company; this data can be divided into structured data and traditional unstructured data. Where the former consists of untapped data a company had not yet been able to find connections between, the latter focuses on text-based data in the form of e-mails, messages, word documents, pdf files, spreadsheets which do not take part of the relational database or tools and techniques to analyze them are not acquired yet.
- Non-traditional unstructured data; includes all data that cannot be mined by the use of traditional analytics techniques. Take for example audio files, video files or still images which require for example machine learning, advances pattern recognition, natural language processing and video and sound analytics to mine the data in non-traditional formats.
- Data from the deep web; all data that is part of the deep web, which is largest body of untapped information. It refers to all information from academics, government agencies, communities and other third parties of which its domain size and unstructured nature makes it difficult to analyze specific data.
An example is a project at the Copenhagen Airport in which the airport is collecting information by crunching the data in the log files or its Wi-Fi routers. Passenger’s smartphones ping routers in case they walk through terminals which offers data on the movement of passengers. This data could offer (commercial) insights on which shops are most visited on the airport (Chowdhury, 2017).
As also seen in the previous example, it is especially the growth of the Internet of Things with the expectation of 20.8 billion connected devices by 2020 which largely expands the volumes of data. And most of this expansion can be labeled as dark (Krambles et al., 2017). Therefore, the ability to access and analyze this data before your rivals results in a great opportunity and possible competitive advantage.
How do we capture this advantage and do not get lost in large volumes of dark data? According to Mittel (2017) it is essential to pay attention to dark analytics as a business strategy instead of a technology. This way, one stays grounded in business questions and will be able to define a scope and measure value. Focus especially on those areas which matter for your business and it will become unlikely to get lost in the dark unstructured bulk of data.
References:
- Krambles, P. Roma, P. Mittel, N. Sharma, S-K. (2017, 7 February). Dark Analytics: Illuminating opportunities hidden within unstructured data. Retrieved from https://dupress.deloitte.com/dup-us-en/focus/tech-trends/2017/dark-data-analyzing-unstructured-data.html
- Mittel, N. (2017, 7 February). Analyzing dark data for hidden opportunities. Retrieved from http://analytics-magazine.org/dark-analytics-shedding-light-new-business-asset/
- Coughlin, T. (2017, 24 July). – Analysis of Dark Data Provides Market Advantages. Retrieved from https://www.forbes.com/sites/tomcoughlin/2017/07/24/analysis-of-dark-data-provides-market-advantages/#6014bcbf872bChowdhury, A.P (2017, 11 May) – Shining light on Dark Analytics in the data driven age. Retrieved form http://analyticsindiamag.com/shining-light-dark-analytics-data-driven-age/
- Deloitte UK (2017, 11 April) Deloitte UK: Tech Trends 2017 – Dark Analytics [Video File]. Retrieved from https://www.youtube.com/watch?v=IWAcOTBYR4s