As indicated in the Deloitte 2017 annual Technology Trends report dark analytics is recognized as one of the disruptive technologies that will disrupt businesses in the next 18-24 months.
Data and the insights derived from them are multiplying at an incredible rate. It is estimated that 90% of all existing data is generated during the last five years. The digital universe with the data we create and copy is doubling in size every 12 months. Its size is expected to be 44 zettabytes in 2020. This results in a digital universe that almost contains as many digital bits as stars in our universe (Krambles, Roma, Mittel & Sharma, 2017).
According to Tom Coughlin (2017) we can divide data into light, grey and dark data. Light data consists all the data that is well-known, structured and readily available. Grey data is data that is only accessible to those player with the means and contact to access. Lastly, dark data includes all the data that is not readily accessible and requires modern data analysis tools to be exposed.
Dark analytics is especially focusing on the last form of data, in which they mine unstructured and inaccessible data sets with the use of modern data analysis tools. By tapping into this data, companies have the opportunity to turn unknown patterns and connections into forceful insights for the development of their market intelligence (Mittel, 2017). Therefore, the purpose of dark analytics is not to catalogue fast quantities of unstructured data but to derive actual insights which can be used in one’s own advantage (Deloitte UK, 2017).
In general, dark analytics is focusing on three dimensions of data (Krambles et al., 2017):
- The untapped data already in the possession of a company; this data can be divided into structured data and traditional unstructured data. Where the former consists of untapped data a company had not yet been able to find connections between, the latter focuses on text-based data in the form of e-mails, messages, word documents, pdf files, spreadsheets which do not take part of the relational database or tools and techniques to analyze them are not acquired yet.
- Non-traditional unstructured data; includes all data that cannot be mined by the use of traditional analytics techniques. Take for example audio files, video files or still images which require for example machine learning, advances pattern recognition, natural language processing and video and sound analytics to mine the data in non-traditional formats.
- Data from the deep web; all data that is part of the deep web, which is largest body of untapped information. It refers to all information from academics, government agencies, communities and other third parties of which its domain size and unstructured nature makes it difficult to analyze specific data.
An example is a project at the Copenhagen Airport in which the airport is collecting information by crunching the data in the log files or its Wi-Fi routers. Passenger’s smartphones ping routers in case they walk through terminals which offers data on the movement of passengers. This data could offer (commercial) insights on which shops are most visited on the airport (Chowdhury, 2017).
As also seen in the previous example, it is especially the growth of the Internet of Things with the expectation of 20.8 billion connected devices by 2020 which largely expands the volumes of data. And most of this expansion can be labeled as dark (Krambles et al., 2017). Therefore, the ability to access and analyze this data before your rivals results in a great opportunity and possible competitive advantage.
How do we capture this advantage and do not get lost in large volumes of dark data? According to Mittel (2017) it is essential to pay attention to dark analytics as a business strategy instead of a technology. This way, one stays grounded in business questions and will be able to define a scope and measure value. Focus especially on those areas which matter for your business and it will become unlikely to get lost in the dark unstructured bulk of data.
References:
- Krambles, P. Roma, P. Mittel, N. Sharma, S-K. (2017, 7 February). Dark Analytics: Illuminating opportunities hidden within unstructured data. Retrieved from https://dupress.deloitte.com/dup-us-en/focus/tech-trends/2017/dark-data-analyzing-unstructured-data.html
- Mittel, N. (2017, 7 February). Analyzing dark data for hidden opportunities. Retrieved from http://analytics-magazine.org/dark-analytics-shedding-light-new-business-asset/
- Coughlin, T. (2017, 24 July). – Analysis of Dark Data Provides Market Advantages. Retrieved from https://www.forbes.com/sites/tomcoughlin/2017/07/24/analysis-of-dark-data-provides-market-advantages/#6014bcbf872bChowdhury, A.P (2017, 11 May) – Shining light on Dark Analytics in the data driven age. Retrieved form http://analyticsindiamag.com/shining-light-dark-analytics-data-driven-age/
- Deloitte UK (2017, 11 April) Deloitte UK: Tech Trends 2017 – Dark Analytics [Video File]. Retrieved from https://www.youtube.com/watch?v=IWAcOTBYR4s
Cool topic and blog Tamara! I would like to contribute to your blog by stating that I think, before we are able to capture the value generated by Dark Analytics, it is important to first overcome its inherent risks. According to Chowdhury (2017), if organizations are to engage in Dark Analytics, it might be that the insights drawn from this unstructured, untapped or dark-web data compromise the organization’s regulatory compliance record – to conform to legislation is often why firms are required to hash data and divide data over different business units, rather than having one division see it all. Furthermore, if the organization suffers from a data breach, it might not only affect the customers of this organization, but also the organization itself, as they might also not fully comprehend the data they might be giving away to its competitors.
To overcome these risks might be a bit unrealistic, however, I think it is possible to provide organizations with the tools to cope with Dark Data and hedge themselves against these risks. If more research will be conducted to elaborate on how Dark Analytics works, its users will comprehend its consequences better. Furthermore, inter-firm policy development on the uses of this data might also prevent misuse of insights retrieved. Forbes actually wrote quite a nice article about data governance (you can find the link in the references if you would like to read the article :)). In this article, the author emphasizes the need for data governance and why it should be a priority to organizations (Forbes, 2016). Key message for me was: if organizations are able to collect your data, they should be able to protect your data. I think data governance will become even more relevant to firms if they will start with Dark Analytics.
Reference:
– Chowdhury, A. P.. 2017. Shining a light on Dark Analytics in the data-driven age. [ONLINE] Available at: http://analyticsindiamag.com/shining-light-dark-analytics-data-driven-age/. [Accessed 17 October 2017].
– Forbes. 2016. The Case For ‘Data Governance’. [ONLINE] Available at: https://www.forbes.com/sites/ciocentral/2016/06/22/the-case-for-data-governance/#47723d9454be. [Accessed 17 October 2017].