THE PROCESS OF BIG DATA SOLUTION ADOPTION

162 Pages·2013·2.8 MB·English

by  Verheij, Bas

THE PROCESS OF
BIG DATA SOLUTION ADOPTION
An exploratory study within the Dutch telecom and energy utility sector

Master Management of Technology
Faculty Technology, Policy and Management
Delft University of Technology
August 2013

Name: Bas Verheij
Student number: 1178938

1st supervisor: Dr. Laurens Rook
2nd supervisor: Prof.dr.ir. Jan van den Berg
Chair: Prof.dr. Cees van Beers

External company: Accenture
External supervisor: Drs. Paul van der Linden Msc 1
2
ACKNOWLEDGEMENT
This thesis covers the development of a conceptual big data solution adoption process model, which has been
designed using empirical findings from interviews with firms adopting big data solutions within the Dutch
telecommunications and energy utilities sector. This document presents the results of my research graduation
research to finalize the master program Management Of Technology (MOT) at Delft University of Technology.
Firstly I would like to thank Paul van der Linden from Accenture for giving me the opportunity to conduct my
research at Accenture and for helping me shape the research through the many meetings we had. Accenture
provided a very fruitful environment for developing knowledge on the topic of big data, and I am thankful to
Paul van der Linden and the DD&A department for giving me the opportunity to conduct my research among
large and interesting companies within the Netherlands.
Also I would like to thank my first supervisor Laurens Rook for supervising my thesis, and mainly for the support
and discussions during the process of my research. Also I would like to thank Jan van den Berg for his critical
and accurate feedback, this has greatly helped my learning experience on performing scientific research.
Thirdly I would to thank my friends who helped me during my time in Delft and my various study activities.
Especially I would like to thank the friends that helped me finish my Bachelor in Mechanical Engineering, which
ultimately made this graduation possible.
Finally I am very grateful to my family for supporting me through my life and my study period, and especially
for supporting me to persevere in my studies.

3
LIST OF ABBREVIATIONS

Abbreviation Term
ACID Atomicity, Consistency, Isolation, Durability
AWS Amazon Web Services
EC2 Amazon Web Services EC2
AWS EMR Amazon Web Services Elastic MapReduce
API Application Programming Interface
BASE Basically Available, Soft state, Eventual consistency
BI Business Intelligence
BI&A Business Intelligence & Analytics
CAP theorem Consistency, Availability, Partition tolerance
DM Data mining
DW Data Warehouse
ETL Extraction-Transformation-Loading
HDFS Hadoop Distributed File System
MVCC Multi-version Concurrency Control
OLAP Online Analytical Processing
RDBMS Relational Database Management System
DSS Decision Support System
DDD Data-driven Decisionmaking
IS Information System

4
5
TABLE OF CONTENTS
1 INTRODUCTION ...................................................................................................................................... 11
1.1 RESEARCH OBJECTIVE .................................................................................................................................. 13
1.2 RESEARCH QUESTIONS ................................................................................................................................ 13
1.3 RESEARCH FRAMEWORK .............................................................................................................................. 15
1.4 STRUCTURE OF THIS REPORT ......................................................................................................................... 15
2 BACKGROUND ........................................................................................................................................ 18
2.1 INTRODUCTION .......................................................................................................................................... 19
2.2 BUSINESS INTELLIGENCE & ANALYTICS AND BIG DATA ......................................................................................... 20
2.3 DATABASE TECHNOLOGY .............................................................................................................................. 24
2.4 BIG DATA SOLUTIONS .................................................................................................................................. 26
2.5 BIG DATA AND ORGANIZATIONS ..................................................................................................................... 31
2.6 THEORY: IT INNOVATION ADOPTION .............................................................................................................. 33
2.7 CONCLUSION: BI GENERATIONS AND ADOPTION ............................................................................................... 35
3 METHODOLOGY ...................................................................................................................................... 38
3.1 CASE-STUDY RESEARCH APPROACH JUSTIFICATION ............................................................................................ 39
3.2 CASE-STUDY RESEARCH STRATEGY .................................................................................................................. 41
3.3 PHASE OF PREPARATION .............................................................................................................................. 43
3.4 PHASE OF DATA COLLECTION ......................................................................................................................... 44
3.5 PHASE OF DATA ANALYSIS ............................................................................................................................ 46
3.6 DESIGN OF CONCEPTUAL MODEL ................................................................................................................... 49
3.7 CASE-STUDY VALIDITY CONCERNS .................................................................................................................. 51
4 BIG PRACTICES IN TWO INDUSTRIES ....................................................................................................... 54
4.1 ENERGY SECTOR OVERVIEW .......................................................................................................................... 55
4.2 TELECOM SECTOR OVERVIEW ........................................................................................................................ 65
4.3 OTHER SECTORS OVERVIEW .......................................................................................................................... 76
5 RESULTS OF THEMATIC ANALYSIS ........................................................................................................... 80
5.1 CASE STUDY RESULTS................................................................................................................................... 81
5.2 BUSINESS CASE DEVELOPMENT ...................................................................................................................... 84
5.3 SOLUTION CHOICE ...................................................................................................................................... 87
5.4 ORGANIZATIONAL CHANGE ........................................................................................................................... 91
5.5 INFORMATION PRIVACY ............................................................................................................................... 94
5.6 IMPLEMENTATION AND FINE-TUNING PROBLEMS IN BIG DATA TECHNOLOGY ........................................................... 95
6 CONCEPTUAL MODEL ............................................................................................................................. 98
6.1 CONSIDERATIONS: REQUIREMENTS FOR BIG DATA ADOPTION............................................................................... 99
6.2 BIG DATA ADOPTION PROCESS PHASES ............................................................................................................ 99
6.3 ISSUES PERCEIVED IN THE ADOPTION PROCESS ................................................................................................ 102
6.4 ADOPTION PROCESS MODEL ....................................................................................................................... 104
7 ANALYSIS .............................................................................................................................................. 106
7.1 IT INNOVATION ADOPTION (BUSINESS CASE DEVELOPMENT) ............................................................................. 107
7.2 RADICAL INNOVATION (ORGANIZATIONAL CHANGE) ........................................................................................ 109
7.3 CHANGES IN IS ARCHITECTURES (SOLUTION CHOICE)........................................................................................ 113
6
7.4 INFORMATION PRIVACY AS A MAIN DRIVER FOR IS CHANGE (PRIVACY) ................................................................. 115
8 REFLECTION .......................................................................................................................................... 120
8.1 REFLECTION ON RESEARCH RESULTS ............................................................................................................. 121
8.2 QUALITY OF THE RESULTS ........................................................................................................................... 121
8.3 REFLECTION ON RESEARCH PROCESS ............................................................................................................. 123
9 CONCLUSION ........................................................................................................................................ 124
9.1 MAIN FINDINGS ....................................................................................................................................... 125
9.2 RESEARCH IMPLICATIONS ........................................................................................................................... 128
9.3 MANAGERIAL IMPLICATIONS ....................................................................................................................... 131
9.4 RESEARCH LIMITATIONS ............................................................................................................................. 134
9.5 FUTURE RESEARCH .................................................................................................................................... 135
9.6 CLOSURE ................................................................................................................................................ 137
10 REFERENCES ...................................................................................................................................... 138
APPENDIX A: PLANNING ............................................................................................................................... 144
APPENDIX B.1: INTERVIEW PROTOCOL ......................................................................................................... 146
APPENDIX B.2: INTERVIEW CASE STUDY DESCRIPTION QUESTIONS ............................................................. 147
APPENDIX B.3: INTERVIEW PERCEPTIONS QUESTIONS ................................................................................. 148
APPENDIX D: SAMPLE DETAILS ..................................................................................................................... 149
D.1 GENERAL FIRM CHARACTERISTICS ...................................................................................................................... 149
D.2 CHARACTERISTICS FIRM CASES .......................................................................................................................... 151
D.3 GROUPING OF ADOPTION PROCESSES ................................................................................................................. 152
APPENDIX F: MOST FREQUENTLY USED PREDICTORS OF ORG. IT ADOPTION (JEYARAJ ET AL., 2006) ........... 153
APPENDIX G: LITERATURE RESEARCH ........................................................................................................... 157
APPENDIX H: DATABASE LANDSCAPE (AS OF JUNE 2013) ............................................................................. 158
APPENDIX I: NOSQL FUNDAMENTAL TECHNICAL CONCEPTS ........................................................................ 159
I.1 FUNDAMENTAL CONCEPTS ................................................................................................................................. 159
I.2 TYPES OF DATA MODELS .................................................................................................................................... 161
I.3 PERFORMANCE AND ELASTICITY .......................................................................................................................... 161

7
EXECUTIVE SUMMARY
Research Objective
Starting with the research objective: the main purpose of this research is to design a conceptual big data
solution adoption model by exploring the process of big data solution adoption within organizations. This
multiple-case study gives a thorough description of the adoption process of big data solutions and the main
issues organizations experience within this process.
Phenomenon
The term “big data” is primarily seen as an umbrella term used within the industry, clear definitions in scientific
literature have not been found at the moment of research. Gartner was the first firm within the industry to
name the big data phenomenon defining it as “challenges and opportunities in data growth” in 2001. Gartner
defined these challenges and opportunities in data growth as having three facets: increasing volume, velocity
and variety ("The three Vs"; Pettey & Goasduff, 2011). The McKinsey Global Institute published an industry
report in 2011, which uses an intentionally subjective definition to capture the essence of the difference
between data and big data. This definition will be used as a principle in this paper (Manyika, Chui, Brown, &
Bughin, 2011; p1): “Big data refers to datasets whose size is beyond the ability of typical database software
tools to capture, store, manage, and analyze.”
The big data phenomenon is accompanied by a new generation of databases – so called NoSQL databases –
which have the capability of distributed analyzing large datasets of unstructured data. These databases are
expected to be adopted within firms as the need of analysis of an increasing volume, velocity and variety of
data demands for these new database technologies.
Although big data is widely and wildly cited in industry papers, analyses of actual activities within firms is
scarce. This is strange, concerning the large potential impact big data has on society. This is also reflected in the
coming MIS Quarterly Executive Dec. 2013 Special edition, which titles “How to succeed in a world of big data1”
and calls specifically for papers on innovative business uses of data, and organizational approaches to
developing and sustaining big data skills and capabilities; topics that are as well described in within this
research.
Methodology
This research was designed as a multiple-case study research (Yin, 2009) in which people involved in big data
practices in Dutch telecommunications and energy utility firms were interviewed. The case studies were
summarized and through a process of open-coding interviews (Strauss & Corbin, 1998) main concepts were
extracted from the codes using Eisenhardts (1989) process of inducting theory using case studies.
Findings
Using eight cases within two Dutch sectors, five phases within the big data solution adoption process were
found: a strategy development phase, a knowledge development phase, a pilot/test-case phase, a platform
implementation phase and a fine-tuning phase.
Issues described within the process of big data solution adoption were categorized in four main concepts:
business case development, technical, organizational, and information privacy related. These findings are
depicted in the conceptual model below.

1 See http://misqe.org/ojs2/MISQE%20Dec%202013%20Special%20Issue.pdf. Retrieved 4/7/2013.
8
Big data adoption process issues
e
sa Strategy Knowledge development Pilot/test-cases Platform implementation Fine-tuning
h
P
- Business case barriers (5) - Business case co-development - Pilot isolated from normal - Top management support
esac ssenisuBtnempoleved - Business case drivers (4) b----t eHIINuTncsao htimren nsdreaton astnlaoosna l gdp ggsyareet oraymtdc kedeseeta snhitnstoae sbl sfduir gepo rpmd soau otrpatt phoerrt p arty -- IDnesftianbei lsitcya &le bouf gims pblieg mdaetnat asytisotnem s
- Solution choice related (combined - Choices on data granularity - Ecosystem complexity - Ecosystem management
seussi lacinhceT in pilot/test-cases) -- SUonlsuttaiobnle c phloaitcfeo rrmelsa ted (15) ---p rDDInoaatctteeaags rsaaaevctscae eil spaslba (itlsifitloyor ’ms) in traditional ---p- PDADrollaaagbtttolaafero ssmitrothmrsuem r arc imlgianiridgngi gpety rps o(crEboaTllbeeLlm esdcmash tsea mseat ’s)
seussi lanoitazinagrO --- WWTehhaeemnn stteoot udstpeav r&et lssoatprvu iancntgau dlryeatt icaa l skills --t-p oARUr ioblvnigalukencsn yin oni newetesendsr ecnpdoar nolt ssocute atmsrksaeeenhrss oalaltdtteiet urasdn easly otinc s ---- DEGAxleeitgvte nesr luneopaxplpt eacoornrnmta alfymlr topiumcanar tiloc iceraagstpa ioannbiz ialittiioesn --(e ISn.pgfe.r arceisfltiircau bccaitlsiutery e ce acnpagapinbaeibleiitlriitesi)se sn neeedededed
ycavirp noitam -p Urivnakcnyo wn consumer attitudes on - Restraints internal law department -(-e CCxohissattnisng ogin fs giyn slatteewgmsr as&)t iroeng purlaivtaiocny st ooling
rofnI
Conceptual model of issues in the big data solution adoption process.

These concepts after analysis led to twelve main issues in the process of big data solution adoption:
Business case development (strategy, knowledge development, and pilot/test-case phases)
1. A paradigm change is needed in order for a firm to recognize the value of big data business cases.
2. Business case development is could be driven by both IT management support or top management
support (in case of new business development), it is hampered due to three main reasons
a. The business case value being unclear and business case return being unclear
b. High financial and time investment before value of business case becomes clear
c. High baseline pressure (especially in small firms)
3. Information privacy concerns significantly impact business case development as will be discussed in
(6).
Solution choice (pilot/test-case phase)
4. Choice for big data solutions (NoSQL databases) are mainly driven by need for scalability, this is also
the main limiting factor on currently used IS. Unstructured data was not reported as a main
determinant for big data solution choice.
5. Parallel databases are currently seen as a viable alternative for big data (NoSQL) solutions by firms
which had already adopted parallel databases for other firm activities.
6. Ecosystem complexity and management problems are reported in both experimentation and
implementation phases, these lead to significant delays in the solution adoption process in
implementation and fine-tuning phase.

9
Organizational change (knowledge development, pilot/test-case, and implementation phases)
7. An analytics department was perceived needed during the process of big data solution adoption and
in all cases were such an department was not yet present an Analytics department was setup during
the process, which caused significant delays in the process.
8. Business process experts (or business consultants) that can translate business case needs in
information systems requirements – taking into account legal and technical constraints – are
perceived a crucial and needed role in the process of big data solution adoption.
9. Specific big data solution infrastructure setup, management and maintenance capabilities were
perceived needed in the process of big data solution adoption.
Information privacy (knowledge development, pilot/test-case, and implementation phases)
10. Information privacy significantly influences the big data solution adoption process in three main ways:
a. Uncertainty regarding consumer attitudes in combination with fear of negative brand image
concerning new big data related products and services hampers business case development
and experimentation, and is seen as a main barrier to new business case development in a
part of cases where personal data is used.
b. Risk of changes in laws and regulations hampering business case development.
c. Business case being unviable due to high costs of privacy tooling (opt-in/out APIs) in the
implementation phase (to connect opt-in/out APIs to current IS) hampers both
implementation and new business case development.
Implementation and fine-tuning of big data technology (implementation and fine-tuning phases)
11. Issues concerning data quality and accuracy problems were related to two main problems:
a. Added complexity layers in big data platforms prevents traditional drill-down in case of data
problems.
b. Scaling effects due to large datasets lead to algorithms having to be retested and
recalibrated.
12. Performance (fine-)tuning of big data solutions is perceived complicated due to high complexity of big
data ecosystems and lack of documentation.

10