Performance Evaluation of Kotlin and Java on Android Runtime

90 Pages·2017·1.16 MB·English

DEGREE PROJECT IN COMPUTER SCIENCE AND ENGINEERING,
SECOND CYCLE, 30 CREDITS
STOCKHOLM, SWEDEN 2018
Performance Evaluation of
Kotlin and Java on Android
Runtime
PATRIK SCHWERMER
KTH ROYAL INSTITUTE OF TECHNOLOGY
SCHOOL OF ELECTRICAL ENGINEERING AND COMPUTER SCIENCE

Performance Evaluation of
Kotlin and Java on Android
Runtime
PATRIK SCHWERMER
Master in Computer Science
Date: July 6, 2018
Supervisor: Elena Troubitsyna
Examiner: Johan Håstad
Swedish title: Prestandautvärdering av Kotlin och Java för Android
Runtime
School of Computer Science and Communication

3
Abstract
This study evaluates the performance of Kotlin and Java
on Android Runtime using four benchmarks from the
Computer Language Benchmarks Game suite, for which
a total of 12 benchmark implementations are studied.
The metrics used to evaluate the performance includes
runtime, memory consumption, garbage collection, box-
ing of primitives as well as bytecode n-grams. To bench-
mark the languages, a benchmark application has been
developedintendedtorunonanAndroidphone. There-
sults indicate that Kotlin is slower than Java for all stud-
iedbenchmarksbyavaryingfactor. Furthermore,theuse
of idiomatic Kotlin features and constructs results in ad-
ditional heap pressure and the need of boxed primitives.
Other interesting results indicate the existence of an un-
derlying garbage collection overhead when reclaiming
Kotlin objects compared to Java. Furthermore, Kotlin
produces larger and more varied bytecode than Java for
amajorityofthebenchmarks.
Sammanfattning
Denna studie utvärderar prestandan mellan Kotlin och
Java på Android Runtime genom 12 implementationer
av fyra benchmarks från The Computer Language
Benchmarks Game. De mätvärden som använts för
att utvärdera prestandan inkluderar körtid, minnesan-
vändning, garbage collection, boxing av primitiver
samt bytekod n-grams. För att benchmarka språken
har en benchmarkapplikation tagits fram för Android.
Resultaten visar att Kotlin är långsammare än Java för
samtliga benchmarks. Vidare resulterar användandet av
idiomatiska Kotlin-funktioner i ökad minnesanvänding
samt behovet att representera primitiver som klasser.
Andra intressanta resultat inkluderar existensen av
en overhead för garbage collectorn för frigörandet av
objektsomallokeratsavKotlinjämförtmedJava. Vidare
producerar Kotlin större bytekodfiler och uppvisar
mer varierad bytekod än Java för en majoritet av de
benchmarkssomstuderats.
Contents
1 Introduction 8
1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.2 Problemstatement . . . . . . . . . . . . . . . . . . . . . . . 9
1.3 ScopeandLimitations . . . . . . . . . . . . . . . . . . . . . 9
1.4 EthicsandSustainability . . . . . . . . . . . . . . . . . . . . 10
1.5 Disposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2 Background 12
2.1 Compilers . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.1.1 Ahead-of-TimeCompiler(AOT) . . . . . . . . . . . 13
2.1.2 Just-in-TimeCompiler(JIT) . . . . . . . . . . . . . . 13
2.1.3 Compileroptimisations . . . . . . . . . . . . . . . . 14
2.2 Android . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.2.1 AndroidArchitecture . . . . . . . . . . . . . . . . . 17
2.2.2 DalvikVirtualMachine(DVM) . . . . . . . . . . . . 19
2.2.3 AndroidRuntime(ART) . . . . . . . . . . . . . . . . 19
2.2.4 AndroidBuildProcessandRuntime . . . . . . . . . 20
2.3 KotlinLanguage . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.3.1 ConstructsandConcepts . . . . . . . . . . . . . . . 24
2.4 Benchmarking . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.4.1 ComputerLanguageBenchmarksGame(CLBG) . . 26
2.4.2 Statisticaltheory . . . . . . . . . . . . . . . . . . . . 27
2.5 Bytecodeanalysis . . . . . . . . . . . . . . . . . . . . . . . . 29
2.5.1 N-gram . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.5.2 PrincipalComponentAnalysis(PCA) . . . . . . . . 30
2.6 Relatedwork . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.6.1 DynamicMetricsforJava . . . . . . . . . . . . . . . 30
2.6.2 JVM-Hosted Languages: They Talk the Talk, but do
theyWalktheWalk? . . . . . . . . . . . . . . . . . . 31
2.6.3 WorkloadcharacterizationofJVMlanguages . . . . 31
4
CONTENTS 5
2.6.4 Cross-language Compiler Benchmarking - Are We
FastYet? . . . . . . . . . . . . . . . . . . . . . . . . . 32
3 Methodology 33
3.1 Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.1.1 Fastabenchmark . . . . . . . . . . . . . . . . . . . . 35
3.1.2 Fannkuch-Reduxbenchmark . . . . . . . . . . . . . 35
3.1.3 N-bodybenchmark . . . . . . . . . . . . . . . . . . . 36
3.1.4 Reverse-complementbenchmark . . . . . . . . . . . 36
3.2 JavaImplementations . . . . . . . . . . . . . . . . . . . . . . 37
3.2.1 Fasta . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.2.2 Fannkuch-Redux . . . . . . . . . . . . . . . . . . . . 38
3.2.3 N-body . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.2.4 Reverse-complement . . . . . . . . . . . . . . . . . . 40
3.3 KotlinImplementations . . . . . . . . . . . . . . . . . . . . 40
3.3.1 Fasta . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.3.2 Fannkuch-Redux . . . . . . . . . . . . . . . . . . . . 42
3.3.3 N-body . . . . . . . . . . . . . . . . . . . . . . . . . . 42
3.3.4 Reverse-Complement . . . . . . . . . . . . . . . . . 43
3.4 Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3.4.1 Runtimemetric . . . . . . . . . . . . . . . . . . . . . 44
3.4.2 Memoryconsumptionmetrics . . . . . . . . . . . . 45
3.4.3 Garbagecollectionmetrics . . . . . . . . . . . . . . . 45
3.4.4 Boxingofprimitivesmetric . . . . . . . . . . . . . . 45
3.4.5 Bytecodemetrics . . . . . . . . . . . . . . . . . . . . 46
3.5 Environmentandsetup . . . . . . . . . . . . . . . . . . . . 46
3.5.1 Softwareandtools . . . . . . . . . . . . . . . . . . . 46
3.5.2 Hardware . . . . . . . . . . . . . . . . . . . . . . . . 49
3.5.3 Datacollectionandanalysis . . . . . . . . . . . . . . 49
4 Results 53
4.1 Runtime . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
4.2 Memoryconsumption . . . . . . . . . . . . . . . . . . . . . 57
4.3 Garbagecollection . . . . . . . . . . . . . . . . . . . . . . . 59
4.4 Boxingofprimitives . . . . . . . . . . . . . . . . . . . . . . 63
4.5 Bytecodeanalysis . . . . . . . . . . . . . . . . . . . . . . . . 64
4.5.1 Unigramresults . . . . . . . . . . . . . . . . . . . . . 64
4.5.2 Bigramresults . . . . . . . . . . . . . . . . . . . . . . 66
4.5.3 Trigramresults . . . . . . . . . . . . . . . . . . . . . 68
6 CONTENTS
5 Discussion 70
5.1 Runtime . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
5.2 Memoryconsumption . . . . . . . . . . . . . . . . . . . . . 72
5.3 Garbagecollection . . . . . . . . . . . . . . . . . . . . . . . 73
5.4 Boxingofprimitives . . . . . . . . . . . . . . . . . . . . . . 73
5.5 Bytecode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
5.6 General . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
6 Conclusion 77
Bibliography 79
A BenchmarkImplementations 83
A.1 Javaimplementations . . . . . . . . . . . . . . . . . . . . . . 83
A.2 Kotlinimplementations . . . . . . . . . . . . . . . . . . . . 83
B BenchmarkApplication 84
C N-gramApplication 85
D Results 86
D.1 Runtime . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
D.2 Memoryconsumption . . . . . . . . . . . . . . . . . . . . . 86
D.3 Garbagecollection . . . . . . . . . . . . . . . . . . . . . . . 86
D.4 Boxingofprimitives . . . . . . . . . . . . . . . . . . . . . . 86
D.5 Bytecoderesults . . . . . . . . . . . . . . . . . . . . . . . . . 87
Acronyms
AOT Ahead-of-Time.
API ApplicationProgrammingInterface.
APK AndroidApplicationPackage.
ART AndroidRuntime.
CLBG ComputerLanguageBenchmarksGame.
DVM DalvikVirtualMachine.
ELF ExecutableandLinkableFormat.
GC GarbageCollector.
JIT Just-in-Time.
JVM JavaVirtualMachine.
LIFO LastInFirstOut.
OS OperatingSystem.
PCA PrincipalComponentAnalysis.
VM VirtualMachine.
7
Chapter 1
Introduction
This chapter introduces the motivation behind this study, specifies the problem
statement and research question, defines the scope and limitations, puts this
study in the context of ethical aspects and sustainability and finally presents
thedispositionofthispaper.
1.1 Introduction
Today, Java is one of the world’s most used programming languages and
for good reason. After its introduction in 1995, it now features a strong
community and rich documentation. A part of Java’s success can also be
attributedtoitswriteonceruneverywherephilosophy. Bycompilingsource
code into intermediate Java bytecode, and then executing it on a virtual
machine on the target system, Java provides portability and platform in-
dependency which few other languages can offer. Although there are
someinherentbenefitsofusingJava,amajordrawbackisitsageshowing
initslackofmanymodernprogrammingconceptsandfeatures. Assuch,
a trend has emerged in the last decade where new languages have been
announced, introducing a modern alternative to Java while still reaping
some of the benefits which contributed to Java’s popularity by targeting
the Java Virtual Machine (JVM). Some popular alternatives to Java are
Scala,GroovyandClojure.
AnotherbigpartofJava’spopularityisitsconnectiontotheAndroidplat-
form as the first officially supported language. However, in May 17 2017,
Google announced official support for another language, namely Kotlin
[19]. Although introduced in 2011, Kotlin has gained most of its popu-
larity since Google’s announcement [34] and is today being used to an
8