Cost Efficient Batch Processing in Amazon Cloud

127 Pages·2016·6.14 MB·English

Cost Efficient Batch
Processing in Amazon Cloud
Cost minimization on batch processing with deadline using
spot instances in Elastic MapReduce Cluster
Kabin Tamrakar
Master’s Thesis Spring 2016

Cost Efficient Batch Processing in Amazon
Cloud
KabinTamrakar
May23,2016
ii
Abstract
Cloud Computing provides computing and storage resources at econom-
ical price with flexibility, mobility and availability. These resources range
from small capacity to very high capacity computes. The cloud providers
alsooffersparecomputeinstancesatsignificantlylowprice.
Amazon Cloud Service provider has a popular bidding scheme
ontheirsparecomputescalledspotinstanceswhichcanberequestedwith
bid price. The spot instances are vulnerable to termination at any time if
spot market price exceeds the bid price. Amazon also rents on-demand
instances which are persistent with fixed price. Spot instance price may
drop up to 90% compared to price of on-demand instance. In this project,
spotinstancesareresortedintaskinstances’groupofAmazonEMRcluster
to process batch jobs with deadline. Amazon EMR makes it convenient to
processbigdatabytheaidofmanagedHadoopframework. Theprocessed
intermediate results in the task nodes of the cluster are lost if the spot
instancesgetsterminatedthatcancauseprocessingdelay.
The cost efficiency can be realized by exploiting non-real time
nature of batch computing for Big Data. Two algorithms are devised
for achieving cost efficient processing in Hadoop MapReduce. Both
algorithms process data in divisions such that abrupt termination of spot
instances affects that division only. Based on progress at some interval
and checkpoints, task group’s capacity is resized to complete processing
in time. Progress is completion of number of divisions of work. The first
algorithm begins with spot instances in estimated quantity. To complete
processing of all data in time, on-demand instances are employed after
threshold time. The second algorithm starts by using higher number
of spot instances than required to complete the work within deadline.
It has higher probability to utilize only spot instances because of faster
work progress. On-demand instances are deployed only in case of slow
progress. Theexperimentsshowthatbothalgorithmsminimizethecostof
processing. Thesecondalgorithmfurtherminimizesthecostinmostcases.
iii
iv
Acknowledgements
Cloud computing is one of the hotcakes of the modern information
technology. Workingonthisthesis,Ilearntalotaboutrecentcloudservices
andsystemautomation. Ithasopeneddoorsformetoresearchmoreinthis
fieldduringmyprofessionallife.
First of all, I would like to express my sincere gratitude to my
supervisorsHårekHaugerudandAnisYazidiforthecontinuousguidance,
remarks, engagement through the learning process and close monitoring
of my master thesis. I would like to take this opportunity to thank them
andexpresshowimmenselyimportantweretheirlecture,weeklymeeting,
suggestion for completion of my master’s thesis. The positive spirit from
mysupervisorshasalwaysencouragedmetothrivemore, workhardand
realizemypotential.
I would also like to mention my professor Kyrre Begnum’s con-
tribution for enhancing my research skills, knowledge about recent tech-
nologies in our field through his amazing lectures and notes. Information
Technologyisthemostdynamicindustry. Hehasdeeplyinfluencedmeto
beupdatedwiththemoderntechnology.
Big Thanks to my fellow classmates for the all the help, support,
important discussions, motivation. They have inspired me to work hard,
helped me when I was stuck in some technical problems. I want to
remember how were they available to help during my thesis, anytime
via social media. My sincere gratitude goes to University of Oslo (UiO)
and Oslo and Akerhus Universiy College (HiOA) for providing quality
educationandinfrastructures.
I would like to thank my wife Mamta Maharjan for all her
love, support and interest. She has always inspired me with insightful
discussionsandconstructivesuggestions. Thanksforkeepingupwithmy
busy schedule and taking care of my health. Last but not the least, my
unlimitedgratitudegoestomymom,dadandsisters. Icannotthankthem
enoughforallthecare,love,moralsupport,encouragement.
-KabinTamrakar
v
vi
Contents
1 Introduction 1
1.1 ProblemStatement . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 StructureoftheReport . . . . . . . . . . . . . . . . . . . . . . 3
2 Background 5
2.1 CloudComputing . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.1.1 DifferentmodelsofCloud . . . . . . . . . . . . . . . . 6
2.1.2 Differentservicecategories . . . . . . . . . . . . . . . 6
2.1.3 CloudServiceProviders . . . . . . . . . . . . . . . . . 7
2.1.4 AmazonWebServices . . . . . . . . . . . . . . . . . . 8
2.1.4.1 AmazonElasticComputeCloud . . . . . . . 8
2.1.4.2 AmazonEC2PurchasingOptions . . . . . . 9
2.1.4.3 AmazonSimpleStorageService . . . . . . . 11
2.1.4.4 Boto-AmazonWebServicesSDK . . . . . . 11
2.1.5 GooglePreemptibleInstances . . . . . . . . . . . . . . 11
2.1.6 Biddingspotinstances . . . . . . . . . . . . . . . . . . 11
2.2 Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.2.1 Hadoop . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.2.2 AmazonElasticMapReduce . . . . . . . . . . . . . . 13
2.2.2.1 AmazonEMRMetrics . . . . . . . . . . . . . 15
vii
2.2.3 CloudWatch . . . . . . . . . . . . . . . . . . . . . . . . 18
2.3 RelevantTheory/LiteratureReview . . . . . . . . . . . . . . 18
2.3.1 Cutting the Cost of Hosting Online Services Using
CloudSpotMarkets . . . . . . . . . . . . . . . . . . . 18
2.3.2 Automatedcloudburstingonahybridcloudplatform 19
2.3.3 WOHA: Deadline-Aware Map-Reduce Workflow
SchedulingFrameworkoverHadoopClusters . . . . 19
2.3.4 Towards a MapReduce Application Performance
Model . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.3.5 HowtoBidtheCloud . . . . . . . . . . . . . . . . . . 20
I Theproject 21
3 Approach 23
3.1 Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.2 DesignStage . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.3 NecessaryComponentsandToolstobuildthemodels . . . . 25
3.3.1 TheCluster . . . . . . . . . . . . . . . . . . . . . . . . 25
3.3.2 MapReduceDataProcessingEngine . . . . . . . . . . 25
3.3.3 Computingvirtualmachines . . . . . . . . . . . . . . 25
3.3.4 PersistentStorage . . . . . . . . . . . . . . . . . . . . . 26
3.3.5 Monitoring . . . . . . . . . . . . . . . . . . . . . . . . 27
3.3.6 ScriptDevelopmentforAutomation . . . . . . . . . . 27
3.3.7 FoxyProxy . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.3.8 R-programming . . . . . . . . . . . . . . . . . . . . . . 27
3.4 ProjectSteps . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.4.1 SampleDataGeneration . . . . . . . . . . . . . . . . . 28
3.4.2 ClusterSetupandConfiguration . . . . . . . . . . . . 29
viii