Context-Dependent Testing of Applications for Mobile Devices

13 Pages·2015·0.78 MB·English

(cid:13)c 2015bytheauthors;licenseeRonPub,Lu¨beck,Germany.Thisarticleisanopenaccessarticledistributedunderthetermsandconditionsof
theCreativeCommonsAttributionlicense(http://creativecommons.org/licenses/by/3.0/).
OpenAccess
Open Journal of Web Technologies (OJWT)
Volume 2, Issue 1, 2015
http://www.ronpub.com/ojwt
ISSN2199-188X
Context-Dependent Testing
of Applications for Mobile Devices
TimA.MajchrzakA,MatthiasSchulteB
AUniversityofAgder,Gimlemoen25,4630Kristiansand,Norway,[email protected]
B viadeeUnternehmensberatungGmbH,Anton-Bruchausen-Straße8,48147Mu¨nster,Germany,
[email protected]
ABSTRACT
Applicationspropeltheversatilityofmobiledevices.Appsenabletherealizationofnewideasandgreatlycontribute
totheproliferationofmobilecomputing. Unfortunately, softwarequalityofappsoftenislow. Thisatleastpartly
can be attributed to problems with testing them. However, it is not a lack of techniques or tools that make app
testing cumbersome. Rather, frequent context changes have to be dealt with. Mobile devices most notably move:
networkparameterssuchaslatencyandusablebandwidthchange,alongwithdatareadfromsensorssuchasGPS
coordinates. Additionally,usagepatternsvary. Toaddresscontextchangesintesting,weproposeanovelconcept.
It is based on identifying blocks of code between which context changes are possible. It helps to greatly reduce
complexity. Besidesintroducingourconcept,wepresentausecase,showitsapplicationandbenefits,anddiscuss
challenges.
TYPE OF PAPER AND KEYWORDS
ShortCommunications: App,mobile,mobileapp,mobiledevice,test,testing,context
1 INTRODUCTION greatlyfromthepossibilitiesofferedbythedevices,for
examplebyusinganintegratedcameraormakinguseof
ItishardtobelievethatthefirstiPhonewasintroduced localizationviaGPS.
onlyafewyearsago(in2007[31],tobeprecise). Con- Developing apps is a relatively new practice with lit-
trastingformerapproachessuchaspersonaldigitalassis- tle experience and even less tradition. While there are
tants(PDAs)andfeature-phones,smartphonesandtablet alreadymanytextbooks,scientificliteratureislimitedto
computers have become devices used by nearly every- articlesfromconferenceproceedingsand–toasmaller
one. Theirpopularityamongconsumersandenterprises extent–journals. Worktypicallytacklesspecificissues
is still rising [16, 17]. Unsurprisingly, companies em- rather than giving a bigger picture. While app devel-
bracethenewpossibilitiesforavarietyofactivities[33]. opment is not too different from developing rich-client
Modern mobile devices challenge the performance applicationsorWebapplicationsforPCs[23],thereare
offered by personal computers shipped few years ago. particularities.
Moreover,theyareequippedwithspecialhardwaresuch Amongothers, softwaretestingisoneofthese. Test-
as cameras and a multitude of sensors. However, their ing software is a cumbersome task [32]. Many tech-
versatility is propelled by the software they use – or niques require sophistication. Effective and efficient
ratherthatmakeuseofthem.Mobileapplications–apps software testing remains a challenge despite decades of
–nowadaysformanecosystemoftheirown. Theydraw research. At the same time, proper testing greatly con-
27
OpenJournalofWebTechnologies(OJWT),Volume2,Issue1,2015
tributestoanapplication’svalue. Insightsfromsoftware app not explicitly takes notice of mobility (say, a com-
engineering including testing strategies and techniques mongame)itisinfluencedbyit: mobilitymightleadto
can be applied to apps. However, testing differs in a changingconditionssuchasvaryingconnectivity.More-
number of ways: Firstly, apps are not developed on the over, app lifecycle models differ from that of applica-
platform they run (mobile device) but on a PC. Testing tions for PCs in that they might be halted on external
on emulators will not yield the same results as testing eventssuchasincomingphonecalls.1 Appsarebuiltto
natively. Moreover,itcanbeinefficient. Secondly,test- beusedindifferentsituationsandinachangingenviron-
ing on mobile devices is laborious and very hard to au- ment. Various kinds of devices are employed. Besides
tomate. Additionally,itistoughtoensureuniformcon- smartphonesandtablets,therearehybridssuchastheso
ditionsduringtesting. Thirdly, toolsupportcurrentlyis calledphablets[48]anddevicessuchassmartTVs[21].
limited. Not all PC tools have mobile device counter- Theaboveconsiderationsleadtoaninsight: Appsare
parts.Fourthly,manyappscombinevarioustechnologies heavily influenced by the context they reside in. At the
andevenprogrammingparadigmsandlanguages. Most sametime,appsarecapableofmakinguseofcontextual
apps contain some mixture of native programming and factors. Togiveanexample: whenyoudriveoutoftown
Webtechnology. yoursmartphonemightneedtoswitchtoacellularsys-
The most profound difference, however, is context. tem providing less bandwidth (adapting to context) but
Mobile devices are subject to many different contexts; help youfind a nearbyswimming lake by matching ge-
the simplest one is location, since mobility typically olocationinformationwithmapdata(utilizationofcon-
means(very)frequentslightchangesofposition. There text).
aremanyfurthercontextchangessuchasnetworkcondi- Inthefollowing,weproposeacategorizationscheme
tion,availabilityofdatafromsensors,andevensocialis- forappcontextthatdistinguishesfivecorecontexts. Itis
suessuchasalternatingusersononedevice. Therefore, alsosketchedinFigures1a-1e(seep.29).
devices need not only be tested as they are but taking Our proposal is not to be seen as (another) formal
contextintoconsiderations. Ourexperienceisthattest- model for context. In fact, it is a framework of context
ingresultsgreatlydifferindependenceofthecontext. onmobiledevices. Moreover,aspresentedhereitisin-
We present a novel approach for software testing of stantiatedforcurrentAndroidplatforms. Sincesomeas-
appsthattakesintoaccountthecontextchangesinherent pectsaredevice-dependent,minoradaptionsareneeded
to mobile devices. Our article makes a number of con- forotherplatforms. Futurechangestotheecosystemof
tributions. Firstly,weexplainthebackgroundofcontext mobileappsmightleadtotheneedforadditionaladjust-
asaninfluencingfactorofapps–ithasimplicationsbe- ments.Whileourconceptmightbeextendedtoaformal-
yond the realm of testing. Secondly, we introduce our conceptatsomepoint,thiswouldbeatopicforaarticle
uniqueapproachforhandlingcontextintesting. Thirdly, ofitsown.
wefacilitateemployingourapproachbyshowingareal-
life scenario. And fourthly, we generalize our findings
2.2 HardwareContext
anddiscussthenextsteps.
This paper is structured as follows. Section 2 high-
The market for mobile devices is greatly fragmented
lightstherelevanceofcontextchangesformobiledevice
[17]. Apps nowadays run on a multitude of devices,
usage. InSection3wedistinguishourworkfromother
ranging from watches (smartwatches) to TVs. While
mobile testing approaches. Section 4 explains our ap-
these devices typically are utilized via touch interfaces,
proach in detail, first with a focus on theory and then
theygreatlydifferinthecomponentstheyarebuiltfrom
by presenting a scenario. Application, Limitations and
(Figure 1a). Screen size, resolution, screen brightness,
ChallengesarediscussedinSection5. Finally,wedraw
contrast and refresh rate, device size, sensors, device
aconclusioninSection6.
extensions, and input means besides touch (e.g. a re-
Thisarticleisangreatlyextendedandrevisedversion
motecontrol)vary. Particularlythelevelofmobility,the
of paper [34]. It extends the concepts proposed by us
screensizeandthesensorequipmentmakesomedevices
in[47].
hardlycomparablebesidesbeingbasedonsiliconchips
and enabled to run apps. An app might require a cam-
2 MOBILE DEVICES AND CONTEXT era,therebylimitingthenumberofdevicesitcanberun
onbysomedegree. If,however,theapprequiresaspe-
2.1 ContextinGeneral cific minimum resolution, the number of excluded de-
vicesrisessharply.
Theusageparadigmofappscontraststhatofapplications
onPCsandthatofWebsites(i.e. Webappswithnoop- 1Applifecyclemodelsarenotfurtherdiscussedhere.Moredetails
timization). Themaindifferenceismobility. Evenifan aregivenby[43].
28
T.A.Majchrzak,M.Schulte:Context-DependentTestingofApplicationsforMobileDevices
battery status
dynamic
currently variation of
available available
memory hardware space
context
memory disk space application Android version
lifecycle
static application software context operating system
system
screen size component 3rd party vendor specific
processing power
and resolution configuration system apps adjustments
(a)Hardware (b)Software
availability accuracy
geolocation
technologies
Bluetooth USB NFC etc.
devices / objects
physical context
around direct com-
connectivity munication available
partly
sensor data measured data cellular
by bandwidth network com. context network roaming
status status
motion position environment latency unavailable
(c)Physical (d)Communication
business private in meeting etc.
has situation Influ-
effects on ences … of week
level of
social context time
attention
… of day
(e)Social
Figure1: Contexts
29
OpenJournalofWebTechnologies(OJWT),Volume2,Issue1,2015
Besidestheparticularitiesofmobiledevices,alsocon- 2.4 PhysicalContext
textualfactorsalreadyknownfromPCsandservershave
Mobility means that the physical context (Figure 1c) is
to be considered. This includes available memory and
continuously changing [45]. Location determines other
diskspace. Batterycapacityisanotherfactor.
contextual factors such as connectivity [46]. Most de-
The rate of technology advances in mobile device
viceshaveoneormoremeanstodeterminetheirlocation
hardware is still very high. This dynamic is accompa-
(more or less accurate, depending on the method and,
niedbythestrivetomakedevices“appenabled”–smart
again,context).
TVs most likely are only a beginning. It is impossible
Thephysicalcontextalsocomprisesotherdevicesthat
fordeveloperstoforecasttheupcomingchanges,thereby
can be contacted with technology such as Bluetooth,
becomingabletodesignappsperfectlyresponsively(cf.
Near Field Communication (NFC) and wireless LANs.
[40]). Thus,thehardwarecontextcanbeproblematic.
Moreover, devices are usually equipped with a number
Different hardware can be handled by a strategy of of sensors such as gyroscopes, accelerometers, fixtures
seekingforakindof“greatestcommondivisor”among for temperature measurement and similar units. Their
devices. Thinkofanappthatletsyouscanbarcodeson feedbackisdependingonthephysicalcontext. Ithasto
goodsinordertocomparepriceswithproductslistedin beparticularlynotedthatnotallphysicalcontextscanbe
a Web database. There is no reason to consider hard- simulatedwithtoday’smeans–thisspecificallyapplies
warethatisnotmobile,suchasTVsets. Theappshould to sensors. Even when using actual devices, testing the
notifyusersifnocameracanbeutilized(insteadofmal- physicalcontextcanbecometremendouslyhard.
functioning). Contexttestingshouldincludereactionto
cameras below the quality deemed to be the minimum
2.5 CommunicationContext
threshold, at the threshold, and above it – similarly to
equivalencepartitioningtesting[32,p.28]. Dependingonthelocation,connectionparametersvary,
whichinfluencescommunication(Figure1d). Themain
parametersareavailability,bandwidthandlatency,along
with minor parameters such as jitter. These are deter-
2.3 SoftwareContext
mined by the service provider used, the cellular ser-
vices supported by the device, the current location, and
Fragmentation in terms of hardware goes along with
tosomeextentexternalcircumstances(i.e. thephysical
softwarefragmentation. Thus,softwareformsacontext
context). Forexample, connectionqualitymightbeim-
ofitsown(Figure1bisanexamplefortheAndroidplat-
pairedbyweatherorbymanypeopleusingthenetwork
form).Withatleastthefourpopularplatforms(Android,
cell(e.g. inasportsstadium).
Blackberry, iOS,andWindowsPhone)[17], eachexist-
Location typically determines whether only “slow”
inginavarietyofversions,appdevelopmentisproblem-
systemslikeGPRSorEDGEareavailable,ortechnolo-
atic. Some versions greatly differ and application pro-
gies such as UMTS, HSPA or LTE can be used. More-
gramminginterfaces(APIs)employedbyolder,stillfre-
over, devices might connect to wireless LANs, circum-
quentlyuseddevicesmightbedeprecatedbynow.More-
passingtheneedforamobiledataconnectionviaaser-
over, vendors tend to amend official releases with addi-
viceprovider.
tionsoftheirown.Thisworsensthesituationparticularly
Thecommunicationcontextgraduallychangescontin-
inthemarketofAndroiddevices.
uously. Thepaceofchangeusuallyalignswiththerate
It is unlikely that soon a middleware layer [1] or a ofmobility. Appsoughttoberobust. Theyshouldalso
cross-platformapproach[22]willalleviatetheproblem. maintainfunctionalityinofflinescenarios. Ideally, they
In fact, apps existing both in a native version and as a shouldadjusttothecontext. Forexample,anappmight
mobile Webapp even complicate testing. Additionally, loadlowresolutionimagesinsteadoffullpicturesifthe
device-specificsoftwarecomponentscontributetocom- availablebandwidthdropsbelowathreshold.
plexity. The communication context is more complex than it
Context-related problems might also arise from the appears. Forexample,ahigh-bandwidthcellularservice
combinationofcontextualfactorsofhardwareandsoft- mightbereadilyavailablebutnotusedbythephonebe-
ware.Appscouldbehaveverysimilarlyusingtwodiffer- causeitislocatedabroadandthusinroamingmodewith
entversionsofamobileplatform, e.g. havingasimilar disableddataservices. Atestthatcheckswhetherband-
performance. This may be different when running on widthisavailable,attemptstoperformanoperationand
slowdevices(whichwouldbeachanginghardwarecon- thenfails(beingunabletosenddata)mightbeveryhard
text). Dataformatsmightbesuitedinoneplacebutnot tointerpretand,thus,confusing.
intheother. Testing includes simulations of small bandwidths,
30
T.A.Majchrzak,M.Schulte:Context-DependentTestingofApplicationsforMobileDevices
high latencies, constant changes in connection quality, ingsurethatappsbehaveasexpectedhastobeaddedon
and abrupt unavailability of service as well as resum- topoftheregulartestingactivities. Duetoamultiplica-
ing service. Unfortunately, which bandwidth, latencies tiverelationship, thismakestestingmuchmorecompli-
etc. are acceptable is highly depending on the app be- cated, requires considerable more time, and can be ex-
ingtestedaswellasontheuser’sexpectationsandusage hausting.Thedescribedchallengescallforsupportinef-
patterns–andoncontext. fectivetesting,andforautomationtoincreaseefficiency.
2.6 SocialContext
3 RELATED WORK
Thesocialcontext(Figure1e)ishardertograspthanthe
There is not much scientific literature on app develop-
other context categories. It comprises of user-specific
mentthatgoesbeyondcasestudiesorapplyingexisting
waysusinganappandofusageparticularities.
methods. In combination with the novelty of our ap-
Firstly, an increasing number of mobile devices are
proachthisexplainswhynotmuchcloselyrelatedwork
usedbothforworkandforprivatepurposes. Bringyour
couldbeidentified.
own device (BYOD) [12] policies allow private smart-
phones within a companies’ premises (and probably to Theliteratureontestingisvast(cf. [32, Chap.2]for
useitsnetwork). However,someappsareusedforwork an overview). App development is too new to be given
whileothersareusedforpersonalreasons. Somemight special attention in textbooks on testing. Nevertheless,
be used for both but not necessarily in the same way. standard literature is a valuable source since testing of
Duetosecurityreasons,appsmightevenneedtorelyon Web-based applications is typically covered (e.g. [44,
different data sources and make sure that company and Chap.22]).Moreover,techniquesfortestingofgraphical
privatedataisnotintermingled. user interfaces (GUI) can typically be applied to apps.
Secondly, while most apps are tailored to single- Security testing of Web applications [4] is different in
personuse,somemobiledevicesareusedbymorethan concept to our approach, despite some context changes
oneperson.Imagineachildplayingagameonaparent’s alsoposingsecurityrisks.
smartphoneoracustomerviewingmaterialona(e.g. in- Mobilityissometimescoveredintextbooks,typically
surance)salesperson’stablet. However,differentpeople alongwithWebapplication[41,Chap.6]. Somerecom-
use apps uniquely and an app’s capabilities might need mendationsarealreadyoutdated;ingeneral,thereislittle
tobeadjusted. Forexample,customersshouldseetheir sophisticated advice. Nevertheless, without mentioning
materialbutmustnotseematerialspreparedinthesame contextdirectly,somediscussionsrelatetoourideas.For
app for other customers (e.g. an estimate of insurance example [41, pp. 166] highlights different connection
premiums). speedsofmobilephoneservices–eventhoughchanges
Thirdly, users’ attention span will not be the same duringconnectiontimearenotmentioned.
in all situations. Typical app usage is characterized by Most textbooks on app development do not address
changingattention. Thiscanbeexplainedwiththemo- testing; exception handling and other app-specific error
bile nature: the simplest example is people that send processing concepts are rather discussed (cf. e.g. [29,
short messages while walking. Naturally, they cannot Chap. 4]). It has to be kept in mind that most titles are
keep staring on the smartphone screens, or they risk notacademicandaddressbeginners,though. Atleastan
bumpingintoastreetlight. Moreover,ithasbeenfound e-bookonAndroidtestingcouldbeidentified[36]. Nev-
in a study that even the time of usage influences which ertheless,thesmallattentiongiventoqualityaspects–let
appsareused[50]. alonetesting–inappdevelopmentbooksisdisappoint-
The social context is very challenging for testing. ing. At the same time, some authors try to disillusion
Whilemostothercontextchangesmightbehardtosim- hobbyistprogrammersabouttheeaseofmakingmoney
ulate,itisratherstraightforwardtodescribeandtomea- byimplementingapps[51].
sure them. Social contextual factors, though, are fuzzy There is a high number of papers that present work
tosomedegree, hardtoestimateandinmanycasesim- on app testing, often in an research-in-progress state.
possibletoexhaustivelysimulateandmeasure. Typical directions of research are test automation [14],
user-centered testing [20], tools [18, 24], testing ap-
proaches [3, 37], and specifically user interface testing
2.7 Consequences
[53,54,9,10]. Allthesepapers–andquiteafewmore
Appshavetocopewithavarietyofcontextsthatarehard that deal with similar issues – only roughly relate for
to be assessed and to be kept track of. For several con- theytackletestingofappsbutareconceptuallydifferent.
textualfactors,frequentchangeshavetobeexpectedand In general, the research papers suggest that app testing
patternsofchangenotnecessarilyarepredictable. Mak- willmature.Someauthorsevencontrivenext-generation
31
OpenJournalofWebTechnologies(OJWT),Volume2,Issue1,2015
business models such as offering Mobile Testing-as-a- for each test case a priori. While being beneficial for
Service [15]. Testing (or rather the level of support for unit tests, asserting that a single code unit produces an
it)alsoisacriterionwhenassessingmobiledeviceplat- expected output in a certain context would not be help-
forms[35]. ful for testing business processes with integration tests.
Strictlyspeaking,evenrandomtestingofapps[30]is Asmanyinfluencingcontextualfactorsarenotstaticbut
related to our work. Rather than comparing concepts it change constantly during usage, a more dynamic ap-
will,however,beataskoffutureworktocomparetheef- proachisneeded.
ficacyofsophisticatedapptestingmethods(suchasthe Providing the possibility (e.g. by an API) to specify
onepresentedinthisarticle)tosimpleones(suchasran- context changes within test cases would allow to sim-
domtesting). ulatechangingcontexts. However, thisapproachisstill
The sole more closely related work we identified is static.Contextchangeshavetobeexplicitlystatedwithin
presented by Amalfitano et al. [2]. They consider con- testcases. Thus,whentestingscenariosincludingdiffer-
textastheresultfromeventsthatcanbetriggeredbythe ent variations of context, many test cases are required.
user,thephone,orexternalactivities. Toconsiderevents Each test case would contain the same test code; only
in testing, they propose to identify patters. These are statementsforchangingcontextswouldvary. Thismul-
used for manual, mutation-based and exploration-based tiplication is not desirable. Test cases would be costly
testing. Ourworkisnotcompetingwiththeirsbutcom- todevelopandexecute,andhardtomaintainletaloneto
plementary. reuse. Moreover, the right sequence of context changes
The relevance of context in mobile computing has hastobeanticipatedbeforehand.
been discussed as early as in 2001 [42]. In most cases
context is used in connection with awareness, i.e. the 4.2 Blocks,AssertionsandContextChanges
abilityofmobiledevicestoperceivecontextchangesand
react accordingly [7]. Context is used to recommend To provide a solution that fosters dynamic changes of
apps [39, 27], adapt automatically [19], and to predict context,weusemodularization. Testcasesaresplitinto
usagepatterns[52]. Whiletheseworkscontributetoour blocks; between each block a change of context is pos-
understanding of contexts, they only support testing by sible. Blocks are reused and combined with different
explainingwhatcancausecontextchanges. context changes. Therefore, testing multiple scenarios
Finally, there is general work on testing that keeps is possible without duplication of test code. It is even
contextinmind.Papersdealwithcontextawareness(e.g. possible to generate scenarios without user-interaction.
in autonomous systems [49] or focusing sensors [8]), As blocks are the foundation of our concept, we call it
modelchecking[13],multithreading[38],andWebser- block-basedcontext-sensitivetesting.
vices[6]. Thenotionofcontext differs; thetermisem- Agiventestcasemayresultinanumberofblocksthat
ployedinvariousfieldsofcomputerscience. Neverthe- formthestructureofacontext-sensitivetest. Eachblock
less, several of the cited articles have a similar under- containsoperationsandassertions. Similartounittest-
standing of context as we do and underline the impor- ing frameworks, operations are needed to simulate user
tanceofbeingsensitivetocontextchanges. interactionandassertionsareusedtoverifyexpectedbe-
Summing up, there is a plethora of roughly related havior. Our idea is to derive blocks from existing test
work that elaborate the importance of context changes cases, such as the so called happy path – a basic test
anddescribedifferentcontexts.However,thereismerely case without extraordinary context changes. Doing so,
one otherapproach thatcombines context changeswith asingletestcasemaybetransformedintoastructureof
testingsofar. blocks that can be used to generate context-dependent
testcases. Topreservetheintentionoftestcases,blocks
havetobeorderedandexecutedconsecutively.
4 CONTEXT-SENSITIVE TESTING
Listing1illustratesinaschematicwayhowatestcase
looks like. If the test code itself would be divided into
4.1 GeneralConsiderations
blocks, this schematic test case contains two of them:
To combine context changes with app testing, it has to onefromlines3to7(blockA)andonefromlines10to
be possible to automatically change context parameters 12 (block B). The scope of each block has to be atomic
whilsttestcasesareexecuted.Withblock-basedcontext- w.r.t. changing contexts. In other words, during execu-
sensitive testing we focus on context elements that are tionoftestoperationsbelongingtoonesingleblock,the
dynamic,i.e. possiblychangingduringappsusage. Do- contextremainsstable. Onlybetweenblockschangesof
ingso,thechangingcontextsthatareinducedbyauser contextarepossible(lines2and9). Torealizedifferent
beingmobilecanbesimulated. scenarios,testoperationsdonotneedtoberepeatedbut
The naive approach would be to specify the context onlythecontextchangesinbetweenhavetobealtered.
32
T.A.Majchrzak,M.Schulte:Context-DependentTestingofApplicationsforMobileDevices
Listing1: Schematictestcasewithcontextchanges
Default
Context 1 Context 2
public void testExample(){ Assertions
1
contextChange(contextA);
2
clickSomewhere() ; ∅
3
enterText () ;
4
clickButton () ;
5 Block 1 A d.1 A 1.1
...
6
7 assertThat () ; es
g
n
8 a
h
9 contextChange(contextB); C Block 2 A d.2 A 1.2 A 2.2
doSomeOtherThings() ; xt
10 e
... nt
11 o
C
assertThat () ;
12
Block 3 A d.3
}
13
The solution is expected to be most beneficial uti-
lizing a generator that automatically creates test cases Block 4 A d.4 A 2.4
fromblocks. Manualeffortforwritingcontext-sensitive
testcasesisreducedandredundanttestcodeminimized.
Moreover,thereisaseparationofconcerns: Operations
Figure 2: Concept of block-based context-sensitive
withinblocksareonlyusedtoaddressanapp’sfunction-
testing
ality;togetherwithcontextchangesinbetweentheyare
usedtoassessthebehaviorunderchangingcontexts.
Asmanyappsmaybehavedifferentlyincertainsitu- Thesecondblockisexecutedsimilarlytothefirstone;
ations(e.g. displayerrormessageswhennoconnection a context-specific assertion part is used as well. This
isavailable),assertionshavetobeextractedfromblocks changeswhenreachingBlock3.AscanbeseeninFig-
duetotheirpotentialdependencyonacontext. Foreach ure 2, this block does not have any specific assertions.
block at least one default assertion has to be assigned. Consequently, the expected behavior is invariant be-
Thisassertionisusedtoverifytheapp’sstandardbehav- tweenvariouscontexts. Therefore, thedefaultassertion
ior. For each known context, a specific assertion may thatverifiesthisbehaviorisevaluated. Finally,Block4
be defined that is used to assess context-dependent be- is executed together with assertion A2.4, which is the
havior. The blocks shown in the schematic test case in assignedassertionforContext2.
Listing 1 therefore have to be tailored smaller. Strictly As the matrix in Figure 2 shows, there are numerous
spoken,assertionsinlines7(BlockA)and12(BlockB) execution paths considering the fact that prior to each
arenotpartoftheblocksbuthavetobetreatedasanother blockthecontextischangeableandalternativeassertions
building block of our concept. Depending on the app’s can be defined. The matrix grows with the number of
expectedbehavior,assertionsmaybedefaultorcontext- contexts but will be sparse when not defining specific
sensitive. assertionpartsforallblocks.
The building blocks of the concept of block-based Using the concept of block-based context sensitive
context-sensitive testing are shown in Figure 2. The testing,thestructureofthetestremainsstaticasthelist
structure of a test case is depicted by an ordered list of of blocks in the sample shows. However, the context
blocks. Each of them has at least one default assertion. inbetweenischangingandalsotheusedassertionscan
Furtherassertionsfordifferentcontextsmaybeadded. vary.
4.3 Context-DependentTestCaseExecution 4.4 SampleScenario
A possible test execution is illustrated in Figure 3. The Toillustratepracticalbenefits,weimplementedaproof-
beginningofthetestcaseisdenotedastheemptyset∅. of-concept tool and evaluated the approach in coopera-
Beforethefirstblockisexecuted, Context1isestab- tion with an industry partner. We use a simple app to
lished. Next, an assertion fitting to the current context explain how it works: a client for the micro blogging
issearched. Sincethisblockhasaspecificassertionfor service Twitter. The app contains two screens, one for
Context1, this one is used. As the assertion holds, logging in to the service and another for posting mes-
execution continues. Between Block1 and Block2 sages. Error messages are shown when communication
thecontextischangedagain. Thistime, Context2is withtheserviceisimpossibleduetomissingconnectiv-
establishedandkeptfortheremainingexecution. ity.
33
OpenJournalofWebTechnologies(OJWT),Volume2,Issue1,2015
Context 1 Context 2
Context Context
∅ Block 1 Block 2 Block 3 Block 4
Change Change
A 1.1 A 2.2 A d.3 A 2.4
Figure3: Exampleoftestexecutionusingblock-basedcontext-sensitivetesting
To prepare the setting, a jar archive containing our context and their default assertions are used for veri-
proof-of-concepthastobeaddedtotheclasspathofthe fying. Before executing the third block, the context is
app’sAndroidtestingproject. Moreover,astheInternet changed. Thecontextdependentassertionforthatblock
connectivityhastobechangedtotesttheappinvarious checkswhetherinthedisconnectedcontextthemessage
contexts, thepermissionsshowninListing2havetobe “Nonetworkconnection”isshown.Itbecomesclearthat
added to the AndroidManifest.xml of the app if eveninasmallscenarioasillustratedherealotofdiffer-
notalreadycontained. ent execution paths are possible, which underlines why
testingwithoutourapproachwouldbeburdensome.
Listing2: RequiredAndroidpermissions
<uses−permission android:name=” ←(cid:45) Listing3: Sampleblockimplementation
1
android.permission. ←(cid:45) 1 Context discContext = new Context( ←(cid:45)
ACCESSNETWORKSTATE”/> ConnectionStatus.DISCONNECTED);
2 <uses−permission android:name=” ←(cid:45) 2
android.permission. ←(cid:45) 3 Block loginIncorrectBlock =new Block(){
CHANGENETWORKSTATE”/> 4 public void operation() {
5 enterText(usrName, ”[email protected]”);
6 enterText(pwd, ”1234”);
Thefollowingsampletestisconductedinformofan
7 clickOnButton(0);
integration test, testing the app in conjunction with the 8 }
real system environment. The high level process steps 9
are: logginginwithinvalidcredentials, logginginwith 10 public void defaultAssertion() {
11 assertTrue(waitForText(” ←(cid:45)
correct credentials, and posting a message. These three Authentication failed!”));
stepsformtheblocksthatareusedtotesttheTwitterapp. 12 }
Eachstepcanlaterbeconductedinadifferentcontext. 13 };
14
Listing3(seenextpage)showsthefirstblockimple- 15 loginIncorrectBlock. ←(cid:45)
mented with our solution. The test operations, namely addAlternativeAssertion(discContext , ←(cid:45)
fillinginusernameandpasswordandclickingthelogin new Assertion() {
buttonafterwards,areimplementedintheoperation-part 16 public void assertion() {
17 assertTrue(waitForText(
of the block (lines 4 to 9). The credentials used in that 18 ”Could not login: No network ←(cid:45)
blockareinvalid;therefore,theappisexpectedtoshow connection.”
acorrespondingmessage. Thisischeckedbythedefault 19 ));
assertion(lines10to12). However, iftheappisindis- 20 }
21 });
connected context, it is expected to show an error mes-
sage. In order to verify this context-dependent behav- We used two sample generators to create test cases
ior,analternativeassertionisaddedforthedisconnected fromtheblocksexplainedabove.Onealternatesthecon-
context(lines15to20). text between each block. Another generates n+1 test
Foreachoftheabovestatedprocessstepsablockwith caseswherenisthenumberofblockscontained.Ineach
context-specificassertionsisimplementedandaddedto test case the generator changes the context at different
a list of blocks. Using a generator that creates differ- steps in the process. For the first test case the context
entmutationsintermsofcontextchangesfromthatlist, ischangedbeforethefirstblockisexecuted. Inthesec-
test cases are generated. Figure 4 shows one possible ond testcase it ischanged after thefirst block hasbeen
testcase. Thefirsttwoblocksareexecutedinconnected executed,andsoon.
34
T.A.Majchrzak,M.Schulte:Context-DependentTestingofApplicationsforMobileDevices
Figure5: ExampleJUnitresult
tested a few apps and evaluated our approach of block-
based context-sensitive testing. By conducting inter-
viewsanddoingtestsasshowninSection4.4weidenti-
fiedthecorecontributions.
We found that by using our proof-of-concept imple-
mentation it is possible to test apps in various contexts
effectively. Blocks of test code are reusable and thus
Loginincorrect Logincorrect Lostconnection theeffortinwritingtestsisreduced. Thisreductionalso
holdstruefortheamountofduplicatedtestcodewhich,
Figure 4: Example process steps for test case execu- inpractice,wouldbehardtomaintain.
tionforvariousconditions By using generators the concept even gets more ef-
fective in terms of finding context-related issues in
apps. Testers only have to implement blocks and de-
Finally, test cases are executed by means of JUnit.
finecontext-specificassertionsonce,butmanytestcases
Like any other test for the Android platform, they are
aregeneratedautomatically. Whenwritingmanualtests,
runningonthedeviceoremulatoritself. Figure5shows
thecontextchangesmayonlybeinsertedatpartswhere
test results for one execution of the above stated exam-
context-related issues are anticipated. However, using
ple.
ourapproachhelpstofinderrorsincodeunitswherethey
arenotexpectedbeforehand.
5 DISCUSSION Manually conducting field tests, i.e. practical tests
in various contexts, is often expensive in terms of time
5.1 ExemplaryTool,ApplicationandMerits and budget. Thus, they are neglected frequently. Our
conceptoffersanalternativetoconductatleastsomeof
Todemonstratetheproducibilityofourconcept,wehave
these tests in laboratory conditions. Doing so, context-
implementedanexemplaryAndroidtool. Itiswrittenin
relatedissuesmaybeidentifiedbeforereleaseofanapp,
JavaandcanbecheckedoutfromGitHub[11]. Release
which could reduce costs for fixing them and increases
undertheApacheLicenseallowsfreeusageandmodifi-
theapp’squality.
cation. Itcanbeinspectedandusedinotherprojects. In
However,ourconceptcannot(yet)beapanaceawhen
particular,studyingthesourcecodewillallowrapidde-
dealing with context during development and testing of
velopment of similar tools for platforms such as Apple
apps. Wedeemitasupportivemeanfortestingpartsof
iOSandMicrosoftWindowsPhone.
appswhichareheavilyinfluencedbycontext.
Whilethetoolisnotacorecontributionofourwork,
it demonstrates the feasibility of our approach. In par-
5.2 Limitations
ticular, it allows to retrace our concept and to check its
practicability for own scenarios. Moreover, the imple-
Duetothenoveltyofourapproach,anumberoflimita-
mentation can be seen as a first step towards powerful
tionshavetobementioned.
context-basedtestingtools.
Firstly,thereareconceptuallimitations.Havinganor-
Together with an industry partner2, we exemplarily
dered linear list of blocks allows for only one way of
2Theindustrypartneristheemployerofoneofus(Schulte)atthe execution.However,changesofcontextmayalsoinduce
sametime. different business processes to be executed. Thus, the
35
OpenJournalofWebTechnologies(OJWT),Volume2,Issue1,2015
order of blocks and the decision if a block is executed thatcoverallpossiblepaths.
mayalsodependonthecontextused. Forinstance,ifin To make the approach more versatile, we also intend
our above sample test (see Section 4.4) the login block tolearnfromotherworksthatdealwithscalingupcom-
would be executed in disconnected context, the test has puter science concepts. Certainly, for larger programs
tobestoppedbecausetheotherprocessstepscannotbe means will be needed to cope with an ever-increasing
reached. Weintroducedamethodforabortingtestcase number of concepts and possibilities. An example of
executioninourproof-of-concepttocopewiththis.That approaches that could be worth a look is the work on
methodhastobeincludedinassertionpartscorrespond- AHEADbyBatory,SarvelaandRauschmayer[5]. Also,
ing with contexts causing the business process to stop. we will try to relate even stronger to existing concepts
Anyhow, our concept is not able to deal with process toovercometheremainingchallenges. Forexample,in-
steps that only have to be executed in certain contexts. sightsfromaspect-orientedprogramming(AOP)regard-
Closelyrelated,assertionpartsareonlydependentonthe ingassertionhandling[25]andeven-driventestingtech-
currentcontextbutnotonthepriorexecutionpathofthe niques[28]mightbeapplicable.
test. Inpractice,theexpectedbehaviourofprocessstep Best practices for context-sensitive testing would
may change according to the context the prior steps are complement the work with our approach. Compil-
executed in. Overcoming this limitation requires a re- ing them requires a base of regular users and a lot of
finement. time. Nevertheless, increased awareness of context is-
Secondly,thecasestudyisafirstexampleoffeasibil- sueswouldfacilitateprogress.
ity,yetnotexhausting. Effectivenesshastobeprovenin Achallengeistobecomeabletocopewithamplecon-
industrial settings, both qualitatively and quantitatively. texts. While our approach theoretically is capable of
We got very positive feedback from an industry partner dealing with arbitrary context changes, actually simu-
and approval from the practitioners’ testing community lating these changes for testing is very hard. Problems
(seeSection5.1). Feedbackhastobeassessedscientifi- particularly arise from parallel (or rather quasi-parallel)
callytobecomeavalidevidence,though. changes of contexts. As a consequence, future work
Thirdly, our tool is a prototype and, thus, poses lim- needstorefineourconceptandinvestigateintothecon-
itations. In particular, it is scarcely commented and it sequencesofcontextchanges(somemighte.g. beharm-
is not yet very user friendly. Due to employing testing less). Researchcouldleadtobettertoolsforsimulation,
bestpracticessuchasrelyingonacommonunittesting makingourapproachevenmoreuseful.
toollikeJUnit[26]itiseasytouseforprogrammersand
technicaltesters. Fordomainexperts,hobbyistprogram-
6 CONCLUSION
mers, and to a lesser extent for pure Web developers it
will, however, becumbersomeduetoamissinggraphi-
In this paper we presented a novel approach for testing
caluserinterface.
apps. Wecoineditblock-basedcontext-sensitivetesting
These limitations have to be kept in mind yet none
sinceweaddresscontextchangeswhiledefiningblocks
ofthemquestionthegeneralfeasibilityofourapproach.
between which they can take place. The approach en-
Theyalsoleadtofuturework.
ables much more versatile and effective testing, while
the overhead is small due to the relatively low level of
5.3 ChallengesandFutureWork
complexity. Ourconceptofcontextchangesextendsthe
literatureofcontext-sensitivityinmobilecomputing.We
Onemajorpracticalchallengeinusingourapproachisto
haveshowntheviabilityofourapproachinacasestudy
findtherightsizeofblocksinordertobalancetheeffort
andbypresentingaprototype.
of slicing test cases into blocks with various assertions
Testingofappsisfarfrombeinghassle-free. Weex-
againsttheprobabilityoffindingadditionalfaultsbydo-
pect new challenges to arise due to the extended capa-
ingso.“Bestpractices”,orguidelinesofhowcontextde-
bilitiesofdevicesandplatforms, e.g. w.r.t. sensorsand
pendenttestcasesmaylooklike,needtobedeveloped.
parallel execution. The proliferation of the Internet of
Forfuturework,anextensionofourtoolcouldprove
Things(IoT)willlikelycontributetothecomplexitythat
beneficial. One idea is to enable graphical modelling,
needs to be coped with. Our work on context-sensitive
whichwouldeasetestcasecreation. Ideally,modelse.g.
testingwillcontinue.
describing business processes could be reused. More-
over,bettersupportfortestcasegenerationw.r.t. context
selectionisdesirable. Forexample, thetoolcouldkeep ACKNOWLEDGMENTS
trackofassertionsandcontextsandprovidetesterswith
reasonablesuggestions. Additionally,itcouldtrackcov- We would like to thank viadee Unternehmensberatung
erage and advise on finding minimal sets of test cases GmbHfortheirsupport.
36