Table Of ContentWhere Do We Stand in Requirements Engineering
Improvement Today? First Results from a Mapping Study
Daniel Méndez Stefan Wagner Maya Daneva
Fernández, Saahil UniversityofStuttgart UniversityofTwente
Ognawala [email protected] [email protected]
TUMünchen,Garching stuttgart.de
mendezfe,[email protected]
ABSTRACT exist secondary studies that deal with the larger context of
7 Requirementsengineeringprocessimprovement(REPI)ap- SPIbutnonesofarforimprovingREconcerningallitspar-
1 proaches have gained much attention in research and prac- ticularities. We aim to consolidate the current understand-
0
tice. Sofar,however,thereisnocomprehensiveviewonthe ing about the state-of-the-art by conducting a systematic
2
research in REPI in terms of solutions and current state of mapping study of all publications on RE process improve-
n reported evidence. This paper aims to provide an overview ment. In this paper, we report on our results and focus, as
a on the existing solutions, their underlying principles and a first step, on categories of publications according to re-
J their research type facets, i.e. their state of empirical ev- search type facets, the contribution phases, paradigms and
9 idence. To this end, we conducted a systematic mapping theirunderlyingprinciples. Thecompletedataandanalysis
1 study of the REPI publication space. This paper reports can be found in [4].
on the first findings regarding research type facets of the
] contributions as well as selected methodological principles. 2. STUDYDESIGN
E
Wefoundastrongfocusintheexistingresearchonsolution
Our study design follows the standard procedures of a sys-
S proposals for REPI approaches that concentrate on norma-
tematicmappingstudy[5]. Wedidthisinconjunctionwith
s. tiveassessmentsandbenchmarksoftheREactivitiesrather the methods of a systematic literature review which entails
c than on holistic RE improvements according to individual
a further in-depth analysis of selected publications.
[ goalsofcompanies. Weconclude,sofar,thatthereisaneed
tobroadentheworkandtoinvestigatemoreproblem-driven
1 2.1 ResearchQuestions
REPI which also targets the improvement of the quality of
v To systematically describe the state-of-the-art, we will an-
the underlying RE artefacts.
7
swerthefollowingresearchquestionsonREPIpublications.
9
4 Keywords
RQ1: Of what type is the research? Asafirststep,we
5
Requirements Engineering Process Improvement, System- willclassifytheREPIpublicationsaccordingtotheresearch
0
atic Mapping Study type facets as described by Wieringa et al. [8]. A research
.
1 type facet is an abstract description of the activity stage in
0 1. INTRODUCTION the engineering cycle that is in scope of a contribution. We
7 alsoaimtospottrendsinthefacetsofREPIpapersoverthe
Requirements engineering (RE) aims at the discovery and
1 years. Welisttheavailableresearchtypefacetcategoriesin
specificationofrequirementsthatunambiguouslyreflectthe
: Tab. 1.
v purpose of a software system. Thus, RE is an important
i factor for productivity and quality. Given the practical
X RQ2: Which process improvement phases are con-
importance of RE, it remains a complex discipline driven
r by uncertainty [2] which eventually makes RE hard to in- sidered? Havingclassifiedtheoverallcontributionsaccord-
a ingtotheirfacet,wewanttoknowwhetherthosecontribu-
vestigate and even harder to improve [3]. Even though a
tionstakeaholisticviewonREPIorwhethertheyfocuson
significant number of contributions have been made in the
selected improvement phases only. We distinguish between
research field of requirements engineering process improve-
(a)Analysis wherethefocusliesonanalysisandassessment
ment (REPI), we do not have exhaustive knowledge about
of a RE, (b) Construction where the focus lies lies on the
the proposed solutions, the problems they address and the
(re-)design of a RE process and, thus, on the actual im-
state of evaluation and validation of these solutions. There
provement realisation, (c) Validation where the focus lies
onthevalidationofanimprovementendeavour,and(d)RE
ProcessImprovementLifecycle(REPI-LC)wherethecontri-
butiontakesaholisticviewonallphasesand/orongeneral
metrics and measurements.
RQ 3: What paradigms do the publications focus
on? We distinguish between activity-oriented and artefact-
orientedparadigmsbasedonwhetherthepublicationsfocus
onimprovingthequalityoftheactivitiesthatformapartof
contributiondataaretechnicalreportsandacademicstudies
Table 1: Definitions of research type facets [8].
informofPhDtheseswhichoftendonotformapartofthe
Validation Techniquesinvestigatedarenovelandhave
searchresultspaceintheabovelistedsources. Wesearchfor
paper not yet been implemented in a large scale
such contributions using Google Scholar which has a wider
industrial or academic setting.
Evaluation Techniquesareimplementedandevaluated spanthatindexestitleslocatedinrepositorieslikeuniversity
paper in a large scale industrial, academic or databases and other independent publications.
other real world setting.
Solution A solution to a problem is proposed, ei-
2.4 InclusionandExclusionCriteria
proposal ther novel or an extension to an existing
solution. Once we have a set of contributions from the publication
Philoso- It proposes a new way of looking at exist- databases,weusealistofinclusion(IC)andexclusion(EC)
phical ing problems by re-structuring the field in
criteria(describedinTab.2)onthisdatasetbeforetheanal-
paper formofataxonomy,conceptualframework
ysis and voting stage.
or systematic literature review.
Opinion The authors present their opinion on a
paper problem space with a critical view on one Table 2: Inclusion and exclusion criteria.
or more solutions described by other re- IC The paper directly relates to REPI.
1
searchers trying to tackle the problem. IC The title and abstract refer to REPI.
2
ExperienceItprovidesaretrospectiveviewontheau- IC The keywords contain related words.
3
paper thors’ experience in developing, applying IC The contribution addresses the research ques-
4
and evaluating a certain technique in the tions, i.e. it
field of engineering process improvement. ...introduces, discusses, compares, or evalu-
Explo- It deals mainly with the problem space ates...
ratory withabird’seyeviewofthecommonprob- ...approaches or experiences, terms and con-
paper lems faced by various solutions proposed. cepts and/or metrics to...
...improve (assess and/or implement and/or
evaluate)requirementsengineeringprocessesor
artefacts.
theREprocessesoronimprovingthequalityoftheREarte-
EC The paper addresses SPI in general (without
facts created. If contributions do not differentiate between 1
clear linkage to RE).
the paradigms (e.g. when providing a set of metrics and
EC The topic does not address approaches, studies
2
measurements)orsimplyincludeideasdealingwithnoneof or experiences for improving requirements en-
thetwoparadigmsinparticular,thenwewillnotassignthe gineering but new approaches and techniques
contribution any paradigm focus. that are claimed to improve RE as an effect of
applying them (e.g. elicitation techniques).
EC No scientific publication, i.e. PowerPoint pre-
RQ4: Aretheunderlyingprinciplesofnormativeor 3
sentations, abstracts or posters.
of problem-driven nature? We categorise a publication
EC The contribution’s language is not English.
4
either as normative, where a given RE activity/artefact is EC The contribution is not available.
5
assessed and improved against a given external norm, or as EC The contribution appears multiple times in the
6
problem-drivenwheretheimprovementisconductedagainst result set (see below).
company-specific goals and problems (see also [6]). EC7 Thecontributioninvestigates(industrial)prob-
lems in RE to be addressed by research to im-
prove RE.
2.2 StudySelection
AsPetersenetal.[5]recommended,westartedourmapping
Among contributions where the same approach is reported,
study with an exhaustive search of a publication database
weonlychooseonetoincludeinourstudy;e.g.PhDtheses
with the key concept terms in REPI. We did a pre-analysis
formingacumulativereportofvariousapproaches. Wetreat
of a selected set of key publications in the REPI area and
papers where several techniques or approaches are reported
made a map of the chief search terms that seemed closely
as a single contribution. Systematic literature reviews are
related to these papers. Next, we performed snowballing
treated as philosophical papers [8] because they define and
ontheselectedpublicationsassuggestedbyKitchenhamet
organise existing concepts and approaches taking a novel
al. [1]. This gave us a large initial dataset with a list of key
view. In case of metrics being introduced in a paper that
publications and the main concept keywords. We can now
can be applied to both artefact or activity orientation, we
formthesearchquerystringsandmodifythembasedonthe
set the paradigm to ”N/A”. Table 3 gives numbers of pa-
quality of the search result set (as compared to the initial
pers at each stage of data processing. Results seen are all
dataset).
the results returned by the database search in the top 20
pages. Included papers are the ones that were kept after
2.3 DataCollectionProcedures
filteringtheseenresultsintheanalysisstagebymakinguse
Our data collection procedure is an automated search on of the inclusion and exclusion criteria in Tab. 2. We then
established web databases including ACM Digital Library, undergothefirstroundofvotingwherewefurtherfilterout
SpringerLink, ScienceDirect and IEEE Xplore. more publications based on their relevance to our research
questions so as to derive the Relevant result set.
Weusethekeywordspresentintheinitialdatasettodefine
search query terms. Lists of prominent contributors in the
domain and their publications are a control mechanism to 2.5 AnalysisandVotingProcedure
filteroutirrelevantsearchresultsandtweakthesearchstring Wedidastagedvotingprocedureonthesetof58papers,as
correspondingly. Another set of notable additions to the indicatedinTab.3. Eachseniorresearcher(Daneva,M´endez
yetproposedforthecontractionphase,i.e. therealisationof
Table 3: Dataset summary.
actualimprovementforecastsidentifiedinananalysisphase.
Database Total Results
Name results seen Included Relevant
ACM 81 81 23 15 3.3 ContributionParadigm(RQ3)
SpringerLink 349 349 31 11 We found that 48 out of 58 papers presented the activity-
ScienceDirect 132 132 12 2
oriented paradigm as the one adopted in REPI. In 7 out of
Google 276 276 16 11
the 58 papers we could not find enough evidence indicating
Scholar
IEEE Ex- 2,819,217 275 18 15 thekindofparadigmadoptedasthefocuswas,forexample,
plore onmetricsandmeasurementsusedinvariousREPIphases.
Misc. 4 4 4 4 However,itremainedunclearifmeasurementssupportedac-
Total 2,820,059 1117 104 58 tivityorientedorartefact-orientedREPI.Thefocusofmost
contributions lies on normative improvements focusing on
the RE activities carried out. This is in tune with our ob-
and Wagner) worked individually to categorise every pub- servation that many contributions are an extension of the
lication according to the research type facets, the lifecycle initialworkproposedbySawyeretal.,thus,thosecontribu-
phase,thetwoparadigmsandthetwounderlyingprinciples. tions focus on how to assess RE processes against a given
The voting procedure allowed us to put forward arguments normconsistingofproposedREbestpracticesconsideredto
regarding our respective choices on assigning a paper to a comprise an external notion of“good RE”.
category of relevance to the RQs. Once each researcher’s
individual categorisation was over, the researchers got to- 3.4 ContributionPrinciple(RQ4)
gether to compare and contrast their categorisations until
Fig. 2 reports our findings on the classification of the prin-
reachingagreement(occasionallybasedonin-depthanalysis
cipleandmapsthemagainsttheparadigms(RQ2). 41out
of the paper). Each voting stage ended with a consensus-
driven discussion among the three researchers on assigning
categories to a paper on which there were disagreements. N/A
We conducted the voting procedure over four stages yield-
ing at each stage the agreement level in the classification
Artefact
subsequently shown:
Orientation
1st stage: 53.4 % (31/58)
2st stage: 72.4 % (42/58)
3st stage: 86.2 % (50/58) Activity
Orientation
4st stage: 100 % (58/58)
Normative Problem-Driven
3. RESULTS
We present our findings structured according to our RQs. Figure 2: Paradigms and principles.
3.1 ResearchTypeFacet(RQ1) of 58 papers fall in the normative category where the fo-
Fig.1,rightside,showsthedistributionofthecontribution cusliesmostlyonimprovingactivitiesratherthanREarte-
over the years and according to their research type facets. facts. The papers in this category mostly deal with sets of
41 out of 58 publications analysed were solution proposals best practices as external norms where the current state of
or evaluation papers. We can see a lack of retrospective RE in a company is assessed and aligned with those best
analysis in the form of experience reports which we con- practices.
sider important given that especially RE and, thus, its im-
provementstronglydependsonsubjectivity(beliefs,desires, 4. CONCLUSION
fears, experiences and expectations), but in general a regu-
In this paper, we presented first results from a mapping
lar distribution over solution proposals and evaluation pa-
study on the current state of REPI. Based on 58 primary
pers. Overall,themapsuggeststhebeginningofresearchin
studies,weshowedthatmostcontributionsfocusontheim-
this area on basis of Sawyer et al. [7] introducing the REPI
provement of RE activities while the improvement of RE
based on a set of what they consider to be best practices.
artefacts is barely discussed. In addition, most improve-
The results also indicate that most of the proposed REPI
ment approaches focus on a normative improvement where
approaches focus on extending the basic ideas introduced
external norms of best practices are taken as a reference.
bySawyeretal. andgroundingREPIonbestpractices(see
Considering that most contributions focus on assessments
also subsequent RQs).
rather than on realising an improvement, we conclude that
availablecontributionsprovidemeanstorateandassessthe
3.2 Phaseofcontribution(RQ2) currentstateofREincompaniesagainstanexternalnorms
38outofthe58publicationsfallundertheanalysisphase of of activity-centric best practices of which most arise from
the REPI lifecycle. Fig. 1, left side, shows the distribution initial contributions made by Sawyer et al. [7].
of the publications over all the phases and depicts which
researchtypefacetareinscope. Themapsuggeststhatmost Ourstudyfurtherrevealedthatveryfewexploratorypapers
of the papers focus on the analysis phase while little seems havebeenpublishedinREPI.Thismeanswehaverelatively
38 2 16 2
6 3 Exploratory 1 1 1 2 1 1 1 1 9
4 Experience 1 1 1 1 4
1 Opinion 1 1
1 Philosophical 1 1
18 2 3 1 Solution 1 3 4 1 1 5 2 3 3 1 24
11 5 1 Evaluation 2 1 1 3 4 3 1 2 17
1 1 Validation 1 1 2
Analysis Construction REPI-LC Validation 1996 1998 2000 2002 2004 2006 2008 2010 2012
Contribution Phase Distribution per Year
Figure 1: Facet distribution according to years and lifecycle phases.
little evidence (and hence, knowledge) about the full range publications of leading REPI authors.
of problems that organisations face. In turn, the general
applicability of the solution proposals might well be com- Second,itmightbepossiblethatwecollectivelycategorised
promised. For example, as currently RE is applied more apaperinawrongway. Wecounteredthisbyimplementing
and more to new domains (e.g. smart city systems), we afour-stagevotingprocedurefocusedonargumentation,re-
think it would be unrealistic to assume that the solutions peated reviews and consensus building. We therefore think
proposals would catch up with the REPI needs of organi- the risk of this threat is minimal. Yet, the RE paper classi-
sations executing projects in those domains. We therefore fication in [8] has not been created with systematic reviews
call for more exploratory studies in REPI to identify and in mind, and at times we found it difficult to categorise a
betterunderstandcommonproblemsfaced. Next,wefound paper to only one facet.
very few (3 out of 58) papers taking an artefact-oriented
perspective on REPI. Clearly, aspects such as effectiveness 5. REFERENCES
and efficiency of RE activities are related to activities and [1] B. Kitchenham, O. Pearl Brereton, D. Budgen,
this might explain the massive amount of papers adopting M. Turner, J. Bailey, and S. Linkman. Systematic
activity-oriented paradigm for REPI. However, are the RE literature reviews in software engineering - a systematic
activitiesactuallyproblematicintherealworld? Infact,em- literature review. In IST.
piricalREpapersreportofproblematicartefacts(e.g.spec-
[2] D. M´endez Ferna´ndez, S. Wagner, K. Lochmann,
ifications, models). We assume that the activity-oriented
A. Baumann, and H. de Carne. Field Study on
paradigmisstudiedsooftenbecauseithasestablisheditself
Requirements Engineering: Investigation of Artefacts,
through the“best-practice”movement. Last, we found the
Project Parameters, and Execution Strategies. IST.
majority of papers were of normative nature. It is tempt-
[3] M´endez Ferna´ndez, D. and Wieringa, R. Improving
ing to assume this finding is traceable to the established
requirements engineering by artefact orientation. In
“best-practice”thinking in the software industry. However,
PROFES’13. Springer.
even best practice gurus (e.g. Capers Jones) suggest that
[4] S. Ognawala, D. M´endez Ferna´ndez, and S. Wagner.
problem-driven improvement might yield greater benefits
Requirements engineering improvement today - a
than a big-bang best-practice based approach.
systematic mapping study. TUM-I145.
[5] K. Petersen, R. Feldt, S. Mujtaba, and M. Mattsson.
We therefore think that investing in problem-driven REPI
Systematic mapping studies in software engineering. In
that also considers the quality of RE artefacts would be
EASE’08.
worthwhileandnecessarytofullyunderstandthebroadspec-
[6] F. Pettersson, M. Ivarsson, T. Gorschek, and
trum of REPI possibilities.
P. O¨hman. A Practitioner’s Guide to Light Weight
Software Process Assessment and Improvement
Planning. JSS, 2008.
Limitations. There are two main limitations of this map-
[7] P. Sawyer, I. Sommerville, and S. Viller. Requirements
pingstudy: First,thepossiblebiasintheselectionofpapers
process improvement through the phased introduction
for inclusion as our access to relevant sources depended on
of good practice. SPI and Practice.
theappropriatenessoftheusedsearchstrings. IntheREPI
area,abroaddiversityoftermsisusedwhichimpliesarisk [8] R. Wieringa, N. Maiden, N. Mead, and C. Rolland.
that we might have missed some relevant studies. We took Requirements engineering paper classification and
extra steps to counter this risk by analysing keywords and evaluation criteria: a proposal and a discussion. In
RE’06.
Copyright ACM. This is the author’s version of the work.
It is posted here for your personal use. Not for redistri-
bution. The definitive Version of Record was published in
Proceedings of the 8th ACM/IEEE International Sympo-
siumonEmpiricalSoftwareEngineeringandMeasurement,
http://dx.doi.org/10.1145/2652524.2652555.
