Organizing to Accomplish the Mission The

110 Pages·2012·29.49 MB·English

The
V
o
lu
m ITEA Journal
e
3
2
N
o
. 2
J
u
n
e
2
0
1
1
, p June 2011
a
g Volume 32, Number 2
e
s
1
1
7
–
2
2
4

O
rg
a
n
iz
in
g
t
o
A
c
c
o
m
p
lis
h
t
h
e
M
is
s
io
n

T
h
e Organizing to Accomplish
IT Published quarterly
E
A Jo by the International the Mission
u Test and Evaluation
rn
a Association
l
CONNECT with ITEA to LEARN, SHARE, and ADVANCE!
The
V
o
lu
m ITEA Journal
e
3
2
N
o
. 2
J
u
n
e
2
0
1
1
, p June 2011
a
g Volume 32, Number 2
e
s
1
1
7
–
2
2
4

O
rg
a
n
iz
in
g
t
o
A
c
c
o
m
p
lis
h
t
h
e
M
is
s
io
n

T
h
e Organizing to Accomplish
IT Published quarterly
E
A Jo by the International the Mission
u Test and Evaluation
rn
a Association
l
CONNECT with ITEA to LEARN, SHARE, and ADVANCE!

C O N T E N T S
OPERATIONAL TEST AGENCIES
The ITEA Journal ORGANIZING TO ACCOMPLISH THE MISSION
June2011
143 AFOTEC:TestersMustbePartoftheSolution:BeginWiththeEndinMindandLeverage
Volume32,Number2
AbilitytoInfluenceandInform.......................................................Maj.Gen.DavidJ.Eichhorn
146 OPTEVFOR:OrganizingtoAccomplishtheOT&EMission(IntegratedTestingThrough
BOARD OF DIRECTORS
MissionBasedTestDesign).............................................................RADMDavidA.Dunaway
StephanieH.Clewer,President
MarkD.J.Brown,Ph.D., 149 JITC:AcceleratingtheNation’sITDominance................................................ColonelJoePuett
VicePresident
CharlesMcKee,Secretary ITEA BEST PAPERS
MichaelT.McFalls,Treasurer
StewartBurley
159 DevelopmentofaTorsionalThrustBalanceforthePerformanceEvaluationof5-NClass
B.GeneHudgins
Thrusters...........MarjorieA.Ingle,JesusR.Flores,NathanielRobinson,andAhsanChoudhuri,Ph.D.
StevenJ.Hutchison,Ph.D.
Charles‘‘Bert’’Johnston 167 SustainabilityviaEnergyHarvestingandScavenging.........JoseBarajas,AnandSagarBokka,Isaac
WilliamT.Keegan Fernandez,JuanGonzalez,AhmadHammad,JavierHernandez,SughoshKuber,DanielMarquis,Ravi
TedMcFarland ChandraMikkilineni,ChaitanyaMotupalli,AbdiMusse,VineetNair,GurNotani,JavittHigmar
GeorgeR.Ryan Padilla-Franco,BalwinderSingh,PrasantSinha,MuraliSithuraj,PaariSivaprakasam,Vanishree
JohnSmith
NarayanVenkatesan,andNadipuram(Ram)R.Prasad,Ph.D.
MarkE.Smith
KeithSutton
TECHNICAL ARTICLES
JohnL.Wiley
SENIOR ADVISORY BOARD 181 BackgroundDiscriminationof Bioaerosols.........JamesJ.Valdes,Ph.D.,RaymondMackay,Ph.D.,
RussellL.‘‘Rusty’’Roberts,Chair JoshuaL.Santarpia,Ph.D.,StephenAdams,Ph.D.,EwelinaTunia,andAndrewMara,Ph.D.
CharlesF.Adolph 191 TheMulti-RelationshipEvaluationDesignFramework:ProducingEvaluationBlueprintsto
BrentM.Bennitt
TestEmerging,Advanced,andIntelligentTechnologies.....BrianA.WeissandLindaC.Schmidt
JohnV.Bolino
EdwardR.Greer 201 AnalyticalToolsofMathematicsandStatistics(ATOMs)—SmallStepstoInnovation............
GeorgeB.Harrison .......................................................................................................................................MarkH.JonesJr.
CharlesE.McQueary,Ph.D.
206 AMeasurefortheDeclineofShort-TermWorkingMemoryCapacityWithCoincidental
J.DanielStewart,Ph.D.
SleepDeprivation.....................................................................................................DavidBate
MarionL.Williams,Ph.D.
210 NASATaskLoadIndexforHuman-RobotInteractionWorkloadMeasurement......................
COMMITTEE CHAIRS
......................................................................JoshuaA.Gomer,Ph.D.andChristopherC.Pagano,Ph.D.
Awards
AlbertA.Sciarretta
Chapter&Individual DEPARTMENTS
MembershipDevelopment
KeithSutton 117 PRESIDENT’SCORNER
CorporateDevelopment
Charles‘‘Bert’’Johnston 119 ISSUEATAGLANCE
Education 123 GUESTEDITORIAL:RETHINKINGTESTANDEVALUATIONFORANEWAGE.....DavidS.Alberts,Ph.D.
RandonR.Herrin
Elections 127 TECHNOTES:TODETERMINEANETWORKEDTELEMETRYTRANSCEIVER
GaryL.Bridgewater RESYNCHRONIZATIONTIME..................................................................................DanielT.Laird
Events 132 HISTORICALPERSPECTIVES:INTEROPERABILITYIMPACTSTHEWARFIGHTER:ORGANIZINGTO
DouglasD.Messer ACCOMPLISHTHEJOINTINTEROPERABILITYTESTINGMISSION................Randon(Randy)R.Herrin
Historian andChrisWatson,WithContributionsFromGeneFenstermacher,Colonel,USAFRetired
JamesWelshans,Ph.D.
Publications 215 CHAPTERLOCATIONS
J.MichaelBarton,Ph.D. 215 T&ENEWS
Technology
Vacant 223 ITEACORPORATEMEMBERS
WaysandMeans 224 ADVERTISINGRATES
MichaelA.Schall
STAFF
ExecutiveDirector
JamesM.Gaidry,CAE
ExecutiveAdministrator ONTHECOVER:Theprecursortocurrentdaysimulationbeganinthe1940swiththedevelopmentof
JeanShivar thefiniteelementmethodforstructuralanalysis,andcomputationalfluiddynamicsforweapondesign,
DirectorofEvents aircraftdesign,andweatherforecasting.Inthe1960sitcameintoitsownwithimprovedunderstanding
EileenG.Redd ofnumericalmethodsanddevelopmentofscientificcomputers,especiallythoseofUnivacandControl
DirectorofEducation
DataCorporation.Vectorcomputers,parallelcomputers,andcommodityclustershavecontinuedto
JohnJ.‘‘Jay’’Weaver
changethefaceofsimulationandmadeevermorecomplexconfigurations,scenariosandmulti-physics
Advertising,Exhibits,and
SponsorshipSalesManager phenomenaaccessible.Insteadofanalyzinganindividualplatformwecanrepresentabattlespacewithits
BillDallas plethoraofvehicles,weapons,sensors,players,dynamics,andinteractions.Itmaynowbethecasewhere
Manager,Membership,and computationalcapacityoutstripsourunderstandingofphenomenatobemodeled,andlegacyalgorithms,
ChapterSupport programminglanguages,andoperatingsystemscannotfullyutilizetheemerginghardware.Thisissue
BonnieSchendell
addressesthestateofsimulation,shedslightonsomelimitationsinrealizingpastexpectations,and
ManagingEditor,ITEAJournal
RitaA.Janssen providessuggestionsformovingforward.(CovergraphiccourtesyofMr.MattGuyoftheAirForce
OperationalTestandEvaluationCenter.)
32(2)N June2011 fmi
CONNECTwithITEAtoLEARN,SHARE,andADVANCE!
&ITEAHeadquarters:4400FairLakesCourt,Suite104,Fairfax,Virginia22033-3899;Tel:[phone];
Fax:[phone],E-mail:[email protected];Website:http://www.itea.org.
&ForoverthirtyyearstheInternationalTestandEvaluationAssociation(ITEA),anot-for-profiteducation
organization,hasbeenadvancingtheexchangeoftechnical,programmatic,andacquisitioninformationamongthe
testandevaluationcommunity.ITEAmemberscometogethertolearnandsharewithothersfromindustry,
government,andacademia,whoareinvolvedwiththedevelopmentandapplicationofthepoliciesandtechniques
usedtoassesseffectiveness,reliability,interoperability,andsafetyofexisting,legacy,andfuturetechnology-based
weaponandnon-weaponsystemsandproductsthroughouttheirlifecycle.ITEAmembersembodyabroadand
diversesetofknowledge,skills,andabilitiesthatspanthefullspectrumofthetestandevaluationprofession.Allof
whichissharedwithothersthroughtheITEAJournal–theindustry’spremiertechnicalpublicationforthe
professionaltester–andatITEA’sAnnualSymposium,regionalworkshops,educationcourses,andlocalChapter
events.JointhethousandsofITEAmembers–yourpeersintheindustry–incontributingtotheITEAJournaland
participatingatITEAeventssothatyoualsocanbenefitfromtheopportunitiestolearnfromothers,shareyour
knowledge,andhelpadvancetheT&Eindustry.
&TheITEAJournal(ISSN1054-0229)ispublishedquarterlybytheInternationalTestandEvaluationAssociation
at4400LakesCourt,Suite104,Fairfax,Virginia22033-3899.SingleissuecoverpriceforTheITEAJournalis$20.
ITEAoffersannualmembershipforindividuals,full-timestudents,andanyorganizationinterestedinadvancingthe
testandevaluationprofession.Annualmembershipduesare$50forindividuals($70iflocatedoutsideoftheU.S.)
and$25forfull-timestudents.AnnualCorporateMembershipduesare$500forsmallorganizations(lessthan50
employees),and$1,000forlargeorganizations(50ormoreemployees).CorporateMembershipsincludethree(3)
IndividualMembershipsforsmallorganizationsandfive(5)IndividualMembershipsforlargeorganizations.Annual
duesincludeaone-yearsubscriptiontoTheITEAJournal.Theannualsubscriptionrateforlibrariesandother
organizationsprovidingtimelyreferencematerialtogroupsis$60.Alloverseasmail(airmailorAOA)requiresan
additional$20.TheITEAJournalservesitsreadersasaforumforthepresentationanddiscussionofissuesrelatedto
testandevaluation.Allarticlesreflecttheindividualviewsoftheauthorsandnotofficialpointsofviewadoptedby
ITEAortheorganizationswithwhichtheauthorsareaffiliated.
’Copyright2011,InternationalTestandEvaluationAssociation,AllRightsReserved.Copyrightisnotclaimedin
theportionsofthisworkwrittenbyU.S.governmentemployeeswithinthescopeoftheirofficialduties.
Reproductioninwholeorinpartprohibitedexceptbypermissionofthepublisher.
POSTMASTER:Sendaddresschangesto:ITEA,4400FairLakesCourt,Suite104,Fairfax,Virginia22033-3899.
fmii TheITEAJournal
President’s Corner
ITEAJournal2011;32:117–118
Copyright’2011bytheInternationalTestandEvaluationAssociation
W
hat is the current Department of The Office of the Deputy Assistant Secretary of
Defense test community (i.e., organi- Defense, Developmental Test and Evaluation
zationsandcapabilities)?Havesevere (DT&E), created in May 2009, is focusing on early
budget constraints led involvement in the acquisition life cycle
thedefensetestandeval- by testing and evaluating earlier and
uation (T&E) organizations to a wrong more effectively. By advocating T&E
focus on investments and expenditures di- infrastructure support to rapid prototyp-
vorced from the mission of preparing and ing and medium- and small-scale mod-
equipping the warfighter? The aging work- ifications to existing weapons systems,
force continually challenges us to find critical theDT&Egoalsarecommensuratewith
skills while more complex systems demand the rest of the acquisition community—
not only a flexible workforce, but more to decrease costs, enhance performance,
‘‘System of System’’ engineers. Insourcing is and retain program schedules.
the administration’s attempt to ameliorate I especially like the theme of this
some of these concerns but it has short- and issue…OrganizingtoAccomplishtheMis-
long-term consequences. These questions sion. As I have reiterated in the last two
and current real world challenges were StephanieH. Clewer issues of the ITEA Journal, that has
theITEAPublicationsCommittee’sfocus been our theme at ITEA this year as
whenplanningthisissue.Theirobjectivewastotakea well. The ITEA mission is to advance the field of test
candid look at defense T&E organizations and andevaluationworldwideingovernment,industryand
capabilities; academic preparation, certifications, proj- academia. Therefore, to remain relevant, ITEA must
ectandprogrammanagementskills;andwhatweneed alsoevolveinstepwiththechangesintheDepartment
to do in the future in training, hiring, and improving of Defense acquisition processes.
our testing—especially as budgets get tighter and the Another mission of DT&E is to develop strategies
scrutiny on programs increases. to enhance the technical management workforce via
Ultimately,T&Eenablestheacquisitioncommunityto practical experience and training, redefine the T&E
learn about limitations (technical or operational) of a acquisition certification standards, and ensure educa-
system under development, so that they can be resolved tion requirements are balanced with job demands for
prior to production and deployment. Of course, all military and civilian personnel. ITEA also recognizes
acquisitionsareplaguedwithcostandscheduleconstraints the importance of these workforce-related strategies
as well. I continue to hear increasing references to the andconsidersitourfundamentalresponsibilitytoplaya
T&E roles in initiatives such as rapid, Agile, and roleinsupportingthem.Iampleasedtoannouncethe
evolutionary acquisition processes. In all of these initia- addition of John ‘‘Jay’’ Weaver to the ITEA Executive
tives, the overarching objective is to balance needs and OfficestaffastheAssociation’sDirectorofEducation.
available capability with resources to put capability into Jay brings a wealth of experience including military
the hands of the user quickly. The success of the service with the U.S. Air Force and positions with
incremental Agile and evolutionary approaches depends other not-for-profit associations where he led profes-
on the phased definition of capability needs and sional development, adult education, and individual
requirements and an awareness of the technology certificationprograms.Jay’sprimaryresponsibilitiesare
maturation that will enable increasing capability over to augment the current ITEA educational content,
time.TheT&Estrategyofthesesystemsmustallowfor develop new curriculum, promote continuous learning
adequate T&E of each increment intended for fielding opportunities, collaborate with other academic institu-
withemphasisonregressiontestingofoperationaland/or tions,enhanceandfacilitateouracademicscholarships,
suitability requirements of subsequent increments. The and develop and manage potential certification pro-
bottom line is that the acquisition process is chang- grams and grant funding.
ing…and the test community must change in concert. The Director of Education is a new staff position
Firstandforemost,incrementalacquisitionsrequireearlier established by the Board of Directors with the
collaborationsamongthetesters,usersandevaluatorsand overarching goal to continue ensuring that ITEA
along-termstrategythatensuresatightlycoupledlinkage remains recognized as the premier provider of a
between evolving technologies and requirements and a ‘‘continuum of learning’’ meeting the evolving needs
responsiveacquisition,testandsustainmentstrategy. of the T&E community through a variety of timely,
32(2)N June2011 117
relevant, and easily accessible education opportunities. The Vice President of the Board, Mark Brown, is
Jay is working closely with the ITEA Education leading a strategic planning tiger team to ensure that
Committee to identify gaps in our current educational the Association’s high priority goals remain aligned
offerings.Ifyouhavenotalreadydoneso,pleasetakea with the needs of our members and the T&E
few minutes to complete their ‘‘Recommended Topics community as a whole. In addition, we have already
for ITEA Education Events Survey’’ on the ITEA begun an exciting Membership Recruitment Promo-
website. Input from our members is very important to tion, the ‘‘Member-Get-A-Member Campaign,’’
us! Also part of our ‘‘organizing to meet the ITEA aimed at both broadening participation and strength-
education mission,’’ we recently partnered with Air ening the ITEA Chapters. Check the website for
Academy Associates to augment our educational further details and be sure to take the opportunity to
content by offering additional courses related to possiblywinafreeAnnualMembershipforyourselfas
Combinatorial Testing with Design of Experiments. wellasadditionalscholarshipfundingforyourchapter!
Althoughimprovingthequalityandrelevanceofour I hope to see many of you at our 2011 Technology
education offerings is one of our primary goals, the
Review in Annapolis in July and at our Annual
BoardofDirectorsisalsotakingahardlookatITEA’s
Symposium in Orlando in September. The theme for
Strategic Plan and the associated Five-Year Action
theAnnualSymposium,FosteringPartnershipsinT&E
Plan for meeting our other high priority goals:
and Acquisition, will be addressing a number of the
1) Broadening participation; challenges I mentioned above, to include both a panel
2) Strengthening our chapters; and poster session on Improving the Current and
3) Increasingopportunitiesforinformationexchange Future T&E Workforce.
and networking through association events, pub- As always, I welcome any comments or suggestions
lications,on-lineforumsandotheravenues; sent to me personally via [email protected].
4) Increasing opportunities to recognize accom-
plishments of T&E professionals; and
5) Ensuring ITEA’s governance is effective and
efficient
118 ITEAJournal
Issue at a Glance
ITEAJournal2011;32:119
Copyright’2011bytheInternationalTestandEvaluationAssociation
O rganizing to Accomplish the Mission. useofintegratedtesting,missionbasedtestdesign,and
Just as we face a rapidly and design of experiments. Colonel Joe Puett, Command-
continuously evolving threat, the er, Joint Interoperability Test Command, presents the
DepartmentofDefensetestcommu- challenges of balancing warfighters’ demands for
nitymustadapttoeverchangingtest increased IT capabilities with strengthening our
requirements and circumstances, for example, more nationalsecuritycapabilitywhileDoDseekstoachieve
complexsystems,shorterschedules,reducedbudgets,a budget efficiencies.
leaner workforce, and rapid acquisition to name a few. Two best paper awards from the 2011 LVC
The aging workforce continually challenges us to find Conference appear in this issue. In the Best Under-
critical skills while more complex systems demand not graduate Student Paper, Marjorie Ingle et al. from the
onlyaflexibleworkforcebutmore‘‘SystemofSystem’’ Center for Space Exploration Technology Research at
engineers. Doing more with less is not new but the University of Texas at El Paso, design a thrust
remains a mantra. This issue addresses ways the test balanceandtestmicropropulsionthrustersintendedfor
community isrespondingand delivering byorganizing space applications. In the Best Graduate Student Paper,
to accomplish the test mission. Jose Barajas and colleagues from New Mexico State
Dr. David Alberts, the Director of Research in the Universityusearigoroussystemsengineeringapproach
Office of the Secretary of Defense Chief Information to identify environmentally benign candidate technol-
Officer (DoD CIO), provides in the Guest Editorial a ogies for energy harvesting and scavenging.
call for partnership between the test and evaluation Dr. James Valdes et al. of the National Defense
(T&E)andresearchcommunities,tobetterunderstand University survey requirements and the state of the art
agility,incorporatetheconsiderationofagilityinT&E, in ground truth determination for bioaerosol detector
thusimprovingtheagilityofthecapabilitiesweacquire. testing. Brian Weiss from the National Institute of
Daniel Laird of the Air Force Flight Test Center Standards and Technology and Dr. Linda Schmidt
analyzes communication link resynchronization, critical from the University of Maryland continue work on
information for the performance of new telemetry tools for complex evaluation designs of advanced and
systems.RandyHerrinoftheJointInteroperabilityTest intelligent systems. Mark Jones reviews five proposed
Command (JITC) and Chris Watson of the Defense changes for test and evaluation to improve our ability
Information Systems Agency present the history and toidentifycritical failure modesanddesignflaws early
evolutionoftheJITCforHistoricalPerspectives. in a system’s lifecycle.
CommandersofthreeoftheDepartmentofDefense The issue closes with two articles from the 2010
Operational Test Agencies offer perspectives on TechnologyReview.DavidBateofScientificResearch
Organizing to Accomplish the Mission. Major General Corporation examines the effects of sleep deprivation
David Eichhorn, Commander of the Air Force on working memory and cognitive performance,
Operational Test and Evaluation Center, discusses concluding that no single metric can characterize the
the impact of the national debt and budget deficits on effects. Dr. Joshua Gomer of Human Factor Engi-
acquisition, and how T&E as the conscience of neering and Dr. Christopher Pagano of Clemson
acquisition ismorecriticaltoday thanit haseverbeen. University demonstrate the utility of the NASA Task
Rear Admiral David Dunaway, Commander, Opera- Load Index for assessing subjective workload data in
tional Test and Evaluation Force, describes improved complex human system technologies, valuable in
operational test efficiencies and better early involve- operator selection and training as well as in design of
mentthroughinternalprocessimprovementsincluding human system technologies.
32(2)N June2011 119
Guest Editorial
ITEAJournal2011;32:123–125
Copyright’2011bytheInternationalTestandEvaluationAssociation
Rethinking Test and Evaluation for a New Age
David S. Alberts, Ph.D.
Director, Research, DoD CIO
T est and Evaluation disseminate information, but they have
(T&E)isacriticalcom- also have had a profound effect on a
ponent of the processes variety of interactions. The products
that develop and/or ac- and services that have been enabled by
quire capability. Done these technologies create a need for
properly, T&E greatly increases the interactionsthatpreviouslywereunim-
likelihood that the capabilities in ques- agined and have facilitated long-de-
tion will be effective and efficient; siredinteractionsthatweretoodifficult
however, organizations and the systems orexpensive.Asaresult,theseproducts
theyconceiveanddevelopactuallyadapt and services have improved both the
to how they are evaluated. If T&E speed and the ‘‘richness’’ of our inter-
criteria and/or processes are not well- actions.2
matched to the demands of the opera-
Dr.David S.Alberts
tional environment and the missions to As a result, we are seeing an
beundertaken,theprobabilityofgettingthecapabilities explosionofinteractionsthat,inandofthemselves,
one really needs at an affordable price can be hasalteredtheworldinwhichwelive.3
significantlyreduced. The explosion in the number of interactions has
Although there is always room for improvement, if created new feedback loops, increased the number of
one knows the capabilities needed and can adequately entities and variables involved in existing feedback
specifyasetofrelevantcharacteristicsandperformance loops, and increased the links between and among
measures,thenexistingT&Eprocessesandpracticesare previousindependentdomains.Asaresult,theworldis
uptothetaskathand.Unfortunately,thisfoundational growing more and more interconnected. This increase
assumptionuponwhichourentireapproachtoT&Eis ininterconnectednesshascreatedbothpossibilitiesand
based,thatis,thatweknowandcanadequatelyspecify interdependencies between and among entities.
thecapabilitiesweneed,isnolongertrue.Furthermore, Our most challenging problems in this New Age
even if one could adequately specify what is currently have two characteristics in common:
neededgiventhenatureoftheAgeinwhichwelive,our N
The problem to be solved involves a multidi-
needsareboundtochange,ifnotduringtheprocessof
mensional effects space.
development,thencertainlywithintheeconomiclifeof N
No single entity, however large or powerful, can
thecapabilitybeingacquired.
make progress by itself; any effective solution
We can no longer ignore the fact that we find
requires collective action.
ourselvesinanew,not-yetnamedAge,anAgeinwhich
weareincreasinglyinterconnected,interdependent,and These aspects of the endeavors we must undertake
pressed for time. We no longer can adequately are, in part, a result of the changes to our world
understand our requirements and predict the nature of brought about by Information Age technologies and
the circumstances in which we operate. This is not the capabilities they have enabled. If we look at these
because we are deficient in some way; it is an endeavorsfromterrorismandcyberterrorismtonatural
unavoidableconsequenceofourAge. and man-made disasters, we find that they have not
Survival in this New Age requires, above all else, only, e.g., a military or security dimension, but also
Agility.1 This editorial briefly explains why this is so, political, social, and economic dimensions. Tackling
and explores the implications for T&E. these multidimensional challenges, as we have repeat-
edly found, cannot be accomplished by a single entity.
A NewAge In some cases, they require a ‘‘whole of government’’
Information Age technologies have not only im- solution,whileinothercases,aninternationalcoalition
proved our ability to collect, store, process, and and/or a public–private effort is necessary.
32(2)N June2011 123
Alberts
As a consequence of these two aspects of complex A way forward
endeavors, the information needed will come from a Thelogical responsetotheresidual uncertaintythat
large number of sources, both known and previously remains after we have tested a capability against the
unknown. This information will need to be widely stated requirement in the predicted environment(s) is
shared among individuals and organizations with to attempt to measure potential agility.
whom we have established relationships and those
Agility is the capability to successfully cope with
with whom we have not. Furthermore, to develop the
and/orexploitchangesincircumstances.4
critical mass of effort, individual entities will need to
coordinate, collaborate, and self-synchronize their Measuring agility involves a Catch 22. While one
efforts in a manner appropriate for the situation. canobservemanifestagilitybyobservinghowanentity
However well intended, narrowly focused actions and/ or system actually responds to observed or simulated
or unilateral action can often have an adverse impact. changes in circumstances, since we cannot adequately
The complex nature of New Age mission challenges predict what future circumstances will be, we cannot
makes it unwise for us to rely on statements of measure an entity’s or system’s future or potential
requirements and prognostications of environmental
agility in this manner. To do so requires both a
conditionsandstressesasthesolebasisforT&E.How
conceptualmodelofagilityandaninstantiationthatis
can we be sure which specific entities will be our
tailored to the capability being acquired.
partners? How can we know exactly what information
Fortunately, experiments with agility as it applies to
wewillneedtohave?Whatinformationwillweneedto
command and control, particularly in the context of
share?Isthe80percentsolutiongoingtobeenough?
complexendeavors,haveprovidedabetterunderstand-
Therefore, testing our systems is necessary, but not
ing of agility. We can now use this knowledge to
sufficient, to ascertain whether or not a real capability
improve current T&E approaches that rely on
is being acquired. Our systems need to support not
statements of requirements alone. However, we can
only our soldiers, sailors, airmen, marines, and civilian
greatly improve both our understanding of agility, and
employees, but also the individuals we partner with in
our ability to measure the potential agility of entities
many other entities—military and civilian, govern-
and systems if we enlist others in this quest.
mental and nongovernmental, public and private. Do
Iamthereforeproposingawayforwardthatinvolves
we think we know these entities’ requirements? If we
a partnership between the T&E and research commu-
onlytestoursystems,howcanweknowthattheoverall
nities. Together we can make progress in better
characteristics and performance of this ever-changing
understanding agility and by incorporating the con-
collection of systems is adequate?
sideration of agility in T&E, we can improve the
To make matters worse, we can no longer act as if
agility of the capabilities we acquire. Such a partner-
cyberspace isa benign environment.Is it reasonable to
ship will create a positive feedback loop by creating a
assumethatwewillbeabletofunctionatanacceptable
better appreciation of the costs of a lack of agility,
levelinthefaceofanever-endingandevolvingstream
accepted metrics for agility, and a definitive, empiri-
of attacks? What we know about the likely conse-
cally supported, quantitative link between levels of
quences of such attacks comes mainly from looking at
agility and performance or effectiveness.
specificsystems,notthedynamiccollectionsofsystems
Thiseditorialisacalltoaction.IfwegetT&Erightwe
needed to support complex endeavors.
will have created a forcing function that will shape the
Can we assume we will have an adequate capabilities we acquire, making them more agile. As a
solution to the insider threat? result,thelikelihoodofmissionsuccesswillbeincreased
while, at the same time, total expenditures can be
Cyber attacks do more than impact system perfor-
reduced.Pastfailureshavetaughtusthat‘‘doingitright
mance—they can destroy trust. Trust is the key to
thefirsttime,’’evenifitcostsmoreandtakesmoretimein
information sharing. A loss of trust, in systems,
theshort run,turns outto provide more capability ata
information sources, and in the partners upon whom
lower cost inthe long run. Inthis case, gettingit right
we rely to accomplish our missions, has at least the
same adverse impact as a loss of connectivity or meansgreatlyincreasedattentiontoagility. C
performance would have. Furthermore, the damage
sustainedcouldbefarmoredifficulttorepairandcould
require much more time to rebuild. Thus, we need to DR. DAVID S. ALBERTS is currently the director of
understand the full consequences of our vulnerabilities research in the Office of the Secretary of Defense Chief
tocyberattacks,notjusttheimmediateimpactsonthe Information Officer (DoD CIO) where his research
systems we are testing. initiatives are focused on issues related to agility,
124 ITEAJournal