Table Of ContentXiaofan He • Huaiyu Dai
Dynamic Games for Network
Security
123
XiaofanHe HuaiyuDai
DepartmentofElectricalEngineering DepartmentofElectricalandComputer
LamarUniversity Engineering
Beaumont,TX,USA NorthCarolinaStateUniversity
Raleigh,NC,USA
ISSN2191-8112 ISSN2191-8120 (electronic)
SpringerBriefsinElectricalandComputerEngineering
ISBN978-3-319-75870-1 ISBN978-3-319-75871-8 (eBook)
https://doi.org/10.1007/978-3-319-75871-8
LibraryofCongressControlNumber:2018933373
©TheAuthor(s)2018
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpartof
thematerialisconcerned,specificallytherightsoftranslation,reprinting,reuseofillustrations,recitation,
broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,andtransmissionorinformation
storageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilarmethodology
nowknownorhereafterdeveloped.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Thepublisher,theauthorsandtheeditorsaresafetoassumethattheadviceandinformationinthisbook
arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsor
theeditorsgiveawarranty,expressorimplied,withrespecttothematerialcontainedhereinorforany
errorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregardtojurisdictional
claimsinpublishedmapsandinstitutionalaffiliations.
Printedonacid-freepaper
ThisSpringerimprintispublishedbytheregisteredcompanySpringerInternationalPublishingAGpart
ofSpringerNature.
Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland
Preface
Therecentemergenceandadvancementofvariousinformationandcyber-physical
networks have brought unprecedented convenience to our daily lives. To ensure
effectiveandcontinuousoperationsofthesemodernnetworks,itisofcrucialimpor-
tancetodeployefficientandreliabledefensemechanismstoprotecttheirsecurity.
However, in the security battles, one challenge is that the adversary is constantly
upgrading their attacking tactics and becoming increasingly intelligent, making
conventional static security mechanisms outdated and incompetent. Considering
this,gametheory,whichisarichsetofanalytictoolsformodelingandanalyzingthe
strategic interactions among intelligent entities, has been widely employed by the
network security community for predicting the adversary’s attacking strategy and
designingthecorrespondingoptimaldefense.Despiteitscelebratedapplicationsin
addressingsomenetworksecurityproblems,theclassicgametheorymainlyfocuses
on static settings, while many practical security competitions often take place in
dynamicscenariosduetofrequentchangesinboththeambientenvironmentandthe
underlying networks. This motivates the recent exploration of the more advanced
stochasticgame(SG)theorythatcancapturenotonlytheinteractionsbetweenthe
defenderandtheattackerbutalsotheenvironmentaldynamics.Theobjectiveofthis
bookistocollectandsystematicallypresentthestateoftheartinthisresearchfield
andtheunderlyinggame-theoreticandlearningtoolstothebroaderaudiencewith
generalnetworksecurityandengineeringbackgrounds.
Our exposition of this book begins with a brief introduction of relevant back-
ground knowledge in Chap.1. Elementary game theory, Markov decision process
(MDP),andSGarecovered,includingthebasicconceptsandmathematicalmodels
aswellasthecorrespondingsolutiontechniques.Withthisnecessarybackground,
inChap.2,weproceedtoreviewexistingapplicationsofSGinaddressingvarious
dynamic security games, in the context of cyber networks, wireless networks, and
cyber-physical networks. In these applications, the defenders and the attackers are
assumedtoholdequalinformationaboutthecorrespondingsecuritycompetitions,
whereasinformationasymmetryoftenexistsinpractice.Consideringthis,wetakea
stepfurtherandexplorehowtodealwithdynamicsecuritygamesinthepresenceof
informationasymmetryinChaps.3–5.Inparticular,ourexplorationincludesthree
vii
viii Preface
aspects of this issue—dynamic security games with extra information, dynamic
security games with incomplete information, and dynamic security games with
deception. It is worth mentioning that, although we mainly take the defender’s
perspective in the discussions, the corresponding results and techniques may be
employedtopredicttheattacker’sbehaviorinsimilarsituations.Morespecifically,
dynamic security games with extra information discussed in Chap.3 concern
security competitions where the defender has an informational advantage over the
adversary.BasedontheexistingSGframework,wepresentanoveltechniquethat
enablesthedefendertofullyexploitsuchadvantagesoastoachievefasteradapta-
tion and learning indynamic security competitions. The complementary scenarios
wherethedefenderlacksinformationabouttheadversaryareexaminedinChap.4
throughthelensofincompleteinformationSG.Toaddressincompleteinformation
SGs, a new algorithm that integrates Bayesian learning and conventional learning
algorithms of SG is presented; the key idea is to allow the defender to gradually
inferthemissinginformationthroughrepeatedinteractionswiththeadversary.The
extra and the incomplete information considered in Chaps.3 and 4 is inherent to
thecorrespondingsecurityproblems.InChap.5,weswitchgearandfurtherexplore
how to proactively create information asymmetry for the defender’s benefit, and
the dynamic deception technique is investigated as an effective tool to achieve
thisobjective.Lastly,concludingremarksandourperspectiveforfutureworksare
presentedinChap.6.
TheauthorswouldliketoacknowledgeProf.RudraDutta,Prof.PengNing,and
Mr. Richeng Jin. Without their contribution, this book could not have been made
possible. We would also like to thank all the colleagues and researchers for their
pioneeringandinspiringworksthatlayoutthesolidfoundationofthisbook.
Wuhan,Hubei,China XiaofanHe
Raleigh,NC,USA HuaiyuDai
Contents
1 Preliminaries .................................................................. 1
1.1 Introduction .............................................................. 1
1.2 ElementaryofGameTheory ............................................ 2
1.3 TheMarkovDecisionProcess........................................... 4
1.3.1 TheMDPModel................................................. 4
1.3.2 SolvingtheMDP ................................................ 5
1.4 TheStochasticGames ................................................... 7
1.4.1 TheModelofSG ................................................ 7
1.4.2 SolvingtheSG................................................... 9
1.5 Summary ................................................................. 15
References...................................................................... 15
2 OverviewofDynamicNetworkSecurityGames .......................... 17
2.1 Introduction .............................................................. 17
2.2 ApplicationsinCyberNetworks ........................................ 18
2.3 ApplicationsinWirelessNetworks ..................................... 19
2.4 ApplicationsinCyber-PhysicalNetworks.............................. 20
2.5 Summary ................................................................. 22
References...................................................................... 22
3 DynamicSecurityGameswithExtraInformation........................ 25
3.1 Introduction .............................................................. 25
3.2 Post-decisionState....................................................... 26
3.3 Multi-AgentPDSLearning.............................................. 28
3.3.1 TheMinimax-PDSAlgorithm .................................. 30
3.3.2 WoLF-PDS....................................................... 32
3.4 SecurityApplications.................................................... 33
3.4.1 Anti-jamminginEnergyHarvestingCommunication
Systems........................................................... 34
3.4.2 Cloud-BasedSecurityGame.................................... 37
3.5 Summary ................................................................. 41
References...................................................................... 41
ix
x Contents
4 DynamicSecurityGameswithIncompleteInformation ................. 43
4.1 Introduction .............................................................. 43
4.2 BackgroundonRepeatedBayesianGame.............................. 44
4.3 BayesianSGs............................................................. 45
4.3.1 BayesianSGModel.............................................. 45
4.3.2 BayesianNashQ-Learning...................................... 47
4.4 SecurityApplications.................................................... 48
4.4.1 DynamicIntrusionDetectionSystemConfiguration........... 49
4.4.2 SpectrumAccessinAdversarialEnvironment ................. 54
4.5 Summary ................................................................. 59
References...................................................................... 59
5 DynamicSecurityGameswithDeception.................................. 61
5.1 Introduction .............................................................. 61
5.2 StochasticDeceptionGames ............................................ 62
5.2.1 TheSDGModel ................................................. 62
5.2.2 SolvingtheSDG................................................. 65
5.3 ASecurityApplication .................................................. 67
5.4 Summary ................................................................. 69
References...................................................................... 71
6 ConclusionandFutureWork................................................ 73
6.1 Summary ................................................................. 73
6.2 FutureWorks............................................................. 74
Chapter 1
Preliminaries
1.1 Introduction
To start our journey in this book, relevant backgrounds on game theory, Markov
decision process (MDP), and stochastic game (SG) will be introduced first in this
chaptertopavethewayforourlaterexposition.
Inessence,manysecurityissuescanbetreatedasagamebetweenthedefender
and the attacker who are intelligent entities that can smartly plan their actions
in the security rivalries. For this reason, our discussion in this chapter begins
with the presentation of some rudimentary concepts of the classic game theory—
afascinating theory thatconcerns themodeling and prediction ofthebehaviors of
intelligent entities in strategical interactions. In particular, the basic elements of a
game will be introduced first. Then, the Nash equilibrium (NE), one of the most
widelyadoptedsolutionconceptinclassicgametheory,willbereviewed.Besides,
severalimportantanalyticresultsregardingtheexistenceanduniquenessoftheNE
arealsodiscussed.
In addition to the interactions with the opponent, practical security problems
often involve with different environmental, system, or network dynamics. As a
fundamental tool for optimal planning in dynamic environments, the MDP is also
reviewedinthischapter.Specifically,twokeynotionsfortheMDP,theQ-function
and the value function, are introduced first, along with the Bellman’s principle
of optimality. To solve the MDP, several computing methods that can be used to
directlyderivetheoptimalsolutionwhenfullinformationoftheMDPisavailable
are presented. To deal with situations with unknown information, the Q-learning
algorithm that enables the agent to gradually adjust its strategy based on repeated
interactionswiththedynamicenvironmentisillustrated.
Asakeyframeworkforaddressingvariousdynamicsecuritygames,theSGwill
be reviewed in this chapter as well. The SG is a natural marriage of the classic
©TheAuthor(s)2018 1
X.He,H.Dai,DynamicGamesforNetworkSecurity,SpringerBriefsinElectrical
andComputerEngineering,https://doi.org/10.1007/978-3-319-75871-8_1
2 1 Preliminaries
game theory and the MDP and therefore can jointly manage the interactions with
the opponent and the environmental dynamics, well suited to practical security
problems.Withthisconsideration,theunderlyingmodelofSGwillbepresentedin
details,andwheneverapplicable,itsconnectionstotheclassicgametheoryandthe
MDPwillbedrawn.SimilartothecaseofMDP,boththecomputingmethodsand
thelearningalgorithmsforSGarepresented,includingthenon-linearprogramming
method, value iteration method, the minimax-Q algorithm, the Nash-Q algorithm
andtheWin-or-Learn-Fast(WoLF)algorithm.
1.2 ElementaryofGameTheory
Game theory [1, 2] is the study of the strategic interactions among multiple
intelligent entities, termed players hereafter. The objective is to acquire certain
reasonable prediction about the behaviors of the entities and help them design
adequatestrategiesthatcanleadtothebestpossiblebenefits.Insomesense,game
theory can be treated as competitive optimization problems in which the players
havetoconsidertheinfluencefromtheotherswhenoptimizingtheirownpayoffs.
Although originated as a celebrated branch of mathematics and economics, game
theory also finds its success in addressing various engineering problems, such as
resourceallocationforwirelessnetworks[3],patrollingstrategydesignforairport
security [4], defense system configuration in cyber-networks [5], among the many
others. It is such a profound subject that covers a large variety of different topics,
and each of them deserves a separate book. Due to space limitation, in this book,
weonlyprovideaverybriefandrudimentaryintroductiontothisfascinatingtheory
soastopreparereaderslackingrelevantbackgroundsforourlaterexpositions.For
more comprehensive and systematic treatments of classic game theory, interested
readersmayreferto[1,2]andotherrelevantmaterials.
Roughlyspeaking,agameisaprocedureofinteractivedecision-makingamong
multipleplayers.Althoughsomeslightdifferencesmayexist,severalbasicelements
are shared by almost all types of games, and will be introduced in the sequel. As
already mentioned above, the set I of players is one of the elementary building
block of a game G. In the game, each player-i (i ∈ I) takes an action ai
from its action set Ai. Based on the actions taken by the players, each player-
i will receive a payoff ri = Ri(ai,a−i) (or sometimes interchangeably called
reward), with Ri(·) the payoff function. Here, we follow the convention of using
a(cid:2)−i to represent the actions fr(cid:3)om all the players other than player-i, i.e., a−i (cid:2)
a1,...,ai−1,ai+1,...,a|I| . The objective of each player is to maximize its
ownpayoffri byproperlyselectinganactionai.Theunderlyingpolicyaccording
to which a player takes its action is called the strategy of that player, and is often
denoted by πi. Two types of strategies are often considered in game theory: pure
strategyandmixedstrategy.Apurestrategydirectlyspecifieswhichactiontotake.
For example, in a rock-paper-scissors game, “play rock” is a pure strategy that
1.2 ElementaryofGameTheory 3
informs the player to always take the action “rock”. A mixed strategy is often
represented by a probability distribution over the action set and only specifies in
a probabilistic manner about which action to take. Taking the rock-paper-scissors
gameasanexampleagain,themixedstrategyπi = [1,1,1]dictatesthatplayer-i
2 4 4
willplayrockwithprobability 1 andplaypaperandscissorswithequalprobability
2
1. In general, a mixed strategy of player-i can be written as a |Ai|-dimensional
4
vector πi = [pi,...,pi ] in which pi (for 1 ≤ j ≤ |Ai|) represents the
1 |Ai| j
|A(cid:4)i|
probability of taking the jth action from the set Ai; clearly, pi = 1. In
j
j=1
addition,itisworthmentioningthatanypurestrategycanbeexpressedasamixed
strategy.Forexample,thepurestrategy“playrock”consideredabovecanbewritten
as πi = [1,0,0]. Nonetheless, as the players often do not have prior information
about which actions will be taken by the others, it is highly non-trivial for each
player to find the “optimal” strategy. In fact, unlike conventional optimization
problems concerned with optimality, game theory considers a different solution
concept—equilibrium; this is mainly because the players in a game are assumed
tobeself-interested,caringonlyabouttheirownpayoffs.Althoughmanydifferent
notions of equilibrium have been developed in literature [1, 2], Nash equilibrium
(NE)[6,7]isprobablythemostwidelyadoptedone,andmanyfundamentalresults
ingametheoryarecenteredaroundthisconcept.Specifically,theNEisdefinedas
follows.
(cid:2) (cid:3)
|I|
Definition1 A tuple of strategies π∗1,...,π∗ form an NE if the following
conditionholds:Foranyplayer-i andstrategyπi,italwayshas
(cid:5) (cid:6) (cid:5) (cid:6)
E(cid:2)π∗1,...,π∗i,...π∗|I|(cid:3) Ri(ai,a−i) ≥E(cid:2)π∗1,...,πi,...π∗|I|(cid:3) Ri(ai,a−i) , (1.1)
(cid:2) (cid:3)
|I|
whereontheleft-handside,thesubscript π∗1,...,π∗i,...π∗ indicatesthatthe
expectation is taken according to the law determined by this strategy tuple, and
similarnotationisusedontheright-handside.
Intuitively, the above definition says, a strategy tuple is an NE if no player can
increaseitsexpectedpayoffbyunilaterallychangingitsstrategy.
Several fundamental results regarding the existence and uniqueness of NE are
presentedbelowwiththecorrespondingproofsomitted.
Theorem1.1([6]) Everyfinitegameinwhichthenumbersofplayersandactions
arefinite(i.e.,|I|<∞and|A1|,...,|A|I| <∞)hasamixedstrategyNE.
Theorem1.2([8]) Foraninfinitegamewith|I|<∞,if
(i) theactionspacesAi’sarenonemptyandcompactmetricspaces;
(ii) thepayofffunctionsRi(ai,a−i)’sarecontinuous;
4 1 Preliminaries
thentherealwaysexistsamixedstrategyNE.If
(i) theactionspacesAi arecompactandconvex;
(ii) thepayofffunctionsRi(ai,a−i)’sarecontinuousina−i;
(iii) thepayofffunctionsRi(ai,a−i)’sarecontinuousand(quasi-)concaveina−i;
thentherealwaysexistsapurestrategyNE.
Theorem1.3([9]) Foraninfinitegamewith|I|<∞,if
(i) thepurestrategysetofeachplayer-i isspecifiedintheform
(cid:7) (cid:8)
Πi = πi|f (πi)≥0 , (1.2)
i
forsomeconcavefunctionf (),andforeachi ∈I,thereexistsatleastapoint
i
x suchthatf (x )isstrictlypositive;
i i i
(ii) thepayofffunctions(R1,...,R|I|)arediagonallystrictlyconcaveovertheset
Π1×···×Π|I|,inwhichΠi isthesetofallpossiblestrategiesofplayer-i;
thenthegamehasauniquepurestrategyNE.
1.3 The MarkovDecisionProcess
1.3.1 TheMDP Model
BeforeintroducingtheSG,thesingle-playerversionofSG,knownastheMDP[10],
is reviewed in this section to better prepare the readers for our later expositions.
As depicted in Fig.1.1, an MDP concerns the interactions between a dynamic
system or environment and an intelligent agent. Particularly, an MDP unfolds as
follows. At the beginning of each timeslot n, the agent first observes the current
state s ∈ S of the system (with S the set of possible states) and then takes
n
an a chosen from its action set A according to its strategy π. The strategy of
n
the agent maps a state s ∈ S into a probability distribution over the action set
A. More specifically, π(s,a) will(cid:4)be used to denote the probability of the agent
takingactiona instates;clearly π(s,a) = 1.Afterthis,ontheonehand,the
a∈A
agentwillreceivearewardr =R(s ,a ),whereR()isthestatedependentpayoff
n n n
functionassociatedwiththisMDP.Ontheotherhand,thesystemwilltransitintoa
newstatesn+1 dictatedbyacontrolledMarkovprocesswithtransitionprobability
P(sn+1|sn,an).Then,thisprocessrepeats.TheobjectiveoftheagentintheMDPis
tomax(cid:4)imizetheexpectedaccumulativelong-termreward,whichisoftenexpressed
as E{ ∞ βn · r }. Here, 0 ≤ β < 1 is the discounting factor of the MDP.
n=0 n
More specifically, β = 0 corresponds to a myopic agent that only cares about its
instant reward, while a non-zero β indicates that the agent concerns its long-term
performance but puts decreasing emphases for the rewards obtained in the further
future due to the increasing uncertainty. In the rest of this book, we will mainly
