Model-Free Control of Thermostatically Controlled Loads Connected to a District Heating Network

1.4 MB·

by  Bert J. Claessens

Model-Free Control of Thermostatically Controlled Loads Connected to a District Heating
Network
BertJ.Claessensa,c,1,D.Vanhoudta,c,J.Desmedta,c,F.Ruelensb,c
aEnergydepartementoftheResearchInstituteVITO,2400Mol,Belgium
bDepartmentofElectricalEngineeringofKULeuven,KasteelparkArenberg10,bus2445,3001Leuven,Belgium
cEnergyVille,ThorPark8310,3600Genk,Belgium
Abstract
Optimal control of thermostatically controlled loads connected to a district heating network is considered a sequential decision-
7makingproblemunderuncertainty. Thepracticalityofadirectmodel-basedapproachiscompromisedbytwochallenges,namely
1
scalabilityduetothelargedimensionalityoftheproblemandthesystemidentificationrequiredtoidentifyanaccuratemodel. To
0
help in mitigating these problems, this paper leverages on recent developments in reinforcement learning in combination with a
2
market-based multi-agent system to obtain a scalable solution that obtains a significant performance improvement in a practical
b
learningtime. Thecontrolapproachisappliedonascenariocomprising100thermostaticallycontrolledloadsconnectedtoaradial
e
Fdistrict heating network supplied by a central combined heat and power plant. Both for an energy arbitrage and a peak shaving
objective,thecontrolapproachrequires60daystoobtainaperformancewithin65%ofatheoreticallowerboundonthecost.
4
2
Keywords: Districtheating,combinedheatandpower,reinforcementlearning,thermostaticallycontrolledloads.

]
Y
1. Introduction canbeusedtostoreheat.
S
. From an operational point of view, controlling a cluster of
s A District Heating Network (DHN) offers the opportunity
TCLs connected to a DHN can be considered as a sequential
c
toprovidethecollectiveheatdemandofaclusterofgeographi-
[ decision-making problem under uncertainty. One well stud-
callyconcentratedbuildingsthroughasetofcentralheatsources.
ied control paradigm for operational management of a DHN
2
Thisallowstheuseofcentralizedproductiontechniqueswithan
isthatofModelPredictiveControl(MPC)[11,12]. Whenpro-
v
efficiencyexceedingthatofdistributedproduction. Combined
4 jectedonthesettingofTCLsconnectedtoaDHN,thisrequires
7HeatandPowerplants(CHPs)areaprominentexample,as80- defining control actions for the central sources as well as for
090%oftheprimaryenergyisconvertedtoheatandelectricity allindividualTCLs. Developingapracticalimplementationre-
8[1,2,3]. Butalsoheatfromageothermalsource[4]orexcess quires one to tackle the problem of scalability, as the state di-
0
heatresultingfromanindustrialprocesscanbeusedasprimary
mensionalityandnumberofcontrolvariablesquicklyresultin
.
1heat source. The heat from the sources is transported through
an intractable optimization problem. This is complicated fur-
0anetworkofpipesusingwaterasamedium. Ateachbuilding,
therbynon-linearsystemdynamics. Asecondimportantchal-
7
1heat is extracted in a local substation, resulting in water at a lenge is that of system identification [13] as identifying an ac-
lowertemperature,beingtransportedbacktothedifferentheat
: curatemodelofboththeDHNandallTCLsrequiressignificant
vsources. A typical operational model at the production side is
amountsofnotreadilyavailabledataandexpertknowledge.
i
Xto modulate the power of the heat sources to keep the supply Thisworkcontributesinmitigatingoperationalcontrolchal-
temperature close to a design setting. This basically results in
r lengesforTCLsconnectedtoaDHNbyworkingonthesetwo
athethermalsupplyfollowingthethermaldemand. Heatstorage
problems.
however, canprovidedemandflexibilityenablingflexibilityat
Scalability: To obtain scalability, a heuristic dispatch ap-
theproductionsidethroughdemandresponseapproaches. This
proach as described in [14], is applied to the setting of TLCs
flexibility allows operational opportunities for cost reduction,
connectedtoaDHN.Insteadofcalculatinganindividualcon-
examples being peak shaving/valley filling [5] and energy ar-
trol action for each TCL, this approach calculates a collective
bitragebysellingtheelectricityproductionoftheCHPonthe
control action for the entire cluster of TCLs. A market-based
wholesalemarket[1,2,6]. Wellreferencedembodimentsoflo-
dispatchalgorithmisusedtotranslatethecollectivecontrolac-
calheatstorageareThermostaticallyControlledLoads(TCLs)
tionintoindividualcontrolactions.
[7]suchasahotwaterstoragetank[8]wheretheheatisstored
System identification: Driven by recent developments in
directlyinthewater,butalsothebuildingenvelope[1,7,9,10] BatchReinforcementLearning(BRL)[15,16],ablindmodel-
freeapproachisconsidered.AsaBRLtechniqueneedsnoprior
information on the system dynamics, this strongly relaxes the
1BertJ.ClaessensiscurrentlyworkingatREstoreandcanbecontactedat
systemidentificationrequirements,atthecostofalearningtime
[email protected]
PreprintsubmittedtoEnergyandBuildings February27,2017
andsub-optimalperformance. actions at device level. Although a simplified first order TCL
In Section 2, an overview is given on related research re- model has been used, the results presented in [13] show that
gardingthecontroloflargeclustersofTLCsandmodelbased careful system identification is required. Moreover, in [25] it
controllersforDHN.InSection3,thedecision-makingproblem wasarguedthatafirstorderTCLmodelisfoundlacking, fur-
is formalized as a Markov Decision Problem (MDP). In Sec- thercomplicatingsystemidentification.
tion 4, the control approach as used in this work, is described Adifferentapproachisthatofdistributedoptimization[26,27],
in detail. In Section 5, an evaluation of the controller perfor- wherethecentralizedoptimizationproblemisdecomposedover
mance is provided based upon a simulation scenario compris- distributed agents who interact through virtual prices. For ex-
ing100TCLsconnectedtoaDHN.Thesimulationscenariois ample in [28], distributed MPC through dual decomposition
sufficiently complex to evaluate the contributions of the con- was presented as a means for energy arbitrage of a large clus-
trolapproach,butalsocomprehensiblyenoughtoallowforan ter of TCLs subjected to a coupling constraint related to an
analysiswhichisnotobfuscatedbycomplexityofthescenario. infrastructure limitation. Although mathematical performance
Finally in Section 6, the conclusions are provided, as is a dis- guarantees can be provided under sufficient assumptions, the
cussionontheresults. methodheavilyreliesontheaccuracyoflocalmodelsandhas
stringentcommunicationandcomputationrequirementsdueto
its iterative character. An example of how an MPC controller
2. RelatedWork
canbeusedatbuildinglevelisdetailedin[12].
Inthissectionanon-exhaustiveoverviewisgivenofrelated
2.2. Model-BasedDHNControl
work regarding both the control of large clusters of TCLs and
model-basedcontrolappliedtoaDHN. Whenimplementingamodel-basedcontrolapproachfora
DHN,beitcentralizedordistributed,oneisconfrontedwith(1)
2.1. ControllingaclusterofTCLs thenon-linearitiesinthedynamicsofaDHN,and(2)theslow
The curse of dimensionality [17] lurks around the corner time scales compared to e.g. an electric network [29]. Taking
whenmanagingtheflexibilitypresentinalargeclusterofTCLs. these effects into account is essential for a good performance
This is attributed to the dimensionality of the state space and ofthecontroller. Severalmodel-basedoptimizationapproaches
the large amount of control variables. To this end, significant have been identified in literature, explicitly incorporating the
recentwork[18,19,20,21,22]hasfocusedonprovidingcom- dynamicsoftheDHN.Forexamplein[11]asimplifiedmodel
putationally tractable solutions for large clusters of TCLs. In hasbeenderivedthatisusedtogetherwithsequentialquadratic
[23],aclusterofflexibilitycarriersrepresentedbygenerictank programming. In[30]approximatedynamicprogramming[31]
modelsisconsideredwiththeobjectiveofprovidingday-ahead has been used taking advantage of permutational symmetries
modulation services to a transmission system operator. Even of the DHN dynamics. A model-based approach using fuzzy
thoughallmodelsarelinear,aformalbranchandboundbased directmatrixcontroltomitigatenon-linearitiesoftheDHNdy-
optimizationapproachquicklybecomesintractable. Themain namicscanbefoundin[32]. Althoughmodel-basedsolutions
contributionoftheworkisaheuristicmethodincludingastate can have excellent performance, accurate models of the DHN
dependent dispatch algorithm in combination with an iterated and the consumers coupled to the DHN are required, tuning
local search technique. High quality solutions to a test prob- andshapingthesemodelsisconsideredanexperttask making
lemarepresented,obtainedwithinapracticalcalculationtime. agenericroll-outofthistechnologychallenging[33].
Theresultsofarelatedapproachappliedinanactualfieldtest Ascalablemodel-freesolutionsolutionispresentedin[34],
comprising54heatpumpshasbeenpresentedin[21]. Herean hereamarket-basedmulti-agentsystemisusedtomatchther-
aggregatedmodelofreduceddimensionalitywasusedtodeter- malandelectricdemandandsupply. Althoughthisapproachis
minepowersetpointsfortheentireclusterinaMPCapproach. scalable, it does not take into account the DHN dynamics and
A heuristic dispatch algorithm was used to convert the aggre- followsamyopiccontrolstrategy. Anapproachcombiningan
gatedsetpointstolocalcontrolactionsintheportfolio. In[22] auction-based multi-agent system with a central optimization,
a data-driven decision framework has been developed using a taking into account a forecast of the total heat demand can be
meta-heuristicoptimizationtechnique. foundin[1].
Anapproachfromthesamesolutionclassispresentedin[13].
HereaproblemofenergyarbitragewithalargeclusterofTCLs 3. Problemdescription
ispresented. Anaggregatedsystemmodelisusedintheform
of a state bin transition model [24]. All TCLs are clustered, InspiredbyGemineetal. [35],thissectionpresentsaprob-
basedupontheirpositionwithintheirdead-band,resultingina lemformulationofthesequentialdecision-makingproblemre-
statevectorcontainingthefractionofTCLsineachstatebin.A lated to optimal control of a cluster of TCLs connected to a
linearstatebintransitionmodeldescribesthedynamicsofthis DHN.Inasecondstep,thecontrolproblemiscastontoaMarkov
statevector,thedimensionalityofwhichisindependentofthe DecisionProcess(MDP).
numberofTCLsinthecluster. ThismodelisusedinanMPC
resultinginacontrolactionforeachstatebin. Asimpleheuris-
tic is used to dispatch the control signals to individual control
2
CHP powerarePe areassociatedto pinanoden. Therelationship
p,t
betweenthesepowersisdefinedas:
n l m
PT +ϕPe =ηPI , ∀(p,t)∈P×T, (1)
p,t p,t p,t
with η the total fuel utilization ratio of the CHP and ϕ defines
(cid:104) (cid:105)
theheattopowerratiooftheCHP.InthisworkPT ∈ 0,PT
p,t p,max
isconsideredthecontrolvariable.
3.2.3. Operationallimits
The main interest of the DHN system is to supply a heat
servicetowardstheTCLsmeetingcomfortconstraints:
∀(d,t)∈D×T :Ta ≤Ta ≤Ta. (2)
d d,t d
Here · and · indicate the lower and upper bound respectively.
Common for buildings is to have a constraint on the opera-
tive temperature. As a simplification the comfort constraints
TCL areheredirectlyrelatedtotheairtemperatureTta. Besidescon-
straintsfortheTCLsalsoconstraintsfortheDHNarerelevant,
Figure1:Illustrationofthedistrictheatingnetworkconfigurationasusedinthis i.e.
work. Asetof100ThermostaticallyControlledLoads(TCLs)isconnected
toadistrictheatingnetwork, thethermalenergyisproducedbyonecentral ∀(n,t)∈N ×T :T ≤T ≤T (3)
n n,t n
CombinedHeatandPower(CHP)plantatthetop.
∀(l,t)∈L×T :T ≤T ≤T , (4)
l l,t l
V ≤V ≤V .
3.1. Testscenarionetwork l l,t l
To make the problem formalism more tangible, references 3.3. Sequentialdecisionmaking
are made to the test scenario as illustrated in Figure 1 and de-
Driven by the possibility of using techniques from Rein-
tailedinSection5. Thetestscenariocomprises100TCLscon-
forcementLearning(RL)[15],thesequentialdecision-making
nected to a DHN. One central CHP is assumed to provide the
process is formulated as a Markov Decision Process (MDP)
heattothebuildingsthroughaDHN.
[17,36]. Thesequentialnatureresultsfrominter-temporalcon-
straintsrelatedtothedynamicsoftheDHNandtheTCLs. De-
3.2. Problemcomponents
cisions made at time step t impact possible actions allowed at
3.2.1. DHN futurestates. AnMDPisdefinedbyitsstatespaceX,itsaction
ADHNcontainsasetofnodesandpipesconnectingthese spaceU,andatransitionfunction f:
nodes. Each pipe l ∈ L connects two nodes n,m ∈ N and is
characterizedbyitslengthL,diameterD andheatlosscoeffi- xt+1 = f(xt,ut,wt), (5)
l l
cient αl. The flow speed of the medium at time t ∈ T in pipe describing the dynamics from state xt ∈ X to xt+1, following
l is denoted by Vl,t and its average temperature by Tl,t. The thecontrolactionsu ∈Usubjecttoarandomprocess,w ∈W,
t t
temperatureineachnodenattimetischaracterizedbyTn,t. wherew isdrawnfromaprobabilitydistributionp (·,x).Each
t w t
transitionisaccompaniedbyacostsignalc:
t
3.2.2. TCLsandheatproduction
The set D contains the TCLs connected to the DHN, each ct(xt,ut,xt+1)=ρ(xt,ut,wt), (6)
TCLisassumedconnectedtoanode. Formodelingpurposesa
withρthecostfunction.
setofrelevanttemperaturesisassociatedtoeachTCLd ∈Dat
timet,i.e. theairtemperatureTa asmeasuredbyalocalther-
d,t 3.3.1. Statedescription
mostat,Tb atemperaturecorrespondingtoabuildingenvelope
d,t Followingthenotationprovidedin[37],thestateofthesys-
[9] and Th the heating system return temperature. The con- temisassumedtobespannedbytimedependentstateinforma-
d,t
trolavailableatthelevelofaTCListodecidewhetherornot tionX,controllablestateinformationX anduncontrollable
t phys
toextractheatfromtheDHN,correspondingtoabinaryvalue exogenousstateinformationX [17].Thetimedependentstate
ex
bd,t ∈ {0,1}. ThethermalinertiaoftheDHNandTCLsisused information Xt describes the time information relevant for the
forstorage,noseparatehotwaterstorageisconsidered. dynamics,e.g.thequarterofanhourinthedayorthedayinthe
Besides TCLs also production units p ∈ P are connected week. InthisworkX ={1,...,96}. Thecontrollablestateinfor-
t
totheDHNinspecificnodes. AsillustratedinFigure1,herea mationX representsthestateoftheDHNandtheTCLs:
phys
singleCHPisusedasheatsource.Ateverytimestept,aninput
power PIp,t, a thermal output power PTp,t and an electric output xphys,t =(T1a,t,T1b,t,T1h,t,...,T|aD|,t,T|bD|,t,T|hD|,t, (7)
T ,...,T ,T ,V ,...,T ,V ). (8)
1,t |N|,t 1,t 1,t |L|,t |L|,t
3
Theuncontrollableexogeneousstateinformationcomprisesthe 3.3.5. Costsignal
physicalparametersrelevantforthedynamicsofthesystemthat Finally also a cost function c needs to be defined: c : X ×
can not be influenced by ut. Examples being the outside tem- U × X → R, in this work, two objectives are regarded, i.e.
→−
peratureTout,solarirradiationS,windspeedanddirection v 2 energyarbitragecλ,respondingtoanexternalpriceλandpeak
andlocalelectricconsumptioned. shaving/valleyfillingcps. Thecostfunctionsaredefinedas:
xex,t =(cid:16)Tout,t,St,→−vt,e1,t,...,e|D|,t(cid:17). (9) cλ(xt,ut,xt+1)= PTp,tλt∆t. (12)
Itisx thatrepresentsw.Ifthereisnocorrelationbetweenw withλ theeffectivepriceforproducingthermalenergyattime
ex,t t t t
andwt+1(pw(.)),xex,tcanbeomittedfromthestateinformation stept. Forthepeakshavingobjective,thecostisexpressedon
[17] in the MDP, however by having x in the state vector a adailybasis:
ex,t
firstordercorrelationisassumed3(p (.|x )).
w ex (cid:110)(cid:13) (cid:13) (cid:111)
cps(x1,u1,...,uT)=E (cid:13)(cid:13)PTp,1,...,PTp,T(cid:13)(cid:13)∞ . (13)
3.3.2. Controlactions
Theobjectiveofthisworkistofindacontrolpolicyπ: X →U
ThecontrolvectorincludesthecontrolactionsoftheTCLs
thatwillminimizetheT-stagereturn Jπ(x )startingfromstate
andthethermaloutputpowerofthecentralCHP: 1
x definedas:
1
u =(PT ,b ,...,b ). (10)
t p,t 1,t |D|,t Jπ(x )=E(Rπ(x ,w ,...,w )). (14)
1 1 1 T
3.3.3. DHNDynamics
with
To model the dynamics of the DHN (as used in the evalu- (cid:88)T
ation), a quasi-dynamic approach is followed [29] as pressure Rπ(x ,w ,...,w )= ρ(x,π(x),w) (15)
1 1 T t t t
and flow change orders of magnitude faster than the tempera- t=1
ture ofthe water. In afirst step, a hydraulicsimulation isper- from the understanding that an optimal policy π∗ satisfies the
formed, in a second step the thermal dynamics are calculated. Bellmanequation:
Forthehydrauliccalculationstheapproachasproposedin[38]
isfollowed. ThisresultsinapplyingKirchhoff’slaws,withthe Jπ∗(x)=min E {ρ(x,π(x),w)+Jπ∗(f(x,π(x),w))} (16)
considerationthatthereexistsanon-linearrelationshipbetween u w∼Pw(.|w)
pressureandflowrate. Whenanaccuratemodelisavailable,typicaltechniquestofind
Tocalculatethethermaldynamics, thenodemodelaspre- near-optimalpoliciesinanMDPframeworkarevalueiteration,
sented by Benonysson [39] has been used, essentially solving policyiteration,directpolicysearch[40]andtreesearchalgo-
thefollowingequationforeverypipesection: rithmssuchasoptimisticplanning[41].In[30]forexample,an
approximatedvalueiterationapproachisfollowedtodetermine
∂mT ∂m˙T
c w +c w +hA(T −T )= Q. (11) acontrolpolicyforthecontrolofaDHN.
pw ∂t pw ∂x w g
In this work, a model-free approach is explored. Driven by
Withmthemassofthewater, c thethermalcapacityofwa- promisingresults[42,16,36],BatchModeReinforcementLearn-
pw
ter, T the water temperature, Q the heat demand and h the ing(BRL)techniquesareinvestigated,detailedinSection4.
w
heattransfercoefficientbetweenthewaterandtheground. The
surfaceofthepipeconsideredisAandT thelocalgroundtem-
g 4. DHNcontrollerapproach
perature.
Thissectiondescribesapragmaticcontrolapproachbuild-
3.3.4. TCLDynamics ing upon recent results in BRL and market-based multi-agent
For the building models, a lumped capacitance model is systems. The control approach illustrated in Figure 2 and fol-
usedi.e.anelectricanaloguefollowingVerbeek[9].Themodel lowedinthisworkisbaseduponaThreeStepApproach(TSA)
includesthetemperaturedynamicsoftheinsideair,abuilding aspresentedin[14,43]followingasimilarstrategyasfollowed
envelope and the heating system return temperature [29]. Be- in[21,24]. Inafirststep,(1)allrelevant(andpracticallyavail-
sidesheatlossestotheambientair,alsowindspeeddependent able)stateinformationiscollected,e.g. thetemperatureinfor-
air infiltration losses are included, as are uncontrolled heating mationfromtheTCLs. Fromthisinformationalimitedsetof
duetosolarirradiationandlocalelectricconsumption[29]. features [17] is extracted, resulting in a low-dimensional rep-
resentation of the system state. In a second step (2) a control
actionfortheentireclusterofTCLsisextractedfromapolicy
2The outside temperature, solar irradiation and wind information are as- π∗ determined offline on given time intervals. In a third and
sumedconstantovertheDHNandthebuildings. laststep(3),thiscontrolactionisdispatchedoverthedifferent
3This can readily be extended to include information several time steps
back. TCLsusingamarket-basedmulti-agentsystem.Thisprocessis
repeatedfollowingarecedinghorizonapproach. Inthefollow-
ing,amoredetaileddescriptionofthethreestepsispresented.
4
Step 1 Step 2 Step 3 functioniszero:
Aggregation Optimal control Dispatch
Ta−Ta
p =1−SoC = d d,t. (21)
c,d Ta−Ta
d d
Determining this heuristic is considered relatively straightfor-
wardasitrequiresonlytheairtemperatureasmeasuredbythe
thermostatandtheupperandlowertemperaturebound. Defin-
ingthethermalpowerextractedfromtheDHNbyaTCLwhen
switched on, is less so [29]. To relax this requirement we as-
sumeanestimateoftheflowrate(L )whenswitchedonisavail-
d
able. The flowrate L is defined as follows. First the set tem-
d
Figure2:Overviewofthethree-stepapproach.Inafirst(left)aggregationstep
peratureoftheindoorheatsupplysystem, whichisafunction
theflexibilityandstateinformationoftheThermostaticallyControlledLoads
oftheoutdoortemperature,iscalculated. Then,amodelforthe
(TCLs)areaggregated. Inasecondoptimizationstep(middle), theoptimal
controlactionu∗fortheentireclusterisdetermined,finallyinstepthree(right) substationheatexchangerisusedtodeterminetheflowrateex-
u∗isdispatchedusingamarket-basedmulti-agentsystem. tracted from the DHN, by which the outlet temperature of the
heatexchangermeetsthesettemperature. Usingthisvaluein-
stead of the actual power results in the following bid function
4.1. Step1: Aggregation
forbuildingd:
In the first step, state information as described in Section
3.3.1 is retrieved from the system. From a practical perspec- b (p )= L (1−H(p +SoC−1)), (22)
d r d r
tive however, not all state information is readily available. At
building level, only the air temperatures as measured by a lo- hereHcorrespondstotheheavisidefunction.
cal thermostat are assumed available, i.e. the air temperatures
measuredinthebuildings,Ta ,∀d ∈D. Furthermoremeasure- Algorithm1OverviewfittedQ-iteration
d,t
ments of the outside air temperature Tout,t are assumed avail- Input: F = (cid:110)(x ,u ,c ,x(cid:48)),l =1,...,n (cid:111), regression algo-
able, asisthewatertemperaturefromasubsetofnodesinthe D ls ls ls ls s s
rithm[47].
supplysideoftheDHN,i.e. Tn,t,∀n ∈ Ns ⊂ N andthereturn letQˆ bezeroeverywhereonX×U
side4 T ,∀n ∈ N ⊂ N. This information is further aggre- 1: 0
n,t r repeat
2:
gated, formally this can be seen as a feature extraction [17], forl =1,...,n do
3: s s
which reduces the dimensionality of the decision-making pro- Q(cid:48) ←r +min Qˆ (x(cid:48),u(cid:48))
cess. Inthiswork,thefeatureextractionishandcrafted,result- 4: l+1,ls ls u(cid:48) l ls
endfor
inginthefollowingeffectivestatevectorxt. 56:: (cid:110)u(cid:16)se regressio(cid:17)n to o(cid:111)btain Qˆl+1 from
xt =((cid:80)(cid:10)Tta(cid:11),TTaout,t,Ts,Tr) (17) 7: unti(lxQlˆsl,+u1lsi)s,sQa(cid:48)lt+is1,flasc,tolsry=1,...,ns
(cid:10)Tta(cid:11)= d∈|DD| d,t (18) 8: return Qˆ = Qˆl+1
(cid:80)
T
T = n∈Ns n,t (19)
s |N |
s 4.2. Step2: BatchReinforcementLearning
(cid:80)
T
Tr = n∈|NNr| n,t (20) In the second step, a control action ut is selected once ev-
r ery15minutes,followingthepolicyπ. Thecontrolactionu is
t
selected for the entire cluster which is projected onto individ-
Although, more generic dimension reduction techniques such
ualcontrolactionsasdescribedinSection4.3. Oneofthemain
asautoencoderscanbeused[44],theaimofthisworkistoun-
goalsofthisworkistoexploretowhatextent(model-free)rein-
derstand what performance can be obtained starting from this
forcementlearningcanbeusedtodetermineπ. Reinforcement
limited state description. Alternatively, a convolutional neural
Learning (RL) is a model-free control approach that learns a
network as presented by the the authors in [45] could be used
policybyinteractionwiththesystem[40]. Inrecentliterature,
toautomaticallyextractrelevantstate-timefeatures,allowingto
RL(mainlyintheformofQ-learning[48,49,50])hasbeenpre-
addhistoricobservationstothestate[16,17].
sentedasaneffectivemodel-freelearningapproachforDRap-
To facilitate the dispatch step as explained in Section 4.3,
plications. Thepracticalityhowever,suffersfromslowconver-
abid-functionisdefinedforeveryTCL[5,14,46]. In[5],the
gence[15,40]andthecurseofdimensionality[17].Thesechal-
bidfunctionofadeviceisexpressedastheelectricpowercon-
sumedversusaheuristic(p ). Aboveacornervalue p thebid lengescanbepartiallymitigatedbyusingpastinteractionsand
r c,d
appropriatefunctionapproximatorsinaBRLstrategy[40,51].
ApopularBRLapproachisthatofFittedQ-Iteration(FQI)in-
4Onlytemperaturesatalimitedsetofnodesareassumedavailable. troducedbyErnstetal. in[52],especiallyincombinationwith
extremely randomized trees as regression technique [47]. In
5
[42]theauthorsconcludethatespeciallyfornon-linearcontrol
Energy
problems,FQIcanbeavaluablealternativetoMPCapproaches market
with the extra advantage that FQI is a blind technique. More-
over, FQI and MPC can strengthen each other [42, 53]. Al-
time slots
though several BRL techniques have been proposed in the lit-
ertharanetdpuoermerfo[iz3rem6d,a5tnr2ce,ee4so4af]so,ttrhheeigsrreBwsRosirLoknafpoaplcguroosraeicsthhomensiF[s4Q7cIo].nusCsiidonemgrepedxartoriseuomtsneidltyoe - P* PI Flow rate event based
Priority
thescopeofthiswork.
Concentrator
Following [52], an approximation of the state-action value agent
function Q(x,u), Qˆ(x,u) is built on a daily basis5 from a
t t t t
batchoffour-tuplesF : Supply
D
(cid:110) (cid:111)
F = (x ,u ,c ,x(cid:48)),l =1,...,n . (23)
D ls ls ls ls s s House agent House agent House agent
(C)HP
Withthestatevectorasdefinedinequation(17). Algorithm1
isusedtoobtainQˆ(xt,ut). Duringtheday,thecontrolactionut TCL TCL TCL
isselectedwithaprobabilitydefinedby[54]:
Return
eQˆ(xt,ut)/τD
P(u|x)= . (24) Figure3:Overviewofthecontrollerapproachasdevelopedinthiswork.
t t (cid:80)u˜eQˆ(xt,u˜)/τD
The temperature τ is decreased on a daily basis according to 5. Evaluation
D
a harmonic sequence [31], a high temperature results in more
explorationwhilstτ →0resultsinagreedyapproach: To evaluate the performance of the controller described in
D
Section 4, a set of simulations have been performed. In this
u =argminQˆ(x,u). (25) section,firstacondenseddescriptionofthesimulationscenario
t t
u will be presented after which the tracking performance of the
controller is evaluated. The performance of the controller is
Inthepeakshavingscenario,acontrolactionisdeterminedon
evaluated for two distinct objectives, i.e. that of energy arbi-
adailybasis,definingtheaveragepowertobefollowedforthe
trage on a day-ahead energy market and peak shaving/valley
nextdayasdetailedin[5].
filling.
4.3. Step3: Real-timecontrol
5.1. Simulationscenario
In the third step, the energy corresponding to u∗ is dis-
t As mentioned in Section 1, the scenario is designed to be
patchedovertheclusterofTCLs, usingamarket-basedmulti-
sufficiently demanding, but also simple enough to allow for
agent system [14, 55]. Compared to the work of [5], there is
asignificantdifferenceasonlytheexpectedflowrateforeach an analysis which is not obfuscated by the complexity of the
scenario. To this end, the (arbitrary) topology as depicted in
TCLisassumedavailable. Tothisend,aProportionalIntegra-
Figure 1 has been used, i.e. a central CHP (η = 0.97,φ =
tor (PI) controller (at a central level) managing the flow rates
atthedifferentbuildingsisused. Sincehydrauliceffectsoccur 0.93,PTp,max = 1100kW)providesheatto100TCLsconnected
nearlyinstantaneousinaDHN,thiswillhaveadirecteffecton toaradialDHN.Eachbuildingislocatedinoneoffourstreets.
Thetotallengthofthegridis2.1km,withpipediametersrang-
theflowrateatthesource,influencingthepoweratthesource
ingfromDN25toDN100. Adetaileddescriptionofthesimu-
sideasthesupplysetpointisassumedtobeconstantinthesim-
lation scenario can be found in [29]. The TCL building mod-
ulations[29]. Anoverviewofthereal-timecontrolcanbeseen
els are lumped capacitance models using an electric analogon
in Figure 3. As described in Section 4.1, every TCL is repre-
comprisingcapacitancesandresistors. Thecapacitancevalues
sented by a bid function b . After a clearing process (26), a
d
clearingpriority p ∗issentbacktothedifferentdevices: arerelatedtothetemperatureoftheinsideair,abuildingenve-
r
lopeandtheheatingsystemreturntemperature[9,29]. All100
(cid:12) (cid:12)
(cid:12)(cid:12)(cid:88)|D| (cid:12)(cid:12) buildings models included in the simulation are derived from
p∗r = argmin(cid:12)(cid:12)(cid:12) bd(pr)−u(cid:12)(cid:12)(cid:12), (26) thesamemodel. Differentmodelparametersareusedforeach
pr (cid:12)d=1 (cid:12) building, by sampling capacitance and resistance values from
a normal distribution with a standard deviation of 20% of the
Thedevicesopenorclosetheirlocalvalveaccordingtob (p∗).
d r standard value. The standard values correspond to a detached
house with a living area of 103 m2 and a protected volume of
452m3. Themaximumstandardpowerdemandofthebuilding
5Thisisdoneforpracticality,asthesimulationscoveratimespanofseveral
months, in a real-life application, Qˆ should be constructed more frequently, is 9.8 kW at an internal temperature of 20◦C and an ambient
followingAlgorithm1. temperature of -8◦C. Besides heat losses to the ambient air,
6
alsowindspeeddependentairinfiltrationlossesareincludedas row,thedailycostisdepicted,bothfortheapproachpresented
isuncontrolledheatingduetosolarirradiationandlocalelectric inthispaperandadefaultcontroller. Thedefaultcontrollerap-
consumption [29]. For simplicity the temperature constraints pliesahysteresiscontrollerateverybuildingandafixedDHN
aresetthesameforallbuildingsat19.5and20.5◦C. inlet temperature [29]. The cumulative cost is depicted in the
secondrow. Itcanbeobservedthatinitiallythedailycostob-
5.2. Trackingperformance tainedwiththeBRLcontrollerissimilartothecostofthede-
faultcontroller.However,astheBRLcontrollerstartsgathering
moreinteractions,itsdailycostforenergystartsdecreasing. To
20.4 evaluate the performance during colder days, the first 20 days
20.2 withaloweraverageoutsidetemperaturewhererepeated,with
C]
°T [ 192.80 athpeorliigchytccoonlsutmrunc.tedThweitphearfllodrmataan.cTehiessesirgensiufilctsanartleydiemppicrtoevdeidn,
19.6 andthedailycostisdecreasedwithabout20%. Althoughthese
19.4 resultsarepositive, theydonotprovideanobjectivemetricof
0 200 400 600 800 1000 1200 1400 1600 1800 2000
Time [minutes] thequalityofthecontrolapproach,sincethispercentageisbi-
asedbytheactualpriceprofiles. Tothisend,alowerboundon
Setpoint thedailycostisprovidedbytakingthetotalenergyconsumed
1500
power source
W] duringeachdayanddistributingthisoverthosehourswiththe
er [ k1000 highestpriceassumingtheCHPrunningatfullcapacity. This
w isconsideredanover-optimisticbenchmark. Inthethirdrowof
Po 500
Figure5thefollowingmetricisdepicted:
00 200 400 600 80T0ime1 [0m0i0nute1s2]00 1400 1600 1800 2000 M = FQI−DC, (27)
LB−DC
Figure4:Topgraph,thetemperaturedynamicsofthe100simulatedbuildings with DC the daily cost of the default controller, LB the lower
asistheaveragetemperatureindicatedbytheblackline. Bottomgraph,the boundonthedailycostandFQIthedailycostobtainedwiththe
requestedthermalpowerateverycontrolstepindicatedbythedashedline,as
solutionpresentedinthiswork. Ametricof0correspondingto
istheactualthermalpowerdeliveredbytheCHPindicatedbytheblackline.
thesameperformanceasthedefaultcontroller,whilstametric
of1correspondingtothelowerboundsolution. FromFigure5
AfirstsupportingresultisdepictedinFigure4, hereanu-
itisobservedthattheperformancemetricgraduallyincreasesto
merical experiment was performed where every 15 minutes a
avalueofaround60-70%,alsoforthecolderdaysasdepicted
randomcontrolactionwasselected(withinthetechnicalspeci-
intherightcolumn.
ficationoftheCHP).Thecorrespondingsetpointsaredepicted
in the lower part of Figure 4, as is the average thermal power
2000 2000
as produced by the CHP. The upper graph of Figure 4 depicts BRL
the internal temperatures of the 100 buildings and the average ost [€]1000 DC ost [€]1000
temperature. The graph shows that when the buildings are on c c
0 0
0 20 40 60 80 0 10 20
averageintheirdeadband,therequestedpowercanbetracked
100 20
accurately. Although the buildings have different physical pa- €] €]
rameters,theytendtosynchronizewithregardtotheirtempera- ost [k 50 ost [k10
turerelativetothecomfortconstraints. Thisisadirecteffectof c 0 c 0
0 20 40 60 80 0 10 20
thedispatchdynamics,asthosebuildingswithahigherpriority 1 1
areservedfirst.
M0.5 M0.5
5.3. Energyarbitrage 0 0
C] 0 20 40 60 80 0 10 20
tprriiccIeensnteahrregeystcaedknieraenrciotflryoofmaetnttheheregByweahlrgobiliaetrnsaagdleea,ymt-haaehrCekaeHdtP[m2c]aa.rnksTeethlle[5ite6sn]e,elrteghcye- °perature [ 200 M0.15
m−20 0
gaspriceissetat38.6e/MWh[57].Asday-aheadpricescanbe Te 0 20 Time4 [0days] 60 80 T−e2m0pera0ture [2 °0 C]
predictedwithareasonableaccuracy[58],theseareconsidered
deterministicinthisevaluation.Furthermore,theofflinepolicy Figure5: Overviewofresultsobtainedforanenergyarbitragescenario. The
calculation (Algorithm 1) is performed only on a daily basis6, uppergraphsdepictthedailycostforheatingforboththeBRLapproachand
andthesimulationperiodcoversupto80days. Theresultsof thedefaultcontrolcase. Thegraphsatthesecondrowpresentthecumulative
cost.Thegraphsinthethirdrow,depictaperformancemetric(M)asexplained
thisnumericalexperimentaredepictedinFigure5.Intheupper
inthetext. Thelowergraphsshowthedailyaverageoutsidetemperatureand
thesensitivityoftheperformancemetricrelativetotheaverageoutsidetemper-
ature.
6Thisisforpracticalreasonsasthecalculationofapolicytypicallytakes
20minutesonaIntel,2.5GHz,8GBRAM Asnapshotofthedailypowerprofilesofamaturecontroller
7
comparedtothedefaultcontrollerisdepictedinFigure6.Itcan 1500
BRL
be seen that the controller produces heat when energy prices
W] DC
arehigh. Itismeaningfultounderstandthatadefaultcontroller [k1000 LB
already has a reasonable performance as the heat demand is
typicallylargestwhenalsothewholesalepriceishighest. This costy 500
correlationishoweverexpectedtodecreaseasmorerenewable ail
D
energycomesintheproductionmix, makingwholesaleprices
0
morevolatile. C] 0 10 20 30 40 50 60 70 80 90 100
e [° 20
€/MWh]15000 Temperatur−2000 10 20 30 40Time5 0[Day]60 70 80 90 100
Price [ 00 20 40 60 80 100 120 140 160 180 200 kW]2000 kW]1000
1400 er [1000 er[ 500
BRL w w
1200 DC Po 00 2000 4000 6000Po 00 2000 4000 6000
numberofq.h. numberofq.h.
W]1000
wer [k 800 Fwiogrukrean7d:Ttohpegdreafpahu,ltdcaoilnytrpoolwleer.rpTehaek,mboidthdlfeorgrtahpehapsphroowaschthperedsaeinltyedavinertahgies
o
p temperature. Thelowerrowpresentsloaddurationcurvesforthefirst50days
e 600
ag andforthelast50days.
er
av 400
900
200 BRL
DC
0 800 set point
0 20 40 60 80 100 120 140 160 180 200
Time [quarter hour]
700
Figure6: Topgraph, thewholesalepriceprofile. Lowergraph, theaverage W]
oinuttphuistwpoowrke.r,bothforthedefaultcontrolcaseandtheBRLapproachpresented wer [600
o
p
e
ag500
er
v
5.4. peakshaving A
Inthesecondexperiment,peakshavingisconsidered. The 400
summarizingresultsaredepictedinFigure7. Inthetopgraph
onecanseethedailypeakinthermalpowerobtainedwiththe 300
default controller and the BRL approach. The second graph
gives the average daily ambient temperature. The daily maxi- 200
0 50 100 150 200 250
mum power peak indeed reduces with time. To make the per- Time [quarter hour]
formancemorevisible,alsotheloaddurationcurvesareplotted
forthefirstandthelast50days,bothfortheBRLapproachand Figure8:Powerprofile,bothforthecontrolleraspresentedinthisworkanda
defaultcontrollerinthepeakshaving/valleyfillingscenario.
thedefaultcontroller.Thefirst50daysthereislimitedimprove-
ment over the default controller, however for the subsequent
periodonecanclearlyobservetheeffectofthecontrollerfrom 6. Conclusionsandfuturework
theloaddurationcurves. Alowerboundisdepictedbyplotting
theaveragepowercorrespondingtothedailyenergyconsumed In this work the control problem of a cluster of thermo-
bythedefaultcontrolapproach. Note,ondayseven(topgraph statically controlled loads connected to a district heating net-
inFigure7),theprofileobtainedwithBRLresultsahighpeak work is addressed by assessing the performance of a control
power, which is attributed to an exploration step (24). The fi- approachcomprisingamodel-freereinforcementlearningtech-
nal performance is visualized more clearly in Figure 8 where niqueincombinationwithamarket-basedmulti-agentsystem.
thepowerprofileisplottedforamaturecontroller. Indeed,the The performance of the controller has been evaluated for two
thermal power follows a constant profile compared to the de- distinctscenarios,i.e.energyarbitrageandpeakshaving.Inthe
faultcontrolcase. Avisualizationofthepolicyobtainedbythe evaluation a detailed district heating network model has been
BRLcontrollerispresentedinFigure9,heretheaveragepower used including hydraulic and thermal dynamics. For the en-
set point is plotted versus the initial indoor temperature state ergy arbitrage scenario, solutions are obtained that reach over
andtheexpectedaveragedailyoutdoortemperature. Asiscon- 65%oftheavailableoptimizationpotentialafteralearningpe-
ceived logical, the setpoint decreases with increasing outside riod of 40 to 60 days. Knowing that the policy is updated on
temperatureandaverageairtemperature. a daily base, this is considered a promising result. Also for
8
set point [kW] [9] G.Verbeeck,OptimisationofExtremelyLowEnergyResidentialBuild-
20 900 ings,PhDthesis,KULeuven,2007.
19.95 [10] J.Kensby,A.Trschel,J.-O.Dalenbck,Potentialofresidentialbuildings
850 asthermalenergystorageindistrictheatingsystems resultsfromapilot
19.9
test,AppliedEnergy137(2015)773–781.
C]19.85 800 [11] G. Sandou, S. Font, S. Tebbani, A. Hiret, C. Mondon, S. Tebbani,
°mperature [ 191.97.58 750 Aitnro.gl,HnEierutewrto,oprCeka,.niMnC:oonnPdtrorooncl,.CP4or4netdhfeicrIetEinvEceEec(CoCnoDtnrCofel-rEeoCnfcCea),oc2no0m0D5pe,lcepixspi.odni7s3tar7in2cdt–7Ch3eo7an7t--.
Inside Te191.96.57 700 [12] Jmd.ooÅid:ier1lo0pk.r,e1Fd1.i0cOt9iv/ldeCeDcwCou.nrt2tre0ol0l,5fJo..r1Ca5in8g3lee.nre,rSgy. Perffivacriean,tEbxupieldriimngenhteaaltianngalsyyssitsemof,
650
19.6 AppliedEnergy88(9)(2011)3079–3087.
[13] J.L.Mathieu,M.Kamgarpour,J.Lygeros,D.S.Callaway,Energyarbi-
19.55 600
tragewiththermostaticallycontrolledloads,in: Proc.EuropeanControl
19.5 Conference(ECC),IEEE,2013,pp.2519–2526.
0 5 10 15
Outside Temperature [° C] [14] S.Vandael,B.J.Claessens,M.Hommelberg,T.Holvoet,G.Deconinck,
Ascalablethree-stepapproachfordemandsidemanagementofplug-in
Figure9: PolicymapasobtainedwiththeBRLapproachforthepeakshav- hybridvehicles,IEEETrans.onSmartGrid4(2)(2013)720–728.
ing/valleyfillingscenario. [15] S. Lange, T. Gabel, M. Riedmiller, Batch reinforcement learning, in:
M.Wiering, M.vanOtterlo(Eds.), ReinforcementLearning: State-of-
the-Art,Springer,NewYork,NYC,2012,pp.45–73.
thepeakshaving/valleyfillingscenario,promisingresultshave [16] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G.
Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski,
been obtained, since a clear performance improvement is ob- etal.,Human-levelcontrolthroughdeepreinforcementlearning,Nature
served. These results support a practical implementation and 518(7540)(2015)529–533.
comingofageofreinforcementlearningtechniques. [17] D.Bertsekas,J.Tsitsiklis,Neuro-DynamicProgramming,AthenaScien-
tific,Nashua,NH,1996.
To understand the potential of completely model-free con-
[18] J.L.Mathieu,M.Kamgarpour,J.Lygeros,G.Andersson,D.S.Callaway,
trol a direct implementation of fitted Q-iteration as presented Arbitragingintradaywholesaleenergymarketpriceswithaggregationsof
in[40]hasbeenused. Howeverasdiscussedin[42], combin- thermostaticloads, IEEETrans.onPowerSystems30(2)(2015)763–
ing general domain knowledge with a model-free approach is 772.
[19] S.Iacovella,F.Ruelens,P.Vingerhoets,B.J.Claessens,G.Deconinck,
expected to result in an improved performance at a reduced
Clustercontrolofheterogeneousthermostaticallycontrolledloadsusing
learning time. This domain knowledge can be incorporated tracerdevices, IEEETrans.onSmartGridPP(99)(2015)1–9. doi:
throughe.g. informationregardingtheshapeofthepolicy[40] 10.1109/TSG.2015.2483506.
[20] E.Georges,B.Corne´lusse,D.Ernst,Q.Louveaux,V.Lemort,S.Mathieu,
orthroughusingamodelsimilarasin[53]. Asecondpointof
Directcontrolservicefromresidentialheatpumpaggregationwithspec-
futureresearchisdirectedatmoreautomatedfeatureextraction
ifiedpayback,in: Proceedingsofthe19thPowerSystemsComputation
techniquessuchasautoencoders[16],whichisalsoexpectedto Conference(PSCC),2016.
resultinareducedlearningtime. [21] B.Biegel,P.Andersen,J.Stoustrup,M.B.Madsen,L.H.Hansen,L.H.
Rasmussen,Aggregationandcontrolofflexibleconsumers-areallife
demonstration,in: Proc.ofthe19thIFACWorldCongress,IFAC,Cape
References Town, South Africa, 2014. doi:10.3182/20140824-6-ZA-1003.
00718.
[22] M.Hu,Adata-drivenfeed-forwarddecisionframeworkforbuildingclus-
References
tersoperationunderuncertainty, AppliedEnergy141(0)(2015)229–
237.
[1] C.Johansson,OnIntelligentDistrictHeating,PhDThesis,BlekingeIn- [23] S.Mathieu,D.Ernst,Q.Louveaux,Anefficientalgorithmfortheprovi-
stituteofTechnology,Blekinge,Sweden,2014.
sionofaday-aheadmodulationservicebyaloadaggregator,in:inProc.
[2] Y.Kitapbayev,J.Moriarty,P.Mancarella,Stochasticcontrolandrealop-
4thIEEEInnovativeSmartGridTechnologiesEurope(ISGTEUROPE),
tionsvaluationofthermalstorage-enableddemandresponsefromflexible
2013,pp.1–5.doi:10.1109/ISGTEurope.2013.6695247.
districtenergysystems,AppliedEnergy137(2015)823–831.
[24] J.Mathieu,S.Koch,D.Callaway,Stateestimationandcontrolofelec-
[3] K. Sartor, S. Quoilin, P. Dewallef, Simulation and optimization of a
tricloadstomanagereal-timeenergyimbalance,IEEETrans.onPower
CHPbiomassplantanddistrictheatingnetwork,AppliedEnergy130(0)
Systems28(1)(2013)430–440.doi:10.1109/TPWRS.2012.2204074.
(2014)474–483.
[25] W. Zhang, K. Kalsi, J. Fuller, M. Elizondo, D. Chassin, Aggregate
[4] L. Ozgener, A. Hepbasli, I. Dincer, Performance investigation of two
modelforheterogeneousthermostaticallycontrolledloadswithdemand
geothermal district heating systems for building applications: Energy
response, in: Proc.IEEEPowerandEnergySocietyGeneralMeeting,
analysis, EnergyandBuildings38(4)(2006)286–292. doi:http:
2012,pp.1–8.
//dx.doi.org/10.1016/j.enbuild.2005.06.021.
[26] N.Gatsis,G.Giannakis,Residentialloadcontrol:Distributedscheduling
[5] B.J.Claessens,S.Vandael,F.Ruelens,M.Hommelberg,Self-learning
andconvergencewithlostAMImessages, IEEETrans.onSmartGrid
demandsidemanagementforaheterogeneousclusterofdeviceswithbi-
3(2)(2012)770–786.doi:10.1109/TSG.2011.2176518.
narycontrolactions,in:Proc.3thIEEEInnov.SmartGridTechnol.Conf.
[27] M.G.C.Bosman,PlanninginSmartGrids,PhDThesis,Universityof
(ISGTEurope),Berlin,Germany,2012,pp.1–8.
Twente,2012.
[6] F.DeRidder,B.Claessens,Atradingstrategyforindustrialchpsonmul-
[28] B.Biegel,J.Stoustrup,P.Andersen,Distributedmodelpredictivecontrol
tiplepowermarkets, InternationalTrans.onElectricalEnergySystems
via dual decomposition, Intelligent Systems, Control and Automation:
24(5)(2014)677–697.
ScienceandEngineering69.
[7] S.Koch,DemandResponseMethodsforAncillaryServicesandRenew-
[29] D.Vanhoudt, B. Claessens, R.Salenbien, J. Desmedt, Theuse ofdis-
ableEnergyIntegrationinElectricPowerSystems,PhDthesis,University
tributedthermalstorageindistrictheatinggridsfordemandsideman-
ofStuttgart,2012.
agement,https://arxiv.org/abs/1702.06005,underreviewatEl-
[8] K.Vanthournout,R.D’hulst,D.Geysen,G.Jacobs,Asmartdomestichot
sevier:Energyandbuildings(feb2017).arXiv:1702.06005.
waterbuffer,IEEETrans.onSmartGrid3(4)(2012)2121–2127.
9
[30] E. Ikonen, I. Selek, J. Kovacs, M. Neuvonen, Z. Szabo, J. Bene, buildingtransferlearning,EnergyandBuildings116(2016)646–655.
J. Peurasaari, Short term optimization of district heating network sup- doi:http://dx.doi.org/10.1016/j.enbuild.2016.01.030.
plytemperatures,in: Proc.IEEEInternationalEnergyConference(EN- URL http://www.sciencedirect.com/science/article/pii/
ERGYCON),2014,pp.996–1003. doi:10.1109/ENERGYCON.2014. S0378778816300305
6850547. [51] X.Xu,L.Zuo,Z.Huang,Reinforcementlearningalgorithmswithfunc-
[31] W.B.Powell,ApproximateDynamicProgramming: SolvingtheCurses tionapproximation: Recentadvancesandapplications,InformationSci-
ofDimensionality,2ndEdition,Wiley,2011. ences261(0)(2014)1–31.
[32] S. Grosswindhager, A. Voigt, M. Kozek, Predictive control of district [52] D.Ernst,P.Geurts,L.Wehenkel,Tree-basedbatchmodereinforcement
heatingnetworkusingfuzzyDMC,in:Proc.ofInternationalConference learning.,JournalofMachineLearningResearch,6(1):503-556doi:10.
onModelling,IdentificationControl(ICMIC),2012,pp.241–246. 1109/TPWRS.2009.2016607.
[33] P.Pinson,T.Nielsen,H.Nielsen,N.Poulsen,H.Madsen,Temperature [53] T.Lampe, M.Riedmiller, Approximatemodel-assistedneuralfittedQ-
predictionatcriticalpointsindistrictheatingsystems,EuropeanJournal iteration, in: IEEEInternationalJointConferenceonNeuralNetworks
ofOperationalResearch194(1)(2009)163–176. (IJCNN2014),Beijing,China,2014.
[34] P.Booij,V.Kamphuis,O.Pruisen,C.Warmer,Multi-agentcontrolfor [54] R. Anderson, A. Boulanger, W. Powell, W. Scott, Adaptive stochastic
integrated heat and electricity management in residential districts, in: controlforthesmartgrid,Vol.99,2011,pp.1098–1115.doi:10.1109/
4thInternationalWorkshoponAgentTechnologiesforEnergySystems, JPROC.2011.2109671.
ATES2013,2013.doi:10.1109/ENERGYCON.2014.6850547. [55] M.Hommelberg,B.vanderVelde,C.Warmer,I.Kamphuis,J.Kok,A
[35] Q. Gemine, D. Ernst, B. Corne´lusse, Active network management for novelarchitectureforreal-timeoperationofmulti-agentbasedcoordina-
electrical distribution systems: problem formulation and benchmark, tionofdemandandsupply,in:PowerandEnergySocietyGeneralMeet-
CoRRabs/1405.2806. ing-ConversionandDeliveryofElectricalEnergyinthe21stCentury,
URLhttp://arxiv.org/abs/1405.2806 2008IEEE,2008,pp.1–5.doi:10.1109/PES.2008.4596531.
[36] R. Fonteneau, S. Murphy, L. Wehenkel, D. Ernst, Batch mode rein- [56] Belpex-Belgianpowerexchange,http://www.belpex.be/,[Online:
forcementlearningbasedonthesynthesisofartificialtrajectories, An- accessedMarch21,2016].
nalsofOperationsResearch208(1)(2013)383–416. doi:10.1007/ [57] Eurostat, Gas prices for industrial consumers 2013, http://epp.
s10479-012-1248-5. eurostat.ec.europa.eu.
URLhttp://dx.doi.org/10.1007/s10479-012-1248-5 [58] A.Conejo,M.Plazas,R.Espinola,A.Molina,Day-aheadelectricityprice
[37] F. Ruelens, B. J. Claessens, S. Vandael, B. De Schutter, R. Babusˇka, forecastingusingthewavelettransformandARIMAmodels,IEEETrans.
R.Belmans,Residentialdemandresponseofthermostaticallycontrolled onPowerSystems20(2)(2005)1035–1042. doi:10.1109/TPWRS.
loads using batch reinforcement learning, IEEE Trans. on Smart Grid 2005.846054.
PP(99)(2016)1–11.doi:10.1109/TSG.2016.2517211.
[38] P.Valdimarsson,ModellingofGeothermalDistrictHeatingSystems,Phd
thesis,UniversityofIceland(1993).
[39] A. Benonysson, Dynamic modellign and operational Optimization of
DistrictHeatingSystems,PhDthesis,TechnicalUniversityofDenmark
(DTU),1991.
[40] L.Busoniu,R.Babusˇka,B.DeSchutter,D.Ernst,Reinforcementlearning
andDynamicProgrammingUsingFunctionApproximators,1stEdition,
CRCPress,2010.
[41] L.Busoniu,R.Munos,R.Babusˇka,ASurveyofOptimisticPlanningin
MarkovDecisionProcesses,JohnWiley&Sons,Inc.,2013,pp.494–516.
doi:10.1002/9781118453988.ch22.
URLhttp://dx.doi.org/10.1002/9781118453988.ch22
[42] D.Ernst,M.Glavic,F.Capitanescu,L.Wehenkel,Reinforcementlearn-
ing versus model predictive control: a comparison on a power system
problem,IEEETrans.Syst.,Man,Cybern.,Syst.39(2)(2009)517–529.
[43] A.Aertgeerts,Demandsidemanagementofthethermalflexibilityina
residentialneighborhoodusingahierarchicalmarket-basedmulti-agent
system,masterthesis,KULeuven,2014.
[44] S.Lange,M.Riedmiller,Deepauto-encoderneuralnetworksinreinforce-
mentlearning,in: Proc.IEEE2010Int.JointConf.onNeuralNetworks
(IJCNN),Barcelona,Spain,2010,pp.1–8.
[45] B. J. Claessens, P. Vrancx, F. Ruelens, Convolutional neural networks
forautomaticstate-timefeatureextractioninreinforcementlearningap-
pliedtoresidentialloadcontrol,IEEETransactionsonSmartGridPP(99)
(2016)1–1.doi:10.1109/TSG.2016.2629450.
[46] K.DeCraemer,G.Deconinck,Balancingtrade-offsincoordinatedphev
chargingwithcontinuousmarket-basedcontrol,in: Proc.3rdIEEEPES
InternationalConferenceandExhibitiononInnovativeSmartGridTech-
nologies(ISGTEurope),,2012,pp.1–8. doi:10.1109/ISGTEurope.
2012.6465685.
[47] P.Geurts,D.Ernst,L.Wehenkel,Extremelyrandomizedtrees,Machine
Learning63(1)(2006)3–42.
[48] Z. Wen, D. ONeill, H. Maei, Optimal demand response using device-
basedreinforcementlearning, IEEETrans.onSmartGrid6(5)(2015)
2312–2324.doi:10.1109/TSG.2015.2396993.
[49] E.C.Kara,M.Berges,B.Krogh,S.Kar,Usingsmartdevicesforsystem-
levelmanagementandcontrolinthesmartgrid: Areinforcementlearn-
ingframework,in: Proc.3rdIEEEInt.Conf.onSmartGridCommun.
(SmartGridComm),Tainan,Taiwan,2012,pp.85–90.
[50] E. Mocanu, P. H. Nguyen, W. L. Kling, M. Gibescu, Unsupervised
energy prediction in a smart grid context using reinforcement cross-
10