mgan: training generative adversarial nets with multiple generators

24 Pages·2017·4.67 MB·English

PublishedasaconferencepaperatICLR2018
MGAN: TRAINING GENERATIVE ADVERSARIAL NETS WITH
MULTIPLE GENERATORS
QuanHoang
UniversityofMassachusetts-Amherst
Amherst,MA,USA
[email protected]
TuDinhNguyen,TrungLe,DinhPhung
PRaDACentre,DeakinUniversity
Geelong,Australia
{tu.nguyen,trung.l,dinh.phung}
@deakin.edu.au
ABSTRACT
WeproposeinthispaperanewapproachtotraintheGenerativeAdversarialNets
(GANs)withamixtureofgeneratorstoovercomethemodecollapsingproblem.
Themainintuitionistoemploymultiplegenerators,insteadofusingasingleone
asintheoriginalGAN.Theideaissimple,yetproventobeextremelyeffectiveat
coveringdiversedatamodes,easilyovercomingthemodecollapsingproblemand
delivering state-of-the-art results. A minimax formulation was able to establish
amongaclassifier,adiscriminator,andasetofgeneratorsinasimilarspiritwith
GAN.Generatorscreatesamplesthatareintendedtocomefromthesamedistribu-
tionasthetrainingdata,whilstthediscriminatordetermineswhethersamplesare
truedataorgeneratedbygenerators,andtheclassifierspecifieswhichgeneratora
samplecomesfrom.Thedistinguishingfeatureisthatinternalsamplesarecreated
frommultiplegenerators,andthenoneofthemwillberandomlyselectedasfinal
output similar to the mechanism of a probabilistic mixture model. We term our
method Mixture Generative Adversarial Nets (MGAN). We develop theoretical
analysis to prove that, at the equilibrium, the Jensen-Shannon divergence (JSD)
between the mixture of generators’ distributions and the empirical data distribu-
tionisminimal,whilsttheJSDamonggenerators’distributionsismaximal,hence
effectivelyavoidingthemodecollapsingproblem.Byutilizingparametersharing,
ourproposedmodeladdsminimalcomputationalcosttothestandardGAN,and
thuscanalsoefficientlyscaletolarge-scaledatasets. Weconductextensiveexper-
imentsonsynthetic2Ddataandnaturalimagedatabases(CIFAR-10,STL-10and
ImageNet) to demonstrate the superior performance of our MGAN in achieving
state-of-the-art Inception scores over latest baselines, generating diverse and ap-
pealingrecognizableobjectsatdifferentresolutions,andspecializingincapturing
differenttypesofobjectsbythegenerators.
1 INTRODUCTION
Generative Adversarial Nets (GANs) (Goodfellow et al., 2014) are a recent novel class of deep
generativemodelsthataresuccessfullyappliedtoalargevarietyofapplicationssuchasimage,video
generation,imageinpainting,semanticsegmentation,image-to-imagetranslation,andtext-to-image
synthesis,tonameafew(Goodfellow,2016).Fromthegametheorymetaphor,themodelconsistsof
adiscriminatorandageneratorplayingatwo-playerminimaxgame,whereinthegeneratoraimsto
generatesamplesthatresemblethoseinthetrainingdatawhilstthediscriminatortriestodistinguish
betweenthetwoasnarratedin(Goodfellowetal.,2014). TrainingGAN,however,ischallengingas
itcanbeeasilytrappedintothemodecollapsingproblemwherethegeneratoronlyconcentrateson
producingsampleslyingonafewmodesinsteadofthewholedataspace(Goodfellow,2016).
1
PublishedasaconferencepaperatICLR2018
ManyGANvariantshavebeenrecentlyproposedtoaddressthisproblem. Theycanbegroupedinto
twomaincategories: trainingeitherasinglegeneratorormanygenerators. Methodsintheformer
includemodifyingthediscriminator’sobjective(Salimansetal.,2016;Metzetal.,2016),modifying
thegenerator’sobjective(Warde-Farley&Bengio,2016),oremployingadditionaldiscriminatorsto
yield more useful gradient signals for the generators (Nguyen et al., 2017; Durugkar et al., 2016).
The common theme in these variants is that generators are shown, at equilibrium, to be able to
recover the data distribution, but convergence remains elusive in practice. Most experiments are
conductedontoydatasetsoronnarrow-domaindatasetssuchasLSUN(Yuetal.,2015)orCelebA
(Liuetal.,2015). Toourknowledge,onlyWarde-Farley&Bengio(2016)andNguyenetal.(2017)
perform quantitative evaluation of models trained on much more diverse datasets such as STL-10
(Coatesetal.,2011)andImageNet(Russakovskyetal.,2015).
Givencurrentlimitationsinthetrainingofsingle-generatorGANs,someveryrecentattemptshave
been made following the multi-generator approach. Tolstikhin et al. (2017) apply boosting tech-
niques to train a mixture of generators by sequentially training and adding new generators to the
mixture. However, sequentially training many generators is computational expensive. Moreover,
this approach is built on the implicit assumption that a single-generator GAN can generate very
goodimagesofsomemodes,soreweighingthetrainingdataandincrementallytrainingnewgener-
atorswillresultinamixturethatcoversthewholedataspace.Thisassumptionisnottrueinpractice
sincecurrentsingle-generatorGANstrainedondiversedatasetssuchasImageNettendtogenerate
imagesofunrecognizableobjects. Aroraetal.(2017)trainamixtureofgeneratorsanddiscrimina-
tors, andoptimizetheminimaxgamewiththerewardfunctionbeingtheweightedaveragereward
function between any pair of generator and discriminator. This model is computationally expen-
siveandlacksamechanismtoenforcethedivergenceamonggenerators. Ghoshetal.(2017)train
many generators by using a multi-class discriminator that, in addition to detecting whether a data
sampleisfake,predictswhichgeneratorproducesthesample. Theobjectivefunctioninthismodel
punishesgeneratorsforgeneratingsamplesthataredetectedasfakebutdoesnotdirectlyencourage
generatorstospecializeingeneratingdifferenttypesofdata.
Weproposeinthispaperanovelapproachtotrainamixtureofgenerators. Unlikeaforementioned
multi-generatorGANs,ourproposedmodelsimultaneouslytrainsasetofgeneratorswiththeobjec-
tive that the mixture of their induced distributions would approximate the data distribution, whilst
encouragingthemtospecializeindifferentdatamodes.Theresultisanoveladversarialarchitecture
formulatedasaminimaxgameamongthreeparties: aclassifier,adiscriminator,andasetofgener-
ators. Generatorscreatesamplesthatareintendedtocomefromthesamedistributionasthetraining
data,whilstthediscriminatordetermineswhethersamplesaretruedataorgeneratedbygenerators,
and the classifier specifies which generator a sample comes from. We term our proposed model
asMixtureGenerativeAdversarialNets(MGAN).Weprovideanalysisthatourmodelisoptimized
towardsminimizingtheJensen-ShannonDivergence(JSD)betweenthemixtureofdistributionsin-
ducedbythegeneratorsandthedatadistributionwhilemaximizingtheJSDamonggenerators.
Empirically, our proposed model can be trained efficiently by utilizing parameter sharing among
generators, and between the classifier and the discriminator. In addition, simultaneously training
manygeneratorswhileenforcingJSDamonggeneratorshelpseachofthemfocusonsomemodes
of the data space and learn better. Trained on CIFAR-10, each generator learned to specialize in
generating samples from a different class such as horse, car, ship, dog, bird or airplane. Overall,
themodelstrainedontheCIFAR-10,STL-10andImageNetdatasetssuccessfullygenerateddiverse,
recognizable objects and achieved state-of-the-art Inception scores (Salimans et al., 2016). The
model trained on the CIFAR-10 even outperformed GANs trained in a semi-supervised fashion
(Salimansetal.,2016;Odenaetal.,2016).
In short, our main contributions are: (i) a novel adversarial model to efficiently train a mixture
of generators while enforcing the JSD among the generators; (ii) a theoretical analysis that our
objectivefunctionisoptimizedtowardsminimizingtheJSDbetweenthemixtureofallgenerators’
distributions and the real data distribution, while maximizing the JSD among generators; and (iii)
a comprehensive evaluation on the performance of our method on both synthetic and real-world
large-scaledatasetsofdiversenaturalscenes.
2
PublishedasaconferencepaperatICLR2018
(cid:1)(cid:2)(cid:3)(cid:4)(cid:5)(cid:2)(cid:6)(cid:2)(cid:7)(cid:8)(cid:9)(cid:10)(cid:5)(cid:11)
(cid:1) (cid:2)
(cid:1)
(cid:1)(cid:2)(cid:3)(cid:4)(cid:2)(cid:5)(cid:6)(cid:7)(cid:2)(cid:3)(cid:8)(cid:9)(cid:10)(cid:11)(cid:4)(cid:12)(cid:11)(cid:11)(cid:5)
(cid:1)(cid:13)(cid:5)(cid:1)(cid:9)(cid:2)(cid:1)(cid:3)(cid:4)(cid:5)
(cid:1) (cid:1)(cid:2)(cid:3)(cid:4)(cid:5)(cid:6)(cid:7)(cid:8)(cid:7)(cid:9)(cid:3)(cid:10)(cid:3)(cid:8)(cid:11) (cid:1)
(cid:1) (cid:3) (cid:1)(cid:2)(cid:3)(cid:4)(cid:5)(cid:6)(cid:7)(cid:8)(cid:7)(cid:9)(cid:3)(cid:10)(cid:3)(cid:8)(cid:11)
M (cid:1)(cid:2)(cid:3)(cid:4)(cid:5)(cid:6)(cid:7)(cid:8)(cid:7)(cid:9)(cid:3)(cid:10)(cid:3)(cid:8)(cid:11) M (cid:1) (cid:12)(cid:8)(cid:2)(cid:14)(cid:8)(cid:9)(cid:6)(cid:11)(cid:5)(cid:11)(cid:15)(cid:13)(cid:4)(cid:16)(cid:15)(cid:9)
(cid:12)(cid:13)(cid:3)(cid:9)(cid:7)(cid:3)(cid:11)(cid:1)(cid:17)
(cid:1)
(cid:4)(cid:12)(cid:8)(cid:3)(cid:3)(cid:2)(cid:13)(cid:2)(cid:14)(cid:5)(cid:11)
(cid:4) (cid:5)(cid:6)(cid:7)(cid:6)
Figure1: MGAN’sarchitecturewithKgenerators,abinarydiscriminator,amulti-classclassifier.
2 GENERATIVE ADVERSARIAL NETS
GiventhediscriminatorDandgeneratorG,bothparameterizedvianeuralnetworks,trainingGAN
canbeformulatedasthefollowingminimaxobjectivefunction:
minmaxE [logD(x)]+E [log(1−D(G(z)))] (1)
G D x∼Pdata(x) z∼Pz
wherexisdrawnfromdatadistributionP ,zisdrawnfromapriordistributionP .Themapping
data z
G(z)inducesageneratordistributionP indataspace. GANalternativelyoptimizesDandG
model
using stochastic gradient-based learning. As a result, the optimization order in 1 can be reversed,
causingtheminimaxformulationtobecomemaximin.Gisthereforeincentivizedtomapeveryztoa
singlexthatismostlikelytobeclassifiedastruedata,leadingtomodecollapsingproblem.Another
commonlyassertedcauseofgeneratinglessdiversesamplesinGANisthat,attheoptimalpointof
D, minimizing G is equivalent to minimizing the JSD between the data and model distributions,
which has been empirically proven to prefer to generate samples around only a few modes whilst
ignoringothermodes(Husza´r,2015;Theisetal.,2015).
3 PROPOSED MIXTURE GANS
Wenowpresentourmaincontributionofanovelapproachthatcaneffectivelytacklemodecollapse
inGAN.Ourideaistouseamixtureofmanydistributionsratherthanasingleoneasinthestandard
GAN,toapproximatethedatadistribution,andsimultaneouslyweenlargethedivergenceofthose
distributionssothattheycoverdifferentdatamodes.
Tothisend,ananalogytoagameamongKgeneratorsG ,adiscriminatorD andaclassifierC
1:K
can be formulated. Each generator G maps z to x = G (z), thus inducing a single distribution
k k
P ;andKgeneratorsaltogetherinduceamixtureoverKdistributions,namelyP inthedata
Gk model
space. An index u is drawn from a multinomial distribution Mult(π) where π = [π ,π ,...,π ]
1 2 K
is the coefficients of the mixture; and then the sample G (z) is used as the output. Here, we use
u
apredefinedπ andfixitinsteadoflearning. ThediscriminatorD aimstodistinguishbetweenthis
sample and the training samples. The classifier C performs multi-class classification to classify
sampleslabeledbytheindicesoftheircorrespondinggenerators. Wetermthiswholeprocessand
ourmodeltheMixtureGenerativeAdversarialNets(MGAN).
Fig.1illustratesthegeneralarchitectureofourproposedMGAN,whereallcomponentsareparam-
eterizedbyneuralnetworks.G (s)tietheirparameterstogetherexcepttheinputlayer,whilstCand
k
D share parameters except the output layer. This parameter sharing scheme enables the networks
toleveragetheircommoninformationsuchasfeaturesatlow-levellayersthatareclosetothedata
layer,hencehelpstotrainmodeleffectively.Inaddition,italsominimizesthenumberofparameters
andaddsminimalcomplexitytothestandardGAN,thusthewholeprocessisstillveryefficient.
Moreformally,D,C andG nowplaythefollowingmulti-playerminimaxoptimizationgame:
1:K
min maxJ (G ,C,D)=E [logD(x)]+E [log(1−D(x))]
G1:K,C D 1:K x∼Pdata x∼Pmodel
(cid:40) K (cid:41)
(cid:88)
−β π E [logC (x)] (2)
k x∼PGk k
k=1
3
PublishedasaconferencepaperatICLR2018
whereC (x)istheprobabilitythatxisgeneratedbyG andβ >0isthediversityhyper-parameter.
k k
Thefirsttwotermsshowtheinteractionbetweengeneratorsandthediscriminatorasinthestandard
GAN.Thelasttermshouldberecognizedasastandardsoftmaxlossforamulti-classificationset-
ting,whichaimstomaximizetheentropyfortheclassifier. Thisrepresentstheinteractionbetween
generatorsandtheclassifier,whichencourageseachgeneratortoproducedataseparablefromthose
producedbyothergenerators. Thestrengthofthisinteractioniscontrolledbyβ. SimilartoGAN,
our proposed network can be trained by alternatively updating D, C and G . We refer to Ap-
1:K
pendixAforthepseudo-codeandalgorithmsforparameterlearningforourproposedMGAN.
3.1 THEORETICALANALYSIS
Assuming all C, D and G have enough capacity, we show below that at the equilibrium point
1:K
of the minimax problem in Eq. (2), the JSD between the mixture induced by G and the data
1:K
distributionisminimal,i.e. p = p ,andtheJSDamongKgeneratorsismaximal,i.e. two
data model
arbitrarygeneratorsalmostneverproducethesamedata. Inwhatfollowswepresentourmathemat-
icalstatementandthesketchoftheirproofs. WerefertoAppendixBforfullderivations.
Proposition 1. For fixed generators G , G , ..., G and their mixture weights π ,π ,...,π , the
1 2 K 1 2 K
optimalsolutionC∗ =C∗ andD∗forJ (G ,C,D)inEq.(2)are:
1:K 1:K
π p (x) p (x)
C∗(x)= k Gk and D∗(x)= data
k (cid:80)Kj=1πjpGj(x) pdata(x)+pmodel(x)
Proof. ItcanbeseenthatthesolutionC∗ isageneralcaseofD∗ whenD classifiessamplesfrom
k
two distributions with equal weight of 1/2. We refer the proofs for D∗ to Prop. 1 in (Goodfellow
etal.,2014),andourproofforC∗toAppendixBinthismanuscript.
k
Based on Prop. 1, we further show that at the equilibrium point of the minimax problem in
Eq.(2), theoptimalgeneratorG∗ = [G∗,...,G∗]inducesthegenerateddistributionp∗ (x) =
1 K model
(cid:80)Kk=1πkpG∗k(x)whichisasclosestaspossibletothetruedatadistributionpdata(x)whilemain-
tainingthemixturecomponentspG∗(x)(s)asfurthestaspossibletoavoidthemodecollapse.
k
Theorem2. AttheequilibriumpointoftheminimaxprobleminEq.(2), theoptimalG∗,D∗, and
C∗satisfy
G∗ =argmin (2·JSD(P (cid:107)P )−β·JSD (P ,P ,...,P )) (3)
data model π G1 G2 GK
G
C∗(x)= πkpG∗k(x) and D∗(x)= pdata(x)
k (cid:80)Kj=1πjpG∗j (x) pdata(x)+pmodel(x)
Proof. Substituting C∗ and D∗ into Eq. (2), we reformulate the objective function for G as
1:K 1:K
follows:
(cid:20) (cid:21) (cid:20) (cid:21)
p (x) p (x)
L(G )=E log data +E log model
1:K x∼Pdata p (x)+p (x) x∼Pmodel p (x)+p (x)
data model data model
−β(cid:40)(cid:88)K π E (cid:34)log πkpGk(x) (cid:35)(cid:41)
k x∼PGk (cid:80)K π p (x)
k=1 j=1 j Gj
=2·JSD(P (cid:107)P )−log4−β(cid:40)(cid:88)K π E (cid:34)log pGk(x) (cid:35)(cid:41)−β(cid:88)K π logπ
data model k x∼PGk (cid:80)K π p (x) k k
k=1 j=1 j Gj k=1
K
(cid:88)
=2·JSD(P (cid:107)P )−β·JSD (P ,P ,...,P )−log4−β π logπ (4)
data model π G1 G2 GK k k
k=1
SincethelasttwotermsinEq.(4)areconstant,thatconcludesourproof.
This theorem shows that progressing towards the equilibrium is equivalently to minimizing
JSD(P (cid:107)P ) while maximizing JSD (P ,P ,...,P ). In the next theorem, we fur-
data model π G1 G2 GK
ther clarify the equilibrium point for the specific case wherein the data distribution has the form
4
PublishedasaconferencepaperatICLR2018
p (x) = (cid:80)K π q (x) where the mixture components q (x)(s) are well-separated in the
data k=1 k k k
sense that E [q (x)] = 0 for j (cid:54)= k, i.e., for almost everywhere x, if q (x) > 0 then
x∼Qk j k
q (x)=0, ∀j (cid:54)=k.
j
Theorem 3. If the data distribution has the form: p (x) = (cid:80)K π q (x) where the mix-
data k=1 k k
ture components q (x)(s) are well-separated, the minimax problem in Eq. (2) or the optimization
k
probleminEq. (3)hasthefollowingsolution:
K
(cid:88)
p (x)=q (x), ∀k =1,...,K and p (x)= π q (x)=p (x)
G∗ k model k k data
k
k=1
, and the corresponding objective value of the optimization problem in Eq. (3) is −βH(π) =
−β(cid:80)K π log 1 ,whereH(π)istheShannonentropy.
k=1 k πk
Proof. PleaserefertoourproofinAppendixBofthismanuscript.
Thm. 3 explicitly offers the optimal solution for the specific case wherein the real data are gen-
erated from a mixture distribution whose components are well-separated. This further reveals
that if the mixture components are well-separated, by setting the number of generators as the
number of mixtures in data and maximizing the divergence between the generated components
p (x)(s), we can exactly recover the mixture components q (x)(s) using the generated com-
Gk k
ponents p (x)(s), hence strongly supporting our motivation when developing MGAN. In prac-
Gk
tice, C, D, and G are parameterized by neural networks and are optimized in the parameter
1:K
space rather than in the function space. As all generators G share the same objective func-
1:K
tion, we can efficiently update their weights using the same backpropagation passes. Empirically,
we set the parameter π = 1,∀k ∈ {1,...,K}, which further minimizes the objective value
k K
−βH(π) = −β(cid:80)K π log 1 w.r.t π in Thm. 3. To simplify the computational graph, we as-
k=1 k πk
sumethateachgeneratorissampledthesamenumberoftimesineachminibatch. Inaddition, we
adoptthenon-saturatingheuristicproposedin(Goodfellowetal.,2014)totrainG bymaximizing
1:K
logD(G (z))insteadofminimizinglogD(1−G (z)).
k k
4 RELATED WORK
Recent attempts to address the mode collapse by modifying the discriminator include minibatch
discrimination (Salimans et al., 2016), Unrolled GAN (Metz et al., 2016) and Denoising Feature
Matching (DFM) (Warde-Farley & Bengio, 2016). The idea of minibatch discrimination is to al-
lowthediscriminatortodetectsamplesthatarenoticeablysimilartoothergeneratedsamples. Al-
thoughthismethod cangeneratevisuallyappealingsamples, itis computationally expensive, thus
normally used in the last hidden layer of discriminator. Unrolled GAN improves the learning by
unrollingcomputationalgraphtoincludeadditionaloptimizationstepsofthediscriminator. Itcould
effectively reduce the mode collapsing problem, but the unrolling step is expensive, rendering it
unscalableuptolarge-scaledatasets. DFMaugmentstheobjectivefunctionofgeneratorwithone
of a Denoising AutoEncoder (DAE) that minimizes the reconstruction error of activations at the
penultimate layer of the discriminator. The idea is that gradient signals from DAE can guide the
generatortowardsproducingsampleswhoseactivationsareclosetothemanifoldofrealdataactiva-
tions. DFMissurprisinglyeffectiveatavoidingmodecollapse,buttheinvolvementofadeepDAE
addsconsiderablecomputationalcosttothemodel.
Analternativeapproachistotrainadditionaldiscriminators.D2GAN(Nguyenetal.,2017)employs
twodiscriminatorstominimizebothKullback-Leibler(KL)andreverseKLdivergences,thusplac-
ingafairdistributionacrossthedatamodes. Thismethodcanavoidthemodecollapsingproblem
toacertainextent,butstillcouldnotoutperformDFM.Anotherworkusesmanydiscriminatorsto
boostthelearningofgenerator(Durugkaretal.,2016). Theauthorsstatethatthismethodisrobust
tomodecollapse,butdidnotprovideexperimentalresultstosupportthatclaim.
Another direction is to train multiple generators. The so-called MIX+GAN (Arora et al., 2017) is
relatedtoourmodelintheuseofmixturebuttheideaisverydifferent. Basedonmin-maxtheorem
(Neumann, 1928), the MIX+GAN trains a mixture of multiple generators and discriminators with
5
PublishedasaconferencepaperatICLR2018
differentparameterstoplaymixedstrategiesinamin-maxgame. Thetotalrewardofthisgameis
computed by weighted averaging rewards over all pairs of generator and discriminator. The lack
ofparametersharingrendersthismethodcomputationallyexpensivetotrain. Moreover,thereisno
mechanismtoenforcethedivergenceamonggeneratorsasinours.
Some attempts have been made to train a mixture of GANs in a similar spirit with boosting algo-
rithms. Wang et al. (2016) propose an additive procedure to incrementally train new GANs on a
subset of the training data that are badly modeled by previous generators. As the discriminator is
expected to classify samples from this subset as real with high confidence, i.e. D(x) is high, the
subset can be chosen to include x where D(x) is larger than a predefined threshold. Tolstikhin
etal.(2017),however,showthatthisheuristicfailstoaddressthemodecollapsingproblem. Thus
theyproposeAdaGANtointroducearobustreweighingschemetopreparetrainingdataforthenext
GAN.AdaGANandboosting-inspiredGANsingeneralarebasedontheassumptionthatasingle-
generatorGANcanlearntogenerateimpressiveimagesofsomemodessuchasdogsorcatsbutfails
tocoverothermodessuchasgiraffe. Therefore,removingimagesofdogsorcatsfromthetraining
data and train a next GAN can create a better mixture. This assumption is not true in practice as
current single-generator GANs trained on diverse data sets such as ImageNet (Russakovsky et al.,
2015)tendtogenerateimagesofunrecognizableobjects.
ThemostcloselyrelatedtooursisMAD-GAN(Ghoshetal.,2017)whichtrainsmanygenerators
andusesamulti-classclassifierasthediscriminator. Inthiswork,twostrategiesareproposedtoad-
dressthemodecollapse:(i)augmentinggenerator’sobjectivefunctionwithauser-definedsimilarity
basedfunctiontoencouragedifferentgeneratorstogeneratediversesamples,and(ii)modifyingdis-
criminator’sobjectivefunctionstopushdifferentgeneratorstowardsdifferentidentifiablemodesby
separating samples of each generator. Our approach is different in that, rather than modifying the
discriminator,weuseanadditionalclassifierthatdiscriminatessamplesproducedbyeachgenerator
from those by others under multi-class classification setting. This nicely results in an optimiza-
tionproblemthatmaximizestheJSDamonggenerators,thusnaturallyenforcingthemtogenerate
diversesamplesandeffectivelyavoidingmodecollapse.
5 EXPERIMENTS
Inthissection, weconductexperimentsonbothsyntheticdataandreal-worldlarge-scaledatasets.
The aim of using synthetic data is to visualize, examine and evaluate the learning behaviors of
our proposed MGAN, whilst using real-world datasets to quantitatively demonstrate its efficacy
and scalability of addressing the mode collapse in a much larger and wider data space. For fair
comparison, weuse experimental settings that areidentical to previous work, and hence we quote
theresultsfromthelateststate-of-the-artGAN-basedmodelstocomparewithours.
WeuseTensorFlow(Abadietal.,2016)toimplementourmodel, andthesourcecodeisavailable
at: https://github.com/qhoangdl/MGAN.Forallexperiments,weuse: (i)sharedparametersamong
generators in all layers except for the weights from the input to the first hidden layer; (ii) shared
parametersbetweendiscriminatorandclassifierinalllayersexceptfortheweightsfromthepenulti-
matelayertotheoutput;(iii)Adamoptimizer(Kingma&Ba,2014)withlearningrateof0.0002and
thefirst-ordermomentumof0.5; (iv)minibatchsizeof64samplesfortrainingdiscriminators; (v)
ReLUactivations(Nair&Hinton,2010)forgenerators;(vi)LeakyReLU(Maasetal.,2013)with
slope of 0.2 for discriminator and classifier; and (vii) weights randomly initialized from Gaussian
distributionN(0,0.02I)andzerobiases. WerefertoAppendixCfordetailedmodelarchitectures
andadditionalexperimentalresults.
5.1 SYNTHETICDATA
Inthefirstexperiment,following(Nguyenetal.,2017)wereusetheexperimentaldesignproposed
in (Metz et al., 2016) to investigate how well our MGAN can explore and capture multiple data
modes. Thetrainingdataissampledfroma2Dmixtureof8isotropicGaussiandistributionswitha
covariancematrixof0.02I andmeansarrangedinacircleofzerocentroidandradiusof2.0. Our
purposeofusingsuchsmallvarianceistocreatelowdensityregionsandseparatethemodes.
We employ 8 generators, each with a simple architecture of an input layer with 256 noise units
drawn from isotropic multivariate Gaussian distribution N (0,I), and two fully connected hidden
6
PublishedasaconferencepaperatICLR2018
����
�P�P�H�W�U�L�F��.�/��G�L�Y�������� �*�8�'�0�Q��$�*�U�*�1�R�$�$ �O�1�O�H�1�G ��*�$�1 GAN
�6�\
��� AN
G
d
e
��. ���. ���. ���. ���. ll
�6�W�H�S ro
n
(a)SymmetricKLdivergence. U
N
A
��� 2G
�P�D�W�H��� D
�:�D�V�V�H�U�V�W�H�L�Q��H�V�W�L������ �*�8�'�0�Q��$�*�U�*�1�R�$�$�O�1�O�H�1�G��*�$�1 AN
��� MG
��� ��. ���. ���. ���. ���.
�6�W�H�S step 5K step 10K step 15K step 20K step 25K
(b)Wassersteindistance. (c)Evolutionofdata(inblue)generatedbyGAN,UnrolledGAN,D2GAN
andourMGANfromthetoprowtothebottom,respectively.Datasampled
fromthetruemixtureof8Gaussiansarered.
Figure2: ThecomparisonofourMGANandGAN’svariantson2Dsyntheticdataset.
layers with 128 ReLU units each. For the discriminator and classifier, one hidden layer with 128
ReLUunitsisused. Thediversityhyperparameterβ issetto0.125.
Fig.2cshowstheevolutionof512samplesgeneratedbyourmodelandbaselinesthroughtime. It
canbeseenthattheregularGANgeneratesdatacollapsingintoasinglemodehoveringaroundthe
validmodesofdatadistribution,thusreflectingthemodecollapseinGANasexpected. Atthesame
time, UnrolledGAN (Metz et al., 2016), D2GAN (Nguyen et al., 2017) and our MGAN distribute
data around all 8 mixture components, and hence demonstrating the abilities to successfully learn
multimodaldatainthiscase. Ourproposedmodel, however, convergesmuchfasterthantheother
two since it successfully explores and neatly covers all modes at the early step 15K, whilst two
baselinesproducesamplescyclingaroundtillthelaststeps. Attheend, ourMGANcapturesdata
modes more precisely than UnrolledGAN and D2GAN since, in each mode, the UnrolledGAN
generates data that concentrate only on several points around the mode’s centroid, thus seems to
producefewersamplesthanourswhosesamplesfairlyspreadouttheentiremode, butnotexceed
theboundarywhilsttheD2GANstillgeneratesmanypointsscatteredbetweentwoadjacentmodes.
Nextwefurtherquantitativelycomparethequalityofgenerateddata. Sinceweknowthetruedis-
tributionP inthiscase,weemploytwomeasures,namelyWassersteindistanceandsymmetric
data
Kullback-Leibler (KL) divergence, which is the average of KL and reverse KL. These measures
computethedistancebetweenthenormalizedhistogramsof10,000pointsgeneratedfromthemodel
totrueP . Figs.2aand2bagainclearlydemonstratethesuperiorityofourapproachoverGAN,
data
UnrolledGANandD2GANw.r.tbothdistances(lowerisbetter);notablytheWassersteindistances
from ours and D2GAN’s to the true distribution almost reduce to zero, and at the same time, our
symmetric KL metric is significantly better than that of D2GAN. These figures also show the sta-
bilityofourMGAN(blackcurves)andD2GAN(redcurves)duringtrainingastheyaremuchless
fluctuatingcomparedwithGAN(greencurves)andUnrolledGAN(bluecurves).
Lastly,weperformexperimentswithdifferentnumbersofgenerators.TheMGANmodelswith2,3,
4and10generatorsallsuccessfullyexplore8modesbutthemodelswithmoregeneratorsgenerate
fewer points scattered between adjacent modes. We also examine the behavior of the diversity
coefficient β by training the 4-generator model with different values of β. Without the JSD force
(β = 0),generatedsamplesclusteraroundonemode. Whenβ = 0.25,theJSDforceisweakand
generateddataclusternear4differentmodes.Whenβ =0.75or1.0,theJSDforceistoostrongand
causesthegeneratorstocollapse,generating4increasinglytightclusters.Whenβ =0.5,generators
successfullycoverallofthe8modes. PleaserefertoAppendixC.1forexperimentaldetails.
7
PublishedasaconferencepaperatICLR2018
5.2 REAL-WORLDDATASETS
Next we train our proposed method on real-world databases from natural scenes to investigate its
performanceandscalabilityonmuchmorechallenginglarge-scaleimagedata.
Datasets. We use 3 widely-adopted datasets: CIFAR-10 (Krizhevsky & Hinton, 2009), STL-10
(Coatesetal.,2011)andImageNet(Russakovskyetal.,2015). CIFAR-10contains50,00032×32
trainingimagesof10classes: airplane,automobile,bird,cat,deer,dog,frog,horse,ship,andtruck.
STL-10, subsampled from ImageNet, is a more diverse dataset than CIFAR-10, containing about
100,00096×96images.ImageNet(2012release)presentsthelargestandmostdiverseconsistingof
over1.2millionimagesfrom1,000classes. Inordertofacilitatefaircomparisonwiththebaselines
in (Warde-Farley & Bengio, 2016; Nguyen et al., 2017), we follow the procedure of (Krizhevsky
etal.,2012)toresizetheSTL-10andImageNetimagesdownto48×48and32×32,respectively.
Evaluationprotocols. Forquantitativeevaluation,weadopttheInceptionscoreproposedin(Sal-
imansetal.,2016),whichcomputesexp(E [KL(p(y|x)(cid:107)p(y))])wherep(y|x)istheconditional
x
labeldistributionfortheimagexestimatedbythereferenceInceptionmodel(Szegedyetal.,2015).
Thismetricrewardsgoodandvariedsamplesandisfoundtobewell-correlatedwithhumanjudg-
ment (Salimans et al., 2016). We use the code provided in (Salimans et al., 2016) to compute the
Inceptionscoresfor10partitionsof50,000randomlygeneratedsamples.Forqualitativedemonstra-
tionofimagequalityobtainedbyourproposedmodel,weshowsamplesgeneratedbythemixtureas
wellassamplesproducedbyeachgenerator.Samplesarerandomlydrawnratherthancherry-picked.
Modelarchitectures. OurgeneratoranddiscriminatorarchitecturescloselyfollowtheDCGAN’s
design(Radfordetal.,2015).Theonlydifferenceisweapplybatchnormalization(Ioffe&Szegedy,
2015)toalllayersinthenetworksexceptfortheoutputlayer. Regardingtheclassifier, weempir-
icallyfindthatourproposedMGANachievesthebestperformance(i.e.,fastconvergencerateand
highinceptionscore)whentheclassifiersharesparametersofalllayerswiththediscriminatorex-
ceptfortheoutputlayer. Thereasonisthatthisparametersharingschemewouldallowtheclassifier
anddiscriminatortoleveragetheircommonfeaturesandrepresentationslearnedateverylayer,thus
helpstoimproveandspeedupthetrainingprogress. Whentheparametersarenottied, themodel
learnsslowlyandeventuallyyieldslowerperformance.
During training we observe that the percentage of active neurons chronically declined (see Ap-
pendixC.2). Onepossiblecauseisthatthebatchnormalizationcenter(offset)isgraduallyshifted
to the negative range, thus deactivating up to 45% of ReLU units of the generator networks. Our
ad-hocsolutionforthisproblemistofixtheoffsetatzeroforalllayersinthegeneratornetworks.
Therationaleisthatforeachfeaturemap,theReLUgateswillopenforabout50%highestinputsin
aminibatchacrossalllocationsandgenerators,andclosefortherest.
Wealsoexperimentwithotheractivationfunctionsofgeneratornetworks.FirstweuseLeakyReLU
and obtain similar results with using ReLU. Then we use MaxOut units (Goodfellow et al., 2013)
and achieves good Inception scores but generates unrecognizable samples. Finally, we try SeLU
(Klambaueretal.,2017)butfailtotrainourmodel.
Hyperparameters. Threekeyhyperparametersofourmodelare: numberofgeneratorsK,coef-
ficient β controlling the diversity and the minibatch size. We use a minibatch size of [128/K] for
each generator, so that the total number of samples for training all generators is about 128. We
trainmodelswith4generatorsand10generatorscorrespondingwithminibatchsizesof32and12
each,andfindthatmodelswith10generatorsperformsbetter. ForImageNet,wetryanadditional
settingwith32generatorsandaminibatchsizeof4foreach. Thebatchof4samplesistoosmall
forupdatingsufficientstatisticsofabatch-normlayer, thuswedropbatch-normintheinputlayer
of each generator. This 32-generator model, however, does not obtain considerably better results
thanthe10-generatorone. Thereforeinwhatfollowsweonlyreporttheresultsofmodelswith10
generators. Forthediversitycoefficientβ,weobservenosignificantdifferenceinInceptionscores
whenvaryingthevalueofβ butthequalityofgeneratedimagesdeclineswhenβ istoolowortoo
high. Generatedsamplesbyeachgeneratorvarymorewhenβ islow,andvarylessbutbecomeless
realisticwhenβ ishigh. Wefindareasonablerangeforβ tobe(0.01,1.0),andfinallysetto0.01
forCIFAR-10,0.1forImageNetand1.0forSTL-10.
8
PublishedasaconferencepaperatICLR2018
Inception results. We now report the Inception scores obtained by our MGAN and baselines in
Tab.1. Itisworthytonotethatonlymodelstrainedinacompletelyunsupervisedmannerwithout
label information are included for fair comparison; and DCGAN’s and D2GAN’s results on STL-
10 are available only for the models trained on 32×32 resolution. Overall, our proposed model
outperformsthebaselinesbylargemarginsandachievesstate-of-the-artperformanceonalldatasets.
Moreover, we would highlight that our MGAN obtains a score of 8.33 on CIFAR-10 that is even
better than those of models trained with labels such as 8.09 of Improved GAN (Salimans et al.,
2016) and 8.25 of AC-GAN (Odena et al., 2016). In addition, we train our model on the original
96×96 resolution of STL-10 and achieve a score of 9.79±0.08. This suggests the MGAN can be
successfullytrainedonhigherresolutionimagesandachievethehigherInceptionscore.
Table1: Inceptionscoresondifferentdatasets. Allmodelsaretrainedinanunsupervisedmanner.
“–”denotesunavailableresult.
Model CIFAR-10 STL-10 ImageNet
Realdata 11.24±0.16 26.08±0.26 25.78±0.47
WGAN(Arjovskyetal.,2017) 3.82±0.06 – –
MIX+WGAN(Aroraetal.,2017) 4.04±0.07 – –
Improved-GAN(Salimansetal.,2016) 4.36±0.04 – –
ALI(Dumoulinetal.,2016) 5.34±0.05 – –
BEGAN(Berthelotetal.,2017) 5.62 – –
MAGAN(Wangetal.,2017) 5.67 – –
GMAN(Durugkaretal.,2016) 6.00±0.19 – –
DCGAN(Radfordetal.,2015) 6.40±0.05 7.54 7.89
DFM(Warde-Farley&Bengio,2016) 7.72±0.13 8.51±0.13 9.18±0.13
D2GAN(Nguyenetal.,2017) 7.15±0.07 7.98 8.25
MGAN 8.33±0.10 9.22±0.11 9.32±0.10
Fre´chet Inception Distance results. One disadvantage of the Inception score is that it does not
comparethestatisticsofrealworldsamplesandthoseofsyntheticexamples. Therefore,wefurther
evaluateMGANusingtheFre´chetInceptionDistance(FID)proposedin(Heuseletal.,2017). Let
pandqbethedistributionsoftherepresentationsobtainedbyprojectingrealandsyntheticsamples
tothelasthiddenlayerinInceptionmodel(Szegedyetal.,2015). Assumingthatpandq areboth
multivariateGaussiandistributions,FIDmeasurestheFre´chetdistance(Dowson&Landau,1982),
whichisalsothe2-Wassersteindistance,betweenthetwodistributions. Tab. 2comparestheFIDs
obtainedbyMGANwithbaselinescollectedin(Heuseletal.,2017).ItisnoteworthythatlowerFID
isbetter,andthatWGAN-GPandWGAN-GP+TTURusestheResNetarchitecturewhileMGAN
employs the DCGAN architecture. In terms of FID, MGAN is roughly 28% better than DCGAN
andDCGAN+TTUR,9%betterthanWGAN-GPand8%weakerthanWGAN-GP+TTUR.This
resultfurtherprovesthatMGANhelpsaddressthemodecollapsingproblem.
Table2: FIDs(lowerisbetter)onCIFAR-10.
Model FID
DCGAN(Radfordetal.,2015) 37.7
DCGAN+TTUR(Heuseletal.,2017) 36.9
WGAN-GP(Gulrajanietal.,2017) 29.3
WGAN-GP+TTUR(Heuseletal.,2017) 24.8
MGAN 26.7
Imagegeneration. Nextwepresentsamplesrandomlygeneratedbyourproposedmodeltrained
onthe3datasetsforqualitativeassessment. Fig.3ashowsCIFAR-1032×32imagescontaininga
widerangeofobjectsinsuchasairplanes,cars,trucks,ships,birds,horsesordogs. Similarly,STL-
1048×48generatedimagesinFig.3bincludecars,ships,airplanesandmanytypesofanimals,but
withwiderrangeofdifferentthemessuchassky,underwater,mountainandforest.Imagesgenerated
forImageNet32×32arediversewithsomerecognizableobjectssuchaslady,oldman,birds,human
eye,livingroom,hat,slippers,tonameafew. Fig.4ashowsseveralcherry-pickedSTL-1096×96
9
PublishedasaconferencepaperatICLR2018
images,whichdemonstratethattheMGANiscapableofgeneratingvisuallyappealingimageswith
complicateddetails.However,manysamplesarestillincompleteandunrealisticasshowninFig.4b,
leavingplentyofroomforimprovement.
(a)CIFAR-1032×32. (b)STL-1048×48. (c)ImageNet32×32.
Figure3: ImagesgeneratedbyourproposedMGANtrainedonnaturalimagedatasets. Duetothe
spacelimit,pleaserefertotheappendixforlargerplots.
(a)Cherry-pickedsamples. (b)Incomplete,unrealisticsamples.
Figure4: ImagesgeneratedbyourMGANtrainedontheoriginal96×96STL10dataset.
Finally,weinvestigatesamplesgeneratedbyeachgeneratoraswellastheevolutionofthesesamples
throughnumbersoftrainingepochs. Fig.5showsimagesgeneratedbyeachofthe10generatorsin
ourMGANtrainedonCIFAR-10atepoch20,50,and250oftraining. Samplesineachrowcorre-
spondtoadifferentgenerator. Generatorsstarttospecializeingeneratingdifferenttypesofobjects
as early as epoch 20 and become more and more consistent: generator 2 and 3 in flying objects
(birds and airplanes), generator 4 in full pictures of cats and dogs, generator 5 in portraits of cats
anddogs,generator8inships,generator9incarandtrucks,andgenerator10inhorses. Generator
6seemstogenerateimagesoffrogoranimalsinabush. Generator7,however,collapsesinepoch
250. One possible explanation for this behavior is that images of different object classes tend to
havedifferentthemes. Lastly,Wangetal.(2016)noticedoneofthecausesfornon-convergencein
GANs is that the generators and discriminators constantly vary; the generators at two consecutive
epochsoftraininggeneratesignificantlydifferentimages. Thisexperimentdemonstratestheeffect
oftheJSDforceinpreventinggeneratorsfrommovingaroundthedataspace.
10