Immersive Video Technologies

686 Pages·2022·27.097 MB·English

by  Giuseppe Valenzise, Martin Alain, Emin Zerman, Cagri Ozcinar

IMMERSIVE VIDEO
TECHNOLOGIES
This page intentionally left blank
IMMERSIVE VIDEO
TECHNOLOGIES
Editedby
GIUSEPPEVALENZISE
UniversitéParis-Saclay,CNRS,CentraleSupélec
Laboratoiredessignauxetsystèmes
Gif-sur-Yvette,France
MARTINALAIN
HuaweiIrelandResearchCentre
Dublin,Ireland
EMINZERMAN
STCResearchCenter
MidSwedenUniversity
Sundsvall,Sweden
CAGRIOZCINAR
SamsungR&DUK
London,UnitedKingdom
AcademicPressisanimprintofElsevier
125LondonWall,LondonEC2Y5AS,UnitedKingdom
525BStreet,Suite1650,SanDiego,CA92101,UnitedStates
50HampshireStreet,5thFloor,Cambridge,MA02139,UnitedStates
TheBoulevard,LangfordLane,Kidlington,OxfordOX51GB,UnitedKingdom
Copyright©2023ElsevierInc.Allrightsreserved.
MATLAB®isatrademarkofTheMathWorks,Inc.andisusedwithpermission.
TheMathWorksdoesnotwarranttheaccuracyofthetextorexercisesinthisbook.
Thisbook’suseordiscussionofMATLAB®softwareorrelatedproductsdoesnotconstituteendorsementor
sponsorshipbyTheMathWorksofaparticularpedagogicalapproachorparticularuseoftheMATLAB®software.
Nopartofthispublicationmaybereproducedortransmittedinanyformorbyanymeans,electronicor
mechanical,includingphotocopying,recording,oranyinformationstorageandretrievalsystem,without
permissioninwritingfromthepublisher.Detailsonhowtoseekpermission,furtherinformationaboutthe
Publisher’spermissionspoliciesandourarrangementswithorganizationssuchastheCopyrightClearanceCenter
andtheCopyrightLicensingAgency,canbefoundatourwebsite:www.elsevier.com/permissions.
ThisbookandtheindividualcontributionscontainedinitareprotectedundercopyrightbythePublisher(other
thanasmaybenotedherein).
Notices
Knowledgeandbestpracticeinthisfieldareconstantlychanging.Asnewresearchandexperiencebroadenour
understanding,changesinresearchmethods,professionalpractices,ormedicaltreatmentmaybecomenecessary.
Practitionersandresearchersmustalwaysrelyontheirownexperienceandknowledgeinevaluatingandusingany
information,methods,compounds,orexperimentsdescribedherein.Inusingsuchinformationormethodsthey
shouldbemindfuloftheirownsafetyandthesafetyofothers,includingpartiesforwhomtheyhaveaprofessional
responsibility.
Tothefullestextentofthelaw,neitherthePublishernortheauthors,contributors,oreditors,assumeanyliability
foranyinjuryand/ordamagetopersonsorpropertyasamatterofproductsliability,negligenceorotherwise,or
fromanyuseoroperationofanymethods,products,instructions,orideascontainedinthematerialherein.
ISBN:978-0-323-91755-1
ForinformationonallAcademicPresspublications
visitourwebsiteathttps://www.elsevier.com/books-and-journals
Publisher:MaraConner
AcquisitionsEditor:TimPitts
EditorialProjectManager:HowiM.DeRamos
ProductionProjectManager:PremKumarKaliamoorthi
CoverDesigner:VickyPearsonEsser
TypesetbyVTeX
Contents
Listofcontributors xi
Preface xvii
PART1 Foundations
1. Introductiontoimmersivevideotechnologies 3
MartinAlain,EminZerman,CagriOzcinar,andGiuseppeValenzise
1.1. Introduction 3
1.2. Whatisimmersion? 4
1.3. Immersivevideo 8
1.4. Challengesintheprocessingandcontentdeliverypipeline 13
1.5. Discussionandperspectives 20
References 21
PART2 Omnidirectionalvideo
2. Acquisition,representation,andrenderingofomnidirectionalvideos 27
ThomasMaugey
2.1. Introduction 27
2.2. Acquisitionandprojectionmodel 29
2.3. Sphererepresentation 36
2.4. Rendering 43
2.5. Towardsmulti-viewacquisition 44
References 45
3. Streaminganduserbehaviorinomnidirectionalvideos 49
SilviaRossi,AlanGuedes,andLauraToni
3.1. Introduction 49
3.2. Streamingpipeline:evolutiontowardsODV 53
3.3. System-centricstreaming 58
3.4. TheroleoftheuserinODV 62
3.5. User-centricODVstreaming 71
3.6. Conclusionsandperspectives 75
References 77
v
vi Contents
4. Subjectiveandobjectivequalityassessmentforomnidirectionalvideo 85
SimoneCroci,AshutoshSingla,StephanFremerey,AlexanderRaake,andAljosaSmolic
4.1. Introduction 85
4.2. Subjectivequalityassessment 88
4.3. Short-termvideoassessment 91
4.4. Long-termvideoassessment 96
4.5. Tile-basedstreaming 97
4.6. Objectivequalityassessment 98
4.7. VIVA-Q:omnidirectionalvideoqualityassessmentbasedonplanarVoronoipatchesand
visualattention 108
4.8. Conclusion 113
References 114
5. Omnidirectionalvideosaliency 123
Fang-YiChao,FedericaBattisti,PierreLebreton,andAlexanderRaake
5.1. Introduction 123
5.2. Collectinguserdata 126
5.3. Userbehavioranalysis 128
5.4. Modeling 130
5.5. Metrics 144
5.6. Conclusionandoutlook 150
References 152
PART3 Lightfields
Preliminariesonlightfields 161
6. Acquisitionoflightfieldimages&videos 163
ThorstenHerfet,KelvinChelli,andMikaelLePendu
6.1. Plenopticcameras 164
6.2. Lightfieldgantries 168
6.3. Cameraarrays 169
References 171
7. Lightfieldrepresentation 173
ThorstenHerfet,KelvinChelli,andMikaelLePendu
7.1. Spacedomainrepresentation 173
7.2. Frequencydomainrepresentation 182
Contents vii
7.3. Depth-basedrepresentation 188
7.4. Alternativerepresentations 195
References 198
8. Compressionoflightfields 201
MilanStepanov,GiuseppeValenzise,andFrédéricDufaux
8.1. Introduction 201
8.2. Transform-basedmethods 201
8.3. Prediction-basedmethods 207
8.4. JPEGPleno 214
8.5. Deeplearning-basedmethods 218
8.6. Performanceanalysis 221
8.7. Conclusionsandperspectives 222
References 223
9. Lightfieldprocessingformediaapplications 227
JoachimKeinert,LauraFink,FlorianGoldmann,MuhammadShahzebKhanGul,
TobiasJaschke,NicoPrappacher,MatthiasZiegler,MichelBätz,andSiegfriedFößel
9.1. Lightfieldprocessingchainoverview 227
9.2. Imageacquisitionandgeometriccameracalibration 229
9.3. Depthreconstruction 230
9.4. Interactivelightfieldprocessing 232
9.5. Lightfieldrendering 233
9.6. Real-timerenderingingameengines 236
9.7. Neuralrenderingand3Dreconstruction 243
9.8. Conclusionandopenchallenges 255
References 255
10. Qualityevaluationoflightfields 265
AliAkandPatrickLeCallet
10.1. Introduction 265
10.2. CharacteristicsofLF-relateddistortions 266
10.3. Subjectivequalityassessment 270
10.4. Objectivequalityassessment 277
10.5. Conclusion 283
References 284
viii Contents
PART4 Volumetricvideo
11. Volumetricvideo–acquisition,interaction,streamingandrendering 289
PeterEisert,OliverSchreer,IngoFeldmann,CorneliusHellge,andAnnaHilsmann
11.1. Creationofvolumetricvideo 290
11.2. Animatingandinteractingwithvolumetricvideo 299
11.3. Streamingvolumetricvideo 310
11.4. Conclusions 320
References 321
12. MPEGimmersivevideo 327
PatrickGarus,MartaMilovanovic´,JoëlJung,andMarcoCagnazzo
12.1. MIVdescriptionandprofiles 329
12.2. TMIVdescription 331
12.3. Commontestconditions 335
12.4. EvaluationofMIVwithdifferentconfigurationsand2Dcodecs 338
12.5. Conclusion 351
References 354
13. Pointcloudcompression 357
GiuseppeValenzise,MauriceQuach,DongTian,JiahaoPang,andFrédéricDufaux
13.1. Introduction 357
13.2. Basictoolsforpointcloudcompression 359
13.3. MPEGPCCstandardization 364
13.4. Learning-basedtechniques 372
13.5. Conclusionsandperspectives 379
References 380
14. Codingofdynamic3Dmeshes 387
Jean-EudesMarvie,MajaKrivokuc´a,andDanilloGraziosi
14.1. Introduction 387
14.2. Meshfundamentals 390
14.3. Staticmeshes 397
14.4. Constant-connectivitymeshsequences 402
14.5. Variable-connectivitymeshsequences 411
14.6. Conclusionandfuturedirections 416
References 417
Contents ix
15. Volumetricvideostreaming 425
IreneViolaandPabloCesar
15.1. Theoreticalapproachestovolumetricvideostreaming 426
15.2. Volumetricvideostreamingsystems 434
15.3. Conclusion 437
References 438
16. Processingofvolumetricvideo 445
SihengChenandJinZeng
16.1. Introduction 445
16.2. Restoration 446
16.3. Semanticsegmentation 454
16.4. Objectdetection 461
16.5. Chapterataglance 465
References 465
17. Computational3Ddisplays 469
JingyuLiu,FangchengZhong,ClaireMantel,SørenForchhammer,andRafałK.Mantiuk
17.1. Introduction 469
17.2. Perceptualconsiderations 470
17.3. Computational3Ddisplays 475
17.4. Summaryanddiscussion 492
18. Subjectiveandobjectivequalityassessmentforvolumetricvideo 501
EvangelosAlexiou,YanaNehmé,EminZerman,IreneViola,GuillaumeLavoué,AliAk,
AljosaSmolic,PatrickLeCallet,andPabloCesar
18.1. Subjectivequalityassessment 503
18.2. Objectivequalityassessment 522
18.3. Conclusion 539
References 544
PART5 Applications
19. MRinvideoguidedliversurgery 555
RafaelPalomar,RahulPrasannaKumar,CongcongWang,EgidijusPelanis,and
FaouziAlayaCheikh
19.1. Introduction 555
19.2. Medicalimagespre-processing 558
19.3. 3Dlivermodeling 562