Table Of ContentSignals and Communication Technology
Pedro Amado Assunção · Atanas Gotchev
Editors
3D Visual
Content
Creation, Coding
and Delivery
Signals and Communication Technology
More information about this series at http://www.springer.com/series/4748
çã
Pedro Amado Assun o Atanas Gotchev
(cid:129)
Editors
3D Visual Content Creation,
Coding and Delivery
123
Editors
PedroAmado Assunção Atanas Gotchev
Instituto deTelecomunicações Department ofSignal Processing
andPolitécnico deLeiria Tampere University of Technology
Leiria Tampere
Portugal Finland
ISSN 1860-4862 ISSN 1860-4870 (electronic)
Signals andCommunication Technology
ISBN978-3-319-77841-9 ISBN978-3-319-77842-6 (eBook)
https://doi.org/10.1007/978-3-319-77842-6
LibraryofCongressControlNumber:2018939315
©SpringerInternationalPublishingAG,partofSpringerNature2019
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpart
of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,
recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission
orinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilar
methodologynowknownorhereafterdeveloped.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfrom
therelevantprotectivelawsandregulationsandthereforefreeforgeneraluse.
The publisher, the authors and the editors are safe to assume that the advice and information in this
book are believed to be true and accurate at the date of publication. Neither the publisher nor the
authorsortheeditorsgiveawarranty,expressorimplied,withrespecttothematerialcontainedhereinor
for any errors or omissions that may have been made. The publisher remains neutral with regard to
jurisdictionalclaimsinpublishedmapsandinstitutionalaffiliations.
Printedonacid-freepaper
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSwitzerlandAG
Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland
Contents
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Pedro Amado Assunção and Atanas Gotchev
2 Emerging Imaging Technologies: Trends and Challenges. . . . . . . . 5
Marek Domański, Tomasz Grajek, Caroline Conti,
Carl James Debono, Sérgio M. M. de Faria, Peter Kovacs,
Luís F. R. Lucas, Paulo Nunes, Cristian Perra,
Nuno M. M. Rodrigues, Mårten Sjöström, Luís Ducla Soares
and Olgierd Stankiewicz
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2 Multiview Video Plus Depth . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.3 Standardization—The Status and Current Activities . . . . . . . . . 14
2.3.1 Standardization in Multimedia. . . . . . . . . . . . . . . . . . . 14
2.3.2 Basic Technologies. . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.3.3 Multiview Video Coding . . . . . . . . . . . . . . . . . . . . . . 17
2.3.4 3D Video Coding. . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.3.5 New Standardization Projects . . . . . . . . . . . . . . . . . . . 21
2.4 Lightfield Super-Multiview with Camera Array . . . . . . . . . . . . 21
2.5 Lightfield with Microlens Array. . . . . . . . . . . . . . . . . . . . . . . . 25
2.5.1 Lightfield Raw Data-Based Approach . . . . . . . . . . . . . 27
2.5.2 Multiview-Based Approach. . . . . . . . . . . . . . . . . . . . . 28
2.5.3 Subsampled Grid of MIs Plus Disparity Approach . . . . 30
2.6 Free Navigation and Free Viewpoint Television . . . . . . . . . . . . 31
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3 3D Content Acquisition and Coding . . . . . . . . . . . . . . . . . . . . . . . . 41
Dragan Kukolj, Libor Bolecek, Ladislav Polak, Tomas Kratochvil,
Ondrej Zach, Jan Kufa, Martin Slanina, Tomasz Grajek,
Jarosłw Samelak, Marek Domański and Dragorad A. Milovanovic
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
v
vi Contents
3.2 Effect of an Incorrect Camera Alignment on the Accuracy
of the Spatial Reconstruction and Stereo Perception . . . . . . . . . 43
3.2.1 The Influence of Inaccurate Camera Alignment . . . . . . 44
3.2.2 Influence of the Camera System Parameters
and Spatial Position of the Object . . . . . . . . . . . . . . . . 48
3.2.3 Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
3.3 Compression Tools for Stereoscopic and Multiview Video . . . . 55
3.3.1 Stereoscopic Frame-Compatible Formats . . . . . . . . . . . 56
3.3.2 Compression Tools for Stereoscopic Video . . . . . . . . . 58
3.3.3 Performance Analysis of Compression Tools . . . . . . . . 61
3.3.4 Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
3.4 Multiview Video Compression for Arbitrary Camera
Locations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
3.4.1 Adaptation of 3D-HEVC to Nonlinear Camera
Arrangements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
3.4.2 Methodology of Evaluation. . . . . . . . . . . . . . . . . . . . . 71
3.4.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
3.4.4 Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
3.5 RecentDevelopmentsinVideoCompressionwithCapabilities
Beyond HEVC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
3.5.1 UltraHD Video Compression Performance Beyond
HEVC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
3.5.2 Conversion and Coding for HDR/WCG Video. . . . . . . 81
3.5.3 Projection Conversions and Coding for 360° Video . . . 86
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
4 Efficient Depth-Based Coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
Carl James Debono, Marek Domański, Sérgio M. M. de Faria,
Krzysztof Klimaszewski, Luís F. R. Lucas, Nuno M. M. Rodrigues
and Krzysztof Wegner
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
4.2 Depth Map Coding for Efficient Virtual View Synthesis . . . . . . 98
4.2.1 Algorithm Overview. . . . . . . . . . . . . . . . . . . . . . . . . . 99
4.2.2 Flexible Block Partitioning . . . . . . . . . . . . . . . . . . . . . 100
4.2.3 Directional Intra Prediction . . . . . . . . . . . . . . . . . . . . . 100
4.2.4 Constrained Depth Modelling Mode . . . . . . . . . . . . . . 102
4.2.5 Residual Signal Coding . . . . . . . . . . . . . . . . . . . . . . . 103
4.2.6 Rate-Distortion Performance . . . . . . . . . . . . . . . . . . . . 104
4.3 Depth Compression Using Standard Coding Techniques . . . . . . 105
4.3.1 Bitrate Distribution. . . . . . . . . . . . . . . . . . . . . . . . . . . 105
4.3.2 Depth Map Quality. . . . . . . . . . . . . . . . . . . . . . . . . . . 106
4.3.3 Bitrate Distribution Between Texture and Depth . . . . . 107
4.3.4 Coding Depth with Reduced Resolution . . . . . . . . . . . 111
Contents vii
4.4 Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
5 Error Concealment Methods for Multiview Video and Depth. . . . . 115
Sérgio M. M. de Faria, Sylvain Marcelino, Carl J. Debono,
Salviano Soares and Pedro Amado Assunção
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
5.2 Error Concealment for Multiview Video. . . . . . . . . . . . . . . . . . 117
5.2.1 Basic Methods Using Neighbouring Regions . . . . . . . . 117
5.2.2 Recent Advances in EC for Multiview Video. . . . . . . . 120
5.3 Methods for Error Concealment of Depth Maps . . . . . . . . . . . . 128
5.4 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
6 Light Field Image Compression . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
Caroline Conti, Luís Ducla Soares, Paulo Nunes, Cristian Perra,
Pedro Amado Assunção, Mårten Sjöström, Yun Li, Roger Olsson
and Ulf Jennehag
6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
6.2 Light Field Image Representation . . . . . . . . . . . . . . . . . . . . . . 145
6.3 Light Field Image Coding Formats . . . . . . . . . . . . . . . . . . . . . 146
6.3.1 Light Field Image Coding Using HEVC . . . . . . . . . . . 147
6.4 Scalable Light Field Coding for Backward Display
Compatibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
6.4.1 Display Scalable Coding Architecture . . . . . . . . . . . . . 155
6.4.2 Hierarchical Content Generation . . . . . . . . . . . . . . . . . 156
6.4.3 Efficient LF Enhancement Layer Coding Solution . . . . 157
6.4.4 Performance Assessment. . . . . . . . . . . . . . . . . . . . . . . 161
6.5 Sparse Set of Micro-lens Images and Disparities for
an Efficient Scalable Coding of Light Field Images. . . . . . . . . . 166
6.5.1 Scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
6.5.2 Displacement Intra and Inter Prediction Scheme. . . . . . 167
6.5.3 Encoding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
6.5.4 Decoding and Reconstruction . . . . . . . . . . . . . . . . . . . 170
6.5.5 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
6.5.6 Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
6.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
7 Impact of Packet Losses in Scalable Light Field
Video Coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
Caroline Conti, Paulo Nunes and Luís Ducla Soares
7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
7.2 Scalable Light Field Coding . . . . . . . . . . . . . . . . . . . . . . . . . . 179
viii Contents
7.3 Mitigation of Packet Loss Impact on Scalable Light Field
Coding. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
7.3.1 Relevant Factors for the Inter-layer Prediction
Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
7.3.2 Proposed Error Concealment Algorithm. . . . . . . . . . . . 183
7.4 Experimental Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
7.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
8 Transmission of 3D Video Content . . . . . . . . . . . . . . . . . . . . . . . . . 195
Emil Dumic, Anamaria Bjelopera, Khaled Boussetta,
Luis A. da Silva Cruz, Yuansong Qiao, A. Murat Tekalp
and Yuhang Ye
8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
8.2 DVB-T/T2, C/C2, and S/S2 Systems . . . . . . . . . . . . . . . . . . . . 196
8.2.1 DVB-T . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
8.2.2 DVB-T2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
8.2.3 DVB-S/S2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
8.2.4 DVB-C/C2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202
8.2.5 Transport of 3D Video in DVB Systems . . . . . . . . . . . 204
8.3 Hybrid Broadcast/Broadband 3DTV. . . . . . . . . . . . . . . . . . . . . 208
8.4 3D Video Delivery Over IP. . . . . . . . . . . . . . . . . . . . . . . . . . . 210
8.4.1 HTTP and RTP-Based 3D/Multi-view Streaming . . . . . 210
8.4.2 3D Video Distribution Over P2P Networks . . . . . . . . . 212
8.5 3D Video Distribution in ICN . . . . . . . . . . . . . . . . . . . . . . . . . 214
8.6 3D Stereo and Multi-view Video in Wireless Networks . . . . . . 215
8.7 Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
9 3D Video Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223
Emil Dumic, Khaled Boussetta, Luis A. da Silva Cruz,
Tasos Dagiuklas, Antonio Liotta, Ilias Politis, Yuansong Qiao,
A. Murat Tekalp, Maria Torres Vega and Yuhang Ye
9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224
9.2 Software Tools for 3D Video Compression . . . . . . . . . . . . . . . 225
9.2.1 H.264 and 3D Extensions. . . . . . . . . . . . . . . . . . . . . . 225
9.2.2 HEVC and 3D Extensions . . . . . . . . . . . . . . . . . . . . . 227
9.2.3 FFmpeg. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228
9.3 Streamers and 3D Video Players . . . . . . . . . . . . . . . . . . . . . . . 231
9.3.1 OpenSVC Decoder. . . . . . . . . . . . . . . . . . . . . . . . . . . 231
9.3.2 VLC Player . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234
9.4 Network Simulators, Emulators, Testbeds and Network
Analysis Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234
9.4.1 Simulators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
Contents ix
9.4.2 Emulators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
9.4.3 Testbeds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
9.4.4 Network Analysis Tools . . . . . . . . . . . . . . . . . . . . . . . 250
9.5 3D Video Evaluation Tools. . . . . . . . . . . . . . . . . . . . . . . . . . . 252
9.5.1 Generator of Degradations in 3D SBS Video
Sequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252
9.5.2 Crowd3D. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254
9.5.3 3D MOS Using DSCQS. . . . . . . . . . . . . . . . . . . . . . . 257
9.6 Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
10 Quality of Experience and Quality of Service Metrics
for 3D Content . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
Miguel Barreda-Ángeles, Federica Battisti, Giulia Boato,
Marco Carli, Emil Dumic, Margrit Gelautz, Chaminda Hewage,
Dragan Kukolj, Patrick Le-Callet, Antonio Liotta, Cecilia Pasquini,
Alexandre Pereda-Baños, Christos Politis, Dragana Sandic,
Murat Tekalp, María Torres-Vega and Vladimir Zlokolica
10.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268
10.2 Perceptual Characteristics of Multiview Content . . . . . . . . . . . . 271
10.2.1 Previous Work on QoE and QoS Assessment . . . . . . . 274
10.2.2 Quality Assessment Based on Geometric
and Spatial Distortions . . . . . . . . . . . . . . . . . . . . . . . . 275
10.2.3 Quality Based on Depth Map Analysis
and Distortion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
10.3 Subjective Quality Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . 280
10.3.1 Standard Methods for Subjective
Quality Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . 280
10.3.2 Psychology/Neuroscience-Based Methodologies. . . . . . 282
10.3.3 High-Level QoE Factors. . . . . . . . . . . . . . . . . . . . . . . 284
10.4 Conclusions and Future Directions. . . . . . . . . . . . . . . . . . . . . . 288
10.4.1 Measurement of Different Perceptual Attributes . . . . . . 289
10.4.2 Lack of 3D Image/Video Databases. . . . . . . . . . . . . . . 289
10.4.3 Visual Attention Models to Develop RR and NR
Quality Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 290
10.4.4 Need for a Standard for Subjective Experiments. . . . . . 290
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291
11 3D Visual Content Datasets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
Karel Fliegel, Federica Battisti, Marco Carli, Margrit Gelautz,
LukÁš Krasula, Patrick Le Callet and Vladimir Zlokolica
11.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300
11.2 Stereoscopic and Multiview Visual Content Datasets . . . . . . . . 301
