Table Of ContentReal-Time Video Content
for Virtual Production &
Live Entertainment
Real-Time Video Content for Virtual Production & Live Entertainment looks at the
evolution of current software and hardware, how these tools are used, and how
to plan for productions dependent on real-time content.
From rock concerts to theatre, live television broadcast to flm production,
art installations to immersive experiences, the book outlines the various
applications of real-time video content – the intersection of gaming and
performance that is revolutionizing how flms are made and how video content
is created for screens. Rather than render out a fxed video fle, new tools
allow for interactive video content that responds to audience activity, camera
position, and performer action in real time. Combining software renderers with
environmental information, video content is generated nearly instantaneously to
simulate depth, creating a new world of Virtual Production. This book provides
an overview of the current software and hardware used to create real-time
content while also reviewing the various external technologies the real-time
content is dependent upon. Case studies from industry experts appear in each
chapter to reinforce the tools described, establish industry practice, and provide
insight on a complex and rapidly growing discipline.
Real-Time Video Content for Virtual Production & Live Entertainment prepares
students and practitioners for a future working with real-time technologies and
informs current entertainment technology professionals how to rethink their old
roles using these new tools.
The book includes access to a companion website featuring web-based and video
resources that expand on topics covered in the text. Each chapter has a unique
page that points to example material, video presentations, and professional
studies on chapter topics. You can visit the companion website at rtv-book.com.
Laura Frank is a creator advocate and founder of frame:work, a professional
organization for the creative video community. She spent her early career as
a lighting programmer with projects spanning rock tours with David Bowie
and Madonna, to Broadway shows like Spamalot and television events like the
Concert for New York City. Laura made the shift to screen content and control
with the growth of the media server market. After a decade of refning a media
delivery workfow, she established herself as a Screens Producer leading a
highly regarded Media Operations team for prominent events around the world.
Her shows included the MTV Video Music Awards, The Game Awards, the Turner
Upfront, and the CMA Music Awards.
Real-Time Video
Content for Virtual
Production & Live
Entertainment
A Learning Roadmap for an Evolving Practice
Laura Frank
Designed cover image: Inside the Interactive Experience at the Al Wasl
Plaza, Dubai, 2022. Image Credit: Bild Studios
First published 2023
by Routledge
605 Third Avenue, New York, NY 10158
and by Routledge
4 Park Square, Milton Park, Abingdon, Oxon, OX14 4RN
Routledge is an imprint of the Taylor & Francis Group, an informa business
© 2023 Laura Frank
The right of Laura Frank to be identifed as author of this work has been
asserted in accordance with sections 77 and 78 of the Copyright, Designs
and Patents Act 1988.
All rights reserved. No part of this book may be reprinted or reproduced or
utilised in any form or by any electronic, mechanical, or other means, now
known or hereafter invented, including photocopying and recording, or in
any information storage or retrieval system, without permission in writing
from the publishers.
Trademark notice: Product or corporate names may be trademarks or
registered trademarks, and are used only for identifcation and explanation
without intent to infringe.
Library of Congress Cataloging-in-Publication Data
Names: Frank, Laura, author.
Title: Real-time video content for virtual production & live entertainment:
A learning roadmap for an evolving practice / Laura Frank.
Description: New York : Routledge, 2023. | Includes index.
Identifers: LCCN 2022034077 (print) | LCCN 2022034078 (ebook) |
ISBN 9781032073477 (hardback) | ISBN 9781032073446 (paperback) |
ISBN 9781003206491 (ebook)
Subjects: LCSH: Performing arts—Production and direction. | Digital video. |
Real-time data processing.
Classifcation: LCC PN1590.P74 F73 2023 (print) | LCC PN1590.P74 (ebook) |
DDC 791—dc23/eng/20220930
LC record available at https://lccn.loc.gov/2022034077
LC ebook record available at https://lccn.loc.gov/2022034078
ISBN: 978-1-032-07347-7 (hbk)
ISBN: 978-1-032-07344-6 (pbk)
ISBN: 978-1-003-20649-1 (ebk)
DOI: 10.4324/9781003206491
Typeset in ITC Offcina Sans
by Apex CoVantage, LLC
Access the companion website: www.rtv-book.com
Contents
List of Contributors x
Acknowledgments xi
CHAPTER 1 Introduction 1
We Are Surrounded by Visual Content 1
Creative Video for Live Visuals 1
Live Entertainment Is Forever Changed 3
Real-Time Video Content Is the Engine that Powers Virtual Production 4
What This Book Is Not 5
SECTION 1 EXPLORING REAL-TIME CONTENT 7
CHAPTER 2 Defning Real-Time Video Content 8
Defning the Term “Real-Time” 8
The Master Variable: Time 11
Variable Triggers Combined with Variable Results 13
CHAPTER 3 Foundations Part 1: The Video Image 14
How Do Computers Display Images? 14
What Defnes a Computer Image? 15
Making Real-Time Images: Generative Content 16
Making Real-Time Images: Software and Game Engines 16
CHAPTER 4 Foundations Part 2: Code and Generative Content 18
Generative Content 18
Video Games 19
The Internet 20
VR and AR 21
Case Study – Blair Neal – Trace Your Road 22
CHAPTER 5 Foundations Part 3: Broadcast 27
Studio Image Compositing 27
Camera POV 30
Sports 31
Broadcast AR 32
Background Replacement 32
v
Contents
Case Study – Bluman Associates – Dave at The BRITS 2020 34
The Project and Use of Real-Time 34
The Challenges 35
The Tests, Research or Development to Provide the Solutions 36
The End Goal (Not Modifed to Be Successful!) and the Result 38
CHAPTER 6 Real-Time Content Generation Applications 39
Custom Code 39
Processing 39
Effects Engines 40
Media Server Effects Engines 41
Node Based 41
Game Engines 42
Beyond Computer Graphics 43
Case Study – Electronic Countermeasures – The Foo Fighters 26th Anniversary Tour 44
CHAPTER 7 A Collision of Technologies 50
What Are All the Tools We Combine? 50
Media Control and Management 50
Projection Mapping and Scenery Tracking 51
Environment Modeling 52
Camera Tracking 52
Content Generation 54
Front Plate and Back Plate 56
Virtual Production Stage Types 56
LED Stages/Volumes 56
LED Stages/XR Stages 57
Modern Green Screen 57
Online Events 57
What Happens Next? 58
Hybrid Audiences 58
Remote Performance 58
AR for Live Events 58
Case Study – Rich & Miyu – AR Livestreamer 58
A TouchDesigner Based Software Solution
for AR Applications 58
SECTION 2 REAL-TIME CONTENT ENTERTAINMENT APPLICATIONS 65
CHAPTER 8 Content Production 66
The No-Render Advantage 66
Creative Flexibility 67
Live Signal Manipulation 67
Interactive Content Production 68
Environmental Responsiveness 68
Camera Perspective Generated Content 69
Case Study – Fray Studio – Real-Time Update 69
CHAPTER 9 Augmented Reality 74
Virtual Graphic Imagery 75
Scenic Augmentation 76
vi
Contents
Human Tracking and Motion Capture 76
Case Study – All of It Now – Coldplay x BTS 77
Project Timeline 77
Technical Considerations 78
Creative Pipeline – The Creative Process 78
On Set Deployment/Challenges 78
Post-Production Challenges 79
A Volumetric Future 79
CHAPTER 10 Background Replacement 82
Scenic Backdrops 82
Depth Simulation 83
LED vs Green Screen 85
Case Study – Fernando González Ortiz – The Lion and The Firebird 88
The New Frontier in Student Filmmaking with Virtual Production 88
CHAPTER 11 Extended Reality and Mixed Reality 92
Front Plate and Back Plate 92
Sync 94
Scenic Extension 95
AR + Depth Sensing 97
Mixed Reality 97
LED vs Green Screen, Part 2 98
Environmental Immersion 98
Lighting and Refections 99
Budget 99
Speed 99
Background Cohesion 100
Camera Frustums 100
Background Quality 101
Case Study – Lux Machina – Worlds 2020 102
CHAPTER 12 The Future of Real-Time Content 106
Developing Industries 106
Developing Technologies 107
Case Study – Bild Studios – Al Wasl – The Interactive Experience 108
How Bild turned Al Wasl Plaza into the world’s largest interactive installation for Expo City Dubai 108
Al Wasl – The Interactive Experience 110
Phase 1 – Technical Feasibility and Creative Exploration 112
Background 112
The Need for a Technical and Creative Feasibility Study 114
Choice of Real-Time Engine 114
Requirements 115
Splitting Up the Real-Time Rendering Power 115
Previz 115
Phase 2 – The App 116
Interactive Audio Setup 119
Phase 3 – Creative Development 120
Phase 4 – Onsite Installation 120
Thoughts by David Bajt, Co-Founder of Bild and Principal Consultant for Al Wasl Expo Projects 121
Thoughts by Expo City Dubai Executive Creative Director, Amna Abulhoul 122
vii
Contents
SECTION 3 STRATEGIES USING REAL-TIME CONTENT 125
CHAPTER 13 Team Leadership and Structure 126
The Video Umbrella 126
Content Creation 126
Media Operations 126
System Engineering 127
Broadcast Engineering 127
Video Team Leadership 127
Partner Teams 128
Scenic Design/Art Department/Virtual Art Department 128
Lighting Design 128
Audio 129
Clients of the Video Team 129
Director 129
Producers 129
Case Study – Visual Noise Creative – MTV VMAs 2021 129
CHAPTER 14 Inter-Department Communication 134
Crafting Good Collaborative Relationships 134
Case Study – EVOKE Studios – Siemens Hannover Messe Digital 2021 137
Doing Something First, It Can Be Tricky 138
Plan, Plan, Plan! 138
Creating the Solution 140
The End Result 140
CHAPTER 15 Production Planning 143
Client Planning 143
Budget 144
Schedule Planning 145
Review 145
Workfow Outline 145
Scenic and Storyboard Review 145
Creative Discussions 146
Team Resources 146
Build and Publish Content Production Workfow 146
Development and Testing, Review and Reiterate 147
Case Study – dandelion & burdock – Johnnie Walker Princes Street Interactive World Map 147
Overview 147
Brief 147
Preparation 148
Process 148
Delivery + Opinion 151
CHAPTER 16 Obstacles to Success 153
Community Education 153
Inversion of Process 154
Digital Is Perceived as Easy 154
Real-Time Can Be Time Consuming 155
viii
Contents
Team Growth and Wellness 155
Case Study – Savages – Practice Makes Process 156
A Case for Intentional Practice 156
CHAPTER 17 Creative First 161
When Technology Leads Design Discussion 161
What Is the Creative Goal? 162
Technology Serves Creative Intent 163
Matt Swoboda – Founder – Notch 163
Ashraf Nehru – Founder – Disguise 163
Greg Hermanovic – Founder and President – Derivative 164
Francis Maes – Founder – SMODE Tech 165
Emric and Martin – Co-Founders – VYV 165
Case Study – Luke Halls Studio – Forest of Us 166
APPENDICES
Appendix A: Real-Time Content Production Tools and Capabilities 172
Appendix B: VFX Map for Real-Time Content 173
Appendix C: Glossary 174
Index 181
ix
