Everything you need to know about Virtual Production
‘Virtual Production’ using LED video walls and Game Engine technology will reshape filmmaking forever. Here is what you need to know.
The pandemic has brought about global upheaval and disruption, every business has been impacted. But while feature film and television productions across the globe were downing tools to await a return to set, the seeds to a quiet revolution were being sown. Giving Disney+ its tentpole title at launch, The Mandalorian became one of the most watched series streamed during lockdown — a period which saw streaming services globally explode as people with nothing to do but watch TV tuned in. The Mandalorian included a number of behind the scenes videos on Disney+, a nice addition and a reminder what we’ve lost from no longer having DVDs with all the bundled bonus content to sift through.
Like many in the spring of 2020 I watched those behind the scenes videos, in awe at the technology John Favreau and the team from Industrial Light & Magic and its partners had put together. They had created a method of shooting that could place actors in any environment imaginable, where they were lit by the environment itself using giant LED screens projecting a photorealistic world that was build in 3D modelling in Unreal Engine. And whats more as they moved the camera, the world moved in sync. You could look around corners into virtual worlds. It looked real. This was Virtual Production. It blew my mind.
Almost synchronously, it was blowing my brothers mind in Los Angeles also. It came up on one of our weekly family zoom calls, another first brought about by the pandemic. We were both hooked.
Fast forward nearly 12 months, and we’ve just finished a major shoot in Hollywood together, creating a promotion for a group of US networks using Virtual Production. I quit my senior job at a big independent producer and we’ve launched a company, Final Pixel, using his experience in the Advertising world, and mine in TV & Film to broaden the adoption of this technology. Bolstered by the producing might of veteran Monica Hinden, Final Pixel’s purpose is to give people the chance to free their imagination and realise their creative dreams. We are making virtual production more accessible to filmmakers so you don’t need a big feature, or Mandalorian, sized budget to afford it and realise the benefits shooting with this technology can bring.
And those benefits are numerous. I sometimes forget and need to remind myself having been so immersed in this world for months just how ground-breaking and game changing this is. Our shoot in LA filmed a group of celebrities across 4 locations, at different times of day, a green screen shoot, 18 pages of dialogue, and in the midst of a pandemic — in just a day and a half. This was all from the comfort of a Hollywood studio, in a Covid-safer environment. If we ever wanted to re-shoot, we can go back to the exact same lighting and setup, the exact same set now exists for ever. Whats more our performers could see the scenes they were being filmed in, not like green screen where they have to imagine it. The Director, DoP, and the whole crew can immediately see in-camera the results of the composited image. The lighting can match exactly. The actors react to events in the scenes they can see, they don’t need to pretend. At a touch of a button we can switch to another scene. The problems this type of production can solve are numerous, and the creative possibilities limitless
So what is Virtual Production?
The convergence of multiple technological developments has brought us to this point. Often the greatest shifts and advances in civilisations are brought about this way, not by one single technology, but by the coming together of multiple maturing existing technologies. LED wall quality and resolution, Graphics Processing Power, Physical-Based Rendering using Epic Games’ Unreal Engine, Camera tracking, Large format Digital cameras, networking speeds, 3D modelling tools and an industry primed for operating in remote teams. A coming together of highly talented artists and computer programmers that have immersed themselves in technology to drive it forward.
Virtual Production works by tracking the ‘real-world’ camera in a studio, itself not a new technology — at the BBC we had News studios fitted with tracked cameras for decades. However, what is new is combining this tracking information with the ‘Virtual’ camera tool within Unreal Engine, live and in real-time. Originally developed to shoot cut scenes for computer games, this Virtual camera can be programmed to move in sync with the real-world camera with little noticeable lag. The result is that we can then project what the virtual camera is seeing onto a massive LED wall, basically a huge television, which is calibrated to be at the correct scale and orientation so that for our actors in the scene — it feels like they are really there.
Whats amazing is that real time render of 3D models makes this possible. The typical VFX pipeline includes a large portion of time devoted to rendering images. With Unreal Engine this happens right before your eyes. Its the technology that has allowed Fortnite to flourish as a massive multiplayer online game like no other. The tools in Unreal Engine allow our Virtual Art Department to create environments, or sets, which are incredibly realistic looking. The prognosis is this will only improve with the release of Unreal Emgine 5 later this year and ever improving Graphics cards from NVIDIA.
[Photorealism in Unreal Engine 4: Link to Australia video ]
The temptation may be to think this means the end for physical set builds, however the reality is that on Final Pixel shoots the art department has been busier than ever, a critical component of the crew. That is because to really create a believable in-camera effect the foreground props and art direction needs to blend seamlessly with the virtual world. This is where the magic really starts to happen, and the collaboration on set to achieve realistic results is like nothing I’ve seen before. Its like a return to the days of early cinema. A magical furnace of creativity awaits, and production designers and artists that can learn to straddle the physical and virtual worlds will have a long career of creative fulfilment ahead of them.
Who makes this?
Which brings us nicely onto the who. Who makes Virtual Productions? Well primarily its traditional ‘physical’ production crew, however they are needing to adapt to this new way of filmmaking and the different workflow steps that goes with it. There is much more emphasis on pre-production, where the virtual sets you are going to shoot on can now be completely created before even getting into the Studio — down to every last detail. But these crews have an uphill course to go, which is why we are working closely with educational establishments and internal training to build the virtual production crews of the future who will master the craft.
Crucially there is also a new department in town on the film set, at least that’s how I like to think of this. Where for decades we’ve had the firm hierarchy and structures of Lighting, Camera, Art etc — we have now, rather abruptly, thrown in the ‘Tech’ department. Finally the truest form of tech and film production coming together on live-action shoots with game developers and software experts on set managing the tools in real time, responding to Directors requests to ‘move that steam vent over there’ and the DoP to ‘spin the world a few degrees to the left’ and the production designer ‘bring the colour of that floor up to match the oak veneer’…. Its truly a sight to behold and this is only in its infancy. Final Pixel pioneered a few firsts of our own, running sets remotely from the other side of the planet for one, having live channels with the stage over Discord so our Virtual Art Department could make final tweaks and edits in the background. We’ve also worked on the craft, learning to shoot dual camera to improve coverage and further push the envelope. It’s a time for innovation. A time like no other.
Whats next?
This technology adoption is in its infancy, there are still bugs to work through, but its improving at a rapid pace. More and more tools are coming onstream to help with stage management, and with every shoot a crew gets more experienced and learns.
As more professionals realise the benefits of this filming, we’ll see more commercials, promotions, TV shows and films use the technology in more and more creative ways. Stretching budgets and precious time with talent further.
Some believe it will go as far to truly democratise film and TV production, making richer scripts available to indie productions by removing some of the traditional barriers to exploring certain genres or periods in time. Combined with proliferation of online streaming and publishing services, there is certainly a future that will see more choice and relevancy for audiences of all backgrounds. Perhaps this will be one of the more wholesome, lasting positive impacts this awful pandemic will have released onto the world.
About the author: Michael McKenna is the CEO and co-founder of Final Pixel, a creative studio established in 2020 to specialise in Virtual Production with LED walls and game engine technology (‘Mandalorian style’). On projects he acts as Director of Virtual Production, bringing together all elements of the tech to deliver results. Final Pixel has operations in the US and UK, based in Los Angeles, New York and London. www.finalpixel.com
