ILM VFX supervisor Julian Foddy and associate VFX supervisor Florian Witzel on how they created the astounding effects for this feature, including the development of an AI face-swap technique to add realism to digi-doubles
Picking up where 2016’s Doctor Strange left off and also continuing the story of WandaVision’s Wanda Maximoff, Marvel Studios’ Doctor Strange in the Multiverse of Madness follows the story of Strange as he protects America Chavez, a teenager capable of traveling the multiverse, from a villainous Maximoff. The feature was directed by Sam Raimi and stars Benedict Cumberbatch as Stephen Strange alongside Elizabeth Olsen as Wanda Maximoff.
ILM worked on everything from action shots and gore to environments, digital magic, Strange’s Third Eye…and even shredding a character into ‘spaghetti’, for which Play-Doh being squashed through a garlic press was used as a reference. Read on to find out how the team put these spectacular effects together…
Tell us about the team you put together for your work on this film.
Julian Foddy: My VFX producer Danielle Legovich and I put a lot of planning into selecting the correct team as we had such a variety of work, which all needed to interact, so it was important that we had a team where the generalist / environment team would work hand in hand with FX and lighting. I was keen to get a workflow planned out in advance of turnover so we had a clear plan for who would render what in the huge FX magic vs powers moments.
What can you say about the filmmaking style of the director and the DP and how that informed the VFX?
Julian Foddy: When Sam Raimi came on board we knew we would be sticking to the original brief that ‘Multiverse of Madness’ would be to some extent, a horror movie. I think Sam’s style of gory, gruesome moments interspersed with comedic beats works very well, and no one’s ever going to complain about a Bruce Campbell cameo!
Myself and Janek also worked closely with the DP John Matheison to decide on a look for the Illuminati Atrium prior to the shoot, so although the actors were on a predominantly green stage, we knew how to light them in such a way that they’d sit well in the CG environment further down the line.
Did you develop any special software or new workflows? Did you use any new tech that you haven’t used very much before?
Julian Foddy: We developed a new face replacement technique which mixes the use of traditional digi-doubles and AI based face-swap tools, for stunt and action shots. We’d originally planned to achieve the many face replacements shots using 2D patches on comp from footage of the actors, or in more challenging cases, the hi res digital double.
During the shoot, I was seconded to Marvel as VFX supervisor for the splinter unit. I also directed ‘AI facial training’ sessions with principle cast, which involved shooting facial close ups of them performing various generic and specific dialogue in controlled lighting with a multi-camera array.
I found that by using this material to train an AI face-swap model, the actor’s real face could be added on top of their digi-double face to bring an extra touch of realism and ‘nuance’ to the digi-double’s performance. There are several fight beats in the Illuminati battle, and as Wanda takes down the line up one by one, each of these fight beats utilizes this technique in some way. We found that not only did the use of AI help in ‘face replace’ shots, but also in any instance where our characters were fully digital, such as the ‘cyclone’ moment as Wanda versus Marvel reaches its conclusion, that the extra level of detail and plausibility the AI face swap provided made a huge difference to the ‘read’ of the shot.
Tell us about the fight scenes you worked on, and your work with digi-doubles and face replacement.
Julian Foddy: Whilst we used AI a lot in the Atrium battle scenes, in the Strange / Mordo square off, things were approached more traditionally. We managed to use a 2D technique for a couple of Mordo face patches, and in the punch-up in the trench, an almost full frame CG Strange face replacement which I’d defy anyone to spot – the reason it’s so successful is that we actually retained the stunt performer’s neck and mouth, so there’s a lot of visible tendon and jaw tension that helps sell the digital disguise above it.
How did you create the digital version of the British Museum?
Julian Foddy: The original plan had been to shoot the sequence on location in the actual atrium of The British Museum but practicality soon got in the way – the amount of SFX and Stunt rig equipment would have been impossible to shoot around, so the sequence was shot on a green stage at Longcross Studios, which was accurate in floor plane and scale to the real location, with just a small set build for the columns and entrance Wanda walks through.
We had, however, been able to gain access to the British Museum (which was closed due to Covid), for scanning and photography. This provided us with a highly detailed LiDAR scan of the whole space and a huge data set of texture / panorama photography under day and night lighting conditions. This allowed us to build a highly-detail, like-for-like model of the atrium in CG. There are some minor design changes in our version of course, the large opening to the bio-monitoring room for example, and the various statues placed around the atrium, including the ‘warrior princess’ statue that ultimately dispatches Maria Rambeau’s Captain Marvel. We took the unusual approach in the 3D modelling stage to build the atrium as per real construction techniques – each stone block is a separate object, as is each floor tile. Inside the walls there are steel joists, cinder block inner walls, plasterboard etc – this meant that in moments of impact or destruction, there was not only plenty of complexity but also extreme realism and plausibility to what we see, and how it all falls apart!
Tell us how you went about redressing the streets in New York.
Julian Foddy: The New York exterior was actually all shot in the UK – the backlot of Longcross Studios became two complete blocks of Greenwich Village. This set build can be predominantly seen in the opening sequence of the film when Strange fights ‘Gargantos’ (the cyclopic tentacle monster). In our ‘alt universe walk and talk’ sequence we were tasked with redressing and augmenting the amazing NYC set to look even more ‘otherworldly’. Our challenge was to give the audience as many visual clues as possible that things are different where we’ve ended up! Firstly, we had to extended buildings vertically as the set was only 2 stories high, so that gave us some scope to inject futuristic architecture etc, but the main way to achieve the brief was the abundance of CG foliage we added – every flat surface is covered in ivy or creeping plants, often with unusual hues not familiar on Earth-616. This plant life also behaves correctly : bigger branches droop under gravity, and obey the rules of phototropism ( plants try to grow toward the sun) .
Other visual clues to being in an alternate universe that we added are signage, road markings, car number plates, traffic lights etc – all are not what we’d normally be used to.
How did you create Zombie Strange?
Julian Foddy: The hero digital version of ‘Zombie Strange’ was actually an asset developed by Weta digital which was then shared with ILM. However in the shot shots ILM worked on, Zombie Strange is predominantly ‘in camera’ – prosthetics and make-up on Benedict Cumberbatch. We gave him a digital cheek / mouth interior with hanging strips of rotting flesh, and a CG hand to allow us to erode a few fingers where the flesh has fallen away!
We did add some interesting alterations to the ‘sling ring’ portal that Zombie Strange conjures – to match his jerky, slightly uncoordinated movements. The portal’s ring of sparks is slightly imperfect and angular, and the emission ‘splutters’ in places.
What was the process for your work on Strange’s Third Eye?
Florian Witzel: In the very beginning there were a lot of different ideas and concepts of what the eye could be. It wasn’t decided yet how much of an independent character or what the relation to Dr. Strange would be. There were multiple ideas all the way from evil monster looks to animal-like shapes to a perfect human eye. Eventually the client settled on a human-like look which was actually the hardest version for us. Not only is it hard to create a perfect human eye and have it right next to his real eyes, but we also needed to figure out how it would sit naturally and anatomically believable on his forehead. Questions like is it a left, right or symmetrical eye, does it have eyelashes, where is the eye brow (which is very important to convey emotion) and how could the eye socket be positioned in his skull. Also the Third Eye can open and close at will, so we needed to figure out how it can do that on his forehead. At the end of the day, the eye needed to convey Dr. Strange’s emotions in a believable way. Any irregularity would send a different message. It was a very artistic and detailed process. Part of our dailies were called “eyelies” where we would only talk about human eyes. It took all the people involved and their careful attention to miniscule details to integrate the Third Eye believably into Dr. Strange’s face.
What references did you use?
Florian Witzel: At first we received a variety of concepts from Marvel. Different studies of animal eyes, evil monster eyes, as well as modified human-looking eyes. On our end, our Model Supervisor Bruce Holcomb made a collection of all kinds of Third Eye’s being created in previous movies. It was a great collection of Cyclops, Monsters and devine Eyes from movie history. David Bocquillon also worked on various concept art of how an eye on Dr. Strange’s forehead could be shaped and convey emotions. All the studies and references really helped to develop a clear idea and also showed us what didn’t work and what we were trying to stay away from. Having done that research helped us communicate much better with Marvel and Janek Sirrs about where we are going and helped us stay focused getting the shots done without going in too many different directions.
Tell us about your work on Black Bolt and Mr Fantastic’s deaths.
Julian Foddy: The shocking moment where Wanda dispatches Black Bolt and Reed Richards in quick succession is when things start to get exciting, and it’s also where the complexity of the VFX work increases. Some shots were a combination of multiple plates as various Illuminati members were shot separately due to covid travel restrictions etc. It’s worth pointing out that in this beat, both Black Bolt and Reed Richards (Mister Fantastic) are wearing 100% CG costumes developed by ILM, even in extreme close ups. The unusual FX simulation work in these character deaths was also an interesting task: For Black Bolt we wanted the ‘sonic blast’ inside his own head to feel physically plausible, so we looked at slow motion reference of objects shattering, and human faces in wind tunnels or skydiving. The complex rippling of the skin, etc., would have been very hard to simulate at normal frame rate so we stretched the whole shot out, simmed at 120fps using cowl, skin, and skull geometry all interacting with each other, then time-compressed the result. There’s something in the sub frame motion blur that really helps sell this effect, along with the very cool ‘squelch’ sound effect brought by Skywalker Sound!
The complex shredding of Reed Richards into ‘spaghetti’ was also a real challenge. We used references of Play-Doh being squished through a garlic press and a 1980s Play-Doh Barber Shop toy as a guide to how a body might get squashed into tubes of flesh… an area that really took some work was to make it feel like the geometry was being torn apart rather than emitting arbitrary ‘sausages’. The trick here was to transfer UVs from the body mesh onto the FX geometry so areas of the costume can be seen tearing away and snaking to the floor. Some of them continue to twitch for a few seconds after they hit the ground!
How did you create the different colours of magic?
Julian Foddy: We actually had great starting points for our digital magic and powers effects – ILM had worked on Scarlet Witch’s Chaos Magic and Captain Marvel’s Binary and Photon powers on previous films. The set-ups needed elaborating and reworking to meet our requirements, but we had great visual ‘match-to’ reference. We were lucky in that our brief here was basically: ‘Different universe, different people, same powers’.
In the world of digital FX and Houdini simulations, changing the colour of flames or magic powers isn’t hard of course, but the use of those colours as a storytelling medium was an interesting visual challenge for us, in the case of the clash of powers between Wanda and Marvel, that culminates in the huge spherical blast shattering the atrium roof. With fast moving action and quick cuts, we had to see that Marvel starts out by blasting Wanda back, and has the upper hand, then Wanda’s chaos magic envelops the Marvel photobeam, ultimately concentrating the energy into a singularity which then explodes. It basically came down to ensuring a change over time in the prominent colour, so we start out yellow / orange and finish almost totally red. This meant I spent a lot of time squinting at the screen to blur my vision while watching the edit to ensure the ‘flow’ was correct!
Author: Tanya Combrinck
Tanya is a writer covering art, design, and visual effects. She has 15 years of experience as a magazine journalist and has written for publications including 3D World, 3D Artist, Computer Arts, net magazine, and Creative Bloq.