Today I was eager for a new Pixar poster. Who would’ve thought that I spent nearly one and half hour talking with the guys and gals from Pixar?

Because I had lost my Ratatouille poster last night I decided that the best thing to do was to attend another RenderMan presentation, get another walking teapot, get another RenderMan sample DVD and to fill out another survey sheet. In the comments section I apologized for showing up again only for a new poster. Sometimes you just got to be honest.

The Plan

The people from the jumping lamp studio

Morning in Stuttgart,
originally uploaded by Phil Strahl.

But instead of hearing the same stuff again about the integration of Massive into the Pixar pipeline by Ziah Fogel there was Afonso Salcedo, one of the lighters at WALL·E who did about 3 or 4 percent of the shots in the whole movie, “a good average”. I learned much about the way of lighting at Pixar, and how they dodge using ray tracing, the Achilles heel of RenderMan. For WALL·E the director wanted a 70s sci-fi movie look so Pixar got Dennis Muren over to set up some of the basic lighting for WALL·E in the atrium with a 70mm-film camera with anamorphic lenses and some WALL·E and Eve mockups in real size. They did tests on the DOF, the cushioning and the way lens flares and iris circles look. That’s what I love about Pixar: Total involvement and research for every project.

I asked Dylan Sisson (?) afterwards and he answered me quite sharp that they don’t because they want to keep the full artistic control over the light¹. They even light characters and background separately. So they set everything by hand, about two hundred lights each shot. No wonder it takes them so long!

When the presentation was over I got myself a nice espresso at the lounge and returned half an hour later to the recruiting presentation which was in fact a Q&A session with all the folks from Pixar that were present in Stuttgart: Carlos Baena, Sharon Calahan, Dylan Sisson, Afonso Salcedo, David Munier, and Ziah Fogel. I seized the day and asked them a lot about anything that came to my mind, most importantly how they do their compositings and what software they use and whether they are looking for a compositor.

That hit me deeply. They don’t even have the job I was hoping to apply for.

After the session there was a buffet in the little room outside the seminar room and the audience mingled with the Pixarians. Paul from the 4th semester was interested to learn from Carlos how the animation of WALL·E’s crawlers played a role in conveying his character. I was still eager to get some of my questions answered by Sharon Calhan, Carlos Baena and Afonso Salcedo at one table - unbelievable. And silly me forgot to take a picture before I left for some presentations with “educational” value for me.

3D and Live Action in 3D. In stereo

Meeting Colleagues,
originally uploaded by Phil Strahl.

Stereoscopy is the buzz of the year, as I mentioned earlier. Disney, Pixar (yes, there’s a difference!), DreamWorks, ILM and Sony ImageWorks are working on features in stereoscopic 3D now and all of us artists in the industry either adapt or become extinct. I chose to adapt but was very skeptical about it.

Journey 3D by Bret St. Clair (from Meteor Studios) and Chris Harvey (from Frantic Films) was the first presentation in stereo³. The two studios in Canada have been working on The Journey to the Center of the Earth 3D starring Brendan Fraser and had to handle a live action/CGI feature to seamlessly blend together - all in stereo.

The two of them talked about their struggles and problems and seemed as if they were telling a terrible tale of how they had survived 9/11 and Katrina. They had to tell so much in such a short time that I noted down like crazy what I felt was important. And there was a lot, starting from the camera rig on set. They were using two different rigs, one that allowed to adjust the interocular separation⁴ even to zero if they had wanted it to thanks to a 45° semitransparent mirror (which produced differences in distortion and lights), the other rig had an interocular separation of 2.7″ as opposed to the 2.5″ space between the average viewers’ eyes.

Zooming, for example, was nearly not possible because the two cameras had to be in totally in sync which was not possible in live action shooting because even an offset of the tenth of a pixel makes a difference to the perceived depth.

Matte paintings should be executed at least in 2.5D, but it is much better to project them onto geometry which shouldn’t be too low-poly because you just can’t fake it in stereo. And don’t paint haze, fog or atmosphere in your matte paintings because that all needs to be done in 3D.

You can’t make a 3D film into a stereo 3D film at one point, you have to be certain about the presentation when you’re working on the concept because from that point on, nearly everything should be executed in stereo, especially the layouting and animatic. Artists that have only worked on non-stereo pictures before (which is the majority of us at the time) tend to cheat to get the image an aesthetic quality instead of being 100% correct in a three dimensional space.

Create optimal viewing conditions. From the emphasis put onto that statement it is the most important one. You have the same projectors on set and in your studio as you will have in the theater, you have to have at least a 20 ft screen for the projection and there should be one 2k stereo-projector instead of having two separate projectors because you will have to stop the footage, scrub through it, etc. to review your work.

Stereo means to double everything: Storage space, amount of footage, time for trouble shooting, compositions and rendering time, when you’re not working with some tricks: In the case of Journey they were able to have the second image rendered in only 20% of the time of the first one by caching a lot of render-data so only the occluded geometry had to be re-rendered. Pretty clever and it saved them a lot of precious rendering time. They used Gelato by nVidia for that, but I have no idea what it exactly does.

And now for the dreaded, feared and gruesome focus on matchmoving in stereo *insert 50’s horror film music here*. Even for ground planes it had to be done absolutely correct to every 0.1 pixel otherwise something would’ve been off in the final depth perception. A tracking cube helps you much in those cases because the perspective its clearly visible at all times. After tracking you should test it in the theater-situation before continuing because it has to be absolutely correct. The guys developed a way of tracking only one view of the stereo and then calculating the correct track for the other “eye” because tracking them separately didn’t result in absolutely the same movement for each eye.

Keeping as much meta data from the camera as possible in each shot. You might not use it at all times but sometimes it really saves your shot.

Another thing to know is that distortions work differently. Instead of having a centered distortion for every eye, the distortion should be in the middle between the eyes, making them off center for the individual camera footage. Lens artifacts should be matching in too: It is distracting, if you have a lens flare in only one of the views or that highlights don’t match and hence “flicker” when looked at. If you can’t avoid this on set you have to either remove the highlight in question or digitally double it for the other eye.

When it comes to compositing you want to have both eyes’ footage in a single composition, although it doubles the complexity. But you can make corrections to one eye where needed and have them share nodes and effects at the same time. Ideally the compositor should be sitting in front of the 20 ft so s/he can always check what it’ll look like in the theater. “Awake” for Fusion does the job ob having both eyes in one composition, by the way.

When evaluating stereo depth you should always revise shots in sequence so you can avoid too harsh differences in depth that the audience’s eyes don’t need too long to adapt each shot and might miss the important action.

Oh, and bump maps should be traded in for displacement-maps!

3D in 3D

Stereoscopic Depth as a Storytelling Tool by Mark Empey and Robert Neuman from Disney Animation was a very insightful presentation because there literally were worlds between their stereo-footage and the previous presentation. Disney showed off a five minute long montage of the stereo version of Meet the Robinsons and the depth perception worked just fabulous, seemed harmonic and believable. The first seconds were terrible, the following 20 seconds were okay and after the first minute I never wanted to watch plain mono footage ever again. Journey looked 3D-ish but not quite that pleasing. Even I could tell, that Disney had mastered 3D as they had mastered 2D. And they gave an insight about their way:

First of all you should avoid using stereo like it was used in the fifties and eighties:

Making it comfortable for the audience is the most important thing. If they get headache, hurting eyes and nausea from watching your stereo movie they will never want to watch one again. So you should keep the interocular distance of 2.5″ in mind. That’s not so much important for near objects that are “floating above” the screen, but for those who are beyond infinity: When looking at objects that are on the horizon both eyes are exactly parallel. But with the use of floating windows (more about them below) it might happens to have the audience look beyond that point where the eyes aren’t looking just parallel, they’re looking outwards which is painful because mother nature never wanted us to look that way. So avoid having offsets bigger than 15 pixels towards infinity.

Just like using a color script it helps much using a depth script where a value from one to ten (sometimes eleven) tells the stereoscopic supervisors how deep a sequence, scene, shot has to be. Neuman (or was it Empey?) said that for him stereoscopic depth equals with emotional attachment to the character. So a depth script is essentially a script where the arch of emotional depth is traced. The closer you want your character to connect with the audience, the more you push her/him out of the screen (not out of the floating window, to be described soon!). For emotional distance keep your character “behind” the screen.

Sometimes you want to have objects come out of the screen very close. It’s okay if you do it in the middle of the screen but when one of the stereo images is being clipped by the screen frame, then a “window violation” occurs. When this happens a viewer perceives the presented images much flatter because her/his brain needs to constantly decide between two conflicting different depth-cues. So what do you do? You produce a “floating window”.

The floating window is created by unequal masking of the different stereo-images, e.g. you give the left image a thick black border on the left and the right image a thick black border on the right. This little trick lets the screen in the theater appear as if it was floating in front of the action, depending on the thickness of the black border because the viewer always associates the floating screen with the physical one.
You can even have the floating window rotate and tilt, all with diagonal masking. Disney uses a Maya plug-in that displays for the stereo camera rig in the viewport the near plane (where nothing can be any closer without risking eye strain), the floating window transformations, the screen plane (where the physical screen in the theater is located) and the far plane (which defines infinity) and can be controlled by eight values.

DOF should be limited and only applied to objects in the background (”behind” the screen). For objects in the foreground you should deemphasize their importance with lighting.

Don’t toe in the two cameras in your rig, because this results in vertical mismatching and is uncomfortable to view. Instead keep the cameras parallel at all times and use only a different portion of their image circle, just like a perspective correction lens works.

At last it is important to have your characters round enough to avoid that your characters look like cardboards which occurs when the roundness of a character drops below 20%. This occurs when using telephoto-lenses where a lot of depth-information is eaten up by blank space. One way would be using a wide angle, but a better way it using multiple camera rigs, one for the foreground with all the necessary depth information and one for the background. So you can skip the blank space between character and background and use this range to enhance the roundness instead.

Disney used all these techniques for their upcoming feature BOLT and from the very raw work-in-progress material that was screened I can say that it works marvelous.

French Stench

Sorry about that headline, I just like cheap rhymes (but no hip-hop!). The presentation titled Effects Tailored to a Director’s Vision by Pierre Buffin and Francois Aubaque from BUF Compagnie sounded rather interesting, because BUF use only proprietary software since they started in the eighties and have grown in technology as well as in style and expertise. They did the effects on Fight Club, Panic Room, Silent Hill, The Prestige and any Astérix and a heap of others. So I was quite thrilled to learn about their work with the directors such as David Fincher.

As it turned out they were talking English with such a terrible French accent that it was hard to make out what they were talking about. But instead of telling much they just rolled making-of after making-of the different productions and just said a few words between them. I was quite disappointed because I can watch all that making-of-stuff alone on any special-edition DVD of the given movie (even without the accompanying French songs). So I left after twenty minutes for another caffeine flash to get ready for Carlos Baena’s much anticipated Realistic vs. Stylized Human Animation presentation.

He showed his work on Star Wars Episode II as example for realistic animation and some shots of Skinner in Ratatouille. The shots he worked on are also on his website. He also showed a short animated clip from the Iron Man Trailer, the shot with the iron man’s reaction as the tank misses him: He dodges, leans back more than he had anticipated so he has to take a step back to keep the balance. He lifts his arms, looks back where the shot came from and shoots back from the gun on his arm. The rebound travels up his arm and reaches his head an chest after a couple of frames, so he shakes a bit, the left arm extended into the other direction to keep balance. Now was that a thorough description of a little animation or what?

Then he showed some reference footage he uses as an animator, a DVD with slowmotion footage from the Olympics in Japan some time ago where you can see most of the actions from the side or the top. Some other references are x-ray films, e.g. of a man raising his arm.

He looks into the audience and continues honestly.

He showed some more footage in slow-mo, of a fist hitting on a table, where you could see the muscles and the skin ripple on the impact. That’s the stuff you should be watching all the time when becoming a character animator!

In terms of stylized character animation Carlos dug up some early tests he did on Skinner, the cook from Ratatouille who was based largely on Louis de Funés, my favorite French comedian. Just watch his performance in Jo and you’ll see why the folks at Pixar decided that Skinner became too French.

Unfortunately the amount of coffee I had pumped into my veins that day made me rather immune to any more caffeine so I had troubles staying awake during the whole presentation. When Carlos was talking something about Woody’s Arms in Toy Story I drifted away for a couple of seconds, or a couple of minutes — I don’t know.

The presentation titled Bringing Nuke and Maya Together by Andreas Frickinger and Daniel Hasenbring from Miromar was held in a way too small room for way too many interested people so I was among those who sat on the floor, when the two Germans showed their tricks of rendering a Mental Ray mia_matrial in a crapload of passes and having a script rebuild the shader in Nuke again, where you can tweak on every nut if you ever wanted to.

The most stunning part, however, was the possibility to re-light your rendered scene within Nuke because their pipeline also exports UV and World coordinates. The only drawback is that your Nuke-lights will not cast any shadows but for subtle re-lighting it’s perfect.

Even some of the scene geometry can be viewed in Nuke’s 3D view — it’s crazy how such an amazing workflow-demonstration came off like a high-school presentation of two totally not entertaining people that seemed not too well prepared either.

Afterwards I talked with Kerschy a little who also witnessed the presentation. He was so amazed about the whole topic that e decided to write his diploma thesis about how to get from Maya to Nuke like the guys from Miromar.

What remains as item on a list of impressions:

Morning Impressions,
originally uploaded by Phil Strahl.

• My first real meal this week, enjoyed at one of the local McSteakhouses, Maredo.
• Strolling around the heart of Stuttgart without knowing a place to go.
• My visit to the Schräglage-lounge with lots of goofiness, caffeine and laughter.
• Another day without warm water at the hotel.

What I have learned today

• That you can’t cheat when working in stereo.
• That zooming is not possible in live-action stereo
• That subpixel-accuracy matters to the tenth of a pixel in stereo
• That matte paintings should be at least in 2.D when working in stereo
• That you can’t paint haze into the matte paintings when working in stereo (= faking it)
• That even the layouting of a stereoscopic shot/scene/film should be done in stereo
• That you should be able to view your dailies the same way your audience will watch the final stereo picture.
• That you should create the same conditions on location for viewing stereo dailies as your final audience will have.
• That camera shakes can be used (moderately!) to hide stereoscopic issues.
• That stereoscopic 3D-rain looks fake in Journey. Really fake, just big blobs dropping.
• That when working in stereo you need to double nearly everything in your production calculation.
• That matchmoving is nearly impossible for live action stereo footage.
• That using a cube for tracking stereo footage is much better than using single tracking marks without any depth information.
• That you should collect as much meta data from the cameras as possible when working in stereo.
• That distortions in 3D share the same center, which lies between the “eyes”.
• That lens artifacts should apply to both eyes. If they don’t match make them match.
• That a composition for stereo shot should be done for both eyes in one composition.
• Stereoscopic depth should be kept persistent throughout a sequence.
• That you should use displacement maps instead of bump-maps for stereo pictures.
• That you just can not cheat when working in stereo!
• That you shouldn’t use stereo for gimmicks
• That you have to make watching stereo as comfortable as possible for the audience.
• That you should avoid “beyond infinity” focusing on your audience in stereo. 15 pixels at 2k are enough!
• That you should use a depth-script when working in stereo.
• That perceived depth can equal emotional depth.
• What a floating window in stereo are and how you can work with them.
• In stereo DOF should only be used for the background, for the foreground deemphasize with lighting.
• That you should use the a different portion of the image circles of parallel cameras in stereo instead of toeing them in.
• That you can counter cardboarding in stereo by using two camera rigs, one for the foreground, one for the background.

1. This question still is a running gag with the folks from the lower semester who witnessed it along with the sharp answer. Looks like I hit a soft spot… [back]
2. RenderMan Scene Description Files [back]
3. from this point on I’ll use the term “stereo” for stereoscopic imagery, both live action and computer generated; “3d” for computer simulated images of digital representation of three dimensional objects and “stereo sound” for audio on two channels each intended for either the left or right ears of the audience. I hope everything is clear now. [back]
4. the distance between the center of the two points of view [back]