AI in animation

This last week I’ve been at the Manchester Animation Festival - a week-long festival for industry, education, and public alike. The panels and short films lit my mind up with an inspiration and drive that I haven’t felt in a while.

On the Industry Day, one of the panels I’d most anticipated was about AI: the fears, the predictions, the ways we’re already using it to improve our work… I’m certainly using AI tools in my photo editing, and as a base for subtitling any fast turnaround work for YouTube etc

For my money, a lot of the practical uses will end up being based on personal/client preferences, past the obvious ethical and legal conversations that are already taking place.

Take voice acting. When we change a line / scene in a script, we usually scratch in the dialogue by getting someone in the studio to record it. The actor then records the line at their next session, or ADRs it if it has already been animated to the scratch.

The current ethical debate, prompted by existing “machine learning” in which AI systems have been “trained” on an actor’s voice and then prompted to generate new speech from a written prompt, is “we can replace that by training an AI on the actor’s voice” vs “actors should be paid, and given the chance to produce the most authentic work”.

Obviously I agree with the latter- audiences deserve the authenticity, and suggesting that AI can replace their training and talent is horrible. But. How about using the AI as a temporary placeholder, instead of the scratch of a completely different voice just jumping in?

+ you get a smoother viewing experience

- the output is misrepresenting what you actually have “in the can” (burn-ins are easily overlooked)

+ it joins up more neatly, and you can judge timing better with a more accurate placeholder

+ if animation for the shot must happen before the actor is next available, a smooth read in the correct cadence and pronunciation will help lipsync

- the actor may be unnerved by hearing their own voice at ADR/ sense the uncanny valley more readily

… all of this assumes that the technology allows for cadence, tone etc. But existing technology such as VocAlign/ Revoice Pro (where you can match an audio clip by the actor to a reading by anyone else) take us a long way towards that already. We can use a written “find” command to find words or syllables in an audio library in Avid’s PhraseFind or DaVinci Resolve Studio. I feel confident that I could get some way towards a useable voice performance… if given enough time. If AI can be used as a tool to cut out that time, and present options to a human, it could be very useful indeed. IF it is desired.

It’s going to be an interesting time. But whether people currently view it as the dawn of a new era or the opening of a new Pandora’s Box… AI is happening. It’s worth being part of the conversation.

Progression of a timeline: animatic

For several years now, the Twitter editing community have been doing #TimelineTuesday - in which you screengrab the timeline of the project you're currently working on, and share it with the other editors. To us, it's an interesting look at the way other people set up their timelines, and some pretty big editors have joined in in the past, sharing various reels of major feature films with varying degrees of VFX and sound work.

I appear to be one of the few people sharing animation timelines, and they're quite different to what most other people are used to seeing. Indeed, they're quite different to my own live-action timelines (a comparison briefly discussed as part of Avid's own #TimelineTuesday series).

So, I'm going to show how I get there. What the various stages are, and a brief explanation where possible of what's changed and why. A previous blog post has a flow chart which shows the 'typical' route through the edit of an episode, for reference to how each part sits in the whole.

I'm using an episode from the series I'm currently editing as an example, and this post covers the animatic stage of production (for additional information see "editing an animatic", a previous blog post based on a different series). For animatics, I use Adobe's Premiere Pro.
Read more

An Animation Editorial Workflow

I put this together pretty quickly for the internal project wiki at work as a basic overview of what goes in and out of my remit as editor on a 52 x 11 min animated series, where within the process sequence reviews typically happen, and where things come from/ go to.

It's not as entirely linear as this in practice (locks happen before all of the animation is done, for example - so that any extensions can be accommodated), a lot of things end up happening as simultaneously, and I'm typically working on anywhere between 5-26 episodes at a time... but this is the basic system.

I'm sharing it here because I get asked a lot where the editing happens in animation ("surely the work's all done by the time you arrive on the scene?"). Answer: everywhere. I haven't even included the script changes and pickups/ ADR here. They basically get added throughout, ideally less frequently as time goes on - the further we are in an episode; the more people are affected if something needs to be changed.

Read more

The Making of Q Pootle 5

Q Pootle 5 was the animated television series I worked on over 2012-13. This 'making of' video was released on one of the series DVDs, and gives a basic overview of the process of creating an animated series: from concept art to foley and compositing.

 



For additional information on the editing side of things running through animatic, animation, and delivery, I wrote a series of blog posts while working on this series, which are linked to from a roundup post which includes examples of how the edit may change throughout production.

A note on notes (from Creativity Inc)

If you haven't read Creativity Inc, on the formation and rise of Pixar by Ed Catmull (one of its founders); you're missing out. As well as a business biography, there are a huge number of philosophical theories on how to best nurture creativity in the film-making process, and how to bring all employees together to make the best product possible.

My Kindle informs me that I've highlighted 35 sections in this book - for non-technical manuals, I usually will make around 2 maximum per book. So, it's fair to say that a lot of it struck a chord.

I'd like to share a short passage on giving meaningful feedback, which applies to many stages in the film-making process, and certainly not just animation. As an editor, I receive a lot of notes - and sometimes send them out, either on cuts I've made or on scripts and films that other people I know are working on.

A good note says what is wrong, what is missing, what isn’t clear, what makes no sense. A good note is offered at a timely moment, not too late to fix the problem. A good note doesn’t make demands; it doesn’t even have to include a proposed fix. But if it does, that fix is offered only to illustrate a potential solution, not to prescribe an answer. Most of all, though, a good note is specific. “I’m writhing with boredom,” is not a good note.

Postchat: Animation and the Post Process

Postchat is a weekly discussion on issues surrounding post-production amongst the Twitter community of which I am a part.

This week I was asked to be featured during a discussion on animation editing. I've summarised the proceedings before and tried to link questions and answers together - although at the time a lot of conversations were occurring in parallel, with diversions - and I went back a few times to questions asked earlier. For a full transcript, see this Storify.

Read more

Editing Animation: a roundup

In drama, it's said that you write a film 3 times: in the script, during filming, and in the edit room.

In animation, there's less filming ("performance" being shared between the voice actors and the animators)... but the editing is constant. The bulk of the editing happens at the start with the animatic, to find the structure of the film in order to eliminate as much unnecessary animation as possible - but as additional detail is added, the edit must be revisited again and again.

The easiest way to illustrate this is to show you two simple sequences from Q Pootle 5, the preschool series I've just finished editing. Even though we used Redboard for the storyboard and the scenes were set up in 3D space with representative backgrounds, things can still change significantly. And, of course, it's still incredibly basic when compared to an animated shot.

Each video shows the animatic more or less as it went into the animation stage (albeit with notes from the producers), a playblast of the animation stage before it went to lighting, and the final version. I've included the full length of each shot from each phase, so there are black sections where there is no video to cover that part of the shot for that stage.

N.B. The animation stage shown is the trimmed version that was sent for animation approval - not the full length of each shot worked up which may have had overlapping actions or excess frames. It matches closely to the final version in terms of action, but the difference lies in the level of detail and any additional fine cutting. A useful additional stage would have been the blocking or early animation - but this wasn't available when I came to put this blog post together.

Example 1: from "Pootle's New Spaceship"

Upper left: Animatic, Lower left: Animation (playblast), Lower right: Final version (as broadcast)
Audio left: Final (as broadcast), Audio right: Temp for animatic

0:00 - Change of angle, changed early in the animation process to better view both characters

0:11 - Action changed to Ray (the bird) closing the nose cone rather than appearing, but the 'beat' of the action remains the same.

0:12 - In the animatic, I wobbled the image to indicate Pootle (green) and Stella (brown) climbing in to the spaceship. This action was eventually replaced by Ray's reaction and the sound effect.

0:23 - Engine start time decided in animation, but the false start matches timing. 

0:24 - Pootle's blustering was cut down in response to animatic feedback from the producers. The shot duration changed accordingly.

0:30 - Hugely different angle, hence different duration. The new angle allowed the point of the shot to be conveyed a lot more quickly.

0:36 - Filling in the animation and letting the animator work the shot as they wished resulted in different timing for this shot, with a more tentative lift-off.

0:40 - A much more cinematic feel for this shot than depicted in the board. Lovely - and a brilliant case for collaboration and encouraging everyone to not be afraid to try things out that are different to what's already there.

0:50 - It was decided after the animatic that Pootle should feel less hesitant about the instructions he's been given, and we shouldn't draw quite so much attention to how bizarre it all seems (in case he doesn't seem in control of the vehicle - preschool audiences aren't as appreciative of mild peril as older children!)

1.01 - End of scene trimmed. The starts and ends of scenes are easy places to add or lose time when cutting to an exact duration.

Angles changing between animatic and animation is a common thing - if there's time and it's a big enough change, new boards will be drawn up to illustrate the intended shot; but often it's quicker and easier for the director to just get it worked up in layout and changed to their specification.

 Example 2: from "The Cosmic Whipple"

Upper right: Animatic, Lower left: Animation (playblast), Lower right: Final version (as broadcast)
Audio left: Final (as broadcast), Audio right: Temp for animatic

0:04 - Adding the rock in the middle of the crater for Oopsy to climb up was a late addition, but added some action to the shot, and was a good opportunity for our most hyperactive character to expend some energy. Of course, this changed the framing a lot from the animatic. And the timing.

0:08 - Blink and you'll miss it: a minor trim to the final at a late stage. A massage of an earlier trim.

0:13 - Shot extended for new action. 

0:20 - There were several shots boarded for this short section of varying duration, but I ended up resizing them in the animatic and merging them at that stage. This accounts for the variation in line thickness and framing.

0:24 - Dead space cleaned up in the fine cut

Often, I would adapt a storyboard to use in an animatic, if it had an element I wanted. This could mean re-sizing, or using a part of a board to replace a part of another board to clarify timing. This often meant a note was added for animatic approval - but I kept my timeline organised to show that it was all part of the same shot, and the metadata reflected that. Sometimes an animatic would end up being the best approximation of the framing from the available boards - I'd put priority on the composition in terms of whether it was a 2-shot, wide, close-up etc.

 

For additional information on the stages noted above, please refer to my previous blog posts on editing animation:

Part 1: Editing an Animatic
Part 2: Editing Animation
Part 3: Editing Animation (the final stages) 

Editing Animation (The Final Stages)

Part 1: Editing an Animatic

Part 2: Editing Animation

 

If you've read the previous parts of this series, this is the part which has the least 'editing' involved - and also the stage where most of the raw materials for the action are together and now fully staged and animated - basically the part where for drama, the editor would start.

However, because we don't have multiple readings from multiple angles and very little in the way of handles, the creative job's basically over. However, there's still work to be done - and elements to be added which will affect the optimal cut point of any two shots.

 

Read more