Presentation
"Dear Upstairs Neighbors": Artist-Centered, AI-Assisted Expressionistic Animation
DescriptionA young woman named Ada, desperate for a good night’s sleep, is kept awake by her exceedingly noisy neighbors. As she struggles to imagine what could be causing the cacophony upstairs, reality drifts into fantasy, and an epic battle for peace and sanity ensues.
That’s the story of “Dear Upstairs Neighbors”, an animated short made by a hybrid team of animation veterans and AI researchers.
The artistic core of the film — storyboards, concept art, character animation — was lovingly hand-crafted by artists using traditional 2D and 3D techniques. Our technical artists then created custom generative image and video models, which they fine-tuned on our own concept art and used, much like a renderer, to unite these hand-crafted components into polished shots that respect all of the artistic choices made along the way.
The film’s expressionistic painterly style evolves, shot by shot, to reflect Ada’s intense and changing emotions. To achieve this evolving look by hand would have meant painting over 7,000 frames, with artists spending so much effort on execution that they would have no time to iterate beyond a first pass. To do it using computer graphics would have required developing dozens of custom new stylized rendering algorithms — a feature film’s worth of R&D for a 6-minute short. Not impossible, but far from practical!
Instead, we built artist-driven, AI-assisted video-to-video workflows that let us iteratively refine the look of each shot, homing in on the exact stylistic choices that best suited each moment in the story. That process was itself a direct conversation between artists: our technical artists would render shots, and receive feedback from our director and production designer, often in the form of new paintings, which they would use to improve the fine-tuning for the next iteration.
This video-to-video approach gave us unprecedented control and flexibility. Each artist could work in their comfort zone, using whatever tool was most natural and creative for them. We could also iterate on every shot, switching freely between our generative models and traditional tools like Premiere and After Effects, to bring the shot closer to perfection. Masked generation allowed us to specify exactly where we wanted the models to improvise.
We feel that generative tools can only be truly useful for making animated films when artists are in control, not only as users of the models, but as co-designers. This film was our opportunity to take the wheel, and guide the technology’s development in a more artist-centered direction.
That’s the story of “Dear Upstairs Neighbors”, an animated short made by a hybrid team of animation veterans and AI researchers.
The artistic core of the film — storyboards, concept art, character animation — was lovingly hand-crafted by artists using traditional 2D and 3D techniques. Our technical artists then created custom generative image and video models, which they fine-tuned on our own concept art and used, much like a renderer, to unite these hand-crafted components into polished shots that respect all of the artistic choices made along the way.
The film’s expressionistic painterly style evolves, shot by shot, to reflect Ada’s intense and changing emotions. To achieve this evolving look by hand would have meant painting over 7,000 frames, with artists spending so much effort on execution that they would have no time to iterate beyond a first pass. To do it using computer graphics would have required developing dozens of custom new stylized rendering algorithms — a feature film’s worth of R&D for a 6-minute short. Not impossible, but far from practical!
Instead, we built artist-driven, AI-assisted video-to-video workflows that let us iteratively refine the look of each shot, homing in on the exact stylistic choices that best suited each moment in the story. That process was itself a direct conversation between artists: our technical artists would render shots, and receive feedback from our director and production designer, often in the form of new paintings, which they would use to improve the fine-tuning for the next iteration.
This video-to-video approach gave us unprecedented control and flexibility. Each artist could work in their comfort zone, using whatever tool was most natural and creative for them. We could also iterate on every shot, switching freely between our generative models and traditional tools like Premiere and After Effects, to bring the shot closer to perfection. Masked generation allowed us to specify exactly where we wanted the models to improvise.
We feel that generative tools can only be truly useful for making animated films when artists are in control, not only as users of the models, but as co-designers. This film was our opportunity to take the wheel, and guide the technology’s development in a more artist-centered direction.
Presenters
Event Type
Production Session
TimeThursday, 23 July 202611:15am - 12:15pm PDT
LocationPetree D
Arts & Design
New Technologies
Production & Animation
Animation
Art
Artificial Intelligence/Machine Learning
Generative AI
Full Conference Supporter
Full Conference
