<aside> 💡 Collected below are some rough explorations using in-painting with AnimateDiff to add motion to both stills and video.

These are created in ComfyUI using simple masks, prompts, IP-Adapter, and the in-painting controlnet with AnimateDiff to create motion.

Exciting because this feels like an entirely new way to approach animation and VFX elements. There are caveats, some things are rough, etc, but it feels quite promising!

These tests were created on Friday through Monday and render in minutes.

</aside>

Nathan Shipley, 11-20-2023 nathanshipley.com twitter.com/citizenplain instagram.com/nathan_shipley_vfx

Some footage & creative input provided by **Paul Trillo.**

Sand

Input RGB & Alpha:

Sand_01_RGB.png

Sand_01_Matte.png

Results:

Sand_to_Sand_01.gif

Water & Octopus variations:

Sand_to_Water_01.gif

Sand_to_Octopus_02.gif

Video In-Painting Explorations:

Hand

Input RGB + Matte:

WS_PaulHand_Tentacle_01_MatteReference.mp4

Result with fire prompt:

WS_PaulHand_Fire_02.mp4

Result with tentacle prompt:

WS_PaulHand_Tentacle_03_NewMatte.mp4

Mime

Input RGB + Matte:

Mime_Matte_01.mp4