<aside> đź’ˇ Collected below are some rough explorations using in-painting with AnimateDiff to add motion to both stills and video.

These are created in ComfyUI using simple masks, prompts, IP-Adapter, and the in-painting controlnet with AnimateDiff to create motion.

Exciting because this feels like an entirely new way to approach animation and VFX elements.

These tests were created in three days and (generally) render in minutes.

</aside>

Nathan Shipley & Paul Trillo, 11-20-2023 nathanshipley.com paultrillo.com

Video In-Painting Explorations:

Hand

Input RGB + Matte:

WS_PaulHand_Tentacle_01_MatteReference.mp4

Result with fire prompt. ”flames, fire, cinematic, beautiful”

WS_PaulHand_Fire_02.mp4

Result with tentacle prompt and IPAdapter:

WS_PaulHand_Tentacle_03_NewMatte.mp4

Mime

Input RGB + Matte:

Mime_Matte_01.mp4

Result with Water Prompt: ”waterfall, cascading water, rushing water falling, cinematic, beautiful”

Mime_Water_01.mp4

Result with Crystal Column prompt & IPAdapter: ”crystal column, quartz, sparkling, detailed, crystalline structure, cinematic, beautiful”

Mime_CrystalColumn_01.mp4

Mime footage is of the fantastic Derek Loughran.

Tree

Source & Result: