Viggle

/

Viggle-Animate

like 116

Follow

Viggle AI 32

you may find more discussion here. it works sometimes, but not stable.

https://www.reddit.com/r/StableDiffusion/comments/1w7b8h9/comment/p7vg8li/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

An example I just ran on minimax H3-max https://fal.ai/models/minimax/h3-max/reference-to-video

https://fal.ai/models/minimax/h3-max/reference-to-video?share=99b9bc28-6de7-4d3e-bc33-814e550430a8

Another example I got before (with prompt enhancement on).

From my personal experience, it's far from satsifying, especially when I directly using the released ref2va ckpt

Good work, character transfer has been an issue with Minimax from the beginning as well as attention issues on background details like hands, mouth and feet etc.

Spotted some flaws, could be a mode limitation or prompt issue?
It seem to struggle with timing with face and hand detail in the video yycc provided:

1. In many of the examples the face doesn't animate much compared to the reference or at all.
   2.also notice in the example of yycc above the Viggle video does not turn his head to look at the camera like person does at 4 seconds and at 5 seconds it does not put the 2 fingers like the person does.
   These are of importance to correct in the future to make it on par with SOTA models.

An example I just ran on minimax H3-max https://fal.ai/models/minimax/h3-max/reference-to-video

https://fal.ai/models/minimax/h3-max/reference-to-video?share=99b9bc28-6de7-4d3e-bc33-814e550430a8

Thanks for the evaluation, it's kinda obvious now original model sometimes drifts away from ref motion, the new fine-tuned model can better preserve the actions.