[Feature] Support AnimateDiff, a popular text2animation method

[ElliotQi]

Motivation

Support AnimateDiff in MMagic.

Modification

• Finish the whole pipeline to implement AnimateDiff method.
• Support xformers to save memory which enables us to generate 16 frames 512*512 videos with ~12GB VRAM.
• Improve the quality of 512*512 videos.
• Finish unit tests for modules of AnimateDiff
• Create a Gradio demo to make AnimateDiff easier to use.
• Train a new Motion Module based on Stable Diffusion XL.
• Support video generation based on a picture.

Use cases (Optional)

1. Download ToonYou and MotionModule checkpoint

#!/bin/bash

mkdir models && cd models
mkdir Motion_Module && mkdir DreamBooth_LoRA
gdown 1RqkQuGPaCO5sGZ6V6KZ-jUWmsRu48Kdq -O Motion_Module/
gdown 1ql0g_Ys4UCz2RnokYlBjyOYPbttbIpbu -O models/Motion_Module/
wget https://civitai.com/api/download/models/78775 -P DreamBooth_LoRA/ --content-disposition --no-check-certificate

2. Modify the config file in configs/animatediff/animatediff_ToonYou.py

    models_path = {Your Checkpoints Path}
    motion_module_cfg=dict(
        path={Your MotionModule Path}
    ),
    dream_booth_lora_cfg=dict(
        type='ToonYou',
        path={Your Dreambooth_Lora Path},
        steps=25,
        guidance_scale=7.5)

3. Enjoy Text2Animation world

from mmengine import Config

from mmagic.registry import MODELS
from mmagic.utils import register_all_modules

import os
import torch
from pathlib import Path
import datetime
from mmagic.models.editors.animatediff import save_videos_grid



register_all_modules()

cfg = Config.fromfile('configs/animatediff/animatediff_ToonYou.py')
animatediff = MODELS.build(cfg.model).cuda()
prompts = [
    "best quality, masterpiece, 1girl, looking at viewer, blurry background, upper body, contemporary, dress",

    "masterpiece, best quality, 1girl, solo, cherry blossoms, hanami, pink flower, white flower, spring season, wisteria, petals, flower, plum blossoms, outdoors, falling petals, white hair, black eyes,",

    "best quality, masterpiece, 1boy, formal, abstract, looking at viewer, masculine, marble pattern",

    "best quality, masterpiece, 1girl, cloudy sky, dandelion, contrapposto, alternate hairstyle,"
]

negative_prompts = [
    "",
    "badhandv4,easynegative,ng_deepnegative_v1_75t,verybadimagenegative_v1.3, bad-artist, bad_prompt_version2-neg, teeth",
    "",
    "",
]

sample_idx = 0
random_seeds = cfg.randomness['seed']
random_seeds = [random_seeds] if isinstance(random_seeds, int) else list(random_seeds)
samples = []
time_str = datetime.datetime.now().strftime("%Y-%m-%dT%H-%M-%S")
savedir = f"samples/{Path(cfg.model['dream_booth_lora_cfg']['type']).stem}-{time_str}"
os.makedirs(savedir)
for prompt_idx, (prompt, n_prompt, random_seed) in enumerate(zip(prompts, negative_prompts, random_seeds)):
    output_dict = animatediff.infer(prompt,negative_prompt=n_prompt, video_length=16, height=512, width=512, seed=random_seed,num_inference_steps=cfg.model['dream_booth_lora_cfg']['steps'])
    sample = output_dict['samples']
    prompt = "-".join((prompt.replace("/", "").split(" ")[:10]))
    save_videos_grid(sample, f"{savedir}/sample/{sample_idx}-{prompt}.gif")
    print(f"save to {savedir}/sample/{prompt}.gif")
    samples.append(sample)
    sample_idx += 1

samples = torch.concat(samples)
save_videos_grid(samples, f"{savedir}/sample.gif", n_rows=4)

Checklist

Submitting this pull request means that,

Before PR:

• I have read and followed the workflow indicated in the CONTRIBUTING.md to create this PR.
• Pre-commit or linting tools indicated in CONTRIBUTING.md are used to fix the potential lint issues.
• Bug fixes are covered by unit tests, the case that causes the bug should be added in the unit tests.
• New functionalities are covered by complete unit tests. If not, please add more unit test to ensure the correctness.
• The documentation has been modified accordingly, including docstring or example tutorials.

After PR:

• If the modification has potential influence on downstream or other related projects, this PR should be tested with some of those projects.
• CLA has been signed and all committers have signed the CLA in this PR.

[CLAassistant]

All committers have signed the CLA.

[ElliotQi]

A demo with ToonYou style

• UPDATE: Improve the quality of 512*512 resolution

[LeoXing1996]

Hey @ElliotQi , thanks you for support this COOL algorithm to MMagic.
Please install pre-commit hook via pip install pre-commit before creating commit in order to fix lint error automatically.

Since you have already create a commit and PR, you can use the following commands to roll back:

git reset --soft HEAD^   # reset commit and remain changes
pip install pre-commit   # install pre-commit hook package
pre-commit install  # install pre-commit hook for current repo
git commit -m "xxxxx"  # create commit
git push -f <repository> <refspec>  # -f for force push

Or, you can run the following commands to run pre-commit hook manually, and then add a new commit for format fixing:

pip install pre-commit   # install pre-commit hook package
pre-commit install  # install pre-commit hook for current repo
pre-commit run --all-files  # run pre-commit checking for all file
git add xxx  # add changed files
git commit -m "xxx"  # create new commit
git push <repository> <refspec>  # push to remote

[ElliotQi]

Hi @LeoXing1996, thanks for your reply.
Apart from first commit for animatediff, all the commits including the latest one delete test_animatediff.py are built after pre-commit.
I find that all the lintings are passed in the CircleCI, but the docstring coverage rate doesn't reach the minimum rate(90%).
I'll commit after adding docstrings.

[codecov]

Codecov Report

Patch coverage is 33.77% of modified lines.

❗ Current head af1288d differs from pull request most recent head 9bad0b9. Consider uploading reports for the commit 9bad0b9 to get more accurate results

Files Changed	Coverage
...ic/models/editors/animatediff/animatediff_utils.py	7.37%
mmagic/models/editors/animatediff/animatediff.py	14.82%
mmagic/models/editors/animatediff/unet_3d.py	15.00%
mmagic/models/editors/animatediff/unet_block.py	46.44%
mmagic/models/editors/animatediff/resnet_3d.py	64.06%
mmagic/models/editors/animatediff/attention_3d.py	66.52%
mmagic/models/editors/animatediff/motion_module.py	71.42%
mmagic/models/editors/__init__.py	100.00%
mmagic/models/editors/animatediff/__init__.py	100.00%

📢 Thoughts on this report? Let us know!.

[OpenMMLab-Assistant-004]

Hi @ElliotQi,

We'd like to express our appreciation for your valuable contributions to the mmagic. Your efforts have significantly aided in enhancing the project's quality.
It is our pleasure to invite you to join our community thorugh Discord_Special Interest Group (SIG) channel. This is a great place to share your experiences, discuss ideas, and connect with other like-minded people. To become a part of the SIG channel, send a message to the moderator, OpenMMLab, briefly introduce yourself and mention your open-source contributions in the #introductions channel. Our team will gladly facilitate your entry. We eagerly await your presence. Please follow this link to join us: ​https://discord.gg/UjgXkPWNqA.

If you're on WeChat, we'd also love for you to join our community there. Just add our assistant using the WeChat ID: openmmlabwx. When sending the friend request, remember to include the remark "mmsig + Github ID".

Thanks again for your awesome contribution, and we're excited to have you as part of our community!

[liuwenran]

Thanks for your contribution!

lgtm