[ECCV 2024] ControlNet++: Improving Conditional Controls with Efficient Consistency Feedback.
Access rate limit exceeded. Please log in to continue.
ControlNet++: Improving Conditional Controls with Efficient Consistency Feedback (ECCV 2024)
ControlNet++: Improving Conditional Controls with Efficient Consistency Feedback
🕹️ Try and Play with ControlNet++
We provide a demo website for you to play with our ControlNet++ models and generate images interactively. For local running, please run the following command:
git clone https://github.com/liming-ai/ControlNet_Plus_Plus.git
git checkout inference
pip3 install -r requirements.txt
Please download the model weights and put them into each subset of checkpoints:
| model | HF weights🤗 |
|---|---|
| LineArt | model |
| Depth | model |
| Segmentation | model |
| Hed (SoftEdge) | model |
| Canny | model |
And then run:
python3 app.py
What's new for ControlNet++?
✨ Cycle Consistency for Conditional Generation
We model image-based controllable generation as an image translation task from input conditional controls cvc_v to output generated images x0′x'_0. If we translate images from one domain to the other (condition cvc_v → generated image x0′x'_0 ), and back again (generated image x0′x'_0 → condition cv′c_v' ) we should arrive where we started (cvc_v = cv′c_v'). Hence, we can directly optimize the cycle consistency loss for better controllability.
✨ Directly Optimization for Controllability:
(a) Existing methods achieve implicit controllability by introducing imagebased conditional control cvc_v into the denoising process of diffusion models, with the guidance of latent-space denoising loss. (b) We utilize discriminative reward models DD to explicitly optimize the controllability of GG via pixel-level cycle consistency loss.
✨ Efficient Reward Strategy:
(a) Pipeline of default reward fine-tuning strategy. Reward fine-tuning requires sampling all the way to the full image. Such a method needs to keep all gradients for each timestep and the memory required is unbearable by current GPUs. (b) We add a small noise (t≤tthret ≤ t_{thre}) to disturb the consistency between input images and conditions, then the single-step denoised image can be directly used for efficient reward fine-tuning.
🔥 Better Controllability Than Existing Methods (Qualitative Results):
🔥 Better Controllability Than Existing Methods (Quantitative Results):
For a deep dive into our analyses, discussions, and evaluations, check out our paper.
License
This project is licensed under the MIT License - see the LICENSE file for details.
Citation
If our work assists your research, feel free to give us a star ⭐ or cite us using:
@inproceedings{controlnet_plus_plus,
author = {Ming Li, Taojiannan Yang, Huafeng Kuang, Jie Wu, Zhaoning Wang, Xuefeng Xiao, Chen Chen},
title = {ControlNet++: Improving Conditional Controls with Efficient Consistency Feedback},
booktitle = {European Conference on Computer Vision (ECCV)},
year = {2024},
}
Introduction
[ECCV 2024] ControlNet++: Improving Conditional Controls with Efficient Consistency Feedback.
Customize your domain