ComfyUI | OminiControl:Flux全能P图神器,乾坤大挪移&重绘&CN控制。

文摘   2024-12-19 00:00   江苏  

OminiControl:FLUX.1的极简通用控制器

点击关注上方公众号并添加公众号小助手加入官方读者交流群,一个有趣有AI的AIGC公众号:关注AI、深度学习、计算机视觉、AIGC、Stable Diffusion、Sora等相关技术,欢迎一起交流学习💗~


OminiControl简介

今天介绍一款 为Diffusion Transformer 模型(如 FLUX)设计的极简而强大的通用控制框架OminiControl。这是一个高度通用且参数高效的框架,它将图像条件集成到预训练的扩散变换器(DiT)模型中。OminiControl的核心在于利用参数复用机制,使DiT能够使用自身作为强大的主干来编码图像条件,并利用其灵活的多模态注意力处理器处理这些条件。与依赖于复杂架构的额外编码器模块的现有方法不同,OminiControl采用:

  • • 极简设计:在保持原始模型结构的同时注入控制信号,仅引入了模型基础参数的0.1%

  • • 通用控制:以统一的方式解决 包括主题驱动生成和空间对齐条件(如边缘、深度等)在内的广泛图像条件任务

值得注意的是,这些能力是通过在DiT自身生成的图像上训练实现的,这对于主题驱动生成特别有益。广泛的评估表明,OminiControl主题驱动空间对齐条件生成方面均优于现有的基于UNet和DiT适应的模型。此外,研发团队还发布了训练数据集Subjects 200K,这是一个包含超过200,000张身份一致图像的多样化集合,以及一个高效的数据合成管道,以推进主题一致生成的研究。

  • • github:https://github.com/Yuanshi9815/OminiControl

  • • 在线体验:https://huggingface.co/spaces/Yuanshi/OminiControl

  • • 模型权重:https://huggingface.co/Yuanshi/OminiControl/tree/main

演示案例

下面展示的演示案例,全部采用格式:左:条件图像;右:生成图像。

主题驱动演示案例

01

A close up view of this item. It is placed on a wooden table. The background is a dark room, the TV is on, and the screen is showing a cooking show. With text on the screen that reads 'Omini Control!.'

02

A film style shot. On the moon, this item drives across the moon surface. A flag on it reads 'Omini'. The background is that Earth looms large in the foreground.

03

In a Bauhaus style room, this item is placed on a shiny glass table, with a vase of flowers next to it. In the afternoon sun, the shadows of the blinds are cast on the wall.

04

"On the beach, a lady sits under a beach umbrella. She's wearing this shirt and has a big smile on her face, with her surfboard hehind her. The sun is setting in the background. The sky is a beautiful shade of orange and purple."

05.更多主题驱动演示

虚拟试衣

场景变化

空间对控制演示案例

06

he Mona Lisa is wearing a white VR headset with 'Omini' written on it.

07

A yellow book with the word 'OMINI' in large font on the cover. The text 'for FLUX' appears at the bottom.

08

A light gray sofa stands against a white wall, featuring a black and white geometric patterned pillow. A white side table sits next to the sofa, topped with a white adjustable desk lamp and some books. Dark hardwood flooring contrasts with the pale walls and furniture.

图像的线稿,深度,着色,去模糊图像处理:


OminiControl ComfyUI体验

首先需要通过插件管理器git安装ComfyUI_RH_OminiControlComfyUI插件。

  • • ComfyUI_RH_OminiControl插件:https://github.com/HM-RunningHub/ComfyUI_RH_OminiControl
  • • FLUX.1-schnell:需要下载模型并放置 /ComfyUI/models/flux 目录下。FLUX.1-schnell下载地址:https://huggingface.co/black-forest-labs/FLUX.1-schnell/tree/main 。
  • • OminiControl:需要下载模型并放置 /ComfyUI/models/flux 目录下。下载地址:https://huggingface.co/Yuanshi/OminiControl ;
  • • depth-anything-small-hf:还需下载depth-anything-small-hf,并放置目录/ComfyUI/models/flux/OminiControl。下载地址:https://huggingface.co/LiheYoung/depth-anything-small-hf/tree/main
  • • 注意:社区还存在另一款插件ComfyUI-Diffusers-OminiControl,但是作者称需要30+ GB VRAM。而ComfyUI_RH_OminiControl称:通过分拆 pipeline 加载,该插件能够在 NVIDIA RTX 4090 显卡上高效运行。同时,空间(spatial)及填充(fill)功能也通过 schnell 模型生成,减少采样步数,提高整体效率

Flux文生图工作流

Flux文生图感兴趣的同学可参考LIBLIB在线运行工作流:FLUX[续篇]:12B参数23G最大开源文生图模型,Dev版直出惊艳美图欣赏。本文涉及ComfyUI工作流和模型均可在LIBLIBAI上下载或在线运行体验:

• F.1-绮梦流光-水湄凝香

https://www.liblib.art/modelinfo/134c6dd95aef48e98a22b24e003e026b

• 工作流-Flux文|图生图+LORA+提示反推一键切换工作流

https://www.liblib.art/modelinfo/782aacd70f604da39e83368c696a02a8


OminiControl工作流

OminiControl工作流已上传LIBLIB平台

https://www.liblib.art/modelinfo/7e6dae483c884c5e81d3db03d66ce08a?versionUuid=4ccccdba98a444c3b14b9e2ca35fe12c

注意

  • • OminiControl也可以使用在线web体验,地址:https://huggingface.co/spaces/Yuanshi/OminiControl
  • • OminiControl还支持线稿、深度、颜色、虚拟换衣等更多功能。

01. Kitty猫

A very close up view of this item. It is placed on a wooden table. The background is a dark room, the TV is on, and the screen is showing a cooking show. With text on the screen that reads 'Omini Control.AGI‘

02.圣诞节

On Christmas evening, on a crowded sidewalk, this item sits on the road, covered in snow and wearing a Christmas hat, holding a sign that reads 'AGI!'

03.闹钟

In a Bauhaus style room, this item is placed on a shiny glass table, with a vase of flowers next to it. In the afternoon sun, the shadows of the blinds are cast on the wall.

04.线稿出图

1girl

05.颜色主题

blue

感谢你看到这里,也欢迎点击关注下方公众号并添加公众号小助手加入官方读者交流群,一个有趣有AI的AIGC公众号:关注AI、深度学习、计算机视觉、AIGC、Stable Diffusion、Sora等相关技术,欢迎一起交流学习💗~


AIGC Studio
一个有趣有AI的AIGC公众号:关注AI、深度学习、计算机视觉、AIGC、Stable Diffusion、Sora等相关技术。这里不仅有简单易懂的AIGC理论实践和AI学习路线,还有大厂工作经历和体会分享。如果有幸能给你一些帮助就更好啦!
 最新文章