OminiControl：FLUX.1的极简通用控制器

OminiControl简介

今天介绍一款 为Diffusion Transformer 模型（如 FLUX）设计的极简而强大的通用控制框架：OminiControl。这是一个高度通用且参数高效的框架，它将图像条件集成到预训练的扩散变换器（DiT）模型中。OminiControl的核心在于利用参数复用机制，使DiT能够使用自身作为强大的主干来编码图像条件，并利用其灵活的多模态注意力处理器处理这些条件。与依赖于复杂架构的额外编码器模块的现有方法不同，OminiControl采用：

• • 极简设计：在保持原始模型结构的同时注入控制信号，仅引入了模型基础参数的0.1%

• • 通用控制：以统一的方式解决 包括主题驱动生成和空间对齐条件（如边缘、深度等）在内的广泛图像条件任务

值得注意的是，这些能力是通过在DiT自身生成的图像上训练实现的，这对于主题驱动生成特别有益。广泛的评估表明，OminiControl在主题驱动和空间对齐条件生成方面均优于现有的基于UNet和DiT适应的模型。此外，研发团队还发布了训练数据集Subjects 200K，这是一个包含超过200,000张身份一致图像的多样化集合，以及一个高效的数据合成管道，以推进主题一致生成的研究。

• • github：https://github.com/Yuanshi9815/OminiControl

• • 在线体验：https://huggingface.co/spaces/Yuanshi/OminiControl

• • 模型权重：https://huggingface.co/Yuanshi/OminiControl/tree/main

演示案例

下面展示的演示案例，全部采用格式：左：条件图像；右：生成图像。

主题驱动演示案例：

01

A close up view of this item. It is placed on a wooden table. The background is a dark room, the TV is on, and the screen is showing a cooking show. With text on the screen that reads 'Omini Control!.'

02

A film style shot. On the moon, this item drives across the moon surface. A flag on it reads 'Omini'. The background is that Earth looms large in the foreground.

03

In a Bauhaus style room, this item is placed on a shiny glass table, with a vase of flowers next to it. In the afternoon sun, the shadows of the blinds are cast on the wall.

04

"On the beach, a lady sits under a beach umbrella. She's wearing this shirt and has a big smile on her face, with her surfboard hehind her. The sun is setting in the background. The sky is a beautiful shade of orange and purple."

05.更多主题驱动演示

虚拟试衣：

场景变化：

空间对控制演示案例：

06

he Mona Lisa is wearing a white VR headset with 'Omini' written on it.

07

A yellow book with the word 'OMINI' in large font on the cover. The text 'for FLUX' appears at the bottom.

08

A light gray sofa stands against a white wall, featuring a black and white geometric patterned pillow. A white side table sits next to the sofa, topped with a white adjustable desk lamp and some books. Dark hardwood flooring contrasts with the pale walls and furniture.

OminiControl ComfyUI体验

首先需要通过插件管理器git安装ComfyUI_RH_OminiControlComfyUI插件。

• • ComfyUI_RH_OminiControl插件：https://github.com/HM-RunningHub/ComfyUI_RH_OminiControl
• • FLUX.1-schnell：需要下载模型并放置 /ComfyUI/models/flux 目录下。FLUX.1-schnell下载地址：https://huggingface.co/black-forest-labs/FLUX.1-schnell/tree/main 。
  
• • OminiControl：需要下载模型并放置 /ComfyUI/models/flux 目录下。下载地址：https://huggingface.co/Yuanshi/OminiControl ；
• • depth-anything-small-hf：还需下载depth-anything-small-hf，并放置目录/ComfyUI/models/flux/OminiControl。下载地址：https://huggingface.co/LiheYoung/depth-anything-small-hf/tree/main
  
• • 注意：社区还存在另一款插件ComfyUI-Diffusers-OminiControl，但是作者称需要30+ GB VRAM。而ComfyUI_RH_OminiControl称：通过分拆 pipeline 加载，该插件能够在 NVIDIA RTX 4090 显卡上高效运行。同时，空间（spatial）及填充（fill）功能也通过 schnell 模型生成，减少采样步数，提高整体效率。

Flux文生图工作流

Flux文生图感兴趣的同学可参考LIBLIB在线运行工作流：FLUX[续篇]：12B参数23G最大开源文生图模型，Dev版直出惊艳美图欣赏。本文涉及ComfyUI工作流和模型均可在LIBLIBAI上下载或在线运行体验：

• F.1-绮梦流光-水湄凝香：

https://www.liblib.art/modelinfo/134c6dd95aef48e98a22b24e003e026b

• 工作流-Flux文|图生图+LORA+提示反推一键切换工作流：

https://www.liblib.art/modelinfo/782aacd70f604da39e83368c696a02a8

OminiControl工作流

OminiControl工作流已上传LIBLIB平台：

https://www.liblib.art/modelinfo/7e6dae483c884c5e81d3db03d66ce08a?versionUuid=4ccccdba98a444c3b14b9e2ca35fe12c

注意：

• • OminiControl也可以使用在线web体验，地址：https://huggingface.co/spaces/Yuanshi/OminiControl
• • OminiControl还支持线稿、深度、颜色、虚拟换衣等更多功能。

01. Kitty猫

A very close up view of this item. It is placed on a wooden table. The background is a dark room, the TV is on, and the screen is showing a cooking show. With text on the screen that reads 'Omini Control.AGI‘

02.圣诞节

On Christmas evening, on a crowded sidewalk, this item sits on the road, covered in snow and wearing a Christmas hat, holding a sign that reads 'AGI!'

03.闹钟

In a Bauhaus style room, this item is placed on a shiny glass table, with a vase of flowers next to it. In the afternoon sun, the shadows of the blinds are cast on the wall.

04.线稿出图

1girl

05.颜色主题

blue