[ComfyUI]LTXV:超高效视频模型!仅需4秒生成5秒24帧高质量视频,运动一致性且消除物体变形

科技   2024-11-24 18:07   浙江  

LTXV:超高效视频模型!仅需4秒生成5秒24帧高质量视频,运动一致性且消除物体变形

🌹大家好!欢迎来到破狼公众号。感谢大家的支持与鼓励。在AIGC探索道路上,我将与你一路同行。喜欢就星标关注破狼公众号或文末扫码加入交流群 !本人仅运营公众号平台,未经授权严禁CSDN等其他平台抄袭和转载!


LTXV简介

今天介绍一款来自 @Lightricks的开创性视频生成模型LTX Video (LTXV)LTXV是一个仅有20亿参数的基于DiT的视频生成模型,能够实时生成高质量视频。它以768x512的分辨率,每秒24帧的速度生成视频,速度之快 。LTXV在ComfyUI的亮点如下所示:

  1. 1. 实时生成速度LTXV能在短短4秒内制作出5秒的24 FPS视频(768x512分辨率),视频生成速度非常的高效快速LTXV是在不牺牲速度或内存效率的情况下保持精度和视觉质量。针对广泛可用的GPU(如RTX 4090)进行了优化,并利用bfloat16精度高效使用内存,同时不牺牲质量。

  2. 2. 高视频质量LTXV提供了无与伦比的速度,仅用20个扩散步骤,在短短四秒内就制作出五秒的视频(121帧,768x512分辨率) 。其扩散变换器架构确保了平滑的运动,并消除了物体变形等常见问题 ,提供了卓越的运动一致性。此外,该模型高度可扩展,能够生成质量一致的长视频,赋予创作者推动叙事边界的能力。

  3. 3. ComfyUI中的原生支持LTXV 现在在最新的ComfyUI中得到了原生支持。Lightricks还为ComfyUI开发了定制节点。这些节点都可在ComfyUI Manager中直接找到。只需搜索“LTXVideo” !

演示案例

01

A woman with blood on her face and a white tank top...A woman with blood on her face and a white tank top looks down and to her right, then back up as she speaks. She has dark hair pulled back, light skin, and her face and chest are covered in blood. The camera angle is a close-up, focused on the woman's face and upper torso. The lighting is dim and blue-toned, creating a somber and intense atmosphere. The scene appears to be from a movie or TV show.

02

A prison guard unlocks and opens a cell door...A prison guard unlocks and opens a cell door to reveal a young man sitting at a table with a woman. The guard, wearing a dark blue uniform with a badge on his left chest, unlocks the cell door with a key held in his right hand and pulls it open; he has short brown hair, light skin, and a neutral expression. The young man, wearing a black and white striped shirt, sits at a table covered with a white tablecloth, facing the woman; he has short brown hair, light skin, and a neutral expression. The woman, wearing a dark blue shirt, sits opposite the young man, her face turned towards him; she has short blonde hair and light skin. The camera remains stationary, capturing the scene from a medium distance, positioned slightly to the right of the guard. The room is dimly lit, with a single light fixture illuminating the table and the two figures. The walls are made of large, grey concrete blocks, and a metal door is visible in the background. The scene is captured in real-life footage.

03

A woman walks away from a white Jeep parked on a city street at night...A woman walks away from a white Jeep parked on a city street at night, then ascends a staircase and knocks on a door. The woman, wearing a dark jacket and jeans, walks away from the Jeep parked on the left side of the street, her back to the camera; she walks at a steady pace, her arms swinging slightly by her sides; the street is dimly lit, with streetlights casting pools of light on the wet pavement; a man in a dark jacket and jeans walks past the Jeep in the opposite direction; the camera follows the woman from behind as she walks up a set of stairs towards a building with a green door; she reaches the top of the stairs and turns left, continuing to walk towards the building; she reaches the door and knocks on it with her right hand; the camera remains stationary, focused on the doorway; the scene is captured in real-life footage.

04

A man walks towards a window, looks out, and then turns around...A man walks towards a window, looks out, and then turns around. He has short, dark hair, dark skin, and is wearing a brown coat over a red and gray scarf. He walks from left to right towards a window, his gaze fixed on something outside. The camera follows him from behind at a medium distance. The room is brightly lit, with white walls and a large window covered by a white curtain. As he approaches the window, he turns his head slightly to the left, then back to the right. He then turns his entire body to the right, facing the window. The camera remains stationary as he stands in front of the window. The scene is captured in real-life footage.

05

A clear, turquoise river flows through a rocky canyon...A clear, turquoise river flows through a rocky canyon, cascading over a small waterfall and forming a pool of water at the bottom.The river is the main focus of the scene, with its clear water reflecting the surrounding trees and rocks. The canyon walls are steep and rocky, with some vegetation growing on them. The trees are mostly pine trees, with their green needles contrasting with the brown and gray rocks. The overall tone of the scene is one of peace and tranquility.

06

A man in a suit enters a room and speaks to two women...A man in a suit enters a room and speaks to two women sitting on a couch. The man, wearing a dark suit with a gold tie, enters the room from the left and walks towards the center of the frame. He has short gray hair, light skin, and a serious expression. He places his right hand on the back of a chair as he approaches the couch. Two women are seated on a light-colored couch in the background. The woman on the left wears a light blue sweater and has short blonde hair. The woman on the right wears a white sweater and has short blonde hair. The camera remains stationary, focusing on the man as he enters the room. The room is brightly lit, with warm tones reflecting off the walls and furniture. The scene appears to be from a film or television show.

LTXV ComfyUI体验

在最新的ComfyUI官方已经本体支持了,需要更新ComfyUI软件到最新版本即可开始体验。另外还需下载下载对应模型模型也可以文末网盘获取)。

  • • 在线体验:https://fal.ai/models/fal-ai/ltx-video

  • • ltx-video-2b-v0.9.safetensors:下载到models/checkpoints文件夹。下载地址:https://huggingface.co/Lightricks/LTX-Video/tree/main

  • • t5xxl_fp16:下载模型放置在models/clip文件夹中。下载地址:https://huggingface.co/Comfy-Org/stable-diffusion-3.5-fp8/blob/main/text_encoders/t5xxl_fp16.safetensors

  • • 另外,Lightricks官方也提供了ComfyUI插件可使用:ComfyUI-LTXVideo本文更推荐使用ComfyUI官方原生支持)。插件地址:https://github.com/Lightricks/ComfyUI-LTXVideo

Flux文生图工作流

Flux文生图感兴趣的同学可参考LIBLIB在线运行工作流:FLUX[续篇]:12B参数23G最大开源文生图模型,Dev版直出惊艳美图欣赏。本文涉及ComfyUI工作流和模型均可在LIBLIBAI上下载或在线运行体验:

• F.1-绮梦流光-水湄凝香

https://www.liblib.art/modelinfo/134c6dd95aef48e98a22b24e003e026b

• 工作流-Flux文|图生图+LORA+提示反推一键切换工作流

https://www.liblib.art/modelinfo/782aacd70f604da39e83368c696a02a8

另外LIBLIBAI已支持本地客户端使用可首页(https://www.liblib.art)下载体验。

LTXV视频工作流

LTXV工作流已上传LIBLIB平台

https://www.liblib.art/modelinfo/9e6c5b586fdc48bd965a88ca274a4879?versionUuid=1881c6745c4a4b02b8fb065a24aacaad

注意

  • • LTXV 视频生成提示词尽量越长详细越好
  • • LTXV 运行速度非常快,在24G机器上文生视频约15秒图生视频约37秒。并且生成的整体质量非常不错

图生视频耗时:

文生视频耗时:

01. 文生视频-金发女人

A woman with long brown hair and light skin smiles at another woman with long blonde hair. The woman with brown hair wears a black jacket and has a small, barely noticeable mole on her right cheek. The camera angle is a close-up, focused on the woman with brown hair's face. The lighting is warm and natural, likely from the setting sun, casting a soft glow on the scene. The scene appears to be real-life footage.

02.文生视频-瀑布探险

A man stands waist-deep in a crystal-clear mountain pool, his back turned to a massive, thundering waterfall that cascades down jagged cliffs behind him. He wears a dark blue swimming shorts and his muscular back glistens with water droplets. The camera moves in a dynamic circular motion around him, starting from his right side and sweeping left, maintaining a slightly low angle that emphasizes the towering height of the waterfall. As the camera moves, the man slowly turns his head to follow its movement, his expression one of awe as he gazes up at the natural wonder. The waterfall creates a misty atmosphere, with sunlight filtering through the spray to create rainbow refractions. The water churns and ripples around him, reflecting the dramatic landscape. The handheld camera movement adds a subtle shake that enhances the raw, untamed energy of the scene. The lighting is natural and bright, with the sun positioned behind the waterfall, creating a backlit effect that silhouettes the falling water and illuminates the mist.

03.图生视频-海浪

best quality, 4k, HDR, a tracking shot of a beautiful scene of the sea waves on the beach with a massive explosion in the water

04.图生视频-演唱会

A teddy bear in sunglasses playing electric guitar and dancing。This is a digitally created CGI image featuring a stylized, anthropomorphic teddy bear as the central figure. The teddy bear is depicted as a rock musician, wearing a black leather jacket and black sunglasses. The bear's fur is a rich, light brown color, and it is holding a red electric guitar with white accents, which it appears to be strumming. The guitar has a glossy finish and visible strings.The background is a concert stage setting, bathed in warm, ambient lighting that includes spotlights creating a dynamic, energetic atmosphere. The stage is dimly lit with a mix of blue and orange hues, enhancing the dramatic effect. Surrounding the bear are various musical instruments: to the left, there is a microphone stand with a microphone positioned at the top, and to the right, a drum set is partially visible, including a bass drum, snare drum, and cymbals. The overall composition conveys a lively, rock-and-roll concert scene with the teddy bear as the charismatic performer.The CGI style is highly detailed, with realistic textures on the bear's fur and the instruments, creating a visually engaging and immersive experience.

05.图生视频-沙滩假期

感谢本节图例来自群友@夜〃岚√LTXV在视频生成中主体保持非常好,很小几率变形。

A Rabbit in sunglasses playing electric guitar and dancing。The image is a digitally created CGI artwork depicting an anthropomorphic white rabbit character standing on a sunlit beach. The rabbit has large, expressive eyes with purple eyeshadow and wears a pair of round, gold-rimmed glasses. Its fur is smooth and white, contrasting with the colorful background. The rabbit is dressed in a casual, layered outfit consisting of a dark green and blue plaid jacket over a white T-shirt with a black graphic of a bat and the word "VAMPIRE" in purple letters. The T-shirt is tucked into a short, dark gray pleated skirt. In its right hand, the rabbit holds a tall glass of orange juice with a green straw, while its left hand rests in its pocket. The rabbit also carries a white tote bag with a floral design on its left shoulder. In the background, there are blurred figures of people, colorful tents, and umbrellas, suggesting a lively beach festival. The sky is clear with a few scattered clouds, and the overall atmosphere is vibrant and cheerful. The image combines elements of realism and cartoonish stylization, with detailed textures on the rabbit’s fur and clothing.

06.图生视频-沙滩假期

感谢本节图例来自群友@一船清梦压星河

A Rabbit i

网盘模型获取:关注公众号口令【AI-Video-Models】获取。
更多推荐文章:
• [ComfyUI]FluxFill:先进与高效重绘和扩图神器,超越阿里等同类模型
• [ComfyUI]FluxRedux:超好玩创意灵感,一丝小遗憾!F1风格化溶图组件
• 智谱CogVideoX1.5:重大升级,可商用开源模型!10秒&增强质量&任意分辨率
• [ComfyUI]Flux:Lovely网红写真,极致细节写实,小红书网红人物写真风格
• 太酷啦!实时人物表情编辑神器,ComfyUI玩转视频表情无限创意
• PixelWave:更真实细节|动漫|摄影多种艺术风格显著提升,基于黑森林F.1D超5周精调模型
• OmniGen:统一图像生成和多任务集成模型,任意人物自由合影,8位量化体验
 15秒F.1D直出,极限无损加速方案,环境大升级敢不敢来试?
 CogVideo:重磅升级!图生视频完美镜头控制和3D环绕,商用级开源AI视频曙光
• [ComfyUI]InstantIR:来自小红书团队模糊图像修复技术,效果是否惊艳?
感兴趣加入[AGI技术交流群]+V

    如果觉得文章不错,就请在看转发三连

破狼
关注AIGC、LLM、绘图作品、软件工程、技术学习。交流+V:shunshizhiwu。
 最新文章