Splitting LTX-2.5 text encoding: 21 minutes to 37 seconds

780mcomfyuivideo-generation

Two luminous paths racing through darkness, with one accelerating into a bright amber streak

The LTX-2.5 setup on this 780M box has one cost I kept paying. Text encoding runs on the CPU, and a fresh prompt takes about twenty minutes before there’s anything to look at. The reason is memory. Gemma-4-12B is 8.9GB, the diffusion model is 9.1GB, and the iGPU gets 14GB, so one of them has to go somewhere else.

Put Gemma-4-12B on the GPU and it finishes in well under a minute. It sits on the CPU so the diffusion model can have the VRAM.

Which raises the question of why they need to be in memory together at all. Encoding a prompt doesn’t touch the diffusion model. If some other program did the encode, wrote the result to disk, and exited, the encoder would have the whole 14GB to itself and ComfyUI would never see it.

A partly finished text encoding moves from sd.cpp to ComfyUI. ComfyUI runs the remaining text-path stage before sampling.

Why not just use the other program for everything

stable-diffusion.cpp runs LTX-2.5 end to end, so the obvious move is to drop ComfyUI entirely. I measured that first. Sampling on its Vulkan backend took about 213 seconds per step against ComfyUI’s 46, so a full clip would have gone from a quarter of an hour to well over an hour.

The gap is backend maturity. ComfyUI on this box runs PyTorch through TheRock’s ROCm build, where matmul and attention land in rocBLAS and MIOpen, both tuned for gfx1103 specifically. ggml-hip is a smaller set of hand-ported kernels validated mostly on other targets.

I tried building sd.cpp against ROCm to see whether that closed the gap. The build succeeds and the binary detects the GPU correctly, printing AMD Radeon 780M Graphics, gfx1103, Wave Size: 32, VRAM: 14143 MiB. Then it segfaults the moment the compute graph runs, at the same point every time, whatever the resolution and whether or not flash attention is on. Vulkan is the only backend that runs here, and it’s the slow one.

So sd.cpp does the encode, ComfyUI does the sampling, and neither one has to give up what it’s good at.

The patch

sd.cpp has no encode-only mode. I checked both the CLI and the HTTP server, and every path runs a full generation through to a saved file. The addition is a --dump-conditioning <path> flag that exits right after get_learned_condition, writing the tensor as safetensors. ComfyUI-LTXVideo already ships LTXVLoadConditioning, a caching node for this tensor format. The metadata handoff uses the small loader patch described below.

write_safetensors_file() couldn’t emit a __metadata__ block, which the format needs, so that grew an optional parameter too.

Because the flag exits before the diffusion model or the VAE ever load, the encoder gets the full 14GB and runs on the GPU. That’s where most of the speedup comes from.

Reproduction details

Everything below was measured against these revisions. LTX-2.5 support is unmerged upstream, so sd.cpp here is pwilkin’s branch.

stable-diffusion.cpp  pwilkin/stable-diffusion.cpp @ 1b75452 (ltx-2-5-support)
ComfyUI               8b2d2917
ComfyUI-LTXVideo      ac4d998

Two patches of mine sit on top of those, and both are needed. --dump-conditioning isn’t in pwilkin’s branch; that one is here, 114 lines across 8 files, applying to 1b75452 with git apply.

LTXVLoadConditioning at ac4d998 builds its options dictionary from tensors alone and never opens the file’s metadata block, so the unprocessed_ltxav_embeds flag written there goes nowhere, the connectors get skipped, and you land straight in the brown noise described below. That patch is here and applies to ac4d998. It comes from the earlier post, where the same flag was getting dropped on its way through ComfyUI’s own save and load nodes.

Building sd.cpp needs VULKAN_SDK set. The installer writes it as a machine-level variable, so a shell opened before the install won’t have it and CMake stops with Could NOT find Vulkan (missing: Vulkan_LIBRARY Vulkan_INCLUDE_DIR glslc). Either export it or pass the three paths directly:

cmake .. -G "Visual Studio 17 2022" -A x64 -DSD_VULKAN=ON \
  -DVulkan_INCLUDE_DIR="$VULKAN_SDK/Include" \
  -DVulkan_LIBRARY="$VULKAN_SDK/Lib/vulkan-1.lib" \
  -DVulkan_GLSLC_EXECUTABLE="$VULKAN_SDK/Bin/glslc.exe"

Command lines below are Git Bash, which is where I ran them. In PowerShell the backslash continuations need to become backticks, or the whole thing goes on one line.

The encode, which writes the conditioning file:

sd-cli.exe --mode vid_gen \
  --diffusion-model ltx-2.5-Q3_K_S.gguf \
  --llm gemma4-12b-with-proj-ltx-2.5-Q5_K_M.gguf \
  --vae ltx-2.5-video-vae-conv-bf16.safetensors \
  --audio-vae ltx-2.5-audio-vae-bf16.safetensors \
  -p "<prompt>" -W 768 -H 512 --video-frames 96 --steps 20 \
  --cfg-scale 7.0 --rng cpu --seed 42 \
  --dump-conditioning cond.safetensors

Copy cond.safetensors into ComfyUI/models/embeddings/ and load it with LTXVLoadConditioning. The -W, -H, --video-frames and --steps values don’t affect the tensor, which is text only, so the same file works at any output geometry.

For the 768x512, 89-frame run measured below, this launch line gave the shortest total time:

python main.py --listen 0.0.0.0 --port 8188 \
  --use-pytorch-cross-attention --reserve-vram 0 --disable-smart-memory --cpu-vae

At 512x288 drop --cpu-vae and let the GPU decode.

The wrong VAE, again

First run failed validation before it got anywhere:

VAE tensor 'first_stage_model.decoder.conv_in.conv.bias' not in model metadata

ltx_vae.hpp expects Diffusers-style names with a .conv. segment. The -bf16 VAE file uses flat names, decoder.conv_in.weight, 396 tensors. The -conv-bf16 file uses the layout the code wants, 170 tensors. This is the same mixup between the diffusion decoder and the convolutional one from the earlier post, biting a second time in a different runtime, and the fix is the same: point at the other file.

Output that decodes to brown noise

The dump then worked. The tensor came out at 6144 wide, which is the number LTX-2.5 wants (4096 for video plus 2048 for audio). It loaded into ComfyUI without complaint. Sampling ran its twenty steps. The video decoded to flat brown static with nothing in it.

No errors anywhere. Correct shape, correct sequence length, no NaNs, sensible min and max.

LTX-2.5’s text path has two stages, and sd.cpp implements one of them. The first is text_embedding_projection, two linear layers called video_aggregate_embed and audio_aggregate_embed whose outputs concatenate to 6144. The second is a pair of small transformers, video_embeddings_connector and audio_embeddings_connector, each two blocks of RoPE self-attention that pad the sequence with 128 learnable register tokens out to at least 1024.

Stage one’s output is already 6144 wide. That’s why nothing complains. av_model.py skips the connectors when the conditioning arrives without an unprocessed_ltxav_embeds flag, and the values that reach cross-attention are meaningless in a way no shape check can see.

Porting the connectors into sd.cpp turned out to be unnecessary. Their weights live in the diffusion checkpoint rather than the text encoder, 258 tensors in ltx-2.5-Q3_K_S.gguf and none in the encoder file, and ComfyUI already runs that stage itself whenever conditioning is marked unprocessed. The dump just has to say so and let ComfyUI finish with the weights it already has loaded.

One more detail. Writing an attention_mask tensor alongside the embeddings crashes cross-attention with expanded size (1024) must match existing size (53). The connector builds its own mask after padding, and a mask sized for the original token count is worse than no mask at all.

What it costs now

StageTime
ComfyUI native encode, CPU20 min 45 s
sd.cpp encode, CPU151 s
sd.cpp encode, GPU37 s

Sampling is untouched by any of this, so the honest way to check the whole pipeline is against the settings the earlier post measured. At 512x288, 97 frames, 8 steps, that post recorded about 5.4 minutes for sampling and decode on banked conditioning. The same geometry here comes in at 4 minutes 18 seconds, of which 3:30 is sampling at 26.31 seconds per step and 48 seconds is decode. The 20% reduction comes from --reserve-vram 0 and --disable-smart-memory, measured below. The encode split changes the text-encoding stage in the table above.

That post also named Q6_K at 1344x768 and 25 steps as its best quality setting without putting a clock on it. On this hardware that geometry runs 1 hour 18 minutes of sampling at 189 seconds per step, plus 8 minutes of decode, for 1 hour 27 minutes. Four seconds of video.

Which points at something that changed. The earlier post found resolution “nearly free”, 7 times the pixels for 60% more time, because generation was dominated by model loading and offload rather than pixel work. With the model fully resident that no longer holds: 1344x768 costs 189 seconds per step against 46 at 768x512, 4.1 times the time for 2.6 times the pixels. Same chip, opposite conclusion, because the memory behaviour underneath it moved.

Two flags, and a fight between them

The launch script had carried --reserve-vram 6 since the first ComfyUI-TheRock setup, which caps what ComfyUI may claim and leaves the rest for everything else on the system. With that reserve in place the model loads partially, 6021MB resident and 3552MB offloaded, at 63.2 seconds per step. Dropping the reserve to zero gives 13065MB usable, the whole model fits, and steps land at 46.1 seconds. Adding --disable-smart-memory takes it to 46.0 and holds it there, which also explains a spread I couldn’t account for at first: the same configuration measured 46 seconds per step one day and 56 the next, because ComfyUI’s memory manager keeps adapting to whatever else is resident.

Then the decode fails.

With the model fully resident, VAE decode at 768x512 and 89 frames dies on a HIP launch failure, twice in a row at exactly the same point, after all twenty sampling steps finish. The logs show why. Asked to load the video VAE, ComfyUI reports Unloaded partially: 5913.34 MB freed, 2374.14 MB remains loaded, and the VAE needs 2770MB on top of that residue plus its compute buffers. --disable-smart-memory makes this worse rather than better, since not releasing VRAM is the thing it exists to do.

Three ways out, all measured on the same clip:

ApproachSamplingDecodeTotal
Keep --reserve-vram 621:001:1722:17
Reserve 0, tiled decode18:392:0120:40
Reserve 0, --disable-smart-memory, --cpu-vae15:193:1618:35

Tiling costs 44 seconds against an untiled GPU decode and showed no seams at 2x2, checked along both tile boundaries. Moving the decode to the CPU costs about two minutes more than the GPU would, and wins anyway because it’s the only option that lets sampling run at full speed. At 512x288 none of this applies and the GPU decodes untroubled in 48 seconds, which is why the fast comparison above uses it.

Three other things I expected to help did nothing.

--use-ck-attention uses Comfy Kitchen’s AMD-tuned kernels, and those kernels are faster per step at comparable offload. It also reserves about 6GB more headroom, which forces partial loading again and nets out slower than plain PyTorch attention at full load. Its memory figures came back byte-identical across two runs on a GPU I’d verified was idle at 469MB, which rules out leftover allocation as the explanation.

--bf16-vae looked free. The VAE file is named -bf16 and gets upcast to fp32 at load time. Forcing bf16 made sampling about 8% slower, which I can’t account for, since the VAE doesn’t run during sampling steps at all.

--force-non-blocking advertises itself for non-Nvidia systems and cost another few percent.

I re-ran the original configuration afterwards and got 46.2 seconds per step against the earlier 46.1, which is what makes the rest of the table worth anything. My first guess for the 46-versus-56 spread was thermal drift, since a long benchmarking session on an iGPU is where that would hide. It was the memory manager, and pinning it with --disable-smart-memory made the number stop moving.

Result

The encode went from twenty minutes to thirty-seven seconds, and the pipeline that gets there runs two different inference engines against the same model files, each doing the part it’s better at. What cost the most time was a model stage one engine doesn’t implement. It fails by handing back a tensor of the right width with nothing in it, so the only check that catches it is decoding the frames and looking at them.