Diffusion model

Running that loop directly on pixels is punishing, because a megapixel image means millions of values passing through the network dozens of times. Latent diffusion adds an autoencoder in front: a compressor that maps the image into a much smaller latent representation, and a decoder that maps it back.

The denoising loop then runs entirely in that compressed space, on a fraction of the values, and only the final latent is decoded into pixels. The saving is large enough that image generation moved from cluster hardware to ordinary consumer GPUs. The cost is that the autoencoder is lossy, which is why very fine texture and small text often come back slightly softened or mangled.

Steps are how many times you run the denoiser. Few steps are fast but leave structure unresolved; more steps refine detail with diminishing returns, and past a point they mostly cost time. The sampler algorithm decides how large a stride each step takes, which is why some samplers reach a usable image in far fewer steps than others.

Guidance controls how hard the model is pushed towards your prompt rather than towards whatever it finds plausible. Low guidance drifts off-brief. High guidance follows the words closely but tends to produce oversaturated colour, harsh contrast and rigid, repetitive composition. The useful setting is a middle range, and it interacts with the number of steps.

A segmentation or matting network runs once. You feed it an image, you get a mask, and the work is one forward pass. A diffusion model runs its network once per sampling step, so the same weights are evaluated dozens of times for a single output image.

The weights are also far larger, and generation needs room for the latents, the text encoder and the decoder at the same time. That combination, many passes over a large model, is why generative editing usually needs a capable GPU or a server, while background removal and upscaling can comfortably run in a browser tab on a laptop.