int8 Compute

Dieser Inhalt ist noch nicht in deiner Sprache verfügbar.

ComfyUI’s int8 builds (int8_tensorwise, see Model Format Support) store every large layer as 8-bit integers. By default InvokeAI keeps those weights in int8 and unpacks each layer to BF16 or FP16 just before it runs: you get the memory saving, but each layer runs at 0.75–0.97× the speed of a full-precision one.

int8_compute is a global setting that does the math in int8 too, on the GPU’s int8 tensor cores, for int8 builds quantized with convrot: each layer’s comfy_quant marker says "convrot": true, as in the published builds. Each layer then rotates and quantizes its input to int8 per token and multiplies int8 by int8.

Denoising time on an RTX 4090, BF16, int8 convrot builds fully in VRAM

Run	Default	`int8_compute`	Speedup
Z-Image Turbo, 8 steps, 1024²	5,04 s	2,93 s	1,72×
Z-Image Turbo, 8 steps, 1536²	12,78 s	8,95 s	1,43×
Z-Image Turbo, 8 steps, 2048²	26,62 s	20,71 s	1,29×
FLUX.1 dev, 20 steps, 1024²	15,25 s	7,85 s	1,94×
FLUX.1 dev, 20 steps, 1536²	36,20 s	23,88 s	1,52×
Krea-2 Turbo, 8 steps, 1024²	9,22 s	4,51 s	2,04×
Krea-2 Turbo, 8 steps, 1536²	21,64 s	12,32 s	1,76×
FLUX.3 Klein, 16 steps, 1024²	7,01 s	4,07 s	1,72×

The gain shrinks at higher resolutions because attention, which int8_compute does not touch, takes a larger share of each step.

Partially loaded models gain the most. A layer still in system RAM is copied to the GPU as int8 for each step, half the bytes of the unpacked weight, and computed exactly like a resident one. With the whole transformer streamed (max_cache_vram_gb: 3, 0% in VRAM), the image was bit-identical to the fully resident one:

Denoising time on an RTX 4090, BF16, transformer streamed from RAM

Run	Default	`int8_compute`	Speedup
Z-Image Turbo, 8 steps, 1024²	34,43 s	9,04 s	3,81×

Other int8 families use the same layers but have not been measured. A convrot int8 text encoder (qwen_3_8b_int8_convrot with FLUX.2 Klein 9B) works with it as well; its speed has not been measured.

It is opt-in because quantizing the inputs changes the numbers: images made with int8_compute are not pixel-identical to the default path at the same seed. In the Z-Image, FLUX.1 and Krea-2 measurements above they were visually equivalent (PSNR 30–46dB against the default path). FLUX.3 Klein was compared with bf16 instead: a mean PSNR of 34.5dB with int8_compute and 33.2dB without. Each path is bit-identical from run to run, and how much of the model is in VRAM does not change the image.

Requirements

An NVIDIA GPU with compute capability 8.0 or newer: RTX 30-series, RTX 40-series, RTX 50-series, A100, H100. Measured on an RTX 4090.
Triton. InvokeAI’s NVIDIA install on Windows x64 and Linux includes it (on Windows, the triton-windows package). A manual install on Windows without the cuda extra has none: install the triton-windows version matching InvokeAI’s PyTorch as described under SageAttention.
An int8 build quantized with convrot, as every published build is.

Everything else falls back to the default path automatically: AMD (ROCm), Apple silicon (MPS), Intel (XPU), the CPU, older NVIDIA GPUs, and Windows on ARM64, whose install has no Triton.

Turning it on

Set it in your invokeai.yaml and restart InvokeAI:

int8_compute: true

The first generation that uses it logs, once per GPU:

int8 compute: rotated int8_tensorwise layers on cuda:0 run on the int8 tensor cores.

If a device can’t use it, the log says why, once (for example that Triton is missing: int8_compute is enabled, but not used on <device>: <reason>. int8 layers there dequantize.), and that device keeps the default path. When a model loads with layers it does not cover, the log counts them, for example int8_compute: 2 of 264 int8 layer(s) keep dequantizing per forward (2 marked full_precision_matrix_mult, 0 not rotated in groups of 256).

What stays on the default path

int8_compute only changes how rotated int8 layers multiply. These keep unpacking their weights:

Layer	Why
Quantized without convrot	Without the rotation, one outlier channel sets the scale for a whole token, where per-token quantization loses the most. No such build has been measured.
Marked full precision by the checkpoint (`full_precision_matrix_mult`)	The model’s author flagged it as sensitive.
Rotated in groups other than 256	The kernels are built for 256, which every published build uses.
Running in FP32	The kernels run in BF16 and FP16 only.

LoRAs work as usual: they are applied beside the int8 layer in the compute precision, never quantized, and do not change which path the layer takes. DoRA patches don’t load on int8 builds, with or without int8_compute.

Memory

Once tuned, the kernels never need more working memory than the default path already reserves for unpacking a layer: they quantize the input in slices of at most twice the layer’s output width, and copy a layer still in system RAM to the GPU as int8, at half the size of the unpacked weight. The first call for each layer shape and token-count bucket may briefly need a little more: tuning uses a buffer the size of the GPU’s L2 cache (72MB on an RTX 4090), and the check against the default path afterwards keeps 256 rows of the layer’s output from each path.

When something goes wrong

InvokeAI checks the first result for each layer shape and token-count bucket against the default path. If it does not match, or a kernel fails, the log shows one warning and that GPU uses the default path until restart. Usually the generation continues; a fault that leaves the GPU unusable (such as an illegal memory access) fails it, and the default path cannot recover from that either.

Running out of memory is not caught: once the kernels are tuned, the default path would not fit either. If an out-of-memory error appears only on the first image at a new size, set int8_compute: false to rule the tuning out.