Quantising to int8 for the download saving, then shipping cut-outs with stepped hair edges nobody traced back to the model.
AI and on-device processing
Model quantisation
Also called quantization, int8 quantisation or fp16 model. Here is what it means, when it changes what you export, and what people get wrong.
AI and on-device processingRuns on your deviceUpdated 10 August 2026
The short answer
Quantisation stores and computes a model's weights at lower numeric precision, typically dropping from 32-bit floats to 16-bit floats or 8-bit integers. The file gets smaller and memory traffic falls, so the model downloads faster and runs on weaker hardware, at some cost in accuracy.
Updated
The short version
Model quantisation at a glance
| Quick fact | Detail |
|---|---|
| fp32 | 4 bytes per weight, the baseline |
| fp16 | 2 bytes per weight, half the size |
| int8 | 1 byte per weight, a quarter the size |
| Safer choice for imaging | fp16 |
| Most at risk | Soft edges, fine alpha detail |
| Two methods | Post-training, or quantisation-aware training |
Slide the table sideways to see every column.
What do you gain at each precision level?
The arithmetic is simple: precision is bytes per weight, and bytes per weight is download size and memory footprint.
| Precision | Bytes per weight | Practical effect |
|---|---|---|
| fp32 | 4 | Baseline quality, largest file |
| fp16 | 2 | Half the size, difference usually hard to see |
| int8 | 1 | Quarter the size, visible cost on soft edges |
Speed does not scale as neatly as size. It depends on whether the hardware has fast paths for that precision, and on whether the runtime can keep values in the lower format throughout rather than converting back and forth around unsupported operators.
Why does int8 hurt edge quality more than you would expect?
Because of where the error lands. Integer quantisation maps a continuous range onto 256 levels, so every value gets nudged to the nearest step, and those nudges accumulate through the network.
For a classifier that mostly does not matter, since the winning label rarely changes. For segmentation and matting it matters a great deal, because the output you care about is a smooth gradient of confidence or alpha across a boundary a few pixels wide. Coarsen that gradient and edges gain a stepped, slightly hard quality, or a faint band of the old background survives. The damage concentrates exactly on the hair, fur and soft edges that people zoom in on. For image work, fp16 is usually the safer trade: most of the size saving, very little of the visible cost.
What is the difference between post-training and quantisation-aware?
Post-training quantisation takes a finished model and rewrites its weights at lower precision. It is quick, needs no training infrastructure, and usually needs a small calibration set of representative images to choose sensible value ranges per layer. It is what most shipped models use.
The middle option is mixed precision: quantise most of the network aggressively but keep the sensitive layers, typically the first and last, at higher precision. It recovers much of the quality for most of the saving.
Quantisation-aware training simulates the rounding during training so the network learns weights that tolerate it. It gives the best int8 results by a clear margin, but it costs a training run and access to the data, so it is only worth it when int8 is genuinely required.
What people get wrong about model quantisation
Each one is a real failure mode, not a style preference.
Calibrating on unrepresentative images, which sets value ranges that clip on the photos users actually upload.
Assuming a smaller file always runs faster, when a runtime lacking fast int8 paths may convert back and lose the benefit.
Judging quantisation quality on a thumbnail rather than at full zoom on a boundary, where the difference actually appears.
Questions people ask
Model quantisation, answered
The follow-up questions people search for once they have the definition.
Will I actually see the difference between fp32 and fp16?
For most image models, no. Half precision keeps enough range and resolution for the values a vision network works with, and side-by-side outputs are usually indistinguishable. It is the standard default for shipping a model to a browser or a phone for exactly that reason.
Does quantisation make the model faster or just smaller?
Smaller is guaranteed, faster is not. Speed improves when the hardware has native support for the lower precision and the runtime can stay in that format throughout. Without both, you keep the download and memory saving while the compute time barely changes.
Is quantisation the same as pruning or distillation?
No. Quantisation lowers the precision of every weight. Pruning removes weights or whole channels entirely. Distillation trains a smaller model to imitate a bigger one. They tackle the same problem from different angles and are often combined.
Can I quantise an already-exported model?
Yes, that is exactly what post-training quantisation is for. It operates on the exported file, needs only a handful of representative images for calibration, and requires no access to the original training data or pipeline. Always compare outputs before and after on real photos.
Same cluster
More from the glossary
Neighbouring entries, so every term in the set is one hop from every other.
From the studio
Tools that do this
Each one runs on your own device, so the photo stays where it is.
Keep reading
Elsewhere on Photomop
Platform image rules
Image size, file type and background rules for 22 selling platforms, each one sourced.
Read itConvert image
JPG, PNG, WebP and AVIF, with an honest answer about HEIC and alpha.
Read itAll the tools
The whole toolbox in three families, with what each one needs before it can run.
Read itOn-device privacy
The pipeline in five steps, what leaves your device, and how to check it yourself.
Read it
Now do it to a photo
Photomop is a photo studio that runs on your own device. Resize, crop, compress, convert, batch edit, remove a background and change one all work in the browser tab you are reading this in, at full resolution and with no watermark. The on-device page shows you how to check that the photo stays put.