Quick answer: how INT8 calibration works
Post-training INT8 quantization converts a trained FP32 model to 8-bit integers without retraining. TensorRT runs your calibration images through the network, records the range of values each layer's activations take, and chooses a scale that maps that range onto 8 bits. The scales are saved in a calibration cache and baked into the engine.
| Step | What happens | What can go wrong |
|---|---|---|
| 1. Calibration set | You supply representative images | Images don't match production (lighting, camera, objects) |
| 2. Preprocessing | Images resized and normalised like inference | Different resize, colour order or scaling than at runtime |
| 3. Calibration | TensorRT collects activation ranges | Too few images; rare classes missing |
| 4. Engine build | Scales applied, layers fused, kernels chosen | Sensitive layers lose precision |
| 5. Validation | INT8 mAP compared with FP16 | Skipped, so the accuracy loss is found in the field |
On Jetson Orin, INT8 typically runs around 1.5x faster than FP16 for YOLO, with a small mAP cost when calibration is done well.
Step 1: build a calibration set
The calibration set decides the result, so it deserves more care than the build command:
- Take the images from your production cameras, not from a public dataset, wherever possible. - Cover the real range of conditions: lighting changes, shifts, backgrounds, motion blur, and both crowded and empty scenes. - Include every class, especially rare and small objects. If a class is missing from calibration, its activations may be clipped. - Use several hundred images. Far fewer risks unstable scales; many thousands mostly adds build time. - Keep calibration images separate from the validation set you will use to measure accuracy.
Labels are not needed for calibration itself, only images. You do need a labelled validation set for Step 4.
Step 2, route A: Ultralytics export (quickest)
For YOLO11 and YOLOv8, the Ultralytics exporter handles calibration for you. Point it at a dataset YAML whose images represent production, and run it on the target device:
# FP16 baseline engine
yolo export model=yolo11s.pt format=engine half=True imgsz=640 device=0
# INT8 engine calibrated on the images listed in your dataset YAML
yolo export model=yolo11s.pt format=engine int8=True data=calib_dataset.yaml imgsz=640 device=0The exporter applies the same letterbox preprocessing it uses at inference, which removes the most common source of calibration mistakes. Its fraction argument limits how much of the dataset is used for calibration if the set is large. Rename the output files so the FP16 and INT8 engines don't overwrite each other.
Step 2, route B: a Python calibrator (full control)
For custom models, custom preprocessing or a pipeline outside Ultralytics, write a calibrator. This example uses the TensorRT 8.6 Python API (JetPack 6.0) with pycuda. The preprocessing must match your inference code exactly, here a 640 x 640 letterbox, BGR to RGB, and scaling to 0-1:
import os, glob, numpy as np, cv2
import tensorrt as trt
import pycuda.driver as cuda
import pycuda.autoinit
def letterbox(img, size=640):
h, w = img.shape[:2]
r = min(size / h, size / w)
nh, nw = int(round(h * r)), int(round(w * r))
out = np.full((size, size, 3), 114, np.uint8)
top, left = (size - nh) // 2, (size - nw) // 2
out[top:top + nh, left:left + nw] = cv2.resize(img, (nw, nh))
return out
class YoloCalibrator(trt.IInt8EntropyCalibrator2):
def __init__(self, image_dir, size=640, cache='calib.cache'):
super().__init__()
self.files = sorted(glob.glob(os.path.join(image_dir, '*.jpg')))
self.size, self.cache, self.i = size, cache, 0
self.dev = cuda.mem_alloc(3 * size * size * 4) # one float32 image
def get_batch_size(self):
return 1 # must match the ONNX input batch size
def get_batch(self, names):
if self.i >= len(self.files):
return None
img = letterbox(cv2.imread(self.files[self.i]), self.size)[:, :, ::-1]
x = np.ascontiguousarray(img.transpose(2, 0, 1)[None], dtype=np.float32) / 255.0
cuda.memcpy_htod(self.dev, x)
self.i += 1
return [int(self.dev)]
def read_calibration_cache(self):
return open(self.cache, 'rb').read() if os.path.exists(self.cache) else None
def write_calibration_cache(self, data):
open(self.cache, 'wb').write(data)Then build the engine, allowing FP16 as well so TensorRT can keep layers that don't benefit from INT8 in FP16:
logger = trt.Logger(trt.Logger.INFO)
builder = trt.Builder(logger)
network = builder.create_network(1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH))
parser = trt.OnnxParser(network, logger)
assert parser.parse(open('yolo11s.onnx', 'rb').read())
config = builder.create_builder_config()
config.set_flag(trt.BuilderFlag.INT8)
config.set_flag(trt.BuilderFlag.FP16)
config.int8_calibrator = YoloCalibrator('calib_images/')
open('yolo11s_int8.engine', 'wb').write(builder.build_serialized_network(network, config))The calibrator writes calib.cache on the first build; later builds reuse it and skip calibration. Delete the cache whenever the model or preprocessing changes. Newer TensorRT 10 releases (JetPack 6.1 and later) deprecate this calibrator API in favour of explicit quantization with Q/DQ nodes, for example via NVIDIA's TensorRT Model Optimizer. The principles, and the importance of good calibration data, stay the same.
Step 3: don't use trtexec --int8 alone for accuracy
trtexec is excellent for measuring speed, but running it with --int8 and no calibration cache uses placeholder scales. The engine will run, and the speed numbers are meaningful, but its accuracy is not. Use trtexec for benchmarking, or pass it a calibration cache produced by a real calibrator:
/usr/src/tensorrt/bin/trtexec --onnx=yolo11s.onnx --int8 --fp16 --calib=calib.cache --saveEngine=yolo11s_int8.engineStep 4: validate INT8 against FP16
Measure accuracy on a labelled validation set the calibration never saw, with both engines, on the target device:
yolo val model=yolo11s_fp16.engine data=val_dataset.yaml imgsz=640 device=0
yolo val model=yolo11s_int8.engine data=val_dataset.yaml imgsz=640 device=0Compare overall mAP50-95, but also look at per-class results and small objects, which INT8 tends to hurt first. A drop of a percent or two overall is common and often acceptable. A large drop on one class means the calibration set is missing something. Then run the end-to-end benchmark from our YOLO11 on Jetson Orin Nano guide to confirm the speed gain survives preprocessing and NMS.
Troubleshooting accuracy drops
| Symptom | Likely cause | Fix |
|---|---|---|
| Large mAP drop on every class | Preprocessing differs between calibration and inference | Use identical resize, colour order and scaling |
| One class or small objects badly hit | Those cases are rare or missing in the calibration set | Add more images containing them, recalibrate |
| Results change after retraining | Old calibration cache reused | Delete the cache and recalibrate for every new model |
| Good mAP in testing, worse in the field | Calibration images don't match production conditions | Calibrate on images from the production cameras |
| Still too much loss after better data | Some layers are sensitive to INT8 | Keep the detection head or other sensitive layers in FP16 |
| Accuracy must match FP16 almost exactly | Post-training quantization isn't enough | Use quantization-aware training (QAT) |
For keeping specific layers in higher precision, trtexec supports --precisionConstraints and --layerPrecisions, and the Python API can set precision per layer. Profile first to find which layers matter.
When INT8 is worth it
INT8 makes sense when FP16 doesn't meet your frame rate, when you need headroom for more cameras or a second model, or when you want to run on the DLA accelerators of Orin NX and AGX Orin, which need INT8 or FP16. If FP16 already meets your targets with headroom, ship FP16: it is simpler and nearly lossless. For choosing the module itself, see Jetson Orin NX vs Orin Nano for YOLO11.
We build and deploy optimised vision pipelines on Jetson for robotics and inspection, including calibration, validation and field monitoring. See our edge AI and embedded ML services or contact us.

