Edge AI

TensorRT INT8 Calibration for YOLO Models, Step by Step, explained simply.

INT8 is the fastest precision TensorRT offers for YOLO on Jetson and data-center GPUs, but it is also where accuracy quietly disappears. The difference between an INT8 engine that matches FP16 and one that misses half your small objects is almost always the calibration. This guide walks through calibration step by step: what it does, how to build a calibration set, two ways to build the engine, and how to prove the result is still accurate.

By Tirth Borda · Co-Founder · Robotics Systems Lead6 min read
TensorRT INT8 calibration pipeline for a YOLO object detection model

Core idea

What this blog covers

INT8 stores weights and activations as 8-bit integers, so TensorRT must choose a scale for every tensor. It chooses those scales from the calibration images you give it. Calibrate on the wrong images, preprocess them differently from inference, or skip validation, and the engine runs fast while detecting the wrong things.

Main discussion

Quick answer: how INT8 calibration works

Post-training INT8 quantization converts a trained FP32 model to 8-bit integers without retraining. TensorRT runs your calibration images through the network, records the range of values each layer's activations take, and chooses a scale that maps that range onto 8 bits. The scales are saved in a calibration cache and baked into the engine.

StepWhat happensWhat can go wrong
1. Calibration setYou supply representative imagesImages don't match production (lighting, camera, objects)
2. PreprocessingImages resized and normalised like inferenceDifferent resize, colour order or scaling than at runtime
3. CalibrationTensorRT collects activation rangesToo few images; rare classes missing
4. Engine buildScales applied, layers fused, kernels chosenSensitive layers lose precision
5. ValidationINT8 mAP compared with FP16Skipped, so the accuracy loss is found in the field

On Jetson Orin, INT8 typically runs around 1.5x faster than FP16 for YOLO, with a small mAP cost when calibration is done well.

Step 1: build a calibration set

The calibration set decides the result, so it deserves more care than the build command:

- Take the images from your production cameras, not from a public dataset, wherever possible. - Cover the real range of conditions: lighting changes, shifts, backgrounds, motion blur, and both crowded and empty scenes. - Include every class, especially rare and small objects. If a class is missing from calibration, its activations may be clipped. - Use several hundred images. Far fewer risks unstable scales; many thousands mostly adds build time. - Keep calibration images separate from the validation set you will use to measure accuracy.

Labels are not needed for calibration itself, only images. You do need a labelled validation set for Step 4.

Step 2, route A: Ultralytics export (quickest)

For YOLO11 and YOLOv8, the Ultralytics exporter handles calibration for you. Point it at a dataset YAML whose images represent production, and run it on the target device:

# FP16 baseline engine
yolo export model=yolo11s.pt format=engine half=True imgsz=640 device=0
# INT8 engine calibrated on the images listed in your dataset YAML
yolo export model=yolo11s.pt format=engine int8=True data=calib_dataset.yaml imgsz=640 device=0

The exporter applies the same letterbox preprocessing it uses at inference, which removes the most common source of calibration mistakes. Its fraction argument limits how much of the dataset is used for calibration if the set is large. Rename the output files so the FP16 and INT8 engines don't overwrite each other.

Step 2, route B: a Python calibrator (full control)

For custom models, custom preprocessing or a pipeline outside Ultralytics, write a calibrator. This example uses the TensorRT 8.6 Python API (JetPack 6.0) with pycuda. The preprocessing must match your inference code exactly, here a 640 x 640 letterbox, BGR to RGB, and scaling to 0-1:

import os, glob, numpy as np, cv2
import tensorrt as trt
import pycuda.driver as cuda
import pycuda.autoinit
def letterbox(img, size=640):
    h, w = img.shape[:2]
    r = min(size / h, size / w)
    nh, nw = int(round(h * r)), int(round(w * r))
    out = np.full((size, size, 3), 114, np.uint8)
    top, left = (size - nh) // 2, (size - nw) // 2
    out[top:top + nh, left:left + nw] = cv2.resize(img, (nw, nh))
    return out
class YoloCalibrator(trt.IInt8EntropyCalibrator2):
    def __init__(self, image_dir, size=640, cache='calib.cache'):
        super().__init__()
        self.files = sorted(glob.glob(os.path.join(image_dir, '*.jpg')))
        self.size, self.cache, self.i = size, cache, 0
        self.dev = cuda.mem_alloc(3 * size * size * 4)  # one float32 image
    def get_batch_size(self):
        return 1  # must match the ONNX input batch size
    def get_batch(self, names):
        if self.i >= len(self.files):
            return None
        img = letterbox(cv2.imread(self.files[self.i]), self.size)[:, :, ::-1]
        x = np.ascontiguousarray(img.transpose(2, 0, 1)[None], dtype=np.float32) / 255.0
        cuda.memcpy_htod(self.dev, x)
        self.i += 1
        return [int(self.dev)]
    def read_calibration_cache(self):
        return open(self.cache, 'rb').read() if os.path.exists(self.cache) else None
    def write_calibration_cache(self, data):
        open(self.cache, 'wb').write(data)

Then build the engine, allowing FP16 as well so TensorRT can keep layers that don't benefit from INT8 in FP16:

logger = trt.Logger(trt.Logger.INFO)
builder = trt.Builder(logger)
network = builder.create_network(1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH))
parser = trt.OnnxParser(network, logger)
assert parser.parse(open('yolo11s.onnx', 'rb').read())
config = builder.create_builder_config()
config.set_flag(trt.BuilderFlag.INT8)
config.set_flag(trt.BuilderFlag.FP16)
config.int8_calibrator = YoloCalibrator('calib_images/')
open('yolo11s_int8.engine', 'wb').write(builder.build_serialized_network(network, config))

The calibrator writes calib.cache on the first build; later builds reuse it and skip calibration. Delete the cache whenever the model or preprocessing changes. Newer TensorRT 10 releases (JetPack 6.1 and later) deprecate this calibrator API in favour of explicit quantization with Q/DQ nodes, for example via NVIDIA's TensorRT Model Optimizer. The principles, and the importance of good calibration data, stay the same.

Step 3: don't use trtexec --int8 alone for accuracy

trtexec is excellent for measuring speed, but running it with --int8 and no calibration cache uses placeholder scales. The engine will run, and the speed numbers are meaningful, but its accuracy is not. Use trtexec for benchmarking, or pass it a calibration cache produced by a real calibrator:

/usr/src/tensorrt/bin/trtexec --onnx=yolo11s.onnx --int8 --fp16 --calib=calib.cache --saveEngine=yolo11s_int8.engine

Step 4: validate INT8 against FP16

Measure accuracy on a labelled validation set the calibration never saw, with both engines, on the target device:

yolo val model=yolo11s_fp16.engine data=val_dataset.yaml imgsz=640 device=0
yolo val model=yolo11s_int8.engine data=val_dataset.yaml imgsz=640 device=0

Compare overall mAP50-95, but also look at per-class results and small objects, which INT8 tends to hurt first. A drop of a percent or two overall is common and often acceptable. A large drop on one class means the calibration set is missing something. Then run the end-to-end benchmark from our YOLO11 on Jetson Orin Nano guide to confirm the speed gain survives preprocessing and NMS.

Troubleshooting accuracy drops

SymptomLikely causeFix
Large mAP drop on every classPreprocessing differs between calibration and inferenceUse identical resize, colour order and scaling
One class or small objects badly hitThose cases are rare or missing in the calibration setAdd more images containing them, recalibrate
Results change after retrainingOld calibration cache reusedDelete the cache and recalibrate for every new model
Good mAP in testing, worse in the fieldCalibration images don't match production conditionsCalibrate on images from the production cameras
Still too much loss after better dataSome layers are sensitive to INT8Keep the detection head or other sensitive layers in FP16
Accuracy must match FP16 almost exactlyPost-training quantization isn't enoughUse quantization-aware training (QAT)

For keeping specific layers in higher precision, trtexec supports --precisionConstraints and --layerPrecisions, and the Python API can set precision per layer. Profile first to find which layers matter.

When INT8 is worth it

INT8 makes sense when FP16 doesn't meet your frame rate, when you need headroom for more cameras or a second model, or when you want to run on the DLA accelerators of Orin NX and AGX Orin, which need INT8 or FP16. If FP16 already meets your targets with headroom, ship FP16: it is simpler and nearly lossless. For choosing the module itself, see Jetson Orin NX vs Orin Nano for YOLO11.

We build and deploy optimised vision pipelines on Jetson for robotics and inspection, including calibration, validation and field monitoring. See our edge AI and embedded ML services or contact us.

Tagged

  • TensorRT
  • INT8
  • Quantization
  • YOLO11
  • Jetson Orin Nano
  • Edge AI

Key takeaways

What readers should remember

  • Calibration picks the INT8 scale for each tensor from real images, so the calibration set must look like production data.
  • Use several hundred representative images, preprocessed exactly as at inference time.
  • The Ultralytics exporter is the quickest route; a custom Python calibrator gives full control.
  • Always compare INT8 mAP against FP16 on the same validation set before shipping.
  • If accuracy drops, fix the calibration data first, then keep sensitive layers in FP16.
Tirth Borda — Co-Founder and Robotics Systems Lead at YantriX Technologies

Written by

Tirth Borda

Co-Founder · Robotics Systems Lead

Leads robotics system architecture and technical execution at YantriX, translating product requirements into integrated mechanical, electronics, embedded, perception and autonomy systems. He guides the work from concept definition through system integration and prototype validation.

Frequently asked questions

Answers from the work itself.

What is INT8 calibration in TensorRT?

It is the step where TensorRT runs representative images through the network to measure the range of each layer's activations, then picks scales that map those ranges onto 8-bit integers. The scales are stored in a calibration cache and used to build the INT8 engine.

How many images do I need for INT8 calibration?

Several hundred representative images is a common starting point. Coverage matters more than count: include every class, small objects and the real range of lighting and backgrounds from your production cameras.

How much accuracy does INT8 lose for YOLO?

With good calibration, YOLO models usually lose a small amount of mAP, often a percent or two, compared with FP16. Poor calibration data or mismatched preprocessing can cause much larger losses, so always validate both engines on the same dataset.

Can I use trtexec --int8 to build a production engine?

Not on its own. Without a calibration cache, trtexec uses placeholder scales, so the engine is fine for speed benchmarks but not for accuracy. Pass it a calibration cache from a real calibrator with --calib.

What is the difference between PTQ and QAT?

Post-training quantization (PTQ) calibrates a trained model without retraining, as described in this guide. Quantization-aware training (QAT) simulates INT8 during training so the model learns to tolerate it, which recovers more accuracy but takes more work.

Related reading

Keep going on embedded & electronics.

Continue exploring

Related services and proof points

Explore our core engineering capabilities and verified hardware case studies related to this technical domain.

Engineering Consultation

Working on a similar engineering problem?

Share your technical specifications, prototype requirements, or CAD models with our engineering team. NDA support is available where required before confidential CAD exchange.