fpga  ·  January 15, 2026  ·  2 min read

Running YOLOv11 on an FPGA: What I Learned

fpgaaihardwareresearch

Getting a modern object detection model to run well on an FPGA is mostly about fighting the compiler, not the math. Here’s what the process actually looked like.

The problem with SiLU

YOLOv11 uses SiLU activations (x * sigmoid(x)). Xilinx’s DPU does not support SiLU — it falls back to the CPU for those layers, which fragments the graph into nine separate DPU subgraphs with expensive host-device transfers in between.

The fix is surgical: substitute every SiLU with ReLU before quantisation.

# Replace SiLU → ReLU before Vitis AI quantisation
import torch.nn as nn

def swap_silu(module: nn.Module) -> nn.Module:
    for name, child in module.named_children():
        if isinstance(child, nn.SiLU):
            setattr(module, name, nn.ReLU(inplace=True))
        else:
            swap_silu(child)
    return module

model = swap_silu(model)

Same treatment for Softmax → HardSigmoid and the attention matmul → element-wise multiply. After substitution: one DPU subgraph instead of nine.

Calibration matters more than dataset size

The standard advice is “use 1000+ images for INT8 calibration.” We found the opposite: a hand-curated 250-image set that covered edge cases outperformed a random 1300-image sample by ~1.2 mAP points.

The intuition: calibration is setting per-layer activation ranges. Outlier images skew those ranges and clip normal activations. Curation beats volume.

Results

MetricCPU (i5-12650H)FPGA (ZCU104)Δ
Throughput17.3 FPS25.77 FPS+49%
Latency57.8 ms38.81 ms−33%
Power draw~45 W~8 W−82%

The paper (IEEE ISVLSI 2026) has the full ablation — operator graph, calibration set analysis, and a comparison against prior FPGA deployments of YOLOv8/v9.

What I’d do differently

  1. Start with the DPU compiler report first. Running vai_c_xir early tells you exactly which ops will fall back before you’ve committed to an architecture.
  2. Profile on-board before optimising. I spent a week tuning layers that weren’t the bottleneck. perf_analyze revealed the real hotspot in 20 minutes.
  3. Quantise-aware training if accuracy matters. PTQ got us to acceptable mAP, but QAT would have closed the remaining gap without extra calibration effort.

If you’re doing something similar, the Vitis AI 3.5 model zoo examples are the best starting point — they show working DPUCZDX8G deployments end to end.

← All posts