AI Inference Optimization at the Edge
Training gets all the glory, but inference is where AI models actually earn their keep. And running inference at the edge — on devices with limited power, memory, and compute — is where the real engineering challenge lives. I've spent a lot of time optimizing neural networks to run on hardware that costs $10 instead of $10,000, and the techniques involved are far more nuanced than most people realize.
What "Edge" Actually Means
The term "edge" covers a huge range of hardware. Let me break it down by compute capability:
- Microcontrollers (MCUs) — ARM Cortex-M series, RISC-V cores, 256KB-2MB RAM, milliwatt power budgets. Running tiny keyword spotting or anomaly detection models. Think STM32, Nordic nRF, Espressif ESP32-S3.
- Mobile/embedded SoCs — Qualcomm Snapdragon, MediaTek Dimensity, Samsung Exynos, Apple A-series. 4-16 GB RAM, integrated NPU/DSP with 10-45 TOPS. Running on-device LLMs, image recognition, video analytics.
- Edge AI accelerators — Google Coral (Edge TPU), Intel Movidius, Hailo-8 (26 TOPS at 2.5W), Nvidia Jetson Orin series. Purpose-built for inference with 5-275 TOPS in 5-60W.
- Edge servers — Nvidia Jetson AGX Orin (275 TOPS), Qualcomm Cloud AI 100, Intel Gaudi. Rack-mounted or box-form-factor systems at the network edge, consuming 40-150W.
The optimization strategies differ dramatically across this spectrum. What works on a Jetson Orin is completely irrelevant to someone trying to fit a model on an ESP32.
Quantization: The Single Most Important Technique
If you do nothing else, quantize your model. Moving from FP32 to INT8 inference cuts model size by 4x, reduces memory bandwidth requirements proportionally, and most hardware has significantly faster integer arithmetic than floating point.
There are several flavors:
Post-training quantization (PTQ) takes a trained FP32 model and converts weights and activations to lower precision with minimal fine-tuning. TensorFlow Lite, ONNX Runtime, and PyTorch all support this. For INT8, you typically see less than 1% accuracy loss on most vision models when done correctly. The trick is calibrating the quantization ranges using representative data — use 500-1000 samples from your training set to compute the activation distributions at each layer.
Quantization-aware training (QAT) inserts fake quantization operations during training so the model learns to tolerate quantization noise. This recovers most of the accuracy loss from PTQ and is essential for aggressive quantization (INT4, mixed precision). Google's research team showed that QAT with INT4 weights and INT8 activations can match FP32 accuracy on EfficientNet and MobileNet architectures.
This connects to the ideas in RISC-V vs ARM 2026: Cuộc Chiến Kiến Trúc Chip.
Weight-only quantization is popular for LLMs on edge devices. Methods like GPTQ, AWQ, and GGML/GGUF quantize weights to 4-bit or even 2-bit while keeping activations in higher precision. This makes the model small enough to fit in RAM while maintaining acceptable output quality. Running Llama-2-7B in 4-bit quantization requires about 3.5 GB of RAM, making it feasible on high-end phones and edge accelerators.
Model Architecture Choices
Not all architectures are created equal for edge deployment. Some observations from real projects:
MobileNet V3 and EfficientNet-Lite remain excellent choices for image classification under tight constraints. MobileNet V3-Small achieves 67.4% ImageNet top-1 accuracy with only 2.5M parameters and runs in under 5ms on a Snapdragon 888 NPU.
YOLO variants (YOLOv8n, YOLOv10n) are the go-to for real-time object detection on edge. YOLOv8-nano does 37.3% mAP on COCO with 3.2M parameters. On a Jetson Orin NX, it runs at 180+ fps in INT8.
Transformers are trickier on edge hardware. The attention mechanism's quadratic memory scaling with sequence length is a problem, and the memory-bandwidth-bound nature of transformer inference makes it slow on hardware with limited bandwidth. That said, mobile-optimized transformers like MobileBERT, TinyBERT, and DistilBERT work reasonably well for NLP tasks. For vision, MobileViT and EfficientViT trade some accuracy for much better edge performance.
For on-device LLMs, I'd look at Phi-3-mini (3.8B parameters), Gemma-2B, or Llama-3.2-1B/3B. These are specifically designed to run on devices with 4-8 GB of available memory.
This connects to the ideas in Semiconductor Talent Crisis: Engineering Shortages and Unive.
Compiler-Level Optimizations
The inference compiler makes a massive difference. The same model running through different compilation stacks can vary by 2-5x in throughput on the same hardware. Here's what's available:
- TensorRT — (Nvidia) — the gold standard for Nvidia hardware. Layer fusion, kernel auto-tuning, precision calibration. A ResNet-50 compiled with TensorRT INT8 on a Jetson Orin is roughly 3x faster than the same model in PyTorch FP32.
- TFLite — with XNNPACK — Google's mobile inference stack. XNNPACK provides optimized NEON/SSE kernels for ARM and x86 CPUs. For CPU-only deployment, it's hard to beat.
- ONNX Runtime — Microsoft's cross-platform runtime with execution providers for different hardware. The NNAPI execution provider delegates to Android's neural networks API, which in turn uses the phone's NPU.
- Apache TVM — an open-source compiler that can target almost any hardware. Its auto-tuning (AutoTVM, Ansor) searches for optimal operator implementations. Results can be excellent but require significant tuning time.
- Qualcomm AI Engine Direct (QNN) — the preferred stack for Snapdragon. It targets the Hexagon DSP and NPU with INT8/INT16 precision and is significantly faster than generic CPU inference on Qualcomm chips.
Pruning and Knowledge Distillation
Beyond quantization, two other techniques consistently deliver results:
Structured pruning removes entire channels or attention heads from the network, yielding a genuinely smaller model that runs faster without needing special sparse hardware support. I've found that you can typically prune 30-50% of channels from a well-trained CNN with less than 1% accuracy loss after fine-tuning. Nvidia's ASP (Automatic SParsity) provides 2:4 structured sparsity that their Ampere and newer GPUs accelerate natively, giving up to 2x speedup.
Knowledge distillation trains a small "student" model to mimic a large "teacher" model's outputs. This works surprisingly well — the student learns the teacher's soft probability distributions, which contain more information than hard labels alone. In my experience, a distilled student model consistently outperforms the same architecture trained from scratch by 2-5 percentage points on classification tasks.
Practical Deployment Considerations
Some things that matter in production but are rarely discussed in papers:
Memory allocation patterns matter a lot on edge devices. Pre-allocate all inference buffers at initialization and reuse them across runs. Dynamic allocation during inference causes heap fragmentation and unpredictable latency spikes. TFLite and TensorRT handle this well by default; custom inference loops often don't.
This connects to the ideas in High-NA EUV Lithography: The Next Step Beyond Current EUV fo.
Thermal throttling is real. A phone's NPU might benchmark at 10 TOPS but sustain only 4-6 TOPS under continuous load because the chip throttles to stay within thermal limits. Always benchmark sustained throughput, not peak. I've seen projects launch with great numbers from 30-second benchmarks that fell apart when the device heated up after 5 minutes of continuous inference.
Power measurement should be part of your optimization loop. Sometimes a model that runs 20% faster also uses 40% more peak power due to activating more accelerator cores, resulting in worse energy efficiency. For battery-powered devices, energy per inference (millijoules per prediction) is often a better metric than latency.
Model versioning and A/B testing on edge is harder than in the cloud. You can't just swap models on a server — firmware updates on embedded devices go through certification, and on mobile you're constrained by app store review cycles. Build your inference pipeline so models are loaded from a file path rather than compiled into the binary. This lets you use over-the-air model updates without a full firmware or app update. MediaPipe and TFLite both support this pattern well.
Input preprocessing is often overlooked. Resizing an image from 4K camera resolution to 224x224 for a classification model can take 2-3 ms on a mobile CPU. If your total inference budget is 16 ms (for 60 fps), that's a significant fraction. Use hardware-accelerated image processing (GPU shader-based resize, ISP output cropping) whenever possible, and avoid unnecessary color space conversions. Some accelerators accept YUV input directly rather than requiring RGB conversion.
Edge AI inference optimization is equal parts science and craft. The theoretical techniques are well-documented, but getting the last 2x of performance on a specific piece of hardware requires hands-on profiling, creative problem-solving, and a willingness to get into the details of the target hardware's architecture. That's what makes it interesting.