Edge / On-Device LLM — Inference Phase

⚡

Inference Specialization

Edge / On-Device LLM

Build AI that runs entirely on-device — no cloud required. Master quantisation, pruning, TFLite, llama.cpp, and hardware-specific deployment for mobile, IoT, and automotive. Massive demand as every device becomes AI-capable.

Days

Projects

$175K

Median

↑68%

YoY Hiring

$130K–$220K

Salary Range

Week 1: Model Compression Fundamentals · Days 1–8  

Understand why edge AI is fundamentally different and master every compression technique: quantisation, pruning, and knowledge distillation.

D1–2

Edge AI hardware taxonomy and constraints

MCUs, NPUs, Apple Neural Engine, Qualcomm Hexagon, Jetson — memory budgets and power envelopes for each

↗ TinyML — Harvard/edX (free audit)

D3–4

Post-training quantisation: INT8, INT4, GGUF formats

Calibration sets, symmetric vs asymmetric, per-channel quantisation with bitsandbytes and llama.cpp

↗ Quantisation Fundamentals — DeepLearning.AI (free)

D5–6

Pruning and knowledge distillation

Magnitude pruning, DistilBERT approach, teacher-student training pipelines for Transformers

↗ HuggingFace Neural Network Compression docs (free)

D7

ONNX and runtime graph optimisation

Export PyTorch to ONNX, operator fusion, ONNX Runtime execution providers for CPU, CUDA, TensorRT

↗ ONNX Runtime official docs (free)

🏗 Project — Day D8: LLM Quantisation Benchmark on CPU 

· Quantise Llama-3-8B to Q4_K_M, Q5_K_M, and Q8_0 with llama.cpp

· Perplexity + tokens/sec + RAM usage comparison table

· Automated Python benchmark script with charts

Week 2: Mobile and Embedded Deployment · Days 9–16  

Deploy real models on Android, iOS, Raspberry Pi, and NVIDIA Jetson. Understand hardware-specific acceleration paths.

D9–10

TensorFlow Lite and Core ML deployment

Android TFLite delegates (NNAPI, GPU), iOS Core ML and Apple Neural Engine, conversion pipelines

↗ TFLite conversion docs — official (free)

D11–12

PyTorch Mobile and ExecuTorch

Meta's ExecuTorch runtime — export, partition, and run PyTorch natively on mobile hardware

↗ ExecuTorch Getting Started — PyTorch (free)

D13–14

MediaPipe Solutions for on-device ML

MediaPipe for face, hand, pose detection — integrate into Android or iOS app with <50ms latency

↗ Google MediaPipe Solutions guide (free)

D15

Raspberry Pi and Jetson Nano deployment

Deploy vision and speech models on edge hardware — ARM NEON optimisation and power profiling

↗ NVIDIA Jetson AI Fundamentals (free)

🏗 Project — Day D16: On-Device Mobile AI App 

· Working Android or iOS app with local inference

· Under 50ms inference latency on real device

· Model under 20MB suitable for app store

· Zero user data leaves the device — privacy proof

Week 3: LLMs on Edge and Advanced Techniques · Days 17–32  

Run full language models on device. Phi-3 Mini, Gemma 2B, speculative decoding, federated learning, and NPU acceleration.

D17–20

Edge LLM models: Phi-3 Mini, Gemma 2B, Qwen-0.5B

Models designed for under 4GB RAM. Benchmark quality vs size on real target hardware

↗ Microsoft Phi-3 Mini Cookbook (free)

D21–24

llama.cpp deep dive and Transformers.js

GGUF internals, Metal and Vulkan backends, and running full LLMs in the browser via Transformers.js

↗ Transformers.js docs — HuggingFace (free)

D25–28

Speculative decoding and early exit strategies

3 to 4 times speedup using draft models, adaptive computation, and token acceptance thresholds

↗ Speculative Decoding blog — HuggingFace (free)

D29–31

Federated learning for private edge training

Train on device data without centralising. Flower framework and privacy accounting basics

↗ Flower Federated Learning Framework (free)

🏗 Project — Day D32: Offline Voice Assistant on Raspberry Pi 

· Whisper.cpp (ASR) + Phi-3 Mini (LLM) + Bark (TTS) — fully offline

· Under 2 second response latency on Pi hardware

· Wake-word detection with Porcupine

· Complete demo video showing full pipeline

Week 4: Production, Safety and Capstone · Days 33–40  

OTA model updates, safety-critical deployment standards, and a complete production edge AI system.

D33–35

OTA model updates and A/B testing on device

Over-the-air model delivery, rollback strategy, canary releases, and version management for edge fleets

↗ Azure IoT Edge ML deployment docs (free)

D36–38

Safety-critical AI: automotive and medical device constraints

AUTOSAR Adaptive, functional safety requirements, real-time guarantees for AI in regulated environments

↗ Self-Driving Car Engineer — Udacity (free preview)

D39

Automated compression CI/CD pipeline

Automate the full cycle: train, prune, quantise, benchmark, promote. Reproducible edge ML pipelines

↗ TensorRT-LLM — NVIDIA (free)

🏗 Project — Day D40: Production Edge AI System — Full Capstone 

· Automated compression pipeline: quantise, prune, benchmark, decide

· Deployment running on mobile plus one additional edge device

· OTA update mechanism with rollback

· Latency, accuracy, and model size trade-off report

· Architecture write-up and deployment guide