AI Infrastructure & Systems — Inference Phase

🏗️

Inference Specialization

AI Infrastructure & Systems

AI Infrastructure engineers design the compute, networking, and storage systems for training and serving the world's largest models. CUDA kernels, distributed training across GPU clusters, inference optimisation. Top 5% global compensation.

Days

Projects

$215K

Median

Top 5%

Global Comp.

$150K–$280K

Salary Range

Week 1: GPU Architecture and CUDA Programming · Days 1–10  

Understand how GPUs execute neural network workloads, write high-performance CUDA kernels, and profile GPU utilisation.

D1–3

GPU architecture: warps, SIMT, memory hierarchy, roofline model

Thread blocks, shared memory, L1/L2/HBM bandwidth hierarchy. Compute-bound vs memory-bound workload analysis

↗ CUDA Programming — NVIDIA training portal (free)

D4–6

Writing high-performance CUDA kernels: tiled matmul and FlashAttention

CUDA C++: tiled matrix multiply, warp-level primitives, fused attention kernels like FlashAttention-2

↗ GPU Puzzles — Sasha Rush GitHub (free)

D7–9

GPU profiling: Nsight Compute and Nsight Systems

Identifying bottlenecks: kernel occupancy, memory bandwidth utilisation, warp divergence, roofline analysis

↗ NVIDIA Nsight Compute official documentation (free)

🏗 Project — Day D10: Custom CUDA Kernel with Performance Engineering Report 

· CUDA C++ kernel integrated into PyTorch via cpp_extension

· Throughput and latency comparison vs PyTorch baseline

· Nsight Compute profiling report with occupancy analysis

· Memory bandwidth utilisation measurement at each optimisation step

Week 2: Distributed Training and High-Performance Networking · Days 11–24  

Train 70B+ parameter models across hundreds of GPUs. Master NCCL collectives, InfiniBand networking, and all parallelism strategies.

D11–14

Distributed training: DDP, FSDP, and Tensor Parallelism

Data, model, tensor, and pipeline parallelism. Training 70B+ models across GPU clusters with PyTorch

↗ HuggingFace Accelerate and DeepSpeed guides (free)

D15–18

High-performance networking: InfiniBand, RDMA, and NCCL

All-Reduce algorithms, NCCL collective operations, topology-aware scheduling for GPU cluster efficiency

↗ Distributed ML Systems — CMU 15-712 (free lecture slides)

D19–22

LLM inference: PagedAttention, continuous batching, speculative decoding

vLLM internals, KV cache management via PagedAttention, 3x speedup with speculative decoding

↗ vLLM engineering blog and source code (free)

D23

Mixed precision: FP16, BF16, FP8, and numerical stability

When to use each floating-point format, loss scaling strategies, preventing gradient underflow in training

↗ PyTorch Automatic Mixed Precision docs (free)

🏗 Project — Day D24: Distributed Training Strong Scaling Benchmark 

· FSDP training script for a 1 billion parameter model

· Strong scaling benchmark from 1 to 4 GPUs with throughput chart

· Communication overhead analysis using Nsight Systems timeline

· Cost per training token estimate across 3 GPU configurations

Week 3: Storage, Scheduling, and AI Systems Design · Days 25–40  

High-performance storage systems, GPU cluster scheduling, and designing complete AI training infrastructure from first principles.

D25–28

High-performance storage: Lustre, WEKA, and streaming datasets

Data loading bottlenecks at scale, parallel file systems, NVIDIA DALI, streaming for terabyte training datasets

↗ PyTorch DataLoader performance tuning guide (free)

D29–32

Cluster scheduling: Slurm, Ray, and multi-tenant fairness

GPU cluster job scheduling, fair-share policies, preemption, Ray for distributed Python ML workflows

↗ Ray distributed computing documentation (free)

D33–36

AI systems design: architecture patterns for hundred-billion-parameter scale

Training and serving infrastructure for 100B+ models. Architecture case studies from Megatron-LM and PaLM

↗ GPU Mode engineering lectures (YouTube, free)

D37–39

Reliability engineering for AI GPU clusters

Checkpoint strategy, failure recovery, NCCL timeout handling, proactive GPU health monitoring with DCGM

↗ NVIDIA GPU Operator and DCGM documentation (free)

🏗 Project — Day D40: End-to-End AI Training Infrastructure — Full Capstone 

· Multi-GPU FSDP training with automatic checkpoint and resume

· Distributed HPO with Ray Tune across 3 hyperparameter dimensions

· Checkpoint versioning and failure recovery demonstration

· Cost optimisation combining spot instances and mixed precision

· Architecture design document with engineering trade-offs