Inference engineering is the practice of making generative AI models faster, less expensive, and more reliable – without sacrificing the quality that makes them so valuable. Both improving performance and preserving quality require a strong intuition for how models work under the hood.
Generative AI models are a composition of big, complex neural networks. The history of neural networks stretches back to the 1950s, when the first perceptrons for simple binary classification were implemented in hardware. In the following decades, perceptrons were abandoned but then reinvented from single to multi-layer perceptrons with a new concept, back-propagation, which introduced hidden states between layers and a learning procedure that repeatedly adjusts weights within the network. These neural networks had only a few layers. In the 2000s, research began into deep neural networks with dozens of layers. In 2012, AlexNet became the first deep neural network to show promising real-world capabilities and the effectiveness of GPUs for deep learning, leading to new architectures like word embedding models for text and Generative Adversarial Networks (GANs) for images.
But the story truly starts in 2017, when Vaswani and colleagues published the seminal paper “Attention Is All You Need,” introducing the transformer. A transformer is a neural network with an attention mechanism that can learn relationships between various parts of a sequence. Transformers are the foundation of generative AI. Transformers aren’t just for LLMs, they power every modality of model from embedding to voice to image and video generation.
Across modalities, there are two important styles of transformer-based models:
- Autoregressive token generation: Start from a tokenized sequence and predict the most likely next token.
- Iterative denoising: Start from random noise and refine toward the most likely output via diffusion.
Neural Networks Fundamentals
To be a productive inference engineer, you need a basic intuition for essential concepts in neural networks. The fundamental unit of a neural network is a node (a.k.a., neuron). A node is a short program that takes an input, multiplies it by some weights, adds some bias, and returns the result.
A group of nodes forms a layer. Nodes within a layer are independent of each other – they do their own calculations. The connection between nodes, or the “network” in a neural network, is between layers, where the nodes in a layer receive the output of the previous layer.
The neural networks behind LLMs contain dozens to hundreds of layers. There are three types of layers:
- Input layer: The first layer, which accepts and processes the input to the neural network.
- Hidden layers: Every layer between the first and last, which iteratively transform the input to arrive at an output.
- Output layer: The final layer, which returns the prediction from the network.
Each layer produces an output that the next layer reads as input. For the hidden layers, these outputs are called hidden states. Internal representations for text input increase the dimensionality, encoding text chunks into vectors of hundreds or thousands of numbers to capture semantic meaning. But internal representations for image models reduce the dimensionality from millions of pixels down to a manageable size.
There are neural networks for creating these internal representations, and there are neural networks for using them:
- Encoder: Takes an input like text or an image and creates an internal representation of the input that includes additional information and semantic meaning.
- Decoder: Uses the internal representation to generate an output like text or an image.
Modern LLMs are decoder-only, while encoder-only models are somewhat rare today, with old-school text embedding models from the BERT family as a prominent example. Many models in other modalities use an encoder-decoder architecture. Whisper, a popular open model for audio transcription, uses an encoder to process audio input and a decoder to generate text tokens.
Linear Layers and Matmul
The most essential operation within a neural network is a matrix multiplication, or matmul. A matmul takes an input vector (a list of numbers) and a matrix (a grid of numbers) and multiplies the vector through the matrix to produce an output vector.
Within a neural network, a linear layer is the simplest form of matmul. Given an input vector, the linear layer applies a weight matrix and adds a bias vector. The weights of any given linear layer are a small part of a generative AI model’s total weights, and the individual values within the weights matrix are set during training.
Activation Functions
Matrix multiplication is composable, meaning that multiplying a vector by two matrices is equivalent to multiplying that vector by the product of those matrices. This is a problem for multi-layer neural networks because a series of linear layers, each one a matmul, would collapse into a single layer with all of the matrices multiplied together.
Neural networks separate layers by breaking linearity with an activation function. Activation functions are non-linear to prevent composable matmul from collapsing layers, and are differentiable or mostly-differentiable to support back propagation.
One of the most basic activation functions in inference is ReLU, which stands for Rectified Linear Unit. ReLU is a simple function: if X is greater than zero, return X, else return zero. There are dozens of activation functions – including one named “Swish” thanks to its resemblance to the Nike logo – but most follow the same general pattern of mapping negative values to zero or near-zero, while keeping positive values unchanged.
Activation functions like ReLU, SiLU, Swish, and SwiGLU are fast to run, easy to train on (as they are mostly differentiable, they have a gradient at least for most values), and break linearity to support multi-layer neural networks.
# Matrix multiplication is the core of inference
import torch
# Simulating a single linear layer
input_tensor = torch.randn(1, 4096) # batch of 1, hidden dim 4096
weight_matrix = torch.randn(4096, 11008) # projecting to FFN dim
# This single operation dominates inference time
output = input_tensor @ weight_matrix # Matrix multiplication
print(f"Input: {input_tensor.shape}")
print(f"Weight: {weight_matrix.shape}")
print(f"Output: {output.shape}")
# FLOPs = 2 × 1 × 4096 × 11008 ≈ 90M floating point operations
