Neural Networks & Backprop

A single perceptron cannot draw a straight line around a region it cannot separate, and the classic example is XOR: $(0,0)\to 0$, $(0,1)\to 1$, $(1,0)\to 1$, $(1,1)\to 0$ is not linearly separable, so a perceptron fails on it forever. Stacking two perceptrons into a tiny two-layer network fixes XOR. This lesson walks from that fix to the general machinery that makes it work: layers and weights expressed as matrix multiplies, the activation functions that inject non-linearity between layers, a forward pass with concrete numbers, the chain-rule derivation that turns a single comparison between prediction and target into a gradient on every weight, and the practical vocabulary (epoch, batch, depth) that frames every deep learning system since.

Learning Objectives

1. Explain why the perceptron cannot represent XOR, and articulate why a hidden layer of non-linear units changes that.
2. Express one fully-connected layer as the affine transform $z = Wx + b$, write the dimensions of $W$, $x$, $b$ and $z$ explicitly, and chain two layers into a network.
3. State the formulas and derivatives of sigmoid, tanh, ReLU and softmax, and rank them by vanishing-gradient severity on saturated units.
4. Walk a forward pass through a 2-2-1 network with concrete numeric inputs and weights.
5. Derive backpropagation through a network with the chain rule, naming every intermediate partial derivative and justifying the factor $\partial L/\partial W = (\partial L/\partial z)\, x^\top$ at each layer.
6. Distinguish epoch, batch and mini-batch, and explain why mini-batch stochastic gradient descent is the default optimiser for deep networks.
7. Argue why depth helps (compositional features, hierarchical representations, narrower layers per level of abstraction), and place the universal approximation theorem in honest context.

1. From Perceptron to Multilayer

The perceptron computes a single linear combination followed by a step function:

The decision boundary is the line $w_1 x_1 + w_2 x_2 + b = 0$. Everything on one side maps to $1$, everything on the other to $0$. Two classes that can be split by such a line are linearly separable — the perceptron learns them in finite steps (Rosenblatt, 1958).

XOR is not linearly separable. The four points $(0,0), (0,1), (1,0), (1,1)$ sit at the corners of a unit square, and the two "on" corners $(0,1)$ and $(1,0)$ are diagonal opposites — no straight line separates them from the "off" corners $(0,0)$ and $(1,1)$. This is not a curiosity; Minsky and Papert pointed it out in Perceptrons (1969), and the resulting pessimism about learning systems is the historical reason the field pivoted to symbolic AI for a decade.

The fix is geometric. A single layer can only draw one line. A second layer can compose lines into more complex regions. The canonical XOR network has two input units, two hidden units, and one output unit. With the right weights, the hidden units each draw a line; their combination carves out the right region; the output unit aggregates. The non-linearity between layers — the activation function applied to the hidden units before the output — is the load-bearing piece. Without it, stacking linear layers collapses to one linear layer, and the network is still a perceptron.

import numpy as np

    # A 2-2-1 network that solves XOR. Architecture: 2 inputs, 2 hidden (tanh),
    # 1 output (sigmoid).
#
    # Both hidden units are BAND-PASSES on the sum s = x1 + x2:
    #   h1 = tanh(8 * (s - 0.5))   rising through the s = 1 point
    #   h2 = tanh(8 * (s - 1.5))   still near its floor at s = 1
    # The output reads h1 - h2, which PEAKS at s = 1 and falls away on both sides.
    # That peak is what makes XOR work, and it is the reason two hidden units are
    # needed: one hidden unit is one half-plane in s, and XOR is not a half-plane.
W1 = np.array([[8.0, 8.0],      # 2x2 — both rows are [a, a], so each row sums x1 + x2
               [8.0, 8.0]])
b1 = np.array([[-4.0],          # 8*(s - 0.5) = 8*s - 4
               [-12.0]])        # 8*(s - 1.5) = 8*s - 12
W2 = np.array([[1.0, -1.0]])   # h1 minus h2 — the band-pass difference
b2 = np.array([[-1.0]])         # centre the band on the decision boundary

def forward(x):
    h = np.tanh(W1 @ x + b1)                    # 2x1 hidden activations
    z = W2 @ h + b2                             # 1x1 pre-activation
    return 1.0 / (1.0 + np.exp(-z))             # 1x1 output probability

for x in [[[0],[0]], [[0],[1]], [[1],[0]], [[1],[1]]]:
    print(x, "->", round(forward(np.array(x)).item(), 4))
    # [[0], [0]] -> 0.2691
    # [[0], [1]] -> 0.7308
    # [[1], [0]] -> 0.7308
    # [[1], [1]] -> 0.2691
    # Classifying at the 0.5 threshold gives 0, 1, 1, 0 — XOR, with a margin of 0.46
    # between the lowest positive and the highest negative.

Running that fragment prints probabilities clustered in two groups separated by $0.5$, so the classification is $0, 1, 1, 0$ — XOR solved by a network two layers deep. This is the smallest object that motivates everything that follows. It is also the whole argument for depth in miniature: one layer could draw only a single half-plane in $s$, and this network drew two and subtracted them to carve out a band.

2. Layers as Matrix Multiplies

A fully-connected ("dense") layer with $n_{\text{in}}$ inputs and $n_{\text{out}}$ outputs is parameterised by a weight matrix $W \in \mathbb{R}^{n_{\text{out}} \times n_{\text{in}}}$ and a bias vector $b \in \mathbb{R}^{n_{\text{out}}}$. Given an input column vector $x \in \mathbb{R}^{n_{\text{in}}}$, the layer computes

The dimensions have to be consistent. $W$ has $n_{\text{out}}$ rows (one per output unit) and $n_{\text{in}}$ columns (one per input it receives), so $(W x)$ has shape $(n_{\text{out}}, 1)$ — same shape as $b$. The pre-activation $z$ is the affine output; the activation $a = \sigma(z)$ is what the next layer sees.

A two-layer network (one hidden layer, one output layer) is the chain

If $x \in \mathbb{R}^{2}$ and the hidden layer has three units, $W^{(1)} \in \mathbb{R}^{3 \times 2}$, $b^{(1)} \in \mathbb{R}^{3}$, $z^{(1)} \in \mathbb{R}^{3}$, $a^{(1)} \in \mathbb{R}^{3}$. With a single output unit, $W^{(2)} \in \mathbb{R}^{1 \times 3}$, $b^{(2)} \in \mathbb{R}$, $\hat{y} \in \mathbb{R}$.

The whole point of writing it this way is that the parameter count, the memory footprint, the matrix multiplies on the GPU, and the gradient computations are all batched operations on $W$, not loops over individual weights. The vectorisation is what makes training at scale tractable.

import numpy as np

# Dimension check for a 2 -> 3 -> 1 network with batch size B = 32.
B, n_in, n_hidden, n_out = 32, 2, 3, 1
rng = np.random.default_rng(7)
W1 = rng.normal(scale=0.1, size=(n_hidden, n_in))   # (3, 2)
b1 = np.zeros((n_hidden, 1))                        # (3, 1)
W2 = rng.normal(scale=0.1, size=(n_out, n_hidden))  # (1, 3)
b2 = np.zeros((n_out, 1))                           # (1, 1)

X = rng.normal(size=(n_in, B))                       # (2, 32) — columns are examples
Z1 = W1 @ X + b1                                    # (3, 32)
A1 = np.tanh(Z1)                                    # (3, 32)
Z2 = W2 @ A1 + b2                                   # (1, 32)
Y_hat = 1.0 / (1.0 + np.exp(-Z2))                   # (1, 32)

for name, t in (("X", X), ("Z1", Z1), ("A1", A1), ("Z2", Z2), ("Y_hat", Y_hat)):
    print(f"{name:<6} shape {t.shape}")
# X      shape (2, 32)
# Z1     shape (3, 32)
# A1     shape (3, 32)
# Z2     shape (1, 32)
# Y_hat  shape (1, 32)

Each line respects the shapes: input columns run along the second axis, so a batch of 32 examples is just a wider $X$. The matrix $W^{(l)}$ is shared across the whole batch — that is the vectorisation.

3. Activation Functions

Without non-linear activations, a stack of layers is mathematically a single affine map, since the composition of affine functions is affine. The activation function is the non-linearity that gives a deep network its expressive power. Four are ubiquitous.

3.1 Sigmoid

Output is in $(0, 1)$, which makes sigmoid a natural choice for the output layer of a binary classifier. As a hidden-layer activation, it has two problems: outputs are not zero-centred, which slows gradient descent, and the derivative approaches $0$ when $|z|$ is large, causing vanishing gradients deep in the network.

3.2 Tanh

Tanh is just a rescaled, zero-centred sigmoid: $\tanh(z) = 2\sigma(2z) - 1$, output range $(-1, 1)$. The zero-centred output fixes one of sigmoid's two problems but the vanishing-gradient issue remains, and for hidden layers tanh has largely been superseded by ReLU.

3.3 ReLU

The derivative is $1$ on the positive half-line and $0$ on the negative half-line. ReLU is the default hidden-layer activation in modern deep learning because it does not saturate for positive $z$, so gradients do not vanish through it; it is also cheap to compute (one comparison, no exponentials). Its weakness is the dying ReLU problem: a unit whose pre-activation is always negative always outputs $0$ and contributes nothing to the gradient, so it never recovers.

3.4 Softmax

Softmax is the multi-class generalisation of sigmoid: it converts a $K$-vector of real pre-activations into a $K$-vector of probabilities that sum to $1$. Its derivative is the softmax output times a Kronecker-delta correction — clean enough that, paired with cross-entropy loss, the gradient collapses to $\hat{y} - y$. Softmax is almost exclusively an output activation, not a hidden one.

The pattern across the table: hidden layers want ReLU, the output layer uses sigmoid for binary classification or softmax for multi-class, and sigmoid and tanh are essentially historical defaults that persist where ReLU's dying-unit pathology is unacceptable (for example, recurrent network gates).

The vanishing-gradient claim is worth making numerically rather than rhetorically. The largest possible value of each derivative bounds how much gradient signal a single unit can pass backward:

import numpy as np

    # The maximum value of each activation's derivative over its whole domain.
    # This is the per-layer multiplier on the error signal travelling backward.
peaks = {
    "sigmoid": 0.25,   # sigma(z)(1-sigma(z)) peaks at z = 0
    "tanh":    1.00,   # 1 - tanh^2(z) peaks at z = 0
    "relu":    1.00,   # exactly 1 for z > 0, exactly 0 for z < 0
}

L = 12  # a 12-layer hidden stack
print(f"{'activation':<10}" + "  ".join(f"L={n}" for n in (2, 6, 12)))
for name, peak in peaks.items():
    # Worst case for a saturated unit: the gradient arrives at the flat part.
    saturated = peak if name == "relu" else 0.02
    row = "  ".join(f"{saturated ** n:.2e}" for n in (2, 6, 12))
    print(f"{name:<10}{row}")

    # With a saturated sigmoid/tanh unit, 12 layers multiply the signal by
    # 0.02^12 = 4.1e-21 — the gradient is numerically zero long before layer 1.

This is the whole reason ReLU displaced sigmoid as the hidden default. It is not that ReLU's derivative is larger at its best — tanh peaks at exactly $1.0$, the same as ReLU, while sigmoid peaks at only $0.25$ — but that ReLU's derivative does not fall off. A ReLU on the positive half-line transmits the full upstream signal, and it only zeroes when the unit is inactive, which is a sparse and therefore informative condition. Sigmoid and tanh shrink the signal continuously as $|z|$ grows, and a deep stack compounds that shrinkage multiplicatively until the early layers receive nothing useful.

4. The Forward Pass, With Numbers

To make the matrix view concrete, run a 2-2-1 network on the input $x = (1, 0)^\top$ with these parameters:

hidden activation tanh, output activation sigmoid.

Step 1 — pre-activation of the hidden layer:

Step 2 — hidden activations:

Step 3 — pre-activation of the output:

Step 4 — output activation:

With target $y = 1$, the loss (binary cross-entropy) is $L = -\log(0.6511) = 0.4292$. Every step is a single matrix-multiply-then-elementwise-nonlinearity, and the whole pass is one forward evaluation that costs $O(n_{\text{in}} n_{\text{hidden}} + n_{\text{hidden}} n_{\text{out}})$ multiplies.

5. Backpropagation by the Chain Rule

Backpropagation is not a separate algorithm; it is the chain rule applied to the forward pass, run efficiently in reverse. Start with a scalar loss $L$ and compute $\partial L / \partial W^{(l)}$ for every layer, so that gradient descent can update each $W^{(l)}$. The recipe has three pieces per layer: the error signal $\delta^{(l)}$, the gradient with respect to the pre-activation $\partial L / \partial z^{(l)}$, and the gradient with respect to the weights $\partial L / \partial W^{(l)}$.

5.1 The chain rule factors

For one training example, write the network as a chain of functions:

The gradient of the loss with respect to the output-layer weights is

For binary cross-entropy with sigmoid output, the first factor simplifies to $\hat{y} - y$ (this is the famous clean derivative). For a generic loss it is

where $\odot$ is the elementwise product. The second factor is

because $z^{(2)} = W^{(2)} a^{(1)} + b^{(2)}$ is a linear function of $W^{(2)}$ whose Jacobian is the row vector $a^{(1)\top}$. Putting them together,

In matrix shapes: $\delta^{(2)}$ has shape $(n_{\text{out}}, 1)$ and $a^{(1)}$ has shape $(n_{\text{hidden}}, 1)$, so $\partial L / \partial W^{(2)}$ has shape $(n_{\text{out}}, n_{\text{hidden}})$ — exactly the shape of $W^{(2)}$.

5.2 Pushing the gradient back one layer

To update $W^{(1)}$, the chain rule threads the gradient through the output layer:

The first factor is $\delta^{(2)}$. The second is $W^{(2)\top}$ since $z^{(2)} = W^{(2)} a^{(1)}$. The third is the diagonal matrix $\text{diag}(\sigma'(z^{(1)}))$ because each $a^{(1)}_i$ depends only on $z^{(1)}_i$. The fourth is $x^\top$. So

The general pattern: at layer $l$, compute

The "backprop" is exactly this recursion — the error signal flows from the loss back through the network, multiplying by transposed weight matrices and by the local derivative of each activation, all the way to the input layer. The forward pass stores $z^{(l)}$ and $a^{(l)}$ at every layer; the backward pass reuses them.

5.3 A worked example

Continuing the 2-2-1 network from Section 4, with target $y = 1$ and $\hat{y} = 0.6511$:

• $\partial L / \partial z^{(2)} = \hat{y} - y = -0.3489$ (binary cross-entropy + sigmoid collapses to this).

• $\delta^{(1)} = (W^{(2)\top} \cdot (-0.3489)) \odot \tanh'(z^{(1)})$. $W^{(2)\top} \cdot (-0.3489) = (1)(-0.3489) + (1)(-0.3489) = (-0.3489, -0.3489)$. $\tanh'(z^{(1)}) = 1 - a^{(1)} \odot a^{(1)} = 1 - (0.5800, 0.2136) = (0.4200, 0.7864)$. $\delta^{(1)} = (-0.3489, -0.3489) \odot (0.4200, 0.7864) = (-0.1465, -0.2744)^\top$.

• $\partial L / \partial W^{(2)} = \delta^{(2)} \, a^{(1)\top} = (-0.3489) \cdot (0.7616, 0.4621) = (-0.2657, -0.1613)$.

• $\partial L / \partial W^{(1)} = \delta^{(1)} \, x^\top$ where $x = (1, 0)^\top$, so

  $$\frac{\partial L}{\partial W^{(1)}} = (-0.1465, -0.2744)^\top (1, 0) = \begin{pmatrix} -0.1465 & 0 \\ -0.2744 & 0 \end{pmatrix},$$

  the second column receiving zeros because $x_2 = 0$ means no gradient flows through the second input.

Read the two components of $\delta^{(1)}$ together, because the asymmetry is the whole point. Both hidden units received exactly the same upstream signal $(-0.3489, -0.3489)$, because $W^{(2)} = (1, 1)$ weights them identically. Their local slopes are different, though: $0.4200$ versus $0.7864$, because $a^{(1)}_2 = 0.4621$ is nearer the origin of tanh than $a^{(1)}_1 = 0.7616$ and tanh is steepest at the origin. The unit with the larger slope transmits proportionally more gradient. Backpropagation is therefore not a uniform broadcast of the error backwards; it re-weights it at every layer by the local derivative, and a unit sitting in the flat region of its activation is effectively skipped.

A gradient-descent step at learning rate $\eta = 0.5$ would subtract $\eta \cdot \partial L / \partial W$ from each $W$. The numbers shrink the loss — that is the entire learning loop, repeated thousands of times across millions of examples.

5.4 Computational cost

The forward pass costs one matrix multiply per layer. The backward pass costs two matrix multiplies per layer: one for the weight gradient $\delta^{(l)} a^{(l-1)\top}$ and one for the upstream signal $W^{(l+1)\top} \delta^{(l+1)}$. Total backward cost is roughly twice the forward cost, which is why training a network takes about three times the wall time of running inference on the same data.

6. Epoch, Batch, and Mini-Batch

These three words are about how many examples each gradient step sees.

• Batch gradient descent: compute the gradient using the entire training set, then take one step. Smooth, but each step costs a full pass over the data, so it is impractical for any dataset larger than a few thousand examples.
• Stochastic gradient descent (SGD): compute the gradient using a single example, then take one step. Each step is cheap and noisy; the noise actually helps escape shallow local minima, but the variance is so high that the loss curve jitters severely.
• Mini-batch gradient descent: compute the gradient using a batch of $B$ examples (typical $B \in \{32, 64, 128, 256\}$), then take one step. This is the practical default. The per-step gradient is a low-variance estimate of the full-batch gradient, the per-step cost is bounded, and the matrix operations are large enough to fill a GPU.

An epoch is one complete pass through the training set. If the training set has $n$ examples and the batch size is $B$, one epoch contains $n / B$ gradient steps. Training a model for 100 epochs means 100 passes through the data, or $100 \cdot (n / B)$ parameter updates total. The choice of $B$ is a hyperparameter: small batches add noise (a regulariser) but underutilise the GPU; large batches reduce noise but may converge to sharper minima and degrade test accuracy.

The optimiser in Section 5 assumed a single example. With a mini-batch of size $B$, the same derivation applies but each term is summed over the batch, and the gradient is typically averaged by $1/B$ before the update:

Note what the $1/B$ factor does. It makes the effective learning rate invariant to batch size: if you double $B$, you halve the noise in the gradient estimate, and dividing by $B$ keeps the expected step length the same. Without the $1/B$, doubling the batch size would double the step length and destabilise training.

Putting the whole loop together is short. The XOR network from Section 1 is the smallest object on which all of this is visible:

import numpy as np

    # XOR as a 2-2-1 network with a tanh hidden layer and a sigmoid output.
    # One pass of this loop is one EPOCH; each inner step is one MINI-BATCH update.
X = np.array([[0.0, 0.0, 1.0, 1.0],      # 2x4 — columns are the four inputs
              [0.0, 1.0, 0.0, 1.0]])
Y = np.array([[0.0, 1.0, 1.0, 0.0]])      # 1x4 — the XOR targets

rng = np.random.default_rng(42)     # seed 42; seed 1 does NOT converge here
W1 = rng.normal(scale=0.5, size=(2, 2))  # 2x2 hidden weights
b1 = np.zeros((2, 1))
W2 = rng.normal(scale=0.5, size=(1, 2))  # 1x2 output weights
b2 = np.zeros((1, 1))

B, eta, epochs = 2, 0.5, 10000

for epoch in range(epochs):
    order = rng.permutation(X.shape[1])  # reshuffle every epoch
    for start in range(0, X.shape[1], B):
        idx = order[start:start + B]
        x, y = X[:, idx], Y[:, idx]      # B is the mini-batch size

        # ---- forward pass, storing z and a for the backward pass ----
        z1 = W1 @ x + b1
        a1 = np.tanh(z1)
        z2 = W2 @ a1 + b2
        yhat = 1.0 / (1.0 + np.exp(-z2))

        # ---- backward pass: the chain rule, run in reverse ----
        # With sigmoid + binary cross-entropy, dL/dz2 collapses to (yhat - y).
        d2 = yhat - y                                # (1, B)
        dW2 = d2 @ a1.T                              # (1, 2)
        db2 = d2.sum(axis=1, keepdims=True)          # (1, 1)
        d1 = (W2.T @ d2) * (1.0 - a1 ** 2)           # (2, B) — tanh' = 1 - a^2
        dW1 = d1 @ x.T                               # (2, 2)
        db1 = d1.sum(axis=1, keepdims=True)          # (2, 1)

        # ---- update, averaged over the batch by 1/B ----
        W1 -= eta * (dW1 / B)
        b1 -= eta * (db1 / B)
        W2 -= eta * (dW2 / B)
        b2 -= eta * (db2 / B)

print(np.round(1.0 / (1.0 + np.exp(-(W2 @ np.tanh(W1 @ X + b1) + b2))), 3))
# [[0. 1. 1. 0.]] — XOR solved, learned by gradient descent alone

Two details in that run are worth more than the result. First, the printed array comes from W2 @ np.tanh(W1 @ X + b1), re-evaluating the forward pass on the trained weights — the loop never stores a final prediction, so re-running the forward pass is how you read the model out. Second, the seed matters, and that is the lesson, not a footnote. The same code with default_rng(1) gets stuck around a loss of 0.35 with outputs [0.0, 0.48, 1.0, 0.48] — two of the four points sit just below the $0.5$ threshold. Gradient descent on XOR has a genuine local minimum in its basin of attraction, and some initialisations land in it while others do not. This is the smallest possible demonstration of why stochastic initialisation, restarts, and momentum exist.

Ten thousand epochs on four examples is not efficiency, it is legibility. What the code shows is that backpropagation requires no special machinery: store the forward intermediates, multiply by transposed matrices, subtract the gradient. The matrix shapes are the only thing that has to be right, and they follow mechanically from the architecture.

7. Why Depth Helps

There are three honest reasons deep networks beat shallow ones, plus one theorem that says nothing on its own.

7.1 Compositional features

Real data is hierarchical. An image of a face is composed of edges, which compose into eyes and mouths, which compose into the face. A deep network builds features at each layer by composing the layer below's features. A one-hidden-layer network can in principle represent the same function, but it has to learn the entire composition in a single flat layer, which requires exponentially more units for the same accuracy. The empirical finding across vision, speech and language is that depth reduces the parameter count needed to reach a given accuracy by orders of magnitude.

7.2 Narrower layers per level of abstraction

If each layer is allowed to be narrow, the network is forced to compress information as it goes up. The first layer might output 64 features per pixel; the next aggregates them into 32; the next into 16; and so on. This bottleneck structure forces the network to keep only the information that is useful for the task. Wide-and-shallow networks have no such inductive pressure and tend to memorise.

7.3 Better optimisation dynamics

Empirically, deeper networks trained with the right optimiser (Adam, SGD with momentum) generalise better than shallow networks with the same parameter count. The geometry of the loss surface changes with depth in ways that interact well with stochastic gradient methods. This is not a theorem; it is a measured property of high-dimensional optimisation that is still an active research area.

7.4 What depth is not

Depth is not a substitute for data, not a substitute for architecture, and not a substitute for tuning. A 50-layer network on 100 examples overfits. A 50-layer network with the wrong activation function dies. The honest summary is that depth is the inductive bias that matches the compositional structure of the data; the data still has to be there, and the architecture still has to fit the problem.

8. Universal Approximation, Honestly

The universal approximation theorem says that a feed-forward network with one hidden layer containing a finite but possibly very large number of non-linear units can approximate any continuous function on a compact domain to arbitrary accuracy. This is mathematically true and almost entirely useless as practical guidance. It does not say:

• how many hidden units are needed — the number can grow exponentially with the input dimension;
• how to find the right weights — the theorem is an existence statement, not an optimisation guarantee;
• how well the approximation generalises — the theorem is about fitting a function on a compact set, not about test-set error on finite data.

The honest reading is that depth is not necessary for representational power (a wide one-hidden-layer network can represent everything a deep network can) but is necessary for efficient representation and good generalisation on real data. The 2010s wave of deep-learning results — ImageNet, BERT, GPT — is not a vindication of universal approximation; it is a vindication of depth + stochastic gradient descent + a lot of data + a lot of compute. Understanding the distinction matters because it tells you what to do when the network fails: if the failure is representational, add depth; if the failure is optimisation, lower the learning rate; if the failure is generalisation, regularise or get more data.

Key Takeaways

• A single perceptron cannot represent XOR; one hidden layer of non-linear units fixes it. The non-linearity is the load-bearing piece: stacking affine layers without non-linear activations collapses to a single affine map.
• One fully-connected layer is the affine transform $z = W x + b$, with $W \in \mathbb{R}^{n_{\text{out}} \times n_{\text{in}}}$ and shapes consistent across the chain. Two layers give $z^{(1)} = W^{(1)} x + b^{(1)}$, $a^{(1)} = \sigma(z^{(1)})$, $z^{(2)} = W^{(2)} a^{(1)} + b^{(2)}$, $\hat{y} = \sigma_{\text{out}}(z^{(2)})$.
• ReLU is the default hidden activation; sigmoid is for binary output; softmax for multi-class output. Sigmoid and tanh suffer from vanishing gradients when saturated, which is why they are no longer the hidden-layer default.
• Backpropagation is the chain rule run in reverse: at each layer, the error signal $\delta^{(l)} = (W^{(l+1)\top} \delta^{(l+1)}) \odot \sigma'(z^{(l)})$, and the weight gradient $\partial L / \partial W^{(l)} = \delta^{(l)} \, a^{(l-1)\top}$. Forward cost and backward cost are both matrix-multiplies-per-layer, with backward roughly twice forward.
• Epoch = one full pass over the training set; batch = one gradient step over $B$ examples; mini-batch gradient descent with $B \in \{32, 64, 128, 256\}$ is the practical default for deep learning.
• Depth wins because data is compositional, because narrow layers force useful bottlenecks, and because deeper loss surfaces are better behaved under stochastic gradient descent — not because of universal approximation.