Classification by Backpropagation in Data Mining

A neural network — a model built from layers of connected artificial neurons, loosely inspired by how brain cells connect to each other — can represent far more complex, non-linear relationships than the simpler classifiers covered elsewhere in this series. But that complexity creates a real problem: with potentially thousands of connection weights, how does the network know which specific weights to adjust, and by how much, when its prediction is wrong? Backpropagation is the algorithm that answers exactly this question, and the answer is precise enough to work through by hand on a tiny network.

What a Neuron Actually Computes

Each neuron performs two steps: it computes a weighted sum of its inputs plus a bias term (z = w·x + b), then passes that sum through an activation function to produce its output, or activation (a). The activation function matters specifically because it has to be differentiable — backpropagation's entire mechanism depends on being able to compute a derivative through every step of the network, which is why a smooth function like the sigmoid (sigmoid(z) = 1 / (1 + e^-z)) is a standard choice for this kind of worked example.

A Complete Forward and Backward Pass, Worked by Hand

The clearest way to actually understand backpropagation is to run one full cycle — forward pass, error calculation, backward pass, weight update — on the smallest network that still has something to backpropagate through: one input, one hidden neuron, one output neuron, both using sigmoid activation. Setup: input x = 1.0, target output y = 1.0, initial weights w1 = 0.5 (input → hidden) and w2 = 1.0 (hidden → output), biases set to 0 for simplicity. Step 1 — Feed forward.

z_h = w1 * x = 0.5 * 1.0 = 0.5
a_h = sigmoid(0.5) ≈ 0.6225
z_o = w2 * a_h = 1.0 * 0.6225 = 0.6225
a_o = sigmoid(0.6225) ≈ 0.6508      <- this is the network's prediction

Step 2 — Calculate the error. Using squared error loss, L = 0.5 * (y - a_o)²:

L = 0.5 * (1.0 - 0.6508)² = 0.5 * (0.3492)² ≈ 0.0610

Step 3 — Propagate the error backward, using the chain rule at every step. The gradient of the loss with respect to w2 requires three factors multiplied together — exactly what "backward propagation" means mechanically: each factor comes from one layer, computed in reverse order from where the error was measured.

dL/da_o   = -(y - a_o)             = -(1.0 - 0.6508)      ≈ -0.3492
da_o/dz_o = a_o * (1 - a_o)        = 0.6508 * 0.3492       ≈  0.2273
dz_o/dw2  = a_h                    = 0.6225
dL/dw2 = (-0.3492) * (0.2273) * (0.6225) ≈ -0.0494

Continuing one layer further back, to w1, reuses the first two factors already computed above and simply continues the chain through the hidden layer:

dL/da_h   = dL/da_o * da_o/dz_o * dz_o/da_h  = (-0.3492) * (0.2273) * 1.0  ≈ -0.0794
da_h/dz_h = a_h * (1 - a_h)                   = 0.6225 * 0.3775            ≈  0.2350
dz_h/dw1  = x                                 = 1.0
dL/dw1 = (-0.0794) * (0.2350) * (1.0) ≈ -0.0187

Step 4 — Update the weights, moving each one in the direction that reduces the loss (learning rate η = 0.5):

w2_new = w2 - η * dL/dw2 = 1.0 - 0.5 * (-0.0494) ≈ 1.0247
w1_new = w1 - η * dL/dw1 = 0.5 - 0.5 * (-0.0187) ≈ 0.5093

Verifying the update actually worked: running the forward pass again with these updated weights gives a_h ≈ 0.6247, a_o ≈ 0.6548, and a new loss of L ≈ 0.0596 — down from 0.0610. The weight update moved the prediction closer to the target, exactly as gradient descent is supposed to do, and this is precisely how a real network with thousands of weights instead of two reaches a good solution: the same four steps, repeated over and over, each time nudging every weight a small amount in the direction its own gradient says will reduce the error.

Where the Algorithm Actually Comes From

The mathematical foundation — using the chain rule to compute how a change deep inside a system affects its final output — traces back to control theory and calculus techniques from the 1960s and 70s, with Paul Werbos first applying this specifically to neural network training in his 1974 doctoral thesis. The algorithm became widely known in the machine learning community after Rumelhart, Hinton, and Williams published "Learning Representations by Back-Propagating Errors" in Nature in 1986 — the paper most often credited with popularizing backpropagation as the standard way to train multi-layer neural networks.

Two Contexts Backpropagation Runs In

Static backpropagation is what the worked example above demonstrates: a fixed input produces a fixed output, appropriate for classification tasks like optical character recognition where each input is independent of every other. Recurrent backpropagation applies to networks whose output depends on previous internal states — rather than settling immediately, the network's activations are updated repeatedly until they converge to a stable output, which takes iterative computation rather than the single, immediate forward pass used above.

Why This Matters for Classification

Backpropagation is what makes a multi-layer neural network trainable at all for the classification tasks covered throughout this series — character and signature recognition, image classification — precisely because it provides an efficient, exact way to compute how every single weight in a large network should change, without which "learning" would mean nothing more than random guessing at weight values and hoping for improvement.