Neural Networks in Data Mining

The actual mechanics of how a neural network learns — a full forward pass, the chain-rule-based backward pass, and a verified weight update that measurably reduces the network's error — are covered step by step in the lesson on classification by backpropagation, on a real (if small) network. This lesson doesn't repeat that derivation; instead, it covers what a neural network architecture choice actually determines, and works through a specific, verifiable reason one activation function replaced another as the default choice in modern networks.

Why Different Network Architectures Exist at All

A neural network's architecture — how its neurons are connected — isn't an arbitrary design choice; each major architecture exists to exploit a specific structural property of a specific kind of data:

• Feedforward Networks (FNNs) connect every layer only to the next one, with no loops and no memory of previous inputs. This is the correct default for ordinary tabular data, where each record is independent of the others (there is no meaningful "previous record" the way there is a previous word in a sentence).
• Recurrent Neural Networks (RNNs) add connections that loop back on themselves, letting the network's output at one step depend on what it processed at previous steps. This directly targets sequential data — text, speech, and the time series covered in the lesson on time series clustering — where the order of the data carries information an FNN, with no memory of what came before, simply cannot use.
• Convolutional Neural Networks (CNNs) apply small filters that slide across an input, exploiting the fact that in an image, nearby pixels are far more related to each other than distant ones. This spatial-locality assumption is what makes CNNs efficient for images in a way an FNN — which would need a separate, independent connection for every single pixel pair — is not. Choosing an architecture is really choosing which structural assumption about the data to build in: no order or spatial relationship at all (FNN), an important sequence (RNN), or an important spatial layout (CNN).

  A Concrete Reason ReLU Replaced Sigmoid

  The lesson on classification by backpropagation uses the sigmoid activation function in its worked example, and sigmoid was in fact the standard choice for years — but modern deep networks default to ReLU (f(x) = max(0, x)) instead, for a reason that becomes visible by computing actual numbers. Backpropagation, as that lesson shows, updates each weight using the derivative of the activation function, multiplied together across every layer via the chain rule. Sigmoid's derivative is sigmoid(x) x (1 - sigmoid(x)), and it shrinks toward zero as its input moves away from zero in either direction:

  At x = 5:
  sigmoid(5) ≈ 0.9933
  sigmoid'(5) = 0.9933 x (1 - 0.9933) = 0.9933 x 0.0067 ≈ 0.00665

  A derivative of 0.00665 is already quite small for a single layer — and because backpropagation multiplies these derivatives together across every layer on the way back through the network (the chain rule, as applied in the lesson on classification by backpropagation), stacking several such small values compounds the shrinkage further, a problem known as the vanishing gradient: in a deep network, the weight updates reaching the earliest layers can become too small to meaningfully change anything, effectively stalling learning in those layers. ReLU's derivative, by contrast, is exactly 1 for any positive input, no matter how large:

  ReLU(5) = 5,  ReLU'(5) = 1
  ReLU(-5) = 0, ReLU'(-5) = 0

  A derivative of exactly 1 (for any positive input) doesn't shrink no matter how many layers it's multiplied across, which is precisely why ReLU allows meaningfully deeper networks to train in practice than sigmoid does — this is a concrete, computable reason for a design choice the source material would otherwise just assert as a fact to memorize.

  Where the Rest of This Topic Is Covered

• The core learning algorithm (forward pass, backpropagation, weight updates) — covered with a full hand-verified worked example in the lesson on classification by backpropagation.
• Overfitting and regularization (dropout, weight decay) — the same overfitting risk covered with a hand-verified cross-validation example in the lesson on major issues in data mining, applied to networks specifically by randomly disabling neurons (dropout) or penalizing large weights (weight decay) during training.
• Interpretability trade-offs — covered in the lesson on major issues in data mining, which applies directly to neural networks: a deep network with many layers is exactly the kind of high-accuracy, low-interpretability model that lesson's trade-off describes.
• Ethical concerns (bias, privacy, adversarial vulnerability) — covered in the lesson on the challenges of data mining.
• Applications (image recognition, fraud detection, healthcare diagnosis) — covered with concrete examples in the lesson on data mining applications.