Essay, 2026

How a neural network actually learns.

Most explanations of a neural network present the trained object as a finished thing and ask you to admire it. This essay does the opposite. We build a network out of nothing, train it in your browser while you watch every weight change, and end with you drawing a digit on a canvas and seeing the network classify it. The whole machine is two hundred lines of matrix arithmetic. No frameworks. No pretrained models. The training runs on this tab.

Reading time about twenty five minutes. Five figures, all default paused. Keyboard, screen reader, and reduced motion supported.

Begin reading

Part one

The task and the data.

A handwritten digit is a small two dimensional grid of greyscale values. By convention it is 28 by 28 pixels, each pixel a number between zero and one, where one is ink and zero is paper. The task we will give the network is the canonical one. Given the 784 pixel values of one such grid, return a probability over the ten possible digit classes. The challenge is that no two human zeros are pixel-identical, and the network has to discover, on its own, the structural commonalities that make a zero a zero.

We will not download a dataset. The figure below generates digits on the fly with a procedural renderer that draws each digit with a random stroke width, a small rotation, and a small offset. The resulting samples vary the way human handwriting varies, which is enough to make the classification task non-trivial while keeping the page weight under fifty kilobytes. Pick a class to filter the grid.

Figure 1. A procedurally generated digit dataset.

Showing 100 samples

Each tile is 28 by 28 pixels. Variation across tiles is the noise model. The network's job is to find a function that maps any tile to the correct class.

Notice that the digits in any single class are not pixel-identical. They are not even close. A four written with a closed top is a different image from a four written with an open top, but a human recognises both. The function we want is a piece of mathematics that does the same.

Part two

A single neuron.

The smallest unit of a neural network is a neuron. A neuron takes its inputs, multiplies each by a weight, sums them, adds a bias, and passes the result through a non-linear function. In symbols, the output of a neuron with weights w, bias b, and input x is y = σ(w · x + b). For binary classification the conventional choice of σ is the logistic sigmoid, which squashes any real number into the interval zero to one. We can read that number as a probability.

The figure below trains exactly one such neuron to distinguish a zero from a one. The inputs are flattened 784 dimensional vectors. The weights are a 784 dimensional vector of the same shape, which we can also reshape into a 28 by 28 image. As training proceeds, watch the weight image evolve. It will end up looking, faintly but unmistakably, like the digit one with negative weight wherever a zero usually has ink.

Figure 2. Train one neuron to classify zero against one.

Neuron weights as a 28 by 28 image

A 28 by 28 heatmap of the neuron's learned weights. Red cells are positive, blue cells are negative. As training proceeds, the pattern resembles the digit one with negative weight where the digit zero usually has ink.

positive weight (favours class 1)   negative weight (favours class 0)

Loss

Epoch
0
Loss
—
Accuracy
—

Sample predictions

The neuron is trained with mini batch stochastic gradient descent on binary cross-entropy loss. The training run is capped at 30 epochs so the figure terminates. The Stop button hard-terminates and clears the busy state.

Two observations are worth holding on to. First, a single neuron is already enough to separate two classes that are linearly separable in pixel space, which is roughly the case for zero versus one. Second, the trained weight image is interpretable. The neuron has discovered, by gradient descent and nothing else, that a one looks like ink in a vertical stripe and that a zero looks like ink in a ring. The whole essay of this section is that the network's knowledge is not hidden. It is the values of the weights.

Part three

Ten neurons, one layer.

Distinguishing two classes is not the task we set out to do. To classify all ten digits we need ten output neurons, one per class. The probability assigned to class k by such a layer is computed by the softmax function, which exponentiates each output and divides by the sum. The loss we minimise is the cross entropy between the predicted distribution and the true one-hot distribution. The optimiser is the same mini batch stochastic gradient descent.

What is striking is what the ten weight images look like after training. Each one is a kind of stencil for its assigned digit. The seven neuron has a strong positive band along the top and along a diagonal. The zero neuron has a ring. The three neuron has two bowls. These stencils are not designed. They are the function the network discovered.

Figure 3. Ten weight images, learning together.

Weight images, one per output class

Training curves

Epoch
0
Loss
—
Accuracy
—

Confusion matrix

Sample predictions

A linear classifier of this kind tops out at about ninety percent accuracy on this dataset. The remaining ten percent are digits whose pixel pattern is not linearly separable from another class. Part four removes that ceiling.

Part four

A hidden layer.

A linear classifier draws straight lines in input space. A digit dataset is not separated by straight lines. The mathematical content of a hidden layer is to introduce a non-linear transformation that warps the input space, in such a way that the warped space is linearly separable. We will use the rectified linear unit, defined as ReLU(z) = max(0, z), applied element-wise. It is the cheapest non-linearity that works.

The figure below puts a 16 neuron hidden layer between the input and the output. The four panels show the cascade. A sample digit on the left. The activations of the sixteen hidden neurons as a small heatmap. The output probabilities as a bar chart. The training curve on the right. Hover a hidden neuron in the heatmap to highlight the corresponding row of the input-to-hidden weight matrix and see what input pattern that neuron is sensitive to.

Figure 4. A multilayer perceptron.

Input · 28×28

True class: —

Hidden · ReLU activations

Output · softmax probabilities

    Hovered neuron, input-to-hidden weights

    Hover any of the 16 hidden neurons to inspect.

    Training curve

    Epoch
    0
    Loss
    —
    Accuracy
    —
    Params
    —

    The MLP trains for 20 epochs. With sixteen hidden units it consistently reaches ninety five percent accuracy on this synthetic dataset, beating the linear classifier in part three by a clear margin. Increasing the hidden size further is informative: the gain tapers because the dataset itself only contains so much structure.

    Part five

    Draw a digit. See it classified.

    The synthesis. The MLP trained in part four is now the engine behind the figure below. Draw a digit in the canvas with your mouse, finger, or pen. Releasing the pointer runs the input through the network and updates the cascade. If you cannot draw, the sample digit picker beside the canvas chooses preset digits keyboard-accessibly and runs the same inference on them.

    Two things to try. First, draw a clean digit and watch the cascade light up appropriately. Second, draw an ambiguous digit, such as a one that leans like a seven, and watch the probabilities split. The split is the network expressing uncertainty in the only language it has.

    Figure 5. Your handwriting, classified live.

    Drawing area · 280×280

    Click and drag, or pick a preset.

    Downsampled · 28×28

    Hidden · activations

    Output · probabilities

      Press Train classifier (or just draw something; the network will auto-train on first interaction).

      If the network from part four has already been trained, this figure reuses those weights. Otherwise, the first preset selection or pointer-up triggers an auto-training pass on the procedural dataset, then runs inference. Trained weights persist for the rest of the session, so you can keep drawing without retraining.

      If the network gets your handwriting wrong, you have learned the most useful lesson of this essay. The function it learned is a function of the data it saw. Our procedural dataset is reasonable, but the strokes are clean, the orientations are limited, and the noise model is generous. Real human handwriting contains shapes the network has never been exposed to, and the failure modes you elicit are the failure modes a real model exhibits when deployed on data that differs from its training distribution. This is called distribution shift, and it is the central practical problem of applied machine learning.

      Colophon

      Every simulation runs client side in vanilla JavaScript. The forward and backward passes are written out in plain matrix arithmetic, around two hundred lines of code. Weights are initialised with He initialisation. Optimisation is mini batch stochastic gradient descent with cross entropy loss. The dataset is generated by drawing each digit with the Canvas API at a random stroke width, rotation, and offset; this avoids any external download and keeps the page weight under fifty kilobytes.

      Sources: Rumelhart, Hinton and Williams (1986), Learning representations by back-propagating errors, Nature 323. LeCun, Bottou, Bengio and Haffner (1998), Gradient-based learning applied to document recognition, Proceedings of the IEEE 86(11), for the canonical statement of the digit task. Glorot and Bengio (2010) and He et al. (2015) on initialisation. Goodfellow, Bengio and Courville (2016), Deep Learning, MIT Press, for the textbook treatment.

      Built as a portfolio piece. The full source is on GitHub. Suggestions are welcome.