One perceptron learns T vs A

Training patterns

Letter T target +1
Letter A target −1

Each square is one input: ink = 1, empty = 0. Click either teaching pattern to edit it.

Audience test

not used for learning

Perceptron

A single perceptron with 25 pixel inputs Twenty-five input nodes connect by weighted lines into a weighted sum and threshold, producing either T or A. 25 pixels weighted sum Σ 0.00 threshold
Tie-break T*
score 0.00
AT
score = Σ(pixel × weight) + bias
Untrained. Press Step once to watch one example, or Train to run.

Weights & progress

Examples seen0
Weight updates0
Epochs0

25 learned weightsneutral

votes Avotes T
bias
0.00

Mistakes per epochgoal: zero

No completed passes yet.

The rule: only a mistake changes the weights. Correct answers leave the machine untouched.

From one decision to a 48-layer language factory

Keep the weighted sums. Replace the hard threshold with smooth activations, add attention, repeat the transformation in layers—and train everything together.

This demo binary classifier
Σ
One perceptron threshold → T or A
25pixel inputs
26learned parameters
GPT-2 XL · 2019 next-token model

GPT-2 1.5B

48 transformer blocks · width 1,600

25 attention heads / blocktokens exchange context
6,400 MLP units / blockneuron-like feature transformation
307,200MLP unit positions across 48 blocks*
1.542Blearned parameters

The adjustable knobs

Demo
26
GPT-2
1,542,000,000
≈59.3 million ×more parameters than this demo
“Neural”one input token
1,600continuous features
Attention25 heads
MLP6,400 units
repeat this transformer block × 48
50,257next-token scores

What happens inside a transformer block?

A token never jumps directly from input to answer. Its 1,600-number representation is refined twice on every floor: first by attention, then by an MLP.

Layer or block

One floor

A transformer block is one repeated processing module: attention, MLP, normalization and residual connections. In GPT‑2’s published “48 layers,” layer usually means one complete block—not one individual neuron.

GPT-2 XL stack× 48
Attention head

One learned view

An attention head lets the current token score earlier tokens and blend their information. GPT‑2 runs 25 heads in parallel, each in a 64-dimensional slice, then combines their results.

parallel views per block25 × 64
MLP units

Feature workshop

The MLP applies the same two learned transformations to each token separately. Its 6,400 intermediate activations are the part most similar to modern neuron-like units.

intermediate units per block6,400

Here, “layer” means one transformer block. Attention heads learn jointly; they do not have predefined jobs.