Back to Home
03
Forward Pass
Watch one token flow through the entire model. Each step transforms the 16-number vector until it becomes 27 probability scores — one for each possible next character.
Select Input
Token:
Position:
Data flowing through the model
Each box shows the data at that stage. Click a step to see its explanation. The 16 numbers get transformed at each step. Colors show values: negative to positive.
Intermediate Vectors (16 dims)
Token embedding: wte['g']
-0.2
-0.224
0.5
0.526
0.8
0.791
0.3
0.330
-0.4
-0.435
-0.8
-0.800
-0.4
-0.429
0.3
0.336
0.8
0.792
0.5
0.520
-0.2
-0.230
-0.8
-0.769
-0.6
-0.601
0.1
0.120
0.7
0.730
0.7
0.669
Position embedding: wpe[0]
-0.4
-0.405
0.4
0.362
0.8
0.796
0.5
0.498
-0.3
-0.257
-0.8
-0.776
-0.6
-0.582
0.1
0.148
0.7
0.741
0.7
0.653
-0.0
-0.035
-0.7
-0.692
-0.7
-0.712
-0.1
-0.078
0.6
0.628
0.8
0.756
Combined (tok + pos)
-0.6
-0.629
0.9
0.887
1.6
1.587
0.8
0.828
-0.7
-0.693
-1.6
-1.576
-1.0
-1.011
0.5
0.484
1.5
1.534
1.2
1.174
-0.3
-0.266
-1.5
-1.461
-1.3
-1.313
0.0
0.042
1.4
1.358
1.4
1.426
After RMSNorm
-0.7
-0.672
0.9
0.949
1.7
1.698
0.9
0.886
-0.7
-0.741
-1.7
-1.686
-1.1
-1.081
0.5
0.518
1.6
1.640
1.3
1.255
-0.3
-0.284
-1.6
-1.562
-1.4
-1.404
0.0
0.045
1.5
1.453
1.5
1.525
After Attention + Residual
-1.4
-1.385
1.9
1.901
3.5
3.482
2.0
1.995
-1.1
-1.136
-3.0
-3.023
-2.0
-1.968
1.0
0.979
3.0
3.011
2.2
2.165
-0.8
-0.850
-3.3
-3.286
-2.9
-2.878
0.1
0.067
2.9
2.937
3.2
3.191
After MLP + Residual
-1.0
-0.985
2.3
2.283
3.8
3.812
2.2
2.244
-1.0
-0.991
-3.0
-2.994
-2.1
-2.059
0.8
0.777
2.7
2.716
1.8
1.803
-1.2
-1.246
-3.7
-3.681
-3.2
-3.236
-0.2
-0.223
2.7
2.741
3.1
3.107
Output: Next Token Probabilities
The model's prediction for what character comes after 'g' at position 0. Higher bars = more likely.
g
16.0%
u
7.0%
o
6.8%
k
6.7%
q
6.4%
h
5.8%
w
5.5%
f
5.5%
l
4.9%
y
4.1%
Key Concepts
Attention
Allows the model to look at previous positions and decide which ones are relevant. Each of 4 heads can focus on different patterns. Q="what am I looking for?", K="what do I contain?", V="what do I offer?"
Residual Connections
Add the output back to the input at certain points. This helps gradients flow during training and preserves original information alongside new insights.
RMSNorm
Normalizes values so they're on a similar scale. Prevents any dimension from dominating and makes training more stable.
MLP
A two-layer network: expand to 4x size (16→64), apply ReLU activation, then compress back (64→16). This refines and transforms the representation.