Overview

From one ReLU to learned functions, network shape, and richer inputs.

01 — Approximation

Fit a 1D target with a two-layer ReLU MLP. Slide H to see how more hidden units improve the piecewise-linear approximation.
02 — Effect of ReLU on Kinks
Shift different functions with bias and see how ReLU creates or removes kinks, including on the Telgarsky sawtooth.
03 — Width vs Depth

Fix the total neuron budget N, then vary the aspect ratio (width/depth). Compare wide-shallow vs narrow-deep networks on the same task.
04 — Sinusoidal Positional Encoding

Side-by-side: plain MLP vs pos-enc MLP fitting a noisy scatter plot. Drag H and L to see how encoding unlocks high-frequency fitting.