Mechanistic Interpretability · WebGPU

In-Browser Attention Visualiser

Attention Lens runs the 124 million parameter GPT-2 model inside your browser, captures the attention matrices from every layer and every head, and renders them as interactive heatmaps, token to token connection views, and a full 12 by 12 model overview. Nothing leaves your device.

Input

Shorter prompts render faster. Nine tokens give a clean 9 by 9 heatmap per head.

Attention Visualisation

Six views share the same underlying tensor, switch with arrow keys inside the tablist.

Heads View

Attention heatmap

Layer 1 · Head 1
Low High
What am I seeing?

Each cell shows the attention weight from a query token (row) onto a key token (column). Rows sum to 1 after softmax with the causal mask. The top axis is the query token, the left axis is the key token. A strong diagonal means the head reads nearby tokens; a first-column stripe is the beginning-of-sequence sink. Use the Mono or Pattern toggles below if the colour ramp is hard to read.

Token to token

Source token connections

Incoming attribution

Sum over all heads and layers

Selection Controls

1
1
1

Ranges respect arrow keys, Home and End, plus Page Up and Page Down.

Performance
Device: detecting
RunTokeniseForwardExtractRenderTotal

About and credits

Attributions and references
  • GPT-2 architecture, Radford et al., 2019.
  • Transformers.js, Xenova, 2023.
  • WebGPU specification, W3C, 2024.
  • ONNX Runtime Web, Microsoft, 2022.
  • Attention is all you need, Vaswani et al., 2017.
  • In-context Learning and Induction Heads, Olsson et al., 2022.
  • Attention Rollout, Abnar and Zuidema, 2020.