Random Rotation vs the Killer Outlier

Low-bit quantization dies on a few giant outlier entries — the step size must span the whole range, so every small value gets crushed to noise. A randomized Hadamard rotation (random ±1 sign flip → fast Walsh–Hadamard transform) smears each outlier's energy across all d coordinates, flattening the distribution so quantization barely hurts. It costs only O(d log d) and is exactly why modern KV-cache / weight quantizers rotate first.

arXiv:2605.13810 · d = 128 · orthogonal transform, so rotate → quantize → rotate-back preserves error exactly

Vector preset

Bit depth · 4-bit

Rotation

Raw vector RAW SPACE

drag on the bars to sculpt / add outliers

value quantized (reconstruction) per-coordinate error grid lines = quantization levels

Naive MSE

—

quantize raw · max err —

Rotated MSE

—

rotate → quant → back · max err —

Error reduction

—

lower is the whole point

Effective range

—

raw vs rotated abs-max

Coefficient magnitude distribution

how spread-out the energy is — flat = easy to quantize

raw |x| (spiky — one bin dominates) rotated |Rx| (flattened bell)

checking WHT orthogonality… R = H·S, H = FWHT/√d, S = diag(±1) · RᵀR = I ⇒ error preserved under rotation