Adaptive Layer Norm

$$\text{LayerNorm}(x) = \gamma \odot \frac{x - \mu}{\sigma} + \beta$$

where $\gamma$ (scale) and $\beta$ (shift) are learned parameters, fixed after training.

$$\text{adaLN}(x, c) = \gamma(c) \odot \frac{x - \mu}{\sigma} + \beta(c)$$

where $\gamma(c)$ and $\beta(c)$ are predicted from a conditioning input $c$ via a small MLP:

# Typical implementation
def adaln(x, c, norm):
    # c: conditioning embedding (e.g., timestep + class)
    gamma, beta = mlp(c).chunk(2, dim=-1)
    return gamma * norm(x) + beta

adaLN-Zero (from # DiT) adds a third parameter $\alpha$ that scales the output of attention/MLP blocks:

$$h = h + \alpha(c) \odot \text{Block}(\text{adaLN}(h, c))$$

The key insight: initialize $\alpha = 0$. This makes each transformer block an identity function at initialization, enabling stable training of deep networks.

adaLN allows conditioning information to modulate feature statistics at every layer without:

• Adding cross-attention layers (computationally expensive)
• Increasing sequence length (in-context tokens)

The conditioning "steers" the network's internal representations by adjusting how features are normalized—a lightweight but powerful mechanism.