BF16

BF16, brain float 16, is a 16-bit floating point format,

• 1 sign bit, 8 exponent bits, 7 mantissa bits
• Lower precision, same range as FP32
• Range: ~1e-38 to 3.4e38
• More stable for training, less likely to overflow