Type promotion semantics¶
JAX’s type promotion rules (i.e., the result of
jax.numpy.promote_types() for each pair of types) are given by the
following table, where, for example
Jax’s type promotion rules differ from those of NumPy, as given by
numpy.promote_types(), in those cells highlighted with a green background
in the table above. There are two key differences:
when promoting an integer or boolean type against a floating-point or complex type, JAX always prefers the type of the floating-point or complex type.
Accelerator devices, such as GPUs and TPUs, either pay a significant performance penalty to use 64-bit floating point types (GPUs) or do not support 64-bit floating point types at all (TPUs). Classic NumPy’s promotion rules are too willing to overpromote to 64-bit types, which is problematic for a system designed to run on accelerators.
JAX uses floating point promotion rules that are more suited to modern accelerator devices and are less aggressive about promoting floating point types. The promotion rules used by JAX for floating-point types are similar to those used by PyTorch.
JAX supports the bfloat16 non-standard 16-bit floating point type (
jax.numpy.bfloat16), which is useful for neural network training. The only notable promotion behavior is with respect to IEEE-754
float16, with which
bfloat16promotes to a