Function marian::rmsNorm(Expr, Expr, Expr, float)¶
Defined in File expression_operators.cpp
Function Documentation¶
-
Expr
marian::rmsNorm(Expr x, Expr gamma, Expr beta = nullptr, float eps = 1e-9) Applies RMS normalization over the last dimension.
See: Biao Zhang; Rico Sennrich (2019). Root Mean Square Layer Normalization. In Advances in Neural Information Processing Systems 32. Vancouver, Canada.
\[ \frac{x}{\sqrt{\frac{1}{N}\sum x^2 + \mathrm{eps}}} \times \gamma + \beta \]