Function marian::gelu(Expr)¶
Defined in File expression_operators.cpp
Function Documentation¶
-
Expr
marian::gelu(Expr a) Gaussian Error Linear Unit (GELU).
Computes an approxmiation to the Gaussian Error Linear Unit
\[ \operatorname{gelu}(x) = x \cdot \Phi(x) = x \cdot \frac{1}{2}\left[ 1 + \operatorname{erf}\left(\frac{x}{\sqrt{2}}\right) \right] \sim \operatorname{swish}(x, 1.702) \]using SwishNodeOp(a, 1.702)- See