Function marian::inits::sinusoidalPositionEmbeddings

Function Documentation

Ptr<NodeInitializer> marian::inits::sinusoidalPositionEmbeddings(int start)

Computes Google’s sinusoidal position embeddings.

Computes Google’s Transformer-style sinusoidal position embeddings starting from position ‘start’ taking into account batch and time dimensions of the tensor. Expected tensor layout {-2: time, -1: model}. Usually gets later reshaped to {time, 1, model} and added with a broadcast to learned embeddings. Positional embeddings are the same for each batch entry and change over time steps.