# Layers In a typical deep neural network, highest-level blocks, which perform different kinds of transformations on their inputs are called layers. A layer wraps a group of nodes and performs a specific mathematical computation, offering a shortcut for building a more complex neural network. In Marian, for example, the `mlp::dense` layer represents a fully connected layer, which implements the operation `output = activation(input * weight + bias)`. A dense layer in the graph can be constructed with the following code: ```cpp // add input node x auto x = graph->constant({120,5}, inits::fromVector(inputData)); // construct a dense layer in the graph auto layer1 = mlp::dense() ("prefix", "layer1") // prefix name is layer1 ("dim", 5) // output dimension is 5 ("activation", (int)mlp::act::tanh) // activation function is tanh .construct(graph)->apply(x); // construct this layer in graph // and link node x as the input ``` The options are passed to the layer using pairs of `(key, value)`, where `key` is a predefined option, and `value` is the option value. Then `construct()` is called to create a layer instance in the graph, and `apply()` to link the input with this layer. Alternatively, the same layer can be created defining nodes and operations directly: ```cpp // construct a dense layer using nodes auto W1 = graph->param("W1", {120, 5}, inits::glorotUniform()); auto b1 = graph->param("b1", {1, 5}, inits::zeros()); auto h = tanh(affine(x, W1, b1)); ``` There are four categories of layers implemented in Marian, described in the sections below. ## Convolution layer To use a `convolution` layer, you first need to install [NVIDIA cuDNN](https://developer.nvidia.com/cudnn). The convolution layer supported by Marian is a 2D [convolution layer](https://en.wikipedia.org/wiki/Convolutional_neural_network#Convolutional_layers). This layer creates a convolution kernel which is used to convolved with the input. The options that can be passed to a `convolution` layer are the following: | Option Name | Definition | Value Type | Default Value | | ------------- |----------------|---------------|---------------| | prefix | Prefix name (used to form the parameter names) | `std::string` | `None` | | kernel-dims | The height and width of the kernel | `std::pair` | `None`| | kernel-num | The number of kernel | `int` | `None` | | paddings | The height and width of paddings | `std::pair` | `(0,0)`| | strides | The height and width of strides | `std::pair` | `(1,1)` | Example: ```cpp // construct a convolution layer auto conv_1 = convolution(graph) // pass graph pointer to the layer ("prefix", "conv_1") // prefix name is conv_1 ("kernel-dims", std::make_pair(3,3)) // kernel is 3*3 ("kernel-num", 32) // kernel no. is 32 .apply(x); // link node x as the input ``` ## MLP layers Marian offers `mlp::mlp`, which creates a [multilayer perceptron (MLP)](https://en.wikipedia.org/wiki/Multilayer_perceptron) network. It is a container which can stack multiple layers using `push_back()` function. There are two types of MLP layers provided by Marian: `mlp::dense` and `mlp::output`. The `mlp::dense` layer, as introduced before, is a fully connected layer, and it accepts the following options: | Option Name | Definition | Value Type | Default Value | | ------------- |----------------|---------------|---------------| | prefix | Prefix name (used to form the parameter names) | `std::string` | `None` | | dim | Output dimension | `int` | `None` | | layer-normalization | Whether to normalise the layer output or not | `bool` | `false` | | nematus-normalization | Whether to use Nematus layer normalisation or not | `bool` | `false` | | activation | Activation function | `int` | `mlp::act::linear` | The available activation functions for mlp are `mlp::act::linear`, `mlp::act::tanh`, `mlp::act::sigmoid`, `mlp::act::ReLU`, `mlp::act::LeakyReLU`, `mlp::act::PReLU`, and `mlp::act::swish`. Example: ```cpp // construct a mlp::dense layer auto dense_layer = mlp::dense() ("prefix", "dense_layer") // prefix name is dense_layer ("dim", 3) // output dimension is 3 ("activation", (int)mlp::act::sigmoid) // activation function is sigmoid .construct(graph)->apply(x); // construct this layer in graph and link node x as the input ``` The `mlp::output` layer is used, as the name suggests, to construct an output layer. You can tie embedding layers to `mlp::output` layer using `tieTransposed()`, or set shortlisted words using `setShortlist()`. The general options of `mlp::output` layer are listed below: | Option Name | Definition | Value Type | Default Value | | ------------- |----------------|---------------|---------------| | prefix | Prefix name (used to form the parameter names) | `std::string` | `None` | | dim | Output dimension | `int` | `None` | | vocab | File path to the factored vocabulary | `std::string` | `None` | | output-omit-bias | Whether this layer has a bias parameter | `bool` | `true` | | lemma-dim-emb | Re-embedding dimension of lemma in factors, must be used with `vocab` option | `int` | `0` | | output-approx-knn | Parameters for LSH-based output approximation, i.e., `k` (the first element) and `nbit` (the second element) | `std::vector` | None | Example: ```cpp // construct a mlp::output layer auto last = mlp::output() ("prefix", "last") // prefix name is dense_layer ("dim", 5); // output dimension is 5 ``` Finally, an example showing how to create a `mlp::mlp` network containing multiple layers: ```cpp // construct a mlp::mlp network auto mlp_networks = mlp::mlp() // construct a mpl container .push_back(mlp::dense() // construct a dense layer ("prefix", "dense") // prefix name is dense ("dim", 5) // dimension is 5 ("activation", (int)mlp::act::tanh))// activation function is tanh .push_back(mlp::output() // construct a output layer ("dim", 5)) // dimension is 5 ("prefix", "mlp_network") // prefix name is mlp_network .construct(graph); // construct this mlp layers in graph ``` ## RNN layers Marian offers `rnn::rnn` for creating a [recurrent neural network (RNN)](https://en.wikipedia.org/wiki/Recurrent_neural_network) network. Just like `mlp::mlp`, `rnn::rnn` is a container which can stack multiple layers using `push_back()` function. Unlike mlp layers, Marian only provides cell-level APIs to construct RNN. RNN cells only process a single timestep instead of the whole batches of input sequences. There are two types of rnn layers provided by Marian: `rnn::cell` and `rnn::stacked_cell`. The `rnn::cell` is the base component of RNN and `rnn::stacked_cell` is a stack of `rnn::cell`. The few options of `rnn::cell` layer are listed below: | Option Name | Definition | Value Type | Default Value | | ------------- |----------------|---------------|---------------| | type | Type of RNN cell | `std::string` | `None` | There are nine types of RNN cells provided by Marian: `gru`, `gru-nematus`, `lstm`, `mlstm`, `mgru`, `tanh`, `relu`, `sru`, `ssru`. The general options for all RNN cells are the following: | Option Name | Definition | Value Type | Default Value | | ------------- |----------------|---------------|---------------| | dimInput | Input dimension | `int` | `None` | | dimState | Dimension of hidden state | `int` | `None` | | prefix | Prefix name (used to form the parameter names) | `std::string` | `None` | | layer-normalization | Whether to normalise the layer output or not | `bool` | `false` | | dropout | Dropout probability | `float` | `0` | | transition | Whether it is a transition layer | `bool` | `false` | | final | Whether it is an RNN final layer or hidden layer | `bool` | `false` | ```{note} Not all the options listed above are available for all the cells. For example, `final` option is only used for `gru` and `gru-nematus` cells. ``` Example for `rnn::cell`: ```cpp // construct a rnn cell auto rnn_cell = rnn::cell() ("type", "gru") // type of rnn cell is gru ("prefix", "gru_cell") // prefix name is gru_cell ("final", false); // this cell is the final layer ``` Example for `rnn::stacked_cell`: ```cpp // construct a stack of rnn cells auto highCell = rnn::stacked_cell(); // for loop to add rnn cells into the stack for(size_t j = 1; j <= 512; j++) { auto paramPrefix ="cell" + std::to_string(j); highCell.push_back(rnn::cell()("prefix", paramPrefix)); } ``` The list of available options for `rnn::rnn` layers: | Option Name | Definition | Value Type | Default Value | | ------------- |----------------|---------------|---------------| | type | Type of RNN layer | `std::string` | `gru` | | direction | RNN direction | `int` | `rnn::dir::forward` | | dimInput | Input dimension | `int` | `None` | | dimState | Dimension of hidden state | `int` | `None` | | prefix | Prefix name (used to form the parameter names) | `std::string` | `None` | | layer-normalization | Whether to normalise the layer output or not | `bool` | `false` | | nematus-normalization | Whether to use Nematus layer normalisation or not | `bool` | `false` | | dropout | Dropout probability | `float` | `0` | | skip | Whether to use skip connections | `bool` | `false` | | skipFirst | Whether to use skip connections for the layer(s) with `index > 0` | `bool` | `false` | Examples for `rnn::rnn()`: ```cpp // construct a `rnn::rnn()` container auto rnn_container = rnn::rnn( "type", "gru", // type of rnn cell is gru "prefix", "rnn_layers", // prefix name is rnn_layers "dimInput", 10, // input dimension is 10 "dimState", 5, // dimension of hidden state is 5 "dropout", 0, // dropout probability is 0 "layer-normalization", false) // do not normalise the layer output .push_back(rnn::cell()) // add a rnn::cell in this rnn container .construct(graph); // construct this rnn container in graph ``` Marian provides four RNN directions in `rnn::dir` enumerator: `rnn::dir::forward`, `rnn::dir::backward`, `rnn::dir::alternating_forward` and `rnn::dir::alternating_backward`. For rnn::rnn(), you can use `transduce()` to map the input state to the output state. An example for `transduce()`: ```cpp auto output = rnn.construct(graph)->transduce(input); ``` ## Embedding layer Marian provides a shortcut to construct a regular embedding layer `embedding` for words embedding. For `embedding` layers, there are following options available: | Option Name | Definition | Value Type | Default Value | | ------------- |----------------|---------------|---------------| | dimVocab | Size of vocabulary| `int` | `None` | | dimEmb | Size of embedding vector | `int` | `None` | | dropout | Dropout probability | `float` | `0` | | inference | Whether it is used for inference | `bool` | `false` | | prefix | Prefix name (used to form the parameter names) | `std::string` | `None` | | fixed | whether this layer is fixed (not trainable) | `bool` | `false` | | dimFactorEmb | Size of factored embedding vector | `int` | `None` | | factorsCombine | Which strategy is chosen to combine the factor embeddings; it can be `"concat"` | `std::string` | `None` | | vocab | File path to the factored vocabulary | `std::string` | `None` | | embFile | Paths to the factored embedding vectors | `std::string>` | `None` | | normalization | Whether to normalise the layer output or not | `bool` | `false` | Example to construct an embedding layer: ```cpp // construct an embedding layer auto embedding_layer = embedding() ("prefix", "embedding") // prefix name is embedding ("dimVocab", 1024) // vocabulary size is 1024 ("dimEmb", 512) // size of embedding vector is 512 .construct(graph); // construct this embedding layer in graph ```