Class ExpressionGraph

Inheritance Relationships

Base Type

  • public std::enable_shared_from_this< ExpressionGraph >

Derived Types

Class Documentation

class ExpressionGraph : public std::enable_shared_from_this<ExpressionGraph>

Main implementation of a computation graph.

Keeps a record of data (tensors) and all operations. Each operation in a computation graph is a Node. Each Node defines its forward and backward steps.

Subclassed by marian::ExpressionGraphONNXExporter, marian::ExpressionGraphPackable

Public Functions

ExpressionGraph(bool inference = false)

Constructs a new expression graph.

Constructor should be used as New<ExpressionGraph>().

virtual ~ExpressionGraph()

Destructor.

Clear everything related to the graph except memoized nodes.

void setDevice(DeviceId deviceId = {0, DeviceType::gpu}, Ptr<Device> device = nullptr)

Set device options used to run the graph.

Parameters

  • deviceId: a struct type which stores device no. (size_t) and device type (DeviceType::cpu or DeviceType::gpu)

  • device: a pointer to the device

DeviceId getDeviceId()

Get device info for the graph.

Return

deviceId a struct type which stores device no. (size_t) and device type (DeviceType::cpu or DeviceType::gpu)

Ptr<Backend> getBackend()

Get backend pointer for the graph.

Return

Ptr<Backend> pointer to backend

void setInference(bool inference)

Set whether the graph is used for inference only.

bool isInference()

Check whether the graph is used for inference only (true) or not.

void setCheckpointing(bool checkpointing)

Set whether the graph uses gradient checkpointing.

Gradient Checkpointing works by trading compute for memory, which reruns a forward-pass segment for each checkpoint segment during backward.

bool isCheckpointing()

Check whether the graph uses gradient checkpointing or not.

void switchParams(const std::string &newNamespace)

Set namespace (std::string) for the graph.

Each graph has its own unique namespace, which is used to form the name of a parameter object.

virtual void copyParams(Ptr<ExpressionGraph> graph)

Copy all parameter objects from one graph to current graph.

Parameters

  • graph: a pointer to a graph object

void reserveWorkspaceMB(size_t num)

Preallocate workspace memory (MB) for the graph.

Sets the size of the memory available for the forward and backward step of the training procedure. This does not include model size and optimizer parameters that are allocated outsize workspace.

void reuseWorkspace(Ptr<ExpressionGraph> graph)

Copy tensor objects from one graph to current graph.

void backprop()

Performs backpropagation on this expression graph.

Backpropagation is implemented by performing first the forward pass and then the backward pass of algorithmic differentiation (AD) on the nodes of the graph.

bool fits()

Perform one backpropagation process on the graph to test whether the graph workspace fits into a given workspace memory.

This function is used for searching the maximum batch size that fits into given workspace memory.

void checkNaN(Tensor t, bool &isNaN, bool &isInf)

Check whether the memory allocated for a tensor object contains a NaN or infinite value.

Parameters

  • t: a Tensor object

  • isNaN: a bool type holds the result whether the tensor contains a NaN value (pass by reference)

  • isInf: a bool type holds the result whether the tensor contains a infinite value (pass by reference)

void forward()

Perform the forward pass on the nodes of the graph.

The forward pass refers to the calculation process. It traverses through all nodes from input layer to output layer.

void forwardNext()

Perform the forward pass without memory allocation for parameters.

Helper function for forward().

void forward(std::list<Expr> &forwardTape, bool finalPass)

Perform forward pass on a given nodes with finalPass flag.

Helper function for forward() and backward().

Parameters

  • forwardTape: a pointer to the nodes used for forward pass

  • finalPass: a bool type which controls whether nodes should be freed with gradient-checkpointing

void backward(bool reset = true, float clipValue = 0.f)

Perform the backward pass on the trainable nodes of the graph.

The back pass refers to the process of computing the output error. It traverses through all nodes from output layer to input layer.

std::string graphviz()

Generate graph layout in Graphviz format for visualisation.

Return

a string presenting graph layout in Graphviz format (dot)

void graphviz(const std::string &filename)

Write graph layout in Graphviz format to a file.

Parameters

  • filename: a string type specifies filename that writes the graph layout

Expr param(const std::string &pname, const Shape &shape, const Ptr<inits::NodeInitializer> &init, const Type elementType, bool fixed = false)

Construct a parameter node in the graph.

Return

a pointer to the parameter node

Parameters

  • pname: a string type holds the name of the parameter node

  • shape: a struct type defines the shape of the parameter tensor e.g., shape={2,3} means 2D matrix with dim[0]=2 and dim[1]=3

  • init: a pointer to a NodeInitializer object, e.g., inits::zeros()

  • elementType: a scoped enumerator (enum class) defines the element type, e.g., Type::float16

  • fixed: a bool type specifies whether the parameter object is fixed (not trainable) or not. The default value is false which means the parameter is trainable.

Expr param(const std::string &pname, const Shape &shape, const Ptr<inits::NodeInitializer> &init, bool fixed = false)

Construct a parameter node in the graph without a specified type, and the type is set to defaultElementType_.

Return

a pointer to the parameter node

Parameters

  • pname: a string type holds the name of the parameter node

  • shape: a struct type defines the shape of the parameter tensor e.g., shape={2,3} means 2D matrix with dim[0]=2 and dim[1]=3

  • init: a pointer to a NodeInitializer object, e.g., inits::zeros()

  • fixed: a bool type specifies whether the parameter object is fixed (not trainable) or not. The default value is false which means the parameter is trainable.

Expr constant(const Shape &shape, const Ptr<inits::NodeInitializer> &init, Type elementType)

Construct a constant node in the graph without a specified type, and the type is set to defaultElementType_.

Return

a pointer to the constant node

Parameters

  • shape: a struct type defines the shape of the constant tensor e.g., shape={2,3} means 2D matrix with dim[0]=2 and dim[1]=3

  • init: a pointer to a NodeInitializer object, e.g., inits::zeros()

  • elementType: a scoped enumerator (enum class) defines the element type, e.g., Type::float16

Expr constant(const Shape &shape, const Ptr<inits::NodeInitializer> &init)

Construct a constant node in the graph without a specified type, and the type is set to defaultElementType_.

Return

a pointer to the constant node

Parameters

  • shape: a struct type defines the shape of the constant tensor e.g., shape={2,3} means 2D matrix with dim[0]=2 and dim[1]=3

  • init: a pointer to a NodeInitializer object, e.g., inits::zeros()

Expr indices(const std::vector<IndexType> &indicesVector)

Turn vector of indices to integer tensor.

A shortcut version to turn vector of indices to integer tensor, to be used with operators like rows() or index_select()

Parameters

  • indicesVector: a vector of IndexType (uint32_t) specifies the indexes

Expr indices(const std::vector<IndexType> &indicesVector, Expr indexee, int axis = -1)

Specify the indexes of elements to be taken from a tensor.

This version sets up the shape such that the indices are in a given axis. Use this if you want to pass these indices to gather(). E.g., indexee shape = (3, 2, 5, 2); axis = 1 -> resulting shape = (1, size of indicesVector, 1, 1):

  • The size of the resulting shape is the same as that of the indexee; here is 4.

  • The shape of the specified axis is equal to the size of given indicesVector.

  • The shapes of the rest axes are filled with 1.

    Parameters

    • indicesVector: a vector of IndexType (uint32_t) specifies the indexes

    • indexee: the source tensor that we want to select elements from

    • axis: specifies the axis that we want to collect along

Expr ones(const Shape &shape, Type elementType)

Construct a constant node filled with 1.

Parameters

  • shape: a struct type defines the shape of the constant dataset e.g., shape={2,3} means 2D matrix with dim[0]=2 and dim[1]=3

  • elementType: a scoped enumerator (enum class) defines the element type, e.g., Type::float16

Expr ones(const Shape &shape)

Construct a constant node filled with 1 without a specified type, and the type is set to defaultElementType_.

Parameters

  • shape: a struct type defines the shape of the constant dataset e.g., shape={2,3} means 2D matrix with dim[0]=2 and dim[1]=3

Expr zeros(const Shape &shape, Type elementType)

Construct a constant node filled with 0.

Parameters

  • shape: a struct type defines the shape of the constant dataset e.g., shape={2,3} means 2D matrix with dim[0]=2 and dim[1]=3

  • elementType: a scoped enumerator (enum class) defines the element type, e.g., Type::float16

Expr zeros(const Shape &shape)

Construct a constant node filled with 0 without a specified type, and the type is set to defaultElementType_.

Parameters

  • shape: a struct type defines the shape of the constant dataset e.g., shape={2,3} means 2D matrix with dim[0]=2 and dim[1]=3

Expr dropoutMask(float dropProb, const Shape &shape, Type elementType)

Construct a dropout mask (a tensor of 0 and 1).

Parameters

  • dropProb: a float type specifies the dropout probability. E.g., dropProb=0.1 means 90% of values are kept.

  • shape: a struct type defines the shape of the constant dataset e.g., shape={2,3} means 2D matrix with dim[0]=2 and dim[1]=3

  • elementType: a scoped enumerator (enum class) defines the element type, e.g., Type::float16

Expr dropoutMask(float dropProb, const Shape &shape)

Construct a dropout mask (a tensor of 0 and 1) without a specified type, and the type is set to defaultElementType_.

Parameters

  • dropProb: a float type specifies the dropout probability. E.g., dropProb=0.1 means 90% of values are kept.

  • shape: a struct type defines the shape of the constant dataset e.g., shape={2,3} means 2D matrix with dim[0]=2 and dim[1]=3

Expr get(std::string name)

Get the parameter object by name.

Parameters

  • name: a string specifies the name of the parameter object

Expr get(std::string name, Type specifiedElementType)

Get the parameter object by name and type.

Parameters

  • name: a string specifies the name of the parameter object

  • elementType: a scoped enumerator (enum class) defines the element type, e.g., Type::float16

Ptr<Parameters> &params()

Return the Parameters object related to the graph.

The Parameters object holds the whole set of the parameter nodes.

Ptr<Parameters> &params(Type elementType)

Return the Parameters object related to the graph by elementType.

The Parameters object holds the whole set of the parameter nodes of the given type.

void setDefaultElementType(Type defaultElementType)

Set default element type for the graph.

The default value is used if some node type is not specified.

Type getDefaultElementType()

Get default element type for the graph.

Expr add(Expr node)

Add a expression node to the graph.

Parameters

  • node: a pointer to a expression node

void allocateForward(Expr node)

Allocate memory for the forward pass of the given node.

Parameters

  • node: a pointer to a expression node

void allocateBackward(Expr node)

Allocate memory for the backward pass of the given node.

Parameters

  • node: a pointer to a expression node

void free(const Tensor &tensor)

Free the memory for a tensor object.

Parameters

  • tensor: a reference to the tensor object

Ptr<Allocator> allocator()

Returns the memory allocator of the graph workspace.

Allocates raw unstructured memory (but 256-byte aligned).

Ptr<TensorAllocator> getTensorAllocator()

Returns the tensor allocator of the graph workspace.

Different from allocator() as proper tensor objects are allocated.

void clear()

Clear everything apart from parameters and memoized nodes.

void setReloaded(bool reloaded)

Set the flag value whether the graph is reloaded (true) or not.

void setThrowNaN(bool throwNaN)

Set the flag value whether the graph throws a NaN exception (true) or not.

bool getThrowNaN()

Get the flag value whether the graph throws a NaN exception (true) or not.

void load(const std::vector<io::Item> &ioItems, bool markReloaded = true)

Load model (mainly parameter objects) from array of io::Items.

void load(const std::string &name, bool markReloaded = true)

Load model by filename.

void load(const void *ptr, bool markReloaded = true)

Load model from buffer (a file pointer)

void mmap(const void *ptr, bool markReloaded = true)

Turn the model (given a file pointer) into a memory-mapped type by converting all the parameter object to memory-mapped version, i.e., MappedParameters.

void save(std::vector<io::Item> &ioItems, Type saveElementType = Type::float32)

Convert all parameters into an array of io::Item elements, for saving.

Parameters

  • ioItems: an array of io::Item elements

  • saveElementType: the element type for saving

void save(const std::string &name, const std::string &meta = "", Type saveElementType = Type::float32)

Save all parameters into a file (.npz or .bin).

Parameters

  • name: a string specifies the filename

  • meta: a string specifies the name of io::Item elements. If not specified, the parameter name is reserved.

  • saveElementType: the element type for saving

Protected Functions

ExpressionGraph(const ExpressionGraph&)
ExpressionGraph(ExpressionGraph&&)

Protected Attributes

std::list<Expr> nodesForward_

contains all nodes used for forward()

std::list<Expr> nodesBackward_

contains trainable nodes used for backward()

Ptr<Tensors> tensors_

A shared pointer to the tensor objects in the graph.

Holds memory and nodes that corresponds to tensors in a graph. Since operations will result in new tensors, this attribute is used to allocate memory for new tensors during forward() and backward(). This gets cleared before a new graph is built.

ElementTypeParamsMap paramsByElementType_

A map holds memory and nodes that corresponds to graph parameters.

The key is Type and the mapped value is a set of parameter objects with corresponding type. Now we can have multiple types of parameters in a separate parameters object per value type. This is currently only accessible through private functions during loading, will abort during training when params() is called (e.g. optimizer) and there is more or other types than the default parameter type. Currently the only usecase is inference. Trying to access params() for non-default parameter type is going to abort. Inference does not need to access a whole set of parameters.

Ptr<Backend> backend_

a shared pointer to the backend for the graph

std::string namespace_

a string defines the namespace of the graph. Each graph has its own unique namespace.