Class ExpressionGraph¶
Defined in File expression_graph.h
Inheritance Relationships¶
Base Type¶
public std::enable_shared_from_this< ExpressionGraph >
Derived Types¶
public marian::ExpressionGraphONNXExporter
(Class ExpressionGraphONNXExporter)public marian::ExpressionGraphPackable
(Class ExpressionGraphPackable)
Class Documentation¶
-
class
ExpressionGraph
: public std::enable_shared_from_this<ExpressionGraph>¶ Main implementation of a computation graph.
Keeps a record of data (tensors) and all operations. Each operation in a computation graph is a Node. Each Node defines its forward and backward steps.
Subclassed by marian::ExpressionGraphONNXExporter, marian::ExpressionGraphPackable
Public Functions
-
ExpressionGraph
(bool inference = false)¶ Constructs a new expression graph.
Constructor should be used as New<ExpressionGraph>().
-
virtual
~ExpressionGraph
()¶ Destructor.
Clear everything related to the graph except memoized nodes.
-
void
setDevice
(DeviceId deviceId = {0, DeviceType::gpu}, Ptr<Device> device = nullptr)¶ Set device options used to run the graph.
- Parameters
deviceId
: a struct type which stores device no. (size_t) and device type (DeviceType::cpu or DeviceType::gpu)device
: a pointer to the device
-
DeviceId
getDeviceId
()¶ Get device info for the graph.
- Return
deviceId a struct type which stores device no. (size_t) and device type (DeviceType::cpu or DeviceType::gpu)
-
void
setInference
(bool inference)¶ Set whether the graph is used for inference only.
-
bool
isInference
()¶ Check whether the graph is used for inference only (true) or not.
-
void
setCheckpointing
(bool checkpointing)¶ Set whether the graph uses gradient checkpointing.
Gradient Checkpointing works by trading compute for memory, which reruns a forward-pass segment for each checkpoint segment during backward.
-
bool
isCheckpointing
()¶ Check whether the graph uses gradient checkpointing or not.
-
void
switchParams
(const std::string &newNamespace)¶ Set namespace (std::string) for the graph.
Each graph has its own unique namespace, which is used to form the name of a parameter object.
-
virtual void
copyParams
(Ptr<ExpressionGraph> graph)¶ Copy all parameter objects from one graph to current graph.
- Parameters
graph
: a pointer to a graph object
-
void
reserveWorkspaceMB
(size_t num)¶ Preallocate workspace memory (MB) for the graph.
Sets the size of the memory available for the forward and backward step of the training procedure. This does not include model size and optimizer parameters that are allocated outsize workspace.
-
void
reuseWorkspace
(Ptr<ExpressionGraph> graph)¶ Copy tensor objects from one graph to current graph.
-
void
backprop
()¶ Performs backpropagation on this expression graph.
Backpropagation is implemented by performing first the forward pass and then the backward pass of algorithmic differentiation (AD) on the nodes of the graph.
-
bool
fits
()¶ Perform one backpropagation process on the graph to test whether the graph workspace fits into a given workspace memory.
This function is used for searching the maximum batch size that fits into given workspace memory.
-
void
checkNaN
(Tensor t, bool &isNaN, bool &isInf)¶ Check whether the memory allocated for a tensor object contains a NaN or infinite value.
- Parameters
t
: a Tensor objectisNaN
: a bool type holds the result whether the tensor contains a NaN value (pass by reference)isInf
: a bool type holds the result whether the tensor contains a infinite value (pass by reference)
-
void
forward
()¶ Perform the forward pass on the nodes of the graph.
The forward pass refers to the calculation process. It traverses through all nodes from input layer to output layer.
-
void
forwardNext
()¶ Perform the forward pass without memory allocation for parameters.
Helper function for forward().
-
void
forward
(std::list<Expr> &forwardTape, bool finalPass)¶ Perform forward pass on a given nodes with finalPass flag.
Helper function for forward() and backward().
- Parameters
forwardTape
: a pointer to the nodes used for forward passfinalPass
: a bool type which controls whether nodes should be freed with gradient-checkpointing
-
void
backward
(bool reset = true, float clipValue = 0.f)¶ Perform the backward pass on the trainable nodes of the graph.
The back pass refers to the process of computing the output error. It traverses through all nodes from output layer to input layer.
-
std::string
graphviz
()¶ Generate graph layout in Graphviz format for visualisation.
- Return
a string presenting graph layout in Graphviz format (dot)
-
void
graphviz
(const std::string &filename)¶ Write graph layout in Graphviz format to a file.
- Parameters
filename
: a string type specifies filename that writes the graph layout
-
Expr
param
(const std::string &pname, const Shape &shape, const Ptr<inits::NodeInitializer> &init, const Type elementType, bool fixed = false)¶ Construct a parameter node in the graph.
- Return
a pointer to the parameter node
- Parameters
pname
: a string type holds the name of the parameter nodeshape
: a struct type defines the shape of the parameter tensor e.g., shape={2,3} means 2D matrix with dim[0]=2 and dim[1]=3init
: a pointer to a NodeInitializer object, e.g., inits::zeros()elementType
: a scoped enumerator (enum class) defines the element type, e.g., Type::float16fixed
: a bool type specifies whether the parameter object is fixed (not trainable) or not. The default value is false which means the parameter is trainable.
-
Expr
param
(const std::string &pname, const Shape &shape, const Ptr<inits::NodeInitializer> &init, bool fixed = false)¶ Construct a parameter node in the graph without a specified type, and the type is set to defaultElementType_.
- Return
a pointer to the parameter node
- Parameters
pname
: a string type holds the name of the parameter nodeshape
: a struct type defines the shape of the parameter tensor e.g., shape={2,3} means 2D matrix with dim[0]=2 and dim[1]=3init
: a pointer to a NodeInitializer object, e.g., inits::zeros()fixed
: a bool type specifies whether the parameter object is fixed (not trainable) or not. The default value is false which means the parameter is trainable.
-
Expr
constant
(const Shape &shape, const Ptr<inits::NodeInitializer> &init, Type elementType)¶ Construct a constant node in the graph without a specified type, and the type is set to defaultElementType_.
- Return
a pointer to the constant node
- Parameters
shape
: a struct type defines the shape of the constant tensor e.g., shape={2,3} means 2D matrix with dim[0]=2 and dim[1]=3init
: a pointer to a NodeInitializer object, e.g., inits::zeros()elementType
: a scoped enumerator (enum class) defines the element type, e.g., Type::float16
-
Expr
constant
(const Shape &shape, const Ptr<inits::NodeInitializer> &init)¶ Construct a constant node in the graph without a specified type, and the type is set to defaultElementType_.
- Return
a pointer to the constant node
- Parameters
shape
: a struct type defines the shape of the constant tensor e.g., shape={2,3} means 2D matrix with dim[0]=2 and dim[1]=3init
: a pointer to a NodeInitializer object, e.g., inits::zeros()
-
Expr
indices
(const std::vector<IndexType> &indicesVector)¶ Turn vector of indices to integer tensor.
A shortcut version to turn vector of indices to integer tensor, to be used with operators like rows() or index_select()
- Parameters
indicesVector
: a vector of IndexType (uint32_t) specifies the indexes
-
Expr
indices
(const std::vector<IndexType> &indicesVector, Expr indexee, int axis = -1)¶ Specify the indexes of elements to be taken from a tensor.
This version sets up the shape such that the indices are in a given axis. Use this if you want to pass these indices to gather(). E.g., indexee shape = (3, 2, 5, 2); axis = 1 -> resulting shape = (1, size of indicesVector, 1, 1):
The size of the resulting shape is the same as that of the indexee; here is 4.
The shape of the specified axis is equal to the size of given indicesVector.
The shapes of the rest axes are filled with 1.
- Parameters
indicesVector
: a vector of IndexType (uint32_t) specifies the indexesindexee
: the source tensor that we want to select elements fromaxis
: specifies the axis that we want to collect along
-
Expr
ones
(const Shape &shape, Type elementType)¶ Construct a constant node filled with
1
.- Parameters
shape
: a struct type defines the shape of the constant dataset e.g., shape={2,3} means 2D matrix with dim[0]=2 and dim[1]=3elementType
: a scoped enumerator (enum class) defines the element type, e.g., Type::float16
-
Expr
ones
(const Shape &shape)¶ Construct a constant node filled with
1
without a specified type, and the type is set to defaultElementType_.- Parameters
shape
: a struct type defines the shape of the constant dataset e.g., shape={2,3} means 2D matrix with dim[0]=2 and dim[1]=3
-
Expr
zeros
(const Shape &shape, Type elementType)¶ Construct a constant node filled with
0
.- Parameters
shape
: a struct type defines the shape of the constant dataset e.g., shape={2,3} means 2D matrix with dim[0]=2 and dim[1]=3elementType
: a scoped enumerator (enum class) defines the element type, e.g., Type::float16
-
Expr
zeros
(const Shape &shape)¶ Construct a constant node filled with
0
without a specified type, and the type is set to defaultElementType_.- Parameters
shape
: a struct type defines the shape of the constant dataset e.g., shape={2,3} means 2D matrix with dim[0]=2 and dim[1]=3
-
Expr
dropoutMask
(float dropProb, const Shape &shape, Type elementType)¶ Construct a dropout mask (a tensor of 0 and 1).
- Parameters
dropProb
: a float type specifies the dropout probability. E.g., dropProb=0.1 means 90% of values are kept.shape
: a struct type defines the shape of the constant dataset e.g., shape={2,3} means 2D matrix with dim[0]=2 and dim[1]=3elementType
: a scoped enumerator (enum class) defines the element type, e.g., Type::float16
-
Expr
dropoutMask
(float dropProb, const Shape &shape)¶ Construct a dropout mask (a tensor of 0 and 1) without a specified type, and the type is set to defaultElementType_.
- Parameters
dropProb
: a float type specifies the dropout probability. E.g., dropProb=0.1 means 90% of values are kept.shape
: a struct type defines the shape of the constant dataset e.g., shape={2,3} means 2D matrix with dim[0]=2 and dim[1]=3
-
Expr
get
(std::string name)¶ Get the parameter object by name.
- Parameters
name
: a string specifies the name of the parameter object
-
Expr
get
(std::string name, Type specifiedElementType)¶ Get the parameter object by name and type.
- Parameters
name
: a string specifies the name of the parameter objectelementType
: a scoped enumerator (enum class) defines the element type, e.g., Type::float16
-
Ptr<Parameters> &
params
()¶ Return the Parameters object related to the graph.
The Parameters object holds the whole set of the parameter nodes.
-
Ptr<Parameters> &
params
(Type elementType)¶ Return the Parameters object related to the graph by elementType.
The Parameters object holds the whole set of the parameter nodes of the given type.
-
void
setDefaultElementType
(Type defaultElementType)¶ Set default element type for the graph.
The default value is used if some node type is not specified.
-
Expr
add
(Expr node)¶ Add a expression node to the graph.
- Parameters
node
: a pointer to a expression node
-
void
allocateForward
(Expr node)¶ Allocate memory for the forward pass of the given node.
- Parameters
node
: a pointer to a expression node
-
void
allocateBackward
(Expr node)¶ Allocate memory for the backward pass of the given node.
- Parameters
node
: a pointer to a expression node
-
void
free
(const Tensor &tensor)¶ Free the memory for a tensor object.
- Parameters
tensor
: a reference to the tensor object
-
Ptr<Allocator>
allocator
()¶ Returns the memory allocator of the graph workspace.
Allocates raw unstructured memory (but 256-byte aligned).
-
Ptr<TensorAllocator>
getTensorAllocator
()¶ Returns the tensor allocator of the graph workspace.
Different from allocator() as proper tensor objects are allocated.
-
void
clear
()¶ Clear everything apart from parameters and memoized nodes.
-
void
setReloaded
(bool reloaded)¶ Set the flag value whether the graph is reloaded (true) or not.
-
void
setThrowNaN
(bool throwNaN)¶ Set the flag value whether the graph throws a NaN exception (true) or not.
-
bool
getThrowNaN
()¶ Get the flag value whether the graph throws a NaN exception (true) or not.
-
void
load
(const std::vector<io::Item> &ioItems, bool markReloaded = true)¶ Load model (mainly parameter objects) from array of io::Items.
-
void
load
(const void *ptr, bool markReloaded = true)¶ Load model from buffer (a file pointer)
-
void
mmap
(const void *ptr, bool markReloaded = true)¶ Turn the model (given a file pointer) into a memory-mapped type by converting all the parameter object to memory-mapped version, i.e., MappedParameters.
-
void
save
(std::vector<io::Item> &ioItems, Type saveElementType = Type::float32)¶ Convert all parameters into an array of io::Item elements, for saving.
- Parameters
ioItems
: an array of io::Item elementssaveElementType
: the element type for saving
-
void
save
(const std::string &name, const std::string &meta = "", Type saveElementType = Type::float32)¶ Save all parameters into a file (.npz or .bin).
- Parameters
name
: a string specifies the filenamemeta
: a string specifies the name of io::Item elements. If not specified, the parameter name is reserved.saveElementType
: the element type for saving
Protected Attributes
-
std::list<Expr>
nodesBackward_
¶ contains trainable nodes used for backward()
-
Ptr<Tensors>
tensors_
¶ A shared pointer to the tensor objects in the graph.
Holds memory and nodes that corresponds to tensors in a graph. Since operations will result in new tensors, this attribute is used to allocate memory for new tensors during forward() and backward(). This gets cleared before a new graph is built.
-
ElementTypeParamsMap
paramsByElementType_
¶ A map holds memory and nodes that corresponds to graph parameters.
The key is Type and the mapped value is a set of parameter objects with corresponding type. Now we can have multiple types of parameters in a separate parameters object per value type. This is currently only accessible through private functions during loading, will abort during training when params() is called (e.g. optimizer) and there is more or other types than the default parameter type. Currently the only usecase is inference. Trying to access params() for non-default parameter type is going to abort. Inference does not need to access a whole set of parameters.
-