MFEM  v4.6.0
Finite element discretization library
Public Types | Public Member Functions | Public Attributes | List of all members
mfem::AmgXSolver Class Reference

#include <amgxsolver.hpp>

Inheritance diagram for mfem::AmgXSolver:
[legend]
Collaboration diagram for mfem::AmgXSolver:
[legend]

Public Types

enum  AMGX_MODE { SOLVER, PRECONDITIONER }
 Flags to configure AmgXSolver as a solver or preconditioner. More...
 
enum  CONFIG_SRC { INTERNAL, EXTERNAL, UNDEFINED }
 
- Public Types inherited from mfem::Operator
enum  DiagonalPolicy { DIAG_ZERO, DIAG_ONE, DIAG_KEEP }
 Defines operator diagonal policy upon elimination of rows and/or columns. More...
 
enum  Type {
  ANY_TYPE, MFEM_SPARSEMAT, Hypre_ParCSR, PETSC_MATAIJ,
  PETSC_MATIS, PETSC_MATSHELL, PETSC_MATNEST, PETSC_MATHYPRE,
  PETSC_MATGENERIC, Complex_Operator, MFEM_ComplexSparseMat, Complex_Hypre_ParCSR,
  Complex_DenseMat, MFEM_Block_Matrix, MFEM_Block_Operator
}
 Enumeration defining IDs for some classes derived from Operator. More...
 

Public Member Functions

 AmgXSolver ()
 
 AmgXSolver (const AMGX_MODE amgxMode_, const bool verbose)
 
void InitSerial ()
 
 AmgXSolver (const MPI_Comm &comm, const AMGX_MODE amgxMode_, const bool verbose)
 
 AmgXSolver (const MPI_Comm &comm, const int nDevs, const AMGX_MODE amgx_Mode_, const bool verbose)
 
void InitExclusiveGPU (const MPI_Comm &comm)
 
void InitMPITeams (const MPI_Comm &comm, const int nDevs)
 
virtual void SetOperator (const Operator &op)
 
void UpdateOperator (const Operator &op)
 
virtual void Mult (const Vector &b, Vector &x) const
 Operator application: y=A(x). More...
 
int GetNumIterations ()
 
void ReadParameters (const std::string config, CONFIG_SRC source)
 
void DefaultParameters (const AMGX_MODE amgxMode_, const bool verbose)
 
void SetConvergenceCheck (bool setConvergenceCheck_=true)
 Add a check for convergence after applying Mult. More...
 
 ~AmgXSolver ()
 
void Finalize ()
 
- Public Member Functions inherited from mfem::Solver
 Solver (int s=0, bool iter_mode=false)
 Initialize a square Solver with size s. More...
 
 Solver (int h, int w, bool iter_mode=false)
 Initialize a Solver with height h and width w. More...
 
- Public Member Functions inherited from mfem::Operator
void InitTVectors (const Operator *Po, const Operator *Ri, const Operator *Pi, Vector &x, Vector &b, Vector &X, Vector &B) const
 Initializes memory for true vectors of linear system. More...
 
 Operator (int s=0)
 Construct a square Operator with given size s (default 0). More...
 
 Operator (int h, int w)
 Construct an Operator with the given height (output size) and width (input size). More...
 
int Height () const
 Get the height (size of output) of the Operator. Synonym with NumRows(). More...
 
int NumRows () const
 Get the number of rows (size of output) of the Operator. Synonym with Height(). More...
 
int Width () const
 Get the width (size of input) of the Operator. Synonym with NumCols(). More...
 
int NumCols () const
 Get the number of columns (size of input) of the Operator. Synonym with Width(). More...
 
virtual MemoryClass GetMemoryClass () const
 Return the MemoryClass preferred by the Operator. More...
 
virtual void MultTranspose (const Vector &x, Vector &y) const
 Action of the transpose operator: y=A^t(x). The default behavior in class Operator is to generate an error. More...
 
virtual void AddMult (const Vector &x, Vector &y, const double a=1.0) const
 Operator application: y+=A(x) (default) or y+=a*A(x). More...
 
virtual void AddMultTranspose (const Vector &x, Vector &y, const double a=1.0) const
 Operator transpose application: y+=A^t(x) (default) or y+=a*A^t(x). More...
 
virtual void ArrayMult (const Array< const Vector *> &X, Array< Vector *> &Y) const
 Operator application on a matrix: Y=A(X). More...
 
virtual void ArrayMultTranspose (const Array< const Vector *> &X, Array< Vector *> &Y) const
 Action of the transpose operator on a matrix: Y=A^t(X). More...
 
virtual void ArrayAddMult (const Array< const Vector *> &X, Array< Vector *> &Y, const double a=1.0) const
 Operator application on a matrix: Y+=A(X) (default) or Y+=a*A(X). More...
 
virtual void ArrayAddMultTranspose (const Array< const Vector *> &X, Array< Vector *> &Y, const double a=1.0) const
 Operator transpose application on a matrix: Y+=A^t(X) (default) or Y+=a*A^t(X). More...
 
virtual OperatorGetGradient (const Vector &x) const
 Evaluate the gradient operator at the point x. The default behavior in class Operator is to generate an error. More...
 
virtual void AssembleDiagonal (Vector &diag) const
 Computes the diagonal entries into diag. Typically, this operation only makes sense for linear Operators. In some cases, only an approximation of the diagonal is computed. More...
 
virtual const OperatorGetProlongation () const
 Prolongation operator from linear algebra (linear system) vectors, to input vectors for the operator. NULL means identity. More...
 
virtual const OperatorGetRestriction () const
 Restriction operator from input vectors for the operator to linear algebra (linear system) vectors. NULL means identity. More...
 
virtual const OperatorGetOutputProlongation () const
 Prolongation operator from linear algebra (linear system) vectors, to output vectors for the operator. NULL means identity. More...
 
virtual const OperatorGetOutputRestrictionTranspose () const
 Transpose of GetOutputRestriction, directly available in this form to facilitate matrix-free RAP-type operators. More...
 
virtual const OperatorGetOutputRestriction () const
 Restriction operator from output vectors for the operator to linear algebra (linear system) vectors. NULL means identity. More...
 
void FormLinearSystem (const Array< int > &ess_tdof_list, Vector &x, Vector &b, Operator *&A, Vector &X, Vector &B, int copy_interior=0)
 Form a constrained linear system using a matrix-free approach. More...
 
void FormRectangularLinearSystem (const Array< int > &trial_tdof_list, const Array< int > &test_tdof_list, Vector &x, Vector &b, Operator *&A, Vector &X, Vector &B)
 Form a column-constrained linear system using a matrix-free approach. More...
 
virtual void RecoverFEMSolution (const Vector &X, const Vector &b, Vector &x)
 Reconstruct a solution vector x (e.g. a GridFunction) from the solution X of a constrained linear system obtained from Operator::FormLinearSystem() or Operator::FormRectangularLinearSystem(). More...
 
void FormSystemOperator (const Array< int > &ess_tdof_list, Operator *&A)
 Return in A a parallel (on truedofs) version of this square operator. More...
 
void FormRectangularSystemOperator (const Array< int > &trial_tdof_list, const Array< int > &test_tdof_list, Operator *&A)
 Return in A a parallel (on truedofs) version of this rectangular operator (including constraints). More...
 
void FormDiscreteOperator (Operator *&A)
 Return in A a parallel (on truedofs) version of this rectangular operator. More...
 
void PrintMatlab (std::ostream &out, int n, int m=0) const
 Prints operator with input size n and output size m in Matlab format. More...
 
virtual void PrintMatlab (std::ostream &out) const
 Prints operator in Matlab format. More...
 
virtual ~Operator ()
 Virtual destructor. More...
 
Type GetType () const
 Return the type ID of the Operator class. More...
 

Public Attributes

bool ConvergenceCheck
 Flag to check for convergence. More...
 
- Public Attributes inherited from mfem::Solver
bool iterative_mode
 If true, use the second argument of Mult() as an initial guess. More...
 

Additional Inherited Members

- Protected Member Functions inherited from mfem::Operator
void FormConstrainedSystemOperator (const Array< int > &ess_tdof_list, ConstrainedOperator *&Aout)
 see FormSystemOperator() More...
 
void FormRectangularConstrainedSystemOperator (const Array< int > &trial_tdof_list, const Array< int > &test_tdof_list, RectangularConstrainedOperator *&Aout)
 see FormRectangularSystemOperator() More...
 
OperatorSetupRAP (const Operator *Pi, const Operator *Po)
 Returns RAP Operator of this, using input/output Prolongation matrices Pi corresponds to "P", Po corresponds to "Rt". More...
 
- Protected Attributes inherited from mfem::Operator
int height
 Dimension of the output / number of rows in the matrix. More...
 
int width
 Dimension of the input / number of columns in the matrix. More...
 

Detailed Description

MFEM wrapper for Nvidia's multigrid library, AmgX (github.com/NVIDIA/AMGX)

AmgX requires building MFEM with CUDA, and AMGX enabled. For distributed memory parallism, MPI and Hypre (version 16.0+) are also required. Although CUDA is required for building, the AmgX solver is compatible with a MFEM CPU device configuration.

The AmgXSolver class is designed to work as a solver or preconditioner for existing MFEM solvers. The AmgX solver class may be configured in one of three ways:

Serial - Takes a SparseMatrix solves on a single GPU and assumes no MPI communication.

Exclusive GPU - Takes a HypreParMatrix and assumes each MPI rank is paired with an Nvidia GPU.

MPI Teams - Takes a HypreParMatrix and enables flexibility between number of MPI ranks, and GPUs. Specifically, MPI ranks are grouped with GPUs and a matrix consolidation step is taken so the MPI root of each team performs the necessary AmgX library calls. The solution is then broadcasted to appropriate ranks. This is particularly useful when configuring MFEM's device as CPU. This work is based on the AmgXWrapper of Chuang and Barba. Routines were adopted and modified for setting up MPI communicators.

Examples 1/1p in the examples/amgx directory demonstrate configuring the wrapper as a solver and preconditioner, as well as configuring and running with exclusive GPU or MPI teams modes.

This work is partially based on:

Pi-Yueh Chuang and Lorena A. Barba (2017). AmgXWrapper: An interface between PETSc and the NVIDIA AmgX library. J. Open Source Software, 2(16):280, doi:10.21105/joss.00280

See https://github.com/barbagroup/AmgXWrapper.

Definition at line 69 of file amgxsolver.hpp.

Member Enumeration Documentation

◆ AMGX_MODE

Flags to configure AmgXSolver as a solver or preconditioner.

Enumerator
SOLVER 
PRECONDITIONER 

Definition at line 74 of file amgxsolver.hpp.

◆ CONFIG_SRC

Flags to determine whether user solver settings are defined internally in the source code or will be read through an external JSON file.

Enumerator
INTERNAL 
EXTERNAL 
UNDEFINED 

Definition at line 83 of file amgxsolver.hpp.

Constructor & Destructor Documentation

◆ AmgXSolver() [1/4]

mfem::AmgXSolver::AmgXSolver ( )

Definition at line 33 of file amgxsolver.cpp.

◆ AmgXSolver() [2/4]

mfem::AmgXSolver::AmgXSolver ( const AMGX_MODE  amgxMode_,
const bool  verbose 
)

Configures AmgX with a default configuration based on the AmgX mode, and verbosity. Assumes no MPI parallism.

Definition at line 36 of file amgxsolver.cpp.

◆ AmgXSolver() [3/4]

mfem::AmgXSolver::AmgXSolver ( const MPI_Comm &  comm,
const AMGX_MODE  amgxMode_,
const bool  verbose 
)

Configures AmgX with a default configuration based on the AmgX mode, and verbosity. Pairs each MPI rank with one GPU.

Definition at line 50 of file amgxsolver.cpp.

◆ AmgXSolver() [4/4]

mfem::AmgXSolver::AmgXSolver ( const MPI_Comm &  comm,
const int  nDevs,
const AMGX_MODE  amgx_Mode_,
const bool  verbose 
)

Configures AmgX with a default configuration based on the AmgX mode, and verbosity. Creates MPI teams around GPUs to support MPI ranks > GPUs. Consolidates linear solver data to avoid multiple ranks sharing GPUs. Requires specifying number of devices in each compute node.

Definition at line 64 of file amgxsolver.cpp.

◆ ~AmgXSolver()

mfem::AmgXSolver::~AmgXSolver ( )

Definition at line 80 of file amgxsolver.cpp.

Member Function Documentation

◆ DefaultParameters()

void mfem::AmgXSolver::DefaultParameters ( const AMGX_MODE  amgxMode_,
const bool  verbose 
)
Parameters
[in]amgxMode_AmgXSolver::PRECONDITIONER, AmgXSolver::SOLVER.
[in]verbosetrue, false. Specifies the level of verbosity.

When configured as a preconditioner, the default configuration is to apply two iterations of an AMG V cycle with AmgX's default smoother (block Jacobi).

As a solver the preconditioned conjugate gradient method is used. The AMG V-cycle with a block Jacobi smoother is used as a preconditioner.

Definition at line 198 of file amgxsolver.cpp.

◆ Finalize()

void mfem::AmgXSolver::Finalize ( )

Definition at line 990 of file amgxsolver.cpp.

◆ GetNumIterations()

int mfem::AmgXSolver::GetNumIterations ( )

Definition at line 983 of file amgxsolver.cpp.

◆ InitExclusiveGPU()

void mfem::AmgXSolver::InitExclusiveGPU ( const MPI_Comm &  comm)

Once the solver configuration has been established, either through the constructor or the ReadParameters method, InitSerial will initialize the library. If configuring with constructor, the constructor will make this call.

Definition at line 120 of file amgxsolver.cpp.

◆ InitMPITeams()

void mfem::AmgXSolver::InitMPITeams ( const MPI_Comm &  comm,
const int  nDevs 
)

Once the solver configuration has been established, either through the ReadParameters method, InitMPITeams will initialize the library and create MPI teams based on the number of devices on each node (nDevs). If configuring with constructor, the constructor will make this call.

Definition at line 149 of file amgxsolver.cpp.

◆ InitSerial()

void mfem::AmgXSolver::InitSerial ( )

Once the solver configuration has been established through either the ReadParameters method or the constructor, InitSerial will initialize the library. If configuring with constructor, the constructor will make this call.

Definition at line 85 of file amgxsolver.cpp.

◆ Mult()

void mfem::AmgXSolver::Mult ( const Vector x,
Vector y 
) const
virtual

Operator application: y=A(x).

Implements mfem::Operator.

Definition at line 902 of file amgxsolver.cpp.

◆ ReadParameters()

void mfem::AmgXSolver::ReadParameters ( const std::string  config,
CONFIG_SRC  source 
)

Definition at line 186 of file amgxsolver.cpp.

◆ SetConvergenceCheck()

void mfem::AmgXSolver::SetConvergenceCheck ( bool  setConvergenceCheck_ = true)

Add a check for convergence after applying Mult.

Definition at line 193 of file amgxsolver.cpp.

◆ SetOperator()

void mfem::AmgXSolver::SetOperator ( const Operator op)
virtual

Sets Operator for AmgX library, either MFEM SparseMatrix or HypreParMatrix

Implements mfem::Solver.

Definition at line 859 of file amgxsolver.cpp.

◆ UpdateOperator()

void mfem::AmgXSolver::UpdateOperator ( const Operator op)

Replaces the matrix coefficients in the AmgX solver.

Definition at line 882 of file amgxsolver.cpp.

Member Data Documentation

◆ ConvergenceCheck

bool mfem::AmgXSolver::ConvergenceCheck

Flag to check for convergence.

Definition at line 77 of file amgxsolver.hpp.


The documentation for this class was generated from the following files: