Derived Data Types with MPI

1. Introduction

The basic MPI communication mechanisms can be used to send or receive a sequence of identical elements that are contiguous in memory. It is often desirable to send data that is not homogeneous or that is not contiguous in memory. This would amortize the fixed overhead of sending and receiving a message over the transmittal of many elements. MPI provides two mechanisms to achieve this:

The user can define derived datatype that specify more general data layouts
A sending process can explicitly pack noncontiguous data into a contiguous buffer and then send it; a reveiving process can explicitly unpack data received in a contiguous buffer and store in noncontiguous locations

This tutorial focuses on the construction and use of derived datatypes, why, how, and when to use them.

2. Why Use Derived Datatypes?

2.1 Basic MPI Datatypes

MPI Basic predefined Datatypes for C

MPI datatype	C datatype
MPI_CHAR	signed char
MPI_SHORT	signed short int
MPI_INT	signed int
MPI_LONG	signed long int
MPI_UNSIGNED_CHAR	unsigned char
MPI_UNSIGNED_SHORT	unsigned short int
MPI_UNSIGNED_LONG	unsigned long_int
MPI_UNSIGNED	unsigned int
MPI_FLOAT	float
MPI_DOUBLE	double
MPI_LONG_DOUBLE	long double
MPI_BYTE
MPI_PACKED

MPI Basic predefined Datatypes for FORTRAN

MPI datatype	FORTRAN datatype
MPI_INTEGER	INTEGER
MPI_REAL	REAL
MPI_REAL8	REAL*8
MPI_DOUBLE_PRECISION	DOUBLE PRECISION
MPI_COMPLEX	COMPLEX
MPI_LOGICAL	LOGICAL
MPI_CHARACTER	CHARACTER
MPI_BYTE
MPI_PACKED

Given these datatypes and a count, you can handle messages of contiguous data of the same type.

2.2 Motivation

What if you want to specify:

non-contiguous data of a single type?
contiguous data of mixed types?
non-contiguous data of mixed types?

A few possible solutions are that you could:

make multiple MPI calls to send and receive each data element
use MPI_PACKED to send data that has been explicitly packed and/or MPI_UNPACKED to receive data that has been explicitly unpacked.
use MPI_BYTE to get around the datatype-matching rules. Like MPI_PACKED, MPI_BYTE can be used to match any byte of storage (on a byte-addressable machine), irrespective of the datatype of the variable that contains this byte.

Generally, however, these solutions are slow, clumsy, and wasteful of memory. Using MPI_BYTE or MPI_PACKED might also result in a program that isn't portable to a heterogeneous system of machines.

The idea of MPI derived datatypes is to provide a portable and efficient way of communicating non-contiguous or mixed types in a message. MPI derived datatypes provide a simpler, cleaner, more elegant and efficient way to handle this type of data.

3. What are Derived Datatypes?

Derived datatypes are datatypes that are built from the basic MPI datatypes. To better understand what is needed to construct such a datatype, you need to understand the general concept of an MPI datatype and something called a typemap.

Formally, the MPI Standard defines a general datatype as an object that specifies two things:

a sequence of basic datatypes
a sequence of integer (byte) displacements

An easy way to represent such an object is as a sequence of pairs of basic datatypes and displacements. MPI calls this sequence a typemap.

4. When and How Do I Use Derived Datatypes?

4.1 When to Use

When you want to create a datatype in C or FORTRAN, you do so by declaring the datatype before executing any statements. Your declarations are read by the compiler that sets up storage for your datatype. In contrast, MPI derived datatypes are created at run-time through calls to MPI library routines. Since MPI derived datatypes are often used to send or receive C or FORTRAN datatypes, in the typical scenario, you first declare your C or FORTRAN datatypes. Later, in the execution part of your program between calls to MPI_INIT and MPI_FINALIZE, you create and use your MPI derived datatypes.

4.2 How to Use

Before you can use a derived datatype, you must create it. Here are the steps you take:

Construct the datatype.
Allocate the datatype.
Use the datatype.
Deallocate the datatype.

You must construct and allocate a datatype before using it. On the other hand, once you have it constructed and allocated, you are not required to use or deallocate it.

4.2.1 Construct the datatype

MPI_Type_contiguous

The simplest constructor. Produces a new datatype by making count copies of an existing data type.

MPI_Type_vector

MPI_Type_hvector

MPI_Type_hvector is identical to MPI _Type_vector except that stride is specified in bytes.

MPI_Type_indexed

MPI_Type_hindexed

An array of displacements of the input data type is provided as the map for the new data type.

MPI_Type_hindexed is identical to MPI_Type_indexed except that offsets are specified in bytes.

MPI_Type_struct

The most general of all derived datatypes. The new data type is formed according to

completely defined map of the component data types.

4.2.2 Allocate the datatype

A constructed datatype must be committed to the system before it can be used in a communication. The constructed datatype is committed with a call to MPI_TYPE_COMMIT. (There is no need to commit basic datatypes; they are pre-committed.) It can then be used in any number of communications. The syntax of MPI_TYPE_COMMIT is:

int MPI_Type_commit (MPI_datatype *datatype)

FORTRAN

MPI_TYPE_COMMIT (DATATYPE, MPIERROR)
  INTEGER DATATYPE, MPIERROR

4.2.3 Use the datatype

Derived datatypes can be used in all send and receive operations. You simply use the handle to the derived datatype as an argument in a send or receive operation instead of a basic datatype argument. Here is a sample C code segment:

MPI_Type_vector(count, blocklength, stride, oldtype, &newtype);
MPI_Type_commit (&newtype);
MPI_Send(buffer, 1, newtype, dest, tag, comm);

4.2.4 Deallocate the datatype

Finally, there is a complementary routine to MPI_TYPE_COMMIT, namely, MPI_TYPE_FREE, which marks a datatype for deallocation. The syntax of MPI_TYPE_FREE is:

int MPI_Type_free (MPI_datatype *datatype)

FORTRAN

MPI_TYPE_FREE (DATATYPE, MPIERROR)
  INTEGER DATATYPE, MPIERROR

5. DERIVED DATATYPES

This section presents the MPI functions for constructing derived datatypes.

5.1 CONTIGUOUS

    C       : MPI_Type_contiguous (count, oldtype, *newtype)
    Fortran : MPI_TYPE_CONTIGUOUS (count, oldtype, newtype,ierr)

count

oldtype

newtype

ierr

newtype

count

oldtype

Example

C example
Fortran example

Sample program output:

rank= 0  b= 1.0 2.0 3.0 4.0
rank= 1  b= 5.0 6.0 7.0 8.0
rank= 2  b= 9.0 10.0 11.0 12.0
rank= 3  b= 13.0 14.0 15.0 16.0

5.2 VECTOR & HVECTOR

    C       : MPI_Type_vector (count,blocklength,stride,oldtype,*newtype)
              MPI_Type_hvector (count,blocklength,stride,oldtype,*newtype)
    Fortran : MPI_TYPE_VECTOR (count, blocklength, stride, oldtype, newtype, ierr)
              MPI_TYPE_HECTOR (count, blocklength, stride, oldtype, newtype, ierr)

count

blocklen

stride

oldtype

newtype

The Vector constructor is similar to contiguous, but allows for regular gaps or overlaps (stride) in the displacements.

Hvector: MPI_Type_hvector (in C) and MPI_TYPE_HVECTOR (in Fortran), respectively, are the same as those for MPI_TYPE_VECTOR given above, except that displacement stride is specified in bytes rather than by length.

Example

C example
Fortran example

Sample program output:

rank= 0  b= 1.0 5.0 9.0 13.0
rank= 1  b= 2.0 6.0 10.0 14.0
rank= 2  b= 3.0 7.0 11.0 15.0
rank= 3  b= 4.0 8.0 12.0 16.0

5.3 INDEXED & HINDEXED

    C       : MPI_Type_indexed (count, blocklens[], offsets[], old_type,*newtype)
              MPI_Type_hindexed (count, blocklens[], offsets[], old_type, *newtype)
    Fortran : MPI_TYPE_INDEXED  (count, blocklens(), offsets(), old_type(),newtype, ierr)
              MPI_TYPE_HINDEXED  (count, blocklens(), offsets(), old_type(),newtype, ierr)

count

blocklens

offsets

oldtype

newtype

This constructor replicates a datatype, taking blocks at different offsets. It allows one to specify a noncontiguous data layout where displacements between successive blocks need not be equal.

Hindexed: MPI_Type_hindexed (in c) and MPI_TYPE_HINDEXED (in FORTRAN), respectively, are the same as those for MPI_TYPE_INDEXED given above, except that offsets array is specified in bytes.

Example

C example
Fortran example
Sample program output:

rank= 0 b= 6.0 7.0 8.0 9.0 13.0 14.0
rank= 1 b= 6.0 7.0 8.0 9.0 13.0 14.0
rank= 2 b= 6.0 7.0 8.0 9.0 13.0 14.0
rank= 3 b= 6.0 7.0 8.0 9.0 13.0 14.0

5.4 STRUCT

    C       : MPI_Type_struct (count, blocklens[], offsets[], old_types[], *newtype)
    Fortran : MPI_TYPE_STRUCT (count,  blocklens(),  offsets(),  old_type(),  newtype,  ierr)

count

blocklens

offsets

oldtype

newtype

To gather a mix of different datatypes scattered at many locations in space into one datatype that can be used for the communication.

Example

C example
Fortran example

Sample program output:

rank= 0   3.00 -3.00 3.00 0.25 3 1
rank= 2   3.00 -3.00 3.00 0.25 3 1
rank= 1   3.00 -3.00 3.00 0.25 3 1
rank= 3   3.00 -3.00 3.00 0.25 3 1

6. Conclusions

Derived datatypes provide a portable and elegant way of communicating non-contiguous or mixed types in a message.
Derived datatypes should provide an efficient method of sending data since the data can be moved from its location from one processor memory to a location on a different processor memory without any intermediate buffering.
Derived datatypes are datatypes that are built from the basic MPI datatypes.
Derived datatypes provide a template of the data that is to be sent. All the data in the datatype is identified by its offset from the base address. The base address is the address passed to the MPI routine using the derived datatype. This allows the same MPI datatype to be used for any number of variables of the same form.
MPI provides a number of different routines for creating derived datatypes, each aimed at certain types of data, i.e., contiguous data, non-contiguous data, and non-contiguous mixed data.
Every derived datatype must be committed before it can be used.
The MPI routine MPI_TYPE_EXTENT is useful for calculating displacements as it takes into account any alignment issues.
extent(Typemap) = ub(Typemap) - lb(Typemap) + pad
where ub is the upper bound (location of the last byte), lb is the lower bound (the location of the first byte) and pad, a concept used by MPI to require the address of a variable (in bytes) to be a multiple of its length (in bytes).
Here is the syntax for the extent routine:
- C
```
int MPI_Type_extent(MPI_Datatype datatype, MPI_Aint *extent)
```
- FORTRAN
```
MPI_TYPE_EXTENT(DATATYPE, EXTENT, MPIERROR)
    INTEGER DATATYPE, EXTENT, MPIERROR
```
where
- datatype is an input datatype handle
- extent is an output integer for FORTRAN, or for C, a special integer type MPI_Aint, that can hold an arbitrary address
A received message does not need to fill the entire receive buffer. MPI defines two routines to help you handle this situation. The MPI_GET_COUNT routine returns the number of received elements of the datatype specified in the receive call. If you want to find out how many basic elements within this datatype were received, use the MPI_GET_ELEMENTS routine.
- C
```
int MPI_Get_count(MPI_Status *status,
                  MPI_Datatype datatype, int *count)
int MPI_Get_elements(MPI_Status *status,
                  MPI_Datatype datatype, int *count)
```
- FORTRAN
```
MPI_GET_COUNT(STATUS, DATATYPE, COUNT, MPIERROR)
MPI_GET_ELEMENTS(STATUS, DATATYPE, COUNT, MPIERROR)
   INTEGER STATUS(MPI_STATUS_SIZE), DATATYPE, COUNT, MPIERROR
```
where MPI_Status is an input variable specifying the status of the receive operation.

7. References

World Wide Web
- www.erc.msstate.edu/mpi/
- www.mcs.anl.gov/mpi/
- www.mpi-forum.org/
- Message Passing Interface Forum (1995)
  MPI: A Message Passing Interface Standard. June 12, 1995.
Freely Available Implementations
- www.erc.mss tate.edu/mpi/implementations.html
Tutorials
- www.msi.umn.edu/sp/tutorials/
- www.erc.mssta te.edu/mpi/presentations.html
Books
- Using MPI, by William Gropp, Ewing Lusk, and Anthony Skjellum
- MPI Annotated Reference Manual, by Marc Snir, et al
- Designing and Building Parallel Programs, by Ian Foster
- Parallel Programming with MPI, by Peter Pacheco
- RS/600 0 SP: Practical MPI Programming by Yukiya Aoyama and Jun Nakano.
Newsgroup
- comp.parallel.mpi