Tutorial 1 - Broadcasting classes using MPI

As well as serialisation to and from HDF5, Anamnesis provides wrapper functionality to allow information to be sent between MPI nodes.

The use of the MPI functions in anamnesis requires the availability of the mpi4py module. If this is not available, you will not be able to use the MPI functions fully. You can, however, set use_mpi=True when creating the MPIHandler() object (see below) and then continue to use the functions. This allows you to write a single code base which will work both when doing multi-processing using MPI and running on a single machine.

The MPI functions require the same setup (primarily the hdf5_outputs class variable) as are used for the HDF5 serialisation / unserialisation, so we suggest that you work through the Serialisation tutorials first.

We are going to re-use some of the classes from the previous example. We place this code in test_mpiclasses.py

#!/usr/bin/python3

from anamnesis import AbstractAnam, register_class


class ComplexPerson(AbstractAnam):

    hdf5_outputs = ['name', 'age']

    hdf5_defaultgroup = 'person'

    def __init__(self, name='Unknown', age=0):
        AbstractAnam.__init__(self)

        self.name = name
        self.age = age


register_class(ComplexPerson)


class ComplexPlace(AbstractAnam):

    hdf5_outputs = ['location']

    hdf5_defaultgroup = 'place'

    def __init__(self, location='Somewhere'):
        AbstractAnam.__init__(self)

        self.location = location


register_class(ComplexPlace)


class ComplexTrain(AbstractAnam):

    hdf5_outputs = ['destination']

    hdf5_defaultgroup = 'train'

    def __init__(self, destination='Edinburgh'):
        AbstractAnam.__init__(self)

        self.destination = destination

    def init_from_hdf5(self):
        print("ComplexTrain.init_from_hdf5")
        print("We have already set destination: {}".format(self.destination))


register_class(ComplexTrain)

We now write a simple Python script which uses the Anamnesis MPI interface. We will design this code so that the master node creates an instance of two of the classes and the slave nodes receive copies of these.

#!/usr/bin/python3

from anamnesis import MPIHandler

from test_mpiclasses import ComplexPerson, ComplexPlace, ComplexTrain

# All nodes must perform this
m = MPIHandler(use_mpi=True)

if m.rank == 0:
    # We are the master node
    print("Master node")

    # Create a person, place and train to broadcast
    s_person = ComplexPerson('Fred', 42)
    s_place = ComplexPlace('York')
    s_train = ComplexTrain('Disastersville')

    print("Master: Person: {} {}".format(s_person.name, s_person.age))
    print("Master: Place: {}".format(s_place.location))
    print("Master: Train to: {}".format(s_train.destination))

    m.bcast(s_person)
    m.bcast(s_place)
    m.bcast(s_train)

else:
    # We are a slave node
    print("Slave node {}".format(m.rank))

    # Wait for our objects to be ready
    s_person = m.bcast()
    s_place = m.bcast()
    s_train = m.bcast()

    print("Slave node {}: Person: {} {}".format(m.rank, s_person.name, s_person.age))
    print("Slave node {}: Place: {}".format(m.rank, s_place.location))
    print("Slave node {}: Train: {}".format(m.rank, s_train.destination))

# We need to make sure that we finalise MPI otherwise
# we will get an error on exit
m.done()

To run this code, we need to execute it in an MPI environment. As usual, make sure that anamnesis is on the PYTHONPATH.

We can then call mpirun directly:

$ mpirun -np 2 python3 test_script1.py

Master node
Master: Person: Fred 42
Master: Place 1: York
Slave node 1
Master: Place 2: Glasgow
Slave node 1: Person: Fred 42
Slave node 1: Place 1: York
Slave node 1: Place 2: Glasgow

If you are using a cluster of some form (for instance gridengine), you will need to make sure that you have a queue with MPI enabled and that you submit your job to that queue. Gridengine in particular has good tight MPI integration which will transparently handle setting up the necessary hostlists.

The first thing which we need to do in the script is to initalise our MPIHandler. This is a singleton object and the use_mpi argument is only examined on the first use. This means that in future calls, you can call it without passing any argument.

m = MPIHandler(use_mpi=True)

In MPI, each instance of the script gets given a node number. By convention, we consider node 0 as the master node. All other nodes are designated as slave nodes. In order to decide whether we are the master node, we can therefore check whether our rank (stored on our MPIHandler object) is 0.

If we are the master, we then create three objects (a Person, a Place and a Train), set their attributes and print them out for reference. We then broadcast each of them in turn to our slave node or nodes.

On the slave node(s), we simply wait to receive the objects which are being sent from the master. There are two things to note. First, we do not need to specify the object type on the slave, this information is included in the MPI transfer. Second, we must make sure that our transmit and receive code is lined up; i.e. if we broadcast three items, every slave must receive three items. Code errors of this form are one of the most common MPI debugging problems. Try and keep your transmit / receive logic tidy and well understood in order to avoid long debugging sessions [#f1].

Once we have recieved the objects, we can simply use them as we normally would. Note that the objects are not shared before the two processes, you now have two distinct copies of each object.

Finally, it is important to call the MPIHandler.done() method to indicate to the MPI library that you have successfully finished.

Fotenotes

[1]	Note that mpi4py under Python3 has an unfortunate tendency to swallow error messages which can make debugging frustrating. This seems to have got worse since the python2 version. Any suggestions as to how to improve this situation would be gratefully recieved by the anamnesis authors.