Running computation concurrently with MPI_Iallreduce
Hi, We have been trying to use MPI_Iallreduce (the non-blocking version) in the SMPI interface. In particular, we want to call it multiple times (in a sequential loop) to all-reduce different messages across multiple processors. But we have some intermediate computations as well which we are trying to hide.
When we do an MPI_Wait for each associated request after all calls to MPI_Iallreduce (like the following snippet), we get the desired non-blocking behavior and the computes occur concurrently with the allreduce operations:
MPI_Iallreduce(buf1,...,rq1)
Compute1()... /* doesn't depend on buf1 */
MPI_Iallreduce(buf2,...,rq2)
Compute2()... /* doesn't depend on buf2 */
MPI_Wait(rq1)
MPI_Wait(rq2)
However, we wanted multi-threaded behavior: basically all computes happen on one thread and all allreduces happen on another thread with potential overlap in comms and certainly with computes (due to the threading).
If we have MPI_Wait in between MPI_Iallreduce calls, it behaves in a blocking manner:
MPI_allreduce(buf1,...,rq1)
Compute1()...
MPI_Wait(rq1) /* We want to do something here as if in a separate thread but not hold up the subsequent execution */
MPI_Iallreduce(buf2,...,rq2)
Compute2()...
MPI_Wait(rq2)
As per the MPI manual, I tried MPI_Test() instead of MPI_Wait() to get non-blocking behavior, but once an MPI_Test() is executed, the request pointer is nulled out, and the receive buffer does not seem to have the allreduced values.
Do you know of an alternate way of achieving the multi-threaded behavior desired above?
Thank you!