[SMPI] Order of the message matching is not guaranteed
This issue is to keep track and discuss about the crashes we can see on HPL_Optimized proxy app (../src/smpi/mpi/smpi_request.cpp:994: [root/CRITICAL] recv - returned MPI_ERR_TRUNCATE instead of MPI_SUCCESS )
The issue seems is that SMPI may not respect the message ordering constraint from the standard in one case.
Standard says:
Order Messages are non-overtaking: If a sender sends two messages in succession to the same destination, and both match the same receive, then this operation cannot receive the second message if the first one is still pending.
HPL sends a lot of messages of various sizes using the same tags, but with different sizes.
Toying with replay of HPL messages, I was able to find a reproducer :
Two processes, one sending and one receiving, and async small threshold set.
First process sends a message big enough to not be asynchronously sent, but small enough to not cause a rendez-vous before returning (detached). Then it sends a smaller message, asynchronously sent.
Second process posts two receives in a row, but the first (big) receive matches first the message already in its "permanent receive" small mailbox, which is alllowed. But the second smaller receive will match the first large message, causing a truncation error.
Message size can't really be used to match the messages, as it is legal to receive in a large buffer (and a legitimate error in the other case). Matching is done using communicator ID+tag+source/dest.
What I don't yet understand is how we could perform a check and ensure that order. We would need to probe both mailboxes and select the best message (if present), but with no guarantee all of them are already posted. And our probe call does not really allow to check the characteristics of a message outside of the match function, which just returns a bool, so even this would be a problem.
Reproducer is for example :
0 init
0 recv 1 1001 15600 0
0 recv 1 1001 29640 0
0 recv 1 1001 1560 0
1 init
1 send 0 1001 15600 0
1 send 0 1001 29640 0
1 send 0 1001 1560 0
with --cfg=smpi/async-small-thresh:65536 (first message is somehow necessary to trigger the issue, but it's not part of it)