Simgrid bug put to mailbox when this last have been timeout'd before.
Hello, sorry for bothereing you again :) Seems like I am facing another but that might be related to simgrid and probably linked to my last issue 120.
I recreated the bug in a little example that you can find here bug-put.zip, but it can also be triggered with my full program falafels.
The bug is in 3 phase, with 2 entities (master/worker):
- Registration phase: workers send their mailbox name to the master, the master send confirmation to the workers, then timeout (line 76) to go to the next phase.
- The master sends a "Go" signal, the workers receive the "Go" signal
- The workers send their data to the master.
The timeout on phase 1 lets us the possibility to not provide the number of worker to the master, which is more flexible in my program scenario which is globally peer-to-peer.
The bug occurs on phase 3, with the communication of the first sending worker to be ignored, resulting in a dead lock because the master is never receiving the worker's data.
[Soraka:master:(1) 2.019590] [s4u_main/INFO] Master start receiving data
[Zoe:worker:(2) 4.015805] [s4u_main/INFO] Zoe ---sending data---> master
[Maokai:worker:(3) 4.016325] [s4u_main/INFO] Maokai ---sending data---> master
[Soraka:master:(1) 4.016845] [s4u_main/INFO] master <--data--- Maokai
[Soraka:master:(1) 4.016845] [s4u_main/INFO] Master start receiving data
[Morgana:worker:(4) 4.019070] [s4u_main/INFO] Morgana ---sending data---> master
[Pyke:worker:(5) 4.019590] [s4u_main/INFO] Pyke ---sending data---> master
[Soraka:master:(1) 4.021816] [s4u_main/INFO] master <--data--- Morgana
[Soraka:master:(1) 4.021816] [s4u_main/INFO] Master start receiving data
[Soraka:master:(1) 4.022335] [s4u_main/INFO] master <--data--- Pyke
[Soraka:master:(1) 4.022335] [s4u_main/INFO] Master start receiving data
[4.022335] ./src/kernel/EngineImpl.cpp:710: [ker_engine/CRITICAL] Oops! Deadlock detected, some activities are still around but will never complete. This usually happens when the user code is not perfectly clean.
In this example here, Zoe's communication is ignored (In the current version of the recreated example I did not included logs to confirm Zoe's call to put() was ended, but indeed it was made and Zoe is not waiting at the end of the program anyways).
This behavior only happens when:
- Running simgrid git -> it running the same program with simgrid v3.35 work perfectly (see test-with-simgrid.3.35.tesh).
- Executing phase 1 -> if I hardcode my nodes in the master, and that the master didn't had to do a timeout before, everything works perfectly.
In the following logs we see that the master receives the data from Zoe.
[Soraka:master:(1) 2.019590] [s4u_main/INFO] Master start receiving data
[Zoe:worker:(2) 4.015805] [s4u_main/INFO] Zoe ---sending data---> master
[Maokai:worker:(3) 4.016325] [s4u_main/INFO] Maokai ---sending data---> master
[Soraka:master:(1) 4.018551] [s4u_main/INFO] master <--data--- Zoe
[Soraka:master:(1) 4.018551] [s4u_main/INFO] Master start receiving data
[Morgana:worker:(4) 4.019070] [s4u_main/INFO] Morgana ---sending data---> master
[Soraka:master:(1) 4.019071] [s4u_main/INFO] master <--data--- Maokai
[Soraka:master:(1) 4.019071] [s4u_main/INFO] Master start receiving data
[Pyke:worker:(5) 4.019590] [s4u_main/INFO] Pyke ---sending data---> master
[Soraka:master:(1) 4.021816] [s4u_main/INFO] master <--data--- Morgana
[Soraka:master:(1) 4.021816] [s4u_main/INFO] Master start receiving data
[Soraka:master:(1) 4.022336] [s4u_main/INFO] master <--data--- Pyke
[4.022336] [s4u_main/INFO] Simulation is over
I tried debugging this bug with lldb and it seems like put() is being called, but no corresponding get() is called.
Thanks in advance for your help!