`smpirun -replay` writes into system directories
I'm following the Simgrid "Tracing and Replay of LU" tutorial ( https://simgrid.org/doc/latest/Tutorial_MPI_Applications.html#lab-2-tracing-and-replay-of-lu ), and smpirun
fails when replaying a trace:
$ smpirun -np 4 -platform ../cluster_crossbar.xml -hostfile ../cluster_hostfile -replay LU.A.4
[0.000000] [xbt_cfg/INFO] Configuration change: Set 'smpi/privatization' to '1'
[0.000000] [xbt_cfg/INFO] Configuration change: Set 'surf/precision' to '1e-9'
[0.000000] [xbt_cfg/INFO] Configuration change: Set 'network/model' to 'SMPI'
[0.000000] [smpi_config/INFO] You did not set the power of the host running the simulation. The timings will certainly not be accurate. Use the option "--cfg=smpi/host-speed:<flops>" to set its value. Check https://simgrid.org/doc/latest/Configuring_SimGrid.html#automatic-benchmarking-of-smpi-code for more information.
[node-0.simgrid.org:0:(1) 0.000000] ../src/smpi/internals/smpi_global.cpp:366: [root/CRITICAL] Cannot write into /usr/lib/simgrid/smpireplaymain_340164_0.so
Backtrace (displayed in actor 0):
0# simgrid::xbt::Backtrace::Backtrace() in /usr/lib/libsimgrid.so.3.25
1# xbt_backtrace_display_current in /usr/lib/libsimgrid.so.3.25
2# 0x00007F7E0B54A7EB in /usr/lib/libsimgrid.so.3.25
3# 0x00007F7E0B54AF04 in /usr/lib/libsimgrid.so.3.25
4# smx_ctx_wrapper in /usr/lib/libsimgrid.so.3.25
Aborted
/usr/lib/simgrid/smpireplaymain --cfg=smpi/privatization:1 --cfg=surf/precision:1e-9 --cfg=network/model:SMPI ../cluster_crossbar.xml smpitmp-apphdzQ8S
Execution failed with code 134.
The problem appears with Simgrid Debian package (libsimgrid3.25), and with Simgrid git version unless it is installed in a user directory (eg $HOME/simgrid
).
The problem seems to come from src/smpi/internals/smpi_global.cpp:41
:
std::string target_executable =
executable + "_" + std::to_string(getpid()) + "_" + std::to_string(rank) + ".so";
smpi_copy_file(executable, target_executable, fdin_size);
From what I understand, simgrid instruments all the libraries that are dlopen'd, but the generated libraries are stored in the same directory as the dlopen'd library (which is a system directory).
These instrumented libraries should probably be stored in a temporary directory in /tmp
.
Regards, François