Tracing SMPI -- Bandwith_used/Speed_used are higher than Bandwith/Speed capacity
Intro
MWE to execute an SMPI trace replay that might generate incorect paje traces.
- Downloads the file here: mwe.tar
- The files for the simulation:
hostfile
platform.xml
trace_files.txt
trace_folder
- The file
./smpi_copy_executabe.patch
is a quick workaround to make smpi replay nix-compliant. - The file
shell.nix
contains the definition of the environment, use withnix-shell shell.nix
.
Do the simulation
make
# or:
# mpirun -replay trace_files.txt -np 16 -platform platform.xml -hostfile hostfile -trace -trace-file trace.paje--cfg=tracing/uncategorized:yes --cfg=host/model:ptask_L07
To run the notebook:
To run the notebook used to generate the issue:
(with nix)
nix-shell --pure --command "make"
# This line will generate a file notebook.html
nix-shell --pure --command "Rscript -e \"library(rmarkdown); render('notebook.Rmd');\""
(without nix)
dependencies
R with:
tidyverse
ggplot
ggrepel
rmarkdown
Simgrid
smpirun --version
SimGrid version 3.22.2
and pajeng.
Run with:
# This line will generate a file notebook.html
Rscript -e "library(rmarkdown); render('notebook.Rmd');"
Transform paje trace
Use pj_dump
(from pajeng utility), to transform a paje trace into a
csv, which is handy to use R.
# bash script
export paje="trace.paje"
export trace="./data/smpi_trace.csv"
mkdir -p ./data
pj_dump --ignore-incomplete-links --float-precision=9 ${paje} > ${trace}
grep Variable $trace > ./data/variable.csv
grep Link $trace > ./data/link.csv
Refine and Visualize with R
Convert the csv from pj_dump
to a dataframe.
df_var = read_csv("data/variable.csv", col_names = FALSE, trim_ws = TRUE)
names(df_var) = c("Container", "Resource_Name", "Type", "Start", "End", "Duration", "Value");
# df_var = df_var[!(names(df_var) %in% c("Container"))];
df_var = df_var %>% mutate(Resource_Name = as.character(Resource_Name));
Have a look at the used bandwidth
df_bw <- df_var %>% filter(Type == "bandwidth_used") %>% filter(!grepl("_loopback", Resource_Name))
# FIXME manage the case where links are symetrical or asymetrical
link_capacity = df_var %>%
filter(Type == "bandwidth") %>%
select(Resource_Name, Value) %>%
filter(!grepl("_loopback", Resource_Name))
bandwith_with_capacity = left_join(df_bw, link_capacity,
by=c("Resource_Name" = "Resource_Name"),
su=c("", "_capacity")) %>% mutate(percentage_usage = Value/Value_capacity)
Have a look at the cpu
df_speed <- df_var %>% filter(Type == "speed_used")
# FIXME manage the case where links are symetrical or asymetrical
host_speed = df_var %>%
filter(Type == "speed") %>%
select(Resource_Name, Value)
speed_witch_capacity = left_join(df_speed, host_speed,
by=c("Resource_Name" = "Resource_Name"),
su=c("", "_capacity")) %>% mutate(percentage_usage = Value/Value_capacity)
resource_usage = bind_rows(speed_witch_capacity, bandwith_with_capacity)
Buggy events
Shows the activity that overshoot the capacity of their speed or link
capacity: column percentage_usage
resource_usage %>% filter(percentage_usage > 1) %>% select(-Container, -Duration, -Resource_Name) %>% arrange(desc(percentage_usage))
## # A tibble: 12 x 6
## Type Start End Value Value_capacity percentage_usage
## <chr> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 speed_used 3.56 4.64 932333027328 466166513664 2
## 2 speed_used 3.56 4.64 932333027328 466166513664 2
## 3 speed_used 1.23 2.32 815791398912 466166513664 1.75
## 4 speed_used 1.23 2.32 815791398912 466166513664 1.75
## 5 speed_used 2.39 3.48 699249770496 466166513664 1.5
## 6 speed_used 4.64 4.65 699249770496 466166513664 1.5
## 7 speed_used 2.39 3.48 699249770496 466166513664 1.5
## 8 speed_used 4.64 4.65 699249770496 466166513664 1.5
## 9 bandwidth_used 1.19 1.20 18749999616 12499999744 1.5
## 10 bandwidth_used 3.51 3.53 18749999616 12499999744 1.5
## 11 bandwidth_used 1.19 1.20 18749999616 12499999744 1.5
## 12 bandwidth_used 3.51 3.53 18749999616 12499999744 1.5
plot
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information