Undestanding Error Log Outputs
I have had some instances stop running but I cannot understand the issue. All instances fail similarly, so I'll just focus on one instance.
I'll list what I can see from the logs of an instance, and I'll upload the relevant log files, since all the logs are too big:
Instance config: auto_iota2.cfg The instance is configured to use a user provided feature, without the use of any other features (not sentinel-2 bands or no generated indices), to learn from an sklearn random forest model
From unpickling IOTA2_tasks_status.txt
...
extraction_T31UDP done
merge_model_1_seed_4_usually done
merge_model_1_seed_2_usually done
learning_model_1_seed_4 failed
learning_model_1_seed_2 failed
classification_T31TCM_model_1_seed_4_0 unlaunchable
classification_T31TCM_model_1_seed_4_4 unlaunchable
classification_T31TCM_model_1_seed_4_3 unlaunchable
...
Looking at logs/learnModel/learning_model_1_seed_4 and 2, .out and .err, I could not see any specific issue, other than dask worker exceeding memory, which seems to be common across all other seeds.
learning_model_1_seed_2.err learning_model_1_seed_2.out learning_model_1_seed_4.err learning_model_1_seed_4.out
I have uploaded stdout
and stderr
for reference, but not much to help from here
sar_rvi_scirf_cviel100k20x9.out.txt
sar_rvi_scirf_cviel100k20x9.err.txt.1
sar_rvi_scirf_cviel100k20x9.err.txt.2
I have also looked at the HTML pages, but could not identify anything that seemed out of the ordinary.
What I can say is that the graph on iota² tasks status graph 2
HTML page was very small and difficult to identify the nodes (obviously, very minor issue).
Using tasks_status_2.svg I can see that extraction completed, but classification was not launched, with merge_model and learn_model nodes left white/uncoloured.
I'm not sure what else I should look at, so let me know, I'm not used to iota, so could be a user error