Process stopped
IOTA2 could be stopped during step computation (noticed during classification step).
Errors are :
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= EXIT CODE: 9
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Killed (signal 9)
This typically refers to a problem with your application
or more simply
Process stopped
The reason is the lack of RAM.
The problem was observed with OTB 6.0 which do not estimate correctly the full pipeline's size when applications are connected together. Indeed, due to this issue, processes could use more RAM than expected (and more than physical RAM) and be killed. This problem is even more obvious if the physical RAM is share between users.
In versions > 6.4 of OTB, pipelines size estimation was fixed. It mean that we can have a better control on RAM consumption per tasks. However, if RAM is shared between users, the problem remains. We can only provide to users a way to tune RAM consumption.
The question is how to set the maximum RAM consumption to steps which needs a lot of it ? (SampleExtraction and Classifications)
Some ideas :
- Add a new field in the configuration file ?
max_RAM:5gb
- Use the configuration file dedicated to resources reservation ? This configuration file is actually use by the PBS file writer.
- Estimate the RAM parameter inside IOTA2 ?