Working with a large number of images - OTB error "too many files open"
Hi,
When working with a large number (~400) of raster images, it may happen during the processing (in my case during the pre-processing of SAR SITS) problems with IO: OTB, via GDAL, complains that "too many files are opened". It seems related to the number of opened files during the process, due to the streaming strategy, which is limited per process by the OS.
A possible work-around is to increase the number of file that can be opened by a process in your OS. The solution is taken from:
- https://www.ibm.com/support/knowledgecenter/SSJMXE_8.3.0/com.ibm.rational.test.ft.install.doc/topics/t_increase_handles.html
- https://linoxide.com/linux-how-to/03-methods-change-number-open-file-limit-linux/
- https://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/
In the shell, before running iota2 set this number to an higher value (initially, on a Ubuntu 16.04.5 LTS, 1024), for instance:
ulimit -n 4096
It is also possible to set it permanently, but to check it I need to reboot the server, something I cannot do.
Mathieu