Too many open files (again)
related to #103 (closed) and #396 (closed)
Until then, the solution to "too many open files" problem was just to raise the limit as mentioned in related issues.
But we might want to take this problem seriously, as it can prevent from running even basic operations. In the following, I describe the status of my investigation and give means to reproduce it.
- hardware/os: dell laptop, ubuntu 20.04, linux 5.13
- environment: iota2_develop released last week (~2022-04-07)
- software: Iota2 on branch develop commit 7b5c624f
- data: zenodo archive (https://zenodo.org/record/5553139)
-
ulimit -u
: 61946
config file
chain: { s1_path: '/home/trentesauxh/dev/tutorial/IOTA2_TEST_S2/hugo_s1.cfg' output_path: '/home/trentesauxh/dev/tutorial/IOTA2_TEST_S2/IOTA2_Outputs/Results_S1' remove_output_path: False list_tile: 'T31TCJ' spatial_resolution: 10 proj: 'EPSG:2154' first_step: "init" last_step: "writing" } external_features: { module: "/home/trentesauxh/dev/tutorial/IOTA2_TEST_S2/external_code/external_code.py" functions: "get_sentinel1" concat_mode: False } python_data_managing: { chunk_size_mode: "split_number" number_of_chunks: 3 data_mode_access: "raw" } builders: { builders_class_name: ["i2_features_map"] }
iota2 command:
Iota2.py -scheduler_type local -config hugo_s1_iota2.cfg -starting_step 1 -ending_step 4 -nb_parallel_tasks 2
result of lsof | grep Iota2 > lsof_iota2.txt
during a iota2 execution
lsof_iota2.txt (17550 lines)
What I see is:
- 13626 lines containing
/home/trentesauxh/dev/tp_zenodo/i2_training_omp/s1_data/
- 4095 lines containing
/home/trentesauxh/mambaforge/envs/iota2-new/lib/
So the number of data-related open files is (only) three times higher than lib/python related files.
We might want to have a look at the whole codebase to see where data-related open files could be closed at some point. An other thing we could do is to avoid opening files when we do not need to. For example, monitoring the number of file opening on configuration file on the classification tutorial with iwatch -e open i2_tutorial_classification.cfg
yields about 500 lines (related to #457 for this specific file).