Learning samples csv extension
This MR implements the steps described in #655 (closed) :
- Create a new parameter to allow the user the possibility to change the learning samples extension
learning_samples_extension
parameter was created to define the file format to use for the samples extraction output (choice between sqlite and csv).
Since it is not possible to use csv files with otb classifiers, a test on the configuration file has been added to check that the choice of classifier and learningSamples file extension are compatible.
- Count the number of features before extracting the samples and inform user to change learningSamples extension in config file when the number of features is superior to 2000
The chain will stop and display an error message indicating that the learning_sample_extension
parameter should be changed if the number of features in at least one tile exceeds 2000.
To do this, a exception was created, as well as a constant that define the maximum number of columns accepted by sqlite.
- Modify the vector sampler module to generate a CSV learning samples file
Use the new learning_sample_extraction
parameter in the configuration file to generate samples in the correct format and select the appropriate driver.
If the number of features is greater than 2000 and the learning_sample_extraction
parameter is set to 'csv' (otherwise the TooManyColums exception is raise), the environment variable OGR_CSV_MAX_FIELD_COUNT
is increased.
- Modify the split samples by region step to deal with a CSV as learning samples file
The split_vector_by_region
function has been split into 4 functions: split_vector_by_region
, get_vect_output_name
, split_vector_by_region_csv
, split_vector_by_region_sqlite_shp
- Modify the Merge Samples by Model step to deal with the a CSV as learning sample file
The merge_wide_sqlite
function has been renamed and split into 2 functions: merge_db_to_netcdf
, get_chunk_iterator
The check_duplicates
function was modified to deal with the a CSV files as input
- Modify the data augmentation step to deal with a CSV as learning samples file
The count_class
function has been and split into 3 functions: count_class
, count_class_in_sqlite
and count_class_in_csv
The do_copy_sqlite
function has been renamed and split into 3 functions: do_copy
, do_copy_sqlite
and do_copy_csv
- Adapt the scikit training step to accept a CSV input file
The get_df_features
function has been created to provide training data to scikit-learn regardless of the extension of the file containing the training samples.
- Add tests
5 unit tests that take over existing tests by replacing the use of sqlite files with csv files.
2 integration tests with training samples characterized by more than 2000 features.
-> to this end 2 functions, one adapted to raw data, the other to interpolated data, have been added to the external_code.py to generate a large number of external features.