Skip to content

Learning samples csv extension

Hélène Touchais requested to merge learningSamples_csv_extension into develop

This MR implements the steps described in #655 (closed) :

  • Create a new parameter to allow the user the possibility to change the learning samples extension

learning_samples_extension parameter was created to define the file format to use for the samples extraction output (choice between sqlite and csv).

Since it is not possible to use csv files with otb classifiers, a test on the configuration file has been added to check that the choice of classifier and learningSamples file extension are compatible.

  • Count the number of features before extracting the samples and inform user to change learningSamples extension in config file when the number of features is superior to 2000

The chain will stop and display an error message indicating that the learning_sample_extension parameter should be changed if the number of features in at least one tile exceeds 2000.

To do this, a exception was created, as well as a constant that define the maximum number of columns accepted by sqlite.

  • Modify the vector sampler module to generate a CSV learning samples file

Use the new learning_sample_extraction parameter in the configuration file to generate samples in the correct format and select the appropriate driver.

If the number of features is greater than 2000 and the learning_sample_extraction parameter is set to 'csv' (otherwise the TooManyColums exception is raise), the environment variable OGR_CSV_MAX_FIELD_COUNT is increased.

  • Modify the split samples by region step to deal with a CSV as learning samples file

The split_vector_by_region function has been split into 4 functions: split_vector_by_region, get_vect_output_name, split_vector_by_region_csv, split_vector_by_region_sqlite_shp

  • Modify the Merge Samples by Model step to deal with the a CSV as learning sample file

The merge_wide_sqlite function has been renamed and split into 2 functions: merge_db_to_netcdf, get_chunk_iterator

The check_duplicates function was modified to deal with the a CSV files as input

  • Modify the data augmentation step to deal with a CSV as learning samples file

The count_class function has been and split into 3 functions: count_class, count_class_in_sqlite and count_class_in_csv

The do_copy_sqlite function has been renamed and split into 3 functions: do_copy, do_copy_sqlite and do_copy_csv

  • Adapt the scikit training step to accept a CSV input file

The get_df_features function has been created to provide training data to scikit-learn regardless of the extension of the file containing the training samples.

  • Add tests

5 unit tests that take over existing tests by replacing the use of sqlite files with csv files.

2 integration tests with training samples characterized by more than 2000 features.

-> to this end 2 functions, one adapted to raw data, the other to interpolated data, have been added to the external_code.py to generate a large number of external features.

Merge request reports