Use dask-ml for learning and classification on large dataset
For learning and vector classification steps, geometry is not required, but iota2 must use vector files to work with OTB.
In OBIA classification, each tile must be cut into small tiles:
- the calculation of statistics on a S2 tile can produce heavy files (several Gb) saved using xml
- OTB VectorClassifier works only with shapefile and sqlites files
Shapefiles have a total size limited to 2 GB. They can handle a large number of columns (the problem of the 256th field is related to the tools with graphical user interface).
Sqlites are limited to 1000 columns. This limit can be quickly reached for a one year time series (480 features) and 5 zonal statistics (mean, min, max, number, stdev).
This split implies the writing of a large number of temporary files.
Dask and dask-ml provides solutions for processing large datasets, saving them in several formats such as hdf, csv etc... Convert xml to csv or hdf is simple with pandas, then it can be combined to avoid splitting manually data.
The issue is that dask-ml requires a recent version of pandas
- pandas >=0.23.0
In the current version of iota2, versions of some packages are locked due to OSSIM.
OSSIM will be removed in OTB 8.0