Patch sample extraction for contextual pixel-based tasks
Intro
The goal of this issue is to specify a workflow for the sample extraction step for image patches instead of pixels. This is a derivation of the discussion started in #194 (closed).
Reminder:
The overall sampling step of iota2 consist in 2 parts:
- sample selection, where the reference data (geometries -- i.e. polygons or points -- and target variable -- i.e. class label) and the image metadata (footprint and resolution) are used to select coordinates (points) that will be used as training samples; this generates sqlite files with positions and labels.
- sample extraction, where the output of the previous step is enriched with the features (spectral bands and indices for each date) for each pixel position.
For contextual approaches (mainly CNN in the spatial domain, but other approaches could benefit from that) the features of the neighboring pixels are needed. One way to build sample files for this case is to modify the sample extraction step to store information about the context of the pixel in the form of a rectangular patch around it.
Please note that this is not particularly useful for semantic segmentation with CNN, which needs that all the pixels in the patch are labeled. Here, we only have the label of the central pixel and the patch is only used to provide a context around it. If we think CNN, we are here dealing with pixel-based classification using patches.
Rationale
When designing such a sampling service, several aspects have to be taken into account:
- How to specify the geometry of the patches
- How to store the features
The geometry of the patches will be the same for all patches and will be a rectangle. The rectangle can be defined by its size in lines and columns in pixel units. Since the sample selection gives single pixel positions, it seems straightforward to use this position as the center of the rectangle, but defining a center pixel for a rectangle means that the number of lines and columns of the patch have to be odd numbers. This seems incompatible with the way patches are used in CNN, where the perceptive field has a shape with powers of 2 in lines and columns.
I propose that we keep odd numbers for the patch size and delegate the cropping to the downstream consumer (i.e. the data loader). In this case, in order to ensure that the shape of the patch has a central pixel, the size can be specified by a radius tuple (w, h) and the effective size will be (2w+1, 2l+1).
If a pixel has F features, a patch is described by F*(2w+1)(2*l+1) features, which can be a huge number for a Sentinel-2 time series. This means that the usual format limitations (file size in shapefiles and number of columns in sqlite) won't allow storing the features as additional columns of the sqlite file generated by the sample selection step.
What can be done is storing the features in a separate file and inserting a link to the appropriate file in the sqlite file which contains the pixel positions and the labels.
If we have many samples, having an external file per patch will be inefficient in terms of I/O. Since the sample extraction step is performed per tile, we can generate a single file containing all the patches for the given tile. Since all the patches have the same size, they can be stacked in the «patch dimension» in the file. For each sample, the sqlite will contain the path of the patch file and index information allowing the consumers to retrieve the appropriate data in the file.
One could use the GeoTIFF format and stack the patches in the band dimension, so that the first N bands correspond to the N spectro-temporal features of the first patch, bands N+1 to 2N correspond to the second patch etc. Another option that seems more appropriate is to use the netCDF format, which is designed for this kind of storage (see https://xarray.pydata.org/en/stable/io.html). Using netCDF, we can separate the line, column, date, band and patch dimension. Also, the different dimensions can have names associated to the coordinates, which can allow to give meaningful names to the bands, the dates and identify the cloud masks, etc. This will allow for an easier index and retrieval of the data downstream in the processing. Furthermore, if the sample extraction is coded using rasterio + xarray, the use of netCDF is straightforward. The same tools can be used to provide generic dataloaders for Pytorch which can stream data from disk. Zarr (https://zarr.readthedocs.io/en/stable/) is another format option, but the project does not seem mature enough yet.
Dealing with patches at the border of images.
The sample selection step can provide pixel positions which are close to the image boundaries yielding patches which run outside of the image. There are 2 ways of dealing with this. Either we suppress those samples (but this may change the sample proportions and may not be acceptable for some applications), or we adapt the validity masks for the parts of the patch which do not contain any data. I think that both options should be available and accessible from the configuration file.
Dealing with patches at the boundary of eco-climatic regions.
In this case, I think we can just ignore the issue, but maybe an option to exclude those patches or adding a band where each pixel in the patch has the value of the region can be foreseen.
Patch sample extraction workflow:
- The work is done at the tile level
- Inputs:
- sample selection file with pixel positions and labels (and eco-climatic region)
- image stacks
- Outputs:
- netCDF file with dimensions: patch_id, date, bands, mask (validity, region), x, y
- sqlite file completed with the file name and patch_id for each sample
- For each point in sample selection
- get the patch from the stack using rasterio windowed reading
- add it to an xarray data structure
We may want to limit the size of the netCDF files and generate files containing a limited number of patches.
Instead of reading a ROI for each patch, we may make things more efficient and read chunks of the stack containing several samples, but some spatial indexing will be needed to ensure that full patches for all chosen samples are included in the chunk.
Please feel free to comment and propose alternatives.