Providing a dataloader utility to feed online/batch learning algorithm (a.k.a., CNN and other stochastic learning algorithm)
Conventional machine learning algorithm (Random Forest, SVM, K-NN) take as input to their learning/fit function the entire training data available and the algorithm optimize their hyperparameters accordingly.
In practice with iota2, we load the entire SQL file into memory. Then, a design matrix X is build (https://en.wikipedia.org/wiki/Design_matrix), where each colon correspond to the features and each line to the samples (pixels): its size is n_samples X n_features. Conventionnaly, each pixel is described by a set of feature and in iota2's conventional learning step no additional spatial information (inter-pixel dependency) is used. In the same time a vector y of measurements (e.g. class labels) is also build that relates each pixel to its measurement (e.g. class label for a classification problem). (X,y) is provided to the algorithm's fit function.
For some kind of algorithms, Deep Learning (DL) based ones in particular, the learning step is a bit different because the way their fit their hyperparameters used online strategies rather than bulk strategies. That is, the training data is provided by batchs (a subset of the training data) iteratively to the algorithm, until convergence is reached (https://en.wikipedia.org/wiki/Online_machine_learning).
Another important difference also appears for algorithms using spatial convolutionnal layer (e.g. CNN, https://en.wikipedia.org/wiki/Convolutional_neural_network). For these algorithms, the pixel is represented by a cube of measurement rather than a single vector, because adjacent pixels are also provided.
Iota2 should provide a facility to end-user to plug their data-loader to their DL algorithm. A cheap solution could to write data to a format that conventional DL dataloader can read. For instance, see Pytorch implementation: here. We should be able in particular to extract patchs from the images, something that is not currently possible with iota2 (up to my knowledge).