Profiling the deep learning workflow
It was mentioned that deep-learning does not use the full capacity of GPUs. At the time of learning it seems that we spend more time reading the data than learning the model on the gpus, several points need to be checked
- how much time is spent reading the data (from disk to RAM)
- how much time is spent transferring the data from RAM to the GPU
- how long it takes the network to consume the data.
- GPU usage
Also, it seems that the amount of memory used at the time of the prediction is greater than expected. It would be necessary to check whether copies of data are not being made unnecessarily during this process.
Some tools to profile the code :
- snakeviz
- cProfile
- memory profiler
- scalene
- nvidia-smi
Then, the documentation can be updated to inform users which configuration (size of the batch, read the database by chunk, ...) is optimum.