Multi-threading read

OTB Development
1

Hello

I’ve noticed that the current GDAL version supports thread-safe reading of multiple files. Is there a plan to incorporate this feature in future versions of OTB for parallel file reading? Is parallel reading already implemented through another method (for TIFF files, for instance)?

I often observe sequential activity when processing a large number of input files, likely due to sequential file-by-file decompression.

Thanks !

2

Hello

I’ve identified a bottleneck in my code, which was the number of threads actually being used by OTB (in this case, just one). From my reading of the code, I found that the number of threads is RAM-driven, and ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS only sets a maximum limit (and defaults to the number of virtual cores). For programs that use multiple filters connected together, it’s advisable to increase the default RAM allocation in order to use the available CPU cores (via OTB_MAX_RAM_HINT).

It would be helpful to display the actual number of threads used in the log, along with the number of blocks. I had to use a printf inside my filter to determine that number.

My previous question still stands, as it is common to observe periods of high computation followed by downtimes where the main thread is at 100%.

Thanks !

3

Hi @ThomasDM , for what I know about OTB when using it with multithreading is that this part is managed by ITK. Indeed the maximum number of thread used is determined by the number of cpu available (can be known with `nproc` on Linux)

The thread management of ITK is the following per application/filter. Let say NP is the number of procs :

  • It determine the size of the output image and divide it vertically in NP parts. Like this you don’t have two threads writting in the same area
  • For each output parts it determine the input part needed to compute it. The same input area can be used to compute different output parts but it is not a problem as multiple threads can read the same input area.
  • When theses area computations are done, each thread can work without problem

As you can see we already have a thread reading part that can be done in parallel, thus I don’t know if it is a good idea to add the GDAL parallel reading to that.