OTB and python's behaviour : RAM usage

OTB Usage
1

Hi,

I would like to share with you what I’ve noticed concerning RAM consumption by using OTB through python. Here a simple a sample code :

def memory_usage_psutil():
    import resource
    return resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1000.0

images = ["raster1.tif", "raster2.tif", ..., "rasterN.tif"]
for image in images:
    bandMath = otb.Registry.CreateApplication("BandMath")
    bandMath.SetParameterStringList("il", [image])
    bandMath.SetParameterString("out", output)
    bandMath.SetParameterString("exp", "im1b1 + 1")
    print("Memory usage {} MB".format(memory_usage_psutil()))
    bandMath.ExecuteAndWriteOutput()
    bandMath = None

If you are running this kind of code, your memory consumption will raise until reach your limit (your code could crash or be kill).
The only solution I’ve found is to wrap ExecuteAndWriteOutput() thanks multiprocessing.Process() as in otb_memory.py (2.6 KB)

I think it is related to python and it’s garbage collector behavior. Is someone know a better solution to clean-up RAM ?

python otb_memory.py -h
highlight otb python bindings RAM usage

optional arguments:
  -h, --help      show this help message and exit
  -in INPUT_IMG
  -loop NB_LOOP   number of loop
  -wrap WRAP      wrap ExecuteAndWriteOutput() ?
  -output OUTPUT  output path

as input, you could use T32TNT_20180407T102019_SCL_20m.tif (354.9 KB)

in advance, thanks

2

Hi Arthur,
We are going to fix this issue as soon as possible.
In the mean time if you want to avoid that you can create an application with the otb.Registry.CreateApplicationWithoutLogger it will solve your issue for now.

3

Indeed, it works ! many thanks

4

Hello,
This is an issue with application logger objects not being removed on app destruction

It should be fixed with this Merge request: https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/merge_requests/569.

Cédric

5

Hi,
I know it’s an old message, but it’s the only one I’ve found on the subject. I have the same problem with tensorflow’s “PatchesExtraction” function - Orfeo ToolBox. multiprocessing.Process () doesn’t solve my problem and otb.Registry.CreateApplicationWithoutLogger doesn’t work.
Do you have any advice for me?

Thanks in advance