Does OTB support Cloud Optimized Geotiffs in any manner?

Just a question. Thank you!

I am not sure what you call Cloud Optimized GeoTIFF, but maybe this can answer your question?

Thanks shall I share my thoughts and progress in OTB7.2.0 in that thread?

GDAL has a driver to support COG files. And OTB relies on GDAL to read and write products. You should give it a try. If it doesn’t work, it would be a good feature to implement.

Don’t hesitate !

1 Like

I recently tried to read cog via OTB it works (via vsicurl) but is there a way we can write internal overviews to GTiff via OTB or directly use COG driver while writing output.?

I created some COG via gdal_translate option then I used otb to read the file and write it to a new file with extended options as follows…
'?&gdal:co:COMPRESS=DEFLATE&gdal:co:TILED=YES&gdal:co:BLOCKXSIZE=512&gdal:co:BLOCKYSIZE=512&gdal:co:NUM_THREADS=ALL_CPUS&gdal:co:COPY_SRC_OVERVIEWS=YES'

Normally as I understand once we have a COG which has internal overviews, using gdal creation option COPY_SRC_OVERVIEWS=YES should copy the existing overviews to new GTiff file but seems like it has no effect while using via extended filename.

Example Code:

  1. Create COG:
    gdal_translate -of COG -co BLOCKSIZE=512 -co COMPRESS=DEFLATE -co PREDICTOR=STANDARD -co NUM_THREADS=ALL_CPUS '/vsicurl/https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/47/Q/RU/2021/12/S2B_47QRU_20211227_0_L2A/B04.tif' gdal_translate_cog.tif

  2. Use gdalinfo to check if it contains internal overviews.
    gdalinfo gdal_translate_cog.tif

Metadata:
  AREA_OR_POINT=Area
  OVR_RESAMPLING_ALG=AVERAGE
Image Structure Metadata:
  COMPRESSION=DEFLATE
  INTERLEAVE=BAND
  LAYOUT=COG
  PREDICTOR=2
Corner Coordinates:
Upper Left  (  799980.000, 1900020.000) (101d49'11.11"E, 17d 9'55.04"N)
Lower Left  (  799980.000, 1790220.000) (101d48'18.93"E, 16d10'26.10"N)
Upper Right (  909780.000, 1900020.000) (102d51' 1.37"E, 17d 8'53.72"N)
Lower Right (  909780.000, 1790220.000) (102d49'50.22"E, 16d 9'28.52"N)
Center      (  854880.000, 1845120.000) (102d19'35.41"E, 16d39'43.18"N)
Band 1 Block=512x512 Type=UInt16, ColorInterp=Gray
  NoData Value=0
  Overviews: 5490x5490, 2745x2745, 1373x1373, 687x687
  1. Read the previous file and write to new raster file via OTB.

otbcli_BandMathX -il gdal_translate_cog.tif -exp "im1b1" -out 'otb_cog.tif?&gdal:co:COMPRESS=DEFLATE&gdal:co:TILED=YES&gdal:co:BLOCKXSIZE=512&gdal:co:BLOCKYSIZE=512&gdal:co:NUM_THREADS=ALL_CPUS&gdal:co:COPY_SRC_OVERVIEWS=YES' uint16

  1. Use gdalinfo to check if it contains internal overviews:
    gdalinfo otb_cog.tif
Metadata:
  AREA_OR_POINT=Area
  DataType=3
  METADATATYPE=OTB
  OTB_VERSION=8.0.1
  TileHintX=512
  TileHintY=512
Image Structure Metadata:
  COMPRESSION=DEFLATE
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  (  799980.000, 1900020.000) (101d49'11.11"E, 17d 9'55.04"N)
Lower Left  (  799980.000, 1790220.000) (101d48'18.93"E, 16d10'26.10"N)
Upper Right (  909780.000, 1900020.000) (102d51' 1.37"E, 17d 8'53.72"N)
Lower Right (  909780.000, 1790220.000) (102d49'50.22"E, 16d 9'28.52"N)
Center      (  854880.000, 1845120.000) (102d19'35.41"E, 16d39'43.18"N)
Band 1 Block=512x512 Type=UInt16, ColorInterp=Gray
  NoData Value=0
  Metadata:
    NoData=0

Overviews missing…

Any suggestion would be helpful
Thank you

1 Like

Hi,

I also have tried to write the overviews in output COG with OTB from a non COG raster, without succes (the output geotiff is a valid COG, but without overviews)

Does someone have a solution?
Maybe something is missing?

I have created an issue on gitlab

1 Like

Thank you @remi.cresson. I hope the issue will be resolved so we can use OTB to write COG.

There is a section in GeoTiff driver page where it is mentioned:

Note that this creation option will have no effect if general options (i.e. options which are not creation options) of gdal_translate are used.

I wasn’t sure what was causing this so gave up… I also did try few other things like creating a cog via gdal translate in memory and then again writing the memory file back to disk via otb using copy_src_overviews but still internal overviews were still missing…

It seems there is an mention here #3917 (COPY_SRC_OVERVIEWS option of gdal_translate incompatible with -outsize and -srcwin) – GDAL when copy_src_overview will have no effect. But I’m not sure if this is somehow related to what is happening here.

I am not really sure what COPY_SRC_OVERVIEW=YES creation option does in OTB, but you definitely want this option along with TILED=YES so that your output geotiff is COG compliant.

I think that generally speaking, the overviews should be computed after the output full-resolution image is ready, and I don’t think that OTB is designed to do otherwise (i.e. process each overview independently, which have no sense is many cases, e.g. neighborhood based processing).

However we should be able to write COGs with overviews. But… the main issue with the GDAL COG driver is that the entire input should already exist. Meaning, in OTB, that the streaming cannot be used, thus the whole image has to be processed in-memory. I have tried something to enable OTB to enforce streaming upstream the image write, when the output file format doesn’t support streaming (as it is the case with COG or JPEG2000 for instance), see here… However you can see the limits, in particular the whole output image has to fit in memory, which is not an option sometimes! The OTB processing with output gtiff + gdal_translate with output cog is more straightforward, with a smaller memory footprint, but it requires temporary files writing…

To enable COG in OTB, you just have to wait that they bump GDAL version >=3.5.3 (of patch manually find_externalGDAL.cmake to update GDAL src link) and you do

sed -i 's/GTiff/COG/g' Modules/IO/IOGDAL/src/otbGDALImageIO.cxx

After what you recompile OTB, and you can do some tests writing COG with overviews, but breaking the streaming of OTB. You can use my MR to see if it helps with the memory (it should!) but again there is a strong limitation with the memory.