Running MAJA on Sentinel-2 from build/install

oscarkramer · November 17, 2022, 4:15pm

I don’t see how to proceed with running MAJA on a single Sentinel-2 scene. The instructions in the repo’s README.md do not explain standalone very much. Virtually all the explanation is for running Start_maja for time-series, which is not my case. Is there a full description of how to run maja that goes into more depth than what is given with the --help option?

I cloned the maja repository from gitlab.orfeo-toolbox.org and checked out branch release_4.6.3. I successfully built it and now have install dir install/maja/4.6.3. I am able to run install/maja/4.6.3/bin/maja --help.

I downloaded Sentinel-2 data and untarred it to reveal the following directory structure:

$ tree -d S2B_MSIL1C_20210216T141739_N0209_R010_T20JKM_20210216T175154.SAFE/
S2B_MSIL1C_20210216T141739_N0209_R010_T20JKM_20210216T175154.SAFE/
├── AUX_DATA
├── DATASTRIP
│ └── DS_VGS2_20210216T175154_S20210216T142743
│ └── QI_DATA
├── GRANULE
│ └── L1C_T20JKM_A020628_20210216T142743
│ ├── AUX_DATA
│ ├── IMG_DATA
│ └── QI_DATA
├── HTML
└── rep_info

Can MAJA handle this? The command line description of the --input option references a lot of files I don’t have, including the DTM. I have the full SRTM 30 m (1 arcsec) dataset for the world, in .hgt format. I can also access 90m/3arcsec data. I read somewhere that the SRTM needs to be converted to a DTM format over the scene rectangle before being used by MAJA. How do I do that?

Regarding the GIPP files, I do see a collection of files that were installed with the build:

install/maja/4.6.3/share/config/SENTINEL2/S2B_TEST_GIP_L2COMM_L_ALLSITES_00001_20190626_21000101.EEF
install/maja/4.6.3/share/config/SENTINEL2/S2A_TEST_GIP_L2COMM_L_ALLSITES_00001_20190626_21000101.EEF
install/maja/4.6.3/share/schemas/SENTINEL2_TM/GIP_L2COMM_L2Commons.xsd
install/maja/4.6.3/share/schemas/SENTINEL2/GIP_L2COMM_L2Commons.xsd

Am I supposed to copy those somewhere, or does maja automagically locate them from the executable path?

Thanks.

thibaut.romain · November 22, 2022, 8:30am

Hi @oscarkramer

Startmaja does a lot of things before calling maja, like downloading the SRTM, convert it in maja readable format, downloading the GIPP etc… maja supports your S2 product you show.

If you want to use maja directly, you have to put the GIPPS, the DTM and the CAMS(if you use them) in a same folder, which also contains your .SAFE product.
There is a tool for converting the SRTM to a maja DTM format, which is called dtmcreation.
dtmcreation -p PathToProduct --type_dem=srtm --dem_dir=/TheDirectoryContainingTheSRTMs
This will create a HDR file and a DBL.DIR directory containing the DTM in maja format.

To download the GIPPs, you can checkout the repository and get the SENTINEL2 folder.

Now let’s create a folder mkdir context_maja and move every *.HDR and *.DBL.DIR and *.EEF in context_maja.

Then move S2B_MSIL1C_20210216T141739_N0209_R010_T20JKM_20210216T175154.SAFE to context_maja.

Then you can call maja :
maja -i context_maja -o context_maja_output -m L2NOMINAL (L2INIT if you only have one product) -l DEBUG

Which will run maja on your context. In the User Manual you have an example of this but we can certainly improve it

Best regards

oscarkramer · November 28, 2022, 9:59pm

Thanks @thibaut.romain for the explanations. I’m still running into a few problems. I am trying to follow the maja readme instructions in the repository for make, but am seeing this regarding test:

xstellar01:~/repos/maja/build$ cmake ../SuperBuild \
-DMAJADATA_SOURCE_DIR=/data/atmcorr/majaData \
-DMAJA_TEST_OUTPUT_ROOT=`pwd`/../testing -DDOWNLOAD_DIR=/data/atmcorr/majaData \
-DENABLE_TV=ON -DENABLE_TVA=ON 
-- MAJADATA_SOURCE_DIR       = /data/atmcorr/majaData
-- MAJADATA_SOURCE_DIR       = /data/atmcorr/majaData
-- Using CMake/MAJA.cmake
-- Using CMake/OTB.cmake
-- Using CMake/GDAL.cmake
-- Custom patches required for ZLIB
-- Custom patches required for SQLITE
-- Using CMake/JPEG.cmake
-- Custom patches required for JPEG
-- Using CMake/HDF4.cmake
-- Using CMake/HDF5.cmake
-- Using CMake/PYTHON.cmake
-- Using CMake/LIBFFI.cmake
-- Using CMake/PATCHELF.cmake
-- Using CMake/OPENSSL.cmake
-- Using CMake/SWIG.cmake
-- Using CMake/BOOST.cmake
-- Custom patches required for BOOST
-- Custom patches required for GDAL
-- Using CMake/ITK.cmake
-- Custom patches required for ITK
-- Custom patches required for OPENTHREADS
-- Custom patches required for OSSIM
-- Custom patches required for TINYXML
-- Using CMake/MUPARSER.cmake
-- Using CMake/NUMPY.cmake
-- Custom patches required for OTB
-- Custom patches required for PUGIXML
-- Using CMake/LXML.cmake
-- Using CMake/SCIPY.cmake
-- Using CMake/REQUESTS.cmake
-- Using CMake/CHARDET.cmake
-- Using CMake/CERTIFI.cmake
-- Using CMake/URLLIB.cmake
-- Using CMake/IDNA.cmake
-- Using CMake/TQDM.cmake
-- VENUS: 
-- VENUS_MUSCATE: 
-- SENTINEL2: 
-- SENTINEL2_MUSCATE: 
-- SENTINEL2_TM: 
-- LANDSAT_MUSCATE: 
-- LANDSAT8_MUSCATE: 
-- LANDSAT8: 
-- List of plugins enabled: 
-- SYSTEM_HOSTNAME           = xstellar01
-- MAJA_VERSION             = 4.6.3
-- MAJA_CORE_VERSION        = 4.6.3
-- MAJA_ALGORITHMS_VERSION  = 4.6.3
-- CMAKE_INSTALL_PREFIX     = /home/oscar.kramer/repos/maja/install/maja/4.6.3
-- PATCHELF_COMMAND         = PATCHELF_COMMAND-NOTFOUND
-- EGREP_COMMAND (for HDF4) = /bin/egrep
-- Configuring done
-- Generating done
-- Build files have been written to: /home/oscar.kramer/repos/maja/build
xstellar01:~/repos/maja/build$ make -j 8
[  4%] Built target LIBFFI
[  8%] Built target FREETYPE
[  8%] Built target EXPAT
[ 12%] Built target SQLITE
[ 12%] Built target ZLIB
[ 14%] Built target BOOST
[ 16%] Built target PATCHELF
[ 18%] Built target HDF5
[ 20%] Built target GEOS
[ 23%] Built target MUPARSER
[ 26%] Built target OPENTHREADS
[ 28%] Built target TINYXML
[ 30%] Built target PUGIXML
[ 33%] Built target JPEG
[ 35%] Built target PNG
[ 38%] Built target PROJ
[ 40%] Built target OPENSSL
[ 43%] Built target TIFF
[ 45%] Built target HDF4
[ 48%] Built target PYTHON
[ 52%] Built target ITK
[ 54%] Built target LIBGD
[ 57%] Built target GEOTIFF
[ 59%] Built target NETCDF
[ 62%] Built target OPENJPEG
[ 64%] Built target SWIG
[ 66%] Built target LXML
[ 68%] Built target NUMPY
[ 72%] Built target TQDM
[ 72%] Built target IDNA
[ 75%] Built target URLLIB
[ 77%] Built target CERTIFI
[ 80%] Built target CHARDET
[ 82%] Built target LIBXML2
[ 84%] Built target OSSIM
[ 87%] Built target GDAL
[ 89%] Built target SCIPY
[ 91%] Built target REQUESTS
[ 94%] Built target LIBXSLT
[ 97%] Built target OTB
[100%] Built target MAJA
xstellar01:~/repos/maja/build$ make install
make: *** No rule to make target 'install'.  Stop.
xstellar01:~/repos/maja/build$ make test
Running tests...
Test project /home/oscar.kramer/repos/maja/build
No tests were found!!!

I ignored the install target issue since the install directory was created and populated during the make.

Running startmaja in data directory. Here is the listing:

xstellar01:/data/atmcorr/majaData$ ls -l
total 28
drwxrwxr-x 7 oscar.kramer oscar.kramer 4096 Oct  1 16:57 S2A_MSIL1C_20221001T102841_N0400_R108_T32TLP_20221001T141232.SAFE
drwxrwxr-x 4 oscar.kramer oscar.kramer 4096 Nov 28 19:55 dtm
-rw-r--r-- 1 oscar.kramer oscar.kramer  281 Nov 28 20:00 folders.txt
drwxrwxr-x 2 oscar.kramer oscar.kramer 4096 Nov 28 19:54 gipp
drwxrwxr-x 2 oscar.kramer oscar.kramer 4096 Nov 28 19:55 l2a
drwxrwxr-x 3 oscar.kramer oscar.kramer 4096 May 31 07:06 maja-gipp2-develop-SENTINEL2
drwxrwxr-x 2 oscar.kramer oscar.kramer 4096 Nov 28 19:54 work

then running maja fails:

xstellar01:/data/atmcorr/majaData$ startmaja --tile 32TLP -f folders.txt
2022-11-28 21:56:00,061 [INFO ] =============This is Start_Maja v4.6.3==============
INFO:root:=============This is Start_Maja v4.6.3==============
Traceback (most recent call last):
  File "/home/oscar.kramer/repos/maja/install/maja/4.6.3/lib/python/StartMaja/Start_maja.py", line 878, in <module>
    ChangeTargetResToR3=args.ChangeTargetResToR3,
  File "/home/oscar.kramer/repos/maja/install/maja/4.6.3/lib/python/StartMaja/Start_maja.py", line 94, in __init__
    ) = self.__set_input_paths()
  File "/home/oscar.kramer/repos/maja/install/maja/4.6.3/lib/python/StartMaja/Start_maja.py", line 350, in __set_input_paths
    path_input_l1 = FileSystem.find_single(r"^T?%s$" % self.tile, self.rep_l1)
  File "/home/oscar.kramer/repos/maja/install/maja/4.6.3/lib/python/StartMaja/Common/FileSystem.py", line 123, in find_single
    return find(pattern, path, case_sensitive=case_sensitive, depth=depth, ftype=ftype)[
  File "/home/oscar.kramer/repos/maja/install/maja/4.6.3/lib/python/StartMaja/Common/FileSystem.py", line 108, in find
    raise ValueError("Cannot find %s in %s" % (pattern, path))
ValueError: Cannot find ^T?32TLP$ in /data/atmcorr/majaData/S2A_MSIL1C_20221001T102841_N0400_R108_T32TLP_20221001T141232.SAFE

Here is my folders.txt:

xstellar01:/data/atmcorr/majaData$ cat folders.txt 
[Maja_Inputs]
repWork=./work
repGipp=./gipp
repL1  =./S2A_MSIL1C_20221001T102841_N0400_R108_T32TLP_20221001T141232.SAFE
repL2  =./l2a
repMNT =./dtm
exeMaja=/home/oscar.kramer/repos/maja/install/maja/4.6.3/bin/maja
repCAMS=./cams

[DTM_Creation]
repRAW =./dtm/raw
repGSW =./dtm/gsw

I’m providing all the console output in the hopes you will recognize my problem. Thanks.

thibaut.romain · November 29, 2022, 8:35am

Hi @oscarkramer

When you type make it is doing a make install silently, for the tests they are built if you provide the option BUILD_TESTING to cmake. Then you can list the tests with ctest -N in your MAJA build folder.

For the problem with startmaja, you have to put your .SAFE in a subdirectory :
./L1C/32TLP/Your.SAFE

in repL1 you have to put ./L1C only

For the gipp I recommend you move maja-gipp2-SENTINEL2 to ./gipp and rename it SENTINEL2 to help Startmaja finding the right GIPP instead of downloading it

Hope that helps

oscarkramer · November 29, 2022, 3:50pm

I added -DBUILD_TESTING=ON to the cmake command line then ran make. I still don’t see any tests:

xstellar01:~/repos/maja/build$ ctest -N
Test project /home/oscar.kramer/repos/maja/build

Total Tests: 0

xstellar01:~/repos/maja/build$ make test
Running tests...
Test project /home/oscar.kramer/repos/maja/build
No tests were found!!!

I also edited the SuperBuild/CMakeLists.txt to force the default to “true” but still no luck.

I modified the data directory as you explained, so that the S2A data is under L1C//, and the gipp directory has the SENTINEL2 subdir. Here is the tree:

xstellar01:/data/atmcorr/majaData$ tree -d
.
|-- dtm
|   |-- gsw
|   `-- raw
|-- gipp
|   `-- SENTINEL2
|       |-- S2A_TEST_GIP_L2ALBD_L_AMMONIUM_00002_20190710_21000101.DBL.DIR
|       |-- S2A_TEST_GIP_L2ALBD_L_BLACKCAR_00002_20190710_21000101.DBL.DIR
|        ... (long list of *.DBL.DIR)
|-- L1C
|   `-- 32TLP
|       `-- S2A_MSIL1C_20221001T102841_N0400_R108_T32TLP_20221001T141232.SAFE
|           |-- AUX_DATA
|           |-- DATASTRIP
|           |   `-- DS_ATOS_20221001T141232_S20221001T103225
|           |       `-- QI_DATA
|           |-- GRANULE
|           |   `-- L1C_T32TLP_A038000_20221001T103225
|           |       |-- AUX_DATA
|           |       |-- IMG_DATA
|           |       `-- QI_DATA
|           |-- HTML
|           `-- rep_info
|-- L2A
|   |-- 32TLP
|   `-- L1C_T32TLP_A038000_20221001T103225
`-- work

The startmaja is failing trying to download the DEM.
I tried running dtmcreation (recall I have srtm data locally as .hgt tiles) but it is still trying to download the DEM and erroring out. Perhaps my specification of --dem_dir is incorrect, or dtmcreation cannot handle .hgt srtm format:

xstellar01:/data/atmcorr/majaData$ dtmcreation -p L1C/32TLP/S2A_MSIL1C_20221001T102841_N0400_R108_T32TLP_20221001T141232.SAFE --type_dem=srtm --dem_dir=/data/elevation/srtm/3arc -o dtm
Product:   S2A_MSIL1C_20221001T102841_N0400_R108_T32TLP_20221001T141232.SAFE
Acq-Date:  2022-10-01 10:28:41
Platform:  sentinel2
Level:     l1c
Tile/Site: 32TLP

Traceback (most recent call last):
  File "/home/oscar.kramer/repos/maja/install/maja/4.6.3/lib/python/StartMaja/prepare_mnt/DTMCreation.py", line 206, in <module>
    creator.run(args.out_dir, args.temp_dir)
  File "/home/oscar.kramer/repos/maja/install/maja/4.6.3/lib/python/StartMaja/prepare_mnt/DTMCreation.py", line 99, in run
    coarse_res = self.coarse_res
  File "/home/oscar.kramer/repos/maja/install/maja/4.6.3/lib/python/StartMaja/Chain/Product.py", line 259, in get_mnt
    **kwargs
  File "/home/oscar.kramer/repos/maja/install/maja/4.6.3/lib/python/StartMaja/prepare_mnt/mnt/MNTFactory.py", line 73, in factory
    full_res_only=self.full_res_only,
  File "/home/oscar.kramer/repos/maja/install/maja/4.6.3/lib/python/StartMaja/prepare_mnt/mnt/MNTBase.py", line 254, in to_maja_format
    mnt_max_res = self.prepare_mnt()
  File "/home/oscar.kramer/repos/maja/install/maja/4.6.3/lib/python/StartMaja/prepare_mnt/mnt/SRTM.py", line 81, in prepare_mnt
    srtm_archives = self.get_raw_data()
  File "/home/oscar.kramer/repos/maja/install/maja/4.6.3/lib/python/StartMaja/prepare_mnt/mnt/SRTM.py", line 68, in get_raw_data
    current_url, output_path, log_level=logging.INFO
  File "/home/oscar.kramer/repos/maja/install/maja/4.6.3/lib/python/StartMaja/Common/FileSystem.py", line 246, in download_file
    ret_code = run_external_app("LD_LIBRARY_PATH='' wget", args, log_level=log_level)
  File "/home/oscar.kramer/repos/maja/install/maja/4.6.3/lib/python/StartMaja/Common/FileSystem.py", line 203, in run_external_app
    raise subprocess.CalledProcessError(return_code, cmd)
subprocess.CalledProcessError: Command 'LD_LIBRARY_PATH='' wget --retry-connrefused --waitretry=1 --read-timeout=20 --timeout=15 -O /tmp/tmpu3a1ptj0 https://srtm.csi.cgiar.org/wp-content/uploads/files/srtm_5x5/TIFF/srtm_38_04.zip -nv' returned non-zero exit status 5.

Any ideas?

thibaut.romain · November 29, 2022, 4:12pm

For the tests, you have to go in the subfolder MAJA/build and run Ctest.

It seems there are some problems sometimes on the srtm website (every 3 months when the TLS certificates expires) making it impossible to download the SRTM you want.By the way, error 5 means there is a proxy error / DNS resolve impossible, are you behind a corporate proxy? This could explain the download problem despite the recurent certificates problem.

can you put your zipped srtm in the “dtm/raw” folder? it should find it next time you call startmaja.

I will have a look at the dem_dir parameter on my side just to be sure if it is used when calling startmaja

Best regards

oscarkramer · November 29, 2022, 5:15pm

For tests, I was in the build directory when I ran ctest. See console output above.

Here is the console output of startmaja. Note that it still has issues with the gipp directory:

xstellar01:/data/atmcorr/majaData$ startmaja --tile 32TLP -f folders.txt
2022-11-29 17:07:12,328 [INFO ] =============This is Start_Maja v4.6.3==============
INFO:root:=============This is Start_Maja v4.6.3==============
2022-11-29 17:07:12,329 [INFO ] Detecting input products...
INFO:root:Detecting input products...
2022-11-29 17:07:12,331 [INFO ] 1 L1C product(s) detected for tile 32TLP in /data/atmcorr/majaData/l1c/32TLP
INFO:root:1 L1C product(s) detected for tile 32TLP in /data/atmcorr/majaData/l1c/32TLP
2022-11-29 17:07:12,331 [WARNI] No L2A products detected for tile 32TLP in /data/atmcorr/majaData/l2a/32TLP
WARNING:root:No L2A products detected for tile 32TLP in /data/atmcorr/majaData/l2a/32TLP
2022-11-29 17:07:12,332 [INFO ] Skipping CAMS file detection.
INFO:root:Skipping CAMS file detection.
2022-11-29 17:07:12,332 [INFO ] Checking GIPP files
INFO:root:Checking GIPP files
2022-11-29 17:07:12,332 [INFO ] Setting up GIPP folder: /data/atmcorr/majaData/gipp
INFO:root:Setting up GIPP folder: /data/atmcorr/majaData/gipp
2022-11-29 17:07:12,343 [INFO ] Cannot find GIPP for SENTINEL2. Will attempt to download it.
INFO:root:Cannot find GIPP for SENTINEL2. Will attempt to download it.
2022-11-29 17:07:12,343 [INFO ] Searching for DTM
INFO:root:Searching for DTM
2022-11-29 17:07:12,343 [INFO ] Cannot find DTM. Will attempt to download it for type 'any'
INFO:root:Cannot find DTM. Will attempt to download it for type 'any'
2022-11-29 17:07:12,343 [INFO ] Attempting to download DTM...
INFO:root:Attempting to download DTM...
2022-11-29 17:07:13,187 [ERROR] Cannot find the following file(s): ['eu_dem_v11_E40N20.zip'].
 Please download it from https://land.copernicus.eu/imagery-in-situ/eu-dem/eu-dem-v1.1?tab=download and place it into the following directory: /data/atmcorr/majaData/dtm/raw.
 Then re-run this script.
ERROR:root:Cannot find the following file(s): ['eu_dem_v11_E40N20.zip'].
 Please download it from https://land.copernicus.eu/imagery-in-situ/eu-dem/eu-dem-v1.1?tab=download and place it into the following directory: /data/atmcorr/majaData/dtm/raw.
 Then re-run this script.
Traceback (most recent call last):
  File "/home/oscar.kramer/repos/maja/install/maja/4.6.3/lib/python/StartMaja/Start_maja.py", line 881, in <module>
    s.run()
  File "/home/oscar.kramer/repos/maja/install/maja/4.6.3/lib/python/StartMaja/Start_maja.py", line 671, in run
    raw_gsw=self.rep_gsw,
  File "/home/oscar.kramer/repos/maja/install/maja/4.6.3/lib/python/StartMaja/Chain/Product.py", line 259, in get_mnt
    **kwargs
  File "/home/oscar.kramer/repos/maja/install/maja/4.6.3/lib/python/StartMaja/prepare_mnt/mnt/MNTFactory.py", line 73, in factory
    full_res_only=self.full_res_only,
  File "/home/oscar.kramer/repos/maja/install/maja/4.6.3/lib/python/StartMaja/prepare_mnt/mnt/MNTBase.py", line 254, in to_maja_format
    mnt_max_res = self.prepare_mnt()
  File "/home/oscar.kramer/repos/maja/install/maja/4.6.3/lib/python/StartMaja/prepare_mnt/mnt/SRTM.py", line 81, in prepare_mnt
    srtm_archives = self.get_raw_data()
  File "/home/oscar.kramer/repos/maja/install/maja/4.6.3/lib/python/StartMaja/prepare_mnt/mnt/SRTM.py", line 68, in get_raw_data
    current_url, output_path, log_level=logging.INFO
  File "/home/oscar.kramer/repos/maja/install/maja/4.6.3/lib/python/StartMaja/Common/FileSystem.py", line 246, in download_file
    ret_code = run_external_app("LD_LIBRARY_PATH='' wget", args, log_level=log_level)
  File "/home/oscar.kramer/repos/maja/install/maja/4.6.3/lib/python/StartMaja/Common/FileSystem.py", line 203, in run_external_app
    raise subprocess.CalledProcessError(return_code, cmd)
subprocess.CalledProcessError: Command 'LD_LIBRARY_PATH='' wget --retry-connrefused --waitretry=1 --read-timeout=20 --timeout=15 -O /tmp/tmp662uwarx https://srtm.csi.cgiar.org/wp-content/uploads/files/srtm_5x5/TIFF/srtm_38_04.zip -nv' returned non-zero exit status 5.

I’m not sure what maja is expecting for elevation. I see reference in the output to ‘eu_dem_v11_E40N20.zip’ and also ‘srtm_38_04.zip’. However, the imagery is over Nice (N43E007), so not sure where it is getting those filenames from. I did zip up my SRTM N43E007.hgt to filename N43E007.hgt.zip and put that in dtm/raw, but it wasn’t picked up.

thibaut.romain · November 30, 2022, 8:34am

for the DEM you can try adding this parameter to your call of startmaja --type_dem=srtm

About the build folder I see that you are calling ctest from the main build folder, which contains the build of all maja dependencies and MAJA itself, if you do a ls in that directory you will see the list of folders. In order to see the list of tests, you have to go down to the MAJA build directory ie in ~/repos/maja/build/MAJA/build. If you launch ccmake from this directory, you will see if BUILD_TESTING is correctly propagated. If it is not (if you have CMAKE < 3.12 it could be the case), reactivate it and launch make from this directory before calling ctest -N

oscarkramer · November 30, 2022, 4:57pm

Thank you Romain for all the help.

I was able to build and run the tests with your instructions. Unfortunately, all 112 tests failed. No description of the failure was printed. Here is the first:

xstellar01:~/repos/maja/build/MAJA/build$ make test
Running tests...
Test project /home/oscar.kramer/repos/maja/build/MAJA/build
        Start   1: pyTv-S2-L2NOMINAL-001-SENTINEL2-ALGO_CHAIN
  1/112 Test   #1: pyTv-S2-L2NOMINAL-001-SENTINEL2-ALGO_CHAIN .........................***Failed    1.35 sec

Is there some config file that and/or env vars that need to be defined prior to running tests? I’m going to hold off trying startmaja until the tests pass. Thanks.

thibaut.romain · November 30, 2022, 5:56pm

You can run the tests with the option -VV for verbosity. I think you don’t have the DATA required by Maja which could explain the failing tests. Those data are not available to users at the moment, but you can create a test with your datas

Best regards

oscarkramer · November 30, 2022, 10:59pm

I’ll skip trying to run the tests since I can’t get to the data.

Running startmaja with “–type_dem=srtm” still errors out. I’m seeing two principal issues:

2022-11-30 22:48:19,571 [INFO ] Checking GIPP files
INFO:root:Checking GIPP files
2022-11-30 22:48:19,572 [INFO ] Setting up GIPP folder: /data/atmcorr/majaData/gipp
INFO:root:Setting up GIPP folder: /data/atmcorr/majaData/gipp
2022-11-30 22:48:19,583 [INFO ] Cannot find GIPP for SENTINEL2. Will attempt to download it.
INFO:root:Cannot find GIPP for SENTINEL2. Will attempt to download it.

As shown in a previous posting with the gipp directory tree, I do have ./gipp/SENTINEL2, populated with a bunch of *.HDR and *.DIR.

The other error is sill the DTM issue:

2022-11-30 22:52:55,721 [INFO ] Searching for DTM
INFO:root:Searching for DTM
2022-11-30 22:52:55,721 [INFO ] Cannot find DTM. Will attempt to download it for type 'srtm'
INFO:root:Cannot find DTM. Will attempt to download it for type 'srtm'

I want it to read my local .hgt SRTM file that I placed in the dtm folder.

Is there a sample dataset that is available, complete with folders.txt in zipped format that works with startmaja? That would probably be the easiest route to get me going. Thanks.

thibaut.romain · December 1, 2022, 10:55am

Here is an example I use for validating a product for maja using startmaja :

[Maja_Inputs]
repWork=/work/03-ResultsValidation/work
repGipp=/work/02-Input/gipp-m4
repL1  =/work/02-Input/L1C_Validation/SENTINEL2/
repL2  =/work/03-ResultsValidation/L2A_Products/SENTINEL2/
repMNT =/work/02-Input/dtm
exeMaja=/softs/maja/4.6.0/bin/maja
repCAMS=/work/02-Input/cams

[DTM_Creation]
repRAW =/work/02-Input/dtm/rawdem
repGSW =/work/02-Input/dtm/gsw

In rawdem I have :

dtm/rawdem/
├── srtm_22_04.zip
├── srtm_35_06.zip
├── srtm_37_03.zip
├── srtm_37_04.zip
├── srtm_38_03.zip
└── srtm_38_04.zip

In gipp-m4 I have :
SENTINEL2 with a bunch of DBL.DIR inside

I call startmaja with :

startmaja -f folders_s2.txt -t 31TFJ -y -v --cams -d 2018-04-23

When looking at the code, dtmcreation looks for zip files in rawdem directory
You can try without dem_dir which is populated using folders.txt when using startmaja

I hope you will succeed, I get back to you if I have more informations about this sample dataset

oscarkramer · December 1, 2022, 4:50pm

I think I succeeded. I manually downloaded the SRTM using the URL startmaja was unsuccessfully attempting to use. Apparently that was in a usable format, unlike my .hgt file.

Regarding the GIPP files, I see in the run log, for example:

2022-12-01 14:58:51 [317134] DEBUG MAJA 4.6 muscate_detect_l1_products:70 Parse the directory: /data/atmcorr/majaData/work/Start_maja_7742d684595e8c1aa63d6aca3d16c34e/S2A_TEST_GIP_L2ALBD_L_SEASALT__00002_20190710_21000101.DBL.DIR

I have the that .DIR in my gipp folder (in “/data/atmcorr/majaData/gipp/SENTINEL2”), but it is reading the one in the work folder, that was cleaned at the end of the run. Were those downloaded or copied out of my ./gipp folder?

The log file in the output directory is quite verbose. My initial goal is to analyse performance at the various stages of processing. I see in the log, on occasion, some timing info such as:

2022-12-01 14:58:52 [317134] DEBUG MAJA 4.6 _run:109 Finished : ConvertCartoToGeoPoint ,Time: 00 min 00 seconds ,RAM current : 424 MB ,RAM max : 424 MB

Almost all timing is 0:00, maybe it is a timing resolution problem with the print out. It isn’t clear what the time applies to. I see “Resampling”, but not a description of where it is in the process – at least it isn’t clear to me. Is there a document that I should be referring to that explains what is happening? Is there a way to get less verbose output?

Merci pour tout ton aide - Oscar

thibaut.romain · December 2, 2022, 2:47pm

Good to hear that you succeed

The DBL.DIR are being copied to the working directory during maja processing.

About the log, you can add a -l INFO to the startmaja call to haves less verbosity (or remove the -v from your call)

For the timing I think you can tweak it to show miliseconds, I will take a look about it. Do you have access to the User Manual ? You can build it with CMAKE (look at the BUILD_DOC option). In this document you have a description of the chain of algorithms

Best regards