Setup python to get the orfeo toolbox work properly together with pyqgis from a pycharm ide or a standalone python script

Ianuam · June 1, 2020, 4:08pm

I am changing my python environment from working with ArcGis and Arcpy to working with QGIS and PyQgis. And I also want to use the machinelearning possibilities from the orfeo toolbox. In doing so
I have succesfully installed the orfeo toolbox (7.0.0) in my QGIS 3.13 installation on my Win10 PC.

Now I want to use stand alone scripts or aan IDE like pycharm to access the orfeo toolbox like other pyqgis processing tools like this:

processing.run("otb:TrainImagesClassifier.......

the error message I receive is:

core.QgsProcessingException: Error: Algorithm otb:TrainImagesClassifier not found.

also the simple command:

import otbApplication as otb

in python results in:

ModuleNotFoundError: No module named 'otbApplication'

How do I have to setup python to get the orfeo toolbox work properly together with pyqgis?

I am using the python 3.7 interpreter from QGIS
I am using the PyCharm ide and try to set things up using:

@echo off
SET OSGEO4W_ROOT=C:\OSGeo4W64
call "%OSGEO4W_ROOT%"\OTB-7.0.0-Win64\otbenv.bat
call "%OSGEO4W_ROOT%"\bin\o4w_env.bat
call "%OSGEO4W_ROOT%\bin\qt5_env.bat"
call "%OSGEO4W_ROOT%\bin\py3_env.bat"
call "%OSGEO4W_ROOT%"\apps\grass\grass78\etc\env.bat
@echo off
path %PATH%;%OSGEO4W_ROOT%\apps\qgis\bin
path %PATH%;%OSGEO4W_ROOT%\apps\grass\grass78\lib
path %PATH%;%OSGEO4W_ROOT%\apps\Qt5\bin
path %PATH%;%OSGEO4W_ROOT%\apps\Python37\Scripts
path %PATH%;C:\qgis-plugins\qgis-otb-plugin
path %PATH%;%OSGEO4W_ROOT%\apps\qgis\python\plugins
path %PATH%;%OSGEO4W_ROOT%\apps\qgis\python\plugins\processing
path %PATH%;C:\Users\...\AppData\Roaming\QGIS\QGIS3\profiles\default\python\plugins
path %PATH%;%OSGEO4W_ROOT%\apps\saga
path %PATH%;%OSGEO4W_ROOT%\apps\gdal2

set GDAL_FILENAME_IS_UTF8=YES

set VSI_CACHE=TRUE
set VSI_CACHE_SIZE=1000000
set QT_PLUGIN_PATH=%OSGEO4W_ROOT%\apps\qgis\qtplugins;%OSGEO4W_ROOT%\apps\qt5\plugins

set PYTHONPATH=%PYTHONPATH%;%OSGEO4W_ROOT%\apps\qgis\python
set PYTHONHOME=%OSGEO4W_ROOT%\apps\Python37

set QT_QPA_PLATFORM_PLUGIN_PATH=%OSGEO4W_ROOT%\apps\Qt5\plugins\platforms
set QGIS_PREFIX_PATH=%OSGEO4W_ROOT%\apps\qgis
SET PYCHARM="C:\Program Files\JetBrains\PyCharm Community Edition 2020.1.1\bin\pycharm64.exe"
start "PyCharm aware of Quantum GIS" /B %PYCHARM% %*

cmd.exe

I hope you can give me some hints where my thinking goes wrong.

remi.cresson · June 2, 2020, 8:55am

Hi,

To use QGIS processing API, I think that you must tell QGIS where is installed OTB.
To use OTB python API (which is not the same thing as the QGIS API), you could define the following environment variables:

PATH <-- otb bin directory
LD_LIBRARY_PATH (PATH on windows?) <-- otb lib directory
OTB_APPLICATION_PATH <-- otb lib/otb/applications directory
PYTHONPATH <-- otb lib/otb/python directory

Hope that helps you!

Ianuam · June 6, 2020, 6:45am

Hi @remi.cresson,

Thank you very much for your answer. It got me one step further, I think. Instead of

`PYTHONPATH`  <-- otb  `lib/otb/python`  directory

I used:

`PYTHONPATH`  <-- otb  `lib/python`  directory

because the lib/otb/python directory does not exist. And the file otbApllication.py is in the otblib/python directory.

The result is the following setup (where changes following your advice are marked:

@echo off
SET OSGEO4W_ROOT=C:\OSGeo4W64
call "%OSGEO4W_ROOT%"\OTB-7.0.0-Win64\otbenv.bat
call "%OSGEO4W_ROOT%"\bin\o4w_env.bat
call "%OSGEO4W_ROOT%\bin\qt5_env.bat"
call "%OSGEO4W_ROOT%\bin\py3_env.bat"
call "%OSGEO4W_ROOT%"\apps\grass\grass78\etc\env.bat
@echo off
path %PATH%;%OSGEO4W_ROOT%\apps\qgis\bin
path %PATH%;%OSGEO4W_ROOT%\apps\grass\grass78\lib
path %PATH%;%OSGEO4W_ROOT%\apps\Qt5\bin
path %PATH%;%OSGEO4W_ROOT%\apps\Python37\Scripts
**path %PATH%;%OSGEO4W_ROOT%\OTB-7.0.0-Win64\bin**
**path %PATH%;%OSGEO4W_ROOT%\OTB-7.0.0-Win64\lib**
path %PATH%;%OSGEO4W_ROOT%\apps\qgis\python\plugins
path %PATH%;%OSGEO4W_ROOT%\apps\qgis\python\plugins\processing
path %PATH%;C:\Users\...\AppData\Roaming\QGIS\QGIS3\profiles\default\python\plugins
path %PATH%;%OSGEO4W_ROOT%\apps\saga
path %PATH%;%OSGEO4W_ROOT%\apps\gdal2

set GDAL_FILENAME_IS_UTF8=YES

set VSI_CACHE=TRUE
set VSI_CACHE_SIZE=1000000
set QT_PLUGIN_PATH=%OSGEO4W_ROOT%\apps\qgis\qtplugins;%OSGEO4W_ROOT%\apps\qt5\plugins

set PYTHONPATH=%PYTHONPATH%;%OSGEO4W_ROOT%\apps\qgis\python;**%OSGEO4W_ROOT%\OTB-7.0.0-Win64\lib\python**
set PYTHONHOME=%OSGEO4W_ROOT%\apps\Python37
**set OTB_APPLICATION_PATH=%OSGEO4W_ROOT%\OTB-7.0.0-Win64\lib\otb\applications**


set QT_QPA_PLATFORM_PLUGIN_PATH=%OSGEO4W_ROOT%\apps\Qt5\plugins\platforms
set QGIS_PREFIX_PATH=%OSGEO4W_ROOT%\apps\qgis
SET PYCHARM="C:\Program Files\JetBrains\PyCharm Community Edition 2020.1.1\bin\pycharm64.exe"
start "PyCharm aware of Quantum GIS" /B %PYCHARM% %*

cmd.exe

When I try to import otbApplication:

import otbApplication as otb

I receive the following error:

Traceback (most recent call last):
  File "C:\OSGEO4~1\OTB-7.0.0-Win64\lib\python\otbApplication.py", line 14, in swig_import_helper
    return importlib.import_module(mname)
  File "C:\OSGEO4~1\apps\Python37\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 670, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 583, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1043, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: DLL load failed: The specified module could not be found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/D.J/panels/test1.py", line 1, in <module>
    import otbApplication as otb
  File "C:\OSGEO4~1\OTB-7.0.0-Win64\lib\python\otbApplication.py", line 17, in <module>
    _otbApplication = swig_import_helper()
  File "C:\OSGEO4~1\OTB-7.0.0-Win64\lib\python\otbApplication.py", line 16, in swig_import_helper
    return importlib.import_module('_otbApplication')
  File "C:\OSGEO4~1\apps\Python37\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ImportError: DLL load failed: The specified module could not be found.

Do you have an idea what causes this error?

Regards!

remi.cresson · June 6, 2020, 9:19am

Hi,
I did not see that at first:

I believe that this should init properly the env vars for OTB.
However refering to the doc, it looks like you must also call the otbenv script to init the python bindings for OTB. Can you try that?
Take a look in those scripts, you will see what env vars you need to declare.

Ianuam · June 7, 2020, 7:42am

Hi @remi.cresson,

I started (again) from scratch using this post.

@echo off
SET OSGEO4W_ROOT=C:\OSGeo4W64

call "%~dp0\o4w_env.bat"
call "%OSGEO4W_ROOT%"\apps\grass\grass78\etc\env.bat
call qt5_env.bat
call py3_env.bat

@echo off
path %OSGEO4W_ROOT%\apps\qgis\bin;%OSGEO4W_ROOT%\apps\grass\grass78\lib;%OSGEO4W_ROOT%\apps\grass\grass78\bin;%PATH%
set QGIS_PREFIX_PATH=%OSGEO4W_ROOT:\=/%/apps/qgis
set GDAL_FILENAME_IS_UTF8=YES
rem Set VSI cache to be used as buffer, see #6448
set VSI_CACHE=TRUE
set VSI_CACHE_SIZE=1000000
set QT_PLUGIN_PATH=%OSGEO4W_ROOT%\apps\qgis\qtplugins;%OSGEO4W_ROOT%\apps\qt5\plugins
SET PYCHARM="C:\Program Files\JetBrains\PyCharm Community Edition 2020.1.1\bin\pycharm64.exe"
start "PyCharm aware of Quantum GIS" /B %PYCHARM% %*

When I add “call “%OSGEO4W_ROOT%”\OTB-7.1.0-Win64\otbenv.bat”, then the following error occurs

Traceback (most recent call last):
  File "C:/Users/D.J/panels/test.py", line 4, in <module>
    from qgis.core import QgsApplication, QgsProcessingFeedback
  File "C:\OSGeo4W64\apps\qgis\python\qgis\core\__init__.py", line 25, in <module>
    from qgis._core import *
ImportError: DLL load failed: Kan opgegeven procedure niet vinden.

When I looked into otbenv.bat I could not quite understand everything that was in there, but t seems that three settings had to be set:

call :prefix_path PATH "%CURRENT_SCRIPT_DIR%bin"
call :prefix_path PYTHONPATH "%CURRENT_SCRIPT_DIR%lib\python"
call :prefix_path OTB_APPLICATION_PATH "%CURRENT_SCRIPT_DIR%lib\otb\applications"

I translated that into:

path %PATH%;%OSGEO4W_ROOT%\OTB-7.1.0-Win64\bin
path %PYTHONPATH%;%OSGEO4W_ROOT%\OTB-7.1.0-Win64\lib\python
path %OTB_APPLICATION_PATH%;%OSGEO4W_ROOT%\OTB-7.1.0-Win64\lib\otb\applications

I inserted that into my .bat file, and it got me the same error.

It seems that the OTB settings are fighting the QGIS settings.

LoutK · June 11, 2020, 2:43pm

Hi @Ianuam and @remi.cresson,

I’m currently getting pretty much the same errors as you do when using otb 7.1.0 in combination with python 3.7.
Switching to an environment with python 3.5 solved the errors for me. However, using 3.5 creates some complications with other packages that I need to use… so it’s definitely not an optimal solution.
If you have found a solution for fixing the problem in 3.7 I’ll gladly hear it.

Ianuam · June 13, 2020, 7:59am

Hi @LoutK,

I am pleased that I am not the only one struggling.
I succeeded in setting up a 3.5 environment with OTB, but then I lost the cgis.core.
Were you able to access both packages within the python 3.5 environment?

Best

alexyshr · November 24, 2021, 11:44pm

Hi guys. Did you manage to run OTB using Pycharm or spyder in Windows 10?. It is really frustrating trying to use python OTB with other libraries or IDEs. That kind of installation in windows is not possible.

LoutK · November 25, 2021, 9:33am

Hi @alexyshr ,

I managed to get in working in Pycharm in Windows 10. To do so I needed to change/add some system variables. Furthermore, I could only get it going with python 3.5.

System variables:

OTB_APPLICATION_PATH = C:\Users\907141\Programs\OTB-7.1.0-Win64\lib\otb\applications

path = C:\Users\907141\Programs\OTB-7.1.0-Win64\bin

PYTHONPATH = C:\Users\907141\Anaconda3\envs\test35 (env with python 3.5)

Use Conda evironment with python 3.5 and numpy 1.18.1

balling7 · November 29, 2021, 10:15pm

Hi @Ianuam and @LoutK , did you manage to get the processing.run (“otb: algorithms”) running in a standalone python script? I can access the otb algorithms in QGIS python console, however could not access it in a python script. thanks.

LoutK · November 30, 2021, 10:00am

Yes, I used the MeanShiftSmoothing and MeanShiftSegmentation.
Example code of the MeanShiftSmoothing, perhaps that helps:

import otbApplication

def MeanShiftSmoothing (in_raster, out_spectral_raster, out_spatial_raster, spatial_radius, range_radius, convergence_threshold, max_iterations):
    Smoothing = otbApplication.Registry.CreateApplication("MeanShiftSmoothing")
    Smoothing.SetParameterString("in", in_raster)
    Smoothing.SetParameterString("fout", out_spectral_raster)
    Smoothing.SetParameterString("foutpos", out_spatial_raster)
    Smoothing.SetParameterInt("spatialr", spatial_radius)
    Smoothing.SetParameterFloat("ranger", range_radius)
    Smoothing.SetParameterFloat("thres", convergence_threshold)
    Smoothing.SetParameterInt("maxiter", max_iterations)
    Smoothing.ExecuteAndWriteOutput()
    return 0

balling7 · November 30, 2021, 1:08pm

Hi @LoutK ,

Thanks heaps for the reply, so did you bypass the QGIS and directly use OTB tools?
And for the sample code you were posted, did you run it in Python 3.5?

I am trying to call the processing.run (“otb: MeanShiftSmoothing”) in a stand-alone python script, and got an error

core.QgsProcessingException: Error: Algorithm MeanShiftSmoothing not found.

If I run the same code in QGIS Python console, it’s working fine.

Thanks

LoutK · December 8, 2021, 12:47pm

Hi @balling7 ,

I used OTB tools directly in Pycharm, yes. So no QGIS or anything like that.
Also yes, I had to use Pyton 3.5, because for whatever reason that was the only version which worked.

The error you are getting is because you are referring to a QGIS function. It is not necessary to access OTB via QGIS in Python. You can use OTB with their library directly.