TrainVectorClassifier returns no result when using Artificial Neural Network

OTB Usage
1

I experimented with the TrainVectorClassifier from ORFEO-Toolbox,
I created trainingdata and tried different classification methods, however for some reason I cannot get the Classifier to work when using ANN as parameter.

Does somebody know what I did wrong?

Training Data Head (there are actually 4 classes in MC_ID):

fid MC_ID value_0 value_1 value_2
1 1 43.5757064819336 67.1139984130859 23.9567775726318
2 1 44.5364456176758 67.6646423339844 24.185676574707
3 1 46.7659378051758 71.6526565551758 24.8563995361328
4 1 40.2990226745605 60.2364349365234 21.3831405639648
5 1 46.2593841552734 70.1774063110352 26.6104145050049

Using Artificial Neural Network:

Eingabeparameter:
{ 'io.vd' : ['/home/raphi/Robonav/Wiesen/test_clip/smpls.gpkg|layername=output'], 'io.stats' : '', 'io.out' : 'TEMPORARY_OUTPUT', 'io.confmatout' : 'TEMPORARY_OUTPUT', 'layer' : 0, 'feat' : '\"value_0 value_1 value_2\"', 'valid.vd' : None, 'valid.layer' : 0, 'cfield' : 'MC_ID', 'v' : True, 'classifier' : 'ann', 'classifier.ann.t' : 'back', 'classifier.ann.sizes' : '\"10 5\"', 'classifier.ann.f' : 'sig', 'classifier.ann.a' : 0, 'classifier.ann.b' : 1, 'classifier.ann.bpdw' : 1, 'classifier.ann.bpms' : 1, 'classifier.ann.rdw' : 0.1, 'classifier.ann.rdwm' : 0, 'classifier.ann.term' : 'all', 'classifier.ann.eps' : 0.01, 'classifier.ann.iter' : 1000, 'rand' : 1 }

2021-04-21 10:56:04 (INFO) TrainVectorClassifier: Default RAM limit for OTB is 256 MB
2021-04-21 10:56:04 (INFO) TrainVectorClassifier: GDAL maximum cache size is 774 MB
2021-04-21 10:56:04 (INFO) TrainVectorClassifier: OTB will use at most 16 threads
2021-04-21 10:56:04 (INFO) TrainVectorClassifier: Reading vector file 1/1
2021-04-21 10:56:04 (WARNING) TrainVectorClassifier: The validation set is empty. The performance estimation is done using the input training set in this case.
2021-04-21 10:56:04 (INFO) TrainVectorClassifier: Computing model file : /tmp/processing_gZqRCM/ac95332ec44f42d8afac20072e049ecc/io_out.file
Training model...: 100% [**************************************************] (0s)
Validation...: 100% [**************************************************] (0s)
2021-04-21 10:56:04 (INFO) TrainVectorClassifier: Predicted list size : 5116
2021-04-21 10:56:04 (INFO) TrainVectorClassifier: ValidationLabeledListSample size : 5116
2021-04-21 10:56:04 (INFO) TrainVectorClassifier: Training performances:
2021-04-21 10:56:04 (INFO) TrainVectorClassifier: Confusion matrix (rows = reference labels, columns = produced labels):
[1] [2] [3] [4]
[ 1] 1279 0 0 0
[ 2] 1279 0 0 0
[ 3] 1279 0 0 0
[ 4] 1279 0 0 0

2021-04-21 10:56:04 (INFO) TrainVectorClassifier: Precision of class [1] vs all: 0.25
2021-04-21 10:56:04 (INFO) TrainVectorClassifier: Recall of class [1] vs all: 1
2021-04-21 10:56:04 (INFO) TrainVectorClassifier: F-score of class [1] vs all: 0.4

2021-04-21 10:56:04 (INFO) TrainVectorClassifier: Precision of class [2] vs all: 0
2021-04-21 10:56:04 (INFO) TrainVectorClassifier: Recall of class [2] vs all: 0
2021-04-21 10:56:04 (INFO) TrainVectorClassifier: F-score of class [2] vs all: 0

2021-04-21 10:56:04 (INFO) TrainVectorClassifier: Precision of class [3] vs all: 0
2021-04-21 10:56:04 (INFO) TrainVectorClassifier: Recall of class [3] vs all: 0
2021-04-21 10:56:04 (INFO) TrainVectorClassifier: F-score of class [3] vs all: 0

2021-04-21 10:56:04 (INFO) TrainVectorClassifier: Precision of class [4] vs all: 0
2021-04-21 10:56:04 (INFO) TrainVectorClassifier: Recall of class [4] vs all: 0
2021-04-21 10:56:04 (INFO) TrainVectorClassifier: F-score of class [4] vs all: 0

2021-04-21 10:56:04 (INFO) TrainVectorClassifier: Global performance, Kappa index: 0
2021-04-21 10:56:04 (INFO) TrainVectorClassifier: mapOfIndicesValid[0] = 1
2021-04-21 10:56:04 (INFO) TrainVectorClassifier: mapOfIndicesValid[1] = 2
2021-04-21 10:56:04 (INFO) TrainVectorClassifier: mapOfIndicesValid[2] = 3
2021-04-21 10:56:04 (INFO) TrainVectorClassifier: mapOfIndicesValid[3] = 4

Using Random Forest:

Eingabeparameter:
{ 'io.vd' : ['/home/raphi/Robonav/Wiesen/test_clip/smpls.gpkg|layername=output'], 'io.stats' : '', 'io.out' : 'TEMPORARY_OUTPUT', 'io.confmatout' : 'TEMPORARY_OUTPUT', 'layer' : 0, 'feat' : '\"value_0 value_1 value_2\"', 'valid.vd' : None, 'valid.layer' : 0, 'cfield' : 'MC_ID', 'v' : True, 'classifier' : 'rf', 'classifier.rf.max' : 5, 'classifier.rf.min' : 10, 'classifier.rf.ra' : 0, 'classifier.rf.cat' : 10, 'classifier.rf.var' : 0, 'classifier.rf.nbtrees' : 100, 'classifier.rf.acc' : 0.01, 'rand' : 1 }

2021-04-21 10:55:51 (INFO) TrainVectorClassifier: Default RAM limit for OTB is 256 MB
2021-04-21 10:55:51 (INFO) TrainVectorClassifier: GDAL maximum cache size is 774 MB
2021-04-21 10:55:51 (INFO) TrainVectorClassifier: OTB will use at most 16 threads
2021-04-21 10:55:51 (INFO) TrainVectorClassifier: Reading vector file 1/1
2021-04-21 10:55:51 (WARNING) TrainVectorClassifier: The validation set is empty. The performance estimation is done using the input training set in this case.
2021-04-21 10:55:51 (INFO) TrainVectorClassifier: Computing model file : /tmp/processing_gZqRCM/15a7b601692c4dcbac7834ab4bb56025/io_out.file
Training model...: 100% [**************************************************] (0s)
Validation...: 100% [**************************************************] (0s)
2021-04-21 10:55:51 (INFO) TrainVectorClassifier: Predicted list size : 5116
2021-04-21 10:55:51 (INFO) TrainVectorClassifier: ValidationLabeledListSample size : 5116
2021-04-21 10:55:51 (INFO) TrainVectorClassifier: Training performances:
2021-04-21 10:55:51 (INFO) TrainVectorClassifier: Confusion matrix (rows = reference labels, columns = produced labels):
[1] [2] [3] [4]
[ 1] 1232 0 47 0
[ 2] 0 1277 0 2
[ 3] 124 0 1155 0
[ 4] 0 5 0 1274

2021-04-21 10:55:51 (INFO) TrainVectorClassifier: Precision of class [1] vs all: 0.908555
2021-04-21 10:55:51 (INFO) TrainVectorClassifier: Recall of class [1] vs all: 0.963253
2021-04-21 10:55:51 (INFO) TrainVectorClassifier: F-score of class [1] vs all: 0.935104

2021-04-21 10:55:51 (INFO) TrainVectorClassifier: Precision of class [2] vs all: 0.9961
2021-04-21 10:55:51 (INFO) TrainVectorClassifier: Recall of class [2] vs all: 0.998436
2021-04-21 10:55:51 (INFO) TrainVectorClassifier: F-score of class [2] vs all: 0.997267

2021-04-21 10:55:51 (INFO) TrainVectorClassifier: Precision of class [3] vs all: 0.960899
2021-04-21 10:55:51 (INFO) TrainVectorClassifier: Recall of class [3] vs all: 0.903049
2021-04-21 10:55:51 (INFO) TrainVectorClassifier: F-score of class [3] vs all: 0.931076

2021-04-21 10:55:51 (INFO) TrainVectorClassifier: Precision of class [4] vs all: 0.998433
2021-04-21 10:55:51 (INFO) TrainVectorClassifier: Recall of class [4] vs all: 0.996091
2021-04-21 10:55:51 (INFO) TrainVectorClassifier: F-score of class [4] vs all: 0.99726

2021-04-21 10:55:51 (INFO) TrainVectorClassifier: Global performance, Kappa index: 0.95361
2021-04-21 10:55:51 (INFO) TrainVectorClassifier: mapOfIndicesValid[0] = 1
2021-04-21 10:55:51 (INFO) TrainVectorClassifier: mapOfIndicesValid[1] = 2
2021-04-21 10:55:51 (INFO) TrainVectorClassifier: mapOfIndicesValid[2] = 3
2021-04-21 10:55:51 (INFO) TrainVectorClassifier: mapOfIndicesValid[3] = 4
Ausführung nach 0.66 Sekunden abgeschlossen
Ergebnisse:
{'io.confmatout': '/tmp/processing_gZqRCM/8cb2b811345b4ac3bb877e16388cff25/io_confmatout.file',
'io.out': '/tmp/processing_gZqRCM/15a7b601692c4dcbac7834ab4bb56025/io_out.file'}
2

Hello Raphael,

I just tried some to perform a classification using neural network and I encountered the same problem (all samples are classified as the first class). I think there is something wrong going on here. I need to investigate.

I the meantime you can try to use an older version of OTB, for example OTB-6.6.1. I managed to train a valid neural network classifier with this version and it should be compatible with QGIS. You can find it on the archive page : Index of /packages/archives/OTB

Cédric

3

Thank you, Cédric, version 6.6 worked perfectly!

Raphael

4

I pushed a fix, this bug should be solved in the next release !

Regards,
Cédric

1 Like