Hello Vasilis,
I think you should use the RGB-NIR image. The information contained in the NIR band is probably useful to discriminate between the objects contained in the image.
The use of the 40 bands makes sense if the objects you are trying to segment are not moving during the year. Please note that the mean shift algorithm is based on Euclidian distances, and therefore might be not be adapted to high dimensions. When trying to segment such images, it might be interesting to apply dimensionality reduction as a pre-processing step of the segmentation.
Normalizing your data before the segmentation might also be useful if the input bands have different ranges.
Ultimately, finding the right input and the right parameters for this algorithm requires some trial and error, there is a recent discussion on this subject here
Please note that I am not an expert in this algorithm or in segmentation in general, so take everything I said with a grain of salt 
I encourage you to look at the Mean Shift literature to find more information. You can take a look at [1], this paper describes the implementation used in OTB and an application to Pleiades imagery.
Cédric
[1] Michel, J., Youssefi, D., & Grizonnet, M. (2015). Stable mean-shift algorithm and its application to the segmentation of arbitrarily large remote sensing images. IEEE Transactions on Geoscience and Remote Sensing, 53(2), 952-964.