Hello Den,
That error means that the Mellanox card isn’t known by Open MPI lib.
As the message suggests this may be affecting performance.
If you’re not doing multi-node training this shouldn’t be a problem for Open MPI not to be able to optimize for your ethernet NIC.
Can you tell me more about your architecture?
How many nodes do you have and how are they connected?
Also, can you, please, send me the command you run with all the parameters?
Do you use Lambda Stack?
Best,
Yanos