Allennlp and Pytorch CUDNN_STATUS_INTERNAL_ERROR

I’ve spent a few days trying to get AllenNLP working on my lambda AI machine. Allennlp uses pytorch. I consistently run into CUDNN_STATUS_INTERNAL_ERROR when the pytorch RNN functions are touched are by AllenNLP. I’ve attempted every form of allennlp and torch install (conda, pip, docker) and continue to run into the issue. We don’t seem to run into the issue using identical commands in our GPU cloud environments, just our lambda AI machine.

Is this something anyone else who uses pytorch has run into?

Grant