Memory Problems with TensorFlow 2.11

Technical Help
1

I have a number of TensorFlow models that have been working fine for months or even years using TensorFlow <= 2.10.

I recently upgraded the Lambda Stack and now it’s running 2.11. The models now fail with out of memory errors (resource exhausted). There were no changes to hardware, software, or data on our end.

Two Questions.

  1. How can I run 2.10 with the current Lambda Stack?

  2. Any insights as to what’s going on with 2.11?

I’ve downgraded on another machine by running the 2.10 Docker container and this does solve the problem i.e. staying with 2.10 works.

Thanks!

2

Also please see Tensorflow 2.11 Memory Issue · Issue #60469 · tensorflow/tensorflow · GitHub for further details. Thanks!