I’ve only run into this issue when I two models training on the same GPU.
Was the output any differet when you gave the memory growth approaches a try?
devices = tf.config.experimental.list_physical_devices('GPU')
for device in devices
tf.config.experimental.set_memory_growth(devices, True)