In case there’s an alternate approach - while going through the steps at Fine tuning Stable Diffusion v2.0 with DreamBooth
I run in to:
The detected CUDA version (11.6) mismatches the version that was used to compile
PyTorch (11.7). Please make sure to use the same CUDA versions.
I wonder if re-compiling pytorch may be the solution?