On a fresh LambdaLabs GPU instance I install Bazel with Bazelisk:
wget https://github.com/bazelbuild/bazelisk/releases/download/v1.8.1/bazelisk-linux-amd64
chmod +x bazelisk-linux-amd64
sudo mv bazelisk-linux-amd64 /usr/local/bin/bazel
Then I download the TF source:
git clone https://github.com/tensorflow/tensorflow.git
cd tensorflow
git checkout r2.11
Then I run the config:
./configure
And this is the result:
ubuntu@*********:~/tensorflow$ ./configure
You have bazel 5.3.0 installed.
Please specify the location of python. [Default is /usr/bin/python3]:
Found possible Python library paths:
/usr/lib/python3/dist-packages
/usr/local/lib/python3.8/dist-packages
Please input the desired Python library path to use. Default is [/usr/lib/python3/dist-packages]
Do you wish to build TensorFlow with ROCm support? [y/N]: n
No ROCm support will be enabled for TensorFlow.
Do you wish to build TensorFlow with CUDA support? [y/N]: y
CUDA support will be enabled for TensorFlow.
Do you wish to build TensorFlow with TensorRT support? [y/N]: y
TensorRT support will be enabled for TensorFlow.
Inconsistent CUDA toolkit path: /usr vs /usr/lib
Asking for detailed CUDA configuration...
Please specify the CUDA SDK version you want to use. [Leave empty to default to CUDA 11]:
Please specify the cuDNN version you want to use. [Leave empty to default to cuDNN 2]:
Please specify the TensorRT version you want to use. [Leave empty to default to TensorRT 6]:
Please specify the locally installed NCCL version you want to use. [Leave empty to use http://github.com/nvidia/nccl]:
Please specify the comma-separated list of base paths to look for CUDA libraries and headers. [Leave empty to use the default]:
Inconsistent CUDA toolkit path: /usr vs /usr/lib
Asking for detailed CUDA configuration...
I have no idea how to tell the compiler which CUDA toolkit path to use, and even which is the correct path. I’m just trying to rebuild TF with TFRT support.
Thanks
EDIT:
When I checked where CUDA is installed:
locate cuda | grep /cuda$
/home/ubuntu/tensorflow/tensorflow/compiler/xla/stream_executor/cuda
/home/ubuntu/tensorflow/tensorflow/stream_executor/cuda
/home/ubuntu/tensorflow/third_party/gpus/cuda
/usr/include/cuda
/usr/include/thrust/system/cuda
/usr/lib/cuda
/usr/lib/python3/dist-packages/pycuda/cuda
/usr/lib/python3/dist-packages/tensorflow/include/tensorflow/stream_executor/cuda
/usr/lib/python3/dist-packages/theano/sandbox/cuda
/usr/lib/python3/dist-packages/torch/cuda
/usr/lib/python3/dist-packages/torch/backends/cuda
/usr/lib/python3/dist-packages/torch/include/ATen/cuda
/usr/lib/python3/dist-packages/torch/include/ATen/native/cuda
/usr/lib/python3/dist-packages/torch/include/c10/cuda
/usr/lib/python3/dist-packages/torch/include/torch/csrc/cuda
/usr/lib/python3/dist-packages/torch/include/torch/csrc/jit/cuda
/usr/lib/python3/dist-packages/torch/include/torch/csrc/jit/codegen/cuda
/usr/lib/python3/dist-packages/torch/include/torch/csrc/jit/codegen/fuser/cuda
/usr/share/doc/libthrust-dev/examples/cuda
Apparently the correct path is /usr/lib/ but I don’t know how to tell the compiler to use that path.