How to run H100 with docker in sm_86 compatibility mode?

[jwatte]

I’m trying to use a PyTorch based container.
The nvidia runtime wasn’t automatically available.
I downloaded and manually installed the NVIDIA container runtime on my H100 instance.
I then rebooted, and started the container with --runtime-nvidia
However, I get a complaint that PyTorch doesn’t work on sm_90 hardware.
Which seems weird – wouldn’t it be backwards compatble? What am I missing?

+ docker run --runtime=nvidia --rm -it --name train -v /mpt:/mpt -v /mpt/cache:/root/.cache train composer train.py observe-help.yaml

==========
== CUDA ==
==========

CUDA Version 11.7.1

Container image Copyright (c) 2016-2022, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

/usr/lib/python3/dist-packages/torch/cuda/__init__.py:155: UserWarning: 
NVIDIA H100 PCIe with CUDA capability sm_90 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70 sm_75 sm_80 sm_86.
If you want to use the NVIDIA H100 PCIe GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

  warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))

[jwatte]

Although nvidia-smi is saying it’s working …

ubuntu@209-20-157-250:~$ nvidia-smi
Thu May 25 23:27:16 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.85.12    Driver Version: 525.85.12    CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA H100 PCIe    On   | 00000000:06:00.0 Off |                    0 |
| N/A   67C    P0   289W / 350W |  36667MiB / 81559MiB |    100%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      3686      C   /usr/bin/python3.10             36664MiB |
+-----------------------------------------------------------------------------+

confused

…

and then, no.

  File "/usr/lib/python3/dist-packages/triton_pre_mlir/compiler.py", line 901, in _compile
    name, asm, shared_mem = _triton.code_gen.compile_ttir(backend, module, device, num_warps, num_stages, extern_libs, cc)
RuntimeError: Internal Triton PTX codegen error: 
ptxas fatal   : Value 'sm_90' is not defined for option 'gpu-name'

[JosephM]

Hello @jwatte,

For H100 instances, you are going to have to install CUDA version 11.8 or higher.
I would suggest using the newest container found here:

PyTorch | NVIDIA NGC

You can also setup a Conda environment with Pytorch 2.0+CUDA 11.8. I can send the instructions if you are interested.

Let me know if this works for you.

[artemis]

Hi @JosephM , Can you please share instruction of setup Conda env with pytorch 2.0+CUDA 11.8 for H100 when you get a chance? Thanks. I tried setup a conda env on H100 but got the following msg: NVIDIA H100 PCIe with CUDA capability sm_90 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70 sm_75 sm_80 sm_86.
I checked pytorch version which is 2.0.1 Thanks

[artemis]

cuda11.8 worked. thanks:)

[JosephM]

Hello @artemis, thanks for confirming that CUDA 11.8 worked.

Here are the steps to setup a Conda environment with Pytorch 2.0+CUDA 11.8:

Install miniconda. You can run it without super user privileges and keep it local. See link here on how to download and configure: https://docs.conda.io/en/latest/miniconda.html.
For my example I will setup Python 3.9 so here’s what I downloaded and ran. Just follow the prompts:

Python 3.9
wget https://repo.anaconda.com/miniconda/Miniconda3-py39_23.3.1-0-Linux-x86_64.sh
chmod u+x Miniconda3-py39_23.3.1-0-Linux-x86_64.sh
bash Miniconda3-py39_23.3.1-0-Linux-x86_64.sh

Create your environment. I created an python3.9 environment named ‘test’. Make sure you activate your environment:

conda create -n test python=3.9
conda activate test

Install the required pytorch packages. You may notice that we use the test environment’s ‘pip’ so that it will be installed only within the domain we are setting up:

/home/ubuntu/miniconda3/envs/test/bin/pip install torch==2.0.0+cu118 torchaudio==2.0.0+cu118 torchvision==0.15.0+cu118 --extra-index-url https://download.pytorch.org/whl/cu118

Install the other CUDA packages:

conda install cudnn
conda install -c "nvidia/label/cuda-11.8.0" cuda-toolkit
conda install -c "nvidia/label/cuda-11.8.0" cuda-nvcc
conda install -c "nvidia/label/cuda-11.8.0" cuda-runtime

After that, test the pytorch installation:

python -c 'import torch ; print("PyTorch Version: ",torch.__version__) ; print("Is available: ", torch.cuda.is_available()) ; print("Current Device: ", torch.cuda.current_device()) ; print("Pytorch CUDA Compiled version: ", torch._C._cuda_getCompiledVersion()) ; print("Pytorch version: ", torch.version) ; print("pytorch file: ", torch.__file__) ; print("Number of GPUs: ",torch.cuda.device_count())'

Here’s what mine looks like:

(test) ubuntu@209-20-158-254:~$ python -c 'import torch ; print("PyTorch Verison: ",torch.__version__) ; print("Is available: ", torch.cuda.is_available()) ; print("Current Device: ", torch.cuda.current_device()) ; print("Pytorch CUDA Compiled version: ", torch._C._cuda_getCompiledVersion()) ; print("Pytorch version: ", torch.version) ; print("pytorch file: ", torch.__file__) ; print("Number of GPUs: ",torch.cuda.device_count())'
PyTorch Verison:  2.0.0+cu118
Is available:  True
Current Device:  0
Pytorch CUDA Compiled version:  11080
Pytorch version:  <module 'torch.version' from '/home/ubuntu/miniconda3/envs/test/lib/python3.9/site-packages/torch/version.py'>
pytorch file:  /home/ubuntu/miniconda3/envs/test/lib/python3.9/site-packages/torch/__init__.py
Number of GPUs:  1