Just a update. At least on Ubuntu 22.04 it looks like there is a ‘nvidia-cudnn’ which does include the C header files.
$ apt search nvidia-cudnn
nvidia-cudnn/jammy,now 8.2.4.15~cuda11.4 amd64 [installed]
NVIDIA CUDA Deep Neural Network library (install script)
$ sudo apt install nvidia-cudnn
It installs the header file in:
/usr/include/x86_64-linux-gnu/cudnn.h
But note this is also install the cudnn library and may conflict with the PyTorch/TensorFlow builds at times.