How to Troubleshooting TensorFlow Error GPU Support on CentOS 7

When setting up TensorFlow with GPU support on CentOS 7, compatibility between TensorFlow, CUDA, and cuDNN versions is essential. In this guide, we'll walk through how to resolve an issue where TensorFlow cannot detect GPUs despite a valid GPU setup.

Problem Description

The error log shows TensorFlow failing to load libcudart.so.10.1, which prevents it from registering available GPUs. Despite having CUDA installed and two NVIDIA GTX 1080 Ti GPUs, the output of tf.config.list_physical_devices('GPU') still returns 0 GPUs.

Diagnosing the Issue

Key points of the diagnostic output:

1. TensorFlow installation includes GPU support (tensorflow-gpu==2.2.0), but it’s failing to detect the GPUs.
2. The CUDA toolkit version (as specified in the environment) is 10.2, but TensorFlow is attempting to load libcudart.so.10.1, suggesting a mismatch in required libraries.
3. There is no /usr/lib/x86_64-linux-gnu directory in CentOS by default, which is causing symbolic link failures.

The steps below detail how to resolve this mismatch and enable TensorFlow to detect the GPUs.

Solution Steps

1. Install the Required CUDA and cuDNN Versions

Check TensorFlow's compatibility chart to ensure that your TensorFlow version (in this case, 2.2.0) is compatible with your installed CUDA and cuDNN versions. For tensorflow-gpu==2.2.0, CUDA 10.1 or CUDA 10.2 and cuDNN 7.6 are recommended.

2. Create Symbolic Links for CUDA Libraries

Since TensorFlow is specifically looking for libcudart.so.10.1 but you have CUDA 10.2, create a symbolic link for libcudart.so.10.1 pointing to libcudart.so.10.2.

sudo ln -s /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudart.so.10.2 /usr/lib/libcudart.so.10.1

This command will link libcudart.so.10.2 to libcudart.so.10.1 in /usr/lib, a path that TensorFlow can access.

3. Update the LD_LIBRARY_PATH

To ensure that TensorFlow can find the linked libraries, add /usr/lib to the LD_LIBRARY_PATH:

export LD_LIBRARY_PATH=/usr/lib:$LD_LIBRARY_PATH

You can add this command to your shell’s startup file (like .bashrc or .bash_profile) to make the change persistent across sessions:

echo 'export LD_LIBRARY_PATH=/usr/lib:$LD_LIBRARY_PATH' >> ~/.bashrc
source ~/.bashrc

4. Verify the GPU Setup with TensorFlow

Restart your environment, and verify that TensorFlow now detects the GPUs:

import tensorflow as tf
print("Num GPUs Available: ", len(tf.config.list_physical_devices('GPU')))

If the output shows the correct number of GPUs, TensorFlow has successfully registered them.

Additional Notes

If these steps do not resolve the issue, consider the following troubleshooting tips:

1. Check GPU Driver Version: Ensure that your GPU driver is compatible with CUDA 10.2.
2. Virtual Environment Dependencies: If TensorFlow is installed in a virtual environment, ensure that LD_LIBRARY_PATH is accessible within it.

By following these steps, TensorFlow should successfully detect and utilize the available GPUs on CentOS 7, allowing you to harness GPU acceleration for your deep learning tasks.