Skip to content

CUDA 12 support (libcublas.so.11 is not found)Β #1250

Closed

Description

@digitalsignalperson

Per https://opennmt.net/CTranslate2/installation.html the requirement says "Install CUDA 11.2 or above to use the GPU."

I'm on arch linux with the latest cuda 12.1.1-3 which has libcublas.so.12 and not libcublas.so.11

I'm trying to use CTranslate2 via faster-whisper and it blows up here
https://github.com/guillaumekln/faster-whisper/blob/1bb7e33b933dde488a17fd556a1114ee0d58d34b/faster_whisper/transcribe.py#L564

with

--> 564 return self.model.encode(features, to_cpu=to_cpu)

RuntimeError: Library libcublas.so.11 is not found or cannot be loaded

Activity

  1. guillaumekln commented on May 30, 2023

    @guillaumekln
    Collaborator

    The documentation is no longer correct since the release of CUDA 12. The released binaries are compiled against CUDA 11. It would be more correct to say "Install CUDA 11.x to use the GPU".

    Do you want to help fixing the documentation?

    Regarding CUDA 12 support, it currently requires to recompile the binaries from the sources. There are no immediate plans to release binaries compiled with CUDA 12.

  2. wizd commented on Nov 17, 2023

    @wizd

    Any plan to support CUDA 12.x?

  3. BBC-Esq commented on Nov 17, 2023

    @BBC-Esq
    Contributor

    I was wondering the same thing actually!

  4. StephennFernandes commented on Nov 20, 2023

    @StephennFernandes

    @guillaumekln is there any update on cuda 12 ?

  5. vince62s commented on Nov 21, 2023

    @vince62s
    Member

    This requires some work in the github ci workflow.
    You know you can build from source quite easily.
    https://opennmt.net/CTranslate2/installation.html#download-the-source-code

  6. wizd commented on Nov 22, 2023

    @wizd

    build manually possible. bellow is a bash history to reference:

       10  apt install git build-essential
       13  apt install cmake
       21  apt-get install libomp5
       23  apt install libomp-dev
       24  apt-get install linux-headers-$(uname -r)
       25  apt-get install cuda-toolkit
       26  wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
       27  apt-get install wget
       28  wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
       29  dpkg -i cuda-keyring_1.1-1_all.deb
       30  rm cuda-keyring_1.1-1_all.deb
       31  cd ..
       32  cd
       33  apt update
       34  apt-get install cuda-toolkit
       35  wget https://developer.nvidia.com/downloads/compute/cudnn/secure/8.9.6/local_installers/12.x/cudnn-local-repo-ubuntu2204-8.9.6.50_1.0-1_amd64.deb/
       36  ls
       37  ls /
       38  ls /root
       39  dpkg -i /root/cudnn-local-repo-ubuntu2204-8.9.6.50_1.0-1_amd64.deb
       40  sudo cp /var/cudnn-local-repo-ubuntu2204-8.9.6.50/cudnn-local-1998375D-keyring.gpg /usr/share/keyrings/
       41  cp /var/cudnn-local-repo-ubuntu2204-8.9.6.50/cudnn-local-1998375D-keyring.gpg /usr/share/keyrings/
       42  dpkg -i /root/cudnn-local-repo-ubuntu2204-8.9.6.50_1.0-1_amd64.deb
       43  apt-get install zlib1g
       44  apt-get update
       45  ls /root
       46  apt-cache search libcudnn
       49  apt-get install libcudnn8 libcudnn8-dev libcudnn8-samples
    
  7. BBC-Esq commented on Nov 22, 2023

    @BBC-Esq
    Contributor

    This requires some work in the github ci workflow. You know you can build from source quite easily. https://opennmt.net/CTranslate2/installation.html#download-the-source-code

    Speaking of building. I use ctranslate2 in my program that other people can download and it's installed with "pip." With "pip" installation method, which backends are installed, if any? I'm assuming that it's the defaults listed at the link you gave? For example, openblas would not be installed...ruy would not be installed...If so, I've made a huge mistake in advising people.

    Also, is there any licensing violation if I build from source with all backends enabled and distribute to others...that way they don't have to "build" on their own to use my program?

    Basically, are these the defaults when pip installed or is everything enabled...
    image

  8. Qubitium commented on Dec 11, 2023

    @Qubitium

    Here is updated ctranslate2 install for Ubuntu 22.04 with cuda 12.x. You should pip uninstall the existing ctranslate2 before proceeding.

    intel mkl

    wget -O- https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB | gpg --dearmor | sudo tee /usr/share/keyrings/oneapi-archive-keyring.gpg > /dev/null
    echo "deb [signed-by=/usr/share/keyrings/oneapi-archive-keyring.gpg] https://apt.repos.intel.com/oneapi all main" | sudo tee /etc/apt/sources.list.d/oneAPI.list
    
    apt update
    apt install intel-oneapi-mkl intel-oneapi-mkl-devel
    

    ubuntu 22.04

    apt install git build-essential
    apt install cmake
    apt install libomp5 libomp-dev
    apt install cuda-toolkit
    apt install zlib1g
    apt install libcudnn8 libcudnn8-dev libcudnn8-samples
    

    clone

    git clone --recursive https://github.com/OpenNMT/CTranslate2.git
    

    build

    cd CTranslate2
    mkdir build 
    cd build
    cmake -DWITH_MKL=ON -DWITH_CUDA=ON -DWITH_CUDNN=ON  -DOPENMP_RUNTIME=INTEL  ..
    make -j8
    make install
    

    python binding

    cd ../python
    pip install -r install_requirements.txt
    CTRANSLATE2_ROOT=/usr/local python setup.py bdist_wheel
    

    final step:

    pip install the file generated in the "dist" folder. The name of the file is different due to python version and arch.

  9. DeserterW commented on Dec 14, 2023

    @DeserterW

    Download

    $ wget https://developer.download.nvidia.com/hpc-sdk/22.9/nvhpc_2022_229_Linux_ppc64le_cuda_11.7.tar.gz
    $ tar xpzf nvhpc_2022_229_Linux_ppc64le_cuda_11.7.tar.gz
    
    

    install download cublas

    u can install cublas where u want and note it.

    echo "/tmp/nvhpc_2022_229_Linux_x86_64_cuda_11.7/install_components/Linux_x86_64/22.9/math_libs/11.7/targets/x86_64-linux/lib" >> /etc/ld.so.conf
    ldconfig
    
  10. BBC-Esq commented on Dec 14, 2023

    @BBC-Esq
    Contributor

    Thanks, I was almost going to switch from using ctranslate2/faster-whisper in my programs because it wasn't supporting CUDA 12+ out of the box (i.e. without compiling from source)...As long as it's being worked on I'd love to keep using it.

    Is there a timeline for when 12+ will be supported via the normal "pip install" method for Windows and other operating systems?

    Also, is there any possible way to allow "flavors" of "pip install" such as with "ruy" and/or "openblas?" Nobody responded whether those options are enabled with a normal "pip install" compared to compiling from source. Thanks for all the great work!
    284901097-57c80041-ba3b-430b-a132-0bf3796b796b

  11. BBC-Esq commented on Dec 14, 2023

    @BBC-Esq
    Contributor

    This requires some work in the github ci workflow. You know you can build from source quite easily. https://opennmt.net/CTranslate2/installation.html#download-the-source-code

    @vince62s , is there any licensing issue if I were to take the time to learn how to compile from source and then distribute the compiled ctranslate2 version (i.e. with all mkl, accelerate, etc. enabled) and then simply distribute that with my program? The only reason I'd want to do that is to simplify the installation for people...

  12. vince62s commented on Dec 14, 2023

    @vince62s
    Member

    There is no licensing issue at all. I understand your point. Of course we will update to CUDA 12, it is not so difficult just requires some time and unfortunately there are very few contributors to this repo.

  13. BBC-Esq commented on Dec 14, 2023

    @BBC-Esq
    Contributor

    Yeah, I hear ya...I wish there were more contributors or, alternatively, that I actually knew C++ myself... :-)

  14. taka-wang commented on Dec 15, 2023

    @taka-wang

    Yeah, I hear ya...I wish there were more contributors or, alternatively, that I actually knew C++ myself... :-)

    Hi @BBC-Esq
    I thought you might find this Dockerfile interesting. I hope it comes in handy.

  15. BBC-Esq commented on Dec 15, 2023

    @BBC-Esq
    Contributor

    Embarrisingly, I don't even know what a docker file is. Isn't it something that allows a user to run a program self contained, like a .exe, instead of having to install all the dependencies?

  16. BBC-Esq commented on Dec 15, 2023

    @BBC-Esq
    Contributor

    Just researched docker so no need to educate me. Thanks!

  17. jncraton commented on Dec 16, 2023

    @jncraton
    Contributor

    If anyone else is encountering this on Colab (or other Ubuntu jammy systems), one workaround for now is to install the libcublas11 package. This can be accomplished by including the following at the start of a Colab notebook:

    !apt install libcublas11
    
  18. Purfview commented on Dec 22, 2023

    @Purfview
    Contributor

    The tests shows that "Faster-Whisper with CUDA v12" has -10% drop in performance.

    RTX 3050 GPU:

    float16:      -10% drop in speed 
    bfloat16:      -8% drop in speed
    int8_bfloat16:  0% same
    
  19. BBC-Esq commented on Dec 22, 2023

    @BBC-Esq
    Contributor

    Do you have any more info on the tests that were run, scripts, logs even? I'd be curious to know because my current plan was to recommend users install CUDA 12+ with my program faster-whisper...

  20. Purfview commented on Dec 22, 2023

    @Purfview
    Contributor

    Nothing more, just that there were 3x tests for each quantization then the results were averaged.
    And the latest CUDA 11 libs vs the latest CUDA 12 libs.

    my current plan was to recommend users install CUDA 12+ with my program faster-whisper...

    I'll keep my repo under CUDA 11.
    Btw, you can have CUDA 12 installed and have CUDA 11 libs separate just for Faster-Whisper.

  21. BBC-Esq commented on Jan 11, 2024

    @BBC-Esq
    Contributor

    This requires some work in the github ci workflow. You know you can build from source quite easily. https://opennmt.net/CTranslate2/installation.html#download-the-source-code

    I know there was a list somewhere, where CUDA 12 support passed all checks but just hadn't been uploaded to pypi I believe? Can't seem to find it, but was generally wondering the status if you know? thanks @vince62s

  22. SnowFox4004 commented on Jan 29, 2024

    @SnowFox4004

    Here is updated ctranslate2 install for Ubuntu 22.04 with cuda 12.x. You should pip uninstall the existing ctranslate2 before proceeding.

    intel mkl

    wget -O- https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB | gpg --dearmor | sudo tee /usr/share/keyrings/oneapi-archive-keyring.gpg > /dev/null
    echo "deb [signed-by=/usr/share/keyrings/oneapi-archive-keyring.gpg] https://apt.repos.intel.com/oneapi all main" | sudo tee /etc/apt/sources.list.d/oneAPI.list
    
    apt update
    apt install intel-oneapi-mkl intel-oneapi-mkl-devel
    

    ubuntu 22.04

    apt install git build-essential
    apt install cmake
    apt install libomp5 libomp-dev
    apt install cuda-toolkit
    apt install zlib1g
    apt install libcudnn8 libcudnn8-dev libcudnn8-samples
    

    clone

    git clone --recursive https://github.com/OpenNMT/CTranslate2.git
    

    build

    cd CTranslate2
    mkdir build 
    cd build
    cmake -DWITH_MKL=ON -DWITH_CUDA=ON -DWITH_CUDNN=ON  -DOPENMP_RUNTIME=INTEL  ..
    make -j8
    make install
    

    python binding

    cd ../python
    pip install -r install_requirements.txt
    CTRANSLATE2_ROOT=/usr/local python setup.py bdist_wheel
    

    final step:

    pip install the file generated in the "dist" folder. The name of the file is different due to python version and arch.

    That works for me, thanks so much! @Qubitium

  23. Yangqing commented on Feb 9, 2024

    @Yangqing

    If you have to use cuda 12 and want a quick hacky fix, one way to try is:

    ln -s /usr/local/cuda/lib64/libcublas.so.12 /usr/local/cuda/lib64/libcublas.so.11

    since ctranslate2 uses standard cublas apis, this usually won't produce an error. You can keep all other configs as-is.

  24. hustshawn commented on Feb 14, 2024

    @hustshawn

    If you have to use cuda 12 and want a quick hacky fix, one way to try is:

    ln -s /usr/local/cuda/lib64/libcublas.so.12 /usr/local/cuda/lib64/libcublas.so.11

    since ctranslate2 uses standard cublas apis, this usually won't produce an error. You can keep all other configs as-is.

    Thanks @Yangqing. It works for me.

  25. minhthuc2502 commented on Feb 15, 2024

    @minhthuc2502
    Collaborator

    The major release 4.0.0 supports CUDA 12 now. Feel free to test it if there is any issue about this. Thanks!

  26. intexcor commented on Oct 28, 2025

    @intexcor

    Need cuda 13 support

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions