Skip to content

BUG: updating from 12.6 to 12.8/12.9 breaks Python import #476

Closed
#626
Closed
BUG: updating from 12.6 to 12.8/12.9 breaks Python import#476
#626
Assignees
leofang
Labels
cuda.bindingsEverything related to the cuda.bindings moduledocumentationImprovements or additions to documentationpackagingAnything related to wheels or Conda packageswontfixThis will not be worked on
Milestone
cuda-python 13.0.0 & 12.9.1

Description

@spec-benno
spec-benno
opened on Feb 27, 2025

On Linux Ubuntu 22.08, I've recently updated the cuda-python package from 12.6 to 12.8.0 in my Python project and when I'm now trying to do:

from cuda import cuda, cudart, nvrtc

I'm getting the error message:

ImportError: cannot import name 'cuda' from 'cuda' (unknown location)

From the examples in the cuda.bindings package, I see that the same import command is still used.

Since, I'm using RDMA, I haven't tried this on Windows.

Activity

  1. github-actions
    added
    triageNeeds the team's attention
    on Feb 27, 2025

  2. leofang commented on Feb 27, 2025

    @leofang
    leofang
    on Feb 27, 2025
    Member

    I cannot reproduce it (obviously, since our CI definitely got this tested), so...

    • How did you install cuda-python? (pip? conda? other means?)
    • Could you show the output of pip list? And if you use conda environment, also conda list?

  3. spec-benno commented on Feb 27, 2025

    @spec-benno
    spec-benno
    on Feb 27, 2025
    Author

    I have an existing virtual environment with cuda-python installed and I simply did a:
    pip install cuda-python==12.8

    The pip list command outputs:

    Package           Version
----------------- -----------
contourpy         1.3.1
cuda-bindings     12.8.0
cuda-core         0.1.1
cuda-python       12.8.0
cupy-cuda12x      13.3.0
cycler            0.12.1
fastrlock         0.8.2
flexcache         0.3
flexparser        0.4
fonttools         4.55.0
freetype-py       2.5.1
h5py              3.13.0
hsluv             5.0.4
kiwisolver        1.4.7
matplotlib        3.9.2
numpy             2.2.3
packaging         24.2
pillow            11.0.0
Pint              0.24.4
pip               22.0.2
platformdirs      4.3.6
pyparsing         3.2.0
PyQt5             5.15.11
PyQt5-Qt5         5.15.15
PyQt5_sip         12.15.0
python-dateutil   2.9.0.post0
scipy             1.14.1
setuptools        59.6.0
six               1.16.0
spcm              1.5.2
spcm_core         1.0.2
typing_extensions 4.12.2
vispy             0.14.3

  4. leofang commented on Feb 27, 2025

    @leofang
    leofang
    on Feb 27, 2025
    Member

    Forgot to ask, what's the Python version?

    Also, with this broken environment, could you check what happens if you do

    from cuda.bindings import driver as cuda

    ? (Same for cudart and nvrtc)

  5. spec-benno commented on Feb 27, 2025

    @spec-benno
    spec-benno
    on Feb 27, 2025
    Author

    The Python version is 3.10.12 and with the command from cuda.bindings import driver as cuda I get:

    Exception has occurred: ModuleNotFoundError
    No module named 'cuda.bindings'
    File "...", line 13, in
    from cuda.bindings import driver as cuda
    ModuleNotFoundError: No module named 'cuda.bindings'

    ps. small correction I'm using Ubuntu LTS 22.04

  6. spec-benno commented on Feb 27, 2025

    @spec-benno
    spec-benno
    on Feb 27, 2025
    Author

    Okay, I've narrowed it down. It has to do with the migration from 12.6 to 12.8, because when setting up a new Virtual Environment with the same packages both imports work:

    from cuda import cuda
from cuda.bindings import driver as cuda2

    Here is the result from the pip list command in the new environment:

    Package           Version
----------------- -----------
contourpy         1.3.1
cuda-bindings     12.8.0
cuda-python       12.8.0
cupy-cuda12x      13.3.0
cycler            0.12.1
fastrlock         0.8.3
flexcache         0.3
flexparser        0.4
fonttools         4.56.0
freetype-py       2.5.1
h5py              3.13.0
hsluv             5.0.4
kiwisolver        1.4.8
matplotlib        3.10.0
numpy             2.2.3
packaging         24.2
pillow            11.1.0
Pint              0.24.4
pip               25.0.1
platformdirs      4.3.6
pyparsing         3.2.1
PyQt5             5.15.11
PyQt5-Qt5         5.15.16
PyQt5_sip         12.17.0
python-dateutil   2.9.0.post0
scipy             1.15.2
six               1.17.0
spcm              1.5.2
spcm_core         1.0.2
typing_extensions 4.12.2
vispy             0.14.3

  7. leofang commented on Feb 27, 2025

    @leofang
    leofang
    on Feb 27, 2025
    Member

    It seems there is a weird cleanup issue going on. With the following steps I hit an error, though different from yours

    1. Create a fresh env
    2. pip install "cuda-python==12.6.*"
    3. pip install "cuda-python==12.8"
    $ python
Python 3.12.9 | packaged by conda-forge | (main, Feb 14 2025, 08:00:06) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from cuda import cuda, cudart, nvrtc
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'cuda'

    @spec-benno could you try a complete removal and then reinstall?

    • pip uninstall -y cuda-python cuda-bindings cuda-core
    • pip install cuda-python==12.8 cuda-core

    and see if it is fixed?

  8. spec-benno commented on Feb 27, 2025

    @spec-benno
    spec-benno
    on Feb 27, 2025
    Author

    we were thinking the same thing. Setting up a new environment does the trick, however migrating is still a problem somehow

  9. spec-benno commented on Feb 27, 2025

    @spec-benno
    spec-benno
    on Feb 27, 2025
    Author

    It seems there is a weird cleanup issue going on. With the following steps I hit an error, though different from yours

    1. Create a fresh env
    2. pip install "cuda-python==12.6.*"
    3. pip install "cuda-python==12.8"
    $ python
Python 3.12.9 | packaged by conda-forge | (main, Feb 14 2025, 08:00:06) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from cuda import cuda, cudart, nvrtc
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'cuda'

    @spec-benno could you try a complete removal and then reinstall?

    • pip uninstall -y cuda-python cuda-bindings cuda-core
    • pip install cuda-python==12.8 cuda-core

    and see if it is fixed?

    I also did the uninstall and reinstall and this works as well, for both import methods. The error you're seeing I've seen as well.

  10. leofang commented on Feb 27, 2025

    @leofang
    leofang
    on Feb 27, 2025
    Member
    1. Create a fresh env
    2. pip install "cuda-python==12.6.*"
    3. pip install "cuda-python==12.8"

    Reproducer is to change the step 2 above to pip install "cuda-python==12.6", which pip interprets as installing 12.6.0 (instead of 12.6.2.post1). Then I hit the same error:

    $ python
Python 3.12.9 | packaged by conda-forge | (main, Feb 14 2025, 08:00:06) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from cuda import cuda, cudart, nvrtc
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'cuda' from 'cuda' (unknown location)

    This seems like another unexpected issue due to layout change that caused some trouble earlier (ex: #226). I don't think there's any fix we can do but I do want to know what caused package updates dirty (since the old version is uninstalled first before the new version is installed)...

  11. leofang
    added
    wontfixThis will not be worked on
    packagingAnything related to wheels or Conda packages
    cuda.bindingsEverything related to the cuda.bindings module
    and removed
    triageNeeds the team's attention
    on May 7, 2025

  12. leofang commented on May 7, 2025

    @leofang
    leofang
    on May 7, 2025
    Member

    @bdice reports offline that this still happens when updating from 12.6.2.post1 to 12.9.0... 😵‍💫

  13. leofang
    pinned this issue on May 12, 2025
  14. leofang
    changed the title [-]BUG: migrating from 12.6 to 12.8 breaks Python import[/-] [+]BUG: updating from 12.6 to 12.8/12.9 breaks Python import[/+] on May 12, 2025

  15. leofang commented on May 12, 2025

    @leofang
    leofang
    on May 12, 2025
    Member

    Marking this issue as a "won't fix." Unfortunately there does not seem to exist any workaround to our knowledge, other than kindly asking users to do a clean installation (uninstall and then install) instead of updating. I'll add this to known issues shortly.

  16. leofang
    self-assigned this
    on May 12, 2025
  17. leofang
    added
    documentationImprovements or additions to documentation
    on May 12, 2025
  18. leofang
    added this to the cuda-python 12-next, 11-next milestone on May 12, 2025
  19. leofang
    closed this as completedin #626on May 12, 2025
  20. leofang
    added this to CCCLon Jun 4, 2025
  21. github-project-automation
    moved this to Todo in CCCLon Jun 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

cuda.bindingsEverything related to the cuda.bindings moduledocumentationImprovements or additions to documentationpackagingAnything related to wheels or Conda packageswontfixThis will not be worked on

Type

No type

Projects

  • CCCL
    Status
    Todo

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions