Repository navigation
Description
Hi, and thank you for making pex!
I'm try to use the new --venv-repository option from pex 2.59.
I have some strange issues (race condition I guess), when using --venv-repository, --layout packed and manylinux wheels.
How to reproduce (pex 2.59.2/ python 3.13, but also on 3.11):
skunnyk@dellboxe:/tmp $ rm -rf ~/.cache/pex/
skunnyk@dellboxe:/tmp $ python -m venv myvenv
skunnyk@dellboxe:/tmp $ source myvenv/bin/activate
(myvenv) skunnyk@dellboxe:/tmp $ pip install tokenizers==0.21.0 pex
Collecting tokenizers==0.21.0
Using cached tokenizers-0.21.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.7 kB)
Collecting pex
Using cached pex-2.59.2-py2.py3-none-any.whl.metadata (7.4 kB)
Collecting huggingface-hub<1.0,>=0.16.4 (from tokenizers==0.21.0)
Using cached huggingface_hub-0.35.3-py3-none-any.whl.metadata (14 kB)
Collecting filelock (from huggingface-hub<1.0,>=0.16.4->tokenizers==0.21.0)
Using cached filelock-3.19.1-py3-none-any.whl.metadata (2.1 kB)
Collecting fsspec>=2023.5.0 (from huggingface-hub<1.0,>=0.16.4->tokenizers==0.21.0)
Using cached fsspec-2025.9.0-py3-none-any.whl.metadata (10 kB)
Collecting packaging>=20.9 (from huggingface-hub<1.0,>=0.16.4->tokenizers==0.21.0)
Using cached packaging-25.0-py3-none-any.whl.metadata (3.3 kB)
Collecting pyyaml>=5.1 (from huggingface-hub<1.0,>=0.16.4->tokenizers==0.21.0)
Using cached pyyaml-6.0.3-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (2.4 kB)
Collecting requests (from huggingface-hub<1.0,>=0.16.4->tokenizers==0.21.0)
Using cached requests-2.32.5-py3-none-any.whl.metadata (4.9 kB)
Collecting tqdm>=4.42.1 (from huggingface-hub<1.0,>=0.16.4->tokenizers==0.21.0)
Using cached tqdm-4.67.1-py3-none-any.whl.metadata (57 kB)
Collecting typing-extensions>=3.7.4.3 (from huggingface-hub<1.0,>=0.16.4->tokenizers==0.21.0)
Using cached typing_extensions-4.15.0-py3-none-any.whl.metadata (3.3 kB)
Collecting hf-xet<2.0.0,>=1.1.3 (from huggingface-hub<1.0,>=0.16.4->tokenizers==0.21.0)
Using cached hf_xet-1.1.10-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.7 kB)
Collecting charset_normalizer<4,>=2 (from requests->huggingface-hub<1.0,>=0.16.4->tokenizers==0.21.0)
Using cached charset_normalizer-3.4.3-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (36 kB)
Collecting idna<4,>=2.5 (from requests->huggingface-hub<1.0,>=0.16.4->tokenizers==0.21.0)
Using cached idna-3.10-py3-none-any.whl.metadata (10 kB)
Collecting urllib3<3,>=1.21.1 (from requests->huggingface-hub<1.0,>=0.16.4->tokenizers==0.21.0)
Using cached urllib3-2.5.0-py3-none-any.whl.metadata (6.5 kB)
Collecting certifi>=2017.4.17 (from requests->huggingface-hub<1.0,>=0.16.4->tokenizers==0.21.0)
Using cached certifi-2025.10.5-py3-none-any.whl.metadata (2.5 kB)
Using cached tokenizers-0.21.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.0 MB)
Using cached huggingface_hub-0.35.3-py3-none-any.whl (564 kB)
Using cached hf_xet-1.1.10-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.2 MB)
Using cached pex-2.59.2-py2.py3-none-any.whl (3.9 MB)
Using cached fsspec-2025.9.0-py3-none-any.whl (199 kB)
Using cached packaging-25.0-py3-none-any.whl (66 kB)
Using cached pyyaml-6.0.3-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (801 kB)
Using cached tqdm-4.67.1-py3-none-any.whl (78 kB)
Using cached typing_extensions-4.15.0-py3-none-any.whl (44 kB)
Using cached filelock-3.19.1-py3-none-any.whl (15 kB)
Using cached requests-2.32.5-py3-none-any.whl (64 kB)
Using cached charset_normalizer-3.4.3-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (151 kB)
Using cached idna-3.10-py3-none-any.whl (70 kB)
Using cached urllib3-2.5.0-py3-none-any.whl (129 kB)
Using cached certifi-2025.10.5-py3-none-any.whl (163 kB)
Installing collected packages: urllib3, typing-extensions, tqdm, pyyaml, pex, packaging, idna, hf-xet, fsspec, filelock, charset_normalizer, certifi, requests, huggingface-hub, tokenizers
Successfully installed certifi-2025.10.5 charset_normalizer-3.4.3 filelock-3.19.1 fsspec-2025.9.0 hf-xet-1.1.10 huggingface-hub-0.35.3 idna-3.10 packaging-25.0 pex-2.59.2 pyyaml-6.0.3 requests-2.32.5 tokenizers-0.21.0 tqdm-4.67.1 typing-extensions-4.15.0 urllib3-2.5.0
(myvenv) skunnyk@dellboxe:/tmp $ pex --venv-repository myvenv/ --layout packed -o mypex
Here we are ok, I have a packed PEX in mypex.
If I rerun the command, errros happens:
(myvenv) skunnyk@dellboxe:/tmp $ pex --venv-repository myvenv/ --layout packed -o mypex
[Errno 2] No such file or directory: '/home/skunnyk/.cache/pex/packed_wheels/0/14d5b2d393a9562265232fce7979b90f65992e51a5e77a697e9412209ba74d4e/hf-xet-1.1.10-cp37-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl' -> 'mypex~/.deps/hf-xet-1.1.10-cp37-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl'
(myvenv) skunnyk@dellboxe:/tmp $ pex --venv-repository myvenv/ --layout packed -o mypex
Previous binary unexpectedly exists, cleaning: mypex
[Errno 2] No such file or directory: '/home/skunnyk/.cache/pex/packed_wheels/0/14d5b2d393a9562265232fce7979b90f65992e51a5e77a697e9412209ba74d4e/hf-xet-1.1.10-cp37-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl' -> 'mypex~/.deps/hf-xet-1.1.10-cp37-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl'
(myvenv) skunnyk@dellboxe:/tmp $ pex --venv-repository myvenv/ --layout packed -o mypex
Previous binary unexpectedly exists, cleaning: mypex
After 4 or 5 attempts, it⦠works.
The strange thing is that it tries to lookup hf-xet-1.1.10-cp37-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl, while hf-xet-1.1.10-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl is present in the cache! (please note the switched manylinux2014 vs manylinux_2_17 in the file name !)
(myvenv) skunnyk@dellboxe:/tmp $ ls -l /home/skunnyk/.cache/pex/packed_wheels/0/14d5b2d393a9562265232fce7979b90f65992e51a5e77a697e9412209ba74d4e/
total 3120
-rw-r--r-- 1 skunnyk skunnyk 3187271 Oct 7 16:23 hf-xet-1.1.10-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
I isolated the problem to have a simple reproducer, but it seems that it happens when using manylinux_2_17_x86_64.manylinux2014 wheels (with the 2 manylinux names in wheel name).
Thank you,
Activity
jsirois commentedon Oct 7, 2025
Thanks @Skunnyk - I repro. This does in fact have something to do with wheels with compressed tag sets as you noticed. Somehow the order of the tags in the compressed set is unstable in some cases.
I haven't sussed the issue yet, but should be able to get a fix released today - repros are gold; thanks again.
I am curious why you used
--layout packed. That's fairly niche and I'm always curious to know what People are doing with Pex. Can you share why you use that option and how you stumbled upon it being beneficial for your use case?jsirois commentedon Oct 7, 2025
Aha, it looks like maturin (of a certain age at least) produces bogus WHEEL metadata: https://packaging.python.org/en/latest/specifications/binary-distribution-format/#file-contents
That should be like so with the compressed tag expanded:
I'm not sure this is a cause, but note to self anyhow.
jsirois commentedon Oct 7, 2025
And lo and behold:
Same issue, same build backend (though newer maturin version this time).
Skunnyk commentedon Oct 7, 2025
Hi,
Thank you! π©΅
We are using
--layout packedbecause the generated PEXes are >4GB on some of our apps (you know, pytorch, cuda and ML stuffs⦠).jsirois commentedon Oct 7, 2025
Ok - makes sense. Are they actually >4GB and not just >2GB,<4GB? There are a few things to note here:
3.143.13 (See: https://docs.python.org/3/library/zipimport.html namelyChanged in version 3.13: ZIP64 is supported). Pex has tests for this:pex/tests/test_pex_builder.py
Lines 499 to 595 in aa641c0
Skunnyk commentedon Oct 7, 2025
Yai, 1GB,2GB, 4GB+, we are using
--layout packedeverywhere to have consistent build options.Noted for python 3.13, will try it when time comes! (we are still on python 3.11)
jsirois commentedon Oct 7, 2025
Ok, yeah - looking deeper: The only place Pex uses WHEEL Tag metadata is to reconstruct the wheel file name from an an installed wheel in a venv (the case here of course!). It uses
packaging.tags.parse_tagto get this metadata:pex/pex/wheel.py
Lines 85 to 92 in 64c869c
That code, in turn does the
.splitting (compressed tag expansion) and collects the tags in afrozenset. As a result, we see the flaky behavior here.Still looking at a workaround...
After finding one, I'll get an issue filed with maturin. I checked and their code still emits bad metadata like this; so this is an ongoing problem.
Stabilize non-conformant WHEEL Tag metadata.
jsirois commentedon Oct 7, 2025
@Skunnyk a fix has been accepted by maturin in PyO3/maturin#2762 that will go out with their 1.9.7 release but the Pex 2.59.3 release is also out and has a workaround which does not rely on the maturin fix.
Using Pex 2.59.3 your original repro case is now fixed:
Stabilize non-conformant WHEEL Tag metadata. (pex-tool#2927)
Skunnyk commentedon Oct 8, 2025
Hey!
Thank you, seems fine on my side!
I now have another issue around
safe_symlink(rel_src, dst)frompex/common.pyininstalled_wheelscache folder with aFileExistsError:, but I don't yet understand what is happening nor have a simple reproducer (that happens when building multiple pex from different venv, with the same pex cache folder). I'll open another issue when I have more information.