Repository navigation
Poetry ignoring credentials in source url?Β #6799
Description
Discussed in https://github.com/orgs/python-poetry/discussions/6629
Originally posted by keriksson-rosenqvist September 26, 2022
I am trying to set up a poetry environment which pulls a certain package from a private pypi-like registry. As I am sharing the project with others via a git repository, I want the setup to be contained within the pyproject.toml file, i.e. not rely on configs, environment variables, or keyring settings that may differ between user devices.
I have therefore added a source with the basic http credentials included in the url. The exact same url is able to install the package and it's dependencies through pip install -i https://<username:password>@[REDACTED]/pypi/simple/ my-private-package
[tool.poetry.dependencies]
my-private-package = { version = "^2.0.0", source = "my_private_package"}
...
[[tool.poetry.source]]
name = "my_private_package"
url = "https://<username:password>@[REDACTED]/pypi/simple/"
secondary = true
I've set secondary = true as that lets dependencies be installed from the normal pypi, otherwise poetry also fails to install them.
According to this issue, others have been seemingly been able to get it working with the credentials in the url, however with the setup above raises the following error for me.
$ poetry update
Updating dependencies
Resolving dependencies... (1.7s)
401 Client Error: for url: https://[REDACTED]/pypi/download/my-private-package/2/my_private_package-2.0.0-py3-none-any.whl#sha256=[REDACTED]
Is this a bug, am I setting it up wrong, or has there been a change to the usage since the mentioned issue was able to get it to work?
Activity
kristang commentedon Oct 14, 2022
Try running
poetry update -vvvand post the output (remember to redact anything you don't want public - like your feed address)Does
pip install -i https://[REDACTED]/pypi/simple/my-private-packagefail with a 401?keriksson-rosenqvist-oqc commentedon Oct 17, 2022
@kristang I've made an MWE using a new
hello-worldpackage in my private pipy registry. Runningpoetry update -vvvgives the following trace:Running
poetry run pip install -i https://[REDACTED]/hello_world/pypi/simple hello-worldwithout providing any user credentials also fails with a 401:kristang commentedon Oct 17, 2022
Only thing I find a little odd is the url returned:
Unauthorized for url: https://[REDACTED]/pypi/download/hello-world/0.1/hello_world-0.1.0-py3-none-any.whl#sha256=[REDACTED]I don't recall seeing that "/download/" part when I install from a private source. What service is providing your private feed?
It might just be my inexperience with pypi feeds in general.
keriksson-rosenqvist-oqc commentedon Oct 17, 2022
I am using Azure Artifacts to store my private feed. As mentioned it works fine with
pip install -ion the command line when providing credentials. If I supply the-vvvflag to my pip command it does use that same urlhttps://[REDACTED]/pypi/download/hello-world/0.1/hello_world-0.1.0-py3-none-any.whl#sha256=[REDACTED]kristang commentedon Oct 17, 2022
Have you tried other authentication methods? I believe Azure Artifacts supports some kind of API token - last time I checked they had a guide somewhere included on the site for your feed on how to use it.
keriksson-rosenqvist-oqc commentedon Oct 18, 2022
@kristang That is what's being used. The tokens are used through the basic auth approach. They have no support for other authentications through pip as far as I have found, and I've been in touch with their support team regarding authentication methods.
kristang commentedon Oct 18, 2022
According to the documentation from Microsoft you can auth with
artifacts-keyring.You can then install with:
https://learn.microsoft.com/en-us/azure/devops/artifacts/quickstarts/python-packages?view=azure-devops#use-artifacts-keyring-to-set-up-authentication
This approach worked for me when I was testing Azure Artifacts. There is also support for using a Personal Access Token, which I assume is what you have been trying to do?
keriksson-rosenqvist-oqc commentedon Oct 18, 2022
A keyring would not solve the problem I've defined:
With regards to the API token versus PAT, the documentation shows that to use the
pipinterface you must use a PAT. I have not found any documentation on another form of token that could be used for this.kristang commentedon Oct 19, 2022
Arh, sorry I completely skipped that.
(Off-topic and sidenote: I would be careful about storing your credentials like that in your git)
squirrel532 commentedon Nov 1, 2022
I'm facing similar problem which happened on a self-host GitLab PyPI repo too.
On version
1.2.2,poetry lockfailed, but it works on version1.1.15.keriksson-rosenqvist-oqc commentedon Nov 1, 2022
I'm currently running version 1.2.1, but if this used to work in an older version I'd say it's definitely a bug that it's not working now.
manulen commentedon Nov 10, 2022
same problem here..
@squirrel532 with version 1.1.15
poetry lockworks, but installation still fails for me.It's interesting that the authentication seems to work in the first step. Depending on which version I set in the pyproject.toml (e.g. ^0.4.0, <=0.4.0) it is able to figure out which package it should install including the correct download link. But then in the second step when it comes to downloading the package, the authentication fails.
keriksson-rosenqvist-oqc commentedon Dec 8, 2022
Is there any further update on this? Can it be tagged as a bug please?
squirrel532 commentedon Jan 21, 2023
As a workaround, I configured
pipwith.netrc, which contains the credentials of my private repo.https://pip.pypa.io/en/stable/topics/authentication/#netrc-support
9 remaining items
squirrel532 commentedon Feb 7, 2023
@bfontaine I think it is risky that someone places their credentials in
pyproject.toml. Maybe we just places a warning in document to prohibit this behavior ?CC @sdispater
bfontaine commentedon Feb 7, 2023
I donβt see any technical reason this shouldnβt work. Right now this is a bug: the URL including credentials is perfectly valid but Poetry appears to silently strip the credentials.
Itβs risky for public packages, but there are valid use-cases in a private/company environment. See what @keriksson-rosenqvist wrote in their initial message:
Strice91 commentedon Feb 7, 2023
Hi! I also experience this issue. Thanks to @squirrel532 for this work around. So we can keep our internal projects going.
I think, people should be careful with their credentials and think twice before putting them anywhere. But in our internal projects at work it is quite useful to checkout private repos which are dependent on each other.
So if I understood it correctly the bug is still there but will not be fixed as the issue was closed? π€
keriksson-rosenqvist-oqc commentedon Feb 8, 2023
Since there seems to be more people experiencing this issue, I've reopened it in hopes the bug will be looked at in the future.
clintonroy commentedon Feb 8, 2023
I cannot imagine the project will ever accept credentials in pyproject.toml, poetry has no idea where that pyproject.toml will be published, in public or private. The first project that accidentally publishes credentials in pyproject.toml will end up with a security bug against poetry.
bfontaine commentedon Feb 9, 2023
@clintonroy The question is not if the project accepts credentials because it doesnβt have a choice: URLs that contain credentials are valid HTTP URLs that shouldnβt be dealt with in a different manner than others. The suggested solution is to put the credentials in
poetry.toml, which you can put in git as well. I donβt remember seeing a security bug against SSH because someone put their private key in a public Git repository.fannigurt commentedon Mar 20, 2023
Hi!, thanks a lot for the instructions above!
Just wanted to mention, that we can add credentials without creating additional
poetry.tomllocally.Here is the config command:
So such steps need:
Add credentials as mentioned
pyproject.tomledit like squirrel532 provided above:Use
poetry add you_awesome_packageorpoetry update you_awesome_packageas usual.bfontaine commentedon Mar 20, 2023
@fannigurt yes, but this issue is specifically about adding credentials inside the repository in order to share them with everyone. By default
poetry configedits your user config. To edit the local config, you can usepoetry config --localand get the same result as the workaround mentioned earlier.amp-matthew-wallace commentedon Aug 1, 2023
This issue is still present in Poetry
1.5.1, and I believe that I've gotten to the bottom of it.In my testing with Google Artifact Registry (GAR), I'm seeing that the simple repository base URL is returning
hrefvalues with absolute URLs. In my testing, if relative URLs are returned, there is no issue. I believe that it is likely that everyone experiencing this issue is using a repository which returns absolute URLs.PEP 503 states:
So there's nothing invalid about GAR's response.
However, within poetry, this does not play nicely with the HTTP basic authentication credentials. When poetry processes these responses,
urllib.parse.urljoinis used to combine the source's authenticated URL with the non-authenticated URLs from the HTML above.poetry/src/poetry/repositories/link_sources/html.py
Line 35 in dfd0db8
urljoindrops the credentials when joining the two URLs.Produces:
https://us-east1-python.pkg.dev/path/to/repo/[REDACTED].tar.gz#sha256=feedface7486 is perhaps the best solution here, as it would intercept the credentials into Poetry's authentication system early on before they have a chance to get mangled.
However, there are some other potential risks caused by the same issue. For example, a user could be connecting to a repository via a proxy, and the returned absolute path would override the proxy URL.
Produces:
https://us-east1-python.pkg.dev/path/to/repo/[REDACTED].tar.gz#sha256=feedfacegresavage commentedon Oct 24, 2023
TLDR: check your
~/.netrcfile for conflicting credentials.I think I am having the same/similar issue. Unfortunately the workarounds did not work in my case. I believe in my case the error has to do with my environment (i.e. the installed packages, libraries, etc.). I'm on Ubuntu 22.04 using Poetry 1.6.1. AFAIK I haven't made any changes to poetry credential configuration between the time everything was working (~1 week ago) and now.
I have traced my error to my system because (after a lot of troubleshooting), I found that running poetry commands (
lock,update,install) in a docker container and using a secret mount to bind a localauth.tomlto~/.config/pypoetry/auth.tomlin the container works to authenticate and pull the packages from the private repo. Also, putting the credentials directly into the source URL works but due to the behavior described in #6799 (comment) (as well as the obvious security risks with storing secrets in distributable code) this is not a full workaround.Therefore I know that a) the credentials are valid and b) on a "clean" system the process works. At some point some package or library poetry uses to authenticate must have gotten corrupted in my local system however attempts to hunt this package down have all failed - downgrading poetry, force installing python and system packages like keyring, urllib, python-dbus, libdbus, etc. have all failed to solve the problem.
Additionally disabling the keyring entirely and either falling back to plaintext or environment variable authentication does not work either. Unfortunately, due to the nature of the problem I cannot produce a reproduction script since I have no idea which library on my system has become "corrupted". I'm hoping someone can at least point me in a direction to look.
Any help would be greatly appreciated. In the meantime I am about to run the
poetry lockcommand through a debugger to introspect what exact requests and metadata are being sent/received so hopefully I can isolate whether the issue lies in poetrys ability to retrieve the credentials from the keyring/plaintext/environment or a malformed request itself.UPDATE 2023-10-24
I've been able to rule out poetry not getting the credentials from the keyring - it gets the credentials, encrypts them, and places them into the header. I haven't checked if the encryption is correct yet. The content of the response claims the token has expired but this doesn't make sense as the repository maintainers checked this and confirmed it was valid. Furthermore the issue occurs with my personal username and token too (I have developer access to the repository).
Just reinstalled requests and cryptography too.
Okay - I finally traced the error to conflicting credentials in
~/.netrcvs the keyring. Apparently,poetrywill userequeststo get credentials for the repository (see here) which uses~/.netrccredentials to form the encoded authorization in the header of the request. So any changes to the configuration of the credentials usingpoetry config, environment variables, etc. is essentially discarded for authenticating the actual request in favor of the contents of~/.netrcSO if you come across this issue and are in a situation like mine - where none of the above workarounds fixed anything - check your
.netrcfile for configurations matching the repository URL and either remove them or modify them so poetry uses the correct credentials.ustcxmwu commentedon Jan 5, 2024
It is 2023-10-24, not 2024
waryak commentedon Feb 2, 2024
woow, if you create default username for token, which is
gitlab+deploy-token-{n}, poetry will consider it as a wrong git url. I advice to create simple usernames likeuser. Thanks for an answer, that helped!kristianeschenburg commentedon Jun 11, 2024
I was having this same issue. Turns out I had my Gitlab credentials stored in the .netrc file as well, so @gresavage solution solved the failed authorization issues for me.
emocibob commentedon Jul 26, 2024
I had a similar issue with GitLab. Since the package I wanted to install was in another repo in GitLab, I had to go to that repo's Settings > CI/CD and add my other repo (the one where the pipelines run) under Token Access > Limit access to this project > Groups and projects with access. More info.
After that, I could use Poetry to add my private package with a pipeline job token.