Repository navigation
What does it actually mean to install a git dependency in editable mode?Β #6390
Description
Hi, I see in the documentation it's possible to install a git dependency in editable mode:
poetry add --editable git+https://github.com/some-repository
But I'm having trouble understanding what this actually means. How do I actually .. you know, edit it? Where does it go? I don't see any repo created in my project directory when I do this, although I am able to import it. What is the expected workflow, what am I meant to do with it?
Reacted by Tommaso Comparin, bersbersbers, Orhun ParmaksΔ±z, yoavweber and Georg Grab
Activity
neersighted commentedon Sep 3, 2022
This would probably be a good FAQ item, or maybe something to add around the dependency management section.
To answer your question, this is a bit of a 'completeness' feature for parity with pip. I wouldn't consider it super useful for most Poetry users, but it was requested and it was easy to implement, plus some external tools might want to make use of it.
When you install a VCS url in editable mode, it hybridizes the install modes. If I fully explain it, this will probably make sense:
$VENV_ROOT/src/<name>where name is the#egg=name (often determined fromsetup.py egg_infoor a PEP 517 build).site-packages), a.pthmarker is added. If you inspect$VENV_ROOT/lib/python3.x/site-packagesyou'll see a.pthentry for your 'root package' (aka your Poetry-managed project).So when you install a VCS dependency in editable mode, the repo is cloned into the
$VENV_ROOT/srcfolder, and a.pthmarker is created to use that VCS repo as the source of imported Python code (as opposed to the usual process of a build-time snapshot and copy of the Python files).I hope that helps -- a MR cleaning up my explanation and adding it to the docs would be very welcome!
ckp95 commentedon Sep 3, 2022
Ah okay that's actually pretty neat, I could make use of that. I sometimes have to simultaneously develop on a "upstream" and "downstream" project at the same time, where the downstream project depends on the upstream project's git repo. So being able to edit the upstream and see the changes straightaway in my local dev version of downstream is nice. I don't have to constantly push and pull the changes.
It is a little awkward to have the repo inside the virtual environment though ... I just tried symlinking it into the root of the project folder for convenience and it seems to work, although it feels a little iffy. Would there be any problems caused by doing this? i.e.
It seems to import the same way.
neersighted commentedon Sep 3, 2022
So by spec, VCS deps have to be present at
$VENV_ROOT/src/<name>... But there are no other assumptions made around them, as far as I am aware.That being said, be cautious of leaning on this too much, as I don't think it was ever intended for longer-term use --
poetry installshould respect your repo, butpoetry updatemay check out files from a different ref in my experience. Basically I would commit often and make use of the reflog as I'm not 100% sure the behavior around updating is fully defined for editable VCS deps.ckp95 commentedon Sep 3, 2022
Yes I just tested it,
poetry updatewill clobber any local commits made, without telling you. It snaps the repo back to whatever the remote origin is, although new files added to the working directory persist but in uncommitted state ... weird.Am I right in thinking a path dependency would be safer?
neersighted commentedon Sep 3, 2022
I think technically what happens is
poetry updatewillgit checkout <ref tracked in lockfile> <paths to packages declared in pyproject toml>instead of changing the checked out ref entirely when the ref is mismatched -- I don't remember why this is, but I believe it's to prevent breaking your project completely by switching to a branch with a different layout or similar.An editable path dependency is the classical solution, yes (putting the dependency back when you are done). One day we might grow support for local replacements or similar, but for now there's not a great way to do that.
Okay, the 1.2 betas did allow for one to have multiple copies of a dep, in different groups, with different source URLs. However that was actually the result of a bug that caused some other problems. With the current architecture, two deps that are identical except in terms of source URL are considered the same and will collide/cause your project to become uninstallable.
ckp95 commentedon Sep 3, 2022
Okay, that makes sense. Oh one other thing, maybe this is a bug: in
pyproject.tomlI changed my dependency from the editable git repo to a editable path, and ranpoetry update. However the.venv/src/upstream-projectdirectory persisted instead of being deleted. Is there a way to make it scrub those files so they don't cause confusion? i.e. if I had a symlink pointing to that repo, I would want that symlink to break, rather than allow me to make useless edits via it by mistake.neersighted commentedon Sep 3, 2022
I think we don't currently clean up as we use pip to manage installations, and pip will just install the new package when you switch back (deleting/creating
.pth, copying source files intosite-packagesand writing the.dist-infoas necessary).It is possible #6205 could change this as then the installer would become part of Poetry, and aware of more than the immediate task (install
xyzpackage). That being said, it does look likepoetry install --syncgets confused by the presence of an orphan Git repo in the virtual environment -- it's possible that that command could be updated to identify this situation and nuke the repo (it's stated purpose is to 'make the venv exactly match the lockfile, nuking anything that doesn't have express permission to be there').lainisourgod commentedon Nov 21, 2023
My current setup is that I have a
commonpackage that is used by-
service_a,service_b,service_cThese four are different repos, but
commoncan't be used outside of one of these services. So I usually developcommonlogic only in conjunction withservice_xlogic.In this way, I need
commonpackage to be both editable install and a git dependency, so I can both interactively see changes incommonfromservice_xwithout need to do constant push-pulls, but also able to commit the changes incommonas I make them.Don't really know how to do this in current poetry workflow.
bersbersbers commentedon Nov 29, 2023
@lainisourgod I was thinking about making
commona Git submodule in allservice_*, and then doing an editable install from that submodule folder. So there is an implicit git dependency if you will. (It would mean you have three working copies ofcommon, though.) I haven't tried this yet as I would have to change topoetryfor that (but I would) - have you tried this by any chance?