Avoid downloading wheels that do not match Python requirement or when platform constraints are specified

[ralbertazzi]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the FAQ and general documentation and believe that my question is not already covered.

Feature Request

Poetry by design includes in its lockfile every wheel matching the version requirements of a package. This is done regardless of the Python version requirement, and is generally fine when ran against pypi. However this results in a big performance penalty wen ran against a repository that does not expose package hashes (pytorch once again 👀 ), which results in Poetry downloading a lot of unnecessary wheels.

Suppose I specify python = "~3.11" as Python version and add torch==specific-version as a dependency from its public repository https://download.pytorch.org/whl/torch. As the repository does not provide hashes, Poetry will (rightfully) download wheels to compute a hash on them. However, it does it regardless of the Python specification - i.e. not only cp311 wheels are downloaded, but also cp39, cp38, ... This becomes increasingly slow as new Python versions are released.

I'm wondering if Poetry would accept a PR where the inclusion of packages in the lockfile (and thus its download) is pre-filtered by filtering on the matching wheels, or if I'm missing something for which this can/should not be done. While I agree that the PyTorch team should improve its repository, I think it doesn't hurt for Poetry to stay on the safe side and exploit this performance improvement if possible (as from real world experience PyTorch is not the only repository that does not expose hashes unfortunately).

---

Related:

• Only search source for designated platform and python Only search source for designated platform and python #8871

[radoering]

In my opinion that makes sense. We should pay attention to the following details:

• There should not be a difference for the different repository types. If we decide that the lockfile should not contain files that do not match the Python constraint that should also apply to PyPI.
• We have to make sure that files are added when we loosen the Python constraint and lock again (even with poetry lock --no-update).

[david-waterworth]

It's not just python version, it's also downloading every architecture as well - I'm using py310 on linux (my pyproject.toml contains python = "^3.10") - and I'm currently waiting for the download of every wheel for windows, linux, macos for every supported python version of polars between the last version I locked and the latest, i.e. polars/1.8.2/polars-1.8.2-cp38-abi3-macosx_10_12_x86_64.whl which is taking ages (possibly because I'm using AWS Code Artifact with PyPi upstream)

[teamclouday]

Is there any update on this? It's still downloading all python versions for torch on poetry 2.1.2

[steve-mavens]

It's also lock file bloat. I think my colleagues and I would be more likely to eyeball the diffs on committed poetry.lock files if they were a bit more manageable. github doesn't even display them by default, but a small poetry.lock file diff (when they occur) is actually pretty human-readable.

Potentially there are other bits and pieces that could be cleaned out: for example I'm seeing python-versions = "!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,!=3.6.*,>=2.7" (for colorama) in the poetry.lock for a project that's already restricted with python = ">=3.10". Fair enough it's copied from the colorama package metadata, but it's information not relevant to the locked dependencies. Even granted that the lockfile supports multiple platforms and Python versions, it doesn't need to exclude 3.0.* again, and I'd like (as far as possible) to pretend that Python 2 no longer exists.

Like radoering warns, though, I guess there's a risk that some of this information is re-used from the poetry.lock file without re-checking it against the wheels, in the case where the Python or platform restrictions are loosened and the environment re-solved? In which case removing it requires code to re-create it.

[Amit-Tsabary]

In my usecase - i use multiple private repositories.
When I run poetry lock - it locks the wheels that are incompatible with my python version from my primary repository, even though my secondary repository does have a compatible wheel.
Therefore after i run poetry lock, It fails on install.
I believe this feature would be helpful in my situation as well.