Poetry 1.4.0 regression: local dependencies aren't discovered when running `mypy`

[RyanGlScott]

Other info

• Poetry version: Poetry 1.4.0 or later

• Python version: Any Python version (I have tested with 3.8, 3.9, 3.10, and 3.11)

• OS version and name: Ubuntu 20.04

• pyproject.toml: The relevant pyproject.toml file in the minimal repro is here:

  [tool.poetry]
  name = "bar"
  version = "0.1.0"
  description = "A library that depends on `foo`"
  authors = ["Ryan Scott"]
  license = "BSD License"
  include = [
      "LICENSE",
  ]
  
  [tool.poetry.dependencies]
  python = "^3.8"
  foo = { path = "../foo/", develop = true }
  
  [tool.poetry.dev-dependencies]
  mypy = "^0.991"
  
  [build-system]
  requires = ["poetry>=1.1.4", "setuptools>=40.8.0"]

  Note the foo = { path = "../foo/", develop = true } line, which is vital to triggering the bug.

• I am on the latest stable Poetry version, installed using a recommended method.
• I have searched the issues of this repo and believe that this is not a duplicate.
• I have consulted the FAQ and blog for any relevant entries or release notes.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

Running mypy with Poetry 1.4.0 or later in a project with a local dependency unexpectedly fails due to a Cannot find implementation or library stub for module error involving the local dependency. This is a regression, as this did not occur with Poetry 1.3.2 or earlier.

I have prepared a minimal reproducer at the https://github.com/RyanGlScott/poetry-bug repository. To reproduce the bug, perform the following steps:

$ git clone https://github.com/RyanGlScott/poetry-bug
$ cd poetry-bug/bar

# First, try Poetry 1.3.2, which is known to work
$ poetry self update 1.3.2
$ rm -rf ~/.cache/pypoetry/ .mypy_cache && poetry update && poetry install
$ poetry run mypy --install-types --non-interactive bar/
Success: no issues found in 1 source file

$ # Next, try Poetry 1.4.2, which fails
$ poetry self update 1.4.2
$ rm -rf ~/.cache/pypoetry/ .mypy_cache && poetry update && poetry install
$ poetry run mypy --install-types --non-interactive bar/
bar/__init__.py:1: error: Cannot find implementation or library stub for module named "foo"  [import]
bar/__init__.py:1: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
Found 1 error in 1 file (checked 1 source file)

[dimbleby]

what makes you think poetry is doing sonething wrong here? Suggest raising an issue with mypy

[RyanGlScott]

@dimbleby, I'm not sure if I understand what you mean. I am using the same version of mypy (0.991) in both invocations in the instructions above. The only thing that has changed is the Poetry version. Is it expected that the behavior of mypy would change depending on the Poetry version?

[dimbleby]

The qustion isn't whether mypy behaviour would change, the question is whether mypy behaviour is correct. What makes you think that mypy is correct and poetry is at fault?

[RyanGlScott]

It's certainly possible that mypy is doing something wrong. However, in order to submit a proper bug report to mypy, I need to know how poetry is invoking mypy under the hood. (Without this information, I worry that I'll submit a mypy issue only for them to tell me "raise an issue upstream with poetry".)

Some information that would be helpful to know:

1. What is the mypy subprocess that gets invoked when poetry run mypy --install-types --non-interactive bar/ is run?
2. Does poetry do anything with environment variables (e.g., MYPYPATH) before invoking the subprocess?

I tried to figure out this information myself by running poetry run -vvv mypy ..., but the extra information it printed out didn't reveal much:

$ poetry -vvv run mypy --install-types --non-interactive bar/
Loading configuration file /home/rscott/.config/pypoetry/config.toml
Using virtualenv: /home/rscott/.cache/pypoetry/virtualenvs/bar-qoePKlzJ-py3.8
bar/__init__.py:1: error: Cannot find implementation or library stub for module named "foo"  [import]
bar/__init__.py:1: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
Found 1 error in 1 file (checked 1 source file)

I looked around in /home/rscott/.config/pypoetry/config.toml and the virtualenv it mentioned, but couldn't find anything that answered my questions.

[Secrus]

To verify whether this is a poetry or mypy issue, activate venv manually (poetry env info -p will give you a path of your current venv) and run mypy in venv. If it works correctly when run manually, then it's Poetry's fault.

[radoering]

As it seems it's caused by a change in poetry (modern installer) but it's probably not a poetry bug but a limitation of mypy. Unfortunately, it seems mypy is not able do much about it. Related issues: pypa/setuptools#3518, python/mypy#13392

Poetry 1.3 (as well as Poetry 1.4 with modern-installation set to false) creates a static foo.pth, which seems to be supported by mypy.

Poetry 1.4 (with modern-installation set to true) uses a more advanced mechanism (creates an __editable__.foo-0.0.0.pth and an __editable___foo_0_0_0_finder.py), which is not supported by mypy.

I don't know yet what exactly caused this change. It might be related to build or installer, which are used for modern-installation instead of pip.

[RyanGlScott]

Thanks for looking into this, @radoering! Also, thanks for teaching me about installer.modern-installation. Indeed, if I run the following command before poetry install:

$ poetry config installer.modern-installation false

Then installing and running mypy will behave as before. As you mention above, this is more of a workaround than a solution to the underlying problem, but this is enough to get me unstuck with Poetry 1.4+ for now.

[radoering]

@RyanGlScott Actually, it's not just related to pypa/setuptools#3518, it's exactly this issue!

You forgot to specify a build-backend for foo in your example:

[build-system]
requires = ["poetry>=1.1.4", "setuptools>=40.8.0"]

Without a build-backend, it will fallback to setuptools. If you want to use poetry as build-backend, you should add the following line:

build-backend = "poetry.core.masonry.api"

Further, you probably do not need poetry in requires for building but only poetry-core.

If you use another build-backend than setuptools, you may not encounter this issue.

[RyanGlScott]

You are quite right, @radoering. After switching to poetry.core.masonry.api and poetry-core, I am able to run the example above with Poetry 1.4.2 without issues. Thanks for the tip!

I'm not sure if there is anything left to be done on the Poetry side (especially since the main problem is in pypa/setuptools#3518), so feel free to close this issue if you wish.