[BUG] ModuleNotFoundError when importing additional local scripts in setup.py

[kdschlosser]

setuptools version

67.8.0

Python version

3.8, 3.9, 3.10, 3.11

OS

Windows, Linux, OSX

Additional environment information

No response

Description

when installing using pip install . for doing a local install or when installing an sdist from pypi using pip install {library_name} under linux if there are additional imports in setup.py for local modules to setup.py the build fails with ModuleNotFound.

On Windows I am able to work around this problem by explicitly adding the parent folder to setup.py to sys.path. this however does not work for Linux.

Expected behavior

To have the installation work properly

How to Reproduce

I have attached an example package that produces the error. Extract the attached file and go into the folder where the setup.py is and use pip install . you will see the error.

pip_failure.zip

Output

****************\pip_failure>pip install .
Processing ****************\pip_failure
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [18 lines of output]
      Traceback (most recent call last):
        File "\python\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 353, in <module>
          main()
        File "\python\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "\python\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "********\AppData\Local\Temp\pip-build-env-ztw59mba\overlay\Lib\site-packages\setuptools\build_meta.py", line 338, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "********\AppData\Local\Temp\pip-build-env-ztw59mba\overlay\Lib\site-packages\setuptools\build_meta.py", line 320, in _get_build_requires
          self.run_setup()
        File "********\AppData\Local\Temp\pip-build-env-ztw59mba\overlay\Lib\site-packages\setuptools\build_meta.py", line 335, in run_setup
          exec(code, locals())
        File "<string>", line 5, in <module>
      ModuleNotFoundError: No module named 'some_module'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

[abravalheri]

if there are additional imports in setup.py for local modules to setup.py the build fails with ModuleNotFound.

Hi @kdscholosser, after the implementation of PEP 517 in setuptools, users can no longer rely on the side effect of having the current working directory on sys.path.

In the past users/pip would run python setup.py ..., so Python itself (not setuptools) would add "" to sys.path, as it does when you run any script file. Now that pip/build use the PEP 517 API (instead of running the setup.py script directly) this coincidence no longer occurs.

If users want to import local modules they are recommended to explicitly add the current directory to sys.path at the top of setup.py.

On Windows I am able to work around this problem by explicitly adding the parent folder to setup.py to sys.path. this however does not work for Linux.

I cannot replicate the error on Linux after explicitly adding the current directory to sys.path:

> docker run --rm -it python:3.9-bullseye /bin/bash
mkdir /tmp/pip_failure
cd /tmp/pip_failure
wget https://github.com/pypa/setuptools/files/11630515/pip_failure.zip
unzip pip_failure.zip
rm pip_failure.zip
sed -i -e "1i import os, sys; sys.path.append(os.path.dirname(__file__))" setup.py
python3 -m venv .venv
.venv/bin/python -m pip install .
# ...
# Successfully installed pip-test-0.0.1

[kdschlosser]

that's because you are installing it using python -m pip install . and not pip install .

[abravalheri]

Hi @kdschlosser, if I change the example above (which explicitly modify sys.path) to use pip install instead of python -m pip install, I am still able to install the project. Both of the examples below will work on a linux container:

> docker run --rm -it python:3.9-bullseye /bin/bash
mkdir /tmp/pip_failure
cd /tmp/pip_failure
wget https://github.com/pypa/setuptools/files/11630515/pip_failure.zip
unzip pip_failure.zip
rm pip_failure.zip
sed -i -e "1i import os, sys; sys.path.append(os.path.dirname(__file__))" setup.py
python3 -m venv .venv
.venv/bin/pip install .
# ...
# Successfully installed pip-test-0.0.1

> docker run --rm -it python:3.9-bullseye /bin/bash
mkdir /tmp/pip_failure
cd /tmp/pip_failure
wget https://github.com/pypa/setuptools/files/11630515/pip_failure.zip
unzip pip_failure.zip
rm pip_failure.zip
sed -i -e "1i import os, sys; sys.path.append(os.path.dirname(__file__))" setup.py
pip install .
# ...
# Successfully installed pip-test-0.0.1

[kdschlosser]

that's really strange. You may or may not want to install the library I am actually working on. see if it fails on your machine. This is driving me nuts because it faisl 100% of the time on my Windows machine unless I add the path statement and another user reported the same error when installing it on linux.

pip install lvgl

That will do an sdist installation. It will compile is you have gcc installed and also SDL2-dev. if you do not have those it will fail and the failure error will be different which is fine so long as it's not the original error. That library has the path statement added to the setup.py file already. the user that reported the problem to me ran python -m pip install lvgl it didn't fail.

My question is why is the path not added to sys.path in the build backend? why is this something that has to be done in the setup script? From what I have read the plan is to eventually stop using the setup.py file. how is that supposed to work with complex build systems?

[abravalheri]

Hi @kdschlosser, any chance you are not distributing the extra files that you need during build time inside the sdist? (This would explain why the path is augmented, but there is no file to be found).

You can find more information on how to configure that in https://setuptools.pypa.io/en/latest/userguide/miscellaneous.html.

[abravalheri]

The comments below reflect my personal opinions and are not necessarily the official position of setuptools or PyPA.

My question is why is the path not added to sys.path in the build backend? why is this something that has to be done in the setup script?

I was not involved during the implementation of PEP 517, but I understand that was a deliberated choice. If I had to guess I would say that sometimes extra Python files in the working directory might obfuscate modules in the stdlib (or in the build tools) and result in errors (for some edge cases).

From what I have read the plan is to eventually stop using the setup.py file. how is that supposed to work with complex build systems?

I would think that the complete removal of setup.py is something that is still up for debate... Right now running python setup.py as a CLI script is deprecated, however the setup.py file itself is a perfectly valid configuration file.

Sometimes writing Python code for customisation is inevitable... We can encourage users to not do that unless strictly necessary, and we can even add controls to pip and OS package managers to "block" packages that run arbitrary code during installation (for sanitisation purposes). However, I don't think that completely removing the possibility of code-driven customisation is a viable move.

[abravalheri]

Ok, I will go ahead and close the issue for now since I have the impression that the root cause of the problem is the one described in #3939 (comment) + there is no plans on setuptools (at least for now) to add the working directory to path.

This was explicitly decided when migrating to PEP 517, and it does not seem that there was a change in circumstances.