pytest cannot import module while python can

Question

I am working on a package in Python. I use virtualenv. I set the path to the root of the module in a .pth path in my virtualenv, so that I can import modules of the package while developing the code and do testing (Question 1: is it a good way to do?). This works fine (here is an example, this is the behavior I want):

(VEnvTestRc) zz@zz:~/Desktop/GitFolders/rc$ python
Python 2.7.12 (default, Jul  1 2016, 15:12:24) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from rc import ns
>>> exit()
(VEnvTestRc) zz@zz:~/Desktop/GitFolders/rc$ python tests/test_ns.py 
issued command: echo hello
command output: hello

However, if I try to use PyTest, I get some import error messages:

(VEnvTestRc) zz@zz:~/Desktop/GitFolders/rc$ pytest
=========================================== test session starts ============================================
platform linux2 -- Python 2.7.12, pytest-3.0.5, py-1.4.31, pluggy-0.4.0
rootdir: /home/zz/Desktop/GitFolders/rc, inifile: 
collected 0 items / 1 errors 

================================================== ERRORS ==================================================
________________________________ ERROR collecting tests/test_ns.py ________________________________
ImportError while importing test module '/home/zz/Desktop/GitFolders/rc/tests/test_ns.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/test_ns.py:2: in <module>
    from rc import ns
E   ImportError: cannot import name ns
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
========================================= 1 error in 0.09 seconds ==========================================
(VEnvTestRc) zz@zz:~/Desktop/GitFolders/rc$ which pytest
/home/zz/Desktop/VirtualEnvs/VEnvTestRc/bin/pytest

I am a bit puzzled, it looks like this indicates an import error, but Python does it fine so why is there a problem specifically with PyTest? Any suggestion to the reason / remedy (Question 2)? I googled and stack-overflowed the 'ImportError: cannot import' error for PyTest, but the hits I got were related to missing python path and remedy to this, which does not seem to be the problem here. Any suggestions?

Answer 1

Found the answer:

DO NOT put a __init__.py file in a folder containing TESTS if you plan on using pytest. I had one such file, deleting it solved the problem.

This was actually buried in the comments to the second answer of PATH issue with pytest 'ImportError: No module named YadaYadaYada' so I did not see it, hope it gets more visibility here.

Answer 2

I can't say I understand why this works, but I had the same problem and the tests work fine if I run python -m pytest.

I'm in a virtualenv, with pytest also available globally:

(proj)tom@neon ~/dev/proj$ type -a python
python is /home/tom/.virtualenvs/proj/bin/python
python is /usr/bin/python

(proj)tom@neon ~/dev/proj$ python -V
Python 3.5.2

(proj)tom@neon ~/dev/proj$ type -a pytest
pytest is /home/tom/.virtualenvs/proj/bin/pytest
pytest is /usr/bin/pytest

(proj)tom@neon ~/dev/proj$ pytest --version
This is pytest version 3.5.0, imported from /home/tom/.virtualenvs/proj/lib/python3.5/site-packages/pytest.py

Answer 3

I just solved this by removing the __init__.py in my project root:

.
├── __init__.py <--- removed
├── models
│   ├── __init__.py
│   ├── address.py
│   ├── appointment.py
│   └── client.py
├── requirements.txt
├── setup.cfg
├── tests
│   ├── __init__.py
│   ├── models
│   │   ├── __init__.py
│   │   ├── appointment_test.py
│   │   └── client_test.py
│   └── other_test.py
└── script.py

Answer 4

I had the same problem but for another reason than the ones mentioned:

I had py.test installed globally, while the packages were installed in a virtual environment.

The solution was to install pytest in the virtual environment. (In case your shell hashes executables, as Bash does, use hash -r, or use the full path to py.test)

Answer 5

Had a similar issue and it worked when I added __init__.py file under tests directory.

Answer 6

Simply put an empty conftest.py file in the project root directory, because when pytest discovers a conftest.py, it modifies sys.path so it can import stuff from the conftest module. A general directory structure can be:

Root
├── conftest.py
├── module1
│   ├── __init__.py
│   └── sample.py
└── tests
    └── test_sample.py

Answer 7

As of pytest 7.0, you can now add pythonpath in pytest.ini. No need to add __init__.py or conftest.py in your root directory.

[pytest]
minversion = 7.0
addopts = --cov=src
pythonpath = src
testpaths =
    tests

You can run pytest without any parameters.

https://docs.pytest.org/en/7.0.x/reference/reference.html#confval-pythonpath

Update:

Here's a sample repo on how I setup my projects which uses the above config:

https://github.com/ianpogi5/pytest-path

Answer 8

2024-02-10 PATH AND VISIBILITY MYSTERIES EXPLAINED

I originally answered this in October 2021. I had struggled with this problem for ages. As described in my original answer below NONE of the solutions here worked. Over 2 years I got 24 upvotes (and 2 downvotes) so it seemed like others were having similar intractable difficulties. I had reluctantly concluded that I had to manipulate sys.path to get pytest to be able to "see" the files that I thought it should see.

I was wrong and failed to understand something about paths. And CRUCIALLY this involves the running of a project (as most of my projects are) where the application is run by using this command (in W10):

> python src\core

In other words, from the start, the CWD is not the current (project root) directory, but a subdirectory [project root directory]\src\core (which contains file __main__.py)

This is where the problem arose. If you are not in fact running a file in the root directory of your project you need to be aware that putting the tests directory in your root directory will plunge you into a world of pain and bafflement: sibling .py files will bafflingly not be "found" in your main "run" directory, i.e. where CWD is, i.e. by virtue of your command line.

Instead, in my case, I need the root "tests" directory to be a subdirectory of src\core (relative to the root directory of my project).

As a more general rule: wherever the CWD of your project is as you run it as an application is the place where you must put your "tests" directory.

Hope this helps someone.

-------------- MY ORIGINAL NAIVE, INCORRECT ANSWER OF 2021-10:

ANOTHER SUGGESTION

I explored this question and various others on SO and elsewhere... all the stuff about adding (or removing) an empty __init__.py in and/or conftest.py in various parts of the project directory structure, all the stuff about PYTHONPATH, etc., etc.: NONE of these solutions worked for me, in what is actually a very simple situation, and which shouldn't be causing any grief.

I regard this as a flaw in pytest's current setup. In fact I got a message recently from someone on SO who clearly knew his stuff. He said that pytest is not designed to work with (as per Java/Groovy/Gradle) separate "src" and "test" directory structures, and that test files should be mingled in amongst the application directories and files. This perhaps goes some way to providing an explanation ... however, tests, particularly integration/functional tests, don't always necessarily correspond neatly to particular directories, and I think pytest should give users more choice in this regard.

Structure of my project:

project_root
    src
        core
            __init__.py
            __main__.py
            my_other_file.py 
    tests
        basic_tests
            test_first_tests.py

The import problem posed: very simply, __main__.py has a line import my_other_file. This (not surprisingly) works OK when I just run the app, i.e. run python src/core from the root directory.

But when I run pytest with a test which imports __main__.py I get

ModuleNotFoundError: No module named 'my_other_file'

on the line in __main__.py that tries to import "my_other_file". Note that the problem here, in my case, is that, in the course of a pytest test, one application file fails to find another application file in the same package.

USING PYTHONPATH
After a lot of experimentation, and putting an __init__.py file and a confest.py file in just about every directory I could find (I think the crucial files were __init__.py added to "tests" and "basic_tests", see above directory structure), and then setting PYTHONPATH to as follows

PYTHONPATH=D:\My Documents\software projects\EclipseWorkspace\my_project\src\core

... I found it worked. Imports in the testing files had to be tweaked a bit, the general pattern being from core import app, project, but the test files were able to "see" core, and crucially there was no need to mess around with the import commands in the app files.

HOWEVER... for some reason the tests now run much slower using this method! Compared to my solution below, where the core package can be seen to be loaded just once, my suspicion is that the PYTHONPATH solution probably results in vast amounts of code being reloaded again and again. I can't yet confirm this, but I can't see any other explanation.

THE ALTERNATIVE

As I say, I regard this as a flaw in pytest's setup. I have absolutely no idea what pytest does to establish a common-sense setup for sys.path, but it's obviously getting it wrong. There should be no need to rely on PYTHONPATH, or whatever, and if indeed this is the "official" solution, the documentation on this subject is sorely lacking.

The ONLY working solution, at least on my machines (OS W10 and Linux, as at pytest 7.3.1), is to add the application packages to sys.path. It is undesirable to mess with sys.path, but in this case nothing else at all works for me.

The most obvious place to do this is conftest.py, which gets executed at the start of each pytest run, and which is kept the project root directory. It involves no modification of the app files. Typically (for the above directory/files setup):

this_file_path = pathlib.Path(__file__)
src_core_dir_path_str = str(this_file_path.parent.joinpath('src', 'core'))
sys.path.insert(0, src_core_dir_path_str)

I am currently (2023-08) using pytest v 7.3.1, and have also tried editing pytest.ini, as suggested by ian's 2022 answer, where it is suggested that these configurations in pytest.ini are a new thing to solve the problem since v7 (release 2021-12). In my machines this does not solve the problem: without manually altering sys.path pytest's attempt to import __main__.py in my example fails as before: because import my_other_file raises ModuleNotFoundError: No module named 'my_other_file'.

PS I have now set up a minimal git repo illustrating the problem: download the .zip at https://github.com/Mrodent/pytest_probs/tree/main, and read what I say in README.md. Interested to know if anyone gets different results to what I get.

--

Answer 9

In my case I am working in a container and unfortuantely pytest has the tendency to use python2.7 rather than my python3 interpreter of choice.

In my case this worked:

python3 -m pytest

My folder structure

/
app/
-module1.py
-module2.py
-tests/
--test_module1.py
--test_module2.py
requirements.txt
README.md

Answer 10

This problem will happen if you have a tests.py file and a tests folder with tests/__init__.py.

During the collection pytest finds the folder, but when it tries to import the test files from the folder, tests.py file will cause the import problem.

To fix simply remove the tests.py file and put all your tests inside the tests/ folder.

For your specific case the fix will be precisely:

• Remove the file /home/zz/Desktop/GitFolders/rc/tests.py
• Make sure /home/zz/Desktop/GitFolders/rc/tests/__init__.py is present

Answer 11

I solved my problem by setting the PYTHONPATH in Environment Variables for the specific configuration I'm running my tests with.

While you're viewing the test file on PyCharm:

1. Ctrl + Shift + A
2. Type Edit Configurations
3. Set your PYTHONPATH under Environment > Environment variables.

UPDATE

1. Move into your project's directory root
2. Create a virtual environment
3. Activate your newly created virtual environment
4. Set the variable $PYTHONPATH to the root of your project and export it:

export PYTHONPATH=$(pwd)

5. Do not remove the __init__.py from the tests/ directory or from the src/ directory.

Also note:

• The root of your directory is not a python module so do NOT add an __init__.py
• conftest.py is not necessary in the root of your project.

6. The $PYTHONPATH var will only be available during the current terminal/console session; so you will need to set this each time. (You can follow the steps previous to this update if you are working in pycharm).

Answer 12

I had a similar issue, exact same error, but different cause. I was running the test code just fine, but against an old version of the module. In the previous version of my code one class existed, while the other did not. After updating my code, I should have run the following to install it.

sudo pip install ./ --upgrade

Upon installing the updated module running pytest produced the correct results (because i was using the correct code base).

Answer 13

In my case, the import error occurred because the package is pointing to another package/directory with the same name and its path is one level above the folder I actually wanted. I think this also explains why some people need to remove _ init _.py while others need to add back.

I just put print(the_root_package.__path__) (after import the_root_package) in both python console and pytest scripts to compare the difference

BOTTOM LINE: When you do python, the package you import may be different from the package when you run pytest.

Answer 14

Please check here: https://docs.pytest.org/en/documentation-restructure/background/pythonpath.html

I has an issue with pytest (that was solved using python -m pytest); the error was

FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.9/site-packages/...

I found the problem was missing __init__.py in tests/ and tests/subfolders.

Answer 15

I had placed all my tests in a tests folder and was getting the same error. I solved this by adding an __init__.py in that folder like so:

.
|-- Pipfile
|-- Pipfile.lock
|-- README.md
|-- api
|-- app.py
|-- config.py
|-- migrations
|-- pull_request_template.md
|-- settings.py
`-- tests
    |-- __init__.py <------
    |-- conftest.py
    `-- test_sample.py

Answer 16

Install the packages into Your virtual environment.
Then start a new shell and source Your virtual environment again.

Answer 17

Here's a medium article! describing the problem!

The issue is which pytest you are using and your use of a virtual environment. If you have installed pytest system-wide, in other words, outside of a virtual environment, pytest has a nasty habit of only looking outside your virtual environment for modules! If your project is using a module that is only installed in your virtual environment, and you’re using a system-wide pytest, it won’t find the module, even if you’ve activated the virtual environment.1

Here’s the step-by-step:1

1. Exit any virtual environment
2. Use Pip to uninstall pytest
3. Activate your project’s virtual environment
4. Install pytest inside the virtual environment
5. pytest will now find your virtual-environment-only packages!

Answer 18

Points to be observed in the same order as below:

1. Make sure that every folder and subfolder of your app has a __init__.py file.
2. Make sure that the tests folder(module) is inside the app folder(module).
3. Try pytest command and if that doesn't work try python3 -m pytest .

Answer 19

The answer above not work for me. I just solved it by appending the absolute path of the module which not found to the sys.path at top of the test_xxx.py (your test module), like:

import sys
sys.path.append('path')

Answer 20

If it is related to python code that was originally developed in python 2.7 and now migrated into python 3.x than the problem is probably related to an import issue.

e.g. when importing an object from a file: base that is located in the same directory this will work in python 2.x:

from base import MyClass

in python 3.x you should replace with base full path or .base not doing so will cause the above problem. so try:

from .base import MyClass

Answer 21

I was getting this using VSCode. I have a conda environment. I don't think the VScode python extension could see the updates I was making.

python c:\Users\brig\.vscode\extensions\ms-python.python-2019.9.34911\pythonFiles\testing_tools\run_adapter.py discover pytest -- -s --cache-clear test
Test Discovery failed:

I had to run pip install ./ --upgrade

Answer 22

I was experiencing this issue today and solved it by calling python -m pytest from the root of my project directory.

Calling pytest from the same location still caused issues.

My Project dir is organized as:

api/
 - server/
  - tests/
      - test_routes.py
  - routes/
      - routes.py
 - app.py

The module routes was imported in my test_routes.py as: from server.routes.routes import Routes

Hope that helps!

Answer 23

I disagree with the posts saying that you must remove any __init__.py files. What you must instead do is alter the sys.path.

Run an experiment where you print sys.path when running the code normally. Then print sys.path while running the code via pytest. I think you will find that there is a difference between these two paths, hence why pytest breaks.

To fix this, insert the path from the first experiment at the 0th index of the second.

Let '/usr/exampleUser/Documents/foo' be the first element of print(sys.path) for experiment 1.

Below is code that should fix your issue:

import sys sys.path[0] = '/usr/exampleUser/Documents/foo'

Put this at the top of your file, before your actual import statement.

Source: I was dealing with this myself and the above process solved it.

Answer 24

The TestCase directory must be a Python Package

Answer 25

For anyone who tried everything and still getting error,I have a work around.

In the folder where pytest is installed,go to pytest-env folder.

Open pyvenv.cfg file.

In the file change include-system-site-packages from false to true.

home = /usr/bin
include-system-site-packages = true
version = 3.6.6

Hope it works .Don't forget to up vote.

Answer 26

Another special case:

I had the problem using tox. So my program ran fine, but unittests via tox kept complaining. After installing packages (needed for the program) you need to additionally specify the packages used in the unittests in the tox.ini

[testenv]
deps =
    package1
    package2 
...

Answer 27

Yet another massive win for Python's import system. I think the reason there is no consensus is that what works probably depends on your environment and the tools you are using on top of it.

I'm using this from VS Code, in the test explorer under Windows in a conda environment, Python 3.8.

The setup I have got to work is:

mypkg/
    __init__.py
    app.py
    view.py
tests/
    test_app.py
    test_view.py

Under this setup intellisense works and so does test discovery.

Note that I originally tried the following, as recommended here.

src/
    mypkg/
        __init__.py
        app.py
        view.py
tests/
    test_app.py
    test_view.py

I could find no way of getting this to work from VS Code because the src folder just blew the mind of the import system. I can imagine there is a way of getting this to work from the command line. As a relatively new convert to Python programming it gives me a nostalgic feeling of working with COM, but being slightly less fun.

Answer 28

My 2 cents on this: pytest will fail at chance if you are not using virtual environments. Sometimes it will just work, sometimes not.

Therefore, the solution is:

• remove pytest with pip uninstall
• create your venv
• activate your venv
• pip install your project path in editable mode, so it will be treated by pytest as a module (otherwise, pytest wont find your internal imports). You will need a setup.py file for that
• install your packages, including pytest
• finally, run your tests

The code, using windows PowerShell:

pip uninstall pytest
python.exe -m venv my_env
.\my_env\Scripts\activate
(my_env) pip install -e .
(my_env) pip install pytest pytest-html pandas numpy

Then finally

(my_env) pytest --html="my_testing_report.html"

An example of setup.py, for pip install -e:

import setuptools

setuptools.setup(
    name='my_package',
    version='devel',
    author='erickfis',
    author_email='[email protected]',
    description='My package',
    long_description='My gooood package',
    packages=setuptools.find_packages(),
    classifiers=[
        'Programming Language :: Python :: 3',
        'Operating System :: OS Independent',
    ],
    include_package_data=True
)

Answer 29

I find the answer in there :Click here

If you have other project structure, place the conftest.py in the package root dir (the one that contains packages but is not a package itself, so does not contain an init.py)

Answer 30

Kept everything same and just added a blank test file at the root folder .. Solved it

Here are the findings, this problem really bugged me for a while. My folder structure was

mathapp/
    - server.py  
    - configuration.py 
    - __init__.py 
    - static/ 
       - home.html  
tests/            
    - functional 
       - test_errors.py 
    - unit  
       - test_add.py

and pytest would complain with the ModuleNotFoundError and gives the hint:

make sure your test modules/packages have valid Python names.

I introduced a mock test file at the same level as mathsapp and tests directory. The file contained nothing. Now pytest does not complain.

Result without the file

$ pytest
============================= test session starts =============================
platform win32 -- Python 3.8.2, pytest-5.4.2, py-1.8.1, pluggy-0.13.1
rootdir: C:\mak2006\workspace\0github\python-rest-app-cont
collected 1 item / 1 error

=================================== ERRORS ====================================
_______________ ERROR collecting tests/functional/test_func.py ________________
ImportError while importing test module 'C:\mainak\workspace\0github\python-rest-app-cont\tests\functional\test_func.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests\functional\test_func.py:4: in <module>
    from mathapp.service import sum
E   ModuleNotFoundError: No module named 'mathapp'
=========================== short test summary info ===========================
ERROR tests/functional/test_func.py
!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
============================== 1 error in 0.24s ===============================

Results with the file

$ pytest
============================= test session starts =============================
platform win32 -- Python 3.8.2, pytest-5.4.2, py-1.8.1, pluggy-0.13.1
rootdir: C:\mak2006\workspace\0github\python-rest-app-cont
collected 2 items

tests\functional\test_func.py .                                          [ 50%]
tests\unit\test_unit.py .                                                [100%]

============================== 2 passed in 0.11s ==============================