\(\newcommand{L}[1]{\| #1 \|}\newcommand{VL}[1]{\L{ \vec{#1} }}\newcommand{R}[1]{\operatorname{Re}\,(#1)}\newcommand{I}[1]{\operatorname{Im}\, (#1)}\)

Where does Python look for modules?

See:

Let’s say we have written a Python module and saved it as a_module.py, in a directory called code.

We also have a script called a_script.py in a directory called scripts.

We want to be able to import the code in a_module.py to use in a_script.py. So, we want to be able to put his line in a_script.py:

import a_module

The module and script might look like this:

Contents of code/a_module.py
def func():
    print("Running useful function")
Contents of scripts/a_script.py
import a_module

a_module.func()

At the moment, a_script.py will fail with:

$ python3 scripts/a_script.py
Traceback (most recent call last):
  File "scripts/a_script.py", line 1, in <module>
    import a_module
ModuleNotFoundError: No module named 'a_module'

When Python hits the line import a_module, it tries to find a package or a module called a_module. A package is a directory containing modules, but we will only consider modules for now. A module is a file with a matching extension, such as .py. So, Python is looking for a file a_module.py, and not finding it.

You will see the same effect at the interactive Python console, or in IPython:


>>> import a_module
Traceback (most recent call last):
  File "<input>", line 1, in <module>
ModuleNotFoundError: No module named 'a_module'

Python looks for modules in “sys.path”

Python has a simple algorithm for finding a module with a given name, such as a_module. It looks for a file called a_module.py in the directories listed in the variable sys.path.


>>> import sys
>>> type(sys.path)
<class 'list'>
>>> for path in sys.path:
...     print(path)
... 
/Users/brettmz-admin/dev_trees/psych-214-fall-2016/sphinxext
/usr/local/Cellar/python/3.7.2_1/Frameworks/Python.framework/Versions/3.7/lib/python37.zip
/usr/local/Cellar/python/3.7.2_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7
/usr/local/Cellar/python/3.7.2_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/lib-dynload
/Users/brettmz-admin/Library/Python/3.7/lib/python/site-packages
/Users/brettmz-admin/dev_trees/grin
/Users/brettmz-admin/dev_trees/rmdex
/usr/local/lib/python3.7/site-packages

The a_module.py file is in the code directory, and this directory is not in the sys.path list.

Because sys.path is just a Python list, like any other, we can make the import work by appending the code directory to the list.


>>> import sys
>>> sys.path.append('code')
>>> # Now the import will work
>>> import a_module

There are various ways of making sure a directory is always on the Python sys.path list when you run Python, including:

As a crude hack, you can also put your code directory on the Python sys.path at the top of the files that need it:

Contents of scripts/a_script_with_hack.py
import sys
sys.path.append('code')

import a_module

a_module.func()

Then:

$ python3 scripts/a_script_with_hack.py
Running useful function

The simple append above will only work when running the script from a directory containing the code subdirectory. For example:

$ mkdir another_dir
$ cd another_dir
$ python3 ../scripts/a_script_with_hack.py
Traceback (most recent call last):
  File "../scripts/a_script_with_hack.py", line 4, in <module>
    import a_module
ModuleNotFoundError: No module named 'a_module'

This is because the directory code that we specified is a relative path, and therefore Python looks for the code directory in the current working directory.

To make the hack work when running the code from any directory, you could use some path manipulation on the The “__file__” variable:

Contents of scripts/a_script_with_better_hack.py
from os.path import dirname, abspath, join
import sys

# Find code directory relative to our directory
THIS_DIR = dirname(__file__)
CODE_DIR = abspath(join(THIS_DIR, '..', 'code'))
sys.path.append(CODE_DIR)

import a_module

a_module.func()

Now the module import does work from another_dir:

$ python3 ../scripts/a_script_with_better_hack.py
Running useful function