Python pkgutil.walk_packages not returning subpackages

Question

I have a package layout:

scenarios/
    __init__.py
    X/
        __init__.py
        Y/
            __init__.py
    Z/
        __init__.py 

I have executed

import scenarios
pkgutil.walk_packages(scenarios.__path__, scenarios.__name__ + '.')

But this generates a list only including the packages X and Z, Y is missing. What can I use to get all the sub directories?

Thanks

Answer 1

Here is a theory: The walk_packages function attempts to import each module listed. When it gets to the sub-package "Y" it attempts to import it, but there is an error. By default, this error is suppressed. A sideeffect is that the walk_packages function doesn't recurse into Y. You can test this theory using the "onerror" keyword argument. For example:

import sys, pkgutil
from traceback import print_tb

def onerror(name):
    print("Error importing module %s" % name)
    type, value, traceback = sys.exc_info()
    print_tb(traceback)

import scenarios
pkgutil.walk_packages(scenarios.__path__, scenarios.__name__ + '.', onerror=onerror)

Answer 2

This works in Python 3.6 at least, for what it's worth. Set up the test scenario:

mkdir -p scenarios/X/Y scenarios/Z
find scenarios -type d -exec touch {}/__init__.py \;

Now run a version of your example code:

import pkgutil
import scenarios
for module_info in pkgutil.walk_packages(scenarios.__path__, scenarios.__name__ + '.'):
    print(module_info.name)

Which prints:

scenarios.X
scenarios.X.Y
scenarios.Z