pip does not detect user-installed packages after upgrading system python?

Question

I use Python on my Fedora KDE (x64) PC. I used to have Fedora 25, and I had installed Python 3.5 using sudo dnf install python3 and several packages like numpy and scipy using commands like sudo dnf install python3-numpy. Now not all the suitable packages or versions were available on fedora repositories, so those alone I installed in ~/.local/lib/python3.5/site-packages/ via pip3 install <packagename> --user.

Fast forward, and I upgraded Fedora to version 28, during which python3 had been upgraded to version 3.6. All the other python3-*packages have also been suitably upgraded. Now pip3 does not recognise any user-installed packages as in pip3 list --user, returning no output at all. But the ~/.local/lib/python3.5/site-packages/ directory still exists and has a bunch of packages.

Now I am not sure what I should do....

• I can maybe delete the contents of site-packages, if I knew exactly what can be deleted? Then I can set about re-investigating what packages really need to be installed via pip now.
• I can maybe rename the ~/.local/lib/python3.5 to ~/.local/lib/python3.6?
• Some other way to make pip import the contents of the old user-installed python packages?

I am kind of lost on how to proceed.

Answer 1

That should be a pretty easy thing to do if you still have Python 3.5 around:

$ pip3.5 list --user --format=freeze | xargs pip3.6 install --user

will reinstall all the user-site packages for Python 3.6. Things get a little bit more complicated when the old Python installation is gone (so no pip3.5 in your case), but still nothing you can't manage. First of all, check that the user site still exists so you have something to work with:

$ find ~/.local -type d -name "site-packages" | grep python3.5
/Users/hoefling/.local/lib/python3.5/site-packages

Now use pkg_resources to list the distributions available in that user site:

$ find ~/.local -type d -name "site-packages" | grep python3.5 | xargs python3.6 -c "import pkg_resources, sys; print(' '.join((f'{pkg.project_name}=={pkg.version}' for pkg in pkg_resources.find_distributions(sys.argv[1]))))"
wheel==0.30.0

The user site dir will be piped to Python code line that will scan the dir using pkg_resources.find_distributions function (see Getting or Creating Distributions section in the docs) which is the same one that pip uses under the hood, IIRC. I just added some formatting so you can then pass the output to the new pip3.6 in the last iteration of extending the one-liner. The full one-liner is:

$ find ~/.local -type d -name "site-packages" | grep python3.5 | xargs python3.6 -c "import pkg_resources, sys; print(' '.join((f'{pkg.project_name}=={pkg.version}' for pkg in pkg_resources.find_distributions(sys.argv[1]))))" | xargs pip3.6 install --user

pkg_resources is part of the setuptools package and should be already preinstalled (at least this is the case when using the installers from https://www.python.org/downloads/). If not, install setuptools first:

$ dnf install python3-setuptools