How to avoid updating PATH by copying SIP module to appropriate directory

Question

I continue to have very time-consuming problems with import statements in the Python scripts that I attempt to write.

In this particular script, I want to use QGIS functionality, and so it begins:

from qgis.core import *

which causes:

Traceback (most recent call last):
  File "C:\Python script for PUI\v20170621.py", line 8, in <module>
    from qgis.core import *
ImportError: No module named qgis.core

So I try:

import sys
sys.path.append('C:/Program Files (x86)/QGIS 2.18/apps/qgis/python')

from qgis.core import *

which causes:

Traceback (most recent call last):
  File "C:\Python script for PUI\v20170621.py", line 8, in <module>
    from qgis.core import *
  File "C:/Program Files (x86)/QGIS 2.18/apps/qgis/python\qgis\__init__.py", line 26, in <module>
    from qgis.PyQt import QtCore
  File "C:/Program Files (x86)/QGIS 2.18/apps/qgis/python\qgis\PyQt\QtCore.py", line 26, in <module>
    import sip
ImportError: No module named sip

There is a folder named 'sip' in 'C:\Program Files (x86)\QGIS 2.18\apps\python27' and so I try:

import sys
sys.path.append('C:/Program Files (x86)/QGIS 2.18/apps/qgis/python')
sys.path.append('C:/Program Files (x86)/QGIS 2.18/apps/python27')

from qgis.core import *

which again causes the 2nd error above:

Traceback (most recent call last):
...
ImportError: No module named sip

I suspect that I have a very fundamental misunderstanding about how modules/libraries are installed/imported/loaded/etc. Why can I not simply copy the necessary folder(s) (for instance, the folder 'sip' in this case) to some folder in 'C:\Python27\ArcGIS10.1' (which is where the 'python.exe' file is located that IDLE calls when I open and execute the script) so that no update to PATH nor PYTHONPATH is required?

Answer 1

As of QGIS 2.18.9, the instructions in the PyQGIS developer cookbook seem to be incomplete and stale but, from what I've been able to piece together from discussions of similar issues on Stack Exchange and elsewhere and digging around within the QGIS bits, it seems PyQGIS is specific to the OSGeo installation of python 2.7 included with QGIS. In Windows installs of QGIS this comes from OSGeo4W (4W indicating "for Windows") and is not set up like a typical Python install.

In this environment, which seems to be what you're using, sip cannot be imported as a module because there's no sip.py or equivalent included with QGIS allowing Python to see sip as a module. If you're running in some other environment sip needs to be installed using its configure.py script as module installs are more than just copying.

If one wants sip to load successfully when running scripts with QGIS's Python install from outside of QGIS, it appears

1. There's a requirement to run Python from under the OSGeo4W Shell. You'll find a link to it in QGIS's start folder.

2. QGIS performs additional undocumented configuration of the OSGeo4W Shell which has to be replicated. There are many flavours of workarounds posted around the web but what seems to be the minimal requirement is

   SET PATH=%PATH%;%OSGEO4W_ROOT%\apps\qgis\bin
   SET PYTHONPATH=%OSGEO4W_ROOT%\apps\qgis\python;%OSGEO4W_ROOT%\apps\Python27;%OSGEO4W_ROOT%\apps\Python27\Lib
   

A corollary of the above is, if you wish to use an IDE for authoring PyQGIS scripts, the IDE has to be launched from the OSGeo4W Shell after PATH and PYTHONPATH have been adjusted so it picks up the proper set of environment variables. In the case of the Python tools for Visual Studio 2017 I'm seeing VS clears PYTHONPATH, meaning sys.path still needs to be adjusted in VS interactive windows and such.