Disable Python requests SSL validation for an imported module

Question

I'm running a Python script that uses the requests package for making web requests. However, the web requests go through a proxy with a self-signed cert. As such, requests raise the following Exception:

requests.exceptions.SSLError: ("bad handshake: Error([('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')],)",)

I know that SSL validation can be disabled in my own code by passing verify=False, e.g.: requests.get("https://www.google.com", verify=False). I also know that if I had the certificate bundle, I could set the REQUESTS_CA_BUNDLE or CURL_CA_BUNDLE environment variables to point to those files. However, I do not have the certificate bundle available.

How can I disable SSL validation for external modules without editing their code?

Answer 1

Note: This solution is a complete hack. And works only for requests < 2.28 (https://github.com/psf/requests/pull/6074)

Short answer: Set the CURL_CA_BUNDLE environment variable to an empty string.

Before:

$ python
import requests
requests.get('http://www.google.com')
<Response [200]>

requests.get('https://www.google.com')
...
File "/usr/local/lib/python2.7/site-packages/requests-2.17.3-py2.7.egg/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: ("bad handshake: Error([('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')],)",)

After:

$ CURL_CA_BUNDLE="" python
import requests
requests.get('http://www.google.com')
<Response [200]>

requests.get('https://www.google.com')
/usr/local/lib/python2.7/site-packages/urllib3-1.21.1-py2.7.egg/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning)
<Response [200]>

How it works

This solution works because Python requests overwrites the default value for verify from the environment variables CURL_CA_BUNDLE and REQUESTS_CA_BUNDLE, as can be seen here:

if verify is True or verify is None:
verify = (os.environ.get('REQUESTS_CA_BUNDLE') or
    os.environ.get('CURL_CA_BUNDLE'))

The environment variables are meant to specify the path to the certificate file or CA_BUNDLE and are copied into verify. However, by setting CURL_CA_BUNDLE to an empty string, the empty string is copied into verify and in Python, an empty string evaluates to False.

Note that this hack only works with the CURL_CA_BUNDLE environment variable - it does not work with the REQUESTS_CA_BUNDLE. This is because verify is set with the following statement:

verify = (os.environ.get('REQUESTS_CA_BUNDLE') or os.environ.get('CURL_CA_BUNDLE'))

It only works with CURL_CA_BUNDLE because '' or None is not the same as None or '', as can be seen below:

print repr(None or "")
# Prints: ''
print repr("" or None )
# Prints: None

Answer 2

I saw this hack only due to some trouble with my private CA.

The given hack with CURL_CA_BUNDLE='' will not longer work in 2022 with next minor release of requests (that means 2.28 ?).

Please refer to GH issue 6071 which was fixed 6 days before in Feb. 2022

Answer 3

My problem was that a third party package sent the request, so that I had no chance to influence the verify = True.

What helped me:

pip install pip-system-certs

And then:

import pip_system_certs #import as first package