How to get Python requests to trust a self signed SSL certificate?

Question

import requests
data = {'foo':'bar'}
url = 'https://foo.com/bar'
r = requests.post(url, data=data)

If the URL uses a self signed certificate, this fails with

requests.exceptions.SSLError: [Errno 1] _ssl.c:507: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

I know that I can pass False to the verify parameter, like this:

r = requests.post(url, data=data, verify=False)

However, what I would like to do is point requests to a copy of the public key on disk and tell it to trust that certificate.

Answer 1

try:

r = requests.post(url, data=data, verify='/path/to/public_key.pem')

Answer 2

The easiest is to export the variable REQUESTS_CA_BUNDLE that points to your private certificate authority, or a specific certificate bundle. On the command line you can do that as follows:

export REQUESTS_CA_BUNDLE=/path/to/your/certificate.pem
python script.py

If you have your certificate authority and you don't want to type the export each time you can add the REQUESTS_CA_BUNDLE to your ~/.bash_profile as follows:

echo "export REQUESTS_CA_BUNDLE=/path/to/your/certificate.pem" >> ~/.bash_profile ; source ~/.bash_profile

Answer 3

With the verify parameter you can provide a custom certificate authority bundle

requests.get(url, verify=path_to_bundle_file)

From the docs:

You can pass verify the path to a CA_BUNDLE file with certificates of trusted CAs. This list of trusted CAs can also be specified through the REQUESTS_CA_BUNDLE environment variable.

Answer 4

All of the answers to this question point to the same path: get the PEM file, but they don't tell you how to get it from the website itself.

Getting the PEM file from the website itself is a valid option if you trust the site, such as on an internal corporate server. If you trust the site, why should you do this? You should do this because it helps protect yourself and others from inadvertently re-using your code on a site that isn't safe.

Here is how you can get the PEM file.

1. Click on the lock next to the url.

2. Navigate to where you can see the certificates and open the certificates.

3. Download the PEM CERT chain.

4. Put the .PEM file somewhere you script can access it and try verify=r"path\to\pem_chain.pem" within your requests call.

r = requests.get(url, verify='\path\to\public_key.pem')

Answer 5

Case where multiple certificates are needed was solved as follows: Concatenate the multiple root pem files, myCert-A-Root.pem and myCert-B-Root.pem, to a file. Then set the requests REQUESTS_CA_BUNDLE var to that file in my ./.bash_profile.

$ cp myCert-A-Root.pem ca_roots.pem
$ cat myCert-B-Root.pem >> ca_roots.pem
$ echo "export REQUESTS_CA_BUNDLE=~/PATH_TO/CA_CHAIN/ca_roots.pem" >> ~/.bash_profile ; source ~/.bash_profile

Answer 6

Setting export SSL_CERT_FILE=/path/file.crt should do the job.

Answer 7

If you're behind a corporate network firewall like I was, ask your network admin where your corporate certificates are, then:

import os
os.environ["REQUESTS_CA_BUNDLE"] = 'path/to/corporate/cert.pem'
os.environ["SSL_CERT_FILE"] = 'path/to/corporate/cert.pem'

This fixed issues I had with requests and openssl.

Answer 8

In a dev environment, using Poetry as virtual env provider on a Mac with Python 3.8 I used this answer https://stackoverflow.com/a/42982144/15484549 as base and appended the content of my self-signed root certificate to the certifi cacert.pem file.

The steps in detail:

cd project_folder
poetry add requests
# or if you use something else, make sure certifi is among the dependencies
poetry shell
python
>>> import certifi
>>> certifi.where()
/path/to/the/certifi/cacert.pem
>>> exit()
cat /path/to/self-signed-root-cert.pem >> /path/to/the/certifi/cacert.pem
python the_script_you_want_to_run.py

Answer 9

I know it is an old thread. However, I run into this issue recently. My python requests code does not accept the self-signed certificate but curl does. It turns out python requests are very strict on the self-signed certificate. It needs to be a root CA certificate. In other words,

Basic Constraints: CA:TRUE

Key Usage: Digital Signature, Non Repudiation, Key Encipherment, Certificate Sign

Answer 10

Incase anyone happens to land here (like I did) looking to add a CA (in my case Charles Proxy) for httplib2, it looks like you can append it to the cacerts.txt file included with the python package.

For example:

cat ~/Desktop/charles-ssl-proxying-certificate.pem >> /usr/local/google-cloud-sdk/lib/third_party/httplib2/cacerts.txt

The environment variables referenced in other solutions appear to be requests-specific and were not picked up by httplib2 in my testing.

Answer 11

You may try:

settings = s.merge_environment_settings(prepped.url, None, None, None, None)

You can read more here: http://docs.python-requests.org/en/master/user/advanced/

Answer 12

In my case, I was using self-signed certificate generated by mkcert.

While curl works fine with such self-signed certificates, the Python requests module does not.

With curl:

# works fine
curl https://tests.localhost/hello-world --cacert /etc/apache2/ssl/SSLforMyHosts-certificate.pem

With Python requests:

# does not work with the self-signed certificate:
response = requests.post(url=url, verify="/etc/apache2/ssl/SSLforMyHosts-certificate.pem")

# works fine with the rootCA that signed the certificate:
requests.post(url=url, verify="/etc/mkcert/rootCA.pem")

in addition: I'm running a webapp inside a Docker container, but creating the self-signed certificates on the host on macOS. So I had to share /etc/mkcert/ across macOS and container -- simply mounted it with Docker compose.