Python 'requests' package throws SSL Exception when verify is set to False (in Apache Nifi )

Question

I have an ExecuteScript processor for NIFI that runs the following script:

import requests

import json import logging

postMap = { 'a':'https://a/v/1', 'b':'https://b/v/2' }

flowFile = session.get() if (flowFile != None):

title = flowFile.getAttribute('title')
tag  = flowFile.getAttribute('tag')
link = flowFile.getAttribute('link')
descp = flowFile.getAttribute('descp')

url = _postMap.get(tag)
headers = { 'content-type': "application/json", 'cache-control': "no-cache" }

payload = { "text": "You have a new Notification",
    "attachments": [
    {
        "title": title,
        "title_link": link,
        "text": descp,
        "color": "#764FA5"
    }
    ]
}
logging.error(json.dumps(payload))
response = requests.post(url, headers=headers, verify=False, data=json.dumps(payload))
flowFile = session.putAttribute(flowFile, "status","posted")
session.transfer(flowFile, REL_SUCCESS)
session.commit() 

Now even though verify=False I get this error:

ExecuteScript[id=9af4a34e-0158-1000-7cf5-2beca58c972e] ExecuteScript[id=9af4a34e-0158-1000-7cf5-2beca58c972e] failed to process due to org.apache.nifi.processor.exception.ProcessException: javax.script.ScriptException: requests.exceptions.SSLError: [Errno 1] General SSLEngine problem (javax.net.ssl.SSLHandshakeException: General SSLEngine problem) in <script> at line number 61; rolling back session: org.apache.nifi.processor.exception.ProcessException: javax.script.ScriptException: requests.exceptions.SSLError: [Errno 1] General SSLEngine problem (javax.net.ssl.SSLHandshakeException: General SSLEngine problem) in <script> at line number 61

What can possibly be causing this ?

Answer 1

The root cause seems to be identified: NiFi's python script engine is actually Jython and not cpython as usually assumed. Now we come to how Jython creates a problem ? So, basically Jython is Python running inside a JVM and a JVM never allows bypassing SSL Certificate Verification without explicitly writing a custom Trust Manager which return true for all certificates.

I came across a few posts on Google implementing a custom TrustManager. For reference:

import java.net.Socket;
import java.security.KeyStore;
import java.security.Provider;
import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;
import javax.net.ssl.ManagerFactoryParameters;
import javax.net.ssl.SSLEngine;
import javax.net.ssl.TrustManager;
import javax.net.ssl.TrustManagerFactorySpi;
import javax.net.ssl.X509ExtendedTrustManager;

public class MyProvider extends Provider {
    public MyProvider() {
        super("MyProvider", 1.0, "Trust certificates");
        put("TrustManagerFactory.TrustAllCertificates", MyTrustManagerFactory.class.getName());
    }

    public static class MyTrustManagerFactory extends TrustManagerFactorySpi {
        public MyTrustManagerFactory() {}
        protected void engineInit( KeyStore keystore ) {}
        protected void engineInit(ManagerFactoryParameters mgrparams ) {}
        protected TrustManager[] engineGetTrustManagers() {
            return new TrustManager[] {
                new X509ExtendedTrustManager() {

                    @Override
                public void checkClientTrusted(X509Certificate[] x509Certificates, String s) throws CertificateException {}

                @Override
                public void checkServerTrusted(X509Certificate[] x509Certificates, String s) throws CertificateException {}

                @Override                           
                public X509Certificate[] getAcceptedIssuers() {
                    return null;
                }

                @Override
                public void checkClientTrusted(X509Certificate[] x509Certificates, String s, Socket socket) throws CertificateException {}

                @Override
                public void checkServerTrusted(X509Certificate[] x509Certificates, String s, Socket socket) throws CertificateException {}

                @Override
                public void checkClientTrusted(X509Certificate[] x509Certificates, String s, SSLEngine sslEngine) throws CertificateException {}

                @Override
                public void checkServerTrusted(X509Certificate[] x509Certificates, String s, SSLEngine sslEngine) throws CertificateException {}
                }                                       
            };
        }
    }
    }

However, the MyProvider object when included inside Nifi JVM does not provide the designated effect, working on it, shall resolve and return with an answer soon.