LWP::UserAgent Insists on verifying hostname

Question

The following script works returns a comprehensive headers on a host running libwww-perl-5.836 but not on the host using libwww-perl-6.30.0. In that case, the script displays the following:

    500 Can't connect to backend.mutegroup.org:443 (certificate verify failed)
    Content-Type: text/plain
    Client-Date: Mon, 28 Jul 2014 21:09:28 GMT
    Client-Warning: Internal response

    Can't connect to backend.mutegroup.org:443 (certificate verify failed)

    LWP::Protocol::https::Socket: SSL connect attempt failed with unknown error error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed at /usr/lib64/perl5/vendor_perl/5.16.3/LWP/Protocol/http.pm line 51.

This is the script:

#!/usr/bin/perl

use strict;
use warnings;

use LWP::UserAgent;
use HTTP::Request::Common;

my $ua      = LWP::UserAgent->new(ssl_opts => { verify_hostname => 0, } );
my $url     = 'https://backend.mutegroup.org/api/getLastId';

my $request = POST $url;
print $ua->request($request)->as_string

Host names are not verified by default for libwww-perl-5.837 and prior. That explains why it works on the old host. However, I am explicitly disabling checks and it is still insisting on doing them.

This is on a Gentoo system.

Answer 1

You have turned off verifying the hostname; you have not turned off verifying the certificate.

verify_hostname controls whether "LWP will for secure protocol schemes ensure it connects to servers that have a valid certificate matching the expected hostname" (my emphasis). Setting that to 0 allows you to connect to a server that has a valid certificate but not issued for the host / hostname that you are trying to reach.

To turn off checking that the certificate is valid (issued by a trusted CA), you want:

use IO::Socket::SSL;

my $ua = LWP::UserAgent->new(
   ssl_opts => {
      verify_hostname => 0,
      SSL_verify_mode => IO::Socket::SSL::SSL_VERIFY_NONE,
   },
);

Note that turning off either of these options is a bad idea if you are transmitting any sensitive information or expect to trust the data returned. With either of these turned off, you are losing the benefits of SSL and are vulnerable to various man-in-the-middle attacks.

Answer 2

We installed cloudflare cert's on our backend machines, and couldn't use our perl scripts to activate URL end points, until we added the SSL_ca_file option to ssl_opts:

my $ua = LWP::UserAgent->new;
$ua->ssl_opts(
    SSL_ca_file => '/etc/ssl/certs/cloudflare-x.pem'
);