Cant restrict SMTP connection by IP in Amazon SES

Question

I use Amazon SES to send transactional emails (using SMTP connection) from my app and I wanna improve the credentials security by restricting access from specific IPs.

I've created an IAM Policy for that and applied to the user:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Deny",
            "Action": "ses:SendRawEmail",
            "Resource": "*",
            "Condition": {
                "NotIpAddress": {
                    "aws:SourceIp": [
                        "1.2.3.4",
                        "5.6.7.8",
                    ]
                }
            }
        }
    ]
}

I also tried a broader version:

{
    "Version": "2012-10-17",
    "Statement": {
        "Effect": "Deny",
        "Action": "*",
        "Resource": "*",
        "Condition": {
            "NotIpAddress": {
                "aws:SourceIp": [
                    "1.2.3.4",
                    "5.6.7.8"
                ]
            }
        }
    }
}

For my surprise, the user can still connect via SMTP from a not authorized IP.

$mail = new \PHPMailer(true);

$mail->Host = SES_HOST; // sets the SMTP server
$mail->Port = SES_PORT;                    // set the SMTP port for the GMAIL server
$mail->Username = SES_USERNAME; // SMTP account username
$mail->Password = SES_PASSWORD;        // SMTP account password

...

if($mail->smtpConnect()) {
    echo 'WTF?';

    $mail->smtpClose();
}

I guess the IAM Policy restriction applies to Amazon API, but not to direct SMTP connections. Am I wrong?

I look for IP restrictions in the SES section of amazon web panel, but with no luck.

What Am I missing?