Forward email but change the FROM address

Question

I have a postfix server running on an EC2 instance. I want to forward all email, via SES, to my personal inbox.

The problem: AWS only allows a FROM address that is verified in the AWS console and the FROM address in this case could be anything, for example: twitter.com. I cannot white-list my server's IP and say: "Accept all emails from this location regardless of sender" (would be a bad idea anyway)

So, I need to figure out a way to forward my email with a verified address but I do not want to lose the original sender's address.

Is there a way of doing this?

Answer 1

Based on our discussion in chat, I'm going to provide you my hackish, customized solution that will change the "FROM" address as we expect it to be, and then deliver to the original destination point but add the "Reply-To" header.

This is a very hackish approach, but should manipulate the messages as you expect before actually sending them via PostFix to where they need to go.

First, PostFix ports need to be changed. We need to change the Postfix SMTP port to something other than 25 so that our python SMTP handler we are going to set up will work on that port instead.

Edit /etc/postfix/master.cf. You're going to be looking for a line like this:

smtp      inet  n       -       y       -       -       smtpd

Comment out this line, and underneath that line, use this instead:

10025      inet  n       -       y       -       -       smtpd

This tells Postfix that we don't want it to listen on the standard SMTP port. Restart the postfix service when you're done with this step.

---

Next, the Python SMTP handler which I mentioned above. This will handle incoming messages, manipulate them, and resend them to the PostFix on your system. Assuming, of course, that all mail is submitted on port 25, even locally.

This code exists on a GitHub GIST and is based off of a generic Python SMTP server code example I got somewhere (but don't remember from where sorry!), and have since manipulated.

The code is also here, it's in Python 3 in case you're curious, and is written with Python 3 as the target Python version:

#!/usr/bin/env python3

# Libraries
import smtplib
import smtpd
import asyncore
import email
import sys
from datetime import datetime

print('Starting custom mail handling server...')

# We need to know where the SMTP server is heh.
SMTP_OUTBOUND = 'localhost'

# We also need to know what we want the "FROM" address to be
FROM_ADDR = "[email protected]"
DESTINATION_ADDRESS = "[email protected]"

#############
#############
# SMTP SERVER
#############
#############


# noinspection PyMissingTypeHints,PyBroadException
class AutoForwardHandlerSMTP(smtpd.SMTPServer):

    def process_message(self, peer, mailfrom, rcpttos, data, **kwargs):
        print('MESSAGE RECEIVED - [%s]' % datetime.now().strftime('%Y-%m-%d %H:%M:%S'))
        print('Receiving message from:', peer)
        print('Message addressed from:', mailfrom)
        print('Message addressed to  :', rcpttos)
        print('Message length        :', len(data))
        print(data)

        # Flush the output buffered (handy with the nohup launch)
        sys.stdout.flush()

        # Analyze and extract data headers
        msg = email.message_from_string(data)
        orig_from = ''
        try:
            orig_from = msg['From']
            msg['Reply-To'] = orig_from
            # We have to use 'replace header' methods to overwrite existing headers.
            msg.replace_header("From", FROM_ADDR)
        except:
            print("Error manipulating headers:", sys.exc_info()[0])

        conn = smtplib.SMTP(SMTP_OUTBOUND, 10025)
        conn.sendmail(FROM_ADDR, msg["To"], msg.as_string())

        # Flush the output buffered (handy with the nohup launch)
        print("\n\n")
        sys.stdout.flush()
        return


# Listen to port 25 ( 0.0.0.0 can be replaced by the ip of your server but that will work with 0.0.0.0 )
server = AutoForwardHandlerSMTP(('0.0.0.0', 25), None)

# Wait for incoming emails
asyncore.loop()

Store this as /opt/PythonAutoForwarderSMTP.py, or whatever you want to call it. Run it with the following as root (either via sudo or by dropping into a root user prompt), first, to make sure it works as we expect:

python3 /opt/PythonAutoForwarderSMTP.py

Once it's confirmed running, send an email through the server. It should be picked up and give you log data from this script that a message was received and processed. You should also then see a connection on Postfix's logs, and this being delivered somewhere after Postfix. If all of this looks OK, and you process the message properly and see it with a different 'From' address wherever the mail message finally ends up, then we can work to get it to autostart now! (You can simply hit Ctrl + C to close out the python process, before continuing).

Assuming we want it to start at boot, then we need to set it up to do so.

As root, run crontab -e, and add the following to the root crontab:

@reboot /usr/bin/python3 /opt/PythonAutoForwarderSMTP.py 2>&1 >> /var/log/PythonSMTP.log &

Save the crontab file. If you don't want to reboot your server, then execute the command line you just added, minus the @reboot part, to run the Python SMTP handler.

Whether run by cron or not, the process that loads the Python will end up forked into the background, and also put all data output (error or otherwise in the Python console) to a log file in /var/log/PythonSMTP.log in Append mode. That way, you can always get logs as you need to.

If all works as expected, this will properly add a Reply-To header, and also adjust the "From" header in the message to be what we expect it to be. I can't guarantee this'll work properly for SPF and DKIM checking, if messages are signed, but I can say that this will properly 'preprocess' messages before using Postfix to relay them elsewhere.

---

OBLIGATORY Security Concerns and Functional Change Notifications:

• Sender DKIM verification may fail. DKIM signature verification fails whenever messages that are signed are manipulated, which means you might have broken DKIM signatures from senders. That means things might get picked up as spam due to failed signature verification. This script can probably be customized to 'work' properly, but I didn't write this to do DKIM/SPF checking.
• We must run this Python SMTP server as root. This is necessary because in Linux by default we can't bind to ports under 1024 unless we are superuser; this is why Postfix has a master 'root' owned process and does sub-processes that don't run as the root user for very long only to port-bind.
• ALL mail on port 25 will end up going through this Python SMTP server. If Postfix also handles mail from outside->in, then the Python SMTP server will be taking its place. This can introduce some evils, but ultimately does what you're asking for.
• This is a fragile solution. While it's not as fragile as some other solutions, if the Python process dies, it doesn't come back up automatically, so you have to handle errors on a case-by-case basis and sometimes bring the Python process back to life if it dies off completely.
• There are no StartTLS or SSL/TLS handlers in this. So everything is Plain Text (which is insecure!)

As always, you should not run anything as root unless you know what you're doing. In this case, I provide the code for this in plain view so you can discern for yourself what this script does, and whether you want to run it as root or not, if you are security-centric and paranoid like I am (I am an IT Security professional as well as a sysadmin, so forgive these obligatory notices)

Answer 2

As well as the great answer by @Thomas Ward AWS does have a "preferred" way, which is very similar, the only difference is that it uses AWS internal tools to accomplish the task instead of an external python script.

There is one key difference between this approach and the other, this approach will do virus/malware scanning and DKIM and SPF checks which you can actually test for and see if they PASS.

So, I followed this GitHub repo's README here: https://github.com/arithmetric/aws-lambda-ses-forwarder

Everything is because of this script. You place it on AWS Lambda and it will post-process the emails for SES rules.

Here is a copy of the setup part of the README:

Note: change things like S3-BUCKET-NAME.

1. Modify the values in the config object at the top of index.js to specify the S3 bucket and object prefix for locating emails stored by SES. Also provide the email forwarding mapping from original destinations to new destination.

2. In AWS Lambda, add a new function and skip selecting a blueprint.

   • Name the function "SesForwarder" and optionally give it a description. Ensure Runtime is set to Node.js 4.3 or 6.10.

   • For the Lambda function code, either copy and paste the contents of index.js into the inline code editor or zip the contents of the repository and upload them directly or via S3.

   • Ensure Handler is set to index.handler.

   • For Role, choose "Basic Execution Role" under Create New Role. In the popup, give the role a name (e.g., LambdaSesForwarder). Configure the role policy to the following: { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents" ], "Resource": "arn:aws:logs:*:*:*" }, { "Effect": "Allow", "Action": "ses:SendRawEmail", "Resource": "*" }, { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:PutObject" ], "Resource": "arn:aws:s3:::S3-BUCKET-NAME/*" } ] }

   • Memory can be left at 128 MB, but set Timeout to 10 seconds to be safe. The task usually takes about 30 MB and a few seconds. After testing the task, you may be able to reduce the Timeout limit.

3. In AWS SES, verify the domains for which you want to receive and forward email. Also configure the DNS MX record for these domains to point to the email receiving (or inbound) SES endpoint. See SES documentation for the email receiving endpoints in each region.

4. If you have the sandbox level of access to SES, then also verify any email addresses to which you want to forward email that are not on verified domains.

5. If you have not configured inbound email handling, create a new Rule Set. Otherwise, you can use an existing one.

6. Create a rule for handling email forwarding functionality.

   • On the Recipients configuration page, add any email addresses from which you want to forward email.

   • On the Actions configuration page, add an S3 action first and then an Lambda action.

   • For the S3 action: Create or choose an existing S3 bucket. Optionally, add an object key prefix. Leave Encrypt Message unchecked and SNS Topic set to [none].

   • For the Lambda action: Choose the SesForwarder Lambda function. Leave Invocation Type set to Event and SNS Topic set to [none].

   • Finish by naming the rule, ensuring it's enabled and that spam and virus checking are used.

   • If you get an error like "Could not write to bucket", follow step 7 before completing this one

   • If you are asked for SES to attempt to add permissions to access lambda:InvokeFunction, agree to it.

7. The S3 bucket policy needs to be configured so that your IAM user has read and write access to the S3 bucket. When you set up the S3 action in SES, it may add a bucket policy statement that denies all users other than root access to get objects. This causes access issues from the Lambda script, so you will likely need to adjust the bucket policy statement with one like this: { "Version": "2012-10-17", "Statement": [ { "Sid": "GiveSESPermissionToWriteEmail", "Effect": "Allow", "Principal": { "Service": "ses.amazonaws.com" }, "Action": "s3:PutObject", "Resource": "arn:aws:s3:::S3-BUCKET-NAME/*", "Condition": { "StringEquals": { "aws:Referer": "AWS-ACCOUNT-ID" } } } ] }

8. Optionally set the S3 lifecycle for this bucket to delete/expire objects after a few days to clean up the saved emails.

I am posting a version of the script from the creation time of this answer with one or two changes.

I noticed that emails that were routed twice through verified domains got changed by this script, so for awesome looking-ness I fixed that

"use strict";

var AWS = require('aws-sdk');

console.log("AWS Lambda SES Forwarder // @arithmetric // Version 4.2.0");

// Configure the S3 bucket and key prefix for stored raw emails, and the
// mapping of email addresses to forward from and to.
//
// Expected keys/values:
//
// - fromEmail: Forwarded emails will come from this verified address
//
// - subjectPrefix: Forwarded emails subject will contain this prefix
//
// - emailBucket: S3 bucket name where SES stores emails.
//
// - emailKeyPrefix: S3 key name prefix where SES stores email. Include the
//   trailing slash.
//
// - forwardMapping: Object where the key is the lowercase email address from
//   which to forward and the value is an array of email addresses to which to
//   send the message.
//
//   To match all email addresses on a domain, use a key without the name part
//   of an email address before the "at" symbol (i.e. `@example.com`).
//
//   To match a mailbox name on all domains, use a key without the "at" symbol
//   and domain part of an email address (i.e. `info`).
var defaultConfig = {
  fromEmail: "",
  subjectPrefix: "",
  emailBucket: "ses-sammaye",
  emailKeyPrefix: "email/",
  forwardMapping: {
    "@vvv.com": [
      "[email protected]"
    ],
    "@fff.com": [
      "[email protected]"
    ],
    "@ggg.com": [
      "[email protected]"
    ],
  },
  verifiedDomains: [
    'vvv.com',
    'fff.com',
    'ggg.com'
  ]
};

/**
 * Parses the SES event record provided for the `mail` and `receipients` data.
 *
 * @param {object} data - Data bundle with context, email, etc.
 *
 * @return {object} - Promise resolved with data.
 */
exports.parseEvent = function(data) {
  // Validate characteristics of a SES event record.
  if (!data.event ||
      !data.event.hasOwnProperty('Records') ||
      data.event.Records.length !== 1 ||
      !data.event.Records[0].hasOwnProperty('eventSource') ||
      data.event.Records[0].eventSource !== 'aws:ses' ||
      data.event.Records[0].eventVersion !== '1.0') {
    data.log({message: "parseEvent() received invalid SES message:",
      level: "error", event: JSON.stringify(data.event)});
    return Promise.reject(new Error('Error: Received invalid SES message.'));
  }

  data.email = data.event.Records[0].ses.mail;
  data.recipients = data.event.Records[0].ses.receipt.recipients;
  return Promise.resolve(data);
};

/**
 * Transforms the original recipients to the desired forwarded destinations.
 *
 * @param {object} data - Data bundle with context, email, etc.
 *
 * @return {object} - Promise resolved with data.
 */
exports.transformRecipients = function(data) {
  var newRecipients = [];
  data.originalRecipients = data.recipients;
  data.recipients.forEach(function(origEmail) {
    var origEmailKey = origEmail.toLowerCase();
    if (data.config.forwardMapping.hasOwnProperty(origEmailKey)) {
      newRecipients = newRecipients.concat(
        data.config.forwardMapping[origEmailKey]);
      data.originalRecipient = origEmail;
    } else {
      var origEmailDomain;
      var origEmailUser;
      var pos = origEmailKey.lastIndexOf("@");
      if (pos === -1) {
        origEmailUser = origEmailKey;
      } else {
        origEmailDomain = origEmailKey.slice(pos);
        origEmailUser = origEmailKey.slice(0, pos);
      }
      if (origEmailDomain &&
          data.config.forwardMapping.hasOwnProperty(origEmailDomain)) {
        newRecipients = newRecipients.concat(
          data.config.forwardMapping[origEmailDomain]);
        data.originalRecipient = origEmail;
      } else if (origEmailUser &&
        data.config.forwardMapping.hasOwnProperty(origEmailUser)) {
        newRecipients = newRecipients.concat(
          data.config.forwardMapping[origEmailUser]);
        data.originalRecipient = origEmail;
      }
    }
  });

  if (!newRecipients.length) {
    data.log({message: "Finishing process. No new recipients found for " +
      "original destinations: " + data.originalRecipients.join(", "),
      level: "info"});
    return data.callback();
  }

  data.recipients = newRecipients;
  return Promise.resolve(data);
};

/**
 * Fetches the message data from S3.
 *
 * @param {object} data - Data bundle with context, email, etc.
 *
 * @return {object} - Promise resolved with data.
 */
exports.fetchMessage = function(data) {
  // Copying email object to ensure read permission
  data.log({level: "info", message: "Fetching email at s3://" +
    data.config.emailBucket + '/' + data.config.emailKeyPrefix +
    data.email.messageId});
  return new Promise(function(resolve, reject) {
    data.s3.copyObject({
      Bucket: data.config.emailBucket,
      CopySource: data.config.emailBucket + '/' + data.config.emailKeyPrefix +
        data.email.messageId,
      Key: data.config.emailKeyPrefix + data.email.messageId,
      ACL: 'private',
      ContentType: 'text/plain',
      StorageClass: 'STANDARD'
    }, function(err) {
      if (err) {
        data.log({level: "error", message: "copyObject() returned error:",
          error: err, stack: err.stack});
        return reject(
          new Error("Error: Could not make readable copy of email."));
      }

      // Load the raw email from S3
      data.s3.getObject({
        Bucket: data.config.emailBucket,
        Key: data.config.emailKeyPrefix + data.email.messageId
      }, function(err, result) {
        if (err) {
          data.log({level: "error", message: "getObject() returned error:",
            error: err, stack: err.stack});
          return reject(
            new Error("Error: Failed to load message body from S3."));
        }
        data.emailData = result.Body.toString();
        return resolve(data);
      });
    });
  });
};

/**
 * Processes the message data, making updates to recipients and other headers
 * before forwarding message.
 *
 * @param {object} data - Data bundle with context, email, etc.
 *
 * @return {object} - Promise resolved with data.
 */
exports.processMessage = function(data) {
  var match = data.emailData.match(/^((?:.+\r?\n)*)(\r?\n(?:.*\s+)*)/m);
  var header = match && match[1] ? match[1] : data.emailData;
  var body = match && match[2] ? match[2] : '';

  // Add "Reply-To:" with the "From" address if it doesn't already exists
  if (!/^Reply-To: /mi.test(header)) {
    match = header.match(/^From: (.*(?:\r?\n\s+.*)*\r?\n)/m);
    var from = match && match[1] ? match[1] : '';
    if (from) {
      header = header + 'Reply-To: ' + from;
      data.log({level: "info", message: "Added Reply-To address of: " + from});
    } else {
      data.log({level: "info", message: "Reply-To address not added because " +
       "From address was not properly extracted."});
    }
  }

  // SES does not allow sending messages from an unverified address,
  // so replace the message's "From:" header with the original
  // recipient (which is a verified domain)
  header = header.replace(
    /^From: (.*(?:\r?\n\s+.*)*)/mg,
    function(match, from) {
      var fromText;
      var fromEmailDomain = from.replace(/(.*)</, '').replace(/.*@/, "").replace('>', '').trim();
      if (data.config.verifiedDomains.indexOf(fromEmailDomain) === -1) {
        if (data.config.fromEmail) {
          fromText = 'From: ' + from.replace(/<(.*)>/, '').trim() +
          ' <' + data.config.fromEmail + '>';
        } else {
          fromText = 'From: ' + from.replace('<', 'at ').replace('>', '') +
          ' <' + data.originalRecipient + '>';
        }
      } else {
          fromText = 'From: ' + from;
      }
      return fromText;
    });

  // Add a prefix to the Subject
  if (data.config.subjectPrefix) {
    header = header.replace(
      /^Subject: (.*)/mg,
      function(match, subject) {
        return 'Subject: ' + data.config.subjectPrefix + subject;
      });
  }

  // Replace original 'To' header with a manually defined one
  if (data.config.toEmail) {
    header = header.replace(/^To: (.*)/mg, () => 'To: ' + data.config.toEmail);
  }

  // Remove the Return-Path header.
  header = header.replace(/^Return-Path: (.*)\r?\n/mg, '');

  // Remove Sender header.
  header = header.replace(/^Sender: (.*)\r?\n/mg, '');

  // Remove Message-ID header.
  header = header.replace(/^Message-ID: (.*)\r?\n/mig, '');

  // Remove all DKIM-Signature headers to prevent triggering an
  // "InvalidParameterValue: Duplicate header 'DKIM-Signature'" error.
  // These signatures will likely be invalid anyways, since the From
  // header was modified.
  header = header.replace(/^DKIM-Signature: .*\r?\n(\s+.*\r?\n)*/mg, '');

  data.emailData = header + body;
  return Promise.resolve(data);
};

/**
 * Send email using the SES sendRawEmail command.
 *
 * @param {object} data - Data bundle with context, email, etc.
 *
 * @return {object} - Promise resolved with data.
 */
exports.sendMessage = function(data) {
  var params = {
    Destinations: data.recipients,
    Source: data.originalRecipient,
    RawMessage: {
      Data: data.emailData
    }
  };
  data.log({level: "info", message: "sendMessage: Sending email via SES. " +
    "Original recipients: " + data.originalRecipients.join(", ") +
    ". Transformed recipients: " + data.recipients.join(", ") + "."});
  return new Promise(function(resolve, reject) {
    data.ses.sendRawEmail(params, function(err, result) {
      if (err) {
        data.log({level: "error", message: "sendRawEmail() returned error.",
          error: err, stack: err.stack});
        return reject(new Error('Error: Email sending failed.'));
      }
      data.log({level: "info", message: "sendRawEmail() successful.",
        result: result});
      resolve(data);
    });
  });
};

/**
 * Handler function to be invoked by AWS Lambda with an inbound SES email as
 * the event.
 *
 * @param {object} event - Lambda event from inbound email received by AWS SES.
 * @param {object} context - Lambda context object.
 * @param {object} callback - Lambda callback object.
 * @param {object} overrides - Overrides for the default data, including the
 * configuration, SES object, and S3 object.
 */
exports.handler = function(event, context, callback, overrides) {
  var steps = overrides && overrides.steps ? overrides.steps :
  [
    exports.parseEvent,
    exports.transformRecipients,
    exports.fetchMessage,
    exports.processMessage,
    exports.sendMessage
  ];
  var data = {
    event: event,
    callback: callback,
    context: context,
    config: overrides && overrides.config ? overrides.config : defaultConfig,
    log: overrides && overrides.log ? overrides.log : console.log,
    ses: overrides && overrides.ses ? overrides.ses : new AWS.SES(),
    s3: overrides && overrides.s3 ?
      overrides.s3 : new AWS.S3({signatureVersion: 'v4'})
  };
  Promise.series(steps, data)
    .then(function(data) {
      data.log({level: "info", message: "Process finished successfully."});
      return data.callback();
    })
    .catch(function(err) {
      data.log({level: "error", message: "Step returned error: " + err.message,
        error: err, stack: err.stack});
      return data.callback(new Error("Error: Step returned error."));
    });
};

Promise.series = function(promises, initValue) {
  return promises.reduce(function(chain, promise) {
    if (typeof promise !== 'function') {
      return Promise.reject(new Error("Error: Invalid promise item: " +
        promise));
    }
    return chain.then(promise);
  }, Promise.resolve(initValue));
};