How to use gnupg in an AWS Lambda function

Question

I have a local python code which GPG encrypts a file. I need to convert this to AWS Lambda, once a file has been added to AWS S3 which triggers this lambda.

My local code

import os
import os.path
import time
import sys
gpg = gnupg.GPG(gnupghome='/home/ec2-user/.gnupg')

path = '/home/ec2-user/2021/05/28/'
ptfile = sys.argv[1]

with open(path + ptfile, 'rb')as f:
        status = gpg.encrypt_file(f, recipients=['[email protected]'], output=path + ptfile + ".gpg")

print(status.ok)
print(status.stderr)

This works great when I execute this file as python3 encrypt.py file.csv and the result is file.csv.gpg

I'm trying to move this to AWS Lambda and invoked when a file.csv is uploaded to S3.

import json
import urllib.parse
import boto3
import gnupg
import os
import os.path
import time

s3 = boto3.client('s3')

def lambda_handler(event, context):
    bucket = event['Records'][0]['s3']['bucket']['name']
    key = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['key'], encoding='utf-8')
    
    try:
        gpg = gnupg.GPG(gnupghome='/.gnupg')
        ind = key.rfind('/')
        ptfile = key[ind + 1:]
        with open(ptfile, 'rb')as f:
            status = gpg.encrypt_file(f, recipients=['[email protected]'], output= ptfile + ".gpg")
        print(status.ok)
        print(status.stderr)

My AWS Lambda code zip created a folder structure in AWS

The error I see at runtime is [ERROR] Runtime.ImportModuleError: Unable to import module 'lambda_function': No module named 'gnupg' Traceback (most recent call last):

Answer 1

You can create a gpg binary suitable for use by python-gnupg on AWS Lambda from the GnuPG 1.4 source. You will need

• GCC and associated tools (sudo yum install -y gcc make glibc-static on Amazon Linux 2)
• pip
• zip

After downloading the GnuPG source package and verifying its signature, build the binary with

$ tar xjf gnupg-1.4.23.tar.bz2
$ cd gnupg-1.4.23
$ ./configure
$ make CFLAGS='-static'
$ cp g10/gpg /path/to/your/lambda/

You will also need the gnupg.py module from python-gnupg, which you can fetch using pip:

$ cd /path/to/your/lambda/
$ pip install -t . python-gnupg

Your Lambda’s source structure will now look something like this:

.
├── gnupg.py
├── gpg
└── lambda_function.py

Update your function to pass the location of the gpg binary to the python-gnupg constructor:

gpg = gnupg.GPG(gnupghome='/.gnupg', gpgbinary='./gpg')

Use zip to package the Lambda function:

$ chmod o+r gnupg.py lambda_function.py 
$ chmod o+rx gpg
$ zip lambda_function.zip gnupg.py gpg lambda_function.py 

Answer 2

Since there are some system dependencies required to use gpg within python i.e gnupg itself, you will need to build your lambda code using the container runtime environment: https://docs.aws.amazon.com/lambda/latest/dg/lambda-images.html

Using docker will allow you to install underlying system dependencies, as well as import your keys.

Dockerfile will look something like this:

FROM public.ecr.aws/lambda/python:3.8

RUN apt-get update && apt-get install gnupg

# copy handler file
COPY app.py <path-to-keys> ./

# Add keys to gpg
RUN gpg --import <path-to-private-key>
RUN gpg --import <path-to-public-key>

# Install dependencies and open port
RUN pip3 install -r requirements.txt

CMD ["app.lambda_handler"]  

app.py would be your lambda code. Feel free to copy any necessary files besides the main lambda handler.

Once the container image is built and uploaded. The lambda can now use the image (including all of its dependencies). The lambda code will run within the containerized environment which contains both gnupg and your imported keys.

Resources:

https://docs.aws.amazon.com/lambda/latest/dg/python-image.html https://docs.aws.amazon.com/lambda/latest/dg/lambda-images.html https://medium.com/@julianespinel/how-to-use-python-gnupg-to-decrypt-a-file-into-a-docker-container-8c4fb05a0593

Answer 3

gpg is now already installed in public.ecr.aws/lambda/python:3.8.

However despite that does not seem to be available from Lambda. So you still need to get the gpg executable into the Lambda environment.

I did it using a docker image.

My Dockerfile is just:

FROM public.ecr.aws/lambda/python:3.8

COPY .venv/lib/python3.8/site-packages/ ./
COPY test_gpg.py .

CMD ["test_gpg.lambda_handler"]

.venv is the directory with my python virtualenv containing the python packages I need.

Answer 4

The best way to do this is to add a lambda layer to your python lambda.

You need to make a virtual environment in which you pip install gnupg and then put all the installed python packages in a zip file, which you upload to aws as a lambda layer. This lambda layer can then be used in all lambdas where you need gnupg. To create the lamba layer you basically do:

python3.9 -m venv my_venv
 ./my_venv/bin/pip3.9 install gnupg
cp -r ./my_venv/lib/python3.9/site-packages/ python
zip -r lambda_layer.zip python

Where the python version above has to match that of the python function in your lambda.

If you don't want to use layers you can additionally do:

zip -r lambda_layer.zip ./.gnupg
zip lambda_layer.zip lambda_funtion.py

And you get a zip file that you can use as a lambda deployment package

Answer 5

The python-gnupg package requires you to have a working installation of the gpg executable, as mentioned in their official docs' Deployment Requirements; I am yet to find a way to access a gpg executable from lambda.

Answer 6

I ended up using a Dockerfile image as the library is already available in one of the Amazon Linux or Lambda Python base images provided by AWS that you can find here https://gallery.ecr.aws/lambda/python (in the tag images tab you will find all the Python versions needed based on your requirements).

You will need to create the following 3 files in your dev environment:

• Dockerfile
• requirements.txt
• lambda script

The requirements.txt contains the python-gnupg for the import and all the other libraries based on your requirements:

boto3==1.15.11            # via -r requirements.in
urllib3==1.25.10          # via botocore
python-gnupg==0.5.0       # required for gpg encryption

This is the Dockerfile:

# Python 3.9 lambda base image
FROM public.ecr.aws/lambda/python:3.9

# Install pip-tools so we can manage requirements
RUN yum install python-pip -y

# Copy requirements.txt file locally
COPY requirements.txt ./

# Install dependencies into current directory
RUN python3.9 -m pip install -r requirements.txt

# Copy lambda file locally
COPY s3_to_sftp_batch.py .

# Define handler file name
CMD ["s3_to_sftp_batch.on_trigger_event_test"]

Then inside your lambda code add:

# define library path to point system libraries
os.environ["LD_LIBRARY_PATH"] = "/usr/bin/"

# create instance of GPG class and specify path that contains gpg binary
gpg = gnupg.GPG(gnupghome='/tmp', gpgbinary='/usr/bin/gpg')

Save these files then go to AWS ECR and create a Private repo then go to the repo and in the top right corner go to View push commands and run them to push your image in AWS. Finally create your Lambda function using the container image.