How to continue using msmtp OAuth 2.0 for Gmail in mutt after oob deprecation?

Question

background

For two years I've been happily accessing my Gmail accounts with neomutt.

I'm sync'ing between neomutt locally and my online Gmail account with mbsync and a "2-Step Verification" app password (Sign in with App Passwords).

sending email with msmtp, until now

To send a Gmail with neomutt is trickier, because msmtp requires an unexpired token from the Gmail API. Fortunately GitHub user tenllado provided the only working open-source solution that I've been able to find, his script oauth2token. I adapted it as oauth2tool.sh. The steps for this to function are:

1 prepare - get my Gmail OAuth 2.0 credentials

1. Use Gmail API's Python Quickstart to get my credentials, which look like this:
   • my Client ID: xxxxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.apps.googleusercontent.com
   • my Client Secret: xxxxxxxxxxxxxxxxxxxxxxxx
2. Grab a copy of oauth2.py (Code "the refresh token lasts indefinitely").
3. Get the immortal refresh token: $ python2 oauth2.py [email protected] --client_id=<myCI> --client_secret=<myCS> --generate_oauth2_token and follow the instructions. It looks like this:
   • refresh token: 1//03xxxxxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxxx

2 prepare - configure msmtprc

account my
auth oauthbearer
host smtp.gmail.com
port 587
from [email protected]
user [email protected]
passwordeval bash oauth2tool.sh my

3 use - send emails from the command line, until now

Then, when I send an email echo "test" | msmtp -a my <target_email>, my oauth2tool.sh pulls up a valid token. The way it does this is it grabs the token with pass if it's not expired, otherwise it grabs a new one with python2 oauth2.py [email protected] --client_id=<myCI> --client_secret=<myCS> --refresh_token=<myRT>.

With all this, I could easily send emails from my Gmail accounts from the command line, until now.

now, oob is no longer allowed

Now my once immortal refresh tokens are being expired, and I can't renew them because Gmail's oauth2.py is using redirect_uri = urn:ietf:wg:oauth:2.0:oob, which is deprecated.

Making Google OAuth interactions safer by using more secure OAuth flows "OAuth out-of-band (oob) flow will be deprecated".

How to continue sending with msmtp?

OAuth 2.0 for Mobile & Desktop Apps "Loopback IP address (macOS, Linux, Windows desktop)" seems to be the way forward, but I'd need a few weeks of free time, which I don't have, to figure out how. Any ideas out there?

Related question: Google Cloud: OAuth clients in test mode that are using the OAuth OOB flow.

Answer 1

getmail6 solution

getmail6 have somewhat solved this problem for Gmail (and Microsoft Office 365) with their Python script getmail-gmail-xoauth-tokens, though with the huge caveat that the Gmail API now limits your refresh token to just 1 hour, so the last step described below, involving brief browser interaction and generating Google security alerts, would need to be repeated regularly, thus making this a solution of little practical worth. (getmailrc-examples does warn of this - "Unfortunately... needs to be repeated regularly.") In discovering this, I successfully worked through Akkana Peck's instructions, Sending Mail via Gmail using OAuth2 (2022 Edition) (as suggested in the comment from @GuenterRoeck).

1 get your OAuth 2.0 Client ID & Secret

(Note: This might not be the only or best path, but it works. In 2020 I got these credentials more easily by the now obsolete route described in the OP, and they're still valid.)

1. Go to Create a Google Cloud project and, as instructed there, open the Google Cloud console which tells you how to navigate on to New Project where you need to choose a Project name - I chose getmail6.
2. In APIs & Services select Enabled APIs and services and click on + ENABLE APIS AND SERVICES.
3. Select the Gmail API, and ENABLE it.
4. Select Credentials and click on + CREATE CREDENTIALS to get to Create OAuth client ID where you click on Configure consent screen, which is where, it seems, we need to now continue as if we're app creators, which seems to me absurd, but there's no choice. Not being a Google Workspace user, I was obliged to select External ("Available to any test user with a Google Account. Your app will start in testing mode and will only be available to users you add to the list of test users. Once your app is ready to push to production, you may need to verify your app."). We're obliged then to specify our imaginary App name - I chose "msmtpGmail", and our User support email, which is that of the account that you're doing all this from, same for our Developer contact information. Then Save and continue.
5. Back to Create OAuth client ID for Application type - I chose Desktop app, and Name - I chose "msmtp", which, finally, got me my Client ID and Client Secret. I downloaded them as client_secret_nnnnnnnnnnnn-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.apps.googleusercontent.com.json, and filed that away safely.
6. Essential last step is to add yourself in Test users (because Google didn't do that for you) at OAuth consent screen.

2 get your OAuth 2.0 access_token (and refresh_token)

Here we leverage some of the hard work of getmail6. You're going to need their getmail-gmail-xoauth-tokens Python script.

1 make your (secret) json

As instructed in getmail6's getmailrc-examples (- search for "mail.google.com"), make this JSON - I named mine getmail6.json:

{"scope": "https://mail.google.com/",
"user": "[email protected]",
"client_id": "yours",
"client_secret": "yours",
"token_uri": "https://accounts.google.com/o/oauth2/token",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs"}

- fill in yours...

2 generate your access_token (and refresh_token)

python3 getmail-gmail-xoauth-tokens --init your.json (fill in your) > "Google hasn’t verified this app" > Continue > "<yourApp> wants access to your Google Account" > Continue, which appends into your.json these lines:

"access_token": "<extremely_long_key>",
"expires_at": <now+3600s>,
"refresh_token": "<another_extremely_long_key>"}

- which are what we've been looking for, but that 1h expiry is prohibitively awkward...

3 adjust your ~/.msmtprc

Again, leveraging getmail-gmail-xoauth-tokens:

tls on
tls_trust_file  /etc/ssl/certs/ca-certificates.crt
logfile ~/.msmtp.log
account your
auth oauthbearer
host smtp.gmail.com
port 587
from [email protected]
user [email protected]
passwordeval python3 getmail-gmail-xoauth-tokens your.json

- fill in instances of your.

4 send a test mail

Quickly, you've got an hour before your refresh token has gone stale (change your):

echo "test send from msmtp using Gmail's OAuth 2.0" | msmtp -a your <a_target_email>