Unattended OAuth2 access to google mailbox using IMAP

Question

I have a program that fetches e-mail from a google mailbox using IMAP. It currently uses username/password ("PLAIN") authentication in the IMAP protocol. I would like to change it to support XOAUTH2 authentication. The program is written in .NET and runs as a service on Windows and so there is no opportunity for the authorization consent request that one normally associates with OAuth2.

It seems to me that the proper procedure would be:

1. Create a "Project" in Google on the e-mail address account for the sole purpose of fetching mail.
2. Create a "Service Account" on that project (which I believe should give the ability to have unattended access)
3. Upload a certificate so Google can use its public key to authenticate the Access Token request, which is signed using the private key by the client Windows service
4. Use ServiceAccountCredential and its Initializer class (part of the Google.Apis.Auth NUGet package) to obtain an Access Token
5. Pass the Access Token during the IMAP protocol setup.

I seem to have a choice of hitting two walls: The first, when I omitted setting the User when building the ServiceAccountCredential, I could obtain an Access Token but when used in IMAP I would get an authorization failure. The second, when I supplied the e-mail address as the User, I would get the following error trying to obtain the access token:

Client is unauthorized to retrieve access tokens using this method, or client not authorized for any of the scopes requested

I am assuming here that the "client" mentioned in the error message refers to the Google Project, though I could be wrong, it could refer to the Service Account. In any case, the scope being requested is https://mail.google.com/

In the setup for the Google Project I can see ("Enabled APIs and Services") where I can enable certain API's for the Project (GMail API is enabled).

I can also see, if I go to the "OAuth Consent Screen", where I can authorize specific scopes for the selected API and in particular where I can enable the https://mail.google.com/ scope. However, the whole goal here is to not have any consent screen (and in any case enabling the https://mail.google.com/ scope here doesn't help).

I don't want access to any other mailbox, so I don't think I need "Domain-Wide Delegation" on the Service Account. Besides, this is just an individual mail account, there is no Google Domain involved.

It seems that I'm missing how to authorize the scope for the Service Account (or the Project), although the "this OR that" nature of the error message may be misleading me. Or perhaps I'm completely misunderstood this...

Suggestions?

Answer 1

Service accounts exist as their own entities – they have access to the project's resources but do not have access to your personal account, because it's a separate account. Only in a project belonging to a Google Workspace domain it's possible to allow a service account to impersonate users of the same domain ("Domain-wide delegation"); this doesn't work for individual Gmail accounts.

So in order to access your personal account's mailbox, the app must go through the OAuth consent screen at least once and get a refresh token. Once you have the refresh token (which is long-lived), the app can store it and obtain access tokens when needed (using the non-interactive "refresh token grant").

Typically you would do this by splitting the app in two: an interactive program which obtains the refresh token (and stores it somewhere) and a service that just uses the refresh token already stored in its configuration, without the need for UI. In case the options for interactivity are limited (e.g. doing the initial setup over SSH), it's possible to use the device authorization flow where the app only needs to output a URL.

Note that the OAuth application must be "published" for the refresh tokens to actually be long-lived, but you don't need to go through Google's restricted-scope verification – you'll only have to click through a few more scare screens than normal. (Alternatively: Add your Gmail account to Thunderbird, let it do the OAuth2 setup, then copy the refresh token from its "Saved passwords" dialog.)

(Also, consider implementing the standard OAUTHBEARER instead of XOAUTH2. Google supports both mechanisms; they don't differ much, but the OAUTHBEARER variant is the "current" one.)