Configuring an Application Specific Password for Gmail Fetchers

Written By Michael Singh
 

An Introduction to Google Mailbox Journaling

For the many organizations worldwide who use Google Workspace for their communication and productivity tools, such as Gmail and Google Docs, one of the methods DataCove can use to gather email for low volume sites (defined as sites that receive less than 10,000 emails per day) is known as a Fetcher. A Fetcher is a specialized login and download process that uses the IMAP4 or POP3 protocols, very similar to that of an email client like Microsoft Outlook, that is given the credentials necessary to get into a Google mailbox to read, download and delete the messages contained within.

This is one of the two primary mechanisms used by DataCove to gather journaled emails from a Google Workspace environment, but is largely becoming a deprecated method due to the simplicity of the SMTP Direct means of journaling (more on that can be found here), which lets Google send the journaled emails directly to the DataCove, rather than to a mailbox that it maintains which DataCove then needs to fetch from.

Given the ever-tightening safeguards for mailbox security from Google, which now require MultiFactor Authentiction (MFA) for mailbox access using the IMAP4 or POP3 protocols, a slightly different methodology is needed for configuration of a DataCove Fetcher to Google.

As DataCove does not inherently possess the ability to pass MFA challenges, we instead use a high security password variant that Google offers known as an Application Specific Password. These are passwords designed for computers to use to access sensitive accounts that utilize a complex character string that would be a pain for any human to use regularly for something as common as email, and are considered ‘high security’ as a result.

This guide covers the steps necessary to configure an Application Specific Password for a Google Mailbox, as well as updating an existing DataCove Fetcher with that password.

Note: This particular article is specific to organizations who are already using this Fetcher functionality and assumes that a Journaling mailbox already exists within Google; it does not cover the creation of such a mailbox for future use. For new deployments or organizations looking to modernize their DataCove configuration, it is strongly recommended to use the aforementioned SMTP Direct method of gathering journaled emails from Google into DataCove, rather than the Fetcher method, as Google may decide to further restrict IMAP4/POP3 mailbox access in the future by deprecating Application Specific Passwords.

 

Prerequisites of Configuring sIMAP4 Fetching for a Google Mailbox

DataCove’s means of accessing a Google mailbox for fetching of messages is traditionally conducted using the sIMAP4 protocol, which uses TLS encryption for security (the “s” in sIMAP4) to protect against anyone “sniffing” the messages to try and read their content. The base protocol of IMAP4 (Internet Message Access Protocol, version four) is a venerable and universal protocol from the 1990’s that allows users to access Gmail messages from various devices and email clients, keeping everything synchronized with Gmail servers.

To configure sIMAP4 fetching from a Gmail account, the account must meet the following prerequisites: 

  • Have a licensed Google Account with Gmail mailbox

  • Enable IMAP4 Protocol Support in Gmail

  • Ensure 2-Step Verification is enabled on the Google Account

  • Creation of an Application Specific Password for DataCove access

Note: Depending on an organization’s policies, there may be preventative rules in place that prevent a user from configuring 2-Step Verification or Application Passwords. In these instances, creating a new, separate Organizational Unit (OU) in the root of the Google Directory and creating/moving the journaling mailbox to it is the preferred approach. Doing so prevents inheritance of such policies from the parent structure (or if policies are defined at the root level, the ability to change them for just this mailbox without impact any other user accounts) and will allow for configuration as per the guidance.

 

Enabling IMAP4 Protocol Support

To enable IMAP4 services in Gmail (which include the Secure variant of sIMAP4), log into the Journaling mailbox at Gmail.com using the existing username and password for that specific account.

From within the Gmail application, click on the Settings cog icon in the top right corner of the screen.

From the pop-up dialog box that appears, click See all settings.

Click the tab titled Forwarding and POP/IMAP.

Scroll down to the section titled “IMAP Access”.

  1. If the status of the mailbox indicates that IMAP is disabled, click Enable IMAP. 

  2. If the status of the maibox indicates that IMAP is enabled, do not change any settings and cancel out of this Settings menu.

If IMAP was enabled in the step above, click Save Changes at the bottom of the screen.

sIMAP4 protocol access will now be enabled for the mailbox.

 

Activating 2-Step Verification in Gmail

To begin setting up Google’s 2-Step Verification, which is a requirement to have enabled before an Application-Specific password can be configured, select the User Identity icon (usually username initials in a colored field or a user-selected image) in the upper right hand corner of the screen.

This will spawn a submenu with multiple additional options.

Select “Manage your Google Account.”

The Google Account Management page will now render.

Select the Security tab on the left hand side of the screen.

Under the “How you sign in to Google” section, locate 2-Step Verification and click on the forward facing arrow.

Under the 2-Step Verification menu, click on the blue button for Turn On 2-Step Verification.

The default 2-Step Verification option Google offers is an SMS verification as a means of providing MultiFactor Authentication to the account (the password you know being the first factor; the device you possess being the second factor) and is the standard approach for this. Google Prompt, OTPs or Passkeys provide superior security and can also be used, but their nuances and setup procedures are beyond the scope of this article.

Enter in the phone number to a device you own and then click Next.

Google will ask to confirm that the phone number entered is accurate.

Verify that the phone number was correctly entered and then click Save.

Google may send an SMS prompt to your phone to verify by entering in the code they provide at this point. If they do, follow the instructions of entering in the provided code into the empty space they provide (the G-prefix can be skipped).

If they do not, 2-Step Verification setup is complete and the SMS prompt will only be needed for future sign-ins to sensitive areas of the account. Click Done to complete the process.

 

Creating an Application Specific Password for a Google Mailbox

With 2-Step Verification now configured, using the Google Account Search at the top of the screen, search for “app” and then select App Passwords from the menu of search results that spawns.

Under the App Passwords menu, add a descriptive name for the Application Password’s purpose (the recommendation is DataCove Email Archiving, so that it is instantly identifiable as to the use case), then click Create.

Google will automatically generate a random password and present it on screen. Copy this password and reserve it for use in the next steps on DataCove.

Note: The Application Specific Password will be entered without spaces when deploying it in the DataCove. The spaces provided in the key are simply for legibility purposes.

After copying the password, click Done to close the window.

 

Adding the Application Specific Password to a Fetcher in DataCove

With the Application Specific password now in hand, navigate to the DataCove web interface, log in with an administrative account and then select Configuration in the top header bar, followed by clicking POP or IMAP Fetchers on the left hand side menu.

Locate the Fetcher that points towards Imap.Gmail.Com and click on the yellow pencil icon to Edit it.

Under the Fetcher details page, locate the Password field and replace the existing password with the Application Specific password that was just generated.

Note: The Application Specific Password will be entered without spaces when deploying it in the DataCove. The spaces that were provided in the key from Google are simply for legibility purposes.

Once entered, click Save.

After saving the fetcher changes, DataCove will be back on the POP or IMAP Fetchers page.

With the password now entered, the final step will be to test the Fetcher to ensure it can get into the mailbox with the new credentials.

For this process, locate the green circled checkmark Test button, right next to the previously used Edit button, and click on it.

The Fetcher Testing subpage will now spawn.

This Test subpage will launch a battery of different tests to validate proper communication with the email server.

Connection is a simple network Ping test, verifying that the DataCove can reach the email server specified by FQDN or IP address. While it will also reference the Port number being used to connect in this section, that is actually being validated as part of the next test. Failure at this level mean the DataCove cannot resolve the FQDN of the mail server (suggesting a DNS issue, such as outdated or incorrect DNS entries on the DataCove’s network configuration or that those servers are not responding to DNS requests) or cannot reach the IP address, perhaps due to being on a disjointed network from the mail server or needing a Level 3 Network Switch-level authorization to send or receive packets from it.

Correct Protocol is affirming that the email server is responding over the Protocol and Port specified in the Fetcher configuration. Errors detected here suggest that the port may be getting blocked by a firewall or at the Managed Switch level.

Log In is ensuring that the Fetcher can log into the mailbox itself. Failing this test can mean an incorrect username or password, a locked account, an account that has another user logged into it currently (only an issue for POP3 and SPOP3 connections) and can sometimes also be rendered from the response delay from the mail server on extremely large mailboxes, with upwards of several hundred thousand messages waiting within.

Select Mailbox verifies that the specific folder that the Fetcher was instructed to reach is present. While nearly all fetchers will pursue the “Inbox” folder, errors or mailbox corruption that affect the Inbox folder can cause this test to fail. For Fetchers that are pursuing other specified folders, errors in the spelling of the folder name is the next most common culprit.

Messages in Mailbox is an identification of how many messages currently reside in the Journaling mailbox that need to be fetched. This is normally going to be some low amount of a few dozen to a few hundred emails at any given time, as the IMAP4 protocol runs its commands for listing, reading and deleting messages in batches, rather than individually the way POP3 does. For a low volume organization or when Journaled email is not being delivered to the account, this can show up with a warning notice that no emails were found in the account. For Journaling mailboxes that may not have been able to be logged into or fetched from for some time, this can also take time to render or report a failure if the query command times out. This latter issue of timeouts on large mailboxes does not necessarily mean they are critical failures and repeated retries will eventually yield results.

Select the Run Tests button to begin.

Within a few moments, the test battery will launch and test connectivity to the mailbox and run a count of the mail contained inside.

Click on the “Show All Details” link in the upper left corner to expand out the fields with full information.

If all was set up properly in the previous steps, and barring any outside factors like networking blockades or additional services being needed outside of the norm, the expected behavior is for all five fields to reflect in green success text. Any warnings or errors should have the Fetcher configuration information reviewed for accuracy and then a rerun of the test sequence until all fields return in green.

  • If the test has successfully completed, mail flow from Google to DataCove will resume within a few minutes as per normal.

  • If the test has failed and the Fetcher Configuration details are believed accurate, please contact DataCove Support for further investigation.

To validate that messages are being fetched once more, select Status in the top header bar, then click System Status on the left hand side menu, locating the “MailRetrievalService” (the internal name for a Fetcher) and clicking on the blue magnifying glass to observe its Details.

Note that the Fetcher will almost always be under the Active Jobs section, but in the brief moment when it takes a moment to “refresh” itself every few minutes with the mail server, it may appear under the Inactive Jobs section of the System Status page and can be found there.

The Fetcher’s Status screen provides a wealth of information about the Fetcher’s current activity and progress.

The most important fields here are:

Fetched This Hour reports on how many messages have been fetched within the current hour. This indicates how fast the DataCove and mail server communicating and generally should match up with how many messages are coming into the account within the same period of time.

Mailbox State shows the current messages inside the mailbox and any Download Failures. Messages inside the mailbox should never exceed more than a few thousand unless the “Delete after Backed-Up” setting was used with the Fetcher configuration, and if so, should never exceed more than 40,000 messages. Anything higher suggests a problem with deleting messages from the Journaling mailbox after fetching them or a backup problem. Download Failures are malformed, damaged, corrupt emails or emails that are not compliant with RFC822 that cannot be read properly by the Fetcher. Such messages are almost always very few and far between since mail servers are designed to recover and/or reject these already, but if any appear, they’ll be incremented on this counter and will reside in the Journaling mailbox indefinitely without any harm.

Activity displays what the Fetcher is doing currently. It could be resetting itself to clear off old counts, making an initial connection or, most commonly, it will be logged in and polling the Journaling mailbox for new messages to come in, downloading them and then deleting them. Any inability to connect or errors will be listed in this field and in the large white box below which contains additional connectivity details.

Fetched Today lists the total amount of messages pulled in by the Fetcher over the course of the entire day. This is largely informational and provides a rapid overview of the Fetcher’s daily workload.

In general, the Fetcher should always looks to have a similar status to what is pictured below, albeit with higher than zero numbers in the Fetched This Hour and Fetched Today fields, with it being logged into the Journaling mailbox and fetching away happily. With the successful receipt of emails now confirmed, configuration of the Application Specific Password in Google and DataCove is now complete.

Michael Singh
Next

Redacting Email Content