Monitoring Gmail Inbox Using Python

Automation

Many automation solutions make use of the functionality provided by mail services as it serves as an important element that allows for communication between humans and the automation process. There are many benefits provided by using Google Mail (Gmail), one of which is cost – for that reason, this post will focus on providing a step-by-step guide of how to monitor emails coming into your Gmail inbox, with the ability to monitor specific labels.

It is also important to note that there are tools and platforms that make it much easier to perform these actions but as developers, we know that life cannot always be “easy”. This post aims at empowering the “not easy” solutions.

What are the steps?

The steps that we will be following are:

  1. Ensure that your Gmail account is configured correctly
  2. Import the libraries
  3. Gather variable values
    1. Email address
    2. Password
    3. Label
    4. Search Criteria
  4. Define methods
    1. Get Body
    2. Search
    3. Get Emails
    4. Authenticate
  5. Authenticate
  6. Extract mails
  7. Extract relevant information from results

Deep Dive

Let’s dive deeper into the steps listed above.

Ensure that your Gmail account is configured correctly

My first few attempts at this left me pulling my hair out with an “Invalid Credentials” error. Upon much Googling and further investigation, I found that it is caused by a Google Account setting. This is quite easily fixable.

In order to interact with my account, I had to allow less secure apps (you can access that setting here):

Allowing less secure apps to communicate with my Gmail account

Allowing less secure apps to communicate with my Gmail account

If you are still experiencing problems, here is a more extensive list of troubleshooting tips.

Import the libraries

Now let’s move over to Python and start scripting!

First, let’s import the libraries that we’ll need:

import imaplib, email

Gather variable values

In order to access the mails from the Gmail account we will need to know the answers to the following questions:

  1. Which Google account (or email address) do we want to monitor?
  2. What is the password to the above account?
  3. What label do we want to monitor?
  4. What is the search criteria?

The best way to find out is to ask and luckily we can do that through code:

imap_url = 'imap.gmail.com' # This is static. We don't ask the questions we know the answer to

user = input("Please enter your email address: ")
password = input("Please enter your password: ")
label = input("Please enter the label that you'd like to search: ") # Example: Inbox or Social
search_criteria = input("Please enter the subject search criteria: ")

Define Methods

It becomes easier to break some of the reusable elements up into methods (or functions) so that larger implementations of this solution are equipped to be easily scalable. Stephen Covey teaches us that starting with the end in mind is one of the habits of highly effective people – some might even call it proactive design thinking. The point is that it is good to think ahead when developing a solution.

Enough rambling, here are the functions:

# Retrieves email content
def get_body(message): 
    if message.is_multipart(): 
        return get_body(message.get_payload(0)) 
    else: 
        return message.get_payload(None, True) 
    
# Search mailbox (or label) for a key value pair
def search(key, value, con):  
    result, data = con.search(None, key, '"{}"'.format(value)) 
    return data 
  
# Retrieve the list of emails that meet the search criteria
def get_emails(result_bytes): 
    messages = [] # all the email data are pushed inside an array 
    for num in result_bytes[0].split(): 
        typ, data = con.fetch(num, '(RFC822)') 
        messages.append(data) 
  
    return messages 

# Authenticate
def authenticate(imap_url, user, password, label):
    
    # SSL connection with Gmail 
    con = imaplib.IMAP4_SSL(imap_url)  

    # Authenticate the user through login
    con.login(user, password)  

    # Search for mails under this label
    con.select(label)

    return con

Authenticate

Before we can extract mails, we first need to call the authenticate method that we had just created and pass through the answers to the questions we asked further up:

con = authenticate(imap_url, user, password, label)

Extract mails

Next, we need to call the search and get_emails methods to extract the mails:

# Retrieve mails
search_results = search('Subject', search_criteria, con)
messages = get_emails(search_results) 
  
# Uncomment to view the mail results
#print(messages)

Extract relevant information from results

Now, let’s work through the results and extract the subject using string manipulation. Feel free to add a “print(subject)” statement underneath the assignment of “subject” for debugging purposes:

for message in messages[::-1]:  
        
    for content in message: 
                       
        if type(content) is tuple:  
  
            # Encoding set as utf-8 
            decoded_content = str(content[1], 'utf-8')  
            data = str(decoded_content) 
            
            # Extracting the subject from the mail content
            subject = data.split('Subject: ')[1].split('Mime-Version')[0]
                                      
            # Handling errors related to unicodenecode 
            try:  
                indexstart = data.find("ltr") 
                
                data2 = data[indexstart + 5: len(data)] 
                
                indexend = data2.find("</div>") 
                
                # Uncomment to see what the content looks like
                #print(data2[0: indexend]) 
  
            except UnicodeEncodeError as e: 
                pass

An Update on Gmail Authentication

Since this post was written, Google has retired the ‘less secure apps’ setting for personal Gmail accounts, so the setting shown above is no longer available. To keep using IMAP with the script above, turn on 2-Step Verification for your Google account and generate an App Password, then use that App Password in place of your normal password.

Alternative: Using the Gmail API

If you would rather not use IMAP at all, the Gmail API is another option. It uses OAuth 2.0 instead of a password and lets you search with the same query syntax you would use in the Gmail search bar (for example, label:Inbox subject:Invoice).

First, enable the Gmail API in the Google Cloud Console, download the OAuth client credentials as credentials.json and install the libraries:

pip install google-api-python-client google-auth-httplib2 google-auth-oauthlib

Then authenticate and list the messages that match your criteria:

from google_auth_oauthlib.flow import InstalledAppFlow
from googleapiclient.discovery import build

SCOPES = ['https://www.googleapis.com/auth/gmail.readonly']

# Opens a browser window to sign in and grant read-only access
flow = InstalledAppFlow.from_client_secrets_file('credentials.json', SCOPES)
creds = flow.run_local_server(port=0)

service = build('gmail', 'v1', credentials=creds)

# Search using Gmail's search syntax
query = 'label:{} subject:"{}"'.format(label, search_criteria)
results = service.users().messages().list(userId='me', q=query).execute()

for msg in results.get('messages', []):
    message = service.users().messages().get(userId='me', id=msg['id'], format='metadata').execute()
    headers = message['payload']['headers']
    subject = next((h['value'] for h in headers if h['name'] == 'Subject'), '')
    print(subject)

Did this work for you? Feel free to drop a comment below or reach out to me through email, jacqui.jm77@gmail.com.

The full Python script is available on Github here.