Analyze Amazon SES occasions at scale utilizing Amazon Redshift

E mail is without doubt one of the most essential strategies for enterprise communication throughout many organizations. It’s additionally one of many major strategies for a lot of companies to speak with their prospects. With the ever-increasing necessity to ship emails at scale, monitoring and evaluation has change into a serious problem.

Amazon Easy E mail Service (Amazon SES) is a cheap, versatile, and scalable e-mail service that lets you ship and obtain emails out of your functions. You should utilize Amazon SES for a number of use instances, comparable to transactional, advertising and marketing, or mass e-mail communications.

An essential good thing about Amazon SES is its native integration with different AWS providers, comparable to Amazon CloudWatch and Amazon Redshift, which lets you monitor and analyze your emails sending at scale seamlessly. You possibly can retailer your e-mail occasions in Amazon Redshift, which is a extensively used, quick, and absolutely managed cloud knowledge warehouse. You possibly can then analyze these occasions utilizing SQL to realize enterprise insights comparable to advertising and marketing marketing campaign success, e-mail bounces, complaints, and so forth.

On this publish, you’ll discover ways to implement an end-to-end answer to automate this e-mail evaluation and monitoring course of.

Answer overview

The next structure diagram highlights the end-to-end answer, which you’ll provision robotically with an AWS CloudFormation template.

On this answer, you publish Amazon SES e-mail occasions to an Amazon Kinesis Information Firehose supply stream that publishes knowledge to Amazon Redshift. You then hook up with the Amazon Redshift database and use a SQL question software to investigate Amazon SES e-mail occasions that meet the given standards. We use the Amazon Redshift SUPER knowledge kind to retailer the occasion (JSON knowledge) in Amazon Redshift. The SUPER knowledge kind handles semi-structured knowledge, which may have various desk attributes and kinds.

The alarm system makes use of Amazon CloudWatch logs that Kinesis Information Firehose generates when an information load to Amazon Redshift fails. We’ve got arrange a metric filter that sample matches the CloudWatch log occasions to find out the error situation and triggers a CloudWatch alarm. This in flip sends out e-mail notifications utilizing Amazon Easy Notification Service (Amazon SNS).

Conditions

As a prerequisite for deploying the answer on this publish, you want to arrange Amazon SES in your account. For extra data, see Getting Began with Amazon Easy E mail Service.

Answer assets and options

The structure constructed by AWS CloudFormation helps AWS finest practices for top availability and safety. The CloudFormation template takes care of the next key assets and options:

• Amazon Redshift cluster – An Amazon Redshift cluster with encryption at relaxation enabled utilizing an AWS Key Administration Service (AWS KMS) buyer managed key (CMK). This cluster acts because the vacation spot for Kinesis Information Firehose and shops all of the Amazon SES e-mail sending occasions within the desk ses, as proven within the following screenshot.
  
• Kinesis Information Firehose configuration – A Kinesis Information Firehose supply stream that acts because the occasion vacation spot for all Amazon SES e-mail sending metrics. The supply stream is ready up with Amazon Redshift because the vacation spot. Server-side encryption is enabled utilizing an AWS KMS CMK, and vacation spot error logging has been enabled as per finest practices.
  
• Amazon SES configuration – A configuration set in Amazon SES that’s used to map Kinesis Information Firehose because the occasion vacation spot to publish e-mail metrics.
  

To make use of the configuration set when sending emails, you possibly can specify a default configuration set to your verified identification, or embody a reference to the configuration set within the headers of the e-mail.

• Exploring and analyzing the information – We use Amazon Redshift question editor v2 for exploring and analyzing the information.
  
• Alarms and notifications for ingestion failures – A knowledge load error notification system utilizing CloudWatch and Amazon SNS generates email-based notifications within the occasion of a failure throughout knowledge load from Kinesis Information Firehose to Amazon Redshift. The setup creates a CloudWatch log metric filter, as proven within the following screenshot.
  

A CloudWatch alarm based mostly on the metric filter triggers an SNS notification when in alarm state. For extra data, see Utilizing Amazon CloudWatch alarms.

Deploy the CloudFormation template

The supplied CloudFormation template robotically creates all of the required assets for this answer in your AWS account. For extra data, see Getting began with AWS CloudFormation.

1. Register to the AWS Administration Console.
2. Select Launch Stack to launch AWS CloudFormation in your AWS account:
   
3. For Stack identify, enter a significant identify for the stack, for instance, ses_events.
4. Present the next values for the stack parameters:
   1. ClusterName – The identify of the Amazon Redshift cluster.
   2. DatabaseName – The identify of the primary database to be created when the Amazon Redshift cluster is created.
   3. DeliveryStreamName – The identify of the Firehose supply stream.
   4. MasterUsername – The consumer identify that’s related to the first consumer account for the Amazon Redshift cluster.
   5. NodeType – The kind of node to be provisioned. (Default dc2.massive)
   6. NotificationEmailId – The e-mail notification record that’s used to configure an SNS subject for sending CloudWatch alarm and occasion notifications.
   7. NumberofNodes – The variety of compute nodes within the Amazon Redshift cluster. For multi-node clusters, the NumberofNodes parameter have to be better than 1.
   8. OnPremisesCIDR – IP vary (CIDR notation) to your current infrastructure to entry the goal and reproduction Amazon Redshift clusters.
   9. SESConfigSetName – Title of the Amazon SES configuration set.
   10. SubnetId – Subnet ID the place supply Amazon Redshift cluster is created.
   11. Vpc – VPC by which Amazon Redshift cluster is launched.
5. Select Subsequent.
   
6. Overview all the data and choose I acknowledge that AWS CloudFormation would possibly create IAM assets.
7. Select Create stack.
   

You possibly can monitor the progress of the stack creation on the Occasions tab. Await the stack to finish and present the standing CREATE_COMPLETE.

Take a look at the answer

To ship a check e-mail, we use the Amazon SES mailbox simulator. Set the configuration-set header to the one created by the CloudFormation template.

We use the Amazon Redshift question editor V2 to question the Amazon Redshift desk (created by the CloudFormation template) and see if the occasions have proven up.

If the information load of the occasion stream fails from Kinesis Information Firehose to Amazon Redshift, the failure notification system is triggered, and also you obtain an e-mail notification by way of Amazon SNS.

Clear up

Among the AWS assets deployed by the CloudFormation stacks on this publish incur a price so long as you proceed to make use of them.

You possibly can delete the CloudFormation stack to delete all AWS assets created by the stack. To wash up all of your stacks, use the AWS CloudFormation console to take away the stacks that you just created in reverse order.

1. On the Stacks web page on the AWS CloudFormation console, select the stack to delete.
2. Within the stack particulars pane, select Delete.
3. Select Delete stack when prompted.

After stack deletion begins, you possibly can’t cease it. The stack proceeds to the DELETE_IN_PROGRESS state. When the stack deletion is full, the stack adjustments to the DELETE_COMPLETE state. The AWS CloudFormation console doesn’t show stacks within the DELETE_COMPLETE state by default. To show deleted stacks, you should change the stack view filter. For extra data, see Viewing deleted stacks on the AWS CloudFormation console.

If the delete fails, the stack enters the DELETE_FAILED state. For options, see Delete stack fails.

Conclusion

On this publish, we walked by the method of establishing Amazon SES and Amazon Redshift to deploy an e-mail reporting service that may scale to assist hundreds of thousands of occasions. We used Amazon Redshift to retailer semi-structured messages utilizing the SUPER knowledge kind in database tables to assist various message sizes and codecs. With this answer, you possibly can simply run analytics at scale and analyze your e-mail occasion knowledge for deliverability-related points comparable to bounces or complaints.

Use the CloudFormation template supplied to hurry up provisioning of the cloud assets required for the answer (Amazon SES, Kinesis Information Firehose, and Amazon Redshift) in your account whereas following safety finest practices. Then you possibly can analyze Amazon SES occasions at scale utilizing Amazon Redshift.

Concerning the Authors

Manash Deb is a Software program Growth Engineer within the AWS Listing Service crew. He has labored on constructing end-to-end functions in numerous database and applied sciences for over 15 years. He likes to study new applied sciences and fixing, automating, and simplifying buyer issues on AWS.

Arnab Ghosh is a Options Architect for AWS in North America serving to enterprise prospects construct resilient and cost-efficient architectures. He has over 13 years of expertise in architecting, designing, and growing enterprise functions fixing advanced enterprise issues.

Sanjoy Thanneer is a Sr. Technical Account Supervisor with AWS based mostly out of New York. He has over 20 years of expertise working in Database and Analytics Domains.  He’s enthusiastic about serving to enterprise prospects construct scalable , resilient and value environment friendly Purposes.

Justin Morris is a E mail Deliverability Supervisor for the Easy E mail Service crew. With over 10 years of expertise within the IT trade, he has developed a pure expertise for diagnosing and resolving buyer points and constantly appears to be like for development alternatives to study new applied sciences and providers.