Archive

If you are using sendgrid as a service for your outbound email, you would want to monitor and be able to answer questions such as:

how much email are you sending

status of sent email – success, bounced, delayed, etc.

trends

etc.

We get questions all the time from $WORK customer support folks on whether an email sent to a customer got there (customer claimed they never got it). There could be any number of reasons why customer do not see email sent from us.

our email is filtered into customer spam folder

email is reject/bounced by customer mail service

any number of network/server/services related errors between us and customer mail service

the email address customer provided is invalid (and email bounced)

If we have access to event logs from sendgrid, we would be able to quickly answer these types of questions.

SendGrid’s Event Webhook will notify a URL of your choice via HTTP POST with information about events that occur as SendGrid processes your email. Common uses of this data are to remove unsubscribes, react to spam reports, determine unengaged recipients, identify bounced email addresses, or create advanced analytics of your email program. With Unique Arguments and Category parameters, you can insert dynamic data that will help build a sharp, clear image of your mailings.

Login to your sendgrid account and click on Mail Settings.

Then click on Event Notification

In HTTP Post URL, enter the DNS name of the service endpoint you are going to setup next.

For example, mine is (not a valid endpoint, but close enough): https://sendlog.mydomain.com/logger

Next is to edit and configure sendgrid-event-logger (SEL for short). If the default config works for you, then no need to do anything. Check and make sure it is pointing to where your ES host is located (mine is running on the same instance, hence localhost). I also left SEL listening on port 8080 as that is available on this instance.

FAQs

How to fix corrupted elasticsearch translog.

In 5.0 there is a tool which can be used to truncate corrupt translog files. This doesn't exist in 2.x but there is a workaround:
POST my_index/_close
PUT my_index/_settings
{ "index.engine.force_new_translog": true }
POST my_index/_open
PUT my_index/_settings
{ "index.engine.force_new_translog": false }
NOTE: Any data in the corrupted translog will be lost.

How to size a cluster?

I want to create a new Elasticsearch cluster. What are the recommended sizing guidelines?
Answer:
This is very much a use case dependent answer. The factors that should be taken into considerations are:

How much data do you expect to index?

Frequency of new data. How often is new data to be indexed? Daily? Hourly?