Weird thoughts from a techie brain...

August 2017

Sun

Mon

Tue

Wed

Thu

Fri

Sat

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

About

This site is an effort to share some of the base knowledge I have gathered through all this years working with Linux, FreeBSD, OpenBSD, Python or Zope, among others. So, take a look around and I hope you will find the contents useful.

Sentry email notifications not arriving?

I've been using Sentry for quite some time now in some projects, mostly all
of them running python based applications and systems, and recently I did a
major upgrade of a Sentry server to the latest version (8.3.1). It was an
update with lots of changes, both in the Sentry internals and the dependencies
needed by it (like the addition of Redis) but everything went more or less
smoothly.

Once I had the server ready, I did upgrade the Raven client accordingly and
I modified something in our staging server so an exception (IOError
actually) was raised and the proper notification was sent to the Sentry server.

It worked just fine, the error message was sent to the server and it appeared
on the Sentry web interface. Just perfect... or not?

Not really. After the error arrived in the Sentry server, a notification should
have been sent to a mailing list where developers/support would see it and act
accordingly. But that email never arrived and there was no error on the Sentry
server logs.

Which basically means there was an incoming connection which was suddently
closed by the client.

Weird.

So, what was happening? Good question indeed. First thing to try was the email
testing tool that comes bundled with Sentry. In the admin panel in the web
interface, there is a Mail section, at the bottom of the page you can find
such tool.

I clicked there, nothing on the sentry logs, same lines in the SMTP server logs.
Not really useful.

And so I started to look at Sentry's source code, to see how those emails are
sent. I ended up looking at this file:

Please note I'm linking to the files for versions 8.1.3 of sentry, which
has django 1.6.11 as a dependency at the moment of writing this article.

A bit further down the chain, django uses smtplib.SMTP from the base
library of python 2. Now, looking at the documentation, it mentions a socket
timeout:

" ... The optional timeout parameter specifies a timeout in seconds for
blocking operations like the connection attempt (if not specified, the global
default timeout setting will be used). If the timeout expires, socket.timeout
is raised. ... "

Interesting. now, let's try to run Sentry's email sending code in a python
shell/console. I'm running sentry inside a virtualenv, so after activating
the environment, I start a shell and give it a try:

At the same time, I was able to see the usual lines in the SMTP server logs.

So, indeed there was a time out. Now, let's check the SMTP configuration. That
server runs sendmail, which has an interesting feature called greet_pause,
which basically makes the SMTP to hold on for some time before sending the
greeting message to the client, after the initial connection. This is a nice
spam protection measure, as most spam bots do not wait for that greeting
message before starting to send commands, and sendmail will simply drop the
connection if the client do not wait for that greeting message before start
talking.

But this was not the case here, the SMTP logs should show a total different
thing for that, something like

Exactly 5 seconds too. So, basically, python's smtplib was opening a connection
with a time out of 5 seconds and sendmail was waiting for 5 seconds after the
initial connection was established to send that greeting message. Obviously,
python's smtplib was correctly and politely waiting for that greeting message
before sending any commands to the SMTP server... but by the time the greeting
message was about to be sent by sendmail, python closed the connection with
a timeout.

WOW

Now, how to fix this?, easy, raise that timeout a bit on the client side.
All we need is to use setdefaulttimeout to adjust it. As sentry is a
django-based app, and still has a django-style configuration file, which is
a python script that is loaded every time you run the app, all we need is to
add the following two lines to sentry.conf.py

# Set the default socket timeout to a value that prevents connections# to our SMTP server from timing out, due to sendmail's greeting pause# feature.importsocketsocket.setdefaulttimeout(10)

(Yeah, better put a good comment there, for the next guy looking into the
config file and wondering why the f*** we set that timeout there)

In this example I've raised it to 10 seconds, adjust that value to whatever
fits for you.

A restart of the sentry server and et voilà, email notifications were back!