Primary Navigation

Re: slow delivery to yahoo

... In that case, you should post summary statistics from the delays=a/b/c/d log entries for yahoo. The a and b numbers don t really matter in this

Message 1 of 7
, Oct 15, 2008

0 Attachment

On Wed, Oct 15, 2008 at 03:48:18PM -0400, Ofer Inbar wrote:

> Victor Duchovni <Victor.Duchovni@...> wrote:
> > > So I wonder if any of you administer high volume postfix sites and
> > > have run into the same problem, and if you've found any workarounds.
> >
> > The work-around is to get on the Yahoo whitelist.
>
> I should've mentioned that all servers in question are whitelisted.
>
> Obviously if they weren't they'd be having problems with the other
> large ISPs too, so I thought it was implicit (large volume sites get
> on the relevant whitelists or they just don't work). Sorry.

In that case, you should post summary statistics from the

delays=a/b/c/d

log entries for yahoo. The "a" and "b" numbers don't really matter in
this context, but "c" and "d" are critical. Simple averages distort the
picture because one slow message can hide a lot of fast messages. So I
tend to compute "moving averages" with slow exponential decay, yielding
output similar to the below:

So, it looks a single connection to Yahoo! takes approximately 5 seconds
to setup, 5 seconds to deliver each message, and (if your message rate
is high enough) capable of delivering up to 5 messages. So for me,
maximal throughput per unit of concurrency is:

msgs/sec = 5 / (5 + 5 * 5) = 1/6

If my destination concurrency to Yahoo! is the default 20, (not sure
they let you make that many connections from the same IP) I can only
deliver 20/6 ~= 3 msgs/sec per sending host.

To send 100 msgs/sec, I would need ~30 hosts. Fortunately, I don't have
nearly that much demand for mail to Yahoo!

--
Viktor.

Disclaimer: off-list followups get on-list replies or get ignored.
Please do not ignore the "Reply-To" header.

If my response solves your problem, the best way to thank me is to not
send an "it worked, thanks" follow-up. If you must respond, please put
"It worked, thanks" in the "Subject" so I can delete these quickly.

Ofer Inbar

... I don t have delays= in my logs, only delay=. I think this is because the setup I m administering now is running postfix-2.2 (yes, I want to upgrade them,

Message 2 of 7
, Oct 16, 2008

0 Attachment

> In that case, you should post summary statistics from the
>
> delays=a/b/c/d

I don't have delays= in my logs, only delay=. I think this is because
the setup I'm administering now is running postfix-2.2 (yes, I want to
upgrade them, but this won't happen for at least a month).

However, independently I came to the same conclusion that concurrency
per-site is the limiting factor. By setting up a separate transport
for yahoo messages only, we were able to increase our delivery rate
from a couple of hundred messages per minute, to a few thousand / min.

Yahoo told us that we were not anywhere close to sending them messages
too quickly, and we should be able to send more quickly, but it's just
a slower process to deliver to yahoo than to the other large ISPs, it
seems, so sending more messages at a time is apparently the solution.

I had previously tried testing by changing the concurrency limit from
20 to 40 but that had a small effect, if any. A large change has a
dramatic effect.

-- Cos

Barney Desmond

... I find this quite interesting. Based on the vague guidelines that Yahoo publishes, it sounds like they consider lots of concurrent connections to also be

Message 3 of 7
, Oct 16, 2008

0 Attachment

Ofer Inbar wrote:

> Yahoo told us that we were not anywhere close to sending them messages
> too quickly, and we should be able to send more quickly, but it's just
> a slower process to deliver to yahoo than to the other large ISPs, it
> seems, so sending more messages at a time is apparently the solution.
>
> Settings for this yahoo-specific transport:
> yahoo_destination_concurrency_limit = 500
> yahoo_connect_timeout = 5s
> yahoo_data_done_timeout = 120s
> yahoo_data_init_timeout = 60s
> yahoo_helo_timeout = 120s
> yahoo_mail_timeout = 120s
> yahoo_quit_timeout = 120s
> yahoo_rcpt_timeout = 120s
> yahoo_rset_timeout = 120s
>
> I had previously tried testing by changing the concurrency limit from
> 20 to 40 but that had a small effect, if any. A large change has a
> dramatic effect.

I find this quite interesting. Based on the vague guidelines that Yahoo
publishes, it sounds like they consider lots of concurrent connections
to also be abusive. If you can really get away with such high
concurrency it may well be worth trying out.

You mentioned going from a couple of hundred messages per minute, to a
few thousand per minute; are these mostly Yahoo addresses, or was mail
to Yahoo just holding up the rest of the queue? I don't suppose you have
any messages-per-minute figures for that yahoo-specific transport?

Victor Duchovni

... These are fine (provided your queue manager does not run out of file descriptors, you really should move to Postfix 2.5 on a system with epoll, devpoll or

If my response solves your problem, the best way to thank me is to not
send an "it worked, thanks" follow-up. If you must respond, please put
"It worked, thanks" in the "Subject" so I can delete these quickly.

Your message has been successfully submitted and would be delivered to recipients shortly.