I am about to start implementing a notification system for my site and would like to know what performance measures I need to take into account before developing it.

I need to make this for a large amount of users (hundreds of thousands) so I need to be very careful with each step I make.

For each notification, I need to create a new term for a custom taxonomy (notifications) to display it on the user's notifications panel and send an email with wp_mail. On the other hand, instead of saving a new term for each notification, I could create a user meta with a meta_key called notifications and then save each notification in meta_value as an array, what do you think about this?

Now, lets say that user A is being followed 10,000 users (the following system has already been implemented). I need to notify all those 10,000 users that user A has posted something. How can I handle saving 10,000 terms (or user_meta) and sending 10,000 email just for that single action?

At the moment, I am thinking of using wp_schedule_event so I could throttle the action to 100 emails sent every 10 minutes.

But, what if 10 users make one post each at the same time and they have thousands of followers. What would be the best way to notify all those followers?

Or, I could just send an email to each user highlighting the top posts of the people they are following. This would certainly be lighter to process, what do you think?

Another thing, how expensive is it to send 10,000 or more emails each day?

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

5

sending 10k emails a day is a quick way to get your domain blacklisted, use extreme caution.
– MiloNov 26 '14 at 16:23

1

getting blacklisted can easily happen, even if you are just in the hundreds area, so preliminary steps to prevent it are pretty much are must
– NicolaiNov 26 '14 at 16:47

2 Answers
2

If you're looking to get in to instant notifications without suffering from backlog, I'd looking in to both Node.js and Socket.io. You should be able to work with them both pretty easily and integrate w/ WordPress. The sample chat application on the Socket.io site isn't a far cry from what you'd be doing.

Basically an event is triggered on your LAMP/LNMP stack, it hits the Node.js / Socket.io server on XXXX port, it responds or errors out. On success, you'd return the calls, thus alerting users, and on an error you'd simply log & hide them such from the public.

Performance wise, the reason for my suggestion is simply because of scale. Let's say that you have 100,000 users that are active and each trigger an alert just a few times a day, that's 200-300,000 calls over a 24 hour period of time. That's a LOT to queue in to a artificial cron system, or even the standard cron on your server. I wouldn't personally trust the cron to take care of it when there's a solution, like what I mentioned, that could easily serve that without much trouble at all, more so because you're not relying on a database, such as MySQL, on the scripting end. All that would be happening in quick logging for analytical purposes (success, errors, usernames, timestamps etc) and the processing of basic text requests. Authentication would be handled on your primary server with WordPress, so you'd essentially lock the Node.js server down to only accept and send responses.

As for e-mails, if you're going to be pushing 100,000 - potentially 1m e-mails a day, same-server e-mail shouldn't even be a consideration, nor should attempting to host it on your own. MailChimp runs Mandrill. The cost there for 500,000 a day would be a little over $100/day. With SendGrid, over $200/day and with MailGun, right at $245.00 a day. It's not cheap by far, but you could go at it on your own and receive 500,000 e-mails a day being returned back to you and as a result, your mail servers(s) being crippled due to the continuous and excessive load of trying to send & receive nearly the same load.

I've used both SendGrid and Mandrill. IMO, Mandrill is the better of the two. I've been with them for over a year now and never had an issue nor failed delivery/bounce. Simple and easy - set up the DKIM & SPF on your domain (they'll tell you what to use) and that's really all there is to it.

As for hitting the user / post meta, it's iffy. There's a really nice plugin called Advanced Custom Fields which utilizes post meta. For simple meta boxes, it's about as fast as anything else; posts go in, data comes out (although the data comes out quite a bit quicker than it goes in). When you start getting more complex, such as what ACF does, that's when you really need to decide whether the meta boxes or perhaps CPT's would be a better option.

With CPT's (Custom Post Types), you could simply utilize the content box as the primary field of storage. It already exists, there's no data serialization (which I believe happens with the meta boxes - I know it happens it happens in the options table) and it can be called using pre-existing queries, so you save yourself from writing new ones.

IMO, if you're dead set on using WordPress as the tool for interception, CPT's are going to be the way to go. At least with CPT's, you have the option to add meta to them.

As @Milo indicated. Be careful with sending too many emails or your IP will be banned/blacklisted.

Personally I would prefer to create a table as email queue and send them with interval with WP Cron API.

By default wp_mail() uses php mail() which is not much reliable and could end up in the Spam folder. Using SMTP for sending email is more reliable way to not end up in spam. It is also very important that your user know that you will be sending them email. Should be very clear to them, that way people hitting "report as spam" case will be minimum.

For more versatile solution you can look for a third party bulk emailing infrastructure. Amazon SES, Send Grid, Mail Gun etc. Create an Interface in your application that connects with those external service and send out the bulk email.