Server and System Status

Our scheduled maintenance to increase the storage space available for email accounts on the mail server (NC036) is complete. The ability to send and receive email was not available between 19:17 and 19:44 UTC. The expanded disk space has been tested and all is running normally.

During this weekend’s maintenance window we will be adding hard drive storage to server NC036 to continue to provide more storage space for a growing number of growing email accounts. This maintenance is scheduled to start at 19:00 UTC and we anticipate it will last less than one hour.

During the maintenance the ability to send and receive email will not be available, both via standalone email programs (e.g., Outlook, Thunderbird, etc.) and the webmail. Incoming email will be queued on the sending servers until our server is back online again, after which it will then be delivered to our server and your email account. This may result in a delay longer than the planned hour of the maintenance though.

Please monitor this status page to be notified of the start and end of the maintenance. If you have any questions or concerns, please contact NinerNet support.

Thank-you for your patience as we continue to work to improve our services to you.

We are aware that the IP address of server NC036 (the primary mail server) has again been blocked by Microsoft’s various mail services, variously known as Outlook.com, MSN, Hotmail, Live.com, etc.

Although we are a member of their Smart Network Data Services programme and Junk Mail Reporting Program, which are supposed to allow us to proactively prevent these kinds of issues, we have been unable to use the service as advertised, or at least as we understand it’s supposed to work. We will continue to attempt to have this server’s IP address removed from their blacklist, and report here when we have success.

In the meantime, outgoing mail to their primary domains (hotmail.ca, hotmail.com, hotmail.co.uk, live.com, msn.com and outlook.com) is being routed through our relay server. If you receive a bounce message that reads similarly the one below to an email you’ve sent, it is probably for a private domain hosted by Microsoft of which we are not aware. Please contact us and we will add it to the list of domains for which email is routed through our relay server:

host 901e3cd0af6f44ab11b5a5e8a49da3.pamx1.hotmail.com[104.47.0.33] said: 550
5.7.1 Unfortunately, messages from [178.62.195.26] weren’t sent. Please
contact your Internet service provider since part of their network is on
our block list (S3140). You can also refer your provider to
http://mail.live.com/mail/troubleshooting.aspx#errors.
[HE1EUR01FT033.eop-EUR01.prod.protection.outlook.com] (in reply to MAIL
FROM command)

Please remember that all email you send through our mail server must be to recipients with whom you already have a business or personal relationship, and all mass email must be explicitly requested — i.e., Confirmed opt-in (COI) or Double opt-in (DOI) email.

Thanks for your cooperation, and our apologies for this inconvenience.

We planned the migration to have absolutely no impact on existing email configurations. We did this by pointing legacy sub-domains of the niner.net domain that named server NC027 — e.g., smtp27.niner.net — to server NC036. At the conclusion of the migration these sub-domains were indeed pointing to the new server. In other words, on Monday morning (4 June) email programs would have thought they were still downloading mail from the same server, not realising (or needing to realise) that they were in fact downloading from a new server.

However, it turned out that a significant minority of email programs were somehow misconfigured with settings that worked on the old server, but stopped working when connecting to the new server. Those clients who were using the correct settings experienced no disruption at all, and when those clients with incorrect settings corrected them on the morning of Monday the 11th, the problems were fixed instantly.

Over the rest of that week (11-15 June) we helped a few clients with some issues unique to how they use email, especially where those practices clashed with current best practices for email transmission. We also dealt with some issues of senders whose mail servers were behaving improperly, causing their emails to be blocked because they looked like spammers. This notably affected email from the ZRA, but their emails are once again flowing unimpeded.

We’re monitoring the spam filtering on the new server. Any message that the server identifies as spam will have the subject of the message prefixed to add “[SPAM]“. You can use this to configure your email program or the webmail to deal with spam automatically, by filtering it into your “junk” folder or deleting it entirely. We recommend filtering to the junk folder so that you can catch the occasional legitimate message that is misclassified as spam.

Finally, in recognition of the fact that the emergency migration of the server to a new data centre on 6 June disrupted all clients’ email, and the fact that those clients with misconfigured email programs experienced a week of disruption before the issue was identified, we will be applying a one-week (quarter month) credit to the accounts of all clients hosted on server NC036. We apologise for the difficulties caused, and will apply what was learned this time to future migrations.

There are two reasons why you may be getting the above error in response to messages you’ve sent to addresses on domains hosted by NinerNet, likely your own domain:

It may be because you’re sending from an address on a domain that we host, but instead of sending your email through our SMTP server (smtp.niner.net) you’re sending through another SMTP server, possibly that of an ISP or another email service provider. In some cases this can happen because of a situation similar to that described in the sixth bullet point of our post “NC036: Migration update 20 — Solutions“, where you’ve sent the email through a third party, perhaps an ISP, or an email account you have with another provider.

If you’re using some cloud-hosted application that tries to send email to you as you (or another user on your own domain), then that email looks like spam to the mail server, because lots of spammers mistakenly try to get their email through by sending their spam from your email address to your email address, or from another address on your own domain to you.

The solutions are, respectively (and respectfully):

Configure your email program to use smtp.niner.net to send email from any domain that we host. If you’re following the configuration instructions we send you, then that is the case by default, and always has been.

Have the provider of the cloud service send those emails from an address — even a “no-reply” address — on their own domain, or use SMTP AUTH to send the email through smtp.niner.net from an address on your own domain, just as you or any other human with an address on your domain would.

Here are the promised screenshots that show how an email program like Thunderbird should be configured.

This is the initial screen when you manually add a new account to Thunderbird. (We will have automatic configuration available by next week.) For POP instead of IMAP, please just select POP from the drop-down list to the right of the “Incoming:” heading, where “IAMP” shows in this screenshot.

These are the advanced incoming (IMAP) settings. Nothing materially different here, just more settings that we suggest.

SMTP servers. Here it is possible that you may have too many or too few SMTP servers. If you’re having problems sending, we suggest that you delete any superfluous SMTP servers and create one SMTP server that corresponds with each incoming email account you have.

Summary

We suspect that clients having problems sending or receiving email have very old legacy configuration settings. Please see the “Email server settings” section below for the definitively correct settings.

Situation

Over the weekend we took a deep breath and stepped back to re-analyse this problem, and consult with a number of you. Between…

a move to a new server in a new data centre,

and then to another data centre to try to outrun the phantom issues at the first data centre,

and the fact that the new server was somehow processing just as many messages as it normally does despite so many people apparently being unable to send and/or receive,

.. we were awash in red herrings to an extent I have never seen in 22 years.

We’ve taken a look at the behaviour of two of the most used email programs (Thunderbird and Outlook) and come to some conclusions about what might be happening:

The fact that most clients carried on connecting with no problems tells us that (a) the server was operating normally, but (b) some clients were using old (in some cases very old) settings that were permitted (but not recommended) on the old server, but no longer permitted on the new server due to the ever-increasing need to raise the bar on server security.

Some email programs (notably Thunderbird and various incarnations of the Apple Mail app) tend to funnel all outgoing email through a single SMTP (outgoing) account. This can lead to situations where someone might be trying to send an email from you@domain1.com, but trying to log in as other-address@domain2.net. Again, with the ongoing need to tighten email security, this is no longer permissible with just about every mail service provider in the world.

A lot (probably most, actually) of email programs and apps try to second-guess your selection of a port number, often after you think you’ve saved your email configuration.

Over the years we’ve seen some email programs and apps treat SSL and TLS in odd and unpredictable ways. The existing settings we’ve always given out still work, but in the interests of getting everyone on the same page we’re starting with a clean slate.

So, if you’re having problems sending, it will likely be worth your while to check your SMTP (outgoing) settings; if you’re having problems receiving, it will likely be worth your time to check your POP or IMAP (incoming) settings. I wanted to have some screenshots ready for this post, but I’d rather get the words up now and post screenshots shortly afterwards, so here are the settings you need to use:

Email server settings

Email address: you@yourdomain.com

User name: you@yourdomain.com (same as your email address)

Password: The correct password on your email account. If you’re not sure what it is, please reset it to a new one through the email control panel (admins only). It can also be reset through the webmail.

Password type: Plain

Incoming (POP/IMAP) mail:

Server: pop.niner.net or imap.niner.net

Port: 110 (POP) or 143 (IMAP)

Encryption: STARTTLS

Outgoing (SMTP) mail:

Server: smtp.niner.net

Port: 587

Encryption: STARTTLS

Authentication: Turned on

To send email, you must log in with the same user name (address) as the address you’re sending from.

Some older mail programs may not offer STARTTLS; if that’s the case for you, try TLS and/or SSL, in that order.

Additional information

I can’t emphasise strongly enough how important it is for you to be precise in setting up this configuration. No setting is “close enough”, and your computer is not smart enough to figure it out; it will just tell you there is an error. Although, having said that, I’d like to emphasise that the niner.net sub-domains with “27” in them — i.e., pop27.niner.net, imap27.niner.net and smtp27.niner.net — do still also work, but they will be phased out; do not use them.

In the case of those email programs that like to railroad you into sending all email through a single SMTP account by default, we suggest that you start with a clean slate there too by deleting all of the saved SMTP accounts (unless you have some on systems that are completely separate from NinerNet) and creating a new one for each of your email accounts. Because your email program may not let you delete the “default” SMTP account, you’ll need to make a new SMTP account the new default, and then delete the old default.

We will post helpful screenshots as soon as possible. In the meantime, please check (and, if necessary, update) your email account settings and ensure that they are correct.

I have just got off the phone with someone in IT security at MTN head office in Lusaka, and they confirm that they have been blocking our new mail server as part of a wrong-headed plan to prevent MTN users from sending spam. It is likely that the first new mail server was also being actively blocked. He says that our IP addresses will be unblocked within the next ten minutes.

This raises the significant question of whether or not this is now an Africa-wide policy with many other ISPs. Other countries manage to prevent their users from sending spam without holding the keys to a gateway to the Internet, forcing companies like NinerNet to supplicate themselves to the likes of big companies like MTN when we find our businesses held hostage.

This is why we sent the questionnaire out yesterday asking you for details on whether nor not you are still having problems, and for the details of your ISP. Please reply to those emails so that we may determine which ISPs are actively blocking our servers and take the appropriate action.

This blog provides information about the status of NinerNet Communications systems. Dates and times of posts to this blog are in the UTC time zone, and dates and times given for events are also in the UTC time zone, although conversions may be offered for some time zones common to our clients. Please use the World Time Server to ensure accurate conversion of dates and times to your own time zone.