Sfitz: From Spark this morning ... it looks like you may be using a Netcomm router, is that correct? If so, some of these modems have beenblocked from checking DNS due to being vulnerable to attacks like we had on the weekend. Applying DNS to each device should still allow each device to get online.

My problem with Spark is they haven't published that they have blocked users and provide more detailed information. Instead they have just left users not working and announce in the media all is good.Setting Google DNS in the router rather than Spark ones does not resolve the problem as it would appear the router itself is blocked from DNS however setting up your own internal DNS or setting DNS on each device does work.

Mines a Dynalink RTA 1025W and I'm in the same boat, so it's not just the Netcomms. I 100% agree without about the announcement that it's all working again, while some users are just sitting in internet darkness.

Same findings about the end device DNS, though to be honest - normal service could be restored but since I've already set all my devices to Google DNS, I haven't bothered testing.

Sfitz: From Spark this morning ... it looks like you may be using a Netcomm router, is that correct? If so, some of these modems have beenblocked from checking DNS due to being vulnerable to attacks like we had on the weekend. Applying DNS to each device should still allow each device to get online.

My problem with Spark is they haven't published that they have blocked users and provide more detailed information. Instead they have just left users not working and announce in the media all is good.Setting Google DNS in the router rather than Spark ones does not resolve the problem as it would appear the router itself is blocked from DNS however setting up your own internal DNS or setting DNS on each device does work.

Mines a Dynalink RTA 1025W and I'm in the same boat, so it's not just the Netcomms. I 100% agree without about the announcement that it's all working again, while some users are just sitting in internet darkness.

Same findings about the end device DNS, though to be honest - normal service could be restored but since I've already set all my devices to Google DNS, I haven't bothered testing.

While using the Google DNS servers is a perfectly reasonable short term fix, you will likely be directed offshore for Akamai content, although Youtube content should still come from an optimal location.

With the tiny number of users currently using non Telecom DNS servers, it makes next to no difference for our international link capacity, but your user experience will be worse.

On top of that your DNS responses are now probably at least 45-70ms, instead of 5-30ms for the Spark DNS servers. (Depending on where i NZ you are and your DSL profile)

Mine is also a Dynalink RT 1025. Luckily I had a Raspberry Pi available and installed dnsmasq which includes DNS caching.Actually response time is now visibly faster when loading pages like NZ Herald with many different links for advertising, than using Spark DNS (even when it was working).

Once again, blaming the modems still doesn't answer the question why only Spark affected. Still seems like we are not getting the full story.

Sfitz: Mine is also a Dynalink RT 1025. Luckily I had a Raspberry Pi available and installed dnsmasq which includes DNS caching.Actually response time is now visibly faster when loading pages like NZ Herald with many different links for advertising, than using Spark DNS (even when it was working).

Once again, blaming the modems still doesn't answer the question why only Spark affected. Still seems like we are not getting the full story.

1) We're still working on it. Giving the full story is less important than resolving all the issues.

2) Yep, if someone wants to install their own caching DNS server then that's great and you will see better performance... Sub millisecond DNS lookups for cached entries are better than even 10ms lookups to our servers. I personally run a caching DNS server at home as well.

Sfitz: From Spark this morning ... it looks like you may be using a Netcomm router, is that correct? If so, some of these modems have beenblocked from checking DNS due to being vulnerable to attacks like we had on the weekend. Applying DNS to each device should still allow each device to get online.

My problem with Spark is they haven't published that they have blocked users and provide more detailed information. Instead they have just left users not working and announce in the media all is good.Setting Google DNS in the router rather than Spark ones does not resolve the problem as it would appear the router itself is blocked from DNS however setting up your own internal DNS or setting DNS on each device does work.

Yep, totally agree with this, the media release should state; we are still under attack/dealing with this issue, however we have managed to largely mitigate the issue, unfortunately our mitigation has inadvertently/(purposefully) blocked a small amount of users from even connecting to our network, in which case DNS server changes wont help them and they need to contact a Spark technician to work through why they have been blocked...

Hi, just come from a school on Telecom Business Fibre that has been cut off, after an age on the line talking to folk with clearly limited knowledge I finally get told that the modem (sic) needs replacing as the current one has a virus. Obviously not happy to have to replace the Mikrotik I ask what it will be replaced with, then ask how I will terminate my VPN's on that. Obviously my immediate concern was that the Mikrotiks DNS server/cache was providing out of LAN scope DNS recursion, I have checked with others I have installed on other ISPs and they dont, clearly I cannot test this one as Telecom have blocked it.

Anyone know what options there are, it seems another two days of no internet for a school wanting to do online exams.

cyril7: Hi, just come from a school on Telecom Business Fibre that has been cut off, after an age on the line talking to folk with clearly limited knowledge I finally get told that the modem (sic) needs replacing as the current one has a virus. Obviously not happy to have to replace the Mikrotik I ask what it will be replaced with, then ask how I will terminate my VPN's on that. Obviously my immediate concern was that the Mikrotiks DNS server/cache was providing out of LAN scope DNS recursion, I have checked with others I have installed on other ISPs and they dont, clearly I cannot test this one as Telecom have blocked it.

Anyone know what options there are, it seems another two days of no internet for a school wanting to do online exams.

Cyril

Hi Cryril, you've pretty much summed up what I have experienced over the past two days. I will PM you.

cyril7: Hi, just come from a school on Telecom Business Fibre that has been cut off, after an age on the line talking to folk with clearly limited knowledge I finally get told that the modem (sic) needs replacing as the current one has a virus. Obviously not happy to have to replace the Mikrotik I ask what it will be replaced with, then ask how I will terminate my VPN's on that. Obviously my immediate concern was that the Mikrotiks DNS server/cache was providing out of LAN scope DNS recursion, I have checked with others I have installed on other ISPs and they dont, clearly I cannot test this one as Telecom have blocked it.

Anyone know what options there are, it seems another two days of no internet for a school wanting to do online exams.

Cyril

PM me details.

When you say business fibre, please be specific about the product, including any CLNE supplied.

Well further to my post on Monday. I did visit the family on Monday. Installed a 'loan' TP-Link ADSL2+ router. Ran anti malware software across their PCs (which turned up a lot of junk). Once I was comfortable in the knowledge all was clear I tried contacting Spark to get their IP unblocked. Oh the joy.

My first call disconnected after 35 minutes. I figured this was going to be a mission, so instead of waiting at their house interrupting their family dinner time, I left. I began my next call in the car.

This "approximately 1 hour wait time" call got to ±115 minutes when I got to speak to somebody. I gave the requisite 'incident number' and asked that she pull it up and use it as reference. I explained what I had done and was now asking that the block be lifted. She then insisted that I go onto the net and see if I could browse to certain addresses. I explained that beside the fact I was no longer onsite the clients IP was being blocked by Spark and none of her online tests would be possible. She was quite insistent. I let rip and asked that I speak to the next tier, and was put on hold. After 2-3 minutes she came back on the line and said that she had now read through the notes relating to the incident number with her supervisor and could now understand what I trying to say.

At ±120 minutes I was told that the team that could remove the block had gone home about hour earlier. Nice. There's a major crisis on, and senior tech people get to go home? I've been doing this stuff for over 30 years, you never go home until it's fixed. Anyway I was then advised that it's been treated as a priority and the customer would have service between 9-10am Tuesday.

Nothing happened Tuesday morning bar the arrival of a shiny new Huawei HG630b. I advsed that they install it, which they did. A rep from Spark called me at around 14:00 to check if there was service yet, if not the I would be put through to the 'specialist' team. When that didn't happen, she advised someone would call be back in 20-30 minutes. No-one did.

Well Tuesday has been and gone and the customer is still waiting for service. I left a voice message for a call back this morning. Sent an email. And also engaged in a 'chat' session with the Broadband Team, first Sarah who upon reading of the customers dilemma, quickly put me through to Colin in Billing. Very helpful. Anyway Colin was quite positive, he took my details and assured me someone will be in touch within 10-15 minutes. That was around 10am, still waiting.

I eventually got a call back from someone at 12:04. They weren't very concise about the delays. Something about how 'some other stuff had to be unblocked before their IP could be released'. "Perhaps sometime in the next 2 to 3 hours".

Then 20 minutes ago the customer recieved the following txt: "From Spark Support: Your Broadband issue has been resolved for 095*****2. If you still experience issues please call 0800225598 using case #2312*******9"

It's still not functioning. She's had to go elsewhere to attend to her internet requirements of banking and account payment. Hopefully on her return there'll be some change.

I can't help think this whole issue could've been handled better. Blaming the non 'Telecom"/"Spark" routers is a bit of a cop out. If there was a vulnerablility in them, why was this not picked up in the Telepermit process?

I have had to talk 8 sites through changing the DNS on their PC so far, last one was 2 minutes before writing this post. restarting the router wasn't effecting a solution.fortunately we only have 1 PC per site and they are all win7 so it's follow the same steps, job done.strange thing, spark is telling me the problem is fixed, waiting for the official response all our routers are dlink couple different varieties.

Just like to update that with the help of Spark staff who frequent here (thanks guys) the schools router was removed from the blacklist, it would seem that a machine in the school seems to be infested with malware that took part in an amplification attack, naturally I have requested the site admin take a look into that aspect.

He assured me that progress is being made. But it will be another 3-4 hours before the client sees service. This clients connection didn't suddenly drop off on Saturday, but has been intermittent since early last week already. Spark, mad that disconnection effectively permanent over the weekend.

If the client is actively taking part in the cause of the issue, then perhaps they should be looking inwards before looking outwards? As far as I know every ISP has rights to kick users off who are affecting their service/core infrastructure.

If the client is actively taking part in the cause of the issue, then perhaps they should be looking inwards before looking outwards? As far as I know every ISP has rights to kick users off who are affecting their service/core infrastructure.

Unless I've misunderstood what your issue is.

but it was working fine before! it cant have been their fault! isp must have made the mistake!

#include <std_disclaimer>

Any comments made are personal opinion and do not reflect directly on the position my current or past employers may have.