Brickset.com site issues

Comments

mere speculation but im going to definitely side with network issue as the cause as opposed to dos.

i agree with other comments the german ip sounds like hes using scraper, and the fact there is network issues is probably inadvertently causing his crappy scraper to make too many requests. if the network was stable he probably would have scraped the entire catalogue overnight and you would be none the wiser unless you study the logs like a hawk each morning

plus if its only a single ip once you banned his entire subnet shouldnt things return to normal?

brickset.com seems up, but very slow and I got a 524 before refreshing.new.brickset.com seems wonderfully whizzy and working like a charm.Are both sites hosted on the same server?

Could you do a 302 or 307 from the old to the new. If any traffic is then seen on the IP for the old site you know it's suspicious (and if the new starts having issues, then it's probably a bad scraper which is following the redirect). Could also try disallowing all in robots.txt but it's unlikely a custom scraper would pay any attention to it.

I've just had a long chat with ORI. They are doing everything they can to sort this out. They are seeing a lot of strange traffic to the site and are blocking it and getting their upstream providers to block it too.

All IPs except those originating via CloudFlare have been blocked. Cloudflare reports security threat originating countries and many are coming from Singapore and China. I have thus blocked them at CloudFlare for now, but will re-enable when this over.

The reason many pages are not loading is because ORI have put a 1 second timeout at the firewall on requests to try and stop the rogue traffic. But that is not enough for many of the pages to load so I've asked it to be increased.

It's tempting to think that rehosting will solve the problems but if it's due to traffic targeted at Brickset directly then obviously that will continue wherever it's hosted.

What I might do tomorrow is sign up for the other hosting package and transfer traffic to it, if only to see what happens. It will be expensive, but if it does turn out to solve the problem it will be worth it.

@Huw I used to have a lot of problem with rogue IPs from Germany and China. They are probably still trying, but my tech person did some of his woodoo and they are no longer a problem. So from that perspective it is possible that your server is under attack. These are usually not people targeting a specific site, just machines looking for a vulnerable site to hack into. If you are able to keep them out and minimize their effect after some time they move on to the next target.

It's actually quite normal for there to be threats showing in your cloudflare analytics, they're just a constant presence on the net these days. If you've ever spent time tailing your logs (I have, it's fun!) you will see some really weird stuff. It's surprising how many script kiddies and scrapers/bots are out there.

Isn't it possible to deny access simply by setting up a deny record in the .htaccess? I can't remember exactly but a site I run encountered massive problems with Baidu a couple of years ago, which was basically spam crawling everything constantly. Somebody suggested I add their IP range to the .htaccess and me being the technical whizz I am (/sarcasm) somehow cobbled it together and the traffic dropped off overnight.

I can access new. and pages either load instantly or not at all, stalling an eventually giving a no page in cache error. Those that won't load, I can usually get to load on a second or third try after waiting a few second. I still can't access the old site at all. In the US.

Yonks ago, when I was on jury service, a more experienced member of the jury told me that you can always tell when someone has previous convictions for doing something, because although you can't disclose them in court, you can be sure that the defence will talk about the suspects immaculate previous recored if he doesn't have any convictions.

Taking this theory back to the Brickset issues. It's very unusual for a small site like Brickset to be targeted by a DDos attack. And if it was just Brickset being targeted, I'm fairly sure that Huw's web host would immediately say that it was all the fault of his site clogging down their bandwidth and shut his site down to keep all of their other customers happy.

However, if it was their entire network being targeted, they'd be extremely reluctant to admit this to any of their customers because they'd face a mass exodus.

It's just a theory, but if I were to have a bet, I'd say its' the entire server not just Bricklink being targeted.

Also, I don't think that it is too much 'legit' traffic going through the page and that the provider isn't able to handle it. Bricklink used to be hosted with the same people until last summer, and BL has a shedload more of traffic (I think!).

^ No, it's the same for everyone unless your browser has the CSS cached from the old site. It's due to the forum attempting to pull the style info from the location that it was in on the old site. I'm sure Huw will sort it shortly :-)

Ah, I see the forum is changed again. Sorry to be that guy, but can't say I like the default version. I'm with @rocao , With categories on the left, the actual thread ends up just hanging in a sea of white, and the glare of the white expanse (below the categories) royally screws my eyes up trying to focus on the text. I hope I can get used to it, but migraines do not show mercy. :oS

I suppose this new layout is okay, but really the categories need to be on the right for me. I also hate the fact that my own posts no longer appear blue, but the same colour as other member's posts, so it's easier for them to get lost in a sea of replies. If you know what I mean. Anything to break up the white a little bit once the categories are out of view! :-)

But maybe that's just me.

I can't say I like the notification and inbox pop ups you get with the default either. I want all my PM's / notifications to appear with one click like it did in the old version.

Glad the site's back up Huw. Was going to make a comment about 1 to 1.5 seconds (ms I could believe, not s) per query sounds very slow to me (especially for a stored proc) but given the 'new' site's speed I'm guessing you've already done something about that. Well done, time for a well earned rest!

I noticed that too. Also, and it might just be me, but I'm finding strange little instances of 'doubled-up' screen 'overlap' when threads open. I'm thinking it's when I have an unposted draft. I have to scroll up and down to clear the overlap.

^ No, i wouldn't make it match. I like the distinction between the two. The forum needs to be simple, as the chat is the attraction. If it looks all glitz and bling, it'll detract from the threads/be less functional, if that makes sense?