Welcome!

Friday, October 21, 2011

Four days ago for reasons unknown, all communication between any scripted object anywhere on the grid and the IP address of my web application (that runs the Galleries of SL kiosks) at http://sasun.info stopped working. Testing shows that my web app is functioning perfectly fine and is in fact working for kiosks on the Avination grid. Avination is able to communicate to my web app, no problem. HTTP communication to other domains from the SL grid work fine. The only combination that does not work is http requests from SL to my website's IP address.

The only sensible conclusion is that either LL or my web host are blocking these http requests. A quick call to my web host confirmed that they do not block any IP addresses from their shared hosting IP addresses at all, it is up to their customers to block IP addresses on a doman-by-domain basis. In other words, if someone is trying to hack into http://sasun.info (which in fact just happened yesterday, unrelated to this case), I have to block any such offending IPs in my application myself. So... no blocking going on there.

So the obvious conclusion here is that LL is for some reason blocking all http requests from any and all of their servers (which means, the entire grid) to my web app. It may be possible that there is something in between that is blocking these, or some technical issue on LL's end that's not blocking them on purpose but some bad side-effect of the recent OS upgrades (hint, hint, LL) that's stopping http requests from working to my domain (and possibly many others, I have no idea).

All of this is perfecty understandable, though unfortunate. Software is buggy, problems crop up, upgrades cause problems, gremlins sneak into the wires and make things go crazy. I may be just the unlucky victim of bad luck somewhere between LL and my web host.

Or am I? It turns out I am not the only IP address that has been blocked by LL. One person I talked to had a very similar story and their solution was a lot of work on their part to move to a different IP address. For me, that would involve quite a bit of expense and work but is something I am going to investigate today since there is no way for me, or any of the "regular" support people at LL, to even tell me what the status of my issue is.

So here is the frustrating and critical failing of LL in this whole debacle - their unbelievably bad handling of this problem so far. As soon as I filed the case I jumped onto a chat session and was politely told that they would escalate the issue to from a normal "case ticket" to a "technical assistance" issue, meaning that it goes to the real support people that don't solve the stupid questions. Great! It's obviously critical to me and a bunch of other SL business owners so this should get fixed soon, right?

Days go by and I hear absolutely nothing. Every day I start a chat session to find out what is going on, and every time I tell them how critical this issue is and how vital it is that I get some kind of update or acknowledgement that it's being looked into. It turns out that the front-line chat support people have NO WAY to communicate with the "technical assistance" people working on any particular support case. From today's chat session:

You: It has been 4 days and affects well over 6,000 kiosks across the grid. Is there any way to get an ETA when this issue will be fixed? Every single scripted object (approximately 7,000) across the grid that communicates to my web site has not been working for 4 days. I've been in the dark this whole time.RebeccaH: I do apologize but I don't have access to the technical support operations information.You: so you guys are cut off from them, there's no way to update customers on what is going on?RebeccaH: I'm sorry but that is correct, I don't have access to their information or a way to communicate with them.

Uh, what? Support team A has no way to communicate with support team B? Seriously, really, the live chat support people have no access to support information or any way to communicate with the technical assistance people?

Furthermore I was told that the technical assistance cases are unprioritized - there is no way to prioritize them. So every non-critical issue is in line with every severe "my business is down" case and is prioritized fist come, first serve. This is a disaster! Any support business in the universe needs to prioritize cases and get to the high-impact, "server down" cases first!

I can guarantee you that LL's own internal IT support department, for example, doesn't prioritize strictly by first-come-first-serve basis. So if an LL intern requests a different colored mouse from the IT department, he's ahead of the CEO whose video card died and is stopping him from getting his job done? I seriously doubt it!!

So why in heavens name would they refuse to prioritize customer cases and make the critical cases that are going to have wide-spread negative effects on the success of Second Life (there are products and content creators that falling into this category, obviously) higher priority? In effect what they are saying is that if LL stops your entire business from functioning, we don't consider that more important than anything else, so you'll need to wait a week for us to get around to it. Great!

I can't begin to express how frustrated I am with LL. I have been sweating over this the whole time, and working like crazy to supply customers with some level of functionality for their broken products. I'm obsessively checking to see if things are working again, since it drives me absolutely nuts to have so many customers unhappy and unable to use my products like this.

I have done some work to port my kiosk network to other grids such as avination and inWorldz, which so far as been very successful. This incident has convinced me I need to step up those efforts so that I can help them compete with LL as much as possible. Having a list of art galleries on avination, inWorldz, and other grids would be very cool and help them grow, not to mention put some products up for sale on those grids as well.

Furthermore I'm taking a hard look at my involvement in such things as the Linden Endowment for the Arts, which I put a rediculous amount of work into for no benefit of my own. I am boycotting any such voluteer work for a company that won't even give me the courtesy of a "hi we got your issue and are working on it, sorry about that" response until such time as this issue is fixed. If that is never, so be it. I don't know why I should work so hard to benefit a company that when the rubber hits the road, really doesn't care, and if they do, they have completely failed to show it.

Posted by
Sasun Steinbeck

6 comments:

The support people at InWorldz are often available inworld and on Twitter and do not like to let things go unresolved. I have always had a speedy response and I know it could be partly to do with the comparitively small size of the grid but ( and this is the major failing of SL) in Iwz the 'right hand' does know what the 'left hand' is doing.I have heard a few people say that the structural incompetence of SL is the poison (Love Machine) chalice left by the previous CEO. It does not seem to function like a normal company.

Omg! WOW! First, I would like to apologize that you and your customers are having to deal with this to begin with. I just found this post after the fact of sending you an inworld message as I want the kiosk product myself. I hope they get this fixed at the very least for you if nothing more. If nothing is done to fix this issue, are you going to move everything or what are your PLAN B if any? I am just completely shocked that they don't have prioritizing and departments reachable to one another. To just think about that makes me see the huge disaster of a mess that is as being a hosting provider myself and having departments such as general support and tech support. If they can't communicate, nothing would ever get done properly. I hope this gets resolved for you, I really do!

Your art gallery tour kiosks have been part of what has kept me holding on to the hope of beauty in SL - i can only begin to comprehend what it would take to get it implemented across the hyper-grided OpenSim networks but if LL is borking it for SL... [snip rant] Thank You - for communicating clearly and sharing valued information-Nyx

Sasun, I have not had a satisfactory response from LL support for years. They loose stuff from my inventory and say "tough nuggies" six months later and then want me to fill out a questionaire on how good the support is - LOL

X fingers you get back on soon. I understand you have finally got thru to Kelly Linden and hopefully will have switched to another, "unborked by SL" provider soon. This kind of bad behavior and terrible support will be the death of this world.