We’re having some load issues today, most likely due to intensive export operations. Things may be up and down as we try to keep things running. If one slice goes down then all the load goes to the other slice and then it becomes unresponsive. It’s not been a fun day. I think as soon as we get both slices up in a stable way, we should be good.

If you’re having trouble exporting, don’t keep trying to export. Just email support@devjavu.com and I’ll do it for you this weekend.

Over the next few months, DevjaVu will be winding down and eventually shutting down. We’ll try to hold on as long as possible, but we don’t have the money to run more than a few months unless we work something out with Engine Yard (perhaps go down to one slice). We highly recommend you start migrating off as soon as possible. A more concrete end of service date will be announced soon.

You have access to your SVN repo and Trac instance in the Admin under Export. If your repository is too big to export, we can do the export for you and give you a link to download. Just contact support@devjavu.com if that’s the case. If worse comes to worse and we do have to shut down before everybody is off, we’ll give plenty of warning and then provide downloadable exports of everybody’s projects linked in an email to the project owner once the service is off.

This is the first time I’ve had to do anything like this, so if you have any feedback on handling this in a better way, let me know. I’d love to give a detailed explanation of what’s going on, but I think I’m going to hold off until the final day. In short: the business never got the momentum I wanted to sustain my personal interest, so while running at breakeven worked so-so for so long, it finally started dying and has been losing money for a while. I started an effort to save it, but I decided I did not want to continue supporting this type of service.

Sorry everybody! I would have liked this to be less of a surprise than it is for some of you, which could have been avoided if I better communicated with you guys up to this point. If you have questions or concerns, email me at jeff@devjavu.com

Engine Yard suffered an outage today from their datacenter. Here’s what they had to say:

We have been in constant contact with our Herakles data center throughout this service interruption. Unfortunately, root cause determination at Herakles for the ISP outage is unknown at this time. Herakles started troubleshooting at the core switching, then to the routers coming into the facility, and are now focused on the distribution switches inside the data center. At this time, Herakles data does not have an ETA for resolution of this problem. We will continue to keep you updated as we hear information from the Herakles data center operations team.

Unfortunately, after the issue was resolved, DevjaVu was still down. Since then I’ve been in contact with them and they’re on it. There’s no ETA, but my guess is within the next hour. Although I’m sure they have a number of follow up issues with their clients after this massive outage.

Anyway, sorry about this outage. If only DevjaVu had an offline mode, no?

Update @ 4/1 10am: No April Fools here. Downtime was extended overnight due to a billing glitch. It’s been resolved and Engine Yard is currently restoring our slices, which should be up momentarily.

Update @ 4/1 1pm: We’re back! After much unnecessary delay from problem after problem after problem. *sigh* I feel like I could have done better. Feel free to yell at me next time you see me.

There seems to be a problem with the network storage on our slices and now they’re going a bit crazy. I’ve pinged Engine Yard and they’re usually quick to help fix the problem. It is something that’s happened before. Sorry if it’s interupting any work!

Just to let you know, we’re experiencing some downtime with Subversion due to a bug with our authentication mechanism. It might be a while to fix, but we’re working on it and we’ll keep you posted here.

I know things have been pretty quiet. Man, it’s tough running a service like this on your own. Don’t worry though, things should start ramping up soon with all the things set in motion behind the scenes.

It’s funny too though because DevjaVu is getting more free signups than ever. It’s funny because free project registration requires an invite code. I know we have a public invite code that anybody can use, but it still means people are going out of their way to search for an invite code.

Unfortunately, we’re dealing with some growing pains and we’re getting a lot of heavy load on our servers from Subversion usage. We’ve got an interesting plan to solve this (which I shouldn’t talk about yet), but for now and because our registration process has been somewhat flaky, we’re going to turn off the invite code for free projects in the next 48 hours. This means you won’t be able to signup for new projects until our big re-release.

Now about that… basically we’re repositioning the DevjaVu service and so it’s going to be renamed and the pricing structure will change a bit. It’s also getting a brand spanking new look both on the site and the default project theme. And if things go as planned, we’ll be releasing with a completely open free project plan.

So yes, after a period of stagnation we have changes abound coming up. But for now, we’re closing free project registration within the next 48 hours. Sorry guys!

Apparently our slices at Engine Yard died because of excessive load and we’re working with them to bring them back up. Sorry for this bit of unexpected downtime in the middle of the day. They should be back up within the next 30 minutes.

Early this morning the Subversion authorization configuration for many of our projects became corrupted, preventing access to those repositories. Unfortunately, I am a major bottleneck in response to DevjaVu service emergencies. This happened to coincide with me moving and not having Internet access, and this morning having my phone die without being able to charge it. I didn’t know there was a problem until very late into the emergency and I wasn’t able to get online to do anything about it until even later.

Obviously this is a terrible excuse for a full day of service interruption. I do the best I can with the resources I have, but I know this is unacceptable. Particularly the lack of communication while handling it. That’s an easy one to fix for next time. I’ll also be looking at hiring a part-time emergency technician and putting together an SLA for premium users.

Not to mention the cause of this outage should not be possible when we migrate to our new system that’s been in development for some time. I want to share more about that soon, but for now I’d just like to appologize for this outage and the lack of communication regarding it.

One of the biggest usability issues with DevjaVu has been inviting new users to your project. Luckily, most of you were smart enough to figure it out and/or put up with the difficulty. The downside of that has been that I’ve been able to put it off for so long. Today we have a solution in the form of a Users section in the Admin. You can now invite users by email and they will get a link to either register if they’re new or login if they have an account.

The main caveat is that you have to invite users into a group. New projects are created with three default groups: admins, developers, and viewers. Older projects might not have groups. If you run into any problems, please let us know.

As a side note, tonight Engine Yard will be moving us to a new load balancer, so there may be some brief downtime after midnight PST. This should relieve some of the recent issues with large checkouts/exports hanging.

We got a couple reports that Subversion service was becoming slow or unresponsive today during peak usage hours. This makes it a bit difficult to debug, but it would help if anybody could report recent SVN downtime or stalls including when it happened to support@devjavu.com