INTERDUBS crossed the 1.5 Million file mark yesterday. Big numbers are hard to imagine. Simplifying matters a bit one can assume that each file at least replaces one DVD. If one would put those 1.5 million DVDs side by side on a shelf then it would be 14 miles long. It was a while back that I walked that distance. But I still remember vividly that it took a while.

Until yesterday we had a perfect uptime history. 1,375 days online, and no interruptions. Today that changed: Starting from 9:16am PDT INTERDUBS.com was not reachable for 19 minutes.

Absolutely our fault. Of all possible scenarios it is actually our preferred one since we can fix it. We did and this problem will never occur again. Still sucks to having lost our outage virginity. All outages are avoidable, and so was this one. And -as usual- it was the lack of imagination that caused us to not see this coming.

What happened?

An upload with 550Mb/s triggered an automated protection system that took a network interface offline. It should have only impacted the one address using that amount of traffic. But it was doing it’s job wrong and shot in both directions: Rendering us unreachable.

This system is meant to guard INTERDUBS against malicious brute force attacks. Not a bad idea, if implemented right. 550Mb/s is of course still very very far away from the limit of our network capabilities. It was the volume and specific traffic pattern that caused the emergency shut off. We did neither envision nor test this specific condition. No two ways around this: Our fault. Embarrassing. Please accept our apologies (and money, if you like - see below for details).

We re-configured the system and are confident todays outage will never happen again.

Since we were unaware of the bug in the system configuration it took us a little while to identify the cause. Since the system is responsible for security we also had to spend a couple minutes testing the changes that then became the fix. 19 Minutes is a long time for an outage. Looking at what needed to get done to bring us back online we feel that we did OK. Not great, but OK. Of course there is also room for improvement, and we started to implement those changes today.

INTERDUBS overall uptime dropped now from 100% to 99.99904%.
For the month we are down to 99.95%. Well below the 99.999% we promise. None of our clients has to pay for INTERDUBS this month. If they choose so. A simple email is enough: we will discount the whole month.

Since counting nines of uptime is not really what most people want to think about, we decided that effective immediately we change our policy: we now guarantee 100% uptime. No longer ‘only’ 99.999%. If a client feels that INTERDUBS wasn’t there for them when they needed it, then they don’t pay. Simple as that.

Back in the day an electron beam was running across the TV screen. NTSC was running with 30 and PAL with 25 frames a second. If the beam would go line by line the screen would flicker. The solution was, to let it run twice over the screen for each frame: Once for all odd lines (1,3,5 etc) and then again for all the even ones (2,4,6). That looked better. It is called ‘interlaced’. Each of these passes is a ‘field’.

Film cameras liked to run at 24 frames per second. Cinema does not flicker since each frames is shown twice, but that is not the point here.

When you have 24 fps footage and your TV runs at 30fps, what do you do? The solution was to insert a so called 3:2 pulldown to make 30 frames out of 24. This was done based on 60 fields to make it look smooth.

Interlacing is dead. There are no electron beams going over glass tubes to make images to speak of.

If you like to compress an NTSC spot that was shot on film, and that has the 3:2 pulldown in it, then you should go back to the 24fps version first. Since I could not find anything that worked I developed this. In 1998. Then, in 2008, I needed it again, and so I looked again. Much to my surprise, nothing really worked the way it should be. Many tools have the button to do an ‘inverse telecine’. But none detect cuts and deal with changing cadence patterns. So, I wrote it again. This time based on quicktime.

With INTERDUBS growing in the US solidly it was time to start to add another dimension to its growth. It is an interesting experience to go through the same motions again. Just on a different continent. Luckily we found a great data center partner. It is pretty cool these days that one can get a virtual test server within minutes. Of course we are building real machines again for the real install. After some research we found some great vendors: ISP Proshop served us extremely well for cases, cables and the like. We found that Alternate.de has very decent inventory in terms of high end server parts.

Some observations along the way:

Calling a vendor can mean that they already have your order on their screen. Before they pick up the phone (after the 1st ring). Caller-ID plus decent software makes this possible.

Ordering parts it can happen that they arrive 16 hours after you did so. Standard shipping. 5 Euros.

Payment is done electronically. Online, bank account to bank account. Securely, since you have a little key generating device. And no credit company sits in the middle, getting their 3 percent, just because banks didn’t get their act together.

Prices are horribly high. Almost 20% tax on top of things.

Another drawback: If you are fond of those plastic packing chips (who isn’t) you will come up empty. Crumpled recycled card board works just as well it seems.

So I wrote a script that will save me a minute. I pretty much assumed that I wrote it, just because I like writing code, and this task was just something that fit into the timeslot before dinner. I chalked up the twenty minutes it took as wasted time. Others check Facebook, I write a script that can be done before the next thing on the schedule.

As i said this one will only save a minute. But it will do so every day. Still no big deal, I thought. But -funny as it goes- it finished it a minute early, so I came to realize that I will have saved six hours after a year. Yes, in my head it takes takes 60 seconds to compute 365 / 60. Anyway: after two years I get one more day in Hawaii. That’s actually not bad at all for something squeezed in before dinner.

It also gets to show how bad we are actually in estimating what the impact our actions is. I didn’t start out to save a work day in two years. I simply had twenty minutes to fill and a repeating task that could be sped up. Guess I got lucky. Again.

“It usually takes about 12 to 18 months to build a new center,” she said. “We’re cutting that down to less than a year.”

from a NY Times article about Microsoft and Google and their respective data center operations.

It is interesting that after years in corporate culture people start saying this kind of thing and feel that there is nothing wrong with it.

The article poses the question whether google benefits from looking at each level of the technology stack and inventing where needs are. It does not come to a conclusive answer. I think it is rather obvious: Google was able to reduce its capex spending simply when it felt oppotune to do so. To my knowledge and own experience there hasn’t been any noticeable impact on the Google useability by this reduction in spending. I would guess google simple turned down the pace innovation while the influx of new equipment was slowing.

On a -in comparison- microscopic scale I experience the benefits of looking at the entire technology stack first hand. Part of what runs INTERDUBS is of the shelve, and other parts are enhanced, customized and severely optimized. Some we even actually build ourselves. We constantly look at the running service and identify room for improvement. Be it, in the user experience, or how efficient internals work. Having an understanding of the entire system on all levels lets us identify clearly where enhancements should be made. Each of these steps might only add a couple of percentage points. Having metrics and detailed information about all aspects of the system at all times not only give us visibility into which areas are to be tunesd and enchanced next. It also reveals, that all those little optimizations add up into a configuration ,that is faster by dimensions than the un-altered and generic one would have been.

Having this culture of change and constant optimization is allot of fun. I was plain scared having to do this on live data and a running service. But the goal was that INTERDUBS is available 24/7. And it turns out that technology - used in the right way - is able to do this now. It is literally flying the airplane and rebuilding it in the same time. You start in LA in a 707 and land in New York on a A380.

An interesting collection of more or less vague ‘cloud sizes’. My guess is that most of these machines are no longer specialized hardware or workstations. Explains why Sun -for instance- is having such a hard time. Once you scale well in software and do handle hardware failures in that layer too there is really no need for expensive irons. I wonder how many of those large footprint installs run Windows like operating systems.

The main purpose of the API for INTERDUBS was to let my clients use it programmatically. Their system control INTERDUBS in a way that fits best into their workflow. That is what the API was made for, and it works. Every day.

Interestingly there are other, somewhat more surprisingly, benefits to having an Application Programmable Interface to a system too. This week I had a discussion with a client where they outlined specific needs in respect to their data that they have in INTERDUBS. They could have updated things manually, but that would have taken a very long time. By using the API myself it actually took me not that long to implement their needs. I spent only slightly more time on it than the actual discussion took that we had about their needs. This was very nice to see. And actually quiet unexpected. If the next client needs something similar I will be able to solve this in five minutes.

Four days of intense work for less than a percent of improvement sounds not like a great use of my time. But I am actually very happy about the outcome: I was able to increase the success rate of clip meta data detection in INTERDUBS by 0.8%. This is great since it went up from 98.8% to 99.6%. Or looking at it from the other end: two third of all flawed detections were and will be corrected with the improved code. One of the benefits of having 100,000s of clips online is to be able run simulations and stats while improving the code. There is a wide variety in what people like to use as their encoding and file format. I’d rather do some more -invsible- work on the backend than to lecture my clients on how exactly they should encode their files. There are recommendations. Sure. But why fail if you don’t have to?

Even though this application of Grubers Broken Windows is seemingly invisible, in the end it certainly is not: A well running system just needs less support per client. Actually so far I was able to decrease the total time spent on support. Despite the fact that the client based tripled last year.

INTERDUBS supported the iPhone 40 days after it came out. Last week Wiredrive came out with their iPhone support. Graphically that looks like:

I don’t think that much can be gained by not acting quickly. At this point my clients have already solid experience with their clients in how to use the iPhone, and how not to. We could make good use of those twenty months.

In an act of ‘active procrastination’ (aka as coding things that nobody needs / wants in order to avoid real work) I wrote a view on my database that would sum up the number of INTERDUBS clients over time.

I was very suprised to find the (somewhat smoothed) result to identify so clearly both growth phases that INTERDUBS had so far. In the beginning I did have only a very very rudimentary public website and growth was only word of mouth. This was intended so that I could spend enough time on the needs of everybody that came on board.

Once what people needed was pretty well covered I made the public site a bit more meaningful and growth increased. Nicely enough support efforts have remained on a constant level: New users need a hand here or there. Often enough it is possible to avoid issues from being repeated by adjusting the code to what the users do expect.

Downtime happens. And it is not pretty. But science can help. And all science starts with data. That is why I monitor how sites like INTERDUBS are availabe to their users. A company that shall remain unnamed was down for four hours this morning, which made me wonder how everybody is doing. I also wanted to play with Google Charts a bit more.

Accumulated downtime since April 2008. Less is better:

Congratulations to Adbeast for being clearly better. I am happy about the second place in this case. It is hard to derive the user experience of outages just based on this one measure. It very well mayb be that the customers of Competitor A - C are also happy with how things are going for them.

In anyway I also feel good in light of thise data about offering 99.999% uptime guarantees with a full months money back.

Update March 2009:
It turned out that I probed a static page for Adbeast. So the praise that I would have loved to give is somewhat unjustified. At this point it is early to tell how the real content pages rank. So far they are much less stable INTERDUBS though. Which is almost a shame: Nobody beliefs a statistic that makes the author of the statistic the clear winner …

The code management system I wrote for INTERDUBS happens to also count the number of updates that I publish to my clients. It just hit 1,000. Of those about 10-20% were cosmetic updates. Like typos or smaller changes in the html to make things more readable etc. Luckily less than 5% were bug fixes. I code in small chunks and those extensively. And maybe it is also a matter of routine. I hope I know what I am doing, and where changes would jeopardize the system. Of course stating this is inviting trouble. A thousand times I changed the running INTERDUBS. While it was in use by clients and admins. And: nobody ever noticed. Flying the airplane and fixing it. I really love this part of the project: Somebody has an idea. I look at it, and can tell them right away how doable it is, or in the best cases the reply email is as terse as “good idea! done”.

The fast majority of all the good things that came in those 1,000 updates were actually customer ideas. The people using the system know best what they need. It is really great being able to listen to them and to implement what they want.

INTERDUBS does not charge for storage. No matter how many Gigabytes you need, it is included in the $285 flat rate. Since there is no financial incentive to clean up, some of my customers amassed quiet a backlog of material.

Which actually was somewhat intended: Other tools were more important than the means to clean up and keep the data pool fresh. With Terrabytes stored -and much of it actually no longer needed- this situation changed. Adding tools to simplify clean up was easy. The database had already all needed information.

Getting people to use them was a different matter though. Somehow I got lucky and had what turned out to be the right idea: Between Sunday and yesterday I had a ‘cleanup drive’ in INTERDUBS. Clients with allot of old material were encouraged to delete as much as possible of it. As an incentive INTERDUBS donates to charity relative to the amount of Gigabytes cleaned up.

And it actually worked rather well: People put in allot of work to clean up their backlog of material, and they did so knowing that it was for a good cause.
In total INTERDUBS will pay for 240 vaccinations.

This little detail is also nice, since I had no idea that this solution would emerge when I decided not to charge for storage. If I can continue to run a flat rate based on the efforts like this then I will be very happy.

The other day a colleague observed me wrangling some obscure firewall / ftp issue that came up for one of my clients. Once I had fixed the problem he proclaimed: “you really leave no bug behind”. I like that expression. It matches what I am trying to do with INTERDUBS. Actually so far each bug got fixed twenty four hours after it had been reported. Other feature wishes can take longer to get implemented: Some people had to wait months before they could create reels via drag and drop.

The ‘all bugs get fixed right away’ mantra has a huge benefit: Low support efforts. Actually I carefully evaluate each support call / email to see if the software / documentation could have helped with this. I sure don’t mind talking to my clients, but I agree with Don Norman that products need to designed so that they work with their users as well as they possibly can.

Today Interdubs crossed the 80K file mark. 59 days for 20,000 more files. Last time it took 83 days to grow that much, and before that 118. Something seems to be working here.

What makes me even happier is to hear the following from a new client: “you must have worked in our field for a while”. That would be true, and it delights me that it seems to show in the application. Intentions are one thing. Seing that they apparently manifested themselves at least somewhat is very rewarding.

Today Interdubs crossed the 60,000 file mark. 83 days ago there where 40,000 files. Before that it took 118 days to go from 20,000 to 40,000. Interdubs grew 42% faster than in the quarter before. There are 25 official clients now that choose to be mentioned on the website. Interestingly, the support amount has actually gone down. Fixing every bug right away makes seems to let the total number go down. Most emails and calls are about new features and concepts. Since I often don’t get things right the first time, people have to make awesome suggestions how new features and concepts could be implemented better. I think that there is huge value in this kind of feedback. Users are used to things working efficientlly in interdubs. When something does not then they point me to it.

Being able to change the code in minutes and doing so frequently is one of the priceless concepts that are hard to imagine. But now I would never attempt to write software in any different way.

That’s an actual quote of a client in an email received a couple of minutes ago. It is his first month with Interdubs, and he is not used to the fact that the bill will only arrive once the month is over. And then he can pay it. Or not. If he should feel like that. Which sounds ‘good hearted’ or ‘weak’. But it makes actually allot of (business) sense: Most of my clients have made more money with the site in their first week of using it, then it will cost them for whole month. A not so significant part of them actually takes just a few hours to make the 285 that the services costs them. Either by direct billing or by improved client relationships. I was aware of this when I designed the system and set the price. The price is solely based on the system working as well as it seems to be. It is arranged around my costs and the future potential of more clients. And maybe on the fact that I like to code fast.

I really hate the business model that tries to leach on to the success of its clients. Network Neutrality is one of those. Phone companies would sure love to charge more for important business conversations than for idle chit chat.

But back to Interdubs: having a super reasonable price that are people actually eager to pay makes everything much easier on everybody. So far people paid their bills. The majority of companies in record time. Thanks again and also from here. If I would try to squeeze more money out of the service, then I might need an accounting department that starts bugging people. I’d rather not.

On the other side with the latest feature additions the price / performance ratio is in danger to tip from “great” to “ridicolously great”. I have feedback from many of my clients saying that the service is too cheap. And I suspect that I could actually sign up more people if the price were higher. Most people think just because the competition is ten times more expensive it also would be better.

? Saas ? Never had heard of it. Till Today. And then it showed up everywhere. SaaS seems to be a fancy acronym for Software as a Service. Turns out that’s what I am doing with Interdubs. Maybe if I would hang out in the Silicon Valley more or spend more time with VC types I would know this kind of language. But actually, I rather not. I just like to go ahead and write software. No need to call it fancy names. I rather check if people can use it for what they would like to do. Chances are they don’t know -or care- about SaaS either. They just have work to do.

Running and developing a system in the same time is allot of fun. An idea can be quickly added and / or tried. Some are more
involved though. At this moment there are 42,658 files in Interdubs. So uploading happens allot. There was a ftp interface, but people
need passwords and needed to remember the folder name.

It could be easier. And now it is. It’s as simple as clicking on a link:

A transmit droplet with the proper parameters get created and downloaded automatically. Those droplets can be kept in the dock or on the desktop, and uploading is even easier than it was.

As with so many nice and easy things the underlying technology is actually not that simple. It was great to be able to draw from the resources and experience of the amazing people at Oneiric to get the backbone for this service addition installed. David Green was super helpful, without him this feature would have taken weeks longer to implement. Working with David is allot of fun, since everything he says he will do he does. And it works, since he has tested and checked it from the get go.

It is truly interesting how a small company with people that care can have so much more impact that larger ones that take weeks to move.

Interdubs had an awesome year in 2007. I had a certain expectation where the service should be by now. Development-wise and feature=wise I am behind. I want more features, and I want to write them now. But doing them right does always take more time than I think it would. And, my clients got what they essentially need months ago. Since then new features have been extra and on top of it.

Looking back at 2007 I particularly like the the fact that Interdubs could scale from a few beta clients to more than 20 customers. Many of them with very diverse needs. And all of them seemingly happy: Even though nobody is contractually obliged to continue their subscription each one renewed month by month. People some times wonder why Interdubs is so inexpensive. Specially compared to it’s feature set. I think it makes sense: Having the most awesome feature vs. price ratio means that I don’t have to spend much time to keep my clients happy otherwise. It also helps with marketing: If anybody interested in an online media solution should happen to talk about it to an existing interdubs user I will get a call. And when I get a call it becomes a sale. Sooner or later it does. Always.

2007 was also nice, since I had not to act on my 99.99% uptime or money back promise. By now it would be not so nice, if I can not charge anybody for a full month. Which is the whole point: I believe in Interdubs’ reliability enough to put my money where where my mouth is. Outages might happen in the future. Nothing is perfect. But by giving my clients their money back for a whole month, if Interdubs should be longer unavailable than for 5 minutes I there is at least a plan. If this should ever happen. The looming penalty of a month long ‘invoice outage’ makes it financially viable to upgrade the servers that Interdubs runs on. So that it does not happen in the first place. Or is at least less likely.

2007 I published 590 times code updates to Interdubs. That’s why I don’t like to call things “Versions”. Version 590? Sometimes I just moved a couple of links around, to make a frequently used choice easier to find. A couple of times I replaced or upgraded the entire engine that runs Interdubs. I might have gotten lucky, but at no point did I loose data during those updates. And only about 10 changes were so stupid, that my users demanded a change back or further alteration of what I did. Knowing that I will hear about things going in the wrong direction allows me to suggest things with great liberty. The same concept looks enabling from the other side as well: Interdubs users know that they will be listened to. Sometimes it takes only minutes between a suggestion and the actual feature / change showing up on the site. Actually a great deal of ideas and features that make Interdubs worthwhile are a result of this collaboration.

2007 was a very successful year for Interdubs, so I had to decide what to do for Holiday presents. I decided not to send any at all. Instead I asked my kids to pick a charity. They suggested “Doctors without Borders” which I liked as well. So instead of sending gift baskets around some people got vaccinations that they needed.

Being able to decide on these things what to do is one of the perks of running your own company. Today I found Charity Navigator and realised with great relief that only a very small percentage of the interdubs donation will go to the adminstration.

Computerworld looks at internet market share data for different devices / operating systems. The headline reads ‘iPhones closing in on 0.1%’. That does sound like laughable little. The author paints a different picture, but I have no interest repainting that here.

I looked at the iPhone user share on Interdubs, and found the following numbers:

November 0.22
October 0.54
September 0.35
August 0.74
July 0.45

Not surprisingly they are by small magnitudes bigger than the general ones found by Net Applications. I had thought that they would be even higher. The iPhone has a nice display. It’s fully supported by Interdubs. Was it a mistake to invest into the iPhone mode? Absolutely not. It is very interesting to see those numbers. With already having iPhone support there is no second guessing ‘what if there were a special iPhone browsing mode’. There is, and that’s what the numbers look like. Right now.