Pinboard servers came under DDOS attack today and the colocation facility (Datacate) has insisted on taking the affected IP addresses offline for 48 hours. In my mind, this accomplishes the goal of the denial of service attack, but I am just a simple web admin.

I've moved the main site to a secondary server and will do the same for the APi in the morning (European time) when there's less chance of me screwing it up. Until then the API will be unreachable.

On December 1st, the World Food Programme (W.F.P.), announced that it was suspending its operations to feed one million seven hundred thousand Syrian refugees—scattered across Lebanon, Turkey, Jordan, and Egypt—because it had run out of money. (The program is under the auspices of the U.N., but funded entirely by voluntary donations.) Under the program, Syrian families received the equivalent of a dollar a person each day to buy food at local shops. This operation cost sixty-four million dollars a month, and, while governments and private donors had helped to fund it throughout most of 2014, there was no longer enough money to carry on. This was “disastrous,” the Programme said in a statement.

If you've wanted a Pinboard account but prefer that your money go to a worthy cause rather than supporting my indolent lifestyle, this is the perfect opportunity. And if you have the means, please consider donating to the WFP directly.

Beginning January 1, 2015, there will be a change in how I charge for Pinboard.

Right now, users pay a one-time signup fee that grows by a fraction of a penny with each new signup. At the moment, this fee is $10.55. Pinboard also offers archiving accounts, which cost $25/year. Users who upgrade after joining Pinboard can deduct the signup fee from the first year of archiving.

Under the new scheme, basic Pinboard accounts will cost $11/year, while archiving will continue to cost $25/year.

My main reason for making the change is so that I don't have to keep explaining how pricing works. An astonishing number of people already believe that they're paying annually for Pinboard. Others accuse me of baiting and switching them when they upgrade to archiving and get a renewal notice. Note how much easier it is to describe the new policy than the old one.

If you already have a Pinboard account, or sign up for one before January 1, 2015, this change will not affect you.

Over two years ago I introduced an API authentication method so people could authorize outside third-party websites to add things to their Pinboard account without sharing their Pinboard password.

Many sites switched over to use the token, but some still have not.

As of today, I'm going to start blocking outside websites that ask you for your Pinboard credentials. First to feel my wrath is Packratius, which angers me especially by asking users to provide their Pinboard password in order to duplicate a native Pinboard feature.

Do not do this

Packratius, I block you!

Next on the chopping block is IFTTT, which has set up an especially ridiculous workflow by requiring users to enter then Pinboard password, and then immediately using it to fetch an API token that they use for all subsequent calls. Their tech team has pleaded for mercy until October 24, and I have heard their pleas. But it's dumb that it is taking two years, multiple engineers, and millions of dollars in funding to begin to promise to fix this. I am itching to block them.

If you know of any other websites that ask for your Pinboard password, please let me know and I will gleefully bring the ban hammer down.

If you run a site that is asking people for their Pinboard passwords, you need to change it to ask for the API token instead. If you need time to do this, email me about your plans with a convincing display of contrition.

If there is something you are able to do with a password and unable to do with an API token, let me know and I will fix it immediately.

I have less of a problem with mobile or desktop apps that ask for Pinboard login credentials, provided those get stored locally. My beef is with sites that ask for passwords that get sent to a server somewhere. There is absolutely no need to do this this given the existence of an API token, and it needlessly puts users' accounts at risk.

Update 4:45 PM Oct 14: I just got news from IFTTT that they've changed their channel form to use the API token. Thanks very much to them for getting that done faster than promised!

Since the birth of the site, Pinboard has always offered a Delicious sync feature. You could enter your Delicious username on the settings page, and the site would periodically poll your public Delicious feed and add any new bookmarks it found.

Because Delicious appears to be in a terminal coma, and because working around bugs in their RSS feed has historically consumed a lot of development time, I am going to turn Delicious sync off effective October 1, 2014. If you want to keep hooking the services together, you will still be able to do it through an outside service like IFTTT or Zapier.

Of course you will still be able to import all your Delicious bookmarks, as well as export Pinboard bookmarks in a format that Delicious can (sometimes) read. The only thing going away is the automatic sync.

This week I'll be running a little experiment in link rot, in preparation for an upcoming conference talk. I'm interested in quantifying long it takes for a typical link to go offline, and if this rate is steady or changing with time. Pinboard now has enough bookmarks (about 100 million) to make this information interesting.

This research is important because we don't have a lot of data for how link rot affects stuff people actually care about. Presumably things that you've bookmarked are more important to you than some random URL off the street.

To run the experiment, I am going to be drawing a few thousand links at random from the entire pool of Pinboard bookmarks. This will include private bookmarks, which make up about half the Pinboard collection. I'll use a combination of scripts and my own weary hands to figure out what proportion of links still point to the original material saved. The URLs I look at will not be associated with your username, and no one except me will look at them.

I will publish some aggregate information about what I find, and use it to seek glory, and persuade people to sign up for archiving. But I won't release anything that could lead back to specific users or links.

If you are uncomfortable with this research and wish to opt out, please email me with your username, and I'll keep your bookmarks out of the pool. If you have questions, ask me on Twitter or email me privately, and I'll be happy to answer them.

Today marks five years since I launched the website that my mom still refers to as 'the other bedbugs'. Happy business birthday to me!

Any site that aspires to be an archive starts life with a credibility problem. The Internet is strewn with the corpses (or in some cases, zombies) of sites that once promised to save your links forever. As people keep discovering, building a bookmarking site is easy, but making a business of bookmarking is hard. Like one of those leathery, spiny plants that is able to thrive in the desert where everything else dies, I have tried to find ways to adapt to this hostile business environment. And I have feasted on the flesh of my rivals!

I raise this brimming skull to the awesome group of users and fellow-travelers who have made it possible.

It's my tradition to post updated statistics about the site:

2010

2011

2012

2013

2014

bookmarks

3.5 M

27 M

53 M

76 M

97 M

tags

11 M

76 M

135 M

178 M

212 M

active users

2.8 K

16 K

23 K

23 K

24 K

bytes archived

200 G

3.0 T

5.9 T

8.8 T

14.2 T

downtime

6 h

29 h

22 h

12 h*

some?

unique URLs

2.5 M

16 M

32 M

48 M

63 M

The biggest surprise (to me) is how predictable Pinboard has been over the past three years. Users come and go, like on every site, but the number of active users stays roughly the same. And the site makes roughly the same amount of money (around $200K) every year.

If you've ever run a small website, you'll recognize how weird this is. Typically everything in a small project—traffic, user count, revenue—is spiky. You spend a long time treading water and then big events happen that dominate everything else. This was true for the first two years I ran the site, but since then, things have settled down remarkably.

I regret that I totally forgot to keep downtime stats this year. There wasn't a lot of it, but I should probably track it better so I can brag about it next year, unless it goes up, in which case I will never mention it again.

Now back to some beard-stroking:

I see my role much like a small-town praire banker in the 1880's. My job is to project an aura of calm, solvency, and permanence in an industry where none of those adjectives applies. People are justifiably risk-averse when it comes to their bookmarks, and they are looking for stability. This means several things at once:

On the most basic level, the site just has to work.

On the design level, it means not futzing with stuff unnecessarily, except for bug fixes and basic improvements. Luckily there is so much work to do on Pinboard that I am immune to the temptations of a redesign. If there is a feature (or bug) you love in 2014, chances are excellent it will still be there, like a cherished friend, years from now when your trembling and aged hands go to make that final click.

Finally, there is stability on the business level. This means persuading people (including myself) that I am going to stick around, and then actually earning enough money to do that.

The money part turns out to be easy. People will pay for a decent service. As long as you stay small and don't forget to have revenue, you too can build a bookmarking website. There is plenty of room to specialize!

My strategy of pre-emptively antagonizing anyone who might possibly have an interest in acquiring or funding the site has worked wonderfully. In five years, I haven't received a single email from an investor or potential acquirer. The closest I came was a few months ago, when the new Delicious owners reached out to me about providing "vision", but I think they were just unfamiliar with my oeuvre. They learned quickly.

So the biggest risk in a project like this remains burnout.

Avoiding burnout is difficult to write about, because the basic premise is obnoxious. Burnout is a rich man's game. Rice farmers don't get burned out and spend long afternoons thinking about whether to switch to sorghum. Most people don't have the luxury of thinking about their lives in those terms. But at the rarefied socioeconomic heights of computerland, it's true that if you run a popular project by yourself for a long time, there's a high risk that it will wear you out.

It's not the fact of working on just one project that's the problem. This dude, for example, has spent much of his life building a Boeing 777 out of manila folders. Another guy (always dudes!) is slowly excavating his basement with toy trucks.

What burns you out is the constant strain of being responsible for a lot of other people's stuff.

The good news is, as you get older, you gain perspective. Perspective helps alleviate burnout.

The bad news is, you gain perspective by having incredibly shitty things happen to you and the people you love. Nature has made it so that perspective is only delivered in bulk quantities. A railcar of perspective arrives and dumps itself on your lawn when all you needed was a microgram. This is a grossly inefficient aspect of the human condition, but I'm sure bright minds in Silicon Valley are working on a fix.

Perspective does not make you immune to burnout. It just makes burnout less scary. I've gone through a few episodes since starting Pinboard, and I'm sure there will be more to come. People have been very understanding about my occasional need to flee the Internet. I find that the longer I run the site, the more resistant I become to the idea of ever giving it up, even if I need to take the occasional break. It is pleasant to work on something that people draw benefit from. It is especially pleasant to work on something lasting. And I enjoy the looking-glass aspect of our industry, where running a mildly profitable small business makes me a crazy maverick not afraid to break all the rules.

Most of all, I'm gratified that people have been patient and considerate over the whole lifetime of the project. There has been a lot of goodwill sent my way that makes my job vastly easier. Thank you to all the people who have used the site over the years, and the many people who have helped me build it and keep it running. To my competitors: I will crush you! To everyone else: you're wonderful! Upgrade!

Amazon S3 is designed to provide 99.999999999% durability of objects over a given year. This durability level corresponds to an average annual expected loss of 0.000000001% of objects. For example, if you store 10,000 objects with Amazon S3, you can on average expect to incur a loss of a single object once every 10,000,000 years. In addition, Amazon S3 is designed to sustain the concurrent loss of data in two facilities.

This is an impressive number, but it's utterly dishonest to make such claims. It implies that there is a less than one-in-one-hundred-billion chance that Amazon will abruptly go out of business, or that a rogue employee will cause massive data loss, or an unexpected bug will result in massive data loss, or a defect in storage media will cause millions of devices to fail silently, or a large solar flare will destroy equipment across three data centers, or that a comet impact will destory three data centers, or that a nuclear exchange will destroy three data centers.

I think these events are all incredibly unlikely, but none of them is one-in-a-hundred-billion unlikely. Yet here is Amazon not only making that argument, but implying that you can safely use S3, a service that launched in 2006, for another ten million years.

Rare events are rare! That's why promises past five or six nines of reliability are functionally meaningless. At that point the "unknown unknowns" must overwhelm any certainty you have about what you think your system is doing.

The risks you failed to model will become obvious in retrospect, and make for an entertaining post-mortem, but that won't get anybody's data back.

Promises like Amazon's should serve as a kind of anti-marketing, suggesting that the company has not thought seriously about the limits of risk assessment and planning.

I suggest the following rule of thumb: if you can't count the number of nines in the reliability claim at a glance, it's specious.

Of course this rant is available in book form, phrased better than I have here. But it's worth repeating at every opportunity.

By now you may have heard about the heartbleed bug, which affected many websites that use encryption. This is the Spanish Flu of security bugs—it hit almost everyone and took a disproportionate toll on the healthiest, those sites that followed security best practices.

Servers affected by the bug (including the Pinboard site and API) could be tricked into sending private information that happened to be in memory. This included authentication cookies, passwords, secret API tokens, and any data you posted to the site. There is also evidence that the bug could expose a site's private key, which would mean anybody eavesdropping on a Pinboard connection could decipher it. The bug was live from the spring of 2012 until I patched the servers on Monday night.

Worst of all, there's no way to tell from logs if anyone's data was exposed. It's possible that no one looked at any Pinboard data; it's also possible that the site was completely compromised.

This morning, I issued a new TLS certificate for the site, with a new private key. Now that the servers are trustworthy, please do these two things:

Reset your API token. On that same page (https://pinboard.in/settings/password), click the reset button. You'll need to update any outside services and apps that use the API token to authenticate. (Remember never to share your Pinboard password with any third party, no matter how nicely they ask. Outside sites should be able to get all the access they need using only the API token.)

In layman's terms, the bug was the equivalent of asking a stranger "hey, what's up?" and having them tell you their most private thoughts, going on about their divorce, sharing their credit card info, whatever was on their mind at the time. You could keep asking "what's up" as often as you wanted, and hear new things each time. Worst of all, the stranger would have no recollection that it had happened.

Of course, I heard about heartbleed before it was cool. The servers were patched by around 7 PM on Monday night, California time, before half the Internet started casually playing with Python scripts that exposed the bug.

So only truly malicious people could have seen your Pinboard secrets. Hooray!

In awful times like these, it's good to stop and reflect on the timeless wisdom of the Pinboard security page:

The site will be down for maintenance this Saturday, March 22 from 19:00-21:00 California time (UTC/GMT -7). For people in Berlin, that's 04:00 Sunday. For Australians and kiwis, that's Sunday afternoon. The database is just about to run out of storage space, and unfortunately there's no way to free up room without shutting the beast down for a few minutes. I'll do my best to keep the downtime brief, and will post updates to Twitter as I go.