Make lots of short urls really really fast

Url Shorteners have
long been an
interest of mine, but my ID generation method up to
this point has been making random strings and checking that they don’t already
exist.

And I must admit, making random strings is really convenient!
The code is relatively small and simple.
But random strings have some issues at scale.

What’s wrong with random strings?

When you need to generate lots of unique IDs, you have to ensure that they never
conflict.
This is fairly avoidable with good hashing algorithms and things like UUIDs.
But those are both veeeeeeeery long, which means you can get away with leaving
them to their randomness.

But when it comes to short random strings, it means you must keep a
canonical set of all strings that have ever been used, then check each time you
make one that you haven’t already made that same one before.
Otherwise, you can’t guarantee their uniqueness.
And that can only really be done by centralizing your
string-uniqueness-monitor-thing.
Which means you now have a single point of failure, and a very overworked
component (at scale, anyway).

So what does sanic do?

sanic gets around these issues by being deterministic.
It creates a timestamp, and then increments a limited counter for every ID it
generates during that timestamp.
If it ever runs out of space in the ID counter, it simply waits until the next
timestamp comes around (which is between 1 millisecond to 1 second, depending on
what you choose).

It also uses a worker ID (which must be unique), so that multiple servers or
processes can all be generating IDs in parallel.

With these pieces of information, it simply appends them all together, so there
can never be the same combination of timestamp + counter + workerID (until a
number of years have elapsed, and the timestamp overflows, anyway :)

So how do I use it?

Right now, you probably shouldn’t ;)
While sanic is fairly complete for its needs, it lacks tests and documentation.
Also, you have to be careful when using it, as it has no built in concurrency
safety.
If you use it with the Golang http standard library, you’re gonna have a bad
time unless you handle the concurrency issues yourself.

Fear not, though!
I’ll be updating sanic with tests, and likely have a followup blog post where I
give an example of how to use it.