Development and UX from Michael Mahemoff. Maker of Player FM. Previously: Google, BT, O'Reilly author.

Menu

Here’s just a few random thoughts on Black Mirror Bandersnatch, which came out yesterday. I’d have posted these as a few lazy tweets, but didn’t want to post spoilers there.

Haven’t watched it yet? Congratulations, you have a life. But go watch it anyway. Binge the first four seasons beforehand if you haven’t seen them yet. Also worth it.

Thoughts with some very mild spoilers ahead:

“Choose Your Own Adventure” previously moved from book form to interactive game form, as depicted in Bandersnatch. Netflix’s experiment takes it another step to streaming form, which is interesting because there’s no way to look at the code and figure out every path. It also means they could change the path dynamically, e.g. introduce an easter egg for just one day.

But what about narrative, in a world where viewer decide the story? This is a popular trope in interactive storytelling, the tension between the producer guiding the story and the viewer feeling free to make decisions. The entire story of Bandersnatch addresses that tension better than anyone could express in a linear sentence. Free will in this medium is an illusion.

Netflix’s long term goal likely involves VR and personalization, i.e. personalized videos similar to the recent bubble of books where $child_name is the star. It’s not hard to see how gaming and movies eventually converge. A degree of interaction is an important step towards this future.and

Netflix’s short term goal likely see this as a popular move for some kids’ content. I’ve noticed Netflix also has Jeaopardy in its catalogue and it’s a no-brainer for the platform to be used for interactive TV game shows, as has long been done with cable remotes, but better.

A few technical observations:

The number input (for the safe) shows this is not just going to be a simple 2-way choice mechanism. I’m guessing they have set up a protocol to support input of any unicode string.

I noticed the episode doesn’t work on my Shield running recent (probably latest?) Android TV OS (Update: it works now, a day later, it must have required an app update.) Netflix has a lot of clients and it will be a big effort to implement – and evolve – this protocol ubiquitiously. They’ll also have to think about different input constraints, e.g. typing on TV might support voice input.

I also noticed the episode isn’t possible to download, and at one point saw it buffering after making a decision (on a bad network). The app probably pre-emptively downloads a minute or two of both decision paths so it can continue seamlessly after each decision. It would also probably hang on to the unchosen path in the event it’s needed later. Overall, there are some very interesting computer science and network topography problems for anyone wanting to optimise a system like this.

Share this:

At some point, a single database instance starts to creak as more objects are added to it, even with read-only replication. A battle-proven strategy here is to scale horizontally via sharding, however there be dragons. Here are general design principles on sharding with relational databases such as MySQL and Postgres.

Only shard when you have to. Premature optimisation is, after all, the root of all evil. Sharding adds more servers to build, maintain, failover, and backup; and it makes apps more complex.

Each object/record has its own GUID to uniquely identify it across all servers. The GUIDs indicate the shard this object lives in. When requests come in, they specify a GUID which the server can then map to a particular shard. Instagram Engineering has a good overview on GUID generation, there are various options with pros and cons.

Use many virtual shards and distribute evenly between them. It’s best to assign each object to one of thousands of virtual shards, and then map those to physical shards (ie database instances running on a particular host). e.g. You might assign an object to shard 1331 out of a possible 10,000 shards, using a simple random number or modulo function to ensure each shard has approximately the same quantity. This virtual shard its permanent home and will never change. You then map 1331 to “database server 3”. The reason for this indirection is so you can easily split up data as the system grows.

Related content lives in the same shard. Typically, content owned by a single user/team/company should live together on the same (virtual) shard. For many applications, the main queries that need a quick user response are all within the same object graph, ie some kind of join between a company, its workers, and their content. It makes sense to store all of this in the same shard, so if you have a natural hierarchy, ensure each class’s shard is initialised with that of the root class (e.g. each “sale” is assigned to the same shard as the shard of the “salesperson” who made them, and each “salesperson” is assigned to the same shard as the “company” they work for).

Slave replication is only for backup/failover. This is advocated by he Pinterest paper. Replication can cause weird “time travel” bugs, where an application reads stale data from a slave and then uses it to update the master. Sharding is sufficient to replace the performance benefits of reading from multiple slaves, so replication should only be used for backup and failover purposes. Each shard (and any central database) gets its own slave.

Share this:

A CDN asked me why they should respect the HTTP “vary” header. My reply was this.

“vary” is needed because the app uses “PJAX” architecture used by many dynamic web apps and embraced in frameworks like Rails’ TurboGears. Simply put, all links are “hijacked” so that when the user clicks on them, the server responds with a slimmed-down version of the full page. It doesn’t have headers or footers.

It still comes from the same URL, so we have “full pages” and “partial pages” both served with the same URL. If the user enters a URL in their web browser and the CDN responds with a “partial page”, it will look like rubbish and fail to function, as there won’t be any stylesheets or scripts (as well as missing page headers and footers). And similarly, if the hijack ends up receiving a full page with headers and footers, it causes initial page scripts to be re-executed, leading to bugs and causes memory leaks.

With the CDN ignoring “vary”, I’ve had to hack the client PJAX library to force the URL to change when it’s hijacked (appending ?xhr=true). However, it still fails on redirects, as I can’t do anything about them on the client side. I added another hack to change it on the server side during redirect, but now people have reported this is still an issue with the homepage, which I can’t even track down, so I’ve now had to turn off caching on that page and still unsure what other pages are affected. Hopefully, that makes it clear why I’m screening for CDNs which respect the “vary” standard.

Share this:

Ansible is very useful for managing multiple servers, but one of its weak points is lack of control over sequencing tasks.

The basic assumption is that you can execute all tasks because it embraces the principle of idempotency. If you add a new task, you should just be able to run the whole playbook again and there will be no side effects because all the other tasks are idempotent.

While that’s true in theory, there are many reasons why you’d want to run only a subset of a playbook:

Most importantly, performance. Some stuff can be slow. Even checking state, to prevent actually making any changes, can be slower than you’d like. Just a second delay for a playbook of 60 tasks would mean waiting a minute every time you want to do something.

While developing playbooks, it’s helpful for debugging purposes to execute only a subset of tasks. (Also, see previous point on performance. You don’t want to wait a minute to find out you had a typo.)

Some tasks should not be done by default. e.g. you may need to restart servers periodically to detect a memory leak. This would have to be “forced” because normally you’d just have a “service” rule indicating the server is “started”. So no action would take place because the server is already running, whereas what you need is a separate rule indicated the server is “restarted”.

Ansible’s primary answer to this is tags – you can specify a list of commands to be run by calling ansible-playbooks with a —-tags argument, and you can skip over other tasks using –skip-tags. These are useful, but limited, mainly because you can’t say “this task should only be executed if it’s tagged”. Yes, you can skip over it with skip-tags, but it will be run by default. e.g. if you have a “restart server” task, it will always be run as part of the standard playbook execution, unless you remember to skip it. Cumbersome.

The common workaround for this is to introduce another concept: variables. Instead of tagging such tasks, you make them happen only when some variable is true. And then default the variable to false. This works well, but things have suddenly got quite confusing to follow. The playbook is now an obstacle course where, for any given task, we have to figure out how to delicately step around some tasks while executing others.

For this reason, I’ve concluded the best approach is to make a front-end control script. It can’t be a “top-level” playbook, because another limitation is that playbooks can’t execute tagged or skip-tagged tasks. So it’s a plain-old bash script. I’m building up all typical invocations of Ansible in this control script. Every time I want to invoke Ansible, I ensure there’s a specific function present. Over time, I’ll be able to consolidate these and adjust the underlying playbooks as necessary.

But no, Google won’t embrace it. Just because something is open source doesn’t mean everyone has equal influence. There’s still control at the top of the repo and this is why Google forked Webkit, an effort to control its own destiny instead of relying on the very same company that controls Swift.

Furthermore, Google and Android is still engaged in a tired, ongoing, battle with Oracle over Java, another language that is – “for varying definitions of” – open. And Oracle isn’t even a direct competitor.

It’s true that there’s a lot to be said for a more dynamic, scripty, language on Android. While Android Studio has done a lot to improve the developer experience for Android devs, a lot of work done is exploratory UI, something that a language like Swift can help with.

If Google were to embrace a dynamic language, and assuming they don’t start from scratch, there are really only three contenders: Go and Dart, since Google completely control them (more likely Dart as it’s more suited to UI). And JavaScript, since it’s immensely popular and under no one company’s control (and as a bonus, Dart is designed to compile nicely to JS, so Google can still support it as a higher level alternative). Just as Swift has made iOS a much more approachable platform for casual developers, embracing the world’s most popular language could be a nice boost for developer traction in the Android world.

The install model has truly gone from being a non-starter to something that more closely maps the needs of users than any native platform has achieved. Here I will reflect on the current state of the web as a platform for apps, identify some remaining concerns, and propose where the biggest wins will come from.

The web can do apps

While the HTML5 era (2008-2012, say) introduced many conventional app components, they emerged in parallel with the mobile revolution. I don’t say “so-called revolution”. It was an actual revolution in every way, changing forever how apps are designed, developed, and distributed. HTML5 was already ushering in a flood of new tech, it was never going to be possible to make it all mobile-savvy at the same time, and this led to a world where people went crazy for apps.

The progressive web movement plugs the gap in several ways:

Push notifications. These are lifeblood for many apps. Critical to the functionality of messaging apps like Skype and Slack, and – in a world of fickle users and high churn rates – vital to retention for all apps.

Background processing. Doing stuff when the user doesn’t have your app open is also vital for a modern app’s functionality, performance and offline capability. This is about the app acting as your digital assistant, not just something you interact with for the time it’s on the screen in front of you.

Low-level APIs. As part of the extensible web manifesto, developers now get access to the low-level innards of the web stack. This not only helps the standards to evolve, but lets developers deliver useful functionality unanticipated by committee-driven standards processes.

Furthermore, it comes at a time when browser performance is strong and web debugging tools are built into all the major browsers and have become stupendously useful. All of which means, it’s now possible to replicate the functionality and interface of many popular native apps.

But how will users install web apps?

Until progressive web movement came along, websites never had a chance on mobiles. Browser bookmarking features were about the same as in 1995 – just a flat list. Users didn’t know how to install to homescreen and even with libraries prompting it, it gave no confidence the site would work offline. Offline tech itself, and the website couldn’t do anything in the background as mentioned above above.

Now – with progressive web apps – two things have changed. Firstly – at least on Android – web apps have been elevated in their presence. The task switcher presents each recent website alongside each recent native app; they are all equivalent. The traditional 38-step “add to homescreen” process has been replaced by a simple menu item in Chrome. And most importantly, the browser will proactively prompt an install.

The progressive web’s install model rocks

A common argument for native apps has been the importance of app stores (insert $legally_acceptable_synonym for app store on your platform of choice). The web’s counter-argument has heretofore been either “Ah but the web has search engines” or “Ah but there are x billion apps on the app store and only the top 10 get any installs”. Neither of these arguments hold much water for me.

SEO is a real thing and search rankings are as much or as little a meritocracy as app rankings are. For every startup blowing $5K a day on Facebook app install ads, there’s another startup paying for fake forum posts in the hope of Google juice. The “long tail” argument also applies just as much to the web.

So how does “progressive web” improve things? By letting users progress from a fly-by visit to a fully installed app, on their own terms.

The conversion funnel from “non-user” to “user” for a traditional app looks like this:

User discovers app.

User installs app. Waits a minute or two for download. App is installed.

Getting users from (1) to (2) is extremely hard for developers. Most users don’t want to clutter up their phone with hundreds of apps, don’t like to go through the hassle of downloading the app, and don’t want to feel the remorse of installing a lemon (even a free lemon). Pulling off a large install base relies on a difficult-to-achieve store ranking, a viral loop that is fleetingly rare in practice, or several dollars of ad spend per paid install.

This is a much more logical transition. Instead of making the install decision based on the store listing, the user makes the decision based on actually interacting with the app. At no point are they obligated to install it, but as they gain confidence in it, they can decide to do so. From the developer’s point of view, it’s easier to win long-term users if you have a product that’s compelling (and if not, why do you care about installs? You will lose users anyway if you don’t have good retention).

Admittedly, going from (3) to (4) is still hard. If the user hasn’t yet installed the app, how confident can you be that they will re-visit your site often enough to be prompted? Part of the answer comes from background notifications, which means the user can still be engaged even without that install. As well, if the user’s social contacts keep recommending the same app, it will likely lead to an install prompt. Compare that to a native share, which would often lead to nothing unless the content was amazingly compelling.

Indeed, it’s likely developers will care less about installs in general, as long as they still have users who are engaged via notifications and spontaneous interactions they wouldn’t see at the moment. Spontaneous usage is particularly compelling when you consider the physical web. There’s no way I’m going to install an app for the restaurant I’m in or the airport I’m flying to … but I’ll gladly open a rich web app from a prompt that shows up on my phone.

Remaining concerns

So that’s it huh? The web came back in the third act and triumphed. No. Not even. There are still many challenges ahead.

Challenge: Apple (“There’s a 586 billion dollar elephant in the room and it’s not happy Jan”)

Here are some facts about Apple, which – when combined – lead to skepticism about any efforts to progress the web:

Apple and iOS are hugely influential. It’s the platform companies care about most when it comes to development efforts.

Browser innovation on iOS is controlled by Apple. Google, Firefox, and Opera may produce their own iOS browsers, but they will still run on Apple’s engine and any progressive web tech on iOS therefore relies entirely on Apple’s whims.

Apple’s incentives for browser innovation are “mixed” at best. While it wants a great user experience – including web interaction – it also has a lot to fear from a platform beyond its control that lets developers “write once, run many”.

Apple is sticky. Most mainstream users don’t care about detailed OS features like homescreen widgets and notification interfaces. Even the tiny iPhone 4 screens were not enough for most users to look elsewhere. That was at least a very clear and visible advantage for the competition. So how much will users care about a better web experience if they can still get the same apps natively? It’s a moderate OS benefit and may help Android (or other platform users) with retention, potentially clawing back market share in the long term, but it’s unlikely to sway many users away from Apple. It’s too difficult a concept to even explain, let alone for anyone to really care about it if they aren’t already using it.

Don’t hold your breath for an iOS which supports native-level video calling, gaming, and podcasting. Some features will make their way over time, but by then, there will be even more features on both native and – on other platforms – web. The only question worth asking is, how much does it matter?

For me, the answer is “not as much as you might think”. People are still making web apps anyway. The whole thing about progressive web apps is they are progressive, not binary. So your web app can still work quite nicely on iOS, but do even more on other platforms. This won’t apply to all genres, e.g. it’s quite useless to make a voice calling app without push notifications.

Furthermore, there is a whole class of developers with iOS apps but lacking apps on Android, Windows, and other platforms. Enhancing their existing web presence is an increasingly bright alternative to hiring dedicated native developers, considering they usually have a web app and developers already (even more so if they are one of many companies now running Node/JavaScript on the server side). It may not give as perfect an experience as a native app, but it’s infinitely better than doing nothing on those platforms and may well be as good as they would produce anyway outside of iOS.

Challenge: Discoverability (“Websites on a plane”)

It’s still difficult to find good, installable, web apps. There are some hints when you’re already using one – e.g. the prompts to install on home screen or receive push notifications, and the color scheme supported by the web manifest protocol. However, if I want to find an installable app to do X, where do I go? On native, I can just search in the store.

On web, I can search in … Google? Nope. I’ll usually get a pile of ugly and ad-ridden sites that happen to be old enough to have reached high rankings. Thankfully, Google does care more about performance and mobile-friendliness now, but it still doesn’t come close to the app browsing experience of a native app store.

This is exactly what Chrome Web Store should be doing in 2016. I hope Google is working to finally bring the web store to native. (Or, amid much controversy, integrate web apps into Google Play.) And other browsers are similarly working on this problem.

Challenge: Native keeps moving forward (“2010 called.”)

Native remains a fast-moving target. The web may have caught up on many features of a modern smartphone, but native has moved on to power virtual reality, cars, home appliances. Additionally, there are still many basic functions that aren’t yet possible on the web, though they are being debated and worked on. e.g. I still can’t make a full-fledged offline podcast app because of SSL and cross-domain restrictions. Bluetooth, USB, background audio … these APIs are all being worked on, but aren’t there yet.

Challenge: Payments (“Shut up and take anyone’s money”)

Frictionless payments obviously drive a huge amount of activity in the app world and this is a realm where the web really hasn’t changed since the introduction of smartphones. In-web payments is a complicated 4-way problem – there are users, browsers, operating systems, and payment providers. Add security, UX, and privacy to the mix, and now you see why there are casual games earning millions each day on native platform but nothing on the web.

If this can be cracked in the context of progressive web apps, game-changer.

Challenge: Native app streaming is also progressive

The progressive engagement model is no longer exclusive to the web. Never afraid to boil the ocean, Google has now begun previewing native apps by streaming them from the cloud. Yes, it relegates your device to a “dumb” display unit and runs the app on Google’s servers, at least until you decide to install it. It’s a very different type of progressive engagement, but it may steal some of the web’s progressive thunder nonetheless, especially if Apple was to follow suit.

Conclusion

Progressive web technologies are making it possible to go beyond just rich websites to “real deal” digital assistants like people have become accustomed to with native apps. The install model mirrors the way an app or service builds trust over time, and for this reason, it goes beyond the binary “installed or not” situation for regular native apps. While many challenges remain, the good news is … it’s progressive. Developers can already see the benefits by sprinkling in these technologies to their existing websites and proceed to build on them as browsers and operating systems increase support.

I admit the former is more concise, bt cncs dsnt lwys mn bttr even if you can parse it.

The rules are simple, please do this when you mention a date:

Include the year. There are 80 trillion web pages and most of them were written before a few months ago, so if I see a date without context, I have no evidence it refers to a time in the future. It could be any time in the last 2 decades.

Name the month. Let’s not get involved in a big debate about MMDD versus DDMM versus YYYYMMDDAAAA🙏🙏🙏🙏ZZZZzzzz. When we’re displaying dates to regular users, keep it simple and use a format everyone immediately understands – the month name. Or an abbreviation thereof. I realise that’s not international-friendly, but the date presumably appears with surrounding text, so use the same language for the month and use one of many i18n frameworks to localise it if you have multiple languages. [1]

Name the weekday. Come on, would it kill you to tell me what day this is on as well? That’s a big deciding factor for many people and helps to plan for the event and remember exactly when it happens.

Count it down. Here’s where digital formats can better traditional printed formats. The date display can be dynamic, so you can show a countdown when it’s appropriate. Again, it helps to make the date more meaningful and can also create some excitement around the event.

Add to calendar. In some cases, you might provide support for adding the date to users’ calendars. There’s unfortunately no great standards for this, but there are tools.

Share this:

1. Swift everywhere

Swift 2 built on Swift’s popularity and it’s clear this language will fly far beyond the confines of iOS. It’s more open than anyone could have expected from Apple and unambiguously the future as they see it, it interops well with the giant legacy of Objective C components, and developers genuinely dig it without a whiff of reality distortion.

Swift is Apple’s answer to Node – app developers will use it to make their first move from client to server. More than that, it has the promise of a “write once, run many” framework. Sure, these frequently lead to mediocre apps, but a lot of developers already have mediocre apps on the web, Android, and (if they have an app at all) Windows. It’s not hard to see the attraction of a turnkey solution that will crank out half-decent apps for platforms that aren’t running in the CEO’s pocket.

2. Microsoft open sources Edge #MoarSatya

Microsoft recently open sourced Edge’s JavaScript Engine (“Chakra”). Edge is already free as in beer, as with its predecessor – MS Internet Explorer – and pretty much every other web browser, ever. Yet, Edge (and IE) is closed source, in much the same way as every other major browser has fully open-source engines and Firefox + Chrome are essentially all open source. I don’t need to go into a long monologue here about the pros and cons of open source; suffice to say, I believe open sourcing Edge will improve its quality and compliance with emerging web standards.

There is an additional reason to open source Edge: it will help to unlock what must be one of the major strategic goals for Edge – to run on platforms beyond Windows. MS has been busy buying and building apps compatible with iOS, Android, and OSX. Browsers are no longer dumb clients – they sync user settings and data across devices, and MS wants Edge users to remain Edge users when they move away from their PC. Furthermore, much of the web is developed on OSX, and MS will make it easier for developers to build first-class Edge experiences if they can ensure Edge is running there without making developers jump through hoops.

3. Google helps users discover mobile web apps

Google has been pushing the progressive web app mantra for a couple of years now, and there’s a huge amount of problems that can now be solved using a web app. While the “install to home page” prompts are helpful if you’re already using an app, how do you discover the app in the first place? Serendipity only gets you so far. Maybe you can use Google Search? Hardly. You’ll see several native apps first (if searching on Android), followed by some crap web page which has top place because it’s been online since 2003 (before its developer had heard of JavaScript, and it shows). Meanwhile, you have Chrome Web Store exclusively for desktop and Play Store not showing web apps.

It’s clear Google cares about the web and making web apps thrive, and its search business depends on this. So how will it bring the app store experience to the mobile web? I won’t be so bold as to predict web apps in the Play store – the last thing Android team wants is millions of “glorified bookmarks” contaminating the listings. Chrome Store for mobile? Maybe. Better mobile web app search? Very likely. However it happens, I believe it will be a lot easier to find the best “timezone web app”, “calculator web app” etc for your phone by the end of 2016.

4. Netflix produces daily news/entertainment show

Netflix will create a new daily show in the mould of The Daily Show and The Late Show. This would be a big departure from their typical evergreen model, which has certainly been vital in building a diverse catalogue under their full control. But there is good reason to expand in this way – ongoing shows mean users can always log in and expect to see something fresh and topical. No more frustrating moments hunting around for something decent when you’ve finished a multi-season binge. Additionally, they benefit from viral clips circulating with that Netflix watermark. (Also, the distinction between evergreen and topical shows is not entirely cleancut; old talk show interviews can still generate giant numbers on YouTube, while series such as House Of Cards will look aged before long.)

The 2016 election is sure to be a perfect backdrop to launch this onto their captive audience, probably with a companion podcast.

5. Podcasting as an art form

Just as TV has become something of an art form in recent years, we will see podcasting viewed in the same light, and as something with distinct properties from radio. Needless to say, the bingeworthy nature of Serial is a big part of this, but it’s also a result of business serials like Startup and fiction like Night Vale, Limetown, and The Message using the medium to its fullest.

6. All-You-Can-Eat video streaming from Google

Google already has all-you-can-eat music streaming subscriptions and also released YouTube Red as an ad-free version of YouTube with offline capability. A natural next step for Google Play would be all-you-can-eat video. It would be similar to Amazon, which still has premium videos for purchase or rental, but has all-you-can-eat on Prime. Indeed, Amazon is part of the reason Google should be doing this – thus far, they have made it all but impossible to consume Prime videos on Android (it requires sideloading Amazon’s marketplace app, and even then, a lot of videos aren’t compatible). This leaves just Netflix as the only viable all-you-can-eat platform in most markets, and Google therefore stands to bolster Android itself as well as generating revenues from such a service.

Share this:

Having worked on both sides of developer relations, here are some thoughts about different levels of maturity for developer relations.

LEVEL 0: No developer relations

No internal effort is made to promote the platform, support developers, or capture their feedback.

LEVEL 1: Informal

No official developer relations staff or programme, but some developer relations handled by other functions. PR may be promoting the platform, business development may be partnering with and supporting developers.

LEVEL 2: High-touch

High-touch, often stealthy, relations with prized partners (i.e. large, established, companies or those with sufficient resources to build showcases for new features). This is a “don’t call us, we’ll call you” outreach which may entail the platform providing funding or direct technical capability to build out the integration, and often working with as-yet unannounced technology so it can be launched with a set of poster-child applications.

LEVEL 3: Evangelism

Promoting, explaining, and supporting the platform at scale via conferences, partnerships, and online media. Proactive efforts to recruit large amounts of developers to use the platform.

LEVEL 4: Advocacy

A 2-way relationship in which the platform’s own staff sees themselves as not just advocating for the platform, but as advocating for developers using the platform. With this mindset, developer relations plays an active role in feeding back real-world bugs and feature requests, and building supporting tools to improve the developer experience.

LEVEL 5: Quantified

Metrics-driven approach in which the return-on-investment for developer relations is understood and outreach efforts are able to quantified, both with high-touch partners and at scale.

Now some caveats about this.

First, how not to use this model. Any maturity model immediately makes you think companies should be ascending to the top level, but that is not the case and not the intention here. Ascending comes at a cost that may not be justified; clearly, a pure platform company (e.g. Twilio, Stripe) has a lot more incentive to get to the top than a product company with an experimental “labs” API, for example. There is financial cost, additional risks, and distraction to the rest of the organisation; all that needs to be weighed up. The purpose of this model, then, is to provide useful labels to facilitate these kinds of decisions. Not to imply one is intrinsically better than another.

So the way to actually use this model is simply to be true to yourself. Where are you now and where do you want to be? If you’re happy at level zero, scale any devrel back. If you want to shoot for level 5, start ramping up. Companies often differ widely between official and actual practices. A company may have no official developer relations programme, but instead have a technical marketing team or a super-engaged developer team who perform the same function. Likewise, no amount of fancy business cards will compensate for a developer relations programme that doesn’t develop and rarely relates. Hopefully, this model helps people to understand where they’re at.

Final caveat: Turns out you can’t pigeonhole a complex organisation into a simple number rating. The lines will blur when applying these definitions to $YOUR_FAVORITE_EXAMPLE. You may apply these definitions to a whole company, a single division, or a single product.

Share this:

I’ll explain here what Bloom filters are and how you might find them useful for recommendation engines. I haven’t used them yet in production — just sharing what I’ve been learning.

Why Bloom Filters?

I was thinking about recommendation system algorithm. Not the main algorithm: how do you generate good recommendations. But instead, an important “side algorithm”: how do you keep track of recommendations users have previously dismissed? All the genius recommendations in the world aren’t going to matter if you keep showing the same results.

The most obvious solution here would be to track everything. Simply store a new record for every “dismissal” the user makes. But that’s a lot to store in a high-scale system, e.g. if 10 million users dismissed 20 items each, you have 200 million records to store and index.

So this is where Bloom filters come in as a highly compressed way to store a set of values. The catch is: it’s fuzzy. It’s not really storing the set; instead, it’s letting you ask the question you want to ask, which is: “Is X in this set?” and coming back with a probabilistic answer. But that’s okay for something like a recommendation system.

Here’s an example. A user Jane has dismissed three articles, identified with their IDs: 123, 456, 789.

Under the traditional model, we perform a standard set inclusion check (e.g. check if a database row exists) and come out with a definite answer:

Jane:
123
456
789

Q: Is article 888 in the "Jane" set?
Algorithm: Check if 888 is in [123, 456, 789]
A: No. I'm sure about that.

Under the fuzzy Bloom filter model, we end up with some funny value as a fuzzy representation of the whole set, and then we can get a probabilistic answer about set inclusion.

Jane:
01101001 (this is the Bloom filter)

Q: Is article 888 in the "Jane" set?
Algorithm: Check against the Bloom filter (details below)
A: Probably not. But maybe. About 5% likelihood it's in the set.

Deriving the Bloom filter

So in the previous example, how did we end up with that representation of the set (what I playfully refer to as 01101001). And what did we do with it?

It’s fairly simple. Remember, this is the only thing we store and the set builds up over time. So the Bloom filter starts out as empty and each new set member adds something to it.

The real representation is a bitwise vector, let’s go with 8 bits:
00000000

So when user Jane is created, her Bloom filter is 0000000.

Jane dismisses article 123. Now what we do is, we compute some hashes of 123 using different algorithms. Since we have decided to make our Bloom filter 8 bits, each hash algorithm should give a number between 0 and 256, so we can store the result. Let’s assume we use two hash algorithms to hash 123. One ends up with 64 (01000000) and the other with 33 (00100001). So now our Bloom filter is:

01100001

When we get a 1, we set the bit to 1. When we get a zero, we do nothing. So yes, over time, this will fill with 1s. That’s why we have to choose a big enough bloom filter size.

Going on, the next dismissal is 456. And maybe we end up with hash values 01001001 and 0110000. So the first of these has added a new “1” to our previous value of 011000001:

01101001

And finally, we might end up with 01001000 and 00100000 for ID 789, neither of which light up any new bits. So we still have the same Bloom filter as before.

01101001

Is X in the set?

Now we have Jane’s Bloom filter, 01101001. This is a fuzzy representation of [123, 456, 789]. We can then ask, for any given value, is it in the set?

e.g. if our recommendation algorithm comes up with 888, should we show it to Jane. We don’t want to show it if it is in that set of previous dismissals. We compute using the same hash algorithms as before and perhaps we end up with 00101100. It lit up a different bit (the 6th one), so we can say categorically, it’s not in the set. We know that for sure because if it was in the set, all those bits would be on. We know for sure it’s not in the set of dismissals, so we can confidently recommend it to Jane.

Take another recommendation we might end up with – 456. Do we show it to Jane? Well, is it in the set of previous dismissals? We compute and get 01101001. It fits within our Bloom filter, so there’s a good chance it was in the list of values that was used to build up the filter. But no guarantee. We might end up with a value of 00001000 for another ID, e.g. 555. This would also fit the Bloom filter and we can be no more certain that it was in the original set than we can be for the 456 value. So, it’s probabilistic. You can be certain some things aren’t in the set, but you can’t be certain something was in the set. For a recommendation of 456 or 555, we can’t be sure. So in either case, we will not show Jane the recommendation and look deeper for more certain values.

Fine tuning

The example above just magically decided to use a Bloom filter of 8 and hand-waved around the algorithms. In practice, you will need to decide on those things and in practice it will probably be hundreds or thousands of bits; otherwise, every bit will quickly fill up to become 1. A cool thing is that there are precise calculations that can help you estimate exactly how big the Bloom filter should be, based on the expected number of items in it, combined with your tolerance for error. (If your algorithm can easily generate lots of good recommendations, you could have quite a high tolerance because it would be easy to skip over any potential matches.)

Share this:

G’Day

Welcome to Michael Mahemoff's blog, soapboxing on software and the web since 2004. I'm presently using HTML5 and the web to make podcasts easier to share, play, and discover at Player FM. I've previously worked at Google and Osmosoft, and built the Ajax Patterns wiki and corresponding book, "Ajax Design Patterns" (O'Reilly 2006).
For avoidance of doubt, I'm not a female, nor ever have been to my knowledge. The title of this blog alludes to English As She Is Spoke, a book so profoundly flawed it reminded me of the maturity of the software industry when this blog began in 2004. I believe the industry has become more sophisticated since then, particularly the importance of UX.
Follow @mahemoff