How can we Estimate Better?

Here’s a fun, surprisingly relevant problem that Aion’s creators doubtless have been wrestling: how do we estimate, and how can we do it better?

You can look at it like this – Go, extended metaphor!

How many Jellybeans are in the jar?

Imagine that you’re working at a summer camp, and today you’re in charge of distributing snacks. All that’s on hand is a large glass jar full of jellybeans. There’s no way that you can see all of the jellybeans from any one angle, and there’s too many to reasonably count out one-by-one if you poured them out. All of the kids in the camp will be heading out for snack in about fifteen minutes, and they will each have a dish into which you should put 10 jellybeans. Any leftover jellybeans are thrown away – all those grimy hands in the jar is just bad news.

Now, there are a couple things you can do in those fifteen minutes. You can run to the store and buy more jellybeans, if you think you’ll need more. You can also siphon off some of the jellybeans for safe-keeping, so they don’t have to be thrown away.

Your goal, then, is to balance the amount of jellybeans on hand to feed all of the kids in the camp with the least possible waste (all that spare money for more jellybeans comes out of your paycheck, by the way!). There’s nothing you can do to be sure that your ratio will 1:1 fit with the number of kids in the class – how best do you get close to it?

There’s a few approaches:

Better safe than sorry is a common answer. Buy extra beans, and no worries if they have to get canned. A typical conclusion, especially from the consumers of America: we have stuff to spare, and a feeling of infinite dump space. A little extra waste won’t hurt anyone, and we’ll guarantee that everyone gets something to eat. Not a bad approach when the resource is cheap, and excess material has no real consequence.

Or perhaps you’re more clever than that: buy the extra jellybeans, then siphon them off to clean storage. From storage, you only replenish the jar as necessary to meet the demand of the kids. This works well if the kids are patient enough for you to refill the jar. Hungry, screaming children are rarely willing to wait any longer than absolutely necessary for food – standing in line while staring at other kids eating their jellybeans just sucks from their perspective.

A third approach, if only to illustrate the multiple options: use a back-off handout system. Kids at the front of the line get the normal amount of jellybeans, but when the jar hits half-full, you ration it out a little more sparingly: 9 beans per kid instead of 10, for instance. Cut back more as your supply dwindles until you see the end in sight. This isn’t a fair distribution, but gets something to everyone with a constant supply of jellybeans.

There’s a lot of options, and each one tries to address the hard balance: how much supply do you put up front, given how many people are coming, when you don’t know how far a given amount will actually go?

The problem can get a lot harder, too. Imagine that, instead of working at a camp, you’re catering youth events at your local park: now you have no measure of how many people will show up today, and you can’t safely extrapolate how many people will come in the future based on today. Essentially, when you remove the constraint on how many people are coming, managing your jellybeans gets even harder. After all, you can’t hold enough jellybeans for everyone in the world, and getting as much as you can would be horrendously wasteful if only a handful of kids show up on a rainy Tuesday.

Talk about a negative edge on approximation, huh? The point I’m trying to illustrate is that estimation is hard. At the same time, estimation is an integral part of our lives, and we need to estimate several things every day. From supplying food for kids to tracking gas used in our cars to guessing how many groceries we need to get through the week, we have to make guesses about what’s going to happen in the future. Sometimes we use measurements to help us: our cars have gas meters to help us track our fuel consumption. Sometimes we use past history: I’ve only needed one gallon of milk each week for the past year, so I don’t think I should buy two this week.

But then, sometimes our easy comparators are unavailable. There’s no scale handy for my huge jar of jellybeans, and this is a new job. How can you hope to gain some intuition into feeding kids that you’ve never met, with an unknown amount of food? Intuition here is going to be crucial – with intuition, we can guess with more certainty, and know that our errors will be less drastic if they do occur.

Know what you know. This may sound trivial, but it’s important. Let’s look at the youth events in a park example again: I said there’s no way to know how many kids are coming, right? We can at least place some of the limitations we know in effect, and that will give us some bounds for estimation. Say the city has only 50,000 inhabitants. The activity at the park is oriented for kids, and probably has some age restrictions; if the kids are all supposed to be within 9 to 14, that’s likely to be within 10% of the town’s population. So, that’s just 5,000 kids, at maximum. Another good question is “how big is the park itself?” if it’s small, you likely can’t fit all 5,000 kids in there at once. If there are sister events happening across town, you can divide your maximum attendance down accordingly. So, I don’t know how many people are coming, but I do know that it won’t be more than X, based on the parameters here.

Ignore precision. I think this is fairly clear from all of my examples, though you might be a bit uneasy in doing it yourself. Just remember that the details in estimation are irrelevant: 38,475 people are the same as 10,000. The paradigm my suite-mate told me was “the only numbers you use in estimation are 1, a few, and 10. Multiply them together as necessary to get the estimates you need.” Even “a few” is extemporaneous. So, for jellybeans: we’ve got a jar, and it’s got some dimensions to it. We know that jellybeans have smaller dimensions, and we can use knowing what we know to figure out how many jellybeans could fit in the space presented (comparing volumes is fun, I promise). So, maybe we can fit 20,000 jellybeans into the space ideally: it’s now 10,000, as far as I’m concerned. How many kids in class? 84? Nah, it’s just 100. If those hold, I should be able to give 100 jellybeans to each kid with the supply in the jar. That’s not going to be accurate, but that’s why we’re estimating – all we want is a ballpark figure.

Know the cost of over- and under-estimating. If you over-estimate and have way too many jellybeans, what happens? If you’re short, what happens to the kids that don’t get any? If failing one way or the other is going to have a substantially worse consequence, weight your estimation accordingly. For instance, starving the kids is bad; throwing away spares is less bad. I’d therefore overstock and expect to throw some away – at least no kid goes hungry. If I’m estimating my gas, however, under-estimating is the way to go; I don’t want to drive my car on fumes unless I absolutely have to.

See, you can be cute AND comfortable with imprecision!

Be comfortable with the imprecision. This one is key. No matter what you do, estimations are never replacements for algorithms or equations. Algorithms and equations operate on known input formats, and produce known output formats. In contrast, estimation is about making a reasonable output for some loosely-bounded inputs. Do not expect accuracy. Just be comfortable with getting a decent guess. More often than not, that’s all you’ll need: do you really need to know how many feet you’re going to get on your tank of gas, or is three hundred miles, give or take, enough information for you? Your only goal is to get a good-enough result; if that’s not enough for you, then move on to fitting your system into an algorithm.

So, why not algorithm everything? Simply put, algorithm implementation takes longer and requires fairly strict bounds on input formats. Estimation sacrifices those bounds for a faster, less-accurate conclusion. If you can force strict bounds on your input, and care about accuracy, then you should take the time to implement the algorithm.

Okay, so we’ve gone over jellybeans a bit more than necessary – why does this stuff matter? Aion and I have both been doing a lot of estimation lately:

Aion:

The Aion team estimated the number of servers and population limitations per server for pre-release and release. This has led to countless complaints across forums and such about very long queues. Remember my second solution for feeding a summer camp? Yeah, all those people in the queue are whining because they’re staring at the other players who are already in the game. Turns out adults aren’t any more patient than little kids – we all feel entitled to have what other people get at the same time. Aion is pretty brilliant in their implementation of this, I have to admit, and yes I get to wait in a queue every day when I try to play. Sometimes I can’t play as a result (too much work). In fact, A lot of this was written while I was waiting in those queues!

So, why are the queues brilliant? Because Aion delivers a promise to their players of a reliable gaming experience. They could have taken the route of other MMOs and ignored server population caps, and watch the game degrade in many ways:

Players would have higher competition for materials because there would be a higher player density.

Players would have to handle more computation on their end to compensate for the player density, compromising any machines that aren’t well above the game’s recommended specifications.

Players would experience substantial connection latency from the server, as it would be operating beyond its own specified capacity (this one may be seen even as we are; those global lag spikes are a by-product of an overclocked server).

Aion: the new exclusive getaway

Aion circumvented these issues by implementing a queue. The game then becomes a fixed population-per-server environment, with transparent information for players in the queue (how long the line is, and an estimate of the queue time – look there, more estimation!). So there’s that benefit. They also knew the cost of over- and under-estimation. Over estimating and providing too many servers would have killed many of the servers. Under estimating simply creates queues, which – contrary to popular belief – actually builds suspense and makes players want to get in. Suddenly, the game world feels a bit like an exclusive party. Not everyone can be inside at once, but we’re willing to wait for half an hour or so to get a chance to hang out inside. Admittedly, the first couple of days were more than half an hour, but it’s gotten better for me with every day. That said, Aion is likely surprised that they underestimated the interest in the game.

Note that I didn’t put “check your history” as part of the intuition for estimation. This is because the past is very often misleading. Just because the past ten coin flips I’ve performed have come up “heads” doesn’t mean the next coin flip is any more likely to be “tails.” Likewise, kids that haven’t been hungry for jellybeans the past week may be really craving them this week, or may be genuinely sickened by the gooey things. Aion likely looked to other game releases and saw just how many pre-subscriptions went from CB to OB to Live, and expected those trends to carry over here. They then prepped for that level of decay; they didn’t expect such a high percentage of their subscribers to actually stay on. I think they shouldn’t have underestimated their potential, though: we kids were craving a new MMO experience more than even we realized.

My only critique is the exploitative nature of private shops. Once open, your character is not checked for inactivity, and is therefore never disconnected. 1.6 is bringing with it a catch for bots that are camping resource nodes; with any luck, they’ll bring a limiter on shop duration (say two or three hours) along with it. Don’t get me wrong – I do like the legit shops that are out on the frontier. They can save me a lot of travel time if they happen to have what I’m looking for, even if it is a little pricier. But then, the afk chumps aren’t adding anything to anyone’s game experience when they’re trying to sell bandages for one billion kinah each.

Me:

I did a lot of estimation in my first post on “What do the Numbers Mean,” and I’ve been doing some more for the second one (Coming soon!). For instance, I assume that Parry stops 50% of the damage on an incoming swing. It may be closer to 30% or 40%, given some reading I’ve done, but I’m not convinced. I estimated 50% based on the admittedly small in-game sample, and compared it to the other avoidance/mitigation stats. Parry is meant to be middle-of-the-road in terms of mitigation, so the amount mitigated fits very nicely at 50%. Again, it could be more or less, and it could even scale with weapon quality – just like the shields. However, 50% is enough of a ballpark for me.

Of course, I could work out algorithms for these things, and for some I’ve figured out the equations. Yet I don’t stress over those too much. Instead, I base my intuition here on the premise that Aion is a diverse game, so stats should likewise be diverse. I find it highly unlikely that any one stat will trump all other available attributes, but I guess time will prove that right or wrong.

Now, if you’ll excuse me, I have a game to go play. My queue time is finally expired, so I’m gonna get back to the game! In the meanwhile, for all of you whiners out there, I challenge you to present here a better solution for Aion’s release. To say “they’re doin it rong” is one thing, but if you can’t do it better, then who are you to judge? Show me that you can; I’m always curious to hear more solutions, and any truly valid solutions will doubtless be good to preserve for future game releases.

Actions

Information

One response

I think you are right. The queue is like a supply, and demand thing. Toy companies use this strategy at Xmas every year. Remember when only so many cabbage patch dolls were gonna be on shelves by xmas, or the Playstation 3 which they allotted for over a year to drive people crazy. The Wii is the same way. They have learned that making someone want something, but not able to get it without waiting works better than shipping 10 million of them at once.