I had considered subtracting bid amounts from the budget, but rejected it for several reasons, all of which are problematic. First, if 36 boxes bid simultaneously, we could overshoot our budget unless we have boxes "lock" the bid amount, but that changes us from parallel to serial bidding and increases response time to the point where bids that should have been one would be lost.

Serial bidding might be appropriate as we get close to the budget, but that raises another issue: the box that receives the notification that we lost is usually not the box that made the corresponding bid. Because we don't receive the bid amount in the loss notification, we'd have to search all 36 boxes to find out who bid what (or tremendously complicate our Redis setup by trying to store huge amounts of bid information in Redis and we're expecting our volume of data to increase by a couple of orders of magnitude).

As for your suggestion of me being overly optimistic, that might be the case, but the bid amounts are (fortunately) small fractions of the budget, so it won't be catastrophic until we scale up.

Riak is not an option because the response times on Riak are slow enough that we'll lose bids we should have one (though we're probably going to put Redis in front of Riak). I don't know about membase.

We're probably going to go with a strategy of allocating separate budgets to separate boxes. It's more complicated, but it seems much safer. We lose a lot of bidding capacity for individual campaigns, but for many campaigns, I think it's doable.

First, if 36 boxes bid simultaneously, we could overshoot our budget unless we have boxes "lock" the bid amount, but that changes us from parallel to serial bidding and increases response time to the point where bids that should have been one would be lost.

I don't see where you need locking. Naturally the check "if ($budget>0-$overshootallowance) { $budget-=$bid }" must be an atomic operation and this is a minimal lock, but this is not a lock over the duration of a bid. The case that only one machine can bid can only occur if the total budget is already so low as to allow exactly one last bid

UPDATE: To clarify, if the subtract operation is the most time-consuming part in a single bid operation, then having a central budget is practically serializing the bidding per campaign. I'm not sure whether you meant that or a lock over a complete bid-and-wait-for-success-transaction

You either have the ability/funds to allow *all* your boxes to overshoot or you don't. In the first case you don't have a problem. In the second case, whatever scheme you are using, you obviously have to lose some bids as the budget gets low.

the box that receives the notification that we lost is usually not the box that made the corresponding bid

Ok, this is a real problem for the central budget idea. But since you want to go with the separate budget idea, note that you have a smaller but similar problem: Since the successful bid arrives at a different machine, that machine has to deduce the amount from its own budget (if you don't want to search all 36 boxes). But what happens if that machine has no budget left? Either it overshoots or it searches the other boxes for budget left over (which takes enough time for the machine with the budget left over to bid although it should not). In the worst case all successful bid notifications go to the same machine with budget==0 and all the other machines can still place further bids until they are notified by that one machine.

You have to be careful that the negotiation traffic between the machines doesn't throttle your network when lots of machines have no budget left and are repeatedly contacting other machines to get another bid out.