Memcache keys not getting saved correctly

Update : We were initially blaming Memcache not correctly setting/getting
keys. We should instead have blamed him for being over-zealous in clearing its
list. See bottom of post for details.

Today we were having a bit of an issue with memcache. We had to write a
thousand keys into it, reading them from a JSON file.

This code had been running happily on our server for the past two weeks. But
as the number of keys increased, we came to spot that some keys seemed to be
randomly cleared.

After further investigation (and a lot of hours), it appears that the keys
were simply not set at all, or set to an empty/false value. This was really
strange because it occured inside a loop on our JSON file, and never occurs
for the same keys, nor everytime.

We took our server under high stress, forcing it to reload data very often, to
trigger the bug once more. After a lot of test, we gather some more data, but
nothing really came out. The missed keys were really random.

Finally, we decided to code a fast and dirty fix, until we found a better
solution. Basically, we simply checks that the value is correctly saved after
saving it, and if not, we retry. And we do so recursively.