The data structure you're talking about is called a Bloom filter. It seems to be one that I
see and think "oh, that'll come in handy sometime" but then it never does.
The nastiest limitation is that you can't ever remove anything from it. So if your
web crawler should eventually rescan the same URL (as I hope it would), it'd be
unsuitable for hashing URLs. I confess that I don't really understand what the
interview question is getting at.