Seeing as how this is probably a one-shot, I might just select data in the main loop, and feed it to Amazon in parallel processes. It shouldn't take ALL that long to select 70G of data. The bottleneck, I would think, would be the Amazon request. Something like: