Hitting Multiple Files Using Bash and Wget or Curl

25 Oct 2013

I needed to hit a number of files on a remote HTTP server with an index ranging from 1 to 150 and my first instinct was to use wget with some sort of a loop (bash script seemed like the most logical thing to do).

Then I realized that I can do the same thing in a simple oneliner using 'curl'

It's super easy using the following info:

You can specify multiple URLs or parts of URLs by writing part sets within braces as in:

http://site.{one,two,three}.com or you can get sequences of alphanumeric series by using [] as in:

Nested sequences are not supported, but you can use several ones next to each other:

http://any.org/archive[1996-1999]/vol[1-4]/part{a,b,c}.html

You can specify any amount of URLs on the command line. They will be fetched in a sequential manner in the specified order. You can specify a step counter for the ranges to get every Nth number or letter: