Mac OS X hack: download multiple files with curl

October 29, 2012

Updated 2017-08-01 to fix typo and use new demo URL. Also note that the Homebrew package manager has matured since this post was first published in 2012. Once you’ve configured Homebrew, installing a tool like wget is as simple as running brew install wget wget is an incredibly useful GNU tool on Linux. Unfortunately, it doesn’t come with OS X (as of Mountain Lion). OS X includes curl, which is a very handy tool but lacks at least one important feature of wget: the ability to use wildcards to get multiple files at the same time. For example, let’s say you want to download a subset of files from an FTP server. With wget, you could type:

wget ftp://ftp.dos.state.fl.us/public/doc/cor/011*.dat

Here is how to mimic that process with curl and a few UNIX command-line tricks. 1. Download the directory listing and save it in a file.

curl -L ftp://ftp.dos.state.fl.us/public/doc/cor/ > index

2. Use grep with regular expressions to parse the .html file, extract the .igs file names and save them in a text file.

grep -o '011[0-9]*c.dat' index > file_list

3. Use a bash loop to iterate over the text file and fetch each file with curl.

while read line; do
curl -O ftp://ftp.dos.state.fl.us/public/doc/cor/$line;
done < file_list

Another advantage of the curl approach is that wget doesn’t support wildcard characters with HTTP URLs (only FTP).

Comments

Comment #1 by Anonymous

Anonymous - Apr 4, 2017

`wget http://www.wiz-worx.com/iges5x/wysiwyg/igs/*.igs` won’t work for wget doesn’t support globbing on http

Comment #2 by Craig

Craig - Apr 5, 2017

You are correct; I updated the wget example to a working public FTP site.

Comment #3 by yep

yep - Jul 1, 2017

there is no “rep” command in OSX anymore…bummer.

Comment #4 by Craig

Craig - Aug 2, 2017

That’s a typo…it was supposed to say grep instead of rep. I fixed it–thanks for pointing that out!