More ftp

I've got a fever,
and the only prescription
is more ftp

by Edward Marczak

Introduction

ftp  whether we're referring to the actual protocol, or just file transfer in general  is something we all need on some basis. When I started toying with computers, I saw punch cards, but never really had to deal with them. But that was only one method of file (or more generally, data) transfer. Then we moved up to tape and floppy disk. Now, very few computers are not connected to a network of some type, and the primary reason is to transfer data in the form of files. Our cover story this month touches on several GUI-based clients, but when you read this column, those utilities get the "Mac In The Shell" treatment. We need to be able to transfer files easily from a shell!

Why?

I often create automated solutions that run on a server without a GUI. There are also plenty of times when a simple, repeated file transfer shouldn't pop up anything visually on a client machine, either. It should 'just happen' simply and reliably with no pomp and circumstance. Enter curl, ftp and wget.

Of the three, "ftp" is the oldest and most simple. wget brings further power, and curl is a veritable Swiss Army Knife of transfer agents. If one of these options can't do what you want, it's most likely not possible (or, consider a different tactic!).

ftp

ftp, the application, implements a client side version of the ftp protocol (which is detailed by Mary Norbury in this month's cover story, "FTP Clients for Mac OS X"). In simple use, you can use it interactively:

Any techy person over the age of 25 should recognize this immediately. They should also remember in the days before big-business-on-the-Internet that it was polite to wait until "after hours" before using ftp against a University server! In the annals of tech-history, though, one needed to be familiar with transferring files this way.

The shell-based ftp application has a good lexicon in its interpreter. It's one that has grown substantially since its inception. However, for purposes of automation, that can get clumsy. You could script it with expect. Some versions of ftp allow creating a script and having ftp simply run through the motions that the script indicates. However, the version of ftp that ships with OS X (at least in Tiger) omits this option. It does keep the macro definition option in, though.

Never fear! The parameters available to you are greatly expanded, including passing a user name and password along. This is ideal for scripting within your own scripting environment. So, if I know in advance the names of the files I need to transfer, I could script this in bash thusly:

ftp -V ftp://user:password@server.example.com/directory/file.txt

...which will download file.txt and name the local file "file.txt". Note the -V switch, which is the opposite of -v  keeping the output quiet. I can also go the other way using:

So, good 'ol ftp provides us with some quick and easy ways to move files around. Not ideal ways, perhaps, as not only is our password sent in the clear as part of the ftp protocol (which may not be an issue for you), but also displays our password in a process listing. That's not really cool.

ftp does offer many more options, so, check the man page if you need to get in deeper.

wget

wget bills itself as the "non-interactive network downloader." So, unlike ftp, there is no interactive mode with which you can use to generally poke around. However, we're here to talk about automated use, so, we don't need no stinkin interactive mode! If you were desperate, though, you could use one of wget's more interesting features: when an ftp directory is requested, it will automatically convert the output into an html listing. That might be a little too esoteric...even for me!

Disapointingly, wget is not installed by default under OS X Tiger. However, it's simple to install one way or another. You can grab the source from the GNU page at http://ftp.gnu.org/pub/gnu/wget/ and compile it yourself. Quentin Stafford-Fraser has a pre-compiled binary here: http://www.statusq.org/images/wget.zip. Finally, you can install wget using fink or MacPorts.

To get right down to it, like ftp, you can use any rational URL to specify your target:

Also like ftp, wget will display your password in a process listing, so, use this with care! Here's where the roads diverge, though, and wget has a few more tricks up its sleeve. You can recursively download an entire ftp or http directory with the "-r" switch:

wget -r -t 5 ftp://emarczak:sekretpass@ftp.example.com/path/

I also threw in the "-t" switch, which will allows for multiple retries if some part of a file download fails. "-t" also allows for a value of "inf" causing infinite retries. Also useful here is the "-l" (ell) switch, which limits the depth of the traversal. So, to grab just the items from the top layer of the directory you specify, use "-l1".

An absolute life saver is the "-c" switch: it tells wget to continue a partial download. So, if your download bombs, or, perhaps you're on a laptop and need to run before the transfer is complete, retry the operation with the "-c" switch and pick up right from where you left off. Nice.

Wget will read a list of URLs from a file, using the "-i" switch. This is handy for scripting, of course. However, it's also a very nice way to keep your password out of a process listing. With your username and password embedded in a file, you're not using it on the command line. This list also comes in handy as a way to store your favorite sites and then recursively them locally using the "-r" switch mentioned above. In fact, toss in the "-A" switch, which will only accept certain files, and you can download only files of a certain type from a site. Next time you want all of the mov files from a given site, try this:

wget -r -l4 http://www.example.com/movies -A.mov -np

This will mirror the given site, and only transfer files ending in ".mov" on the given pages up to 4 levels deep. We also ensure that we don't follow links back up to the parent directory ("-np", or, "--no-parent").

Again, wget has many, many tricks up its sleeve. Too many to list here, but the brief introduction should convince you of its utility above the standard "ftp" application. Check out the man page for much more. (Specifically, check out the "-k" switch!).

curl

Like the other utilities mentioned here, curl will accept any valid URL as its file description. Unlike wget, curl is installed as a part of OS X. One very cool curl trick is that it dumps files to standard out unless it's told where to write them. Why is that cool?

Sometimes, you just want to view a remote document, be it an actual file  like a README or index.html file  or a directory listing. So, you could easily:

curl http://server2.example.com/instructions/how_to_do_it.txt | less

...which will get the file from the server and pipe it into less. When you quit less, there will be no file remaining to clutter up your disk. I sometimes use that with http://www.whatismyip.com, and then pipe the output to a script that simply reports back the machine's external IP address. This is also a cool way to run a remote script:

curl ftp://server.example.local/script.sh | bash

If you are interested in downloading a file, use the "-O" switch (capital O), which names the local file the same as the remote:

curl -O ftp://ftp.example.com/path/to/file/some_file.zip

This will anonymously download some_file.zip, and store it in the current working directory as some_file.zip. I also particularly like the "-L" switch (capital ell) when used with http servers as this will make curl follow http redirects.

Of course, curl will upload files, too. The "-T" switch will take care of this for you:

curl -T "pix[1-100].jpg" ftp://ftp.example.com/pictures/

I also threw in the fact that curl will respect globbing and regular expressions. So, the previous example will upload pix1.jpg, pix2.jpg...up through pix100.jpg. Clearly very handy.

Both upload and download can be resumed using the "-C -" switch (capital C followed by a hyphen). The hyphen tells curl to figure out where to resume from automatically. This does require server-side support in the form of telling the server at which byte to start appending at (the SIZE command for upload) or which byte to start the transfer from (ftp resume or HTTP 1.1 for downloads).

If you're an all-OS X shop, you'll be happy to hear that curl supports Kerberos. You can get your initial ticket the usual way (kinit), and then tell curl to use that authentication via the "--krb" switch.

curl --krb private ftp://krb4site.com -u username:boguspw

If this uses Kerberos, why did I still supply a name and password? This is a bit of a hack, but with no password, curl will still want to prompt you for one. However, if you supply one, but use Kerberos, it'll just ignore the password you supply  so use a bogus password as this will appear in process listings.

Conclusion

Being able to script data transfer is an important part of every system administrator's toolkit. While good 'old ftp will do the job in many cases, wget and curl give you much more flexibility. Both utilities have overlap in functionality, but curl goes deeper in many cases. Case in point: when I said that curl accepts any valid URL syntax, try TELNET://, dict:// and even LDAP:// (although, you'll currently need to build your own curl for LDAP support as the Apple supplied version isn't linked correctly with the LDAP framework).

Of course, there are other file transfer options available to you, including scp, sftp, ditto and rsync, to name a few. However, I focused mainly on ftp options here, as ftp is alive and well, but sometimes overlooked. While perhaps a deceiving name, sftp is not true ftp, but file transfer over ssh, requiring no ftp server at all. Of course, over "hostile" networks, you should use no less that an encrypted solution. However, with the right internal setup, and in certain other cases, ftp can be the perfect solution.

Please practice this in a test environment and then press it into real-world use where appropriate. Until next month, I think you'll find this a great tool in your automation arsenal.

Ed Marczak owns and operates Radiotope, a technology consulting company that guides companies to use what they have as efficiently as possible. He is also the Executive Editor of MacTech Magazine, a husband and father of two. His spare time is spent editing MacTech Magazine and enjoying his family. He finds keeping it all running smoothly good practice. Improve your practice at http://www.radiotope.com.

PCalc is a full-featured, scriptable scientific calculator with support for hexadecimal, octal, and binary calculations, as well as an RPN mode, programmable functions, and an extensive set of unit... Read more

FileZilla 3.10.2 - Fast and reliable FTP...

FileZilla (ported from Windows) is a fast and reliable FTP client and server with lots of useful features and an intuitive interface.
Version 3.10.2:
Note: Now requires a 64-bit Intel processor.... Read more

The Hit List 1.1.11 - Advanced reminder...

The Hit List manages the daily chaos of your modern life. It's easy to learn - it's as easy as making lists. And it's powerful enough to let you plan, then forget, then act when the time is right.... Read more

Bartender 1.2.32 - Organize your menu ba...

Bartender lets you organize your menu bar apps.
Features:
Lets you tidy your menu bar apps how you want.
See your menu bar apps when you want.
Hide the apps you need to run, but do not need to... Read more

ClamXav 2.7.5 - Free virus checker, base...

ClamXav is a free virus checker for OS X. It uses the tried, tested, and very popular ClamAV open source antivirus engine as a back end.
I hope you like and use ClamXav a lot and that it helps keep... Read more

xScope 4.1.2 - Onscreen graphic measurem...

xScope is powerful set of tools that are ideal for measuring, inspecting, and testing on-screen graphics and layouts. Its tools float above your desktop windows and can be accessed via a toolbar,... Read more

MacFamilyTree 7.3.3 - Create and explore...

MacFamilyTree gives genealogy a facelift: it's modern, interactive, incredibly fast, and easy to use. We're convinced that generations of chroniclers would have loved to trade in their genealogy... Read more

Skype 7.5.0.738 - Voice-over-internet ph...

Skype allows you to talk to friends, family and co-workers across the Internet without the inconvenience of long distance telephone charges. Using peer-to-peer data transmission technology, Skype... Read more

PushPal 3.0 - Mirror Android notificatio...

PushPal is a client for Pushbullet, which automatically shows you all of your phone's notifications right on your computer. This means you can see who's calling or read text messages even if your... Read more

At this point it’s pretty safe to say that no MOBA is going to dethrone Dota 2 and League of Legends anytime soon. After all, if Batman can’t do it, nobody can. However, with a genre as popular and profitable as this one, there’s still room for... | Read more »

Final February Fun at 148Apps
How do you know what apps are worth your time and money? Just look to the review team at 148Apps. We sort through the chaos and find the apps you’re looking for. The ones we love become Editor’s Choice, standing out... | Read more »

GDC 2015 – Does Not Commute is Definitel...

GDC 2015 – Does Not Commute is Definitely a Game You Should Keep an Eye on
Posted by Rob Rich on March 2nd, 2015 [ permalink ]
We were teased about Mediocre Games’ (Smash Hit,
| Read more »

F84 Games & POW! Announce Stan Lee V...

F84 Games has announced that it is working with legendary comic creator Stan Lee and POW! Entertainment to produce Stan Lee’s Hero Command. The game will be a action adventure of heroic proportions.
| Read more »

Setlyst Keeps Your Set Straight So You C...

Setlyst Keeps Your Set Straight So You Can Focus On Rocking Out.
Posted by Jessica Fisher on March 2nd, 2015 [ permalink ]
Universal App - Designed for iPhone and iPad
| Read more »

Space is Vast, So Space Agency Has a Vas...

Space is Vast, So Space Agency Has a Vast New Update!
Posted by Jessica Fisher on March 2nd, 2015 [ permalink ]
Universal App - Designed for iPhone and iPad
| Read more »

Size DOES Matter Review

Size DOES Matter Review
By Campbell Bird on March 2nd, 2015
Our Rating: :: HARD TO BEATUniversal App - Designed for iPhone and iPad
This rhythm game has a unique control scheme and performance system that make it feel like a true... | Read more »

The first ever action 3D card battler Al...

On the other hand, you probably haven’t played an action 3D card battler – until now. Step forward, All Star Legion.
All Star Legion is a 3D QTE-based action RPG card battler, but fear not – the game itself isn’t as convoluted as its description.... | Read more »

Travel Back to the 1980s With the Making...

Headup Games has released a hilarious making of video for its upcoming title, Pixel Heroes: Byte & Magic. The game is a RPG/Roguelike where you control three heroes set to save the township of Pixton from an evil cult called The Sons of Dawn.... | Read more »

Price Scanner via MacPrices.net

Sale! 15-inch 2.2GHz Retina MacBook Pro for $...

Best Buy has the 15″ 2.2GHz Retina MacBook Pro on sale for $1774.99 $1799.99, or $225 off MSRP. Choose free home shipping or free local store pickup (if available). Price valid for online orders... Read more

27-inch 3.5GHz 5K iMac in stock today and on...

B&H Photo has the 27″ 3.5GHz 5K iMac in stock today and on sale for $2299 including free shipping plus NY sales tax only. Their price is $200 off MSRP, and it’s the lowest price available for... Read more

Apple Launches Free Web-Based Pages and Other...

Apple’s new Web-only access to iWork productivity apps is a free level of iCloud service available to anyone, including people who don’t own or use Apple devices. The service includes access to Apple... Read more

Survey Reveals Solid State Disk (SSD) Technol...

In a recent SSD technology use survey, Kroll Ontrack, a firm specializing in data recovery, found that while nearly 90 percent of respondents leverage the performance and reliability benefits of SSD... Read more

Save up to $600 with Apple refurbished Mac Pr...

The Apple Store is offering Apple Certified Refurbished Mac Pros for up to $600 off the cost of new models. An Apple one-year warranty is included with each Mac Pro, and shipping is free. The... Read more

Apple CEO Tim Cook will deliver the George Washington University’s Commencement address to GWU grads on May 17, at which time he will also be awarded an honorary doctorate of public service from the... Read more

Apple restocks refurbished Mac minis for up t...

The Apple Store has restocked Apple Certified Refurbished 2014 Mac minis, with models available starting at $419. Apple’s one-year warranty is included with each mini, and shipping is free:
- 1.4GHz... Read more

Walmart has the 16GB iPad Air 2 WiFi on sale for $446.99 on their online store for a limited time. Choose free shipping or free local store pickup (if available). Sale price for online orders only,... Read more

Jobs Board

*Apple* Solutions Consultant - Retail Sales...

**Job Summary** As an Apple Solutions Consultant (ASC) you are the link between our customers and our products. Your role is to drive the Apple business in a retail
Read more

*Apple* Pay Automation Engineer - iOS System...

**Job Summary** At Apple , great ideas have a way of becoming great products, services, and customer experiences very quickly. Bring passion and dedication to your job
Read more

Sr. Technical Services Consultant, *Apple*...

**Job Summary** Apple Professional Services (APS) has an opening for a senior technical position that contributes to Apple 's efforts for strategic and transactional
Read more

Event Director, *Apple* Retail Marketing -...

…This senior level position is responsible for leading and imagining the Apple Retail Team's global engagement strategy and team. Delivering an overarching brand
Read more

*Apple* Pay - Site Reliability Engineer - Ap...

**Job Summary** Imagine what you could do here. At Apple , great ideas have a way of becoming great products, services, and customer experiences very quickly. Bring
Read more

MacTech is a registered trademark of Xplain Corporation. Xplain, "The journal of Apple technology", Apple Expo, Explain It, MacDev, MacDev-1, THINK Reference, NetProfessional, Apple Expo, MacTech Central, MacTech Domains, MacNews, MacForge, and the MacTutorMan are trademarks or service marks of Xplain Corporation. Sprocket is a registered trademark of eSprocket Corporation. Other trademarks and copyrights appearing in this printing or software remain the property of their respective holders. Not responsible for typographical errors.

All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.