BT

Article: Automating File Uploads with SSH and Ruby

by Werner Schuster on May 30, 2007 |
Matthew Bass introduces us to the Net::SFTP and Net::SSH libraries in Ruby by learning by doing. He implements a simple clone of rsync, a tool to upload and synchronize local file systems with remote ones. By using the secure SSH protocol, this is secure and production ready.

In the process of building the tool, Matthew also shows some of the pitfalls that come with APIs and how to avoid them. For instance, uploading files to directories that don't exist will fail with an exception instead of creating the directories. Discovering little details such as this can make programming tedious at times, and the article also shows how to tackle these issues.

If you're interested in this kind of networking work with Ruby or just want to see a little real world tool in Ruby, be sure to read  Automating File Uploads with SSH and Ruby.

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Good Example by Bruce Zuo

Thanks for the good write up. But if you only care about to get the job done, wouldn't rsync be a good solution?

Re: Good Example by Matthew Bass

Absolutely. But it's more fun to write it in Ruby. :)

Re: Good Example by Alex Popescu

You may be required to have something similar in your app. And your app is required to work on a platform without rsync or at least to not require external dependencies that may be missing from the system. So, you are left to creating something, and this article shows you how to do it.

./alex
--
.w( the_mindstorm )p.
________________________
Alexandru Popescu
Senior Software Eng.
InfoQ TechLead&CoFounder

Re: Good Example by Phi Sanders

FYI to other slack hackers trying to follow along :

I had to add "require 'rubygems'" as the first line of code to get net/ssh and net/sftp to load, presumably because I used the "ports" command on Mac OS X to setyp my ruby/gems environment...

ERROR: Error installing gem net-ssh[.gem]: buffer error in Win XP by Raru Panta

I got ERROR: Error installing gem net-ssh[.gem]: buffer while trying to install net-ssh,net-sftp in Windows, I installed Ruby via latest one click installer version 1.8.4.0. After searching on google I found out comment on Jamies Buck's blog weblog.jamisbuck.org/2007/5/10/net-ssh-1-1-1 for the solution.
After following command
C:\>gem update rubygems-update

and gem install net-ssh and net-sftp
both gems installed successfully.


Thanks for great article

Nasty code by Kris Leech

Shame about the two bits of nasty code otherwise this would make a great introduction to Ruby.
Great article all the same!

There are several serious problems with this script. by David Richards

The first one is that sftp.mkdir(remote_dir, :permissions => dir_perm) is _wrong_ it should be sftp.mkdir(remote_dir, :mode => dir_perm)

This will get by on some platforms, but not others like ubuntu feisty. I'm really not sure why. #setstat is the same way, it should be :mode.

Also, the way in which this recurses will break for many people. If you have a file two directorys up from the directory you started recursing from, and the inbetween directory has not been created yet, this script will break.

Re: There are several serious problems with this script. by David Richards

Also its not very scalable because its calling stat on the remote directory for every single file, and then on each remote file. This is really slow. This is from the net-sftp faq:

handle = sftp.opendir("/usr/lib")
items = sftp.readdir(handle)
items.each do |item|
puts item.filename
puts item.longname
p item.attributes # permissions, atime, etc.
end
sftp.close_handle(handle)

So if you could create a hash or something like that to check against for each file rather then it could scale.

I'm working on a rails backup system that uses this technique and will post code when it gets good enough.

thanks by Richard Finegan

thanks, this article was very helpful for what i was doing

Re: Good Example by janna fierst

This example was great- I've never done anything with ruby before and I was able to modify it for my needs in two days, saving me hours and hours of tedious file uploads and downloads. I had to change a few things though, the biggest one being that sftp.put_file() doesn't seem to be supported anymore and I had to use ssh.sftp.upload() instead. Maybe it's something with the new versions of net::ssh and net::sftp? Thanks for the great intro to ruby!

Improvements by Dominic Rose

Thanks for this I was leaning how to use SSH and this is just what I needed!

Now I have to work on this to make it handle something else:
using a zip archiver with `` commands and zip only the files with a timestamp > X.
So it could be really fast if I disable outputs like "Copying D:/html/images/stop.gif to /home/public_html/images/stop.gif", and it would only send one file before asking the server to unzip it.
Thanks very much maybe I will soon only need to do some keyboard shorcut to synchronise all I need.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

11 Discuss

Educational Content

General Feedback
Bugs
Advertising
Editorial
InfoQ.com and all content copyright © 2006-2014 C4Media Inc. InfoQ.com hosted at Contegix, the best ISP we've ever worked with.
Privacy policy
BT