InfoQ Homepage News Article: Automating File Uploads with SSH and Ruby

# Article: Automating File Uploads with SSH and Ruby

Matthew Bass introduces us to the Net::SFTP and Net::SSH libraries in Ruby by learning by doing. He implements a simple clone of rsync, a tool to upload and synchronize local file systems with remote ones. By using the secure SSH protocol, this is secure and production ready.

In the process of building the tool, Matthew also shows some of the pitfalls that come with APIs and how to avoid them. For instance, uploading files to directories that don't exist will fail with an exception instead of creating the directories. Discovering little details such as this can make programming tedious at times, and the article also shows how to tackle these issues.

If you're interested in this kind of networking work with Ruby or just want to see a little real world tool in Ruby, be sure to read  Automating File Uploads with SSH and Ruby.

Style

## Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

• ##### Good Example

by Bruce Zuo /

• ##### Re: Good Example

by Matthew Bass /

• ##### Re: Good Example

by Alex Popescu /

• ##### Re: Good Example

by Phi Sanders /

• ##### Re: Good Example

by janna fierst /

• ##### ERROR: Error installing gem net-ssh[.gem]: buffer error in Win XP

by Raru Panta /

• ##### Nasty code

by Kris Leech /

• ##### There are several serious problems with this script.

by David Richards /

• ##### Re: There are several serious problems with this script.

by David Richards /

• ##### thanks

by Richard Finegan /

• ##### Improvements

by Dominic Rose /

• ##### Good Example

by Bruce Zuo /

Your message is awaiting moderation. Thank you for participating in the discussion.

Thanks for the good write up. But if you only care about to get the job done, wouldn't rsync be a good solution?

• ##### Re: Good Example

by Matthew Bass /

Your message is awaiting moderation. Thank you for participating in the discussion.

Absolutely. But it's more fun to write it in Ruby. :)

• ##### Re: Good Example

by Alex Popescu /

Your message is awaiting moderation. Thank you for participating in the discussion.

You may be required to have something similar in your app. And your app is required to work on a platform without rsync or at least to not require external dependencies that may be missing from the system. So, you are left to creating something, and this article shows you how to do it.

./alex
--
.w( the_mindstorm )p.
________________________
Alexandru Popescu
Senior Software Eng.

• ##### Re: Good Example

by Phi Sanders /

Your message is awaiting moderation. Thank you for participating in the discussion.

FYI to other slack hackers trying to follow along :

I had to add "require 'rubygems'" as the first line of code to get net/ssh and net/sftp to load, presumably because I used the "ports" command on Mac OS X to setyp my ruby/gems environment...

• ##### ERROR: Error installing gem net-ssh[.gem]: buffer error in Win XP

by Raru Panta /

Your message is awaiting moderation. Thank you for participating in the discussion.

I got ERROR: Error installing gem net-ssh[.gem]: buffer while trying to install net-ssh,net-sftp in Windows, I installed Ruby via latest one click installer version 1.8.4.0. After searching on google I found out comment on Jamies Buck's blog weblog.jamisbuck.org/2007/5/10/net-ssh-1-1-1 for the solution.
After following command
C:\>gem update rubygems-update

and gem install net-ssh and net-sftp
both gems installed successfully.

Thanks for great article

• ##### Nasty code

by Kris Leech /

Your message is awaiting moderation. Thank you for participating in the discussion.

Shame about the two bits of nasty code otherwise this would make a great introduction to Ruby.
Great article all the same!

• ##### There are several serious problems with this script.

Your message is awaiting moderation. Thank you for participating in the discussion.

The first one is that sftp.mkdir(remote_dir, :permissions => dir_perm) is _wrong_ it should be sftp.mkdir(remote_dir, :mode => dir_perm)

This will get by on some platforms, but not others like ubuntu feisty. I'm really not sure why. #setstat is the same way, it should be :mode.

Also, the way in which this recurses will break for many people. If you have a file two directorys up from the directory you started recursing from, and the inbetween directory has not been created yet, this script will break.

• ##### Re: There are several serious problems with this script.

Your message is awaiting moderation. Thank you for participating in the discussion.

Also its not very scalable because its calling stat on the remote directory for every single file, and then on each remote file. This is really slow. This is from the net-sftp faq:

handle = sftp.opendir("/usr/lib")
items.each do |item|
puts item.filename
puts item.longname
p item.attributes # permissions, atime, etc.
end
sftp.close_handle(handle)

So if you could create a hash or something like that to check against for each file rather then it could scale.

I'm working on a rails backup system that uses this technique and will post code when it gets good enough.

• ##### thanks

Your message is awaiting moderation. Thank you for participating in the discussion.

• ##### Re: Good Example

by janna fierst /

Your message is awaiting moderation. Thank you for participating in the discussion.

This example was great- I've never done anything with ruby before and I was able to modify it for my needs in two days, saving me hours and hours of tedious file uploads and downloads. I had to change a few things though, the biggest one being that sftp.put_file() doesn't seem to be supported anymore and I had to use ssh.sftp.upload() instead. Maybe it's something with the new versions of net::ssh and net::sftp? Thanks for the great intro to ruby!

• ##### Improvements

by Dominic Rose /

Your message is awaiting moderation. Thank you for participating in the discussion.

Thanks for this I was leaning how to use SSH and this is just what I needed!

Now I have to work on this to make it handle something else:
using a zip archiver with  commands and zip only the files with a timestamp > X.
So it could be really fast if I disable outputs like "Copying D:/html/images/stop.gif to /home/public_html/images/stop.gif", and it would only send one file before asking the server to unzip it.
Thanks very much maybe I will soon only need to do some keyboard shorcut to synchronise all I need.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Is your profile up-to-date? Please take a moment to review and update.

Note: If updating/changing your email, a validation request will be sent

Company name:
Company role:
Company size:
Country/Zone:
State/Province/Region:
You will be sent an email to validate the new email address. This pop-up will close itself in a few moments.