Over three years ago, I was faced with a dilemma. I had recently switched to the Mac (from Linux) and was still using my text editor of choice (vim), but at the time, vim’s “integration” with OS X was pretty minimal (and that’s putting it optimistically). I experimented with emacs, but it never clicked for me, and honestly, emacs on OS X wasn’t all that better than vim at the time. Sadly, reluctantly, I said good-bye to vim and switched to TextMate.

TextMate was (and certainly still is) a fantastic text editor. The project drawer was awesome, finding files via cmd-T was super powerful, and smarter auto-completion and snippets promised a new and faster way to pound code. After a couple of months of reteaching my fingers how to edit text, I was happy.

Sometimes, though, late at night, I would think again of vim.

Fast forward three years. The vim landscape is different now. There is actually a Mac-friendly GUI version of vim now, MacVim, which actually looks like it belongs on OS X. Vim 7 supports UI tabs, and a much more powerful auto-completion mechanism than before. And plugins like rails.vim and fuzzyfinder.vim mean that TextMate no longer has a corner on powerful project navigation.

For the last few weeks I’ve been toying with switching back to vim. TextMate’s “snippet” feature never clicked for me, and the only times I used it were by accident (when it annoyed me more than it helped me), but I really was hooked on the project browser, and cmd-T, and a few other things. I realized that, with a little work, perhaps a way could be found to reimplement most of the things I loved about TextMate, in vim.

This last week I’ve worked exclusively in vim, to test that theory. It’s like coming home. As I said, TextMate is a powerful and wonderful editor, too, but differently powerful and wonderful. Vim’s wonderfulness and power is the wonderfulness and power of git, or linux, where the learning curve is steep (ridiculously steep at times), but the rewards of mastery are sublime. I didn’t even realize I had missed a sane shift-J, or using the dot key to repeat the last command. Fix transposition typos with ‘xp’. Select a single word with ‘viw’. Drop bookmarks with ‘m’, and jump right back to them with single quote.

If any of that makes your stomach roil, then vim is not for you. :) But to me, it’s like being embraced by a long-lost friend after years apart. And vim holds no grudges.

There was still the issue of the TextMate features that I had particularly come to love. First to tackle was cmd-T, since my workflow had become so dependent on that for finding files. Takeshi NISHIDA’s fuzzyfinder.vim script seemed like exactly what I wanted…at first. It’s definitely a powerful tool, but the fuzzy finder for files was not TextMate’s cmd-T, and my instinctive attempts to treat it so were causing me a lot of aggravation.

So I took an evening and wrote fuzzy_file_finder, a Ruby library that mimics (and improves on, if I do say so myself) TextMate’s cmd-T functionality. Then, I extended fuzzyfinder.vim with fuzzyfinder_textmate, which bound the fuzzy_file_finder to vim. The result?

See for yourself: http://s3.amazonaws.com/buckblog/videos/fuzzyfinder_textmate.mov (600K, QuickTime video).

I’m still working on a solution for the project browser. Yes, I know there are several (“countless” might be a more accurate term) vim plugins that present a project drawer in a split window, but even before being spoiled by TextMate those didn’t feel right to me. I’m experimenting with a cocoa tree view that sends files to a specific vim server, and it mostly works, but I’m still not sure it’s the right solution. If I do come up with something, I’ll definitely open it up and share it. (On the other hand, if any of you out there in readerland already know of such a thing, please point me at it!)

So, I’m still reacquaiting myself with all my old muscle memories, but here are some commands I wasn’t previously very familiar with which are proving useful in conquering my TextMate habits:

• :e [file]. This is great if you don’t want to use fuzzy finding. Vim will even do tab completion to make things easier.
• :ls. Shows all of your buffers.
• :buffer [file]. Fantastic for quickly switching between buffers. You can give it just part of a file name and it will jump to the buffer that matches.
• The ctrl-6 (technically ctrl-caret) key is awesome for switching back and forth between two buffers. For instance, if I just need to quickly look at one buffer, I can jump right back to where I was with ctrl-caret.
• ctrl-W introduces a whole host of options for working with split windows.

And lastly, can I just say that Vim is seriously the poster-child for documentation? I recommend spending 15-30 minutes, every day, in :help, just exploring. There is a LOT there, and all excellently documented.

So, all you vimsters out there: what commands do you frequently use? What features of vim are you so dependent on that you’d be useless without them? Do share!

Capistrano 2.5.0 is now available! You can read the full release announcement on the capify.org news blog.

SQLite3-Ruby version 1.2.3 is now available. It is a maintenance release, fixing just a few things:

• The permissions on the database.rb and translator.rb files in 1.2.2 were incorrect, resulting in broken sqlite3-ruby installations for many *nix users. This is now fixed.
• A few more Ruby 1.9 compatibility issues were patched.
• Some optimizations were applied to speed up iterating over result sets.

To install or upgrade:

  gem install sqlite3-ruby

Thanks!

Are you currently using Capistrano 1.4.1? If so, drop everything (I mean it, do this RIGHT NOW) and install Capistrano 1.4.2.

Why, you ask?

Capistrano 1.4.1 will work just fine, right up until you decide you want to experiment with Capistrano 2. When you do that, Cap 2.3+ will install net-ssh 2.x, which kills Capistrano 1.4.1 in all kinds of really obscure ways.

The good news is that Cap 1.4.2 is completely compatible with Cap 1.4.1. It adds no new features. The only “bug” it fixes is that if you ever install net-ssh 2.x, Cap 1.4.2 will still happily continue to work.

Ultimately, an upgrade to Cap 2 is recommended, but I understand it’s not feasible for everyone. So, if you’re one of those who can’t go cap2 yet, please please please PLEASE PLEASE FOR THE LOVE OF ALL THAT IS HOLY upgrade to Cap 1.4.2. It’ll make your life easier, and it’ll make my life easier (because I won’t have to keep troubleshooting the same issues over and over). Thanks!

Having spent the better part of a day googling and struggling, I figured it would possibly benefit others if I took a minute to post the steps I took to clone a VMWare Fusion image. The image in question is of Ubuntu Server (Hardy). I’m using VMWare Fusion 2 (beta 2).

Just find your “Virtual Machines” folder (should be in your Documents folder), and copy the image in question to a new location. (The images are actually folders; a simple “cp -R” worked fine for me.)

Then, open the copied image in VMWare Fusion and boot it. VMWare Fusion will ask if you if you copied or moved the image—be sure to say you copied it (that let’s VMWare set up a new MAC address for your image).

Go ahead and log in once the server boots. You’ll find networking is all hosed. To fix networking, this worked for me:

• “sudo hostname blah”, to set the hostname. This doesn’t change it permanently, so you’ll also want to:
• “sudo vim /etc/hostname”. Change the contents of the file to the hostname you want. Then:
• “sudo vim /etc/hosts”. Replace all mentions of the old hostname with the new hostname.
• “sudo vim /etc/udev/rules.d/70-persistent-net.rules”. There will be two entries in this file. The first points eth0 at the old MAC address, and the second points eth1 at the new. Go ahead and delete the first entry, and change “eth1” to “eth0” in the second (and now only) entry.
• “sudo shutdown -r now” to restart your virtual machine.

Once your machine comes back up, you should have a network again! Now, if only VMWare Fusion could bake all this in somehow… :/

(Related: installing VMWare Tools on Ubuntu server. It’s from 2006, but it still worked well enough for me, though I followed the instructions for tweaking the network that the VMWare Tools install gave at the end, rather than what this gent said.)

So, the second Capistrano tutorial session has come and gone. It was a great session, though it started pretty rough. (I was very unprepared for some windows-related configuration issues during the first half-hour. That won’t happen again.) I appreciate everyone’s patience who participated!

I’m going to be quiet on the tutorial front for the next several weeks, but its not because I’ve given up on them. I’m going to be rethinking some aspects of them. I really want these to be opportunities for people to master capistrano, and while these first two tutorials have been very useful to those who attended (their words, not mine), I think they could be even better.

Stay tuned!

The first online Capistrano tutorial session was very successful, thanks to everyone who participated. Now, it’s time to try again!

The second online Capistrano tutorial session is now open for registration: http://events.capify.org/events/2.

It is scheduled for Wednesday, August 13th, at 7pm (Mountain time), and will last two hours (plus up to an hour at the end for Q&A). As before, this tutorial is targeted at beginning Capistrano users, those will little or no prior experience. (If you’re looking for a session for more advanced users, keep watching this space—I’ll probably do one of those eventually, as soon as I can write a curriculum for it.)

Tyler Bird, the director of the Capistrano documentation effort, has begun collecting links about Capistrano, to build a centralized de.licio.us account that people can use to find cap-related info around the web. You can read more about it on the capify.org/news site.

Today I’m pleased to announce the first online Capistrano tutorial session. It will be held via Campfire on July 15 at 7pm MDT, focusing on Capistrano basics for Rails deployment. The session will accomodate only ten people. For all details, read up at capify.org:

http://www.capify.org/2008/7/6/online-capistrano-tutorial-session

If this proves popular enough, and if the format works out well, I’ll consider hosting more of these on more advanced topics.

Capistrano 2.4.3 is now available!

gem install capistrano

This is a maintenance release. The changes since 2.4.0 include:

• In 2.4.0 and earlier, port settings in ssh_config files were not honored by Capistrano. 2.4.3 will now honor those settings.
• The capture() helper no longer raises an exception if stderr text is received, but instead emits a warning.

Please report bugs to the Capistrano bug tracker at Lighthouse:

http://capistrano.lighthouseapp.com/projects/8716/tickets

Patches should be generated against the latest version of the code at GitHub:

http://github.com/jamis/capistrano

You may submit patches via email, as submissions to the Capistrano bug tracker, or as GitHub pull-requests.

Net::SSH 2.0.3 is now available!

gem install net-ssh

This is a maintenance release that fixes the following issues:

• Net::SSH 2.0.2 and earlier wouldn’t work with Mocana SSH and a few other esoteric SSH servers commonly used on routers and other hardware. (Net::SSH truncated the server’s identity string at the first space, rather than at the end of the line as required by the specification, causing subsequent algorithm negotiations to fail.) 2.0.3 now preserves the entire identity string as reported by the remote server.
• Malformed entries in ssh_config files are silently ignored now.
• Entries in ssh_config files that are delimited with equals-signs are now parsed correctly.
• The NameError raised when a forwarded port request could not be satisfied now raises the correct exception.
• Net::SSH::Version is now Comparable, so you can test two version objects using comparison operators.

Please report bugs to the bug tracker for net-ssh on rubyforge:

http://rubyforge.org/tracker/?atid=1123&group_id=274&func=browse

If you have a patch you want to submit, please make sure to create it against the latest version of the code in the Net::SSH repository on GitHub:

http://github.com/jamis/net-ssh/commits/master

Patches may be submitted via email, as GitHub pull-requests, or as posts to the net-ssh tracker on rubyforge.

Capistrano 2.4.0 is now available.

gem install capistrano

Report bugs to Capistrano’s Lighthouse project. And if you have some ideas for patches, please patch against the code at Capistrano’s GitHub repository.

Here’s the skinny on 2.4.0:

death to “git fetch --tags“

You git users out there will be pleased to learn that this release will no longer do “git fetch --tags”, meaning your git-based deploys will finally work again. Sorry that took so long to fix. It was really ridiculously simple. :(

cap -d

Mark Imbriaco added a debug switch for enabling conditional execution of commands. This is fantastic for debugging and testing tasks, since you can basically step through your tasks and approve or deny each remote command as it is executed. Just give cap the “-d” switch to enable this.

New and improved sudo helper

No, really. This time I mean it. I pulled the version of sudo introduced in 2.3.0 (where each sudo command was wrapped in an explicit sh invocation), and put it back almost exactly as it was. However, if you call sudo without a command to execute, it will instead return the sudo command to use. On top of that, if the run() helper detects that you’re using sudo, it will listen for password prompts. So you can now do arbitrarily complex sudo commands like this:

1

run "if [ -d /some/directory ]; then #{sudo} chmod -R g+w /some/directory; fi"

In other words, just dump the sudo() call into your command as an interpolated value, and the real sudo command gets substituted. You can pass options to it as well:

1

run "#{sudo :as => "bob"} something"

And, naturally, the original sudo() syntax we all know and love remains as before:

sudo "something", :as => "bob" 

The deployment recipes themselves have been updated to use this new syntax, as needed.

:runner vs. :admin_runner

Some cappers have noted that having deploy:setup and deploy:cleanup run as the :runner user messed up their carefully crafted permissions. I agreed that this was a problem. With this release, deploy:start, deploy:stop, and deploy:restart all continue to use the :runner user when sudoing, but deploy:setup and deploy:cleanup will use the :admin_runner user. The :admin_runner variable is unset, by default, meaning those tasks will sudo as root, but if you want them to run as :runner, just do “set :admin_runner, runner”.

deploy:upload with globs

You can now specify glob patterns with deploy:upload:

1

$ cap deploy:upload FILES="config/apache/*.conf" 

As before, you can also specify entire folders to upload, but now the upload is being done via the upload() helper (introduced in 2.3.0) so the behavior is more standardized.

before/after hooks use well-defined server scope

Prior to this release, if you defined a before hook using a block, the block would be executed within the same server scope as the original task that invoked task that this hook was attached to. (whew!)

In other words:

1
2
3
4
5
6
7
8
9
10
11
12

task :first, :roles => :app do 
  # ... 
  second 
end 

task :second, :roles => :db do 
  # ... 
end 

before :second do 
  # ... 
end 

Prior to this release, when “first” calls “second”, the before hook at the bottom would get called, but it would get called with the :roles => :app server constraint active, instead of :roles => :db. This preview release makes those hooks use the server scope of the task they are attached to.

host reported correctly from SCM output

Prior to this release, when any output was processed by the :checkout, :export, or :remote_cache strategies, you’d not be able to tell what host the output was from (it’d just be prefixed [err] or [out]). With this release, that output is correctly tagged with the host that generated it, making it easier to troubleshoot issues with SCM command execution.

Disable asset timestamp normalization

Some SCM’s give you the option of forcing the modification times of files that are being checked out to be the time that they were last modified in the repository. If your SCM gives you that capability, and if you enable it (however that works in your SCM), then you don’t need the massive touch command that Capistrano’s default deployment tasks run on each deploy, which is cap’s way of forcing all assets on all of your servers to have the same timestamp.

To disable Capistrano’s default timestamp normalization step, just set the :normalize_asset_timestamps variable to false:

1

set :normalize_asset_timestamps, false

You could also do this if you are deploying to a single server only. Note, though, that if you do this when deploying to multiple servers, and you haven’t configured your SCM to set the timestamps on the files on checkout, then Rails may use inconsistent timestamps on the assets, causing caching of your assets to fail and resulting in longer load times for your users.

Other lesser fixes, straight from the CHANGELOG

• Make git SCM honor the :scm_verbose option [Jamis Buck]
• Don’t follow symlinks when using :copy_cache [Jamis Buck]
• If :mode is given to upload() helper, do a chmod after to set the mode [Jamis Buck]
• Fix load_from_file method for windows users [Neil Wilson]
• Display a deprecation error if a remote git branch is specified [Tim Harper]
• Make set(:default_shell, false) work for not using a shell on a per-command basis [Ryan McGeary]
• Use upload() instead of put() with the copy strategy [Jamis Buck]
• Improved test coverage [Ryan McGeary]
• Fixed “coverage” take task [Ryan McGeary]
• Fix deploy:pending when using git SCM [Ryan McGeary]
• Make sure deploy:check works with :none scm (which has no default command) [Jamis Buck]

On February 16 I set myself to travel a path that I hoped would eventually help me deal with the project overload I was feeling. Today I get to release Capistrano 2.3, which is very near to the end of that path!

gem install capistrano

Capistrano 2.3 is primarily significant in that it switches to the new Net::SSH v2 library, which is faster and slimmer than the older Net::SSH v1 library. It also pulls in Net::SFTP v2, and the new Net::SSH::Gateway and Net::SCP libraries.

In addition to that, cap 2.3 adds several exciting (if I do say so myself) new features, and a few bug fixes that had lingered for far too long.

FEATURE: The :copy strategy has been significantly improved. Instead of doing a blind checkout/export, archiving the result, and copying it over, you can specify that a local cached copy of your repository should be used. If the cached copy does not exist, it is created (via a checkout), otherwise it is simply updated (a much faster operation than a checkout, typically). This functions much like the :remote_cache strategy, but locally. To use this, simply set the :copy_cache variable to true:

1
2

set :deploy_via, :copy
set :copy_cache, true

By default, the cached copy will be in your machine’s temporary directory (/tmp, for example), but you can specify your own location by setting :copy_cache to the desired path:

1
2

set :deploy_via, :copy
set :copy_cache, "/u/caches/#{application}"

But wait! There’s more! Suppose you have certain files that you don’t want to deploy, like photoshop files or your .git directory. You can set the :copy_exclude variable to a file glob (or an array of globs):

1
2
3

set :deploy_via, :copy
set :copy_cache, true
set :copy_exclude, [".git", "materials"]

This should make your deploys faster than ever, especially using a tip I hope to share in the next day or two that uses this in conjunction with some vendor/rails symlinking.

FEATURE: Even though I strenuously believe it is a mistake to deploy anything that is not under source control, I’ve finally caved and added a dumb :none SCM module. It can be used to deploy a specific directory if used with the :copy strategy:

1
2
3

set :repository, "."
set :scm, :none
set :deploy_via, :copy

Again, there are very, very few cases when I think use of this technique is justified, but because I ran into one myself a month ago, I decided it was worth adding.

FEATURE: Support was added for “depend :remote, file” to test for the existence of a specific file:

depend :remote, "/etc/syslog.conf"

This is used in conjunction with the deploy:check task.

FEATURE: You can specify ssh_options per-server, now, simply by giving an :ssh_options key and corresponding hash with the server definition:

role "some.host", :ssh_options => { :keys => "/path/to/key" }

FEATURE: There are two new file transfer helpers, upload and download, which are much more powerful and resource-friendly than the old ‘put’ and ‘get’ helpers. You can use upload and download to transfer single files, or entire directory trees:

1
2

upload "/local/file", "/remote/file"
download "/remote/file", "/local/file"

This will transfer to or from all active servers, which is particular tricky when using the download helper, since it will download the file simultaneously from all active hosts. To make this work, you need to make sure each is downloaded to a different location:

1
2

Dir.mkdir("destination")
download "/remote/file", "destination/file-$CAPISTRANO:HOST$"

The above will download the file from each host to a file on the local host, where the local file includes the name of the source host. Tricky!

Also, you can now specify that you want to upload or download via SCP instead of SFTP:

1
2

upload "local", "remote", :via => :scp
download "remote", "local", :via => :scp

The default is :sftp.

The less exciting (but still mildly titillating) things are the bug fixes and pleasure-enhancing behavioral changes:

• The default run options are now mixed into the command options when executing a command from the capistrano shell.
• The git SCM now uses git-ls-remote instead of git-rev-parse to resolve the revision of the checkout.
• A trivial export method has been added to the git SCM.
• The git SCM will include tags when it fetches.`
• The sudo() helper now works nicely with complex and chained commands.
• The deploy:setup task will now use sudo if :use_sudo is true (the default)

So, have at it! Remember to report bugs to http://capistrano.lighthouseapp.com. And patches are always welcome via git—just fork the capistrano repository at git://github.com/jamis/capistrano.git (thanks GitHub!).

Cheers!

At last! Net::SSH 2.0 is available! Also available are Net::SFTP 2.0, Net::SCP 1.0, Net::SSH::Gateway 1.0, and Net::SSH::Multi 1.0.

1
2
3
4
5

$ gem install net-ssh \
  net-sftp \
  net-scp \
  net-ssh-gateway \
  net-ssh-multi

All of these Ruby libraries are for communicating with remote servers via the SSH protocol in different ways.

Net::SSH and Net::SFTP are both significant upgrades from their previous incarnations; if you have used either library in the past, you’ll want to read the documentation (Net::SSH, Net::SFTP). The Net::SSH API is still fairly similar to the way it was before, but the Net::SFTP API is entirely different.

All have pretty complete RDoc documentation, so you should be able to employ “ri” to good effect to find your way around the libraries. (Try “ri Net::SSH”, for example, to get started.)

Per-library synopses follow.

Net::SSH 2.0

This is a significant upgrade from Net::SSH 1.x. Changes from 1.x include (but are not limited to):

• Net::SSH 2.0 no longer requires the Needle dependency-injection library. This has made the library faster, lighter, and easier to understand.
• The “sync” and “sync-shell” services have been removed and will not return. (If you need the functionality of those services, I strongly encourage you to port them to Net::SSH 2 and release them as separate extensions.)
• Options in ~/.ssh/config files are (partially) supported, and are loaded by default now.
• The Net::SSH::start method now requires both a host and a username as the first two parameters, rather than inferring the username.
• There is now a Session#exec method that makes it easier than ever to just execute and interact with a command.
• Channel now sports an #on_process method that you can send a block to, which will be executed on every pass of the event loop.
• Channel#on_request now accepts a string to indicate which request you want to respond to.
• Channel open failures may be captured via Channel#on_open_failure.
• Lots, lots, LOTS, more—refer to the rdocs for all the gory details.

1
2
3
4
5
6

require 'net/ssh'

Net::SSH.start('localhost', 'jamis') do |ssh|
  ssh.exec('hostname') # prints the results to $stdout
  ssh.loop
end

Net::SFTP 2.0

This is a complete rewrite of the original Net::SFTP 1.x code, and shares very, very little in common with it. The new version has a much cleaner implementation and API, and provides some really handy methods for transferring files and directories.

1
2
3
4
5
6
7
8
9
10

require 'net/sftp'

Net::SFTP.start('localhost', 'jamis') do |sftp|
  sftp.upload! "/local/file", "/remote/file"
  sftp.download! "/remote/file", "/local/file"

  sftp.file.open("/remote/file", "w") do |file|
    file.puts "here is some data"
  end
end

Net::SCP 1.0

This provides a way to transfer files and directories via the SCP protocol, over Net::SSH.

1
2
3
4
5
6

require 'net/scp'

Net::SCP.start('localhost', 'jamis') do |scp|
  scp.upload! "/local/file", "/remote/file"
  scp.download! "/remote/file", "/local/file"
end

Net::SSH::Gateway 1.0

This library makes it easy to tunnel connections though firewalls. You simply connect to the gateway machine, and then specify which ports you want forwarded.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

require 'net/ssh/gateway'

gateway = Net::SSH::Gateway.new('host', 'user')

gateway.ssh("host.private", "user") do |ssh|
  puts ssh.exec!("hostname")
end

gateway.open("host.private", 80) do |port|
  require 'net/http'
  Net::HTTP.get_print("127.0.0.1", "/path", port)
end

gateway.shutdown!

Net::SSH::Multi 1.0

This library makes it simple to open multiple Net::SSH connections and tie them all together, running commands in parallel (much like Capistrano does).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

require 'net/ssh/multi'

Net::SSH::Multi.start do |session|
  # access servers via a gateway
  session.via 'gateway', 'gateway-user'

  # define the servers we want to use
  session.use 'user1@host1'
  session.use 'user2@host2'

  # define servers in groups for more granular access
  session.group :app do
    session.use 'user@app1'
    session.use 'user@app2'
  end

  # execute commands on all servers
  session.exec "uptime"

  # execute commands on a subset of servers
  session.with(:app).exec "hostname"

  # run the aggregated event loop
  session.loop
end

Enjoy!

With the impending releases of the improved Net::SSH family of libraries, and the next Capistrano release that will depend on them, it was time to go back and make sure that the oldies-but-goodies will still work along side these new-comers.

To that end, I present:

Net::SSH 1.1.3

This maintenance release will hopefully fix the hangs that people have been reporting when dealing with long-running requests in Net::SSH, Net::SFTP, and Capistrano.

gem install net-ssh

It is strongly recommended that all Net::SSH users upgrade to this version.

Capistrano 1.4.2

This release merely adds an explicit dependency on Net::SSH and Net::SFTP versions less than 1.99.0, allowing you to use both cap1 and cap2 side-by-side after Net::SSH v2 (and friends) are released.

gem install --version 1.4.2 capistrano

If you are needing to run cap1 and cap2 side-by-side (for instance, if you haven’t yet upgraded some recipes, but want to enjoy the cap2 goodies on newer projects), you should upgrade your cap1 installation to 1.4.2. If you are completely upgraded to cap2, you can safely ignore this release.