Asynchronous SCP with Vim over mulitplexed SSH connection

To homepage

One of the problems with working with remote machines is that you expose yourself to fluctuations in network conditions that cause a laggy and inconsistent command line experience. I often alleviate this problem by editing my files locally and running them remotely. To automate the copy of edited files to remote machines, I have relied in the past on a long-running process that monitored a directory for file changes, and triggered an rsync to remote host when file changes are detected. That method had some serious performance and stability problems.

So, since I’m an avid user of vim, I was thinking whether I could use vim autocommands to trigger automatic file transfers for edited files. It would not be as comprehensive as rsync, especially when it came to any file changes done outside of vim or to deleted files, but it at least it could be simpler, more stable, hopefully more performant, and would satisfy my needs 90% of the time.

So I thought I’d just launch SCP calls whenever the BufWritePost event is triggered, which easily is done easily enough by using the :! command in vim to launch the SCP executable. It passes the executable the name of the current buffer, which just got saved, to the SCP process with %:p where % is the name of the file in the buffer, and :p is a filename-modifier which expands the file name to a full absolute path. So the autocommand would look something like:

au BufWritePost /code_path/* :!scp %:p remote_host:/

So if a file called /code_path/some_cool_script.rb was changed and saved in vim, it will match the glob expansion /code_path/*, and vim will pass its full path to scp to copy to the remote host.

The problem with this approach is two-fold:

  1. Vim will block until SCP finishes transferring the file.
  2. SCP will attempt to open a new SSH connection everytime it is invoked. For small files – that is, most files you edit on a daily basis – that means that the connection creation time will dominate.

The first problem can be dealt with by wrapping the scp command in a shell script and invoking scp in a forked subshell in the background, so that when vim invokes the script, the script returns immediately after forking and vim is not blocked. The second problem can be dealt with by reusing a long-living ssh connection. OpenSSH provides a cool connection sharing feature that is controlled with options ControlMaster and ControlPath, which basically places a fifo (or named input) file in the path specified by ControlPath. The ssh process which controls the “master” connection listens to incoming connections on the socket file and tunnels them through the master connection. Client ssh sessions communicate with the socket file instead of with the remote host directly. All in all, the forked subshell invocation that will launch scp now could look like:

SSH_MUX_SOCKET=~/.ssh/async-scp.socket

(if [ ! -w $SSH_MUX_SOCKET ]; then
    echo "Starting the Master SSH connection for the first time.." 
    # launch SSH connection in the background (-f) with no shell (-N)
    # that will be shared (-M) with a ControlPath (-S) of $SSH_MUX_SOCKET
    ssh -N -f -M -S $SSH_MUX_SOCKET $HOST
fi
/usr/bin/time scp -v -o "ControlPath $SSH_MUX_SOCKET" $1 $HOST:$DEST
) &

and the vim autocommand would look something like:

au BufWritePost /code_path/* :!~/scp_wrapper.sh %:p

For this to work, you will have to have a passwordless ssh key, since you can not enter your credentials in a forked subshell (not easily, anyway). Multiplexing SSH connections can significantly reduce the time files take to transfer over to the remote machine.

I’ve put all these ideas together and wrote a small and simple vim plugin that allows you to do asynchronous scp between arbitrary mirroring directories and hosts. You can see the code for that here, and you can look through it and adapt it to your needs if you like it.