March 13, 2011

At the urging of @kevin from Instagram, about 6 months ago I bought a little 11" MacBook Air, for what he calls "admin anywhere." Combined with a power supply, an iPhone charger, and a Sprint EVDO air-card--a configuration that most of our engineering team subscribes to--this means we can securely manage the Hearsay datacenter from anywhere, whether that means getting work done, just checking up on stats, or debugging the occasional issue.

In order to satisfy the "secure" requirement, we've got a multi-layered system involving (among other things) a VPN, SSH requiring key-based auth, and only a DMZ that's exposed to inbound external traffic. Rather than needing to repeadedly connect to every server I care about (monitoring, operations, front-end machine, database replicas, etc), I keep a screen session open on the jump box with the ability to quickly connect to specific servers that I care about.

However, since you don't want to leave your private keys lying around on remote servers (such that if a single server was compromised, it would compromise the rest of the datacenter), we use ssh-agent to forward key-based auth requests to the original host (eg. my laptop). Unfortunately, the way that ssh-agent works is by setting environment variables in your remote login session (I SSH from laptop -> jump box, and then want to SSH from jump box -> database replica ... jump box has environment variables set that tell it where to forward the SSH auth requests to ... a temporary socket that points to ssh-agent on my laptop).

But since the screen session can be long-lived (but I don't want the remote SSH sessions to necessarily be long-lived), the environment variables in each of the screen sessions are out of date after I disconnect for the first time. After getting annoyed with having to reset this manually too much (and being at #pycon 2011 this weekend with lots of connect/disconnect/reconnect happening between sessions), I finally did the research to solve this problem. Many of the solutions that I found online only mostly solve this problem, but not completely, so I wanted to share my tried-and-true final solution.

Most of the credit for this solution goes Samat Jain, and I just tried to make it a little easier.

First, I set up a script on the jump box to save out my screen environment vars--I call this file "~/":

# Wrapper script for screen

echo 'saving out screen vars from ~/'
if [ -n "$SSH_CLIENT" ]; then
  mkdir -p "$HOME/.screen"

  # Variables to save

  for x in ${SSHVARS} ; do
    (eval echo $x=\$$x) | sed  -e 's/=/="/' -e 's/$/"/' -e 's/^/export /'
  done 1> "$HOME/.screen/session-variables"

Then we set up aliases in .bashrc (or really, .bash_aliases which gets included from .bashrc) so that we can use normal commands ("screen -rd" to reattach on login, and "ssh" to connect to internal machines) but still ensure environment variables are properly set.

if [ -e ~/.screen/session-variables ]; then
  alias ssh='source ~/.screen/session-variables; ssh'

if [ -e ~/ ]; then
  alias screen='~/; screen'

That's it!


blog comments powered by Disqus