Is the Server Up Yet?

I hate waiting, but I have to do a lot of it. Waiting for servers to restart. Waiting for services to come up. Waiting around for a Yes or a No. To that end I have a couple of BASH functions that can monitor a server and pop up a notification when it’s up.

First we need something to notify us. I’m a fan of Growl, which was popular before Apple introduced it’s User Notifications. But, it’s just a personal preference both will get the job done in a very similar fashion. In both cases you need to install a command line component.

For Growl, it’s growlnotify, which is available from the deleveloper:

http://growl.info/downloads.php#generaldownloads

Alternatively, there’s a Homebrew Cask that can provide it:

brew install Caskroom/cask/growlnotify

For User Notifications I like terminal-notifier. There are other options, including AppleScript, but I think terminal-notifier wraps it up neatly. Download it or install it with brew

brew install terminal-notifier

growlnotify and terminal-notifier have different (yet annoyingly similar) syntax. To avoid hard-coding the details in all of my functions, I first created a notifier function, allowing me to swap out the notifier in one place.

Growl:

notify() { growlnotify -s -m "$@"; }

User Notifications:

notify() { terminal-notifier -sound default -message "$@"; }

The -s option to growlnotify make the message sticky, it will stay on the screen until clicked (the default behavior is to stay on the screen for 5 seconds), which I like.

Notifications don’t have a “sticky” option and will automatically dismiss themselves. Because of that, I have it play an alert with the -sound default option. If you find that annoying, remove it (or nerd out by using “Submarine” instead of “default”).

Now that we have a way to get notified, let’s do something with it. We want to know when a server is up, and the easiest measure of that is can we ping it. It’s easy enough to get ping to retry until it gets a response:

while ! ping -c1 www.example.com &>/dev/null; do sleep 5; done

The -c1 says “send one packet”, if it receive a response to that packet, it will exit with a status of “0”, if it times out the status will non-zero. To turn it into a function:

wait-for-host() {
    while ! ping -c1 $1 &>/dev/null; do sleep 5; done
    notify "$1 is up: `date`";
}

Which can then be called as wait-for-host www.example.com. The loop will continue until the ping succeeds and when it does, notify will be called. Works, but a couple of refinements:

If you forget the hostname, ping will fail with a usage error and the loop will be infinite. So, I test for that:

if [ -z "$1" ]; then echo "... specify a host."; return; fi

Most of the time, 5 seconds is a good interval to check at. However, if something is going to be down a long time, to cut down on traffic I like to bump up the sleep:

sleep ${2:-5};

This takes advantage of Bash’s Parameter Substitution. If $2, the second argument to the function, is set, it is used, otherwise the default value of “5” is. Now I can limit the pinging to every 30 seconds with wait-for-host www.example.com 30 or stick to 5 with wait-for-host www.example.com. Put it all together and we have

wait-for-host () {
    if [ -z "$1" ]; then echo "... specify a host"; return; fi
    while ! ping -c1 $1 &>/dev/null; do sleep ${2:-5}; done
    notify "$1 is up: `date`";
}

Now, I know what you’re saying. “I use AWS and it doesn’t respond to ICMP traffic, ping tells me nothing.”. The way to tell if a host that doesn’t respond to ping is up is to check and see if a service is up, and the most basic service for most servers is SSH. If only we had a program that could see if a port is open on a server. Oh, wait, we do, Netcat.

Actually, that’s really underselling it. Netcat bills itself as the the TCP/IP swiss army. Some of it’s features include:

• Outbound or inbound connections, TCP or UDP, to or from any ports
• Full DNS forward/reverse checking, with appropriate warnings
• Ability to use any local source port
• Ability to use any locally-configured network source address
• Built-in port-scanning capabilities, with randomizer
• Built-in loose source-routing capability
• Can read command line arguments from standard input
• Slow-send mode, one line every N seconds
• Hex dump of transmitted and received data
• Optional ability to let another program service established connections

It’s worth reading up on as it can help with all sorts of network debugging tasks. However, for our situation, we need one of it’s most basic features, port scanning (-z). Netcat will scan a range of TCP ports and report which are open:

nc -z www.example.com 22-25
Connection to www.example.com port 22 [tcp/ssh] succeeded!
Connection to www.example.com port 25 [tcp/smtp] succeeded!

If any ports are open it returns an exit status of “0”, otherwise it returns “1”. That combined with the fact that a range of one port is perfectly valid gives us the command we need:

while ! nc -z www.example.com 22 &>/dev/null; do sleep 5; done

22 being the SSH port. We could use that in our function, but I find it more helpful to have a generic version that can wait for any port:

wait-for-service() {
    while ! nc -z $1 $2 &>/dev/null; do sleep ${3:-5}; done
    notify "$1:$2 is up: `date`"
}

Giving us wait-for-service www.example.com 80 with an optional third option of how long to wait between tries wait-for-service www.example.com 80 60

Now we can wrap that to create a specify SSH version:

wait-for-ssh() {
    wait-for-service $1 22 $2
}

Again, the optional last argument sets the retry time. Throw in a:

alias wait-for-aws=wait-for-ssh

if you’d like (or wait-for-ec2 if you’re pedantic), which finally boils us down to:

wait-for-aws my-ec2-instance.us-west-1.compute.amazonaws.com

I should pause and give credit where credit is do. However, I can’t. I didn’t originally write these alias. Over time, I’ve refactored them down to their current form, creating the generic notification function, adding the optional delay, and general simplifying the form. However, at this point, I no longer remember where I got the original idea. So, thank you, whoever you were.

I think once you have these tools, you won’t remember how you lived without them. And once you understand the concept of using notifiers, you’ll find plenty of other uses. But that’s another show.