I hate waiting, but I have to do a lot of it. Waiting for servers to restart. Waiting for services to come up. Waiting around for a Yes or a No. To that end I have a couple of BASH functions that can monitor a server and pop up a notification when it’s up.
First we need something to notify us. I’m a fan of Growl, which was popular before Apple introduced it’s User Notifications. But, it’s just a personal preference both will get the job done in a very similar fashion. In both cases you need to install a command line component.
For Growl, it’s
growlnotify, which is available from the deleveloper:
Alternatively, there’s a Homebrew Cask that can provide it:
terminal-notifier have different (yet annoyingly
similar) syntax. To avoid hard-coding the details in all of my
functions, I first created a notifier function, allowing me to swap
out the notifier in one place.
-s option to
growlnotify make the message sticky, it will stay
on the screen until clicked (the default behavior is to stay on the
screen for 5 seconds), which I like.
Notifications don’t have a “sticky” option and will automatically
dismiss themselves. Because of that, I have it play an alert with the
-sound default option. If you find that annoying, remove it (or nerd
out by using “Submarine” instead of “default”).
Now that we have a way to get notified, let’s do something with it. We want to know when a server is up, and the easiest measure of that is can we ping it. It’s easy enough to get ping to retry until it gets a response:
-c1 says “send one packet”, if it receive a response to that
packet, it will exit with a status of “0”, if it times out the status
will non-zero. To turn it into a function:
1 2 3 4
Which can then be called as
wait-for-host www.example.com. The loop
will continue until the ping succeeds and when it does,
be called. Works, but a couple of refinements:
If you forget the hostname, ping will fail with a usage error and the loop will be infinite. So, I test for that:
Most of the time, 5 seconds is a good interval to check at. However, if something is going to be down a long time, to cut down on traffic I like to bump up the sleep:
This takes advantage of Bash’s
Parameter Substitution. If
$2, the second argument to the function, is set, it is used, otherwise
the default value of “5” is. Now I can limit the pinging to every 30
wait-for-host www.example.com 30 or stick to 5 with
wait-for-host www.example.com. Put it all together and we have
1 2 3 4 5
Now, I know what you’re saying. “I use AWS and it doesn’t respond to ICMP traffic, ping tells me nothing.”. The way to tell if a host that doesn’t respond to ping is up is to check and see if a service is up, and the most basic service for most servers is SSH. If only we had a program that could see if a port is open on a server. Oh, wait, we do, Netcat.
Actually, that’s really underselling it. Netcat bills itself as the the TCP/IP swiss army. Some of it’s features include:
- Outbound or inbound connections, TCP or UDP, to or from any ports
- Full DNS forward/reverse checking, with appropriate warnings
- Ability to use any local source port
- Ability to use any locally-configured network source address
- Built-in port-scanning capabilities, with randomizer
- Built-in loose source-routing capability
- Can read command line arguments from standard input
- Slow-send mode, one line every N seconds
- Hex dump of transmitted and received data
- Optional ability to let another program service established connections
It’s worth reading up on as it can help with all sorts of network debugging tasks. However, for our situation, we need one of it’s most basic features, port scanning (-z). Netcat will scan a range of TCP ports and report which are open:
1 2 3
If any ports are open it returns an exit status of “0”, otherwise it returns “1”. That combined with the fact that a range of one port is perfectly valid gives us the command we need:
22 being the SSH port. We could use that in our function, but I find it more helpful to have a generic version that can wait for any port:
1 2 3 4
wait-for-service www.example.com 80 with an optional third
option of how long to wait between tries
www.example.com 80 60
Now we can wrap that to create a specify SSH version:
1 2 3
Again, the optional last argument sets the retry time. Throw in a:
if you’d like (or wait-for-ec2 if you’re pedantic), which finally boils us down to:
I should pause and give credit where credit is do. However, I can’t. I didn’t originally write these alias. Over time, I’ve refactored them down to their current form, creating the generic notification function, adding the optional delay, and general simplifying the form. However, at this point, I no longer remember where I got the original idea. So, thank you, whoever you were.
I think once you have these tools, you won’t remember how you lived without them. And once you understand the concept of using notifiers, you’ll find plenty of other uses. But that’s another show.