Simple Ubuntu Upstart Alerts via Email

July 20, 2015Lance Lakey

Ubuntu Upstart is an asynchronous, event-driven equivalent to init or systemd. Recently, on an Upstart system, I needed a quick script which would follow a log file, search for some patterns, and send an email when it found matches.

Start With `bash`

I started with this quick and dirty bash one-liner:

exec tail -F /var/log/nginx/access.log | grep --line-buffered -iP '^(?=.*<PHRASE>)(?=.*<IP ADDRESS>)' | while read LINE; do echo "$LINE" | mail -s 'Match' -a "From: user@host.example.com" me@example.com; done

Upstart runs any scripts which live inside /etc/init, so I turned this into an Upstart config file and saved it at /etc/init/match.conf.

# Tail a log file, grep for matches, send email

description	"match-mail"

start on runlevel [2345]
stop on runlevel [!2345]

respawn
respawn limit 10 5
umask 022

console log

exec tail -F /var/log/nginx/access.log | grep --line-buffered -iP '^(?=.*<PHRASE>)(?=.*<IP ADDRESS>)' | while read LINE; do echo "$LINE" | mail -s 'Match' -a "From: user@host.example.com" me@example.com; done

The Upstart commands prior to the one-liner are pretty standard:

run this command in any/all configuration states
restart this process if it stops
if you respawn more than 10 times in 5 seconds, there’s a problem
set a standard file creation mask
log output to the console

The bash one-liner did the job. However, I also wanted more details. I wanted the script to extract a timestamp, convert it to a different format, and then format the email body with only the information I needed: the date, IP address, and URL. At this point it made sense to switch to Ruby.

Second Version, Mostly In Ruby

I kept the config file almost the same, but renamed it to match-rb.conf, and changed the one-liner to this:

exec tail -F /var/log/nginx/access.log | /usr/local/example/match.rb

We’re still piping tail into the match script, but that’s now a Ruby script rather than bash. Here it is:

#!/usr/bin/env ruby
# mail gem required, gem install mail

require 'pp'
require 'mail'

ARGF.each_line do |line|
  regex = /"http.*<PHRASE>=(\d+)"/
  matched = line.match regex

  if matched
    line_split = line.split
    ip = line_split[0]
    url = line_split[10]

    if ip == '<IP ADDRESS>'
      millis = matched[1]
      date = Time.at(millis.to_f / 1000.0)
      body = "Date:  #{date} \nIP:  #{ip} \nURL:  #{url} \n"

      mail = Mail.new do
        from    'user@host.example.com'
        to      'me@example.com'
        subject 'Match'
        body    body
      end

      mail.delivery_method :sendmail
      mail.deliver!
    end

  end

end

And there you have it. In the spirit of blogging little things, a concise solution for a common problem.

Simple Ubuntu Upstart Alerts via Email

Start With bash

Second Version, Mostly In Ruby

Start With `bash`