For a while I have been looking for a simple service monitoring tool that didn’t cost anything to run and gave me what I wanted to know: when services are in trouble. Everything I found was either too complicated or partially filled what I was looking for. So I decided to quickly write something up and created go-monitor.
TL;DR
go-monitor is a simple service monitoring tool. Its primary function is to monitor a list of services specified by the user, and then
to notify one or many users if/when these services go down. In my implementation I opted to send an SMS to myself when a service goes down,
and I end up received something like this:
node not running on server ksred-server1 server1.ksred.me with IP 198.162.10.1!
Notifications for the same service being down are sent out at an interval specified by the user. For example, if the interval is set to 30 minutes the user will receive an SMS every 30 minutes the service is down. Services are checked at an interval of 60 seconds by default.
How it works
This was written to be used on Unix systems. In order to get a process list all I do is make the program run ps aux | grep PROCESS_NAME. In Go,
that is done as follows:
c1 := exec.Command("ps", "aux")
c2 := exec.Command("grep", processName)
r, w := io.Pipe()
c1.Stdout = w
c2.Stdin = r
var b2 bytes.Buffer
c2.Stdout = &b2
c1.Start()
c2.Start()
c1.Wait()
w.Close()
c2.Wait()
lines, err := lineCounter(&b2)
It’s a little bit complicated due to the commands having to be piped into one another. I then send the output to a separate function lineCounter which
counts the number of lines in the output. If there are 0 lines, the process is dead and an alert is sent.
Config is loaded through a yaml file, making it quick and easy to edit any configuration.
processes: [ "list", "of", "processes" ]
config:
messagebirdtoken: "test_TOKEN"
messagebirdsender: "+sender-number"
recipients: "+recipient-numbers,+one-or-many"
defaultttl: 30 // This is a TTL in minutes
servernicename: "my-server-name"
I also wrote an init script so it can be used as a service, like service go-monitor start.
Channels
I spent quite some time figuring out the best way to do this using channels. I haven’t done any programming in Go using channels, and it took a bit of experimentation and searching to get the correct implementation. I think I’m there, and I ended up doing the following.
I create a parent WaitGroup which waits for two processes to complete before exiting the application. This WaitGroup never completes and so the function never
exits. This makes the program long running, which is what we are looking for.
I then created two separate goroutines. The first routine goes through the list of processes and checks if they are running. If they are not running, a message is passed onto an error channel. This loops forever at intervals of 60 seconds.
This first goroutine has its own WaitGroup which waits on all processes to complete before looping again. If you have 5 processes, the WaitGroup is 5.
The second goroutine waits and listens on the error channel for any incoming messages. When a message is received it calls the notify function which sends out the message. This notify function is called as a goroutine to be non-blocking.
MessageBird
I decided to go with MessageBird as the delivery mechanism due to their ease of integration (a simple POST) as well as their pricing. They also have loads of channels of delivery which makes switching to another easy.
// Send text message
authToken := conf.Config.MessageBirdToken
urlStr := "https://rest.messagebird.com/messages"
v := url.Values{}
v.Set("recipients", recipientNumber)
v.Set("originator", conf.Config.MessageBirdSender)
v.Set("body", proc+" not running on server "+server+"!")
rb := *strings.NewReader(v.Encode())
client := &http.Client{}
req, _ := http.NewRequest("POST", urlStr, &rb)
req.SetBasicAuth("AccessKey", authToken)
req.Header.Add("Accept", "application/json")
req.Header.Add("Content-Type", "application/x-www-form-urlencoded")
// Make request
_, err := client.Do(req)
if err != nil {
//fmt.Printf("Error: %s\n", err.Error())
return
}
Conclusion
The ease of writing in Go is still one of the biggest reasons I love the language. Building a simple version of the functionality I want and having it run on my servers in a few hours makes for happy development.
The code is available on Github.