I’ve been wanting to learn Golang for some while. I own the amazing, and every technical, Programming in Go and have been making my way through it over time. I’ve found though that learning a language is much easier when you have a clear project you want to build out. I took a short course on Udemy to give me a primer on the basics of Go and then was ready to build.
A non-trivial application I have been wanting to build for some time is a stock notifier.
The easy problem
I trade shares and want to keep on top of them throughout the day without manually having to check online. I want to be sent an email peridiocally, at intervals of my choosing, of a list of stocks I have chosen ordered by biggest gainers to biggest losers.
The slightly tougher problem
I want to be notified of daily trends in stocks. The trend is simple: every increasing or decreasing closing price over three days, and an increase in volume in one of the three days. I want this email to be sent to me before open or after close.
After doing some research around data sources, I felt that a JSON API with basic information would more than suffice. Google provided some great data across all of the stocks I was interested in on the Johannesburg Stock Exchange, so I chose their officially defunct finance API. Althought it is no longer advertised, it is still available for use. One day it will be shut off, so I have made the program to be able to use a variety of APIs, with the ability to switch on the source. The idea would be for the JSON to be parsed and cleaned according to the source, and for it to be stored in a suitable variable for use later.
Go uses structs as storage for specific variables. I wanted to make the config for the site as easy as possible, so I decided to use a JSON config file that would get read on program start up, and used throughout the lifecycle. The config file has the following format:
{
"MailSMTPServer" : "mail.server.com",
"MailSMTPPort" : "587",
"MailUser" : "user@mail.com",
"MailPass" : "plaintext_password",
"MailRecipient" : "recipient@mail.com",
"MailSender" : "sender@mail.com",
"Symbols" : ["NASDAQ:GOOGL", "NYSE:BLK"] //Format: exhange:stock
"UpdateInterval" : "100", // in seconds
"TimeZone" : "America/New_York",
"MySQLUser" : "mysql_user",
"MySQLPass" : "mysql_password",
"MySQLHost" : "mysql_ip",
"MySQLPort" : "mysql_port"
"MySQLDB" : "mysql_database"
}
Which is read into the the following matching struct:
type Configuration struct {
MailUser string
MailPass string
MailSMTPServer string
MailSMTPPort string
MailRecipient string
MailSender string
Symbols []string
UpdateInterval string
TimeZone string
MySQLUser string
MySQLPass string
MySQLHost string
MySQLPort string
MySQLDB string
}
The symbols are a slice (slices in Go are equatable to arrays in other languages) and can be looped through. Go uses some pretty powerful JSON parsing built in, which maps JSON data to a struct. As such, each field in the struct has a matching field in the JSON. The config is then loaded:
configuration := Configuration{}
loadConfig(&configuration)
...
func loadConfig(configuration *Configuration) {
// Get config
file, _ := os.Open("config.json")
decoder := json.NewDecoder(file)
err := decoder.Decode(&configuration)
if err != nil {
fmt.Println("error:", err)
}
As every stock has certain known fields, these would be parsed into the struct objects and used as these variables for saving and data analysis. The structs were based on the Google API return values:
type Stock struct {
Symbol string `json:"t"`
Exchange string `json:"e"`
Name string `json:"name"`
Change string `json:"c"`
Close string `json:"l"`
PercentageChange string `json:"cp"`
Open string `json:"op"`
High string `json:"hi"`
Low string `json:"lo"`
Volume string `json:"vo"`
AverageVolume string `json:"avvo"`
High52 string `json:"hi52"`
Low52 string `json:"lo52"`
MarketCap string `json:"mc"`
EPS string `json:"eps"`
Shares string `json:"shares"`
}
Once this is parsed and everything is happy, we can move onto storage and analysis.
Getting the data
We use the config Symbols variable to form the url for data retrieval. The response is then massaged and the result set parsed. There are some sanitizing functions which I won’t go into, the most important function is the parseJSONData.
symbolString := convertStocksString(configuration.Symbols)
var urlStocks string = "https://www.google.com/finance/info?infotype=infoquoteall&q=" + symbolString
body := getDataFromURL(urlStocks)
jsonString := sanitizeBody("google", body)
stockList := make([]Stock, 0)
stockList = parseJSONData(jsonString)
...
func parseJSONData(jsonString []byte) (stockList []Stock) {
raw := make([]json.RawMessage, 10)
if err := json.Unmarshal(jsonString, &raw); err != nil {
log.Fatalf("error %v", err)
}
for i := 0; i < len(raw); i += 1 {
stock := Stock{}
if err := json.Unmarshal(raw[i], &stock); err != nil {
fmt.Println("error %v", err)
}
stockList = append(stockList, stock)
}
return
}
Now that we have the data in a usable format, the string representation of numbers is adapted into full digits: M = X * 1 000 000, B = X * 1 000 000 000. This data can now finally be stored.
Storage
I chose a relational database for the data storage, namely MySQL. It’s common and scales pretty well, and Go has really good support for it. The following code illustrates a write operation for saving the data:
// Prepare statement for inserting data
insertStatement := "INSERT INTO st_data (`symbol`, `exchange`, `name`, `change`, `close`, `percentageChange`, `open`, `high`, `low`, `volume` , `avgVolume`, `high52` , `low52`, `marketCap`, `eps`, `shares`, `time`, `minute`, `hour`, `day`, `month`, `year`) "
insertStatement += "VALUES( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ? )"
stmtIns, err := db.Prepare(insertStatement)
if err != nil {
panic(err.Error()) // proper error handling instead of panic in your app
}
defer stmtIns.Close() // Close the statement when we leave main() / the program terminates
... Massage data ...
_, err = stmtIns.Exec(stock.Symbol, stock.Exchange, stock.Name, sqlChange, sqlClose,
sqlPercChange, sqlOpen, sqlHigh, sqlLow, sqlVolume, sqlAvgVolume,
sqlHigh52, sqlLow52, sqlMarketCap, sqlEps, sqlShares,
sqlTime, sqlMinute, sqlHour, sqlDay, sqlMonth, sqlYear)
if err != nil {
fmt.Println("Could not save results: " + err.Error())
}
At the end of this we have:
- Config loaded from file
- Stock data being retrieved
- Stock data parsed and massaged
- Stock data saved into a database
We can now get onto the interesting part: analysis.
Analysis of data: Gainers and Losers
I wanted to receive a nice (HTML) email periodically with a list of stocks ordered by gain. This requires: an interval, a HTML parser, and an SMTP mail call.
The interval
Interval timing is relatively easy in Go:
for _ = range time.Tick(n * time.Second) {
...
}
During the interval various checks are done to make sure the emails are only sent when they should be: during stock opening hours, Monday to Friday. The hours they are sent on is also harcoded, but can easily be set into the config.
The HTML parser
Go also has a great parsing library. An HTML template is injected with a variable and parsed accordingly:
mailTpl.Title = "Stock update"
t, err := template.ParseFiles("tpl/notification.html")
if err != nil {
fmt.Println("template parse error: ", err)
return
}
err = t.Execute(&templateString, mailTpl)
if err != nil {
fmt.Println("template executing error: ", err)
return
}
Where mailTpl is a struct containing stock data and some meta, such as the title above. The template contains variables to be parsed, as well as looping functions. A section:
{{ with .Stocks }}
{{ range . }}
<table with="100%" border="1px solid #ccc">
<tr width="100%" style="border: 1px solid #ccc">
<td><strong>Name</strong></td><td>{{ .Name }}</td>
<td></td><td></td>
</tr>
...
</table>
<br /><br />
{{ end }}
{{ end }}
With the HTML, we are now ready to send the email.
Sending the formatted email
Sending SMTP mail with Go is straightforward:
func sendMail(configuration Configuration, notifyMail string) {
// Send email
// Set up authentication information.
auth := smtp.PlainAuth("", configuration.MailUser, configuration.MailPass, configuration.MailSMTPServer)
// Connect to the server, authenticate, set the sender and recipient,
// and send the email all in one step.
mime := "MIME-version: 1.0;\nContent-Type: text/html; charset=\"UTF-8\";\n\n"
to := []string{configuration.MailRecipient}
msg := []byte("To: " + configuration.MailRecipient + "\r\n" +
"Subject: Quote update!\r\n" +
mime + "\r\n" +
"\r\n" +
notifyMail +
"\r\n")
err := smtp.SendMail(configuration.MailSMTPServer+":"+configuration.MailSMTPPort, auth, configuration.MailSender, to, msg)
//err = smtp.SendMail("mail.messagingengine.com:587", auth, "ksred@fastmail.fm", []string{"kyle@ksred.me"}, msg)
if err != nil {
log.Fatal(err)
}
}
Deeper analysis: Trends
After we have gathered enough data (I was doing calls in 15 minute intervals), we can start doing some good analysis. My trend analysis is really simple. You take a 3 day sample of data, check for increasing (or decreasing) prices day on day, and a once-off increase in volume over the same period.
if closes[0] > closes[1] && closes[1] > closes[2] && (volumes[0] > volumes[2] || volumes[0] > volumes[1]) {
return true
}
I then also added in a standard deviation function to get some idea of the volatility of the stock in question. The function is fairly long, but you can see the full calculation here.
The result is a separate mail, sent end of day or before market open, with the trending stocks as calculated above including an indication of volatility. This can be extended to include other more complex technical analysis, such as overlaying bollinger bands, stochastic graphs for price points, and more.
Conclusion
As my first work in Go, I am really pleased with the results. Loads of improvements can be made for sure, and I’ll be updating periodically with these are I learn more about Go and it’s deeper functionality.
I’ve also fallen in love with Go. It is a beautiful language to code in, and I am definitely going to do more projects in it. If you find this project useful, or feel you can improve some of the functionality, feel to to submit a pull request. The code to this project lives on Github.