This past Monday 29 February, one of my favourite companies in the fintech space Mondo launched their public funding round on the crowd funding platform CrowdCube. This event has been built up over the last few months and there was a significant level of hype around it.
The launch, however, was by all accounts a failure. This post aims to break down the reasons for failure, and hopefully give some insight into preventative measures.
Proposition
Mondo decided to raise 500k GBP (and then increased this to 1 million GBP) through Crowdcube, an investment crowdfunding platform. This meant that any person would be able to invest in Mondo and in turn receive equity. The investment amounts ranged from 10GBP to 1 000GBP.
The idea was great, and as Monday rolled around and the mailers were sent out things started to go south.
Crowdcube did not scale
Within minutes of the campaign going live the Crowdcube site went down. Handling traffic has historically been difficult, but these days there are hundreds of ways to handle scaling issues and testing load in preparation for a big launch.
The Crowdcube platform looks to be predominantly PHP and MySQL with some microservices thrown in. In my experience, PHP and MySQL can be tested and scaled fairly easily.
How much was too much
The numbers of visitors were fair, with a few thousand pre-registrants and a mailing list of ~60k being notified. This might lead to around ~10k concurrents, which is far from any upper limit for most consumer facing sites.
[Note: I remember reading this somewhere during the commotion, but cannot find it now. I will update if I do]
Load testing
There are dozens of load testing frameworks for many different use cases, so it’s surprising that a site would go down from load alone. It should be standard practice to be able to stand up during heavy load testing in preparation for real world usage. This is especially true when you deal with people’s money.
Caching
Cahing is always a good first answer for getting a solution to scale. In my experience, good caching and good cache invalidation can help a site handle traffic to a few orders of magnitude.
Database caching
I’m a big fan of caching database result sets. Depending on the freshness needed, these can be stored for varied lengths with the longer being the more efficient. The approach I would take:
- Query database
- Read database result set from cache (memcache, redis, etc)
- This cache has a predefined expiry
- Update database
- Database writes are done straight to the DB
- Cache is checked for write time. If write time is less than expiry time, the cache is refreshed with new result set
- Cache is only ever refreshed on database update
The above “cache first” solution has saved me loads of hassle when trying to write an efficient, high load, database-driven system.
Varnish
Varnish is a life saver. We went from 40% CPU load across four servers, to a quarter load and number of servers after installing and configuring Varnish.
There are various ways to invalidate some aspects of the caching, and any time put into getting Varnish to work for your particular set up is worth its weight in gold. Varnish is now at a stage where all the complexities we were dealing with years ago (SSL, Cookies, etc) are now handled far better. This alone might have been enough for keeping the system up.
Horizontal scaling
There are many solutions, paid and free, that offer horizontal scaling capabilities. MySQL can be a pain to put on more than one server, but the frontends should be trivial to swap across. This, coupled with the two methods above, should be more than enough to handle (almost) anything.
Reputation / Fallout
An unfortunate result of a site going down for a funding round is a bad reflection on both the system as well as the company raising funds. This is especially true when the company raising funds is a bank. A bank’s biggest asset is trust.
After Monday, Mondo did a really good job of communicating the next steps. What seemed to be planned was a pretty haphazard, confusing process though. I was eager to invest but the new process was confusing. When sent the mail on Thursday I was unable to register my interest for the funding round (even though I had registered my interest before, why should I register again?). When the round opened in an hour, I was none the wiser.
The result
After all is said and done, Mondo absolutely crushed it. They managed to raise 1 million GBP in 96 seconds. This shows the massive demand for their product, as well as their ability to market and advertise themselves. You can read more here.
Conclusion
I’ve been in both of the situations above and neither of them is where you want to be. I have no idea how much hustle the team from Crowdcube had to put in during this week to make sure there was no further outages.
A massive congrats to Tom, Jonas and the rest of the Mondo team.