Load testing on Heroku

There are plenty of load testing tools out there, like ab, siege, locust, gatling, vegeta and many more, just google it. With no particular objective i though it would be interesting if i can use Heroku as a medium to perform load testing for some of my projects. In fact, you should be doing some sort of testing to determine bottlenecks in your systems.

Vegeta looked promising so i decided that i'd give it a try. It has a pretty basic CLI, could be deployed on multiple hosts and has ability to generate multiple report types (including hit chart). Not to mention that it's written in Go and ships as a single binary.

First of all, why Heroku? You can host apps for free with the an entry-tier plan, 10s' of apps, not just one. While scaling is not free, we can have another way to scale out by deploying more apps from the same codebase. Free apps must sleep for some time, but we don't really care about that since those apps are going to be discarded within an hour.

heroku-vegeta

As a result of my experiments, i came up with a simple repository that contains all necessary dependencies and scripts to automate load testing. You can clone it from Github. Check the README file for instructions to get it up and running. Basically, once you done with configuration there are a few scripts available:

./script/setup    - Create heroku applications (from config.json) and deploy the app.
./script/run      - Execute the load test.
./script/teardown - Delete all created heroku applicaitons.
./script/restart  - Restart all apps in case if you need to cancel the test.

Lets go over how this whole operation works.

Vegeta tool does not provide any way to communicate with it using HTTP interface. For the load test to work correctly we need to have a lightweight api shim that will accept incoming requests and handle process spawning in the background. I ended up creating a simple go app that exposes endpoints to start the load test, check on status and download result files. See main.go for reference.

To simplify repository structure and app creation process i decided to drop the idea of making a custom buildpack. Instead, vegeta binary is just embedded into the repo so that api app can just spawn it.

The actual vegeta process runs in the background, which is not limited in time as long as the dyno stays up (for longer tests). Availability of the dyno is maintained by the runner script, which basically pings the app endpoint until "done" state is returned. After load testing is complete, runner script will automatically download binary result files from each app that participated in the test. Finally, results are fed into vegeta command line tool locally on your machine and the html file with chart and other report information is generated.

Example report page:

Using vegeta on Heroku turned out to be an interesting experiment. While my solution is not exactly appropriate for the platform, it is definitely good enough for random load testing and prototyping. Keep in mind that for "best results" your app must be in AWS us-east zone because majority of Heroku's infra is hosted there.

Feel free to drop me a line if you have any questions or ideas.