Opsworks and HAProxy

At Integrate.io, with much of our infrastructure on AWS, we try to make use of the various AWS services available to us. One of these is Amazon Opsworks (http://docs.aws.amazon.com/opsworks/latest/userguide/welcome.html). Amazon Opsworks is basically a Chef-as-a-Service tool for DevOps automation. You can use most of the Chef recipes available on Opscode, and it just makes everything easier to setup. Amazon Opsworks only runs Chef Solo and it’s a bit limited (i.e., no secure databags), but for a small team, this works. In this article, we are going to describe how we use HAProxy with our Layers (http://docs.aws.amazon.com/opsworks/latest/userguide/workinglayers-custom.html).

Replacing ELB with HAProxy

Due to the way ELB functions and how it is built for HTTP, we decided to stop using ELB for our needs. One of the things we have noticed was that, if there were no available instances the ELB is load-balancing for, the ELB would still respond to requests and “answer” them as though everything was fine. This wasn’t a behavior we could work with. With our HAProxy configuration, we changed this behavior so that it closes the port on connections if no cluster was responding. This way, the clients would not try to send data unknowingly, thinking that the cluster was still available. With this change, we increased our reliability. We are now able to provide more assurances that our customer’s data is in sync. By using Layers as clusters, and with a bit of regex, we created custom recipes based on the haproxy cookbook (https://supermarket.chef.io/cookbooks/haproxy) to setup the haproxy configuration (along with the rest of the instance setup). With the use of the shortname, we can associate custom HAProxy layers with the application layer. By using the configure lifecycle event, we are able to re-configure the haproxy configuration with any update to the stack (ie. new instances launched, etc). We could have used the Opsworks HAProxy cookbook, but since the way we use Layers is a bit different than what it was intended for, we decided to use custom recipes for our Stacks.