In recent articles I covered how I've built a Continuous Delivery pipeline for my blog. These articles talk about using Docker to build a container for my blog, using Travis CI to test and build that container, and finally using a Masterless SaltStack configuration to deploy the blog. Once setup, this pipeline enables me to publish new posts by simply managing them within a GitHub repository.
The nice thing about this setup is that not only are blog posts managed hands-free.
Over the past two articles I've described building a Continuous Delivery pipeline for my blog (the one you are currently reading). The first article covered packaging the blog into a Docker container and the second covered using Travis CI to build the Docker image and perform automated testing against it.
While the first two articles covered quite a bit of the CD pipeline there is one piece missing; automating deployment. While there are many infrastructure and application tools for automated deployments I've chosen to use Saltstack.
Self healing infrastructure is something that has always piqued my interested. The first iteration of self healing infrastructure that I came across was the Solaris Service Management Facility aka “SMF”. SMF would restart services if they crashed due to hardware errors or general errors outside of the service itself.
For today's article we are going to explore another way of creating a self healing environment; going beyond restarting failed services. In today's article we are going to take a snippet of code that connects to a database service and give that application not only the ability to reconnect during database failure but also give it the ability to automatically resolve the database issues.
Recently I have been looking for ways to allow external tools and services to perform corrective actions across my infrastructure automatically. As an example, I want to allow a monitoring tool to monitor my nginx availability and if for whatever reason nginx is down I want that monitoring tool/service to do something to fix it.
While I was looking at how to implement this, I remembered that SaltStack has an API and that API can provide exactly the functionality I wanted.
Configuration management and automation tools like SaltStack are great, they allow us to deploy a configuration change to thousands of servers with out much effort. However, while these tools are powerful and give us greater control of our environment they can also be dangerous. Since you can roll out a configuration change to all of your servers at once, it is easy for that change to break all of your servers at once.
Today's article is an item I covered briefly during my presentation at SaltConf 2014 (which was a pretty awesome conference by the way). One of the lesser known features of SaltStack is the ability to configure multiple master servers. Having an additional master server allows for some extra redundancy as well as capacity for large implementations. While I covered the benefits of having an additional master server in my presentation I didn't cover in full detail how to set this up, today I will cover the details of configuring multiple salt masters.
A few weeks back I wrote an article Getting started with SaltStack; that article covered Configuration and Package Automation with Saltstack. In Today's article I am going to cover SaltStack's Remote Execution abilities, a feature that I feel Saltstack has implemented better than other automation tools.
Running a command in a State If you remember from the previous article SaltStack's states are permanent configurations. Adding a command in a Salt state is used when you want to have a command that is run after provisioning a server, run every time Salt manages the state of the system or run when certain conditions are true.