Deployment/Monitoring Strategies

This article finishes the series on building a continuous deployment environment using Python and Django.

If you have been following along, hopefully you're already on your way to building a continuous deployment environment. The final touch is to setup a deployment and monitoring strategy. You can use a variety of services for this, such as Hudson, Continuum, CruiseControl, etc. At Votizen.com we decided to use CruiseControl, but regardless of the service used, it should manage your deployment process, complain loudly when deployment fails, and regularly monitor the health of your servers.

Getting ready

This article will discuss how I setup CruiseControl (CC) at Votizen. CC will require JAVA 1.6, so make sure you have it installed before you start. Begin by installing CC from http://cruisecontrol.sourceforge.net/ (this example will install it to your home directory, but it can be installed whereever you like it):

cd ~
wget DIRECT_LINK_HERE  cruisecontrol-bin-2.8.4.zip
unzip cruisecontrol*.zip
rm cruisecontrol*.zip
ln -s ~/cruisecontrol-bin-2.8.4 ~/cruisecontrol

CC has really good remote monitoring for CSV and SVN, but for GIT it requires a local clone of a repository to monitor for chanages. Clone your repository into the CC projects directory:

cd ~/cruisecontrol/projects/
git clone {YOUR_REPOSITORY_HERE}

Use whatever service manager you like. If you like init.d, here is the script Votizen uses CruiseControl Init.d. Put it in /etc/init.d/:

sudo cp {PATH_TO_SCRIPT} /etc/init.d/cruisecontrol
sudo chmod a+x /etc/init.d/cruisecontrol

Lastly, if 'java' on your machine doesn't point to JAVA 1.6, then replace java $CC_OPTS with {PATH_TO_JAVA}/java $CC_OPTS in ~/cruisecontrol/cruisecontrol.sh. Now, you can start, stop, and restart CC with the installed init.d command:

sudo /etc/init.d/cruisecontrol start
sudo /etc/init.d/cruisecontrol stop
sudo /etc/init.d/cruisecontrol restart

How to do it…

Once you have the CC environment setup, you need to configure it for your projet. Here is a sample configuration ~/cruisecontrol/config.xml:

<cruisecontrol>
    <property name="buildmaster.email" value="{YOUR_EMAIL}"/>
    <property name="buildmaster.name" value="{YOUR_NAME}"/>
    <property name="buildmaster.returnaddress" value="{YOUR_EMAIL}"/>
    
    <property name="mail.host" value="{YOUR_HOST_NAME}"/>
    <property name="mail.name" value="{YOUR_USER_NAME}"/>
    <property name="mail.pass" value="{YOUR_PASSWORD}"/>

	<property name="pathToProject" value="projects/{YOUR_REPO_NAME}" />

	<project name="{PROJECT_NAME}" buildafterfailed="false">

		<property name="run_scripts" value="${pathToProject}/cruisecontrol/{PROJECT_NAME}.xml"/>

		 <listeners>
			<currentbuildstatuslistener file="logs/${project.name}/buildstatus.txt"/>
		 </listeners>

		<!-- Bootstrappers are run every time the build runs,
			*before* the modification checks -->
		<bootstrappers>
			<antbootstrapper anthome="apache-ant-1.7.0" buildfile="${run_scripts}" target="clean" />
		</bootstrappers>

		<!-- Defines where cruise looks for changes, to decide whether to run the build -->
		<modificationset quietperiod="30">
			<git LocalWorkingCopy="${pathToProject}"/>
		</modificationset>

		<!-- Configures the actual build loop, how often and which build file/target -->
		<schedule interval="300">
			<exec workingdir="${pathToProject}/cruisecontrol/" command="./deploy.sh" timeout="600" />
		</schedule>

		<!-- directory to write build logs to -->
		<log dir="logs/${project.name}"/>

		<!-- Publishers are run *after* a build completes -->
		<publishers>
            <htmlemail returnaddress="${buildmaster.returnaddress}"
                mailhost="${mail.host}"
                username="${mail.name}"
                password="${mail.pass}"
                usessl="True"
                subjectprefix="[CruiseControl]">
                <always address="${buildmaster.email}"/>
				<failure address="${buildmaster.email}"/>
			</htmlemail>
		</publishers>
	</project>

	<project name="monitorStatus" buildafterfailed="true">
		 <listeners>
			<currentbuildstatuslistener file="logs/${project.name}/buildstatus.txt"/>
		 </listeners>

		<!-- Defines where cruise looks for changes, to decide whether to run the build -->
		<modificationset quietperiod="30">
			<alwaysbuild/>
		</modificationset>

		<!-- Configures the actual build loop, how often and which build file/target -->
		<schedule interval="300">
			<exec workingdir="${pathToProject}/cruisecontrol/" command="./monitorServer.sh" timeout="20" />
		</schedule>

		<!-- directory to write build logs to -->
		<log dir="logs/${project.name}"/>

		<!-- Publishers are run *after* a build completes -->
		<publishers>
            <htmlemail returnaddress="${buildmaster.returnaddress}"
                mailhost="${mail.host}"
                username="${mail.name}"
                password="${mail.pass}"
                usessl="True"
                subjectprefix="[CruiseControl]">
                <always address="${buildmaster.email}"/>
				<failure address="${buildmaster.email}"/>
			</htmlemail>
		</publishers>

	</project>

	<project name="pollGit" buildafterfailed="true">
		 <listeners>
			<currentbuildstatuslistener file="logs/${project.name}/buildstatus.txt"/>
		 </listeners>

		<!-- Defines where cruise looks for changes, to decide whether to run the build -->
		<modificationset quietperiod="30">
			<alwaysbuild/>
		</modificationset>

		<!-- Configures the actual build loop, how often and which build file/target -->
		<schedule interval="300">
			<exec workingdir="${pathToProject}/cruisecontrol/" command="./updateGitRepos.sh" timeout="60" />
		</schedule>

		<!-- directory to write build logs to -->
		<log dir="logs/${project.name}"/>

	</project>

</cruisecontrol>

Replace all the {UPPER_CASE} values with your own. The values prefaced with a $ are variables that will be used by CC. In your project's repository create a cruisecontrol/ directory and add the following files:

deploy.sh
monitorServer.sh
updateGitRepos.sh

If any of the shell scripts print to stderr, then CC will treat it as a failure, otherwise it will be considered successful.

Here is a simple monitorServer.sh that executes a fab script for monitoring the web server and related services (assumes there is a fabfile.py with a monitor_server task defined in the root of your repo):

#!/bin/bash
cd ../ && fab prod monitor_server

Here is a simple updateGitRepos.sh that simply updates the clone of your git repo:

#!/bin/bash
cd ../ && git pull origin master

Here is a simple deploy.sh that deploys your project:

#!/bin/bash
cd ../ && fab prod deploy

Make sure all the files have +x permissions.

How it works…

When CC starts it parses the config.xml file and determines the projects to monitor. The modificationset tag determines when the CC project task should execute. Two of the projects in the XML above use the alwaysbuild directive and one the git directive, which points to the local git clone and monitors for changes. The schedule tag indicates the frequency that the modificationset should be evaluated and what to do when the modificationset returns true. The exec comman can be used inside the schedule tag to run a shell script. The publishers tag is used to indicate who should be notified of failures and success. The remaining tags are used internally by CC and shouldn't be modified, unless you know what you are doing.

A complete list of all the tags is available at the Configuration Reference.

The cruisecontrol directory added to the main repo, makes maintaining CC for that project easy and under version control. Additionally, it creates a template that is easy to clone for additional projects. I use fabric to do any real scripting, but you could use bash or any other scripting tool to monitor and deploy servers.

Running CC with this configuration causes CC to do three things every five minutes: pull from the repo to keep the local clone up-to-date, check the local repo for changes and deploy when a change is detected, and execute a script that monitors the servers for errors/problems.

The monitoring strategy Votizen uses is to expose a protected web URL that prints out each service and its status. This task hits that URL and fails if failed appears anywhere on that webpage. Ultimately, you will want to monitor logs for errors, and this task could be improved for that, or you could use another service like sentry.

That wraps up our discussion on using CC to deploy a project. I hope this was not too confusing. Leave comments below, if you have questions.

There's more…

There is a great tool, CCMenu, that is available for most operating systems making the monitoring of CC even easier. I use this exclusively to keep track of status of my servers.

Here are some useful links: