This article finishes the series on building a continuous deployment environment using Python and Django.
- Starting Your First Django Project
- Testing and Django
- Mock and Coverage
- Using Fabric for Painless Scripting
- Using Celery to handle Asynchronous Processes
- Deployment/Monitoring Strategies
If you have been following along, hopefully you're already on your way to building a continuous deployment environment. The final touch is to setup a deployment and monitoring strategy. You can use a variety of services for this, such as Hudson, Continuum, CruiseControl, etc. At Votizen.com we decided to use CruiseControl, but regardless of the service used, it should manage your deployment process, complain loudly when deployment fails, and regularly monitor the health of your servers.
Getting ready
This article will discuss how I setup CruiseControl (CC) at Votizen. CC will require JAVA 1.6, so make sure you have it installed before you start. Begin by installing CC from http://cruisecontrol.sourceforge.net/ (this example will install it to your home directory, but it can be installed whereever you like it):
cd ~ wget DIRECT_LINK_HERE cruisecontrol-bin-2.8.4.zip unzip cruisecontrol*.zip rm cruisecontrol*.zip ln -s ~/cruisecontrol-bin-2.8.4 ~/cruisecontrol
CC has really good remote monitoring for CSV and SVN, but for GIT it requires a local clone of a repository to monitor for chanages. Clone your repository into the CC projects
directory:
cd ~/cruisecontrol/projects/ git clone {YOUR_REPOSITORY_HERE}
Use whatever service manager you like. If you like init.d
, here is the script Votizen uses CruiseControl Init.d. Put it in /etc/init.d/
:
sudo cp {PATH_TO_SCRIPT} /etc/init.d/cruisecontrol sudo chmod a+x /etc/init.d/cruisecontrol
Lastly, if 'java' on your machine doesn't point to JAVA 1.6, then replace java $CC_OPTS
with {PATH_TO_JAVA}/java $CC_OPTS
in ~/cruisecontrol/cruisecontrol.sh
. Now, you can start, stop, and restart CC with the installed init.d command:
sudo /etc/init.d/cruisecontrol start sudo /etc/init.d/cruisecontrol stop sudo /etc/init.d/cruisecontrol restart
How to do it…
Once you have the CC environment setup, you need to configure it for your projet. Here is a sample configuration ~/cruisecontrol/config.xml
:
<cruisecontrol> <property name="buildmaster.email" value="{YOUR_EMAIL}"/> <property name="buildmaster.name" value="{YOUR_NAME}"/> <property name="buildmaster.returnaddress" value="{YOUR_EMAIL}"/> <property name="mail.host" value="{YOUR_HOST_NAME}"/> <property name="mail.name" value="{YOUR_USER_NAME}"/> <property name="mail.pass" value="{YOUR_PASSWORD}"/> <property name="pathToProject" value="projects/{YOUR_REPO_NAME}" /> <project name="{PROJECT_NAME}" buildafterfailed="false"> <property name="run_scripts" value="${pathToProject}/cruisecontrol/{PROJECT_NAME}.xml"/> <listeners> <currentbuildstatuslistener file="logs/${project.name}/buildstatus.txt"/> </listeners> <!-- Bootstrappers are run every time the build runs, *before* the modification checks --> <bootstrappers> <antbootstrapper anthome="apache-ant-1.7.0" buildfile="${run_scripts}" target="clean" /> </bootstrappers> <!-- Defines where cruise looks for changes, to decide whether to run the build --> <modificationset quietperiod="30"> <git LocalWorkingCopy="${pathToProject}"/> </modificationset> <!-- Configures the actual build loop, how often and which build file/target --> <schedule interval="300"> <exec workingdir="${pathToProject}/cruisecontrol/" command="./deploy.sh" timeout="600" /> </schedule> <!-- directory to write build logs to --> <log dir="logs/${project.name}"/> <!-- Publishers are run *after* a build completes --> <publishers> <htmlemail returnaddress="${buildmaster.returnaddress}" mailhost="${mail.host}" username="${mail.name}" password="${mail.pass}" usessl="True" subjectprefix="[CruiseControl]"> <always address="${buildmaster.email}"/> <failure address="${buildmaster.email}"/> </htmlemail> </publishers> </project> <project name="monitorStatus" buildafterfailed="true"> <listeners> <currentbuildstatuslistener file="logs/${project.name}/buildstatus.txt"/> </listeners> <!-- Defines where cruise looks for changes, to decide whether to run the build --> <modificationset quietperiod="30"> <alwaysbuild/> </modificationset> <!-- Configures the actual build loop, how often and which build file/target --> <schedule interval="300"> <exec workingdir="${pathToProject}/cruisecontrol/" command="./monitorServer.sh" timeout="20" /> </schedule> <!-- directory to write build logs to --> <log dir="logs/${project.name}"/> <!-- Publishers are run *after* a build completes --> <publishers> <htmlemail returnaddress="${buildmaster.returnaddress}" mailhost="${mail.host}" username="${mail.name}" password="${mail.pass}" usessl="True" subjectprefix="[CruiseControl]"> <always address="${buildmaster.email}"/> <failure address="${buildmaster.email}"/> </htmlemail> </publishers> </project> <project name="pollGit" buildafterfailed="true"> <listeners> <currentbuildstatuslistener file="logs/${project.name}/buildstatus.txt"/> </listeners> <!-- Defines where cruise looks for changes, to decide whether to run the build --> <modificationset quietperiod="30"> <alwaysbuild/> </modificationset> <!-- Configures the actual build loop, how often and which build file/target --> <schedule interval="300"> <exec workingdir="${pathToProject}/cruisecontrol/" command="./updateGitRepos.sh" timeout="60" /> </schedule> <!-- directory to write build logs to --> <log dir="logs/${project.name}"/> </project> </cruisecontrol>
Replace all the {UPPER_CASE} values with your own. The values prefaced with a $
are variables that will be used by CC. In your project's repository create a cruisecontrol/
directory and add the following files:
deploy.sh monitorServer.sh updateGitRepos.sh
If any of the shell scripts print to stderr, then CC will treat it as a failure, otherwise it will be considered successful.
Here is a simple monitorServer.sh
that executes a fab script for monitoring the web server and related services (assumes there is a fabfile.py with a monitor_server task defined in the root of your repo):
#!/bin/bash cd ../ && fab prod monitor_server
Here is a simple updateGitRepos.sh
that simply updates the clone of your git repo:
#!/bin/bash cd ../ && git pull origin master
Here is a simple deploy.sh
that deploys your project:
#!/bin/bash cd ../ && fab prod deploy
Make sure all the files have +x
permissions.
How it works…
When CC starts it parses the config.xml
file and determines the projects to monitor. The modificationset
tag determines when the CC project task should execute. Two of the projects in the XML above use the alwaysbuild
directive and one the git
directive, which points to the local git clone and monitors for changes. The schedule
tag indicates the frequency that the modificationset
should be evaluated and what to do when the modificationset
returns true. The exec
comman can be used inside the schedule
tag to run a shell script. The publishers
tag is used to indicate who should be notified of failures and success. The remaining tags are used internally by CC and shouldn't be modified, unless you know what you are doing.
A complete list of all the tags is available at the Configuration Reference.
The cruisecontrol
directory added to the main repo, makes maintaining CC for that project easy and under version control. Additionally, it creates a template that is easy to clone for additional projects. I use fabric to do any real scripting, but you could use bash or any other scripting tool to monitor and deploy servers.
Running CC with this configuration causes CC to do three things every five minutes: pull from the repo to keep the local clone up-to-date, check the local repo for changes and deploy when a change is detected, and execute a script that monitors the servers for errors/problems.
The monitoring strategy Votizen uses is to expose a protected web URL that prints out each service and its status. This task hits that URL and fails if failed
appears anywhere on that webpage. Ultimately, you will want to monitor logs for errors, and this task could be improved for that, or you could use another service like sentry.
That wraps up our discussion on using CC to deploy a project. I hope this was not too confusing. Leave comments below, if you have questions.
There's more…
There is a great tool, CCMenu, that is available for most operating systems making the monitoring of CC even easier. I use this exclusively to keep track of status of my servers.
Here are some useful links: