SaaS CI For Startups – Finding the right fit

The Premise

As part of developing Foover we needed a CI environment.  We are using EC2 for our server, but didn’t want anything else running on production that would consume resources.

I also wanted professional deployment notifications to multiple stakeholders, and deploys on each push without thinking about it, with a high degree of confidence that, if the tests pass, the code is up in da cloud within a few minutes.

Obviously you need a codebase with a decent suite of tests first =)

In this post I will review Magnum CI, CodeshipShippable and Circle CI with their pros and the cons.

First, let me summarise why you should consider SaaS CI, and what tech stack my codebase uses.

The Benefits

If you are a startup, you probably know a bit about setting up a Linux server, SSH, deploying artifacts and restarting services.  But that’s not what is gonna earn you the $$$. If you consider your time valuable, let someone else manage this infrastructure for you. Especially if you have multiple projects.

Also if you have multiple developers, especially across multiple time zones, accessible CI is important to see what happened to the tests across builds, and it provides you with good transparency and a separate environment in which everything works.  Doing this research helped me uncover some environment variables I had just assumed on my development machine and EC2 server which is great, now it’s documented and is infrastructure as code.

One difference in our repository worth mentioning is that the Rails root of the project was in a sub-directory of the Git repository.  I always set Rails projects up like this so you can put other stuff in here.  You could also use subtrees or submodules but this I found was a good test to the flexibility of configuring the SaaS CI build scripts.

The Tech Stack

We have a Rails-based API using MongoDB, deployment with Capistrano checked into Git. The repository is hosted in Bitbucket but all the CI services I review below support Github as well.

Tip: Deploying with Capistrano to EC2

When I was deploying manually from my machine, i’d tell Capistrano to use my personal EC2 SSH Private PEM key which was generated with the EC2 instance.  I’d suggest not putting this key in your codebase, this should be kept separate from prying eyes since it’s the keys to production… literally!

What I did is I took the SSH public key generated for the SaaS CI build and put it in ~/.ssh/authorized_keys on my EC2 server.  This way it is trivial to revoke access and do passwordless deployments.  To specify the SSH private key to use when on SaaS CI build slaves, I found it safe to assume it is in ~/.ssh/id_rsa just have Capistrano reference this when it runs the deploy task with the ssh_options[:keys] variable (Capistrano v2).

NOTE: Time to build in the below reviews was determined by 3 consecutive green builds (after an initial green build) and averaged to yield the mean build time.


First impressions: It is free and seems fairly new, so the interface didn’t feel super-slick but it was clean and functional.

Magnum CI Screenshot

The UI and builds felt slow, it seemed each time it started a new VM from scratch (which may be the price you pay on a free tier) but it increased the built time incredibly. It had to install all the ruby gems each time which probably took at least 2 minutes.

It’s simple to setup a build, you add your build commands to the UI and authorise Magnum CI to access your repository via SSH keys, and add a web-hook into Bitbucket so it doesn’t need to poll for changes (it gets notified – push not pull).

It abstracts away installing RVM – you select your Ruby version.  There isn’t every version possible, but there’s a common list: 1.9.2, 1.9.3, 2.0.0 and 2.1.0 (at the time of writing).

The UI does give a nice scrolling window like Bamboo / Jenkins to show build console status.  It may sound simple but I feel like this is an important feature.

These guys have integrations with Hipchat, Slack and IRC.

One feature I thought could be very useful (but should not be abused) is to have your CI platform *not* perform a build if you enter [ci-skip] or [ci skip] in your commit message.

Time to deploy: 6:28s

Cost: Free (no parallel builds)

Cost Rating: 4.8 / 5

Support Rating: 3.8 / 5

Speed Rating: 1.5 / 5

Overall Rating: 3.4 / 5

First impressions: It has a very professional interface, leads you via prompts like “I see you changed your build script – would you like to re-run your last build?” – Why yes, J.A.R.V.I.S, I would!  Awesome Ux – reminds me of Heroku. The UI feels fast and responsive.

Codeship Screenshot

Codeship has an option to SSH onto the slave VM to debug what is actually happening (or not happening) on the metal, which is really what I want – when the sh*&t hits the fan, I don’t want to log a support ticket, I want to see if there’s a way I can work around it and get a build out.  This is a killer feature.

They have plenty of integrations for Slack, Hipchat, Basecamp and add-ons for deployment tasks – too many to mention.

Codeship has been around for a long time, so they have great support and have dealt with most teething issues by now.  They are also highly focused on performance which is reflected in their build times and their fast UI.  They also allow [skip ci] to skip builds in commit.

The only possible gripe I can muster is that between a $0 to $49 plan there could be a mid-way plan, for $10 or $25 per month.

Time to Deploy: 1:09s

Cost: Free (100 builds per month) or $49 per month (1 concurrent build / unlimited repos)

Cost Rating: 3 / 5

Support Rating: 4.5 / 5

Speed Rating: 5 / 5

Overall Rating: 4.2 / 5

First Impressions: Having to check-in your shippable.yml (similiar to a .travis.yml file) to run builds for a CI service the first few times is painpain pain.  I did at least 15 commit and pray git pushes until I got the right configuration for my project which ran the tests and passed the build.  This litters your commit history and is a painful exercise.  The upside is you have audit tracking on your build tasks which is a good thing, but I feel like I just wanted to dive into the slave VM, adjust the file in-line (until it worked) then back-port it to my codebase.

Shippable Screenshot

The ability to edit that file in-line and re-run as an override function would be a killer feature here.  I felt like I had lost control of the CI, since there was no SSH function either.  This is the feeling I wanted to avoid. Also seemed to run 4 builds whenever I committed code to one branch, couldn’t figure out why, I only had 3 branches, in any case.  Strange.

It also popped up with a Success message when a build was queued, which is a bit misleading.  The UI feels like a beta at times with uncompiled angular.js variable names and the like 😦

I had serious problems with Nokogiri 1.5.x.x due to native dependencies – in this case I upgraded to and didn’t have any issues.  I didn’t have this problem on any other CI in this blog with the same configuration.

The worst part was all my builds were labelled as Success even though Capistrano rolled back from the deployment!  Not good.

Once it was working, I was happy enough although it seemed slow like Magnum CI.  I just didn’t have enough confidence based on my experience which for me is a #fail

Time to Deploy: 3:56s

Cost: Free (no concurrent builds / 5 private repos) or $10 per month (2 concurrent builds / 10 private repos)

Cost Rating: 4.7 / 5

Support Rating: 4.5 / 5

Speed Rating: 3 / 5

Overall Rating: 3 / 5   Took 1 point off due to Capistrano rollbacks being labelled ‘success’ and other bugs / issues mentioned.


First Impressions: After having defined my shippable.yml I cloned it into circle.yml and that pretty much gave me the configuration I needed to get started.  Same gripes as Shippable re: YAML file for configuration – commit and pray sucks.

Circle CI Screenshot

NOTE:  These guys do not support Bitbucket.  For me this is a #fail since their status link for adding Bitbucket support is over 6 months old, with lots of +1 comments.

I guess Circle CI is Travis CI for private repositories.  It gets great points for the Ux feels very spacious and clean, it takes a lot of time and effort to get it this slick.  It almost feels too minimalist though.

They allow you to SSH to the slave VM, but if you ask for this option it leaves the connection open for 30 minutes, so it blocks your builds.  You can close it manually but it’s an extra step and was not intuitive.

You can generate API tokens for doing your own integration with this CI – quite an awesome feature (if you want it).  One specific gripe about their web application – their main assets file is 2MB of uncompressed javascript – 739kb compressed (gzip).  As a web-dev during the sunlight hours, this is not a good sign.

Since my project was in Bitbucket I couldn’t proceed much further, I tinkered around using a Github repository I maintain to test it conceptually but cannot comment on a comparable build time.

In term of their build output, it was great – showing details then folding, and turning green or red for each step which is great use of colours. Probably the most advanced UI in this aspect – competitors take note!


I feel like this deserves it’s own segment since, for any system you rely on for technology or profit, you need to have confidence that it will do it’s job 95% of the time.  The other 5% of the time, you want to have the utmost confidence in the human support factor.

This is based on my own prejudices but I did contact support for each CI service to get some understanding.  Your mileage may vary =)

Magnum CI – Quick and helpful response.

Codeship – Bloody great.  CEO and CTO are quite available (Skype / Twitter) and they seem very hands-on and focused on improving and keeping customers happy.  While finishing editing of this blog post, Codeship put out an announcement extending their Free tier from 50 monthly builds to 100, which is awesome.

Shippable (via issues) – Good – troubleshooted an issue I had and took 2 minor bug reports.

Circle CI – No reply (from their hipchat channel twice within 30 minutes… disappointing.) I did however, get a thoughtful response a day later from marketing about cancelling my membership.

Review Notes

Trying your tech stack on several providers is an ideal way to discover environment variables, apt-get native packages or unspecified version dependencies you need for your deployed code to run.

One realisation I have come to is that shipping your build file with your code, although it works well once you have a stable pipeline / environment, to get up and running can be a slow process.  Ideally you could override this file or SSH in and tweak settings to find the right configuration without multiple commits – you could easily waste an hour or two doing this with an existing project if build times are > 5 minutes.

After setting up the deployment pipeline, I also found something missing.

Sure, I can run tests before deploying, then deploy, then regression test.  But my CI should do more than that.  It should keep running tests, even in the absence of commits or deployments, to ensure the environment (and any dependencies) are fully functional.

Call them functional tests, smoke tests, contract tests, health checks – i’ve found these core acceptance tests alert me to failures in an online system better than more specific tools (eg. New Relic or Nagios). Running these tests regularly in-between deployments on a running system will uncover any fragility in your production system and ensure your tests are rock-solid.  Every CI system I have ever setup I have ended up doing this – not every functional test will run after every commit.

Editors Choice

All tested CI services support Bitbucket (except Circle CI) and Github and all support web-hooks (which is far more efficient than polling) and Magnum CI and Codeship support various add-ons for integration and notifications.

I have picked Codeship as my preferred SaaS CI service.  For me, currently it will be free, if I use them for another month or so I will start donating if I am still on the free tier, however I encourage you to evaluate Magnum CI and Shippable as well – your tech stack may be different and you may get a different experience.

Thanks for reading!


Scala, come out and Play

Last week I started work on a Play 2.1 Scala project, which is a nice departure from Spring and Hibernate, so it’s a good time to share some first thoughts on the differences between the old traditional Java webapp, Grails, Rails and Play 2.1 with Scala.

First off, installing Play is easy as pie. Most modern frameworks can create a skeleton app quite easily, however the integration with sbt feels a bit awkward but is still much better than good ol’ Maven. Re-compiling scala files can seem to take awhile, but running tests are nice and quick in IntelliJ.

IntelliJ is a great choice for Scala/Play projects, as v12.1 has excellent auto-completion, syntax highlighting and click-through method links and good test integration. The contextual help is quite good at telling you what’s wrong – this seems to be a strength of the Scala compiler. However, for whatever reason we spent some time trying to get IntelliJ to register our Play app as such when importing the project. Possibly because the /app folder wasn’t off the root of the repository, (we have it in /play/app) but even after running play idea to generate the necessary project files, IntelliJ refused to import it as a Play app.

After upgrading the Play plugin, installing a Scala plugin and ensuring Scala, Play and Java were all in the path and had appropriate %_HOME directories set, it still wouldn’t give in. Creating a new Play app from IntelliJ seemed to bootstrap it with the necessary vision it needed, and after doing this, it imported our existing app no worries.

Be careful as lots of documentation on the web is disinformation, since Play 1.0 is different, and Play 2.1 have made some refinements from previous revisions. Also the fact that we are using Specs2 means you have to sanity check all blog posts and stackoverflows before assuming they are correct for you.

Testing in Scala is beautiful and clean, feels natural and all the crazy syntax of Mockito in Java seems to have been repaired. Specs2 reminds me more of RSpec or Jasmine Specs, which make it easier to read and group similiar tests.

Scala as a language feels quite wierd. Ruby and underscore.js already have nice functional methods like reduce, map, filter which are pretty straight-forward. Things I like are the Option class and the case matching aspects. Like any new framework (and language) you can get stumped and need a good reference on how to do things the idiomatic way.

Scala templates have been carefully designed with a set of opinions and good conventions (such as auto-escaping and immutable parameters, and the ability to create templates and helpers which have multiple closures as parameters changes the way you can call templates.

As an example:

To get all the goodness I am used to in other projects, I created a @toggle macro, which takes in a single String parameter, followed by 2 closures (one to render when enabled and, you guess it, the other to render when disabled).

Place the following in the directory views/helpers/toggle.scala.html as a helper template:

@(toggle: String)(enabledContent: Any)(disabledContent: Any)

@import play.api._

@defining(Play.current.configuration.getBoolean("toggles." + toggle)) { toggleValue =>

  @toggleValue match {
    case Some(toggleEnabled) => {

      @if(toggleEnabled) {
      } else {

    case None => {
      <h1> Error invalid toggle specified: @toggle</h1>

Then simply set a value in conf/application.conf ie.


Where you want to switch the behaviour (in a Scala template), do the following:

  @toggle("search_box") { 
    <label>Search:<input type="text" id="search"></label>
  } {
    <div>Search is currently disabled</div>

Additionally, Play allows you to toggle the value in the application.conf file dynamically, no restart needed =)


Tips to increase your dev-fu (java devs mainly):


  • switch to IntelliJ / RubyMine – fast searching (maybe notepad++ / grep / sublime is faster?) and if you are doing CSS / Javascript development I find it’s the most productive IDE for me.
  • spend 10 minutes every day asking yourself how to improve your IDE / environment… what do you spend the most time doing each day that is repetitive?
  • learn REGEX. It’s everywhere… if you have to replace/modify more than 2-3 LOC, use a find/replace. If that doesn’t fit, REGEX is your friend. Also learn SED once you’ve learned Linux 🙂
  • printout that keymap of your IDE… circle shortcuts you wanna learn then make a concious effort to learn them.  Test yourself the next day / week by trying to recall it with your muscle memory. Practising on different computers / keyboards reinforces it… especially mac vs. non-mac keyboards 🙂
  • Pair!!! I have learned the most shortcuts / configuration options from pairing or looking at others IDEs and asking questions.
  • Consider Chrome your browser of choice.  I was Firefox for a long time, but Chrome seems quicker and i’m more productive on it now.
  • if you are a frontend dev, research a bit on browser plugins for helpful tools like screen resizing, URL testing, JSON parsing, CSS reloading etc.
  • learn Linux, learn the shell and the command line…
  • learn Git (if you haven’t already), setup a few repos, patch some code on – it’s good for the soul 🙂