SaaS CI For Startups – Finding the right fit

The Premise

As part of developing Foover we needed a CI environment.  We are using EC2 for our server, but didn’t want anything else running on production that would consume resources.

I also wanted professional deployment notifications to multiple stakeholders, and deploys on each push without thinking about it, with a high degree of confidence that, if the tests pass, the code is up in da cloud within a few minutes.

Obviously you need a codebase with a decent suite of tests first =)

In this post I will review Magnum CI, CodeshipShippable and Circle CI with their pros and the cons.

First, let me summarise why you should consider SaaS CI, and what tech stack my codebase uses.

The Benefits

If you are a startup, you probably know a bit about setting up a Linux server, SSH, deploying artifacts and restarting services.  But that’s not what is gonna earn you the $$$. If you consider your time valuable, let someone else manage this infrastructure for you. Especially if you have multiple projects.

Also if you have multiple developers, especially across multiple time zones, accessible CI is important to see what happened to the tests across builds, and it provides you with good transparency and a separate environment in which everything works.  Doing this research helped me uncover some environment variables I had just assumed on my development machine and EC2 server which is great, now it’s documented and is infrastructure as code.

One difference in our repository worth mentioning is that the Rails root of the project was in a sub-directory of the Git repository.  I always set Rails projects up like this so you can put other stuff in here.  You could also use subtrees or submodules but this I found was a good test to the flexibility of configuring the SaaS CI build scripts.

The Tech Stack

We have a Rails-based API using MongoDB, deployment with Capistrano checked into Git. The repository is hosted in Bitbucket but all the CI services I review below support Github as well.

Tip: Deploying with Capistrano to EC2

When I was deploying manually from my machine, i’d tell Capistrano to use my personal EC2 SSH Private PEM key which was generated with the EC2 instance.  I’d suggest not putting this key in your codebase, this should be kept separate from prying eyes since it’s the keys to production… literally!

What I did is I took the SSH public key generated for the SaaS CI build and put it in ~/.ssh/authorized_keys on my EC2 server.  This way it is trivial to revoke access and do passwordless deployments.  To specify the SSH private key to use when on SaaS CI build slaves, I found it safe to assume it is in ~/.ssh/id_rsa just have Capistrano reference this when it runs the deploy task with the ssh_options[:keys] variable (Capistrano v2).

NOTE: Time to build in the below reviews was determined by 3 consecutive green builds (after an initial green build) and averaged to yield the mean build time.

magnum-ci

First impressions: It is free and seems fairly new, so the interface didn’t feel super-slick but it was clean and functional.

Magnum CI Screenshot

The UI and builds felt slow, it seemed each time it started a new VM from scratch (which may be the price you pay on a free tier) but it increased the built time incredibly. It had to install all the ruby gems each time which probably took at least 2 minutes.

It’s simple to setup a build, you add your build commands to the UI and authorise Magnum CI to access your repository via SSH keys, and add a web-hook into Bitbucket so it doesn’t need to poll for changes (it gets notified – push not pull).

It abstracts away installing RVM – you select your Ruby version.  There isn’t every version possible, but there’s a common list: 1.9.2, 1.9.3, 2.0.0 and 2.1.0 (at the time of writing).

The UI does give a nice scrolling window like Bamboo / Jenkins to show build console status.  It may sound simple but I feel like this is an important feature.

These guys have integrations with Hipchat, Slack and IRC.

One feature I thought could be very useful (but should not be abused) is to have your CI platform *not* perform a build if you enter [ci-skip] or [ci skip] in your commit message.

Time to deploy: 6:28s

Cost: Free (no parallel builds)

Cost Rating: 4.8 / 5

Support Rating: 3.8 / 5

Speed Rating: 1.5 / 5

Overall Rating: 3.4 / 5

First impressions: It has a very professional interface, leads you via prompts like “I see you changed your build script – would you like to re-run your last build?” – Why yes, J.A.R.V.I.S, I would!  Awesome Ux – reminds me of Heroku. The UI feels fast and responsive.

Codeship Screenshot

Codeship has an option to SSH onto the slave VM to debug what is actually happening (or not happening) on the metal, which is really what I want – when the sh*&t hits the fan, I don’t want to log a support ticket, I want to see if there’s a way I can work around it and get a build out.  This is a killer feature.

They have plenty of integrations for Slack, Hipchat, Basecamp and add-ons for deployment tasks – too many to mention.

Codeship has been around for a long time, so they have great support and have dealt with most teething issues by now.  They are also highly focused on performance which is reflected in their build times and their fast UI.  They also allow [skip ci] to skip builds in commit.

The only possible gripe I can muster is that between a $0 to $49 plan there could be a mid-way plan, for $10 or $25 per month.

Time to Deploy: 1:09s

Cost: Free (100 builds per month) or $49 per month (1 concurrent build / unlimited repos)

Cost Rating: 3 / 5

Support Rating: 4.5 / 5

Speed Rating: 5 / 5

Overall Rating: 4.2 / 5

First Impressions: Having to check-in your shippable.yml (similiar to a .travis.yml file) to run builds for a CI service the first few times is painpain pain.  I did at least 15 commit and pray git pushes until I got the right configuration for my project which ran the tests and passed the build.  This litters your commit history and is a painful exercise.  The upside is you have audit tracking on your build tasks which is a good thing, but I feel like I just wanted to dive into the slave VM, adjust the file in-line (until it worked) then back-port it to my codebase.

Shippable Screenshot

The ability to edit that file in-line and re-run as an override function would be a killer feature here.  I felt like I had lost control of the CI, since there was no SSH function either.  This is the feeling I wanted to avoid. Also seemed to run 4 builds whenever I committed code to one branch, couldn’t figure out why, I only had 3 branches, in any case.  Strange.

It also popped up with a Success message when a build was queued, which is a bit misleading.  The UI feels like a beta at times with uncompiled angular.js variable names and the like :(

I had serious problems with Nokogiri 1.5.x.x due to native dependencies – in this case I upgraded to 1.6.2.1. and didn’t have any issues.  I didn’t have this problem on any other CI in this blog with the same configuration.

The worst part was all my builds were labelled as Success even though Capistrano rolled back from the deployment!  Not good.

Once it was working, I was happy enough although it seemed slow like Magnum CI.  I just didn’t have enough confidence based on my experience which for me is a #fail

Time to Deploy: 3:56s

Cost: Free (no concurrent builds / 5 private repos) or $10 per month (2 concurrent builds / 10 private repos)

Cost Rating: 4.7 / 5

Support Rating: 4.5 / 5

Speed Rating: 3 / 5

Overall Rating: 3 / 5   Took 1 point off due to Capistrano rollbacks being labelled ‘success’ and other bugs / issues mentioned.

circleci

First Impressions: After having defined my shippable.yml I cloned it into circle.yml and that pretty much gave me the configuration I needed to get started.  Same gripes as Shippable re: YAML file for configuration – commit and pray sucks.

Circle CI Screenshot

NOTE:  These guys do not support Bitbucket.  For me this is a #fail since their status link for adding Bitbucket support is over 6 months old, with lots of +1 comments.

I guess Circle CI is Travis CI for private repositories.  It gets great points for the Ux feels very spacious and clean, it takes a lot of time and effort to get it this slick.  It almost feels too minimalist though.

They allow you to SSH to the slave VM, but if you ask for this option it leaves the connection open for 30 minutes, so it blocks your builds.  You can close it manually but it’s an extra step and was not intuitive.

You can generate API tokens for doing your own integration with this CI – quite an awesome feature (if you want it).  One specific gripe about their web application – their main assets file is 2MB of uncompressed javascript – 739kb compressed (gzip).  As a web-dev during the sunlight hours, this is not a good sign.

Since my project was in Bitbucket I couldn’t proceed much further, I tinkered around using a Github repository I maintain to test it conceptually but cannot comment on a comparable build time.

In term of their build output, it was great – showing details then folding, and turning green or red for each step which is great use of colours. Probably the most advanced UI in this aspect – competitors take note!

Support

I feel like this deserves it’s own segment since, for any system you rely on for technology or profit, you need to have confidence that it will do it’s job 95% of the time.  The other 5% of the time, you want to have the utmost confidence in the human support factor.

This is based on my own prejudices but I did contact support for each CI service to get some understanding.  Your mileage may vary =)

Magnum CI – Quick and helpful response.

Codeship - Bloody great.  CEO and CTO are quite available (Skype / Twitter) and they seem very hands-on and focused on improving and keeping customers happy.  While finishing editing of this blog post, Codeship put out an announcement extending their Free tier from 50 monthly builds to 100, which is awesome.

Shippable (via github.com issues) – Good – troubleshooted an issue I had and took 2 minor bug reports.

Circle CI – No reply (from their hipchat channel twice within 30 minutes… disappointing.) I did however, get a thoughtful response a day later from marketing about cancelling my membership.

Review Notes

Trying your tech stack on several providers is an ideal way to discover environment variables, apt-get native packages or unspecified version dependencies you need for your deployed code to run.

One realisation I have come to is that shipping your build file with your code, although it works well once you have a stable pipeline / environment, to get up and running can be a slow process.  Ideally you could override this file or SSH in and tweak settings to find the right configuration without multiple commits – you could easily waste an hour or two doing this with an existing project if build times are > 5 minutes.

After setting up the deployment pipeline, I also found something missing.

Sure, I can run tests before deploying, then deploy, then regression test.  But my CI should do more than that.  It should keep running tests, even in the absence of commits or deployments, to ensure the environment (and any dependencies) are fully functional.

Call them functional tests, smoke tests, contract tests, health checks – i’ve found these core acceptance tests alert me to failures in an online system better than more specific tools (eg. New Relic or Nagios). Running these tests regularly in-between deployments on a running system will uncover any fragility in your production system and ensure your tests are rock-solid.  Every CI system I have ever setup I have ended up doing this – not every functional test will run after every commit.

Editors Choice

All tested CI services support Bitbucket (except Circle CI) and Github and all support web-hooks (which is far more efficient than polling) and Magnum CI and Codeship support various add-ons for integration and notifications.

I have picked Codeship as my preferred SaaS CI service.  For me, currently it will be free, if I use them for another month or so I will start donating if I am still on the free tier, however I encourage you to evaluate Magnum CI and Shippable as well – your tech stack may be different and you may get a different experience.

Thanks for reading!

@rpocklin

About these ads

3 thoughts on “SaaS CI For Startups – Finding the right fit

  1. Response from Circle CI: Thanks for the writeup. Sorry that we lacked BitBucket support for you to try out Circle properly! I know that’s not helpful for those not on GitHub, but our customers love the effort we’ve put into parallelizing large test suites, our support for many different tech stacks and builds that start instantly. It’s a hard tradeoff to make, but we prefer to make an amazing experience for our customers than spread ourselves thin with a substandard product!

  2. This is a great roundup Rob.

    Very timely too as many devs would be looking into this. The more visibility you have about these builds the better it seems.
    Only thing you could have added is some screenshots of the UI’s given you were comparing them based on it.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s