SaaS CI For Startups – Finding the right fit

The Premise

As part of developing Foover we needed a CI environment.  We are using EC2 for our server, but didn’t want anything else running on production that would consume resources.

I also wanted professional deployment notifications to multiple stakeholders, and deploys on each push without thinking about it, with a high degree of confidence that, if the tests pass, the code is up in da cloud within a few minutes.

Obviously you need a codebase with a decent suite of tests first =)

In this post I will review Magnum CI, Codeship, Shippable and Circle CI with their pros and the cons.

First, let me summarise why you should consider SaaS CI, and what tech stack my codebase uses.

The Benefits

If you are a startup, you probably know a bit about setting up a Linux server, SSH, deploying artifacts and restarting services.  But that’s not what is gonna earn you the $$$. If you consider your time valuable, let someone else manage this infrastructure for you. Especially if you have multiple projects.

Also if you have multiple developers, especially across multiple time zones, accessible CI is important to see what happened to the tests across builds, and it provides you with good transparency and a separate environment in which everything works.  Doing this research helped me uncover some environment variables I had just assumed on my development machine and EC2 server which is great, now it’s documented and is infrastructure as code.

One difference in our repository worth mentioning is that the Rails root of the project was in a sub-directory of the Git repository.  I always set Rails projects up like this so you can put other stuff in here.  You could also use subtrees or submodules but this I found was a good test to the flexibility of configuring the SaaS CI build scripts.

The Tech Stack

We have a Rails-based API using MongoDB, deployment with Capistrano checked into Git. The repository is hosted in Bitbucket but all the CI services I review below support Github as well.

Tip: Deploying with Capistrano to EC2

When I was deploying manually from my machine, i’d tell Capistrano to use my personal EC2 SSH Private PEM key which was generated with the EC2 instance.  I’d suggest not putting this key in your codebase, this should be kept separate from prying eyes since it’s the keys to production… literally!

What I did is I took the SSH public key generated for the SaaS CI build and put it in ~/.ssh/authorized_keys on my EC2 server.  This way it is trivial to revoke access and do passwordless deployments.  To specify the SSH private key to use when on SaaS CI build slaves, I found it safe to assume it is in ~/.ssh/id_rsa just have Capistrano reference this when it runs the deploy task with the ssh_options[:keys] variable (Capistrano v2).

NOTE: Time to build in the below reviews was determined by 3 consecutive green builds (after an initial green build) and averaged to yield the mean build time.

First impressions: It is free and seems fairly new, so the interface didn’t feel super-slick but it was clean and functional.

The UI and builds felt slow, it seemed each time it started a new VM from scratch (which may be the price you pay on a free tier) but it increased the built time incredibly. It had to install all the ruby gems each time which probably took at least 2 minutes.

It’s simple to setup a build, you add your build commands to the UI and authorise Magnum CI to access your repository via SSH keys, and add a web-hook into Bitbucket so it doesn’t need to poll for changes (it gets notified – push not pull).

It abstracts away installing RVM – you select your Ruby version.  There isn’t every version possible, but there’s a common list: 1.9.2, 1.9.3, 2.0.0 and 2.1.0 (at the time of writing).

The UI does give a nice scrolling window like Bamboo / Jenkins to show build console status.  It may sound simple but I feel like this is an important feature.

These guys have integrations with Hipchat, Slack and IRC.

One feature I thought could be very useful (but should not be abused) is to have your CI platform *not* perform a build if you enter [ci-skip] or [ci skip] in your commit message.

Time to deploy: 6:28s

Cost: Free (no parallel builds)

Cost Rating: 4.8 / 5

Support Rating: 3.8 / 5

Speed Rating: 1.5 / 5

Overall Rating: 3.4 / 5

First impressions: It has a very professional interface, leads you via prompts like “I see you changed your build script – would you like to re-run your last build?” – Why yes, J.A.R.V.I.S, I would!  Awesome Ux – reminds me of Heroku. The UI feels fast and responsive.

Codeship has an option to SSH onto the slave VM to debug what is actually happening (or not happening) on the metal, which is really what I want – when the sh*&t hits the fan, I don’t want to log a support ticket, I want to see if there’s a way I can work around it and get a build out.  This is a killer feature.

They have plenty of integrations for Slack, Hipchat, Basecamp and add-ons for deployment tasks – too many to mention.

Codeship has been around for a long time, so they have great support and have dealt with most teething issues by now.  They are also highly focused on performance which is reflected in their build times and their fast UI.  They also allow [skip ci] to skip builds in commit.

The only possible gripe I can muster is that between a $0 to $49 plan there could be a mid-way plan, for $10 or $25 per month.

Time to Deploy: 1:09s

Cost: Free (100 builds per month) or $49 per month (1 concurrent build / unlimited repos)

Cost Rating: 3 / 5

Support Rating: 4.5 / 5

Speed Rating: 5 / 5

Overall Rating: 4.2 / 5

First Impressions: Having to check-in your shippable.yml (similiar to a .travis.yml file) to run builds for a CI service the first few times is pain, pain pain.  I did at least 15 commit and pray git pushes until I got the right configuration for my project which ran the tests and passed the build.  This litters your commit history and is a painful exercise.  The upside is you have audit tracking on your build tasks which is a good thing, but I feel like I just wanted to dive into the slave VM, adjust the file in-line (until it worked) then back-port it to my codebase.

The ability to edit that file in-line and re-run as an override function would be a killer feature here.  I felt like I had lost control of the CI, since there was no SSH function either.  This is the feeling I wanted to avoid. Also seemed to run 4 builds whenever I committed code to one branch, couldn’t figure out why, I only had 3 branches, in any case.  Strange.

It also popped up with a Success message when a build was queued, which is a bit misleading.  The UI feels like a beta at times with uncompiled angular.js variable names and the like 😦

I had serious problems with Nokogiri 1.5.x.x due to native dependencies – in this case I upgraded to 1.6.2.1. and didn’t have any issues.  I didn’t have this problem on any other CI in this blog with the same configuration.

The worst part was all my builds were labelled as Success even though Capistrano rolled back from the deployment!  Not good.

Once it was working, I was happy enough although it seemed slow like Magnum CI.  I just didn’t have enough confidence based on my experience which for me is a #fail

Time to Deploy: 3:56s

Cost: Free (no concurrent builds / 5 private repos) or $10 per month (2 concurrent builds / 10 private repos)

Cost Rating: 4.7 / 5

Support Rating: 4.5 / 5

Speed Rating: 3 / 5

Overall Rating: 3 / 5   Took 1 point off due to Capistrano rollbacks being labelled ‘success’ and other bugs / issues mentioned.

First Impressions: After having defined my shippable.yml I cloned it into circle.yml and that pretty much gave me the configuration I needed to get started.  Same gripes as Shippable re: YAML file for configuration – commit and pray sucks.

NOTE:  These guys do not support Bitbucket.  For me this is a #fail since their status link for adding Bitbucket support is over 6 months old, with lots of +1 comments.

I guess Circle CI is Travis CI for private repositories.  It gets great points for the Ux feels very spacious and clean, it takes a lot of time and effort to get it this slick.  It almost feels too minimalist though.

They allow you to SSH to the slave VM, but if you ask for this option it leaves the connection open for 30 minutes, so it blocks your builds.  You can close it manually but it’s an extra step and was not intuitive.

You can generate API tokens for doing your own integration with this CI – quite an awesome feature (if you want it).  One specific gripe about their web application – their main assets file is 2MB of uncompressed javascript – 739kb compressed (gzip).  As a web-dev during the sunlight hours, this is not a good sign.

Since my project was in Bitbucket I couldn’t proceed much further, I tinkered around using a Github repository I maintain to test it conceptually but cannot comment on a comparable build time.

In term of their build output, it was great – showing details then folding, and turning green or red for each step which is great use of colours. Probably the most advanced UI in this aspect – competitors take note!

Support

I feel like this deserves it’s own segment since, for any system you rely on for technology or profit, you need to have confidence that it will do it’s job 95% of the time.  The other 5% of the time, you want to have the utmost confidence in the human support factor.

This is based on my own prejudices but I did contact support for each CI service to get some understanding.  Your mileage may vary =)

Magnum CI – Quick and helpful response.

Codeship – Bloody great.  CEO and CTO are quite available (Skype / Twitter) and they seem very hands-on and focused on improving and keeping customers happy.  While finishing editing of this blog post, Codeship put out an announcement extending their Free tier from 50 monthly builds to 100, which is awesome.

Shippable (via github.com issues) – Good – troubleshooted an issue I had and took 2 minor bug reports.

Circle CI – No reply (from their hipchat channel twice within 30 minutes… disappointing.) I did however, get a thoughtful response a day later from marketing about cancelling my membership.

Review Notes

Trying your tech stack on several providers is an ideal way to discover environment variables, apt-get native packages or unspecified version dependencies you need for your deployed code to run.

One realisation I have come to is that shipping your build file with your code, although it works well once you have a stable pipeline / environment, to get up and running can be a slow process.  Ideally you could override this file or SSH in and tweak settings to find the right configuration without multiple commits – you could easily waste an hour or two doing this with an existing project if build times are > 5 minutes.

After setting up the deployment pipeline, I also found something missing.

Sure, I can run tests before deploying, then deploy, then regression test.  But my CI should do more than that.  It should keep running tests, even in the absence of commits or deployments, to ensure the environment (and any dependencies) are fully functional.

Call them functional tests, smoke tests, contract tests, health checks – i’ve found these core acceptance tests alert me to failures in an online system better than more specific tools (eg. New Relic or Nagios). Running these tests regularly in-between deployments on a running system will uncover any fragility in your production system and ensure your tests are rock-solid.  Every CI system I have ever setup I have ended up doing this – not every functional test will run after every commit.

Editors Choice

All tested CI services support Bitbucket (except Circle CI) and Github and all support web-hooks (which is far more efficient than polling) and Magnum CI and Codeship support various add-ons for integration and notifications.

I have picked Codeship as my preferred SaaS CI service.  For me, currently it will be free, if I use them for another month or so I will start donating if I am still on the free tier, however I encourage you to evaluate Magnum CI and Shippable as well – your tech stack may be different and you may get a different experience.

Thanks for reading!

@rpocklin

Scala, come out and Play

Last week I started work on a Play 2.1 Scala project, which is a nice departure from Spring and Hibernate, so it’s a good time to share some first thoughts on the differences between the old traditional Java webapp, Grails, Rails and Play 2.1 with Scala.

First off, installing Play is easy as pie. Most modern frameworks can create a skeleton app quite easily, however the integration with sbt feels a bit awkward but is still much better than good ol’ Maven. Re-compiling scala files can seem to take awhile, but running tests are nice and quick in IntelliJ.

IntelliJ is a great choice for Scala/Play projects, as v12.1 has excellent auto-completion, syntax highlighting and click-through method links and good test integration. The contextual help is quite good at telling you what’s wrong – this seems to be a strength of the Scala compiler. However, for whatever reason we spent some time trying to get IntelliJ to register our Play app as such when importing the project. Possibly because the /app folder wasn’t off the root of the repository, (we have it in /play/app) but even after running play idea to generate the necessary project files, IntelliJ refused to import it as a Play app.

After upgrading the Play plugin, installing a Scala plugin and ensuring Scala, Play and Java were all in the path and had appropriate %_HOME directories set, it still wouldn’t give in. Creating a new Play app from IntelliJ seemed to bootstrap it with the necessary vision it needed, and after doing this, it imported our existing app no worries.

Be careful as lots of documentation on the web is disinformation, since Play 1.0 is different, and Play 2.1 have made some refinements from previous revisions. Also the fact that we are using Specs2 means you have to sanity check all blog posts and stackoverflows before assuming they are correct for you.

Testing in Scala is beautiful and clean, feels natural and all the crazy syntax of Mockito in Java seems to have been repaired. Specs2 reminds me more of RSpec or Jasmine Specs, which make it easier to read and group similiar tests.

Scala as a language feels quite wierd. Ruby and underscore.js already have nice functional methods like reduce, map, filter which are pretty straight-forward. Things I like are the Option class and the case matching aspects. Like any new framework (and language) you can get stumped and need a good reference on how to do things the idiomatic way.

Scala templates have been carefully designed with a set of opinions and good conventions (such as auto-escaping and immutable parameters, and the ability to create templates and helpers which have multiple closures as parameters changes the way you can call templates.

As an example:

To get all the goodness I am used to in other projects, I created a @toggle macro, which takes in a single String parameter, followed by 2 closures (one to render when enabled and, you guess it, the other to render when disabled).

Place the following in the directory views/helpers/toggle.scala.html as a helper template:

@(toggle: String)(enabledContent: Any)(disabledContent: Any)

@import play.api._

@defining(Play.current.configuration.getBoolean("toggles." + toggle)) { toggleValue =>

  @toggleValue match {
    case Some(toggleEnabled) => {

      @if(toggleEnabled) {
          @enabledContent
      } else {
         @disabledContent
      }

    }
    case None => {
      <h1> Error invalid toggle specified: @toggle</h1>
    }
  }
}

Then simply set a value in conf/application.conf ie.

toggles.search_box=true

Where you want to switch the behaviour (in a Scala template), do the following:

  @toggle("search_box") { 
    <label>Search:<input type="text" id="search"></label>
  } {
    <div>Search is currently disabled</div>
  }

Additionally, Play allows you to toggle the value in the application.conf file dynamically, no restart needed =)

$(‘#2012’).on(‘success’, function(events) {..});

Personally, for me this year, my biggest technical advances were:

• Learning bash scripting, provisioning servers, deployment scripts.
• Developing on a Mac… yes I know i’m behind the curve here 🙂
• Changing to IntelliJ from Eclipse… it’s worth the $$$.
• Forcing myself to use Sinatra to build an app rather than Rails… and building it up to the point where it is becoming a Rails app.
• I will migrate it across once Rails 4 goes standard. This helps me appreciate what Rails gives you, what it forces you into, and understand more about which modules it wraps up and the conventions it gives you.
• Learning the in-s and out-s of underscore.js and backbone.js – you can read and read but to grok it you need to code in it.  Underscore.js in particular is like an ant… a small creature that can lift 100 times it’s weight. Kudos guys it’s invaluable and gives much cleaner / functional code in javascript.
• Getting a nice 14″ Lenovo i5/SSD/8GB to use on the train… any larger is uncomfortable. The new Mac Pro 13″ looks like a good size too.
• Spending time *away* from code to give my ideas time to settle / step away from the keyboard. As valuable as coding itself.
• Getting a Telstra 4G phone to use the internet on the train… for better or worse, the Telstra mobile network is very fast and reliable.

error: 2013 is undefined

Dev.productivity.increase();

Tips to increase your dev-fu (java devs mainly):

 

• switch to IntelliJ / RubyMine – fast searching (maybe notepad++ / grep / sublime is faster?) and if you are doing CSS / Javascript development I find it’s the most productive IDE for me.
• spend 10 minutes every day asking yourself how to improve your IDE / environment… what do you spend the most time doing each day that is repetitive?
• learn REGEX. It’s everywhere… if you have to replace/modify more than 2-3 LOC, use a find/replace. If that doesn’t fit, REGEX is your friend. Also learn SED once you’ve learned Linux 🙂
• printout that keymap of your IDE… circle shortcuts you wanna learn then make a concious effort to learn them.  Test yourself the next day / week by trying to recall it with your muscle memory. Practising on different computers / keyboards reinforces it… especially mac vs. non-mac keyboards 🙂
• Pair!!! I have learned the most shortcuts / configuration options from pairing or looking at others IDEs and asking questions.
• Consider Chrome your browser of choice.  I was Firefox for a long time, but Chrome seems quicker and i’m more productive on it now.
• if you are a frontend dev, research a bit on browser plugins for helpful tools like screen resizing, URL testing, JSON parsing, CSS reloading etc.
• learn Linux, learn the shell and the command line…
• learn Git (if you haven’t already), setup a few repos, patch some code on github.com – it’s good for the soul 🙂

developer.life.com/js/

Lately i’ve been doing alot of javascript development in my spare time. Spikes, little libraries, like HTML validation and logging.  When you’re righting event-driven code, TDD and logging help you quickly track down problems in your code.

The playing field has changed in Javascript and on the frontend over the last few years, and i’d like to share the tools and utilities I now use for my spare-time projects:

XDate : gives you nice decorators over a vanilla Date object (which is pretty basic and outdated) including parsing, formatting and manipulation without all the headaches.

Normalize.css : Best of the CSS resets.

Modernizr: Good for targetting many platforms (especially non-PC eg. mobile and IPad – helps with feature detection)

JQuery + Bootstrap 2: This one is a no-brainer.  Gives you responsive design (*-fluid), thoroughly tested and customisable and using Less or bootstrap-sass, can easily change styles globally.  Becoming a defacto standard.

Underscore.js:  I’ve found most people that have used Ruby like this library, it’s neat and powerful…

Backbone.js + Model.Binder – Backbone is useful even if you are grafting it to a legacy app, since it gives you basic object-oriented grouping of data and methods.  It encourages OO design in javascript, which also encourages a basic separation of DOM and data/functions.  There are some aspects I think are too basic, especially once you get into non-trivial models (toJSON() for example, is not recursive) but for alot of simple stuff it gives you cleaner, testable code and good separation of concerns.

Jasmine: If you are familiar with RSpec or keen to learn, start using Jasmine.  It’s probably the strongest testing suite for javascript and has good integration to both ruby/rails and java.  I use it even on vanilla javascript projects.

Working Together

Working on a new project is a great way to review what you think you know in a different application. I think as a veteran developer, the hardest part of doing our job is to avoid doing just the ‘cool’ stuff and being continuously objective about the work you are doing.

Devs will often spend more time on things they think are important which really aren’t in the scheme of things. A good dev will figure this out him/herself without help, and benefit the team and the product for doing so.

The other side of the coin is that the business isn’t always aware of things which would add efficiency to the product or process and can be hard to convince, even with several team members’ support.

Establishing trust in the environment is paramount to working together, and eroding that trust is so easy that it is a cautious game. The other issue is that building trust happens best when you achieve common goals, and many people will change jobs within a few years. It also seems like the less time you have been at a job, the more passionate you are about it, and the more energy you give the project.

Perhaps the solution is to only have short projects? =)

A good question is how engaged is the team? How often do you see people working together? You will normally see groups of people in the same process (streams) working together, and eating together. This is because you need a level of teamwork day to day to do your work. Examples are developers with developers, testers with testers, BAs and product owners.

A more important question is how often you see people from different streams working together, having whiteboard discussions, creating stories or eating lunch together or socialising. Forget metrics, KPIs, meetings, building a good relationship will pay off in orders of magnitude. Why?

If you build a relationship, you build respect and trust. Even if it’s not work related. You also build empathy.

By building empathy you tune your listening skills. You put yourself in the other person’s shoes and try to understand their points of view. Having two-way empathy and understanding is the ultimate level of communication.

Apache on RedHat

Seems many sites have partial suggestions to installing mod_jk.so in Apache but fall short of the whole process.

This works for installing for Apache 2.2 / 2.2.3 on RedHat Linux.

This is what we found worked:

1. yum install httpd-devel
2. Download the latest tomcat connectors source from Apache to /tmp/tomcat-connectors.tar.gz
3. cd /tmp
4. tar -zxvf tomcat-connectors.tar.gz
5. cd native
6. ./buildconf.sh
7. ./configure --with-apxs=/usr/sbin/apxs
8. make
9. make install

You can verify this worked by reading the results on screen, also by verifying the following file exists: /usr/lib64/httpd/modules/mod_jk.so

Some of the complexity is in compiling the tomcat connectors from the source for your environment.

The reason for yum install httpd-devel, is to install the Apache Extension Tools, which is required to build mod_jk.so.

Hope this saves someone (else) some time!

Spring 3.1 not showing field errors using spring tags

After upgrading from Spring 3.1 we noted that many of our forms which have validation logic in the server side stopped displaying the error messages back to the original form submission.

Seems in certain circumstances when you create and return a new ModelAndView the BindingResult object is not appended to the Model automatically.

Adding the following lines is a quick work-around.  

Assuming method signature of:

public ModelAndView someMethod(@Valid User user, BindingResult result) {

…

if (result.hasErrors()) {

  ModelAndView modelAndView = new ModelAndView(PAGE);

  modelAndView.addAllObjects(result.getModel()); // added this line

  return modelAndView;

}

…

}

In our case we are not going to continue with this way of posting forms so we won’t invest more time in this.

I also noticed Spring has adopted the rails/grails ‘flash’ object which uses the session as a shared data store to pass messages or objects in between server side redirects, not a great pattern really.  Most Client MVC apps don’t need this complexity.

Learning new languages and their expressiveness

Syntax can be hard to learn. Although I have trouble reading lisp / clojure / haskell, I respect that for those who understand it is a simple pleasure to write code in.

The problem, as highlighted is not limited to computer languages. Often people who try to learn another language from English continually say “but why do they do it like that? English does it better.” Unfortunately, learning your first language (eg. mother tongue) is vastly different in approach from learning your second and third.

Many times learning the second is the hardest – why is this? You have to open up your cemented neural pathways to the idea that something can be expressed in a foreign way. Unlocking those pathways is hard, as it is like fighting an inertia you have had for a long time.

I learned Spanish, then French. After that I find my ability to learn languages is vastly easier. Why? Because I understand the concepts of tenses, gender, case, mood and respect the subtleties of each language.

I no longer learn concepts, I learn the implementations and the nuances of each language. The same now for most programming languages. In a language, I need to express conditionality. I need collections and looping, filtering
and transforming. I don’t need to learn these concepts again so I can pick up languages quicker and quicker as I learn more concepts.

I now see a trend towards less shared-state/synchronised realtime applications and a strengthening towards functional (map/reducible) programming and languages supporting shared-nothing and actor patterns. The interesting thing is that we used to think by sharing state an synchronising it was faster and more scalable than copying objects and using more memory. Now it is more scalable to do exactly that.

We now have the programming power, the memory and the disk space for almost anything – we are now stripping back our programming languages to make it ‘easier’ for us to write code. By easier I don’t mean we are writing simpler code. As Ola Bini points out in his blog programmers are wanting a higher level of expressiveness. To do this, we have to make some assumptions like we are not going to share and synchronise data/state everywhere in our application. Sure, it may happen on the database or in a 3rd party library like an aggregator, but we as expressive functional programmers don’t want the bother of covering all those horrid edge cases which are hard to debug and harder to reproduce. I’ve seen some shocking issues with ThreadLocal being used and abused. Which happens alot.

In my own code I have found issues where I have overridden variables or options to my own detriment. Each time thinking ‘if only I had put that into another (usually local) variable. I believe this is the biggest (current) pitfall in Java. It is too easy to modify the contents of objects from any library which has access to it. It almost sounds like a security flaw. It’s calling out saying “hey, you there. You want to use me somewhere else? Don’t clone me, oh no. Just change my state a bit. It’s alright. Surely.”. No.

I am finishing a book on Beginning Scala at the moment, which sounds interesting but mostly just copying the good things in Ruby and cramming it into Java-esque syntax. For these languages it is more important to learn the ‘idoms’ of the language, which is what should give you more expressiveness.

Ruby has alot of this. I am awed that this language was created over 15 years ago. It is a beautiful and expressive language. It reminds me of going from Windows background and learning Linux. You can do some really powerful stuff,
and you can really get yourself into trouble (metaclass?). Guard the gates of expressiveness and you will be treated with a powerful tool.

My next blog post will talk about JRuby in detail.

Agile… enough?

We’ve been doing a few inceptions in agile lately, and the question always comes up… how much work should be done for ‘this’ story? Well, you define your acceptance criteria, then you estimate your work based on that being completed and functional.

Honestly, there is so much more to it than that. There will always be acceptance criteria you miss out at the start, but perhaps you will cover the 80% of use cases. Perhaps you have to design a new pattern, so that other stories like the current one can be done quicker in the future, or perhaps you need to spike first because of the unknown estimate. Perhaps you need a 3rd party library to incorporate, so you need to wire that in and test it too.

Even in agile, I find myself asking, how much (or rather, how little) design is just enough? It really depends on the project. If the code is being rapidly developed, screens changing every day, database schema in flux, then perhaps you don’t want to do anything you don’t have to for that story, lest it be thrown away tomorrow and your efforts wasted.

Any developer hates the idea of their code being thrown away, it’s in our nature 🙂

Two areas I sometimes struggle with is a) only do enough to make the feature work and b) don’t think about things in the future. If you take b) as gospel, it means you will never refactor. I have seen developers go down this path. The code works, they iron out the bugs (with scant test cases) and it goes to production. They develop work quickly, but slowly but surely the code is polluted by duplication, wrong responsibilities in the wrong classes, or tight coupling due to bad design and/or lack of tests.

The begging question is how do you monitor this to stop it happening?