Shipit vs Flightplan for Automated Administration

First off, let me say one really important thing. If you want your next project to succeed, I believe you should automate your server setup and deployment from day one. If you find yourself doing a particular thing even twice, it’s time to write down the steps to do it in a form you can run again; not just make a note of it in Evernote or on a piece of paper.

If you do that you will always be keen to do another deployment whenever you want to make a small improvement or fix a bug. Especially when you’re returning to the project weeks or months later and the process to do things is no longer fresh in your mind. Any solution at all is better than no solution at that point.

PaperQuik and ClearAndDraw

Last year I threw together a couple of small projects, grabbed a DigitalOcean server (thanks for their $60 credit from ng-conf last year), figured out how to setup my own server and deploy everything to create some new websites. I did everything from start to finish myself and I felt duly proud about having built something from scratch and launched it no matter how small it was.

But there’s a siren song that every developer feels when working in a particular language, be it Ruby or Java, JavaScript or Python, that all your tools should be written in your favorite language. You want your web server, build tools, continuous integration server, and sometimes even your editor built using the language you use most of the time. The same things get rewritten over and over to support that idea and although I hate the wastefulness of it, I certainly understand the feeling and everything I used to do my administration tasks for my projects was Bash shell scripts and I didn’t really like them much.

Then I started seeing new deployment and automation tools like Flightplan and Shipit come out specifically for JavaScript developers. Both are focused on the deployment part of things and not the building and development automation tasks which Grunt or Gulp focus on. So I thought it might be interesting to try and replace my shell scripts with these tools to see how easily they could do the same jobs. The tasks covered by my scripts were: initial machine configuration, updating of Ubuntu (mainly to get security fixes), deployment, and creating a SSH shell to the remote server. It’s not a lot, but that covered everything I found myself doing repeatedly as I put together my projects.

Flightplan

First I started with Flightplan. It’s a tool supposedly similar to Python’s Fabric tool. Never having used the latter I cannot say anything about that. What I can say is that I was able to cobble together a flightplan.js file which allowed me to do three out of my four tasks (you can run an SSH shell under it but not easily interact with it, so I abandoned that).

I might not be making the absolute best use of Flightplan because I was trying to import the same commands I had used in my shell scripts into the flightplan.js file to create tasks in it. However, it worked and it was pretty straightforward.

// Running this requires installing flightplan (https://github.com/pstadler/flightplan).
// Then use commands like:
//   fly install:production
//   fly deploy:production
//   fly upgrade:production
var plan = require('flightplan');

plan.target('production', [
  {
    host: 'PocketChange',
    username: 'root',
    agent: process.env.SSH_AUTH_SOCK
  }
]);

var tmpDir = 'PaperQuik-com-' + new Date().getTime();

// Install software on the server necessary to run this application.
// Then ensure that Apache is properly configured to serve the 
// application.
plan.remote('install', function (remote) {
  remote.sudo('apt-get update');
  remote.sudo('apt-get -y install apache2 emacs23 git unzip');
});

plan.local('install', function (local) {
  local.echo("We couldn't copy this file earlier because there isn't a spot for it until after Apache is installed.");
  local.transfer('paperquik.conf', '/etc/apache2/sites-available/');
});

plan.remote('install', function (remote) {
  remote.sudo('a2enmod expires headers rewrite proxy_http');

  remote.sudo('a2dissite 000-default');

  remote.sudo('a2ensite paperquik mdm');

  remote.sudo('service apache2 reload');
});

// Deploy the application.
plan.local('deploy', function(local) {
  local.log('Deploy the current build of PaperQuik.com.');
  local.log('Run build');
  local.exec('grunt build');

  local.log('Copy files to remote hosts');
  local.with('cd dist', function() {
    var filesToCopy = local.exec('find .');

    // rsync files to all the target's remote hosts
    local.transfer(filesToCopy, '/tmp/' + tmpDir);
  });
});

plan.remote('deploy', function(remote) {
  remote.log('Move folder to web root');
  remote.sudo('cp -R /tmp/' + tmpDir + '/*' + ' /var/www/paperquik');
  remote.rm('-rf /tmp/' + tmpDir);
});

// Upgrade Ubuntu to the latest.
plan.remote('upgrade', function (remote) {
  remote.log('Fetches the list of available Ubuntu upgrades.');
  remote.sudo('apt-get update');

  // And then actually does them.
  remote.sudo('apt-get -y dist-upgrade');
});

Notice the way the install task (or the deploy task) switches back and forth between sections which are for local and remote. I don’t think I messed that up, it seems like the normal structure for Flightplan and it seems odd to me. Doubly so because Flightplan doesn’t seem to have a mechanism for task dependencies (I need to do task A before I do task B or task C and I don’t want to repeat that code).

Pros:

  • All the commands within a task execute sequentially so it’s an easier transition from something like a shell script.
  • Allows you to specify multiple servers and will allow you to run tasks simultaneously against all of them. Not anything I need at this time, but you never know when a project could grow from one server to two.

Cons:

  • A given task is broken up into local and remote sections and they run sequentially based upon them all having the same name. There doesn’t seem to be any way for a given task to specify that it has dependencies upon other tasks being executed first (for example, maybe I do a directory cleanup before several different tasks).
  • Although it’s easier to deal with serially executing code, if you have several actions which would execute more quickly in parallel Flightplan doesn’t really support that.

Shipit

Then I built the same thing again in Shipit. As with Flightplan, it claims similarity to another tool as well, the Ruby deployment tool Capistrano in this case. Again I have to claim ignorance on this never having used Capistrano. Here’s the same set of commands (install, deploy, and upgrade) using a Shipit file:

// Running this requires installing Shipit (https://github.com/shipitjs/shipit).
// Then use commands like:
//   shipit production install
//   shipit production deploy
//   shipit production upgrade
module.exports = function (shipit) {
  require('shipit-deploy')(shipit);

  shipit.initConfig({
    production: {
      servers: 'root@PocketChange'
    }
  });

  var tmpDir = 'PaperQuik-com-' + new Date().getTime();

  shipit.task('install', function () {
    shipit.remote('sudo apt-get update').then(function () {
      // We'll wait for the update to complete before installing some software I like to have on the
      // server.
      shipit.remote('sudo apt-get -y install apache2 emacs23 git unzip').then(function () {
        // We don't need the following set of actions to happen in any particular order. For example,
        // we're good if the disables happen before the enables.
        var promises = [ ];

        // We couldn't copy this file earlier because there isn't a spot for it until after Apache is installed.
        promises.push(shipit.remoteCopy('paperquik.conf', '/etc/apache2/sites-available/'));

        promises.push(shipit.remote('sudo a2enmod expires headers rewrite proxy_http'));

        promises.push(shipit.remote('sudo a2dissite 000-default'));

        promises.push(shipit.remote('sudo a2ensite paperquik mdm'));

        // But we do need this to wait until we've complete all of the above. So we have it wait until
        // all of their promises have resolved.
        Promise.all(promises).then(function () {
          shipit.remote('sudo service apache2 reload');
        });
      });
    });
  });

  // This shipit file doesn't yet use the official shipit deploy functionality. It may in the future but
  // this is my old sequence and I know it works. Note: I also know theirs seems like it might be
  // better because it can roll back and I definitely do not have that.
  shipit.task('deploy', function () {
    shipit.log('Deploy the current build of PaperQuik.com.');
    shipit.local('grunt build')
        .then(function () {
          return shipit.remoteCopy('dist/*', '/tmp/' + tmpDir);
        })
        .then(function () {
          shipit.log('Move folder to web root');
          return shipit.remote('sudo cp -R /tmp/' + tmpDir + '/*' + ' /var/www/paperquik')          
        })
        .then(function () {
          shipit.remote('rm -rf /tmp/' + tmpDir);        
        });
  });

  shipit.task('upgrade', function () {
    shipit.log('Fetches the list of available Ubuntu upgrades.');
    shipit.remote('sudo apt-get update').then(function () {
      shipit.log('Now perform the upgrade.');
      shipit.remote('sudo apt-get -y dist-upgrade');
    });
  });
};

Sorry for the small text above, the line wrapping was bad if I didn’t reduce it. Here’s the original over on Github. The huge and most obvious difference here is that Shipit wants to do all of those Apache configuration commands in parallel. So I let it. I just added a little bit of code to delay restarting the server until all of them have been completed (you can see that around line 37 above). Likewise the deploy and upgrade tasks want to execute steps in parallel and I can’t always let it do that. Since all of the asynchronous actions in Shipit return promises I just added a little bit of code in each task where I need to control the order in which things happen and it works.

Pros:

  • Executes commands within a task in parallel to achieve maximum speed.
  • Allows you to specify multiple servers and will allow you to run tasks simultaneously against all of them. Not anything I need at this time, but you never know when a project could grow from one server to two.
  • Supports tasks which run other tasks (or which broadcast/sink events). Thus dependencies for tasks can be handled.

Cons:

  • The documentation. Seriously, come on. I’m going to have to contribute to this project just to fill out the documentation some.
  • Harder to structure serial commands which need to execute in a particular sequence.

Thoughts

You see what I mean about people rebuilding the same tools over and over again just using different languages. Both Shipit and Flightplan claim similarity to previous tools for Ruby and Python. However, at the same time I have to confess I don’t find either of those particularly appealing to use when all I use day to day is JavaScript. I used Java for over ten years and I still don’t want to do all of my build and deployment with Ant. When I wanted to control the order of the asynchronous events in Shipit, it was nice that I could easily see how to do that from my experience with JavaScript promises in AngularJS and Node.js.

Both tools allow you to run tasks against multiple servers simultaneously. Both allow you to have multiple sets of servers so you can have staging servers or, if your just playing around like me, a Vagrant server you bring up and down just for testing purposes. Either could probably do your administration jobs, but I just liked Shipit a little bit better because it seemed more powerful. Going forward I’m going to probably pull the Flightplan files out of the master branch of my projects and leave them up only for reference from this blog post. Now I just need to see if I can do something about that Shipit documentation.

Advertisements

9 thoughts on “Shipit vs Flightplan for Automated Administration

  1. John

    Thanks for the post, very helpful. Two quick questions:
    1) the shipit functions return promises and then works fine. However, Promise.all throws an error saying Promise is not defined. I haven’t required bluebird, but it looks like you didn’t either. Any light you could shed would be helpful, thanks!
    2) What is ‘deployunction? I couldn’t find any reference to it being sugar for function deploy(){}

    Reply
    1. John Munsch Post author

      1) I did not include Promise and yet I’ve had no trouble with that. Shipit itself includes it in the shipit.js file here: https://github.com/shipitjs/shipit/blob/master/lib/shipit.js

      So that’s a bit of a mystery for me. However, since you don’t actually have to use the same specific instance of the library as what Shipit used, you _could_ do your own require if you want to solve that problem.

      2) I’m not sure exactly what you meant about ‘deployunction’. I searched my blog entry for something like that and did not see it anywhere within the text. Can you point to where you saw that?

      Reply
  2. John

    1 is still a mystery, for now i’ll just include bluebird. I saw that shipit uses bluebird and includes it, so no clue either. Promise.all should work. I redefined it and it works fine. Weird.

    2) i think might just be a misplaced span tag. The code on chrome reads shipit.task(‘deployunction () { and shipit.task(‘deployunction () { . I suspect it’s just an html issue rather than some new funky ES6 functionality I hadn’t run across 🙂 You might have a look at the source.

    Reply
    1. John Munsch Post author

      Looks fine in Chrome on my Mac. However, you definitely want to take a look at the copy over on Github (https://github.com/JohnMunsch/PaperQuik/blob/master/shipitfile.js) anyway if you decide to base anything on that. The install task in particular was out-of-date and didn’t work very well when I tried it last weekend, but now I believe I’ve got it well sorted out.

      I’m hoping to do a follow up article to this one within the week where I talk about installing Vagrant and using that to test out all your Shipit tasks prior to deploying on real servers. I was able to get solid install, deploy, and upgrades against practice servers which should look exactly like what I could stand up at DigitalOcean, Atlantic.net, or Vultr.

      Reply
  3. sosh101

    Thanks for this writeup! I have been looking at shipit too, but am close to passing on it due to the terrible documentation and lack of examples (your article has convinced me to investigate a little longer though). Did you find any examples/docs/tips other than the readme on the github page? I notice you are require()ing the shipit-deploy plugin, but are you using any of the features provided by that? I couldn’t see anything in your code, but I’m really not sure whats provided by plain shipit, and the shipit-deploy plugin.

    (PS. I see installunction, deployunction too – I initially assumed they were comical task names 😉

    Reply
    1. John Munsch Post author

      I really get what you’re saying about the documentation and examples (and the github page was pretty much my only source of info, there’s not a lot out there about Shipit right now). I’m not using the deploy task at all yet.

      Here’s the thing. There are a million tools out there for the people who are administering dozens or hundreds of servers because they basically cannot do so without tools, it’s just impossible to do so without everything falling down around their heads. But those of us who only want to administrate one or two servers for our own projects now have the same gift of inexpensive servers that everyone else has. I can run a couple of small apps on a $5/month server and I don’t want to do it all with a bunch of shell scripts. I want an easy to learn, easy to use tool which is aimed at the little guy.

      In fact, I’ll go further. There was a time when you got programming frameworks like AngularJS and they did not come with any kind of automated testing. I think one of the big frameworks needs to adopt provisioning and deployment as something that comes as a standard part of the package. Wouldn’t it be nice if Node.js had a “make me a server” and “put this project on that server” right there in the box?

      Ruby on Rails was great at establishing a toolset that encompassed 80% of what developers needed most of the time and you swapped out a particular tool only if it didn’t work for your needs but everything else was still covered by the one-size-fits-all blanket. I see these things as just as fundamental.

      Reply
  4. TinTopHat (@TinTopHat)

    Personally I’m a big fan of Ansible for automated deployment and server management. Its written in Python but the “playbooks” as they’re called are written in YAML. I suppose Ansible is closer to Puppet but all of these tools from Fabric to Ansible toi Puppet do have some overlap.

    Just wondered if since writing this blog post you’d come across Ansible and what your thoughts on it were?

    Reply
    1. John Munsch Post author

      After trying out both of these I too ended up rejecting them both in favor of Ansible. If you look at my PaperQuik project on Github, it now does everything via Ansible.

      I won’t say Ansible is perfect. I’m unhappy with the non-uniform quality of the various plugins and also the problems it seems to have with copying files (it either takes forever to copy or you use rsync and encounter a bunch of problems there). However, I love the fact that it doesn’t require something installed on each target machine and I figure now that they have real money behind them, some of my complaints may disappear over time.

      Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s