Monthly Archives: March 2015

Shipit vs Flightplan for Automated Administration

First off, let me say one really important thing. If you want your next project to succeed, I believe you should automate your server setup and deployment from day one. If you find yourself doing a particular thing even twice, it’s time to write down the steps to do it in a form you can run again; not just make a note of it in Evernote or on a piece of paper.

If you do that you will always be keen to do another deployment whenever you want to make a small improvement or fix a bug. Especially when you’re returning to the project weeks or months later and the process to do things is no longer fresh in your mind. Any solution at all is better than no solution at that point.

PaperQuik and ClearAndDraw

Last year I threw together a couple of small projects, grabbed a DigitalOcean server (thanks for their $60 credit from ng-conf last year), figured out how to setup my own server and deploy everything to create some new websites. I did everything from start to finish myself and I felt duly proud about having built something from scratch and launched it no matter how small it was.

But there’s a siren song that every developer feels when working in a particular language, be it Ruby or Java, JavaScript or Python, that all your tools should be written in your favorite language. You want your web server, build tools, continuous integration server, and sometimes even your editor built using the language you use most of the time. The same things get rewritten over and over to support that idea and although I hate the wastefulness of it, I certainly understand the feeling and everything I used to do my administration tasks for my projects was Bash shell scripts and I didn’t really like them much.

Then I started seeing new deployment and automation tools like Flightplan and Shipit come out specifically for JavaScript developers. Both are focused on the deployment part of things and not the building and development automation tasks which Grunt or Gulp focus on. So I thought it might be interesting to try and replace my shell scripts with these tools to see how easily they could do the same jobs. The tasks covered by my scripts were: initial machine configuration, updating of Ubuntu (mainly to get security fixes), deployment, and creating a SSH shell to the remote server. It’s not a lot, but that covered everything I found myself doing repeatedly as I put together my projects.

Flightplan

First I started with Flightplan. It’s a tool supposedly similar to Python’s Fabric tool. Never having used the latter I cannot say anything about that. What I can say is that I was able to cobble together a flightplan.js file which allowed me to do three out of my four tasks (you can run an SSH shell under it but not easily interact with it, so I abandoned that).

I might not be making the absolute best use of Flightplan because I was trying to import the same commands I had used in my shell scripts into the flightplan.js file to create tasks in it. However, it worked and it was pretty straightforward.

// Running this requires installing flightplan (https://github.com/pstadler/flightplan).
// Then use commands like:
//   fly install:production
//   fly deploy:production
//   fly upgrade:production
var plan = require('flightplan');

plan.target('production', [
  {
    host: 'PocketChange',
    username: 'root',
    agent: process.env.SSH_AUTH_SOCK
  }
]);

var tmpDir = 'PaperQuik-com-' + new Date().getTime();

// Install software on the server necessary to run this application.
// Then ensure that Apache is properly configured to serve the 
// application.
plan.remote('install', function (remote) {
  remote.sudo('apt-get update');
  remote.sudo('apt-get -y install apache2 emacs23 git unzip');
});

plan.local('install', function (local) {
  local.echo("We couldn't copy this file earlier because there isn't a spot for it until after Apache is installed.");
  local.transfer('paperquik.conf', '/etc/apache2/sites-available/');
});

plan.remote('install', function (remote) {
  remote.sudo('a2enmod expires headers rewrite proxy_http');

  remote.sudo('a2dissite 000-default');

  remote.sudo('a2ensite paperquik mdm');

  remote.sudo('service apache2 reload');
});

// Deploy the application.
plan.local('deploy', function(local) {
  local.log('Deploy the current build of PaperQuik.com.');
  local.log('Run build');
  local.exec('grunt build');

  local.log('Copy files to remote hosts');
  local.with('cd dist', function() {
    var filesToCopy = local.exec('find .');

    // rsync files to all the target's remote hosts
    local.transfer(filesToCopy, '/tmp/' + tmpDir);
  });
});

plan.remote('deploy', function(remote) {
  remote.log('Move folder to web root');
  remote.sudo('cp -R /tmp/' + tmpDir + '/*' + ' /var/www/paperquik');
  remote.rm('-rf /tmp/' + tmpDir);
});

// Upgrade Ubuntu to the latest.
plan.remote('upgrade', function (remote) {
  remote.log('Fetches the list of available Ubuntu upgrades.');
  remote.sudo('apt-get update');

  // And then actually does them.
  remote.sudo('apt-get -y dist-upgrade');
});

Notice the way the install task (or the deploy task) switches back and forth between sections which are for local and remote. I don’t think I messed that up, it seems like the normal structure for Flightplan and it seems odd to me. Doubly so because Flightplan doesn’t seem to have a mechanism for task dependencies (I need to do task A before I do task B or task C and I don’t want to repeat that code).

Pros:

  • All the commands within a task execute sequentially so it’s an easier transition from something like a shell script.
  • Allows you to specify multiple servers and will allow you to run tasks simultaneously against all of them. Not anything I need at this time, but you never know when a project could grow from one server to two.

Cons:

  • A given task is broken up into local and remote sections and they run sequentially based upon them all having the same name. There doesn’t seem to be any way for a given task to specify that it has dependencies upon other tasks being executed first (for example, maybe I do a directory cleanup before several different tasks).
  • Although it’s easier to deal with serially executing code, if you have several actions which would execute more quickly in parallel Flightplan doesn’t really support that.

Shipit

Then I built the same thing again in Shipit. As with Flightplan, it claims similarity to another tool as well, the Ruby deployment tool Capistrano in this case. Again I have to claim ignorance on this never having used Capistrano. Here’s the same set of commands (install, deploy, and upgrade) using a Shipit file:

// Running this requires installing Shipit (https://github.com/shipitjs/shipit).
// Then use commands like:
//   shipit production install
//   shipit production deploy
//   shipit production upgrade
module.exports = function (shipit) {
  require('shipit-deploy')(shipit);

  shipit.initConfig({
    production: {
      servers: 'root@PocketChange'
    }
  });

  var tmpDir = 'PaperQuik-com-' + new Date().getTime();

  shipit.task('install', function () {
    shipit.remote('sudo apt-get update').then(function () {
      // We'll wait for the update to complete before installing some software I like to have on the
      // server.
      shipit.remote('sudo apt-get -y install apache2 emacs23 git unzip').then(function () {
        // We don't need the following set of actions to happen in any particular order. For example,
        // we're good if the disables happen before the enables.
        var promises = [ ];

        // We couldn't copy this file earlier because there isn't a spot for it until after Apache is installed.
        promises.push(shipit.remoteCopy('paperquik.conf', '/etc/apache2/sites-available/'));

        promises.push(shipit.remote('sudo a2enmod expires headers rewrite proxy_http'));

        promises.push(shipit.remote('sudo a2dissite 000-default'));

        promises.push(shipit.remote('sudo a2ensite paperquik mdm'));

        // But we do need this to wait until we've complete all of the above. So we have it wait until
        // all of their promises have resolved.
        Promise.all(promises).then(function () {
          shipit.remote('sudo service apache2 reload');
        });
      });
    });
  });

  // This shipit file doesn't yet use the official shipit deploy functionality. It may in the future but
  // this is my old sequence and I know it works. Note: I also know theirs seems like it might be
  // better because it can roll back and I definitely do not have that.
  shipit.task('deploy', function () {
    shipit.log('Deploy the current build of PaperQuik.com.');
    shipit.local('grunt build')
        .then(function () {
          return shipit.remoteCopy('dist/*', '/tmp/' + tmpDir);
        })
        .then(function () {
          shipit.log('Move folder to web root');
          return shipit.remote('sudo cp -R /tmp/' + tmpDir + '/*' + ' /var/www/paperquik')          
        })
        .then(function () {
          shipit.remote('rm -rf /tmp/' + tmpDir);        
        });
  });

  shipit.task('upgrade', function () {
    shipit.log('Fetches the list of available Ubuntu upgrades.');
    shipit.remote('sudo apt-get update').then(function () {
      shipit.log('Now perform the upgrade.');
      shipit.remote('sudo apt-get -y dist-upgrade');
    });
  });
};

Sorry for the small text above, the line wrapping was bad if I didn’t reduce it. Here’s the original over on Github. The huge and most obvious difference here is that Shipit wants to do all of those Apache configuration commands in parallel. So I let it. I just added a little bit of code to delay restarting the server until all of them have been completed (you can see that around line 37 above). Likewise the deploy and upgrade tasks want to execute steps in parallel and I can’t always let it do that. Since all of the asynchronous actions in Shipit return promises I just added a little bit of code in each task where I need to control the order in which things happen and it works.

Pros:

  • Executes commands within a task in parallel to achieve maximum speed.
  • Allows you to specify multiple servers and will allow you to run tasks simultaneously against all of them. Not anything I need at this time, but you never know when a project could grow from one server to two.
  • Supports tasks which run other tasks (or which broadcast/sink events). Thus dependencies for tasks can be handled.

Cons:

  • The documentation. Seriously, come on. I’m going to have to contribute to this project just to fill out the documentation some.
  • Harder to structure serial commands which need to execute in a particular sequence.

Thoughts

You see what I mean about people rebuilding the same tools over and over again just using different languages. Both Shipit and Flightplan claim similarity to previous tools for Ruby and Python. However, at the same time I have to confess I don’t find either of those particularly appealing to use when all I use day to day is JavaScript. I used Java for over ten years and I still don’t want to do all of my build and deployment with Ant. When I wanted to control the order of the asynchronous events in Shipit, it was nice that I could easily see how to do that from my experience with JavaScript promises in AngularJS and Node.js.

Both tools allow you to run tasks against multiple servers simultaneously. Both allow you to have multiple sets of servers so you can have staging servers or, if your just playing around like me, a Vagrant server you bring up and down just for testing purposes. Either could probably do your administration jobs, but I just liked Shipit a little bit better because it seemed more powerful. Going forward I’m going to probably pull the Flightplan files out of the master branch of my projects and leave them up only for reference from this blog post. Now I just need to see if I can do something about that Shipit documentation.

Advertisements