Tag Archives: JavaScript

Ideas are like Legos

Want to learn about how I think? No. No, you don’t; but I’m going to tell you anyway. Every day that goes by I add to this long list of stuff which interests me. It is neat from a technical standpoint, or it allows me to do something I didn’t know how to do or didn’t want to figure out for myself, etc.

Then I let those pieces rattle around in my head until something occurs to me about how I could combine them with what I already know to make something interesting.

So here’s a brain dump of all the stuff rattling around up there right now:

Tools

  • Map/Reduce
    • pjs
  • Flow
    • NodeRed
    • NoFlo
    • dat

Browser Compatible Libraries

  • OAuth
    • hello.js
  • Encryption
    • TweetNaCl
  • QR Codes
    • qrcode.js
    • jquery.qrcode.js
  • Color
    • randomColor
  • Generate Files
    • FileSaver.js
  • Data
    • TingoDB
  • Markdown Editor
    • EpicEditor
  • Communications
    • WebRTC
    • Socket.IO
  • Genetic Algorithms
    • genetic-js
  • Game Development
    • Phaser.js

Server

  • Authentication
    • Passwordless
  • Framework
    • MEAN.IO/MEAN.JS

“That’s a mess of divergent crap you’ve got there John,” you might say; and you’d be right. But it’s kind of like looking at a huge pile of Legos. What do you see to build when you look?

About these ads

I’m Open Sourcing Two AngularJS Projects

By far, the most successful open source thing I’ve done in years is the project I called airquotes. It was my first project built using AngularJS and I published it early on to give others a chance to see something finished which had been built using it other than a to-do list.

airquotes on Github

Since then I’ve built some other projects outside of my day job using AngularJS and though not particularly profitable they are diverse (to say the least) and I’ve decided I’d like to open source them as well to see if they can help people.

The first up is PaperQuik (PaperQuik.com). It’s an app which asks a few simple questions and then generates a printable sheet of paper (lined, dot paper, graph paper, etc.) in the browser. Unlike most sites like this, it doesn’t just have a canned set of PDF files it dispenses, nor does it have a server process building them. Instead it uses the HTML5 canvas to draw an image of the paper and then helps you print that image.

PaperQuik on Github

The second project is ClearAndDraw (ClearAndDraw.com). It’s a simple webapp that I threw together in just a few evenings because I wanted to keep track of my cards and dice for the game Marvel Dicemasters: Avengers vs. X-Men. It’s not nearly as complicated as the paper generation in PaperQuik, but it does show real time filtering using AngularJS and it stores all of the information you give it in localStorage of the user’s browser so it doesn’t forget anything they enter.

ClearAndDraw on Github

Neither of these projects has any back-end at all, they are served up strictly as a set of static HTML, CSS, JavaScript, and images and do all of their work client side. That’s not to say that I don’t want to build a back-end; ClearAndDraw.com in particular cries out for one to be added so users can enter in card/dice information and then retrieve it from any browser and any machine, rather than always having to return to the same place previous cataloging was done. But the initial solution was simple and worked as a starting point. It also presents an example of how a site might save data locally even for unregistered users and then later save that to a back-end data store if the user does create an account later.

I also took an evening and updated airquotes to the current version of AngularJS (1.2.25) and deployed it to a GitHub page so people can play with it without having to deploy it locally (like PaperQuik and ClearAndDraw).

A couple of alternative ways of data preloading in AngularJS

Gabe Scholz recently wrote the blog post: Frictionless data preloading in AngularJS If you want to skip my explanation and go straight to the code, here’s my version of the same idea: http://plnkr.co/edit/23u3BI?p=preview I’ll be honest, I wasn’t in love with the solution presented in the blog post. We don’t actually do preloading for our work at my current employer, we just load the page and do one (or many) calls to populate the page. But even so, I’ve still given the idea of preloading some thought and all of the solutions which were presented in this article left me thinking that they didn’t make good enough use of the built in object creation and injection capabilities built into AngularJS itself. So, if you check out my solution you’ll see that I create an AngularJS constant object (with examples of that created in either the index.html file or one of the JavaScript files, depending upon your preference) or I create an AngularJS service which returns a promise which is resolved with the data immediately. The latter solution has the advantage of working well if you find yourself wanting to sometimes pull from a service, sometimes use canned data, or even sometimes pull from a cache like localStorage in the browser. By separating it out to another object which is simply injected into the controller, you’ll have lots of options of how you want to inject your data and you don’t have to create any new directives to do it.

Why we abandoned server generated web pages

1. You’re constantly transporting state back and forth between client and server (and often neglecting to do so).

The next time you’re in GMail, notice something interesting about selecting emails. If you check several emails for a mass operation and then realize you’re not altogether sure about one email, so you click to read it, then you come back out to the list, the same emails are still checked. Now do the same test in an app written in any of the classic Java, PHP, Python, etc. frameworks. In those, when the user clicks on the email to view it:

  1. In all likelihood it was just a link so all of the client-side state (the checked and unchecked checkboxes) is immediately discarded by the browser.
  2. Some “clever” frameworks make that link not really a link, instead, they generate JavaScript client side which invokes a POST (as if a form was submitted), the checkboxes do get submitted to the server, stored somewhere in the user’s session, and then hopefully sent back down with the next request for the page that had the original list of checkboxes. Our links aren’t really links, there’s lots of JavaScript magic going on behind the scenes that most people you work with don’t even begin to understand and, trust me, trust me when I say this… It breaks. And it breaks badly.

2. It’s slow, slow, and did I mention slow?

  1. The development cycle itself is slower. I can throw up a temporary API using a variety of tools (or mock it client side using something like $httpBackend) and start work on the UI immediately. As I make each change I just have to refresh the browser to see my change in place. In fact, if I use tools like the Grunt server the browser refresh is automatically triggered when I save a changed file so I can just glance over at it and evaluate the results without leaving my code.

    Most server side frameworks involve a compilation step of some type (for example, Java’s JSPs are translated to servlets and then those are compiled to a byte code for the VM). Thus I have a longer wait to view each change I make.

  2. If my UI is built client-side with HTML, CSS, and JavaScript then it’s way more efficient because the only data which is being transported back and forth is that sent via the API calls to the server. I don’t have to ship a complete list of all the products wrapped in HTML layout one minute, and then minutes later what is effectively the exact same list (minus one or two and plus one or two) again wrapped in a bunch of HTML formatting. Note: Some of this can carry over to content delivery networks as well because an all static files front-end works really well around the world.
  3. Users get tired of every thing they click upon meaning another trip to the server. If they just looked at that data a minute ago, it may well be in memory locally so showing it again is free. But if I have to go to the server again to get it re-rendered for re-display, I can get really old fast. I think we can all point to a site where we are frequently frustrated by the slow performance.
  4. And why is it slow? Well, because it’s not just persisting data and performing queries on it, it’s also combining that data with HTML templates of some flavor to generate full pages and doing so over and over and over again. More work for the server means more servers needed, more time spent on performance tuning, etc.

3. It’s a more complicated programming model.

  1. If you do a lot of web applications where the pages don’t use any JavaScript to pull data on the fly, validate user’s forms in real time, etc. then that’s great. But if you are then you’ve got a wonderful hybrid going where sometimes you work in one language server side but you also use JavaScript client-side and you’ve got two different models just for one UI.
  2. For many frameworks, lots of middle tier complexity can leak into the pages, making them way more complicated than HTML and thus much harder for designers (and programmers) to work on. I’ve seen this most often in frameworks where designers are expected not to use standard HTML. Instead they’re supposed to use perfect XML where every tag perfectly matches a closing tag and not a bracket is out of place, or they can’t use the <a>, <form>, <input>, etc. tags, instead each has some replacement which is supposed to be used which typically functions about 70% or so the same as the regular tag.
  3. I loved this recent quote about JSF, a technology that I rejected at a previous employer because it was very clear that it was designed by committee and not extracted from real projects (like say Ruby on Rails):

4. Debugging is much more complicated.

Something as simple as how some HTML is rendering may require me to setup a debugger on a server on a remote machine because it is being generated from an intermediate file and data on the server.

With AngularJS or similar JavaScript frameworks, I can first look to see if the data came to the browser without problems. That involves just looking at the JSON I received from my API calls. If that’s good then I can set breakpoints in the JavaScript in the browser to see how the JavaScript code flow is going awry. On data flowing from client to server I can usually just consult the browser to see what was sent and I again know whether I’m looking at a client or server problem.

Software Rants

Mine is not a real software rant, I wrote it tongue-in-cheek after reading one just this week which spun up a bunch of people. Like most of the others, I skipped gracefully past some problems:

  • I still have non-API uses for servers. If I need a CSV or XLS file for download, it’s still way easier to have that generated on the server than to try and craft it client side with JavaScript.
  • I glossed over areas where the server generated web pages have advantages (for example, if your users are developmentally disabled and insist on staying on IE 7 or keeping JavaScript turned off).
  • I also skipped over the fact that there are browsers like older versions of IE where the debuggers are very poor and debugging can be just as painful client-side as it is for server-side.

Every framework that has achieved some level of popularity or notoriety has had its share of famous rants (Rails is a Ghetto, Node.js is stupid, and if you use it, so are you!, Why we left AngularJS which has since been renamed to 5 surprisingly painful things about client-side JS). Somebody doesn’t like the language, the framework, the community which goes with it and in frustration they vent. Sometimes they’re right about their complaints and sometimes they’re wrong, often it’s somewhere in between. Just relax and try to read it with an eye to whether the points being made are good ones and ignore the vitriol.

“selfhelp” – a self-upgrading Node.js application

I have a strange little project I’ve been working on over the last few weeks. It’s an example app I wrote using Node.js which is capable of self-upgrading itself to new versions. Thus you could use it as the basis of a cross-platform application with a browser based UI like SABnzbd+, the Plex Media Server, Couch Potato, and Sick Beard. All of which work on different operating systems (Mac OS X, Windows, and Linux) and which offer the same UI across all of them. So, this isn’t really intended for use on servers as much as it is intended for use on desktop machines so you can build apps using Node.js and whatever front-end technologies you happen to fancy (my particular flavor is AngularJS, though there is none in the example application).

I got it to where it works well on Mac OS X so I thought I’d write about it and maybe someone else would find it interesting enough to see if they could add a Windows batch file to it, maybe test it on Linux, or see if it suits a project they have been wanting to build. It’s called “selfhelp” and it’s available over on Github:

https://github.com/JohnMunsch/selfhelp

6 Tips for building JavaScript apps

I’ve actually built a few JavaScript applications in the new style (AngularJS, Backbone.js, or other front-end JavaScript framework on the front-end and only APIs on the back-end) over the last couple of years. Here are some tips on what I think has worked well on those projects:

  1. Understand this, above all else, the front-end code is not real security! If you’re an American you can understand this via an analogy. The JavaScript code running in the browser is the TSA, it is security theater which exists just to make some user’s experience better. For example, it might hide buttons which the user is not allowed to click. But that doesn’t mean that the user cannot hack the JavaScript to turn on the forbidden button anyway. All of the real security in your application exists at the API layer. It must check every single value passed to it and confirm that the user has the permissions to perform the action he/she is trying to perform before actually doing anything. Likewise, it must not return any information which the logged in user should not have access to. Relying on the JavaScript code to hide part of the data will not work. Put all of your security focus on having a bulletproof API and you will never have real security problems.
  2. People use HTTP error codes to communicate back data for their APIs. In my opinion that’s a really bad idea and often not very adaptable to the actual errors you’re having. Instead use the JSend protocol for all the JSON you return. It’s the same objects you would probably send back today except that it is wrapped with an object that tells you status (‘success’, ‘fail’, or ‘error’) and messages/codes when appropriate because there were errors. Going this route will simplify your JavaScript service calling code and help you differentiate API errors from actual transport layer problems like servers being down or problems on the network.
  3. Don’t try to sequence operations from the front-end. I once answered a question on Stack Overflow where the asker wanted to know about how to sequence a seven step process for paying for something. I answered it once telling how to do it and then again to say never to do that. You should not have your front-end be the conductor and the back-end be the orchestra. If you do, you will be sorry because eventually someone will lose their web connection, close their laptop, or just shut down their browser in the middle of your carefully choreographed sequence. Instead, always try to make API calls from front to back that provide complete units of work, complete transactions with all the information needed for multi-step operations so you won’t end up with only part of an operation completing.
  4. Please, please, please, please don’t do things that break basic conventions in your apps. There’s no reason the user shouldn’t be able to hit the back button or the forward button. It requires very little thought to support (especially if you use modern JavaScript frameworks). Ditto bookmarks and multiple tabs. There shouldn’t be any reason I can’t copy a URL and send it to somebody else or make a bookmark of my location so I can get back to the same spot. Nevertheless, I’ve worked on so many apps over the years where these basic operations acted weird or wouldn’t work at all. Don’t be one of those apps.
  5. Spend some time thinking about what happens when the user sits on a page so long his/her session expires on the server. If you’re following suggestion two above then you can send back a standard error in your JSend and catch it in your JavaScript code. Then just prompt the user to login without ever leaving the page. Likewise, think about what happens when the user clicks on a bookmark in the browser or an email and goes to the site but is not yet logged in.
  6. Please, don’t be afraid to reject some ancient browsers. There’s good code out there to help you do it and make it look nice, but ultimately you’re doing yourself, your users, and everybody else a service if you refuse service for IE 6/7/8 and maybe more than that depending upon your needs.

A post-mortem on my first AngularJS project

Before I started my current employer down the road to AngularJS I decided to do a small (but complete) project using it just to see what it felt like. I had a colleague from my previous job who needed a website at precisely this point in time when I needed a project to do. It was small, self-contained, downloaded all of the data it needed at the time it initialized (so I wouldn’t have to start using AJAX with AngularJS and come up with a service layer when he didn’t already have one in place). In other words, it was almost like it had been designed for what I needed to learn.

It was to quote prices for custom t-shirts and it had some fairly complicated calculations it needed to do based upon shirt cost, shirt color, number of colors printed front and back, quantity of shirts, and surcharges for large shirts. Here’s the finished UI:

The interface as the user first sees it.

The interface as the user first sees it.

His original design called for buttons the user clicked to get a quote generated and to clear out a quote when the user wanted to get a new one. I wanted to see if I could use AngularJS to generate two versions of the same website using the same controller; one would have the buttons of the original design, the other would simply update all of the prices in real-time as the user selected colors, sizes, etc.

When I was done I had a couple of functions and flags the “quote button” version of the site used that the real-time update version of the page didn’t need but otherwise the same controller worked quite happily with two different versions of the UI (the original and my design). Needless to say, the real-time quote updates version of the site was much less clunky to use and is the only version of the source code I’ve made available on Github. My colleague (who also provided me with the web design) preferred it as well.

Prices and the quantity discounts update in real-time as the user changes colors, printing colors front and back, quantities, etc.

Prices and the quantity discounts update in real-time as the user changes colors, printing colors front and back, quantities, etc.

If you want to download the source code to an AngularJS app that isn’t another To-Do List or something to pull down Twitter tweets then feel free to check out this project here: https://github.com/JohnMunsch/airquotes

Now for the retrospective part of the blog. Surprisingly, there aren’t a huge number of things about the code that stand out as glaring mistakes almost five months later, despite spending hundreds of hours working with AngularJS since. Given how much I’ve learned in other parts of the framework, I think that speaks well to how much I could learn in a short time to get the project done and done fairly well. But that’s not to say that it’s perfect. Three changes I could pinpoint off the top of my head were:

  1. All of the shirt data and pricing data were just dumped as JSON into a file and used as global variables by the controller I built. Now I know that I could put them into an AngularJS service and inject them into the controller. That would be cleaner, but you won’t see it that way in the current code.
  2. I need to do another view to demonstrate how the user could flip over to a catalog of shirts in order to pick a different type of short-sleeved shirt, long-sleeved shirt, crop-top, etc. and then that would change the route to pick a different shirt. The routing for a selected shirt is in place, but the picking page isn’t. The fellow I was working with said he could provide that, but I think my example would be better if I went ahead and put one in.
  3. I do a lot of calculations and sub-calculations to bfigure the price and to figure the price at different price-break levels for the display. That code should really be unit tested. I actually did something about this recently, read on for details.

So I recently returned to the project with an eye to fixing a few things and the largest change I made was to use it to figure out how to do both unit tests and figure out how to know how much of my code was covered by the tests. The coverage report was, let’s just say highly motivating, when there were only a few tests in place so I spent a few hours writing a simple suite of tests to get me to 100% coverage.

So, if you’re interested, I can show you what was involved in getting unit tests and a coverage report added to an existing AngularJS project. First, if you haven’t already installed Node.js onto your machine, you should do so from the big green install button over here. Then run:

sudo npm install -g karma

The sudo won’t be necessary on Windows systems but on Mac OS X and Linux you’ll want it because we’re trying to install Karma, our unit test runner, globally so you can use it from any of your projects. Once you’ve done this once, you shouldn’t have to do it for each project.

npm install karma-coverage --save-dev

This gives us a Karma component which can check the percentage of our code which our tests reach. It’s uses the Istanbul project and it will show you at the line level exactly which lines are getting hit by your unit tests. The –save-dev adds it to the dev-dependencies part of your package.json file so npm knows it’s only a development dependency and not a runtime dependency.

bower install angular-mocks --save-dev

They’re different tools but both bower and npm recognize the –save-dev flag and use it the same way. In this case angular-mocks becomes a development dependency in your bower.json file. It will help us when we’re building our unit tests for AngularJS code.

karma init

The installation parts are done, now it’s time to add unit testing to a particular project. This will get Karma to ask you a series of questions and generate a configuration file. With a few tweaks to that file you’ll be ready to run unit tests and get code coverage information for your tests.

Karma init options.

Karma init options.

This screenshot shows me setting up a configuration file the same way I set up mine for this project. I picked Jasmine for writing the tests (other choices are Mocha and QUnit), I don’t use RequireJS, and you’ll notice it gave me an error message for no matching JavaScript files. That happened in this case because I was doing this test run in an empty directory, you wouldn’t see that normally.

If you actually open up the config file Karma generated at this point you’ll find it pretty easy to read. It has nice comments so you can tell where it wants you to put library files, the files you want to test, the browsers in which you want to test, etc.

You’re going to make two edits to the file at this point to get some things we want in there. This should be a one-time thing again.

There’s a files array part way down the page. Make sure you add the path to the angular-mocks.js in there.

And you’ll need to add a preprocessors section like you see below so the code coverage tool knows exactly which source it’s checking for test coverage and add ‘coverage’ to the list of reporters to turn on the coverage checker.

preprocessors: { 'app/scripts/**/*.js': 'coverage' },
reporters: ['progress', 'coverage'],

Note: If you use the latest angular-generator together with the latest Yeoman it seems like the generator now creates a Karma config file (karma.conf.js) and installs the angular-mocks so you can skip those two steps for your new project if you’re using Yeoman to generate it.

Finally there’s the writing of the tests themselves and the checking your coverage to see where else you need to craft tests.

To kick off this process, run “karma start” at the command line and Karma should now kick into gear, start up the browser or browsers you wanted your tests to run in, and start watching all of your source files for changes. This is super handy because you can save a change to a source file or a test file, Karma will notice that and rerun your tests so you get immediate feedback on whether everything is passing or not.

When I had only one test completed, this is what the coverage report looked like. Look for it in a subdirectory under the coverage directory after you've run some tests through Karma.

When I had only one test completed, this is what the coverage report looked like. Look for it in a subdirectory under the coverage directory after you’ve run some tests through Karma.

I’m not going to give instructions on writing Jasmine tests or even the specifics of writing them for AngularJS. There’s good examples for Jasmine on their website and I have to confess that most of the tests I’ve written so far focused on testing JavaScript rather than AngularJS specific items like services and directives. So I still have some learning yet to do. But for basic tests, if you look at the test/spec/airquoteSpec.js you can see 20+ tests I wrote for my code to achieve complete coverage of my main.js file.

With tests written for all the code in main.js there's lots of green and 100% coverage indicators at the top of the page.

With tests written for all the code in main.js there’s lots of green and 100% coverage indicators at the top of the page.

Good luck, I hope some of my code is helpful to you in your own projects, and feel free to ask questions about any of it.