Turbulent World

In web development, trends quickly come and go. For a time, Ruby on Rails was all the rage; now the hype has shifted away from the framework and is focused on JavaScript, the only programming language to run on the client and on the server, if you subscribe to the credo of JavaScript's followers.

Server-side JavaScript frameworks have been around for a while, such as Helma [1], on which the website of the Austrian Broadcasting Corporation (ORF) is based. Helma is slowly being mothballed, but a successor named Ringo is already in the starting blocks. Projects like these, however, have only ever been niche products compared with Rails and PHP frameworks.

The situation is different with the Node.js platform (or Node for short) [2], which has been extremely popular for several years and has attracted a large community of developers. Node was developed by Ryan Dahl, who coupled Google's V8 JavaScript compiler with some non-blocking C libraries whose functions are available in JavaScript. This approach allows event-based processing of work requests and, thus, more concurrent requests than the Apache server can handle – although Apache also has an event-based processing module [3] in its more recent incarnations.

Spoiled for Choice

One of the strengths of Node.js, and a result of its massive popularity, is the huge number of modules that extend its functionality. The hype now goes so far that Node is no longer limited to web applications but is even used for classic system tools. Thus, the administration tools in SmartOS [4] are based on Node – which is perhaps less surprising if you know that Joyent, the company behind SmartOS, is involved in the Node project and sees itself as the "steward" of the project. Even Microsoft has released a tool that is based on Node [5] for managing the Azure cloud via the Linux and OS X command line.

Installing Node is pretty simple. Most Linux distributions now have it in their repositories, although you will rarely find this to be a current version. It is better to download the source code, unpack, and then run make and make install to build and install the program. You need various development packages in place for this, such as Libev. For Ubuntu, current Node versions exist in the ppa:chris-lea/node.js repository.

After the install, you will find two executables, node and npm, on your hard drive, typically in the /usr/local/bin directory. The modules are stored by default in /usr/local/lib/node_modules/. Besides Linux, Node runs on OS X, (Open)Solaris, and Windows.

As indicated, Node is primarily a JavaScript compiler/interpreter with additional libraries. If you launch node at the command line, you are taken to a prompt where you can enter JavaScript commands and functions:

> "admin".toUpperCase()
'ADMIN'

Programmers typically work with Node just as with any other scripting language: You write your code in a file and execute it with node. Because Node reveals its capabilities especially as a web framework, the classic Hello World example is a small web server that is programmed in a few lines. The http needed for this module is loaded in typical JavaScript style using require and is bound to a normal variable:

var http = require('http');

The createServer function of the http module expects a function that is executed when a request event occurs. In Listing 1, this takes the form of an anonymous function with one parameter for the request and one for the response to the client. In the function body, the writeHead function first writes the header and then expects the response code and an optional hash for the headers. The end function usually stipulates the end of the header and body, which is provided by the write function. In this case, end does everything in one fell swoop if you pass in the payload data to it.

01 var http = require('http');
02 http.createServer(function (req, res) {
03    res.writeHead(200, {'Content-Type': 'text/plain'});
04    res.end('Hello from Node.js\n');
05 }).listen(3000, '127.0.0.1');

Suppose the code in this example resides in a file named server.js; in this case, a call to node server.js launches the server on port 3000. On this basis, you could start to build your own web server by incorporating case distinctions into the request method that jump to other functions depending on the URL. This process is quite a chore, so Node users have posted several modules that simplify URL routing. The most popular is Express [6], which is used in most web applications and offers even more practical capabilities. An alternative would be Director from the Flatiron project, for example.

Node Packages

External modules are installed using the NPM package manager, which is included in the Node distribution. You can install a package manually with a call to npm install module. The modules end up in the node_modules subdirectory below the current directory. System global Node modules are installed with -g, but this technique is usually frowned upon unless you are installing a module that contains an executable command.

The idea is that each Node application can work with exactly those versions of the modules that it needs, which are not necessarily identical to those of another Node application. If you use require to integrate a module into a Node application, Node examines not only the current directory but also all higher levels, up to the root directory. This means you could also maintain a repository of Node modules available to multiple applications if desired.

After installing the Express module by typing npm install express (Figure 1), you integrate in the normal way using var express = require('express'). Listing 2 shows how to structure your web application with Express. An Express application is generated by calling the express() function, which acts as a switching point from then on. Routing takes place in line 11. A link between the URL and the calling function is created by calling the Express method with the same name as the required HTTP method.

01 var http = require('http'),
02     express = require('express');
03
04 function root(req, res) {
05    res.end("hello");
06 }
07
08 var app = express();
09 app.set('port', process.env.PORT || 3000);
10
11 app.get('/', root);
12
13 var server = http.createServer(app);
14 server.listen(app.get('port'), function() {
15    console.log('Express server listening on port ' + app.get('port'));
16 });

In the example, the call to app.get() means that when the / URL is called by HTTP GET, the root function is executed. To assign an action to a HTTP POST request for the same address, you would use app.post() instead.

Again, the server runs on port 3000 if you launched the program by typing node app.js, for example. If you then use your web browser to access http://servername:3000, you will see the Hello World message.

Templates

Outputting HTML in such a primitive way also works, of course, but far better ways are available, namely templates. Node and Express also offer a variety of template engines that you can use to define the page layout, which is filled with content at run time. The Jade templates, for example, which completely do without angle brackets and closing tags  – instead using the indent level for structuring  – are extremely compact. A Jade template can thus look like this:

html(lang="en")
  body
  h1 Hello

As you can see, this approach saves a huge amount of typing. Node converts the Jade code to HTML in real time when you call the render function of the response object with the template file as its first parameter. Express refers to the suffix to determine the template engine. If it is missing, Express uses the default engine, which defines a call to app.set('view engine', 'jade'). If you don't want to learn yet another language, in the form of Jade, you can use, for example, Swig for a template syntax that enriches classic HTML with variables [7]. By default, Express applications look for templates in the views subdirectory. If you want to define a different location, you can do so with app.set('views', directory). The global __dirname variable of a JavaScript Node file is useful here. It contains the directory in which the file is located.

Of course, templates only become really interesting if you also pass variables to them; otherwise, you could write static HTML files. In Swig templates, variables are enclosed in curly brackets on the left and right. Filters, loops, functions, and template inheritance can also be used. The latter is interesting if you want to outsource, for example, the HTML header or visible footer of a page to a single file and then use it on all your pages.

To transfer the variables from the Node code to the template, you pass them in as JavaScript objects. The corresponding code is shown in Listing 3. If this looks strange, remember that the first occurrence of firstname is the keyname of the object, whereas the second is the JavaScript variable of the same name. In the Swig template, {{firstname}} outputs the corresponding variable.

01 var firstname = "John";
02 var lastname = "Doe";
03 app.get('/', function(req, res){
04   res.render('index.html', { firstname: firstname, lastname: lastname });
05 });

The data in Node applications usually originate from NoSQL databases, such as MongoDB, for which matching NPM modules exist. However, classic relational databases such as MySQL, MariaDB, and PostgreSQL are also supported. If you look around on the web, you will continually find interesting projects that depart from the familiar framework. An example is Bookshelf, which is an object-relational layer for MySQL, PostgreSQL, and SQLite that supports transactions and JavaScript. It promises to provide an unusual but powerful programming model for asynchronous processing of requests and queries.

Structure

Basically, neither Node nor Express demands a specific structure for web applications. This feature could also be understood as a weakness, because as a beginner, you may initially find yourself somewhat disoriented. However, when you install Express with the -g (for global) switch, it also gives you a command-line tool of the same name. It creates the public directory for static files, routes for routes, and views for templates. The main function of the application resides in the app.js file (Figure 2).

In the figure, you can see another aspect of NPM package management  – that is, the package.json file. If you develop a Node application manually, you do not have to use it, but it is recommended for structuring your projects. It contains the metadata of a project and the packets needed to run it. Yet another section contains the packages needed to "develop" the project – more on that later. If package.json is complete, you just need to call npm in the same directory for NPM to install all the packages listed there, including their dependencies.

The npm command also offers two practical switches that save the developer from entering each required package manually in package.json. Using

npm install package --save

enters the package to be installed in package.json in the dependencies section, whereas

npm install package --save-dev

puts the package in devDependencies. Listing 4 shows an example of an NPM package file of this type.

01 {
02   "name": "random",
03   "version": "0.0.1",
04   "private": true,
05   "scripts": {
06      "start": "node app.js"
07   },
08   "dependencies": {
09     "express": "*",
10     "swig": "*",
11     "consolidate": "*"
12   },
13   "devDependencies": {
14     "grunt": "~0.4.1",
15     "grunt-contrib-copy": "*",
16     "grunt-contrib-clean": "*",
17     "grunt-contrib-uglify": "*",
18     "grunt-htmlrefs": "*"
19   }
20 }

Development dependencies include just the Grunt package [8], along with a few of its subprojects, about which I also need to say a few words. To structure the development process, JavaScript or Node developers show an increasing tendency to use build tools, as known from compiled languages, such as Make/CMake from C/C++, or Ant/Maven from Java. In principle, the compile step is dropped in JavaScript; Node handles this for server code, whereas the user's web browser does the same for client code. However, there are quite a number of steps in web development at which projects can be optimized thanks to "compilers," or preprocessors; that's where tools like Grunt enter the game.

For example, many meta-dialects exist for CSS stylesheets, such as SASS and LESS, that introduce includes and variables like the HTML templates mentioned previously or abstractions for the differences between different browsers. This approach greatly simplifies the task of maintaining style sheets in larger projects. At some point before delivery to the browser, these dialects must be translated back into classic CSS, either at run time or offline using a build tool like Grunt, which in turn calls the appropriate preprocessors.

The same goes for JavaScript code that needs to meet different requirements for development and production: In development, the focus is on structuring, readability, and maintainability, but when the project moves into the production phase, it shifts primarily to performance. It goes without saying that the code should be error-free to the greatest extent possible in both cases.

For web projects, performance not only means efficient algorithms but also using small files and as few as you can effect. You typically need to find a compromise that is defined by the specified key data (maximum number of concurrent HTTP connections per browser/server combination) or that can be determined experimentally. Grunt also supports this process by allowing the developer to integrate various tools for compressing (e.g., UglifyJS), merging, or otherwise transforming JavaScript files.

Future

Grunt offers many opportunities, but it also seems a bit strange to reverse-engineer a program like GNU Make in JavaScript only to then have to formulate its tasks in a fairly awkward way in JSON. Every community, however, is probably doomed to reinvent its own very special wheel. This, therefore, conveniently brings up the biggest weakness in the Node world, which is also its greatest strength: its enormous enthusiasm, which unfortunately dies out as quickly as it flares.

Grunt is one example. Although it was recently hailed as the greatest invention of all time, it is already regarded as clumsy. The new emperor in Node-build country goes by the name of Gulp, if you believe the propagandists [9]. However, nobody knows what developments next week will bring. This situation in itself is not bad, but if you are new to Node, the huge selection of packages and frequently changing favorites can lead to despair.

In this article, I only touched on the topic of programming complex applications with Node. Next time, I will look at circumnavigating the pitfalls of asynchronous processing and helping components communicate with one another via events.