JavaScript Functions That Run Beautifully on Client and Server

One of the compelling benefits of Meteor/Node.js is that a single language, JavaScript, can be used on client and server.  Meteor developers use the word isomorphic to describe JavaScript code that has been written to run anywhere.  To make that possible, your JavaScript functions must adhere to certain coding conventions.

After some experimentation and false starts, we have devised helpful programming standards that foster isomorphic code.  I will share these standards with you through concrete examples.

I’m currently working on an MVP for a startup called Pract.us.  This system uses a variety of Meteor packages, plus some custom-written UI code to deal with forms, animations and so on.  In contrast to this “mechanical” code, which has nothing to do with the problem domain, the system has dozens of domain functions which are germane exclusively to Pract.us.  Many of these domain functions can be classified either as queries or transactions because they either read from or write to the database.

Domain functions are the heart of the system.  They encapsulate logic that updates the database in response to UI events.  A large part of the Pract.us development effort goes into writing and refining these domain functions.

In Pract.us, a UI event handler will usually delegate to one or more domain functions.  The domain function(s) will perform a number of database queries and updates, and some of them are complex, involving updates to several collections.

Pract.us domain functions are anchored in a global JavaScript object named Practus.  Because Pract.us is a small system, it is feasible to anchor all domain functions in Practus.

Domain functions are technically stateless because they have no instance variables.  They operate exclusively on passed parameter values and data stored in Mini-Mongo and/or MongoDB.

To facilitate isomorphism, we first declare a Practus anchor object in a Meteor project folder that is shared by both client and server:

Practus = {};

Next, we create three JavaScript files named practus.js: one in a client folder, one in a server folder and one in a shared folder.  All three files contain the following skeletal code:

/*
 * Pract.us domain functions.
 */
"use strict"
Practus = _.extend(Practus || {}, {
 // *** DOMAIN FUNCTIONS GO HERE ***
});

By using the Underscore extend function, we’re appending functions to the Practus anchor object.  Since shared functions are visible to both client and server, this results in the following arrangement:

  • Client
  • Shared Functions + Client Functions
  • Server
  • Shared Functions + Server Functions

Technically, only the shared functions need to be isomorphic, but we apply the same programming conventions to all domain functions. This allows us to move functions around without costly refactoring.

To invoke a domain function, we make a call via the Practus anchor. Example:

Practus.setCardOpen(cardId, true);

Since client, server and shared functions are all invoked via the Practus anchor, the impact of moving functions is minimized. This is particularly helpful for nested calls (i.e., when one Practus function calls another Practus function).

When you write domain functions, you must restrict your code to use only those services that are available on both client and server.  A client-side event handler may use jQuery to extract information from the DOM, then pass that information as parameters to isomorphic domain functions.

card = this;
$switch = $(event.target);
open = $switch.bootstrapSwitch("state");
result = Practus.setCardOpen(card._id, open);

Any jQuery calls must be performed outside the domain function, because jQuery calls inside the domain function would prevent it from being used on the server side.

In Pract.us, domain functions always return a result object that contains a success/failure indicator, an i18n message key, message variables and a message severity level.  Domain functions never throw errors, but instead catch-and-return them as failure-type result objects. The UI will render the response object to the user via PNotify, resulting in a pop-up message in the lower-right corner of the page:

pnotifyPNotify can be used to report results that are returned from both client-side and server-side domain functions.

When the UI calls a client-side domain function, that function will return the result object directly:

var result = Practus.rejectEnrollment(cardId);
UX.createAlertForResult(result);

When the UI calls a server-side domain function, that function will return the result object via a callback:

Meteor.call("rejectEnrollment", cardId, function(err, result) {
 UX.createAlertForResult(result);
});

Meteor.methods({
 rejectEnrollement: function(cardId) {
  return Practus.rejectEnrollment(cardId);
 })
});

PNotify is well-suited for reporting results from domain functions because it can work both synchronously and asynchronously. When the UI invokes server-side domain functions, there will be a delay before the results come back. Using PNotify, even if that delay is long, and the user has moved on to a new page, the message will be displayed properly once it arrives; thus, the UI can call server-side domain functions without locking the UI and forcing the user to wait for the response.

A well-written isomorphic function can be moved from client to server or vice-versa with negligible impact on user experience; the only difference will be latency.

Mini-Mongo makes isomorphism feasible because it provides a client-side API that is identical to the server-side MongoDB API.  In a data-centric system, most domain functions either query or update the database.  Since Meteor provides identical APIs on both client and server, most domain functions can be written isomorphically.

Although domain functions can run equally well on client or server, there can be tantalizing advantages to running them on the client.  First, there are UI performance advantages. UI events, such as toggling a switch, will trigger an immediate UI update as Meteor responds to Mini-Mongo state changes and re-renders the DOM locally without a server round trip (Meteor developers call this latency compensation).  Moreover, client-centric coding can reduce the server-side workload and costs to a level that would be difficult to achieve with a traditional page-oriented application.

When you write isomorphic functions, it will make things easier if you use Mini-Mongo and MongoDB to hold all of your state.  You must publish and subscribe collections carefully so that all necessary data is available in Mini-Mongo before attempting to promote server-side functions to the client. On occasion, I’ve tried to move functions from server to client only to discover that some of the data that those functions needed had not been published to the client. Such problems can be remedied by refactoring domain functions or changing the publishing rules. Domain functions may be split so that part of the work is done on the client, while other parts remain on the server.

With good programming standards and proper design, you can develop your domain functions isomorphically, allowing you to easily move code from client to server or vice-versa as you see fit.  The option to move functions with impunity can improve quality and performance, while simultaneously reducing development and operational costs.

Meteor Development Moved to Windows

For almost two years, I’ve been developing Meteor using several Ubuntu VMWare virtual machines running inside a Windows host; this weekend, I moved all of my Meteor development functions to Windows native, taking advantage of the new Meteor Windows support.

meteorwindows

You might ask: why would one do this?  The main reason is that it will smooth out some kinks in my development process.  Much of this has to do with Eclipse.  Many months back, I used to run Eclipse on Ubuntu (Unity) on VMWare, and I’d do all of my editing from within the Ubuntu operating system.

This approach had a couple of drawbacks: speed, and also VMWare Ubuntu limits you to only a single display.  This is a significant drawback with Meteor, because there is a huge efficiency advantage to having HTML, CSS and JavaScript side-by-side on separate displays.  Most Meteor developers tend to do this, because all three types of artifacts are interdependent, and you can see how they work together at a glance.  I’ll often rough out the HTML and CSS for a page, then start building JavaScript that refers to the HTML elements by IDs or classes.  These artifacts evolve in tandem, and with a single display you are constantly having to switch back and forth trying to remember the names of things as you make changes.

Because I desperately needed multiple displays, I was doing something kludgy:  I had created a network mount on Ubuntu (which contained my source code), then using Eclipse on Windows, I was able to map a network drive to that Ubuntu network share using Samba.  The effect was my Windows Eclipse would modify the Ubuntu-resident source code, and I could therefore use multiple displays.  In order to build or do other things, I’d switch over to Ubuntu, but editing was done on Windows.

This worked, and I’ve had this setup for several months, but there were some problems.  Some of the Eclipse plug-ins do a lot of I/O to the folders, and I think the Eclipse developers didn’t anticipate that developers would have their source folders on Ubuntu network drive.  Many operations, such as simply changing a line of JavaScript would cause the Eclipse client to hang for as much as 30 seconds.  This would not happen every time, so it was tolerable.  Also, checking into GitHub was extremely slow, a commit could take several minutes for a large check-in.  Lastly, the Eclipse syntax highlighting logic apparently does a lot of I/O to determine whether a JavaScript variable is local, argument or global.  This would mean that after making a change, I’d often have to wait 30 seconds or more for the system to properly color the variables.

Now, all of that waiting is history.  Changes to JavaScript are instant, and syntax highlighting is almost instant.   Moreover, my MongoDB is now on a native SSD, so performance is better (on my system, all VMWare machines are on a spinning hard drive because SSD space is at a premium).

I am now building Meteor on Windows.  I have created a bunch of batch scripts that support the build/release process, including scripts to automatically SSH or FTP builds to my test Ubuntu machine or EC2.

So, things are better, but not perfect.  Because Meteor Windows support is bleeding-edge, there are some warning messages that are generated during the build and install processes.  Apparently these messages are non-fatal, but they can give you pause because during those operations you tend to want everything to go perfectly.  I believe that when Meteor upgrades to a later version of Node.js these problems will be solved.  Currently Meteor is on Node.js 0.10.36, which is a couple of releases back (Node.js is now at 0.12.X).

I still have one mundane task left to complete this process, and that is to get my Windows MongoDB database set up as a replica set so that it closely matches production.  Replica set can really help Meteor performance, because Meteor can leverage the so-called OPLOG to get notified of state changes in MongoDB the second they happen.  Without a replica set, Meteor goes into a different mode where it polls for state changes.   It also spends a significant amount of time doing comparisons (before and after) of result sets to infer state changes.  None of this is needed with OPLOG enabled, so once this is done I should be getting snappy performance from my development environment.  We have OPLOG tailing set up on on all of our production instances, and I can attest it makes a big difference.

 

 

The Iceberg

iceberg

Beneath any modern software project lies a vast, invisible effort.  The typical user sees only the the tip of an enormous iceberg.

Seemingly-simple ideas can take longer to implement than one would expect.  Pushing an application into production requires many crucial decisions in disciplines ranging from systems design to engineering to security to performance management.

It takes balance and maturity to face the realities of internet projects without delusion.  Over-zealous belief in the market potential of an idea combined with the notion that one can get to market more rapidly than is humanly possible is a recipe for disaster.  Discipline, patience and time are the great levelers.

Reign of the Front End

Since my last post in January 2014, I have thrown myself head-first into the world of JavaScript, Meteor, Node.js and MongoDB, working on two major (i.e., paid) projects plus one unpaid project.

The first paid project is a system to support sales functions of an established firm in the nutritional supplements space.  The system is essentially an add-on to an existing CRM system. It uses the APIs to discover incomplete orders in the CRM system, then issues reminder emails to the prospects, and provides a convenient single-click method of completing those orders.

The second paid project is a multitenant SaaS in the eLearning space.  The goal is to create a minimum viable product (MVP) to serve as a beta, and potentially to attract seed capital.  This eLearning project has consumed most of my attention this year, and the challenges I have encountered are the impetus for this post.

In an earlier post, I suggested that Node.js and Meteor might yield an important indirect benefit: since JavaScript, a single language, is used for both client and server, JavaScript developers need not be pigeon-holed as back-end or front-end developers.  Because the Meteor framework reduces friction on the back end, more intellectual horsepower might be freed up for the front end.

This could not come a moment too soon.  Increasingly, modern websites are dynamic, responsive and reactive.  They behave like installed apps.  They push the limits of HTML5 and CSS3 leveraging complex stacks of JavaScript code to deliver the best possible user experience.  They use animations, drag/drop, collapsible sections, momentum scrolling, popovers, type-ahead boxes, layer sliders, image galleries, maps, video feeds and graphs.  Modern systems are expected to work equally well on touch devices and desktops, across all browsers.

We have little choice but to allocate more engineering bandwidth to the UI.  Applications that look like they were designed before the tablet revolution will go the way of the dodo bird.

Nine Months

In the nine months since my last post, I’ve been immersed in the front end.  I have come to see myself as a curator, scanning the globe for best-of-breed open-source offerings to facilitate our ambitious UX.  Our front-end stack currently includes:

  • Bootstrap 3-LESS
  • Bootstrap Context Menus
  • Bootstrap DateTime Picker
  • Bootstrap Switches
  • Bootstrap Tags Input
  • Jasny Bootstrap Extensions
  • Holder.js
  • jQuery Sortable
  • jQuery Touch Punch
  • Selectize
  • Amplify
  • Touchspin
  • Nearest

Evaluating open-source offerings can be time consuming.  Bootstrap and jQuery add-ons tend to be in a constant state of flux as developers struggle to maintain compatibility with other systems, while simultaneously dealing with bugs and feature backlogs.

There are competing offerings, each with strengths and weaknesses.  It can be helpful to first research what other developers are saying about the tools, and to take into account the sizes of their user bases.

It is necessary to install the tools and use them first-hand to prove that they will work harmoniously with the rest of the stack.  You need to dive into the JavaScript code, frequently single-stepping through it with a debugger to diagnose issues.  This process can disqualify a tool, resulting in a new round of searching and evaluation.

Be forewarned that choosing any tool will entail compromises: some features may work great but at the expense of others.  You have to pick your poison.

Meteor Blaze

The Meteor Blaze release has been lauded as the solution for reconciling Meteor with jQuery.  If you are not aware of the issue, it boils down to this: both Meteor and jQuery-based offerings want to control the document object model (DOM).  When conflicts occur, you must intercede.  Earlier versions of Meteor had template directives to allow you to declare certain areas of your web page constant (i.e., off limits for Meteor’s reactive updates).  This would allow you to use tools such as jQuery Sortable, albeit foregoing reactive updates for constant regions.

The Blaze release is widely believed to have reconciled Meteor with many jQuery-based tools, but it is only a first step.  The Meteor development team deprecated the constant template directive, implying that Blaze has solved the problem completely.  I can say with considerable authority that this is not true.  I have spent hours reconciling the behaviors of tools such as jQuery Sortable and Jasny Bootstrap with Blaze.  I have reported issues to the Meteor development team and they have graciously acknowledged them.

Notwithstanding, Blaze is an enormous improvement.  But there is still work to do to make it truly seamless, and until then UI developers will face extra effort, in some cases patching jQuery tools to get them to work correctly with Meteor.

Touch-Friendly

Desktop computers and touch devices have different event models.   jQuery Touch Punch provides a means to convert touch events into traditional events so the same code can work on desktop and touch devices.  But even when using Touch Punch, you have to be careful about the set of gestures your application supports.  For example, there is no obvious gesture corresponding to right-mouse-click on a touch device, even though right-mouse-click is the standard event to display context menu or properties on desktop apps.  This means that a modern web application will tend to avoid context menus, or at minimum provide a touchable icon (i.e., button) to display the menu, for example, a down-chevron, which seems to be all the rage these days:

menua

The down-chevron should not always be visible, but should appear only when the division that contains the menu is “hovered” over or touched.  I refer to this as a hover control.  This clutter-resisting design pattern has been made popular by Facebook.

There is no touch equivalent for double-clicking, so that gesture should be avoided entirely.

Other differences include the absence of movable dividers (sashes) in the touch world, and the fact that drag/drop gestures must be reconciled with scrolling: in the touch world the two gestures can be ambiguous unless explicit “handles” are designated for dragging.

Layout Challenges

A modern web site must work well on desktop and touch devices, and with all popular browsers at any screen resolution.  Bootstrap is helpful for coping with differences in screen sizes, but it is no panacea.  Media queries can help in limited cases by targeting CSS rules to particular devices, but complexity grows with each exception added; therefore, it is necessary to code your HTML and CSS carefully and test frequently to avoid changes that break the UI on one or more platforms.

Speaking with other UI developers, I’ve come to realize that we’re all in the same boat.  We’re all looking at our iPads wondering why our text doesn’t properly center on certain devices, or why our scrolling area extends beyond the right margin of the containing division, or why when iOS momentum scrolling is enabled we cannot drag items out of the containing division without the item being rendered invisible.  With each new problem, we will use Google to find hits that might give us more insight, or might yield a solution.  Time and time again, we will eventually find the magic recipe of CSS and/or JavaScript to overcome the problem.

Consider this: the popular website JSFiddle is loaded with tens of thousands of hacks to beat the browser into submission.   That such a web site is even necessary speaks volumes about the current frustrating state of affairs.

Occasionally, when I complain about this, some hot-shot UI developer will respond “we use LESS” as if LESS somehow ameliorates these issues.  The implication is that the problems are attributable to the CSS language; unfortunately, the problem isn’t the syntax of CSS, it is how browsers interpret the rules, particularly in complex cases with heavily nested divisions and dealing with issues like dynamic resizing, justification, visibility, scrolling and truncation.

We use LESS for all of our projects, specifically the advanced configuration offered by package Bootstrap-LESS.  But LESS doesn’t address the core issue: the difficulty of finding the magic set of rules that will yield the desired dynamic-resize behaviors.  Your pages must behave beautifully on any desktop or mobile device, at any size.

CSS requires discipline and continuous refactoring.  The never-ending battle to keep your CSS DRY (don’t repeat yourself) is thankless.  Unless you pay careful attention to class naming conventions and rule granularity, your CSS can spin out of control.  Seemingly innocent changes will begin to have unwelcome side-effects.

Lately, I’ve been adding detailed comments to my CSS.  Whenever I code a complex set of rules to overcome some layout issue, I write comments as clearly as possible to explain the intent of those rules, particularly in cases where the intent is to circumvent a browser bug.  When the CSS has to be changed, comments regarding the original intent can help minimize costs.

The Front End Remains the Same

Last year, I went on a quest to find the best full-stack framework to develop modern web applications.  I evaluated Ruby on Rails, Grails, ASP.NET and Meteor/Node.js.  I imagined that I would find the best framework, try to learn it inside-out, and then use it to develop exciting applications.

But there was one thing that I didn’t fully appreciate: regardless of which framework you choose, the front-end issues will always be there.  JavaScript, HTML5, CSS3 and all of the related challenges will be virtually identical.  You’re going to have to worry about responsive design, layout, advanced behaviors touch devices and browser compatibility.

If you are a full-stack developer, you’ll spend the majority of your time in the front-end world of JavaScript and jQuery, regardless of whether your back end may be PHP, Django, Rails, JSP, ASP.NET or Node.js.

In my view, this reinforces the notion that Node.js (and particularly Meteor) is a great choice for new systems.  Since there is no viable way to avoid JavaScript on the front end, why not go all the way and use it for everything?

When Will Our MVP Be Ready?

Notwithstanding great tools, project durations can be difficult to estimate, because no one can anticipate the snags that will arise.  One thing is predictable: a given project will take longer than one would wish. A modern web site that works beautifully on all devices and has cutting-edge behaviors will be expensive.

Today, one factor seems irreducible: the cost of research.  When you embark on creating a new subsystem that requires advanced behaviors, you’ll be forced into research mode to evaluate open-source offerings or custom coding.  When you run into a snag, such as unacceptable behavior on some device, you will have choices (1) ignore the issue (2) consult with a knowledgeable colleague (3) fight harder by Googling, looking at the source code and experimenting.

In research mode, it is difficult to estimate how long it might take to find a solution.  To estimate the cost, you must rely on your instincts, comparing each new challenge to similar challenges in the past.  Sometimes you get lucky, but more often you don’t.

There will be plenty of dead ends, because no one can make the right choices every single time.  New JavaScript offerings, platforms and services are constantly being introduced, and competitive pressures will compel you to learn about them and incorporate them into your applications.

Today, it may be drag and drop.  Tomorrow it may be streaming video.  The next day you may be working with 3D graphics in WebGL.  You’ll have to learn many APIs, and use them defensively with robust error control and graceful degradation.  Your system is not an island but an amalgamation of services, each of which is another potential point of failure.

We are all constantly learning.  What we don’t know today, we will learn tomorrow.  We must look at new challenges with a sense of wonder, with the eyes of a child.  We must take the time to help and support one another, to share what we have learned.

Move Over LAMP, Make Way for JAMM

After working heads-down with Meteor and MongoDB for four months, I haven’t run into any show-stopping snags.

You’ve likely run into the acronym LAMP which describes an enormously popular application architecture based on Linux, Apache, MySQL and PHP.  The combination of JavaScript, Meteor and MongoDB is so powerful and cost-effective, it deserves its own acronym.  So move over LAMP and make room for JAMM (Javascript, Meteor, MongoDB).

Working with these tools has made me appreciate scrappy JavaScript, particularly when combined with powerful add-ons like jQuery and Underscore.  JavaScript can be made modular, yielding key benefits of encapsulation and object-orientation (purists may argue but they’re splitting hairs).  My own fears about leaving the Java “tribe” seem unfounded: everything you need for a typical data-centric project is available, open source, lovingly crafted by a growing list of true believers in Node.js and Meteor.

I think we’ll see more and more enterprise-class, cloud-based applications written using nothing but JavaScript.  Many forward-thinking companies have been infected by this trend and have made enormous investments in JavaScript, albeit still using mature technologies such as PHP and AJAX.   With deep pockets, they’ve spared no expense on their front-end code.  Meteor lowers the bar, making JavaScript-centric, reactive front ends available to the masses at low cost.

But adjusting to these new tools can be disorienting, and I find myself comparing and contrasting JAMM to things with which I am more familiar.  Over the past decade, I’ve been working on rule-based systems that conform to three-tier architecture.

Three Tier Architecture

With this architecture, the client tier, typically a web page or hand held device, communicates with an application server that comprises the middle tier.  The middle tier contains business rules that operate on a domain model (i.e., object instances that contain enterprise data).  In most cases, the domain model is mapped to a database, and in the Java world, often using Hibernate, the leading object-relational mapping system.

By convention, in three-tier architecture, the client tier is typically regarded as “dumb” in that it merely renders facts that are derived by business rules in the middle tier.  Granted, the front end has mechanical JavaScript code for things like animations, tabs and managing powerful widgets such as trees and tables, but typically no business logic.

In the three-tier approach, the database tier is frequently seen as nothing more than a storage system, but admittedly in many cases, firms fudge by coding some business logic in database-resident stored procedures and triggers that can technically be characterized as rules.  People will argue about the merits of this, but I’ve honestly become fatigued by the controversy; it is mostly religion.

With three-tier architecture, the middle tier is king.  The middle tier encapsulates a centrally-controlled domain model that is in a consistent, canonical form (i.e., objects).  In theory, by centralizing business rules in the middle tier, cost can be reduced, and software quality can be improved.

Irrespective of whether potential benefits are realized, three-tier architecture is pervasive.  A consensus about the viability of this architecture emerged in the early 1990’s and it has remained largely unchallenged to this day.

Tangible cost benefits of three-tier architecture can be difficult to achieve due to complexity.  Object-relational mapping is not straightforward due to a set of technical problems collectively referred to as impedance mismatch.  It has taken many years of focused effort for Hibernate to be regarded as production-ready.  Even now, performance considerations make it likely that caching systems will be necessary, effectively carrying an object-oriented copy of frequently-referenced data in the middle tier.

It takes extraordinary effort to maintain the illusion that our data is represented as neutral objects when in fact the underlying data is stored in relational form.  The cost of dealing with this complexity is real, including the difficulty of recruiting developers that have mastered object/relational technologies, but also in operations and technical support, where problems can be difficult to diagnose and correct.  Much of this complexity and cost can be reduced by JAMM.

MongoDB as Object Authority

MongoDB stores and maintains data in a form that is functionally equivalent to a traditional middle-tier domain model.  Because of this, there is no need to maintain a extra object-oriented copy of the data in the middle tier; instead, middle-tier rules operate upon objects in the database tier, which becomes the central authority for persistent and non-persistent facts.  With this approach, the middle tier can be characterized as stateless.

JAMM Architecture

To an engineer familiar with traditional three-tier architecture, this shift in design can trigger alarms.  I explored my own feelings about this, and found that they boiled down to visceral concerns about database performance.

To counter these concerns, it is important to understand that accessing data in MongoDB is much faster than an object/relational system.  Given a Mongo ID of a document (tantamount to primary key), data can be accessed rapidly because:

  • In many cases, the data to be accessed will already be memory
  • The data may be indexed in extraordinary ways, including indexing every single field of a given record
  • The database tier and middle tiers may in fact be in the same machine, reducing communications overhead
  • There is no mapping overhead because the data is stored in object form

Rules in the middle tier can access just the data they need using granular MongoDB selectors (akin to database SELECT statements), and make state changes to MongoDB without carrying an object-oriented copy of the data.

With this approach MongoDB can be viewed as the state authority for both persistent and non-persistent data, supplanting the middle tier.  I stress non-persistent, because MongoDB is so fast and convenient, it can be used to handle non-persistent data that would traditionally be kept only in the middle tier.

In short, we need to resist the tendency to look at MongoDB as just a storage system.  It can be elevated to a higher plane to become the state authority, the keeper of canonical objects.

Rules on Client Tier

Because JAMM is implemented in 100% JavaScript, it is possible to deploy business rules to the client tier, the middle tier or both.  Although purists may resist the idea of deploying business rules to the client, I have come to see this as a stupendous opportunity.  Example opportunities for rules on the client include:

  • Data validation without requiring a server round trip
  • Data validation via calls to external systems without involving the middle tier (e.g., Google Places API)
  • Conditional forms (i.e., certain parts of a web page being displayed or hidden based on answers to questions)

Meteor allows JavaScript rules to be moved indiscriminately from middle tier to the client or vice-versa, without programming changes.  This permits you to deploy rules where they make the most sense, resulting in improved performance and user experience, and allowing the client to behave reasonably in the face of a spotty connection to the server.

Look forward to more applications pushing rules out to the client tier, and using middle tier services more sparingly for heavy number-crunching or for financial transactions where security is a concern.

 

Why I’m Sold on Meteor and Node.js

It has been almost two weeks since my last blog post.  During this time, I’ve been working hands-on with Meteor, Node.js, MongoDB and Twitter’s Bootstrap framework.  One of my main objectives was to find gotchas that might disqualify Meteor as a viable solution for upcoming projects.

During this time, I spent quite a bit of time working with HTML5 and CSS3 in the context of Bootstrap.  Meteor,  Node.js and MongoDB are so expeditious at dealing with the core issues such as data storage and retrieval that a larger percentage of time can be spent working on aesthetics, for example a responsive layout that works equally well on both hand-held and desktop clients, and also ensure that the application simply looks good and professional.

I broke down and purchased the fine book Discover Meteor: Building Real-Time JavaScript Web Apps:

Meteor

This book helped me resolve a key mystery about Meteor, namely how the system determines which templates need to be refreshed when back-end data changes.  Now that I understand it, I have to say that it is awesome, but please bear with me because you’ll need a little background to understand.

A meteor application is comprised of two “things” that must be developed in tandem:

1.  Templates, which are HTML plus special Handlebars tags.
2.  JavaScript template functions, helpers and event handlers.

Meteor templates are cleaner than traditional template-based development approaches like JSP, because there is never any JavaScript code in the template, and the interplay between the template and JavaScript code follows well-defined rules.  Microsoft has gone in this direction with ASP.NET using the so-called code behind philosophy, where most C# or VB code can be in a separate file.

When Meteor processes your template, it will call into your JavaScript code at key control points to retrieve necessary data.   For data-centric applications, the JavaScript code will typically call MongoDB to retrieve data, and that data will be cached in the browser-side Mini-Mongo as JavaScript objects.  The Mini-Mongo data is directly accessible to the templates and is in the necessary format to feed the templates.  There is no conversion overhead.

Now here is the magic: after Meteor processes your template, the system is smart enough to detect whether any of the data referenced by your JavaScript code has been changed, and to dynamically re-render the corresponding template(s).   This automatic update occurs without a single additional line of coding, and is the key to Meteor’s so-called reactive behavior.  Please take note of the word reactive, because I suspect that you’ll be hearing that more frequently in coming years.  This is because Meteor, and similar platforms such as Derby, are poised to change user’s expectations regarding how web apps should behave, and there is no turning back.

But there was one central mystery: how can Meteor know if any of the data elements that happen to be referenced by my JavaScript code might have been changed, requiring that my template(s) be re-rendered?  Meteor solves this problem in an elegant way that is transparent to the developer; moreover, by mastering a few simple concepts, developers can control the underlying machinery to connect to new data sources that are not known to Meteor.

Whenever Meteor calls into your JavaScript code, it first constructs an object called a Computation.     The Computation contains many Dependency instances.  Each Dependency instance represents a single data source that might dynamically change after your template has rendered.  In practice, most of these Dependency instances correspond to MongoDB collections.  The relationship between Computation and Dependency is many-to-many, so a given Dependency may be contained inside any number of Computation instances.

At any point in time, there is a so-called current Computation which is “active” at the time your JavaScript function is executing.

Meteor getter functions, particularly those that involve retrieval of MongoDB data, will automatically add Dependency instances to the current Computation.   These Dependency instances are listeners for any changes in the underlying data source, and each Dependency knows the set of Computations that might be affected my underlying data changes.

Together, the Computation and its Dependency instances comprise a complete record of all reactive data sources that your template references, both directly and indirectly.  In other words, as a byproduct of executing your JavaScript template-rendering function the first time, the system constructs a record of all reactive data sources that are inputs into your template.

Since the relationship between Dependency and Computation is many-to-many, a given Dependency knows the set of all Computations (and associated templates) that should be re-rendered if the underlying data changes.  With the help of Websockets, Meteor listens to data state changes, and selectively re-renders only the affected templates.  This is in contrast to standard JSP, ASP or PHP where the entire page would have to be refreshed on demand by the user, or via AJAX and jQuery calls to patch HTML after the initial rendering, requiring two separate code paths that do essentially the same thing.

Meteor unifies the code paths: the same code that handles the initial rendering handles dynamic refresh, so making a real-time web application carries no additional cost.  Since real-time behaviors are inexpensive, you’ll see them more and more frequently, and static approaches such as JSP will seem clunky and “legacy” by comparison.

I’m particularly excited about the possibilities using Meteor together with HTML5/CSS3 animations, Google Charts and RIA frameworks such as Sencha ExtJS.   Consider these possibilities:

1.  Smooth-scrolling tables of data that dynamically roll as data is added similar to the ending credits of a movie
2.  Google Charts that dynamically update as data is changing
3.  Table and tree controls that are dynamically updated as user collaborate on a shared data model

Such reactive behaviors will likely become hallmarks of modern web apps.  Start-ups can take advantage of these technologies to differentiate themselves from well-established competitors, at least in the near term.

So, if Grails is a carrier battle group, and Ruby on Rails is a rickety pirate ship, I would say that Meteor/Node.js are the Oracle catamaran: minimal, streamlined, modern and super fast to develop.