Node.js + MongoDB = Love: Guest Post from MongoLab

October 26, 2011 - by alexsalkever

Share:

Node.js with the popular document-oriented MongoDB make for a deeply powerful and robust application platform. Or in other words, they rock.

(Note: This blog post was contributed by Ben Wen of MongoLab - a Joyent Partner and provider of MongoDB hosting, support and analytics)

Pair Joyent Cloud’s hosted node.js SmartMachine Appliance with MongoLab’s hosted MongoDB and the integration becomes downright operatic. Angels sing. Trumpets blare. Grey storm thunderheads of object-relational-mapping haze part. Revealed are golden rays of low-impedance JSON object storage and query. All in the fertile green valley of asynchronous JavaScript on the unflappable, cool bedrock of Joyent’s SmartMachine hosting platform. Songbirds tweet. Life is good. Metaphors strain.

More prosaically, the high performance asynchronous design of node.js and the tunable latency/consistency of MongoDB mean a high throughput application can be assembled in a compressed timeframe and with standard tools you probably have laying around the home. Since MongoLab runs managed hosted MongoDB instances on Joyent’s Cloud near a node.js SmartMachine, you get world-class operation of both environments.

Below, we’ll take a quick spin setting up a MongoLab database and a no.de account. We’ll build a minimalistic Web Server that can do some data inserts and queries and display it through a gratuitous 3D guestbook demo.

For the impatient

1. Sign up at mongolab.com and create a MongoLab database on Joyent Cloud and note database name,hostname, port, database username/password
2. Sign up at no.de and start a SmartMachine
3. git clone git://github.com/mongolab/demo-node-01.git
4. Modify config.js with database credentials and connection info from Step 1.
5. git commit -a -m "updated config" git remote add mongolabdemo <your no.de machine>.no.de git push mongolabdemo master
6. Point your WebGL capable browser to <your no.de machine>.no.de and enjoy.

For the really impatient

1. Go to nodejs.mongolab.com with your WebGL compatible browser

What is MongoDB?

First a quick word about MongoDB for the newly initiated. MongoDB is a non-relational database system that emphasizes horizontal scale across multiple servers, tunable consistency, and high performance. Being a document-database, it uses JSON notation to describe data and sports a rich query language with indexes to enhance query speed. It also has a map-reduce framework for more intense data analysis and transformation. There is growing adoption of MongoDB for large stores of documents like in a Content Management System or in data analytics, for feature-rich Web 2.0 sites and games, and for persistent stores for mobile applications. Its code is open source licensed under the Gnu AGPL v3.0 and is commercially licensed from its author, 10Gen. Large corporations and smaller outfits are using MongoDB in production today. New users, you are in good company.

MongoLab and Joyent

MongoLab is a cloud database service specifically for optimized production of and development on MongoDB. MongoLab hosts instances in Joyent’s SmartMachine cloud for high performance connectivity to SmartMachines and no.de. MongoLab takes care of operational tasks of MongoDB like replication, backup, and redundancy. Precious development hours can be focused on unique features and less focused on unproductive database maintenance. MongoLab has unique ad hoc Web query and data manipulation tools that inspect and manipulate data easily. Of course, standing up a MongoDB instance on a SmartMachine is also straightforward. Free accounts up to 240MB at MongoLab make a compelling case to get started quickly. MongoLab makes using MongoDB as easy as belting out a U2 tune in the shower, or a Katy Perry one if you’re into that.

The node.js Appliance is practically instantaneous for launching development, test, and production servers running node.js. Joyent also hosts a free (as in beer) node.js little brother (128MB of RAM little) at no.de. You can get started in experimental development and try out new design vignettes with no cost or risk. Joyent’s SmartMachine infrastructure that underpins the Appliances and no.de is a beauty of a design, having been built on OpenSolaris and decades of real-world stress testing. Cloud Analytics and the magical DTrace mean you can find bugs and anomalies with minimal thrashing.

Gratuitous 3D Demo

Ok, onto the demo. The client side requires WebGL, a 3D graphics standard for browsers (http://www.khronos.org/webgl/wiki/GettingaWebGL_Implementation) that is supported on the latest versions of Firefox and Chrome. Safari and Opera previews or nightly builds have support.

Don’t fret if you don’t have a browser that supports the right version of WebGL. The 3D stuff is pure eye-candy. It's gratuitous! The main idea can be demonstrated with any browser and two URLs detailed below.

If you do have a WebGL browser there's an instance of the 3D guestbook running on the Joyent Cloud: nodejs.mongolab.com The server is a node.js instance that returns some static content and also handles two dynamic requests: One for querying a MongoLab database and another for inserting new messages.

The client side uses the GLGE Library to render a 3D scene and animate some scrolling text. The camera view can be rotated by dragging the mouse across the canvas. The text is pulled from a URL mongolab.no.de/getentry. Each request returns a different message from the MongoLab database. If you don't have WebGL, you can point your browser right to that directory and get the next entry.

The URL mongolab.no.de/addentry?entry=testmessage does some validation, truncates to 20 characters and inserts it into a MongoLab database for later recovery. That also works in an non-WebGL browser.

Back in the 3D interface, the text field in the front of the canvas lets a viewer of the scene to sign the guestbook with a 20 character message by invoking the /addentry URL.

Pretty simple on the server side, right?

Getting started

1. Here are the things you’ll need beyond a text editor and browser:
2. copy of the source code, which you can pull from github: https://github.com/mongolab/demo-node-01
3. A MongoLab account
4. A no.de account
5. Git for the command line or a git visual client (see git-scm.com for details.).

For the source code, you can git clone or just go to Downloads and get the .zip bundle. For reference, this description is based on the 5931f55cd4 commit of the repository. The command line is

git clone git://github.com/mongolab/demo-node-01.git

MongoLab setup

Now let’s get signed up at MongoLab.

1. Point your browser to mongolab.com and click on "sign up"

1. Fill out the signup form. Note that this “first” account name and username will allow you to create multiple databases on MongoLab. Each database will have its own users independent of the MongoLab account username.

1. Click Add for a Database

1. Select “JoyentCloud” for the Cloud Provider and the Free Plan for 240MB of space.

1. Fill out a database name. Pick something unique. The system will tell you if there's another database with that name.
2. The DB username and DB password are the “second” credentials and these are used by the driver to connect to the database. Be sure to note them down.
3. Click on the row that depicts your new database. You'll see a box that has the exact command line and URI info to connect to your database. Note them down.

Great, you’re set on the MongoLab side. The next steps populate the database with a sample entry. They're optional.

1. Now click Add in the Collections section.

1. Name this new collection “guestbook” in the popup and click Create.
2. In the new row under “Collections” click on “guestbook” and well now add a couple of guestbook entries.
3. Click Add under the Documents / Objects section.
4. Type “entry”:”hello” in the curly brackets such that you get ‘{“entry”:”hello”}’ in the large black input field. Click “Create and go back”.
5. Add another Document ‘{“entry”: “world”}’.
6. Note that the list view shows the two documents. More information about the List View can be found here. http://blog.mongolab.com/2011/10/video-tutorial-list-view-and-json-editing-for-mongodb-right-in-your-browser/

No.de setup

Onto signing up at no.de. Remember that no.de is a free service for development. Joyent’s public cloud SmartMachine node.js appliance is what you’ll want for more capable deployments (available at https://my.joyentcloud.com/)

1. Point your browser to no.de.

1. Click Sign Up
2. Fill out the Sign up form

1. If you have an SSH public key, provide it on the SSH Key form and click Add This Key. If you don’t have one, then follow this link to get one. You’ll need it for other purposes, including being able to push git repositories to the no.de instance. http://wiki.joyent.com/display/node/How+to+use+SSH+Keys+with+Node.js+Smart+Machines

1. Click “Order a Machine”, provide a unique machine name, click Provision and one will be provisioned for you in a few minutes. The Status column will go from “provisioning” to “running” when it’s up and running.

1. While you’re waiting for the status column to change, click on the name of your machine and you’ll get some instructions to add the following to your $HOME/.ssh/config file. It’ll look something like this:

   Host <your no.de machine>.no.de Port <number> User node ForwardAgent yes

2. Once it’s running, you should “ssh” into it by typing ssh <your no.de machine>.no.de SSH is important because we’ll need it to move the code from your workstation to the no.de SmartMachine instance. You’ll see a warning that your machine hasn’t ever connected to that server and the fingerprint is unknown. That’s expected only the first time you connect to the machine. You can type “yes” to accept.

   Making config.js changes

   config.js in the root of the git repository demo-node-01 is where you’ll put the info we collected from the MongoLab database creation. There's a snippet below. The exports.databaseUrl should look like “dbh.mongolab.com:27007” and are chosen by the MongoLab system. The exports.databaseName, username, and password are provided by you when you created the database. Yes, the username and password are the “second” ones. Save your changes to disk.

    // // MongoDB connectivity configuration // exports.databaseUrl = “dbh.mongolab.com:27007”; exports.databaseName = “nodelove”; exports.username = “databaseuser”; exports.password = “myseekrit”; 

   High level overview

   Before we commit the changes to the no.de server, let’s look at the high level stack here. At the top is the 3D browser client in Javascript that calls to the no.de instance running node.js. That node.js instance runs the server.js program from our git repository that itself connects to a MongoLab hosted database running on a Joyent SmartMachine. The node.js server connects to a MongoLab instance running in Joyent’s cloud and makes queries and insert requests. Node.js itself is covered in detail nodejs.org. Download and installation instructions are there if you want to get them to run on your own workstation. But for the rest of this discussion, we’ll be wallking through a free no.de instance.

   server.js major functions

   Diving in deeper into the server.js running in node.js, the major functions are:
   1. The server is a a minimalistic Web server that serves static content: the index.html page, attendant client-side JavaScript files (3dmongodemo.js), libraries (glge-compiled-min.js), and 3D/graphics data (level.xml and texture maps).
   2. The server queries the MongoDB database and sends back the result to the client. The server also tracks a little meta-data to keep remember what guestbook entry was most recently sent. Clients would see a different entry most of the time. Yes, this is not idempotent, but we’re not looking for pure REST semantics; we’re making a demo not a banking app.
   3. The server inserts new guestbook entries. On insert requests, the server does a little data validation and cleansing before connecting over the MongoDB wire protocol to do an insert into the database. Let's take a look at server.js internally. There are three steps that the server goes through when it's loaded.

    // // Start running here. // processCmd(); // Process Command Line args. initializeDb(); // Start connection to MongoDb startServer(); // Start the http server. 

   The first is to parse a single command line option which allows you to override the default network port (port 80). That’s useful if you’re running it locally and don’t have / want authority to bind to those system level ports. Be aware if you fiddle with the port number that there may be firewall issues in your network that block non-standard ports. The second step is to initialize the connection to MongoLab. The third step is to start the server listening on the designated port. Most of the work is done in a function called dispatch() each time a new request is made.

   MongoLab connectivity

   In initializeDb(), you can see the configuration parameters, the database’s network hostname and the authorized username and password, are pulled from the file config.js. The file is ‘require’d in the preamble of server.js.

    // // Opens connection to MongoDB database, authenticates, logs successful connection. // function initializeDb() { mongoose.connection.on(“open”, function() { console.log(“Connected to MongoDB successfully!”);}); mongoose.connect(“mongodb://” + loginCredentials + “@” + dbUrl + “/” + dbName); } 

   Note there are programming language specific drivers for connecting to MongoDB. We use an enhanced driver package called mongoose that works with node.js. As a bonus, it provides a measure of object-document mapping prettiness wrapped around the driver, even if we don’t use those features in our demo. The package.json file specifies to no.de that we need mongoose to run the demo. If you want to get mongoose directly, there’s an installation program called “npm” which you can remember as “node package manager” even though it stands for something slightly different and less memorable. npm can be found at npmjs.org.

   dispatch handler

   In the third step to start the http server in startServer(), where the serverCallback gets called when a new http message is received from the network. In turn the dispatch() routine is the called that handles the two special URL paths “/getentry” and “/addentry”. Failing that, it sees if the path is one of the allowed filenames. allowedFiles is also a configuration entry in config.js. If the pathname matches then the file is served to the requesting client. If the path was “/getentry” the server makes a query to the database and returns the next guestbook entry (from most recent to oldest) to the client. Before it does this, it sees if the guestbook had any new entries added since the last query, in which case, it returns the newest and starts counting down from the top again.

    if (urlparts.pathname == “/getentry”) { // Query MongoLab query(“guestbook”, {“entry”:{“$exists”: true}}, function (err, docs) { if (err) { console.log(“Query error”, err); return; } if (maxEntry < docs.length) { // Someone added a guestbook entry since my last visit, start from top currentEntry = maxEntry - 1; maxEntry = docs.length; } else { currentEntry = currentEntry - 1; } if (currentEntry < 0) currentEntry = docs.length - 1; renderHtml(docs[currentEntry].entry); }); } 

   Stop for a moment to look at the query. The call to query() has as parameters the name of the collection: “guestbook” followed by the query: {“entry”:{“$exists”: true}}. A collection is a MongoDB database’s set of documents. The documents in a collection get searched and indexed as a group. The query is just a JSON document with the name of the field: “query” being used to test for a filter: {“$exists”: true}. There’s a nice set of query predicates in MongoDB: conditional operators, regex’s, array matches, etc. Details can be found here: http://www.mongodb.org/display/DOCS/Querying If the path was “/addentry”, the server looks for a query parameter “entry=” and after doing some data validation, inserts it into the database. The inserted document in our demo is a trivial key-value: {“entry”:<data>}. In a more real-world use case, the entire rich, nested constructs of JSON would be used.

    if (urlparts.pathname == "/addentry") { var querystring = urlparts.query.split("&"); var value = ""; var patt = /entry=/; querystring.forEach(function (item) { if (patt.test(item)) { value = item.replace(patt,""); value = decodeURI (value); value = value.replace(/\+/g," "); // TODO: better way to pass in " " in parameters like a post body. value = value.substr(0,20); // For safety, truncate to 20 chars. }}); if (value == "") { renderHtml("query malformed: " + urlparts.query); } else { insert ("guestbook", {"entry": value}, function (err, docs) { if (err) { console.log("Insert error", err); return; }}); renderHtml(value); } 

   Pushing code to no.de

   So we’ve made the necessary changes to the config.js file and we now can push the code to the no.de instance. We’ll use the git command line below. If you’re using a visual git tool, the steps are similar. Git uses ssh to authenticate from your workstation to the no.de SmartMachine instance, so again it’s important that it’s set up properly, including the .ssh/config file above. If you haven’t used git before, it’s a distributed source code manager and it has become popular for deploying to remote cloud machines. The basic idea is to push a local repository from your workstation to the remote Joyent cloud repository. So first we have to commit the config.js changes to the file to the repository: git commit -a -m “Updated config.js.” Next we have to tell your local workstation to set up the remote repository. git remote add mongolabdemo <your no.de machine>.no.de:repo That creates a remote alias ‘mongolabdemo’ that points to your new no.de SmartMachine in a directory ‘repo’. The directory name “repo” is special because there are scripts expecting your code to be dropped there. git push mongolabdemo master That pushes the code to the alias you created. Master is a special git keyword meaning the latest changes. You should see something like the screenshot below.

   Debugging

   You can point your browser to <your no.de machine>.no.de and the 3D screen should appear with scrolling guestbook entries. Remember to add your own entries in the text field. Congrats! Now if things don’t go exactly well, there are a few places you can look to debug. If you’re ssh’d into the no.de server, there are a number of script commands that are helpful. They’re highlighted in the greeting when you first ssh in. Notably node-service-log connects your ssh session to the console.log output of the node.js server. Run it and you should see “http server open for business.” if the http server starts successfully. And “Connected to MongoDB successfully!” appears if the database connection is alive. Once you point your browser to the URL, the server emits log entries for each URL request. You should also be able to point your browser directly to the URL http//:<your no.de machine>.no.de/getentry and get back one of your guestbook entries. If you’re getting something else, check that the database credentials and hostname are correct in config.js. Each time you make changes to the code, do a git commit -a -m “<your new message>” and a git push mongolabdemo master to update the no.de instance.

   Conclusion

   So that was a whirlwind tour of creating an http server on Joyent’s node.js SmartMachine service with dynamic data from MongoLab. We didn’t review any of the 3D client side code, which is in the webroot subdirectory. There are a lot of interesting things that could be done on the client side. It could be a mobile or tablet application, or a rich HTML5 client (if you peek in the webroot code, you’ll see some jQuery!). I hope that you learned a bit about node.js and MongoDB and how to use them in the cloud. In the process maybe the gratuitous 3D client sowed some creative seeds in the the fertile terraced valley of your mind. I’m hoping to seeing your magum opus on no.de and mongolab.com!

Share: