Category Archives: node.js

Node for your ArcGIS Dev

Posted by on March 6, 2013 No comments

Need a local server anyway

One of the things that tripped a lot of users up when the ArcGIS JavaScript API moved to version 3.0 and started using Dojo 1.7 as it’s core was the fact that you needed to use a server to test your applications. This is by no fault of the ArcGIS API team, it’s just the way Dojo works as part of the AMD loading. This isn’t the kind of thing Esri can fix, nor is going away, so users will need to deal with it. And let’s be honest, best practices would dictate that you probably should have been developing with a local server of some sort anyway.

There are various solutions to work with this. If you use a flavor of Visual Studio, you are in the clear. You could use something like XAMPP, which is a great choice especially if you will be deploying to an Apache environment (it even comes in a portable version for usb drives). I still use both of these methods on a regular basis. But as usual, I was looking for something better. I wanted an easy way to to develop, build tools, test, debug and the all critical build and deploy.

So I turned to Node.

The Node less traveled

I have dabbled in some node in the past, but it has been a while as I have had other projects to work on. So I was sitting there drifting off to sleep in my vanpool (eco-friendly, of course) on the way home from work when it hit me. Why can’t I just build a template app that I could run in node to serve my app to test and debug? It’s just another web page after all. One of the reasons I like to use MVC to build my apps is it’s easy to write an underlying REST API side by side with my web app. Node devs do this kind of thing with their eyes closed while sipping lattes from a Tardis mug (adding to wishlist) all the time.

So I put a template project together on github.

The Point

Using this method makes it very easy to get started. One of the great things about working with Node and the connect framework that expressjs uses is the ability to drop in middleware, which is basically another word for extensions.

So in this case, I am using Jade for an HTML template engine, although you can use any engine you desire or none at all. There is a less-middleware which not only compile less css for you when it is served, but optimize it as well. I am also using express-uglify, which will minify my js files from the server. This means I don’t need to create a release build using r.js or some other method to deploy my application. That to me is very cool. If you come from a .NET/MVC4 background, this is all very analogous to the Web Optimization tools.

I came across a method on stack overflow to load scripts into a jade template that you can see in router/index.js. This is where you can set the title of your page and tell it what scripts you may want to load to shim up your application.

The bulk of the work is done under the public/ folder. This is all stuff I use in production on a daily basis. This would be where you do most of your actual ArcGIS JS dev. I use a config.json that could probably use a README.MD for options, but for now you can load dynamic and feature layers. If you review the code, you should be able to figure it out. Ideally, this config json would be a web service responseThis config json is served from a url endpoint, but could come from MongoDB. I’m still playing with this in my development.

You may notice lack of tests. I am a horrible person. I had some QUnit tests, but nothing comprehensive. I’ll be adding in some mocha tests as soon as I can, which would make it easy to run my test from the command line and probably even on save in my vim environment.

The Proxy

A lot of ArcGIS JS dev requires the use of a proxy, for long queries and CORS and edits, it just comes in handy quite a bit. Obviously, Esri doesn’t provide a node version. I attempted to make one with http-proxy and I think I was somewhat successful. I got it to proxy to external sites ok, but it has not been extensively tested. This is where my node-fu is weak, so no guarantees here, but I am more than open to assitance (or ridicule) on this subject on how I did it. –Update: I think I may have the proxy working as expected. Doesn’t handle tokens just yet. Keep an eye on git repo for further updates.

Will this work?

So you may be asking if all this works. It is totally experimental on my part. I really like developing in this environment and I hope to use it to deploy a couple of projects in the future. You can host node apps in IIS with iisnode if you are an IIS shop. I’m hoping some people will be open to try a solution like this. You could deploy directly to nodejitsu to streamline the process. I was easily able to deploy it to Nodejitsu, which you can see here.

Are you high?

This was a project I got really excited about because I think it opens up some cool possibilities. Say for example you have some bit of code that doesn’t interact with the DOM, but could be a little complex for the client to handle. Offload it to a node module in your development environment. I’m going to add in the esri/geojson-utils as a node module for geojson/esrijson conversions. I have a lot of ideas on how I can use this approach for some projects that have proven a little tricky.

In the end, I do all this for fun and every once in a while, something useful comes out of it. So take it, play with it and maybe you’ll find some use out of it as well.

CoffeeScript, Visual Studio and Sublime Text 2

Posted by on June 7, 2012 No comments

I had previously written about simplifying your dev environment. Most of the day to day work that I do, I can manage with vim just fine. But I also use Visual Studio quite a bit for ASP/MVC3 and (don’t laugh) Silverlight. So when I do web dev in Visual Studio, I like to set up my environment in a way that makes me effective and productive. I never really thought about this process until I started doing some pair-programming and had to share this convoluted information with another human being. I’m going to do this step-by-step, so there’s no confusion (please don’t get confused).

1: MVC 3

Of course you’ll need Visual Studio 2010. In my case, I work mainly with ASP.NET MVC 3 when using Visual Studio, so be sure to install that here.

2: Mindscape Workbench

You can get Mindscape Workbench here. It’s a great plugin for Visual Studio that allows you to create CoffeeScript files as well as Less/Sass files. We’ll also need to install CoffeeScript and Sass/Compass for this to work. You can read more about the plugin from Scott Hanselman here. One drawback it has is that there is no intellisense for CoffeeScript files. This is the next step.

3: Sublime Text 2

This is not a required step, you could just as easily use vim or some other text editor, but I wanted to set things up in a somewhat easier to use environment for someone that doesn’t care about vim power. Download it here. Leave it alone for now, we’ll come back to this later.

4: Git

Now the fun begins, install Git for Windows from here. From here on out, I recommend using the Git Bash shell for simplicity sake. Now the tricky part is working with these command line tools behind a proxy. If you’re lucky, you don’t work behind a proxy, if you’re not, these command line tools will not automatically be aware if your internet settings, so you need to manually set them up. You can use the following command or git as discussed in this Stackoverflow question.

git config –global http.proxy http://proxy_host:port

5: Ruby

This is a simple one, install Ruby from here. The Ruby installer comes with gem. Once again, per instructions here and here, we may need to set up gem behind a proxy. Run this command before using gem.

set HTTP_PROXY=http://proxy_host:port

This should set HTTP_PROXY for this current bash session, so you’ll do it each time you reopen bash before running gem. I tried setting up a permanent environment variable for this, but it interfered with my internet connection. You may have better luck. I could have sworn there was a config file you could create that gem could read for proxy info, but can’t remember it.

6: Node.js

Simple enough these days, use the windows installer for Node.js and you should be all set. Before the windows installer, this process sucked, I would say run Linux in a Virtual Box and spare the stress. Thanks Node devs for making this a clean process for Windows guys. The installer comes with npm, so if you need to work behind a proxy, read this and do the following.

npm config set proxy http://proxy_host:port
npm config set registry “http://registry.npmjs.org/”

7: CoffeeScript

Now you can install CoffeeScript, finally!

npm install -g coffee-script

Now you the Mindbench plugin for Visual Studio can compile your Less/Sass files.

8: Sass/Compass

Install Compass per these instructions.

gem install compass

9: CoffeeScript bundle for Sublime Text 2

Get the CoffeeScript bundle here. Sublime Text 2 can use Textmate bundles, so this works great. On Windows 7, you are looking for $HOME/AppData/Roaming/SublimeText 2/packages/, where $HOME is your user documents and settings folders. Turn on the ability to show hidden files and folders if you don’t see the AppData folder. Run the following command.

git clone git://github.com/jashkenas/coffee-script-tmbundle CoffeeScript

Add the ability to build the CoffeeScript files per this link. Open the CoffeeScript/Commands folder and look for a file named CoffeeScript.sublime-build. Make sure it looks like this. I did not need to have the “path” option for it to work for me.

{
“cmd”: ["coffee.cmd","-c","$file"],
“file_regex”: “^(…*?):([0-9]*):?([0-9]*)”,
“selector”: “source.coffee”
}

Now when you edit a *.coffee file in Sublime Text 2, you can use ctrl+b to build and save the file at the same time.

10: Take it for a spin

At this point, start up Visual Studio, start a new MVC 3 Web project. Add a new item, and you should have the option to add CoffeeScript files.
Right click the CoffeeScript file -> Open With, choose “Sublime Text 2″, if not an option, browse to the sublime.exe in Program Files\Sublime Text 2. Set it as the default and now all *.coffee files will open in Sublime Text 2 when double-clicked in Visual Studio. Again, press ctrl+b in Sublime Text 2 to build and save the *.coffee files and you should be good to go.

Whew…

Ok, I know this whole process seemed a little long, but I find it to be worthwhile in my dev workflow. The trickiest part for me was getting everything working behind a proxy. After I had to do this on a couple of machines and asked to explain it, I thought I should document this somehow. Many of the search results out there are also written specific to OS X and Linux users, so there are little bits here and there that took some translating to Windows, not much, but slight nuances like installing the CoffeeScript bundle. I hope everything works for you. If you run into to trouble, I’ll see if I can help, but some Google-fu worked when I got stuck.

I did not cover each tool in much detail, like how to use git or the benefits of npm and gem. I would encourage you to read on these subjects further if you are interested. I would like to state that Node.js and Ruby are more than development languages (Ok, Node is a library, give me a break), but they open up a whole world of tools that you can use in your day to day development, like testing, building of your web applications.

I hope someone finds this information useful. It took me hours to figure out when I first started it some time ago, so hopefully I saved you some time.

ESRI-JSON to GeoJSON, with Node.js

Posted by on November 23, 2011 1 comment

I’ll admit, my professional GIS development experience is in the ESRI sphere. It’s what we use at work and it’s what I’ve been using for going on 10 years now. I’m familiar with FOSS4G, but have not really had the opportunity to develop any projects incorporating it. Because of this, I was not really aware that the ESRI REST spec for their JSON response really differed much from the standard GeoJSON spec. My naivety is a little embarrassing as a GIS professional.

So my interest was piqued a few days ago when I saw someone post a tweet about the request to have ESRI add a GeoJSON response to their REST API. For whatever reason, I got it on my head that it couldn’t be that hard to do. A quick Google search showed their were a couple of ways to do this with third party tools, but I figured there had to be a way to do this quickly in the browser or natively server side.

Recently, I have been practicing on little projects in various languages and frameworks and I thought Node.js would be a great solution to get this rolling. It’s fast, can work with JSON objects natively and should work pretty simple without having to worry about browser compatibility issues because it runs through Google’s V8 engine.

So I set out looking at what would need to be done to convert ESRI-JSON to GeoJSON. As an aside, this entire project was done inside Google Chrome using the Cloud9 IDE and deployed directly to Heroku from within Cloud9. I could probably do a whole write up on that experience.

It took me a couple of days and lots of debugging and running various ESRI-JSON through the converter, but I think I’ve got a very usable application running to demonstrate the conversion. If you’ve worked with this before, GeoJSON has a standard array called “coordinates” to hold x,y information. ESRI-JSON uses “rings” for polygons, “paths” for lines and just an array of x,y pairs for points. A little annoying to convert, but not a big deal.

ESRI-JSON to GeoJSON Converter

You can also send your ESRI-JSON as a parameter to http://esritogeo.herokuapp.com/parse/esrijosngoeshere and get a JSON response. I should point out a caveat is that you need to make sure you are sending the correct projected data. GeoJSON uses standard Lat/Long, so your ESRI-JSON should be in Spatial Reference 4326.

I put a couple of demo pages up. This sample uses the ESRI JavaScript API to load a FeatureLayer of points. This sample uses Leaflet to load the GeoJSON that was converted. The GeoJSON sample takes a few seconds to load and would not load in IE8 for me, but worked in Chrome. I don’t know if this is an issue with Leaflet or not, sorry.

The ESRI-JSON to GeoJSON application is on Heroku for now as an experiment. I don’t have any permanent Node.js hosting. So if it gets a lot of traffic it may just disappear. Heroku is for quick testing, not production. I don’t really have funds to put this up for wide use. I will get the whole application up on Github, as soon as I figure out how from within Cloud9.

I would however like to share the code that does the conversion, which is shown below.

I’m sure this functionality could be incorporated using Server Object Extensions on ArcGIS Server, but it works so cleanly this way. Maybe some people will find this useful and could incorporate it into their workflows.

Thanks.

edit: Project is now available on github.

Node.js native extension with a hammer and a prayer

Posted by on October 20, 2011 4 comments

I get excited about dumb stuff when it comes to programming, so when I first heard about Node.js, it seemed so out there I was like, Ice Cream Ballz, I have to try this! I experimented, read the books, guides, online tutorials and decided, I am in, you got me.

When a good man goes to war

Node.js is built using the Google V8 Engine. For most people that use Node.js, this probably doesn’t matter to you at all. However, if you are interested in building native extensions to Node.js, you better learn to embrace it. The Node.js docs sort of cover a simple “Hello World” example of writing a native extension in C++. Then they link to the node_potsgres for an example of a real world project. This is like teaching a 5-year old how to drive a Tonka Truck, then giving him a big rig to drive around the block. Bless their hearts, but the node_postgres extension is 600+ lines of code and although well documented, unless you familiarize yourself with some V8 stuff and review the Node.js source, you may as well be reading a Celtic translation of lorem ipsum. Or maybe you’re just way smarter than me.

I decided at some point I wanted to build a native extension for Node.js using the ESRI C++ File Geodatabase API. I’ll write more details on this project at a later date when I can actually make it do something useful. For now, the point is I am a novice at C++ at best but very eager. I can struct, point and make, but I don’t work in C++ on a day-to-day basis. So I did what anyone else would do. I raised the mighty Goolgle hammer and I pounded out “Hello World” examples and nailed down some erroneous errors in my attempts.

WTF… I mean WAF

I did ok at first, knocking out some simple stuff. When I tried adding other libraries to a project was when I hit the first wall. Node.js uses WAF as it’s build system for native extensions. You write a wscript file and use the node-waf command. WAF as they state “… is provided for the ease of users”. Sure, I use WAF all the time, great, this should be easy (I have never heard of WAF before). Turns out, WAF is a Python framework for compiling, similar to ANT, Make, and Rake. So time to switch from C++ mode to automation mode. Turns out WAF is pretty cool, and you can pretty much just assign your flags to a build object and let it rip. I had a tough time figuring out how to define some environment variables, and the stuff in WAF book didn’t quite work as expected, but once I had wrapped my head around WAF, I felt like I had conquered a small country.

In search of the Node way…

When I got my first Node.js extension to compile, I was smiles ear to ear. It was a simple ext.speak(“Say my name”) and it would print “Say my name” in the console. But that did not seem like a non-blocking Node.js way of doing things. So I set off to figure out how to ride the callback train to crazy town. First, I attempted to replicate what was happening in the node-postgres project. They extend the EventEmitter which is part of Node.js and go from there. I still had trouble following as I had trouble figuring out if some functions came from the Node.js headers or the V8 headers. Then I came across this issue on the Node.js github page. That little comment was like a small spark as to how to handle an event in C++. Researching the EventEmitter lead me to an example on github called node-event-emitter that did exactly what I had been looking to do this entire time. It was the revelation I needed to continue. You don’t have to write your Node.js native extension to extend EventEmitter, just write the handlers in C++, then use JavaScript to create a shim to inherit EventEmitter and voila! Remember that feeling you got when you first learned how to read? Me neither, but I imagine this is how I would have felt. Just to illustrate, below is a generic example of a JavaScript shim copied from node-event-emitter.

On the Node to find out

So what the hell is point of all this rambling? Well, I think I shared some pretty good resources and really I wanted to highlight some of the stuff you will probably encounter if you attempt to write some Node.js native extensions. There are a couple of really good tutorials out there to write an extension and really it’s not that hard. My biggest stumbling block was trying to figure out how to write an asynchronous extension. There simply is not much guidance (that I could find) on how to accomplish this. Node.js is still very new and I have yet to see anything beyond a couple of blog posts on how to write native extensions. You won’t find this stuff in any of the Node.js books available (as far as I know, if there is a book that covers some of this, please let me know). Granted, I did not hit the google groups or an IRC, which admittedly may have lead to quicker results. But at least I was forced to read lots of source code, get more familiar with some V8 Engine stuff and learned me some C++ along the way. By the way, don’t attempt to learn C++ by diving into writing Node.js native extensions. Just my opinion, but maybe start with brisk walks before running a marathon.

In the end, I am well on my way to writing what I think could be a pretty cool Node.js native extension and if you want to check it out, you can find it here.

A really big thanks to Ben Noordhuis and his event-emitter example on github. Without that example, I’d be wallowing in a sea of google search inferno.

Again, since I am new to this, if I have misinterpreted anything or lack understanding on some stuff, please point it out. I’m here to learn and I’m taking names.

I posted on Google Plus for this too. Talk about recursive blogging.