Been there, done that, got the T-shirt …

So, how was your week? Added a few visitors, improved the way the storage is accessed, and as a bonus fooled around with WebJobs. Let’s dive in!

My Store Client

One of the things I was complaining about is the fact that the actual storage of data could only be performed if my script was running in the Windows Azure cloud. This is because the scripts were using the global ‘tables’ object to access the data store, and that one is only available at the server side, and not on my development machine. This gives a nasty round trip in order to test and troubleshoot the actual storage part of the solution. But not anymore …

The Windows Azure Storage is accessible through the Mobile Services API, that’s the whole point of Windows Azure Mobile Services, duhhh. So why not have our own visitor scripts also use that API  to work with the data. For that purpose, I created a ‘store client’. It’s a piece of JavaScript code that performs the CRUD operations on our ‘Stores’ table through the Mobile Services API. Out of the box, the API supports the OData protocol, so our client only needs to be a thin wrapper around a NodeJS package that does OData. This is what it looks like:

So … what are we looking at? Starting with the ‘odata-cli’ NodeJS package with gives us a beauty of an OData client. This is what our CRUD operations AddStore, GetStore, UpdateStore and DeleteStore are using to communicate. The ‘onBeforeRequest’ function is injected as the request handler, for the purpose of assigning my super secret application key into the header of each request, by setting the X-ZUMO-APPLICATION value in the header.

Also part of this new client is my store definition and the code to synchronize the stores found during crawling with the already existing stores in the database. The code for comparison is already a bit more condensed and the public objects and functions are exposed through a nice module interface. Not yet as good as it can get, but we’re improving. This module provides a central reusable artifact for any other code that want to interact with the ‘store’ entity in our information model. Nice!

The Hoogvliet visitor

The first one to benefit from the Store Client is the brand new Hoogvliet visitor. This is what their page looks like:


And this is the code that crawls it, nice and compact:

Nothing much to say, or excitement going on. Just some JavaScript object embedded in the page that contains the data we need to fill our store object. Been there, done that, got the T-shirt …

A WebJob Jumbo crawler


Based on a nice post that Scott Hanselman published on his blog on Windows Azure WebJobs, I just wanted to take a look at what it is compared to the Mobile Services Scheduler and NodeJS. An amazing experience! Why? The simplicity of it! All you have to do is create a command-line application, push it to a Windows Azure Website in a zip file and it is ready to rock and roll. So productive to be able to use C# and .NET again, after experiencing the newbie pain with asynchronous JavaScript in NodeJS. It was really a breeze. Here is the code:

While simple, a new crawl pattern is encountered. This site has some of the store details exclusively available through dedicated pages for each store. So after having indexed the number of stores and some location information, each individual page for each store needs to be retrieved and crawled to get the proper information to fill the store object. Well, something we would have had to deal with sooner or later. But not really a showstopper …

Again, WebJobs are a very simple and productive way to extend websites with scheduled jobs executing on the backend. But for our little pet project, it also brings in a dependency again on Azure Websites, because that’s where the scripts are hosted. And we’ve just done a nice piece of effort to get the storage scripts support off-server development. So we’ll consolidate the Jumbo, and our previously generated Digros / Bas / Dirk C# implementation crawlers on the NodeJS platform as well.

NodeJS tools for Visual Studio

Just before we ‘call it a week’, wanted to extend my thanks to Scott Hanselman again, also for another post on the NodeJS tools for Visual Studio. This tool set allows to debug NodeJS scripts on a development system as if it was native code, with all functionality of setting breakpoints, step-by-step execution, variable value evaluation and all other good stuff we are used to in Visual Studio. Great job guys! Now I’ve got no good reasons left anymore NOT to continue on this platform 🙂

Next post, we’ll see the C# crawlers ported to JavaScript and attached to the store client, and have some better consolidated logging to improve the monitoring in the Windows Azure console. And if there is time left, we’ll throw in some email sending capabilities to notify us when changes are happening during synchronization. Go, go, go!

Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *