A briefing on Cloud and .NET development

We all know that when we are looking for the best performance accessing DBs in .NET there is nothing faster than plain old ADO.NET DataReaders… but that could also make your code so verbose. Yes, you could build some helpers, but you could easily end up rebuilding the wheel. That’s what I loved about Dapper.NET – it is just a thin layer on top of ADO.NET, so you get the best performance, a lean implementation in your repositories (3 or 4 LoC), and full control of what’s going to the DB engine. I am not against EF, but many times you don’t know what is this guy doing down there.. and when comparing the performance, Dapper.Net is a clear winner. This is just another pragmatic library shared by the guys at StackExchange.

I have been using Azure Table Storage—ATS in a couple of my personal projects, and I just love it. It is simple, the performance was decent and the storage quite cheap. A NoSQL key-value store like ATS is just perfect for storing lots of unrelated records like audit, and error. In our case, around 70% of our data fall into this category.

I had the perception that ATS was not that fast, but I did not notice much impact on the performance of the site. Anyway, the audit and error reporting operation were asynchronous. Probably the only major drawback during the project was the ATS poor API – I still cannot conceive the anemic LINQ support.

During the architecture definition of a new web-based project I started to consider some other options for the data storage. This new project required a data model with way more relations between entities – a productive API was key, although I wanted to stick to a NoSQL store for future-proof scalability.

One of the serious options that we started to contemplate was MongoDB. I had some quick experiences in the past with it but nothing serious. I knew that their support of LINQ was phenomenal, the option of growing to a massive scale thanks to sharding and replica sets, but what about performance…? How would ATS performance for read/write operations compare against MongoDB, or Azure SQL?

I built a simple MVC 5 application and deployed it in a Azure Web Role (XS). Using the ATS .NET Storage Client Library 2.2, I built a simple page which reads 100 records in a ATS table, and another page for reading each one of the records in that same ATS table. The average latency of each write and read operation is displayed. I based my application on the tutorials and walkthroughs available from Microsoft. The idea was to build it with the techniques available, without any tuning.

I did the same thing with Azure SQL, procuring the smallest database I could. I used plain old fast ADO.NET DataReaders for implementing the operations.

For MongoDB, I launched a Extra Small VM with Linux CentOS. This is a 1Ghz CPU, and 786 MB RAM VM. MongoDB was deployed with default settings.

Naturally, all these resources were located in the same region (US West). This diagram summarizes the topology:

And…these are the results:

Yes, the performance of plain vanilla ATS is just disappointing. After some research I found blog post with similar findings, which indicated how to improve the performance turning-off the nagle before the calls:

The performance benefit is impressive – I wonder why this is not a default setting. Note how the read operation were not affected by that.

The performance of Azure SQL operations was really good (under 10 msecs on average), but the winner as you can see was MongoDB – impressive, with both operations under 2 msecs!

Well, that was a eye-opener. It is pretty obvious what we are going to use for next projects. Unfortunately, neither Azure nor Amazon Web Services offer a managed MongoDB service at this time, so I would need to setup and maintain my own set of VMs running MongoDB, which is not a big deal, but I would need to pay for this in addition to the storage.

Yesterday I found this gem while creating a simple spreadsheet with a portfolio I am defining. Instead copy/pasting values of the stocks in my portfolio from any of the financial information providers, you can simply use the GoogleFinance(SYMBOL, ATTRIBUTE) function to get the latest indicators of your specified stock symbol.

Where attributes could be:

price – Current Price of the stock

closeyest – Last closing price of the stock

priceopen – Current opening price of the stock

high – Daily high price of the stock

low – Daily low price of the stock

change – Change since the last posted closing price.

changepct – Percentage change since the last posted closing price

high52 – The 52 week high price for the stock

low52 – The 52 week low price for the stock.

eps – The calculated earnings per share

pe – The calculated price to earnings ratio. (Note that companies with negative earnings will not have a pe ratio).

volume – Number of shares traded.

marketcap – Market Capitalization

tradetime – Time of the last trade.

datadelay – The time delay from Google’s servers.

volumeavg – Average volume for the stock.

beta – Beta value of a stock.

shares – Outstanding shares of the stock.

The GoogleFinance() function could also pull historic data, allowing you to track the performance of any stock across a certain time period. To show historical data, type =GoogleFinance(“symbol”, “attribute”, “start_date”, “num_days|end_date”, “interval”) into a spreadsheet cell.

You’ll find details about the syntax of this formula below:

“Symbol” and “attribute” follow the same rules as above

“Start date” is the day you’d like to start showing information from

“num_days” | “end_date” [Optional] can be either the end date for the time period over which you want to see historical data, or the number of days from the start date. Any number less than 50 is considered to be num_days. Otherwise it is considered an end_date.

“interval” specifies granularity at which the stock data is shown, either daily or weekly. You can enter either “Daily” or “1” if you would like daily information and either “Weekly” or “7” for weekly data.

In a previous post, I detailed how to automate the creation of a standard multi-server environment using the IaaS capabilities in Azure. During the last days I had the opportunity of enhancing these scripts a bit. This second part of the post describe the enhancements.

The IaaS capabilities of Azure could be very handy when you need to create temporary development/test environments during the SDLC. Automating the creation and clean-up of these environments could save a lot of time and compute time ($$).

Azure exposes an interface based on PowerShell to automate all the steps required to do this, and I spent some time researching how to properly do it. You will find many references and blog post on how to create VMs using the PowerShell API in Azure; however I did not find many updated, accurate references of how to do it for an entire environment. Probably because the API has evolved so quickly and these articles are no longer relevant.. The results are summarized in the following script which demonstrates the creation of a standard deployment of an enterprise multi-tier environment (web front-end, application server and database server).

“There is something about investing your humanity, your eccentricity, your exuberance in the things you do. Not everything you do is going to be successful, but that’s part of the allure. It is also what makes the work valuable: that you are really present and invested in what you’re doing.”

By default, any typo or mistake in MVC razor views will only be detected during execution.. However, you can enable compiling of views and detect those errors earlier in the development cycle.

Just open your project file (.csproj) as text (unload your project first), and find the < MvcBuildViews> and change its value to true:

<MvcBuildViews>true</MvcBuildViews>

Beware that compilation time will be increased (almost double in my case). You may also get a weird “Unrecognized attribute ‘xmlns:xdt’on the web.config” error during compilation (especially after switching between build configurations ). To work around it, delete the \obj folder in your project folder, or use the pre-build step described in here.

During PDC 2010 in Redmond, WA Microsoft announced a bunch of improvements to the whole Azure platform, some of them desperately needed:

Support for the new Virtual Machine role, in addition to the existing Web and Worker roles. This could allow PaaS scenarios, where you can build, configure and upload your own Windows Server 2008 R2 VMs as VHDs – quiet similar to the AWS model. (Great!!!) In addition, the pricing model for the Windows Azure VM role is the same as the existing pricing model for Web and Worker roles.

Enhancements to the Web and Worker roles: with the introduction of Elevated Privileges and Full IIS support!!! – so we now can have multiple IIS sites per Web role and the ability to install IIS modules. (Cool!!)

Windows Azure will also provide Remote Desktop functionality, which enables customers to connect to a running instance of their application or service in order to monitor activity and troubleshoot common problems. So basically your Azure computing instances are no longer black-boxes. (Finally!!!! OMG, I am going to cry…)

The introduction of an Extra Small Windows Azure instance – great!!, now you can configure an instance to run low-priority Worker Roles, or Admin apps without ruining your budget.

Compute Instance Size

CPU

Memory

Instance Storage

I/O Performance

Cost per hour

Extra Small

1.0 GHz

768 MB

20 GB

Low

$0.05

A range of new networking functionality under the Windows Azure Virtual Network name was introduced. Windows Azure Connect (formerly Project Sydney), which enables a simple and easy-to-manage mechanism to setup IP-based network connectivity between on-premises and Windows Azure resources, is the first Virtual Network feature that we’ll be making available as a CTP later this year. With this, your can establish VPN between your on-premises servers and your cloud machines. Much needed for some enterprise scenarios.

The Windows Azure portal will also be improved with SL technologies, and with access to new diagnostic information including the ability to click on a role to see type, and deployment time. (Finally, for god sake!!!)

A much needed update for the pretty basic Database Manager for SQL Azure (formerly “Project Houston”) was also announced.

The situation is fairly common, you have been using TFS 2008 for ALM in your work environment for a while, but suddenly, you have access to the latest version: TFS 2010, and wonder how cool could be make use of all the new features included in this new major version (like collections, work item hierarchies and finally a better integration with MS Project). You followed the Upgrade Guide (kindly provided by the TFS Rangers) and after some work you finally have your precious Team Projects safe in their new 2010 home. Clean and straightforward upgrade, wasn’t it?

Opps, wait a minute!, why I do not have access to those new cool dashboards, and reports from my upgraded project portals? Ok, let’s take a look at those functional testing capabilities I’ve been waiting for a long time.. opps.. The new MS Test Manager cannot connect to my upgraded projects!! The reason for all this and many other symptoms: the upgrade process did not upgrade the definition of your Team Projects.

You ended up with the same Work Items, Reports, etc. that you had in TFS 2008. No way!!

Branch the code from each upgraded project to a brand new Team Project in TFS 2010. Remove permissions to the upgraded project. This way you could have your source code history accessible from the new Team Project. Obviously, you will have trouble if you accidentally delete your old upgraded project from TFS. If you could live without your history then simply move the latest version to the new Team Project. This is very easy, and it is a valid option, but you will probably have this uneasy feeling that it could be done in a more elegant way (see option 2).

Rebuild the entire definition of your Team Project to the latest ones included in TFS 2010!! This could sound a little drastic and unfortunately you will not easily find a comprehensive guide in the Web on how to accomplish this. That is exactly the goal of this article.

Guide to Rebuilding your Team Project Template

Note: this guide is based on the assumption that you can delete all the preexisting Work Items created in your upgraded projects. If you know how to overcome this limitation just let me know.

Log in your TFS 2010 server or a desktop machine with Team Explorer 2010 installed. You must be Project Collection Administrator to execute the following steps.

Download the latest CMMI and Agile Templates from your TFS 2010. Open the Team Explorer 2010 (or VS 2010) as an Administrator. Right-click on the Collection..Team project Collection Settings..Process Template Manager…and download both templates to a folder in your disk. We will use the path C:MSF for CMMI Process Improvement v5.0 in the following steps.

Open a command line prompt and change dir to C:<Program Files>Microsoft Visual Studio 10.0Common7IDE. You will need to execute the following steps for each one of the upgraded projects. In this case I will use a Collection named “Migrated”, and a Team Project called “TemplateChgTest”:

4.1 – Open Notepad and create a XML file called “ImCategories.xml” with the following content:

4.2 – Clean-up the current category list for the project. This would allow us to delete all the Work Item definitions. Import the empty Category list (ImCategories.xml) for the TemplateChgTest project.

4.6 – At his point you should be able to connect to your upgraded projects from MS Test Manager. You can also check your new definitions from Team Explorer (remember to refresh your projects or restart VS).

5 – Let’s rebuild the project portal and the new dashboards and reports. Details of this process are further explained here, but in summary you will need to do this:

5.1 – In case you want your portal in the same path, backup your upgraded content and delete the portal. Open the WSS-based project portal and delete it (Site Actions…Site Settings…Site Administration…Delete this site).

5.2 – Delete your upgraded SSRS reports. Find the SSRS Server URL in the TFS Admin Console applet (under the Reporting node). Open the SSRS Report Manager in your browser and delete the reports folder for the upgraded team project.

5.3 – Manually copy the Queries from a new Project using the same template. Just copy and paste the queries in the Team Explorer tree view.

5.4 – Create the following XML file in notepad (C:TFSUpgradeAddDashboards.xml):

Configure the parameter based on your environment. Remember that you can get the ProjectSiteWebApplication parameter from the TFS Admin Console (Under the Name column in the SharePoint Web Applications node.

5.5 Execute the rebuild process. For this, open the Command Window in VS2010 and enter the following command:

File.BatchNewTeamProject C:TFSUpgradeAddDashboards.xml

Wait for the completion of the command, and the check the generated log file for exceptions. This must contain a line to consider the process as successfully completed:

Recently, one of my customers asked me that question: “Based on the updated SQL Azure plans, the maximum database size is now 50GB. What if my DB requires more storage?

The first recommendation could be: try to measure how your DB is growing, and (if possible) try to have there only the most relevant information – SSIS is a great option to download all that historic data to your on-premises servers. Another option is Data Sync. Some good articles on measuring your DB size are:

Well, according to Microsoft 50GB is the maximum size, and if you need more space you will need to partition your data (either horizontally or vertically). Unfortunately, SQL Azure won’t help you much with this, and you will need to make some changes in your app logic to handle this. This should be done in your Data Access Layer, and it will not be an easy process to implement, let me warn you. Following articles could give you some insight on the details and limitations of this process: