Practical Microsoft Azure Active Directory Blog Series

I finally had a chance to play with Microsoft Azure Active Directory in a recent project. I found the experience to be very interesting – Azure AD itself is an amazing, powerful product with a lot of potential. It certainly has a few rough edges here and there, but it’s pretty clear Microsoft are putting a lot of effort into it as it’s forming the cornerstone of how it authenticates all of it’s services including Office 365.

Azure AD gives you the ability to securely manage a set of users and also gives the added benefit of allowing two-factor authentication (2FA), single-sign-on across applications, multi-tenancy support and ability to allow external organisations to authenticate against your application.

This blog series will outline the minimum set of steps that you need to perform to quickly and easily add Azure AD authentication to an existing ASP.NET MVC 5 (or 3 or 4) site (or a new one if you select the No Authentication option when creating it) as well as configure things like API authentication, role authorisation, programmatic logins and deployments to different environments.

There are already tools and libraries out there for this – why are you writing this series?

Microsoft have made it fairly easy to integrate Azure AD authentication with your applications by providing NuGet packages with most of the code you need and also tooling support to configure your project in Visual Studio. This is combined with a slew of MSDN and technet posts covering most of it.

When it comes to trying to understand the code that is added to your solution however, things become a bit tricky as the documentation is hard to navigate through unless you want to spend a lot of time. Also, if you have Visual Studio 2013 rather than Visual Studio 2012 you can only add authentication to a new app as part of the File -> New Project workflow by choosing the Organizational Authentication option:

ASP.NET Organizational Authentication option
Visual Studio: File > New Project > ASP.NET > Change Authentication > Organizational Authentication

If you have an existing ASP.NET web application and you are using Visual Studio 2013 then you are out of luck.

Furthermore, the default code you get requires you to have Entity Framework and a database set up, despite the fact this is only really required if you are using multiple Azure AD tenants (unlikely unless you are creating a fairly hardcore multi-tenant application).

If you then want to add role-based authentication based on membership in Azure AD groups then there is no direction for this either.

For these reasons I’m developing a reference application that contains the simplest possible implementation of adding these features in an easy to follow commit-by-commit manner as a quick reference. I will also provide explanations of what all the code means in this blog series so you can understand how it all works if you want to.

You can see the source code of this application here and an example deployment here. The GitHub page outlines information such as example user logins and what infrastructure I set up in Azure.

What are you planning on covering?

This will be the rough structure of the posts I am planning in no particular order (I’ll update this list with links to the posts over time):

I’m notoriously bad at finishing blog series‘ that I start, so no promises on when I will complete this, but I have all of the code figured out in one way or another and the GitHub should at least contain commits with all of the above before I finish the accompanying posts so *fingers crossed*! Feel free to comment below if you want me to expedite a particular post.

More resources

I came across some great posts that have helped me so far so I thought I’d link to them here to provide further reading if you are interested in digging deeper:

Introduction to Windows/Microsoft Azure

I often have people ask me how they can get started with Windows Microsoft Azure (that one’s gonna take a while for me). Generally I start by telling them to grab a free trial and start off by publishing some web apps using Azure Web Sites. It’s great to start there because it’s so easy to get up and running and it gives you a nice introduction to the Azure portal and some of the awesome diagnostics and configuration features that appear for a lot of the Azure services.

Recently, I did a few hours of presenting and lab facilitation at the Perth event for Global Windows Azure Bootcamp with Matt Davies. In preparation for that event we prepared some content (slides, live demos and hands-on labs) on our open source organisation (MRCollective). If you are learning Azure then I encourage you to check it out, since I think it gives a nice overview of the main features of Azure you are likely to start out using as well as some examples of how to use them and get started with them.

It’s a tiny bit out of date after the huge amount of announcements that were made about Azure after that event, but it’s still 95% right and all the hands-on labs and demos should still work.

Windows Azure Whitepapers

My latest whitepaper for my employer, Readify, has been published. That makes two whitepapers I’ve written now.

  • Cloud and Windows Azure Introduction
    Cloud Computing is an often talked about and often misunderstood phenomenon. While many people disagree about what it is, most people agree that it is a revolutionary shift in the way computing resources are consumed and delivered and will have a big impact on the future. This paper consists of two parts; firstly, it discusses what cloud computing is, what benefits cloud computing can deliver to your business and what things to look out for when making a decision about cloud computing. The second part of the paper gives an overview of the Microsoft Windows Azure cloud computing platform and outlines the benefits that it can deliver to your business.
  • Azure Web Apps: Developing and Deploying Web Applications in Windows Azure
    When deploying web applications in Windows Azure there are a number of options available to you. It’s not always immediately clear from the documentation which of the options are the right ones to use in any given situation.This whitepaper will explain the three main options that are available in Windows Azure to deploy web applications and will cover the similarities and differences between them in order to make the decision of which to use easier. It will also cover the different deployment strategies available to you after you have made the decision.

Announcing AzureWebFarm.OctopusDeploy

I’m very proud to announce the public 1.0 release of a new project that I’ve been working on with Matt Davies over the last couple of weeks – AzureWebFarm.OctopusDeploy.

This project allows you to easily create an infinitely-scalable farm of IIS 8 / Windows Server 2012 web servers using Windows Azure Web Roles that are deployed to by an OctopusDeploy server.

If you haven’t used OctopusDeploy before then head over to the homepage and check it out now because it’s AMAZING.

The installation instructions are on the AzureWebFarm.OctopusDeploy homepage (including a screencast!), but in short it amounts to:

  1. Configure a standard Web Role project in Visual Studio
  2. Install-Package AzureWebFarm.OctopusDeploy
  3. Configure 4 cloud service variables – OctopusServerOctopusApiKeyTentacleEnvironment and TentacleRole
  4. Deploy to Azure and watch the magic happen!

We also have a really cool logo that a friend of ours, Aoife Doyle, drew and graciously let us use!AzureWebFarm.OctopusDeploy logo

 

It’s been a hell of a lot of fun developing this project as it’s not only been very technically challenging, but the end result is just plain cool! In particular the install.ps1 file for the NuGet package was very fun to write and results in a seamless installation experience!

Also, a big thanks to Nicholas Blumhardt, who gave me some assistance for a few difficulties I had with Octopus and implemented a new feature I needed really quickly!

The Idempotency issue when retrying commands with Azure SQL Database (SQL Azure)

There is a lot of information available about dealing with transient errors that occur when using Azure SQL Database. If you are using it then it’s really important to take the transient errors into account since it’s one of the main differences that Azure SQL has when compared to SQL Server.

If you are using .NET then you are in luck because Microsoft have provided an open source library to detect and retry for these transient errors (The Transient Fault Handling Application Block). I have blogged previously about how the guidance that comes with the application block (along with most of the posts, tutorials and forum posts about it) indicate that you need to completely re-architect your system to wrap every single database statement with a retry.

I wasn’t satisfied with that advice and hence I created NHibernate.SqlAzure and more recently ReliableDbProvider (works with ADO.NET, EntityFramework, LinqToSql, etc.). These frameworks allow you to drop in a couple of lines of configuration at one place in your application and unobtrusively get transient fault handling in your application.

Easy right? A silver bullet even? Unfortunately, no.

The Idempotency issue

Today I was made aware of a post made by a Senior Program Manager on the SQL Server team that was posted a few months ago about the Idempotency issue with Azure SQL Database. Unfortunately, I haven’t been able to find any more information about it – if you know anything please leave a comment.

The crux of the problem is that it is possible for a transient error to be experienced by the application when in fact the command that was sent to the server was successfully processed. Obviously, that won’t have any ill-effect for a SELECT statement, and if the SELECT is retried then there is no problem. When you have write operations (e.g. INSERTs, UPDATEs and DELETEs) then you can start running into trouble unless those commands are repeatable (i.e. idempotent).

This is particularly troubling (although in retrospect not terribly surprising) and the frustrations of one of the commenters from the post sums up the situation fairly well (and in particular calls out how impractical the suggested workaround given in the post is):

How exactly would this work with higher abstraction ORMs such as Entity Framework? The updates to a whole entity graph are saved as a whole, along with complex relationships between entities. Can entity updates be mapped to stored procedures such as this in EF? I completely appreciate this post from an academic perspective, but it seems like an insane amount of work (and extremely error-prone) to map every single update/delete operation to a stored procedure such as this.

Approaches

After giving it some consideration and conferring with some of my colleagues, I can see a number of ways to deal with this (you could do something like what was suggested in the post linked to above, but frankly I don’t think it’s practical so I’m not including it). If you have any other ideas then please leave a comment below.

  1. Do nothing: transient faults (if you aren’t loading the database heavily) are pretty rare and within that the likelihood of coming across the idempotency issue is very low
    • In this case you would be making a decision that the potential for “corrupt” data is of a lower concern than application complexity / overhead / effort to re-architect
    • If you do go down this approach I’d consider if there is some way you can monitor / check the data to try and detect if any corruption has occurred
    • Unique keys are your friend (e.g. if you had a Member table with an identity primary key and some business logic that said emails must be unique per member then you can use a unique key on Member.Email to protect duplicate entries)
  2. Architect your system so that all work to the database is abstracted behind some sort of unit of work pattern and that the central code that executes your unit of work contains your retry logic
    • For instance if using NHibernate you could throw away the session on a transient error, get another one and retry the unit of work
    • While this ensures the integrity of your transactions it does have the potential side-effect of making everything a lot slower since any transient errors will cause the whole unit of work to retry (which could potentially be slow)
  3. Ensure all of your commands are idempotent
    • While on the surface this doesn’t sound much better than having to wrap all commands with transient retry logic it can be quite straightforward depending on the application because most update and delete commands are probably idempotent already
    • Instead of using database-generated identities for new records use application generated identities (for instance generate a GUID and assign it to the id before inserting an entity) and then your insert statements will also be idempotent (assuming the database has a primary key on the id column)
    • NHibernate has automatic GUID generation capabilities for you and you can use the Comb GUID algorithm to avoid index fragmentation issues within the database storage
    • Alternatively, you can use strategies to generate unique integers like HiLo in NHibernate or SnowMaker
    • If you are doing delete or update statements then you simply need to ensure that they can be executed multiple times with the same result – e.g. if you are updating a column based on it’s current value (e.g. UPDATE [table] SET [column] = [column] + 1 WHERE Id = [Id]) then that could be a problem if it executed twice
  4. Retry for connections only, but not commands
  5. Retry for select statements only, but not others (e.g. INSERT, UPDATE, DELETE)
  6. Don’t use Azure SQL, but instead use SQL Server on an Azure Virtual Machine

Recommendations

With all that in mind, here are my recommendations:

  • Don’t shy away from protecting against transient errors – it’s still important and the transient errors are far more likely to happen than this problem
  • Use application-generated ids for table identifiers
  • Consider what approach you will take to deal with the idempotency issue (as per above list)

Unobtrusive transient fault handling in Azure SQL Database with Entity Framework, Linq To Sql, NHibernate and ADO.NET

It’s fairly well known know that when connecting to an Azure SQL Database you need to protect your application from transient errors that occur when using it.

Luckily, Microsoft provide a library that takes care of most of the effort of detecting these errors and retrying your operations according to a retry policy.

The problem with this library is that most of the documentation out there tells you to essentially wrap every database call in your system in a retry call using the library. Let’s be honest – this is a crappy solution to the problem, particularly for legacy code bases where this might result in a major set of changes to the codebase.

Luckily, there is a way around it:

  • If you are using Entity Framework 6 then check out the new Connection Resiliency functionality, although note that the default implementation doesn’t retry for timeout errors (see the documentation of the below mentioned libraries to see why you might care about that)
  • If you are using NHibernate then you can use the library I’ve previously announced/released; NHibernate.SqlAzure
  • If you are using EntityFramework then the best unobtrusive solutions on the net previously have retries for connection establishment (that have other potential connection management side-effects) and save operations, but not command retries – I’d like to announce the release of a library I have created called ReliableDbProvider that solves these problems in an unobtrusive way (generally you will only need to change your config and possibly one or two lines of code)
  • If you are using ADO.NET (provided you use the Db Provider functionality for managing connections and commands) or Linq To Sql then the ReliableDbProvider library can be used as well

Simulating transient errors when using Azure SQL Database

I recently gave a presentation on the ins and outs of using Azure SQL for the Perth Ms Cloud User Group on Azure SQL Database. If you are curious feel free to check out the presentation file at: https://github.com/robdmoore/SQLAzurePresentation

One of the things I did for the presentation was come up with a demo that illustrated the need for, and impact of, having transient error protection when talking to an Azure SQL Database. I have taken this code and cleaned it up a bit and released it on Github at: https://github.com/robdmoore/SQLAzureTransientDemo

This code is quite unique since I have managed to find a way to reliably elicit the transient exceptions (something I haven’t seen anywhere else on the net). If you need to test some code that needs these exceptions to happen to run and you want to be really sure the code works then feel free to check out the code.

Windows Azure Web Sites Programmatic Web Deploy

I was setting up a continuous integration / deployment pipeline last night for my team at GovHack Perth and I had a lot of trouble getting the web deploy deployment working against an Azure Web Sites site.

I had downloaded the .publishsettings file and copied out the username, password and server to pass through to the .cmd file that is generated by MSBuild when creating a web deploy package. I do this because I create the package separately as part of the CI build and attach it as a build artifact and then separately do the deployment with msdeploy as another CI build step.

The problem I came across was that I kept getting the following error when running the deployment:

C:\Path\To\Project\obj\Release\Package>projectname.deploy.cmd /T /M:https://waws-prod-hk1-001.pu
blish.azurewebsites.windows.net:443/msdeploy.axd /U:$sitename /P:password /A:Basic
SetParameters from:
"C:\Path\To\Project\obj\Release\Package\projectname.SetParameters.xml"
You can change IIS Application Name, Physical path, connectionString
or other deploy parameters in the above file.
-------------------------------------------------------
Start executing msdeploy.exe
-------------------------------------------------------
"C:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe" -source:package='C:
\Path\To\Project\obj\Release\Package\projectname.zip' -dest:auto,computerName="https://waws-prod
-hk1-001.publish.azurewebsites.windows.net:443/msdeploy.axd",userName="$sitename"
,password="password",authtyp
e="Basic",includeAcls="False" -verb:sync -disableLink:AppPoolExtension -disableL
ink:ContentExtension -disableLink:CertificateExtension -setParamFile:"C:\Path\To\Project\obj
\Release\Package\projectname.SetParameters.xml" -whatif
Info: Using ID '744a18b1-51c2-4d05-a674-6c407a7f80c5' for connections to the rem
ote server.
Error Code: ERROR_USER_UNAUTHORIZED
More Information: Connected to the remote computer ("waws-prod-hk1-001.publish.a
zurewebsites.windows.net") using the Web Management Service, but could not autho
rize. Make sure that you are using the correct user name and password, that the
site you are connecting to exists, and that the credentials represent a user who
has permissions to access the site. Learn more at: http://go.microsoft.com/fwl
ink/?LinkId=221672#ERROR_USER_UNAUTHORIZED.
Error: The remote server returned an error: (401) Unauthorized.
Error count: 1.

It turns out the problem was I needed to include the ?site=sitename parameter in the msdeploy destination otherwise (I assume) the load balancer on the publish server can’t authenticate your username and password properly. This meant I needed to switch to using msdeploy.exe directly like so:

"C:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe" -source:package='projectname.zip' -dest:auto,computerName="https://waws-prod-hk1-001.publish.azurewebsites.windows.net:443/msdeploy.axd?site=sitename",userName="$sitename",password="password",authtype="Basic",includeAcls="False" -verb:sync -disableLink:AppPoolExtension -disableLink:ContentExtension -disableLink:CertificateExtension -setParamFile:"projectname.SetParameters.xml"

Hopefully this post helps other people stuck in the same situation (I couldn’t find anything on Google to help with this)!

AzureWebFarm 0.9.2 Released

I’m happy to announce that I’ve just released AzureWebFarm 0.9.2 to Nuget.

This release involves months of work by myself and Matt Davies to really tighten up the farm, perform a lot of stress testing on it (including some reasonably heavy production loads) and take the (larger than we expected – I’ll do a separate blog post on this in the next few days) step to upgrade to Windows Server 2012 / .NET 4.5 support. We will be deploying this version to production farms over the next few weeks and if there are no issues then we will be releasing v1.0 in the near future.

I’m pleased to announce that with Windows Server 2012 and the Azure SDK 1.8 (which we’ve discovered comes with some quite massive and seemingly undocumented deployment speed improvements that we are reliably getting deployment times (including package upload) of around 9 minutes for a new web farm! We think this is incredible that you can get an enterprise-grade, clustered, load-balanced, IIS8 / Windows Server 2012 powered web farm with background execution capabilities that you can scale up and down in less than 10 minutes!!!

What’s new?

The changes we’ve introduced in this version are (in no particular order):

  • If a web.config file is included with a background worker application then it will no longer cause an exception in the web farm and in fact will not be overwritten
  • Upgraded to Azure SDK 1.8 along with the deployment speed improvements this entails
  • Decreased the number of files that are needed in the web farm web project significantly (now it’s just web.config, app.config, Probe.aspx, Probe.aspx.cs and WebRole.cs)
  • Added missing HTTP certificate config in the example cloud project config files
  • Set a bunch of internally used classes to `internal` from `public` to reduce the possibility of any confusion
  • Refactored core code to make it easier to unit test and maintain
  • Improved the test coverage of the codebase
  • Changes to config settings while the farm is deployed will now update the farm without requiring the roles to be manually restarted (the roles won’t automatically restart either – they will always use the latest version of the config settings)
  • Added handling to OnStop to ensure all ASP.NET requests are served before a role is restarted as per http://blogs.msdn.com/b/windowsazure/archive/2013/01/14/the-right-way-to-handle-azure-onstop-events.aspx
  • Added configurable logging via Castle.Core so you can adjust the verbosity of the trace logs and intercept the logs if you would like to redirect it to different instrumentation
  • Removed dependencies on Azure Storage within uncaught code called from OnRun() – this means that the web farm should not go down if there is an Azure Storage outage
  • Added configuration setting to allow for syncing to be disabled without needing to redeploy the farm
  • Added functionality to ensure all web deploy connections get funnelled to a single role instance at a time (with robust failover if that role instance fails). This resolves an issue when using MsDeploy v3+ on the server that we discovered and will be documented in a later blog post.
  • Changed the example config files to use Windows Server 2012
  • Opened up a readme.txt file when you install the NuGet package to outline any breaking changes
  • Improved documentation on github

Breaking Changes

If you are upgrading from a previous version of Azure Web Farm then make sure you look at the breaking changes file to see upgrade instructions as there are a number of breaking changes in this release.

Future Direction

What’s in store for Azure Web Farm in the future? This is a rough idea of the sorts of things we are looking into (we don’t have a timeline on this – if you want to help out to make it happen faster though please contact us):

  • Out of the box auto-scaling support using Wasabi
  • Better installation experience – using nuget install scripts to take away some of the tedious manual configuration when creating a new farm
  • Improving the syncing speed – it currently takes 30-90s to sync a new deployment across the farm and we are confident we can get that down significantly

Announcing NHibernate.SqlAzure version 1.0!

I’m proud to be able to announce the release of version 1.0 of NHibernate.SqlAzure!

This library takes care of retrying when the transient errors that SQL Azure throws at you occur while using the NHibernate ORM. It’s been in Beta for the last few months and has been successfully used on a number of production websites.

Changes from 0.9 to 1.0

  • Bug fix when using Schema validation (thanks to @hmvs)
  • There is now a transient error detection strategy and associated NHibernate driver (SqlAzureClientDriverWithTimeoutRetries; say that 10 times fast!!) that retries for timeout exceptions (see the Github page for details and also thanks to @hmvs for a contribution towards this)
  • Some instances where exceptions were wrapped in NHibernate exceptions (batching, transactions) are now picked up as transient exceptions when before they were ignored
  • You can now easily log connection and command exceptions separately (see the CommandRetry and ConnectionRetry virtual methods on the driver class you use)
  • The documentation is a bit more comprehensive now
  • I finished writing all the automated tests I wanted to
  • Been road-tested on a number of sites over the last few months in production

This project is a collaborative effort along with my partner in code crime – Matt Davies – all code was either pair programmed together or reviewed by the other party.

Go and grab it from NuGet today and let me know how you go! Installation / usage instructions are on the Github page.