Deployment and Development using PhoneGap Build for a Cordova/PhoneGap app

I recently worked on a project to build a mobile app in iOS and Android. The team decided to use Cordova and Ionic Framework to build it. I highly recommend Ionic – it’s an amazingly productive and feature-rich framework with a really responsive development team! It enabled us to use AngularJS (and the application structure and maintenance advantages that affords) and really sped up the development time in a number of ways. We were also able to get an almost-native performance experience for the application using features like collection repeat (virtualised list) as well as the in-built performance tuning the Ionic team have performed.

When it was time to set up an automated build pipeline we settled on using PhoneGap Build and its API which we then called from OctopusDeploy as part of a NuGet package we created.

PhoneGap Build is Adobe’s cloud build service that can build iOS, Android, Windows Phone etc. apps for you based on a zip file with your Cordova/PhoneGap files.

We decided on PhoneGap Build because:

  • It has been around a while (i.e. it’s relatively mature)
  • It’s backed by a big name (Adobe)
  • It has a decent API
  • It meant we didn’t have to set up a Mac server, which would have been a pain
  • It’s free for a single project 🙂

This post covers how we set up our automated deployment pipeline for our native apps using OctopusDeploy and PhoneGap Build. I have also published a companion post that covers everything you need to know about PhoneGap Build (read: what I wish I knew before I started using PhoneGap Build because it’s darn confusing!).

Deployment pipeline

This is the deployment pipeline we ended up with:

  1. Developer commits a change, submits a pull request and that pull request gets merged
  2. Our TeamCity continuous integration server compiles the server-side code, runs server-side code tests and runs client-side (app) tests
  3. If all is well then a NuGet package is created for the server-side code and a NuGet package is created for the app – both are pushed to OctopusDeploy and the server-side code is deployed to a staging environment (including a web server version of the app)
    • Native functionality wouldn’t work, but you could access and use the bulk of the app to test it out
    • To test native functionality developers could easily manually provision a staging version of the app if they plugged a phone into their computer via a USB cable and executed a particular script
  4. The product owner logs into OctopusDeploy and promotes the app to production from staging whereupon OctopusDeploy would:
    • Deploy the server-side code to the production environment
    • Use the PhoneGap Build API to build an iOS version with an over-the-air compatible provisioning profile
    • Download the over-the-air .ipa
      • Upload this .ipa to Azure blob storage
      • Beta testers could then download the latest version by visiting a html page linking to the package (see below for details)
    • Build the production Android and iOS versions of the app using the PhoneGap Build API
  5. The product owner then logs on to the PhoneGap Build site, downloads the .ipa / .apk files and uploads them to the respective app stores

End result

I personally wasn’t completely satisfied with this because I didn’t like the manual step at the end, however, apparently app store deployments are stuck in the 90s so unless you want to screen-scrape the respective app stores you have to manually log in and upload the deployment file. Regardless, the end result was good:

  • Production deployments were the same binaries/content that were tested on the CI server and that had been successfully deployed and manually tested in the staging environment
  • We could go from developer commit to production in a few minutes (ignoring app store approval/roll-out times; remember – stuck in the 90s)
  • The product owner had complete control to deploy any version of the app he wanted to and it was easy for him to do so

How we did it

I have published the NodeJS script we used to automate the PhoneGap Build API to a Gist. It contains a very reusable wrapper around a PhoneGap Build API NodeJS library that makes it dead easy to write your own workflow in a very succinct manner.

To generate the zip file the NodeJS script used we simply grabbed the www directory (including config.xml) from our repository and zipped it up. That zip file and the NodeJS scripts went into the NuGet package that OctopusDeploy deployed when the app was promoted to production (we made this happen on a Tentacle residing on the OctopusDeploy server). I have published how we generated that NuGet package to a Gist as well (including how to set up over-the-air).

Further notes

Some notes about our approach:

  • Because we were able to make our backend apis deploy to production at the same time as generating our production app packages we were assured that the correct version of the backend was always available when the apps were deployed to the respective app stores.
  • It was important for us to make sure that our backend api didn’t have any breaking changes otherwise existing app deployments (and any lingering ones where people haven’t updated) would break – usual API management/versioning practices apply.
  • Whenever we wanted to add new Beta testers the provisioning profile for over-the-air needed to be re-uploaded to PhoneGap Build
    • Unfortunately, PhoneGap Build requires you upload this alongside the private key – thus requiring someone with access to PhoneGap Build, the private key and the iOS developer account – it’s actually quite a convoluted process so I’ve also published the instructions we provided to the person responsible for managing the Beta testing group to a Gist to illustrate
  • We stored the passwords for the Android Keystore and iOS private keys as sensitive variables in OctopusDeploy so nobody needed to ever see them after they had been generated and stored in a password safe
    • This meant the only secret that was ever exposed on a regular basis was the password-protected over-the-air private key whenever a Beta tester was added

TestFlight

While we didn’t use it, it’s worth pointing out TestFlight. It’s a free service that gives you the ability to more easily keep track of a list of testers, their UUIDs, informing them of updates and automatically creating and hosting the necessary files for over-the-air. It contains an API that allows you to upload your .ipa to and it should be simple enough to take the scripts I published above and upload a .ipa to TestFlight rather than Azure Blob Storage if you are interested in doing that.

Final Thoughts

PhoneGap Build is a useful and powerful service. As outlined in my companion post, it’s very different to Cordova/PhoneGap locally and this, combined with confusing documentation, resulted in me growing a strong dislike of PhoneGap Build while using it.

In writing my companion post however, I’ve been forced to understand it a lot more and I now think that as long as you have weighed up the pros and cons and still feel it’s the right fit then it’s a useful service and shouldn’t be discounted outright.

Announcing AzureWebFarm.OctopusDeploy

I’m very proud to announce the public 1.0 release of a new project that I’ve been working on with Matt Davies over the last couple of weeks – AzureWebFarm.OctopusDeploy.

This project allows you to easily create an infinitely-scalable farm of IIS 8 / Windows Server 2012 web servers using Windows Azure Web Roles that are deployed to by an OctopusDeploy server.

If you haven’t used OctopusDeploy before then head over to the homepage and check it out now because it’s AMAZING.

The installation instructions are on the AzureWebFarm.OctopusDeploy homepage (including a screencast!), but in short it amounts to:

  1. Configure a standard Web Role project in Visual Studio
  2. Install-Package AzureWebFarm.OctopusDeploy
  3. Configure 4 cloud service variables – OctopusServerOctopusApiKeyTentacleEnvironment and TentacleRole
  4. Deploy to Azure and watch the magic happen!

We also have a really cool logo that a friend of ours, Aoife Doyle, drew and graciously let us use!AzureWebFarm.OctopusDeploy logo

 

It’s been a hell of a lot of fun developing this project as it’s not only been very technically challenging, but the end result is just plain cool! In particular the install.ps1 file for the NuGet package was very fun to write and results in a seamless installation experience!

Also, a big thanks to Nicholas Blumhardt, who gave me some assistance for a few difficulties I had with Octopus and implemented a new feature I needed really quickly!

ChameleonForms 1.0 Released

I’m incredibly excited and proud to finally announce the release of 1.0 of the ChameleonForms library I’ve been working on with Matt Davies. My blog has been fairly quiet the last couple of months while I’ve poured time and energy in finally getting ChameleonForms to 1.0.

WOOOOOOOOOOOOOOOO

(Apologies; I’m releasing months of built-up anticipation and excitement here).

I’m biased of course, but I think this library is amazing to use and results in immensely more maintainable form generation code when using ASP.NET MVC. It extends on the knowledge that MVC developers would have in generating forms using the (already pretty awesome) built-in stuff, but adds the things I think are missing. For me, this library epitomises over 7 years of exploration in the best way to do web-based forms and I’m excited to be able to share the beginnings of my current vision via this library.

What is ChameleonForms?

In short, ChameleonForms takes away the pain and repetition of building forms with ASP.NET MVC by following a philosophy of:

  • Model-driven defaults (e.g. enum is drop-down, [DataType(DataType.Password)] is password textbox)
  • DRY up your forms – your forms will be quicker to write and easier to maintain and you won’t get stuck writing the same form boilerplate markup form after form after form
  • Consistent – consistency of the API and form structure within your forms and consistency across all forms in your site via templating
  • Declarative syntax – specify how the form is structured rather than worrying about the boilerplate HTML markup of the form; this has the same beneficial effect as separating HTML markup and CSS
  • Beautiful, terse, fluent APIs – it’s a pleasure to read and write the code
  • Extensible and flexible core – you can extend or completely change anything you want at any layer of ChameleonForms and you can drop out to plain HTML at any point in your form for those moments where pre-prepared field types and templates just don’t cut it

More info.

What are the big improvements in 1.0?

We’ve been releasing pretty often so that depends on what version you are currently using, but these are the most important things:

  • Extensive usage across a number of production websites – we are happy that this library is mature, stable and ready for prime-time
  • Twitter Bootstrap 3 Template out-of-the-box supported by a NuGet package to get you up and running faster – this is HUGE for a number of reasons:
    • Bootstrap is pretty darn popular right now so this is immediately useful to a lot of people
    • In creating this template we had to do some pretty sophisticated changes to allow the template to drive a lot of changes unobtrusively to the form structure you are adding in your views – this is great because it means it’s really easy for you to create your own form templates and accomplish similarly complex transformations of your form markup
    • The ASP.NET MVC templates that come with Visual Studio 2013 come with Bootstrap by default now – and boy do they have gross repetitive boilerplate in them, which you can clean right up using this library
    • The vision that we have for this library is coming to fruition, which is personally gratifying – this is a beautiful demonstration of being able to declaratively specify the structure of your form and then completely change the markup/template of your form across a whole application with a single line of code when it changes
  • Really comprehensive documentation of everything in the library – we’ve spent many hours writing up the documentation – the idea was to make it comprehensive, but accessible/terse; hopefully we’ve met that goal!
  • Really solid code coverage to help prevent regressions or breaking changes as well as some refactorings that give us a solid codebase to continue with the other features we want to add – hopefully this can support us into the future with minimal breaking changes

How can I get it?

Checkout the GitHub release or go to NuGet.

Versioning

From this point on we are following semver thanks to the GitHubFlowVersion project. The fourth number in the NuGet version number is actually build metadata.

Borderlands 2 here I come!

Over a year ago now (yes it’s been a long journey – our first NuGet package was published on November 1, 2012) Matt Davies and I made an agreement to each other that neither of us would play the recently released Borderlands 2 game (we were both huge fans of Borderlands so this was a big deal) until we released 1.0 of ChameleonForms so that we would remain focussed on it and not get distracted. Now, while we both didn’t realise that it would take this long and while the last couple of months have seemed like forever (I’m pretty sure we had a phone conversation at least once a week where one of us would say “dude, we are sooooo close to 1.0 and BORDERLANDS 2!”) we are both incredibly proud of the library and are happy with what we’ve managed to get into 1.0.

Needless to say, we will probably be taking a break from open source for a few weekends to play Borderlands 2 🙂

We hope you enjoy using the library!

As usual hit us up with issues and pull requests on GitHub – they make our day 🙂

NQUnit update

I’ve just pushed a new version of NQUnit and NQUnit.NUnit (1.0.5). This update updates the libraries to use the latest version of Watin, NUnit, jQuery and QUnit. I’ve also made some documentation improvements. You can see the latest code and documentation on GitHub.

It’s worth noting that while NQUnit still works perfectly well, there is a better option that I usually recommend in preference: Chutzpah. I’ve updated NQUnit as per pull requests that the library received – while there are still people using the library I’m happy to keep it up to date.

Test Harness for NuGet install PowerShell scripts (init.ps1, install.ps1, uninstall.ps1)

One thing that I find frustrating when creating NuGet packages is the debug experience when it comes to creating the PowerShell install scripts (init.ps1, install.ps1, uninstall.ps1).

In order to make it easier to do the debugging I’ve created a test harness Visual Studio solution that allows you to make changes to a file, compile the solution, run a single command in the package manager and then have the package uninstall and then install again. That way you can change a line of code, do a few key strokes and then see the result straight away.

To see the code you can head to the GitHub repository. The basic instructions are on the readme:

  1. [Once off] Checkout the code
  2. [Once off] Create a NuGet source directory in checkout directory
  3. Repeat in a loop:
    1. Write the code (the structure of the solution is the structure of your nuget package, so put the appropriately named .ps1 scripts in the tools folder)
    2. Compile the solution (this creates a nuget package in the root of the solution with the name Package and version {yymm}.{ddHH}.{mmss}.nupkg – this means that the package version will increase with time so if you install from that directory it will always install the latest build) <F6> or <Ctrl+Shift+B)
    3. Switch to the Package Manager Console <Ctrl+P, Ctrl+M>
    4. [First time] Uninstall-Package Package; Install-Package Package <enter> / [Subsequent times] <Up arrow> <enter>
  4. When done simply copy the relevant files out and reset master to get a clean slate

Other handy hints

AzureWebFarm 0.9.2 Released

I’m happy to announce that I’ve just released AzureWebFarm 0.9.2 to Nuget.

This release involves months of work by myself and Matt Davies to really tighten up the farm, perform a lot of stress testing on it (including some reasonably heavy production loads) and take the (larger than we expected – I’ll do a separate blog post on this in the next few days) step to upgrade to Windows Server 2012 / .NET 4.5 support. We will be deploying this version to production farms over the next few weeks and if there are no issues then we will be releasing v1.0 in the near future.

I’m pleased to announce that with Windows Server 2012 and the Azure SDK 1.8 (which we’ve discovered comes with some quite massive and seemingly undocumented deployment speed improvements that we are reliably getting deployment times (including package upload) of around 9 minutes for a new web farm! We think this is incredible that you can get an enterprise-grade, clustered, load-balanced, IIS8 / Windows Server 2012 powered web farm with background execution capabilities that you can scale up and down in less than 10 minutes!!!

What’s new?

The changes we’ve introduced in this version are (in no particular order):

  • If a web.config file is included with a background worker application then it will no longer cause an exception in the web farm and in fact will not be overwritten
  • Upgraded to Azure SDK 1.8 along with the deployment speed improvements this entails
  • Decreased the number of files that are needed in the web farm web project significantly (now it’s just web.config, app.config, Probe.aspx, Probe.aspx.cs and WebRole.cs)
  • Added missing HTTP certificate config in the example cloud project config files
  • Set a bunch of internally used classes to `internal` from `public` to reduce the possibility of any confusion
  • Refactored core code to make it easier to unit test and maintain
  • Improved the test coverage of the codebase
  • Changes to config settings while the farm is deployed will now update the farm without requiring the roles to be manually restarted (the roles won’t automatically restart either – they will always use the latest version of the config settings)
  • Added handling to OnStop to ensure all ASP.NET requests are served before a role is restarted as per http://blogs.msdn.com/b/windowsazure/archive/2013/01/14/the-right-way-to-handle-azure-onstop-events.aspx
  • Added configurable logging via Castle.Core so you can adjust the verbosity of the trace logs and intercept the logs if you would like to redirect it to different instrumentation
  • Removed dependencies on Azure Storage within uncaught code called from OnRun() – this means that the web farm should not go down if there is an Azure Storage outage
  • Added configuration setting to allow for syncing to be disabled without needing to redeploy the farm
  • Added functionality to ensure all web deploy connections get funnelled to a single role instance at a time (with robust failover if that role instance fails). This resolves an issue when using MsDeploy v3+ on the server that we discovered and will be documented in a later blog post.
  • Changed the example config files to use Windows Server 2012
  • Opened up a readme.txt file when you install the NuGet package to outline any breaking changes
  • Improved documentation on github

Breaking Changes

If you are upgrading from a previous version of Azure Web Farm then make sure you look at the breaking changes file to see upgrade instructions as there are a number of breaking changes in this release.

Future Direction

What’s in store for Azure Web Farm in the future? This is a rough idea of the sorts of things we are looking into (we don’t have a timeline on this – if you want to help out to make it happen faster though please contact us):

  • Out of the box auto-scaling support using Wasabi
  • Better installation experience – using nuget install scripts to take away some of the tedious manual configuration when creating a new farm
  • Improving the syncing speed – it currently takes 30-90s to sync a new deployment across the farm and we are confident we can get that down significantly

On myself and FluentMVCTesting joining TestStack

I’d like to announce that I’m joining TestStack and my FluentMCVTesting library will become a TestStack library.

What is TestStack?

I think it can be summed up nicely by a post from one of the core TestStack team, Michael Whelan, in the blog post he wrote. tl;dr: TestStack is a collection of open source projects which mainly focus on making the left side of Marrick’s Agile Testing Quadrant easier!

Why am I joining TestStack?

I think that the libraries in TestStack are awesome and the core team are a group of people I trust and respect. It’s a pleasure to be invited into the core team and I think it’s really great for the FluentMVCTesting library to be present alongside the other awesome testing libraries.

What will happen to FluentMVCTesting?

The nuget package and namespace is changing to have a TestStack prefix for consistency with other TestStack libraries, but other than that nothing much – the library is the same library, but it’s more likely to have additions made to it and hopefully more people using it.

If you update the existing FluentMVCTesting package then it will include the new package as a dependency and wrap around the new classes, but mark the old classes as deprecated. This should provide a smooth transition to existing users of the package as well as letting them know there is a new package out there.

I’m also dropping the strong signing support for now to keep consistency with the other TestStack libraries. We will add that in due course to all the libraries if we feel there is enough demand for it.

Announcing ChameleonForms 0.1

One of the things I find frustrating with ASP.NET MVC out of the box is creating forms. That’s not to say I find it more frustrating than other frameworks or libraries though; on the contrary I think it’s painful almost always. I think ASP.NET MVC has an amazingly powerful and useful capacity to handle forms. However, out of the box I find it fiddly and annoying to output the HTML associated with the form in a consistent and maintainable way across a whole website.

The other problem occurs when you want to change the layout of the form. For instance, imagine I create a proof-of-concept application using Twitter Bootstrap to get up and running with something that looks nice quickly. I may well want to scrap the application because it turns out to not be popular or useful. On the other hand, if it became really popular then I might want to spend some more time and effort customising the look-and-feel. At that point I will more than likely want to move away from the HTML template that Twitter Bootstrap imposes on you to use a more semantic markup (e.g. definition lists).

Enter Chameleon Forms

For the last 2 years, while I was working at Curtin University I was lucky enough to be able to use a library that we developed for tersely and maintainably developing forms that conformed to the forms standard that we had at Curtin. We had the advantage that the library was able to be specific to the Curtin standard for forms and didn’t have to be very generic. We still ensured that the library was very flexible though, because we want to make sure we could use it for most of the use cases we came across and still be able to break out into straight HTML for the weird edge cases.

ChameleonForms is my attempt to use the learnings and concepts from the previous forms library I was using and apply them to a new, open source library that I can use on my personal and professional projects to ease the pain of creating forms and improve the maintainability of them.

I am implementing this library alongside Matt Davies, who I often work closely with.

This blog post is announcing the initial release of this library, which contains basic functionality only. There is more information about the library, an example of how to use it and the roadmap we have for it at the Github page.

If you want to install this initial version from NuGet then simply use:

Install-Package ChameleonForms

Enjoy!

FluentMVCTesting: Fluent, terse and maintainable ASP.NET MVC controller testing

I would like to announce the release of a software library I have been using for the last year or so as open source via NuGet / Github: FluentMVCTesting.

The motivation behind this library is to provide a way to test MVC actions quickly, tersely and maintainably. Most examples I find on MVC controller testing are incredibly verbose, repetitive and time-consuming to write and maintain.

Given how quickly you can write controller actions and how simple they are (assuming you are following best practices and keeping them lean) the time to test them generally isn’t worth it given you can glance at most of your controller actions and know they are right or wrong instantly.

This library aims to make the time to implement the tests inconsequential and then the value your tests are providing is worth it.

The other problem that I’ve noticed with most examples of controller testing is that there are a lot of magic strings being used to test view and action names; this library also aims to (where possible) utilise the type system to resolve a lot of those magic strings, thus ensuring your tests are more maintainable and require less re-work when you perform major refactoring of your code.

I came up with this library after using the MVCContrib.TestHelper library for quite a while, but becoming frustrated with it; the library was initially created during the experiment I conducted to try and create terse controller tests (which I still have to write a follow-up post for at some point!). I have been using the library for over a year on a number of projects both personally and professionally.

A small code snippet to whet your appetite (go to the NuGet site for more detailed documentation and examples):

// Check that a post to the Login action with model errors returns the Login view passing through the model
var vm = new LoginViewModel();
_controller.WithModelErrors().WithCallTo(c => c.Login(vm)).ShouldRenderDefaultView().WithModel(vm);

// Check that a post to the Login action without model errors redirects to the site homepage
_controller.WithCallTo(c => c.Login(new LoginViewModel())).ShouldRedirectTo<HomeController>(c => c.Index());

Enjoy!

Getting up and running with Database testing quickly in .NET

When I first started learning about how to do database code in .NET and in particular how to test your database code I came across a really useful post on the Code Project.

It basically goes through a tutorial on how to get a base class set up that sets up a brand new SQL Server CE database every test run and populates it from an NHibernate configuration.

This was great, but there were a number of issues that caused it to not work for me (including the fact I decided to use Fluent NHibernate rather than writing a heap of obscure XML). As well as this, there were a couple of fundamental flaws in the way the test worked and it didn’t work with the latest versions of SQL Server CE and NHibernate at the time.

Consequently, I fixed up the code so it worked and I wanted to outline those changes and the final result here in case anyone finds it useful.

Continue reading “Getting up and running with Database testing quickly in .NET”