MsDeploy to Azure Web App with Application Insights extension enabled when deleting additional destination files

When performing an MsDeploy to an Azure Web App and you have the App Insights extension enabled you may find something interesting happens if you use the option to delete additional files on the destination that don’t appear in the source. If you look at the deployment log you may see something like this:

2017-01-30T07:29:27.5515545Z Info: Deleting file ({sitename}\ApplicationInsights.config).
2017-01-30T07:29:27.5515545Z Info: Deleting file ({sitename}\App_Data\packages\Microsoft.ApplicationInsights.2.2.0\Microsoft.ApplicationInsights.2.2.0.nupkg).
2017-01-30T07:29:27.5515545Z Info: Deleting directory ({sitename}\App_Data\packages\Microsoft.ApplicationInsights.2.2.0).
2017-01-30T07:29:27.5515545Z Info: Deleting file ({sitename}\App_Data\packages\Microsoft.ApplicationInsights.Agent.Intercept.2.0.6\Microsoft.ApplicationInsights.Agent.Intercept.2.0.6.nupkg).
2017-01-30T07:29:27.5515545Z Info: Deleting directory ({sitename}\App_Data\packages\Microsoft.ApplicationInsights.Agent.Intercept.2.0.6).
2017-01-30T07:29:27.5515545Z Info: Deleting file ({sitename}\App_Data\packages\Microsoft.ApplicationInsights.Azure.WebSites.2.2.0\Microsoft.ApplicationInsights.Azure.WebSites.2.2.0.nupkg).
2017-01-30T07:29:27.5515545Z Info: Deleting directory ({sitename}\App_Data\packages\Microsoft.ApplicationInsights.Azure.WebSites.2.2.0).
2017-01-30T07:29:27.5525645Z Info: Deleting file ({sitename}\App_Data\packages\Microsoft.ApplicationInsights.DependencyCollector.2.2.0\Microsoft.ApplicationInsights.DependencyCollector.2.2.0.nupkg).
2017-01-30T07:29:27.5525645Z Info: Deleting directory ({sitename}\App_Data\packages\Microsoft.ApplicationInsights.DependencyCollector.2.2.0).
2017-01-30T07:29:27.5525645Z Info: Deleting file ({sitename}\App_Data\packages\Microsoft.ApplicationInsights.PerfCounterCollector.2.2.0\Microsoft.ApplicationInsights.PerfCounterCollector.2.2.0.nupkg).
2017-01-30T07:29:27.5525645Z Info: Deleting directory ({sitename}\App_Data\packages\Microsoft.ApplicationInsights.PerfCounterCollector.2.2.0).
2017-01-30T07:29:27.5525645Z Info: Deleting file ({sitename}\App_Data\packages\Microsoft.ApplicationInsights.Web.2.2.0\Microsoft.ApplicationInsights.Web.2.2.0.nupkg).
2017-01-30T07:29:27.5525645Z Info: Deleting directory ({sitename}\App_Data\packages\Microsoft.ApplicationInsights.Web.2.2.0).
2017-01-30T07:29:27.5525645Z Info: Deleting file ({sitename}\App_Data\packages\Microsoft.ApplicationInsights.WindowsServer.2.2.0\Microsoft.ApplicationInsights.WindowsServer.2.2.0.nupkg).
2017-01-30T07:29:27.5525645Z Info: Deleting directory ({sitename}\App_Data\packages\Microsoft.ApplicationInsights.WindowsServer.2.2.0).
2017-01-30T07:29:27.5525645Z Info: Deleting file ({sitename}\App_Data\packages\Microsoft.ApplicationInsights.WindowsServer.TelemetryChannel.2.2.0\Microsoft.ApplicationInsights.WindowsServer.TelemetryChannel.2.2.0.nupkg).
2017-01-30T07:29:27.5525645Z Info: Deleting directory ({sitename}\App_Data\packages\Microsoft.ApplicationInsights.WindowsServer.TelemetryChannel.2.2.0).
2017-01-30T07:29:27.5525645Z Info: Deleting file ({sitename}\App_Data\packages\Microsoft.Web.Infrastructure.1.0.0.0\Microsoft.Web.Infrastructure.1.0.0.0.nupkg).
2017-01-30T07:29:27.5525645Z Info: Deleting directory ({sitename}\App_Data\packages\Microsoft.Web.Infrastructure.1.0.0.0).
2017-01-30T07:29:27.5535680Z Info: Deleting directory ({sitename}\App_Data\packages).
2017-01-30T07:29:27.5535680Z Info: Deleting file ({sitename}\bin\Microsoft.AI.Agent.Intercept.dll).
2017-01-30T07:29:27.5535680Z Info: Deleting file ({sitename}\bin\Microsoft.AI.DependencyCollector.dll).
2017-01-30T07:29:27.5535680Z Info: Deleting file ({sitename}\bin\Microsoft.AI.HttpModule.dll).
2017-01-30T07:29:27.5535680Z Info: Deleting file ({sitename}\bin\Microsoft.AI.PerfCounterCollector.dll).
2017-01-30T07:29:27.5535680Z Info: Deleting file ({sitename}\bin\Microsoft.AI.ServerTelemetryChannel.dll).
2017-01-30T07:29:27.5535680Z Info: Deleting file ({sitename}\bin\Microsoft.AI.Web.dll).
2017-01-30T07:29:27.5535680Z Info: Deleting file ({sitename}\bin\Microsoft.AI.WindowsServer.dll).
2017-01-30T07:29:27.5535680Z Info: Deleting file ({sitename}\bin\Microsoft.ApplicationInsights.AzureWebSites.dll).
2017-01-30T07:29:27.5535680Z Info: Deleting file ({sitename}\bin\Microsoft.ApplicationInsights.dll).

The cool thing about this is it gives you an indication of what the extension actually does. The fact there is an App_Data/packages folder with what is clearly unpacked NuGet packages tells us that the extension is installing a NuGet package into your site for you. That makes a lot of sense given you don’t need to install the extension if you installed the NuGet package yourself (I generally don’t bother because I don’t need App Insights locally and see it as a deployment concern so I like App Service adding it for me :)).

Setting the MsDeploy option to delete extraneous files is very useful so it’s not something I want to simply turn off. However, a knowledge of MsDeploy gives us some indication as to a possible solution. In this case we can make use of the skip option to specify that MSDeploy should ignore the above affected files.

Putting it all together, if you specify the following rules in your msdeploy.exe call then you should have success:

-skip:objectname='filePath',absolutepath='ApplicationInsights.config' -skip:objectname='dirPath',absolutepath='App_Data\\packages\\*.*' -skip:objectname='filePath',absolutepath='bin\\Microsoft.AI.*.dll'  -skip:objectname='filePath',absolutepath='bin\\Microsoft.ApplicationInsights.*.dll'

After doing that your deployment log should look something like this:

2017-01-30T08:10:26.7172425Z Info: Object filePath ({sitename}\ApplicationInsights.config) skipped due to skip directive 'CommandLineSkipDirective 1'.
2017-01-30T08:10:26.7182428Z Info: Object dirPath ({sitename}\App_Data\packages) skipped due to skip directive 'CommandLineSkipDirective 2'.
2017-01-30T08:10:26.7192429Z Info: Object filePath ({sitename}\bin\Microsoft.AI.Agent.Intercept.dll) skipped due to skip directive 'CommandLineSkipDirective 3'.
2017-01-30T08:10:26.7192429Z Info: Object filePath ({sitename}\bin\Microsoft.AI.DependencyCollector.dll) skipped due to skip directive 'CommandLineSkipDirective 3'.
2017-01-30T08:10:26.7192429Z Info: Object filePath ({sitename}\bin\Microsoft.AI.HttpModule.dll) skipped due to skip directive 'CommandLineSkipDirective 3'.
2017-01-30T08:10:26.7192429Z Info: Object filePath ({sitename}\bin\Microsoft.AI.PerfCounterCollector.dll) skipped due to skip directive 'CommandLineSkipDirective 3'.
2017-01-30T08:10:26.7192429Z Info: Object filePath ({sitename}\bin\Microsoft.AI.ServerTelemetryChannel.dll) skipped due to skip directive 'CommandLineSkipDirective 3'.
2017-01-30T08:10:26.7202428Z Info: Object filePath ({sitename}\bin\Microsoft.AI.Web.dll) skipped due to skip directive 'CommandLineSkipDirective 3'.
2017-01-30T08:10:26.7202428Z Info: Object filePath ({sitename}\bin\Microsoft.AI.WindowsServer.dll) skipped due to skip directive 'CommandLineSkipDirective 3'.
2017-01-30T08:10:26.7202428Z Info: Object filePath ({sitename}\bin\Microsoft.ApplicationInsights.AzureWebSites.dll) skipped due to skip directive 'CommandLineSkipDirective 4'.
2017-01-30T08:10:26.7202428Z Info: Object filePath ({sitename}\bin\Microsoft.ApplicationInsights.dll) skipped due to skip directive 'CommandLineSkipDirective 4'.

What to do when accidentally deleting App Insights files

If you find yourself in the position that you have accidentally deleted the App Insights files you simply need to delete the App Insights extension and then re-add it and it should work again.

Adding the SDK to your application

If you eventually end up adding the App Insights SDK to your application then take note that you will need to make sure the version of the extension and the version of the SDK DLLs match otherwise you’ll get a version mismatch exception on app startup.

You still need to install the exception because the SDK alone doesn’t collect all information.

My stance on Azure Worker Roles

tl;dr 99% of the time Worker Role is not the right solution. Read on for more info.

Worker Role Deployments

I quite often get asked by people about the best way to deploy Worker Roles because it is a pain – as an Azure Cloud Service the deployment time of a Worker Role is 8-15+ minutes. In the age of continuous delivery and short feedback loops this is unacceptable (as I have said all along).

On the surface though, Worker Roles are the most appropriate and robust way to deploy heavy background processing workloads in Azure. So what do we do?

Web Jobs

The advice I generally give people when deploying websites to Azure is to use Azure Web Sites unless there is something that requires them to use Web Roles (and use Virtual Machines as a last resort). That way you are left with the best possible development, deployment, debugging and support experience possible for your application.

Now that Web Jobs have been released for a while and have a level of maturity and stability I have been giving the same sort of advice when it comes to background processing: if you have a workload that can run on the Azure Web Sites platform (e.g. doesn’t need registry/COM+/GDI+/elevated privileges/custom software installed/mounted drives/Virtual Network/custom certificates etc.) and it doesn’t have intensive CPU or memory resource usage then use Web Jobs.

I should note that when deploying Web Jobs you can deploy them automatically using the WebJobsVs Visual Studio extension.

As a side note: some of my colleagues at Readify have recently started using Web Jobs as a platform for deploying Microservices in asynchronous messaging based systems. It’s actually quite a nice combination because you can put any configuration / management / monitoring information associated with the micro-service in the web site portion of the deployment and it’s intrinsically linked to the Web Job in both the source code and the deployment.

Worker Roles

If you are in a situation where you have an intense workload, you need to scale the workload independently  of your Web Sites instances or your workload isn’t supported by the Azure Web Sites platform (and thus can be run as a Web Job) then you need to start looking at Worker Roles or some other form of background processing.

Treat Worker Roles as infrastructure

One thing that I’ve been trying to push for a number of years now (particularly via my AzureWebFarm and AzureWebFarm.OctopusDeploy projects) is for people to think of Cloud Services deployments as infrastructure rather than applications.

With that mindset shift, Cloud Services becomes amazing rather than a deployment pain:

  • Within 8-15+ minutes a number of customised, RDP-accessible, Virtual Machines are being provisioned for you on a static IP address and those machines can be scaled up or down at any time and they have health monitoring and diagnostics capabilities built-in as well as a powerful load balancer and ability to arbitrarily install software or perform configurations with elevated privileges!
  • To reiterate: waiting 8-15+ minutes for a VM to be provisioned is amazing; waiting 8-15+ minutes for the latest version of your software application to be deployed is unacceptably slow!

By treating Cloud Services as stateless, scalable infrastructure you will rarely perform deployments and the deployment time is then a non-issue – you will only perform deployments when scaling up or rolling out infrastructure updates (which should be a rare event and if it rolls out seamlessly then it doesn’t matter how long it takes).

Advantages of Web/Worker Roles as infrastructure

  • As above, slow deployments don’t matter since they are rare events that should be able to happen seamlessly without taking out the applications hosted on them.
  • As above, you can use all of the capabilities available in Cloud Services.
  • Your applications don’t have to have a separate Azure project in them making the Visual Studio solution simpler / load faster etc.
  • Your applications don’t have any Azure-specific code in them (e.g. CloudConfiguationManager, RoleEnvironment, RoleEntryPoint, etc.) anymore
    • This makes your apps simpler and also means that you aren’t coding anything in them that indicates how/where they should be deployed – this is important and how it should be!
    • It also means you can deploy the same code on-premise and in Azure seamlessly and easily

How do I deploy a background processing workload to Worker Role as infrastructure?

So how does this work you might ask? Well, apart from rolling your own code in the Worker Role to detect, deploy and run your application(s) (say, from blob storage) you have two main options that I know of (both of which are open source projects I own along with Matt Davies):

  • AzureWebFarm and its background worker functionality
    • This would see you deploying the background work as part of MSDeploying a web application and it works quite similar to (but admittedly probably less robust than) Web Jobs – this is suitable for light workloads
  • AzureWebFarm.OctopusDeploy and using OctopusDeploy to deploy a Windows Service
    • In general I recommend using Topshelf to develop Windows Services because it allows a nicer development experience (single console app project that you can F5) and deployment experience (you pass an install argument to install it)
    • You should be able to deploy heavyweight workloads using this approach (just make sure your role size is suitable)

The thing to note about both of these approaches is that you are actually using Web Roles, not Worker Roles! This is fine because there isn’t actually any difference between them apart from the fact that Web Roles have IIS installed and configured. If you don’t want anyone to access the servers over HTTP because they are only used for background processing then simply don’t expose a public endpoint.

So, when should I actually use Worker Roles (aka you said they aren’t applicable 99% of the time – what about the other 1%)?

OK, so there is definitely some situations I can think of and have come across before occasionally that warrant the application actually being coded as a Worker Role – remember to be pragmmatic and use the right tool for the job! Here are some examples (but it’s by no means exhaustive):

  • You need the role to restart if there are any uncaught exceptions
  • You need the ability to control the server as part of the processing – e.g. request the server start / stop
  • You want to connect to internal endpoints in a cloud service deployment or do other complex things that require you to use RoleEnvironment
  • There isn’t really an application-component (or it’s tiny) – e.g. you need to install a custom application when the role starts up and then you invoke that application in some way

What about Virtual Machines?

Sometimes Cloud Services aren’t going to work either – in a scenario where you need persistent storage and can’t code your background processing code to be stateless via RoleEntryPoint then you might need to consider standing up one or more Virtual Machines. If you can avoid this at all then I highly recommend it since you then need to maintain the VMs rather than using a managed service.

Other workloads

This post is targeted at the types of background processing workloads you would likely deploy to a Worker Role. There are other background processing technologies in Azure that I have deliberately not covered in this post such as Hadoop.

Windows Azure Web Sites Programmatic Web Deploy

I was setting up a continuous integration / deployment pipeline last night for my team at GovHack Perth and I had a lot of trouble getting the web deploy deployment working against an Azure Web Sites site.

I had downloaded the .publishsettings file and copied out the username, password and server to pass through to the .cmd file that is generated by MSBuild when creating a web deploy package. I do this because I create the package separately as part of the CI build and attach it as a build artifact and then separately do the deployment with msdeploy as another CI build step.

The problem I came across was that I kept getting the following error when running the deployment:

C:\Path\To\Project\obj\Release\Package>projectname.deploy.cmd /T /M:https://waws-prod-hk1-001.pu
blish.azurewebsites.windows.net:443/msdeploy.axd /U:$sitename /P:password /A:Basic
SetParameters from:
"C:\Path\To\Project\obj\Release\Package\projectname.SetParameters.xml"
You can change IIS Application Name, Physical path, connectionString
or other deploy parameters in the above file.
-------------------------------------------------------
Start executing msdeploy.exe
-------------------------------------------------------
"C:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe" -source:package='C:
\Path\To\Project\obj\Release\Package\projectname.zip' -dest:auto,computerName="https://waws-prod
-hk1-001.publish.azurewebsites.windows.net:443/msdeploy.axd",userName="$sitename"
,password="password",authtyp
e="Basic",includeAcls="False" -verb:sync -disableLink:AppPoolExtension -disableL
ink:ContentExtension -disableLink:CertificateExtension -setParamFile:"C:\Path\To\Project\obj
\Release\Package\projectname.SetParameters.xml" -whatif
Info: Using ID '744a18b1-51c2-4d05-a674-6c407a7f80c5' for connections to the rem
ote server.
Error Code: ERROR_USER_UNAUTHORIZED
More Information: Connected to the remote computer ("waws-prod-hk1-001.publish.a
zurewebsites.windows.net") using the Web Management Service, but could not autho
rize. Make sure that you are using the correct user name and password, that the
site you are connecting to exists, and that the credentials represent a user who
has permissions to access the site. Learn more at: http://go.microsoft.com/fwl
ink/?LinkId=221672#ERROR_USER_UNAUTHORIZED.
Error: The remote server returned an error: (401) Unauthorized.
Error count: 1.

It turns out the problem was I needed to include the ?site=sitename parameter in the msdeploy destination otherwise (I assume) the load balancer on the publish server can’t authenticate your username and password properly. This meant I needed to switch to using msdeploy.exe directly like so:

"C:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe" -source:package='projectname.zip' -dest:auto,computerName="https://waws-prod-hk1-001.publish.azurewebsites.windows.net:443/msdeploy.axd?site=sitename",userName="$sitename",password="password",authtype="Basic",includeAcls="False" -verb:sync -disableLink:AppPoolExtension -disableLink:ContentExtension -disableLink:CertificateExtension -setParamFile:"projectname.SetParameters.xml"

Hopefully this post helps other people stuck in the same situation (I couldn’t find anything on Google to help with this)!

Running AspNetCompiler after creating web deploy package using VisualStudio 2012 with round-tripping to 2010

I’ve previously blogged (as a side-note of an unrelated problem I was having) the approach that I use to overcome the problem that the AspNetCompiler has when you have the extraneous web.config files in the obj folder after creating a web deploy package.

The usual way you will come across this problem is running MvcBuildViews in a subsequent build after creating the package. The solution that I previously posted included a few MSBuild targets that can be called prior to running MvcBuildViews to clean up the web deploy files.

If you have tried using these targets after upgrading your project to Visual Studio 2012 then you will have noticed that some of the targets no longer work because they have been renamed. The following code is what I now use and it is compatible with the Visual Studio 2010 round-tripping support.

  <Target Name="CleanWebDeploy" BeforeTargets="MvcBuildViews" Condition="'$(MvcBuildViews)'=='true'">
    <Message Text="Running Clean Web Deploy Target" />
    <CallTarget Targets="CleanWebsitesPackage" />
    <CallTarget Targets="CleanWebsitesPackageTempDir" Condition="'$(VisualStudioVersion)'=='10.0'" />
    <CallTarget Targets="CleanWebsitesTransformParametersFiles" Condition="'$(VisualStudioVersion)'=='10.0'" />
    <CallTarget Targets="CleanWebsitesWPPAllFilesInSingleFolder" Condition="'$(VisualStudioVersion)'!='10.0'" />
    <CallTarget Targets="CleanWebPublishPipelineIntermediateOutput" Condition="'$(VisualStudioVersion)'!='10.0'" />
  </Target>

Maintainable, large-scale continuous delivery with TeamCity Blog Series

I’ve been a ardent supporter of continuous delivery since I first learnt about it from a presentation by Martin Fowler and Jez Humble. At the time I loved how it encouraged less risky deployments and changed the decision of when/what to deploy from being a technical decision to being a business decision.

I personally think that embracing continuous delivery is an important intermediate step on the journey towards moving from technical agility to strategic agility.

This post was first written in August 2012, but has since been largely rewritten in February 2014 to keep it up to date.

This post forms the introduction to a blog series that is jointly written by myself and Matt Davies.

  1. Intro
  2. TeamCity deployment pipeline
  3. Deploying Web Applications
    • MsDeploy (onprem and Azure Web Sites)
    • OctopusDeploy (nuget)
    • Git push (Windows Azure Web Sites)
  4. Deploying Windows Services
    • MsDeploy
    • OctopusDeploy
    • Git push (Windows Azure Web Sites Web Jobs)
  5. Deploying Windows Azure Cloud Services
    • OctopusDeploy
    • PowerShell
  6. How to choose your deployment technology

Continuous Delivery with TeamCity

One of the key concepts in continuous delivery is the creation of a clear deployment pipeline that provides a clear set of sequential steps or stages to move software from a developer’s commits to being deployed in production.

We have been using TeamCity to develop deployment pipelines that facilitate continuous delivery for the last few years. While TeamCity is principally a continuous integration tool, it is more than adequate for creating a deployment pipeline and comes with the advantage that you are then using a single tool for build and deployment.

We have also tried combining TeamCity with other tools that are more dedicated to deployments, such as OctopusDeploy. Those tools provide better deployment focussed features such as visualisation of the versions of your application deployed to each environment. This approach does create the disadvantage of needing to rely on configuring and using two separate tools rather than just one, but can still be useful depending on your situation.

There are a number of articles that you will quickly come across in this space that give some really great advice on how to set up a continuous delivery pipeline with TeamCity and complement our blog series:

Purpose of this blog series

The purpose of this series is three-fold:

  • Document any findings that myself and Matt Davies have found from implementing continuous delivery pipelines using TeamCity that differ from the articles above;
  • Outline the techniques we have developed to specifically set up the TeamCity installation in a way that is maintainable for a large number of projects;
  • Cover the deployment of Windows Services and Azure Roles as well as IIS websites.

Our intention is to build of the work of our predecessors rather than provide a stand-alone series on how to set up continuous delivery pipelines with TeamCity.

Other options

TeamCity is by no means the only choice when it comes to creating a deployment pipeline. Feel free to explore some of the other options:

Web Deploy Package Connection String Parameterisation Problem

I recently came across a problem using web deploy packages, and more specifically with connection string parameterisation. I was generating a web deploy package using the “Package” MSBuild command and then deploying to a remote server using MSDeploy. The problem I came across was that the parameterised connection strings weren’t being replaced.

That resulted in the deployed web.config file looking something like this:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  ...
  <connectionStrings>
      <add name="SomeConnectionString" connectionString="$(ReplacableToken_SomeConnectionString-Web.config Connection String_0)" />
  </connectionStrings>
  ...
</configuration>

I used a variety of approaches – DeployOnBuild MSBuild command, Visual Studio GUI interface, the generated .cmd file and direct msdeploy call – to deploy the package to check whether the problem was simply that I was somehow deploying it incorrectly, but they all resulted in the same problem.

Most of the Google results that came up simply said to turn off parameterisation, but in this instance I wanted to use parameterisation. While I was using XDT transformations, I also wanted to use connection string parameterisation so I could substitute the connection strings for entries in a password safe as part of an automated deployment step.

The build output wasn’t terribly helpful, so I passed the verbose option to msdeploy and got the following interesting output:

...
[16:08:33][Step 2/2] Verbose: Parameter entry 'SomeConnectionString-Web.config Connection String/1' could not be applied anywhere.
[16:08:33][Step 2/2] Verbose: The dependency check 'DependencyCheckInUse' found no issues.
[16:08:33][Step 2/2] Verbose: The synchronization completed in 2 pass(es).
[16:08:33][Step 2/2] Total changes: 10 (0 added, 0 deleted, 10 updated, 0 parameters changed, 1004778 bytes copied)
[16:08:33][Step 2/2] Process exited with code 0

This didn’t really make a huge amount of sense and after some sanity checking to make sure parameterisation does actually work (created a new ASP.NET website and deployed it successfully) I started comparing all the differences in my .csproj configuration to a blank project. After some tedious trial and error I managed to narrow the errant configuration to the following (seemingly) innocuous line:

<BaseIntermediateOutputPath>$(ProjectDir)..obj</BaseIntermediateOutputPath>

You might be asking why I had that line in there in the first place? It is certainly a bit weird and you would normally expect it to be set to “obj”. In this case it was there so that when MvcBuildViews was next executed after building a package it didn’t choke with the following error (on the intermediate web.config files in the obj folder):

It is an error to use a section registered as allowDefinition='MachineToApplication' beyond application level.  This error can be caused by a virtual directory not being configured as an application in IIS.

When I initially came across that problem I changed the location of the obj folder since it seemed the best option compared to the other options such as running clean before all builds or deleting the obj folder, both of which would slow down the build and be tedious and annoying. Also, I didn’t want to turn MvcBuildViews off since I want assurance on the build server that the views all compiled fine.

Since this was no longer an option (up until this point all web deploys were done manually so we hadn’t actually come across this new problem!) I came up with the following configuration as a solution (props to my Senior Dev, Matt Davies, for finding the relevant StackOverflow post for the inspiration):

  <Target Name="CleanPackage" BeforeTargets="MvcBuildViews" Condition="'$(MvcBuildViews)'=='true'">
    <CallTarget Targets="CleanWebsitesPackage" />
    <CallTarget Targets="CleanWebsitesPackageTempDir" />
    <CallTarget Targets="CleanWebsitesTransformParametersFiles" />
  </Target>
  <Target Name="MvcBuildViews" AfterTargets="Build" Condition="'$(MvcBuildViews)'=='true'">
    <AspNetCompiler VirtualPath="temp" PhysicalPath="$(WebProjectOutputDir)" />
  </Target>

Basically, if you have MvcBuildViews set to true then it will run MvcBuildViews directly after the build and before running MvcBuildViews it will run the CleanWebsitesPackage, CleanWebsitesPackageTempDir and CleanWebsitesTransformParametersFiles targets which clean up the web deploy package files and thus the web.config files that mess with MvcBuildViews.

Update (17 Feb 2013)

If you are trying to use the above MSBuild with Visual Studio 2012 then it won’t work. I have blogged a solution that works in both Visual Studio 2012 and Visual Studio 2010 though.