Guy Shahine's Blog

Windows Azure: Why is my service not starting?

Have you found yourself in a situation where your service goes from Initializing…Busy…Stopped…Initializing… when you deploy to Windows Azure? That’s when I value the “hint button” that pops up in some games after struggling to figure out how to go to the next level. Unfortunately, there is no hint button on the Windows Azure Portal, so this blog post will help you put a checklist of things to go over before you give up.

Deployment Initializing Busy Deployment

Possible Issues

Deploying With Local Service Configuration File

A typical mistake is to deploy your service with your local configuration file where usually the diagnostic store and/or other storage endpoints are pointing to the local development storage. To avoid this problem, always create separate configuration files, e.g. ServiceConfiguration-Local.cscfg, ServiceConfiguration-Staging.cscfg and ServiceConfiguration-Production.cscfg…etc

Local Service Configuration

Wrong Service Configuration Settings

There is a range of wrong configuration settings which could cause your role to crash. I’ll enumerate the most common mistakes:

  • Wrong storage name: You should be aware that the storage account name is not your LiveID alias. When you create a new storage account, you usually choose an account name and it’s displayed in the following fashion “http://[account-name].blob.core.windows.net”.
  • Wrong storage account key: The storage account key is not your LiveID password. When you create a storage account, you should get two base-64 keys primary and secondary key.
  • Regenerated account key: When you regenerate your primary and/or secondary key then the old key will become invalid.
  • Unreachable endpoints: Many factors could cause your endpoints to be unreachable, for instance, power failure, network failure … etc (that’s another reason to move those services to Windows Azure 🙂 so you don’t suffer from downtimes ). When an endpoint is not reachable and your service is relying on it, then your service will crash.

Administrator Privileges Required

Today, Windows Azure doesn’t allow you to perform administrative actions within your role, so if your service requires:

  • Installing some software
  • Installing/Configuring a DCOM service
  • Editing Registries

Or anything that requires admin privilege then your role will crash and your service won’t start.

ATTENTION: When your service is running locally in Full Trust in DevFabric then administrative actions will work and they won’t crash your role. This is caused by the DevFabric running in administrator mode on your machine and the Windows Azure team hasn’t implemented yet a way to block those actions when running locally. So this will make it harder locally to identify that your service will not work when deployed to the cloud.

Full Trust vs Partial Trust

In Windows Azure you’re allowed to run your service in Full Trust or Partial Trust so if you set it to run in Partial Trust, then Full Trust assemblies will cause your service to crash.

.NET Trust Level Option in Visual Studio

ASP.NET Custom Error Page

By default, custom error pages are enabled when creating an new Windows Azure web role. The custom page is meant to hide the ugly crash details from the end-user (ugly to some people, very beautiful and informational to others 😉 ). It’s possible to turn off the custom error page in the project web.config, check out the documentation: http://msdn.microsoft.com/en-us/library/h0hfz6fc.aspx.

Windows Azure Web Role with Custom Error Page On:

ASP.NET Custom Error Page

Windows Azure Web Role with Custom Error Page Off:

ASP.NET Page Details Crash Information

Key Takeaways

The key takeaways from this post is to double check your application’s settings and to be aware of what’s not supported by Windows Azure before contacting the support team.

  • Always make sure to run your service locally in the DevFabric, as it will identify many of the issues that might cause your service to crash when deployed to the cloud.
  • Maintain separate service configuration files for each environment: Local, Staging, Production, … etc
  • Make sure to keep your settings file up-to-date in case you renew storage keys or you rename resources (e.g. rename the queue instance that your service relies on)
  • Make sure that other services that you rely on are up and running.
  • Turn off ASP.NET custom error pages as you’ll get more details about the crash, which will allow you to get more traction on the areas to focus on to fix your service.

Good Luck!

  • Nice posts, great tips.

  • This could be the beginning of an Azure debugging FAQ.

  • Guy,
    thanks for solving *endless* frustration for me on this one. There are about 30 other posts on the topic that I went through prior to yours, and it turned out my mistake was using the wrong storage account name (used the storage service name not the account name). Thanks for clarifying, very helpful!
    Steve

  • What about posting a screenshot of an actual populated .cscfg config file that actually works, instead of showing us the wizard-generated one that does not?

    After 4 hours of investigations, I still don’t know how my configuration file is supposed to look, and how a well formed diagnostic and storage connection string should look…

    Thanks

  • Axel, Thanks for pointing that out. A well formed connection string should look like:

    Setting name=”MyCloudStorageAccount” value=”DefaultEndpointsProtocol=https;AccountName=mystorage;AccountKey=[mystorage key]”

    NOTE: You cannot use HTTP as your default endpoint protocol for pushing diagnostics.

    If you’re using Visual Studio then the Windows Azure tools have a nice UI to populate the connection string.

    For more detailed example, take a look at the Windows Azure Storage team blog post: http://bit.ly/alZuji

  • A nice post on troubleshooting deployments from the Windows Azure team is posted on the MSDN forum, found here: http://bit.ly/cc1yaw

  • CodeGrue

    Can you elaborate on how to have separate configuration files? When I add the -local, -staging, -production and build, I her the error “Only one service configuration file per Windows Azure application project”.

    • Unfotunately you cannot use multiple configuration files and publish within visual studio today. I recommend that you use the configuration file that’s associated to your service within your service for local deployment and testing. And maintain different configuration files for different environments (this would prevent you from publishing from Visual Studio). I know that it’s not too convenient and that every time you need to add or remove a config setting, you’d have to update multiple files, that’s true and I wish there was a better support out of the box.