ASP.NET MVC2 in the web farm – slow page load with high traffic – where is the bottleneck?

Monday 24th, October 2011 / 10:59 Written by

Recently I have implemented quite complex e-commerce web application using technologies such as: ASP.NET MVC2, Linq2SQL , .NET 3.5 , SQL Server 2008 R2.

We have the whole thing hosted in the web farm.

We have 4 x IIS7 + 1 SQL SERVER 2008 load balanced with MS NLB

Static content is cached by external cache provider – Akamai, which reduces 86% of the requests.

Each web server has 32 GB of RAM and 4 x quad core CPUs so there are 64 cores on the front-end.

We store session state in tables of SQL server.

It works prefectly with medium traffic (page load = 0.2 -0.3 s)

but the company does the tv ads and during those ads the traffic hits up to 20,000 -30,000 users within 20-30 seconds.

In this moment the page slows down to 39 seconds. However, the usage of CPU’s and the memory doesn’t even arrive to 40% on any machine.

The bandwidth of the data centre does not arrive to the half of its limits.

Pages which are slow generate data from simple SELECTs of maximum 10 records from 1-2 tables only.

I gone through all possible things: I already had my queries compiled but I also optimized selects, created additional indexes studied a lot the problems of storing session in tables that sql server 2005 had (temdb) and I couldn’t figure out what was going on.

You could reproduce the the problem by launching stress tools against the web. The worst thing was that we could freeze the web just by launching stress tools from only two pc’s located in our offices.

I have also posted the question on stackoverflow
http://stackoverflow.com/questions/7644339/asp-net-mvc2-in-the-web-farm-slow-page-load-with-high-traffic
where people suggested to check indexes and use CompiledQuery.

In practice 90% of the bottlenecks are caused by database. After the whole week of tests and studying the problem I have possibly found a bug in the ASP.NET MVC2 library.
It was incredible how silly the thing was:

Let’s have a look at this code:


public ActionResult Index(){
var model = new SomeModel();
// Get data
....
return View(model);
}

In this code the View is generated using file Index.aspx.
However, this View needs to be found in some way and I guess the search of the this View is causing the problem.

Unbelievable but true – adding the name of the View removed the problem:


public ActionResult Index() {
 var model = new SomeModel();
// Get data
....
return View("Index",model);
}

I say it again, this problem does exists only when the web has a high number of users.
The MVC version I’m using is

Version: 2.0.0.0
Runtime: v2.0.50727

I have contacted the geeks responsible for the design and development of this great library to see what they say.

By all means I don’t  want to discourage anyone to use this library as it is absolutely fantastic and I use it a lot and I will carry on using it.

 

 

 

, ,
  • Blah

    This was insanely helpful for us and our e-Commerce website. You’ve saved Xmas! :)