Did you know there are 5 places where ASP.NET requests can become queued on an IIS server?
Not all these queues are documented, and it can be very difficult to tell when and where requests are queued.
As part of LeanSentry’s automatic hang detection and troubleshooting, we had to figure out the IIS/ASP.NET request queueing behavior. So, we wanted to share the knowledge with everyone so you can properly track down queued requests.
Read on to learn all about these queues, how to tell when requests are queued, and how to identify the exact requests that are actually queued!
(UPDATE: Want to learn how to troubleshoot common ASP.NET issues like hangs, high CPU, etc?
Take our new LeanSentry Production Troubleshooting course. Its a free 5-7 email course that teaches the production troubleshooting techniques we’ve been using for years.)
The details on IIS and ASP.NET queues
When a request is received by your IIS server, here are all the queues it must clear in order to be processed:
1. HTTP.SYS: Application pool queue
Requests are always first queued here, for the IIS worker process to dequeue.
Behavior: Requests begin to accumulate when IIS falls behind in dequeueing requests. The limit is set by the application pool’s configured queueLength attribute, and defaults to 1000. When limit is reached, HTTP.SYS returns 503 Service Unavailable.
Monitor: “Http Service Request QueuesCurrentQueueSize” performance counter
2. IIS worker process: completion port
The dequeued requests queue up here, waiting for IIS i/o threads to pick them up.
Behavior: There is usually up to 20 possible requests queued here, and they are dispatched up to N at a time (where N = number of processor cores).
Monitor: This is an undocumented queue, with no available reporting.
3. ASP.NET: CLR threadpool queue
ASP.NET queues all incoming requests to the CLR threadpool.
Behavior: If all CLR threads are busy, requests can queue up here up to the configured processModel/requestQueueLimit. When there are more than this many total requests (executing + queued), ASP.NET returns 503 Service Unavailable.
NOTE: Any async modules also re-post requests to the CLR threadpool, so requests can become “re-queued” again later in the request processing.
Monitor: “ASP.NET v4.0.30319Requests Queued” performance counter
NOTE: This counter is global for the entire server, there is no way to tell which website/apppool has queued requests. It also does not work correctly in Integrated mode for .NET 2.0/3.5.
4. ASP.NET: Integrated mode global queue
In Integrated mode, ASP.NET will queue all incoming requests after the configured concurrency limit is reached.
Behavior: Concurrency limit is set by the MaxConcurrentRequestsPerCPU registry key or applicationPool/maxConcurrentRequestsPerCPU attribute (Defaults to 12 on .NET 2.0/3.5, and 5000 on .NET 4.0+) and MaxConcurrentThreadsPerCPU registry key or the applicationPool/MaxConcurrentThreadsPerCPU attribute (defaults to 0, disabled).
Monitor: “ASP.NET v4.0.30319Requests Queued” performance counter
5. ASP.NET: Classic mode application queue
In Classic mode, ASP.NET will queue all incoming requests to the per-application queue when there are not enough threads.
Behavior: The threads available for request processing are determined by available threads in the CLR thread pool, minus the reserved threads set by the httpRuntime/minFreeThreads and httpRuntime/minFreeLocalThreads attributes.
NOTE: This queue has poor performance, and does not guarantee FIFO in application pools with multiple applications (because threads are shared between multiple apps, so a single app can starve the other applications of available threads).
Monitor: “ASP.NET ApplicationsRequests in Application Queue” performance counter, with instances per application.
How to tell which requests are queued
Ok, so we can now tell whether requests are queued, but how we can tell which requests are queued vs. which requests are processing?
This helps us identify which requests are causing blocking in the system (and possibly causing a hang), vs. the requests that are simply queued as a result.
Well, we can’t tell which requests are queued in queues 1 & 2, because they have not yet been picked up by IIS. We also cant tell which requests are queued in the Classic mode per-application queue. Lucky for us, most queueing for ASP.NET apps in Integrated mode (default) happens in queue #3 and #4. And while we can’t always determine his 100%, there is a heuristic that can help us separated queued from processing requests 90% of the time. In my experience, that has been good enough!
Here is the trick:
1. Snapshot the currently executing requests
> %windir%system32inetsrvappcmd list requests /elapsed:1000
You’ll get a list like this:
// hanging!
REQUEST "7000000780000548" (url:GET /test.aspx, time:30465 msec, client:localhost, stage:ExecuteRequestHandler, module:ManagedPipelineHandler)
REQUEST "f200000280000777" (url:GET /test.aspx, time:29071 msec, client:localhost, stage:ExecuteRequestHandler, module:ManagedPipelineHandler)
…
// queued!
REQUEST "6f00000780000567" (url:GET /, time:1279 msec, client:localhost, stage:AuthenticateRequest, module:WindowsAuthentication)
REQUEST "7500020080000648" (url:GET /login, time:764 msec, client:localhost, stage:AuthenticateRequest, module:WindowsAuthentication)
2. Use this rule to identify the queued requests:
A group of requests to an Integrated pipeline ASP.NET app are queued if they are:
- Processing in an ASP.NET module
- There are no other requests to the same app in an ASP.NET module in an earlier pipeline stage
- There are no other requests to the same app in a different ASP.NET module/stage with higher avg. latency.
Basically, this takes advantage of the fact that the first ASP.NET module in the request processing pipeline will cause ASP.NET to queue the request, showing it as processing in that module in the executing request list. The request at the front of the list have been executing the longest, which means they are NOT queued (queueing is FIFO).
Practically, this just means that the last block of requests in the list the same ASP.NET module/stage are queued requests. Think about it. From experience, these usually show as blocks of:
REQUEST "6f00000780000567" (url:GET /, time:8076 msec, client:localhost, stage:AuthenticateRequest, module:WindowsAuthentication)
REQUEST "6f00000780000567" (url:GET /login, time:5601 msec, client:localhost, stage:AuthenticateRequest, module:WindowsAuthentication)
REQUEST "6f00000780000567" (url:GET /, time:5200 msec, client:localhost, stage:AuthenticateRequest, module:WindowsAuthentication)
REQUEST "6f00000780000567" (url:GET /test.aspx, time:3209 msec, client:localhost, stage:AuthenticateRequest, module:WindowsAuthentication)
REQUEST "6f00000780000567" (url:GET /test.aspx, time:1690 msec, client:localhost, stage:AuthenticateRequest, module:WindowsAuthentication)
REQUEST "6f00000780000567" (url:GET /, time:1279 msec, client:localhost, stage:AuthenticateRequest, module:WindowsAuthentication)
Simply because for most ASP.NET apps, WindowsAuthentication is the first ASP.NET module to process the request in the AuthenticateRequest stage. If you have a custom module or global.asax processing BeginRequest, expect to see that.
We hope this will help you make sense of queued requests when troubleshooting slow or hung ASP.NET requests.
To learn more about LeanSentry’s automatic hang detection and troubleshooting, check out The holy grail: Automatically diagnose IIS & ASP.NET website hangs.
Best,
The LeanSentry Team