Have you ever stopped an IIS application pool, just to find it running later? This a quick post to explain why this happens, and how to correctly stop the app pool so it stays down.
(check our Reset, Restart and Recycle IIS guide for more on effectively restarting your application pools and avoiding the problematic side effects).
Why stop an application pool in the first place?
You may choose to stop one of your app pools if:
It’s having an excessive impact on your server.
It is hosting applications that you don’t want running (unused, broken, or for security reasons).
It’s a copy of another application that’s already running (and you don’t want it conflicting or running twice).
You are doing a performance test on another app, and you want everything else stopped.
You are updating the applications and you don’t want them restarting during the update.
Stopping apppools for security and performance reasons can actually be an effective strategy when dealing with many customers/apps on the server, or removing unused applications that can become a DOS target.
Why does the stopped application pool come back to life?
The reason for this is simple. The Windows Process Activation Service (WAS, WPAS) will start all application pools when it starts.
So, if WAS restarts for any reason, it will re-start your stopped application pools.
This will happen in all of these cases:
IISRESET
Net stop was /yes && net start w3svc (restarting IIS services)
Rebooting your server VM
When someone makes a request to the application that’s mapped to your application pool, it will then start a worker process.
The application pool will also start a worker process when using the AlwaysRunning mode. Which we totally recommend for your important production sites, as you can see in our Maximize IIS Application pool availability guide.
How do I stop the pool so it stays stopped?
Easy.
You must also configure the application pool to set autoStart=false. This way, WAS knows not to start it unless you explicitly choose to start it.
Here is the compact command:
%windir%\system32\inetsrv\appcmd set apppool POOLNAME /autoStart:false
%windir%\system32\inetsrv\appcmd stop apppool POOLNAME
This causes this applicationHost.config configuration to be written in your application pool’s definition:
<add name="TestApp" autoStart="false" ...>
To start the application pool back up
If and when you want to get the pool started back up:
%windir%\system32\inetsrv\appcmd set apppool POOLNAME /autoStart:true
%windir%\system32\inetsrv\appcmd start apppool POOLNAME
You can skip the autoStart:true command if you want to continue manually managing the pool’s uptime, and just start/stop it when needed.
That’s it. No more zombie application pools.
P.S. If you haven’t already, check out our Reset, Restart, Recycle IIS guide for useful tips on how to correctly restart production apppools with minimum downtime.
But what you may not know is that when it fails, it often leaves your IIS services in a downed state. Meaning your web server is now permanently down.
You may be using IISRESET as a last resort to fix a website performance issue … but instead you take your entire web server down… Probably not what you intended!
Don’t use IISRESET to fix performance issues … actually fix them instead!
If you are relying on IISRESET to resolve hangs or high CPU, check out our Reset, Restart and Recycle IIS guide for better case-by-case alternatives.
Then, use LeanSentry to diagnose and actually resolve your performance issues so you don’t have to keep restarting as a bandaid.
IISRESET fails with Access Denied (and leaves IIS stopped)
The most common reason why the command fails to restart IIS services, is the frustrating Access Denied error. This happens even when you are executing IISRESET from an elevated “As Administrator” command line:
IISRESET fails to restart IIS and leaves the WAS and W3SVC services stopped
When this happens, you get this error:
Attempting stop...
Stop attempt failed.
Access denied, you must be an administrator of the remote computer to use this command. Either have your account added to the administrator local group of the remote computer or to the domain administrator global group.
This confusing error is actually due to IISRESET timing out when attempting to stop the Windows Process Activation Service (WAS), which probably happens more than 50% of the time.
This is entirely due to the discrepancy between the IISRESET restart timeout (a short 20 seconds) and the default shutdownTimeLimit for IIS worker processes which is 90 seconds by default.
So, this happens:
IISRESET tries to stop WAS.
WAS tells all IIS worker processes to stop, and begins to wait up to 90 seconds for each.
Each worker process tries to wait for any executing requests to finish, which may take anywhere from 0 to 90 seconds (because the process will be killed after that).
IISRESET times out waiting for WAS to stop, and tries to kill it.
This throws the “Access Denied” error.
IISRESET reports the error and bails, without trying to restart IIS services!
Seconds later, your WAS and W3SVC services are stopped and your webserver is now down. Any traffic to your website is now getting connection refused:
ERR_CONNECTION_REFUSED in chrome because W3SVC removed your websites bindings when it stopped.
You may attempt a quick fix for that IISRESET command:
IISRESET /TIMEOUT:100
This should do it in 90% of cases, but if WAS does legitimately get hung and fail to stop (we’ve never seen that ourselves, but hey it could happen), you are going to get Access Denied again and the services will be left stopped.
IISRESET NOFORCE is a no-go
We also DON’T recommend the NOFORCE option despite what some Microsoft support content out there suggests, for the same reason … because IISRESET /NOFORCE won’t try to kill the services if they are stuck, and they could get left in a bad state.
If you use the NOFORCE parameter, you may also get these errors:
There was an error while performing this operation.
The service cannot accept control messages at this time. (2147943461, 80070425)
or:
Restart attempt failed.
The service cannot accept control messages at this time. (2147943461, 80070425)
So, if I absolutely have to give you a better IISRESET command, here it is:
This makes sure that if WAS does not stop in the reasonable 100 seconds, we kill its host process (this will be the worst case because we are giving enough time for all apppools to stop).
We also make sure the services are terminated if they don’t stop, to avoid leaving them in a bad state.
Then, we always re-start the services! That’s the key part.
How to recover when IISRESET fails
Finally, you just ran IISRESET, and it failed. Your server now may be down.
Recover with these commands:
net stop was /yes
taskkill /F /FI "SERVICES eq was"
net start w3svc
The idea here is that, since we don’t exactly know what state the services are in, we’ll just make sure they are stopped, and then start em clean.
Note that we start W3SVC, NOT WAS. This also starts WAS. Sometimes people forget that you need both, so if you just started WAS, you’d still be getting “This site can’t be reached” errors because the web server would not be listening for any incoming requests to your sites.
That’s it. Now that we recovered your server, be sure to never use IISRESET again. Instead, head over to our Reset, Restart and Recycle IIS guide to learn how to restart your websites with zero downtime.
Other IISRESET errors we’ve seen
Restart attempt failed. The IIS Admin Service or the World Wide Web Publishing Service, or a service dependent on them failed to start. The service, or dependent services, may had an error during its startup or may be disabled.
IISRESET may be failing to start some services that depend on WAS or W3SVC. These services may be disabled or not used on the machine.
Instead of this, just start W3SVC directly with net start w3svc.
Some support content suggests installing or enabling these dependent services to make IISRESET work, but I think that’s completely backwards. If you don’t use those services, why do you have to enable them just to make a broken command work?
Access denied, you must be an administrator of the remote computer to use this command. Either have your account added to the administrator local group of the remote computer or to the domain administrator global group.
This happens because you are trying to run IISRESET from an non-elevated command line, so it’s blocked by User Access Control (UAC). Run the command from a command line prompt started with “As Administrator” instead.
Have you ever had an Azure instance that just was not performing up to your expectations?
We have. And in the past, they were very hard to get rid of without affecting your entire cloud deployment.
How-To: Remove Azure instances that became slow or unhealthy
In this week’s How-To post, we’ll cover the new API that lets us easily remove an Azure instance that is performing poorly or has become unhealthy. We also share a tool we wrote to automate the removal, so you can remove dead Azure instances quickly whenever you need.
Back when we launched LeanSentry 2 years ago, we had a lot of issues with Azure instances not performing up to our expectations. In particular, the Azure host processes would die whenever the instances experienced high memory utilization from our custom cache layer, and begin to constantly recycle the role/reboot the VM. This caused service outages during times of peak usage.
Back then, the only way to take an instance out of rotation was to do VIP swap, or to scale down the service to a point where the offending instance would be removed (and so would all other instances with higher instance ids). Because we maintained a lot of in-memory and on-disk state on the instance, both of those options would be a huge no-no. So, we lobbied Microsoft to create an option to remove a specific instance, instead of trashing half of your service.
The Azure team came through and finally released an API to do this. This has been a godsend, allowing us to intelligently manage how we scale up and down so we can keep the instances with the highest efficiency / warmest cache.
For more on how to when to do this, and a tool to quickly delete instances, check out the How-To post here.
In the last 2 years, we’ve helped thousands of big and small companies to improve their IIS servers & ASP.NET applications.
Today, we are announcing LeanSentry How-To: a series of best-practice guides to help you solve common performance problems & improve your site health.
We’ll cover topics like:
1. Azure best practices
2. IIS & ASP.NET performance
3. Improving application & server stability
4. Advanced tools & techniques for troubleshooting production problems
