DotCom’s Blog Space

Recently we deployed some of our Web Services to a new and shiney 64 bit OS and 8 processor machine. A few days later I was pinged with cacti graphs showing that the web service was using 1.4 gb of memory. At first I assumed that this was due to some kinda of memory leak so I started getting the dump off the box for that w3wp process hosting the web service. Pulled the data up in windbg and noted a very large number of System.XML.DocumentXpathNavigator objects, in excess of 1.2 million. So with this I started investigating roots of these object of which none I could find. Started banging my head because the same code in 32 bit OS only consumed 250 meg max and ran just fine.

After digging and opening up a ticket with MS we finally figured out what the issue really was. Memory allocation and usage is different in 64 bit vs 32 bit applications. Of course I knew this, but to the extent is what I didnt know. One thing they have in common is that in IIS all applications default to gcServer enabled=true. This will cause the GC manager to spawn 1 heap and Garbage Collector per Processor(multiply this by 2 when using hyperthreading). The difference in 2.0 sp1 .net 64 bit, for example, is that the initial allocations segment sizes the GC Manager reserves is 512mb for small object segment, and 128 mb for the LOH. The 32 bit version will allocate 64 mb for small object segment, and 32 mb for the LOH.

Now if you take that and multiply this out across 8 processors it becomes a pretty substantial amount. Now people are saying to themselves, but I dont see that much memory used so how can that be. Explanation follows: The commited memory does not show this maybe because you have not created or allocated that many managed objects. If you take an application and just loop it creating and destroying objects you would see the committed memory climb until the garbage collector kicks in when it reaches that segments size at which point it will clean out all non rooted objects and bring the committed memory back down to earth. The other side effect to this is that garbage collections happen a LOT less often as it takes your application a lot longer to fill the segment. What does this mean to the troubleshooter? A lot of objects left in memory without roots. This goes back to the original issue I talked about above.

So one asks, how do I solve this problem? You have to think about how much memory does your application need. If you don’t see the need for more then the 4gb limit(2gb application) then the recommendation is run the application in 32 bit mode as this will cause a lot less memory pressure on the box. It may cause more garbage collections but if your are running in gcServer mode you probably will not see difference. I have actually noticed a few points drop in the CPU Cycles being used. This I am guessing is the numbers being crunched in the background are much larger due to being 64 bit in nature.

If you don’t have the option of running the application in 32 bit mode another workaround is to make the application run in workstation mode. To enable workstation mode you must add this to the applications config file(in iis its the frameworks aspnet.config).

<configuration>
  <runtime>
    <gcServer enabled="false" />
    <gcConcurrent enabled="false" />
  </runtime>
</configuration>

The difference between these are explained in greater detail in Tess’s blog entry here: How does the gc work and what are the sizes of the different generations

With the workaround above, something to also note is that in WinForms and Windows Services they default to gcConcurrent enabled=true/gcServer enabled = false mode and you have to specifically configure them to gcServer enabled=true to take advantage of the multiple GC heaps on a multiproc machine. Applications and Services default to a single GC Heap, and the thread causing the allocation to the heap that triggers GC is responsible for running the collection.