DotCom’s Blog Space

Recently we were trying to fix our High CPU utilization by moving from 2.0.x iron python to 2.6 as it has a interpreter built in and they claim it will use a lot less CPU in the compilation of the python scripts. Well when we upgraded to the new versions and put anything more then 1 request simulataneously to the site it would crash it with a Kernel.dll error in the Event Viewer claiming an error of 0xe053534f which indicates a Stack Overflow. This was a difficult one to chase down because just attaching ADPlus in crash mode slowed the site down so bad it was causing site to stack and not work properly. So eventually we would ultimately attach windbg and only break on exception 0xe053534f using command:

sxe 0xe053534f

Then we could investigate the stack properly to see what was going on. When this breakpoint happens you will be on the thread that is the offending code that caused the stack overflow. Also note if you just do a crash dump instead of the windbg sxe method all other threads will most likely be shutdown as the crash dump happens after your exception was raised. You must use the following commands to allow you to see both managed and unmanaged stacks:

.loadby sos mscorwks
.load psscor2.dll (if you have it)

When doing a !dumpstack we ended up finding that the DLR was using recursive algorithms for some of there code exection/compilation. In our Python script we had a case where there were 139 if/elsif conditions. This was enough to cause the stack overflow in the recursive interpreter of the new 1.0 Dynamice Library Runtime. If we brought it below 125 in the block the overflow would go away.

Another note when using windbg on checking these errors some other helpful commands are:

kb 2000 – show up to 2000 lines back into the stack
!dso – Dump Stack Objects
!dae – Dump all exceptions and there data

Our Stack indicating the exception is as follows:
OS Thread Id: 0×1788 (37)
Child-SP RetAddr Call Site
00000000095564d0 000007fef9ca12e6 KERNELBASE!RaiseException+0×39
00000000095565a0 000007fef9ecfdd5 mscorwks!`string’+0×59046
00000000095566f0 000007fef8baa137 mscorwks!RuntimeTypeHandle::GetInstantiation+0×175
00000000095568f0 000007ff009d5750 mscorlib_ni!System.RuntimeType.GetGenericArguments()+0×37
0000000009556950 000007ff009d5351 Microsoft_Scripting_Core!System.Dynamic.Utils.TypeUtils.CanCache(System.Type)+0×60
0000000009556990 000007ff00b11276 Microsoft_Scripting_Core!System.Dynamic.Utils.TypeExtensions.GetParametersCached(System.Reflection.MethodBase)+0×71
00000000095569f0 000007ff00b11115 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Expression.GetValidMethodForDynamic(System.Type)+0×26
0000000009556a30 000007ff00c17f15 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Expression.MakeDynamic(System.Type, System.Runtime.CompilerServices.CallSiteBinder, Microsoft.Scripting.Ast.Expression, Microsoft.Scripting.Ast.Expression)+0×75
0000000009556aa0 000007ff00c171ca Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteDynamicExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xe5
0000000009556b10 000007ff00c180cd Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xd2a
0000000009556f70 000007ff00c1803f Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller+ChildRewriter.Add(Microsoft.Scripting.Ast.Expression)+0×4d
0000000009556fc0 000007ff00c17eb8 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller+ChildRewriter.AddArguments(Microsoft.Scripting.Ast.IArgumentProvider)+0×4f
0000000009557010 000007ff00c171ca Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteDynamicExpression(Microsoft.Scripting.Ast.Expression, Stack)+0×88
0000000009557080 000007ff00c180cd Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xd2a
00000000095574e0 000007ff00c1803f Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller+ChildRewriter.Add(Microsoft.Scripting.Ast.Expression)+0×4d
0000000009557530 000007ff00c17eb8 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller+ChildRewriter.AddArguments(Microsoft.Scripting.Ast.IArgumentProvider)+0×4f
0000000009557580 000007ff00c171ca Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteDynamicExpression(Microsoft.Scripting.Ast.Expression, Stack)+0×88
00000000095575f0 000007ff00c17993 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xd2a
0000000009557a50 000007ff00c1691a Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteConditionalExpression(Microsoft.Scripting.Ast.Expression, Stack)+0×93
0000000009557b20 000007ff00c17803 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0×47a
0000000009557f80 000007ff00c17156 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteBlockExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xa3
0000000009558030 000007ff00c179f9 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xcb6

last 4 lines Repeated 122 times

00000000095aa980 000007ff00c1691a Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteConditionalExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xf9
00000000095aaa50 000007ff00c17803 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0×47a
00000000095aaeb0 000007ff00c17156 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteBlockExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xa3
00000000095aaf60 000007ff00c179f9 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xcb6
00000000095ab3c0 000007ff00c1691a Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteConditionalExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xf9
00000000095ab490 000007ff00c17803 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0×47a
00000000095ab8f0 000007ff00c17156 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteBlockExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xa3
00000000095ab9a0 000007ff00c17803 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xcb6
00000000095abe00 000007ff00c17156 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteBlockExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xa3
00000000095abeb0 000007ff012b14af Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xcb6
00000000095ac310 000007ff00c17206 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExtensionExpression(Microsoft.Scripting.Ast.Expression, Stack)+0×6f
00000000095ac390 000007ff00c17803 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xd66
00000000095ac7f0 000007ff00c17156 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteBlockExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xa3
00000000095ac8a0 000007ff012b10f6 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xcb6
00000000095acd00 000007ff00c173da Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteTryExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xe6
00000000095ace50 000007ff00c17803 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xf3a
00000000095ad2b0 000007ff00c17156 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteBlockExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xa3
00000000095ad360 000007ff00c17803 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xcb6
00000000095ad7c0 000007ff00c17156 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteBlockExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xa3
00000000095ad870 000007ff00c18453 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xcb6
00000000095adcd0 000007ff00c172f0 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteLabelExpression(Microsoft.Scripting.Ast.Expression, Stack)+0×73
00000000095add70 000007ff00c163e8 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xe50
00000000095ae1d0 000007ff00c16245 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpressionFreeTemps(Microsoft.Scripting.Ast.Expression, Stack)+0×48
00000000095ae240 000007ff00c161d5 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.Rewrite[[System.__Canon, mscorlib]](Microsoft.Scripting.Ast.Expression`1)+0×45
00000000095ae2d0 000007ff00c16014 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Expression`1[[System.__Canon, mscorlib]].Accept(Microsoft.Scripting.Ast.Compiler.StackSpiller)+0×55
00000000095ae320 000007ff012b0355 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.LambdaCompiler.Compile(Microsoft.Scripting.Ast.LambdaExpression, System.Runtime.CompilerServices.DebugInfoGenerator)+0×44
00000000095ae370 000007fef8b3dd38 Microsoft_Dynamic!Microsoft.Scripting.Interpreter.LightDelegateCreator.Compile(System.Object)+0xb5
00000000095ae3d0 000007fef9aad502 mscorlib_ni!System.Threading.ExecutionContext.runTryCode(System.Object)+0×178
00000000095ae490 000007fef9969fd3 mscorwks!CallDescrWorker+0×82
00000000095ae4e0 000007fef997a3af mscorwks!CallDescrWorkerWithHandler+0xd3
00000000095ae580 000007fef995f916 mscorwks!MethodDesc::CallDescr+0×24f
00000000095ae7e0 000007fef9f0a302 mscorwks!ExecuteCodeWithGuaranteedCleanupHelper+0×12a
00000000095aea70 000007fef8b22b82 mscorwks!ReflectionInvocation::ExecuteCodeWithGuaranteedCleanup+0×172
00000000095aec80 000007fef8b97411 mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)+0×62
00000000095aecd0 000007fef8b9724f mscorlib_ni!System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(System.Threading._ThreadPoolWaitCallback)+0×61
00000000095aed20 000007fef9aad502 mscorlib_ni!System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(System.Object)+0×4f
00000000095aed70 000007fef9969fd3 mscorwks!CallDescrWorker+0×82
00000000095aedc0 000007fef9961c3a mscorwks!CallDescrWorkerWithHandler+0xd3
00000000095aee60 000007fef991d0bf mscorwks!DispatchCallDebuggerWrapper+0×3e
00000000095aeec0 000007fef9a22f17 mscorwks!DispatchCallNoEH+0×5f
00000000095aef40 000007fef98ec1a0 mscorwks!QueueUserWorkItemManagedCallback+0×83
00000000095aefd0 000007fef9928725 mscorwks!Thread::DoADCallBack+0×488
00000000095af020 000007fef98e0195 mscorwks!SVR::gc_heap::make_heap_segment+0×155
00000000095af0f0 000007fef98c0a21 mscorwks!AssemblySecurityDescriptor::GetZone+0×169
00000000095af130 000007fef98ebe5d mscorwks!ClassCompat::MethodTableBuilder::BuildInteropVTable_PlaceVtableMethods+0×441
00000000095af160 000007fef98ec1c5 mscorwks!Thread::DoADCallBack+0×145
00000000095af2d0 000007fef9928725 mscorwks!Thread::DoADCallBack+0×4ad
00000000095af320 000007fef98e0195 mscorwks!SVR::gc_heap::make_heap_segment+0×155
00000000095af3f0 000007fef98c4cc5 mscorwks!AssemblySecurityDescriptor::GetZone+0×169
00000000095af430 000007fef98f102a mscorwks!ThreadNative::KickOffThread+0×401
00000000095af490 000007fef99ca4ea mscorwks!ManagedPerAppDomainTPCount::DispatchWorkItem+0xee
00000000095af540 000007fef9a2095c mscorwks!ThreadpoolMgr::WorkerThreadStart+0×1ba
00000000095af5e0 00000000775cf56d mscorwks!Thread::intermediateThreadProc+0×78
00000000095af7b0 0000000077703281 kernel32!BaseThreadInitThunk+0xd
00000000095af7e0 0000000000000000 ntdll!RtlUserThreadStart+0×1d

DLR Forums where we were discussing the problem:
http://dlr.codeplex.com/Thread/View.aspx?ThreadId=212185

OS: 2008 R2 IIS 7.5 using MVC Framework with IronPython