Hi,
We are having this issue on our prod server from yesterday. Everytime this happens Server becomes unresponsive and I have to reboot the machine. Looks like this is being caused due to high activity on the server. I don't know where to start on this. Can someone advise on this?
SQL Server 2008 R2 (64 bit) enterprise edition
Error log says the below:
Message
New queries assigned to process on Node 3 have not been picked up by a worker thread in the last 60 seconds. Blocking or long-running queries can contribute to this condition, and may degrade client response time. Use the "max worker threads" configuration
option to increase number of allowable threads, or optimize current running queries. SQL Process Utilization: 35%. System Idle: 63%.
Detailed log :
01/27/2013 14:36:08,Server,Unknown,**Dump thread - spid = 0<c/> EC = 0x0000000000000000
01/27/2013 14:36:08,Server,Unknown,***Stack Dump being sent to R:\MSSQL10_50.GBIDC2DW01\MSSQL\LOG\SQLDump0004.txt
01/27/2013 14:36:08,Server,Unknown,* *******************************************************************************
01/27/2013 14:36:08,Server,Unknown,*
01/27/2013 14:36:08,Server,Unknown,* BEGIN STACK DUMP:
01/27/2013 14:36:08,Server,Unknown,* 01/27/13 14:36:08 spid 30576
01/27/2013 14:36:08,Server,Unknown,*
01/27/2013 14:36:08,Server,Unknown,* Deadlocked Schedulers
01/27/2013 14:36:08,Server,Unknown,*
01/27/2013 14:36:08,Server,Unknown,* *******************************************************************************
01/27/2013 14:36:08,Server,Unknown,* -------------------------------------------------------------------------------
01/27/2013 14:36:08,Server,Unknown,* Short Stack Dump
01/27/2013 14:36:08,Server,Unknown,Stack Signature for the dump is 0x00000000000003BB
01/27/2013 14:36:13,Server,Unknown,External dump process return code 0x20000001.<nl/>External dump process returned no errors.
01/27/2013 14:36:13,Server,Unknown,New queries assigned to process on Node 3 have not been picked up by a worker thread in the last 60 seconds. Blocking or long-running queries can contribute to this condition<c/> and may degrade clientresponse time. Use the "max worker threads" configuration option to increase number of allowable threads<c/> oroptimize current running queries. SQL Process Utilization: 35%. System Idle: 63%.
01/27/2013 14:37:13,Server,Unknown,New queries assigned to process on Node 3 have not been picked up by a worker thread inthe last 120 seconds. Blocking or long-running queries can contribute to this condition<c/> and may degrade clientresponse time. Use the "max worker threads" configuration option to increase number of allowable threads<c/> oroptimize current running queries. SQL Process Utilization: 22%. System Idle: 76%.
01/27/2013 14:38:13,Server,Unknown,New queries assigned to process on Node 3 have not been picked up by a worker thread inthe last 180 seconds. Blocking or long-running queries can contribute to this condition<c/> and may degrade clientresponse time. Use the "max worker threads" configuration option to increase number of allowable threads<c/> oroptimize current running queries. SQL Process Utilization: 17%. System Idle: 81%.
01/27/2013 14:39:13,Server,Unknown,New queries assigned to process on Node 3 have not been picked up by a worker thread inthe last 240 seconds. Blocking or long-running queries can contribute to this condition<c/> and may degrade clientresponse time. Use the "max worker threads" configuration option to increase number of allowable threads<c/> oroptimize current running queries. SQL Process Utilization: 15%. System Idle: 84%.
01/27/2013 14:40:13,Server,Unknown,New queries assigned to process on Node 3 have not been picked up by a worker thread inthe last 300 seconds. Blocking or long-running queries can contribute to this condition<c/> and may degrade clientresponse time. Use the "max worker threads" configuration option to increase number of allowable threads<c/> oroptimize current running queries. SQL Process Utilization: 15%. System Idle: 84%.
01/27/2013 14:40:38,spid25s,Unknown,deadlock-list
01/27/2013 14:40:38,spid25s,Unknown,deadlock victim=process181e452748
01/27/2013 14:40:38,spid25s,Unknown,process-list
01/27/2013 14:40:38,spid25s,Unknown,process id=process181e452748 taskpriority=0 logused=0 waitresource=RID:7:1:64811:24 waittime=2933 ownerId=253627277 transactionname=SELECT lasttranstarted=2013-01-27T14:40:35.903XDES=0x3af72b9d0 lockMode=S schedulerid=16 kpid=20604 status=suspended spid=121 sbid=0 ecid=0 priority=0trancount=0 lastbatchstarted=2013-01-27T14:40:35.903 lastbatchcompleted=2013-01-27T14:40:35.727lastattention=1900-01-01T00:00:00.727 hostpid=3124 loginname=bi_login isolationlevel=read committed (2)xactid=253627277 currentdb=7 lockTimeout=4294967295 clientoption1=671088672 clientoption2=128056
01/27/2013 14:40:38,spid25s,Unknown,executionStack
01/27/2013 14:40:38,spid25s,Unknown,frame procname=adhoc line=1sqlhandle=0x020000004426a90b6f82b59f2072e1d0dbfeafece4835376
01/27/2013 14:40:38,spid25s,Unknown,(@1 varchar(8000)<c/>@2 tinyint<c/>@3 tinyint)SELECT [dbo].[CMS_InfoObjects6].[ObjectID] FROM [dbo].[CMS_InfoObjects6] WHERE [dbo].[CMS_InfoObjects6].[ObjName]=CONVERT([varbinary](255)<c/>@1<c/>0) AND ([dbo].[CMS_InfoObjects6].[TypeID]=@2 AND [SI_PLUGIN_OBJECT]=@3) ORDER BY [dbo].[CMS_InfoObjects6].[ObjectID] ASC
01/27/2013 14:40:38,spid25s,Unknown,frame procname=adhoc line=1sqlhandle=0x0200000023120f04db700ff458e3a2aca66f79aa4485317f
01/27/2013 14:40:38,spid25s,Unknown,SELECT dbo.CMS_InfoObjects6.ObjectID FROM dbo.CMS_InfoObjects6 WHERE(dbo.CMS_InfoObjects6.ObjName = CAST('IEMM?)O)K51O!;J' AS VARBINARY(255)) AND (dbo.CMS_InfoObjects6.TypeID =19 AND SI_PLUGIN_OBJECT = 0)) ORDER BY dbo.CMS_InfoObjects6.ObjectID ASC
01/27/2013 14:40:38,spid25s,Unknown,inputbuf
01/27/2013 14:40:38,spid25s,Unknown,SELECT dbo.CMS_InfoObjects6.ObjectID FROM dbo.CMS_InfoObjects6 WHERE(dbo.CMS_InfoObjects6.ObjName = CAST('IEMM?)O)K51O!;J' AS VARBINARY(255)) AND (dbo.CMS_InfoObjects6.TypeID =19 AND SI_PLUGIN_OBJECT = 0)) ORDER BY dbo.CMS_InfoObjects6.ObjectID ASC
01/27/2013 14:40:38,spid25s,Unknown,process id=processfe1d048 taskpriority=0 logused=2228 waitresource=KEY:7:72057594071285760 (9be724bd0e79) waittime=2933 ownerId=253627124 transactionname=user_transactionlasttranstarted=2013-01-27T14:40:35.867 XDES=0x1826982e80 lockMode=X schedulerid=15 kpid=30976status=suspended spid=100 sbid=0 ecid=0 priority=0 trancount=2 lastbatchstarted=2013-01-27T14:40:35.897lastbatchcompleted=2013-01-27T14:40:35.897 lastattention=1900-01-01T00:00:00.897 hostpid=3172loginname=bi_login isolationlevel=read committed (2) xactid=253627124 currentdb=7 lockTimeout=4294967295clientoption1=671088672 clientoption2=128056
01/27/2013 14:40:38,spid25s,Unknown,executionStack
01/27/2013 14:40:38,spid25s,Unknown,frame procname=adhoc line=3 stmtstart=150 stmtend=292sqlhandle=0x02000000eb585c3ad22432fda61435961b20d19447b0457d
01/27/2013 14:40:38,spid25s,Unknown,DELETE FROM dbo.CMS_InfoObjects6 WHERE ObjectID IN (15956911<c/>15956912);
01/27/2013 14:40:38,spid25s,Unknown,inputbuf
01/27/2013 14:40:38,spid25s,Unknown,BEGIN<nl/>DELETE FROM dbo.CMS_Aliases6 WHERE ObjectID IN (15956911<c/>15956912);<nl/>DELETE FROM dbo.CMS_InfoObjects6 WHERE ObjectID IN (15956911<c/>15956912);<nl/>END;
01/27/2013 14:40:38,spid25s,Unknown,resource-list
01/27/2013 14:40:38,spid25s,Unknown,ridlock fileid=1 pageid=64811 dbid=7 objectname=PROD_CMS.dbo.CMS_InfoObjects6id=lock1c6d10ca80 mode=X associatedObjectId=72057594050379776
01/27/2013 14:40:38,spid25s,Unknown,owner-list
01/27/2013 14:40:38,spid25s,Unknown,owner id=processfe1d048 mode=X
01/27/2013 14:40:38,spid25s,Unknown,waiter-list
01/27/2013 14:40:38,spid25s,Unknown,waiter id=process181e452748 mode=S requestType=wait
01/27/2013 14:40:38,spid25s,Unknown,keylock hobtid=72057594071285760 dbid=7objectname=PROD_CMS.dbo.CMS_InfoObjects6 indexname=ObjName_I6 id=lockb5d8e1100 mode=SassociatedObjectId=72057594071285760
01/27/2013 14:40:38,spid25s,Unknown,owner-list
01/27/2013 14:40:38,spid25s,Unknown,owner id=process181e452748 mode=S
01/27/2013 14:40:38,spid25s,Unknown,waiter-list
01/27/2013 14:40:38,spid25s,Unknown,waiter id=processfe1d048 mode=X requestType=wait
01/27/2013 14:41:13,Server,Unknown,New queries assigned to process on Node 3 have not been picked up by a worker thread inthe last 360 seconds. Blocking or long-running queries can contribute to this condition<c/> and may degrade clientresponse time. Use the "max worker threads" configuration option to increase number of allowable threads<c/> oroptimize current running queries. SQL Process Utilization: 16%. System Idle: 82%.
01/27/2013 14:42:13,Server,Unknown,New queries assigned to process on Node 3 have not been picked up by a worker thread inthe last 420 seconds. Blocking or long-running queries can contribute to this condition<c/> and may degrade clientresponse time. Use the "max worker threads" configuration option to increase number of allowable threads<c/> oroptimize current running queries. SQL Process Utilization: 16%. System Idle: 83%.
01/27/2013 14:43:14,Server,Unknown,New queries assigned to process on Node 3 have not been picked up by a worker thread inthe last 480 seconds. Blocking or long-running queries can contribute to this condition<c/> and may degrade clientresponse time. Use the "max worker threads" configuration option to increase number of allowable threads<c/> oroptimize current running queries. SQL Process Utilization: 15%. System Idle: 84%.
01/27/2013 14:44:14,Server,Unknown,New queries assigned to process on Node 3 have not been picked up by a worker thread inthe last 540 seconds. Blocking or long-running queries can contribute to this condition<c/> and may degrade clientresponse time. Use the "max worker threads" configuration option to increase number of allowable threads<c/> oroptimize current running queries. SQL Process Utilization: 14%. System Idle: 84%.
01/27/2013 14:45:14,Server,Unknown,New queries assigned to process on Node 3 have not been picked up by a worker thread inthe last 600 seconds. Blocking or long-running queries can contribute to this condition<c/> and may degrade clientresponse time. Use the "max worker threads" configuration option to increase number of allowable threads<c/> oroptimize current running queries. SQL Process Utilization: 14%. System Idle: 84%.
01/27/2013 14:46:14,Server,Unknown,New queries assigned to process on Node 3 have not been picked up by a worker thread inthe last 660 seconds. Blocking or long-running queries can contribute to this condition<c/> and may degrade clientresponse time. Use the "max worker threads" configuration option to increase number of allowable threads<c/> oroptimize current running queries. SQL Process Utilization: 14%. System Idle: 84%.
01/27/2013 14:47:14,Server,Unknown,New queries assigned to process on Node 3 have not been picked up by a worker thread inthe last 720 seconds. Blocking or long-running queries can contribute to this condition<c/> and may degrade clientresponse time. Use the "max worker threads" configuration option to increase number of allowable threads<c/> oroptimize current running queries. SQL Process Utilization: 14%. System Idle: 84%.
01/27/2013 14:48:14,Server,Unknown,New queries assigned to process on Node 3 have not been picked up by a worker thread inthe last 780 seconds. Blocking or long-running queries can contribute to this condition<c/> and may degrade clientresponse time. Use the "max worker threads" configuration option to increase number of allowable threads<c/> oroptimize current running queries. SQL Process Utilization: 14%. System Idle: 85%.
01/27/2013 14:49:14,Server,Unknown,New queries assigned to process on Node 3 have not been picked up by a worker thread inthe last 840 seconds. Blocking or long-running queries can contribute to this condition<c/> and may degrade clientresponse time. Use the "max worker threads" configuration option to increase number of allowable threads<c/> oroptimize current running queries. SQL Process Utilization: 13%. System Idle: 85%.
01/27/2013 14:50:14,Server,Unknown,New queries assigned to process on Node 3 have not been picked up by a worker thread inthe last 900 seconds. Blocking or long-running queries can contribute to this condition<c/> and may degrade clientresponse time. Use the "max worker threads" configuration option to increase number of allowable threads<c/> oroptimize current running queries. SQL Process Utilization: 13%. System Idle: 85%.
01/27/2013 14:51:14,Server,Unknown,New queries assigned to process on Node 3 have not been picked up by a worker thread inthe last 960 seconds. Blocking or long-running queries can contribute to this condition<c/> and may degrade clientresponse time. Use the "max worker threads" configuration option to increase number of allowable threads<c/> oroptimize current running queries. SQL Process Utilization: 13%. System Idle: 86%.
01/27/2013 14:52:14,Server,Unknown,New queries assigned to process on Node 3 have not been picked up by a worker thread inthe last 1020 seconds. Blocking or long-running queries can contribute to this condition<c/> and may degrade clientresponse time. Use the "max worker threads" configuration option to increase number of allowable threads<c/> oroptimize current running queries. SQL Process Utilization: 13%. System Idle: 85%.
01/27/2013 14:53:14,Server,Unknown,New queries assigned to process on Node 3 have not been picked up by a worker thread inthe last 1080 seconds. Blocking or long-running queries can contribute to this condition<c/> and may degrade clientresponse time. Use the "max worker threads" configuration option to increase number of allowable threads<c/> oroptimize current running queries. SQL Process Utilization: 13%. System Idle: 85%.
01/27/2013 14:54:15,Server,Unknown,New queries assigned to process on Node 3 have not been picked up by a worker thread inthe last 1140 seconds. Blocking or long-running queries can contribute to this condition<c/> and may degrade clientresponse time. Use the "max worker threads" configuration option to increase number of allowable threads<c/> oroptimize current running queries. SQL Process Utilization: 13%. System Idle: 86%.
01/27/2013 14:55:15,Server,Unknown,New queries assigned to process on Node 3 have not been picked up by a worker thread inthe last 1200 seconds. Blocking or long-running queries can contribute to this condition<c/> and may degrade client response time. Use the "max worker threads" configuration option to increase number
of allowable threads<c/> or optimize current running queries. SQL Process Utilization: 12%. System Idle: 86%.
01/27/2013 14:56:15,Server,Unknown,New queries assigned to process on Node 3 have not been picked up by a worker thread in the last 1260 seconds. Blocking or long-running queries can contribute to this condition<c/> and may degrade client response time.
Use the "max worker threads" configuration option to increase number of allowable threads<c/> or optimize current running queries. SQL Process Utilization: 12%. System Idle: 86%.
01/27/2013 14:57:15,Server,Unknown,New queries assigned to process on Node 3 have not been picked up by a worker thread in the last 1320 seconds. Blocking or long-running queries can contribute to this condition<c/> and may degrade client response time.
Use the "max worker threads" configuration option to increase number of allowable threads<c/> or optimize current running queries. SQL Process Utilization: 12%. System Idle: 86%.
01/27/2013 14:58:15,Server,Unknown,New queries assigned to process on Node 3 have not been picked up by a worker thread in the last 1380 seconds. Blocking or long-running queries can contribute to this condition<c/> and may degrade client response time.
Use the "max worker threads" configuration option to increase number of allowable threads<c/> or optimize current running queries. SQL Process Utilization: 12%. System Idle: 87%.
01/27/2013 14:59:15,Server,Unknown,New queries assigned to process on Node 3 have not been picked up by a worker thread in the last 1440 seconds. Blocking or long-running queries can contribute to this condition<c/> and may degrade client response time.
Use the "max worker threads" configuration option to increase number of allowable threads<c/> or optimize current running queries. SQL Process Utilization: 11%. System Idle: 87%.
01/27/2013 15:00:15,Server,Unknown,New queries assigned to process on Node 3 have not been picked up by a worker thread in the last 1500 seconds. Blocking or long-running queries can contribute to this condition<c/> and may degrade client response time.
Use the "max worker threads" configuration option to increase number of allowable threads<c/> or optimize current running queries. SQL Process Utilization: 12%. System Idle: 87%.
01/27/2013 15:01:15,Server,Unknown,New queries assigned to process on Node 3 have not been picked up by a worker thread in the last 1560 seconds. Blocking or long-running queries can contribute to this condition<c/> and may degrade client response time.
Use the "max worker threads" configuration option to increase number of allowable threads<c/> or optimize current running queries. SQL Process Utilization: 12%. System Idle: 87%.
01/27/2013 15:02:16,Server,Unknown,New queries assigned to process on Node 3 have not been picked up by a worker thread in the last 1620 seconds. Blocking or long-running queries can contribute to this condition<c/> and may degrade client response time.
Use the "max worker threads" configuration option to increase number of allowable threads<c/> or optimize current running queries. SQL Process Utilization: 12%. System Idle: 86%.
01/27/2013 15:03:16,Server,Unknown,New queries assigned to process on Node 3 have not been picked up by a worker thread in the last 1680 seconds. Blocking or long-running queries can contribute to this condition<c/> and may degrade client response time.
Use the "max worker threads" configuration option to increase number of allowable threads<c/> or optimize current running queries. SQL Process Utilization: 13%. System Idle: 85%.
01/27/2013 15:04:16,Server,Unknown,New queries assigned to process on Node 3 have not been picked up by a worker thread in the last 1740 seconds. Blocking or long-running queries can contribute to this condition<c/> and may degrade client response time.
Use the "max worker threads" configuration option to increase number of allowable threads<c/> or optimize current running queries. SQL Process Utilization: 14%. System Idle: 85%.
01/27/2013 15:05:16,Server,Unknown,New queries assigned to process on Node 3 have not been picked up by a worker thread in the last 1800 seconds. Blocking or long-running queries can contribute to this condition<c/> and may degrade client response time.
Use the "max worker threads" configuration option to increase number of allowable threads<c/> or optimize current running queries. SQL Process Utilization: 14%. System Idle: 84%.
01/27/2013 15:05:24,spid242,Unknown,Configuration option 'Agent XPs' changed from 1 to 0. Run the RECONFIGURE statement to install.
01/27/2013 15:05:24,spid242,Unknown,FILESTREAM: effective level = 0<c/> configured level = 0<c/> file system access share name = 'server01'.