Hi All,
I have a strange issue with my SCOM 2007 R2 CU4.
My management server grayes out frequently and when i check the operations manager console it throws me these two errors. Can anyone give me a solution for this.
Note: The following information was gathered when the operation was attempted. The information may appear cryptic but provides context for the error. The application will continue to run.
Microsoft.EnterpriseManagement.Common.UnknownAuthorizationStoreException: Unable to perform the operation because of authorization store errors. ---> System.Runtime.InteropServices.COMException (0x800705AA): Insufficient system resources exist to complete
the requested service. (Exception from HRESULT: 0x800705AA)
at Microsoft.Interop.Security.AzRoles.IAzApplication2.InitializeClientContextFromToken(UInt64 ullTokenHandle, Object varReserved)
at Microsoft.EnterpriseManagement.Mom.Sdk.Authorization.AzManHelper.AccessCheck(String accessCheckContext, Int32[] operationIds, IntPtr hToken, String stringSid, Int32[] accessCheckReturnCodes, List`1[] accessCheckScopes)
--- End of inner exception stack trace ---
at Microsoft.EnterpriseManagement.DataAbstractionLayer.SdkDataAbstractionLayer.HandleIndigoExceptions(Exception ex)
at Microsoft.EnterpriseManagement.DataAbstractionLayer.InstanceSpaceOperations.GetMonitoringObjectByMonitoringObjectIds(List`1 monitoringObjectIds, String languageCode, MonitoringObjectMode monitoringObjectMode)
at Microsoft.EnterpriseManagement.ManagementGroup.GetPartialMonitoringObjects(ICollection`1 ids)
at Microsoft.EnterpriseManagement.Mom.Internal.UI.Common.TaskPaneContext.GetManagedEntities(ICollection`1 items)
at Microsoft.EnterpriseManagement.Mom.Internal.UI.Common.TaskPaneContext.UpdateContextJob(Object sender, ConsoleJobEventArgs args)
at Microsoft.EnterpriseManagement.Mom.Internal.UI.Console.ConsoleJobExceptionHandler.ExecuteJob(IComponent component, EventHandler`1 job, Object sender, ConsoleJobEventArgs args)
System.Runtime.InteropServices.COMException (0x800705AA): Insufficient system resources exist to complete the requested service. (Exception from HRESULT: 0x800705AA)
at Microsoft.Interop.Security.AzRoles.IAzApplication2.InitializeClientContextFromToken(UInt64 ullTokenHandle, Object varReserved)
at Microsoft.EnterpriseManagement.Mom.Sdk.Authorization.AzManHelper.AccessCheck(String accessCheckContext, Int32[] operationIds, IntPtr hToken, String stringSid, Int32[] accessCheckReturnCodes, List`1[] accessCheckScopes)
====================================================================================
Note: The following information was gathered when the operation was attempted. The information may appear cryptic but provides context for the error. The application will continue to run.
Microsoft.Mom.Isam.IsamOutOfMemoryException: Out of Memory (-1011)
at Microsoft.Mom.Isam.?A0x5359b89f.HandleError(Int32 err)
at Microsoft.Mom.Isam.EseInterop.JetGetIndexInfo(JET_SESID sesid, JET_DBID dbid, String table, String index)
at Microsoft.Mom.Isam.IndexCollection.get_Item(String indexName)
at Microsoft.Mom.Isam.Cursor.get_CurrentIndexDefinition()
at Microsoft.Mom.Isam.Cursor.MakeKey(Key key, Boolean end)
at Microsoft.Mom.Isam.Cursor.GotoKey(Key key)
at Microsoft.EnterpriseManagement.Mom.Internal.UI.Cache.EseCursor.Find(Object[] values)
at Microsoft.EnterpriseManagement.Mom.Internal.UI.MPImageCache.RetrieveImageIdFromCache(CacheTable classToImageMapTable, ManagementPackElementImageKey imageKey, Guid& imageId)
at Microsoft.EnterpriseManagement.Mom.Internal.UI.MPImageCache.GetImageCore(CacheSession session, ManagementGroup managementGroup, ManagementPackElementImageKey imageKey, Guid& imageId)
at Microsoft.EnterpriseManagement.Mom.Internal.UI.MPImageCache.GetImageJob(Object sender, ConsoleJobEventArgs jobArgs)
at Microsoft.EnterpriseManagement.Mom.Internal.UI.Console.ConsoleJobExceptionHandler.ExecuteJob(IComponent component, EventHandler`1 job, Object sender, ConsoleJobEventArgs args)
Screenshot of the error:
Also i restarted the health service and the health service fails to start below is the screenshot of that.
Analysed the event logs and found the below:
Event id 29104 - OpsMgr Config Service failed to send the dirty state notifications to the dirty OpsMgr Health Services. This may be happening because the Root OpsMgr Health Service is not running.
Event id : 4000 - A monitoring host is unresponsive or has crashed. The status code for the host failure was 2164195371.
Event id 4503 - A module reported an error 0x8007000E from a callback which was running as part of rule "_44E9A997_6A02_4298_8430_8E01952AB6F3_.RaiseAlert" running for instance "Root managementserver FQDN" with id:"{6C3444C3-3990-BB5A-F25E-289D8F427570}" in management group "CINSCOM".
Event id - 21044 - The OpsMgr Connector cannot uncompress package, received from IP XXXXXXXXXXXX
Event id 1103 - Summary: 28 rule(s)/monitor(s) failed and got unloaded, 0 of them reached the failure limit that prevents automatic reload. Management group "CINSCOM". This is summary only event, please see other events with descriptions of unloaded rule(s)/monitor(s).
26380 - The System Center Operations Manager SDK Service failed due to an unhandled exception.
The service will attempt to restart.
Exception:
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at Bid.TraceError(String fmtPrintfW, Object a1, Object a2)
at Microsoft.EnterpriseManagement.Mom.DataAccess.SqlRetryHandler.Execute[T](ExecuteArguments executeArguments, RetryPolicy retryPolicy, GenericExecute`1 genericExecute)
at Microsoft.EnterpriseManagement.Mom.DataAccess.SqlRetryHandler.ExecuteReaderSingleRow(SqlDataReader sqlDataReader, SqlConnection sqlConnection, IList`1 prologEpilogList, RetryPolicy retryPolicy)
at Microsoft.EnterpriseManagement.Mom.DataAccess.QueryResultsReader.Read()
at Microsoft.EnterpriseManagement.Mom.ServiceDataLayer.ClientReaderManager.GetObjects(Guid id, Int32 count)
at Microsoft.EnterpriseManagement.Mom.ServiceDataLayer.SdkDataAccess.GetObjectsFromReader(Guid readerId, Int32 count)
at Microsoft.EnterpriseManagement.Mom.ServiceDataLayer.SdkDataAccessTieringWrapper.GetObjectsFromReader(Guid readerId, Int32 count)
at Microsoft.EnterpriseManagement.Mom.ServiceDataLayer.SdkDataAccessExceptionTracingWrapper.GetObjectsFromReader(Guid readerId, Int32 count)
at SyncInvokeGetObjectsFromReader(Object , Object[] , Object[] )
at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage4(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)
at System.ServiceModel.Dispatcher.ChannelHandler.DispatchAndReleasePump(RequestContext request, Boolean cleanThread, OperationContext currentOperationContext)
at System.ServiceModel.Dispatcher.ChannelHandler.HandleRequest(RequestContext request, OperationContext currentOperationContext)
at System.ServiceModel.Dispatcher.ChannelHandler.AsyncMessagePump(IAsyncResult result)
at System.ServiceModel.Diagnostics.Utility.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
at System.ServiceModel.AsyncResult.Complete(Boolean completedSynchronously)
at System.ServiceModel.Channels.FramingDuplexSessionChannel.TryReceiveAsyncResult.OnReceive(IAsyncResult result)
at System.ServiceModel.Diagnostics.Utility.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
at System.ServiceModel.AsyncResult.Complete(Boolean completedSynchronously)
at System.ServiceModel.Channels.SynchronizedMessageSource.SynchronizedAsyncResult`1.CompleteWithUnlock(Boolean synchronous, Exception exception)
at System.ServiceModel.Channels.SynchronizedMessageSource.ReceiveAsyncResult.OnReceiveComplete(Object state)
at System.ServiceModel.Channels.SessionConnectionReader.OnAsyncReadComplete(Object state)
at System.ServiceModel.Channels.StreamConnection.OnRead(IAsyncResult result)
at System.ServiceModel.Diagnostics.Utility.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)
at System.Net.Security.NegotiateStream.ProcessFrameBody(Int32 readBytes, Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.NegotiateStream.ReadCallback(AsyncProtocolRequest asyncRequest)
at System.Net.FixedSizeReader.CheckCompletionBeforeNextRead(Int32 bytes)
at System.Net.FixedSizeReader.ReadCallback(IAsyncResult transportResult)
at System.ServiceModel.AsyncResult.Complete(Boolean completedSynchronously)
at System.ServiceModel.Channels.ConnectionStream.ReadAsyncResult.OnAsyncReadComplete(Object state)
at System.ServiceModel.Channels.SocketConnection.FinishRead()
at System.ServiceModel.Channels.SocketConnection.AsyncReadCallback(Boolean haveResult, Int32 error, Int32 bytesRead)
at System.ServiceModel.Diagnostics.Utility.IOCompletionThunk.UnhandledExceptionFrame(UInt32 error, UInt32 bytesRead, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
======================================================
Below is the configuration of the server:
Windows server 2008 r2 enterprise (Without SP1)
SQL server standard edition 2008.
Intel Xeon 2.6 Ghz (2 Processors) Totally 8 cores.
12GB Physical memory (Always 95 - 98% used)
427 agents being monitored (Exchange 2010, Blackberry etc)
All the roles (SCOM, RMS, SQL all are in the same box)
Can anyone please help.
Gautam.75801