Hi, I'm currently facing issues connecting to the Operations Manager Console on SCOM 2016, been reading and trying out all methods and suggestions from people who faced this issue but I'm still unable to resolve it. It seems that the Data Access Service crashes after a short period of time.
Below are the logs I have collected from the console and event logs from the system.
Operations Manager Error Log
Date: 7/30/2019 4:57:59 PM
Application: Operations Manager
Application Version: 7.2.12066.0
Severity: Error
Message: Failed to connect to server 'SCOM.CONTOSO.COM'
Microsoft.EnterpriseManagement.Common.ServiceNotRunningException: The Data Access service is either not running or not yet initialized. Check the event log for more information. ---> System.ServiceModel.EndpointNotFoundException: Could not connect to net.tcp://scom.contoso.com:5724/DispatcherService.
The connection attempt lasted for a time span of 00:00:02.0045240. TCP error code 10061: No connection could be made because the target machine actively refused it 192.168.1.20:5724. ---> System.Net.Sockets.SocketException: No connection could be
made because the target machine actively refused it 192.168.1.20:5724
at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
--- End of inner exception stack trace ---
Server stack trace:
at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
at System.ServiceModel.Channels.BufferedConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
at System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnection(TimeSpan timeout)
at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.LayeredChannel`1.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.CallOpenOnce.System.ServiceModel.Channels.ServiceChannel.ICallOnce.Call(ServiceChannel channel, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.CallOnceManager.CallOnce(TimeSpan timeout, CallOnceManager cascade)
at System.ServiceModel.Channels.ServiceChannel.EnsureOpened(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Microsoft.EnterpriseManagement.Common.Internal.IDispatcherService.Connect(SdkClientConnectionOptions connectionOptions)
at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.Initialize(EnterpriseManagementConnectionSettings connectionSettings, SdkChannelObject`1 channelObjectDispatcherService)
at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.CreateEndpoint[T](EnterpriseManagementConnectionSettings connectionSettings, SdkChannelObject`1 channelObjectDispatcherService)
--- End of inner exception stack trace ---
at Microsoft.EnterpriseManagement.Common.Internal.ExceptionHandlers.HandleChannelExceptions(Exception ex)
at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.CreateEndpoint[T](EnterpriseManagementConnectionSettings connectionSettings, SdkChannelObject`1 channelObjectDispatcherService)
at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.ConstructEnterpriseManagementGroupInternal[T,P](EnterpriseManagementConnectionSettings connectionSettings, ClientDataAccessCore clientCallback)
at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.RetrieveEnterpriseManagementGroupInternal[T,P](EnterpriseManagementConnectionSettings connectionSettings, ClientDataAccessCore callbackDispatcherService)
at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.Connect[T,P](EnterpriseManagementConnectionSettings connectionSettings, ClientDataAccessCore callbackDispatcherService)
at Microsoft.EnterpriseManagement.ManagementGroup.InternalInitialize(EnterpriseManagementConnectionSettings connectionSettings, ManagementGroupInternal internals)
at Microsoft.EnterpriseManagement.Mom.Internal.UI.Common.ManagementGroupSessionManager.Connect(String server)
at Microsoft.EnterpriseManagement.Monitoring.Console.Internal.ConsoleWindowBase.TryConnectToManagementGroupJob(Object sender, ConsoleJobEventArgs args)
System.ServiceModel.EndpointNotFoundException: Could not connect to net.tcp://scom.contoso.com:5724/DispatcherService. The connection attempt lasted for a time span of 00:00:02.0045240. TCP error code 10061: No connection could be made because the target machine
actively refused it 192.168.1.20:5724. ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 192.168.1.20:5724
at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
--- End of inner exception stack trace ---
Server stack trace:
at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
at System.ServiceModel.Channels.BufferedConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
at System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnection(TimeSpan timeout)
at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.LayeredChannel`1.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.CallOpenOnce.System.ServiceModel.Channels.ServiceChannel.ICallOnce.Call(ServiceChannel channel, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.CallOnceManager.CallOnce(TimeSpan timeout, CallOnceManager cascade)
at System.ServiceModel.Channels.ServiceChannel.EnsureOpened(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Microsoft.EnterpriseManagement.Common.Internal.IDispatcherService.Connect(SdkClientConnectionOptions connectionOptions)
at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.Initialize(EnterpriseManagementConnectionSettings connectionSettings, SdkChannelObject`1 channelObjectDispatcherService)
at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.CreateEndpoint[T](EnterpriseManagementConnectionSettings connectionSettings, SdkChannelObject`1 channelObjectDispatcherService)
System.Net.Sockets.SocketException (0x80004005): No connection could be made because the target machine actively refused it 192.168.1.20:5724
at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
Event Log - OpsMgr SDK Service
The System Center Data Access service failed due to an unhandled exception.
The service will attempt to restart.
Exception:
Microsoft.EnterpriseManagement.Common.SdkServiceNotInitializedException: The Data Access service has not yet initialized. Please try again.
at Microsoft.EnterpriseManagement.ServiceDataLayer.DispatcherService.get_Container()
at Microsoft.EnterpriseManagement.Mom.Sdk.Service.SdkSubService.SdkChannel.Start()
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
Event Log - OpsMgr Management Configuration
OpsMgr Management Configuration Service failed to communicate with System Center Data Access Service due to the following exception
System.Runtime.Remoting.RemotingException: Unable to get ISdkService interface. Please make sure local Sdk Service is running.
at Microsoft.EnterpriseManagement.ManagementConfiguration.Communication.CredentialDataProvider.CreateSdkConnector()
at Microsoft.EnterpriseManagement.ManagementConfiguration.Communication.CredentialDataProvider.GetSecureData(Guid agentId, ICollection`1 addedReferenceList, ICollection`1 deletedReferenceList, ICollection`1 addedCredentialList, ICollection`1 deletedCredentialList,
Byte[]& hashValue)
at Microsoft.EnterpriseManagement.ManagementConfiguration.Engine.TracingCredentialDataProvider.GetSecureData(Guid agentId, ICollection`1 addedReferenceList, ICollection`1 deletedReferenceList, ICollection`1 addedCredentialList, ICollection`1 deletedCredentialList,
Byte[]& hashValue)
at Microsoft.EnterpriseManagement.ManagementConfiguration.Engine.AgentConfigurationFormatter.WriteSecureData(AgentConfigurationStream stream, XmlWriter writer, Guid agentId, Hashtable credentialAssociationList, Hashtable credentialList)
Event Log (Application) - .NET Runtime Error Log
Application: Microsoft.Mom.Sdk.ServiceHost.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: Microsoft.EnterpriseManagement.Common.SdkServiceNotInitializedException
at Microsoft.EnterpriseManagement.ServiceDataLayer.DispatcherService.get_Container()
at Microsoft.EnterpriseManagement.Mom.Sdk.Service.SdkSubService+SdkChannel.Start()
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
Event Log (Application) - Application Error Log
Faulting application name: Microsoft.Mom.Sdk.ServiceHost.exe, version: 7.2.12150.0, time stamp: 0x5c9cd835
Faulting module name: KERNELBASE.dll, version: 10.0.14393.2848, time stamp: 0x5c7f626e
Exception code: 0xe0434352
Fault offset: 0x0000000000034078
Faulting process id: 0x154c
Faulting application start time: 0x01d546b1f30ae0d1
Faulting application path: C:\Program Files\Microsoft System Center 2016\Operations Manager\Server\Microsoft.Mom.Sdk.ServiceHost.exe
Faulting module path: C:\Windows\System32\KERNELBASE.dll
Report Id: ff3db2d7-97dd-4d5e-8255-bd205da286bd
Faulting package full name:
Faulting package-relative application ID:
------------------------------------ End of Logs ------------------------------------
SCOM 2016 have been installed on to a Windows Server 2016 virtual machine. No Update Rollups have been installed on the SCOM 2016.
I have Microsoft SQL Server 2016 installed on another set of Windows Server 2016 host machines configured in Active/Passive failover configuration which is handled by Failover cluster. This serves as my database for the SCOM 2016.
Below are the steps I took trying to resolve the issue.
- All firewalls have been turned off
- Operations databases and warehouse databases are no where nearing their disk capacity(healthy)
- Configured the Data Access Service and Management Configuration Service to use a Domain Account as their service account
- Configured MSOMSdkSvc SPNs on the designated Service Account and the MSOMHSvc on the computer account that the SCOM is installed on. (According to this article https://kevinholman.com/2011/08/08/opsmgr-2012-what-should-the-spns-look-like/)
- The Service Account used by the database is the same service account used by both the Data Access Service and Management Configuration Service.
- The Service Account have been granted the following rights:
- apm_datareader, apm_datawriter
- ConfigService
- ConfigServiceMonitoringUsers
- configsvc_users
- db_accessadmin
- db_datareader, db_datawriter
- db_ddladmin
- db_owner,
- db_securityadmin,
- dbmodule_users
- public,
- sdk_users
- sql_dependency_subscriber
- Ensured that both SCOM VM and Database machines are using TLS 1.0 (TLS 1.1 and 1.2 keys have been removed entirely from the registry)
- Ensured that the Service Account used is added into the local policy for "Logon Locally", "Logon as a service"
- Added the Service Account to the Local Administrators of the SCOM VM and Database machines
- Tested the connection on port 5724 of the SCOM VM but failed as the service is not listening on port 5724
Been trying to resolve this issue for weeks but it seems like I'm back at where I started.