TCadmin Service Manager crashing after some uptime

[gijs007]

In the last few months I'm starting to see issues with the TCadmin Service Manager on my servers.

The problem happens regularly on the servers with a higher amount of services, but has also been seen on servers which host less than 30 game servers.

 

What I see is that the TCadmin Service Manager service crashes, restarting the Service Manager service results in a new crash within 2-60 minutes.

Additionally symptoms are 10010 event ID's related to DistributedCOM errors in the Windows Eventlog.

 

The only quick fix is to restart Windows.

In some cases I was able to run the TCadmin Service Manager in console mode, which somehow doesn't crash (as often).

 

I've already been in touch with TCadmin support since the end of January regarding this issue, but we haven't found a solution yet which is quite frustrating..

Luis suggested applying the registry settings for Windows Server 2008/2012 from: https://clients.tcadmin.com/knowledgebase.php?action=displayarticle&id=9

Unfortunately applying these registry settings did not improve the situation on my servers (using Windows Server 2016).

 

Are there any other people who experience this problem?

Any help in finding the cause and/or resolving this problem is appreciated!

Edited March 9, 2018 by gijs007

[Dennis]

Could your share your error logs?

[gijs007]

The service manager logs show several of the following items:

System System.ComponentModel.Win32Exception: Access is denied

at System.Diagnostics.Process.set_ProcessorAffinity(IntPtr value)

at TCAdmin.ServiceManLib.ProcessController.SetProcessAffinity(Process p, Int64 affinity)

 

mscorlib System.Security.Principal.IdentityNotMappedException: Some or all identity references could not be translated.

at System.Security.Principal.NTAccount.Translate(IdentityReferenceCollection sourceAccounts, Type targetType, Boolean forceSuccess)

at System.Security.Principal.NTAccount.Translate(Type targetType)

at System.Security.AccessControl.CommonObjectSecurity.ModifyAccess(AccessControlModification modification, AccessRule rule, Boolean& modified)

at System.Security.AccessControl.CommonObjectSecurity.AddAccessRule(AccessRule rule)

at TCAdmin.ServiceManLib.***1309;.***1353;(ProcessController ***1264;, Int32 ***1320;, String ***1354;, String ***1356;)

at TCAdmin.ServiceManLib.WindowsProcessController.Start2()

 

System.InvalidOperationException: Cannot process request because the process (PID) has exited.

at System.Diagnostics.ProcessManager.OpenProcess(Int32 processId, Int32 access, Boolean throwIfExited)

at System.Diagnostics.Process.GetProcessHandle(Int32 access, Boolean throwIfExited)

at System.Diagnostics.Process.set_PriorityClass(ProcessPriorityClass value)

at TCAdmin.ServiceManLib.WindowsProcessController.SetAffinityAndPriority()

[Dennis]

Could you try reinstalling TCAdmin? It could seem like either like the TCAGame or TCAdmin user doesn't have the correct permissions.

[gijs007]

Nah the permissions are fine. Reinstalling won't help.

These errors appear in the log when the TCadmin Service Manger becomes unstable.

 

After analysing the issue I found that one of the services was constantly being restarted (due to a configuration error). This results in a lot of files being generated in the service's subdirectory located in: C:\Program Files\TCAdmin2\Monitor\Services\Launchers

 

Every time the game server is started TCadmin generates three files:

One batch file to execute the custom script.

One batch file to launch the other batch file..

And also a .txt file which serves as a log by keeping track of the output from the batch file.

 

I'm not sure why this is, but it seems like a very silly and inefficient design.

These scripts don't or rarely change. Why would we need many identical files with different file names for each time the game server process is started?

And why does TCadmin store this temporary data to disk? We can use RAM for this kind of data.

 

To make this worse, the TCadmin Service Manager reads all of these files when starting a game server. Even though it doesn't actually need to use all of these files to start the game server.

In my case this resulted in 30.000+ file operations in less than one minute, as can be seen in the screenshots below. Which cases the OS to run out of permission or filesystem handlers after some time, as such the read request from the TCadmin Service Manager fail with the error message I mentioned earlier.

 

 

Hopefully this silly design will be changed so that it's efficient and doesn't use unnecessary filesystem resources.

 

We can disable auto restarts all together, as workaround. However this is undesirable because it also prevents restarts from legitimate random processes crashes.

What we would need is to have a limit in the amount of times a service can be auto restarted within a given time frame. (E.g. if the game server crashes more than 5 times in 10 minutes it will not be restarted automatically).

Edited March 10, 2018 by gijs007

[mouseymars]

HI,

 

This is on Windows 2016 right?

 

We have a similar issue on 2016.

[gijs007]

That is correct. This is on Windows Server 2016.

 

Note: Luis has added a feature which allows us to specify the max amount of restarts in a given time period. This prevents the constant restart, and so far it looks like our servers are stable again.

[mouseymars]

That is correct. This is on Windows Server 2016.

 

Note: Luis has added a feature which allows us to specify the max amount of restarts in a given time period. This prevents the constant restart, and so far it looks like our servers are stable again.

 

Where is that specified?

[Dennis]

Probably in an unreleased version. Changes will be pushed to the upcoming release, I guess.