Servers crashing and restarting 2x

[TroyT]

Looking for some insight here. We're having issues with Arma 3 servers crashing and not shutting down correctly. I was originally looking to this as a mod issue, but now I'm not so sure.

 

If no players are connected, the server will run its 4-hour course without issue but when it receives the #shutdown command from infiStar it doesn't shut down as much as it nosedives.

 

This is what we're used to seeing from our RPT on our previous GSP's TCAdmin servers:

22:00:13 Successfull attempt to execute serverCommand '#unlock' by server.
22:00:13 Successfull attempt to execute serverCommand '#shutdown' by server.
22:00:14 EPE manager release (299|595|0)
22:00:14 Number of actors in scene after release: 288
22:00:14 EPE manager release (0|288|0)
22:00:15 Deinitialized shape [Class: "C_Soldier_VR_F"; Shape: "a3\characters_f_bootcamp\common\vr_soldier_f.p3d";]
22:00:15 Deinitialized shape [Class: "B_CTRG_Soldier_2_F"; Shape: "a3\characters_f_exp\blufor\b_ctrg_soldier_02_f.p3d";]
22:00:15 Deinitialized shape [Class: "B_GEN_Soldier_F"; Shape: "a3\characters_f_exp\gendarmerie\b_gen_soldier_f.p3d";]
22:00:15 Deinitialized shape [Class: "I_C_Soldier_Bandit_4_F"; Shape: "a3\characters_f_exp\syndikat\i_c_soldier_bandit_4_f.p3d";]

*** About 300 timestamped lines of this... ***

22:00:15 Class LaserTarget destroyed with lock count 2
22:00:15 Class SuppressTarget destroyed with lock count 1
22:00:16 Extensions:
22:00:16   armalog (@infiSTAR_Exile\armalog.dll) [K]

22:00:16   updatearmalog (@infiSTAR_Exile\updatearmalog.dll) [***65533;***65533;]

22:00:16   ArmAdmin (@ArmAdmin\ArmAdmin.dll) [0]

22:00:16   extDB2 (@ExileServer\extDB2.dll) [@***65533;***65533;]

 

But we're getting this with our own servers:

2:00:13 Successfull attempt to execute serverCommand '#shutdown' by server.
2:00:13 Successfull attempt to execute serverCommand '#unlock' by server.
2:00:14 EPE manager release (262|629|0)
2:00:14 Number of actors in scene after release: 262
2:00:14 EPE manager release (0|262|0)
2:00:20 Deinitialized shape [Class: "C_Soldier_VR_F"; Shape: "a3\characters_f_bootcamp\common\vr_soldier_f.p3d";]
=======================================================
-------------------------------------------------------
Exception code: C0000005 ACCESS_VIOLATION at C0CEED88
cpu:  Intel(R) Xeon(R) CPU E5-1630 v4 @ 3.70GHz
graphics:  No
resolution:  160x120x32
Addons:
 rhssaf_editorpreviews in rhssaf\addons\rhssaf_editorpreviews\
 A3_Structures_F_Argo_Signs_City in a3\structures_f_argo\signs\city\
 rhs_bmp_camo in rhsafrf\addons\rhs_bmp_camo\

*** About 1,500 lines of this... ***

 A3_Structures_F_EPA_Civ_Constructions in a3\structures_f_epa\civ\constructions\
 A3_Weapons_F_EPB in a3\weapons_f_epb\
 A3_Weapons_F_Gamma_Rifles_MX in a3\weapons_f_gamma\rifles\mx\
Mods: @ExileServer;@A3XAI;@infiSTAR_Exile;@Adv_Rap;@Adv_Urban_Rap;A3;curator;kart;heli;mark;expansion;jets;argo;orange;tacops;tank;dlcbundle;dlcbundle2;@Exile;@RZ;@RZI;@RHSAFRF;@BSF;@RHSGREF;@RHSUSAF;@RHSSAF;@EBM;@EM;@MAS_Veh;@PI_AIO;@Fox_SC;@Max_Women
ServerMods: @Adv_Urban_Rap;@Adv_Rap;@infiSTAR_Exile;@A3XAI;@ExileServer
Version 1.92.145639
Fault time: 2019/06/09 02:00:20
Fault address:  C0CEED88 00:C0CEED88 Unknown module
file:     Exile (__cur_mp)
world:    Altis
Prev. code bytes: 70 ED CE C0 F6 7F 00 00 00 00 00 00 00 00 00 00
Fault code bytes: 88 ED CE C0 F6 7F 00 00 88 ED CE C0 F6 7F 00 00
=======================================================
note: Minidump has been generated into the file C:\TCAFiles\Users\troyuser\3\A3Master\arma3server_x64_2019-06-09_01-49-58.mdmp

 

Now the thing that makes me question whether it's mod or configuration related is that when there's a crash the server restarts twice. These are 3 consecutive RPT samples, notice the timestamps:

 

9:14:16 BEServer: registering a new player #733419447
...
9:20:52 rhsafrf\addons\rhs_tigr\sts_proxies\sts_ags_feedtray.p3d: No geometry and no visual shape
9:20:52 Duplicate HitPoint name 'HitGun' in 'O_HMG_01_high_F'
9:20:52 Duplicate HitPoint name 'HitTurret' in 'O_HMG_01_high_F'
9:20:52 Dimensions in class O_HMG_01_high_F should be an array of size 2.
9:21:16 "ExileZ Mod: Monitored Zombies	|	2	"
9:21:46 "ExileZ Mod: Monitored Zombies	|	2	"
9:22:17 "ExileZ Mod: Monitored Zombies	|	2	"
9:22:47 "ExileZ Mod: Monitored Zombies	|	2	"

 

*** CRASH

 

=====================================================================
== C:\TCAFiles\Users\troyuser\3\arma3server_x64.exe
== "C:\TCAFiles\Users\troyuser\3\arma3server_x64.exe" -ip=151.80.19.11 -port=2322 -profiles=C:\TCAFiles\Users\troyuser\3\A3Master -world=empty -loadMissionToMemory -bepath=C:\TCAFiles\Users\troyuser\3\battleye -config=C:\TCAFiles\Users\troyuser\3\@ExileServer\exileconfig.cfg -cfg=C:\TCAFiles\Users\troyuser\3\@ExileServer\basic.cfg -mod=@Exile;@RZ;@RZI;@RHSAFRF;@BSF;@RHSGREF;@RHSUSAF;@RHSSAF;@EBM;@EM;@MAS_Veh;@PI_AIO;@Fox_SC;@Max_Women; -servermod=@ExileServer;@A3XAI;@infiSTAR_Exile;@Adv_Rap;@Adv_Urban_Rap; -autoinit -nosound -nosplash -maxmem=6142

Original output filename: Arma3Retail_Server_x64
Exe timestamp: 2019/04/26 12:56:39
Current time:  2019/06/09 09:22:54

Type: Public
Build: Stable
Version: 1.92.145639

Allocator: C:\TCAFiles\Users\troyuser\3\Dll\tbb4malloc_bi_x64.dll [2017.0.0.0] [2017.0.0.0]
PhysMem: 64 GiB, VirtMem : 131072 GiB, AvailPhys : 60 GiB, AvailVirt : 131072 GiB, AvailPage : 63 GiB
=====================================================================

9:22:54 SteamAPI initialization failed. Steam features won't be accessible!
9:22:55 Initializing stats manager.
9:22:55 Stats config disabled.
9:22:55 sessionID: db571ab013c5718e3f83b23d7a6039b421b2b323

 

*** CRASH or RESTART

 

=====================================================================
== C:\TCAFiles\Users\troyuser\3\arma3server_x64.exe
== "C:\TCAFiles\Users\troyuser\3\arma3server_x64.exe" -ip=151.80.19.11 -port=2322 -profiles=C:\TCAFiles\Users\troyuser\3\A3Master -world=empty -loadMissionToMemory -bepath=C:\TCAFiles\Users\troyuser\3\battleye -config=C:\TCAFiles\Users\troyuser\3\@ExileServer\exileconfig.cfg -cfg=C:\TCAFiles\Users\troyuser\3\@ExileServer\basic.cfg -mod=@Exile;@RZ;@RZI;@RHSAFRF;@BSF;@RHSGREF;@RHSUSAF;@RHSSAF;@EBM;@EM;@MAS_Veh;@PI_AIO;@Fox_SC;@Max_Women; -servermod=@ExileServer;@A3XAI;@infiSTAR_Exile;@Adv_Rap;@Adv_Urban_Rap; -autoinit -nosound -nosplash -maxmem=6142

Original output filename: Arma3Retail_Server_x64
Exe timestamp: 2019/04/26 12:56:39
Current time:  2019/06/09 09:22:59

Type: Public
Build: Stable
Version: 1.92.145639

Allocator: C:\TCAFiles\Users\troyuser\3\Dll\tbb4malloc_bi_x64.dll [2017.0.0.0] [2017.0.0.0]
PhysMem: 64 GiB, VirtMem : 131072 GiB, AvailPhys : 60 GiB, AvailVirt : 131072 GiB, AvailPage : 63 GiB
=====================================================================

9:22:59 SteamAPI initialization failed. Steam features won't be accessible!
9:23:00 Initializing stats manager.
9:23:00 Stats config disabled.
9:23:00 sessionID: a80e25ae96bc1cb88e18cb5c2dc87230c6adae7b
9:23:07 Unsupported language English in stringtable
9:23:07 Unsupported language English in stringtable
9:23:07 Unsupported language English in stringtable
9:23:12 Updating base class None->, by Shemagh\config.bin/CfgGlasses/TRYK_headset_glasses/ (original ESSGoggles\config.bin)

 

It's the intermediate restart that raises my doubts about it being mod or mission related.

 

The current server that this is happening on is a temporary remote box with about 250 Mbps bandwidth. We will be installing permanently on a new server with 1Gbps bandwidth. I've tried tweaking the bandwidth setting for the game instance but haven't seen any change.

 

We've changed the server configuration to utilize the 64-bit exe and are running EXTDB3 instead of the 32-bit with EXTDB2. Other than that, these servers are exactly as they were with the old GSP.

 

Also, the TCAdmin database is cohabitating with the game server's database if that matters.