Speeding up the setup/provisioning process

[gijs007]

The setup process of a game server is currently relatively slow:

 

27/03/2017 12:28:18	Using API settings for user id 1. Servers: * Selection: LessSlots
27/03/2017 12:28:18	Checking server... id: 1 name: Amsterdam, The Netherlands
27/03/2017 12:28:18	Server is in different datacenter.
27/03/2017 12:28:18	Checking server... id: 10 name: Dallas, Texas
27/03/2017 12:28:18	Server is in different datacenter.
27/03/2017 12:28:18	Checking server... id: 8 name: London, United Kingdom
27/03/2017 12:28:18	Assetto Corsa (115) is selected in the server's API configuration. Selected games: *
27/03/2017 12:28:18	Server is available: id: 8 name: London, United Kingdom Slots: 1088 of 0 Services: 60 of 0
27/03/2017 12:28:19	Checking server... id: 9 name: Los Angeles, California
27/03/2017 12:28:19	Server is in different datacenter.
27/03/2017 12:28:19	Checking server... id: 6 name: Montreal, Quebec
27/03/2017 12:28:20	Server is in different datacenter.
27/03/2017 12:28:20	Checking server... id: 5 name: Roubaix, France
27/03/2017 12:28:20	Server is in different datacenter.
27/03/2017 12:28:20	Checking server... id: 4 name: Washington D.C.
27/03/2017 12:28:20	Server is in different datacenter.
27/03/2017 12:28:20	Number of servers available: 1 Selection method: LessSlots
27/03/2017 12:28:20	Checking for files and ports on server Id: 8 name: London, United Kingdom
27/03/2017 12:28:31	Selected server: id: 8 name: London, United Kingdom This task: 34108 This step: 1 Updating task step: 2

It appears that the database search is looking for all datacenters which match the ID. And continues looking after it already found a match..

 

This doesn't make sense, as each datacenter has a unique ID. There is no reason to look for other matches.

I think this function can perform better by changing the SQL query, so that it only selects servers with a certain datacenter ID.

 

Also, we only have one server. Selecting this server from the list of servers in this datacenter shouldn't take more than a second.

 

---

I think the server selection method could be improved by preparing some data to speed up the provisioning process.

We can decide which server should be used to provision the next service, before we get to the part where the next service is actually being provisioned.

 

For example in the case of round robin: after creating a service we can update the database to use server X next time. This can then simply be looked up quickly when the provisioning process is started.

When using slot/service counts for load balancing, we can also calculate the total slots/services on a machine (e.g. after creating or deleting a service). During provisioning we only need to lookup which machine has the fewest slots / services used.

 

 

27/03/2017 12:28:39 Searching for an ip that has the required ports

27/03/2017 12:31:03 Selected ip x.x.x.142 and port 9796

Seems like TCAdmin checks all ports manually to see if they are used, as I can't possibly see why this would take 2 minutes and 30 seconds.

 

Idea 1:

This can be optimized by using netstat to find all the used ports on a system.

 

These ports can now be excluded from the selection.

Now simply select a port (range) that is available and matches the criteria.

Finally do a check to verify these ports are indeed free and then proceed.

 

This shouldn't take more than 10 seconds on a modern server.

 

Idea 2:

We already know which ports are in use by TCAdmin, this information could be stored on the local server and the used ports can be excluded from the existing process.

 

Idea 3: When idle, check which port (ranges) are free for each template game/voice server.

These free port ranges can be used during provisioning later.

The preselected ports should be verified when provisioning the server, as an application could theoretically have started using this port between the last check and the provisioning process.

Edited December 15, 2018 by gijs007

[Dennis]

It appears that the database search is looking for all datacenters with the same ID. And continues looking after it found a match..

This take SECONDS, you won't earn anything from "optimizing" this at all. Checking if the files are present is what takes a long time. When we provision a Minecraft service, the setup happens within 6 seconds (WHMCS -> TCAdmin -> Select one out of 6 servers -> Setup). TCAdmin started setup of Minecraft at 13:00:41 and finished at 13:00:47.

On our setup, finding the server on which to install the service usually takes 1 second.

Finding the IP didn't even take a second.

[gijs007]

This take SECONDS, you won't earn anything from "optimizing" this at all. Checking if the files are present is what takes a long time. When we provision a Minecraft service, the setup happens within 6 seconds (WHMCS -> TCAdmin -> Select one out of 6 servers -> Setup). TCAdmin started setup of Minecraft at 13:00:41 and finished at 13:00:47.

On our setup, finding the server on which to install the service usually takes 1 second.

Finding the IP didn't even take a second.

It takes over 10 seconds, as you can see in the earlier quoted debug log.

As we start adding more locations this will only take longer and longer.

 

I agree with you that the end user won't notice the difference with <10 locations, but my point was that this process is illogical, very inefficient and could be much faster with a few changes.

 

On a side note:

I think there is a lot of inefficient code in TCadmin v2, as I often see CPU spikes on my servers, caused by the TCadmin monitor using one entire core for no good reason.

I do get comments from people that they are frustrated about the TCadmin website responding relatively slowly.

 

Hence why I address such issues, as I hope it will change the focus on improving the code quality of TCadmin in general and thus improve the overall performance and stability.

[ECF]

The slowness is due to the data transfer from 10 servers that are collecting player count CPU, RAM and disk space. Everyone wants all this info to display to the client. All this data needs to be sent back to the master server to be stored in the database.

 

Take 10 dedicated servers and multiply say 20 game servers on each server. That is 200 games that have to send their CPU, RAM and disk usage back to the master server.

 

Now factor in that most hosters want to display as much historical data as they can such as players, RAM, CPU over a month etc...

 

Then factor in the storage into the database from each of these data files which takes CPU as well.

 

Now we need to factor in all the log files that people wanted for security, such as website access, database access and commands that are run.

 

The code is fine, the data that hosters want to keep and display will cause a huge slowdown if there is gigs of it being sent back to the master.

 

Luckily we added the option to set the amount of data to save and store. But no one wants to lose their precious Player count, RAM and CPU graphs for a entire month. That would just be silly talk

 

A simple server snapshot query of what the server is doing at the moment would be fine in my opinion. But everyone is all about the historical stats.

[gijs007]

The slowness is due to the data transfer from 10 servers that are collecting player count CPU, RAM and disk space. Everyone wants all this info to display to the client. All this data needs to be sent back to the master server to be stored in the database.

 

Take 10 dedicated servers and multiply say 20 game servers on each server. That is 200 games that have to send their CPU, RAM and disk usage back to the master server.

 

Now factor in that most hosters want to display as much historical data as they can such as players, RAM, CPU over a month etc...

 

Then factor in the storage into the database from each of these data files which takes CPU as well.

 

Now we need to factor in all the log files that people wanted for security, such as website access, database access and commands that are run.

 

The code is fine, the data that hosters want to keep and display will cause a huge slowdown if there is gigs of it being sent back to the master.

 

Luckily we added the option to set the amount of data to save and store. But no one wants to lose their precious Player count, RAM and CPU graphs for a entire month. That would just be silly talk

 

A simple server snapshot query of what the server is doing at the moment would be fine in my opinion. But everyone is all about the historical stats.

So if I disable: "Enable activity and resource usage stats" (for each game template), the historic stats are disabled? And this would reduce the monitor's CPU usage on the master?