100% Cpu when web is idle

[Duvy]

When nobody uses the web interface, (default 8880 port), the xsp2.exe starts to use 100% of one Cpu.

It goes back to normal, when i load up the site, i dont have to even log in, just visit the page.

I have found the following charts:

 

Linux

Debian 2.6.32-5-amd64

mono 2.10.8-1 amd64 Mono build for TCAdmin

lsof 4.86+dfsg-1 amd64

ia32-libs 1:0.4 amd64, also i-386

libpcap0.8:amd64 1.3.0-1 amd64

dpkg -l |grep schedutils returns nothing, but it is installed from the testing rep. and it says the newest version is installed.

 

To prevent that, i inserted to crontab:

0 * * * * (cd ~/; wget localhost:8880 && rm index.html)

It prevents using one Cpu, as loading the webpage in every hour, but i'm sure theres a more elegant way to solve that.

There are nothing relevant in the log files.

[Bubka3]

That's interesting, possibly a bug. It shouldn't eat up an entire core when idle...

[LFA]

That's the first time I see that. I haven't test it but another client mentioned today that adding this to the mono-config reduces cpu usage:

export MONO_THREADS_PER_CPU=300

[Duvy]

Well, i exported, in the nano config the MONO_THREADS_PER_CPU=300, also using the crontab wget script, but it seems, it just haven't been solved.

 

Only logging in, with a user makes it using idle Cpu.

 

Wgeting the localhost:8880, makes only to use the CPU in "nice", not un user.

 

By the was, this is the process, I'm talking about:

/opt/mono-2.10.8/bin/mono /opt/mono-2.10.8/lib/2.0/xsp2.exe --nostop --verbose --port 8880 --root ../Configure --pidfile /home/tcadmin/Monitor/TCAdminWeb.lock

Also in that time, when nobody is logged in, the interrups, and context switches goes up by 10k.

[LFA]

Try uncommenting the SECURE_WEB_PORT in tcadmin-config and restart the monitor. That will make it use a different web server.

[Duvy]

This solved the thing, the TCAdminPublicWebServer.exe runs at around 5-10% cpu on idle, that is better than 100, but the older was around 0 on idle without the bug, or something

 

As for the older web, heres the console.log:

http://pastebin.com/0THN0Vse

 

Exporting thread limit isn't helped, it's not making new processes or threads, the originally running processes somewhy starts to make interupts, and system_calls really fast.

I tried what export MONO_LOG_LEVEL=debug says, but i couldnt find any logfile, or anything to see.

 

Now Im realy excited,why it does that, I'm goung to dig it further tomorrow, maybe not logging out from the web can couse problem.

[LFA]

The output I see if from mono in debug mode. Don't enable debug mode in your mono_config. It's going to be slower and use more cpu.

 

You can execute these commands on your server to optimize mono.

/opt/mono-2.10.8/bin/mono --aot -O=all /opt/mono-2.10.8/lib/mono/2.0/mscorlib.dll
for i in /opt/mono-2.10.8/lib/mono/gac/*/*/*.dll; do /opt/mono-2.10.8/bin/mono --aot -O=all $i; done
/home/tcadmin/Monitor/serviceman-service restart
/home/tcadmin/Monitor/monitor-service restart

[Duvy]

Dear LFA!

 

Actually I did everything, but it seems just not to work properly, but luckily, i could make that cron script working, as i wanted to in the first place.

 

When the Cpu usage is more than 20% (In heavly load it might be increased), i wrote a wget script, to log in, with post data ( it seems logging in solves the problem, load only the login.aspx not ), and if the Cpu is still above 20%, for the 5th time, it restarts itself.

 

Maybe theres something bad on my server, and also i'm happy with that solution, so you can focus on other developing things.

 

Also if you have free time, and if you are interested in massive debugging , i can give you ssh access.

( I'm relatively new to mono, and never thought to run windows binary on linux, it's like "magic" for me )

Thanks again the answers!

Edited January 31, 2013 by Duvy

[jeeringsole]

Try uncommenting the SECURE_WEB_PORT in tcadmin-config and restart the monitor. That will make it use a different web server.

 

I'm having this same issue and tried your suggestion. Still no dice, eating up 50% cpu causing the CPU usage to max out at 100%

 

I uncommented the secure web port and there is no fix. This is on Server 2008 Web Edition R2. My install has been running fine for almost a year. Any other suggestions?

 

The process doing it is the TCAdminMonitor.exe

[Duvy]

Well, if you need a part solution, here's mine:

In Linux theres a base binary called "wget", basically it downloads something from the net.

I made it, to download a "sending logging in data" from the webserver, and it reacts like i was realy logging in.

It can be found to Windows as well here:

http://users.ugent.be/~bpuype/wget/

If you rename to wget.exe, and copy to C:/Windows, you can open the program just typing wget in from command prompt ( run -> cmd.exe ).

 

Heres the login:

 

wget --user-agent="Mozilla/5.0 (Windows NT 6.2; WOW64; rv:18.0) Gecko/20100101 Firefox/18.0" --keep-session-cookies --save-cookies cookies.txt --post-data "__SCROLLPOSITIONX=0&ctl00_DefaultScriptManager_TSM=%3B%3BSystem.Web.Extensions%2C+Version%3D3.5.0.0%2C+Culture%3Dneutral%2C+PublicKeyToken%3D31bf3856ad364e35%3Aen-US%3A2e279a6f-f959-4904-bdf3-719a47e8c014%3Aea597d4b%3Ab25378d2%3BTelerik.Web.UI%2C+Version%3D2012.3.1016.35%2C+Culture%3Dneutral%2C+PublicKeyToken%3D121fae78165ba3d4%3Aen-US%3A30c1931d-6456-454d-bb0f-48f5b0851c34%3A16e4e7cd%3Af7645509%3Aed16cbdc%3A874f8ea2%3A24ee1bba%3A19620875%3Af46195d3%3A39040b5c%3Af85f9819%3A490a9d4e%3Abd8f85e4&__VIEWSTATE=&__SCROLLPOSITIONY=0&__EVENTTARGET=ctl00%24ContentPlaceHolderMain%24Login1%24ButtonLogin&__VSTATE=4c03f7cd-d676-4abb-9fbb-c821c3a1fce8&__EVENTARGUMENT=&ctl00%24ContentPlaceHolderMain%24Login1%24TextBoxUserId=USERNAME&ctl00%24ContentPlaceHolderMain%24Login1%24TextBoxPassword=PASSWORD&ctl00%24ContentPlaceHolderMain%24Login1%24DropDownListLanguages=en-US&ctl00%24ContentPlaceHolderMain%24Login1%24DropDownListTheme=1%3A00000000-0000-0000-0000-000000000001&ctl00_DefaultToolTipMessages_ClientState=&ctl00_ToolTipManager_ClientState=&ctl00_DefaultDialog_ClientState=&ctl00_GridSettingsDialog_ClientState=&ctl00_FileSelectDialog_ClientState=&ctl00_ConsoleWindow_ClientState=&ctl00_UploadWindow_ClientState=&ctl00_MonitorRestartDialog_ClientState=&ctl00_DefaultWindowManager_ClientState=&__EVENTVALIDATION=qPwpO%2B39G3DYNeHES2eXYJZayoJfgWANuHx8if%2FvPWCC8n0iCLdJ5UuDVi6DN8pWS9xISrlsw1xfCwfpo6YaHJhE0EWuhQ%2FzI%2BX3mQaGusoy0xVBsJcrp%2F5Ybni%2BDxr560vgyoRWIfIk2cSR%2FiRrDBJ9W5%2FIV98K0K71IMmpIoFjSOZxvk6g1KLB6SFdNFfqxuaG489eyU0ke%2B47KRYOZYlKlrFpUdWiByQny2TU%2FGwMABHZYxi9auYUv6P4lkRU6vPtzgQMQgCpYHoifvji4jeUA2ypI%2BuXIx7KTqLL1dlRU3r1akzMEaqNrZe8XtGH8l1D5gW8lahtOPC9NDr8t7NIhJls0K9tD%2FIYpkUhP%2BOZkz%2FPyCUHk61Jvj9XUw5uT76axg%3D%3D" http://DOMAIN:8880/Interface/Base/Login.aspx

 

//END

 

After that just to be sure, if it worked, i download a protected page, if it downloads fine, logging in is succesful, and my Cpu usage goes back to normal:

 

wget --load-cookies cookies.txt http://DOMAIN:8880/Interface/Base/Home.aspx

 

It might be rude for first.

 

I'm not realy familiar with Windows servers, but I'm pretty sure, you can figure it out, to take the 2 lines into a .bat file (Also you can delete Home.apsx, and Login.aspx there), and schedule to run every 10 minutes, or so.

Thats all i could suggest Jeeringsole

Edited February 1, 2013 by Duvy

[Racer]

I am sorry to bring this up again, But If this was fixed in a recent update, It didn't work. If it can be fixed soon, that would be awesome.

[Duvy]

Still I'm having troubles with the Cpu usage, the only solution is to restart the tcadmin, now i have 2 restarts in a day, and still i can find, when it loops one cpu.

It's not only a peak, if i don't restart, it hangs on 100% for days.

1-2 Users are logged in, and doing nearly nothing.

 

Debian 2.6.32-5-amd64

 

I ran the console, and service-man, with debugging true, and there was nothing happening, when the cpu usage went up to 100%, just some sql update, for live stat on one, and affinity settings on the other console.

But around 1 printing per 5 minutes.

 

My only ideia is a kernel upgrade, but that can mess other things up, so that would be the last option.

LFA, could we figure out something in private, if you could look in through ssh, you might be able to find something, i ran out of ideas :/

Edited April 22, 2013 by Duvy

[LFA]

Try commenting/uncommenting "export MONO_MANAGED_WATCHER=true" in your mono-config. You can also try compiling the latest mono and configure tcadmin to use that instead of 2.10.8. Create a support ticket and I can send you the commands.

[Duvy]

Thanks LFA!

 

Meanwhile, i compiled a version of 2.11.3.

Now it seems to be working smoothly.