Server Lags out

[donjski]

I'm running a Gigabyte K7 Triton with nForce chipset (GA-7N400-L)AMD XP2500+ with 1gb Ram and 7200 rpm harddrive on a dedicated T1 line. Using RedHat 9.0 kernel 2.4.20-18.9 UT 436 patch.

I can get 8 players on and the pings stay pretty good. 9 or more the pings start to sky rocket.

Does anyone know why? How many players should I be able to get on with this machine.

When I have 8 peeps on, the cpu is 99% IDLE according to 'top'. And my memory is 806mb free of the 1 gb.

any Ideas?

[Rush]

Maybe you should try to set your MaxClientNetspeeed on a lower value....

[50BMG]

I second Rush's suggestion as being most likely.

You need to tell us about your server's internet connection and what the MaxClientRate is before you change it, to get more assistance on this.

The usual thing with this problem isn't server (CPU) load, but available bandwidth from your server to the internet.

You should actually TEST your server's connection on a site like www.dslreports.com or www.speedguide.net. Results from tools on these sites will provide you with actual connection capacities, both to and from the internet. We'll be able to better assist you if you already know these figures.

In less common cases,we've seen instances where equipment between the server and the inernet connection introduced unintended limits on bandwidth. Devices like less expensive firewalls or routers. This kind of thing could get important if the connection is not the limiting factor.

In general, a full T1 (1.5mb up/down) would be good for about 16 players, just to give you an idea.

[donjski]

MaxClientRate=5500

NetServerMaxTickRate=64

The T1 averages 1.2 up and down

Do you know how to do a bandwidth test via the commandline, maybe using Lynx?

This machine has no Xwindows

[echo]

Originally Posted by donjski

MaxClientRate=5500

NetServerMaxTickRate=64

The T1 averages 1.2 up and down

Do you know how to do a bandwidth test via the commandline, maybe using Lynx?

This machine has no Xwindows

The MaxClientRate looks good but I think the tickrate is set to high... In my opinion a tickrate around 30 would be much better.

[ouch]

Originally Posted by donjski

MaxClientRate=5500
NetServerMaxTickRate=64

There's your problem. The server is trying to update the clients 64 times a second, but with a 5500 bandwidth cap your server is having to drop a bunch of stuff, creating artificial lag in the process. Try 7500/30 and see how she runs.

[50BMG]

"Ditto" on the Tic rate .... it's more than double what it should be.

Can I ask how you got the "Averages 1.2 up/down" if you haven't measured it?

I am not sure you can get the numbers we need without an internet browser. The measurement tools are usually java applets that get the co-operation of another internet host to do the test. You don't need to run the test from the server... any computer on that connection would do, as long as it's connected at the same level as the server.

Anyway, I think you've found the problem already.

Have you read TNSe's network tutorial?[list:97519a2cc6]http://www.unrealadmin.org/modules.p...amp;a rtid=15[/list:u:97519a2cc6]

Invest some time, it will help.

[donjski]

Thanks,
Yes, I did read the tutorial. BUt after reading it I was a little more lost than when I started. Had he give some examples, it would have been a big help. What I mean about examples is:

If you have so much bandwidth, use this much clientrate...etc
I think he did this, but in a way that I was confused.

I did test the connection from another computer that is on the same connection, that is how I got the 1.2 up/down. But for some reason I think that my firewall might be interfering. Everyonce and a while, no matter how many peeps are on, everyone will get booted. Looks like a big lag spike. Then after a min or so you can reconnect and continu play. This firewall is on the server itself. That is why I want to try some type of command line test.

Thanks for all the help guys,

Try it out and tell me what you think.
66.84.149.84:7777

Much appreciated.

[Naked_Ape]

Originally Posted by donjski

... But for some reason I think that my firewall might be interfering. ... This firewall is on the server itself.

Just a little heads up: The kind of software "personla firewalls" that are so prolific in the Windows world today (e.g. ZoneAlarm) are usually not suited to running a server behind them. They have less than stellar implementations for UDP state tracking which is detrimental to game data connections and they can't handle the storm of many small packages very well in my experience (read: never could server a game properly behind ZA). Another problem is that since they are heavyhanded with the typical gameserver traffic they consume quite a lot of CPU cycles which detract from your servers performance.

This is all IMO and YMMV but you might want to consider another firewall. I'm not to familiar with the offerings on the Windows side but WinRoute seems to be popular (I believe they have a try-before-you-buy version).

[donjski]

Isn't this the LINUX Specific section?

[50BMG]

donjski Wrote: But for some reason I think that my firewall might be interfering. Everyonce and a while, no matter how many peeps are on, everyone will get booted. Looks like a big lag spike. Then after a min or so you can reconnect and continu play. This firewall is on the server itself. That is why I want to try some type of command line test.

Ok, I think I can offer some concrete suggestions.

First, let's look at that 1.2mbit result....

Sounds like the way you got it is good enough to conclude the following:
[list:5bd076421f]1.2mb (MegaBits) = 150000 Bytes Per Second

This represents the absolute maximum data per second that can be sent to the internet. The way you measured it, this figure already includes some of the transmission overhead that usually occurs. We'll add an additional fudge factor at the end to account for the rest.[/list:u:5bd076421f]
You say:

I can get 8 players on and the pings stay pretty good. 9 or more the pings start to sky rocket.

This 8-9 player boundry is interesting.
[list:5bd076421f]150000 / 8 Players = 18750 MaxClientRate

This is certainly high enough that your server's connection could not be the problem, especially with your MaxClientRate set at 5500.[/list:u:5bd076421f]
Now let's factor in the effect of your 64 Tic Rate..
[list:5bd076421f]18750 Bytes Per Second / 64 Tics = 292 Bytes Per tic

This too, is a good number unless it goes any lower. Here's your limiting factor.[/list:u:5bd076421f]However, with a MaxClientRate of 5500 this is nearly unworkable.
[list:5bd076421f]5500 Bytes Per Second / 64 Tics = 85 Bytes Per Tic

85 Bytes per tic isn't enough. Not by a factor of 3. The miracle here is that you could play it this way at all. I wouldn't have predicted that LAG would be the symptom. I would have expected phantom kills and missing sounds.[/list:u:5bd076421f]So, without further testing, this is your remedy:
[list:5bd076421f]If you lower your Tic Rate to something more reasonable, say 30, this will have the effect of moving this impass up to around 16 players. If your 1.2mb is correct, you could then increase the MaxClientRate to 7000 or so. This would be a good combination.[/list:u:5bd076421f]
Now let's consider that firewall.
[list:5bd076421f]Certainly, running the firewall on the server will have some impact. The issue is to define a test that will confirm that the firewall is, or is not, the problem.

The most obvious thing I can think of is to drop the firewall during a pre-planned test when many of your friends will be available. This involves some risk, but it would only be needed for a game or so. Intentionally overload the server with players to confirm that the lag still happens.[/list:u:5bd076421f]
Do this test with the settings you have now... 5500 MaxClientRate /64 tic rate. Play part of a round with 8 players and then add 2 or 3 more.
[list:5bd076421f]That should confirm it. If there's no problem then your firewall was causing at least some of the issue.[/list:u:5bd076421f]If the problem still occurs, then reduce the Tic Rate from 64 to 30 and re-test.
[list:5bd076421f]The problem should be gone.[/list:u:5bd076421f]IF it is, then re-enable the firewall and test again.
[list:5bd076421f]If the problem is still gone, then have fun. You're all fixed[/list:u:5bd076421f]If the problem comes back, we'll work the firewall issue.

Does that all make sense?

Lastly, let's consider that LAG that is so bad everyone gets kicked.[list:5bd076421f]I didn't see from your posts if you have a re-direct server. Would you tell us about that?[/list:u:5bd076421f]
Please let us know what you think, and what happens with the test, if you decide to do it. :gulp:

[donjski]

Thanks much, I will make those changes and see what happens.

Redirect? Yes. I have another server running at another remote loacation just for that. It is on a dsl line. Has some traffic, but not a whole lot during the times of game play. Which is usually a night time.

Although, this is a good question, cause I used to use my UT server as the redirect too. I can run Apache for this. Seeing as how it is a T1. I figured that the download times would be fast enough. I don't think that it ever was an issue in the past, except that it just seemed that a lot of the time peeps weren't being rediected, even though all the files were there

[50BMG]

I need to know if the "major LAG" that kicks everyone occurs now that the redirect server is off the T1 that the game server is on?

Having run this configuration myself, I can explain why it usually doesn't work out.

The game server uses only the bandwith allowed by the ut.ini's settings. This usage is well controlled, and can allow for headroom on the T1 line to provide bandwidth for other services running on the line.

However, when using Apache' on the same service (the T1) there is usually no limit on the bandwidth which the Apache' server would consume. When it gets a request for data, the Apache server would send the data as rapidly as possible, consuming at least some, if not all, of the data capacity required for the Game Server. This could result in "Major LAG", just as described above.

If Apache' were to self limit, or another mechanism were to "throttle" Apache's use of the T1, then the situation would be workable. Placing the Reditrect on a line not involved with the Game server is usually the best policy, and eliminates this possibility. Many admins use a personal web page someplace for this purpose.

I don't think that it ever was an issue in the past, except that it just seemed that a lot of the time peeps weren't being rediected, even though all the files were there

There are three things I can think of that might cause this.
[list:c2077320b7]1) If there is an error while getting the file from the redirect server, the client will eventually give up and take the file from the game server. Often, even if the error is momentary, the client takes the file AND ALL SUCCESSIVE FILES from the game server anyway. I'm not certain what type of errors don't cause this, but a missing file on the redirect will definitely cause it.

2) If the redirect server does not exist at the named redirect location or if it is not accessible at that locale. This can be caused by a failure to forward the redirect port in a firewall, a blocked port by the ISP, or an incorrect server spec in the redirect entry in the ini file.

3) the Apache' server has not been configured with sufficient permissions to access or serve the files to the domain requesting them.[/list:u:c2077320b7]

Unless you still are having trouble with this feature, I expect you've solved all this correctly already.

Let us know how those new settings work out.

[Naked_Ape]

Originally Posted by donjski

Isn't this the LINUX Specific section?

DOH!

Hand me a napkin, please. I need to wipe the egg of my face...

[donjski]

Thanks again,

The redirect is on a different server. It hasn't been on this server for quite some time. Sorry if I didn't make that clear above. So it shouldn't be the issue. What I was trying to say is that even when I did have it on this server, the 'Major Lag ' never occured

The booting of players seems to have stopped. I tried with firewall up and down. Worked great both ways, no change in pings either.

The settings? Kool, I was able to get 11 peeps on before the pings went up this time. the settings were 7000/30.

I then changed to 7500/30 and 6500/30 both were laggy without it filling up.

I have since changed it to 7000/35 and it seems to run pretty smooth, but i could only get 10 peeps on to test it.

I just played around with the settings, with no clue what would be better. So hopefully from the info I gave you, we can get this puppy running a little bit better.

I really appreciate all your help. I have been trying to find someone with a clue for somtime now.

[50BMG]

Thanks for the numbers.

Sounds like you should lower the Tic Rate even further.

The 7000 for the MaxClientRate looks pretty good.

I am a bit surprised that you say the firewall makes no difference. I expected that with TicRate = 30 and 11 players causing lag, that removing the firewall would allow that number with no lag.
[list:8abca6dd3b]With a Tic Rate of 30 and 11 players, the firewall is having to process 2 x 30 x 11 = 660 packets per second before it lagged.

With the old settings, 9 players and 64 tics, it was having problems at 2 x 9 x 64 = 1152 packets per second.[/list:u:8abca6dd3b]
These numbers support what you are saying about the firewall having no effect since it is now handling fewer packets than before and the system still lags. :wall:

Normally, when I see this pattern it means that the server is having trouble keeping up. In your case, since the firewall is in the server, I thought that disabling it would ease the problem.

Your processor speed is good, so I guess we should look elsewhere. :crazy:

Could you tell me what the intervening equipment is between the server and the T1 interface?
[list:8abca6dd3b]Particularly I need to know line rates and whether the links are full duplex or not. Are there hubs, or switches in the system? Is there a router? If you can observe the system in operation, look for indications of collisions and the like.[/list:u:8abca6dd3b]
The fact is that you shouldn't have any problem with greater than 11 players. Many servers are run with TicRates of 30, but we need to experiment with lesser numbers if the evidence pushes that way.
[list:8abca6dd3b]Could you please repeat the test of 11 or more players at 30/7000 both with and without the firewall?

Then repeat it with 25/7000 to see how many players it will support with and without firewall.[/list:u:8abca6dd3b]
The only other thing I can think of right now is to have a look at your ini to see if anything pops out.

I've had another thought.
[list:8abca6dd3b]Connect to your server, and enter the following command on the console of your client:

Inject userflag 1

Some stuff will print on the screen. Let it do so for a few seconds. Then enter the following command:

Inject userflag 0

Make a screen snapshot (F9) of the junk it was printing on the console and send it to us.[/list:u:8abca6dd3b]

That'll give us something to chew on for a while! :angry2:

[donjski]

Could you tell me what the intervening equipment is between the server and the T1 interface?

The only thing between the server and the T1 is a T1 router. The router has 6 of its own ports on the back. I am plugged in there. There is only one other line plugged in there with me and the goes to the internal lan.

I'm not sure what you mean by line rates :withstupid: , but the router is 10mb lan side. I did read somewhere that due to the fact that I am behind a router I should change the max mtu of eth0 to 1492 because the router always adds on 8 bits of information to all packets. The other players did notice a good change in game play after doing this, I however didn't see much change at all.

Could you please repeat the test of 11 or more players at 30/7000 both with and without the firewall?

Then repeat it with 25/7000 to see how many players it will support with and without firewall.

I will be happy to do so. Give me sometime though, we have a scrim tonight. Was hard enough to get 12 the first time.

So what I'm hearing from you is 'the higher the tickrate, the more lag?'

Here is the link for output of the inject command.
http://www.donjski.com/screenshots/inject.JPG

Heres my ini :http://www.donjski.com/screenshots/UT-INI

thanks again

[50BMG]

One thing did pop out at me from a quick glance at your ini.

You have:
[list:b0c4f5e3dd][Engine.GameEngine]
CacheSizeMegs=4[/list:u:b0c4f5e3dd]
Usually in the Windows world, we would have:
[list:b0c4f5e3dd][Engine.GameEngine]
CacheSizeMegs=128[/list:u:b0c4f5e3dd]

Also, I take it that you sanitized it a little for those who might want to exploit special knowledge. LOL (If not, we should talk)

The 1492 is a common MTU, and if it feels good, DO IT!

Sounds like an optimal connection scheme, right into the router! Can I ask what kind of router? Some Cisco routers have bandwidth allocation capabilities. If there were any for the port you're plugged into, they could effect your maximum player capacity. (Remember you said you measured the 1.2m from another point in the system)

The only thing to check is that the server's link to the router is operating FULL DUPLEX. Usually there are status lights on the router interface. You could plug in a FDX capable switch with indicators between the server and the router to see how it comes up. (This is a trick I've used when no other method was available) If you do this be sure to look at the status of both the router and server links.

I'll digest the rest of the info and let you know what else I see.

Oh, and regarding the tic rate, we're experimenting to see what's up with your system. A Tic rate of 30 is demanding, but not impossible. We're looking for the point where lowering it doesn't improve the player capacity any more. The fact is, if you're happy with 11 palyers as a limit, we can just leave it be. If you want the full monte, then we need to keep tweaking and testing. Ok?

[50BMG]

One more annoying test to perform.

The next time you experience the LAG induced by too many players, please use the "Inject USERFLAG" command during the round while the LAG and players are present, and send me the screenshot, just as you did before.

I hope I'm in time to get you this evening.

[donjski]

Usually in the Windows world, we would have:

[Engine.GameEngine]
CacheSizeMegs=128

Mine is at the default, never changed it.

Also, I take it that you sanitized it a little for those who might want to exploit special knowledge. LOL (If not, we should talk)

If by getting rid of passwords, login names etc, yes. If not. No clue, we need to talk.

Can I ask what kind of router?

Netopia 5300R

Oh, and regarding the tic rate, we're experimenting to see what's up with your system. A Tic rate of 30 is demanding, but not impossible

I changed it to 7000/25 and it was worse. Seems as though 30 is working best right now. But that CacheSizeMegs=128 is something I think needs to be addressed.

If you want the full monte, then we need to keep tweaking and testing. Ok?

I want the full Monte!

As far as the FULL DUPLEX part, I will have to ask my brother, the owner of it, about it.

Hope you get this tonight too!!