Memory leakage ?

[The_Hitman]

As some of you now I run a bunch of servers, I run them all on a rent-machine with windows 2003, the weird thing is that i got some sort of memory leakage on all of them, they are slowly raise up, even no one is on it ?? When it starts up its still at about 40MB but as soon as someone enter it starts raise up and keep doing it even after they leave and could go as high as 200 for no reason...., so I thout it was some mod doing this and created a vanilla-server (no adds) to test, but that one act the same way ?? any one else seen this ?

[UTrustedPlayer]

I'll email you hitty. It's probably in the way the server machine is set up to segregate the different installations.

[The_Hitman]

The machine I rent got 4 cores and i separate the servers with psexec.exe like this for the nali “ psexec.exe -a 0 ucc server” so since that one use the most I have set it to core0 alone.
and it still act this way, even if I only run the nali alone with no other servers running??!! This makes me think it can be some setting in windows or..?


MH-Nali core0
MH core1
BT1 core2
BT2 core3
FunMix1 core1
FunMix2 core2
Insta CTF core3
Sniper core1
US core2

[UTrustedPlayer]

How much memory does the server have? If it's forcing itself to go to page file often due to low RAM availability that could be part of the problem.

Edit- Just looked at your list closely and that's a lot to ask from a server. 9 installs running on one server, you'd better have some serious RAM installed as well as a fast HDD. This really feels like a hardware issue.

If you want you can upload the server setup to my home server and fire it up there. Password it, let it run and see if you get the same leakage. That would isolate the windows versus hardware issue at least.

[The_Hitman]

Its a rent-machine at gaming deluxe, a Dell Poweredge R200 quad-core with 2 gig of ram and shows 1,28gig free when the servers have about 20-25 players
spitted up on the servers so it dont use that much over all.
2 hard-drives at 70/232 Gb. I used Microsofts freeware process explorer to watch this, maybe you can try it to see if it act the same if you on one of yours.http://technet.microsoft.com/en-us/s.../bb896653.aspx
Stable at start-enter for min and leave and it start climb up slowly if I watch it in processXP.

[Feralidragon]

Just one thing I must note: it's perfectly normal seeing UT consuming more and more memory over time, specially during an active game since UT garbage collector only purges on map change.

For example: if you fire a rocket, that rocket occupies memory, even after it explodes (disappears/gets destroyed) and even if it's garbage collected (it's like the memory recycle bin, if I can put it that way) it will still remain in memory nonetheless.

So, if you or "something" in the server keeps spawning or creating new instances of something (like a rocket for example), this will add up in memory, thus increasing the memory usage over time.

At the beginning everything is stopped and nothing gets spawned/instantiated but the initial stuff, but once someone joins and starts playing, plenty of things get spawned, either from him/her or bots/monsters or even triggered/scripted stuff in the game.
And even if the player leaves the game, if other things are supposed to keep the game going nonetheless, things will still get spawned and thus raising memory usage (for example, in a MH server, if monsters start to fight each others, which happens plenty of times, even if the player leaves, they will keep fighting as the server is still processing all this, it doesn't stop).

Now, of course, there are things that may rise memory usage too damn much (like mods which spawn things recklessly and in high numbers), so to understand it better, you mentioned a 40Mb to a 200Mb leap: how much of a time frame are we exactly talking about and which gametype/maps?
Does it go back to 40Mb (more or less) on map change?

[SC]-[LO]\[G_{HoF}]

Originally Posted by The_Hitman

As some of you now I run a bunch of servers, I run them all on a rent-machine with windows 2003, the weird thing is that i got some sort of memory leakage on all of them, they are slowly raise up, even no one is on it ?? When it starts up its still at about 40MB but as soon as someone enter it starts raise up and keep doing it even after they leave and could go as high as 200 for no reason...., so I thout it was some mod doing this and created a vanilla-server (no adds) to test, but that one act the same way ?? any one else seen this ?

Are you talking about memory or talking about ping when you refer to 200. 2gb of ram isn't much for that many servers you run on that machine. Also what another thing that really concerns me is that your big issue might not be the machine but the bandwidth instead.

[UTrustedPlayer]

Hitty and I have talked on this issue at length and I've highlighted all this to him multiple times. Even though UT itself will run on a toaster at 800X600, the server is a different story. The engine wants a lot of room to breathe and having multiple servers to a core is a bad idea, especially since all these servers share the RAM that a low-end laptop has now. It's not surprising that bad things happen given the environment these servers are having to share.

My advice is the same as it was 3 years ago: scale back or get another server at another location to share the load.

[SC]-[LO]\[G_{HoF}]

Yeah listen to UTrusted FFS. Get a new server bro.

[SoNY_scarface]

From experience I'd agree with what's been said. I allocate 500mb ram per server to be on the safe side (at least) and that's on Linux. If you will use page ram then you should think about getting a faster disk 10k or ssd. This is a server after all not a client. Currently siege is bursting to 2.5gb, combo upto 1gb and BT averages 400mb. I guess you can use that as a benchmark. I would assume MH would be more resource hungry if it was available....

You should also consider collocation. It'd be much cheaper than renting in the long run and you'd probably pick up a pretty good server for 600£

Our box two years ago was a Core i7 920 and even that lagged with 6 ut servers. On out new box it's a complete different story though. You should think about virtualisation as well which brings so much more performance....

I hope (for security reasons) that the servers root ip is not the gamserver ip addresses. If so you should purchase a second ip and not disclose that to anybody. If you are unfortunate to be under DDoS you can just disable your gamserver ip address and your server will no longer be under attack consume traffic etc. On the flip side you could close all ssh FTP etc ports and just leave gameserver ports open while on the other ip you have all your other services running but close them with a firewall using only rule based whitelist entry for yourself.

[The_Hitman]

Right now I am most in to Feralidragons thouts since as I said it act like this even with only 1 single vanilla-server running with just 1 map and I have the exact same ping/packet-loss as when I enter when I got all 8 running.
I think I am going to do some more study of windows2003 behavior and do a
mem-test on the machine just in case.

[UTrustedPlayer]

I hope this doesn't come across as harsh as it seems in print but I think you are choosing Ferali because he is telling you closest to what you want to hear. Make no mistake, if there is a coding/development/engine question I default to Ferali's response every time. He doesn't run servers though, and this is something you learn more about by doing than theory.

You have Schlong, Sony and myself telling you the same thing. This is coming from guys who have set up, troubleshot, and ran more servers than 90% of the people in the game (not to mention they run two of the perpetual top 10 servers). Sony's contribution alone would make me stop and listen because I know what that guy has been through. If he gives advice on how to configure a server or what it needs to operate correctly you should listen intently. He's been to battle over his stuff and the fact his servers carry on despite this tells me he knows what he's talking about.

One server per core, double your RAM (at least), and find out the number of allowable connections through your ISP. These three issues will cause all kinds of problems for you until you address them each.

[The_Hitman]

Of course my friend, I will look in to all that was said here to and I appreciate all ideas, thats what forums are for, I just believe that was closer to my problem specific and the ram-usage goes slightly down when a map shifts, just weird that its still high when its emty, all 8 servers together with windows only use 0,8 of the total 2,0 gig of ram and most same bunch of ppl that jump around so it would be different if I had 20 players on each server LOL
so I will do some diff tests and let you now if I find something...

[2399Skillz]

Well that is a lot of servers, but I've seen more run on less.

As for the network issue, he is renting the server from a server rental company that has multiple Tier 1 providers. Doubt he will have an issue with number of connections.

[Feralidragon]

Please don't misunderstand my post above, it was just a plain explanation in "why" memory usage rises and keeps rising over time in a UT server, I didn't provide a solution nor anything along those lines.
As said above, I never hosted a UT server of my own so there are things which are out of my domain there.

And btw, as for a single UT server consuming 800Mb, 1.5Gb and over 2Gb during a single game, that may be avoided (look here: http://www.unrealadmin.org/forums/sh...ad.php?t=30865).
Basically, if you purge things mid game, you will certainly lower the memory usage, and if the mod is optimized for garbage collection (by dereferencing stuff after destroy) then purging shall be even more effective and lower a whole lot more the memory usage.
Unfortunately however I have the impression there is no released mod that make this optimization though, but having something to purge mid-game is possible (I am not sure what the outcome would be though, afaik it was never tried before?).

[UTrustedPlayer]

Years ago there was a rather intricate discussion on garbage collection in the Epic forums (I think). I'll see if I can find it. I didn't understand most of it so that probably means it's something you would find very useful.

Skillz I know that some groups can exceed the >1 server per core without a problem but it's generally been lightly modded stuff. Servers running NW2 and MH need so much more than those, especially when they are busy as Hitty's.

[qwerty]

Originally Posted by Feralidragon

Please don't misunderstand my post above, it was just a plain explanation in "why" memory usage rises and keeps rising over time in a UT server, I didn't provide a solution nor anything along those lines.
As said above, I never hosted a UT server of my own so there are things which are out of my domain there.

And btw, as for a single UT server consuming 800Mb, 1.5Gb and over 2Gb during a single game, that may be avoided (look here: http://www.unrealadmin.org/forums/sh...ad.php?t=30865).
Basically, if you purge things mid game, you will certainly lower the memory usage, and if the mod is optimized for garbage collection (by dereferencing stuff after destroy) then purging shall be even more effective and lower a whole lot more the memory usage.
Unfortunately however I have the impression there is no released mod that make this optimization though, but having something to purge mid-game is possible (I am not sure what the outcome would be though, afaik it was never tried before?).

your points are the only ones that make sense. Mh will eat up tons of ram due to monsters spawning etc.

its has been long known that ut has memory leaks in windows server envoirments. Server 2008 seems to manage ram better than 2003 in my experience.

restarting the ut service is highly recomended at least once a week.