Debugging performance issues & memory usage on Linux server?

[dizzy]

I run several game servers from a Linux (Red Hat) dedicated server with very good hardware - plenty of RAM, CPU cores, SSDs, etc. however occasionally players are experiencing issues, especially when the servers are quite busy (i.e. 8+ players at a time).

My setup is fairly basic (no heavy mods except a MapVote and Nexgen server controller) and I don't use NewNet or any modified engine code.

Issues I'm seeing include:
- Nexgen "log in" time increasing dramatically (30 seconds+) as server fills up
- All players' ping increasing slightly as server fills up
- Memory usage increasing to 100% if ucc-bin isn't restarted regularly

So: does anyone have any novel ways to debug these problems and see what's causing them, without actually needing 8-10 players to test it out? Can I just fire up 8 instances of UT and connect to the server to emulate stress testing, or are there more advanced tools to do this kind of thing?

Secondly, are there any relevant tools/mods which you'd say are essential for your server? For example, I've been reading about a basic Garbage Collection script, NewNet, and XC_Engine -- are those considered modern essentials yet?

Finally, are there any forum threads or web pages out there which act as a "best practices" guide for Linux servers? If not, I might just make one once I clear all this up.

Cheers.