The Unreal Admins Page - Forums - Automating UTStats Import

The Unreal Admins Page - Forums (https://unrealadmin.org/forums/index.php)

- General Chat (https://unrealadmin.org/forums/forumdisplay.php?f=174)

- - Automating UTStats Import (https://unrealadmin.org/forums/showthread.php?t=10241)

Automating UTStats Import

Here are a few ways to make it automated @ GNU/Linux:

1) UTServer and UTStats @ the same BOX, using Cron

Let's prepare the import.php first, open it and comment the 6th line, it should look now like:

Code:

// Get key from web browser

//$adminkey = $_GET[key];

Then uncomment the 13th line:

Code:

$adminkey = $argv[1];

* don't forget about setting the $akey in the 3rd line to somewhat like "mypassword" :)
*php command may require a full path to the binary for example /usr/local/php5/bin/php

You should try running the script manually first to find out if everything is ok.

Code:

cd /your/utstats/main/path

php import.php "mypassword"

Now you should see some crappy html output and import procedure if there were any logs.

Ok, we're on the good way to make it working. In your /home directory create a script "utstatsimport"

Code:

#!/bin/bash

for file in `ls /your/path/ut-server/Logs/Unreal.ngLog*.log 2> /dev/null`; do

mv $file /your/utstats/main/path/logs > /dev/null

done

cd /your/utstats/main/path

php import.php "mypassword" > /dev/null

*CHANGE THE PATHS TO UTSERVER AND UTSTATS!! don't forget about the password !
.. and make it executable

Code:

chmod +x utstatsimport

*php command may require a full path to the binary for example /usr/local/php5/bin/php

Better check this script on few logs now to avoid searching the mistakes in the future.

Now setting the cron. Create a file called "mycrontab" with this content

Code:

SHELL=/bin/bash

HOME=/home/me



# run-parts

* * * * * ./utstatsimport

*change the home to your own !!!

now the last one is to run a little command in your home:

Code:

crontab ./mycrontab

The script should be runned now about every one minute.
---------
2) UTServer and UTStats @ the same BOX, without Cron(stupid admin)

Use the point's 1 hints to the moment you have to create the utstatsimport script, the script itself has to be different:

Code:

#!/bin/bash

while [ 1 ];do

for file in `ls /your/path/ut-server/Logs/Unreal.ngLog*.log 2> /dev/null`; do

mv $file /your/utstats/main/path/logs > /dev/null

done

cd /your/utstats/main/path

php import.php "mypassword" > /dev/null

sleep 1m

done

*CHANGE THE PATHS TO UTSERVER AND UTSTATS!! don't forget about the password !
*php command may require a full path to the binary for example /usr/local/php5/bin/php

Make it executable

Code:

chmod +x ./utstatsimport

and just run it by

Code:

./utstatsimport &

To stop it use the command

Code:

killall -9 utstatsimport

---------
3) UTServer and UTStats on different boxes .. whatever. :P

I won't describe how to use the ftp stuff in utstats php part here cause I just don't use it, haha. I'll just tell you how to trigger the import.php script.

3a) With cron ....
Create a script in your home, called "utstatsimport" with content:

Code:

#!/bin/bash

lynx -dump "http://utstats.mydomain.com/import.php?key='mypassword'"  > /dev/null

*Change the mypassword to your own !!!
Make it executable with

Code:

chmod +x ./utstatsimport

Now create a file called "mycrontab" also in your home with this content:

Code:

SHELL=/bin/bash

HOME=/home/me



# run-parts

* * * * * ./utstatsimport

*change the home to your own !!!

now the last thing you should do is to run a little command in your home:

Code:

crontab ./mycrontab

3b)If the admin is lame and you cannot use cron ...

Create a script in your home, called "utstatsimport" with content:

Code:

#!/bin/bash

while [ 1 ];do

lynx -dump "http://utstats.mydomain.com/import.php?key='mypassword'"  > /dev/null

sleep 1m

done

*Change the mypassword to your own !!!
Make it executable with

Code:

chmod +x ./utstatsimport

and just run it by

Code:

./utstatsimport &

. To stop it use the command

Code:

killall -9 utstatsimport

Nice work Rush as always :)

I just hope it is clear enough and do not contains any bugs. :P

A little suggestion for those people who don't have access to cron jobs (and/or lame admins :)): WebCron.
You just have to enter "http://utstats.mydomain.com/import.php?key=mypassword" in the URL field. Don't forget to click 'enable' after you created your task.

Quote:

Originally Posted by toa

Nice one :)

hmm I'm having trouble with import.

I had to run webcron. I have my account, and all is well webcron wise. It is enabled, and runs every hour. Problem is, when I go to see RECENT MATCHES, it doesn't appear that they have been processed. I check webcron, and it said all was well. I opened the cron's log. The stats FTPed fine, and then were deleted, as they are supposed to be. Well now for some reason those new games are not getting truely imported. Any suggestions?

I have my cron task pointed at:
http://www.atomicunreal.com/stats/import.php?key=*******
The task is ENBALED and runs fine, no errors :)

EDIT:

I will mention that:
UTStats, when I manually import, changes the current page automatically about 4 times as it imports the tables. Could webcron just not be allowing the tables to be imported?

Also just after a manual import, a small amount of files were FTPed, but many many matches were just imported and were already up there... so I believe they are uploading correctly.

Hey I got it... not sure if it is the "right" way to do it but it works so... :D
In IMPORT.PHP I have changed

Code:

$logdir = opendir('logs');



while (false !== ($filename = readdir($logdir)))

{

//Our (self set) timelimit exceeded => reload the page to prevent srcipt abort

        if (!empty($import_reload_after) and $start_time + $import_reload_after <= time()) {

                if (!$html) die('Time limit exceeded - unable to reload page (no HTML output)' ."\n");



                $elapsed = $elapsed + time() - $start_time;

                $target = $PHP_SELF ."?key=". urlencode($adminkey) ."&amp;".str_rand()."=".str_rand()."&amp;no_ftp=1&amp;debug=$debug&amp;files=$files&amp;elapsed=$elapsed";

                echo '<meta http-equiv="refresh" content="2;URL='. $target .'">';



                echo'<br /><table border="0" cellpadding="1" cellspacing="2" width="720">

                  <tr>

                        <td class="heading" align="center" colspan="2">Maximum execution time exeeded; restarting ...</td>

                  </tr>

                  </table>';



                include("includes/footer.php");

                return;

        }



        $oldfilename = $filename;

        $filename = 'logs/' . $filename;

        $backupfilename = 'logs/backup/' . $oldfilename;

To:

Code:

$logdir = opendir('logs');



while (false !== ($filename = readdir($logdir)))

{

//////////////////////////////////////////////////////////////////

// ----------------------- EDIT FOR WEBCRON-----------------------

//////////////////////////////////////////////////////////////////

//Our (self set) timelimit exceeded => reload the page to prevent srcipt abort

//        if (!empty($import_reload_after) and $start_time + $import_reload_after <= time()) {

//                if (!$html) die('Time limit exceeded - unable to reload page (no HTML output)' ."\n");

//

//                $elapsed = $elapsed + time() - $start_time;

//                $target = $PHP_SELF ."?key=". urlencode($adminkey) ."&amp;".str_rand()."=".str_rand()."&amp;no_ftp=1&amp;debug=$debug&amp;files=$files&amp;elapsed=$elapsed";

//                echo '<meta http-equiv="refresh" content="2;URL='. $target .'">';

//

//                echo'<br /><table border="0" cellpadding="1" cellspacing="2" width="720">

//                  <tr>

//                        <td class="heading" align="center" colspan="2">Maximum execution time exeeded; restarting ...</td>

//                  </tr>

//                  </table>';

//

//                include("includes/footer.php");

//                return;

//        }

//

//////////////////////////////////////////////////////////////////

        $oldfilename = $filename;

        $filename = 'logs/' . $filename;

        $backupfilename = 'logs/backup/' . $oldfilename;

Quote:

Originally Posted by Matthew

I had to run webcron. I have my account, and all is well webcron wise. It is enabled, and runs every hour. Problem is, when I go to see RECENT MATCHES, it doesn't appear that they have been processed.

Tecnically there is no difference between a connection from your browser and the webcron thinghie. Both request a webpage.
The importer will stop after $import_reload_after seconds (config.php). It asks your browser to reload the same page again until it processed all the files. Webcron does not obey these requests, so your import script won't run long enough to do all the work. You may increase this value depending on your php's max_execution_time (check phpinfo()) or even set it to 0 if safe mode is disabled.

What about just putting the script in the LocalBatcherURL and making sure bBatchLocal is on? Works for my server so far, and no cron script.

I rolled my own script to handle the moving of logs and running import, nothing new there though. Pretty crappy bash kludge really. :)

Think we can get an update to this?

Lines 6 and 13 are totally different than what you have up there.

Apparently they aren't needed anyway.

I didn't have to do most of the stuff in that tutorial for the new version to get the stats automated, though your cronjob scripts did do the work just fine.

Editing the import.php file as you mention didn't require anything.

Well I take that back, I did comment out the 21st line.
if (!isset($adminkey)) $adminkey = 'password;

Took me a while to figure it out, I thought it was the method I was working, when come to find out the web server user didn't have access to the Logs directory on the server, once I set those permissions it was able to move the files, and add them to the database just fine.

I take that back, something isn't working.

The cronjob didn't appear to work correctly, so I tested the file the cronjob called for and it gives me two errors. So I ran the manual script (stupid admins) and it did something but I have no idea what.

Why not use crowbar's mutator and save yourself a lot of physical pain?

Because I have access to cron jobs on my server and I'd rather do it that way, than adding another serveractor to the game server.