Well i’ve hit the other odd thing that I know has effected
these type of stats before and thats when the data
contains charactors like , ; space tab etc etc.
Got it writing out a CSV file to a file called lifemapper.dat
just imported it into excel while most of it was correct it
has fallen down where people have , in some of there data.
At least now its raw data at only 57K with a CSV file made
up of no label tags and just data of UserID/Team/UserName/Computingtime/Jobsdone/speciesmapped.
I don’t have any experiences with CSV files.
I’m using a MySQL database and I try to catch all characters that are special for SQL…
BTW:
I assume you’re talking about size in when you mention that 57K?? 
If that’s the case… it’s bigger than I expected.
I’ve just made a full MySQL export of the statistics of:
Of the entire database I’m keeping, both data and database structure (=everything)
My stats site (see link below) is now running for somewhat more then a month.
Total size: 244KB
(with ‘drop tables’ option and ‘full and extended inserts’)
:hat:
Yes that is the raw size for all members report
57,000 bytes against the exported xml from lifemapper
of 490161 bytes.
i’m only at the stage of exporting selected data fields to a
database.
That filesize would be a lot smaller if it was sorted to give
say only TPR members and export that only.
Hi guys,
Only just seen this thread. I am currently talking with barcode about creating a full stats database on his server for the seti stats siggy counters. Perhaps we could integrate all the stats counters into one database so that we can have multiple projects on each counter? If someone can supply me with the code you’ve been working on to parse the LM XML, I can use it to populate a table for LM counters too.
Any thoughts on this idea?
Perhaps we can do Genome, Folding, etc aswell?
Matt. 
Originally posted by Mincer
If someone can supply me with the code you’ve been working on to parse the LM XML, I can use it to populate a table for LM counters too.
PMM did the parsing XML section of the code, I’ve created the db and added the writing to db section on the end:
<?php
if(!include('./xmlize.php')){
$error = 'Error. Incorrect location for xmlize.php.<br />';
die($error);
}
$xmlfile = 'http://www.lifemapper.org/lm-individualmembers/lm-memberstats/FullMembersReport.xml';
$fp = fopen($xmlfile,"rb");
if($fp){
$xml = '';
while (!feof ($fp)) {
$xml .= fgets ($fp, 1024);
}
fclose ($fp);
}else{
$error = 'Error. File <b>'.$xmlfile.'</b> does not exist.<br />';
die($error);
}
$xml_array = xmlize($xml);
$xml = xmlize($xml);
$team_stats=$xml["xml"]["#"]["rs:data"]["0"]["#"]["z:row"];
for($i = 0; $i < sizeof($team_stats); $i++) {
$person = $team_stats[$i];
$statix = $person["#"];
$ID = $person["@"]["ID"];
$Name = addslashes($person["@"]["Name"]);
$AddressWWW = $person["@"]["AddressWWW"];
$TotCPUTime = $person["@"]["TotCPUTime"];
$JoinDate = $person["@"]["JoinDate"];
$EMail = $person["@"]["EMail"];
$TaxaComputed = $person["@"]["TaxaComputed"];
$NumOfJobs = $person["@"]["NumOfJobs"];
$Country = $person["@"]["Country"];
$PrefCountry = $person["@"]["PrefCountry"];
$CPUGroup = $person["@"]["CPUGroup"];
$CPUGroupID = $person["@"]["CPUGroupID"];
$PrefDataProvider = $person["@"]["PrefDataProvide"];
$GroupAddressWWW = $person["@"]["GroupAddressWWW"];
$PrefTaxonGroup = $person["@"]["PrefTaxonGroup"];
$link= mysql_connect("localhost", "username","password") or
die(mysql_error());
$db = mysql_select_db("new_dbase", $link) or die(mysql_error());
$query = "insert into lm_user (ID, Name, EMail, JoinDate, Country) values ('$ID', '$Name', '$EMail', '$JoinDate', '$Country')";
$result = @mysql_query($query) or die (mysql_error());
$query2 = "insert into lm_variablestats (ID, TotCPUTime, NumOfJobs, TaxaComputed) values ('$ID', '$TotCPUTime', '$NumOfJobs', '$TaxaComputed')";
$result2 = mysql_query($query2) or die (mysql_error());
}
?>
I’ve just been having a quick fiddle whilst working (…
…) but have created two tables so far, haven’t gotten round to the third yet:
lm_user - to hold fixed user details
fields: ID, Name, Email, JoinDate,Country
lm_variablestats - to hold variable stats details
fields:ID, TotCPUTime, NumOfJobs, TaxaCompleted
lm_preferences - to hold user preferences
fields: ID, PrefCountry, AddessWWW, CPUGroup, CPUGroupID, GroupAddressWWW, PrefTaxonGroup
The above is just a first thought, it’ll prob change as/when it develops further…
The thing is, we were thinking more than just counters - I’m looking at some web-based stats that will be self-sustaining (ie not having to be uploaded every day at 5am). Obviously not as in-depth as the Stat-O-Matic but will be something to keep the punters interested 
So… we have three choices:[ul]
[li]move the whole LM stats to barcode’s server
[/li][li]have a feed from the LM stats db to the counter db on barcode’s server
[/li][li]It’d be feasible enough to just link the counter script to feed from the db on my server…?
[/li][/ul]What do you reckon…? In theory we can host the stats pages on the current stats site and host the db on my www.riddlermarc.co.uk server, my bandwidth is easily enough to handle the calls…
The output of the table lm_variablestats can be seen here: http://www.riddlermarc.co.uk/dev/lm/show_variablestats.php should hold most of the necessary data for LM counters?
Originally posted by riddlermarc
The thing is, we were thinking more than just counters - I’m looking at some web-based stats that will be self-sustaining (ie not having to be uploaded every day at 5am). Obviously not as in-depth as the Stat-O-Matic but will be something to keep the punters interested
This was exactly my idea too. I’ll see what I can come up with and we can merge efforts when we can.
wish I could understand more of this thread but for my sins I am a biologist and get a bit lost in code 
if you guys need any coffee making I can help there 
Originally posted by lab_boy
if you guys need any coffee making I can help there
And I can dial up a pizza for you. 
OK, I’ve had a crack at modifying the PHP code used for the updating WU counter in my siggy so that it will now poll the lifemapper site and auto update the number of species I’ve processed. I’ve had to make it poll the team site and look for my user name as I can’t work out the URL (with arguments) needed to directly enter my user stats area. The basic URL is http://www.lifemapper.org/lm-apps/ViewMember/viewMember.asp but then I guess I have to add various arguments after this (id=emailaddress??) to get to the stats page.
I would much rather have the script poll this page as it’s updated regularly rather than the team page which only gets updated every 12 hours.
can anyone help me? I’m a complete noob with stuff like this and I’ve just tried to guess my way through
Getting the html stats is easy
But as per Seti think concerns were that is would be a large
hit on Lifemapper everytime the counter was created esp
if all members were to use it on the basis the HTML if i remember
is some 400K .
So the idea as far as I see was to get something set up where
lifemapper is hit once for data and that is put into a database
where its stripped to its bare minimum for creating any counters
and and team stat pages.
I know Riddler was going to attempt getting this sorted but I
suspect he’s currently side tracked with other TPR issues.
I was going to have a bash at it but at this time…I have no time
to do so or the willing to do so again down to TPR issues.
I’m sure it will come with time
—>
Would someone be interested in starting a sourceforge project to do this. Then the whole DC community can benefit?
Any thoughts?