unit caching util for unix?

I can’t seem to find one… searched the sticky etc.

Anyone know of one?

Cheers

Originally posted by MrTFWitt
[B]You can point it at a queue by running with the -proxy flag.

Takes about 20 lines of shell script to make a cache of sorts.

Want me to cobble some stuff together ? [/B]

Yeh thats the problem - i cant use a queue as the machines are firewalled and pretty locked down security wise. I’d like to be able to tar up a cache of units and have the machines crunch them - then after they are all complete i can ftp them off the machine and upload them.

If you fancy having a go at a script that would be great :slight_smile: I’m sure some other team member could find it useful as well.

Cheers

blimey charlie ! good work TFW :slight_smile:

I will give it a go later, looks good - thanks

had a couple of probs to sort out but i will give it a go tomorrow mate.

it certainly looks like it will work :slight_smile:

TF - you da man ! :hail: :hail:

Works a treat mate, no probs whatsoever :slight_smile:

I say we stickify this NOW !

Thanks again mate :):banana:

You can point it at a queue by running with the -proxy flag.

Takes about 20 lines of shell script to make a cache of sorts.

Want me to cobble some stuff together ?

Caching units on a Linux system

Create an account for seti processing if you havent done so already.

Either use linuxconf or the following lines will do it

groupadd seti
useradd -g seti -d /home/seti -s /bin/ksh -c “Seti User” -m seti
passwd seti

The next thing to do is put the setiathome binary in the right place

cp setiathome ~seti

If you have already run seti on this system you will have a user_info.sah.
copy that into the same directory

cp user_info.sah ~seti

If not you will have to do a login as a returning user to create one

su - seti
./setiathome -login

Right so thats a userid and a working install of seti, now we need to create the cache and leech some units.

Setup script



#!/bin/bash
cd ~seti
if [ ! -f setiathome -o ! -f user_info.sah ]
then
        echo expected to find setiathome and user_info.sah in this directory
        exit 1
fi

Proxy=www.adventurevision.com:5517
Clients=20
i=0
while [ "$Clients"  -gt "$i" ]
do
        dirname=`echo $i|awk '{printf("sc%2.2d",$1)}'`
        if [ -d ~seti/$dirname ]
        then
                if [ ! -f ~seti/$dirname/work_unit.sah ]
                then
                        cd ~seti/$dirname
                        ../setiathome -stop_after_xfer -proxy $Proxy
                fi
        else
                echo Creating ~seti/$dirname
                mkdir ~seti/$dirname
                cp user_info.sah ~seti/$dirname
                cd ~seti/$dirname
                ../setiathome -stop_after_xfer -proxy $Proxy
        fi
        i=`expr $i + 1`
done

Cut and paste the above into a file

save the file as setup and type chmod u+x setup to set the execute bit.

There are a couple of lines you can change
Proxy= If you are on NTL pick a proxy that lets you see Berkeley !
Clients= 99 is the maximum unless you change the awk statement.

Now your cache is chocka with units its time for some crunching.
Cut and paste the following into a file called runit saved in the seti users directory


#!/bin/bash
cd ~seti
while :
do
        for unit in `find . -name work_unit.sah`
        do
                cd `dirname $unit`
                echo Starting on `pwd`
                ~seti/setiathome -stop_after_process
                cd ~seti
        done
        sleep 10
done

then type chmod u+x runit

Launch this with nohup ./runit & and it will keep running after you logoff.

To run at boot time add the following to /etc/rc.local

su - seti -c “nohup /home/seti/runit &”

Uploading units can be done with this snippet


#!/bin/bash
cd ~seti
Proxy=www.adventurevision.com:5517
for unit in `find . -name result.sah`
do
        cd `dirname $unit`
        echo Sending `pwd`
        ~seti/setiathome -stop_after_xfer -proxy $Proxy
        cd ~seti
done

Right those that are still awake will be asking about progress checking. This will tell you a bit more detail on completed units and time per unit. Nowt flash as laziness was setting in when I did this bit.


#!/bin/ksh
for file in `find . -name state.sah` `find . -name wtemp.sah`
do
cpu=`grep "^cpu=" $file |cut -d "=" -f2|cut -d"." -f1`
prog=`grep "^prog=" $file |cut -d "=" -f2`
if [ ! -z ${cpu} ]
then

time=$cpu
hours=`expr $time / 3600`
time=`expr $time - \( 3600 \* $hours \)`
mins=`expr $time / 60`
time=`expr $time - \( 60 \* $mins \)`
secs=$time

timestring=`echo $hours $mins $secs |awk '{printf("%2.2d:%2.2d:%2.2d
",$1,$2,$3 )}'`
echo $file $timestring $prog
fi
done

All feedback welcome, I’ve tested most of this but not from square one on a clean build like many of you will be doing.
Oh and I used Redhat 7.2 with the INSTALL EVERYthing button selected.

Did you try any of this out ?

Just wondering if any of this is sticky-worthy stuff.

I didn’t save the post so once it drops off the forum its gone for good.

Tick Tick Tick Tick . . . . . . . . . .

I’m at work tomorrow so any feedback from me may be on the slow side (Mondays are generally bedlam).

It should work but your way of using it may be different to mine.
I believe you are intending to use TAR to move wedges of units around.
I might try and find the NFS version I wrote that eats from one directory and deposits the finished ones in another. Works quite well with a sizeable network :smiley:

Done :slight_smile:

A re-read of this and I suspect the start at boot option is broken.

It should probably read

su - seti -c “nohup /home/seti/runit &”

Otherwise it will run it as root and mangle the permissions.

I did a bit of editing and removed the need for the seti user - this is quicker for me to deploy :slight_smile:

Maybe we could tar the scripts up and make them available to other members for download?

The non root bit was planning ahead for NFS use.
Allowing root access to a shared volume makes the already weak validation of Userid’s laughable. Fine for a home lan but not for use at work.

By all means tar up what you have and feel free to host it somewhere.

Damn fine work Mr TFW and Muppet, a most excellent addition to the sticky!