Skip to Content.
Sympa Menu

sm-discuss - Re: [SM-Discuss] Sorcery Fall back mirror needs patchting!

sm-discuss AT lists.ibiblio.org

Subject: Public SourceMage Discussion List

List archive

Chronological Thread  
  • From: Andrew <afrayedknot AT thefrayedknot.armory.com>
  • To: sm-discuss AT lists.ibiblio.org
  • Subject: Re: [SM-Discuss] Sorcery Fall back mirror needs patchting!
  • Date: Wed, 21 May 2003 10:48:54 -0700

heres what i use

basically it downloads everything it possibly can from the grimoires
using summon

then links everything across to the mirror directories (using
hardlinks). after that it rsyncs, the reason I hardlink everything across
is so that way it tries minimize the amount of work rsync has to do,
rsync of course promptly deletes any files it doesnt need, but since
they are hardlinks the files stay in /var/spool/sorcery/

theres another script that seth has which scans for md5sums which i also
run on everything.

On Wed, May 21, 2003 at 10:32:46AM -0700, Glenn Shannon wrote:
> I'd be willing to set up a mirror as well.
>
> I've got a beefy server that has mass bandwidth I could devote to it :)
>
> Does anyone have the commands they use to update/check the repositories? I
> could just script it out then, and send myself emails if I get a warning.
> ----- Original Message -----
> From: "Andrew" <afrayedknot AT thefrayedknot.armory.com>
> To: <sm-discuss AT lists.ibiblio.org>
> Sent: Wednesday, May 21, 2003 10:29 AM
> Subject: Re: [SM-Discuss] Sorcery Fall back mirror needs patchting!
>
>
> > sure its fine with me, i dont have too much bandwidth but perhaps if we
> > spread across some other peoples sites we'd be fine.
> >
> > the site is ftp://thefrayedknot.armory.com/ or
> > ftp://t.armory.com/ for short
> >
> > there is a directory called sorcery which has everything thats been
> > summonable for the past 3 weeks. 3 weeks ago i had an annoying xfs crash
> > and lost the 8gbs (and 4 months worth) of mirror data, moral? dont use
> > xfs if you care about your data.
> >
> > currently im missing about 20 tarballs over the entire grimoire.
> >
> > there is also a directory called smgl-mirror which used to rsync against
> > ibiblio but currently isnt, if anyone knows whats wrong with the following
> > line please tell me
> >
> > On Wed, May 21, 2003 at 10:17:34AM -0700, Eric Sandall wrote:
> > >
> > > Andrew said:
> > > > fyi I have a full mirror of stable test and devel, i update it every
> > > > night and scan for bad md5's. Seth does the same thing.
> > >
> > > Is it available for us to use? If so, what's the URL? Any requirements
> > > and/or prohibitions?
> > >
> > > -One of Four
> > > a.k.a. sandalle
> > >
> > > --
> > > PGP Key Fingerprint: FCFF 26A1 BE21 08F4 BB91 FAED 1D7B 7D74 A8EF DD61
> > > http://search.keyserver.net:11371/pks/lookup?op=get&search=0xA8EFDD61
> > >
> > > Eric Sandall | Source Mage GNU/Linux Developer
> > > eric AT sandall.us | http://www.sourcemage.org/
> > > http://eric.sandall.us/ | SysAdmin @ Inst. Shock Physics @ WSU
> > > http://counter.li.org/ #196285 | http://www.shock.wsu.edu/
> > > _______________________________________________
> > > SM-Discuss mailing list
> > > SM-Discuss AT lists.ibiblio.org
> > > http://lists.ibiblio.org/mailman/listinfo/sm-discuss
> > _______________________________________________
> > SM-Discuss mailing list
> > SM-Discuss AT lists.ibiblio.org
> > http://lists.ibiblio.org/mailman/listinfo/sm-discuss
> >
> >
> >
>
>
#!/bin/bash
summon --all -g test
summon --all -g stable
summon --all -g devel
TARGET=/usr/share/media/smgl-mirror
LOCALG=/var/spool/sorcery
GRIMOIRES="devel test"
cd /tmp
ls $LOCALG|sort|uniq >spooldir
echo updating mirror from spool
for stuff in $GRIMOIRES ; do
echo $stuff
ls $TARGET/mirror/$stuff|sort|uniq >mirror
comm -3 spooldir mirror |cut -f1 > linkme
for each in `cat linkme` ; do
echo $each
ln $LOCALG/$each $TARGET/mirror/$stuff
done
done

rsync --delete -av --bwlimit=20 ibiblio.org::sourcemage $TARGET

for stuff in $GRIMOIRES; do
ls $LOCALG|sort|uniq >spooldir
ls $TARGET/mirror/$stuff|sort|uniq >mirror
comm -3 mirror spooldir |cut -f1 > linkme
echo updating spool from $stuff
for each in `cat linkme` ; do
echo $each
ln $TARGET/mirror/$stuff/$each $LOCALG
done
done
#!/bin/bash


#---------------------------------------------------------------------
##=item gaze_checkmd5 <spelldirectory> <sourcenumber>
##
## spelldirectory is the spell's directory
## sourcenumber is '' or '2', '3', '4', etc.
## Checks the md5 of a single source file sourcenumber in spelldirectory
##
#---------------------------------------------------------------------
gaze_checkmd5 () {
i=$1/DETAILS
SOURCEnum=$2
SOURCEvar=SOURCE$SOURCEnum

SOURCE=${!SOURCEvar}
if [ -n "$SOURCEnum" ]
then
MD5num="$(($SOURCEnum-1))"
else
MD5num="0"
fi
tMD5=${MD5[$MD5num]}

GRIMOIRE=`echo $i | sed -e 's/\/[^\/]*\/[^\/]*\/DETAILS//' -e 's/\/.*\///'`
SECTION=`echo $i | sed -e 's/\/[^\/]*\/DETAILS//' -e 's/\/.*\///'`
if test "$SECTION" == "$SPELL" || test "$SECTION" == "DETAILS"
then
echo -en "${SPELL_COLOR}$SPELL${DEFAULT} "
else
echo -en "$GRIMOIRE: $SECTION: ${SPELL_COLOR}$SPELL${DEFAULT} "
fi

if test "$SOURCE" != ''
then
if [ -f $SOURCE_CACHE/$SOURCE ]
then
APPEND=
else
OLDESTSOURCE=$SOURCE
OLDSOURCE=`echo $SOURCE | sed -e 's/\./\\\./g'`
SOURCE=`echo $SOURCE | sed -e 's/\.[bt].*/\./'`
SOURCE=`ls $SOURCE_CACHE/$SOURCE* 2> /dev/null | cut -d'/' -f 5 | head
-1`
if [ "$SOURCE" ]
then
APPEND="EDIT s/$OLDSOURCE/$SOURCE/ "
else
APPEND=
SOURCE=$OLDESTSOURCE
fi
fi
if test -z "$PRECACHEDF"
then
if test `guess_compressor $SOURCE_CACHE/$SOURCE` != 'gzip' &&
test `guess_compressor $SOURCE_CACHE/$SOURCE` != 'bzip2' &&
test `guess_compressor $SOURCE_CACHE/$SOURCE` != 'compress'"'"'d' &&
test `guess_compressor $SOURCE_CACHE/$SOURCE` != 'Zip' &&
test `guess_compressor $SOURCE_CACHE/$SOURCE` != 'RPM' &&
test `guess_compressor $SOURCE_CACHE/$SOURCE` != 'tar'
then
fMD5=`cat $SOURCE_CACHE/$SOURCE 2> /dev/null | md5sum | cut -d' ' -f1`
else
fMD5=$(
uncompress $SOURCE_CACHE/$SOURCE `
guess_compressor $SOURCE_CACHE/$SOURCE` |
md5sum | cut -d' ' -f1
)
fi
else
fMD5="`grep " $SOURCE\$" $PRECACHEDF | grep -Ev '\.[0-9]{14}$' | grep
-v '\.asc$' | cut -d' ' -f1`"
fi
if test "$fMD5" == "$tMD5"
then
echo -en "${GREEN}VERIFIED SOURCE[$MD5num] $SOURCE${DEFAULT}"
else
if test "$fMD5" == "d41d8cd98f00b204e9800998ecf8427e" || test "$fMD5"
== ""
then
echo -n "SKIPPED SOURCE[$MD5num] $SOURCE"
else
if test "$tMD5" == ""
then
echo -en "${YELLOW}${BOLD}UNCHECKED SOURCE[$MD5num] $SOURCE INSERT
MD5[$MD5num]=$fMD5 or MD5[$MD5num]=IGNORE${DEFAULT}"
else
if test "$tMD5" == "IGNORE"
then
echo -en "${YELLOW}IGNORED SOURCE[$MD5num] $SOURCE${DEFAULT}"
else
if test -z "$PRECACHEDB"
then
bMD5="`md5sum $SOURCE_CACHE/$SOURCE | cut -d' ' -f1`"
else
bMD5="`grep " $SOURCE\$" $PRECACHEDB | grep -Ev '\.[0-9]{14}$'
| grep -v '\.asc$'`" # | cut -d' ' -f1`"
fi
if test "$tMD5" == "$bMD5"
then
echo -en "${RED}${BOLD}MALFORMED SOURCE[$MD5num] $SOURCE EDIT
s/$tMD5/$fMD5/${DEFAULT}"
elif test "$tMD5" == "d41d8cd98f00b204e9800998ecf8427e"
then
echo -en "${RED}${BOLD}EMPTY SOURCE[$MD5num] $SOURCE EDIT
s/$tMD5/$fMD5/${DEFAULT}"
elif test -n "$PRECACHEDB" && test -n "`grep $tMD5 $PRECACHEDF`"
then
echo -en "${RED}${BOLD}DIFFERENT SOURCE[$MD5num] $SOURCE EDIT
s/$tMD5/$fMD5/${DEFAULT} MATCHES "`grep "$tMD5" "$PRECACHEDF" | cut -c60-`
else
echo -en "${RED}${BOLD}INCORRECT SOURCE[$MD5num] $SOURCE EDIT
s/$tMD5/$fMD5/${DEFAULT}"
fi
fi
fi
fi
fi
echo -n " $APPEND"
if test "$tMD5" != "IGNORE" ; then
gaze_checkmd5syntax "$1" "$2"
fi
else
echo -n "NONSOURCE "
fi
echo
}

#---------------------------------------------------------------------
##=item gaze_checkmd5syntax <spelldirectory> <sourcenumber>
##
## spelldirectory is the spell's directory
## sourcenumber is '' or '2', '3', '4', etc.
## Checks the syntax related to md5 checking of a single source file
sourcenumber in spelldirectory
##
#---------------------------------------------------------------------
gaze_checkmd5syntax () {
rp=$1
sn=$2
echo -en "${YELLOW}${BOLD}"
if test -f $rp/PRE_BUILD ; then
#echo $rp $sn
if test "`grep unpack $rp/* | grep SOURCE$sn[^0-9] | grep MD5`" = "" ;
then
if test "$sn" != "" ; then
echo -n "ADD unpack md5 arg to EXISTING PRE_BUILD"
else
if test "`grep default_pre_build $rp/*`" = "" ; then
echo -n "ADD unpack md5 arg to EXISTING PRE_BUILD"
fi
fi
fi
else
if test -f $rp/POST_INSTALL ; then
if test "`grep unpack $rp/* | grep SOURCE$sn[^0-9] | grep MD5`" = "" ;
then
if test "$sn" != "" ; then
echo -n "ADD unpack md5 arg to EXISTING POST_INSTALL or CREATED
PRE_BUILD"
fi
fi
else
if test "$sn" != "" ; then
echo -n "APPEND unpack md5 arg to CREATED PRE_BUILD"
else
if test "$SOURCE2" != "" ; then
echo -n "INIT unpack md5 arg or default_pre_build to NEW PRE_BUILD"
fi
fi
fi
fi
echo -en "${DEFAULT}"
}

#---------------------------------------------------------------------
##=item gaze_checkmd5s <spelldirectory>
##
## spelldirectory is the spell's directory
## Checks the md5s of all source files in spelldirectory
##
#---------------------------------------------------------------------
gaze_checkmd5s() {
spellroot=$1
SCRIPT_DIRECTORY=$spellroot
spellpath=$spellroot/DETAILS
source $spellpath > /dev/null 2> /dev/null

gaze_checkmd5 $spellroot ''

unset SOURCE
unset MD5[0]
j=2
jj=SOURCE$j
while [ -n "${!jj}" ]; do

SOURCEvar=SOURCE$j
SOURCE=${!SOURCEvar}
if test "$SOURCE" != " " ; then

gaze_checkmd5 $spellroot $j

fi
unset $jj
unset MD5[$(($j-1))]
j=$(($j+1))
jj=SOURCE$j
done
unset VERSION
}

#---------------------------------------------------------------------
##=item gaze_md5check [<item> ...]
##
## item is a spell or section name.
## Checks the md5s of various spells, sections, or if called with no
arguments, the entire grimoire
##
#---------------------------------------------------------------------
gaze_md5check() {

source /etc/sorcery/config

#if CACHED is defined, then use some caches for speedy checking.
if ! test -z "$CACHED" ; then
PRECACHEDF=/var/spool/sorcery/md5unpack
PRECACHEDB=/var/spool/sorcery/md5sum
fi

unset SECTIONS SPELLS UNKNOWN

[ -z "$1" ] && SECTIONS=`codex_get_all_sections`

for spell_or_section in $@; do

if codex_find_spell_or_section_by_name $spell_or_section; then
[ -n "$CODEX_FOUND_SECTION" ] && SECTIONS="$SECTIONS
$CODEX_FOUND_SECTION"
[ -n "$CODEX_FOUND_SPELL" ] && SPELLS="$SPELLS
$CODEX_FOUND_SPELL"
else
UNKNOWN="$spell_or_section $UNKNOWN"
fi

done

for i in $SECTIONS ; do
SPELLS="$SPELLS `codex_get_spells $i`"
done

for i in $SPELLS ; do
gaze_checkmd5s $i
done

if [ -n "$UNKNOWN" ] ; then
for i in $UNKNOWN ; do
echo "unknown: $i"
done
fi

}

gaze_md5check $*



Archive powered by MHonArc 2.6.24.

Top of Page