Public SourceMage Discussion List

[Andrew <afrayedknot AT thefrayedknot.armory.com>]

heres what i use

basically it downloads everything it possibly can from the grimoires
using summon

then links everything across to the mirror directories (using
hardlinks). after that it rsyncs, the reason I hardlink everything across
is so that way it tries minimize the amount of work rsync has to do,
rsync of course promptly deletes any files it doesnt need, but since
they are hardlinks the files stay in /var/spool/sorcery/

theres another script that seth has which scans for md5sums which i also
run on everything.

On Wed, May 21, 2003 at 10:32:46AM -0700, Glenn Shannon wrote:
> I'd be willing to set up a mirror as well.
> 
> I've got a beefy server that has mass bandwidth I could devote to it :)
> 
> Does anyone have the commands they use to update/check the repositories? I
> could just script it out then, and send myself emails if I get a warning.
> ----- Original Message ----- 
> From: "Andrew" <afrayedknot AT thefrayedknot.armory.com>
> To: <sm-discuss AT lists.ibiblio.org>
> Sent: Wednesday, May 21, 2003 10:29 AM
> Subject: Re: [SM-Discuss] Sorcery Fall back mirror needs patchting!
> 
> 
> > sure its fine with me, i dont have too much bandwidth but perhaps if we
> > spread across some other peoples sites we'd be fine.
> >
> > the site is ftp://thefrayedknot.armory.com/ or
> > ftp://t.armory.com/ for short
> >
> > there is a directory called sorcery which has everything thats been
> > summonable for the past 3 weeks. 3 weeks ago i had an annoying xfs crash
> > and lost the 8gbs (and 4 months worth) of mirror data, moral? dont use
> > xfs if you care about your data.
> >
> > currently im missing about 20 tarballs over the entire grimoire.
> >
> > there is also a directory called smgl-mirror which used to rsync against
> > ibiblio but currently isnt, if anyone knows whats wrong with the following
> > line please tell me
> >
> > On Wed, May 21, 2003 at 10:17:34AM -0700, Eric Sandall wrote:
> > >
> > > Andrew said:
> > > > fyi I have a full mirror of stable test and devel, i update it every
> > > > night and scan for bad md5's. Seth does the same thing.
> > >
> > > Is it available for us to use?  If so, what's the URL?  Any requirements
> > > and/or prohibitions?
> > >
> > > -One of Four
> > > a.k.a. sandalle
> > >
> > > -- 
> > > PGP Key Fingerprint:  FCFF 26A1 BE21 08F4 BB91  FAED 1D7B 7D74 A8EF DD61
> > > http://search.keyserver.net:11371/pks/lookup?op=get&search=0xA8EFDD61
> > >
> > > Eric Sandall                     |  Source Mage GNU/Linux Developer
> > > eric AT sandall.us                  |  http://www.sourcemage.org/
> > > http://eric.sandall.us/          |  SysAdmin @ Inst. Shock Physics @ WSU
> > > http://counter.li.org/  #196285  |  http://www.shock.wsu.edu/
> > > _______________________________________________
> > > SM-Discuss mailing list
> > > SM-Discuss AT lists.ibiblio.org
> > > http://lists.ibiblio.org/mailman/listinfo/sm-discuss
> > _______________________________________________
> > SM-Discuss mailing list
> > SM-Discuss AT lists.ibiblio.org
> > http://lists.ibiblio.org/mailman/listinfo/sm-discuss
> >
> >
> >
> 
> 
#!/bin/bash
summon --all -g test
summon --all -g stable
summon --all -g devel
TARGET=/usr/share/media/smgl-mirror
LOCALG=/var/spool/sorcery
GRIMOIRES="devel test"
cd /tmp
ls $LOCALG|sort|uniq >spooldir
echo updating mirror from spool
for stuff in $GRIMOIRES ; do 
        echo $stuff
        ls $TARGET/mirror/$stuff|sort|uniq >mirror
        comm -3 spooldir mirror |cut -f1 > linkme
        for each in `cat linkme` ; do
                echo $each
                ln $LOCALG/$each $TARGET/mirror/$stuff 
         done
done

rsync --delete -av --bwlimit=20 ibiblio.org::sourcemage $TARGET

for stuff in $GRIMOIRES; do 
        ls $LOCALG|sort|uniq >spooldir
        ls $TARGET/mirror/$stuff|sort|uniq >mirror
        comm -3 mirror spooldir |cut -f1 > linkme
        echo updating spool from $stuff
        for each in `cat linkme` ; do
                echo $each
                ln $TARGET/mirror/$stuff/$each  $LOCALG
         done
done
#!/bin/bash


#---------------------------------------------------------------------
##=item gaze_checkmd5 <spelldirectory> <sourcenumber>
##
## spelldirectory is the spell's directory
## sourcenumber is '' or '2', '3', '4', etc.
## Checks the md5 of a single source file sourcenumber in spelldirectory
##
#---------------------------------------------------------------------
gaze_checkmd5 () {
  i=$1/DETAILS
  SOURCEnum=$2
  SOURCEvar=SOURCE$SOURCEnum

  SOURCE=${!SOURCEvar}
  if [ -n "$SOURCEnum" ]
  then
    MD5num="$(($SOURCEnum-1))"
  else
    MD5num="0"
  fi
  tMD5=${MD5[$MD5num]}

  GRIMOIRE=`echo $i | sed -e 's/\/[^\/]*\/[^\/]*\/DETAILS//' -e 's/\/.*\///'`
  SECTION=`echo $i | sed -e 's/\/[^\/]*\/DETAILS//' -e 's/\/.*\///'`
  if test "$SECTION" == "$SPELL" || test "$SECTION" == "DETAILS"
  then
    echo -en "${SPELL_COLOR}$SPELL${DEFAULT} "
  else
    echo -en "$GRIMOIRE: $SECTION: ${SPELL_COLOR}$SPELL${DEFAULT} "
  fi

  if test "$SOURCE" != '' 
  then
    if [ -f $SOURCE_CACHE/$SOURCE ]
    then
      APPEND=
    else
      OLDESTSOURCE=$SOURCE
      OLDSOURCE=`echo $SOURCE | sed -e 's/\./\\\./g'`
      SOURCE=`echo $SOURCE | sed -e 's/\.[bt].*/\./'`
      SOURCE=`ls $SOURCE_CACHE/$SOURCE* 2> /dev/null | cut -d'/' -f 5 | head 
-1`
      if [ "$SOURCE" ]
      then
        APPEND="EDIT s/$OLDSOURCE/$SOURCE/ "
      else 
        APPEND=
        SOURCE=$OLDESTSOURCE
      fi
    fi
    if test -z "$PRECACHEDF"
    then
      if test `guess_compressor $SOURCE_CACHE/$SOURCE` != 'gzip'   && 
         test `guess_compressor $SOURCE_CACHE/$SOURCE` != 'bzip2'  &&
         test `guess_compressor $SOURCE_CACHE/$SOURCE` != 'compress'"'"'d' &&
         test `guess_compressor $SOURCE_CACHE/$SOURCE` != 'Zip'  &&
         test `guess_compressor $SOURCE_CACHE/$SOURCE` != 'RPM'  &&
         test `guess_compressor $SOURCE_CACHE/$SOURCE` != 'tar'
      then
        fMD5=`cat $SOURCE_CACHE/$SOURCE 2> /dev/null | md5sum | cut -d' ' -f1`
      else
        fMD5=$(
          uncompress $SOURCE_CACHE/$SOURCE `
            guess_compressor $SOURCE_CACHE/$SOURCE` | 
          md5sum | cut -d' ' -f1
        )
      fi
    else
      fMD5="`grep " $SOURCE\$" $PRECACHEDF | grep -Ev '\.[0-9]{14}$' | grep 
-v '\.asc$' | cut -d' ' -f1`"
    fi
    if test "$fMD5" == "$tMD5"
    then
      echo -en "${GREEN}VERIFIED SOURCE[$MD5num] $SOURCE${DEFAULT}"
    else
      if test "$fMD5" == "d41d8cd98f00b204e9800998ecf8427e" || test "$fMD5" 
== ""
      then
        echo -n "SKIPPED SOURCE[$MD5num] $SOURCE"
      else
        if test "$tMD5" == ""
        then
          echo -en "${YELLOW}${BOLD}UNCHECKED SOURCE[$MD5num] $SOURCE INSERT 
MD5[$MD5num]=$fMD5 or MD5[$MD5num]=IGNORE${DEFAULT}"
        else   
          if test "$tMD5" == "IGNORE"
          then
            echo -en "${YELLOW}IGNORED SOURCE[$MD5num] $SOURCE${DEFAULT}"
          else
            if test -z "$PRECACHEDB"
            then
              bMD5="`md5sum $SOURCE_CACHE/$SOURCE | cut -d' ' -f1`"
            else
              bMD5="`grep " $SOURCE\$" $PRECACHEDB | grep -Ev '\.[0-9]{14}$' 
| grep -v '\.asc$'`" # | cut -d' ' -f1`"
            fi
            if test "$tMD5" == "$bMD5"
            then
              echo -en "${RED}${BOLD}MALFORMED SOURCE[$MD5num] $SOURCE EDIT 
s/$tMD5/$fMD5/${DEFAULT}"
            elif test "$tMD5" == "d41d8cd98f00b204e9800998ecf8427e"
            then
              echo -en "${RED}${BOLD}EMPTY SOURCE[$MD5num] $SOURCE EDIT 
s/$tMD5/$fMD5/${DEFAULT}"
            elif test -n "$PRECACHEDB" && test -n "`grep $tMD5 $PRECACHEDF`"
            then
              echo -en "${RED}${BOLD}DIFFERENT SOURCE[$MD5num] $SOURCE EDIT 
s/$tMD5/$fMD5/${DEFAULT} MATCHES "`grep "$tMD5" "$PRECACHEDF" | cut -c60-`
            else
              echo -en "${RED}${BOLD}INCORRECT SOURCE[$MD5num] $SOURCE EDIT 
s/$tMD5/$fMD5/${DEFAULT}"
            fi
          fi
        fi
      fi
    fi  
    echo -n " $APPEND"
    if test "$tMD5" != "IGNORE" ; then
      gaze_checkmd5syntax "$1" "$2"
    fi
  else
    echo -n "NONSOURCE "
  fi
  echo
}

#---------------------------------------------------------------------
##=item gaze_checkmd5syntax <spelldirectory> <sourcenumber>
##
## spelldirectory is the spell's directory
## sourcenumber is '' or '2', '3', '4', etc.
## Checks the syntax related to md5 checking of a single source file 
sourcenumber in spelldirectory
##
#---------------------------------------------------------------------
gaze_checkmd5syntax () {
  rp=$1
  sn=$2
  echo -en "${YELLOW}${BOLD}"
  if test -f $rp/PRE_BUILD ; then
    #echo $rp $sn
    if test "`grep unpack $rp/* | grep SOURCE$sn[^0-9] | grep MD5`" = "" ; 
then
      if test "$sn" != "" ; then
        echo -n "ADD unpack md5 arg to EXISTING PRE_BUILD"
      else
        if test "`grep default_pre_build $rp/*`" = "" ; then
          echo -n "ADD unpack md5 arg to EXISTING PRE_BUILD"
        fi
      fi
    fi
  else
    if test -f $rp/POST_INSTALL ; then
      if test "`grep unpack $rp/* | grep SOURCE$sn[^0-9] | grep MD5`" = "" ; 
then
        if test "$sn" != "" ; then
          echo -n "ADD unpack md5 arg to EXISTING POST_INSTALL or CREATED 
PRE_BUILD"
        fi
      fi
    else
      if test "$sn" != "" ; then
        echo -n "APPEND unpack md5 arg to CREATED PRE_BUILD"
      else
        if test "$SOURCE2" != "" ; then
          echo -n "INIT unpack md5 arg or default_pre_build to NEW PRE_BUILD"
        fi
      fi
    fi
  fi
  echo -en "${DEFAULT}"
}

#---------------------------------------------------------------------
##=item gaze_checkmd5s <spelldirectory>
##
## spelldirectory is the spell's directory
## Checks the md5s of all source files in spelldirectory
##
#---------------------------------------------------------------------
gaze_checkmd5s() {
  spellroot=$1
  SCRIPT_DIRECTORY=$spellroot
  spellpath=$spellroot/DETAILS
  source $spellpath > /dev/null 2> /dev/null

  gaze_checkmd5  $spellroot  ''

  unset SOURCE
  unset MD5[0]
  j=2
  jj=SOURCE$j
  while  [  -n "${!jj}"  ];  do

    SOURCEvar=SOURCE$j
    SOURCE=${!SOURCEvar}
    if test "$SOURCE" != " " ; then

      gaze_checkmd5  $spellroot  $j

    fi
    unset $jj
    unset MD5[$(($j-1))]
    j=$(($j+1))
    jj=SOURCE$j
  done
  unset VERSION
}

#---------------------------------------------------------------------
##=item gaze_md5check [<item> ...]
##
## item is a spell or section name.
## Checks the md5s of various spells, sections, or if called with no 
arguments, the entire grimoire
##
#---------------------------------------------------------------------
gaze_md5check() {

  source /etc/sorcery/config

  #if CACHED is defined, then use some caches for speedy checking.
  if ! test -z "$CACHED" ; then
    PRECACHEDF=/var/spool/sorcery/md5unpack
    PRECACHEDB=/var/spool/sorcery/md5sum
  fi

  unset  SECTIONS  SPELLS  UNKNOWN

  [  -z  "$1"  ]  &&  SECTIONS=`codex_get_all_sections`

  for  spell_or_section  in  $@;  do

    if  codex_find_spell_or_section_by_name  $spell_or_section;  then
      [  -n  "$CODEX_FOUND_SECTION"  ]  &&  SECTIONS="$SECTIONS 
$CODEX_FOUND_SECTION"
      [  -n  "$CODEX_FOUND_SPELL"  ]    &&  SPELLS="$SPELLS 
$CODEX_FOUND_SPELL"
    else
      UNKNOWN="$spell_or_section $UNKNOWN"
    fi

  done

  for i in $SECTIONS ; do
    SPELLS="$SPELLS `codex_get_spells $i`"
  done

  for i in $SPELLS ; do
    gaze_checkmd5s   $i
  done

  if [  -n  "$UNKNOWN"  ] ; then
    for i in $UNKNOWN ; do
      echo "unknown: $i"
    done
  fi

}

gaze_md5check $*