Skip to Content.
Sympa Menu

sm-discuss - [SM-Discuss] Lowish level sorcery changes

sm-discuss AT lists.ibiblio.org

Subject: Public SourceMage Discussion List

List archive

Chronological Thread  
  • From: Dufflebunk <dufflebunk AT dufflebunk.homeip.net>
  • To: sm-discuss <sm-discuss AT lists.ibiblio.org>
  • Subject: [SM-Discuss] Lowish level sorcery changes
  • Date: Mon, 11 Aug 2003 22:25:15 -0000

Well, due to the drop in discussion, I figured I'd toss some stuff out
on sm-discuss for discussion. I've redone liblock, it no longer uses
lockexec. That is good, in that the only reason there was a spell was so
the program could be used for liblock, but bad because the reason it was
used was so that it wasn't necessary to sleep for long random periods.

The new liblock is much faster at serial locking: lock a, unlock a, lock
a unlock, ...
But slower if two processes are trying to lock the same file. The first
one will get the lock, but the second will sleep up to 5 seconds after
the lock has been released.
For serial uncontested locks, I clocked it at about 90 lock-unlocks /
second. I'm not sure what the old liblock was... somewhere closer to 5 I
think.

I've attached the new liblock. It's not in devel sorcery yet. It won't
go in until I can properly run devel sorcery (that's when devel works
with test grimoire again). However, that's no reason other people can't
test it. Do backup your local setting in /etc/sorcery, and stuff in
/var/state/sorcery. People have tested this so it shouldn't cause Bad
Things. If you do want to try it out, just replace liblock (it's a drop
in replacement), and let me know if there are any issues that come up.

And just to stir things up I'll also mention this, Andrew has written a
thin wrapper around the sysv message passing stuff for me and I've
written an interface lib in bash for it. The makes it possible to write
bash script that pass messages efficiently. I would like to add this to
sorcery. It would enable us to remove the nasty hacks in there, like
using files in /tmp for inter-process notices
(/tmp/casting.*.some-spell.download.success for example). I clocked it
at 600 ping-pongs / second. It would make the asynchronus stuff much
nicer to work with. It would be a very nice thing to have, however, it
requires that the sysv kernel stuff be compiled in and of course the
binary wrapper. Are those reasonable requirements? Does anyone not have
sysv stuff compiled in?


--


Quidquid latine dictum sit, altum sonatur.
-----------------
PGP public key at
http://wwwkeys.pgp.net:11371/pks/lookup?op=get&search=0x3327A9A5
F1


[[ $LOCK_DIR ]] || LOCK_DIR=/tmp/liblock-$UID
[ -d $LOCK_DIR ] || mkdir $LOCK_DIR
[[ $LOCK_TRANSACTION ]] || LOCK_TRANSACTION="/tmp/locklist"

[[ $MAX_SLEEP ]] || MAX_SLEEP=5

set -x


#---------------------------------------------------------------------
##=item lock_file <file> [<file>] ...
##
## Locks files for access to only this PID. Will cause attempts by
## other processes that want to lock the file(s) to block until this
## PID unlocks the file, or this PID dies.
## It cannot prevent bad programs from modifying the file w/o
## permission.
## Note: Files with funky chars may break things. No colons or stuff.
## (Blocking)
##
#---------------------------------------------------------------------
function lock_file()
{
debug "liblock" "$FUNCNAME - $*"
local FILE
lock_resources $(
for FILE in "$@" ; do
echo $FILE file
done
)
}


#---------------------------------------------------------------------
##=item unlock_file <file> [<file>] ...
##
## Unlocks a file so that another process can lock and modify it.
##
#---------------------------------------------------------------------
function unlock_file()
{
debug "liblock" "$FUNCNAME - $*"
local FILE
unlock_resources $(
for FILE in "$@" ; do
echo $FILE file
done
)
}


#---------------------------------------------------------------------
##=item lock_resources resource-type resource-name [<type> <name>]...
##
## The generalized locking mechanism. Every resource to be locked
## is accompanied by a type. Currently used types are "lockfunction"
## and "file".
## This is the function that makes use of lockexec to make the
## resource tracking safe in a multiprocess environment.
## (Blocking)
##
#---------------------------------------------------------------------
function lock_resources()
{
debug "liblock" "$FUNCNAME - $*"
local lockfile good_locks
local i ALL_LOCKED="no"

lock_clean_locklist

while [[ $ALL_LOCKED == no ]] ; do
ALL_LOCKED="yes"
for (( i=0; i<$#; i+=2 )) ; do
lockfile="$LOCK_DIR/${1//\//^}.$2.$$.lock"
touch $lockfile
A=$(eval echo $LOCK_DIR/${1//\//^}.*.lock )
B=${A// /}
if [ $(( ${#A}-${#B}+1 )) -gt 1 ] ; then
ALL_LOCKED="no"
echo "Trying to lock ${1//\//^} ($2), with $$
($(ls -l $LOCK_DIR/${1//\//^}.*.lock))."
rm "$lockfile" "${good_locks[@]}" 2>/dev/null
good_locks=()
sleep $(( RANDOM % MAX_SLEEP ))
break
fi
good_locks=("${good_locks[@]}" "$lock_file")
done
done
}


#---------------------------------------------------------------------
##item=unlock_resources <type> <name> [<type> <name] ...
##
## Unlocks exclusive access to the resource so other processes can
## use it.
##
#---------------------------------------------------------------------
function unlock_resources()
{
debug "liblock" "$FUNCNAME - $*"
while [ $# -gt 1 ] ; do
rm "$LOCK_DIR/${1//\//^}.$2.$$.lock" 2>/dev/null || debug
"liblock" "unlock_resources - $* wasn't locked!"
shift 2
done
}


#---------------------------------------------------------------------
##=item is_locked <type> <name>
##
## Prints the PID of the process owning a lock on the file
## Returns 0 if a lock exists
## Returns 1 if the file is not locked
## Note this is slower that lock_owned_by, but obtains a lock and
## cleans losck list
##
#---------------------------------------------------------------------
function is_locked()
{ # $1=type $2=name
debug "liblock" "$FUNCNAME - $*"

# lock_clean_locklist

local A=( $( echo $LOCK_FILE/${2//\//^}.${1}.*.lock ) )
[[ -e "${A[1]}" ]] && echo "Yeah, it's locked :("


}


#---------------------------------------------------------------------
##=item lock_owned_by <type> <name>
##
## Prints the PID of the process owning a lock on the file
## Returns 0 if a lock exists
## Returns 1 if the file is not locked
## Note: This function does not obtain a lock on the lock list. It
## only reads and it checks that each line conforms to the format.
##
#---------------------------------------------------------------------
function lock_owned_by()
{
debug "liblock" "$FUNCNAME - $*"
echo $LOCK_FILE/${2//\//^}.${1}.*.lock | \
sed -n 's/.*\.\([^.]*\)\.lock/\1/p'
}


#---------------------------------------------------------------------
##=lock_start_transaction <file> [<file>] ...
##
## A transaction locks a file and ensures that changes made to the
## file are all made at once. No changes are made to the file until
## the transaction is commited. Furthermore, a transaction can be
## killed before it is commited, ending the transaction and not
## making any changes to the files.
## This function returns the name of a temporary file that should
## be written to instead of the real file.
##
#---------------------------------------------------------------------
function lock_start_transaction()
{
debug "liblock" "$FUNCNAME - $*"
local i TRANSNAME NUMTRANS RET=""

#debug liblock lock_start_transaction - $*

if lock_file "$@" ; then

lock_file $LOCK_TRANSACTIONS
[ -s $LOCK_TRANSACTIONS ] || echo "0:::" > $LOCK_TRANSACTIONS

NUMTRANS=`tail -n 1 $LOCK_TRANSACTIONS | cut -d : -f 1`
NUMTRANS=${NUMTRANS:-1}
RET=()

for i in $* ; do
let NUMTRANS++
# copy the file to the temp file, if it doesn't exist then create an
empty file.
if [ -e "$i" ] ; then
cp "$i" "$LOCK_TRANSACTIONS.$NUMTRANS"
else
echo -n "" > $LOCK_TRANSACTIONS.$NUMTRANS
fi
echo "$NUMTRANS:$i:$$" >$LOCK_TRANSACTIONS
RET[$NUMTRANS]="${LOCK_TRANSACTIONS}.${NUMTRANS}"
done

unlock_file $LOCK_TRANSACTIONS
echo ${RET[*]}
return 0

else
return 1
fi
}

#---------------------------------------------------------------------
##=item lock_commit_transaction <file> [<file>]...
##
## Commits the changes made to the files as atomicaly as possible
## Before this func is called, the files remain unchanged.
##
#---------------------------------------------------------------------
function lock_commit_transaction()
{
debug "liblock" "$FUNCNAME - $*"
local TMP_FILE unlockList=""

lock_file $LOCK_TRANSACTIONS
for i in $* ; do
TMP_FILE=${LOCK_TRANSACTIONS}.$(grep ".*:$i:$$" $LOCK_TRANSACTIONS | cut
-d : -f 1)
[ -e $TMP_FILE ] && cp $TMP_FILE $i && rm $TMP_FILE
unlock_file $i
cp $LOCK_TRANSACTIONS $LOCK_TRANSACTIONS.new
grep -v ".*:$i:$$" $LOCK_TRANSACTIONS.new >$LOCK_TRANSACTIONS
rm $LOCK_TRANSACTIONS.new
done
unlock_file $LOCK_TRANSACTIONS

}

#---------------------------------------------------------------------
##=item lock_kill_transaction
##
## Stops a transaction. Causes the changes to the file(s) to be
## ignored/undone.
##
#---------------------------------------------------------------------
function lock_kill_transaction()
{
debug "liblock" "$FUNCNAME - $*"
local TMP_FILE

lock_file $LOCK_TRANSACTIONS
for i in $* ; do
TMP_FILE=$(grep `esc_str ".*:$i:$$"` $LOCK_TRANSACTIONS | cut -d : -f 1)
## [ -e $TMP_FILE ] && cp $TMP_FILE $i && rm $TMP_FILE <-- this is the
only diff between commit and kill
unlock_file $i
cp $LOCK_TRANSACTIONS $LOCK_TRANSACTIONS.1
grep -v `esc_str ".*:$i:$$"` $LOCK_TRANSACTIONS.1 >$LOCK_TRANSACTIONS
done
unlock_file $LOCK_TRANSACTIONS
}

#---------------------------------------------------------------------
##=item SYNCHRONIZE
##
## Prevents more than one process from entering a section of code
## at a time
## NOTE: This assumes that two functions of the same name will
## not havethe SYNCHRONIZE command on the same line in different
## files.
##
## Example:
## #!/bin/bash
## echo "Script started with PID=$$
## eval "$SYNCHRONIZE" && {
## echo "PID $$ is in the synched section of code."
## sleep 10
## } && eval "$UNSYNCHRONIZE"
## echo "PID $$ done."
##
## Note: SYNCHRONIZE and UNSYNCHRONIZE must be in the same local
## scope. Nested SYNCHs are not allowed in the same local scopes.
## Local scope usualy being a function.
## (Blocking)
##
#---------------------------------------------------------------------
SYNCHRONIZE='
__llSYNCH_LINE=$LINENO
debug "liblock" "+++ in synch code"
lock_resources "lockfunction" "${FUNCNAME}/${__llSYNCH_LINE}"'

#---------------------------------------------------------------------
##=item UNSYNCHRONIZE
##
## Terminates a section of SYNCHRONIZED code
##
#---------------------------------------------------------------------
UNSYNCHRONIZE='
debug "liblock" "+++ in unsynch code"
unlock_resources "lockfunction" "${FUNCNAME}/${__llSYNCH_LINE}"
unset __llSYNCH_LINE'


######################################################################
##
## LOCK STUFF _PRIVATE_!
##
######################################################################
#---------------------------------------------------------------------
##=item lock_clean_locklist
##
## Removes locks owned by PIDs that don't exist anymore.
##
#---------------------------------------------------------------------
function lock_clean_locklist() #IF YOU CALL THIS I WILL HUNT YOU DOWN
{
debug "liblock" "clean_locklist() - Cleaning lock list"

local locklist_clean=$LOCK_DIR/locklist.clean
local locklist=$LOCK_DIR/locklist

# If this file doesn't exist, then the locklist will never be cleaned
# automatiacly of dead locks
[ -e $locklist_clean ] || touch $locklist_clean

#Obtain a lock, do this manualy as we don't want an infinite loop
# by using lock_file ;)
while touch $locklist.$$.lock
A=$(eval echo $LOCK_DIR/${1//\//^}.*.lock )
B=${A// /}
[ $(( ${#A}-${#B}+1 )) -gt 1 ]
do
for i in $A ; do
B=${i%.lock}
B=${B##*.}
if ! [ -d /proc/$i ] ; then
debug "liblock" "$FUNCNAME - cleaning out $i"
rm $i 2>/dev/null
fi
done
rm $locklist.$$.lock

sleep $(( RANDOM % MAX_SLEEP ))
done
#We now have a lock on the locklist
# We can now mess with it without fear of other cleaners coming around.

#Clean the lock list only if necessary (clean every minute)

filen=`basename $locklist_clean`
if [[ `find "$LOCK_DIR" -name "$filen" -maxdepth 1 -mindepth 1 -mmin +1` ]]
; then

echo "" > ${locklist}.1

for i in `cat $locklist` ; do
debug "liblock" "aftualy looking at locks to clean"
if ! [[ $i ]] ; then continue; fi
PID=`echo $i | cut -d : -f 3`

if [ -d $( echo /proc/$PID) ] ; then
echo "$i" >> ${locklist}.1
else
debug "liblock" "lock_clean_locklist() - Removing $i from list"
fi
done

for i in $LOCK_DIR/*.*.lock ; do
PID=${i%.lock}
PID=${PID##*.}
if ! [ -d $( echo /proc/$PID) ] ; then
debug "liblock" "lock_clean_locklist() - Removing $i"
rm $i
fi
done

mv ${locklist}.1 locklist
date > $locklist_clean
fi
rm $locklist.$$.lock

}




Archive powered by MHonArc 2.6.24.

Top of Page