Skip to Content.
Sympa Menu

sm-commit - [SM-Commit] GIT changes to master sorcery by Thomas Orgis (18a0f4352f1ecc392d8b7ac1fba2e2502a55290b)

sm-commit AT lists.ibiblio.org

Subject: Source Mage code commit list

List archive

Chronological Thread  
  • From: Thomas Orgis <scm AT sourcemage.org>
  • To: sm-commit AT lists.ibiblio.org
  • Subject: [SM-Commit] GIT changes to master sorcery by Thomas Orgis (18a0f4352f1ecc392d8b7ac1fba2e2502a55290b)
  • Date: Fri, 26 Feb 2021 11:30:58 +0000

GIT changes to master sorcery by Thomas Orgis <sobukus AT sourcemage.org>:

ChangeLog | 6 ++++++
usr/sbin/cast | 7 +++++++
var/lib/sorcery/modules/libdepengine | 29 ++++++++++++++++++++++++-----
var/lib/sorcery/modules/libgrimoire | 12 ++++++++++--
var/lib/sorcery/protected | 2 ++
5 files changed, 49 insertions(+), 7 deletions(-)

New commits:
commit 18a0f4352f1ecc392d8b7ac1fba2e2502a55290b
Author: Thomas Orgis <sobukus AT sourcemage.org>
Commit: Thomas Orgis <sobukus AT sourcemage.org>

protected: also /bin/login, got bitten by failed shadow install

commit f64e9a892b0c57f763137718a652ccde048d5fca
Author: Thomas Orgis <sobukus AT sourcemage.org>
Commit: Thomas Orgis <sobukus AT sourcemage.org>

libgrimoire: half of RAM as default size of TMPFS source directories

commit e7ab3e8f6918b496012cb51307c2243b6d95da22
Author: Thomas Orgis <sobukus AT sourcemage.org>
Commit: Thomas Orgis <sobukus AT sourcemage.org>

protected: egrep is needed in sorcery itself, even, but also installs

commit 4b741dfb06c3f332bdb6f16531d449f22e1e0163
Author: Thomas Orgis <sobukus AT sourcemage.org>
Commit: Thomas Orgis <sobukus AT sourcemage.org>

libdepengine: add short cut for grey dependency loops

The repeated dive into the same branches of the dependency tree
can be very wasteful. Lots of calls to recurse_depends can be
avoided when it is already known that a spell recursion will
turn up aborted because of a dependency loop (grey dependency).

This assumes that the situation only possibly changes when a spell
is actually cast.

I am not sure if my analysis is correct. I had the rather fatal
situation of a sorcery rebuild spinning for hours in a rendering
of dependency hell trying to decide what to build next after some
successful initial casts. This patch avoids that situation at least.

I also for some time whitnessed an immense delay on removing dependees
of a failed spell. I am not sure if this is now gone or could still
reappear. The whole dependency engine needs an overhaul in all its
wastefulness.

commit aae3c866337f7325d7703d99989ab13979a859f9
Author: Thomas Orgis <sobukus AT sourcemage.org>
Commit: Thomas Orgis <sobukus AT sourcemage.org>

cast: more debugging, tell differing subshells apart

diff --git a/ChangeLog b/ChangeLog
index 5f4207c..6cf9156 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,9 @@
+2020-10-01 Thomas Orgis <sobukus AT sourcemage.org>
+ * protected: add /bin/login (!!)
+
+2020-09-30 Thomas Orgis <sobukus AT sourcemage.org>
+ * libgrimoire: default to default size of tmpfs (half of RAM)
+
2020-09-19 Thomas Orgis <sobukus AT sourcemage.org>
* protected: added readline, ncurses, isl, icu (bug 16112)
* protected: also added guile libs, in case you make make
diff --git a/usr/sbin/cast b/usr/sbin/cast
index 09204f0..51441ef 100755
--- a/usr/sbin/cast
+++ b/usr/sbin/cast
@@ -435,22 +435,26 @@ pass_two() {
MINUS_K=yes
if [ "$SEPARATE" ]; then
( CAST_PASS="three"
+ debug "cast" "pass $CAST_PASS subshell (separate)"
depengine_entry_point "$SPELLS" "$SPELLS_TO_CAST"
if [[ $SCREEN_NAME ]] ; then
screen_notify "$SCREEN_NAME" "Done downloading"
screen_kill_window "$SCREEN_NAME" $SCREEN_SUMMON_WIN
fi
+ debug "cast" "pass $CAST_PASS subshell done"
touch $TMP_DIR/pass_three.done
)
else
( CAST_PASS="three"
trap "exit 1" INT TERM # this keep a rather nasty message
# from appearing if we get killed
+ debug "cast" "pass $CAST_PASS subshell"
depengine_entry_point "$SPELLS" "$SPELLS_TO_CAST"
if [[ $SCREEN_NAME ]] ; then
screen_notify "$SCREEN_NAME" "Done downloading"
screen_kill_window "$SCREEN_NAME" $SCREEN_SUMMON_WIN
fi
+ debug "cast" "pass $CAST_PASS subshell done"
touch $TMP_DIR/pass_three.done &>/dev/null
) &
fi
@@ -459,7 +463,9 @@ pass_two() {

(
CAST_PASS="four"
+ debug "cast" "pass $CAST_PASS subshell"
depengine_entry_point "$SPELLS" "$SPELLS_TO_CAST"
+ debug "cast" "pass $CAST_PASS subshell done"
touch $TMP_DIR/pass_four.done
)

@@ -610,6 +616,7 @@ pass_four() {
else
cast_spell $*
fi
+ debug "cast" "pass_four done for $SPELL"
}


diff --git a/var/lib/sorcery/modules/libdepengine
b/var/lib/sorcery/modules/libdepengine
index 8644ab5..02e9b32 100755
--- a/var/lib/sorcery/modules/libdepengine
+++ b/var/lib/sorcery/modules/libdepengine
@@ -45,6 +45,11 @@ depengine_entry_point() {
local spell pending_list spell_status rc
local spell_depends spell_sub_depends spell_rsub_depends

+ # Shortcut for avoiding grey dependency loop detection,
+ # see depengine_cast_engine(). Empty list to begin with.
+ # This is a global variable!
+ dpgn_grey_loop_spells=
+
for spell in $(hash_get_table_fields dep_f_hash); do
dpgn_set_spell_color $spell white
done
@@ -101,14 +106,28 @@ depengine_cast_engine() {
local org_color=$(dpgn_get_spell_color $spell)
local rc

- dpgn_set_spell_color $spell grey
-
- recurse_depends $spell $in_trigger
- rc=$?
- if [[ $in_trigger != 0 ]] && [[ $rc == 2 ]] ; then
+ # Dependency recursion can hit a grey dependency loop many times
+ # and waste a lot of CPU cycles on that. The shortcut turns a pathological
+ # seemingly endless loop during a whole-system rebuild (about 1000 spells)
+ # into a non-issue.
+ if [[ $in_trigger != 0 ]] && list_find "$looped_spell_list" $spell; then
+ debug "libdepengine" "grey loop shortcut for $spell"
dpgn_set_spell_color $spell $org_color
return 2
+ else
+ dpgn_set_spell_color $spell grey
+ recurse_depends $spell $in_trigger
+ rc=$?
+ if [[ $in_trigger != 0 ]] && [[ $rc == 2 ]] ; then
+ debug "libdepengine" "got a grey loop for $spell"
+ dpgn_set_spell_color $spell $org_color
+ list_add looped_spell_list $spell
+ return 2
+ fi
fi
+ # The grey loop detection shall start fresh on the next round of recursions
+ # once an actual action takes place that may change the situation.
+ dpgn_grey_loop_spells=

spell_status=$(dpgn_get_spell_color $spell)
if [[ $spell_status == grey ]] && [[ $rc == 0 ]] ; then
diff --git a/var/lib/sorcery/modules/libgrimoire
b/var/lib/sorcery/modules/libgrimoire
index bc69995..7389b54 100755
--- a/var/lib/sorcery/modules/libgrimoire
+++ b/var/lib/sorcery/modules/libgrimoire
@@ -39,14 +39,22 @@ real_mk_source_dir() {
local NEW_DIR=${NEW_DIR:=$SOURCE_DIRECTORY}

local SIZE=$2
- local SIZE=${SIZE:=2g}
+ # See bug 15946. Even simpler approach: Just specify no size, wich
+ # also defaults to half of RAM in Linux.
+ # TODO: Use given size to decide for TMPFS or not, early bail out
+ # if not enough disk.
+ local sizepar
+ if [[ -n "$SIZE" ]]; then
+ sizepar="size=$SIZE,"
+ fi
+ # Could also think about inode count being necessary.

if [ -n "$NEW_DIR" ]; then

rm_source_dir $NEW_DIR
mkdir -p $NEW_DIR &&
if [[ $TMPFS == on ]]; then
- mount -o size=$SIZE,nr_inodes=1m -t tmpfs tmpfs $NEW_DIR
+ mount -o "$sizepar"nr_inodes=1m -t tmpfs tmpfs $NEW_DIR
fi
fi

diff --git a/var/lib/sorcery/protected b/var/lib/sorcery/protected
index 145c1ce..56dbe57 100755
--- a/var/lib/sorcery/protected
+++ b/var/lib/sorcery/protected
@@ -3,6 +3,7 @@
^/bin/mawk$
^/bin/sed$
^/bin/bash$
+^/bin/login$
^/bin/cat$
^/bin/cp$
^/bin/date$
@@ -60,6 +61,7 @@
^/usr/bin/file$
^/usr/bin/find$
^/usr/bin/grep$
+^/usr/bin/egrep$
^/usr/bin/make$
^/usr/bin/ranlib$
^/usr/bin/wget$



  • [SM-Commit] GIT changes to master sorcery by Thomas Orgis (18a0f4352f1ecc392d8b7ac1fba2e2502a55290b), Thomas Orgis, 02/26/2021

Archive powered by MHonArc 2.6.24.

Top of Page