Skip to Content.
Sympa Menu

sm-commit - [SM-Commit] GIT changes to devel-sobukus-protection sorcery by Thomas Orgis (0c944ad58ccf09d2a666076dec0021c46178ea91)

sm-commit AT lists.ibiblio.org

Subject: Source Mage code commit list

List archive

Chronological Thread  
  • From: Thomas Orgis <scm AT sourcemage.org>
  • To: sm-commit AT lists.ibiblio.org
  • Subject: [SM-Commit] GIT changes to devel-sobukus-protection sorcery by Thomas Orgis (0c944ad58ccf09d2a666076dec0021c46178ea91)
  • Date: Fri, 28 Aug 2020 18:17:57 +0000

GIT changes to devel-sobukus-protection sorcery by Thomas Orgis
<sobukus AT sourcemage.org>:

ChangeLog | 6 ++++++
usr/sbin/cast | 7 +++++++
var/lib/sorcery/modules/libdepengine | 33 ++++++++++++++++++++++++++++-----
var/lib/sorcery/modules/libmisc | 12 +++++++++---
var/lib/sorcery/protected | 8 ++++++++
5 files changed, 58 insertions(+), 8 deletions(-)

New commits:
commit 0c944ad58ccf09d2a666076dec0021c46178ea91
Author: Thomas Orgis <sobukus AT sourcemage.org>
Commit: Thomas Orgis <sobukus AT sourcemage.org>

libdepengine: add short cut for grey dependency loops

The repeated dive into the same branches of the dependency tree
can be very wasteful. Lots of calls to recurse_depends can be
avoided when it is already known that a spell recursion will
turn up aborted because of a dependency loop (grey dependency).

This assumes that the situation only possibly changes when a spell
is actually cast.

I am not sure if my analysis is correct. I had the rather fatal
situation of a sorcery rebuild spinning for hours in a rendering
of dependency hell trying to decide what to build next after some
successful initial casts. This patch avoids that situation at least.

I also for some time whitnessed an immense delay on removing dependees
of a failed spell. I am not sure if this is now gone or could still
reappear. The whole dependency engine needs an overhaul in all its
wastefulness.

commit 9811c7f9cc31a7d77b18ecfd76b95f36ddcdc898
Author: Thomas Orgis <sobukus AT sourcemage.org>
Commit: Thomas Orgis <sobukus AT sourcemage.org>

libdepengine: more debugging

commit b6cfef16e0abbba6ff8d1b16e58a1d5dacbd580c
Author: Thomas Orgis <sobukus AT sourcemage.org>
Commit: Thomas Orgis <sobukus AT sourcemage.org>

libmisc: more robust debugging of parallel work

This prints out the whole debugging line on one go to reduce interference
of printouts from multiple sorcery processes (subshells for cast passes).
Debugging is a nightmare when data within a line is already interleaved!
This is still not proper, but we don't want to bring things to a crawl by
i/o locking. An alternative might be to write output to per-process files.
But then, we also should implement timestamps in the output. Pro-tip: use
ts from moreutils in a tail -f | ts configuration to put them on closely
after the fact.

Also, to actually tell the differing subshells of the cast process apart,
the variable $$ is not enough. Bash offers an alternative. This is used
with $$ as fallback now. We may want to check what to do for other shells
that we want to be able to run sorcery properly.

commit 09b31a32fb004e9f7daeb39e3662f84218e80451
Author: Thomas Orgis <sobukus AT sourcemage.org>
Commit: Thomas Orgis <sobukus AT sourcemage.org>

cast: more debugging, tell differing subshells apart

commit 6a810b9b9b44eb0aa58a831998ce91b243d89897
Author: Thomas Orgis <sobukus AT sourcemage.org>
Commit: Thomas Orgis <sobukus AT sourcemage.org>

protected: also adding guile stuff

GNU Make has a possible dependency on guile. If you enable it and upgrade
guile, you could break make and have a hosed system again.

commit 96b4be0743c9ab8f0c3e00cf3ad50ae2aa13d635
Author: Thomas Orgis <sobukus AT sourcemage.org>
Commit: Thomas Orgis <sobukus AT sourcemage.org>

protected: add more core libs that hose the system on updates

Bash is broken when you remove libreadline. GCC is broken when you remove
libisl.
All hell is broken if you remove ICU libs. Just don't do that.

This fixes bug 16112. A proper fix for the whole library removal business
is
still to come. Hopefully (see bug 16100 for one recent approach).

diff --git a/ChangeLog b/ChangeLog
index 003a8e2..5f4207c 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,9 @@
+2020-09-19 Thomas Orgis <sobukus AT sourcemage.org>
+ * protected: added readline, ncurses, isl, icu (bug 16112)
+ * protected: also added guile libs, in case you make make
+ depend on it (untested if normal make operation still works without
+ other guile files, though)
+
2020-08-05 Ismael Luceno <ismael AT sourcemage.org>
* protected: Added libintl
* libsorcery: Removed -DPIC from prelink optimization option
diff --git a/usr/sbin/cast b/usr/sbin/cast
index 09204f0..51441ef 100755
--- a/usr/sbin/cast
+++ b/usr/sbin/cast
@@ -435,22 +435,26 @@ pass_two() {
MINUS_K=yes
if [ "$SEPARATE" ]; then
( CAST_PASS="three"
+ debug "cast" "pass $CAST_PASS subshell (separate)"
depengine_entry_point "$SPELLS" "$SPELLS_TO_CAST"
if [[ $SCREEN_NAME ]] ; then
screen_notify "$SCREEN_NAME" "Done downloading"
screen_kill_window "$SCREEN_NAME" $SCREEN_SUMMON_WIN
fi
+ debug "cast" "pass $CAST_PASS subshell done"
touch $TMP_DIR/pass_three.done
)
else
( CAST_PASS="three"
trap "exit 1" INT TERM # this keep a rather nasty message
# from appearing if we get killed
+ debug "cast" "pass $CAST_PASS subshell"
depengine_entry_point "$SPELLS" "$SPELLS_TO_CAST"
if [[ $SCREEN_NAME ]] ; then
screen_notify "$SCREEN_NAME" "Done downloading"
screen_kill_window "$SCREEN_NAME" $SCREEN_SUMMON_WIN
fi
+ debug "cast" "pass $CAST_PASS subshell done"
touch $TMP_DIR/pass_three.done &>/dev/null
) &
fi
@@ -459,7 +463,9 @@ pass_two() {

(
CAST_PASS="four"
+ debug "cast" "pass $CAST_PASS subshell"
depengine_entry_point "$SPELLS" "$SPELLS_TO_CAST"
+ debug "cast" "pass $CAST_PASS subshell done"
touch $TMP_DIR/pass_four.done
)

@@ -610,6 +616,7 @@ pass_four() {
else
cast_spell $*
fi
+ debug "cast" "pass_four done for $SPELL"
}


diff --git a/var/lib/sorcery/modules/libdepengine
b/var/lib/sorcery/modules/libdepengine
index df4b9fe..6ec0e1d 100755
--- a/var/lib/sorcery/modules/libdepengine
+++ b/var/lib/sorcery/modules/libdepengine
@@ -45,6 +45,11 @@ depengine_entry_point() {
local spell pending_list spell_status rc
local spell_depends spell_sub_depends spell_rsub_depends

+ # Shortcut for avoiding grey dependency loop detection,
+ # see depengine_cast_engine(). Empty list to begin with.
+ # This is a global variable!
+ dpgn_grey_loop_spells=
+
for spell in $(hash_get_table_fields dep_f_hash); do
dpgn_set_spell_color $spell white
done
@@ -101,14 +106,28 @@ depengine_cast_engine() {
local org_color=$(dpgn_get_spell_color $spell)
local rc

- dpgn_set_spell_color $spell grey
-
- recurse_depends $spell $in_trigger
- rc=$?
- if [[ $in_trigger != 0 ]] && [[ $rc == 2 ]] ; then
+ # Dependency recursion can hit a grey dependency loop many times
+ # and waste a lot of CPU cycles on that. The shortcut turns a pathological
+ # seemingly endless loop during a whole-system rebuild (about 1000 spells)
+ # into a non-issue.
+ if [[ $in_trigger != 0 ]] && list_find "$looped_spell_list" $spell; then
+ debug "libdepengine" "grey loop shortcut for $spell"
dpgn_set_spell_color $spell $org_color
return 2
+ else
+ dpgn_set_spell_color $spell grey
+ recurse_depends $spell $in_trigger
+ rc=$?
+ if [[ $in_trigger != 0 ]] && [[ $rc == 2 ]] ; then
+ debug "libdepengine" "got a grey loop for $spell"
+ dpgn_set_spell_color $spell $org_color
+ list_add looped_spell_list $spell
+ return 2
+ fi
fi
+ # The grey loop detection shall start fresh on the next round of recursions
+ # once an actual action takes place that may change the situation.
+ dpgn_grey_loop_spells=

spell_status=$(dpgn_get_spell_color $spell)
if [[ $spell_status == grey ]] && [[ $rc == 0 ]] ; then
@@ -147,6 +166,7 @@ depengine_cast_engine() {

dpgn_set_spell_color $spell black:$rc

+ debug "libdepengine" "done with depengine_cast_engine $@"
return $rc
}

@@ -225,6 +245,7 @@ recurse_depends() {
# catches those cases where we dont, and doesn't worry too much
# about a failure
if [[ $rc == 0 ]] ; then
+ debug "libdepengine" "build triggerers of $spell"
local hash_triggerers
hash_get_ref trg_r_hash $spell:cast_self hash_triggerers
for triggerer in $hash_triggerers; do
@@ -236,6 +257,7 @@ recurse_depends() {
fi
done
fi
+ debug "libdepengine" "done with recurse_depends $@"
return $rc
}

@@ -344,6 +366,7 @@ execute_triggers() {
## frontend to cast spells, unregisters pending triggers if any
#------------------------------------------------------------------------
dpgn_cast_spell_front() {
+ $STD_DEBUG
if ! [[ $CAST_PASS ]] ; then
message "ERROR: Missing cast pass variable!!"
exit 1
diff --git a/var/lib/sorcery/modules/libmisc b/var/lib/sorcery/modules/libmisc
index b7116a0..fdfa2f7 100755
--- a/var/lib/sorcery/modules/libmisc
+++ b/var/lib/sorcery/modules/libmisc
@@ -230,12 +230,18 @@ debug() {
debugVar=${debugVar//[^a-zA-Z0-9_]/_}
local i
if [[ ${!debugVar} != "no" ]] ; then
- echo -n "$1($$): " >>$DEBUG
+ # Construct line and print out in one go, to lessen interference from
+ # parallel work, and some strange interference between printouts of one
+ # and the same process that I do not understand (yet, or ever).
+ # In ( ... ) subshells, $$ still refers to the mother shell, trying
+ # the bashism with fallback to be able to tell apart parallel cast passes
+ # three and four.
+ local line="$1(${BASHPID:-$$}):"
shift
for i in "$@" ; do
- echo -n " \"$i\"" >>$DEBUG
+ line="$line \"$i\""
done
- echo >>$DEBUG
+ printf "%s\n" "$line" >>$DEBUG
fi

true
diff --git a/var/lib/sorcery/protected b/var/lib/sorcery/protected
index 11704bf..145c1ce 100755
--- a/var/lib/sorcery/protected
+++ b/var/lib/sorcery/protected
@@ -71,3 +71,11 @@
^/usr/share/info/dir$
^/bin/mount$
^/lib/libmount\.so\.
+^/lib/libreadline\.so\.
+^/lib/libhistory\.so\.
+^/lib/libncurses\.so\.
+^/lib/libncursesw\.so\.
+^/usr/lib/libisl\.so\.
+^/usr/lib/libicu.*\.so\.
+^/usr/lib/libguile-.*\.so\.
+^/usr/lib/libguilereadline-.*\.so\.



  • [SM-Commit] GIT changes to devel-sobukus-protection sorcery by Thomas Orgis (0c944ad58ccf09d2a666076dec0021c46178ea91), Thomas Orgis, 08/28/2020

Archive powered by MHonArc 2.6.24.

Top of Page