Source Mage code commit list

[Thomas Orgis <scm AT sourcemage.org>]

GIT changes to devel-sobukus-protection sorcery by Thomas Orgis 
<sobukus AT sourcemage.org>:

 ChangeLog                            |    6 ++++++
 usr/sbin/cast                        |    7 +++++++
 var/lib/sorcery/modules/libdepengine |   33 ++++++++++++++++++++++++++++-----
 var/lib/sorcery/modules/libmisc      |   12 +++++++++---
 var/lib/sorcery/protected            |    8 ++++++++
 5 files changed, 58 insertions(+), 8 deletions(-)

New commits:
commit 0c944ad58ccf09d2a666076dec0021c46178ea91
Author: Thomas Orgis <sobukus AT sourcemage.org>
Commit: Thomas Orgis <sobukus AT sourcemage.org>

    libdepengine: add short cut for grey dependency loops
    
    The repeated dive into the same branches of the dependency tree
    can be very wasteful. Lots of calls to recurse_depends can be
    avoided when it is already known that a spell recursion will
    turn up aborted because of a dependency loop (grey dependency).
    
    This assumes that the situation only possibly changes when a spell
    is actually cast.
    
    I am not sure if my analysis is correct. I had the rather fatal
    situation of a sorcery rebuild spinning for hours in a rendering
    of dependency hell trying to decide what to build next after some
    successful initial casts. This patch avoids that situation at least.
    
    I also for some time whitnessed an immense delay on removing dependees
    of a failed spell. I am not sure if this is now gone or could still
    reappear. The whole dependency engine needs an overhaul in all its
    wastefulness.

commit 9811c7f9cc31a7d77b18ecfd76b95f36ddcdc898
Author: Thomas Orgis <sobukus AT sourcemage.org>
Commit: Thomas Orgis <sobukus AT sourcemage.org>

    libdepengine: more debugging

commit b6cfef16e0abbba6ff8d1b16e58a1d5dacbd580c
Author: Thomas Orgis <sobukus AT sourcemage.org>
Commit: Thomas Orgis <sobukus AT sourcemage.org>

    libmisc: more robust debugging of parallel work
    
    This prints out the whole debugging line on one go to reduce interference
    of printouts from multiple sorcery processes (subshells for cast passes).
    Debugging is a nightmare when data within a line is already interleaved!
    This is still not proper, but we don't want to bring things to a crawl by
    i/o locking. An alternative might be to write output to per-process files.
    But then, we also should implement timestamps in the output. Pro-tip: use
    ts from moreutils in a tail -f | ts configuration to put them on closely
    after the fact.
    
    Also, to actually tell the differing subshells of the cast process apart,
    the variable $$ is not enough. Bash offers an alternative. This is used
    with $$ as fallback now. We may want to check what to do for other shells
    that we want to be able to run sorcery properly.

commit 09b31a32fb004e9f7daeb39e3662f84218e80451
Author: Thomas Orgis <sobukus AT sourcemage.org>
Commit: Thomas Orgis <sobukus AT sourcemage.org>

    cast: more debugging, tell differing subshells apart

commit 6a810b9b9b44eb0aa58a831998ce91b243d89897
Author: Thomas Orgis <sobukus AT sourcemage.org>
Commit: Thomas Orgis <sobukus AT sourcemage.org>

    protected: also adding guile stuff
    
    GNU Make has a possible dependency on guile. If you enable it and upgrade
    guile, you could break make and have a hosed system again.

commit 96b4be0743c9ab8f0c3e00cf3ad50ae2aa13d635
Author: Thomas Orgis <sobukus AT sourcemage.org>
Commit: Thomas Orgis <sobukus AT sourcemage.org>

    protected: add more core libs that hose the system on updates
    
    Bash is broken when you remove libreadline. GCC is broken when you remove 
libisl.
    All hell is broken if you remove ICU libs. Just don't do that.
    
    This fixes bug 16112. A proper fix for the whole library removal business 
is
    still to come. Hopefully (see bug 16100 for one recent approach).

diff --git a/ChangeLog b/ChangeLog
index 003a8e2..5f4207c 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,9 @@
+2020-09-19 Thomas Orgis <sobukus AT sourcemage.org>
+       * protected: added readline, ncurses, isl, icu (bug 16112)
+       * protected: also added guile libs, in case you make make
+         depend on it (untested if normal make operation still works without
+         other guile files, though)
+
 2020-08-05 Ismael Luceno <ismael AT sourcemage.org>
        * protected: Added libintl
        * libsorcery: Removed -DPIC from prelink optimization option
diff --git a/usr/sbin/cast b/usr/sbin/cast
index 09204f0..51441ef 100755
--- a/usr/sbin/cast
+++ b/usr/sbin/cast
@@ -435,22 +435,26 @@ pass_two()  {
   MINUS_K=yes
   if [ "$SEPARATE" ]; then
     ( CAST_PASS="three"
+      debug "cast" "pass $CAST_PASS subshell (separate)"
       depengine_entry_point "$SPELLS" "$SPELLS_TO_CAST"
       if [[ $SCREEN_NAME ]] ; then
         screen_notify "$SCREEN_NAME" "Done downloading"
         screen_kill_window "$SCREEN_NAME" $SCREEN_SUMMON_WIN
       fi
+      debug "cast" "pass $CAST_PASS subshell done"
       touch $TMP_DIR/pass_three.done
     )
   else
     ( CAST_PASS="three"
       trap "exit 1" INT TERM # this keep a rather nasty message
                                     # from appearing if we get killed
+      debug "cast" "pass $CAST_PASS subshell"
       depengine_entry_point "$SPELLS" "$SPELLS_TO_CAST"
       if [[ $SCREEN_NAME ]] ; then
         screen_notify "$SCREEN_NAME" "Done downloading"
         screen_kill_window "$SCREEN_NAME" $SCREEN_SUMMON_WIN
       fi
+      debug "cast" "pass $CAST_PASS subshell done"
       touch $TMP_DIR/pass_three.done &>/dev/null
     ) &
   fi
@@ -459,7 +463,9 @@ pass_two()  {
 
   (
     CAST_PASS="four"
+    debug "cast" "pass $CAST_PASS subshell"
     depengine_entry_point "$SPELLS" "$SPELLS_TO_CAST"
+    debug "cast" "pass $CAST_PASS subshell done"
     touch $TMP_DIR/pass_four.done
   )
 
@@ -610,6 +616,7 @@ pass_four()  {
   else
     cast_spell $*
   fi
+  debug "cast" "pass_four done for $SPELL"
 }
 
 
diff --git a/var/lib/sorcery/modules/libdepengine 
b/var/lib/sorcery/modules/libdepengine
index df4b9fe..6ec0e1d 100755
--- a/var/lib/sorcery/modules/libdepengine
+++ b/var/lib/sorcery/modules/libdepengine
@@ -45,6 +45,11 @@ depengine_entry_point() {
   local spell pending_list spell_status rc
   local spell_depends spell_sub_depends spell_rsub_depends
 
+  # Shortcut for avoiding grey dependency loop detection,
+  # see depengine_cast_engine(). Empty list to begin with.
+  # This is a global variable!
+  dpgn_grey_loop_spells=
+
   for spell in $(hash_get_table_fields dep_f_hash); do
     dpgn_set_spell_color $spell white
   done
@@ -101,14 +106,28 @@ depengine_cast_engine() {
   local org_color=$(dpgn_get_spell_color $spell)
   local rc
 
-  dpgn_set_spell_color $spell grey
-
-  recurse_depends $spell $in_trigger
-  rc=$?
-  if [[ $in_trigger != 0 ]] && [[ $rc == 2 ]] ; then
+  # Dependency recursion can hit a grey dependency loop many times
+  # and waste a lot of CPU cycles on that. The shortcut turns a pathological
+  # seemingly endless loop during a whole-system rebuild (about 1000 spells)
+  # into a non-issue.
+  if [[ $in_trigger != 0 ]] && list_find "$looped_spell_list" $spell; then
+    debug "libdepengine" "grey loop shortcut for $spell"
     dpgn_set_spell_color $spell $org_color
     return 2
+  else
+    dpgn_set_spell_color $spell grey
+    recurse_depends $spell $in_trigger
+    rc=$?
+    if [[ $in_trigger != 0 ]] && [[ $rc == 2 ]] ; then
+      debug "libdepengine" "got a grey loop for $spell"
+      dpgn_set_spell_color $spell $org_color
+      list_add looped_spell_list $spell
+      return 2
+    fi
   fi
+  # The grey loop detection shall start fresh on the next round of recursions
+  # once an actual action takes place that may change the situation.
+  dpgn_grey_loop_spells=
 
   spell_status=$(dpgn_get_spell_color $spell)
   if [[ $spell_status == grey ]] && [[ $rc == 0 ]] ; then
@@ -147,6 +166,7 @@ depengine_cast_engine() {
 
   dpgn_set_spell_color $spell black:$rc
 
+  debug "libdepengine" "done with depengine_cast_engine $@"
   return $rc
 }
 
@@ -225,6 +245,7 @@ recurse_depends() {
   # catches those cases where we dont, and doesn't worry too much
   # about a failure
   if [[ $rc == 0 ]] ; then
+    debug "libdepengine" "build triggerers of $spell"
     local hash_triggerers
     hash_get_ref trg_r_hash $spell:cast_self hash_triggerers
     for triggerer in $hash_triggerers; do
@@ -236,6 +257,7 @@ recurse_depends() {
       fi
     done
   fi
+  debug "libdepengine" "done with recurse_depends $@"
   return $rc
 }
 
@@ -344,6 +366,7 @@ execute_triggers() {
 ## frontend to cast spells, unregisters pending triggers if any
 #------------------------------------------------------------------------
 dpgn_cast_spell_front() {
+  $STD_DEBUG
   if ! [[ $CAST_PASS ]] ; then
     message "ERROR: Missing cast pass variable!!"
     exit 1
diff --git a/var/lib/sorcery/modules/libmisc b/var/lib/sorcery/modules/libmisc
index b7116a0..fdfa2f7 100755
--- a/var/lib/sorcery/modules/libmisc
+++ b/var/lib/sorcery/modules/libmisc
@@ -230,12 +230,18 @@ debug()  {
   debugVar=${debugVar//[^a-zA-Z0-9_]/_}
   local i
   if [[ ${!debugVar} != "no" ]] ; then
-    echo -n "$1($$): " >>$DEBUG
+    # Construct line and print out in one go, to lessen interference from
+    # parallel work, and some strange interference between printouts of one
+    # and the same process that I do not understand (yet, or ever).
+    # In ( ... ) subshells, $$ still refers to the mother shell, trying
+    # the bashism with fallback to be able to tell apart parallel cast passes
+    # three and four.
+    local line="$1(${BASHPID:-$$}):"
     shift
     for i in "$@" ; do
-      echo -n " \"$i\"" >>$DEBUG
+      line="$line \"$i\""
     done
-    echo >>$DEBUG
+    printf "%s\n" "$line" >>$DEBUG
   fi
 
   true
diff --git a/var/lib/sorcery/protected b/var/lib/sorcery/protected
index 11704bf..145c1ce 100755
--- a/var/lib/sorcery/protected
+++ b/var/lib/sorcery/protected
@@ -71,3 +71,11 @@
 ^/usr/share/info/dir$
 ^/bin/mount$
 ^/lib/libmount\.so\.
+^/lib/libreadline\.so\.
+^/lib/libhistory\.so\.
+^/lib/libncurses\.so\.
+^/lib/libncursesw\.so\.
+^/usr/lib/libisl\.so\.
+^/usr/lib/libicu.*\.so\.
+^/usr/lib/libguile-.*\.so\.
+^/usr/lib/libguilereadline-.*\.so\.