Public SourceMage Discussion List

[evraire AT tuwg.com]

I guess it's appropriate that this happened on Halloween, but I wanted to 
share my latest horror story.  (Someone else did the same recently and I 
thought it was cool to read. :)  Unfortunately, it's very long: I tend to do 
that when I start writing. :)

First, I'd just like to say I learned my lesson: NEVER do an unnatended 
"sorcery rebuild"!!!

So here goes...  I've been getting these little fidgety problems on and off 
resulting in an environment which is much less stable than Linux should be.  
For instance, I tried using gaim, but everytime I would add a user it would 
crash.  I tried using EveryBuddy, but it would never receive events.  
Yesterday for some reason everytime I used Galeon to fill out form 
information it would crash on sending.

Now, I've very recently started building my system from scratch.  About two 
weeks ago I popped in the "secret" 1001 ISO and, with a lot of help from 
Dufflebunk (thanks!) I finally got a working sorcery with an up-to-date 
grimoire!  Satisfied, I went along my happy way installing everything from 
Gnome2 to ssh.  Never did I rebuild, too weary of what would happen.  
(Especially since there were still issues with dependancies, before sorcery 
0.9 came out.)

Last night, I was tired of all these little problems.  Afterall, if I wanted 
such an unstable environment, I would just run Wintendo!  I finally decided 
it was time for a sorcery rebuild.  So I started by running a "cast fix", 
then once that was completed I did a "sorcery rebuild".

Of course, I'm not that trusting to believe everything would go perfectly 
smoothly, especially with all the glibc talk that's been going on in the 
list, so I dilligently edit the queue removing the candidates for what I 
thought were the more nasty problems should they fail.  With this, I started 
the long procedure of a "sorcery rebuild".

Sorcery was happily chugging along in a window when my girlfriend started 
pestering me to go to bed.  Everything seemed fine and I had removed the 
worse of the bunch from the queue, so I sighed, turned off the monitor and 
headed for bed.

Then Halloween arrived...

Now I've been sick lately, so I woke up around 2am with a coughing fit.  
Feeling bad for waking up my girlfriend and keeping her up, I decided to 
leave the bedroom 'til it passed.  Having nothing better to do, I strolled up 
to the computer (like any good geek would :) to see how things were coming 
along.  I turned on my monitor and was horrified!!!  From then on, my 
coughing didn't seem to bother me so much.

The screensaver was on, but the system was frozen up SOLID!!!  No numlock, no 
mouse, no CTL-ALT-DEL, CTL-ALT-BS, CTL-ALT-ESC... NOTHING!  I couldn't see 
any other choice: I hit the reset button and prayed.

Much as I feared, rather than booting into my gorgeous graphical GDM greeter, 
the horrible red [FAILED] messages started scrolling by and I watched as 
ReiserFS recovered links (good thing I'm using a journaled filesystem!) until 
Linux finally dropped me to a login prompt.  "It's not so bad" I remember 
thinking to myself, "at least I have a login prompt and no kernel panic!"

Throwing caution to the wind, I logged in as root.  The first thing I did was 
check the /var/log/sorcery/activity to see what succeeded and what failed.  I 
was scrolling down the list and much to my surprised, it looked like most 
everything had succeeded and what failed were for obvious reasons (like 
missing source file).  Then BAM!!!  The last few lines of the log where filed 
with '@' characters in reverse color.  "That can't be good" I said to no one 
in the room.

This may sound stupid, but given the recent thread on glibc, I decided to run 
ldconfig with no change.

I tried a "cast --fix", but that did nothing obvious.  Since I was up, I 
decided I'd do another "sorcery rebuild", sticking around this time to see 
where things went to hell, but then it occured to me one of those blood red 
[FAILED] meant my network was down.  A quick "ifconfig" showed this was the 
case.  I had put the "linux" spell on hold (and manually removed it from the 
queue... what the hell?) but I decided to make sure my module was still 
there... it was!  I knew it wouldn't work, but I wanted to see what it said 
so I did "modprobe via-rhine" and, instead of the expected unresolved 
symbols, I see a pile of text.

Now remember, I'm coughing my head off and it's like 2:30 in the morning, so 
it took me awhile before I realized it was snippets (comments to be more 
precise) from a source file.  There was a name there (something Owens), and 
using my laptop I was able to discover he was responsible for some kernel 
related stuff and modutils related stuff.  Doing an lsmod gave me the same 
result (at the time I didn't realize most of these commands were symlinks to 
insmod. :)  I realized modutils was screwed.

Having no network, I popped in the 1001 ISO and rebooted.  Thank God I've got 
that one because it uses the same kernel as me (2.4.19)!  I copied over what 
I thought were the necessary binaries to get modutils running again (making a 
backup of mine, just in case).  I tried "modprobe via-rhine" again: SUCCESS!  
I was thrilled!  I tried to "cast modutils" to get my own copy running again 
and right away it complained about the "ps" command not being found.  I 
CTL-C'd out of the compilation, copied over /bin/ps and libproc-whatever.so 
tried it again: success!  I had my own copy of modutils running again.

Realizing how important modutils really is now, I thought that maybe fixing 
it might alleviate a lot of problems, so I typed "shutdown -r now", popped 
out the CD and crossed my fingers.  At best, I was hoping to see a lot less 
[FAILED] messages and getting back the prompt to further fix up my system, 
but to my ecstatic surprise my beautiful GDM returned!  I quickly logged in 
as my normal user and everything seemed normal once again.

Deciding to wait 'til I could monitor the PC before doing another "sorcery 
rebuild", I shut down the PC and went to bed close to 3am satisfied.  I 
realized how, had the same happened under Wintendo, it most certainly 
would've meant a complete reinstallation.  I also thought about sorcery and 
how, being implemented in bash, it was so easy to use to fix my system.  I 
also realized how Linux can usually be fixed no matter what happens.

I'm still not sure what caused the original crash (I'm starting to suspect 
hardware though), and I've still got this mysterious "win" file appearing on 
my gnome-desktop everytime I log in, but at least I have a running system.  
Here are some lessons I learned in less than an hour:

1) as I said before, NEVER do a "sorcery rebuild" unnattended, much less a 
"sorcery system-update"!

2) because SourceMage relies on the Net so much, it makes sense to compile 
your network driver directly in the kernel.  This is what I had before, but 
the new networking scripts didn't support this well so I switched to the 
module.  This should be fixed in the neworking scripts.

3) ALWAYS have an ISO nearby, especially one with the same kernel as you're 
running.  It will certainly be nice when the RC1 ISO is released (and I hope 
this one works with USB keyboards ;)

  Anyway, is anyone still reading this?  I hope you didn't find my scary 
story too boring. :)  (It was scary to me anyway :) Sorry for the long post!

  Jonathan Evraire
  evraire at tuwg.com

PS: this may have something to do with bug 1523, since the first nano that 
pops up doesn't accept any input and needs to be killed, although I don't see 
how this could freeze the rest of the system.




--------------------------------------------------------------
Message sent via Toad-One
http://www.toad-one.org

.