Internetworkers: http://www.ibiblio.org/internetworkers/

["Tony Spencer" <tony AT tonyspencer.com>]

Thanks so much for the ideas Sil.

I think it may be that the firewall is to blame.  I think it may have been
configured incorrectly.  I installed the latest version, configured it, and
started testing ports and it seems to be working.  I haven't had to reboot
since then and the firewall is no longer reporting 'offline' every 5
minutes.

The funky timestamps were really worrying me that the box had been hacked
and someone did a sloppy job of editing the logs.  However, I have now
noticed this phenomenon across many logs so I think you must be correct
about the assumption that the system is updating its time from a timeserver.

Crossing fingers.

> -----Original Message-----
> From: Sil Greene [mailto:quack AT ibiblio.org] 
> Sent: Wednesday, December 01, 2004 5:19 PM
> To: Internetworkers: http://www.ibiblio.org/internetworkers/
> Subject: RE: [internetworkers] Server Disappears Sporadically
> 
> 
> Tony,
> 
> I'm not a firewall expert, and I haven't worked with APF 
> before.  With 
> that said, I'm not seeing the APF logs that indicate it's 
> opening inbound 
> ports for HTTP, FTP, and other traffic.  It does seem to be 
> opening SSH 
> (port 22).  I suspect all traffic should be allowed when APF 
> indicates 
> it's offline.
> 
> The "default (ingress) input drop" message suggests that, at 
> that point in 
> time, everything but SSH and ping (and a telnet straight to port 22) 
> should fail.  I'm not sure what the significance is of the 
> skip-backwards-1-hour phenomenon (notice this is visible in 
> both Apache 
> and APF logs) -- perhaps the server is updating the system 
> time from a 
> timeserver?  Is this seen in your system logs as well?  Any 
> clues from an 
> NTP daemon in any logs?
> 
> How often do you have to reboot the system?
> 
> --Sil
> 
> 
> 
> from Tony Spencer:
> .:
> .:Yes.  Much thanks in advance for taking a look.
> .:
> .:Dec 01 13:00:00 server1 apf(10221): flushing & zeroing 
> chain policies
> .:Dec 01 13:00:00 server1 apf(10221): firewall offline
> .:Dec 01 13:05:00 server1 apf(10515): flushing & zeroing 
> chain policies
> .:Dec 01 13:05:00 server1 apf(10515): firewall offline
> .:Dec 01 13:10:00 server1 apf(10741): flushing & zeroing 
> chain policies
> .:Dec 01 13:10:00 server1 apf(10741): firewall offline
> .:Dec 01 13:15:01 server1 apf(10991): flushing & zeroing 
> chain policies
> .:Dec 01 13:15:01 server1 apf(10991): firewall offline
> .:Dec 01 13:20:00 server1 apf(11281): flushing & zeroing 
> chain policies
> .:Dec 01 13:20:00 server1 apf(11281): firewall offline
> .:Dec 01 13:25:00 server1 apf(11516): flushing & zeroing 
> chain policies
> .:Dec 01 13:25:00 server1 apf(11516): firewall offline
> .:Dec 01 13:30:00 server1 apf(11805): flushing & zeroing 
> chain policies
> .:Dec 01 13:30:00 server1 apf(11805): firewall offline
> .:Dec 01 14:38:24 server1 apf(1868): activating firewall
> .:Dec 01 14:38:25 server1 apf(1904): determined eth0 has 
> address xx.xx.xx.xx
> .:Dec 01 14:38:25 server1 apf(1904): loading sysctl.rules
> .:Dec 01 14:38:25 server1 apf(1904): loading preroute.rules
> .:Dec 01 14:38:26 server1 apf(1904): loading prelog.rules
> .:Dec 01 14:38:26 server1 apf(1904): loading bt.rules
> .:Dec 01 14:38:26 server1 apf(1904): loading 192.168.234.235.rules
> .:Dec 01 14:38:26 server1 apf(1904): opening inbound tcp port 22 on
> .:192.168.234.235
> .:Dec 01 14:38:26 server1 apf(1904): opening inbound icmp type 3 on
> .:192.168.234.235
> .:Dec 01 14:38:26 server1 apf(1904): opening inbound icmp type 5 on
> .:192.168.234.235
> .:Dec 01 14:38:26 server1 apf(1904): opening inbound icmp type 11 on
> .:192.168.234.235
> .:Dec 01 14:38:26 server1 apf(1904): opening inbound icmp type 0 on
> .:192.168.234.235
> .:Dec 01 14:38:26 server1 apf(1904): opening inbound icmp type 30 on
> .:192.168.234.235
> .:Dec 01 14:38:26 server1 apf(1904): opening inbound icmp type 8 on
> .:192.168.234.235
> .:Dec 01 14:38:26 server1 apf(1904): loading main.rules
> .:Dec 01 14:38:26 server1 apf(1904): opening inbound tcp port 22 on
> .:xx.xx.xx.xx
> .:Dec 01 14:38:26 server1 apf(1904): opening inbound icmp type 3 on
> .:xx.xx.xx.xx
> .:Dec 01 14:38:26 server1 apf(1904): opening inbound icmp type 5 on
> .:xx.xx.xx.xx
> .:Dec 01 14:38:26 server1 apf(1904): opening inbound icmp type 11 on
> .:xx.xx.xx.xx
> .:Dec 01 14:38:26 server1 apf(1904): opening inbound icmp type 0 on
> .:xx.xx.xx.xx
> .:Dec 01 14:38:26 server1 apf(1904): opening inbound icmp type 30 on
> .:xx.xx.xx.xx
> .:Dec 01 14:38:26 server1 apf(1904): opening inbound icmp type 8 on
> .:xx.xx.xx.xx
> .:Dec 01 14:38:26 server1 apf(1904): default (egress) output accept
> .:Dec 01 14:38:26 server1 apf(1904): default (ingress) input drop
> .:Dec 01 14:38:27 server1 apf(1868): firewall initalized
> .:Dec 01 14:38:51 server1 apf(2957): activating firewall
> .:Dec 01 13:35:21 server1 apf(2998): determined eth0 has 
> address xx.xx.xx.xx
> .:Dec 01 13:35:21 server1 apf(2998): loading sysctl.rules
> .:Dec 01 13:35:21 server1 apf(2998): loading preroute.rules
> .:Dec 01 13:35:21 server1 apf(2998): loading prelog.rules
> .:Dec 01 13:35:21 server1 apf(2998): loading bt.rules
> .:Dec 01 13:35:21 server1 apf(2998): loading 192.168.234.235.rules
> .:Dec 01 13:35:21 server1 apf(2998): opening inbound tcp port 22 on
> .:192.168.234.235
> .:Dec 01 13:35:21 server1 apf(2998): opening inbound icmp type 3 on
> .:192.168.234.235
> .:Dec 01 13:35:21 server1 apf(2998): opening inbound icmp type 5 on
> .:192.168.234.235
> .:Dec 01 13:35:21 server1 apf(2998): opening inbound icmp type 11 on
> .:192.168.234.235
> .:Dec 01 13:35:21 server1 apf(2998): opening inbound icmp type 0 on
> .:192.168.234.235
> .:Dec 01 13:35:21 server1 apf(2998): opening inbound icmp type 30 on
> .:192.168.234.235
> .:Dec 01 13:35:21 server1 apf(2998): opening inbound icmp type 8 on
> .:192.168.234.235
> .:Dec 01 13:35:21 server1 apf(2998): loading main.rules
> .:Dec 01 13:35:21 server1 apf(2998): opening inbound tcp port 22 on
> .:xx.xx.xx.xx
> .:Dec 01 13:35:21 server1 apf(2998): opening inbound icmp type 3 on
> .:xx.xx.xx.xx
> .:Dec 01 13:35:21 server1 apf(2998): opening inbound icmp type 5 on
> .:xx.xx.xx.xx
> .:Dec 01 13:35:21 server1 apf(2998): opening inbound icmp type 11 on
> .:xx.xx.xx.xx
> .:Dec 01 13:35:21 server1 apf(2998): opening inbound icmp type 0 on
> .:xx.xx.xx.xx
> .:Dec 01 13:35:21 server1 apf(2998): opening inbound icmp type 30 on
> .:xx.xx.xx.xx
> .:Dec 01 13:35:21 server1 apf(2998): opening inbound icmp type 8 on
> .:xx.xx.xx.xx
> .:Dec 01 13:35:21 server1 apf(2998): default (egress) output accept
> .:Dec 01 13:35:21 server1 apf(2998): default (ingress) input drop
> .:Dec 01 13:35:22 server1 apf(2957): firewall initalized
> .:Dec 01 13:40:00 server1 apf(3582): flushing & zeroing chain policies
> .:Dec 01 13:40:00 server1 apf(3582): firewall offline
> .:Dec 01 13:45:00 server1 apf(3969): flushing & zeroing chain policies
> .:Dec 01 13:45:00 server1 apf(3969): firewall offline 
> .:
> .:> -----Original Message-----
> .:> From: Sil Greene [mailto:quack AT ibiblio.org] 
> .:> Sent: Wednesday, December 01, 2004 3:12 PM
> .:> To: Internetworkers: http://www.ibiblio.org/internetworkers/
> .:> Subject: Re: [internetworkers] Server Disappears Sporadically
> .:> 
> .:> 
> .:> Tony, do you also have the firewall logs from that time period?
> .:> --s
> .:> 
> .:> 
> .:> 
> .:> 
> .:> from Tony Spencer:
> .:> .:
> .:> .:I lease a server running RedHat Linux.  Recently it has 
> .:> started disappearing .:for no apparent reason.  When I say 
> .:> disappearing I mean that there is no .:response on port 80, I 
> .:> can't FTP to it, I can't SSH2, and I can't even ping .:it.  I 
> .:> have to request my ISP to reboot it.  It always comes back up 
> .:> with no .:problem after a reboot.  I'm not a sysadmin and  
> .:> I've usually managed fine .:but this one is boggling me.  Can 
> .:> anyone give me any more pointers for what .:to look for?  I'm 
> .:> wondering if the firewall is closing everything for some .:reason.
> .:> .:
> .:> .:Here is the situation:
> .:> .:RedHat v3.1.0
> .:> .:2 GB of RAM
> .:> .:Running two websites.  Load rarely tops 0.2 and usually is 
> .:> 0 to 0.1 I ran .:chkrootkit as well as Rootkit Hunter and 
> .:> found no problems.
> .:> .:Telnet is off.  
> .:> .:Runs APF firewall with the following ports open:
> .:> .:TCP_CPORTS="
> .:> 
> .:21,22,25,26,53,80,110,143,443,2082,2083,2086,2087,2095,2096,19638" 
> .:> .:UDP_CPORTS="53"
> .:> .:
> .:> .:Here are logs just before and after the disappearing act.  
> .:> I think the .:proftpd stuff is just a monitor on the box 
> .:> checking whether or not FTP is .:working.  I don't understand 
> .:> why the apache logs actually go backwards in .:time at one 
> .:> point (14:38:37 to 13:35:21).
> .:> .:
> .:> .:-----------------------------------------/var/log/messages--
> .:> ----------------
> .:> .:----------------
> .:> .:Dec  1 11:48:47 server1 proftpd[3892]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP session closed. 
> .:> .:Dec  1 11:57:07 server1 proftpd[4500]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP no transfer timeout, 
> .:> disconnected Dec  1
> .:> .:11:57:07 server1 proftpd[4500]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP session closed. 
> .:> .:Dec  1 12:05:28 server1 proftpd[5466]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP no transfer timeout, 
> .:> disconnected Dec  1
> .:> .:12:05:28 server1 proftpd[5466]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP session closed. 
> .:> .:Dec  1 12:13:48 server1 proftpd[6336]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP no transfer timeout, 
> .:> disconnected Dec  1
> .:> .:12:13:48 server1 proftpd[6336]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP session closed. 
> .:> .:Dec  1 12:22:08 server1 proftpd[7163]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP no transfer timeout, 
> .:> disconnected Dec  1
> .:> .:12:22:08 server1 proftpd[7163]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP session closed. 
> .:> .:Dec  1 12:30:28 server1 proftpd[8289]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP no transfer timeout, 
> .:> disconnected Dec  1
> .:> .:12:30:28 server1 proftpd[8289]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP session closed. 
> .:> .:Dec  1 12:38:48 server1 proftpd[8787]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP no transfer timeout, 
> .:> disconnected Dec  1
> .:> .:12:38:48 server1 proftpd[8787]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP session closed. 
> .:> .:Dec  1 12:47:08 server1 proftpd[9280]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP no transfer timeout, 
> .:> disconnected Dec  1
> .:> .:12:47:09 server1 proftpd[9280]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP session closed. 
> .:> .:Dec  1 12:55:29 server1 proftpd[9713]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP no transfer timeout, 
> .:> disconnected Dec  1
> .:> .:12:55:29 server1 proftpd[9713]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP session closed. 
> .:> .:Dec  1 13:03:49 server1 proftpd[10167]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP no transfer timeout, 
> .:> disconnected Dec  1
> .:> .:13:03:49 server1 proftpd[10167]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP session closed. 
> .:> .:Dec  1 13:12:09 server1 proftpd[10635]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP no transfer timeout, 
> .:> disconnected Dec  1
> .:> .:13:12:09 server1 proftpd[10635]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP session closed. 
> .:> .:Dec  1 13:15:29 server1 proftpd[11067]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP session opened. 
> .:> .:Dec  1 13:15:29 server1 proftpd[11067]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP session closed. 
> .:> .:Dec  1 13:28:49 server1 proftpd[11484]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP no transfer timeout, 
> .:> disconnected Dec  1 .:13:28:50 server1 proftpd[11484]: 
> .:> server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP session closed. 
> .:> .:Dec  1 14:38:13 server1 syslogd 1.4.1: restart.
> .:> .:
> .:> .:--------------------------------Apache 
> .:> error_log---------------------------
> .:> .:[Wed Dec  1 13:20:14 2004] [error] [client 65.33.122.230] 
> .:> File does not
> .:> .:exist: /home/myuser/public_html/images/trans.gif
> .:> .:[Wed Dec  1 13:20:18 2004] [error] [client 66.243.50.46] 
> .:> File does not
> .:> .:exist: /home/myuser/public_html/images/trans.gif
> .:> .:[Wed Dec  1 13:23:02 2004] [error] [client 66.206.96.196] 
> .:> File does not
> .:> .:exist: /home/myuser/public_html/images/trans.gif
> .:> .:[Wed Dec  1 13:24:24 2004] [error] [client 209.30.109.245] 
> .:> File does not
> .:> .:exist: /home/myuser/public_html/images/trans.gif
> .:> .:[Wed Dec  1 13:25:04 2004] [error] [client 24.119.238.68] 
> .:> File does not
> .:> .:exist: /home/myuser/public_html/images/trans.gif
> .:> .:[Wed Dec  1 13:26:26 2004] [error] [client 66.139.27.254] 
> .:> File does not
> .:> .:exist: /home/myuser/public_html/images/trans.gif
> .:> .:[Wed Dec  1 14:38:37 2004] [warn] pid file 
> .:> /usr/local/apache/logs/httpd.pid .:overwritten -- Unclean 
> .:> shutdown of previous Apache run?
> .:> .:[Wed Dec  1 14:38:37 2004] [crit] (98)Address already in 
> .:> use: make_sock:
> .:> .:could not bind to port 443
> .:> .:[Wed Dec  1 14:38:37 2004] [notice] Apache/1.3.29 (Unix) 
> .:> Resin/2.1.12
> .:> .:mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 P
> .:> .:HP/4.3.3 FrontPage/5.0.2.2634 mod_ssl/2.8.16 OpenSSL/0.9.7a 
> .:> configured -- .:resuming normal operations [Wed Dec  1 
> .:> 14:38:37 2004] [notice] suEXEC .:mechanism enabled (wrapper:
> .:> .:/usr/local/apache/bin/suexec)
> .:> .:[Wed Dec  1 14:38:37 2004] [notice] Accept mutex: sysvsem 
> .:> (Default: sysvsem) .:[Wed Dec  1 13:35:21 2004] [notice] 
> .:> SIGUSR1 received.  Doing graceful .:restart [Wed Dec  1 
> .:> 13:35:21 2004] [notice] Apache/1.3.29 (Unix)
> .:> .:Resin/2.1.12
> .:> .:mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 P
> .:> .:HP/4.3.3 FrontPage/5.0.2.2634 mod_ssl/2.8.16 OpenSSL/0.9.7a 
> .:> configured -- .:resuming normal operations [Wed Dec  1 
> .:> 13:35:21 2004] [notice] suEXEC .:mechanism enabled (wrapper:
> .:> .:/usr/local/apache/bin/suexec)
> .:> .:[Wed Dec  1 13:35:21 2004] [notice] Accept mutex: sysvsem 
> .:> (Default: sysvsem) .:[Wed Dec  1 13:35:21 2004] [error] 
> .:> [client 12.219.125.81] File does not
> .:> .:exist: /home/myuser/public_html/free-stuff/free-download-ant
> .:> .:
> .:> .:
> .:> .:
> .:> .:---
> .:> .:Come and play at the InterNetWorkers Web site!  
> .:> http://www.ibiblio.org/internetworkers/
> .:> .:You are currently subscribed to InterNetWorkers mailing 
> .:> list .:To unsubscribe visit 
> .:> http://lists.ibiblio.org/mailman/listinfo/internetworkers
> .:> .:
> .:> 
> .:> --
> .:> "If you put your supper dish to your ears you can hear the 
> .:> sounds  of a restaurant."  --Snoopy
> .:> ---
> .:> Come and play at the InterNetWorkers Web site!  
> .:> http://www.ibiblio.org/internetworkers/
> .:> You are currently subscribed to InterNetWorkers mailing list 
> .:> To unsubscribe visit 
> .:> http://lists.ibiblio.org/mailman/listinfo/internetworkers
> .:> 
> .:> 
> .:
> .:---
> .:Come and play at the InterNetWorkers Web site!  
> http://www.ibiblio.org/internetworkers/
> .:You are currently subscribed to InterNetWorkers mailing list
> .:To unsubscribe visit 
> http://lists.ibiblio.org/mailman/listinfo/internetworkers
> .:
> 
> -- 
> "If you put your supper dish to your ears you can hear the sounds
>  of a restaurant."  --Snoopy
> ---
> Come and play at the InterNetWorkers Web site!  
> http://www.ibiblio.org/internetworkers/
> You are currently subscribed to InterNetWorkers mailing list
> To unsubscribe visit 
> http://lists.ibiblio.org/mailman/listinfo/internetworkers
> 
>