Skip to Content.
Sympa Menu

internetworkers - RE: [internetworkers] Server Disappears Sporadically

internetworkers AT lists.ibiblio.org

Subject: Internetworkers: http://www.ibiblio.org/internetworkers/

List archive

Chronological Thread  
  • From: Sil Greene <quack AT ibiblio.org>
  • To: "Internetworkers: http://www.ibiblio.org/internetworkers/" <internetworkers AT lists.ibiblio.org>
  • Subject: RE: [internetworkers] Server Disappears Sporadically
  • Date: Wed, 1 Dec 2004 17:18:53 -0500 (EST)


Tony,

I'm not a firewall expert, and I haven't worked with APF before. With
that said, I'm not seeing the APF logs that indicate it's opening inbound
ports for HTTP, FTP, and other traffic. It does seem to be opening SSH
(port 22). I suspect all traffic should be allowed when APF indicates
it's offline.

The "default (ingress) input drop" message suggests that, at that point in
time, everything but SSH and ping (and a telnet straight to port 22)
should fail. I'm not sure what the significance is of the
skip-backwards-1-hour phenomenon (notice this is visible in both Apache
and APF logs) -- perhaps the server is updating the system time from a
timeserver? Is this seen in your system logs as well? Any clues from an
NTP daemon in any logs?

How often do you have to reboot the system?

--Sil



from Tony Spencer:
.:
.:Yes. Much thanks in advance for taking a look.
.:
.:Dec 01 13:00:00 server1 apf(10221): flushing & zeroing chain policies
.:Dec 01 13:00:00 server1 apf(10221): firewall offline
.:Dec 01 13:05:00 server1 apf(10515): flushing & zeroing chain policies
.:Dec 01 13:05:00 server1 apf(10515): firewall offline
.:Dec 01 13:10:00 server1 apf(10741): flushing & zeroing chain policies
.:Dec 01 13:10:00 server1 apf(10741): firewall offline
.:Dec 01 13:15:01 server1 apf(10991): flushing & zeroing chain policies
.:Dec 01 13:15:01 server1 apf(10991): firewall offline
.:Dec 01 13:20:00 server1 apf(11281): flushing & zeroing chain policies
.:Dec 01 13:20:00 server1 apf(11281): firewall offline
.:Dec 01 13:25:00 server1 apf(11516): flushing & zeroing chain policies
.:Dec 01 13:25:00 server1 apf(11516): firewall offline
.:Dec 01 13:30:00 server1 apf(11805): flushing & zeroing chain policies
.:Dec 01 13:30:00 server1 apf(11805): firewall offline
.:Dec 01 14:38:24 server1 apf(1868): activating firewall
.:Dec 01 14:38:25 server1 apf(1904): determined eth0 has address xx.xx.xx.xx
.:Dec 01 14:38:25 server1 apf(1904): loading sysctl.rules
.:Dec 01 14:38:25 server1 apf(1904): loading preroute.rules
.:Dec 01 14:38:26 server1 apf(1904): loading prelog.rules
.:Dec 01 14:38:26 server1 apf(1904): loading bt.rules
.:Dec 01 14:38:26 server1 apf(1904): loading 192.168.234.235.rules
.:Dec 01 14:38:26 server1 apf(1904): opening inbound tcp port 22 on
.:192.168.234.235
.:Dec 01 14:38:26 server1 apf(1904): opening inbound icmp type 3 on
.:192.168.234.235
.:Dec 01 14:38:26 server1 apf(1904): opening inbound icmp type 5 on
.:192.168.234.235
.:Dec 01 14:38:26 server1 apf(1904): opening inbound icmp type 11 on
.:192.168.234.235
.:Dec 01 14:38:26 server1 apf(1904): opening inbound icmp type 0 on
.:192.168.234.235
.:Dec 01 14:38:26 server1 apf(1904): opening inbound icmp type 30 on
.:192.168.234.235
.:Dec 01 14:38:26 server1 apf(1904): opening inbound icmp type 8 on
.:192.168.234.235
.:Dec 01 14:38:26 server1 apf(1904): loading main.rules
.:Dec 01 14:38:26 server1 apf(1904): opening inbound tcp port 22 on
.:xx.xx.xx.xx
.:Dec 01 14:38:26 server1 apf(1904): opening inbound icmp type 3 on
.:xx.xx.xx.xx
.:Dec 01 14:38:26 server1 apf(1904): opening inbound icmp type 5 on
.:xx.xx.xx.xx
.:Dec 01 14:38:26 server1 apf(1904): opening inbound icmp type 11 on
.:xx.xx.xx.xx
.:Dec 01 14:38:26 server1 apf(1904): opening inbound icmp type 0 on
.:xx.xx.xx.xx
.:Dec 01 14:38:26 server1 apf(1904): opening inbound icmp type 30 on
.:xx.xx.xx.xx
.:Dec 01 14:38:26 server1 apf(1904): opening inbound icmp type 8 on
.:xx.xx.xx.xx
.:Dec 01 14:38:26 server1 apf(1904): default (egress) output accept
.:Dec 01 14:38:26 server1 apf(1904): default (ingress) input drop
.:Dec 01 14:38:27 server1 apf(1868): firewall initalized
.:Dec 01 14:38:51 server1 apf(2957): activating firewall
.:Dec 01 13:35:21 server1 apf(2998): determined eth0 has address xx.xx.xx.xx
.:Dec 01 13:35:21 server1 apf(2998): loading sysctl.rules
.:Dec 01 13:35:21 server1 apf(2998): loading preroute.rules
.:Dec 01 13:35:21 server1 apf(2998): loading prelog.rules
.:Dec 01 13:35:21 server1 apf(2998): loading bt.rules
.:Dec 01 13:35:21 server1 apf(2998): loading 192.168.234.235.rules
.:Dec 01 13:35:21 server1 apf(2998): opening inbound tcp port 22 on
.:192.168.234.235
.:Dec 01 13:35:21 server1 apf(2998): opening inbound icmp type 3 on
.:192.168.234.235
.:Dec 01 13:35:21 server1 apf(2998): opening inbound icmp type 5 on
.:192.168.234.235
.:Dec 01 13:35:21 server1 apf(2998): opening inbound icmp type 11 on
.:192.168.234.235
.:Dec 01 13:35:21 server1 apf(2998): opening inbound icmp type 0 on
.:192.168.234.235
.:Dec 01 13:35:21 server1 apf(2998): opening inbound icmp type 30 on
.:192.168.234.235
.:Dec 01 13:35:21 server1 apf(2998): opening inbound icmp type 8 on
.:192.168.234.235
.:Dec 01 13:35:21 server1 apf(2998): loading main.rules
.:Dec 01 13:35:21 server1 apf(2998): opening inbound tcp port 22 on
.:xx.xx.xx.xx
.:Dec 01 13:35:21 server1 apf(2998): opening inbound icmp type 3 on
.:xx.xx.xx.xx
.:Dec 01 13:35:21 server1 apf(2998): opening inbound icmp type 5 on
.:xx.xx.xx.xx
.:Dec 01 13:35:21 server1 apf(2998): opening inbound icmp type 11 on
.:xx.xx.xx.xx
.:Dec 01 13:35:21 server1 apf(2998): opening inbound icmp type 0 on
.:xx.xx.xx.xx
.:Dec 01 13:35:21 server1 apf(2998): opening inbound icmp type 30 on
.:xx.xx.xx.xx
.:Dec 01 13:35:21 server1 apf(2998): opening inbound icmp type 8 on
.:xx.xx.xx.xx
.:Dec 01 13:35:21 server1 apf(2998): default (egress) output accept
.:Dec 01 13:35:21 server1 apf(2998): default (ingress) input drop
.:Dec 01 13:35:22 server1 apf(2957): firewall initalized
.:Dec 01 13:40:00 server1 apf(3582): flushing & zeroing chain policies
.:Dec 01 13:40:00 server1 apf(3582): firewall offline
.:Dec 01 13:45:00 server1 apf(3969): flushing & zeroing chain policies
.:Dec 01 13:45:00 server1 apf(3969): firewall offline
.:
.:> -----Original Message-----
.:> From: Sil Greene [mailto:quack AT ibiblio.org]
.:> Sent: Wednesday, December 01, 2004 3:12 PM
.:> To: Internetworkers: http://www.ibiblio.org/internetworkers/
.:> Subject: Re: [internetworkers] Server Disappears Sporadically
.:>
.:>
.:> Tony, do you also have the firewall logs from that time period?
.:> --s
.:>
.:>
.:>
.:>
.:> from Tony Spencer:
.:> .:
.:> .:I lease a server running RedHat Linux. Recently it has
.:> started disappearing .:for no apparent reason. When I say
.:> disappearing I mean that there is no .:response on port 80, I
.:> can't FTP to it, I can't SSH2, and I can't even ping .:it. I
.:> have to request my ISP to reboot it. It always comes back up
.:> with no .:problem after a reboot. I'm not a sysadmin and
.:> I've usually managed fine .:but this one is boggling me. Can
.:> anyone give me any more pointers for what .:to look for? I'm
.:> wondering if the firewall is closing everything for some .:reason.
.:> .:
.:> .:Here is the situation:
.:> .:RedHat v3.1.0
.:> .:2 GB of RAM
.:> .:Running two websites. Load rarely tops 0.2 and usually is
.:> 0 to 0.1 I ran .:chkrootkit as well as Rootkit Hunter and
.:> found no problems.
.:> .:Telnet is off.
.:> .:Runs APF firewall with the following ports open:
.:> .:TCP_CPORTS="
.:> .:21,22,25,26,53,80,110,143,443,2082,2083,2086,2087,2095,2096,19638"
.:> .:UDP_CPORTS="53"
.:> .:
.:> .:Here are logs just before and after the disappearing act.
.:> I think the .:proftpd stuff is just a monitor on the box
.:> checking whether or not FTP is .:working. I don't understand
.:> why the apache logs actually go backwards in .:time at one
.:> point (14:38:37 to 13:35:21).
.:> .:
.:> .:-----------------------------------------/var/log/messages--
.:> ----------------
.:> .:----------------
.:> .:Dec 1 11:48:47 server1 proftpd[3892]: server1.mydomain.com
.:> .:(localhost[127.0.0.1]) - FTP session closed.
.:> .:Dec 1 11:57:07 server1 proftpd[4500]: server1.mydomain.com
.:> .:(localhost[127.0.0.1]) - FTP no transfer timeout,
.:> disconnected Dec 1
.:> .:11:57:07 server1 proftpd[4500]: server1.mydomain.com
.:> .:(localhost[127.0.0.1]) - FTP session closed.
.:> .:Dec 1 12:05:28 server1 proftpd[5466]: server1.mydomain.com
.:> .:(localhost[127.0.0.1]) - FTP no transfer timeout,
.:> disconnected Dec 1
.:> .:12:05:28 server1 proftpd[5466]: server1.mydomain.com
.:> .:(localhost[127.0.0.1]) - FTP session closed.
.:> .:Dec 1 12:13:48 server1 proftpd[6336]: server1.mydomain.com
.:> .:(localhost[127.0.0.1]) - FTP no transfer timeout,
.:> disconnected Dec 1
.:> .:12:13:48 server1 proftpd[6336]: server1.mydomain.com
.:> .:(localhost[127.0.0.1]) - FTP session closed.
.:> .:Dec 1 12:22:08 server1 proftpd[7163]: server1.mydomain.com
.:> .:(localhost[127.0.0.1]) - FTP no transfer timeout,
.:> disconnected Dec 1
.:> .:12:22:08 server1 proftpd[7163]: server1.mydomain.com
.:> .:(localhost[127.0.0.1]) - FTP session closed.
.:> .:Dec 1 12:30:28 server1 proftpd[8289]: server1.mydomain.com
.:> .:(localhost[127.0.0.1]) - FTP no transfer timeout,
.:> disconnected Dec 1
.:> .:12:30:28 server1 proftpd[8289]: server1.mydomain.com
.:> .:(localhost[127.0.0.1]) - FTP session closed.
.:> .:Dec 1 12:38:48 server1 proftpd[8787]: server1.mydomain.com
.:> .:(localhost[127.0.0.1]) - FTP no transfer timeout,
.:> disconnected Dec 1
.:> .:12:38:48 server1 proftpd[8787]: server1.mydomain.com
.:> .:(localhost[127.0.0.1]) - FTP session closed.
.:> .:Dec 1 12:47:08 server1 proftpd[9280]: server1.mydomain.com
.:> .:(localhost[127.0.0.1]) - FTP no transfer timeout,
.:> disconnected Dec 1
.:> .:12:47:09 server1 proftpd[9280]: server1.mydomain.com
.:> .:(localhost[127.0.0.1]) - FTP session closed.
.:> .:Dec 1 12:55:29 server1 proftpd[9713]: server1.mydomain.com
.:> .:(localhost[127.0.0.1]) - FTP no transfer timeout,
.:> disconnected Dec 1
.:> .:12:55:29 server1 proftpd[9713]: server1.mydomain.com
.:> .:(localhost[127.0.0.1]) - FTP session closed.
.:> .:Dec 1 13:03:49 server1 proftpd[10167]: server1.mydomain.com
.:> .:(localhost[127.0.0.1]) - FTP no transfer timeout,
.:> disconnected Dec 1
.:> .:13:03:49 server1 proftpd[10167]: server1.mydomain.com
.:> .:(localhost[127.0.0.1]) - FTP session closed.
.:> .:Dec 1 13:12:09 server1 proftpd[10635]: server1.mydomain.com
.:> .:(localhost[127.0.0.1]) - FTP no transfer timeout,
.:> disconnected Dec 1
.:> .:13:12:09 server1 proftpd[10635]: server1.mydomain.com
.:> .:(localhost[127.0.0.1]) - FTP session closed.
.:> .:Dec 1 13:15:29 server1 proftpd[11067]: server1.mydomain.com
.:> .:(localhost[127.0.0.1]) - FTP session opened.
.:> .:Dec 1 13:15:29 server1 proftpd[11067]: server1.mydomain.com
.:> .:(localhost[127.0.0.1]) - FTP session closed.
.:> .:Dec 1 13:28:49 server1 proftpd[11484]: server1.mydomain.com
.:> .:(localhost[127.0.0.1]) - FTP no transfer timeout,
.:> disconnected Dec 1 .:13:28:50 server1 proftpd[11484]:
.:> server1.mydomain.com
.:> .:(localhost[127.0.0.1]) - FTP session closed.
.:> .:Dec 1 14:38:13 server1 syslogd 1.4.1: restart.
.:> .:
.:> .:--------------------------------Apache
.:> error_log---------------------------
.:> .:[Wed Dec 1 13:20:14 2004] [error] [client 65.33.122.230]
.:> File does not
.:> .:exist: /home/myuser/public_html/images/trans.gif
.:> .:[Wed Dec 1 13:20:18 2004] [error] [client 66.243.50.46]
.:> File does not
.:> .:exist: /home/myuser/public_html/images/trans.gif
.:> .:[Wed Dec 1 13:23:02 2004] [error] [client 66.206.96.196]
.:> File does not
.:> .:exist: /home/myuser/public_html/images/trans.gif
.:> .:[Wed Dec 1 13:24:24 2004] [error] [client 209.30.109.245]
.:> File does not
.:> .:exist: /home/myuser/public_html/images/trans.gif
.:> .:[Wed Dec 1 13:25:04 2004] [error] [client 24.119.238.68]
.:> File does not
.:> .:exist: /home/myuser/public_html/images/trans.gif
.:> .:[Wed Dec 1 13:26:26 2004] [error] [client 66.139.27.254]
.:> File does not
.:> .:exist: /home/myuser/public_html/images/trans.gif
.:> .:[Wed Dec 1 14:38:37 2004] [warn] pid file
.:> /usr/local/apache/logs/httpd.pid .:overwritten -- Unclean
.:> shutdown of previous Apache run?
.:> .:[Wed Dec 1 14:38:37 2004] [crit] (98)Address already in
.:> use: make_sock:
.:> .:could not bind to port 443
.:> .:[Wed Dec 1 14:38:37 2004] [notice] Apache/1.3.29 (Unix)
.:> Resin/2.1.12
.:> .:mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 P
.:> .:HP/4.3.3 FrontPage/5.0.2.2634 mod_ssl/2.8.16 OpenSSL/0.9.7a
.:> configured -- .:resuming normal operations [Wed Dec 1
.:> 14:38:37 2004] [notice] suEXEC .:mechanism enabled (wrapper:
.:> .:/usr/local/apache/bin/suexec)
.:> .:[Wed Dec 1 14:38:37 2004] [notice] Accept mutex: sysvsem
.:> (Default: sysvsem) .:[Wed Dec 1 13:35:21 2004] [notice]
.:> SIGUSR1 received. Doing graceful .:restart [Wed Dec 1
.:> 13:35:21 2004] [notice] Apache/1.3.29 (Unix)
.:> .:Resin/2.1.12
.:> .:mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 P
.:> .:HP/4.3.3 FrontPage/5.0.2.2634 mod_ssl/2.8.16 OpenSSL/0.9.7a
.:> configured -- .:resuming normal operations [Wed Dec 1
.:> 13:35:21 2004] [notice] suEXEC .:mechanism enabled (wrapper:
.:> .:/usr/local/apache/bin/suexec)
.:> .:[Wed Dec 1 13:35:21 2004] [notice] Accept mutex: sysvsem
.:> (Default: sysvsem) .:[Wed Dec 1 13:35:21 2004] [error]
.:> [client 12.219.125.81] File does not
.:> .:exist: /home/myuser/public_html/free-stuff/free-download-ant
.:> .:
.:> .:
.:> .:
.:> .:---
.:> .:Come and play at the InterNetWorkers Web site!
.:> http://www.ibiblio.org/internetworkers/
.:> .:You are currently subscribed to InterNetWorkers mailing
.:> list .:To unsubscribe visit
.:> http://lists.ibiblio.org/mailman/listinfo/internetworkers
.:> .:
.:>
.:> --
.:> "If you put your supper dish to your ears you can hear the
.:> sounds of a restaurant." --Snoopy
.:> ---
.:> Come and play at the InterNetWorkers Web site!
.:> http://www.ibiblio.org/internetworkers/
.:> You are currently subscribed to InterNetWorkers mailing list
.:> To unsubscribe visit
.:> http://lists.ibiblio.org/mailman/listinfo/internetworkers
.:>
.:>
.:
.:---
.:Come and play at the InterNetWorkers Web site!
http://www.ibiblio.org/internetworkers/
.:You are currently subscribed to InterNetWorkers mailing list
.:To unsubscribe visit
http://lists.ibiblio.org/mailman/listinfo/internetworkers
.:

--
"If you put your supper dish to your ears you can hear the sounds
of a restaurant." --Snoopy




Archive powered by MHonArc 2.6.24.

Top of Page